diff --git a/.agents/skills/ci-failure-analysis/SKILL.md b/.agents/skills/ci-failure-analysis/SKILL.md new file mode 100644 index 00000000000..7a0bf81b0cc --- /dev/null +++ b/.agents/skills/ci-failure-analysis/SKILL.md @@ -0,0 +1,100 @@ +--- +name: ci-failure-analysis +description: Analyze Vortex GitHub Actions CI failures. Use when asked to investigate failed CI runs, failed jobs, or when the user mentions "/ci-failure-analysis". +--- + +# Vortex CI Failure Analysis Skill + +Analyze failed GitHub Actions runs for the Vortex repository and identify whether the failure is +caused by the PR, pre-existing flakiness, infrastructure, or an unrelated main-branch issue. + +## Inputs + +Use any PR number, repository, run ID, failed job metadata, or log snippets supplied by the user or +automation prompt. If a needed value is missing, discover it with the narrowest `gh` command that +can answer the question. + +## Workflow + +1. List failed jobs for the workflow run: + + ```bash + gh run view --repo --json jobs + ``` + +2. Fetch only failed job logs first: + + ```bash + gh run view --repo --job --log-failed + ``` + + If that fails, use the Actions API: + + ```bash + gh api repos//actions/jobs//logs + ``` + +3. If any `gh` command fails with `error connecting to api.github.com` in a sandbox, rerun it with + escalated network permissions immediately. + +4. Classify each failure: + + - Rust build errors: compiler diagnostics, spans, trait bound failures, feature-gate issues. + - Rust test failures: failing test name, panic/assertion output, expected vs actual values, + source path and line. + - Clippy failures: lint name, file path, line, and suggested fix if shown. + - Formatting or public API failures: changed files and commands needed to regenerate output. + - Python/docs failures: pytest, maturin, Sphinx, doctest, or packaging output. + - Infrastructure failures: toolchain download, cache, runner, network, disk, timeout, or + service issues. + +5. Fetch the PR diff and metadata only after the failing log section is understood: + + ```bash + gh pr view --repo --json title,body,baseRefName,headRefName,files,commits + gh pr diff --repo + ``` + +6. Reproduce narrowly when practical: + + ```bash + cargo test -p + cargo clippy -p --all-targets --all-features + make -C docs doctest + uv run --all-packages pytest + ``` + +7. Check whether the same failure appears on recent main-branch runs or open issues before calling + it PR-caused. + +## Report Format + +Post or return one concise Markdown report: + +````markdown +## CI Failure Analysis + +### Status + + +### Failed Jobs +- ``: + +### Relevant Log Output +```text + +``` + +### Correlation With PR Changes + + +### Recommended Next Step + +```` + +## Rules + +- Show relevant failure excerpts, not full logs. +- If many tests fail, detail the first few distinct failures and summarize the rest. +- Do not guess. If causation is unclear, say what was checked and what would resolve it. +- Prefer one PR comment or one final report over multiple fragmented updates. diff --git a/.agents/skills/pr-review/SKILL.md b/.agents/skills/pr-review/SKILL.md new file mode 100644 index 00000000000..4c073dd85fe --- /dev/null +++ b/.agents/skills/pr-review/SKILL.md @@ -0,0 +1,93 @@ +--- +name: pr-review +description: Review Vortex pull requests for correctness, Rust soundness, performance, API compatibility, and test coverage. Use when reviewing PRs, reviewing code changes, or when the user mentions "/pr-review". +--- + +# Vortex PR Review Skill + +Review Vortex changes for issues that CI may miss: semantic correctness, Rust soundness, +zero-copy and alignment invariants, performance, API compatibility, and missing regression +coverage. + +## Usage Modes + +### GitHub Actions Mode + +When invoked by a PR automation, the prompt may already include PR metadata, issue body, +comments, and changed files. In this mode, use git commands for the diff and history: + +```bash +git diff origin/...HEAD +git diff --stat origin/...HEAD +git log origin/..HEAD --oneline +``` + +If the base ref is missing, fetch only that ref: + +```bash +git fetch origin --depth=1 +``` + +Do not refetch data already provided in the prompt. Use the supplied comments and metadata as +review context. + +### Local CLI Mode + +When the user provides a PR number or URL, use `gh` to fetch the PR metadata, comments, and diff: + +```bash +gh pr view --json title,body,author,baseRefName,headRefName,files,additions,deletions,commits +gh pr view --json comments,reviews +gh pr diff +``` + +If `gh` cannot connect to `api.github.com` because of sandbox networking, rerun with escalated +network permissions. + +## Review Workflow + +1. Read `AGENTS.md` and any nested `AGENTS.md` for project conventions. +2. Identify the change intent from the PR title, body, commits, and tests. +3. Group changed files by area: arrays, encodings, buffers, file/layout, integrations, bindings, + docs, or CI. +4. Trace changed behavior through callers, trait implementations, dtype/nullability handling, + validity masks, and tests. +5. Focus findings on actionable defects. Avoid commenting on formatting or issues already covered + by clippy, rustfmt, or generated API checks. +6. Scope verification to the change. Rust/API changes need Rust checks; docs-only, agent-only, + symlink-only, and metadata-only changes should use targeted validation such as Markdown review, + `ls`, `find`, `git status`, or relevant config linters. + +## Review Areas + +| Area | Focus | +| --- | --- | +| Correctness | Length and dtype invariants, nullability, validity masks, offset math, canonicalization, boundary conditions, empty arrays, scalar vs array behavior | +| Rust soundness | `unsafe` blocks, aliasing, lifetimes, alignment, FFI boundaries, panic safety, ownership of buffers and arrays | +| Compression and IO | Encoding metadata, statistics, layout evolution, file compatibility, scan projection/filter behavior, async IO edge cases | +| Performance | Unnecessary copies, lost zero-copy behavior, avoidable allocations, poor cache locality, quadratic loops, excessive dynamic dispatch in hot paths | +| Error handling | Correct `vortex_err!` and `vortex_bail!` usage, useful messages, no accidental panics on user data | +| API compatibility | Public API docs, public-api lock updates, feature flags, crate boundaries, Python/Java binding impacts | +| Tests | Regression coverage, edge cases, parameterized cases with `rstest`, use of `assert_arrays_eq!`, docs doctests when docs change | +| Verification scope | Avoid requesting or running expensive workspace checks when the PR only changes docs, agent files, symlinks, or metadata | + +## Output + +Lead with findings, ordered by severity. For each finding include: + +- File and line reference. +- Why the issue is a real bug or material risk. +- A concrete fix or verification path when possible. + +Use inline review comments when the environment supports them and a precise changed line is the +best place for the feedback. Keep broad design feedback in the summary. + +If no issues are found, say so explicitly and mention any residual risk or tests not run. + +## Principles + +- Review the code that changed, but inspect enough surrounding code to validate invariants. +- Do not infer causation from commit messages alone. Verify with code, tests, or logs. +- Do not ask for broad rewrites when a narrow fix would address the risk. +- Do not downgrade a correctness or soundness issue to a nit because it is inconvenient. +- Be specific and proportionate. diff --git a/.agents/skills/query/SKILL.md b/.agents/skills/query/SKILL.md new file mode 100644 index 00000000000..2a07f3ef312 --- /dev/null +++ b/.agents/skills/query/SKILL.md @@ -0,0 +1,36 @@ +--- +name: query +description: Answer questions about the Vortex codebase or pull requests. Use when asked a question via "/query" or when the user wants to understand code, architecture, behavior, or implementation details. +--- + +# Vortex Query Skill + +Answer questions about the Vortex project, its pull requests, and its implementation. + +## Key Context + +- Vortex is a Rust workspace for columnar arrays, compression encodings, file IO, and scan + integrations. +- `vortex-array` defines the core array traits, dtype system, canonical arrays, and base + encodings. +- `vortex-buffer` owns aligned zero-copy buffers. +- `vortex-file` and `vortex-layout` implement file and layout reading. +- `encodings/*` contains specialized compressed encodings. +- Python, Java, DuckDB, and DataFusion integrations live in their own workspace areas. + +## Workflow + +1. Read `AGENTS.md` and any closer scoped `AGENTS.md` before relying on conventions. +2. Use `rg` and targeted file reads to identify the relevant crate, module, and tests. +3. If the question is about a PR, inspect the diff and comments before answering. +4. If the question is about behavior, trace the implementation through public entry points, + encoding-specific implementations, and tests. +5. Answer with concrete file paths and line numbers when they help. + +## Answering Guidelines + +- Separate confirmed facts from inference. +- Prefer precise code references over broad descriptions. +- Mention important uncertainty and describe what would verify it. +- Do not invent architecture. If the repository does not answer the question, say what you + checked and what is still missing. diff --git a/.cargo/config.toml b/.cargo/config.toml index d5b9f580321..466422a80b1 100644 --- a/.cargo/config.toml +++ b/.cargo/config.toml @@ -3,9 +3,6 @@ rustflags = [ "-C", "force-frame-pointers=yes", ] -[target.wasm32-unknown-unknown] -rustflags = ['--cfg', 'getrandom_backend="wasm_js"', '-C', 'target-feature=+atomics'] - [alias] xtask = "run -p xtask --" vx = "run -p vortex-tui --" diff --git a/.claude b/.claude new file mode 120000 index 00000000000..c0ca4685663 --- /dev/null +++ b/.claude @@ -0,0 +1 @@ +.agents \ No newline at end of file diff --git a/.github/AGENTS.md b/.github/AGENTS.md index 5d43ab24300..dd85b76afdd 100644 --- a/.github/AGENTS.md +++ b/.github/AGENTS.md @@ -5,7 +5,6 @@ Nightly is required for: - `-Z` flags: sanitizers (`-Zsanitizer=address`), miri (`-Zmiri-*`), publish (`-Zpublish-timeout`) - `cargo-fuzz` (requires nightly) - `public-api` xtask (nightly rustdoc JSON) -- `--cfg vortex_nightly` (enables `portable_simd` feature gate) Everything else (build, clippy, tests, docs, benchmarks, packaging) should use stable. diff --git a/.github/ISSUE_TEMPLATE/epic.md b/.github/ISSUE_TEMPLATE/epic.md new file mode 100644 index 00000000000..7e9a32ab454 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/epic.md @@ -0,0 +1,77 @@ +--- +name: Epic +about: Umbrella issue for a major initiative tracked via sub-issues. +title: "Epic: " +labels: epic +--- + + + +This Epic is for ... + + + +## Status + + + +**Proposed.** + +## Goal + + + +## Motivation + + + +## Unresolved questions + + + +- [ ] None yet. diff --git a/.github/ISSUE_TEMPLATE/tracking_issue.md b/.github/ISSUE_TEMPLATE/tracking_issue.md new file mode 100644 index 00000000000..a5a8a66d869 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/tracking_issue.md @@ -0,0 +1,98 @@ +--- +name: Tracking Issue +about: A tracking issue for a feature or initiative in Vortex. +title: "Tracking Issue: " +labels: tracking-issue +--- + + + +This is a tracking issue for ... + + + +## Motivation + + + +## Design + + + +## Steps + + + +- [ ] Initial implementation +- [ ] Documentation +- [ ] Public API stabilization + +## Unresolved questions + + + +- [ ] None yet. + +## Implementation history + + diff --git a/.github/actions/alert-incident-io/action.yml b/.github/actions/alert-incident-io/action.yml index 7eb835431dc..e375ffdc5a0 100644 --- a/.github/actions/alert-incident-io/action.yml +++ b/.github/actions/alert-incident-io/action.yml @@ -27,7 +27,7 @@ inputs: runs: using: "composite" steps: - - uses: incident-io/github-action@v1 + - uses: incident-io/github-action@7aa5f85e67679cd8fdf2a19aed3d5450335a004b # v1 with: api-key: ${{ inputs.api-key }} alert-source-id: 01KH4EYTH3HA4PDZPRAPEV1Q10 diff --git a/.github/actions/prepare-java-linux/action.yml b/.github/actions/prepare-java-linux/action.yml deleted file mode 100644 index 06c51d5245a..00000000000 --- a/.github/actions/prepare-java-linux/action.yml +++ /dev/null @@ -1,18 +0,0 @@ -name: "Prepare vortex dependencies for java" -description: "Install required packages for vortex java build" -runs: - using: "composite" - steps: - - shell: bash - run: | - apt update - apt install -y wget curl build-essential unzip clang - - name: Print GLIBC version - shell: bash - run: ldd --version - - name: Verify GLIBC version - shell: bash - run: | - set -eux - - ldd --version | grep 'GLIBC 2.31' diff --git a/.github/actions/setup-rust/action.yml b/.github/actions/setup-rust/action.yml index 5fea9acaa53..4f42d38167a 100644 --- a/.github/actions/setup-rust/action.yml +++ b/.github/actions/setup-rust/action.yml @@ -36,7 +36,7 @@ runs: - name: Install Mold if: runner.os == 'Linux' - uses: rui314/setup-mold@v1 + uses: rui314/setup-mold@9c9c13bf4c3f1adef0cc596abc155580bcb04444 # v1 - name: Check for rustup id: check-rustup @@ -45,7 +45,7 @@ runs: - name: Rust Toolchain id: rust-toolchain - uses: dtolnay/rust-toolchain@stable + uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable if: steps.check-rustup.outputs.exists != 'true' with: toolchain: "${{ steps.toolchain-config.outputs.toolchain }}" @@ -73,7 +73,7 @@ runs: - name: Rust Compile Cache if: inputs.enable-sccache == 'true' - uses: mozilla-actions/sccache-action@v0.0.9 + uses: mozilla-actions/sccache-action@9e7fa8a12102821edf02ca5dbea1acd0f89a2696 # v0.0.10 - name: Pre-start sccache server if: inputs.enable-sccache == 'true' @@ -83,13 +83,3 @@ runs: - name: Install Protoc (for lance-encoding build step) if: runner.os != 'Windows' uses: ./.github/actions/setup-protoc - - - name: Install Sweep - shell: bash - if: ${{ inputs.timestamp == 'true' && github.ref_name == 'develop' }} - run: cargo install cargo-sweep - - - name: Timestamp Cache - shell: bash - if: ${{ inputs.timestamp == 'true' && github.ref_name == 'develop' }} - run: cargo sweep --stamp diff --git a/.github/release-drafter.yml b/.github/release-drafter.yml index c28b97aba4c..1608fb15570 100644 --- a/.github/release-drafter.yml +++ b/.github/release-drafter.yml @@ -4,6 +4,9 @@ categories: - title: "⚠️ Breaks" labels: - "changelog/break" + - title: "🚧 Deprecation" + labels: + - "changelog/deprecation" - title: "✨ Features" labels: - "changelog/feature" diff --git a/.github/runs-on.yml b/.github/runs-on.yml index b1bfde5c961..6a904a6a10d 100644 --- a/.github/runs-on.yml +++ b/.github/runs-on.yml @@ -1 +1,6 @@ _extends: .github-private +images: + ubuntu24-full-arm64-pre: + platform: "linux" + arch: "arm64" + ami: "ami-08f97c6362847dc5d" diff --git a/.github/scripts/close_fixed_fuzzer_issues.py b/.github/scripts/close_fixed_fuzzer_issues.py index ab637e725de..48513a3ceb2 100755 --- a/.github/scripts/close_fixed_fuzzer_issues.py +++ b/.github/scripts/close_fixed_fuzzer_issues.py @@ -162,7 +162,6 @@ def build_fuzz_target(target: str) -> bool: """Build the fuzz target once. Returns True on success.""" print(f"\nBuilding fuzz target: {target}") env = os.environ.copy() - env["RUSTFLAGS"] = "--cfg vortex_nightly" result = run( ["cargo", "+nightly", "fuzz", "build", "--dev", "--sanitizer=none", target], env=env, @@ -173,7 +172,6 @@ def build_fuzz_target(target: str) -> bool: def retest_crash(target: str, crash_path: str, timeout_secs: int = 120) -> str: """Run the fuzz target with the crash file. Returns 'fixed', 'reproduces', or 'timeout'.""" env = os.environ.copy() - env["RUSTFLAGS"] = "--cfg vortex_nightly" try: result = run( [ diff --git a/.github/scripts/run-sql-bench.sh b/.github/scripts/run-sql-bench.sh deleted file mode 100755 index 96c2e2f19dc..00000000000 --- a/.github/scripts/run-sql-bench.sh +++ /dev/null @@ -1,138 +0,0 @@ -#!/bin/bash -# SPDX-License-Identifier: Apache-2.0 -# SPDX-FileCopyrightText: Copyright the Vortex contributors -# -# Runs SQL benchmarks (datafusion-bench, duckdb-bench, lance-bench) for the given targets. -# This script is used by the sql-benchmarks.yml workflow. -# -# Usage: -# run-sql-bench.sh [options] -# -# Arguments: -# subcommand The benchmark subcommand (e.g., tpch, clickbench, tpcds) -# targets Comma-separated list of engine:format pairs -# (e.g., "datafusion:parquet,datafusion:vortex,duckdb:parquet") -# -# Options: -# --scale-factor Scale factor for the benchmark (e.g., 1.0, 10.0) -# --iterations Number of iterations to pass to each benchmark binary -# --remote-storage Remote storage URL (e.g., s3://bucket/path/) -# If provided, runs in remote mode (no lance support). -# --benchmark-id Benchmark ID for error messages (e.g., tpch-s3) - -set -Eeu -o pipefail - -subcommand="$1" -targets="$2" -shift 2 - -scale_factor="" -iterations="" -remote_storage="" -benchmark_id="" - -while [[ $# -gt 0 ]]; do - case "$1" in - --scale-factor) - scale_factor="$2" - shift 2 - ;; - --iterations) - iterations="$2" - shift 2 - ;; - --remote-storage) - remote_storage="$2" - shift 2 - ;; - --benchmark-id) - benchmark_id="$2" - shift 2 - ;; - *) - echo "Unknown option: $1" >&2 - exit 1 - ;; - esac -done - -is_remote=false -if [[ -n "$remote_storage" ]]; then - is_remote=true -fi - -# Lance on remote storage is not supported. The infrastructure to generate and upload lance files -# to S3 does not exist. If you need lance on S3, you must first implement: -# 1. Lance data generation in data-gen (or a separate step) -# 2. Lance data upload to S3 before this step runs -if $is_remote && echo "$targets" | grep -q 'lance'; then - echo "ERROR: Lance format is not supported for remote storage benchmarks." - echo "Remove 'datafusion:lance' from targets for benchmark '${benchmark_id:-unknown}'." - exit 1 -fi - -# Extract formats for each engine from the targets string. -# Example input: "datafusion:parquet,datafusion:vortex,datafusion:lance,duckdb:parquet" -# -# Pipeline: split by comma -> filter by engine prefix -> remove prefix -> rejoin with commas -# -# Lance is filtered out of df_formats because it uses a separate binary (lance-bench). -# -# The `|| true` is needed because some benchmarks don't use all engines (e.g., statpopgen only has -# duckdb targets). grep returns exit code 1 when no matches are found. Both greps must be in the -# subshell so that `|| true` covers the case where grep -v receives empty input. -df_formats=$(echo "$targets" | tr ',' '\n' | (grep '^datafusion:' | grep -v ':lance$' || true) | sed 's/datafusion://' | tr '\n' ',' | sed 's/,$//') -ddb_formats=$(echo "$targets" | tr ',' '\n' | (grep '^duckdb:' || true) | sed 's/duckdb://' | tr '\n' ',' | sed 's/,$//') -has_lance=$(echo "$targets" | grep -q 'datafusion:lance' && echo "true" || echo "false") - -# Build options string. -opts="" -if $is_remote; then - opts="--opt remote-data-dir=$remote_storage" -fi -if [[ -n "$scale_factor" ]]; then - if [[ -n "$opts" ]]; then - opts="--opt scale-factor=$scale_factor $opts" - else - opts="--opt scale-factor=$scale_factor" - fi -fi -if [[ -n "$iterations" ]]; then - opts="-i $iterations $opts" -fi - -touch results.json - -if [[ -n "$df_formats" ]]; then - # shellcheck disable=SC2086 - target/release_debug/datafusion-bench "$subcommand" \ - -d gh-json \ - --formats "$df_formats" \ - $opts \ - -o df-results.json - - cat df-results.json >> results.json -fi - -if [[ -n "$ddb_formats" ]]; then - # shellcheck disable=SC2086 - target/release_debug/duckdb-bench "$subcommand" \ - -d gh-json \ - --formats "$ddb_formats" \ - $opts \ - --delete-duckdb-database \ - -o ddb-results.json - - cat ddb-results.json >> results.json -fi - -# Lance-bench only runs for local benchmarks. -if ! $is_remote && [[ "$has_lance" == "true" ]] && [[ -f "target/release_debug/lance-bench" ]]; then - # shellcheck disable=SC2086 - target/release_debug/lance-bench "$subcommand" \ - -d gh-json \ - $opts \ - -o lance-results.json - - cat lance-results.json >> results.json -fi diff --git a/.github/workflows/approvals.yml b/.github/workflows/approvals.yml index 8939e8dec66..7448b02ecd2 100644 --- a/.github/workflows/approvals.yml +++ b/.github/workflows/approvals.yml @@ -3,10 +3,14 @@ name: PR Approval Check on: pull_request_review: types: [submitted, dismissed] + pull_request: + branches: + - "develop" jobs: check-approvals: runs-on: ubuntu-latest + timeout-minutes: 10 steps: - name: Check required approvals uses: actions/github-script@450193c5abd4cdb17ba9f3ffcfe8f635c4bb6c2a @@ -32,13 +36,15 @@ jobs: const authorType = pr.user.type; // 'Bot' vs 'User' const authorLogin = pr.user.login; // e.g. 'github-actions[bot]' const isBot = authorType === 'Bot' || authorLogin.endsWith('[bot]'); + const oneApprovalBotAuthors = new Set(['renovate[bot]']); - const required = isBot ? 2 : 1; + const required = isBot && !oneApprovalBotAuthors.has(authorLogin) ? 2 : 1; console.log(`PR author: ${authorLogin} (${authorType}), isBot: ${isBot}`); + console.log(`One-approval bot allowlist: ${oneApprovalBotAuthors.has(authorLogin)}`); console.log(`Approvals: ${approvalCount} / ${required} required`); - if (approvalCount < required) { + if (isBot && (approvalCount < required)) { core.setFailed( `This PR needs ${required} human approval(s) but has ${approvalCount}. ` + `(Author is ${isBot ? 'a bot' : 'human'})` diff --git a/.github/workflows/bench-dispatch.yml b/.github/workflows/bench-dispatch.yml index 264dfaf94e8..48bf6fbe4af 100644 --- a/.github/workflows/bench-dispatch.yml +++ b/.github/workflows/bench-dispatch.yml @@ -18,10 +18,10 @@ permissions: jobs: remove-bench-label: runs-on: ubuntu-latest - timeout-minutes: 2 + timeout-minutes: 10 if: github.event.label.name == 'action/benchmark' steps: - - uses: actions-ecosystem/action-remove-labels@v1 + - uses: actions-ecosystem/action-remove-labels@2ce5d41b4b6aa8503e285553f75ed56e0a40bae0 # v1 if: github.event.pull_request.head.repo.full_name == 'vortex-data/vortex' with: labels: action/benchmark @@ -34,10 +34,10 @@ jobs: remove-sql-label: runs-on: ubuntu-latest - timeout-minutes: 2 + timeout-minutes: 10 if: github.event.label.name == 'action/benchmark-sql' steps: - - uses: actions-ecosystem/action-remove-labels@v1 + - uses: actions-ecosystem/action-remove-labels@2ce5d41b4b6aa8503e285553f75ed56e0a40bae0 # v1 if: github.event.pull_request.head.repo.full_name == 'vortex-data/vortex' with: labels: action/benchmark-sql diff --git a/.github/workflows/bench-pr.yml b/.github/workflows/bench-pr.yml index f392f1a6e45..d50a5f04633 100644 --- a/.github/workflows/bench-pr.yml +++ b/.github/workflows/bench-pr.yml @@ -38,7 +38,7 @@ jobs: if: github.event.pull_request.head.repo.fork == false with: sccache: s3 - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: ref: ${{ github.event.pull_request.head.sha }} - name: Setup benchmark environment @@ -49,7 +49,7 @@ jobs: - name: Install DuckDB run: | - wget -qO- https://github.com/duckdb/duckdb/releases/download/v1.4.2/duckdb_cli-linux-amd64.zip | funzip > duckdb + wget -qO- https://github.com/duckdb/duckdb/releases/download/v1.5.2/duckdb_cli-linux-amd64.zip | funzip > duckdb chmod +x duckdb echo "$PWD" >> $GITHUB_PATH @@ -60,11 +60,11 @@ jobs: env: RUSTFLAGS: "-C target-cpu=native" run: | - cargo build --package ${{ matrix.benchmark.id }} --profile release_debug ${{ matrix.benchmark.build_args }} + cargo build --package ${{ matrix.benchmark.id }} --profile release_debug ${{ matrix.benchmark.build_args }} --features unstable_encodings - name: Setup Polar Signals if: github.event.pull_request.head.repo.fork == false - uses: polarsignals/gh-actions-ps-profiling@v0.8.1 + uses: polarsignals/gh-actions-ps-profiling@68ae857e375a826606352016e5b90f01a2a7ff7a # v0.8.1 with: polarsignals_cloud_token: ${{ secrets.POLAR_SIGNALS_API_KEY }} labels: "branch=${{ github.ref_name }};gh_run_id=${{ github.run_id }};benchmark=${{ matrix.benchmark.id }}" @@ -76,18 +76,20 @@ jobs: shell: bash env: RUST_BACKTRACE: full + VORTEX_EXPERIMENTAL_PATCHED_ARRAY: "1" + FLAT_LAYOUT_INLINE_ARRAY_NODE: "1" run: | bash scripts/bench-taskset.sh target/release_debug/${{ matrix.benchmark.id }} -d gh-json -o results.json - name: Setup AWS CLI if: github.event.pull_request.head.repo.fork == false - uses: aws-actions/configure-aws-credentials@v6 + uses: aws-actions/configure-aws-credentials@ec61189d14ec14c8efccab744f656cffd0e33f37 # v6 with: role-to-assume: arn:aws:iam::245040174862:role/GitHubBenchmarkRole aws-region: us-east-1 - name: Install uv - uses: spiraldb/actions/.github/actions/setup-uv@0.18.5 + uses: spiraldb/actions/.github/actions/setup-uv@a746510eafaa926484c354541cfc49b2ec06cc63 # 0.18.6 with: sync: false @@ -115,14 +117,14 @@ jobs: - name: Comment PR if: github.event.pull_request.head.repo.fork == false - uses: thollander/actions-comment-pull-request@v3 + uses: thollander/actions-comment-pull-request@24bffb9b452ba05a4f3f77933840a6a841d1b32b # v3 with: file-path: comment.md comment-tag: bench-pr-comment-${{ matrix.benchmark.id }} - name: Comment PR on failure if: failure() && github.event.pull_request.head.repo.fork == false - uses: thollander/actions-comment-pull-request@v3 + uses: thollander/actions-comment-pull-request@24bffb9b452ba05a4f3f77933840a6a841d1b32b # v3 with: message: | # BENCHMARK FAILED @@ -135,82 +137,3 @@ jobs: secrets: inherit with: mode: "pr" - benchmark_matrix: | - [ - { - "id": "clickbench-nvme", - "subcommand": "clickbench", - "name": "Clickbench on NVME", - "targets": "datafusion:parquet,datafusion:vortex,duckdb:parquet,duckdb:vortex,duckdb:duckdb", - "extra_data_formats": "vortex-compact" - }, - { - "id": "tpch-nvme", - "subcommand": "tpch", - "name": "TPC-H SF=1 on NVME", - "targets": "datafusion:arrow,datafusion:parquet,datafusion:vortex,datafusion:vortex-compact,duckdb:parquet,duckdb:vortex,duckdb:vortex-compact,duckdb:duckdb", - "scale_factor": "1.0" - }, - { - "id": "tpch-s3", - "subcommand": "tpch", - "name": "TPC-H SF=1 on S3", - "local_dir": "vortex-bench/data/tpch/1.0", - "remote_storage": "s3://vortex-ci-benchmark-datasets/${{github.ref_name}}/${{github.run_id}}/tpch/1.0/", - "targets": "datafusion:parquet,datafusion:vortex,datafusion:vortex-compact,duckdb:parquet,duckdb:vortex,duckdb:vortex-compact", - "scale_factor": "1.0" - }, - { - "id": "tpch-nvme-10", - "subcommand": "tpch", - "name": "TPC-H SF=10 on NVME", - "targets": "datafusion:arrow,datafusion:parquet,datafusion:vortex,datafusion:vortex-compact,duckdb:parquet,duckdb:vortex,duckdb:vortex-compact,duckdb:duckdb", - "scale_factor": "10.0" - }, - { - "id": "tpch-s3-10", - "subcommand": "tpch", - "name": "TPC-H SF=10 on S3", - "local_dir": "vortex-bench/data/tpch/10.0", - "remote_storage": "s3://vortex-ci-benchmark-datasets/${{github.ref_name}}/${{github.run_id}}/tpch/10.0/", - "targets": "datafusion:parquet,datafusion:vortex,datafusion:vortex-compact,duckdb:parquet,duckdb:vortex,duckdb:vortex-compact", - "scale_factor": "10.0" - }, - { - "id": "tpcds-nvme", - "subcommand": "tpcds", - "name": "TPC-DS SF=1 on NVME", - "targets": "datafusion:parquet,datafusion:vortex,datafusion:vortex-compact,duckdb:parquet,duckdb:vortex,duckdb:vortex-compact,duckdb:duckdb", - "scale_factor": "1.0" - }, - { - "id": "statpopgen", - "subcommand": "statpopgen", - "name": "Statistical and Population Genetics", - "targets": "duckdb:parquet,duckdb:vortex,duckdb:vortex-compact", - "scale_factor": "100" - }, - { - "id": "fineweb", - "subcommand": "fineweb", - "name": "FineWeb NVMe", - "targets": "datafusion:parquet,datafusion:vortex,datafusion:vortex-compact,duckdb:parquet,duckdb:vortex,duckdb:vortex-compact", - "scale_factor": "100" - }, - { - "id": "fineweb-s3", - "subcommand": "fineweb", - "name": "FineWeb S3", - "local_dir": "vortex-bench/data/fineweb", - "remote_storage": "s3://vortex-ci-benchmark-datasets/${{github.ref_name}}/${{github.run_id}}/fineweb/", - "targets": "datafusion:parquet,datafusion:vortex,datafusion:vortex-compact,duckdb:parquet,duckdb:vortex,duckdb:vortex-compact", - "scale_factor": "100" - }, - { - "id": "polarsignals", - "subcommand": "polarsignals", - "name": "PolarSignals Profiling", - "targets": "datafusion:vortex", - "scale_factor": "1" - }, - ] diff --git a/.github/workflows/bench.yml b/.github/workflows/bench.yml index 7be829d80bd..a437523b9be 100644 --- a/.github/workflows/bench.yml +++ b/.github/workflows/bench.yml @@ -14,11 +14,11 @@ permissions: jobs: commit-metadata: runs-on: ubuntu-latest - timeout-minutes: 120 + timeout-minutes: 10 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - name: Setup AWS CLI - uses: aws-actions/configure-aws-credentials@v6 + uses: aws-actions/configure-aws-credentials@ec61189d14ec14c8efccab744f656cffd0e33f37 # v6 with: role-to-assume: arn:aws:iam::245040174862:role/GitHubBenchmarkRole aws-region: us-east-1 @@ -38,6 +38,7 @@ jobs: && format('runs-on={0}/runner=bench-dedicated/extras=s3-cache/tag={1}', github.run_id, matrix.benchmark.id) || 'ubuntu-latest' }} strategy: + fail-fast: false matrix: benchmark: - id: random-access-bench @@ -53,7 +54,7 @@ jobs: if: github.repository == 'vortex-data/vortex' with: sccache: s3 - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - name: Setup benchmark environment run: sudo bash scripts/setup-benchmark.sh - uses: ./.github/actions/setup-rust @@ -62,7 +63,7 @@ jobs: - name: Install DuckDB run: | - wget -qO- https://github.com/duckdb/duckdb/releases/download/v1.4.2/duckdb_cli-linux-amd64.zip | funzip > duckdb + wget -qO- https://github.com/duckdb/duckdb/releases/download/v1.5.2/duckdb_cli-linux-amd64.zip | funzip > duckdb chmod +x duckdb echo "$PWD" >> $GITHUB_PATH @@ -73,10 +74,10 @@ jobs: env: RUSTFLAGS: "-C target-cpu=native" run: | - cargo build --bin ${{ matrix.benchmark.id }} --profile release_debug ${{ matrix.benchmark.build_args }} + cargo build --bin ${{ matrix.benchmark.id }} --profile release_debug ${{ matrix.benchmark.build_args }} --features unstable_encodings - name: Setup Polar Signals - uses: polarsignals/gh-actions-ps-profiling@v0.8.1 + uses: polarsignals/gh-actions-ps-profiling@68ae857e375a826606352016e5b90f01a2a7ff7a # v0.8.1 with: polarsignals_cloud_token: ${{ secrets.POLAR_SIGNALS_API_KEY }} labels: "branch=${{ github.ref_name }};gh_run_id=${{ github.run_id }};benchmark=${{ matrix.benchmark.id }}" @@ -88,11 +89,13 @@ jobs: shell: bash env: RUST_BACKTRACE: full + VORTEX_EXPERIMENTAL_PATCHED_ARRAY: "1" + FLAT_LAYOUT_INLINE_ARRAY_NODE: "1" run: | bash scripts/bench-taskset.sh target/release_debug/${{ matrix.benchmark.id }} --formats ${{ matrix.benchmark.formats }} -d gh-json -o results.json - name: Setup AWS CLI - uses: aws-actions/configure-aws-credentials@v6 + uses: aws-actions/configure-aws-credentials@ec61189d14ec14c8efccab744f656cffd0e33f37 # v6 with: role-to-assume: arn:aws:iam::245040174862:role/GitHubBenchmarkRole aws-region: us-east-1 @@ -115,83 +118,3 @@ jobs: secrets: inherit with: mode: "develop" - benchmark_matrix: | - [ - { - "id": "clickbench-nvme", - "subcommand": "clickbench", - "name": "Clickbench on NVME", - "targets": "datafusion:parquet,datafusion:vortex,datafusion:vortex-compact,datafusion:lance,duckdb:parquet,duckdb:vortex,duckdb:vortex-compact,duckdb:duckdb", - "build_lance": true - }, - { - "id": "tpch-nvme", - "subcommand": "tpch", - "name": "TPC-H SF=1 on NVME", - "targets": "datafusion:arrow,datafusion:parquet,datafusion:vortex,datafusion:vortex-compact,datafusion:lance,duckdb:parquet,duckdb:vortex,duckdb:vortex-compact,duckdb:duckdb", - "scale_factor": "1.0", - "build_lance": true - }, - { - "id": "tpch-s3", - "subcommand": "tpch", - "name": "TPC-H SF=1 on S3", - "local_dir": "vortex-bench/data/tpch/1.0", - "remote_storage": "s3://vortex-ci-benchmark-datasets/${{github.ref_name}}/${{github.run_id}}/tpch/1.0/", - "targets": "datafusion:parquet,datafusion:vortex,datafusion:vortex-compact,duckdb:parquet,duckdb:vortex,duckdb:vortex-compact", - "scale_factor": "1.0" - }, - { - "id": "tpch-nvme-10", - "subcommand": "tpch", - "name": "TPC-H SF=10 on NVME", - "targets": "datafusion:arrow,datafusion:parquet,datafusion:vortex,datafusion:vortex-compact,datafusion:lance,duckdb:parquet,duckdb:vortex,duckdb:vortex-compact,duckdb:duckdb", - "scale_factor": "10.0", - "build_lance": true - }, - { - "id": "tpch-s3-10", - "subcommand": "tpch", - "name": "TPC-H SF=10 on S3", - "local_dir": "vortex-bench/data/tpch/10.0", - "remote_storage": "s3://vortex-ci-benchmark-datasets/${{github.ref_name}}/${{github.run_id}}/tpch/10.0/", - "targets": "datafusion:parquet,datafusion:vortex,datafusion:vortex-compact,duckdb:parquet,duckdb:vortex,duckdb:vortex-compact", - "scale_factor": "10.0" - }, - { - "id": "tpcds-nvme", - "subcommand": "tpcds", - "name": "TPC-DS SF=1 on NVME", - "targets": "datafusion:parquet,datafusion:vortex,datafusion:vortex-compact,duckdb:parquet,duckdb:vortex,duckdb:vortex-compact,duckdb:duckdb", - "scale_factor": "1.0" - }, - { - "id": "statpopgen", - "subcommand": "statpopgen", - "name": "Statistical and Population Genetics", - "local_dir": "vortex-bench/data/statpopgen", - "targets": "duckdb:parquet,duckdb:vortex,duckdb:vortex-compact", - "scale_factor": "100" - }, - { - "id": "fineweb", - "subcommand": "fineweb", - "name": "FineWeb NVMe", - "targets": "datafusion:parquet,datafusion:vortex,datafusion:vortex-compact,duckdb:parquet,duckdb:vortex,duckdb:vortex-compact" - }, - { - "id": "fineweb-s3", - "subcommand": "fineweb", - "name": "FineWeb S3", - "local_dir": "vortex-bench/data/fineweb", - "remote_storage": "s3://vortex-ci-benchmark-datasets/${{github.ref_name}}/${{github.run_id}}/fineweb/", - "targets": "datafusion:parquet,datafusion:vortex,datafusion:vortex-compact,duckdb:parquet,duckdb:vortex,duckdb:vortex-compact" - }, - { - "id": "polarsignals", - "subcommand": "polarsignals", - "name": "PolarSignals Profiling", - "targets": "datafusion:vortex", - "scale_factor": "1" - }, - ] diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e706b15b99f..3f0740e4eec 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,9 +1,8 @@ -name: CI +name: Linters and Tests # Concurrency control: # - PRs: new commits on a feature branch will cancel in-progress (outdated) runs. -# - Push to develop: runs queue sequentially, never cancelled. This allows us to have benchmarks -# run on every commit for our benchmarks website. +# - Push to develop: runs queue sequentially, never cancelled. # - `workflow_dispatch`: groups by branch and queues if run on develop. concurrency: group: ${{ github.workflow }}-${{ github.ref }} @@ -21,23 +20,23 @@ permissions: issues: write # audit-check creates issues env: - CARGO_TERM_COLOR: auto + CARGO_TERM_COLOR: always RUST_BACKTRACE: 1 NIGHTLY_TOOLCHAIN: nightly-2026-02-05 jobs: lint-toml: runs-on: ubuntu-latest - timeout-minutes: 40 + timeout-minutes: 10 steps: - - uses: actions/checkout@v6 - - uses: spiraldb/actions/.github/actions/lint-toml@0.18.5 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + - uses: spiraldb/actions/.github/actions/lint-toml@a746510eafaa926484c354541cfc49b2ec06cc63 # 0.18.6 validate-workflow-yaml: runs-on: ubuntu-latest - timeout-minutes: 40 + timeout-minutes: 10 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - name: Validate YAML file run: | # Lint the workflows and yamllint's configuration file. @@ -52,13 +51,13 @@ jobs: ${{ github.repository == 'vortex-data/vortex' && format('runs-on={0}/runner=amd64-medium/image=ubuntu24-full-x64-pre-v2/tag=python-lint', github.run_id) || 'ubuntu-latest' }} - timeout-minutes: 40 + timeout-minutes: 10 steps: - uses: runs-on/action@v2 if: github.repository == 'vortex-data/vortex' with: sccache: s3 - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - uses: ./.github/actions/setup-prebuild - name: Install uv if: github.repository != 'vortex-data/vortex' @@ -85,7 +84,7 @@ jobs: ${{ github.repository == 'vortex-data/vortex' && format('runs-on={0}/runner=amd64-large/image=ubuntu24-full-x64-pre-v2/tag=python-test', github.run_id) || 'ubuntu-latest' }} - timeout-minutes: 40 + timeout-minutes: 30 env: RUST_LOG: "info,maturin=off,uv=debug" MATURIN_PEP517_ARGS: "--profile ci" @@ -94,7 +93,7 @@ jobs: if: github.repository == 'vortex-data/vortex' with: sccache: s3 - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - uses: ./.github/actions/setup-prebuild - name: Pytest - Vortex @@ -120,50 +119,9 @@ jobs: uv run --all-packages make html working-directory: docs/ - python-wheel-build: - name: "Python (wheel build)" - runs-on: ubuntu-latest - timeout-minutes: 40 - steps: - - uses: actions/checkout@v6 - - name: Rust Dependency Cache - uses: Swatinem/rust-cache@v2 - with: - save-if: ${{ github.ref_name == 'develop' }} - - uses: ./.github/actions/setup-rust - with: - repo-token: ${{ secrets.GITHUB_TOKEN }} - enable-sccache: "false" - - uses: mlugg/setup-zig@v2.2.1 - - name: Install uv - uses: spiraldb/actions/.github/actions/setup-uv@0.18.5 - with: - sync: false - prune-cache: false - - name: Ensure wheel and sdist can be built on Linux - PyVortex - shell: bash - run: | - echo "Clearing wheel target directory" - rm -rf ../target/wheels/ - - uv venv - uv tool run maturin@1.10 build --interpreter python3.11 --zig - uv tool run maturin@1.10 build --interpreter python3.11 --zig --sdist - - file_count=$(ls -1 ../target/wheels/ | wc -l) - - if [[ $file_count -ne 2 ]]; then - echo "Unexpected number of files detected ${file_count}:" - ls ../target/wheels/ - exit 1 - else - echo "Generated two files" - fi - working-directory: vortex-python/ - rust-docs: name: "Rust (docs)" - timeout-minutes: 40 + timeout-minutes: 30 runs-on: >- ${{ github.repository == 'vortex-data/vortex' && format('runs-on={0}/runner=amd64-small/image=ubuntu24-full-x64-pre-v2/tag=rust-docs', github.run_id) @@ -173,7 +131,7 @@ jobs: if: github.repository == 'vortex-data/vortex' with: sccache: s3 - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - uses: ./.github/actions/setup-prebuild - name: Docs run: | @@ -183,7 +141,7 @@ jobs: build-rust: name: "Rust build (${{matrix.config.name}})" - timeout-minutes: 40 + timeout-minutes: 30 runs-on: >- ${{ github.repository == 'vortex-data/vortex' && format('runs-on={0}/runner={1}/image=ubuntu24-full-x64-pre-v2/tag={2}', github.run_id, matrix.config.runner, matrix.config.name) @@ -209,14 +167,14 @@ jobs: runner: amd64-medium target: wasm32-unknown-unknown env: - rustflags: "RUSTFLAGS='-A warnings'" + rustflags: "RUSTFLAGS='-A warnings --cfg getrandom_backend=\"unsupported\"'" args: "--target wasm32-unknown-unknown --exclude vortex --exclude vortex-cuda --exclude vortex-cub --exclude vortex-nvcomp --exclude vortex-datafusion --exclude vortex-duckdb --exclude vortex-tui --exclude vortex-zstd --exclude vortex-test-e2e-cuda --exclude vortex-sqllogictest" steps: - uses: runs-on/action@v2 if: github.repository == 'vortex-data/vortex' with: sccache: s3 - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - uses: ./.github/actions/setup-prebuild - name: Install wasm32 target if: ${{ matrix.config.target == 'wasm32-unknown-unknown' }} @@ -231,7 +189,7 @@ jobs: check-min-deps: name: "Check build with minimal dependencies" - timeout-minutes: 40 + timeout-minutes: 30 runs-on: >- ${{ github.repository == 'vortex-data/vortex' && format('runs-on={0}/runner=amd64-medium/image=ubuntu24-full-x64-pre-v2/tag=rust-min-deps', github.run_id) @@ -241,13 +199,13 @@ jobs: if: github.repository == 'vortex-data/vortex' with: sccache: s3 - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - uses: ./.github/actions/setup-prebuild - run: cargo minimal-versions check --direct --workspace --ignore-private rust-lint: name: "Rust (lint)" - timeout-minutes: 40 + timeout-minutes: 30 runs-on: >- ${{ github.repository == 'vortex-data/vortex' && format('runs-on={0}/runner=amd64-large/image=ubuntu24-full-x64-pre-v2/tag=rust-lint', github.run_id) @@ -257,37 +215,74 @@ jobs: if: github.repository == 'vortex-data/vortex' with: sccache: s3 - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - uses: ./.github/actions/setup-prebuild - name: Install nightly for fmt run: rustup toolchain install $NIGHTLY_TOOLCHAIN --component rustfmt - name: Rust Lint - Format + id: fmt + continue-on-error: true run: cargo +$NIGHTLY_TOOLCHAIN fmt --all --check - name: Rustc check + id: check + continue-on-error: true run: RUSTFLAGS="-D warnings" cargo check --profile ci --locked --all-features --all-targets - name: Rustc check (release) + id: check-release + continue-on-error: true run: RUSTFLAGS="-D warnings" cargo check --locked --all-features --all-targets --release - name: Rust Lint - Clippy All Features + id: clippy-all + continue-on-error: true run: cargo clippy --profile ci --locked --all-features --all-targets -- -D warnings - name: Rust Lint - Clippy Default Features + id: clippy-default + continue-on-error: true run: cargo clippy --profile ci --locked --all-targets -- -D warnings + - name: Check lint results + if: always() + uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9 + with: + script: | + const failed = Object.entries({ + fmt: '${{ steps.fmt.outcome }}', + check: '${{ steps.check.outcome }}', + 'check-release': '${{ steps.check-release.outcome }}', + 'clippy-all': '${{ steps.clippy-all.outcome }}', + 'clippy-default': '${{ steps.clippy-default.outcome }}', + }).filter(([, o]) => o === 'failure').map(([n]) => n); + if (!failed.length) return; + + const { data: { jobs } } = await github.rest.actions.listJobsForWorkflowRun({ + ...context.repo, run_id: context.runId, + }); + const url = jobs.find(j => j.name === context.job)?.check_run_url; + if (url) { + await github.rest.checks.update({ + ...context.repo, + check_run_id: Number(url.split('/').pop()), + output: { title: `${failed.join(', ')} — failing`, summary: '' }, + }); + } + core.setFailed(`Lint failed: ${failed.join(', ')}`); cpp-lint: name: "C/C++ (lint)" runs-on: ubuntu-latest timeout-minutes: 10 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - name: C/C++ Lint - clang-format run: | git ls-files vortex-cuda vortex-cxx vortex-duckdb vortex-ffi \ | grep -E '\.(cpp|hpp|cu|cuh|h)$' \ | grep -v 'kernels/src/bit_unpack_.*\.cu$' \ + | grep -v 'kernels/src/bit_unpack_.*_lanes\.cuh$' \ | xargs clang-format --dry-run --Werror --style=file rust-lint-no-default: name: "Rust (lint, no default)" - timeout-minutes: 40 + timeout-minutes: 30 runs-on: >- ${{ github.repository == 'vortex-data/vortex' && format('runs-on={0}/runner=amd64-medium/image=ubuntu24-full-x64-pre-v2/tag=rust-lint-no-default', github.run_id) @@ -297,7 +292,7 @@ jobs: if: github.repository == 'vortex-data/vortex' with: sccache: s3 - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - uses: ./.github/actions/setup-prebuild - name: Install cargo-hack if: github.repository != 'vortex-data/vortex' @@ -310,7 +305,7 @@ jobs: public-api: name: "Public API lock files" - timeout-minutes: 40 + timeout-minutes: 30 runs-on: >- ${{ github.repository == 'vortex-data/vortex' && format('runs-on={0}/runner=amd64-xsmall/image=ubuntu24-full-x64-pre-v2/tag=public-api', github.run_id) @@ -320,7 +315,7 @@ jobs: if: github.repository == 'vortex-data/vortex' with: sccache: s3 - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - uses: ./.github/actions/setup-prebuild - name: Install nightly for public-api run: rustup toolchain install $NIGHTLY_TOOLCHAIN @@ -342,317 +337,9 @@ jobs: exit 1 fi - rust-coverage: - name: "Rust tests (coverage) (${{ matrix.suite }})" - timeout-minutes: 40 - permissions: - id-token: write - strategy: - matrix: - include: - - suite: tests - runs-on: >- - ${{ github.repository == 'vortex-data/vortex' - && format('runs-on={0}/runner=amd64-large/image=ubuntu24-full-x64-pre-v2/tag=rust-coverage-suite-{1}', github.run_id, matrix.suite) - || 'ubuntu-latest' }} - env: - RUSTFLAGS: "-Cinstrument-coverage -A warnings" - CARGO_INCREMENTAL: 0 # Disable incremental compilation to get accurate coverage - LLVM_PROFILE_FILE: "target/coverage/vortex-%p-%m.profraw" - GRCOV_OUTPUT_FILE: "target/coverage/vortex.lcov" - steps: - - uses: runs-on/action@v2 - if: github.repository == 'vortex-data/vortex' - with: - sccache: s3 - - uses: actions/checkout@v6 - - uses: ./.github/actions/setup-prebuild - - name: Rust Tests - if: ${{ matrix.suite == 'tests' }} - run: | - cargo nextest run --locked --workspace --all-features --no-fail-fast - - name: Generate coverage report - run: | - grcov . --binary-path target/debug/ -s . -t lcov --llvm --ignore-not-existing \ - --threads $(nproc) \ - --ignore '../*' --ignore '/*' --ignore 'fuzz/*' --ignore 'vortex-bench/*' \ - --ignore 'home/*' --ignore 'xtask/*' --ignore 'target/*' --ignore 'vortex-error/*' \ - --ignore 'vortex-python/*' --ignore 'vortex-jni/*' --ignore 'vortex-flatbuffers/*' \ - --ignore 'vortex-proto/*' --ignore 'vortex-tui/*' --ignore 'vortex-datafusion/examples/*' \ - --ignore 'vortex-ffi/examples/*' --ignore '*/arbitrary/*' --ignore '*/arbitrary.rs' --ignore 'vortex-cxx/*' \ - --ignore benchmarks/* --ignore 'vortex-test/*' \ - -o ${{ env.GRCOV_OUTPUT_FILE }} - - name: Codecov - uses: codecov/codecov-action@v5 - with: - name: run-${{ matrix.suite }} - files: ${{ env.GRCOV_OUTPUT_FILE }} - disable_search: true - flags: ${{ matrix.suite }} - use_oidc: true - - rust-test-sanitizer: - strategy: - fail-fast: false - matrix: - include: - # We don't run memory sanitizer as it provides many false positives - # for std - - sanitizer: asan - sanitizer_flags: "-Zsanitizer=address,leak" - - sanitizer: tsan - sanitizer_flags: "-Zsanitizer=thread" - name: "Rust tests (${{ matrix.sanitizer }})" - runs-on: >- - ${{ github.repository == 'vortex-data/vortex' - && format('runs-on={0}/pool=amd64-medium-pre-v2/tag=rust-test-sanitizer', github.run_id) - || 'ubuntu-latest' }} - timeout-minutes: 40 - env: - ASAN_OPTIONS: "symbolize=1:check_initialization_order=1:detect_leaks=1:leak_check_at_exit=1" - LSAN_OPTIONS: "report_objects=1" - ASAN_SYMBOLIZER_PATH: "/usr/bin/llvm-symbolizer" - MSAN_OPTIONS: "symbolize=1" - MSAN_SYMBOLIZER_PATH: "/usr/bin/llvm-symbolizer" - TSAN_OPTIONS: "symbolize=1:suppressions=${{ github.workspace }}/vortex-ffi/tsan_suppressions.txt" - TSAN_SYMBOLIZER_PATH: "/usr/bin/llvm-symbolizer" - VORTEX_SKIP_SLOW_TESTS: "1" - # -Cunsafe-allow-abi-mismatch=sanitizer: libraries like compiler_builtins - # unset -Zsanitizer flag and we should allow that. - RUSTFLAGS: "-A warnings -Cunsafe-allow-abi-mismatch=sanitizer --cfg disable_loom --cfg vortex_nightly -C debuginfo=2 -C opt-level=0 -C strip=none" - steps: - - uses: runs-on/action@v2 - if: github.repository == 'vortex-data/vortex' - with: - sccache: s3 - - uses: actions/checkout@v6 - - uses: ./.github/actions/setup-prebuild - - name: Install Rust nightly toolchain - run: | - rustup toolchain install $NIGHTLY_TOOLCHAIN - rustup component add --toolchain $NIGHTLY_TOOLCHAIN rust-src rustfmt clippy llvm-tools-preview - - name: Build tests with sanitizer - run: | - RUSTFLAGS="${RUSTFLAGS} ${{ matrix.sanitizer_flags }}" \ - cargo +$NIGHTLY_TOOLCHAIN build --locked --all-features \ - --target x86_64-unknown-linux-gnu -Zbuild-std \ - -p vortex-buffer -p vortex-fastlanes -p vortex-fsst -p vortex-alp -p vortex-array - - - name: Run tests with sanitizer - run: | - RUSTFLAGS="${RUSTFLAGS} ${{ matrix.sanitizer_flags }}" \ - cargo +$NIGHTLY_TOOLCHAIN nextest run --locked --all-features \ - --target x86_64-unknown-linux-gnu --no-fail-fast -Zbuild-std \ - -p vortex-buffer -p vortex-fastlanes -p vortex-fsst -p vortex-alp -p vortex-array - - # vortex-ffi requires --no-default-features as otherwise we pull in - # Mimalloc which interferes with sanitizers - # cargo nextest reports less sanitizer issues than cargo test - # TODO(myrrc): remove --no-default-features once we make Mimalloc opt-in - - name: Run vortex-ffi tests with sanitizer - run: | - RUSTFLAGS="${RUSTFLAGS} ${{ matrix.sanitizer_flags }}" \ - cargo +$NIGHTLY_TOOLCHAIN test --locked --no-default-features \ - --target x86_64-unknown-linux-gnu --no-fail-fast -Zbuild-std \ - -p vortex-ffi -- --no-capture - - rust-ffi-test-sanitizer: - strategy: - fail-fast: false - matrix: - include: - # We don't run memory sanitizer as it's clang-only and provides many - # false positives for Catch2 - - sanitizer: asan - sanitizer_flags: "-Zsanitizer=address,leak" - - sanitizer: tsan - sanitizer_flags: "-Zsanitizer=thread" - name: "Rust/C++ FFI tests (${{ matrix.sanitizer }})" - timeout-minutes: 40 - env: - ASAN_OPTIONS: "symbolize=1:check_initialization_order=1:detect_leaks=1:leak_check_at_exit=1" - LSAN_OPTIONS: "report_objects=1" - ASAN_SYMBOLIZER_PATH: "/usr/bin/llvm-symbolizer" - MSAN_OPTIONS: "symbolize=1" - MSAN_SYMBOLIZER_PATH: "/usr/bin/llvm-symbolizer" - TSAN_OPTIONS: "symbolize=1" - TSAN_SYMBOLIZER_PATH: "/usr/bin/llvm-symbolizer" - VORTEX_SKIP_SLOW_TESTS: "1" - # -Cunsafe-allow-abi-mismatch=sanitizer: libraries like compiler_builtins - # unset -Zsanitizer flag and we should allow that. - runs-on: >- - ${{ github.repository == 'vortex-data/vortex' - && format('runs-on={0}/runner=amd64-medium/image=ubuntu24-full-x64-pre-v2/tag=rust-ffi-test-sanitizer', github.run_id) - || 'ubuntu-latest' }} - steps: - - uses: runs-on/action@v2 - if: github.repository == 'vortex-data/vortex' - with: - sccache: s3 - - uses: actions/checkout@v6 - - uses: ./.github/actions/setup-prebuild - - name: Install rustfilt - run: | - cargo install rustfilt - - name: Install Rust nightly toolchain - run: | - rustup toolchain install $NIGHTLY_TOOLCHAIN - rustup component add --toolchain $NIGHTLY_TOOLCHAIN rust-src rustfmt clippy llvm-tools-preview - - name: Build FFI library - run: | - # TODO(myrrc): remove --no-default-features - RUSTFLAGS="-A warnings -Cunsafe-allow-abi-mismatch=sanitizer \ - --cfg disable_loom --cfg vortex_nightly -C debuginfo=2 \ - -C opt-level=0 -C strip=none -Zexternal-clangrt \ - ${{ matrix.sanitizer_flags }}" \ - cargo +$NIGHTLY_TOOLCHAIN build --locked --no-default-features \ - --target x86_64-unknown-linux-gnu -Zbuild-std \ - -p vortex-ffi - - name: Build FFI library tests - run: | - cd vortex-ffi - cmake -Bbuild -DBUILD_TESTS=1 -DSANITIZER=${{ matrix.sanitizer }} -DTARGET_TRIPLE="x86_64-unknown-linux-gnu" - cmake --build build -j - - name: Run tests - run: | - set -o pipefail - ./vortex-ffi/build/test/vortex_ffi_test 2>&1 | rustfilt -i- - - cuda-build-lint: - if: github.repository == 'vortex-data/vortex' - name: "CUDA build & lint" - timeout-minutes: 40 - runs-on: runs-on=${{ github.run_id }}/runner=gpu/tag=cuda-build - steps: - - uses: runs-on/action@v2 - with: - sccache: s3 - - uses: actions/checkout@v6 - - uses: ./.github/actions/setup-rust - with: - repo-token: ${{ secrets.GITHUB_TOKEN }} - - uses: ./.github/actions/check-rebuild - with: - command: >- - cargo build --profile ci --locked --all-features --all-targets - -p vortex-cuda -p vortex-cub -p vortex-nvcomp - -p gpu-scan-cli -p vortex-test-e2e-cuda - - name: Clippy CUDA crates - run: | - cargo clippy --profile ci --locked --all-features --all-targets \ - -p vortex-cuda \ - -p vortex-cub \ - -p vortex-nvcomp \ - -p gpu-scan-cli \ - -p vortex-test-e2e-cuda \ - -- -D warnings - - cuda-test: - if: github.repository == 'vortex-data/vortex' - name: "CUDA tests" - timeout-minutes: 30 - runs-on: runs-on=${{ github.run_id }}/runner=gpu/tag=cuda-tests - steps: - - uses: runs-on/action@v2 - with: - sccache: s3 - - name: Display NVIDIA SMI details - run: | - nvidia-smi - nvidia-smi -L - nvidia-smi -q -d Memory - - uses: actions/checkout@v6 - - uses: ./.github/actions/setup-rust - with: - repo-token: ${{ secrets.GITHUB_TOKEN }} - - name: Install nextest - uses: taiki-e/install-action@v2 - with: - tool: nextest - - name: Rust Tests - env: - FLAT_LAYOUT_INLINE_ARRAY_NODE: true - run: | - cargo nextest run \ - --cargo-profile ci \ - --locked \ - -p vortex-file \ - -p vortex-cuda \ - -p vortex-cub \ - -p vortex-nvcomp \ - -p vortex-test-e2e-cuda \ - --all-features \ - --no-fail-fast \ - --target x86_64-unknown-linux-gnu \ - --verbose - - cuda-test-sanitizer: - if: github.repository == 'vortex-data/vortex' - name: "CUDA tests (${{ matrix.sanitizer }})" - timeout-minutes: 30 - runs-on: runs-on=${{ github.run_id }}/runner=gpu/tag=cuda-test-sanitizer - strategy: - fail-fast: false - matrix: - include: - - sanitizer: memcheck - runner_flags: "--tool memcheck --leak-check=full --error-exitcode 1" - # TODO(joe): try to re-enable racecheck, it is hanging in CI. - # - sanitizer: racecheck - # runner_flags: "--tool racecheck --error-exitcode 1" - - sanitizer: synccheck - runner_flags: "--tool synccheck --error-exitcode 1" - - sanitizer: initcheck - runner_flags: "--tool initcheck --error-exitcode 1" - steps: - - uses: runs-on/action@v2 - with: - sccache: s3 - - name: Display NVIDIA SMI details - run: | - nvidia-smi - nvidia-smi -L - nvidia-smi -q -d Memory - - uses: actions/checkout@v6 - - uses: ./.github/actions/setup-rust - with: - repo-token: ${{ secrets.GITHUB_TOKEN }} - - name: Build tests - run: cargo test --profile ci --locked -p vortex-cuda --all-features --target x86_64-unknown-linux-gnu --no-run - - name: "CUDA - ${{ matrix.sanitizer }}" - env: - CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUNNER: "compute-sanitizer ${{ matrix.runner_flags }}" - run: cargo test --profile ci --locked -p vortex-cuda --all-features --target x86_64-unknown-linux-gnu - - cuda-test-cudf: - if: github.repository == 'vortex-data/vortex' - name: "CUDA tests (cudf)" - timeout-minutes: 30 - runs-on: runs-on=${{ github.run_id }}/runner=gpu/tag=cuda-test-cudf - steps: - - uses: runs-on/action@v2 - with: - sccache: s3 - - name: Display NVIDIA SMI details - run: | - nvidia-smi - nvidia-smi -L - nvidia-smi -q -d Memory - - uses: actions/checkout@v6 - - uses: ./.github/actions/setup-rust - with: - repo-token: ${{ secrets.GITHUB_TOKEN }} - - name: Build cudf test library - run: cargo build --profile ci --locked -p vortex-test-e2e-cuda --target x86_64-unknown-linux-gnu - - name: Download and run cudf-test-harness - run: | - curl -fsSL https://github.com/vortex-data/cudf-test-harness/releases/latest/download/cudf-test-harness-x86_64.tar.gz | tar -xz - cd cudf-test-harness-x86_64 - compute-sanitizer --tool memcheck --error-exitcode 1 ./cudf-test-harness check $GITHUB_WORKSPACE/target/x86_64-unknown-linux-gnu/ci/libvortex_test_e2e_cuda.so - rust-test-other: name: "Rust tests (${{ matrix.os }})" - timeout-minutes: 40 + timeout-minutes: 30 strategy: fail-fast: false matrix: @@ -671,7 +358,7 @@ jobs: if: github.repository == 'vortex-data/vortex' with: sccache: s3 - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - name: Setup (Windows) if: matrix.os == 'windows-x64' run: | @@ -710,64 +397,21 @@ jobs: ${{ github.repository == 'vortex-data/vortex' && format('runs-on={0}/pool=amd64-medium-pre-v2/tag=java', github.run_id) || 'ubuntu-latest' }} - timeout-minutes: 40 + timeout-minutes: 30 steps: - uses: runs-on/action@v2 if: github.repository == 'vortex-data/vortex' with: sccache: s3 - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - uses: ./.github/actions/setup-prebuild - run: ./gradlew test --parallel working-directory: ./java - bench-codspeed: - strategy: - matrix: - include: - - { shard: 1, name: "Core foundation", packages: "vortex-buffer vortex-error" } - - { shard: 2, name: "Arrays", packages: "vortex-array", features: "--features _test-harness" } - - { shard: 3, name: "Main library", packages: "vortex" } - - { shard: 4, name: "Encodings 1", packages: "vortex-alp vortex-bytebool vortex-datetime-parts" } - - { shard: 5, name: "Encodings 2", packages: "vortex-decimal-byte-parts vortex-fastlanes vortex-fsst", features: "--features _test-harness" } - - { shard: 6, name: "Encodings 3", packages: "vortex-pco vortex-runend vortex-sequence" } - - { shard: 7, name: "Encodings 4", packages: "vortex-sparse vortex-zigzag vortex-zstd" } - - { shard: 8, name: "Storage formats", packages: "vortex-flatbuffers vortex-proto vortex-btrblocks" } - name: "Benchmark with Codspeed (Shard #${{ matrix.shard }})" - timeout-minutes: 40 - runs-on: >- - ${{ github.repository == 'vortex-data/vortex' - && format('runs-on={0}/runner=amd64-medium/image=ubuntu24-full-x64-pre-v2/tag=bench-codspeed-{1}', github.run_id, matrix.shard) - || 'ubuntu-latest' }} - steps: - - uses: runs-on/action@v2 - if: github.repository == 'vortex-data/vortex' - with: - sccache: s3 - - uses: actions/checkout@v6 - - name: Setup benchmark environment - run: sudo bash scripts/setup-benchmark.sh - - uses: ./.github/actions/setup-prebuild - - uses: ./.github/actions/system-info - - name: Install Codspeed - uses: taiki-e/cache-cargo-install-action@66c9585ef5ca780ee69399975a5e911f47905995 - with: - tool: cargo-codspeed - - name: Build benchmarks - env: - RUSTFLAGS: "-C target-feature=+avx2" - run: cargo codspeed build ${{ matrix.features }} $(printf -- '-p %s ' ${{ matrix.packages }}) --profile bench - - name: Run benchmarks - uses: CodSpeedHQ/action@d872884a306dd4853acf0f584f4b706cf0cc72a2 - with: - run: bash scripts/bench-taskset.sh cargo codspeed run - token: ${{ secrets.CODSPEED_TOKEN }} - mode: "simulation" - license-check-and-audit-check: name: License Check and Audit Check runs-on: ubuntu-latest - timeout-minutes: 40 + timeout-minutes: 30 strategy: matrix: checks: @@ -776,14 +420,14 @@ jobs: # Prevent sudden announcement of a new advisory from failing ci: continue-on-error: ${{ matrix.checks == 'advisories' }} steps: - - uses: actions/checkout@v6 - - uses: EmbarkStudios/cargo-deny-action@v2 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + - uses: EmbarkStudios/cargo-deny-action@91bf2b620e09e18d6eb78b92e7861937469acedb # v2 with: command: check ${{ matrix.checks }} cxx-test: name: "C++ build" - timeout-minutes: 40 + timeout-minutes: 30 runs-on: >- ${{ github.repository == 'vortex-data/vortex' && format('runs-on={0}/runner=amd64-medium/image=ubuntu24-full-x64-pre-v2/tag=cxx-build', github.run_id) @@ -793,7 +437,7 @@ jobs: if: github.repository == 'vortex-data/vortex' with: sccache: s3 - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - uses: ./.github/actions/setup-prebuild - name: Build and run C++ unit tests run: | @@ -816,12 +460,13 @@ jobs: ${{ github.repository == 'vortex-data/vortex' && format('runs-on={0}/runner=amd64-medium/image=ubuntu24-full-x64-pre-v2/tag=sql-logic-test', github.run_id) || 'ubuntu-latest' }} + timeout-minutes: 30 steps: - uses: runs-on/action@v2 if: github.repository == 'vortex-data/vortex' with: sccache: s3 - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - uses: ./.github/actions/setup-prebuild - name: Run sqllogictest tests run: | @@ -834,9 +479,9 @@ jobs: wasm-integration: name: "WASM integration smoke test" runs-on: ubuntu-latest - timeout-minutes: 40 + timeout-minutes: 30 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - uses: ./.github/actions/setup-rust with: repo-token: ${{ secrets.GITHUB_TOKEN }} @@ -851,42 +496,19 @@ jobs: - run: wasmer run ./target/wasm32-wasip1/ci/wasm-test.wasm working-directory: ./wasm-test - miri: - name: "Rust tests (miri)" - runs-on: >- - ${{ github.repository == 'vortex-data/vortex' - && format('runs-on={0}/runner=amd64-medium/image=ubuntu24-full-x64-pre-v2/tag=rust-miri', github.run_id) - || 'ubuntu-latest' }} - timeout-minutes: 40 - env: - MIRIFLAGS: -Zmiri-strict-provenance -Zmiri-symbolic-alignment-check -Zmiri-disable-isolation -Zmiri-env-forward=RUST_BACKTRACE - RUSTFLAGS: "-A warnings --cfg vortex_nightly" - RUST_BACKTRACE: full - steps: - - uses: runs-on/action@v2 - if: github.repository == 'vortex-data/vortex' - with: - sccache: s3 - - uses: actions/checkout@v6 - - uses: ./.github/actions/setup-prebuild - - name: Install nightly with miri - run: rustup toolchain install $NIGHTLY_TOOLCHAIN --component rust-src,rustfmt,clippy,miri - - name: Run Miri - run: cargo +$NIGHTLY_TOOLCHAIN miri nextest run --no-fail-fast -p vortex-buffer -p vortex-ffi - generated-files: name: "Check generated source files are up to date" runs-on: >- ${{ github.repository == 'vortex-data/vortex' && format('runs-on={0}/runner=amd64-medium/image=ubuntu24-full-x64-pre-v2/tag=generated-files', github.run_id) || 'ubuntu-latest' }} - timeout-minutes: 40 + timeout-minutes: 30 steps: - uses: runs-on/action@v2 if: github.repository == 'vortex-data/vortex' with: sccache: s3 - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - uses: ./.github/actions/setup-prebuild - name: Install nightly for cbindgen macro expansion run: rustup toolchain install $NIGHTLY_TOOLCHAIN @@ -925,7 +547,7 @@ jobs: if: github.repository == 'vortex-data/vortex' with: sccache: s3 - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - uses: ./.github/actions/setup-prebuild - name: "regenerate FFI header file" run: | @@ -937,62 +559,3 @@ jobs: cmake -Bbuild -DRUST_BUILD_PROFILE=ci cmake --build build -j $(nproc) ctest --test-dir build -j $(nproc) - - check-java-publish-build: - runs-on: ${{ matrix.target.runs-on }} - container: - image: "ubuntu:20.04" - timeout-minutes: 40 - strategy: - fail-fast: false - matrix: - target: - - { os: ubuntu, runs-on: "ubuntu-24.04-arm", target: aarch64-unknown-linux-gnu } - - { os: ubuntu, runs-on: "ubuntu-24.04", target: x86_64-unknown-linux-gnu } - steps: - - uses: actions/checkout@v6 - with: - fetch-depth: 0 - - uses: ./.github/actions/prepare-java-linux - - uses: actions/setup-java@v5 - with: - distribution: "corretto" - java-version: "17" - - uses: ./.github/actions/setup-rust - with: - targets: ${{ matrix.target.target }} - repo-token: ${{ secrets.GITHUB_TOKEN }} - enable-sccache: "false" - - run: cargo build --profile ci --package vortex-jni - - compat-check: - name: "Compat check" - uses: ./.github/workflows/compat-validation.yml - with: - mode: last - - rust-publish-dry-run: - name: "Rust publish dry-run" - timeout-minutes: 40 - runs-on: >- - ${{ github.repository == 'vortex-data/vortex' - && format('runs-on={0}/runner=amd64-xsmall/image=ubuntu24-full-x64-pre-v2/tag=rust-publish-dry-run', github.run_id) - || 'ubuntu-latest' }} - steps: - - uses: runs-on/action@v2 - if: github.repository == 'vortex-data/vortex' - with: - sccache: s3 - - uses: actions/checkout@v6 - - uses: ./.github/actions/setup-prebuild - - name: Install cargo-edit - uses: taiki-e/cache-cargo-install-action@66c9585ef5ca780ee69399975a5e911f47905995 - with: - tool: cargo-edit - - name: Set Version - run: | - # This is mostly a dummy version, we don't actually publish anything but it cannot exist in crates.io - cargo set-version --workspace 0.100000.0 - - name: Test Release - run: | - cargo publish --dry-run --no-verify --allow-dirty --workspace diff --git a/.github/workflows/claude-review.yml b/.github/workflows/claude-review.yml new file mode 100644 index 00000000000..7802904c99f --- /dev/null +++ b/.github/workflows/claude-review.yml @@ -0,0 +1,259 @@ +name: Claude Code (Review) + +# Setup +# - This workflow does not need the GitHub App private key. +# - Do not attach the `claude-automation` environment here. +# - Store `CLAUDE_CODE_OAUTH_TOKEN` as a repository or organization Actions secret. +# - Create a repository or organization Actions variable: +# - CLAUDE_APP_LOGIN +# Set this to the bot login for the GitHub App, usually `[bot]`. +# - It may use the default GITHUB_TOKEN to read repository data and post review +# comments, but it must never be able to push commits or open branches. +# +# Why this workflow exists separately +# - PR review traffic is a different trust boundary from issue automation. +# - This workflow is intentionally read-only with respect to repository contents. +# - Fork PRs are refused outright. We do not "promote" or manually bless fork +# content into Claude. If a contributor wants Claude to implement something, +# a maintainer should restate the task on an issue and use claude-write.yml. +# - PR conversation comments on PRs already opened by the Claude App are handled by +# claude-write.yml so maintainers can ask Claude to make follow-up changes there. + +concurrency: + # `issue_comment` events on PRs expose the PR number via `github.event.issue.number`, + # so this falls back there when `github.event.pull_request.number` is unset. + group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.event.issue.number || github.run_id }} + cancel-in-progress: true + +on: + issue_comment: + types: [created] + pull_request_review_comment: + types: [created] + pull_request_review: + types: [submitted] + +jobs: + gate: + name: Gate PR Trigger + runs-on: ubuntu-latest + permissions: + contents: read + issues: read + pull-requests: read + outputs: + should_run: ${{ steps.gate.outputs.should_run }} + reason: ${{ steps.gate.outputs.reason }} + pull_number: ${{ steps.gate.outputs.pull_number }} + checkout_ref: ${{ steps.gate.outputs.checkout_ref }} + actor_has_write: ${{ steps.gate.outputs.actor_has_write }} + steps: + - name: Check whether this PR event is allowed to reach Claude + id: gate + uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9 + with: + github-token: ${{ github.token }} + script: | + const sender = context.payload.sender?.login ?? ''; + const senderType = context.payload.sender?.type ?? ''; + const trustedClaudeLogin = process.env.CLAUDE_APP_LOGIN ?? ''; + + async function getPermission(username) { + try { + const { data } = await github.rest.repos.getCollaboratorPermissionLevel({ + owner: context.repo.owner, + repo: context.repo.repo, + username, + }); + return data.permission; + } catch (error) { + if (error.status === 404) { + return 'none'; + } + throw error; + } + } + + let mentioned = false; + let pullNumber = null; + + if (context.eventName === 'issue_comment') { + mentioned = (context.payload.comment?.body ?? '').includes('@claude'); + if (context.payload.issue?.pull_request) { + pullNumber = context.payload.issue.number; + } + } else if (context.eventName === 'pull_request_review_comment') { + mentioned = (context.payload.comment?.body ?? '').includes('@claude'); + pullNumber = context.payload.pull_request?.number ?? null; + } else if (context.eventName === 'pull_request_review') { + mentioned = (context.payload.review?.body ?? '').includes('@claude'); + pullNumber = context.payload.pull_request?.number ?? null; + } + + if (pullNumber) { + core.setOutput('pull_number', String(pullNumber)); + } + + let reason = ''; + + if (!mentioned) { + reason = 'not_mentioned'; + } else if (!pullNumber) { + reason = 'not_a_pr_event'; + } else if (senderType === 'Bot') { + reason = 'bot_sender_refused'; + } + + let actorHasWrite = 'false'; + if (!reason) { + const permission = await getPermission(sender); + core.setOutput('actor_permission', permission); + actorHasWrite = ['admin', 'maintain', 'write'].includes(permission) ? 'true' : 'false'; + if (actorHasWrite !== 'true') { + reason = 'actor_lacks_write'; + } + } + + if (!reason && context.eventName === 'issue_comment' && !trustedClaudeLogin) { + reason = 'missing_claude_app_login'; + } + + if (!reason) { + let pr = null; + const response = await github.rest.pulls.get({ + owner: context.repo.owner, + repo: context.repo.repo, + pull_number: pullNumber, + }); + pr = response.data; + + const headRepo = pr.head.repo; + const isFork = !headRepo || headRepo.fork || headRepo.full_name !== `${context.repo.owner}/${context.repo.repo}`; + core.setOutput('checkout_ref', pr.head.sha); + + if (isFork) { + reason = 'fork_pr_refused'; + } else if ( + context.eventName === 'issue_comment' && + trustedClaudeLogin && + (pr.user?.login ?? '') === trustedClaudeLogin + ) { + reason = 'claude_pr_uses_write_workflow'; + } + + if (!reason) { + const files = await github.paginate(github.rest.pulls.listFiles, { + owner: context.repo.owner, + repo: context.repo.repo, + pull_number: pullNumber, + per_page: 100, + }); + // Refuse PRs that modify `.github/` because workflow files define the + // automation policy this review job is enforcing. + if (files.some(f => f.filename.startsWith('.github/'))) { + reason = 'modifies_github_dir'; + } + } + } + + core.setOutput('actor_has_write', actorHasWrite); + core.setOutput('should_run', !reason ? 'true' : 'false'); + core.setOutput('reason', reason || 'allowed'); + env: + CLAUDE_APP_LOGIN: ${{ vars.CLAUDE_APP_LOGIN }} + + explain-refusal: + name: Explain Refusal + needs: gate + if: | + needs.gate.outputs.actor_has_write == 'true' && + ( + needs.gate.outputs.reason == 'fork_pr_refused' || + needs.gate.outputs.reason == 'modifies_github_dir' || + needs.gate.outputs.reason == 'missing_claude_app_login' + ) + runs-on: ubuntu-latest + permissions: + issues: write + pull-requests: write + steps: + - name: Comment with the refusal reason + uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9 + with: + script: | + const reason = ${{ toJSON(needs.gate.outputs.reason) }}; + const messages = { + fork_pr_refused: [ + "Claude review automation is disabled for fork pull requests.", + "", + "Why:", + "- fork content is untrusted input", + "- this repository does not allow Claude to run against fork content", + "- there is no promotion path for forks", + "", + "If maintainers want Claude to implement a change, restate the task on an issue and use the issue-driven Claude workflow instead." + ].join("\n"), + modifies_github_dir: [ + "Claude review automation is disabled for pull requests that modify `.github/` files.", + "", + "Why:", + "- workflow and action files are part of the automation policy", + "- this review workflow refuses to evaluate automation changes from the same PR", + "", + "Ask a human reviewer to inspect workflow changes directly." + ].join("\n"), + missing_claude_app_login: [ + "Claude review automation is misconfigured for issue-comment triggers.", + "", + "Why:", + "- `CLAUDE_APP_LOGIN` is not set", + "- without that bot login, the review workflow cannot safely route comments on Claude-owned PRs to the write workflow", + "", + "Set the `CLAUDE_APP_LOGIN` Actions variable, then retry the command." + ].join("\n"), + }; + + await github.rest.issues.createComment({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: Number(${{ needs.gate.outputs.pull_number }}), + body: messages[reason], + }); + + claude: + name: Run Claude PR Review + needs: gate + if: needs.gate.outputs.should_run == 'true' + runs-on: ubuntu-latest + timeout-minutes: 45 + permissions: + contents: read + issues: write + pull-requests: write + actions: read + steps: + - name: Checkout same-repo PR contents + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + ref: ${{ needs.gate.outputs.checkout_ref }} + fetch-depth: 1 + # Keep git credentials out of the workspace. This workflow is review-only + # and should never push changes. + persist-credentials: false + + - name: Run Claude Code in review-only mode + id: claude + uses: anthropics/claude-code-action@6cad158a175744eb2e76f7f5fd108ec63145598c + with: + claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} + # This workflow deliberately uses the built-in token because it only needs + # to read repository state and post review comments. It cannot write repo + # contents, and it has no access to the GitHub App private key. + github_token: ${{ github.token }} + + additional_permissions: | + actions: read + + claude_args: | + --allowedTools "Read,Grep,Glob,Bash(git diff:*),Bash(git show:*),Bash(git log:*),Bash(head:*),Bash(jq:*),Bash(rg:*)" + --system-prompt "You are the repository's read-only Claude review workflow. Review the current same-repo pull request and respond in GitHub. Never modify files, never create commits, never push branches, and never open or update pull requests. Fork pull requests are blocked before this job starts." diff --git a/.github/workflows/claude-write.yml b/.github/workflows/claude-write.yml new file mode 100644 index 00000000000..252bb8f9382 --- /dev/null +++ b/.github/workflows/claude-write.yml @@ -0,0 +1,213 @@ +name: Claude Code (Write) + +# Setup +# 1. Create a dedicated GitHub App and install it only on this repository. +# 2. Grant the App only: +# - Contents: Read & write +# - Issues: Read & write +# - Pull requests: Read & write +# 3. Create a GitHub Actions environment named `claude-automation`. +# 4. Store these environment secrets in `claude-automation`: +# - APP_ID +# - APP_PRIVATE_KEY +# - CLAUDE_CODE_OAUTH_TOKEN +# 5. Create a repository or organization Actions variable: +# - CLAUDE_APP_LOGIN +# Set this to the bot login for the GitHub App, usually `[bot]`. +# +# Why this workflow exists separately +# - This is the only workflow that can read the GitHub App private key. +# - It is primarily issue-driven. The only PR path it allows is a follow-up comment +# on a same-repo PR that was already opened by the Claude GitHub App. +# - Claude uses a short-lived GitHub App installation token, not GITHUB_TOKEN, so +# PRs opened by Claude trigger normal `pull_request` workflows. +# - A gate job runs before the environment is attached so untrusted events are +# rejected before the private key is exposed. +# - GitHub exposes the author of an app-created PR as the app's bot login rather +# than the numeric App ID in the PR payload, so the gate checks `CLAUDE_APP_LOGIN` +# instead of comparing directly to `APP_ID`. + +concurrency: + group: ${{ github.workflow }}-${{ github.event.issue.number || github.run_id }} + cancel-in-progress: true + +on: + issues: + types: [opened, assigned] + issue_comment: + types: [created] + +jobs: + gate: + name: Gate Issue Trigger + runs-on: ubuntu-latest + permissions: + contents: read + issues: read + pull-requests: read + outputs: + should_run: ${{ steps.gate.outputs.should_run }} + reason: ${{ steps.gate.outputs.reason }} + checkout_ref: ${{ steps.gate.outputs.checkout_ref }} + steps: + - name: Check whether this event is allowed to reach Claude + id: gate + uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9 + with: + github-token: ${{ github.token }} + script: | + const sender = context.payload.sender?.login ?? ''; + const senderType = context.payload.sender?.type ?? ''; + const trustedClaudeLogin = process.env.CLAUDE_APP_LOGIN ?? ''; + + async function getPermission(username) { + try { + const { data } = await github.rest.repos.getCollaboratorPermissionLevel({ + owner: context.repo.owner, + repo: context.repo.repo, + username, + }); + return data.permission; + } catch (error) { + if (error.status === 404) { + return 'none'; + } + throw error; + } + } + + let mentioned = false; + let reason = ''; + let checkoutRef = context.payload.repository?.default_branch ?? ''; + + if (context.eventName === 'issues') { + const title = context.payload.issue?.title ?? ''; + const body = context.payload.issue?.body ?? ''; + mentioned = title.includes('@claude') || body.includes('@claude'); + } else if (context.eventName === 'issue_comment') { + const body = context.payload.comment?.body ?? ''; + mentioned = body.includes('@claude'); + + if (context.payload.issue?.pull_request) { + const response = await github.rest.pulls.get({ + owner: context.repo.owner, + repo: context.repo.repo, + pull_number: context.payload.issue.number, + }); + const pr = response.data; + + checkoutRef = pr.head.sha; + + const headRepo = pr.head.repo; + const isFork = !headRepo || headRepo.fork || headRepo.full_name !== `${context.repo.owner}/${context.repo.repo}`; + if (isFork) { + reason = 'fork_pr_refused'; + } else if (!trustedClaudeLogin) { + reason = 'missing_claude_app_login'; + } else if ((pr.user?.login ?? '') !== trustedClaudeLogin) { + reason = 'pr_not_owned_by_claude_app'; + } + + if (!reason) { + const files = await github.paginate(github.rest.pulls.listFiles, { + owner: context.repo.owner, + repo: context.repo.repo, + pull_number: context.payload.issue.number, + per_page: 100, + }); + // Refuse workflow changes so write-capable automation never acts on + // `.github/` modifications from the same pull request. + if (files.some(f => f.filename.startsWith('.github/'))) { + reason = 'modifies_github_dir'; + } + } + } + } + + if (!reason && !mentioned) { + reason = 'not_mentioned'; + } + + if (!reason && senderType === 'Bot') { + reason = 'bot_sender_refused'; + } + + if (!reason) { + const permission = await getPermission(sender); + core.setOutput('actor_permission', permission); + if (!['admin', 'maintain', 'write'].includes(permission)) { + reason = 'actor_lacks_write'; + } + } + + core.setOutput('checkout_ref', checkoutRef); + core.setOutput('should_run', !reason ? 'true' : 'false'); + core.setOutput('reason', reason || 'allowed'); + env: + CLAUDE_APP_LOGIN: ${{ vars.CLAUDE_APP_LOGIN }} + + claude: + name: Run Claude Code + needs: gate + if: needs.gate.outputs.should_run == 'true' + runs-on: ubuntu-latest + timeout-minutes: 60 + + environment: + # The App private key lives only in this environment so only this single job + # can mint a GitHub App installation token. + name: claude-automation + deployment: false + + permissions: + contents: read + issues: read + pull-requests: read + actions: read + id-token: write + + steps: + - name: Checkout repository + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + ref: ${{ needs.gate.outputs.checkout_ref }} + fetch-depth: 0 + # Do not leave the built-in GITHUB_TOKEN in git config. Claude should use + # only the GitHub App token generated below so its PRs trigger CI normally. + persist-credentials: false + + - name: Generate short-lived GitHub App token + id: app-token + uses: actions/create-github-app-token@f8d387b68d61c58ab83c6c016672934102569859 + with: + app-id: ${{ secrets.APP_ID }} + private-key: ${{ secrets.APP_PRIVATE_KEY }} + permission-contents: write + permission-issues: write + permission-pull-requests: write + + - name: Setup Rust toolchain + uses: ./.github/actions/setup-rust + with: + enable-sccache: "false" + + - name: Install uv + uses: spiraldb/actions/.github/actions/setup-uv@a746510eafaa926484c354541cfc49b2ec06cc63 # 0.18.6 + with: + sync: false + + - name: Run Claude Code + id: claude + uses: anthropics/claude-code-action@6cad158a175744eb2e76f7f5fd108ec63145598c + with: + claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} + github_token: ${{ steps.app-token.outputs.token }} + + # Claude may inspect CI state on related PRs, but this workflow is otherwise + # issue-driven and is the only place with write-capable GitHub credentials. + additional_permissions: | + actions: read + + claude_args: | + --allowedTools "Bash(cargo nextest:*),Bash(cargo check:*),Bash(cargo clippy:*),Bash(cargo fmt:*),Bash(uv run:*)" + --system-prompt "You are the repository's write-capable Claude workflow. You run only from trusted issue traffic and from trusted PR conversation comments on same-repo pull requests that were previously opened by the repository's Claude GitHub App. Create or update branches and pull requests using the provided GitHub App token. Do not use fork pull request content because those runs are blocked before this job starts." diff --git a/.github/workflows/claude.yml b/.github/workflows/claude.yml deleted file mode 100644 index 448cc5e2c50..00000000000 --- a/.github/workflows/claude.yml +++ /dev/null @@ -1,65 +0,0 @@ -name: Claude Code - -concurrency: - # We shouldn't have multiple instances of Claude running on the same PR. - group: ${{ github.workflow }}-${{ github.head_ref || github.ref }} - cancel-in-progress: ${{ github.event_name == 'pull_request' }} - -on: - issue_comment: - types: [created] - pull_request_review_comment: - types: [created] - issues: - types: [opened, assigned] - pull_request_review: - types: [submitted] - -jobs: - claude: - if: | - (github.event_name == 'issue_comment' && contains(github.event.comment.body, '@claude')) || - (github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude')) || - (github.event_name == 'pull_request_review' && contains(github.event.review.body, '@claude')) || - (github.event_name == 'issues' && (contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude'))) - runs-on: ubuntu-latest - permissions: - contents: read - pull-requests: read - issues: read - id-token: write - actions: read # Required for Claude to read CI results on PRs - steps: - - name: Checkout repository - uses: actions/checkout@v6 - - uses: ./.github/actions/setup-rust - with: - repo-token: ${{ secrets.GITHUB_TOKEN }} - - name: Install uv - uses: spiraldb/actions/.github/actions/setup-uv@0.18.5 - with: - sync: false - - - name: Run Claude Code - id: claude - uses: anthropics/claude-code-action@v1 - with: - claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} - - # This is an optional setting that allows Claude to read CI results on PRs - additional_permissions: | - actions: read - - # Optional: Customize the trigger phrase (default: @claude) - # trigger_phrase: "/claude" - - # Optional: Trigger when specific user is assigned to an issue - # assignee_trigger: "claude-bot" - - claude_args: | - --allowedTools "Bash(cargo nextest:*),Bash(cargo check:*),Bash(cargo clippy:*),Bash(cargo fmt:*),Bash(uv run:*)" - --system-prompt "You have been granted tools for editing files and running cargo commands (cargo nextest, cargo check, cargo clippy, cargo fmt) and uv for running pytest (e.g. via uv run --all-packages pytest)" - - # Optional: Custom environment variables for Claude - # claude_env: | - # NODE_ENV: test diff --git a/.github/workflows/close-fixed-fuzzer-issues.yml b/.github/workflows/close-fixed-fuzzer-issues.yml index 44d51663bb5..bd365070e3c 100644 --- a/.github/workflows/close-fixed-fuzzer-issues.yml +++ b/.github/workflows/close-fixed-fuzzer-issues.yml @@ -29,14 +29,14 @@ jobs: ${{ github.repository == 'vortex-data/vortex' && format('runs-on={0}/runner=arm64-medium/disk=large/tag=fuzzer-cleanup-{1}', github.run_id, matrix.target) || 'ubuntu-latest' }} - timeout-minutes: 60 + timeout-minutes: 30 steps: - uses: runs-on/action@v2 if: github.repository == 'vortex-data/vortex' with: sccache: s3 - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - uses: ./.github/actions/setup-rust with: diff --git a/.github/workflows/codspeed.yml b/.github/workflows/codspeed.yml new file mode 100644 index 00000000000..ff42ff589b9 --- /dev/null +++ b/.github/workflows/codspeed.yml @@ -0,0 +1,106 @@ +name: Codspeed Benchmarking + +# Concurrency control: +# - PRs: new commits on a feature branch will cancel in-progress (outdated) runs. +# - Push to develop: runs queue sequentially, never cancelled. This allows us to have benchmarks +# run on every commit for our benchmarks website. +# - `workflow_dispatch`: groups by branch and queues if run on develop. +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: ${{ github.ref != 'refs/heads/develop' }} +on: + push: + branches: [develop] + pull_request: { } + workflow_dispatch: { } + +permissions: + contents: read + +env: + CARGO_TERM_COLOR: always + RUST_BACKTRACE: 1 + NIGHTLY_TOOLCHAIN: nightly-2026-02-05 + +jobs: + bench-codspeed: + strategy: + matrix: + include: + - { shard: 1, name: "Core foundation", packages: "vortex-buffer vortex-error" } + - { shard: 2, name: "Arrays", packages: "vortex-array", features: "--features _test-harness" } + - { shard: 3, name: "Main library", packages: "vortex" } + - { shard: 4, name: "Encodings 1", packages: "vortex-alp vortex-bytebool vortex-datetime-parts" } + - { shard: 5, name: "Encodings 2", packages: "vortex-decimal-byte-parts vortex-fastlanes vortex-fsst", features: "--features _test-harness" } + - { shard: 6, name: "Encodings 3", packages: "vortex-pco vortex-runend vortex-sequence" } + - { shard: 7, name: "Encodings 4", packages: "vortex-sparse vortex-zigzag vortex-zstd" } + - { shard: 8, name: "Storage formats", packages: "vortex-flatbuffers vortex-proto vortex-btrblocks" } + name: "Benchmark with Codspeed (Shard #${{ matrix.shard }})" + timeout-minutes: 30 + runs-on: >- + ${{ github.repository == 'vortex-data/vortex' + && format('runs-on={0}/runner=amd64-medium/image=ubuntu24-full-x64-pre-v2/tag=bench-codspeed-{1}', github.run_id, matrix.shard) + || 'ubuntu-latest' }} + steps: + - uses: runs-on/action@v2 + if: github.repository == 'vortex-data/vortex' + with: + sccache: s3 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + - name: Setup benchmark environment + run: sudo bash scripts/setup-benchmark.sh + - uses: ./.github/actions/setup-prebuild + - uses: ./.github/actions/system-info + - name: Install Codspeed + uses: taiki-e/cache-cargo-install-action@66c9585ef5ca780ee69399975a5e911f47905995 + with: + tool: cargo-codspeed + - name: Build benchmarks + env: + RUSTFLAGS: "-C target-feature=+avx2" + run: cargo codspeed build ${{ matrix.features }} $(printf -- '-p %s ' ${{ matrix.packages }}) --profile bench + - name: Run benchmarks + uses: CodSpeedHQ/action@d872884a306dd4853acf0f584f4b706cf0cc72a2 + with: + run: bash scripts/bench-taskset.sh cargo codspeed run + token: ${{ secrets.CODSPEED_TOKEN }} + mode: "simulation" + + bench-codspeed-cuda: + if: github.repository == 'vortex-data/vortex' + strategy: + matrix: + include: + - { shard: 1, name: "Bitpacked", benches: "bitpacked_cuda" } + - { shard: 2, name: "Dynamic dispatch", benches: "dynamic_dispatch_cuda" } + - { shard: 3, name: "Standalone kernels", benches: "alp_cuda date_time_parts_cuda dict_cuda runend_cuda" } + name: "Benchmark with Codspeed (CUDA Shard #${{ matrix.shard }} - ${{ matrix.name }})" + timeout-minutes: 30 + runs-on: runs-on=${{ github.run_id }}/family=g5/image=ubuntu24-gpu-x64/tag=bench-codspeed-cuda-${{ matrix.shard }} + steps: + - uses: runs-on/action@v2 + with: + sccache: s3 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + - uses: ./.github/actions/setup-rust + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + - name: Display NVIDIA SMI details + run: | + nvidia-smi + nvidia-smi -L + nvidia-smi -q -d Memory + - name: Install Codspeed + uses: taiki-e/cache-cargo-install-action@66c9585ef5ca780ee69399975a5e911f47905995 + with: + tool: cargo-codspeed + - name: Build benchmarks + run: cargo codspeed build -m walltime $(printf -- '--bench %s ' ${{ matrix.benches }}) --profile bench + - name: Run benchmarks + uses: CodSpeedHQ/action@d872884a306dd4853acf0f584f4b706cf0cc72a2 + env: + CARGO_MANIFEST_DIR: ${{ github.workspace }}/vortex-cuda + with: + run: cargo codspeed run $(printf -- '--bench %s ' ${{ matrix.benches }}) + token: ${{ secrets.CODSPEED_TOKEN }} + mode: "walltime" diff --git a/.github/workflows/compat-gen-upload.yml b/.github/workflows/compat-gen-upload.yml index da17ec9a28c..2e67341bcee 100644 --- a/.github/workflows/compat-gen-upload.yml +++ b/.github/workflows/compat-gen-upload.yml @@ -27,6 +27,7 @@ jobs: ${{ github.repository == 'vortex-data/vortex' && format('runs-on={0}/runner=amd64-medium/image=ubuntu24-full-x64-pre-v2/tag=compat-gen-dry-run', github.run_id) || 'ubuntu-latest' }} + timeout-minutes: 30 permissions: id-token: write contents: read @@ -37,13 +38,13 @@ jobs: if: github.repository == 'vortex-data/vortex' with: sccache: s3 - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: fetch-depth: 0 - uses: ./.github/actions/setup-prebuild - name: Configure AWS credentials - uses: aws-actions/configure-aws-credentials@v6 + uses: aws-actions/configure-aws-credentials@ec61189d14ec14c8efccab744f656cffd0e33f37 # v6 with: role-to-assume: "arn:aws:iam::245040174862:role/GitHubCompatUploadRole" aws-region: us-east-1 @@ -77,6 +78,7 @@ jobs: ${{ github.repository == 'vortex-data/vortex' && format('runs-on={0}/runner=amd64-medium/image=ubuntu24-full-x64-pre-v2/tag=compat-gen-upload', github.run_id) || 'ubuntu-latest' }} + timeout-minutes: 30 environment: compat-upload permissions: id-token: write @@ -86,13 +88,13 @@ jobs: if: github.repository == 'vortex-data/vortex' with: sccache: s3 - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: fetch-depth: 0 - uses: ./.github/actions/setup-prebuild - name: Configure AWS credentials - uses: aws-actions/configure-aws-credentials@v6 + uses: aws-actions/configure-aws-credentials@ec61189d14ec14c8efccab744f656cffd0e33f37 # v6 with: role-to-assume: "arn:aws:iam::245040174862:role/GitHubCompatUploadRole" aws-region: us-east-1 diff --git a/.github/workflows/compat-validation.yml b/.github/workflows/compat-validation.yml index f479420cd48..7be13aec5b3 100644 --- a/.github/workflows/compat-validation.yml +++ b/.github/workflows/compat-validation.yml @@ -27,13 +27,13 @@ jobs: ${{ github.repository == 'vortex-data/vortex' && format('runs-on={0}/runner=amd64-medium/image=ubuntu24-full-x64-pre-v2/tag=compat-validation', github.run_id) || 'ubuntu-latest' }} - timeout-minutes: 120 + timeout-minutes: 30 steps: - uses: runs-on/action@v2 if: github.repository == 'vortex-data/vortex' with: sccache: s3 - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - uses: ./.github/actions/setup-prebuild - name: Run compat tests run: | diff --git a/.github/workflows/cuda.yaml b/.github/workflows/cuda.yaml new file mode 100644 index 00000000000..24eb9add90d --- /dev/null +++ b/.github/workflows/cuda.yaml @@ -0,0 +1,154 @@ +name: CUDA + +# Concurrency control: +# - PRs: new commits on a feature branch will cancel in-progress (outdated) runs. +# - Push to develop: runs queue sequentially, never cancelled. +# - `workflow_dispatch`: groups by branch and queues if run on develop. +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: ${{ github.ref != 'refs/heads/develop' }} +on: + push: + branches: [develop] + pull_request: { } + workflow_dispatch: { } + +permissions: + contents: read + +env: + CARGO_TERM_COLOR: always + RUST_BACKTRACE: 1 + +jobs: + cuda-build-lint: + if: github.repository == 'vortex-data/vortex' + name: "CUDA build & lint" + timeout-minutes: 30 + runs-on: runs-on=${{ github.run_id }}/runner=gpu/tag=cuda-build + steps: + - uses: runs-on/action@v2 + with: + sccache: s3 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + - uses: ./.github/actions/setup-rust + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + - uses: ./.github/actions/check-rebuild + with: + command: >- + cargo build --profile ci --locked --all-features --all-targets + -p vortex-cuda -p vortex-cub -p vortex-nvcomp + -p gpu-scan-cli -p vortex-test-e2e-cuda + - name: Clippy CUDA crates + run: | + cargo clippy --profile ci --locked --all-features --all-targets \ + -p vortex-cuda \ + -p vortex-cub \ + -p vortex-nvcomp \ + -p gpu-scan-cli \ + -p vortex-test-e2e-cuda \ + -- -D warnings + + cuda-test: + if: github.repository == 'vortex-data/vortex' + name: "CUDA tests" + timeout-minutes: 30 + runs-on: runs-on=${{ github.run_id }}/runner=gpu/tag=cuda-tests + steps: + - uses: runs-on/action@v2 + with: + sccache: s3 + - name: Display NVIDIA SMI details + run: | + nvidia-smi + nvidia-smi -L + nvidia-smi -q -d Memory + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + - uses: ./.github/actions/setup-rust + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + - name: Install nextest + uses: taiki-e/install-action@cf525cb33f51aca27cd6fa02034117ab963ff9f1 # v2 + with: + tool: nextest + - name: Rust Tests + env: + FLAT_LAYOUT_INLINE_ARRAY_NODE: true + run: | + cargo nextest run \ + --cargo-profile ci \ + --locked \ + -p vortex-file \ + -p vortex-cuda \ + -p vortex-cub \ + -p vortex-nvcomp \ + -p vortex-test-e2e-cuda \ + --all-features \ + --no-fail-fast \ + --target x86_64-unknown-linux-gnu \ + --verbose + + cuda-test-sanitizer: + if: github.repository == 'vortex-data/vortex' + name: "CUDA tests (${{ matrix.sanitizer }})" + timeout-minutes: 30 + runs-on: runs-on=${{ github.run_id }}/runner=gpu/tag=cuda-test-sanitizer + strategy: + fail-fast: false + matrix: + include: + - sanitizer: memcheck + runner_flags: "--tool memcheck --leak-check=full --error-exitcode 1" + # TODO(joe): try to re-enable racecheck, it is hanging in CI. + # - sanitizer: racecheck + # runner_flags: "--tool racecheck --error-exitcode 1" + - sanitizer: synccheck + runner_flags: "--tool synccheck --error-exitcode 1" + - sanitizer: initcheck + runner_flags: "--tool initcheck --error-exitcode 1" + steps: + - uses: runs-on/action@v2 + with: + sccache: s3 + - name: Display NVIDIA SMI details + run: | + nvidia-smi + nvidia-smi -L + nvidia-smi -q -d Memory + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + - uses: ./.github/actions/setup-rust + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + - name: Build tests + run: cargo test --profile ci --locked -p vortex-cuda --all-features --target x86_64-unknown-linux-gnu --no-run + - name: "CUDA - ${{ matrix.sanitizer }}" + env: + CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUNNER: "compute-sanitizer ${{ matrix.runner_flags }}" + run: cargo test --profile ci --locked -p vortex-cuda --all-features --target x86_64-unknown-linux-gnu + + cuda-test-cudf: + if: github.repository == 'vortex-data/vortex' + name: "CUDA tests (cudf)" + timeout-minutes: 30 + runs-on: runs-on=${{ github.run_id }}/runner=gpu/tag=cuda-test-cudf + steps: + - uses: runs-on/action@v2 + with: + sccache: s3 + - name: Display NVIDIA SMI details + run: | + nvidia-smi + nvidia-smi -L + nvidia-smi -q -d Memory + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + - uses: ./.github/actions/setup-rust + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + - name: Build cudf test library + run: cargo build --profile ci --locked -p vortex-test-e2e-cuda --target x86_64-unknown-linux-gnu + - name: Download and run cudf-test-harness + run: | + curl -fsSL https://github.com/vortex-data/cudf-test-harness/releases/latest/download/cudf-test-harness-x86_64.tar.gz | tar -xz + cd cudf-test-harness-x86_64 + compute-sanitizer --tool memcheck --error-exitcode 1 ./cudf-test-harness check $GITHUB_WORKSPACE/target/x86_64-unknown-linux-gnu/ci/libvortex_test_e2e_cuda.so diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 6a7b6c47f4f..8cee65e3a9e 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -17,20 +17,20 @@ permissions: jobs: build: runs-on: ubuntu-latest - timeout-minutes: 120 + timeout-minutes: 30 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - uses: ./.github/actions/setup-rust with: repo-token: ${{ secrets.GITHUB_TOKEN }} enable-sccache: "false" - name: Set up JDK 17 - uses: actions/setup-java@v5 + uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5 with: java-version: "17" distribution: "temurin" - name: Install uv - uses: spiraldb/actions/.github/actions/setup-uv@0.18.5 + uses: spiraldb/actions/.github/actions/setup-uv@a746510eafaa926484c354541cfc49b2ec06cc63 # 0.18.6 with: sync: false prune-cache: false @@ -39,7 +39,7 @@ jobs: env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} - name: Install Doxygen - uses: ssciwr/doxygen-install@v2 + uses: ssciwr/doxygen-install@329d88f5a303066a5bd006db7516b1925b86350e # v2 - name: Generate Javadoc for Java projects run: | cd java @@ -55,7 +55,7 @@ jobs: uv run --all-packages make -C docs html - name: Upload static files as artifact id: deployment - uses: actions/upload-pages-artifact@v4 + uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5 with: path: docs/_build/html @@ -66,19 +66,19 @@ jobs: name: github-pages url: ${{ steps.deployment.outputs.page_url }} runs-on: ubuntu-latest - timeout-minutes: 120 + timeout-minutes: 10 needs: build steps: # Note, since we provide the job with a CloudFlare scoped API token, we run it in a separate job that doesn't # execute any repository code. - - uses: actions/download-artifact@v8 + - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: name: github-pages - name: Extract Pages Artifact run: mkdir docs && tar -xvf artifact.tar -C docs shell: bash - name: Upload artifacts to CloudFlare Pages - uses: cloudflare/wrangler-action@v3 + uses: cloudflare/wrangler-action@9acf94ace14e7dc412b076f2c5c20b8ce93c79cd # v3 with: apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }} accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }} diff --git a/.github/workflows/fuzz-coverage.yml b/.github/workflows/fuzz-coverage.yml index 8db58c1a5f6..0cc2e211e18 100644 --- a/.github/workflows/fuzz-coverage.yml +++ b/.github/workflows/fuzz-coverage.yml @@ -15,7 +15,7 @@ jobs: fail-fast: false matrix: fuzz_target: [array_ops, file_io, compress_roundtrip] - timeout-minutes: 60 + timeout-minutes: 120 runs-on: >- ${{ github.repository == 'vortex-data/vortex' && format('runs-on={0}/runner=arm64-medium/disk=large', github.run_id) @@ -26,7 +26,7 @@ jobs: with: sccache: s3 - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - uses: ./.github/actions/setup-rust with: @@ -75,8 +75,7 @@ jobs: - name: Generate coverage data run: | - RUSTFLAGS="--cfg vortex_nightly" \ - cargo +$NIGHTLY_TOOLCHAIN fuzz coverage --release --debug-assertions \ + cargo +$NIGHTLY_TOOLCHAIN fuzz coverage --release --debug-assertions \ ${{ matrix.fuzz_target }} \ -- -rss_limit_mb=4096 @@ -115,7 +114,7 @@ jobs: --ignore-filename-regex='(\.cargo|rustc|registry)' - name: Upload coverage report - uses: actions/upload-artifact@v7 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: name: ${{ matrix.fuzz_target }}-coverage path: coverage_html/ diff --git a/.github/workflows/fuzzer-fix-automation.yml b/.github/workflows/fuzzer-fix-automation.yml index 08618237c95..b45e0b866a4 100644 --- a/.github/workflows/fuzzer-fix-automation.yml +++ b/.github/workflows/fuzzer-fix-automation.yml @@ -32,7 +32,7 @@ jobs: github.event_name == 'workflow_dispatch' runs-on: ubuntu-latest - timeout-minutes: 90 + timeout-minutes: 120 permissions: contents: write @@ -42,7 +42,7 @@ jobs: steps: - name: Checkout repository - uses: actions/checkout@v6 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - name: Fetch issue details id: fetch_issue @@ -163,7 +163,7 @@ jobs: echo "Building fuzzer target: ${{ steps.extract.outputs.target }} (debug mode for faster build)" # Build the fuzzer target in debug mode (faster than release) - if RUSTFLAGS="--cfg vortex_nightly" cargo +$NIGHTLY_TOOLCHAIN fuzz build --dev --sanitizer=none "${{ steps.extract.outputs.target }}" 2>&1 | tee fuzzer_build.log; then + if cargo +$NIGHTLY_TOOLCHAIN fuzz build --dev --sanitizer=none "${{ steps.extract.outputs.target }}" 2>&1 | tee fuzzer_build.log; then echo "✅ Fuzzer target built successfully" echo "build_success=true" >> $GITHUB_OUTPUT else @@ -183,7 +183,7 @@ jobs: echo "Attempting to reproduce crash with fuzzer (debug mode)..." # Run fuzzer with crash file (debug mode, no sanitizer, full backtrace) - RUSTFLAGS="--cfg vortex_nightly" RUST_BACKTRACE=full timeout 30s cargo +$NIGHTLY_TOOLCHAIN fuzz run --dev --sanitizer=none "${{ steps.extract.outputs.target }}" "${{ steps.download.outputs.crash_file_path }}" -- -runs=1 -rss_limit_mb=0 2>&1 | tee crash_reproduction.log + RUST_BACKTRACE=full timeout 30s cargo +$NIGHTLY_TOOLCHAIN fuzz run --dev --sanitizer=none "${{ steps.extract.outputs.target }}" "${{ steps.download.outputs.crash_file_path }}" -- -runs=1 -rss_limit_mb=0 2>&1 | tee crash_reproduction.log FUZZ_EXIT_CODE=${PIPESTATUS[0]} @@ -236,7 +236,7 @@ jobs: CRASH_FILE: ${{ steps.extract.outputs.crash_file }} CRASH_FILE_PATH: ${{ steps.download.outputs.crash_file_path }} ARTIFACT_URL: ${{ steps.extract.outputs.artifact_url }} - uses: anthropics/claude-code-action@v1 + uses: anthropics/claude-code-action@567fe954a4527e81f132d87d1bdbcc94f7737434 # v1 with: claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }} github_token: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/labels.yml b/.github/workflows/labels.yml index a6d4e9364e3..4934dbad72e 100644 --- a/.github/workflows/labels.yml +++ b/.github/workflows/labels.yml @@ -13,12 +13,13 @@ jobs: check_changelog_label: name: Validate Changelog Label runs-on: ubuntu-latest + timeout-minutes: 10 permissions: pull-requests: read # Grant permission to read PR information steps: - name: Get PR Labels from API id: get_labels_api - uses: octokit/request-action@v2.4.0 # Use an action to make API requests + uses: octokit/request-action@b91aabaa861c777dcdb14e2387e30eddf04619ae # v3.0.0 # Use an action to make API requests with: route: GET /repos/{owner}/{repo}/issues/{pull_number}/labels owner: ${{ github.repository_owner }} @@ -33,6 +34,7 @@ jobs: run: | REQUIRED_LABELS=( "changelog/break" + "changelog/deprecation" "changelog/feature" "changelog/performance" "changelog/fix" diff --git a/.github/workflows/minimize_fuzz_corpus_workflow.yml b/.github/workflows/minimize_fuzz_corpus_workflow.yml index f4fcdbc39cb..e3b2a9c11e6 100644 --- a/.github/workflows/minimize_fuzz_corpus_workflow.yml +++ b/.github/workflows/minimize_fuzz_corpus_workflow.yml @@ -38,6 +38,7 @@ jobs: ${{ github.repository == 'vortex-data/vortex' && format('runs-on={0}/runner=arm64-medium/disk=large/tag={1}-minimize', github.run_id, inputs.fuzz_target) || 'ubuntu-latest' }} + timeout-minutes: 240 steps: - name: Bail if not on develop if: github.ref != 'refs/heads/develop' @@ -50,7 +51,7 @@ jobs: with: sccache: s3 - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - uses: ./.github/actions/setup-rust with: @@ -91,8 +92,7 @@ jobs: CORPUS_DIR="fuzz/corpus/${{ inputs.fuzz_target }}" MINIMIZED_DIR="${CORPUS_DIR}_minimized" mkdir -p "$MINIMIZED_DIR" - RUSTFLAGS="--cfg vortex_nightly" \ - cargo +$NIGHTLY_TOOLCHAIN fuzz cmin $FEATURES_FLAG \ + cargo +$NIGHTLY_TOOLCHAIN fuzz cmin $FEATURES_FLAG \ ${{ inputs.fuzz_target }} "$CORPUS_DIR" -- "$MINIMIZED_DIR" rm -rf "$CORPUS_DIR" mv "$MINIMIZED_DIR" "$CORPUS_DIR" diff --git a/.github/workflows/nightly-bench.yml b/.github/workflows/nightly-bench.yml index fa8e46cb6b9..b89395516be 100644 --- a/.github/workflows/nightly-bench.yml +++ b/.github/workflows/nightly-bench.yml @@ -29,7 +29,19 @@ jobs: "id": "tpch-nvme", "subcommand": "tpch", "name": "TPC-H on NVME", - "targets": "datafusion:parquet,datafusion:vortex,duckdb:parquet,duckdb:vortex", + "data_formats": ["parquet", "vortex"], + "pr_targets": [ + {"engine": "datafusion", "format": "parquet"}, + {"engine": "datafusion", "format": "vortex"}, + {"engine": "duckdb", "format": "parquet"}, + {"engine": "duckdb", "format": "vortex"} + ], + "develop_targets": [ + {"engine": "datafusion", "format": "parquet"}, + {"engine": "datafusion", "format": "vortex"}, + {"engine": "duckdb", "format": "parquet"}, + {"engine": "duckdb", "format": "vortex"} + ], "scale_factor": "100" }, { @@ -38,9 +50,21 @@ jobs: "name": "TPC-H on S3", "local_dir": "vortex-bench/data/tpch/100.0", "remote_storage": "s3://vortex-ci-benchmark-datasets/${{github.ref_name}}/${{github.run_id}}/tpch/100.0/", - "targets": "datafusion:parquet,datafusion:vortex,duckdb:parquet,duckdb:vortex", + "data_formats": ["parquet", "vortex"], + "pr_targets": [ + {"engine": "datafusion", "format": "parquet"}, + {"engine": "datafusion", "format": "vortex"}, + {"engine": "duckdb", "format": "parquet"}, + {"engine": "duckdb", "format": "vortex"} + ], + "develop_targets": [ + {"engine": "datafusion", "format": "parquet"}, + {"engine": "datafusion", "format": "vortex"}, + {"engine": "duckdb", "format": "parquet"}, + {"engine": "duckdb", "format": "vortex"} + ], "scale_factor": "100.0" - }, + } ] strategy: # A single run not should kill the others @@ -48,4 +72,4 @@ jobs: matrix: machine_type: - id: x86 - instance_name: c6id.8xlarge + instance_name: i7i.metal-24xl diff --git a/.github/workflows/package.yml b/.github/workflows/package.yml index fbaca0f8407..e720a5d43c9 100644 --- a/.github/workflows/package.yml +++ b/.github/workflows/package.yml @@ -31,7 +31,7 @@ jobs: - { os: ubuntu, runs-on: "ubuntu-latest", target: aarch64-unknown-linux-gnu } - { os: ubuntu, runs-on: "ubuntu-latest", target: x86_64-unknown-linux-gnu } steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: fetch-depth: 0 @@ -51,14 +51,14 @@ jobs: cargo set-version --workspace ${{ inputs.version }} - name: Set up Python - uses: actions/setup-python@v6 + uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6 # Latest macOS doesn't allow maturin to install stuff into the global Python interpreter if: "${{ matrix.target.runs-on == 'macos-latest' }}" with: python-version: "3.11" - name: Build wheel - uses: PyO3/maturin-action@v1 + uses: PyO3/maturin-action@e83996d129638aa358a18fbd1dfb82f0b0fb5d3b # v1 with: command: build maturin-version: 1.10.2 @@ -76,7 +76,7 @@ jobs: PYO3_ENVIRONMENT_SIGNATURE: "cpython3" - name: Upload wheels - uses: actions/upload-artifact@v7 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: name: "wheels-${{ matrix.target.target }}.zip" path: dist/ @@ -84,11 +84,12 @@ jobs: prepare-java-macos: runs-on: "macos-latest" + timeout-minutes: 30 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: fetch-depth: 0 - - uses: actions/setup-java@v5 + - uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5 with: distribution: "corretto" java-version: "17" @@ -97,7 +98,7 @@ jobs: repo-token: ${{ secrets.GITHUB_TOKEN }} enable-sccache: "false" - run: cargo build --release --package vortex-jni - - uses: actions/upload-artifact@v7 + - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: name: "libvortex_jni_aarch64-apple-darwin.zip" path: "target/release/libvortex_jni.dylib" @@ -105,40 +106,45 @@ jobs: if-no-files-found: error prepare-java-linux: - runs-on: ${{ matrix.target.runs-on }} - container: - image: "ubuntu:20.04" - timeout-minutes: 120 + timeout-minutes: 30 strategy: fail-fast: false matrix: - target: - - { os: ubuntu, runs-on: "ubuntu-24.04-arm", target: aarch64-unknown-linux-gnu } - - { os: ubuntu, runs-on: "ubuntu-24.04", target: x86_64-unknown-linux-gnu } + include: + - target: x86_64-unknown-linux-gnu + runner: runs-on=${{ github.run_id }}/runner=amd64-medium/image=ubuntu24-full-x64-pre-v2/tag=prepare-java-linux-amd64 + - target: aarch64-unknown-linux-gnu + runner: runs-on=${{ github.run_id }}/runner=arm64-medium/image=ubuntu24-full-arm64-pre-v2/tag=prepare-java-linux-arm64 + runs-on: ${{ matrix.runner }} steps: - - uses: actions/checkout@v6 + - uses: runs-on/action@v2 + if: github.repository == 'vortex-data/vortex' + with: + sccache: s3 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: fetch-depth: 0 - - uses: ./.github/actions/prepare-java-linux - - uses: actions/setup-java@v5 + - uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5 with: distribution: "corretto" java-version: "17" - - uses: ./.github/actions/setup-rust + - uses: ./.github/actions/setup-prebuild + - uses: mlugg/setup-zig@d1434d08867e3ee9daa34448df10607b98908d29 # v2.2.1 + - name: Install cargo-zigbuild + uses: taiki-e/cache-cargo-install-action@66c9585ef5ca780ee69399975a5e911f47905995 with: - targets: ${{ matrix.target.target }} - repo-token: ${{ secrets.GITHUB_TOKEN }} - enable-sccache: "false" - - run: cargo build --release --package vortex-jni - - uses: actions/upload-artifact@v7 + tool: cargo-zigbuild + - run: cargo zigbuild --release --target ${{ matrix.target }}.2.31 --package vortex-jni + - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: - name: "libvortex_jni_${{ matrix.target.target }}.zip" - path: "target/release/libvortex_jni.so" + name: "libvortex_jni_${{ matrix.target }}.zip" + path: "target/${{ matrix.target }}/release/libvortex_jni.so" retention-days: 7 if-no-files-found: error prepare-all: needs: [prepare-python, prepare-java-macos, prepare-java-linux] runs-on: ubuntu-latest + timeout-minutes: 10 steps: - run: echo "All packages built successfully" diff --git a/.github/workflows/publish-benchmarks-website.yml b/.github/workflows/publish-benchmarks-website.yml index 824ff80e6a8..a85ba0124e1 100644 --- a/.github/workflows/publish-benchmarks-website.yml +++ b/.github/workflows/publish-benchmarks-website.yml @@ -9,27 +9,28 @@ on: jobs: publish: runs-on: ubuntu-latest + timeout-minutes: 10 permissions: contents: read packages: write steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - name: Log in to GHCR - uses: docker/login-action@v4 + uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4 with: registry: ghcr.io username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - name: Set up QEMU - uses: docker/setup-qemu-action@v4 + uses: docker/setup-qemu-action@ce360397dd3f832beb865e1373c09c0e9f86d70a # v4 - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v4 + uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4 - name: Build and push - uses: docker/build-push-action@v7 + uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7 with: context: ./benchmarks-website platforms: linux/arm64 diff --git a/.github/workflows/publish-dry-runs.yml b/.github/workflows/publish-dry-runs.yml new file mode 100644 index 00000000000..4b834935777 --- /dev/null +++ b/.github/workflows/publish-dry-runs.yml @@ -0,0 +1,127 @@ +name: Publish Dry Runs + +# Concurrency control: +# - PRs: new commits on a feature branch will cancel in-progress (outdated) runs. +# - Push to develop: runs queue sequentially, never cancelled. +# - `workflow_dispatch`: groups by branch and queues if run on develop. +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: ${{ github.ref != 'refs/heads/develop' }} +on: + push: + branches: [develop] + pull_request: { } + workflow_dispatch: { } + +permissions: + contents: read + +env: + CARGO_TERM_COLOR: always + RUST_BACKTRACE: 1 + NIGHTLY_TOOLCHAIN: nightly-2026-02-05 + +jobs: + python-wheel-build: + name: "Python (wheel build)" + runs-on: ubuntu-latest + timeout-minutes: 30 + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + - name: Rust Dependency Cache + uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2 + with: + save-if: ${{ github.ref_name == 'develop' }} + - uses: ./.github/actions/setup-rust + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + enable-sccache: "false" + - uses: mlugg/setup-zig@d1434d08867e3ee9daa34448df10607b98908d29 # v2.2.1 + - name: Install uv + uses: spiraldb/actions/.github/actions/setup-uv@a746510eafaa926484c354541cfc49b2ec06cc63 # 0.18.6 + with: + sync: false + prune-cache: false + - name: Ensure wheel and sdist can be built on Linux - PyVortex + shell: bash + run: | + echo "Clearing wheel target directory" + rm -rf ../target/wheels/ + + uv venv + uv tool run maturin@1.10.2 build --interpreter python3.11 --zig + uv tool run maturin@1.10.2 build --interpreter python3.11 --zig --sdist + + file_count=$(ls -1 ../target/wheels/ | wc -l) + + if [[ $file_count -ne 2 ]]; then + echo "Unexpected number of files detected ${file_count}:" + ls ../target/wheels/ + exit 1 + else + echo "Generated two files" + fi + working-directory: vortex-python/ + + check-java-publish-build: + timeout-minutes: 30 + strategy: + fail-fast: false + matrix: + include: + - target: x86_64-unknown-linux-gnu + runner: runs-on=${{ github.run_id }}/runner=amd64-medium/image=ubuntu24-full-x64-pre-v2/tag=check-java-publish-build-amd64 + - target: aarch64-unknown-linux-gnu + runner: runs-on=${{ github.run_id }}/runner=arm64-medium/image=ubuntu24-full-arm64-pre-v2/tag=check-java-publish-build-arm64 + runs-on: ${{ matrix.runner }} + steps: + - uses: runs-on/action@v2 + if: github.repository == 'vortex-data/vortex' + with: + sccache: s3 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + fetch-depth: 0 + - uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5 + with: + distribution: "corretto" + java-version: "17" + - uses: ./.github/actions/setup-prebuild + - uses: mlugg/setup-zig@d1434d08867e3ee9daa34448df10607b98908d29 # v2.2.1 + - name: Install cargo-zigbuild + uses: taiki-e/cache-cargo-install-action@66c9585ef5ca780ee69399975a5e911f47905995 + with: + tool: cargo-zigbuild + - run: cargo zigbuild --profile ci --target ${{ matrix.target }}.2.31 --package vortex-jni + + compat-check: + name: "Compat check" + uses: ./.github/workflows/compat-validation.yml + with: + mode: last + + rust-publish-dry-run: + name: "Rust publish dry-run" + timeout-minutes: 30 + runs-on: >- + ${{ github.repository == 'vortex-data/vortex' + && format('runs-on={0}/runner=amd64-xsmall/image=ubuntu24-full-x64-pre-v2/tag=rust-publish-dry-run', github.run_id) + || 'ubuntu-latest' }} + steps: + - uses: runs-on/action@v2 + if: github.repository == 'vortex-data/vortex' + with: + sccache: s3 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + - uses: ./.github/actions/setup-prebuild + - name: Install cargo-edit + uses: taiki-e/cache-cargo-install-action@66c9585ef5ca780ee69399975a5e911f47905995 + with: + tool: cargo-edit + - name: Set Version + run: | + # This is mostly a dummy version, we don't actually publish anything but it cannot exist in crates.io + cargo set-version --workspace 0.100000.0 + - name: Test Release + run: | + cargo publish --dry-run --no-verify --allow-dirty --workspace diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 2d3e2388922..1a015fb451f 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -20,10 +20,10 @@ jobs: publish-rust: runs-on: ubuntu-latest - timeout-minutes: 120 + timeout-minutes: 30 needs: [package] steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: fetch-depth: 0 - uses: ./.github/actions/setup-rust @@ -49,7 +49,7 @@ jobs: publish-python: needs: [package] runs-on: ubuntu-latest - timeout-minutes: 120 + timeout-minutes: 30 permissions: id-token: write # IMPORTANT: mandatory for trusted publishing deployments: write @@ -57,7 +57,7 @@ jobs: name: push-to-pypi url: https://pypi.org/p/vortex-data steps: - - uses: actions/download-artifact@v8 + - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: pattern: wheels-*.zip # https://github.com/actions/download-artifact?tab=readme-ov-file#download-all-artifacts @@ -67,7 +67,7 @@ jobs: - name: Display structure of downloaded files run: ls -R dist/ - name: Publish to PyPI - uses: pypa/gh-action-pypi-publish@release/v1 + uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # release/v1 with: attestations: true verbose: true @@ -76,7 +76,7 @@ jobs: publish-java: needs: [package] runs-on: ubuntu-latest - timeout-minutes: 120 + timeout-minutes: 30 env: # Picked up by gradle-git-version GIT_VERSION: ${{ github.event.release.tag_name }} @@ -84,13 +84,13 @@ jobs: run: working-directory: ./java steps: - - uses: actions/checkout@v6 - - uses: actions/setup-java@v5 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + - uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5 with: distribution: "corretto" java-version: "17" - - uses: gradle/actions/setup-gradle@v5 - - uses: actions/download-artifact@v8 + - uses: gradle/actions/setup-gradle@50e97c2cd7a37755bbfafc9c5b7cafaece252f6e # v6 + - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: pattern: libvortex_jni_*.zip - name: Copy native JNI libs diff --git a/.github/workflows/release-binaries.yml b/.github/workflows/release-binaries.yml new file mode 100644 index 00000000000..1e5cd70e69c --- /dev/null +++ b/.github/workflows/release-binaries.yml @@ -0,0 +1,60 @@ +name: Release Binaries + +on: + release: + types: [published] + +permissions: + contents: write + +jobs: + build: + name: Build ${{ matrix.target }} + runs-on: ${{ matrix.runs-on }} + timeout-minutes: 120 + strategy: + fail-fast: false + matrix: + include: + - target: aarch64-apple-darwin + runs-on: macos-latest + archive: tgz + - target: x86_64-apple-darwin + runs-on: macos-15-intel + archive: tgz + - target: aarch64-unknown-linux-gnu + runs-on: ubuntu-24.04-arm + archive: tgz + - target: x86_64-unknown-linux-gnu + runs-on: ubuntu-24.04 + archive: tgz + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + with: + fetch-depth: 0 + + - uses: ./.github/actions/setup-rust + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + targets: ${{ matrix.target }} + enable-sccache: "false" + + - name: Build release binary + run: cargo build --release --package vortex-tui --bin vx --target ${{ matrix.target }} + + - name: Create archive (tgz) + if: matrix.archive == 'tgz' + run: | + cd target/${{ matrix.target }}/release + tar -czvf ../../../vx-${{ matrix.target }}.tar.gz vx + + - name: Create archive (zip) + if: matrix.archive == 'zip' + run: | + cd target/${{ matrix.target }}/release + zip ../../../vx-${{ matrix.target }}.zip vx.exe + + - name: Upload release asset + run: gh release upload "${{ github.event.release.tag_name }}" vx-${{ matrix.target }}.${{ matrix.archive == 'tgz' && 'tar.gz' || 'zip' }} + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/release-drafter.yml b/.github/workflows/release-drafter.yml index df9bc92e455..a4d2bc4e677 100644 --- a/.github/workflows/release-drafter.yml +++ b/.github/workflows/release-drafter.yml @@ -25,8 +25,9 @@ jobs: # write permission is required to create a github release contents: write runs-on: ubuntu-latest + timeout-minutes: 10 steps: - - uses: release-drafter/release-drafter@v7.1.1 + - uses: release-drafter/release-drafter@5de93583980a40bd78603b6dfdcda5b4df377b32 # v7.2.0 with: commitish: ${{ github.ref_name }} prerelease: ${{ github.event_name == 'workflow_dispatch' }} diff --git a/.github/workflows/report-fuzz-crash.yml b/.github/workflows/report-fuzz-crash.yml index dd4026641f6..83d75fcb19e 100644 --- a/.github/workflows/report-fuzz-crash.yml +++ b/.github/workflows/report-fuzz-crash.yml @@ -49,16 +49,16 @@ jobs: steps: - name: Checkout repository - uses: actions/checkout@v6 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - name: Download fuzzer logs - uses: actions/download-artifact@v8 + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: name: ${{ inputs.logs_artifact_name }} path: ./logs - name: Download crash artifacts - uses: actions/download-artifact@v8 + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: name: ${{ inputs.artifact_name }} path: ./crash_artifacts @@ -118,7 +118,7 @@ jobs: steps.dedup.outputs.duplicate != 'true' || steps.dedup.outputs.confidence != 'exact' continue-on-error: true - uses: anthropics/claude-code-action@v1 + uses: anthropics/claude-code-action@567fe954a4527e81f132d87d1bdbcc94f7737434 # v1 with: claude_code_oauth_token: ${{ secrets.claude_code_oauth_token }} github_token: ${{ secrets.gh_token }} diff --git a/.github/workflows/reuse.yml b/.github/workflows/reuse.yml index e2891f48165..6d871fd20d4 100644 --- a/.github/workflows/reuse.yml +++ b/.github/workflows/reuse.yml @@ -15,7 +15,8 @@ on: jobs: reuse-check: runs-on: ubuntu-latest + timeout-minutes: 10 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - name: REUSE Compliance Check - uses: fsfe/reuse-action@v6 + uses: fsfe/reuse-action@676e2d560c9a403aa252096d99fcab3e1132b0f5 # v6 diff --git a/.github/workflows/run-fuzzer.yml b/.github/workflows/run-fuzzer.yml index 251ee31a5b7..55dc3268d3b 100644 --- a/.github/workflows/run-fuzzer.yml +++ b/.github/workflows/run-fuzzer.yml @@ -61,7 +61,7 @@ jobs: name: "Run ${{ inputs.fuzz_name || inputs.fuzz_target }}" env: FUZZ_NAME: ${{ inputs.fuzz_name || inputs.fuzz_target }} - timeout-minutes: 230 # almost 4 hours + timeout-minutes: 240 # 4 hours runs-on: >- ${{ github.repository == 'vortex-data/vortex' && format('runs-on={0}/runner={1}/disk=large/tag={2}-fuzz', github.run_id, inputs.runner, inputs.fuzz_name || inputs.fuzz_target) @@ -76,7 +76,7 @@ jobs: with: sccache: s3 - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - uses: ./.github/actions/setup-rust with: @@ -102,7 +102,7 @@ jobs: AWS_ENDPOINT_URL: "https://01e9655179bbec953276890b183039bc.r2.cloudflarestorage.com" run: | CORPUS_KEY="${FUZZ_NAME}_corpus.tar.zst" - CORPUS_DIR="fuzz/corpus/${FUZZ_NAME}-${{ inputs.extra_features }}" + CORPUS_DIR="fuzz/corpus/${FUZZ_NAME}" # Try to download corpus if python3 scripts/s3-download.py "s3://vortex-fuzz-corpus/$CORPUS_KEY" "$CORPUS_KEY"; then @@ -124,7 +124,7 @@ jobs: if [ "${{ inputs.jobs }}" -gt 1 ]; then FORK_FLAG="-fork=${{ inputs.jobs }}" fi - ${{ inputs.extra_env }} RUSTFLAGS="--cfg vortex_nightly" RUST_BACKTRACE=1 \ + ${{ inputs.extra_env }} RUST_BACKTRACE=1 \ cargo +$NIGHTLY_TOOLCHAIN fuzz run --release --debug-assertions \ $FEATURES_FLAG \ ${{ inputs.fuzz_target }} -- \ @@ -163,7 +163,7 @@ jobs: if [ -n "${{ inputs.extra_features }}" ]; then FEATURES_FLAG="--features ${{ inputs.extra_features }}" fi - RUSTFLAGS="--cfg vortex_nightly" RUST_BACKTRACE=1 \ + RUST_BACKTRACE=1 \ cargo +$NIGHTLY_TOOLCHAIN fuzz run --release --debug-assertions \ $FEATURES_FLAG \ ${{ inputs.fuzz_target }} \ @@ -173,7 +173,7 @@ jobs: - name: Archive crash artifacts id: upload_artifacts if: steps.check.outputs.crashes_found == 'true' - uses: actions/upload-artifact@v7 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: name: ${{ env.FUZZ_NAME }}-crash-artifacts path: fuzz/artifacts @@ -181,7 +181,7 @@ jobs: - name: Archive fuzzer output log if: steps.check.outputs.crashes_found == 'true' - uses: actions/upload-artifact@v7 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: name: ${{ env.FUZZ_NAME }}-logs path: fuzz_output.log diff --git a/.github/workflows/rust-instrumented.yml b/.github/workflows/rust-instrumented.yml new file mode 100644 index 00000000000..e9178f04635 --- /dev/null +++ b/.github/workflows/rust-instrumented.yml @@ -0,0 +1,234 @@ +name: Rust Instrumented + +# Concurrency control: +# - PRs: new commits on a feature branch will cancel in-progress (outdated) runs. +# - Push to develop: runs queue sequentially, never cancelled. +# - `workflow_dispatch`: groups by branch and queues if run on develop. +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: ${{ github.ref != 'refs/heads/develop' }} +on: + push: + branches: [develop] + pull_request: { } + workflow_dispatch: { } + +permissions: + contents: read + +env: + CARGO_TERM_COLOR: always + RUST_BACKTRACE: 1 + NIGHTLY_TOOLCHAIN: nightly-2026-02-05 + +jobs: + rust-coverage: + name: "Rust tests (coverage) (${{ matrix.suite }})" + timeout-minutes: 30 + permissions: + id-token: write + strategy: + matrix: + include: + - suite: tests + runs-on: >- + ${{ github.repository == 'vortex-data/vortex' + && format('runs-on={0}/runner=amd64-large/image=ubuntu24-full-x64-pre-v2/tag=rust-coverage-suite-{1}', github.run_id, matrix.suite) + || 'ubuntu-latest' }} + env: + RUSTFLAGS: "-Cinstrument-coverage -A warnings" + CARGO_INCREMENTAL: 0 # Disable incremental compilation to get accurate coverage + LLVM_PROFILE_FILE: "target/coverage/vortex-%p-%m.profraw" + GRCOV_OUTPUT_FILE: "target/coverage/vortex.lcov" + steps: + - uses: runs-on/action@v2 + if: github.repository == 'vortex-data/vortex' + with: + sccache: s3 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + - uses: ./.github/actions/setup-prebuild + - name: Rust Tests + if: ${{ matrix.suite == 'tests' }} + run: | + cargo nextest run --locked --workspace --all-features --no-fail-fast + - name: Generate coverage report + run: | + grcov . --binary-path target/debug/ -s . -t lcov --llvm --ignore-not-existing \ + --threads $(nproc) \ + --ignore '../*' --ignore '/*' --ignore 'fuzz/*' --ignore 'vortex-bench/*' \ + --ignore 'home/*' --ignore 'xtask/*' --ignore 'target/*' --ignore 'vortex-error/*' \ + --ignore 'vortex-python/*' --ignore 'vortex-jni/*' --ignore 'vortex-flatbuffers/*' \ + --ignore 'vortex-proto/*' --ignore 'vortex-tui/*' --ignore 'vortex-datafusion/examples/*' \ + --ignore 'vortex-ffi/examples/*' --ignore '*/arbitrary/*' --ignore '*/arbitrary.rs' --ignore 'vortex-cxx/*' \ + --ignore benchmarks/* --ignore 'vortex-test/*' \ + -o ${{ env.GRCOV_OUTPUT_FILE }} + - name: Codecov + uses: codecov/codecov-action@57e3a136b779b570ffcdbf80b3bdc90e7fab3de2 # v6 + with: + name: run-${{ matrix.suite }} + files: ${{ env.GRCOV_OUTPUT_FILE }} + disable_search: true + flags: ${{ matrix.suite }} + use_oidc: true + + rust-test-sanitizer: + strategy: + fail-fast: false + matrix: + include: + # We don't run memory sanitizer as it provides many false positives + # for std + - sanitizer: asan + sanitizer_flags: "-Zsanitizer=address,leak" + - sanitizer: tsan + sanitizer_flags: "-Zsanitizer=thread" + name: "Rust tests (${{ matrix.sanitizer }})" + runs-on: >- + ${{ github.repository == 'vortex-data/vortex' + && format('runs-on={0}/pool=amd64-medium-pre-v2/tag=rust-test-sanitizer', github.run_id) + || 'ubuntu-latest' }} + timeout-minutes: 30 + env: + ASAN_OPTIONS: "symbolize=1:check_initialization_order=1:detect_leaks=1:leak_check_at_exit=1" + LSAN_OPTIONS: "report_objects=1" + ASAN_SYMBOLIZER_PATH: "/usr/bin/llvm-symbolizer" + MSAN_OPTIONS: "symbolize=1" + MSAN_SYMBOLIZER_PATH: "/usr/bin/llvm-symbolizer" + TSAN_OPTIONS: "symbolize=1:suppressions=${{ github.workspace }}/vortex-ffi/tsan_suppressions.txt" + TSAN_SYMBOLIZER_PATH: "/usr/bin/llvm-symbolizer" + VORTEX_SKIP_SLOW_TESTS: "1" + # -Cunsafe-allow-abi-mismatch=sanitizer: libraries like compiler_builtins + # unset -Zsanitizer flag and we should allow that. + RUSTFLAGS: "-A warnings -Cunsafe-allow-abi-mismatch=sanitizer -C debuginfo=2 -C opt-level=0 -C strip=none" + steps: + - uses: runs-on/action@v2 + if: github.repository == 'vortex-data/vortex' + with: + sccache: s3 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + - uses: ./.github/actions/setup-prebuild + - name: Install Rust nightly toolchain + run: | + rustup toolchain install $NIGHTLY_TOOLCHAIN + rustup component add --toolchain $NIGHTLY_TOOLCHAIN rust-src rustfmt clippy llvm-tools-preview + - name: Build tests with sanitizer + run: | + RUSTFLAGS="${RUSTFLAGS} ${{ matrix.sanitizer_flags }}" \ + cargo +$NIGHTLY_TOOLCHAIN build --locked --all-features \ + --target x86_64-unknown-linux-gnu -Zbuild-std \ + -p vortex-buffer -p vortex-fastlanes -p vortex-fsst -p vortex-alp -p vortex-array + + - name: Run tests with sanitizer + run: | + RUSTFLAGS="${RUSTFLAGS} ${{ matrix.sanitizer_flags }}" \ + cargo +$NIGHTLY_TOOLCHAIN nextest run --locked --all-features \ + --target x86_64-unknown-linux-gnu --no-fail-fast -Zbuild-std \ + -p vortex-buffer -p vortex-fastlanes -p vortex-fsst -p vortex-alp -p vortex-array + + # vortex-ffi requires --no-default-features as otherwise we pull in + # Mimalloc which interferes with sanitizers + # cargo nextest reports less sanitizer issues than cargo test + # TODO(myrrc): remove --no-default-features once we make Mimalloc opt-in + - name: Run vortex-ffi tests with sanitizer + run: | + RUSTFLAGS="${RUSTFLAGS} ${{ matrix.sanitizer_flags }}" \ + cargo +$NIGHTLY_TOOLCHAIN test --locked --no-default-features \ + --target x86_64-unknown-linux-gnu --no-fail-fast -Zbuild-std \ + -p vortex-ffi -- --no-capture + + rust-ffi-test-sanitizer: + strategy: + fail-fast: false + matrix: + include: + # We don't run memory sanitizer as it's clang-only and provides many + # false positives for Catch2 + - sanitizer: asan + sanitizer_flags: "-Zsanitizer=address,leak" + - sanitizer: tsan + sanitizer_flags: "-Zsanitizer=thread" + name: "Rust/C++ FFI tests (${{ matrix.sanitizer }})" + timeout-minutes: 30 + env: + ASAN_OPTIONS: "symbolize=1:check_initialization_order=1:detect_leaks=1:leak_check_at_exit=1" + LSAN_OPTIONS: "report_objects=1" + ASAN_SYMBOLIZER_PATH: "/usr/bin/llvm-symbolizer" + MSAN_OPTIONS: "symbolize=1" + MSAN_SYMBOLIZER_PATH: "/usr/bin/llvm-symbolizer" + TSAN_OPTIONS: "symbolize=1:suppressions=${{ github.workspace }}/vortex-ffi/tsan_suppressions.txt" + TSAN_SYMBOLIZER_PATH: "/usr/bin/llvm-symbolizer" + VORTEX_SKIP_SLOW_TESTS: "1" + # -Cunsafe-allow-abi-mismatch=sanitizer: libraries like compiler_builtins + # unset -Zsanitizer flag and we should allow that. + runs-on: >- + ${{ github.repository == 'vortex-data/vortex' + && format('runs-on={0}/runner=amd64-medium/image=ubuntu24-full-x64-pre-v2/tag=rust-ffi-test-sanitizer', github.run_id) + || 'ubuntu-latest' }} + steps: + - uses: runs-on/action@v2 + if: github.repository == 'vortex-data/vortex' + with: + sccache: s3 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + - uses: ./.github/actions/setup-prebuild + - name: Install rustfilt + run: | + cargo install rustfilt + - name: Install Rust nightly toolchain + run: | + rustup toolchain install $NIGHTLY_TOOLCHAIN + rustup component add --toolchain $NIGHTLY_TOOLCHAIN rust-src rustfmt clippy llvm-tools-preview + - name: Build FFI library + run: | + # TODO(myrrc): remove --no-default-features + RUSTFLAGS="-A warnings -Cunsafe-allow-abi-mismatch=sanitizer \ + -C debuginfo=2 -C opt-level=0 -C strip=none -Zexternal-clangrt \ + ${{ matrix.sanitizer_flags }}" \ + cargo +$NIGHTLY_TOOLCHAIN build --locked --no-default-features \ + --target x86_64-unknown-linux-gnu -Zbuild-std \ + -p vortex-ffi + - name: Build FFI library tests and examples + run: | + cd vortex-ffi + cmake -Bbuild -DBUILD_TESTS=1 -DBUILD_EXAMPLES=1 -DSANITIZER=${{ matrix.sanitizer }} -DTARGET_TRIPLE="x86_64-unknown-linux-gnu" + cmake --build build -j + - name: Run tests + run: | + set -o pipefail + ./vortex-ffi/build/test/vortex_ffi_test 2>&1 | rustfilt + - name: Run examples + run: | + set -o pipefail + + # Failed to create data source: Object store error: Generic LocalFileSystem + # error: Unable to walk dir: File system loop found + rm -fr vortex-ffi/build/_deps/nanoarrow-src/python + + ./vortex-ffi/build/examples/write_sample file.vortex 2>&1 | rustfilt + ./vortex-ffi/build/examples/write_sample file2.vortex 2>&1 | rustfilt + ./vortex-ffi/build/examples/dtype '*.vortex' 2>&1 | rustfilt + ./vortex-ffi/build/examples/scan '*.vortex' 2>&1 | rustfilt + ./vortex-ffi/build/examples/scan_to_arrow '*.vortex' 2>&1 | rustfilt + + miri: + name: "Rust tests (miri)" + runs-on: >- + ${{ github.repository == 'vortex-data/vortex' + && format('runs-on={0}/runner=amd64-medium/image=ubuntu24-full-x64-pre-v2/tag=rust-miri', github.run_id) + || 'ubuntu-latest' }} + timeout-minutes: 30 + env: + MIRIFLAGS: -Zmiri-strict-provenance -Zmiri-symbolic-alignment-check -Zmiri-disable-isolation -Zmiri-env-forward=RUST_BACKTRACE + RUSTFLAGS: "-A warnings" + RUST_BACKTRACE: full + steps: + - uses: runs-on/action@v2 + if: github.repository == 'vortex-data/vortex' + with: + sccache: s3 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + - uses: ./.github/actions/setup-prebuild + - name: Install nightly with miri + run: rustup toolchain install $NIGHTLY_TOOLCHAIN --component rust-src,rustfmt,clippy,miri + - name: Run Miri + run: cargo +$NIGHTLY_TOOLCHAIN miri nextest run --no-fail-fast -p vortex-buffer -p vortex-ffi diff --git a/.github/workflows/sql-benchmarks.yml b/.github/workflows/sql-benchmarks.yml index 4c114f00bda..8be259fa562 100644 --- a/.github/workflows/sql-benchmarks.yml +++ b/.github/workflows/sql-benchmarks.yml @@ -9,26 +9,64 @@ on: machine_type: required: false type: string - default: c6id.8xlarge + default: i7i.metal-24xl benchmark_matrix: required: false type: string description: "JSON string containing the matrix configuration" - # We do not include lance in the default configuration. default: | [ { "id": "clickbench-nvme", "subcommand": "clickbench", "name": "Clickbench on NVME", - "targets": "datafusion:parquet,datafusion:vortex,duckdb:parquet,duckdb:vortex,duckdb:duckdb" + "data_formats": ["parquet", "vortex", "vortex-compact", "duckdb"], + "pr_targets": [ + {"engine": "datafusion", "format": "parquet"}, + {"engine": "datafusion", "format": "vortex"}, + {"engine": "duckdb", "format": "parquet"}, + {"engine": "duckdb", "format": "vortex"}, + {"engine": "duckdb", "format": "duckdb"} + ], + "develop_targets": [ + {"engine": "datafusion", "format": "parquet"}, + {"engine": "datafusion", "format": "vortex"}, + {"engine": "datafusion", "format": "vortex-compact"}, + {"engine": "datafusion", "format": "lance"}, + {"engine": "duckdb", "format": "parquet"}, + {"engine": "duckdb", "format": "vortex"}, + {"engine": "duckdb", "format": "vortex-compact"}, + {"engine": "duckdb", "format": "duckdb"} + ] }, { "id": "tpch-nvme", "subcommand": "tpch", "name": "TPC-H SF=1 on NVME", - "targets": "datafusion:arrow,datafusion:parquet,datafusion:vortex,datafusion:vortex-compact,duckdb:parquet,duckdb:vortex,duckdb:vortex-compact,duckdb:duckdb", - "scale_factor": "1.0" + "data_formats": ["parquet", "vortex", "vortex-compact", "duckdb"], + "pr_targets": [ + {"engine": "datafusion", "format": "arrow"}, + {"engine": "datafusion", "format": "parquet"}, + {"engine": "datafusion", "format": "vortex"}, + {"engine": "datafusion", "format": "vortex-compact"}, + {"engine": "duckdb", "format": "parquet"}, + {"engine": "duckdb", "format": "vortex"}, + {"engine": "duckdb", "format": "vortex-compact"}, + {"engine": "duckdb", "format": "duckdb"} + ], + "develop_targets": [ + {"engine": "datafusion", "format": "arrow"}, + {"engine": "datafusion", "format": "parquet"}, + {"engine": "datafusion", "format": "vortex"}, + {"engine": "datafusion", "format": "vortex-compact"}, + {"engine": "datafusion", "format": "lance"}, + {"engine": "duckdb", "format": "parquet"}, + {"engine": "duckdb", "format": "vortex"}, + {"engine": "duckdb", "format": "vortex-compact"}, + {"engine": "duckdb", "format": "duckdb"} + ], + "scale_factor": "1.0", + "iterations": "10" }, { "id": "tpch-s3", @@ -36,7 +74,23 @@ on: "name": "TPC-H SF=1 on S3", "local_dir": "vortex-bench/data/tpch/1.0", "remote_storage": "s3://vortex-ci-benchmark-datasets/${{github.ref_name}}/${{github.run_id}}/tpch/1.0/", - "targets": "datafusion:parquet,datafusion:vortex,datafusion:vortex-compact,duckdb:parquet,duckdb:vortex,duckdb:vortex-compact", + "data_formats": ["parquet", "vortex", "vortex-compact"], + "pr_targets": [ + {"engine": "datafusion", "format": "parquet"}, + {"engine": "datafusion", "format": "vortex"}, + {"engine": "datafusion", "format": "vortex-compact"}, + {"engine": "duckdb", "format": "parquet"}, + {"engine": "duckdb", "format": "vortex"}, + {"engine": "duckdb", "format": "vortex-compact"} + ], + "develop_targets": [ + {"engine": "datafusion", "format": "parquet"}, + {"engine": "datafusion", "format": "vortex"}, + {"engine": "datafusion", "format": "vortex-compact"}, + {"engine": "duckdb", "format": "parquet"}, + {"engine": "duckdb", "format": "vortex"}, + {"engine": "duckdb", "format": "vortex-compact"} + ], "scale_factor": "1.0", "iterations": "10" }, @@ -44,8 +98,30 @@ on: "id": "tpch-nvme-10", "subcommand": "tpch", "name": "TPC-H SF=10 on NVME", - "targets": "datafusion:arrow,datafusion:parquet,datafusion:vortex,datafusion:vortex-compact,duckdb:parquet,duckdb:vortex,duckdb:vortex-compact,duckdb:duckdb", - "scale_factor": "10.0" + "data_formats": ["parquet", "vortex", "vortex-compact", "duckdb"], + "pr_targets": [ + {"engine": "datafusion", "format": "arrow"}, + {"engine": "datafusion", "format": "parquet"}, + {"engine": "datafusion", "format": "vortex"}, + {"engine": "datafusion", "format": "vortex-compact"}, + {"engine": "duckdb", "format": "parquet"}, + {"engine": "duckdb", "format": "vortex"}, + {"engine": "duckdb", "format": "vortex-compact"}, + {"engine": "duckdb", "format": "duckdb"} + ], + "develop_targets": [ + {"engine": "datafusion", "format": "arrow"}, + {"engine": "datafusion", "format": "parquet"}, + {"engine": "datafusion", "format": "vortex"}, + {"engine": "datafusion", "format": "vortex-compact"}, + {"engine": "datafusion", "format": "lance"}, + {"engine": "duckdb", "format": "parquet"}, + {"engine": "duckdb", "format": "vortex"}, + {"engine": "duckdb", "format": "vortex-compact"}, + {"engine": "duckdb", "format": "duckdb"} + ], + "scale_factor": "10.0", + "iterations": "10" }, { "id": "tpch-s3-10", @@ -53,7 +129,23 @@ on: "name": "TPC-H SF=10 on S3", "local_dir": "vortex-bench/data/tpch/10.0", "remote_storage": "s3://vortex-ci-benchmark-datasets/${{github.ref_name}}/${{github.run_id}}/tpch/10.0/", - "targets": "datafusion:parquet,datafusion:vortex,datafusion:vortex-compact,duckdb:parquet,duckdb:vortex,duckdb:vortex-compact", + "data_formats": ["parquet", "vortex", "vortex-compact"], + "pr_targets": [ + {"engine": "datafusion", "format": "parquet"}, + {"engine": "datafusion", "format": "vortex"}, + {"engine": "datafusion", "format": "vortex-compact"}, + {"engine": "duckdb", "format": "parquet"}, + {"engine": "duckdb", "format": "vortex"}, + {"engine": "duckdb", "format": "vortex-compact"} + ], + "develop_targets": [ + {"engine": "datafusion", "format": "parquet"}, + {"engine": "datafusion", "format": "vortex"}, + {"engine": "datafusion", "format": "vortex-compact"}, + {"engine": "duckdb", "format": "parquet"}, + {"engine": "duckdb", "format": "vortex"}, + {"engine": "duckdb", "format": "vortex-compact"} + ], "scale_factor": "10.0", "iterations": "10" }, @@ -61,21 +153,66 @@ on: "id": "tpcds-nvme", "subcommand": "tpcds", "name": "TPC-DS SF=1 on NVME", - "targets": "datafusion:parquet,datafusion:vortex,datafusion:vortex-compact,duckdb:parquet,duckdb:vortex,duckdb:vortex-compact,duckdb:duckdb", + "data_formats": ["parquet", "vortex", "vortex-compact", "duckdb"], + "pr_targets": [ + {"engine": "datafusion", "format": "parquet"}, + {"engine": "datafusion", "format": "vortex"}, + {"engine": "datafusion", "format": "vortex-compact"}, + {"engine": "duckdb", "format": "parquet"}, + {"engine": "duckdb", "format": "vortex"}, + {"engine": "duckdb", "format": "vortex-compact"}, + {"engine": "duckdb", "format": "duckdb"} + ], + "develop_targets": [ + {"engine": "datafusion", "format": "parquet"}, + {"engine": "datafusion", "format": "vortex"}, + {"engine": "datafusion", "format": "vortex-compact"}, + {"engine": "duckdb", "format": "parquet"}, + {"engine": "duckdb", "format": "vortex"}, + {"engine": "duckdb", "format": "vortex-compact"}, + {"engine": "duckdb", "format": "duckdb"} + ], "scale_factor": "1.0" }, { "id": "statpopgen", "subcommand": "statpopgen", "name": "Statistical and Population Genetics", - "targets": "duckdb:parquet,duckdb:vortex,duckdb:vortex-compact", + "local_dir": "vortex-bench/data/statpopgen", + "data_formats": ["parquet", "vortex", "vortex-compact"], + "pr_targets": [ + {"engine": "duckdb", "format": "parquet"}, + {"engine": "duckdb", "format": "vortex"}, + {"engine": "duckdb", "format": "vortex-compact"} + ], + "develop_targets": [ + {"engine": "duckdb", "format": "parquet"}, + {"engine": "duckdb", "format": "vortex"}, + {"engine": "duckdb", "format": "vortex-compact"} + ], "scale_factor": "100" }, { "id": "fineweb", "subcommand": "fineweb", "name": "FineWeb NVMe", - "targets": "datafusion:parquet,datafusion:vortex,datafusion:vortex-compact,duckdb:parquet,duckdb:vortex,duckdb:vortex-compact", + "data_formats": ["parquet", "vortex", "vortex-compact"], + "pr_targets": [ + {"engine": "datafusion", "format": "parquet"}, + {"engine": "datafusion", "format": "vortex"}, + {"engine": "datafusion", "format": "vortex-compact"}, + {"engine": "duckdb", "format": "parquet"}, + {"engine": "duckdb", "format": "vortex"}, + {"engine": "duckdb", "format": "vortex-compact"} + ], + "develop_targets": [ + {"engine": "datafusion", "format": "parquet"}, + {"engine": "datafusion", "format": "vortex"}, + {"engine": "datafusion", "format": "vortex-compact"}, + {"engine": "duckdb", "format": "parquet"}, + {"engine": "duckdb", "format": "vortex"}, + {"engine": "duckdb", "format": "vortex-compact"} + ], "scale_factor": "100" }, { @@ -84,22 +221,48 @@ on: "name": "FineWeb S3", "local_dir": "vortex-bench/data/fineweb", "remote_storage": "s3://vortex-ci-benchmark-datasets/${{github.ref_name}}/${{github.run_id}}/fineweb/", - "targets": "datafusion:parquet,datafusion:vortex,datafusion:vortex-compact,duckdb:parquet,duckdb:vortex,duckdb:vortex-compact", - "scale_factor": "100", - "iterations": "10" + "data_formats": ["parquet", "vortex", "vortex-compact"], + "pr_targets": [ + {"engine": "datafusion", "format": "parquet"}, + {"engine": "datafusion", "format": "vortex"}, + {"engine": "datafusion", "format": "vortex-compact"}, + {"engine": "duckdb", "format": "parquet"}, + {"engine": "duckdb", "format": "vortex"}, + {"engine": "duckdb", "format": "vortex-compact"} + ], + "develop_targets": [ + {"engine": "datafusion", "format": "parquet"}, + {"engine": "datafusion", "format": "vortex"}, + {"engine": "datafusion", "format": "vortex-compact"}, + {"engine": "duckdb", "format": "parquet"}, + {"engine": "duckdb", "format": "vortex"}, + {"engine": "duckdb", "format": "vortex-compact"} + ], + "scale_factor": "100" }, { "id": "polarsignals", "subcommand": "polarsignals", "name": "PolarSignals Profiling", - "targets": "datafusion:vortex", + "data_formats": ["vortex"], + "pr_targets": [ + {"engine": "datafusion", "format": "vortex"} + ], + "develop_targets": [ + {"engine": "datafusion", "format": "vortex"} + ], "scale_factor": "1" - }, + } ] jobs: bench: timeout-minutes: 120 + env: + VORTEX_EXPERIMENTAL_PATCHED_ARRAY: "1" + FLAT_LAYOUT_INLINE_ARRAY_NODE: "1" + # Makes python output nicer + COLUMNS: 120 strategy: fail-fast: false matrix: @@ -107,72 +270,76 @@ jobs: runs-on: >- ${{ github.repository == 'vortex-data/vortex' - && format('runs-on={0}/runner=bench-dedicated/tag={1}{2}', github.run_id, matrix.id, (inputs.mode != 'pr' || github.event.pull_request.head.repo.fork == false) && '/extras=s3-cache' || '') + && format('runs-on={0}/runner=bench-dedicated/instance-type={1}/tag={2}{3}', github.run_id, inputs.machine_type, matrix.id, (inputs.mode != 'pr' || github.event.pull_request.head.repo.fork == false) && '/extras=s3-cache' || '') || 'ubuntu-latest' }} steps: - uses: runs-on/action@v2 if: inputs.mode != 'pr' || github.event.pull_request.head.repo.fork == false with: sccache: s3 - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 if: inputs.mode == 'pr' with: ref: ${{ github.event.pull_request.head.sha }} - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 if: inputs.mode != 'pr' - name: Setup benchmark environment run: sudo bash scripts/setup-benchmark.sh - uses: ./.github/actions/setup-rust with: repo-token: ${{ secrets.GITHUB_TOKEN }} + - name: Install uv + uses: spiraldb/actions/.github/actions/setup-uv@a746510eafaa926484c354541cfc49b2ec06cc63 # 0.18.6 + with: + sync: false - name: Install DuckDB run: | - wget -qO- https://github.com/duckdb/duckdb/releases/download/v1.4.2/duckdb_cli-linux-amd64.zip | funzip > duckdb + wget -qO- https://github.com/duckdb/duckdb/releases/download/v1.5.2/duckdb_cli-linux-amd64.zip | funzip > duckdb chmod +x duckdb echo "$PWD" >> $GITHUB_PATH - uses: ./.github/actions/system-info + - name: Resolve benchmark targets + id: targets + shell: bash + run: | + if [ "${{ inputs.mode }}" = "pr" ]; then + targets='${{ toJSON(matrix.pr_targets) }}' + else + targets='${{ toJSON(matrix.develop_targets) }}' + fi + { + echo 'targets_json<<__TARGETS_JSON__' + echo "$targets" + echo '__TARGETS_JSON__' + } >> "$GITHUB_OUTPUT" + - name: Build binaries shell: bash env: RUSTFLAGS: "-C target-cpu=native" run: | packages="--bin data-gen --bin datafusion-bench --bin duckdb-bench" - if [ "${{ matrix.build_lance }}" = "true" ]; then + if [ "${{ inputs.mode }}" != "pr" ]; then packages="$packages --bin lance-bench" fi - cargo build $packages --profile release_debug + cargo build $packages --profile release_debug --features unstable_encodings - name: Generate data shell: bash env: RUST_BACKTRACE: full run: | - # Extract all unique formats from targets (e.g., "datafusion:parquet,duckdb:vortex" -> "parquet,vortex") - all_formats=$(echo "${{ matrix.targets }}" | tr ',' '\n' | sed 's/^[^:]*://' | sort -u | tr '\n' ',' | sed 's/,$//') - - # Append extra data formats if specified (for file size tracking without benchmarking) - if [ -n "${{ matrix.extra_data_formats }}" ]; then - all_formats="$all_formats,${{ matrix.extra_data_formats }}" - fi - - # Build options string if scale_factor is set - opts="" - if [ -n "${{ matrix.scale_factor }}" ]; then - opts="--opt scale-factor=${{ matrix.scale_factor }}" - fi - - # Generate all data formats with a single command - target/release_debug/data-gen ${{ matrix.subcommand }} \ - --formats "$all_formats" \ - $opts + uv run --project bench-orchestrator vx-bench prepare-data "${{ matrix.subcommand }}" \ + --formats-json '${{ toJSON(matrix.data_formats) }}' \ + ${{ matrix.scale_factor && format('--opt scale-factor={0}', matrix.scale_factor) || '' }} - name: Setup AWS CLI if: inputs.mode != 'pr' || github.event.pull_request.head.repo.fork == false - uses: aws-actions/configure-aws-credentials@v6 + uses: aws-actions/configure-aws-credentials@ec61189d14ec14c8efccab744f656cffd0e33f37 # v6 with: role-to-assume: arn:aws:iam::245040174862:role/GitHubBenchmarkRole aws-region: us-east-1 @@ -188,7 +355,7 @@ jobs: - name: Setup Polar Signals if: inputs.mode != 'pr' || github.event.pull_request.head.repo.fork == false - uses: polarsignals/gh-actions-ps-profiling@v0.8.1 + uses: polarsignals/gh-actions-ps-profiling@68ae857e375a826606352016e5b90f01a2a7ff7a # v0.8.1 with: polarsignals_cloud_token: ${{ secrets.POLAR_SIGNALS_API_KEY }} labels: "branch=${{ github.ref_name }};gh_run_id=${{ github.run_id }};benchmark=${{ matrix.id }}" @@ -206,9 +373,13 @@ jobs: OTEL_EXPORTER_OTLP_HEADERS: "${{ (inputs.mode != 'pr' || github.event.pull_request.head.repo.fork == false) && secrets.OTEL_EXPORTER_OTLP_HEADERS || '' }}" OTEL_RESOURCE_ATTRIBUTES: "bench-name=${{ matrix.id }}" run: | - bash scripts/bench-taskset.sh .github/scripts/run-sql-bench.sh "${{ matrix.subcommand }}" "${{ matrix.targets }}" \ + bash scripts/bench-taskset.sh uv run --project bench-orchestrator vx-bench run "${{ matrix.subcommand }}" \ + --targets-json '${{ steps.targets.outputs.targets_json }}' \ + --output results.json \ + --no-build \ + --runner "ec2_${{ inputs.machine_type }}" \ ${{ matrix.iterations && format('--iterations {0}', matrix.iterations) || '' }} \ - ${{ matrix.scale_factor && format('--scale-factor {0}', matrix.scale_factor) || '' }} + ${{ matrix.scale_factor && format('--opt scale-factor={0}', matrix.scale_factor) || '' }} - name: Run ${{ matrix.name }} benchmark (remote) if: matrix.remote_storage != null && (inputs.mode != 'pr' || github.event.pull_request.head.repo.fork == false) @@ -221,16 +392,14 @@ jobs: OTEL_EXPORTER_OTLP_HEADERS: "${{ (inputs.mode != 'pr' || github.event.pull_request.head.repo.fork == false) && secrets.OTEL_EXPORTER_OTLP_HEADERS || '' }}" OTEL_RESOURCE_ATTRIBUTES: "bench-name=${{ matrix.id }}" run: | - bash scripts/bench-taskset.sh .github/scripts/run-sql-bench.sh "${{ matrix.subcommand }}" "${{ matrix.targets }}" \ + bash scripts/bench-taskset.sh uv run --project bench-orchestrator vx-bench run "${{ matrix.subcommand }}" \ + --targets-json '${{ steps.targets.outputs.targets_json }}' \ + --output results.json \ + --no-build \ + --runner "ec2_${{ inputs.machine_type }}" \ ${{ matrix.iterations && format('--iterations {0}', matrix.iterations) || '' }} \ - --remote-storage "${{ matrix.remote_storage }}" \ - --benchmark-id "${{ matrix.id }}" \ - ${{ matrix.scale_factor && format('--scale-factor {0}', matrix.scale_factor) || '' }} - - - name: Install uv - uses: spiraldb/actions/.github/actions/setup-uv@0.18.5 - with: - sync: false + --opt remote-data-dir=${{ matrix.remote_storage }} \ + ${{ matrix.scale_factor && format('--opt scale-factor={0}', matrix.scale_factor) || '' }} - name: Compare results if: inputs.mode == 'pr' shell: bash @@ -256,7 +425,7 @@ jobs: - name: Comment PR if: inputs.mode == 'pr' && github.event.pull_request.head.repo.fork == false - uses: thollander/actions-comment-pull-request@v3 + uses: thollander/actions-comment-pull-request@24bffb9b452ba05a4f3f77933840a6a841d1b32b # v3 with: file-path: comment.md # There is exactly one comment per comment-tag. If a comment with this tag already exists, @@ -309,14 +478,14 @@ jobs: - name: Comment PR with file sizes if: inputs.mode == 'pr' && matrix.remote_storage == null && github.event.pull_request.head.repo.fork == false - uses: thollander/actions-comment-pull-request@v3 + uses: thollander/actions-comment-pull-request@24bffb9b452ba05a4f3f77933840a6a841d1b32b # v3 with: file-path: sizes-comment.md comment-tag: file-sizes-${{ matrix.id }} - name: Comment PR on failure if: failure() && inputs.mode == 'pr' && github.event.pull_request.head.repo.fork == false - uses: thollander/actions-comment-pull-request@v3 + uses: thollander/actions-comment-pull-request@24bffb9b452ba05a4f3f77933840a6a841d1b32b # v3 with: message: | # 🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨 diff --git a/.github/workflows/stale.yml b/.github/workflows/stale.yml index 672be333f07..0acb0a15a55 100644 --- a/.github/workflows/stale.yml +++ b/.github/workflows/stale.yml @@ -2,22 +2,23 @@ name: Close inactive PRs on: schedule: - cron: "30 1 * * *" - workflow_dispatch: + workflow_dispatch: { } jobs: close-issues: runs-on: ubuntu-latest + timeout-minutes: 10 permissions: pull-requests: write steps: - - uses: actions/stale@v10 + - uses: actions/stale@b5d41d4e1d5dceea10e7104786b73624c18a190f # v10 with: # After 30 days of no activity, a PR will be marked as stale - days-before-pr-stale: 30 + days-before-pr-stale: 14 # PR has 7 more days to become active, otherwise it will be closed. days-before-pr-close: 7 stale-pr-label: "stale" - stale-pr-message: "This PR has been marked as stale because it has been open for 30 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days" + stale-pr-message: "This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days" close-pr-message: "This PR was closed because it has been inactive for 7 days since being marked as stale." days-before-issue-stale: -1 days-before-issue-close: -1 diff --git a/.github/workflows/typos.yml b/.github/workflows/typos.yml index 6f51e873c61..f73ca0e0a0f 100644 --- a/.github/workflows/typos.yml +++ b/.github/workflows/typos.yml @@ -16,8 +16,9 @@ jobs: spelling: name: Spell Check with Typos runs-on: ubuntu-latest + timeout-minutes: 10 steps: - name: Checkout Actions Repository - uses: actions/checkout@v6 + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - name: Spell Check Repo - uses: crate-ci/typos@v1.44.0 + uses: crate-ci/typos@cf5f1c29a8ac336af8568821ec41919923b05a83 # v1.45.1 diff --git a/.github/workflows/wasm-fuzz.yml b/.github/workflows/wasm-fuzz.yml index 4ff51ee78f3..f076890c5b1 100644 --- a/.github/workflows/wasm-fuzz.yml +++ b/.github/workflows/wasm-fuzz.yml @@ -23,9 +23,9 @@ jobs: wasm-fuzz: name: "Build & Fuzz WASM" runs-on: ubuntu-latest - timeout-minutes: 270 + timeout-minutes: 240 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - uses: ./.github/actions/setup-rust with: @@ -63,7 +63,7 @@ jobs: # Capture exit code - wasmfuzz exits with error on crash set +e wasmfuzz fuzz \ - --timeout=4h \ + --timeout=3h30m \ --cores 2 \ --dir corpus-wasm/ \ target/wasm32-wasip1/release/array_ops_wasm.wasm @@ -85,7 +85,7 @@ jobs: - name: Upload corpus if: always() - uses: actions/upload-artifact@v7 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: name: corpus-wasm path: corpus-wasm/ diff --git a/.github/workflows/web.yml b/.github/workflows/web.yml index 6fd0a72ea83..b429e398b3b 100644 --- a/.github/workflows/web.yml +++ b/.github/workflows/web.yml @@ -7,7 +7,7 @@ concurrency: cancel-in-progress: true on: - pull_request: + pull_request: { } push: branches: [develop] @@ -22,13 +22,14 @@ jobs: changes: name: Detect Changes runs-on: ubuntu-latest + timeout-minutes: 10 permissions: pull-requests: read outputs: web: ${{ steps.filter.outputs.web }} steps: - - uses: actions/checkout@v6 - - uses: dorny/paths-filter@v3 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 + - uses: dorny/paths-filter@fbd0ab8f3e69293af611ebaee6363fc25e6d187d # v4 id: filter with: filters: | @@ -43,7 +44,7 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 30 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - uses: ./.github/actions/setup-rust with: repo-token: ${{ secrets.GITHUB_TOKEN }} @@ -51,13 +52,15 @@ jobs: - name: Install wasm-pack run: curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh working-directory: . - - uses: actions/setup-node@v4 + - uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6 with: - node-version: "22" + node-version: "24" cache: "npm" cache-dependency-path: vortex-web/package-lock.json - run: npm ci - - run: npm run wasm + - env: + RUSTFLAGS: --cfg getrandom_backend="unsupported" + run: npm run wasm - run: npm run format:check - run: npm run lint - run: npm run typecheck @@ -70,7 +73,7 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 30 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 - uses: ./.github/actions/setup-rust with: repo-token: ${{ secrets.GITHUB_TOKEN }} @@ -78,15 +81,17 @@ jobs: - name: Install wasm-pack run: curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh working-directory: . - - uses: actions/setup-node@v4 + - uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6 with: - node-version: "22" + node-version: "24" cache: "npm" cache-dependency-path: vortex-web/package-lock.json - run: npm ci - - run: npm run build + - env: + RUSTFLAGS: --cfg getrandom_backend="unsupported" + run: npm run build - name: Upload build artifact - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: name: vortex-explorer path: vortex-web/dist/ @@ -102,13 +107,13 @@ jobs: name: github-pages url: ${{ steps.deploy.outputs.deployment-url }} steps: - - uses: actions/download-artifact@v4 + - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 with: name: vortex-explorer path: dist - name: Deploy to Cloudflare Pages id: deploy - uses: cloudflare/wrangler-action@v3 + uses: cloudflare/wrangler-action@9acf94ace14e7dc412b076f2c5c20b8ce93c79cd # v3 with: apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }} accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }} diff --git a/.gitignore b/.gitignore index 3eaf3c2579f..7fa79fb2162 100644 --- a/.gitignore +++ b/.gitignore @@ -223,7 +223,8 @@ compile_commands.json /justfile # AI Agents -.claude +.agents/settings.local.json +.claude/settings.local.json .opencode # cargo sweep output diff --git a/.yamllint.yaml b/.yamllint.yaml index 5fd2694dabf..85f7142a9a9 100644 --- a/.yamllint.yaml +++ b/.yamllint.yaml @@ -29,7 +29,7 @@ rules: document-end: disable document-start: disable empty-lines: enable - empty-values: disable + empty-values: enable float-values: disable hyphens: enable indentation: enable @@ -38,9 +38,10 @@ rules: line-length: disable new-line-at-end-of-file: enable new-lines: enable - octal-values: disable + octal-values: enable quoted-strings: quote-type: double required: false trailing-spaces: enable - truthy: disable + truthy: + check-keys: false diff --git a/AGENTS.md b/AGENTS.md index f4cc4620f00..6ba6006251f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,81 +1,174 @@ -# Vortex - -## Development Guidelines - -- project is a monorepo Rust workspace, java bindings in `/java`, python bindings in - `/vortex-python` -- run `cargo build -p` to build a specific crate -- use `cargo clippy --all-targets --all-features` to make sure a project is free of lint issues. - Please do this every time you reach a stopping point or think you've finished work. -- run `cargo +nightly fmt --all` to format Rust source files. Please do this every time you reach a - stopping point or think you've finished work. -- run `./scripts/public-api.sh` to re-generate the public API lock files. Please do this every time - you reach a stopping point or think you've finished work. -- you can try running +# AGENTS.md + +Guidance for Claude, Codex and other coding agents working in the Vortex repository. + +## Task Routing + +- When asked to review a PR, especially via `/pr-review`, use the + `.agents/skills/pr-review` skill. +- When asked a question about the PR or codebase, especially via `/query`, use the + `.agents/skills/query` skill. +- When asked to investigate a CI failure, especially via `/ci-failure-analysis`, use the + `.agents/skills/ci-failure-analysis` skill. + +## Overview + +Vortex is a Rust monorepo for columnar array processing, compression encodings, and file IO. +The workspace also contains Java bindings in `java/`, Python bindings in `vortex-python/`, +documentation in `docs/`, and benchmark tooling in `vortex-bench/` and `benchmarks/`. + +## Repository Layout + +- `vortex-buffer` defines zero-copy aligned `Buffer` and `BufferMut`, guaranteed to + be aligned to `T` or to a requested runtime alignment. +- `vortex-array/src/dtype` contains the `DType` logical type system used throughout Vortex. +- `vortex-array` contains the core `Array` trait and the base encodings, including most + Apache Arrow-style encodings. +- `encodings/*` contains more specialized compressed encodings. +- `vortex-file` implements file IO. It uses `LayoutReader` from `vortex-layout`. +- `vortex-scan`, `vortex-session`, `vortex-datafusion`, and `vortex-duckdb` contain scan + and execution integrations. +- `vortex-python` contains Python bindings. RST-flavored project docs live in `docs/`. + +## Build + +Prefer narrow crate builds while iterating: + +```bash +cargo build -p +``` + +Use workspace-wide builds only when the change spans crate boundaries or before handing off a +broad refactor: + +```bash +cargo build --workspace +``` + +## Testing + +Run tests for the crate or binding you touched before broader checks: + +```bash +cargo nextest run -p +``` + +if cargo-nextest is not available you can install it with +```bash +cargo install --locked cargo-nextest +``` + +Examples: + +```bash +cargo nextest run -p vortex-array +make -C docs doctest +uv run --all-packages pytest vortex-python/test +cargo test --doc +``` + +Run docs doctests from the docs directory with `make -C docs doctest` so the correct Sphinx +Makefile target is used. + +If you touch documentation run doc tests via `cargo test --doc`. + +## Linting, Formatting, and Generated Files + +Run verification that matches the files changed. Do not run expensive Rust checks for changes that +only touch Markdown, agent configuration, comments outside Rust code, symlinks, or other metadata +with no Rust/API behavior impact. For docs/config-only changes, validate formatting by inspection +or with a targeted doc/config command, and verify symlink or path changes with `ls`, `find`, and +`git status`. + +For Rust code, public API, feature flag, or generated-file changes, run these before stopping: + +```bash +cargo +nightly fmt --all +./scripts/public-api.sh +cargo clippy --all-targets --all-features +``` + +Notes: + +- `./scripts/public-api.sh` regenerates all `public-api.lock` files through the `xtask` + wrapper. Run it when public Rust APIs may have changed, not for docs-only or agent-only edits. +- For `.github/` changes, follow `.github/AGENTS.md` and run + `yamllint --strict -c .yamllint.yaml` on changed workflow files. +- You can try `cargo fix --lib --allow-dirty --allow-staged && cargo clippy --fix --lib --allow-dirty --allow-staged` - to automatically many fix minor errors. -- when iterating on CI failures, fetch only failed job logs first - (`gh run view --job --log-failed`) and run narrow local repro commands for the - affected crate/tests before running workspace-wide checks. -- if `gh` commands fail with `error connecting to api.github.com` in sandbox, immediately rerun with - escalated network permissions instead of retrying in sandbox. -- if cargo fails with `sccache: error: Operation not permitted`, rerun the command with - `RUSTC_WRAPPER=` so rustc runs directly. -- run docs doctests from the docs directory (`make -C docs doctest`) so the correct Sphinx Makefile - target is used. - -## Architecture - -- `vortex-buffer` defines zero-copy aligned `Buffer` and `BufferMut` that are guaranteed to be - aligned to `T` (or whatever requested runtime alignment). -- `vortex-array/src/dtype` contains the basic `DType` logical type enum that is the basis of the - Vortex type system -- `vortex-array` contains the basic `Array` trait, as well as several encodings which impl that - trait for each encoding. It includes all of most of the Apache Arrow encodings. -- More exotic compressed encodings live in the crates inside of `/encodings/*` -- File IO is defined in `vortex-file`. It uses the concept of a `LayoutReader` defined in - `vortex-layout` crate. -- `/vortex-python` contains the python bindings. rst flavored docs for the project are in `/docs` - -## Code Style - -- Prefer `impl AsRef` to `&T` for public interfaces where possible, e.g. `impl AsRef` -- Avoid usage of unsafe where not necessary, use zero-cost safe abstractions wherever possible, or - cheap non-zero-cost abstractions. -- Every new public API definition must have a doc comment. Examples are nice to have but not - strictly required. -- Use `vortex_err!` to create a `VortexError` with a format string and `vortex_bail!` to do the same - but immediately return it as a `VortexResult` to the surrounding context. -- When writing tests, strongly consider using `rstest` cases to parameterize repetitive test logic. -- If you want to create a large number of tests to an existing file module called `foo.rs`, and if - you think doing so would be too many to inline in a `tests` submodule within `foo.rs`, then first - promote `foo` to a directory module. You can do this by running - `mkdir foo && mv foo.rs foo/mod.rs`. Then, you can create a test file `foo/tests.rs` that you - include in `foo/mod.rs` with the appropriate test config flag. -- If you encounter clippy errors in tests that should only pertain to production code (e.g., - prohibiting panic/unwrap, possible numerical truncation, etc.), then consider allowing those lints - at the test module level. -- Prefer naming test modules `tests`, not `test`. -- Prefer having test return VortexResult<()> and use ? over unwrap. -- All imports must be at the top of the module, never inside functions. The only exception is - `#[cfg(test)]` blocks, where imports should be at the top of the test module. Function-scoped - imports are only acceptable when (a) required, or (b) it would be exceptionally verbose otherwise, - such as a match statement where left and right sides have similar names. -- Imports should be preferred over qualified identifiers. -- Only write comments that explain non-obvious logic or important context. Avoid commenting simple - or self-explanatory code. -- Use `assert_arrays_eq!` macro for comparing arrays in tests instead of element-by-element - comparison. -- Keep tests concise and to the point - avoid unnecessary setup or verbose assertions. -- Run tests for a specific crate with `cargo test -p ` (e.g., - `cargo test -p vortex-array`). - -## Other - -- When summarizing your work, please produce summaries in valid Markdown that can be easily - copied/pasted to Github. + to fix minor Rust diagnostics automatically when working on Rust code. +- If cargo fails with exactly `sccache: error: Operation not permitted`, rerun that command + with `RUSTC_WRAPPER=` so rustc runs directly. Only do this for that exact error. + +## CI Investigation + +- When iterating on CI failures, fetch only failed job logs first: + `gh run view --job --log-failed`. +- Run narrow local repro commands for the affected crate, test, docs target, or binding before + running workspace-wide checks. +- If a `gh` command fails with `error connecting to api.github.com` in the sandbox, immediately + rerun it with escalated network permissions instead of retrying in the sandbox. +- Verify causation from logs, diffs, and local repros before attributing a failure to a PR. + +## Rust Code Style + +- Prefer `impl AsRef` to `&T` for public interfaces where practical, for example + `impl AsRef`. +- Avoid `unsafe` unless it is necessary. Prefer zero-cost safe abstractions, or cheap + non-zero-cost safe abstractions, over hand-written unsafe code. +- Every new public API definition must have a doc comment. Examples are useful but not required. +- Use `vortex_err!` to create a `VortexError` with a format string. +- Use `vortex_bail!` to create and immediately return a `VortexError` as a `VortexResult`. +- Keep imports at the top of the module. The only exception is a `#[cfg(test)]` test module, + where imports should be at the top of that module. +- Prefer imports over qualified identifiers when the name is used repeatedly. +- Avoid function-scoped imports unless required or unless fully qualifying both sides would be + exceptionally verbose. +- Only write comments that explain non-obvious logic or important context. Do not comment + self-explanatory code. +- Keep public APIs small and consistent with neighboring crates. + +## Tests + +- Strongly consider `rstest` cases when parameterizing repetitive test logic. +- Prefer test functions that return `VortexResult<()>` and use `?` instead of `unwrap`. +- Prefer test module names `tests`, not `test`. +- Use `assert_arrays_eq!` for array comparisons instead of element-by-element assertions. +- Keep tests concise and focused on behavior, edge cases, and regressions. +- If a bug fix is requested, add or identify a failing test first when practical. A test that + passes before and after the fix does not prove the fix. +- If clippy lints in tests prohibit patterns that are acceptable only in test code, consider + allowing the lint at the test module level. +- If an existing `foo.rs` module needs many tests, promote it to a directory module: + `foo/mod.rs` plus `foo/tests.rs`, included from `foo/mod.rs` behind the appropriate test + configuration. + +## Common Mistakes + +Check new and modified lines against this list before finishing: + +- Public API changes without doc comments or refreshed `public-api.lock` files. +- Running `cargo fmt`, `./scripts/public-api.sh`, or workspace clippy for docs-only, agent-only, + symlink-only, or other metadata-only changes. +- Running broad CI-style commands before trying a narrow local repro. +- Using `unwrap`, `expect`, or panic-oriented assertions in tests where `VortexResult<()>` and + `?` would be clearer. +- Comparing arrays element by element instead of using `assert_arrays_eq!`. +- Adding imports inside functions when module-level imports would work. +- Introducing `unsafe` without proving that safe Rust cannot express the same operation. +- Updating expected test output to match buggy behavior without independently verifying the + intended semantics. +- Silently reducing the scope of an approved plan when implementation is harder than expected. + +## Summaries + +When summarizing work, write valid Markdown that can be copied into GitHub. Include the checks +you ran and call out any checks you could not run. ## Commits -- All commits must be signed of by the committers in the form - `Signed-off-by: "COMMITTER" `. +All commits must be signed off by the committers in this form: + +```text +Signed-off-by: "COMMITTER" +``` diff --git a/Cargo.lock b/Cargo.lock index 5c7f200d756..d223d120703 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -10,13 +10,13 @@ checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" [[package]] name = "aes" -version = "0.8.4" +version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0" +checksum = "66bd29a732b644c0431c6140f370d097879203d79b80c94a6747ba0872adaef8" dependencies = [ - "cfg-if", "cipher", - "cpufeatures 0.2.17", + "cpubits", + "cpufeatures 0.3.0", ] [[package]] @@ -89,21 +89,6 @@ version = "0.1.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4b46cbb362ab8752921c97e041f5e366ee6297bd428a31275b9fcf1e380f7299" -[[package]] -name = "anstream" -version = "0.6.21" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "43d5b281e737544384e969a5ccad3f1cdd24b48086a0fc1b2a5262a26b8f4f4a" -dependencies = [ - "anstyle", - "anstyle-parse 0.2.7", - "anstyle-query", - "anstyle-wincon", - "colorchoice", - "is_terminal_polyfill", - "utf8parse", -] - [[package]] name = "anstream" version = "1.0.0" @@ -111,7 +96,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "824a212faf96e9acacdbd09febd34438f8f711fb84e09a8916013cd7815ca28d" dependencies = [ "anstyle", - "anstyle-parse 1.0.0", + "anstyle-parse", "anstyle-query", "anstyle-wincon", "colorchoice", @@ -125,15 +110,6 @@ version = "1.0.14" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000" -[[package]] -name = "anstyle-parse" -version = "0.2.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4e7644824f0aa2c7b9384579234ef10eb7efb6a0deb83f9630a49594dd9c15c2" -dependencies = [ - "utf8parse", -] - [[package]] name = "anstyle-parse" version = "1.0.0" @@ -179,13 +155,13 @@ dependencies = [ "bon", "bzip2", "crc32fast", - "digest", + "digest 0.10.7", "liblzma", "log", "miniz_oxide", "num-bigint", "quad-rand", - "rand 0.9.2", + "rand 0.9.4", "regex-lite", "serde", "serde_bytes", @@ -224,9 +200,9 @@ checksum = "c3d036a3c4ab069c7b410a2ce876bd74808d2d0888a82667669f8e783a898bf1" [[package]] name = "arc-swap" -version = "1.9.0" +version = "1.9.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a07d1f37ff60921c83bdfc7407723bdefe89b44b98a9b772f225c8f9d67141a6" +checksum = "6a3a1fd6f75306b68087b831f025c712524bcb19aad54e557b1129cfa0a2b207" dependencies = [ "rustversion", ] @@ -272,23 +248,23 @@ dependencies = [ [[package]] name = "arrow" -version = "58.0.0" +version = "58.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "602268ce9f569f282cedb9a9f6bac569b680af47b9b077d515900c03c5d190da" +checksum = "d441fdda254b65f3e9025910eb2c2066b6295d9c8ed409522b8d2ace1ff8574c" dependencies = [ - "arrow-arith 58.0.0", - "arrow-array 58.0.0", - "arrow-buffer 58.0.0", - "arrow-cast 58.0.0", - "arrow-csv 58.0.0", - "arrow-data 58.0.0", - "arrow-ipc 58.0.0", - "arrow-json 58.0.0", - "arrow-ord 58.0.0", - "arrow-row 58.0.0", - "arrow-schema 58.0.0", - "arrow-select 58.0.0", - "arrow-string 58.0.0", + "arrow-arith 58.1.0", + "arrow-array 58.1.0", + "arrow-buffer 58.1.0", + "arrow-cast 58.1.0", + "arrow-csv 58.1.0", + "arrow-data 58.1.0", + "arrow-ipc 58.1.0", + "arrow-json 58.1.0", + "arrow-ord 58.1.0", + "arrow-row 58.1.0", + "arrow-schema 58.1.0", + "arrow-select 58.1.0", + "arrow-string 58.1.0", ] [[package]] @@ -307,14 +283,14 @@ dependencies = [ [[package]] name = "arrow-arith" -version = "58.0.0" +version = "58.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cd53c6bf277dea91f136ae8e3a5d7041b44b5e489e244e637d00ae302051f56f" +checksum = "ced5406f8b720cc0bc3aa9cf5758f93e8593cda5490677aa194e4b4b383f9a59" dependencies = [ - "arrow-array 58.0.0", - "arrow-buffer 58.0.0", - "arrow-data 58.0.0", - "arrow-schema 58.0.0", + "arrow-array 58.1.0", + "arrow-buffer 58.1.0", + "arrow-data 58.1.0", + "arrow-schema 58.1.0", "chrono", "num-traits", ] @@ -340,14 +316,14 @@ dependencies = [ [[package]] name = "arrow-array" -version = "58.0.0" +version = "58.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e53796e07a6525edaf7dc28b540d477a934aff14af97967ad1d5550878969b9e" +checksum = "772bd34cacdda8baec9418d80d23d0fb4d50ef0735685bd45158b83dfeb6e62d" dependencies = [ "ahash 0.8.12", - "arrow-buffer 58.0.0", - "arrow-data 58.0.0", - "arrow-schema 58.0.0", + "arrow-buffer 58.1.0", + "arrow-data 58.1.0", + "arrow-schema 58.1.0", "chrono", "chrono-tz", "half", @@ -371,9 +347,9 @@ dependencies = [ [[package]] name = "arrow-buffer" -version = "58.0.0" +version = "58.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2c1a85bb2e94ee10b76531d8bc3ce9b7b4c0d508cabfb17d477f63f2617bd20" +checksum = "898f4cf1e9598fdb77f356fdf2134feedfd0ee8d5a4e0a5f573e7d0aec16baa4" dependencies = [ "bytes", "half", @@ -405,16 +381,16 @@ dependencies = [ [[package]] name = "arrow-cast" -version = "58.0.0" +version = "58.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "89fb245db6b0e234ed8e15b644edb8664673fefe630575e94e62cd9d489a8a26" +checksum = "b0127816c96533d20fc938729f48c52d3e48f99717e7a0b5ade77d742510736d" dependencies = [ - "arrow-array 58.0.0", - "arrow-buffer 58.0.0", - "arrow-data 58.0.0", - "arrow-ord 58.0.0", - "arrow-schema 58.0.0", - "arrow-select 58.0.0", + "arrow-array 58.1.0", + "arrow-buffer 58.1.0", + "arrow-data 58.1.0", + "arrow-ord 58.1.0", + "arrow-schema 58.1.0", + "arrow-select 58.1.0", "atoi", "base64", "chrono", @@ -442,13 +418,13 @@ dependencies = [ [[package]] name = "arrow-csv" -version = "58.0.0" +version = "58.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d374882fb465a194462527c0c15a93aa19a554cf690a6b77a26b2a02539937a7" +checksum = "ca025bd0f38eeecb57c2153c0123b960494138e6a957bbda10da2b25415209fe" dependencies = [ - "arrow-array 58.0.0", - "arrow-cast 58.0.0", - "arrow-schema 58.0.0", + "arrow-array 58.1.0", + "arrow-cast 58.1.0", + "arrow-schema 58.1.0", "chrono", "csv", "csv-core", @@ -470,12 +446,12 @@ dependencies = [ [[package]] name = "arrow-data" -version = "58.0.0" +version = "58.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "189d210bc4244c715fa3ed9e6e22864673cccb73d5da28c2723fb2e527329b33" +checksum = "42d10beeab2b1c3bb0b53a00f7c944a178b622173a5c7bcabc3cb45d90238df4" dependencies = [ - "arrow-buffer 58.0.0", - "arrow-schema 58.0.0", + "arrow-buffer 58.1.0", + "arrow-schema 58.1.0", "half", "num-integer", "num-traits", @@ -499,17 +475,17 @@ dependencies = [ [[package]] name = "arrow-ipc" -version = "58.0.0" +version = "58.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7968c2e5210c41f4909b2ef76f6e05e172b99021c2def5edf3cc48fdd39d1d6c" +checksum = "609a441080e338147a84e8e6904b6da482cefb957c5cdc0f3398872f69a315d0" dependencies = [ - "arrow-array 58.0.0", - "arrow-buffer 58.0.0", - "arrow-data 58.0.0", - "arrow-schema 58.0.0", - "arrow-select 58.0.0", + "arrow-array 58.1.0", + "arrow-buffer 58.1.0", + "arrow-data 58.1.0", + "arrow-schema 58.1.0", + "arrow-select 58.1.0", "flatbuffers", - "lz4_flex 0.12.1", + "lz4_flex 0.13.0", "zstd", ] @@ -539,15 +515,15 @@ dependencies = [ [[package]] name = "arrow-json" -version = "58.0.0" +version = "58.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "92111dba5bf900f443488e01f00d8c4ddc2f47f5c50039d18120287b580baa22" +checksum = "6ead0914e4861a531be48fe05858265cf854a4880b9ed12618b1d08cba9bebc8" dependencies = [ - "arrow-array 58.0.0", - "arrow-buffer 58.0.0", - "arrow-cast 58.0.0", - "arrow-data 58.0.0", - "arrow-schema 58.0.0", + "arrow-array 58.1.0", + "arrow-buffer 58.1.0", + "arrow-cast 58.1.0", + "arrow-data 58.1.0", + "arrow-schema 58.1.0", "chrono", "half", "indexmap", @@ -576,15 +552,15 @@ dependencies = [ [[package]] name = "arrow-ord" -version = "58.0.0" +version = "58.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "211136cb253577ee1a6665f741a13136d4e563f64f5093ffd6fb837af90b9495" +checksum = "763a7ba279b20b52dad300e68cfc37c17efa65e68623169076855b3a9e941ca5" dependencies = [ - "arrow-array 58.0.0", - "arrow-buffer 58.0.0", - "arrow-data 58.0.0", - "arrow-schema 58.0.0", - "arrow-select 58.0.0", + "arrow-array 58.1.0", + "arrow-buffer 58.1.0", + "arrow-data 58.1.0", + "arrow-schema 58.1.0", + "arrow-select 58.1.0", ] [[package]] @@ -602,14 +578,14 @@ dependencies = [ [[package]] name = "arrow-row" -version = "58.0.0" +version = "58.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8e0f20145f9f5ea3fe383e2ba7a7487bf19be36aa9dbf5dd6a1f92f657179663" +checksum = "e14fe367802f16d7668163ff647830258e6e0aeea9a4d79aaedf273af3bdcd3e" dependencies = [ - "arrow-array 58.0.0", - "arrow-buffer 58.0.0", - "arrow-data 58.0.0", - "arrow-schema 58.0.0", + "arrow-array 58.1.0", + "arrow-buffer 58.1.0", + "arrow-data 58.1.0", + "arrow-schema 58.1.0", "half", ] @@ -626,9 +602,9 @@ dependencies = [ [[package]] name = "arrow-schema" -version = "58.0.0" +version = "58.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1b47e0ca91cc438d2c7879fe95e0bca5329fff28649e30a88c6f760b1faeddcb" +checksum = "c30a1365d7a7dc50cc847e54154e6af49e4c4b0fddc9f607b687f29212082743" dependencies = [ "bitflags", "serde_core", @@ -651,15 +627,15 @@ dependencies = [ [[package]] name = "arrow-select" -version = "58.0.0" +version = "58.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "750a7d1dda177735f5e82a314485b6915c7cccdbb278262ac44090f4aba4a325" +checksum = "78694888660a9e8ac949853db393af2a8b8fc82c19ce333132dfa2e72cc1a7fe" dependencies = [ "ahash 0.8.12", - "arrow-array 58.0.0", - "arrow-buffer 58.0.0", - "arrow-data 58.0.0", - "arrow-schema 58.0.0", + "arrow-array 58.1.0", + "arrow-buffer 58.1.0", + "arrow-data 58.1.0", + "arrow-schema 58.1.0", "num-traits", ] @@ -682,15 +658,15 @@ dependencies = [ [[package]] name = "arrow-string" -version = "58.0.0" +version = "58.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e1eab1208bc4fe55d768cdc9b9f3d9df5a794cdb3ee2586bf89f9b30dc31ad8c" +checksum = "61e04a01f8bb73ce54437514c5fd3ee2aa3e8abe4c777ee5cc55853b1652f79e" dependencies = [ - "arrow-array 58.0.0", - "arrow-buffer 58.0.0", - "arrow-data 58.0.0", - "arrow-schema 58.0.0", - "arrow-select 58.0.0", + "arrow-array 58.1.0", + "arrow-buffer 58.1.0", + "arrow-data 58.1.0", + "arrow-schema 58.1.0", + "arrow-select 58.1.0", "memchr", "num-traits", "regex", @@ -817,9 +793,9 @@ dependencies = [ [[package]] name = "async-signal" -version = "0.2.13" +version = "0.2.14" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "43c070bbf59cd3570b6b2dd54cd772527c7c3620fce8be898406dd3ed6adc64c" +checksum = "52b5aaafa020cf5053a01f2a60e8ff5dccf550f0f77ec54a4e47285ac2bab485" dependencies = [ "async-io", "async-lock", @@ -904,9 +880,9 @@ checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8" [[package]] name = "aws-lc-rs" -version = "1.16.2" +version = "1.16.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a054912289d18629dc78375ba2c3726a3afe3ff71b4edba9dedfca0e3446d1fc" +checksum = "0ec6fb3fe69024a75fa7e1bfb48aa6cf59706a101658ea01bfd33b2b248a038f" dependencies = [ "aws-lc-sys", "zeroize", @@ -914,9 +890,9 @@ dependencies = [ [[package]] name = "aws-lc-sys" -version = "0.39.1" +version = "0.40.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "83a25cf98105baa966497416dbd42565ce3a8cf8dbfd59803ec9ad46f3126399" +checksum = "f50037ee5e1e41e7b8f9d161680a725bd1626cb6f8c7e901f91f942850852fe7" dependencies = [ "cc", "cmake", @@ -924,6 +900,12 @@ dependencies = [ "fs_extra", ] +[[package]] +name = "base16ct" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fd307490d624467aa6f74b0eabb77633d1f758a7b25f12bceb0b22e08d9726f6" + [[package]] name = "base64" version = "0.22.1" @@ -1007,9 +989,9 @@ dependencies = [ [[package]] name = "bitflags" -version = "2.11.0" +version = "2.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "843867be96c8daad0d758b57df9392b6d8d271134fce549de6ce169ff98a92af" +checksum = "c4512299f36f043ab09a583e57bceb5a5aab7a73db1805848e8fef3c9e8c78b3" [[package]] name = "bitpacking" @@ -1038,21 +1020,21 @@ version = "0.10.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "46502ad458c9a52b69d4d4d32775c788b7a1b85e8bc9d482d92250fc0e3f8efe" dependencies = [ - "digest", + "digest 0.10.7", ] [[package]] name = "blake3" -version = "1.8.3" +version = "1.8.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2468ef7d57b3fb7e16b576e8377cdbde2320c60e1491e961d11da40fc4f02a2d" +checksum = "4d2d5991425dfd0785aed03aedcf0b321d61975c9b5b3689c774a2610ae0b51e" dependencies = [ "arrayref", "arrayvec", "cc", "cfg-if", "constant_time_eq", - "cpufeatures 0.2.17", + "cpufeatures 0.3.0", ] [[package]] @@ -1064,6 +1046,16 @@ dependencies = [ "generic-array", ] +[[package]] +name = "block-buffer" +version = "0.12.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cdd35008169921d80bc60d3d0ab416eecb028c4cd653352907921d95084790be" +dependencies = [ + "hybrid-array", + "zeroize", +] + [[package]] name = "blocking" version = "1.6.2" @@ -1248,9 +1240,9 @@ dependencies = [ [[package]] name = "cargo-platform" -version = "0.3.2" +version = "0.3.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "87a0c0e6148f11f01f32650a2ea02d532b2ad4e81d8bd41e6e565b5adc5e6082" +checksum = "dd0061da739915fae12ea00e16397555ed4371a6bb285431aab930f61b0aa4ba" dependencies = [ "serde", "serde_core", @@ -1306,9 +1298,9 @@ dependencies = [ [[package]] name = "cc" -version = "1.2.57" +version = "1.2.61" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7a0dd1ca384932ff3641c8718a02769f1698e7563dc6974ffd03346116310423" +checksum = "d16d90359e986641506914ba71350897565610e87ce0ad9e6f28569db3dd5c6d" dependencies = [ "find-msvc-tools", "jobserver", @@ -1357,7 +1349,7 @@ checksum = "6f8d983286843e49675a4b7a2d174efe136dc93a18d69130dd18198a6c167601" dependencies = [ "cfg-if", "cpufeatures 0.3.0", - "rand_core 0.10.0", + "rand_core 0.10.1", ] [[package]] @@ -1413,11 +1405,11 @@ dependencies = [ [[package]] name = "cipher" -version = "0.4.4" +version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad" +checksum = "e34d8227fe1ba289043aeb13792056ff80fd6de1a9f49137a5f499de8e8c78ea" dependencies = [ - "crypto-common", + "crypto-common 0.2.1", "inout", ] @@ -1434,9 +1426,9 @@ dependencies = [ [[package]] name = "clap" -version = "4.6.0" +version = "4.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b193af5b67834b676abd72466a96c1024e6a6ad978a1f484bd90b85c94041351" +checksum = "1ddb117e43bbf7dacf0a4190fef4d345b9bad68dfc649cb349e7d17d28428e51" dependencies = [ "clap_builder", "clap_derive", @@ -1448,7 +1440,7 @@ version = "4.6.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "714a53001bf66416adb0e2ef5ac857140e7dc3a0c48fb28b2f10762fc4b5069f" dependencies = [ - "anstream 1.0.0", + "anstream", "anstyle", "clap_lex", "strsim", @@ -1457,9 +1449,9 @@ dependencies = [ [[package]] name = "clap_derive" -version = "4.6.0" +version = "4.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1110bd8a634a1ab8cb04345d8d878267d57c3cf1b38d91b71af6686408bbca6a" +checksum = "f2ce8604710f6733aa641a2b3731eaa1e8b3d9973d5e3565da11800813f997a9" dependencies = [ "heck", "proc-macro2", @@ -1482,6 +1474,12 @@ dependencies = [ "cc", ] +[[package]] +name = "cmov" +version = "0.5.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3f88a43d011fc4a6876cb7344703e297c71dda42494fee094d5f7c76bf13f746" + [[package]] name = "codespan-reporting" version = "0.13.1" @@ -1495,9 +1493,9 @@ dependencies = [ [[package]] name = "codspeed" -version = "4.4.1" +version = "4.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b684e94583e85a5ca7e1a6454a89d76a5121240f2fb67eb564129d9bafdb9db0" +checksum = "c83592369519f73d5731f9c91aa562e213a33cc14bb0f27ac2f5730fd1ecbcec" dependencies = [ "anyhow", "cc", @@ -1513,9 +1511,9 @@ dependencies = [ [[package]] name = "codspeed-criterion-compat-walltime" -version = "4.4.1" +version = "4.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "96389aaa4bbb872ea4924dc0335b2bb181bcf28d6eedbe8fea29afcc5bde36a6" +checksum = "1601803d919d1bca7338b98948a319f0e40275c304516792aeabdd98db6dcd4f" dependencies = [ "anes", "cast", @@ -1540,9 +1538,9 @@ dependencies = [ [[package]] name = "codspeed-divan-compat" -version = "4.4.1" +version = "4.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "89e4bf8c7793c170fd0fcf3be97b9032b2ae39c2b9e8818aba3cc10ca0f0c6c0" +checksum = "a717deb83a7472e31b38244f836c92947ac17890970f61a083495b3a6653bf1b" dependencies = [ "clap", "codspeed", @@ -1553,9 +1551,9 @@ dependencies = [ [[package]] name = "codspeed-divan-compat-macros" -version = "4.4.1" +version = "4.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "78aae02f2a278588e16e8ca62ea1915b8ab30f8230a09926671bba19ede801a4" +checksum = "eed8dc7bd913f259c57e2b94657a4c9fd20b3bded18462b22c647877feb43b53" dependencies = [ "divan-macros", "itertools 0.14.0", @@ -1567,9 +1565,9 @@ dependencies = [ [[package]] name = "codspeed-divan-compat-walltime" -version = "4.4.1" +version = "4.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "59ffd32c0c59ab8b674b15be65ba7c59aebac047036cfa7fa1e11bc2c178b81f" +checksum = "96500330271308ca89e68e1885951716cf2d3895ef9508192376dd125321a8f2" dependencies = [ "cfg-if", "clap", @@ -1593,7 +1591,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "117725a109d387c937a1533ce01b450cbde6b88abceea8473c4d7a85853cda3c" dependencies = [ "lazy_static", - "windows-sys 0.59.0", + "windows-sys 0.52.0", ] [[package]] @@ -1635,8 +1633,8 @@ name = "compress-bench" version = "0.1.0" dependencies = [ "anyhow", - "arrow-array 58.0.0", - "arrow-schema 58.0.0", + "arrow-array 58.1.0", + "arrow-schema 58.1.0", "async-trait", "bytes", "clap", @@ -1644,7 +1642,7 @@ dependencies = [ "indicatif", "itertools 0.14.0", "lance-bench", - "parquet 58.0.0", + "parquet 58.1.0", "regex", "tokio", "tracing", @@ -1722,6 +1720,12 @@ dependencies = [ "wasm-bindgen", ] +[[package]] +name = "const-oid" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c" + [[package]] name = "const-random" version = "0.1.18" @@ -1750,11 +1754,12 @@ checksum = "d9c50fcfdf972929aff202c16b80086aa3cfc6a3a820af714096c58c7c1d0582" [[package]] name = "const_format" -version = "0.2.35" +version = "0.2.36" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7faa7469a93a566e9ccc1c73fe783b4a65c274c5ace346038dca9c39fe0030ad" +checksum = "4481a617ad9a412be3b97c5d403fef8ed023103368908b9c50af598ff467cc1e" dependencies = [ "const_format_proc_macros", + "konst", ] [[package]] @@ -1809,6 +1814,12 @@ version = "0.8.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" +[[package]] +name = "cpubits" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5ef0c543070d296ea414df2dd7625d1b24866ce206709d8a4a424f28377f5861" + [[package]] name = "cpufeatures" version = "0.2.17" @@ -1883,6 +1894,16 @@ dependencies = [ "crossbeam-utils", ] +[[package]] +name = "crossbeam-skiplist" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df29de440c58ca2cc6e587ec3d22347551a32435fbde9d2bff64e78a9ffa151b" +dependencies = [ + "crossbeam-epoch", + "crossbeam-utils", +] + [[package]] name = "crossbeam-utils" version = "0.8.21" @@ -1933,6 +1954,15 @@ dependencies = [ "typenum", ] +[[package]] +name = "crypto-common" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "77727bb15fa921304124b128af125e7e3b968275d1b108b379190264f4423710" +dependencies = [ + "hybrid-array", +] + [[package]] name = "csv" version = "1.4.0" @@ -1954,21 +1984,30 @@ dependencies = [ "memchr", ] +[[package]] +name = "ctutils" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d5515a3834141de9eafb9717ad39eea8247b5674e6066c404e8c4b365d2a29e" +dependencies = [ + "cmov", +] + [[package]] name = "cudarc" -version = "0.18.2" +version = "0.19.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3aa12038120eb13347a6ae2ffab1d34efe78150125108627fd85044dd4d6ff1e" +checksum = "f071cd6a7b5d51607df76aa2d426aaabc7a74bc6bdb885b8afa63a880572ad9b" dependencies = [ "half", - "libloading 0.8.9", + "libloading 0.9.0", ] [[package]] name = "custom-labels" -version = "0.4.5" +version = "0.4.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a750ea4bdb7dbf9584b5d5c668bfa3835f88275781a947b5ea0212945bbdd41f" +checksum = "4121f4f539659ad975da5bc6702b7d7d6de59ff1bbaec328dfe7dd5058052ef2" dependencies = [ "bindgen", "cc", @@ -2088,47 +2127,46 @@ dependencies = [ [[package]] name = "datafusion" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8ba7cb113e9c0bedf9e9765926031e132fa05a1b09ba6e93a6d1a4d7044457b8" +checksum = "7541353e77dc7262b71ca27be07d8393661737e3a73b5d1b1c6f7d814c64fa2a" dependencies = [ "arrow 57.3.0", "arrow-schema 57.3.0", "async-trait", "bytes", "chrono", - "datafusion-catalog 51.0.0", - "datafusion-catalog-listing 51.0.0", - "datafusion-common 51.0.0", - "datafusion-common-runtime 51.0.0", - "datafusion-datasource 51.0.0", - "datafusion-datasource-arrow 51.0.0", - "datafusion-datasource-csv 51.0.0", - "datafusion-datasource-json 51.0.0", - "datafusion-execution 51.0.0", - "datafusion-expr 51.0.0", - "datafusion-expr-common 51.0.0", - "datafusion-functions 51.0.0", - "datafusion-functions-aggregate 51.0.0", - "datafusion-functions-nested 51.0.0", - "datafusion-functions-table 51.0.0", - "datafusion-functions-window 51.0.0", - "datafusion-optimizer 51.0.0", - "datafusion-physical-expr 51.0.0", - "datafusion-physical-expr-adapter 51.0.0", - "datafusion-physical-expr-common 51.0.0", - "datafusion-physical-optimizer 51.0.0", - "datafusion-physical-plan 51.0.0", - "datafusion-session 51.0.0", - "datafusion-sql 51.0.0", + "datafusion-catalog 52.5.0", + "datafusion-catalog-listing 52.5.0", + "datafusion-common 52.5.0", + "datafusion-common-runtime 52.5.0", + "datafusion-datasource 52.5.0", + "datafusion-datasource-arrow 52.5.0", + "datafusion-datasource-csv 52.5.0", + "datafusion-datasource-json 52.5.0", + "datafusion-execution 52.5.0", + "datafusion-expr 52.5.0", + "datafusion-expr-common 52.5.0", + "datafusion-functions 52.5.0", + "datafusion-functions-aggregate 52.5.0", + "datafusion-functions-nested 52.5.0", + "datafusion-functions-table 52.5.0", + "datafusion-functions-window 52.5.0", + "datafusion-optimizer 52.5.0", + "datafusion-physical-expr 52.5.0", + "datafusion-physical-expr-adapter 52.5.0", + "datafusion-physical-expr-common 52.5.0", + "datafusion-physical-optimizer 52.5.0", + "datafusion-physical-plan 52.5.0", + "datafusion-session 52.5.0", + "datafusion-sql 52.5.0", "futures", "itertools 0.14.0", "log", "object_store 0.12.5", "parking_lot", - "rand 0.9.2", + "rand 0.9.4", "regex", - "rstest", "sqlparser 0.59.0", "tempfile", "tokio", @@ -2138,51 +2176,51 @@ dependencies = [ [[package]] name = "datafusion" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "de9f8117889ba9503440f1dd79ebab32ba52ccf1720bb83cd718a29d4edc0d16" +checksum = "93db0e623840612f7f2cd757f7e8a8922064192363732c88692e0870016e141b" dependencies = [ - "arrow 58.0.0", - "arrow-schema 58.0.0", + "arrow 58.1.0", + "arrow-schema 58.1.0", "async-trait", "bytes", "bzip2", "chrono", - "datafusion-catalog 53.0.0", - "datafusion-catalog-listing 53.0.0", - "datafusion-common 53.0.0", - "datafusion-common-runtime 53.0.0", - "datafusion-datasource 53.0.0", - "datafusion-datasource-arrow 53.0.0", + "datafusion-catalog 53.1.0", + "datafusion-catalog-listing 53.1.0", + "datafusion-common 53.1.0", + "datafusion-common-runtime 53.1.0", + "datafusion-datasource 53.1.0", + "datafusion-datasource-arrow 53.1.0", "datafusion-datasource-avro", - "datafusion-datasource-csv 53.0.0", - "datafusion-datasource-json 53.0.0", + "datafusion-datasource-csv 53.1.0", + "datafusion-datasource-json 53.1.0", "datafusion-datasource-parquet", - "datafusion-execution 53.0.0", - "datafusion-expr 53.0.0", - "datafusion-expr-common 53.0.0", - "datafusion-functions 53.0.0", - "datafusion-functions-aggregate 53.0.0", - "datafusion-functions-nested 53.0.0", - "datafusion-functions-table 53.0.0", - "datafusion-functions-window 53.0.0", - "datafusion-optimizer 53.0.0", - "datafusion-physical-expr 53.0.0", - "datafusion-physical-expr-adapter 53.0.0", - "datafusion-physical-expr-common 53.0.0", - "datafusion-physical-optimizer 53.0.0", - "datafusion-physical-plan 53.0.0", - "datafusion-session 53.0.0", - "datafusion-sql 53.0.0", + "datafusion-execution 53.1.0", + "datafusion-expr 53.1.0", + "datafusion-expr-common 53.1.0", + "datafusion-functions 53.1.0", + "datafusion-functions-aggregate 53.1.0", + "datafusion-functions-nested 53.1.0", + "datafusion-functions-table 53.1.0", + "datafusion-functions-window 53.1.0", + "datafusion-optimizer 53.1.0", + "datafusion-physical-expr 53.1.0", + "datafusion-physical-expr-adapter 53.1.0", + "datafusion-physical-expr-common 53.1.0", + "datafusion-physical-optimizer 53.1.0", + "datafusion-physical-plan 53.1.0", + "datafusion-session 53.1.0", + "datafusion-sql 53.1.0", "flate2", "futures", "itertools 0.14.0", "liblzma", "log", - "object_store 0.13.1", + "object_store 0.13.2", "parking_lot", - "parquet 58.0.0", - "rand 0.9.2", + "parquet 58.1.0", + "rand 0.9.4", "regex", "sqlparser 0.61.0", "tempfile", @@ -2199,12 +2237,12 @@ dependencies = [ "anyhow", "clap", "custom-labels", - "datafusion 53.0.0", - "datafusion-common 53.0.0", - "datafusion-physical-plan 53.0.0", + "datafusion 53.1.0", + "datafusion-common 53.1.0", + "datafusion-physical-plan 53.1.0", "futures", "itertools 0.14.0", - "object_store 0.13.1", + "object_store 0.13.2", "opentelemetry", "opentelemetry-otlp", "opentelemetry_sdk", @@ -2220,21 +2258,21 @@ dependencies = [ [[package]] name = "datafusion-catalog" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "66a3a799f914a59b1ea343906a0486f17061f39509af74e874a866428951130d" +checksum = "9997731f90fa5398ef831ad0e69600f92c861b79c0d38bd1a29b6f0e3a0ce4c8" dependencies = [ "arrow 57.3.0", "async-trait", "dashmap", - "datafusion-common 51.0.0", - "datafusion-common-runtime 51.0.0", - "datafusion-datasource 51.0.0", - "datafusion-execution 51.0.0", - "datafusion-expr 51.0.0", - "datafusion-physical-expr 51.0.0", - "datafusion-physical-plan 51.0.0", - "datafusion-session 51.0.0", + "datafusion-common 52.5.0", + "datafusion-common-runtime 52.5.0", + "datafusion-datasource 52.5.0", + "datafusion-execution 52.5.0", + "datafusion-expr 52.5.0", + "datafusion-physical-expr 52.5.0", + "datafusion-physical-plan 52.5.0", + "datafusion-session 52.5.0", "futures", "itertools 0.14.0", "log", @@ -2245,88 +2283,87 @@ dependencies = [ [[package]] name = "datafusion-catalog" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "be893b73a13671f310ffcc8da2c546b81efcc54c22e0382c0a28aa3537017137" +checksum = "37cefde60b26a7f4ff61e9d2ff2833322f91df2b568d7238afe67bde5bdffb66" dependencies = [ - "arrow 58.0.0", + "arrow 58.1.0", "async-trait", "dashmap", - "datafusion-common 53.0.0", - "datafusion-common-runtime 53.0.0", - "datafusion-datasource 53.0.0", - "datafusion-execution 53.0.0", - "datafusion-expr 53.0.0", - "datafusion-physical-expr 53.0.0", - "datafusion-physical-plan 53.0.0", - "datafusion-session 53.0.0", + "datafusion-common 53.1.0", + "datafusion-common-runtime 53.1.0", + "datafusion-datasource 53.1.0", + "datafusion-execution 53.1.0", + "datafusion-expr 53.1.0", + "datafusion-physical-expr 53.1.0", + "datafusion-physical-plan 53.1.0", + "datafusion-session 53.1.0", "futures", "itertools 0.14.0", "log", - "object_store 0.13.1", + "object_store 0.13.2", "parking_lot", "tokio", ] [[package]] name = "datafusion-catalog-listing" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6db1b113c80d7a0febcd901476a57aef378e717c54517a163ed51417d87621b0" +checksum = "2b30a3dd50dec860c9559275c8d97d9de602e611237a6ecfbda0b3b63b872352" dependencies = [ "arrow 57.3.0", "async-trait", - "datafusion-catalog 51.0.0", - "datafusion-common 51.0.0", - "datafusion-datasource 51.0.0", - "datafusion-execution 51.0.0", - "datafusion-expr 51.0.0", - "datafusion-physical-expr 51.0.0", - "datafusion-physical-expr-adapter 51.0.0", - "datafusion-physical-expr-common 51.0.0", - "datafusion-physical-plan 51.0.0", + "datafusion-catalog 52.5.0", + "datafusion-common 52.5.0", + "datafusion-datasource 52.5.0", + "datafusion-execution 52.5.0", + "datafusion-expr 52.5.0", + "datafusion-physical-expr 52.5.0", + "datafusion-physical-expr-adapter 52.5.0", + "datafusion-physical-expr-common 52.5.0", + "datafusion-physical-plan 52.5.0", "futures", "itertools 0.14.0", "log", "object_store 0.12.5", - "tokio", ] [[package]] name = "datafusion-catalog-listing" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "830487b51ed83807d6b32d6325f349c3144ae0c9bf772cf2a712db180c31d5e6" +checksum = "17e112307715d6a7a331111a4c2330ff54bc237183511c319e3708a4cff431fb" dependencies = [ - "arrow 58.0.0", + "arrow 58.1.0", "async-trait", - "datafusion-catalog 53.0.0", - "datafusion-common 53.0.0", - "datafusion-datasource 53.0.0", - "datafusion-execution 53.0.0", - "datafusion-expr 53.0.0", - "datafusion-physical-expr 53.0.0", - "datafusion-physical-expr-adapter 53.0.0", - "datafusion-physical-expr-common 53.0.0", - "datafusion-physical-plan 53.0.0", + "datafusion-catalog 53.1.0", + "datafusion-common 53.1.0", + "datafusion-datasource 53.1.0", + "datafusion-execution 53.1.0", + "datafusion-expr 53.1.0", + "datafusion-physical-expr 53.1.0", + "datafusion-physical-expr-adapter 53.1.0", + "datafusion-physical-expr-common 53.1.0", + "datafusion-physical-plan 53.1.0", "futures", "itertools 0.14.0", "log", - "object_store 0.13.1", + "object_store 0.13.2", ] [[package]] name = "datafusion-common" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7c10f7659e96127d25e8366be7c8be4109595d6a2c3eac70421f380a7006a1b0" +checksum = "d551054acec0398ca604512310b77ce05c46f66e54b54d48200a686e385cca4e" dependencies = [ "ahash 0.8.12", "arrow 57.3.0", "arrow-ipc 57.3.0", "chrono", "half", - "hashbrown 0.14.5", + "hashbrown 0.16.1", "indexmap", "libc", "log", @@ -2339,14 +2376,14 @@ dependencies = [ [[package]] name = "datafusion-common" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0d7663f3af955292f8004e74bcaf8f7ea3d66cc38438749615bb84815b61a293" +checksum = "d72a11ca44a95e1081870d3abb80c717496e8a7acb467a1d3e932bb636af5cc2" dependencies = [ "ahash 0.8.12", "apache-avro", - "arrow 58.0.0", - "arrow-ipc 58.0.0", + "arrow 58.1.0", + "arrow-ipc 58.1.0", "chrono", "half", "hashbrown 0.16.1", @@ -2354,8 +2391,8 @@ dependencies = [ "itertools 0.14.0", "libc", "log", - "object_store 0.13.1", - "parquet 58.0.0", + "object_store 0.13.2", + "parquet 58.1.0", "paste", "recursive", "sqlparser 0.61.0", @@ -2365,9 +2402,9 @@ dependencies = [ [[package]] name = "datafusion-common-runtime" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b92065bbc6532c6651e2f7dd30b55cba0c7a14f860c7e1d15f165c41a1868d95" +checksum = "567d40e285f5b79f8737b576605721cd6c1133b5d2b00bdbd5d9838d90d0812f" dependencies = [ "futures", "log", @@ -2376,9 +2413,9 @@ dependencies = [ [[package]] name = "datafusion-common-runtime" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5f590205c7e32fe1fea48dd53ffb406e56ae0e7a062213a3ac848db8771641bd" +checksum = "89f4afaed29670ec4fd6053643adc749fe3f4bc9d1ce1b8c5679b22c67d12def" dependencies = [ "futures", "log", @@ -2387,62 +2424,62 @@ dependencies = [ [[package]] name = "datafusion-datasource" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fde13794244bc7581cd82f6fff217068ed79cdc344cafe4ab2c3a1c3510b38d6" +checksum = "27d2668f51b3b30befae2207472569e37807fdedd1d14da58acc6f8ca6257eae" dependencies = [ "arrow 57.3.0", "async-trait", "bytes", "chrono", - "datafusion-common 51.0.0", - "datafusion-common-runtime 51.0.0", - "datafusion-execution 51.0.0", - "datafusion-expr 51.0.0", - "datafusion-physical-expr 51.0.0", - "datafusion-physical-expr-adapter 51.0.0", - "datafusion-physical-expr-common 51.0.0", - "datafusion-physical-plan 51.0.0", - "datafusion-session 51.0.0", + "datafusion-common 52.5.0", + "datafusion-common-runtime 52.5.0", + "datafusion-execution 52.5.0", + "datafusion-expr 52.5.0", + "datafusion-physical-expr 52.5.0", + "datafusion-physical-expr-adapter 52.5.0", + "datafusion-physical-expr-common 52.5.0", + "datafusion-physical-plan 52.5.0", + "datafusion-session 52.5.0", "futures", "glob", "itertools 0.14.0", "log", "object_store 0.12.5", - "rand 0.9.2", + "rand 0.9.4", "tokio", "url", ] [[package]] name = "datafusion-datasource" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fde1e030a9dc87b743c806fbd631f5ecfa2ccaa4ffb61fa19144a07fea406b79" +checksum = "e9fb386e1691355355a96419978a0022b7947b44d4a24a6ea99f00b6b485cbb6" dependencies = [ - "arrow 58.0.0", + "arrow 58.1.0", "async-compression", "async-trait", "bytes", "bzip2", "chrono", - "datafusion-common 53.0.0", - "datafusion-common-runtime 53.0.0", - "datafusion-execution 53.0.0", - "datafusion-expr 53.0.0", - "datafusion-physical-expr 53.0.0", - "datafusion-physical-expr-adapter 53.0.0", - "datafusion-physical-expr-common 53.0.0", - "datafusion-physical-plan 53.0.0", - "datafusion-session 53.0.0", + "datafusion-common 53.1.0", + "datafusion-common-runtime 53.1.0", + "datafusion-execution 53.1.0", + "datafusion-expr 53.1.0", + "datafusion-physical-expr 53.1.0", + "datafusion-physical-expr-adapter 53.1.0", + "datafusion-physical-expr-common 53.1.0", + "datafusion-physical-plan 53.1.0", + "datafusion-session 53.1.0", "flate2", "futures", "glob", "itertools 0.14.0", "liblzma", "log", - "object_store 0.13.1", - "rand 0.9.2", + "object_store 0.13.2", + "rand 0.9.4", "tokio", "tokio-util", "url", @@ -2451,22 +2488,22 @@ dependencies = [ [[package]] name = "datafusion-datasource-arrow" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "804fa9b4ecf3157982021770617200ef7c1b2979d57bec9044748314775a9aea" +checksum = "e02e1b3e3a8ec55f1f62de4252b0407c8567363d056078769a197e24fc834a0f" dependencies = [ "arrow 57.3.0", "arrow-ipc 57.3.0", "async-trait", "bytes", - "datafusion-common 51.0.0", - "datafusion-common-runtime 51.0.0", - "datafusion-datasource 51.0.0", - "datafusion-execution 51.0.0", - "datafusion-expr 51.0.0", - "datafusion-physical-expr-common 51.0.0", - "datafusion-physical-plan 51.0.0", - "datafusion-session 51.0.0", + "datafusion-common 52.5.0", + "datafusion-common-runtime 52.5.0", + "datafusion-datasource 52.5.0", + "datafusion-execution 52.5.0", + "datafusion-expr 52.5.0", + "datafusion-physical-expr-common 52.5.0", + "datafusion-physical-plan 52.5.0", + "datafusion-session 52.5.0", "futures", "itertools 0.14.0", "object_store 0.12.5", @@ -2475,65 +2512,65 @@ dependencies = [ [[package]] name = "datafusion-datasource-arrow" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "331ebae7055dc108f9b54994b93dff91f3a17445539efe5b74e89264f7b36e15" +checksum = "ffa6c52cfed0734c5f93754d1c0175f558175248bf686c944fb05c373e5fc096" dependencies = [ - "arrow 58.0.0", - "arrow-ipc 58.0.0", + "arrow 58.1.0", + "arrow-ipc 58.1.0", "async-trait", "bytes", - "datafusion-common 53.0.0", - "datafusion-common-runtime 53.0.0", - "datafusion-datasource 53.0.0", - "datafusion-execution 53.0.0", - "datafusion-expr 53.0.0", - "datafusion-physical-expr-common 53.0.0", - "datafusion-physical-plan 53.0.0", - "datafusion-session 53.0.0", + "datafusion-common 53.1.0", + "datafusion-common-runtime 53.1.0", + "datafusion-datasource 53.1.0", + "datafusion-execution 53.1.0", + "datafusion-expr 53.1.0", + "datafusion-physical-expr-common 53.1.0", + "datafusion-physical-plan 53.1.0", + "datafusion-session 53.1.0", "futures", "itertools 0.14.0", - "object_store 0.13.1", + "object_store 0.13.2", "tokio", ] [[package]] name = "datafusion-datasource-avro" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "49dda81c79b6ba57b1853a9158abc66eb85a3aa1cede0c517dabec6d8a4ed3aa" +checksum = "a579c3bd290c66ea4b269493e75e8a3ed42c9c895a651f10210a29538aee50c4" dependencies = [ "apache-avro", - "arrow 58.0.0", + "arrow 58.1.0", "async-trait", "bytes", - "datafusion-common 53.0.0", - "datafusion-datasource 53.0.0", - "datafusion-physical-expr-common 53.0.0", - "datafusion-physical-plan 53.0.0", - "datafusion-session 53.0.0", + "datafusion-common 53.1.0", + "datafusion-datasource 53.1.0", + "datafusion-physical-expr-common 53.1.0", + "datafusion-physical-plan 53.1.0", + "datafusion-session 53.1.0", "futures", "num-traits", - "object_store 0.13.1", + "object_store 0.13.2", ] [[package]] name = "datafusion-datasource-csv" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "61a1641a40b259bab38131c5e6f48fac0717bedb7dc93690e604142a849e0568" +checksum = "b559d7bf87d4f900f847baba8509634f838d9718695389e903604cdcccdb01f3" dependencies = [ "arrow 57.3.0", "async-trait", "bytes", - "datafusion-common 51.0.0", - "datafusion-common-runtime 51.0.0", - "datafusion-datasource 51.0.0", - "datafusion-execution 51.0.0", - "datafusion-expr 51.0.0", - "datafusion-physical-expr-common 51.0.0", - "datafusion-physical-plan 51.0.0", - "datafusion-session 51.0.0", + "datafusion-common 52.5.0", + "datafusion-common-runtime 52.5.0", + "datafusion-datasource 52.5.0", + "datafusion-execution 52.5.0", + "datafusion-expr 52.5.0", + "datafusion-physical-expr-common 52.5.0", + "datafusion-physical-plan 52.5.0", + "datafusion-session 52.5.0", "futures", "object_store 0.12.5", "regex", @@ -2542,44 +2579,44 @@ dependencies = [ [[package]] name = "datafusion-datasource-csv" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9e0d475088325e2986876aa27bb30d0574f72a22955a527d202f454681d55c5c" +checksum = "503f29e0582c1fc189578d665ff57d9300da1f80c282777d7eb67bb79fb8cdca" dependencies = [ - "arrow 58.0.0", + "arrow 58.1.0", "async-trait", "bytes", - "datafusion-common 53.0.0", - "datafusion-common-runtime 53.0.0", - "datafusion-datasource 53.0.0", - "datafusion-execution 53.0.0", - "datafusion-expr 53.0.0", - "datafusion-physical-expr-common 53.0.0", - "datafusion-physical-plan 53.0.0", - "datafusion-session 53.0.0", + "datafusion-common 53.1.0", + "datafusion-common-runtime 53.1.0", + "datafusion-datasource 53.1.0", + "datafusion-execution 53.1.0", + "datafusion-expr 53.1.0", + "datafusion-physical-expr-common 53.1.0", + "datafusion-physical-plan 53.1.0", + "datafusion-session 53.1.0", "futures", - "object_store 0.13.1", + "object_store 0.13.2", "regex", "tokio", ] [[package]] name = "datafusion-datasource-json" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "adeacdb00c1d37271176f8fb6a1d8ce096baba16ea7a4b2671840c5c9c64fe85" +checksum = "250e2d7591ba8b638f063854650faa40bca4e8bd4059b2ece8836f6388d02db4" dependencies = [ "arrow 57.3.0", "async-trait", "bytes", - "datafusion-common 51.0.0", - "datafusion-common-runtime 51.0.0", - "datafusion-datasource 51.0.0", - "datafusion-execution 51.0.0", - "datafusion-expr 51.0.0", - "datafusion-physical-expr-common 51.0.0", - "datafusion-physical-plan 51.0.0", - "datafusion-session 51.0.0", + "datafusion-common 52.5.0", + "datafusion-common-runtime 52.5.0", + "datafusion-datasource 52.5.0", + "datafusion-execution 52.5.0", + "datafusion-expr 52.5.0", + "datafusion-physical-expr-common 52.5.0", + "datafusion-physical-plan 52.5.0", + "datafusion-session 52.5.0", "futures", "object_store 0.12.5", "tokio", @@ -2587,23 +2624,23 @@ dependencies = [ [[package]] name = "datafusion-datasource-json" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ea1520d81f31770f3ad6ee98b391e75e87a68a5bb90de70064ace5e0a7182fe8" +checksum = "e33804749abc8d0c8cb7473228483cb8070e524c6f6086ee1b85a64debe2b3d2" dependencies = [ - "arrow 58.0.0", + "arrow 58.1.0", "async-trait", "bytes", - "datafusion-common 53.0.0", - "datafusion-common-runtime 53.0.0", - "datafusion-datasource 53.0.0", - "datafusion-execution 53.0.0", - "datafusion-expr 53.0.0", - "datafusion-physical-expr-common 53.0.0", - "datafusion-physical-plan 53.0.0", - "datafusion-session 53.0.0", + "datafusion-common 53.1.0", + "datafusion-common-runtime 53.1.0", + "datafusion-datasource 53.1.0", + "datafusion-execution 53.1.0", + "datafusion-expr 53.1.0", + "datafusion-physical-expr-common 53.1.0", + "datafusion-physical-plan 53.1.0", + "datafusion-session 53.1.0", "futures", - "object_store 0.13.1", + "object_store 0.13.2", "serde_json", "tokio", "tokio-stream", @@ -2611,104 +2648,105 @@ dependencies = [ [[package]] name = "datafusion-datasource-parquet" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "95be805d0742ab129720f4c51ad9242cd872599cdb076098b03f061fcdc7f946" +checksum = "32a8e0365e0e08e8ff94d912f0ababcf9065a1a304018ba90b1fc83c855b4997" dependencies = [ - "arrow 58.0.0", + "arrow 58.1.0", "async-trait", "bytes", - "datafusion-common 53.0.0", - "datafusion-common-runtime 53.0.0", - "datafusion-datasource 53.0.0", - "datafusion-execution 53.0.0", - "datafusion-expr 53.0.0", - "datafusion-functions-aggregate-common 53.0.0", - "datafusion-physical-expr 53.0.0", - "datafusion-physical-expr-adapter 53.0.0", - "datafusion-physical-expr-common 53.0.0", - "datafusion-physical-plan 53.0.0", - "datafusion-pruning 53.0.0", - "datafusion-session 53.0.0", + "datafusion-common 53.1.0", + "datafusion-common-runtime 53.1.0", + "datafusion-datasource 53.1.0", + "datafusion-execution 53.1.0", + "datafusion-expr 53.1.0", + "datafusion-functions-aggregate-common 53.1.0", + "datafusion-physical-expr 53.1.0", + "datafusion-physical-expr-adapter 53.1.0", + "datafusion-physical-expr-common 53.1.0", + "datafusion-physical-plan 53.1.0", + "datafusion-pruning 53.1.0", + "datafusion-session 53.1.0", "futures", "itertools 0.14.0", "log", - "object_store 0.13.1", + "object_store 0.13.2", "parking_lot", - "parquet 58.0.0", + "parquet 58.1.0", "tokio", ] [[package]] name = "datafusion-doc" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2b99e13947667b36ad713549237362afb054b2d8f8cc447751e23ec61202db07" +checksum = "b9496cb0db222dbb9a3735760ceca7fc56f35e1d5502c38d0caa77a81e9c1f6a" [[package]] name = "datafusion-doc" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5c93ad9e37730d2c7196e68616f3f2dd3b04c892e03acd3a8eeca6e177f3c06a" +checksum = "8de6ac0df1662b9148ad3c987978b32cbec7c772f199b1d53520c8fa764a87ee" [[package]] name = "datafusion-execution" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "63695643190679037bc946ad46a263b62016931547bf119859c511f7ff2f5178" +checksum = "dc45d23c516ed8d3637751e44e09e21b45b3f58b473c802dddd1f1ad4fe435ff" dependencies = [ "arrow 57.3.0", "async-trait", + "chrono", "dashmap", - "datafusion-common 51.0.0", - "datafusion-expr 51.0.0", + "datafusion-common 52.5.0", + "datafusion-expr 52.5.0", "futures", "log", "object_store 0.12.5", "parking_lot", - "rand 0.9.2", + "rand 0.9.4", "tempfile", "url", ] [[package]] name = "datafusion-execution" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9437d3cd5d363f9319f8122182d4d233427de79c7eb748f23054c9aaa0fdd8df" +checksum = "c03c7fbdaefcca4ef6ffe425a5fc2325763bfb426599bb0bf4536466efabe709" dependencies = [ - "arrow 58.0.0", - "arrow-buffer 58.0.0", + "arrow 58.1.0", + "arrow-buffer 58.1.0", "async-trait", "chrono", "dashmap", - "datafusion-common 53.0.0", - "datafusion-expr 53.0.0", - "datafusion-physical-expr-common 53.0.0", + "datafusion-common 53.1.0", + "datafusion-expr 53.1.0", + "datafusion-physical-expr-common 53.1.0", "futures", "log", - "object_store 0.13.1", + "object_store 0.13.2", "parking_lot", - "rand 0.9.2", + "rand 0.9.4", "tempfile", "url", ] [[package]] name = "datafusion-expr" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f9a4787cbf5feb1ab351f789063398f67654a6df75c4d37d7f637dc96f951a91" +checksum = "63dd30526d2db4fda6440806a41e4676334a94bc0596cc9cc2a0efed20ef2c44" dependencies = [ "arrow 57.3.0", "async-trait", "chrono", - "datafusion-common 51.0.0", - "datafusion-doc 51.0.0", - "datafusion-expr-common 51.0.0", - "datafusion-functions-aggregate-common 51.0.0", - "datafusion-functions-window-common 51.0.0", - "datafusion-physical-expr-common 51.0.0", + "datafusion-common 52.5.0", + "datafusion-doc 52.5.0", + "datafusion-expr-common 52.5.0", + "datafusion-functions-aggregate-common 52.5.0", + "datafusion-functions-window-common 52.5.0", + "datafusion-physical-expr-common 52.5.0", "indexmap", "itertools 0.14.0", "paste", @@ -2718,19 +2756,19 @@ dependencies = [ [[package]] name = "datafusion-expr" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "67164333342b86521d6d93fa54081ee39839894fb10f7a700c099af96d7552cf" +checksum = "574b9b6977fedbd2a611cbff12e5caf90f31640ad9dc5870f152836d94bad0dd" dependencies = [ - "arrow 58.0.0", + "arrow 58.1.0", "async-trait", "chrono", - "datafusion-common 53.0.0", - "datafusion-doc 53.0.0", - "datafusion-expr-common 53.0.0", - "datafusion-functions-aggregate-common 53.0.0", - "datafusion-functions-window-common 53.0.0", - "datafusion-physical-expr-common 53.0.0", + "datafusion-common 53.1.0", + "datafusion-doc 53.1.0", + "datafusion-expr-common 53.1.0", + "datafusion-functions-aggregate-common 53.1.0", + "datafusion-functions-window-common 53.1.0", + "datafusion-physical-expr-common 53.1.0", "indexmap", "itertools 0.14.0", "paste", @@ -2741,12 +2779,12 @@ dependencies = [ [[package]] name = "datafusion-expr-common" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5ce2fb1b8c15c9ac45b0863c30b268c69dc9ee7a1ee13ecf5d067738338173dc" +checksum = "1b486b5f6255d40976b88bb83813b0d035a8333e0ec39864824e78068cf42fa6" dependencies = [ "arrow 57.3.0", - "datafusion-common 51.0.0", + "datafusion-common 52.5.0", "indexmap", "itertools 0.14.0", "paste", @@ -2754,12 +2792,12 @@ dependencies = [ [[package]] name = "datafusion-expr-common" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ab05fdd00e05d5a6ee362882546d29d6d3df43a6c55355164a7fbee12d163bc9" +checksum = "7d7c3adf3db8bf61e92eb90cb659c8e8b734593a8f7c8e12a843c7ddba24b87e" dependencies = [ - "arrow 58.0.0", - "datafusion-common 53.0.0", + "arrow 58.1.0", + "datafusion-common 53.1.0", "indexmap", "itertools 0.14.0", "paste", @@ -2767,9 +2805,9 @@ dependencies = [ [[package]] name = "datafusion-functions" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "794a9db7f7b96b3346fc007ff25e994f09b8f0511b4cf7dff651fadfe3ebb28f" +checksum = "07356c94118d881130dd0ffbff127540407d969c8978736e324edcd6c41cd48f" dependencies = [ "arrow 57.3.0", "arrow-buffer 57.3.0", @@ -2777,72 +2815,73 @@ dependencies = [ "blake2", "blake3", "chrono", - "datafusion-common 51.0.0", - "datafusion-doc 51.0.0", - "datafusion-execution 51.0.0", - "datafusion-expr 51.0.0", - "datafusion-expr-common 51.0.0", - "datafusion-macros 51.0.0", + "chrono-tz", + "datafusion-common 52.5.0", + "datafusion-doc 52.5.0", + "datafusion-execution 52.5.0", + "datafusion-expr 52.5.0", + "datafusion-expr-common 52.5.0", + "datafusion-macros 52.5.0", "hex", "itertools 0.14.0", "log", "md-5", "num-traits", - "rand 0.9.2", + "rand 0.9.4", "regex", - "sha2", + "sha2 0.10.9", "unicode-segmentation", "uuid", ] [[package]] name = "datafusion-functions" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "04fb863482d987cf938db2079e07ab0d3bb64595f28907a6c2f8671ad71cca7e" +checksum = "f28aa4e10384e782774b10e72aca4d93ef7b31aa653095d9d4536b0a3dbc51b6" dependencies = [ - "arrow 58.0.0", - "arrow-buffer 58.0.0", + "arrow 58.1.0", + "arrow-buffer 58.1.0", "base64", "blake2", "blake3", "chrono", "chrono-tz", - "datafusion-common 53.0.0", - "datafusion-doc 53.0.0", - "datafusion-execution 53.0.0", - "datafusion-expr 53.0.0", - "datafusion-expr-common 53.0.0", - "datafusion-macros 53.0.0", + "datafusion-common 53.1.0", + "datafusion-doc 53.1.0", + "datafusion-execution 53.1.0", + "datafusion-expr 53.1.0", + "datafusion-expr-common 53.1.0", + "datafusion-macros 53.1.0", "hex", "itertools 0.14.0", "log", "md-5", "memchr", "num-traits", - "rand 0.9.2", + "rand 0.9.4", "regex", - "sha2", + "sha2 0.10.9", "unicode-segmentation", "uuid", ] [[package]] name = "datafusion-functions-aggregate" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1c25210520a9dcf9c2b2cbbce31ebd4131ef5af7fc60ee92b266dc7d159cb305" +checksum = "b644f9cf696df9233ce6958b9807666d78563b56f923267474dd6c07795f1f8f" dependencies = [ "ahash 0.8.12", "arrow 57.3.0", - "datafusion-common 51.0.0", - "datafusion-doc 51.0.0", - "datafusion-execution 51.0.0", - "datafusion-expr 51.0.0", - "datafusion-functions-aggregate-common 51.0.0", - "datafusion-macros 51.0.0", - "datafusion-physical-expr 51.0.0", - "datafusion-physical-expr-common 51.0.0", + "datafusion-common 52.5.0", + "datafusion-doc 52.5.0", + "datafusion-execution 52.5.0", + "datafusion-expr 52.5.0", + "datafusion-functions-aggregate-common 52.5.0", + "datafusion-macros 52.5.0", + "datafusion-physical-expr 52.5.0", + "datafusion-physical-expr-common 52.5.0", "half", "log", "paste", @@ -2850,20 +2889,20 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "829856f4e14275fb376c104f27cbf3c3b57a9cfe24885d98677525f5e43ce8d6" +checksum = "00aa6217e56098ba84e0a338176fe52f0a84cca398021512c6c8c5eff806d0ad" dependencies = [ "ahash 0.8.12", - "arrow 58.0.0", - "datafusion-common 53.0.0", - "datafusion-doc 53.0.0", - "datafusion-execution 53.0.0", - "datafusion-expr 53.0.0", - "datafusion-functions-aggregate-common 53.0.0", - "datafusion-macros 53.0.0", - "datafusion-physical-expr 53.0.0", - "datafusion-physical-expr-common 53.0.0", + "arrow 58.1.0", + "datafusion-common 53.1.0", + "datafusion-doc 53.1.0", + "datafusion-execution 53.1.0", + "datafusion-expr 53.1.0", + "datafusion-functions-aggregate-common 53.1.0", + "datafusion-macros 53.1.0", + "datafusion-physical-expr 53.1.0", + "datafusion-physical-expr-common 53.1.0", "half", "log", "num-traits", @@ -2872,48 +2911,48 @@ dependencies = [ [[package]] name = "datafusion-functions-aggregate-common" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "62f4a66f3b87300bb70f4124b55434d2ae3fe80455f3574701d0348da040b55d" +checksum = "c1de2deaaabe8923ce9ea9f29c47bbb4ee14f67ea2fe1ab5398d9bbebcf86e56" dependencies = [ "ahash 0.8.12", "arrow 57.3.0", - "datafusion-common 51.0.0", - "datafusion-expr-common 51.0.0", - "datafusion-physical-expr-common 51.0.0", + "datafusion-common 52.5.0", + "datafusion-expr-common 52.5.0", + "datafusion-physical-expr-common 52.5.0", ] [[package]] name = "datafusion-functions-aggregate-common" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "08af79cc3d2aa874a362fb97decfcbd73d687190cb096f16a6c85a7780cce311" +checksum = "b511250349407db7c43832ab2de63f5557b19a20dfd236b39ca2c04468b50d47" dependencies = [ "ahash 0.8.12", - "arrow 58.0.0", - "datafusion-common 53.0.0", - "datafusion-expr-common 53.0.0", - "datafusion-physical-expr-common 53.0.0", + "arrow 58.1.0", + "datafusion-common 53.1.0", + "datafusion-expr-common 53.1.0", + "datafusion-physical-expr-common 53.1.0", ] [[package]] name = "datafusion-functions-nested" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae5c06eed03918dc7fe7a9f082a284050f0e9ecf95d72f57712d1496da03b8c4" +checksum = "552f8d92e4331ee91d23c02d12bb6acf32cbfd5215117e01c0fb63cd4b15af1a" dependencies = [ "arrow 57.3.0", "arrow-ord 57.3.0", - "datafusion-common 51.0.0", - "datafusion-doc 51.0.0", - "datafusion-execution 51.0.0", - "datafusion-expr 51.0.0", - "datafusion-expr-common 51.0.0", - "datafusion-functions 51.0.0", - "datafusion-functions-aggregate 51.0.0", - "datafusion-functions-aggregate-common 51.0.0", - "datafusion-macros 51.0.0", - "datafusion-physical-expr-common 51.0.0", + "datafusion-common 52.5.0", + "datafusion-doc 52.5.0", + "datafusion-execution 52.5.0", + "datafusion-expr 52.5.0", + "datafusion-expr-common 52.5.0", + "datafusion-functions 52.5.0", + "datafusion-functions-aggregate 52.5.0", + "datafusion-functions-aggregate-common 52.5.0", + "datafusion-macros 52.5.0", + "datafusion-physical-expr-common 52.5.0", "itertools 0.14.0", "log", "paste", @@ -2921,22 +2960,22 @@ dependencies = [ [[package]] name = "datafusion-functions-nested" -version = "53.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "465ae3368146d49c2eda3e2c0ef114424c87e8a6b509ab34c1026ace6497e790" -dependencies = [ - "arrow 58.0.0", - "arrow-ord 58.0.0", - "datafusion-common 53.0.0", - "datafusion-doc 53.0.0", - "datafusion-execution 53.0.0", - "datafusion-expr 53.0.0", - "datafusion-expr-common 53.0.0", - "datafusion-functions 53.0.0", - "datafusion-functions-aggregate 53.0.0", - "datafusion-functions-aggregate-common 53.0.0", - "datafusion-macros 53.0.0", - "datafusion-physical-expr-common 53.0.0", +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ef13a858e20d50f0a9bb5e96e7ac82b4e7597f247515bccca4fdd2992df0212a" +dependencies = [ + "arrow 58.1.0", + "arrow-ord 58.1.0", + "datafusion-common 53.1.0", + "datafusion-doc 53.1.0", + "datafusion-execution 53.1.0", + "datafusion-expr 53.1.0", + "datafusion-expr-common 53.1.0", + "datafusion-functions 53.1.0", + "datafusion-functions-aggregate 53.1.0", + "datafusion-functions-aggregate-common 53.1.0", + "datafusion-macros 53.1.0", + "datafusion-physical-expr-common 53.1.0", "hashbrown 0.16.1", "itertools 0.14.0", "itoa", @@ -2946,126 +2985,126 @@ dependencies = [ [[package]] name = "datafusion-functions-table" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "db4fed1d71738fbe22e2712d71396db04c25de4111f1ec252b8f4c6d3b25d7f5" +checksum = "970fd0cdd3df8802b9a9975ff600998289ba9d46682a4f7285cba4820c9ada78" dependencies = [ "arrow 57.3.0", "async-trait", - "datafusion-catalog 51.0.0", - "datafusion-common 51.0.0", - "datafusion-expr 51.0.0", - "datafusion-physical-plan 51.0.0", + "datafusion-catalog 52.5.0", + "datafusion-common 52.5.0", + "datafusion-expr 52.5.0", + "datafusion-physical-plan 52.5.0", "parking_lot", "paste", ] [[package]] name = "datafusion-functions-table" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6156e6b22fcf1784112fc0173f3ae6e78c8fdb4d3ed0eace9543873b437e2af6" +checksum = "72b40d3f5bbb3905f9ccb1ce9485a9595c77b69758a7c24d3ba79e334ff51e7e" dependencies = [ - "arrow 58.0.0", + "arrow 58.1.0", "async-trait", - "datafusion-catalog 53.0.0", - "datafusion-common 53.0.0", - "datafusion-expr 53.0.0", - "datafusion-physical-plan 53.0.0", + "datafusion-catalog 53.1.0", + "datafusion-common 53.1.0", + "datafusion-expr 53.1.0", + "datafusion-physical-plan 53.1.0", "parking_lot", "paste", ] [[package]] name = "datafusion-functions-window" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1d92206aa5ae21892f1552b4d61758a862a70956e6fd7a95cb85db1de74bc6d1" +checksum = "40b4c21a7c8a986a1866c0a87ab756d0bbf7b5f41f306009fa2d9af79c52ed31" dependencies = [ "arrow 57.3.0", - "datafusion-common 51.0.0", - "datafusion-doc 51.0.0", - "datafusion-expr 51.0.0", - "datafusion-functions-window-common 51.0.0", - "datafusion-macros 51.0.0", - "datafusion-physical-expr 51.0.0", - "datafusion-physical-expr-common 51.0.0", + "datafusion-common 52.5.0", + "datafusion-doc 52.5.0", + "datafusion-expr 52.5.0", + "datafusion-functions-window-common 52.5.0", + "datafusion-macros 52.5.0", + "datafusion-physical-expr 52.5.0", + "datafusion-physical-expr-common 52.5.0", "log", "paste", ] [[package]] name = "datafusion-functions-window" -version = "53.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ca7baec14f866729012efb89011a6973f3a346dc8090c567bfcd328deff551c1" -dependencies = [ - "arrow 58.0.0", - "datafusion-common 53.0.0", - "datafusion-doc 53.0.0", - "datafusion-expr 53.0.0", - "datafusion-functions-window-common 53.0.0", - "datafusion-macros 53.0.0", - "datafusion-physical-expr 53.0.0", - "datafusion-physical-expr-common 53.0.0", +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d4e88ec9d57c9b685d02f58bfee7be62d72610430ddcedb82a08e5d9925dbfb6" +dependencies = [ + "arrow 58.1.0", + "datafusion-common 53.1.0", + "datafusion-doc 53.1.0", + "datafusion-expr 53.1.0", + "datafusion-functions-window-common 53.1.0", + "datafusion-macros 53.1.0", + "datafusion-physical-expr 53.1.0", + "datafusion-physical-expr-common 53.1.0", "log", "paste", ] [[package]] name = "datafusion-functions-window-common" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "53ae9bcc39800820d53a22d758b3b8726ff84a5a3e24cecef04ef4e5fdf1c7cc" +checksum = "b1210ad73b8b3211aeaf4a42bef9bd7a2b7fce3ec119a478831f18c6ff7f7b93" dependencies = [ - "datafusion-common 51.0.0", - "datafusion-physical-expr-common 51.0.0", + "datafusion-common 52.5.0", + "datafusion-physical-expr-common 52.5.0", ] [[package]] name = "datafusion-functions-window-common" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "159228c3280d342658466bb556dc24de30047fe1d7e559dc5d16ccc5324166f9" +checksum = "8307bb93519b1a91913723a1130cfafeee3f72200d870d88e91a6fc5470ede5c" dependencies = [ - "datafusion-common 53.0.0", - "datafusion-physical-expr-common 53.0.0", + "datafusion-common 53.1.0", + "datafusion-physical-expr-common 53.1.0", ] [[package]] name = "datafusion-macros" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1063ad4c9e094b3f798acee16d9a47bd7372d9699be2de21b05c3bd3f34ab848" +checksum = "aaa566a963013a38681ad82a727a654bc7feb19632426aea8c3412d415d200c5" dependencies = [ - "datafusion-doc 51.0.0", + "datafusion-doc 52.5.0", "quote", "syn 2.0.117", ] [[package]] name = "datafusion-macros" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e5427e5da5edca4d21ea1c7f50e1c9421775fe33d7d5726e5641a833566e7578" +checksum = "2e367e6a71051d0ebdd29b2f85d12059b38b1d1f172c6906e80016da662226bd" dependencies = [ - "datafusion-doc 53.0.0", + "datafusion-doc 53.1.0", "quote", "syn 2.0.117", ] [[package]] name = "datafusion-optimizer" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f35f9ec5d08b87fd1893a30c2929f2559c2f9806ca072d8fefca5009dc0f06a" +checksum = "ff9aa82b240252a88dee118372f9b9757c545ab9e53c0736bebab2e7da0ef1f2" dependencies = [ "arrow 57.3.0", "chrono", - "datafusion-common 51.0.0", - "datafusion-expr 51.0.0", - "datafusion-expr-common 51.0.0", - "datafusion-physical-expr 51.0.0", + "datafusion-common 52.5.0", + "datafusion-expr 52.5.0", + "datafusion-expr-common 52.5.0", + "datafusion-physical-expr 52.5.0", "indexmap", "itertools 0.14.0", "log", @@ -3075,16 +3114,16 @@ dependencies = [ [[package]] name = "datafusion-optimizer" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "89099eefcd5b223ec685c36a41d35c69239236310d71d339f2af0fa4383f3f46" +checksum = "e929015451a67f77d9d8b727b2bf3a40c4445fdef6cdc53281d7d97c76888ace" dependencies = [ - "arrow 58.0.0", + "arrow 58.1.0", "chrono", - "datafusion-common 53.0.0", - "datafusion-expr 53.0.0", - "datafusion-expr-common 53.0.0", - "datafusion-physical-expr 53.0.0", + "datafusion-common 53.1.0", + "datafusion-expr 53.1.0", + "datafusion-expr-common 53.1.0", + "datafusion-physical-expr 53.1.0", "indexmap", "itertools 0.14.0", "log", @@ -3095,39 +3134,40 @@ dependencies = [ [[package]] name = "datafusion-physical-expr" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c30cc8012e9eedcb48bbe112c6eff4ae5ed19cf3003cb0f505662e88b7014c5d" +checksum = "7d48022b8af9988c1d852644f9e8b5584c490659769a550c5e8d39457a1da0a5" dependencies = [ "ahash 0.8.12", "arrow 57.3.0", - "datafusion-common 51.0.0", - "datafusion-expr 51.0.0", - "datafusion-expr-common 51.0.0", - "datafusion-functions-aggregate-common 51.0.0", - "datafusion-physical-expr-common 51.0.0", + "datafusion-common 52.5.0", + "datafusion-expr 52.5.0", + "datafusion-expr-common 52.5.0", + "datafusion-functions-aggregate-common 52.5.0", + "datafusion-physical-expr-common 52.5.0", "half", - "hashbrown 0.14.5", + "hashbrown 0.16.1", "indexmap", "itertools 0.14.0", "parking_lot", "paste", "petgraph", + "tokio", ] [[package]] name = "datafusion-physical-expr" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0f222df5195d605d79098ef37bdd5323bff0131c9d877a24da6ec98dfca9fe36" +checksum = "4b1e68aba7a4b350401cfdf25a3d6f989ad898a7410164afe9ca52080244cb59" dependencies = [ "ahash 0.8.12", - "arrow 58.0.0", - "datafusion-common 53.0.0", - "datafusion-expr 53.0.0", - "datafusion-expr-common 53.0.0", - "datafusion-functions-aggregate-common 53.0.0", - "datafusion-physical-expr-common 53.0.0", + "arrow 58.1.0", + "datafusion-common 53.1.0", + "datafusion-expr 53.1.0", + "datafusion-expr-common 53.1.0", + "datafusion-functions-aggregate-common 53.1.0", + "datafusion-physical-expr-common 53.1.0", "half", "hashbrown 0.16.1", "indexmap", @@ -3141,59 +3181,62 @@ dependencies = [ [[package]] name = "datafusion-physical-expr-adapter" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f9ff2dbd476221b1f67337699eff432781c4e6e1713d2aefdaa517dfbf79768" +checksum = "ae7a8abc0b4fe624000972a9b145b30b7f1b680bffaa950ea53f78d9b21c27c3" dependencies = [ "arrow 57.3.0", - "datafusion-common 51.0.0", - "datafusion-expr 51.0.0", - "datafusion-functions 51.0.0", - "datafusion-physical-expr 51.0.0", - "datafusion-physical-expr-common 51.0.0", + "datafusion-common 52.5.0", + "datafusion-expr 52.5.0", + "datafusion-functions 52.5.0", + "datafusion-physical-expr 52.5.0", + "datafusion-physical-expr-common 52.5.0", "itertools 0.14.0", ] [[package]] name = "datafusion-physical-expr-adapter" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40838625d63d9c12549d81979db3dd675d159055eb9135009ba272ab0e8d0f64" +checksum = "ea22315f33cf2e0adc104e8ec42e285f6ed93998d565c65e82fec6a9ee9f9db4" dependencies = [ - "arrow 58.0.0", - "datafusion-common 53.0.0", - "datafusion-expr 53.0.0", - "datafusion-functions 53.0.0", - "datafusion-physical-expr 53.0.0", - "datafusion-physical-expr-common 53.0.0", + "arrow 58.1.0", + "datafusion-common 53.1.0", + "datafusion-expr 53.1.0", + "datafusion-functions 53.1.0", + "datafusion-physical-expr 53.1.0", + "datafusion-physical-expr-common 53.1.0", "itertools 0.14.0", ] [[package]] name = "datafusion-physical-expr-common" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "90da43e1ec550b172f34c87ec68161986ced70fd05c8d2a2add66eef9c276f03" +checksum = "147253ca3e6b9d59c162de64c02800973018660e13340dd1886dd038d17ac429" dependencies = [ "ahash 0.8.12", "arrow 57.3.0", - "datafusion-common 51.0.0", - "datafusion-expr-common 51.0.0", - "hashbrown 0.14.5", + "chrono", + "datafusion-common 52.5.0", + "datafusion-expr-common 52.5.0", + "hashbrown 0.16.1", + "indexmap", "itertools 0.14.0", + "parking_lot", ] [[package]] name = "datafusion-physical-expr-common" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "eacbcc4cfd502558184ed58fa3c72e775ec65bf077eef5fd2b3453db676f893c" +checksum = "b04b45ea8ad3ac2d78f2ea2a76053e06591c9629c7a603eda16c10649ecf4362" dependencies = [ "ahash 0.8.12", - "arrow 58.0.0", + "arrow 58.1.0", "chrono", - "datafusion-common 53.0.0", - "datafusion-expr-common 53.0.0", + "datafusion-common 53.1.0", + "datafusion-expr-common 53.1.0", "hashbrown 0.16.1", "indexmap", "itertools 0.14.0", @@ -3202,64 +3245,64 @@ dependencies = [ [[package]] name = "datafusion-physical-optimizer" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ce9804f799acd7daef3be7aaffe77c0033768ed8fdbf5fb82fc4c5f2e6bc14e6" +checksum = "689156bb2282107b6239db8d7ef44b4dab10a9b33d3491a0c74acac5e4fedd72" dependencies = [ "arrow 57.3.0", - "datafusion-common 51.0.0", - "datafusion-execution 51.0.0", - "datafusion-expr 51.0.0", - "datafusion-expr-common 51.0.0", - "datafusion-physical-expr 51.0.0", - "datafusion-physical-expr-common 51.0.0", - "datafusion-physical-plan 51.0.0", - "datafusion-pruning 51.0.0", + "datafusion-common 52.5.0", + "datafusion-execution 52.5.0", + "datafusion-expr 52.5.0", + "datafusion-expr-common 52.5.0", + "datafusion-physical-expr 52.5.0", + "datafusion-physical-expr-common 52.5.0", + "datafusion-physical-plan 52.5.0", + "datafusion-pruning 52.5.0", "itertools 0.14.0", ] [[package]] name = "datafusion-physical-optimizer" -version = "53.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d501d0e1d0910f015677121601ac177ec59272ef5c9324d1147b394988f40941" -dependencies = [ - "arrow 58.0.0", - "datafusion-common 53.0.0", - "datafusion-execution 53.0.0", - "datafusion-expr 53.0.0", - "datafusion-expr-common 53.0.0", - "datafusion-physical-expr 53.0.0", - "datafusion-physical-expr-common 53.0.0", - "datafusion-physical-plan 53.0.0", - "datafusion-pruning 53.0.0", +version = "53.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7cb13397809a425918f608dfe8653f332015a3e330004ab191b4404187238b95" +dependencies = [ + "arrow 58.1.0", + "datafusion-common 53.1.0", + "datafusion-execution 53.1.0", + "datafusion-expr 53.1.0", + "datafusion-expr-common 53.1.0", + "datafusion-physical-expr 53.1.0", + "datafusion-physical-expr-common 53.1.0", + "datafusion-physical-plan 53.1.0", + "datafusion-pruning 53.1.0", "itertools 0.14.0", "recursive", ] [[package]] name = "datafusion-physical-plan" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0acf0ad6b6924c6b1aa7d213b181e012e2d3ec0a64ff5b10ee6282ab0f8532ac" +checksum = "68253dc0ee5330aa558b2549c9b0da5af9fc17d753ae73022939014ad616fc28" dependencies = [ "ahash 0.8.12", "arrow 57.3.0", "arrow-ord 57.3.0", "arrow-schema 57.3.0", "async-trait", - "chrono", - "datafusion-common 51.0.0", - "datafusion-common-runtime 51.0.0", - "datafusion-execution 51.0.0", - "datafusion-expr 51.0.0", - "datafusion-functions-aggregate-common 51.0.0", - "datafusion-functions-window-common 51.0.0", - "datafusion-physical-expr 51.0.0", - "datafusion-physical-expr-common 51.0.0", + "datafusion-common 52.5.0", + "datafusion-common-runtime 52.5.0", + "datafusion-execution 52.5.0", + "datafusion-expr 52.5.0", + "datafusion-functions 52.5.0", + "datafusion-functions-aggregate-common 52.5.0", + "datafusion-functions-window-common 52.5.0", + "datafusion-physical-expr 52.5.0", + "datafusion-physical-expr-common 52.5.0", "futures", "half", - "hashbrown 0.14.5", + "hashbrown 0.16.1", "indexmap", "itertools 0.14.0", "log", @@ -3270,24 +3313,24 @@ dependencies = [ [[package]] name = "datafusion-physical-plan" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "463c88ad6f1ecab1810f4c9f046898bee035b370137eb79b2b2db925e270631d" +checksum = "5edc023675791af9d5fb4cc4c24abf5f7bd3bd4dcf9e5bd90ea1eff6976dcc79" dependencies = [ "ahash 0.8.12", - "arrow 58.0.0", - "arrow-ord 58.0.0", - "arrow-schema 58.0.0", + "arrow 58.1.0", + "arrow-ord 58.1.0", + "arrow-schema 58.1.0", "async-trait", - "datafusion-common 53.0.0", - "datafusion-common-runtime 53.0.0", - "datafusion-execution 53.0.0", - "datafusion-expr 53.0.0", - "datafusion-functions 53.0.0", - "datafusion-functions-aggregate-common 53.0.0", - "datafusion-functions-window-common 53.0.0", - "datafusion-physical-expr 53.0.0", - "datafusion-physical-expr-common 53.0.0", + "datafusion-common 53.1.0", + "datafusion-common-runtime 53.1.0", + "datafusion-execution 53.1.0", + "datafusion-expr 53.1.0", + "datafusion-functions 53.1.0", + "datafusion-functions-aggregate-common 53.1.0", + "datafusion-functions-window-common 53.1.0", + "datafusion-physical-expr 53.1.0", + "datafusion-physical-expr-common 53.1.0", "futures", "half", "hashbrown 0.16.1", @@ -3302,104 +3345,104 @@ dependencies = [ [[package]] name = "datafusion-pruning" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ac2c2498a1f134a9e11a9f5ed202a2a7d7e9774bd9249295593053ea3be999db" +checksum = "0fcad240a54d0b1d3e8f668398900260a53122d522b2102ab57218590decacd6" dependencies = [ "arrow 57.3.0", - "datafusion-common 51.0.0", - "datafusion-datasource 51.0.0", - "datafusion-expr-common 51.0.0", - "datafusion-physical-expr 51.0.0", - "datafusion-physical-expr-common 51.0.0", - "datafusion-physical-plan 51.0.0", + "datafusion-common 52.5.0", + "datafusion-datasource 52.5.0", + "datafusion-expr-common 52.5.0", + "datafusion-physical-expr 52.5.0", + "datafusion-physical-expr-common 52.5.0", + "datafusion-physical-plan 52.5.0", "itertools 0.14.0", "log", ] [[package]] name = "datafusion-pruning" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2857618a0ecbd8cd0cf29826889edd3a25774ec26b2995fc3862095c95d88fc6" +checksum = "ac8c76860e355616555081cab5968cec1af7a80701ff374510860bcd567e365a" dependencies = [ - "arrow 58.0.0", - "datafusion-common 53.0.0", - "datafusion-datasource 53.0.0", - "datafusion-expr-common 53.0.0", - "datafusion-physical-expr 53.0.0", - "datafusion-physical-expr-common 53.0.0", - "datafusion-physical-plan 53.0.0", + "arrow 58.1.0", + "datafusion-common 53.1.0", + "datafusion-datasource 53.1.0", + "datafusion-expr-common 53.1.0", + "datafusion-physical-expr 53.1.0", + "datafusion-physical-expr-common 53.1.0", + "datafusion-physical-plan 53.1.0", "itertools 0.14.0", "log", ] [[package]] name = "datafusion-session" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8f96eebd17555386f459037c65ab73aae8df09f464524c709d6a3134ad4f4776" +checksum = "f58e83a68bb67007a8fcbf005c44cefe441270c7ee7f6dee10c0e0109b556f6d" dependencies = [ "async-trait", - "datafusion-common 51.0.0", - "datafusion-execution 51.0.0", - "datafusion-expr 51.0.0", - "datafusion-physical-plan 51.0.0", + "datafusion-common 52.5.0", + "datafusion-execution 52.5.0", + "datafusion-expr 52.5.0", + "datafusion-physical-plan 52.5.0", "parking_lot", ] [[package]] name = "datafusion-session" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ef8637e35022c5c775003b3ab1debc6b4a8f0eb41b069bdd5475dd3aa93f6eba" +checksum = "5412111aa48e2424ba926112e192f7a6b7e4ccb450145d25ce5ede9f19dc491e" dependencies = [ "async-trait", - "datafusion-common 53.0.0", - "datafusion-execution 53.0.0", - "datafusion-expr 53.0.0", - "datafusion-physical-plan 53.0.0", + "datafusion-common 53.1.0", + "datafusion-execution 53.1.0", + "datafusion-expr 53.1.0", + "datafusion-physical-plan 53.1.0", "parking_lot", ] [[package]] name = "datafusion-spark" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "923a8b871962a9d860f036f743a20af50ff04729f1da2468ed220dab4f61c97d" +checksum = "e059dcf8544da0d6598d0235be3cc29c209094a5976b2e4822e4a2cf91c2b5c5" dependencies = [ - "arrow 58.0.0", + "arrow 58.1.0", "bigdecimal", "chrono", "crc32fast", - "datafusion 53.0.0", - "datafusion-catalog 53.0.0", - "datafusion-common 53.0.0", - "datafusion-execution 53.0.0", - "datafusion-expr 53.0.0", - "datafusion-functions 53.0.0", - "datafusion-functions-aggregate 53.0.0", - "datafusion-functions-nested 53.0.0", + "datafusion 53.1.0", + "datafusion-catalog 53.1.0", + "datafusion-common 53.1.0", + "datafusion-execution 53.1.0", + "datafusion-expr 53.1.0", + "datafusion-functions 53.1.0", + "datafusion-functions-aggregate 53.1.0", + "datafusion-functions-nested 53.1.0", "log", "percent-encoding", - "rand 0.9.2", + "rand 0.9.4", "serde_json", - "sha1", - "sha2", + "sha1 0.10.6", + "sha2 0.10.9", "url", ] [[package]] name = "datafusion-sql" -version = "51.0.0" +version = "52.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3fc195fe60634b2c6ccfd131b487de46dc30eccae8a3c35a13f136e7f440414f" +checksum = "be53e9eb55db0fbb8980bb6d87f2435b0524acf4c718ed54a57cabbb299b2ab3" dependencies = [ "arrow 57.3.0", "bigdecimal", "chrono", - "datafusion-common 51.0.0", - "datafusion-expr 51.0.0", + "datafusion-common 52.5.0", + "datafusion-expr 52.5.0", "indexmap", "log", "regex", @@ -3408,16 +3451,16 @@ dependencies = [ [[package]] name = "datafusion-sql" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "12d9e9f16a1692a11c94bcc418191fa15fd2b4d72a0c1a0c607db93c0b84dd81" +checksum = "fa0d133ddf8b9b3b872acac900157f783e7b879fe9a6bccf389abebbfac45ec1" dependencies = [ - "arrow 58.0.0", + "arrow 58.1.0", "bigdecimal", "chrono", - "datafusion-common 53.0.0", - "datafusion-expr 53.0.0", - "datafusion-functions-nested 53.0.0", + "datafusion-common 53.1.0", + "datafusion-expr 53.1.0", + "datafusion-functions-nested 53.1.0", "indexmap", "log", "recursive", @@ -3427,15 +3470,15 @@ dependencies = [ [[package]] name = "datafusion-sqllogictest" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a43746bd59e7f2655be4c5553ede4a1ceb1cd34005932fa9e2bd0641c714c46e" +checksum = "04e5a4a7a49143a68936992b6dbb0db44121c635e9992b2482817278f1e69c56" dependencies = [ - "arrow 58.0.0", + "arrow 58.1.0", "async-trait", "bigdecimal", "clap", - "datafusion 53.0.0", + "datafusion 53.1.0", "datafusion-spark", "datafusion-substrait", "futures", @@ -3443,7 +3486,7 @@ dependencies = [ "indicatif", "itertools 0.14.0", "log", - "object_store 0.13.1", + "object_store 0.13.2", "sqllogictest", "sqlparser 0.61.0", "tempfile", @@ -3453,17 +3496,17 @@ dependencies = [ [[package]] name = "datafusion-substrait" -version = "53.0.0" +version = "53.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d5e5656a7e63d51dd3e5af3dbd347ea83bbe993a77c66b854b74961570d16490" +checksum = "98494539a5468979cc42d86c7bc5f0f8cb71ee5c742694c26fc34efdd29dd2e5" dependencies = [ "async-recursion", "async-trait", "chrono", - "datafusion 53.0.0", + "datafusion 53.1.0", "half", "itertools 0.14.0", - "object_store 0.13.1", + "object_store 0.13.2", "pbjson-types", "prost 0.14.3", "substrait", @@ -3493,9 +3536,9 @@ dependencies = [ [[package]] name = "deflate64" -version = "0.1.11" +version = "0.1.12" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "807800ff3288b621186fe0a8f3392c4652068257302709c24efd918c3dffcdc2" +checksum = "ac6b926516df9c60bfa16e107b21086399f8285a44ca9711344b9e553c5146e2" [[package]] name = "deranged" @@ -3535,11 +3578,24 @@ version = "0.10.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" dependencies = [ - "block-buffer", - "crypto-common", + "block-buffer 0.10.4", + "crypto-common 0.1.7", "subtle", ] +[[package]] +name = "digest" +version = "0.11.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4850db49bf08e663084f7fb5c87d202ef91a3907271aff24a94eb97ff039153c" +dependencies = [ + "block-buffer 0.12.0", + "const-oid", + "crypto-common 0.2.1", + "ctutils", + "zeroize", +] + [[package]] name = "dirs" version = "6.0.0" @@ -3610,7 +3666,7 @@ version = "0.1.0" dependencies = [ "anyhow", "clap", - "similar", + "similar 3.1.0", "tokio", "tracing", "vortex", @@ -3631,16 +3687,6 @@ version = "1.0.20" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555" -[[package]] -name = "earcutr" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "79127ed59a85d7687c409e9978547cffb7dc79675355ed22da6b66fd5f6ead01" -dependencies = [ - "itertools 0.11.0", - "num-traits", -] - [[package]] name = "educe" version = "0.6.0" @@ -3716,9 +3762,9 @@ dependencies = [ [[package]] name = "env_filter" -version = "1.0.0" +version = "1.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7a1c3cc8e57274ec99de65301228b537f1e4eedc1b8e0f9411c6caac8ae7308f" +checksum = "32e90c2accc4b07a8456ea0debdc2e7587bdd890680d71173a15d4ae604f6eef" dependencies = [ "log", "regex", @@ -3726,11 +3772,11 @@ dependencies = [ [[package]] name = "env_logger" -version = "0.11.9" +version = "0.11.10" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b2daee4ea451f429a58296525ddf28b45a3b64f1acf6587e2067437bb11e218d" +checksum = "0621c04f2196ac3f488dd583365b9c09be011a4ab8b9f37248ffcc8f6198b56a" dependencies = [ - "anstream 0.6.21", + "anstream", "anstyle", "env_filter", "jiff", @@ -3761,9 +3807,9 @@ checksum = "5692dd7b5a1978a5aeb0ce83b7655c58ca8efdcb79d21036ea249da95afec2c6" [[package]] name = "ethnum" -version = "1.5.2" +version = "1.5.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ca81e6b4777c89fd810c25a4be2b1bd93ea034fbe58e6a75216a34c6b82c539b" +checksum = "40404c3f5f511ec4da6fe866ddf6a717c309fdbb69fbbad7b0f3edab8f2e835f" [[package]] name = "event-listener" @@ -3842,9 +3888,9 @@ dependencies = [ [[package]] name = "fastrand" -version = "2.3.0" +version = "2.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "37909eebbb50d72f9059c3b6d82c0463f2ff062c9e95845c43a6c9c0355411be" +checksum = "9f1f227452a390804cdb637b74a86990f2a7d7ba4b7d5693aac9b4dd6defd8d6" [[package]] name = "filetime" @@ -3871,7 +3917,7 @@ checksum = "835c052cb0c08c1acf6ffd71c022172e18723949c8282f2b9f27efbc51e64534" dependencies = [ "arbitrary", "byteorder", - "rand 0.8.5", + "rand 0.8.6", "rustc-hex", "static_assertions", ] @@ -3903,12 +3949,6 @@ dependencies = [ "zlib-rs", ] -[[package]] -name = "float_next_after" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8bf7cc16383c4b8d58b9905a8509f02926ce3058053c056376248d958c9df1e8" - [[package]] name = "fnv" version = "1.0.7" @@ -3963,19 +4003,19 @@ checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c" [[package]] name = "fsst" -version = "2.0.1" +version = "4.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5f9e5c0b1c67a38cb92b41535d44623483beb9511592ae23a3bf42ddec758690" +checksum = "2b3a6f3550e61b999febd7168d462db953948eff4fc3448276b3d10d10324dbb" dependencies = [ "arrow-array 57.3.0", - "rand 0.9.2", + "rand 0.9.4", ] [[package]] name = "fsst-rs" -version = "0.5.9" +version = "0.5.10" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cdf65e16e100438be0030d113042e07a62bed67203998640ca6fae0404eed71e" +checksum = "3bf53d7c403a2b76873d4d66ba7d79c54bde2784cdaba6083f223d6e33270708" dependencies = [ "rustc-hash", ] @@ -4136,128 +4176,6 @@ dependencies = [ "version_check", ] -[[package]] -name = "geo" -version = "0.31.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2fc1a1678e54befc9b4bcab6cd43b8e7f834ae8ea121118b0fd8c42747675b4a" -dependencies = [ - "earcutr", - "float_next_after", - "geo-types", - "geographiclib-rs", - "i_overlay", - "log", - "num-traits", - "robust", - "rstar", - "spade", -] - -[[package]] -name = "geo-traits" -version = "0.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2e7c353d12a704ccfab1ba8bfb1a7fe6cb18b665bf89d37f4f7890edcd260206" -dependencies = [ - "geo-types", -] - -[[package]] -name = "geo-types" -version = "0.7.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "24f8647af4005fa11da47cd56252c6ef030be8fa97bdbf355e7dfb6348f0a82c" -dependencies = [ - "approx", - "num-traits", - "rayon", - "rstar", - "serde", -] - -[[package]] -name = "geoarrow-array" -version = "0.7.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc1cc4106ac0a0a512c398961ce95d8150475c84a84e17c4511c3643fa120a17" -dependencies = [ - "arrow-array 57.3.0", - "arrow-buffer 57.3.0", - "arrow-schema 57.3.0", - "geo-traits", - "geoarrow-schema", - "num-traits", - "wkb", - "wkt", -] - -[[package]] -name = "geoarrow-expr-geo" -version = "0.7.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fa84300361ce57fb875bcaa6e32b95b0aff5c6b1af692b936bdd58ff343f4394" -dependencies = [ - "arrow-array 57.3.0", - "arrow-buffer 57.3.0", - "geo", - "geo-traits", - "geoarrow-array", - "geoarrow-schema", -] - -[[package]] -name = "geoarrow-schema" -version = "0.7.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e97be4e9f523f92bd6a0e0458323f4b783d073d011664decd8dbf05651704f34" -dependencies = [ - "arrow-schema 57.3.0", - "geo-traits", - "serde", - "serde_json", - "thiserror 1.0.69", -] - -[[package]] -name = "geodatafusion" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "773cfa1fb0d7f7661b76b3fde00f3ffd8e0ff7b3635096f0ff6294fe5ca62a2b" -dependencies = [ - "arrow-arith 57.3.0", - "arrow-array 57.3.0", - "arrow-schema 57.3.0", - "datafusion 51.0.0", - "geo", - "geo-traits", - "geoarrow-array", - "geoarrow-expr-geo", - "geoarrow-schema", - "geohash", - "thiserror 1.0.69", - "wkt", -] - -[[package]] -name = "geographiclib-rs" -version = "0.2.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c5a7f08910fd98737a6eda7568e7c5e645093e073328eeef49758cfe8b0489c7" -dependencies = [ - "libm", -] - -[[package]] -name = "geohash" -version = "0.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0fb94b1a65401d6cbf22958a9040aa364812c26674f841bee538b12c135db1e6" -dependencies = [ - "geo-types", - "libm", -] - [[package]] name = "get_dir" version = "0.5.0" @@ -4301,7 +4219,7 @@ dependencies = [ "js-sys", "libc", "r-efi 6.0.0", - "rand_core 0.10.0", + "rand_core 0.10.1", "wasip2", "wasip3", "wasm-bindgen", @@ -4343,9 +4261,9 @@ dependencies = [ [[package]] name = "grid" -version = "1.0.0" +version = "1.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f9e2d4c0a8296178d8802098410ca05d86b17a10bb5ab559b3fb404c1f948220" +checksum = "b40ca9252762c466af32d0b1002e91e4e1bc5398f77455e55474deb466355ff5" [[package]] name = "h2" @@ -4375,26 +4293,11 @@ dependencies = [ "cfg-if", "crunchy", "num-traits", - "rand 0.9.2", + "rand 0.9.4", "rand_distr 0.5.1", "zerocopy", ] -[[package]] -name = "handle" -version = "1.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "52ed72152641d513a6084a3907bfcba3f35ae2d3335c22ce2242969c25ff8e46" - -[[package]] -name = "hash32" -version = "0.3.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "47d60b12902ba28e2730cd37e95b8c9223af2808df9e902d4df49588d1470606" -dependencies = [ - "byteorder", -] - [[package]] name = "hashbag" version = "0.1.13" @@ -4443,13 +4346,14 @@ dependencies = [ ] [[package]] -name = "heapless" -version = "0.8.0" +name = "hashbrown" +version = "0.17.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0bfb9eb618601c89945a70e254898da93b13be0388091d42117462b265bb3fad" +checksum = "4f467dd6dccf739c208452f8014c75c18bb8301b050ad1cfb27153803edb0f51" dependencies = [ - "hash32", - "stable_deref_trait", + "allocator-api2", + "equivalent", + "foldhash 0.2.0", ] [[package]] @@ -4472,11 +4376,11 @@ checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" [[package]] name = "hmac" -version = "0.12.1" +version = "0.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6c49c37c09c17a53d937dfbb742eb3a961d65a994e6bcdcf37e7399d0cc8ab5e" +checksum = "6303bc9732ae41b04cb554b844a762b4115a61bfaa81e3e83050991eeb56863f" dependencies = [ - "digest", + "digest 0.11.2", ] [[package]] @@ -4539,11 +4443,20 @@ version = "2.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "135b12329e5e3ce057a9f972339ea52bc954fe1e9358ef27f95e89716fbc5424" +[[package]] +name = "hybrid-array" +version = "0.4.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3944cf8cf766b40e2a1a333ee5e9b563f854d5fa49d6a8ca2764e97c6eddb214" +dependencies = [ + "typenum", +] + [[package]] name = "hyper" -version = "1.8.1" +version = "1.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2ab2d4f250c3d7b1c9fcdff1cece94ea4e2dfbec68614f7b87cb205f24ca9d11" +checksum = "6299f016b246a94207e63da54dbe807655bf9e00044f73ded42c3ac5305fbcca" dependencies = [ "atomic-waker", "bytes", @@ -4555,7 +4468,6 @@ dependencies = [ "httparse", "itoa", "pin-project-lite", - "pin-utils", "smallvec", "tokio", "want", @@ -4563,20 +4475,18 @@ dependencies = [ [[package]] name = "hyper-rustls" -version = "0.27.7" +version = "0.27.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e3c93eb611681b207e1fe55d5a71ecf91572ec8a6705cdb6857f7d8d5242cf58" +checksum = "33ca68d021ef39cf6463ab54c1d0f5daf03377b70561305bb89a8f83aab66e0f" dependencies = [ "http", "hyper", "hyper-util", "rustls", "rustls-native-certs", - "rustls-pki-types", "tokio", "tokio-rustls", "tower-service", - "webpki-roots", ] [[package]] @@ -4613,49 +4523,6 @@ dependencies = [ "serde", ] -[[package]] -name = "i_float" -version = "1.15.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "010025c2c532c8d82e42d0b8bb5184afa449fa6f06c709ea9adcb16c49ae405b" -dependencies = [ - "libm", -] - -[[package]] -name = "i_key_sort" -version = "0.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9190f86706ca38ac8add223b2aed8b1330002b5cdbbce28fb58b10914d38fc27" - -[[package]] -name = "i_overlay" -version = "4.0.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "413183068e6e0289e18d7d0a1f661b81546e6918d5453a44570b9ab30cbed1b3" -dependencies = [ - "i_float", - "i_key_sort", - "i_shape", - "i_tree", - "rayon", -] - -[[package]] -name = "i_shape" -version = "1.14.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1ea154b742f7d43dae2897fcd5ead86bc7b5eefcedd305a7ebf9f69d44d61082" -dependencies = [ - "i_float", -] - -[[package]] -name = "i_tree" -version = "0.16.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "35e6d558e6d4c7b82bc51d9c771e7a927862a161a7d87bf2b0541450e0e20915" - [[package]] name = "iana-time-zone" version = "0.1.65" @@ -4682,12 +4549,13 @@ dependencies = [ [[package]] name = "icu_collections" -version = "2.1.1" +version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4c6b649701667bbe825c3b7e6388cb521c23d88644678e83c0c4d0a621a34b43" +checksum = "2984d1cd16c883d7935b9e07e44071dca8d917fd52ecc02c04d5fa0b5a3f191c" dependencies = [ "displaydoc", "potential_utf", + "utf8_iter", "yoke", "zerofrom", "zerovec", @@ -4695,9 +4563,9 @@ dependencies = [ [[package]] name = "icu_locale_core" -version = "2.1.1" +version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "edba7861004dd3714265b4db54a3c390e880ab658fec5f7db895fae2046b5bb6" +checksum = "92219b62b3e2b4d88ac5119f8904c10f8f61bf7e95b640d25ba3075e6cac2c29" dependencies = [ "displaydoc", "litemap", @@ -4708,9 +4576,9 @@ dependencies = [ [[package]] name = "icu_normalizer" -version = "2.1.1" +version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5f6c8828b67bf8908d82127b2054ea1b4427ff0230ee9141c54251934ab1b599" +checksum = "c56e5ee99d6e3d33bd91c5d85458b6005a22140021cc324cea84dd0e72cff3b4" dependencies = [ "icu_collections", "icu_normalizer_data", @@ -4722,15 +4590,15 @@ dependencies = [ [[package]] name = "icu_normalizer_data" -version = "2.1.1" +version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7aedcccd01fc5fe81e6b489c15b247b8b0690feb23304303a9e560f37efc560a" +checksum = "da3be0ae77ea334f4da67c12f149704f19f81d1adf7c51cf482943e84a2bad38" [[package]] name = "icu_properties" -version = "2.1.2" +version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "020bfc02fe870ec3a66d93e677ccca0562506e5872c650f893269e08615d74ec" +checksum = "bee3b67d0ea5c2cca5003417989af8996f8604e34fb9ddf96208a033901e70de" dependencies = [ "icu_collections", "icu_locale_core", @@ -4742,15 +4610,15 @@ dependencies = [ [[package]] name = "icu_properties_data" -version = "2.1.2" +version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "616c294cf8d725c6afcd8f55abc17c56464ef6211f9ed59cccffe534129c77af" +checksum = "8e2bbb201e0c04f7b4b3e14382af113e17ba4f63e2c9d2ee626b720cbce54a14" [[package]] name = "icu_provider" -version = "2.1.1" +version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "85962cf0ce02e1e0a629cc34e7ca3e373ce20dda4c4d7294bbd0bf1fdb59e614" +checksum = "139c4cf31c8b5f33d7e199446eff9c1e02decfc2f0eec2c8d71f65befa45b421" dependencies = [ "displaydoc", "icu_locale_core", @@ -4816,12 +4684,12 @@ dependencies = [ [[package]] name = "indexmap" -version = "2.13.0" +version = "2.14.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7714e70437a7dc3ac8eb7e6f8df75fd8eb422675fc7678aff7364301092b1017" +checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" dependencies = [ "equivalent", - "hashbrown 0.16.1", + "hashbrown 0.17.0", "serde", "serde_core", ] @@ -4851,11 +4719,11 @@ dependencies = [ [[package]] name = "inout" -version = "0.1.4" +version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "879f10e63c20629ecabbb64a8010319738c66a5cd0c29b02d63d272b03751d01" +checksum = "4250ce6452e92010fdf7268ccc5d14faa80bb12fc741938534c58f16804e03c7" dependencies = [ - "generic-array", + "hybrid-array", ] [[package]] @@ -4866,7 +4734,7 @@ checksum = "7b4a6248eb93a4401ed2f37dfe8ea592d3cf05b7cf4f8efa867b6895af7e094e" dependencies = [ "console 0.16.3", "once_cell", - "similar", + "similar 2.7.0", "tempfile", ] @@ -4891,9 +4759,9 @@ checksum = "8bb03732005da905c88227371639bf1ad885cc712789c011c31c5fb3ab3ccf02" [[package]] name = "inventory" -version = "0.3.22" +version = "0.3.24" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "009ae045c87e7082cb72dab0ccd01ae075dd00141ddc108f43a0ea150a9e7227" +checksum = "a4f0c30c76f2f4ccee3fe55a2435f691ca00c0e4bd87abe4f4a851b1d4dac39b" dependencies = [ "rustversion", ] @@ -4906,9 +4774,9 @@ checksum = "d98f6fed1fde3f8c21bc40a1abb88dd75e67924f9cffc3ef95607bad8017f8e2" [[package]] name = "iri-string" -version = "0.7.11" +version = "0.7.12" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d8e7418f59cc01c88316161279a7f665217ae316b388e58a0d10e29f54f1e5eb" +checksum = "25e659a4bb38e810ebc252e53b5814ff908a8c58c2a9ce2fae1bbec24cbf4e20" dependencies = [ "memchr", "serde", @@ -4940,15 +4808,6 @@ dependencies = [ "either", ] -[[package]] -name = "itertools" -version = "0.11.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b1c173a5686ce8bfa551b3563d0c2170bf24ca44da99c7ca4bfdab5418c3fe57" -dependencies = [ - "either", -] - [[package]] name = "itertools" version = "0.12.1" @@ -4993,9 +4852,9 @@ dependencies = [ [[package]] name = "jiff" -version = "0.2.23" +version = "0.2.24" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1a3546dc96b6d42c5f24902af9e2538e82e39ad350b0c766eb3fbf2d8f3d8359" +checksum = "f00b5dbd620d61dfdcb6007c9c1f6054ebd75319f163d886a9055cec1155073d" dependencies = [ "jiff-static", "jiff-tzdb-platform", @@ -5008,9 +4867,9 @@ dependencies = [ [[package]] name = "jiff-static" -version = "0.2.23" +version = "0.2.24" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2a8c8b344124222efd714b73bb41f8b5120b27a7cc1c75593a6ff768d9d05aa4" +checksum = "e000de030ff8022ea1da3f466fbb0f3a809f5e51ed31f6dd931c35181ad8e6d7" dependencies = [ "proc-macro2", "quote", @@ -5041,15 +4900,45 @@ dependencies = [ "cesu8", "cfg-if", "combine", - "java-locator", "jni-sys 0.3.1", - "libloading 0.7.4", "log", "thiserror 1.0.69", "walkdir", "windows-sys 0.45.0", ] +[[package]] +name = "jni" +version = "0.22.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5efd9a482cf3a427f00d6b35f14332adc7902ce91efb778580e180ff90fa3498" +dependencies = [ + "cfg-if", + "combine", + "java-locator", + "jni-macros", + "jni-sys 0.4.1", + "libloading 0.8.9", + "log", + "simd_cesu8", + "thiserror 2.0.18", + "walkdir", + "windows-link", +] + +[[package]] +name = "jni-macros" +version = "0.22.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a00109accc170f0bdb141fed3e393c565b6f5e072365c3bd58f5b062591560a3" +dependencies = [ + "proc-macro2", + "quote", + "rustc_version", + "simd_cesu8", + "syn 2.0.117", +] + [[package]] name = "jni-sys" version = "0.3.1" @@ -5090,19 +4979,21 @@ dependencies = [ [[package]] name = "js-sys" -version = "0.3.91" +version = "0.3.95" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b49715b7073f385ba4bc528e5747d02e66cb39c6146efb66b781f131f0fb399c" +checksum = "2964e92d1d9dc3364cae4d718d93f227e3abb088e747d92e0395bfdedf1c12ca" dependencies = [ + "cfg-if", + "futures-util", "once_cell", "wasm-bindgen", ] [[package]] name = "jsonb" -version = "0.5.5" +version = "0.5.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2a901f06163d352fbe41c3c2ff5e08b75330a003cc941e988fb501022f5421e6" +checksum = "eb98fb29636087c40ad0d1274d9a30c0c1e83e03ae93f6e7e89247b37fcc6953" dependencies = [ "byteorder", "ethnum", @@ -5111,11 +5002,11 @@ dependencies = [ "jiff", "nom 8.0.0", "num-traits", - "ordered-float 5.1.0", - "rand 0.9.2", - "ryu", + "ordered-float 5.3.0", + "rand 0.9.4", "serde", "serde_json", + "zmij", ] [[package]] @@ -5139,11 +5030,26 @@ dependencies = [ "thiserror 2.0.18", ] +[[package]] +name = "konst" +version = "0.2.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "128133ed7824fcd73d6e7b17957c5eb7bacb885649bd8c69708b2331a10bcefb" +dependencies = [ + "konst_macro_rules", +] + +[[package]] +name = "konst_macro_rules" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4933f3f57a8e9d9da04db23fb153356ecaf00cbd14aee46279c33dc80925c37" + [[package]] name = "lance" -version = "2.0.1" +version = "4.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2b7f07b905df393a5554eba19055c620f9ea25a3e40a013bda4bd8dc4ca66f01" +checksum = "f63e285ceee2b4ca8eb3a8742266cc1ac8161599767a8ecb4d8c2f9fd43d8b29" dependencies = [ "arrow 57.3.0", "arrow-arith 57.3.0", @@ -5160,12 +5066,13 @@ dependencies = [ "byteorder", "bytes", "chrono", + "crossbeam-skiplist", "dashmap", - "datafusion 51.0.0", - "datafusion-expr 51.0.0", - "datafusion-functions 51.0.0", - "datafusion-physical-expr 51.0.0", - "datafusion-physical-plan 51.0.0", + "datafusion 52.5.0", + "datafusion-expr 52.5.0", + "datafusion-functions 52.5.0", + "datafusion-physical-expr 52.5.0", + "datafusion-physical-plan 52.5.0", "deepsize", "either", "futures", @@ -5177,7 +5084,6 @@ dependencies = [ "lance-datafusion", "lance-encoding", "lance-file", - "lance-geo", "lance-index", "lance-io", "lance-linalg", @@ -5190,8 +5096,8 @@ dependencies = [ "pin-project", "prost 0.14.3", "prost-types", - "rand 0.9.2", - "roaring 0.10.12", + "rand 0.9.4", + "roaring", "semver", "serde", "serde_json", @@ -5199,6 +5105,7 @@ dependencies = [ "tantivy", "tokio", "tokio-stream", + "tokio-util", "tracing", "url", "uuid", @@ -5206,9 +5113,9 @@ dependencies = [ [[package]] name = "lance-arrow" -version = "2.0.1" +version = "4.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "100e076cb81c8f0c24cd2881c706fc53e037c7d6e81eb320e929e265d157effb" +checksum = "5c55e62fc04422ef4cd4af6f863ada32641ae23124f9b2e9c567a40d617e8c97" dependencies = [ "arrow-array 57.3.0", "arrow-buffer 57.3.0", @@ -5218,11 +5125,12 @@ dependencies = [ "arrow-schema 57.3.0", "arrow-select 57.3.0", "bytes", + "futures", "getrandom 0.2.17", "half", "jsonb", "num-traits", - "rand 0.9.2", + "rand 0.9.4", ] [[package]] @@ -5245,9 +5153,9 @@ dependencies = [ [[package]] name = "lance-bitpacking" -version = "2.0.1" +version = "4.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "588318d3d1ba0f97162fab39a323a0a49866bb35b32af42572c6b6a12296fa27" +checksum = "a48d232a2908645af0040f96c60a6387fea2df75e762d7033e93e17bb420c6a1" dependencies = [ "arrayref", "paste", @@ -5256,9 +5164,9 @@ dependencies = [ [[package]] name = "lance-core" -version = "2.0.1" +version = "4.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6fa01d1cf490ccfd3b8eaeee2781415d0419e6be8366040e57e43677abf2644e" +checksum = "ce071baaff88fcdcf67f1dd0af54e17656f52ae75aaeb75f25f9cf4da29241f2" dependencies = [ "arrow-array 57.3.0", "arrow-buffer 57.3.0", @@ -5267,8 +5175,8 @@ dependencies = [ "byteorder", "bytes", "chrono", - "datafusion-common 51.0.0", - "datafusion-sql 51.0.0", + "datafusion-common 52.5.0", + "datafusion-sql 52.5.0", "deepsize", "futures", "itertools 0.13.0", @@ -5281,8 +5189,8 @@ dependencies = [ "object_store 0.12.5", "pin-project", "prost 0.14.3", - "rand 0.9.2", - "roaring 0.10.12", + "rand 0.9.4", + "roaring", "serde_json", "snafu", "tempfile", @@ -5295,9 +5203,9 @@ dependencies = [ [[package]] name = "lance-datafusion" -version = "2.0.1" +version = "4.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ef89a39e3284eef76f79e63f23de8881a0583ad6feb20ed39f47eadd847a2b88" +checksum = "11ebc97ee94fa8e1af6fd0520066c7e7e0eab38a100e750ba9aabad644c5aa57" dependencies = [ "arrow 57.3.0", "arrow-array 57.3.0", @@ -5307,19 +5215,19 @@ dependencies = [ "arrow-select 57.3.0", "async-trait", "chrono", - "datafusion 51.0.0", - "datafusion-common 51.0.0", - "datafusion-functions 51.0.0", - "datafusion-physical-expr 51.0.0", + "datafusion 52.5.0", + "datafusion-common 52.5.0", + "datafusion-functions 52.5.0", + "datafusion-physical-expr 52.5.0", "futures", "jsonb", "lance-arrow", "lance-core", "lance-datagen", - "lance-geo", "log", "pin-project", "prost 0.14.3", + "prost-build", "snafu", "tokio", "tracing", @@ -5327,9 +5235,9 @@ dependencies = [ [[package]] name = "lance-datagen" -version = "2.0.1" +version = "4.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fc2a60eef5c47e65d91e2ffa8e7e1629c52e7190c8b88a371a1a60601dc49371" +checksum = "9b90dbb2829875b3a3d00f88fd3a3e39a9e4c7d34c266f67da6550fcda54c76e" dependencies = [ "arrow 57.3.0", "arrow-array 57.3.0", @@ -5339,7 +5247,7 @@ dependencies = [ "futures", "half", "hex", - "rand 0.9.2", + "rand 0.9.4", "rand_distr 0.5.1", "rand_xoshiro 0.7.0", "random_word", @@ -5347,9 +5255,9 @@ dependencies = [ [[package]] name = "lance-encoding" -version = "2.0.1" +version = "4.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "95ce4a6631308aa681b2671af8f2a845ff781f8d4e755a2a7ccd012379467094" +checksum = "65ec429cc2e18ad1b7e43cc7ec57a2f2e49229cfbd934da45e619751a886b8cd" dependencies = [ "arrow-arith 57.3.0", "arrow-array 57.3.0", @@ -5375,7 +5283,7 @@ dependencies = [ "prost 0.14.3", "prost-build", "prost-types", - "rand 0.9.2", + "rand 0.9.4", "snafu", "strum 0.26.3", "tokio", @@ -5386,9 +5294,9 @@ dependencies = [ [[package]] name = "lance-file" -version = "2.0.1" +version = "4.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e2d4d82357cbfaa1a18494226c15b1cb3c8ed0b6c84b91146323c82047ede419" +checksum = "418afe3f82487615fa09222b95a4b5853103f3f0425996d24a537ca750381f83" dependencies = [ "arrow-arith 57.3.0", "arrow-array 57.3.0", @@ -5400,7 +5308,7 @@ dependencies = [ "async-trait", "byteorder", "bytes", - "datafusion-common 51.0.0", + "datafusion-common 52.5.0", "deepsize", "futures", "lance-arrow", @@ -5418,27 +5326,11 @@ dependencies = [ "tracing", ] -[[package]] -name = "lance-geo" -version = "2.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a7183fc870da62826f0f97df8007b634da053eb310157856efe1dc74f446951c" -dependencies = [ - "datafusion 51.0.0", - "geo-traits", - "geo-types", - "geoarrow-array", - "geoarrow-schema", - "geodatafusion", - "lance-core", - "serde", -] - [[package]] name = "lance-index" -version = "2.0.1" +version = "4.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "20e9c5aa7024a63af9ae89ee8c0f23c8421b7896742e5cd4a271a60f9956cb80" +checksum = "936b3deeb6ee075646d18f27b01cf2d2e846c3f5f6c5fa45b30aa41dd5b4c4e2" dependencies = [ "arrow 57.3.0", "arrow-arith 57.3.0", @@ -5452,19 +5344,17 @@ dependencies = [ "bitpacking", "bitvec", "bytes", + "chrono", "crossbeam-queue", - "datafusion 51.0.0", - "datafusion-common 51.0.0", - "datafusion-expr 51.0.0", - "datafusion-physical-expr 51.0.0", - "datafusion-sql 51.0.0", + "datafusion 52.5.0", + "datafusion-common 52.5.0", + "datafusion-expr 52.5.0", + "datafusion-physical-expr 52.5.0", + "datafusion-sql 52.5.0", "deepsize", "dirs", "fst", "futures", - "geo-types", - "geoarrow-array", - "geoarrow-schema", "half", "itertools 0.13.0", "jsonb", @@ -5474,7 +5364,6 @@ dependencies = [ "lance-datagen", "lance-encoding", "lance-file", - "lance-geo", "lance-io", "lance-linalg", "lance-table", @@ -5486,11 +5375,11 @@ dependencies = [ "prost 0.14.3", "prost-build", "prost-types", - "rand 0.9.2", + "rand 0.9.4", "rand_distr 0.5.1", "rangemap", "rayon", - "roaring 0.10.12", + "roaring", "serde", "serde_json", "smallvec", @@ -5505,9 +5394,9 @@ dependencies = [ [[package]] name = "lance-io" -version = "2.0.1" +version = "4.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c7d2af0b17fb374a8181bcf1a10bce5703ae3ee4373c1587ce4bba23e15e45c8" +checksum = "4103e4cebe146af15bfb198c8142d6ea37d5b25fa04158bf2d9be4597bf174d3" dependencies = [ "arrow 57.3.0", "arrow-arith 57.3.0", @@ -5524,6 +5413,7 @@ dependencies = [ "chrono", "deepsize", "futures", + "http", "lance-arrow", "lance-core", "lance-namespace", @@ -5532,10 +5422,10 @@ dependencies = [ "path_abs", "pin-project", "prost 0.14.3", - "rand 0.9.2", + "rand 0.9.4", "serde", - "shellexpand", "snafu", + "tempfile", "tokio", "tracing", "url", @@ -5543,9 +5433,9 @@ dependencies = [ [[package]] name = "lance-linalg" -version = "2.0.1" +version = "4.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5125aa62696e75a7475807564b4921f252d8815be606b84bc00e6def0f5c24bb" +checksum = "c00c7ad71eca93635404519e77add6689947c9342134bb2133578f81249bf809" dependencies = [ "arrow-array 57.3.0", "arrow-buffer 57.3.0", @@ -5556,28 +5446,29 @@ dependencies = [ "lance-arrow", "lance-core", "num-traits", - "rand 0.9.2", + "rand 0.9.4", ] [[package]] name = "lance-namespace" -version = "2.0.1" +version = "4.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "70545c2676ce954dfd801da5c6a631a70bba967826cd3a8f31b47d1f04bbfed3" +checksum = "e0c59a574e72a4b72da8096bcaaa1b1e5b44f6a83da164cc714c286fab30c369" dependencies = [ "arrow 57.3.0", "async-trait", "bytes", "lance-core", "lance-namespace-reqwest-client", + "serde", "snafu", ] [[package]] name = "lance-namespace-reqwest-client" -version = "0.4.5" +version = "0.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a2acdba67f84190067532fce07b51a435dd390d7cdc1129a05003e5cb3274cf0" +checksum = "ee2e48de899e2931afb67fcddd0a08e439bf5d8b6ea2a2ed9cb8f4df669bd5cc" dependencies = [ "reqwest 0.12.28", "serde", @@ -5588,9 +5479,9 @@ dependencies = [ [[package]] name = "lance-table" -version = "2.0.1" +version = "4.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b06ad37bd90045de8ef533df170c6098e6ff6ecb427aade47d7db8e2c86f2678" +checksum = "943b9c503f23ebab9e0dbee356f528bc4cbcafded87a6848451f205b0bb473d7" dependencies = [ "arrow 57.3.0", "arrow-array 57.3.0", @@ -5612,9 +5503,9 @@ dependencies = [ "prost 0.14.3", "prost-build", "prost-types", - "rand 0.9.2", + "rand 0.9.4", "rangemap", - "roaring 0.10.12", + "roaring", "semver", "serde", "serde_json", @@ -5625,6 +5516,16 @@ dependencies = [ "uuid", ] +[[package]] +name = "lasso" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6e14eda50a3494b3bf7b9ce51c52434a761e383d7238ce1dd5dcec2fbc13e9fb" +dependencies = [ + "dashmap", + "hashbrown 0.14.5", +] + [[package]] name = "lazy_static" version = "1.5.0" @@ -5727,15 +5628,15 @@ dependencies = [ [[package]] name = "libbz2-rs-sys" -version = "0.2.2" +version = "0.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2c4a545a15244c7d945065b5d392b2d2d7f21526fba56ce51467b06ed445e8f7" +checksum = "b3a6a8c165077efc8f3a971534c50ea6a1a18b329ef4a66e897a7e3a1494565f" [[package]] name = "libc" -version = "0.2.183" +version = "0.2.185" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b5b646652bf6661599e1da8901b3b9522896f01e736bad5f723fe7a3a27f899d" +checksum = "52ff2c0fe9bc6cb6b14a0592c2ff4fa9ceb83eea9db979b0487cd054946a2b8f" [[package]] name = "libfuzzer-sys" @@ -5749,19 +5650,19 @@ dependencies = [ [[package]] name = "libloading" -version = "0.7.4" +version = "0.8.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b67380fd3b2fbe7527a606e18729d21c6f3951633d0500574c4dc22d2d638b9f" +checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55" dependencies = [ "cfg-if", - "winapi", + "windows-link", ] [[package]] name = "libloading" -version = "0.8.9" +version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55" +checksum = "754ca22de805bb5744484a5b151a9e1a8e837d5dc232c2d7d8c2e3492edc8b60" dependencies = [ "cfg-if", "windows-link", @@ -5778,9 +5679,9 @@ dependencies = [ [[package]] name = "liblzma-sys" -version = "0.4.5" +version = "0.4.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f2db66f3268487b5033077f266da6777d057949b8f93c8ad82e441df25e6186" +checksum = "1a60851d15cd8c5346eca4ab8babff585be2ae4bc8097c067291d3ffe2add3b6" dependencies = [ "cc", "libc", @@ -5795,24 +5696,23 @@ checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981" [[package]] name = "libmimalloc-sys" -version = "0.1.44" +version = "0.1.47" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "667f4fec20f29dfc6bc7357c582d91796c169ad7e2fce709468aefeb2c099870" +checksum = "2d1eacfa31c33ec25e873c136ba5669f00f9866d0688bea7be4d3f7e43067df6" dependencies = [ "cc", - "libc", ] [[package]] name = "libredox" -version = "0.1.14" +version = "0.1.16" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1744e39d1d6a9948f4f388969627434e31128196de472883b39f148769bfe30a" +checksum = "e02f3bb43d335493c96bf3fd3a321600bf6bd07ed34bc64118e9293bdffea46c" dependencies = [ "bitflags", "libc", "plain", - "redox_syscall 0.7.3", + "redox_syscall 0.7.4", ] [[package]] @@ -5821,7 +5721,7 @@ version = "0.8.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "14e6ba06f0ade6e504aff834d7c34298e5155c6baca353cc6a4aaff2f9fd7f33" dependencies = [ - "anstream 1.0.0", + "anstream", "anstyle", "clap", "escape8259", @@ -5829,9 +5729,9 @@ dependencies = [ [[package]] name = "line-clipping" -version = "0.3.5" +version = "0.3.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5f4de44e98ddbf09375cbf4d17714d18f39195f4f4894e8524501726fd9a8a4a" +checksum = "3f50e8f47623268b5407192d26876c4d7f89d686ca130fdc53bced4814cd29f8" dependencies = [ "bitflags", ] @@ -5859,9 +5759,9 @@ checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" [[package]] name = "litemap" -version = "0.8.1" +version = "0.8.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6373607a59f0be73a39b6fe456b8192fcc3585f602af20751600e974dd455e77" +checksum = "92daf443525c4cce67b150400bc2316076100ce0b3686209eb8cf3c31612e6f0" [[package]] name = "litrs" @@ -5908,9 +5808,9 @@ dependencies = [ [[package]] name = "lru" -version = "0.16.3" +version = "0.16.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a1dc47f592c06f33f8e3aea9591776ec7c9f9e4124778ff8a3c3b87159f7e593" +checksum = "7f66e8d5d03f609abc3a39e6f08e4164ebf1447a732906d39eb9b99b7919ef39" dependencies = [ "hashbrown 0.16.1", ] @@ -5955,13 +5855,22 @@ dependencies = [ "twox-hash", ] +[[package]] +name = "lz4_flex" +version = "0.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "db9a0d582c2874f68138a16ce1867e0ffde6c0bb0a0df85e1f36d04146db488a" +dependencies = [ + "twox-hash", +] + [[package]] name = "lzma-rust2" version = "0.16.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "47bb1e988e6fb779cf720ad431242d3f03167c1b3f2b1aae7f1a94b2495b36ae" dependencies = [ - "sha2", + "sha2 0.10.9", ] [[package]] @@ -6009,7 +5918,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d89e7ee0cfbedfc4da3340218492196241d89eefb6dab27de5df917a6d2e78cf" dependencies = [ "cfg-if", - "digest", + "digest 0.10.7", ] [[package]] @@ -6038,9 +5947,9 @@ dependencies = [ [[package]] name = "mimalloc" -version = "0.1.48" +version = "0.1.50" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e1ee66a4b64c74f4ef288bcbb9192ad9c3feaad75193129ac8509af543894fd8" +checksum = "b3627c4272df786b9260cabaa46aec1d59c93ede723d4c3ef646c503816b0640" dependencies = [ "libmimalloc-sys", ] @@ -6079,9 +5988,9 @@ dependencies = [ [[package]] name = "mio" -version = "1.1.1" +version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a69bcab0ad47271a0234d9422b131806bf3968021e5dc9328caf2d4cd58557fc" +checksum = "50b7e5b27aa02a74bac8c3f23f448f8d87ff11f92d3aac1a6ed369ee08cc56c1" dependencies = [ "libc", "log", @@ -6257,9 +6166,9 @@ dependencies = [ [[package]] name = "noodles-vcf" -version = "0.86.0" +version = "0.87.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0131147080a5aeadcd6c2ab43e7ca1f9e8018cdd482eed797d081c40e8f0a3c7" +checksum = "12b339e85269ab7689661c690f9108dceb606bf3d230e1768799ae107d505e76" dependencies = [ "futures", "indexmap", @@ -6334,9 +6243,9 @@ dependencies = [ [[package]] name = "num-conv" -version = "0.2.0" +version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf97ec579c3c42f953ef76dbf8d55ac91fb219dde70e49aa4a6b7d74e9919050" +checksum = "c6673768db2d862beb9b39a78fdcb1a69439615d5794a1be50caa9bc92c81967" [[package]] name = "num-integer" @@ -6383,7 +6292,6 @@ version = "0.7.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "680998035259dcfcafe653688bf2aa6d3e2dc05e98be6ab46afb089dc84f1df8" dependencies = [ - "proc-macro-crate", "proc-macro2", "quote", "syn 2.0.117", @@ -6452,16 +6360,18 @@ dependencies = [ [[package]] name = "object_store" -version = "0.13.1" +version = "0.13.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2858065e55c148d294a9f3aae3b0fa9458edadb41a108397094566f4e3c0dfb" +checksum = "622acbc9100d3c10e2ee15804b0caa40e55c933d5aa53814cd520805b7958a49" dependencies = [ "async-trait", "base64", "bytes", "chrono", "form_urlencoded", - "futures", + "futures-channel", + "futures-core", + "futures-util", "http", "http-body-util", "httparse", @@ -6472,7 +6382,7 @@ dependencies = [ "parking_lot", "percent-encoding", "quick-xml", - "rand 0.9.2", + "rand 0.10.1", "reqwest 0.12.28", "ring", "rustls-pki-types", @@ -6595,7 +6505,7 @@ dependencies = [ "futures-util", "opentelemetry", "percent-encoding", - "rand 0.9.2", + "rand 0.9.4", "thiserror 2.0.18", ] @@ -6616,9 +6526,9 @@ dependencies = [ [[package]] name = "ordered-float" -version = "5.1.0" +version = "5.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f4779c6901a562440c3786d08192c6fbda7c1c2060edd10006b05ee35d10f2d" +checksum = "b7d950ca161dc355eaf28f82b11345ed76c6e1f6eb1f4f4479e0323b9e2fbd0e" dependencies = [ "num-traits", ] @@ -6742,17 +6652,17 @@ dependencies = [ [[package]] name = "parquet" -version = "58.0.0" +version = "58.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3f491d0ef1b510194426ee67ddc18a9b747ef3c42050c19322a2cd2e1666c29b" +checksum = "7d3f9f2205199603564127932b89695f52b62322f541d0fc7179d57c2e1c9877" dependencies = [ "ahash 0.8.12", - "arrow-array 58.0.0", - "arrow-buffer 58.0.0", - "arrow-data 58.0.0", - "arrow-ipc 58.0.0", - "arrow-schema 58.0.0", - "arrow-select 58.0.0", + "arrow-array 58.1.0", + "arrow-buffer 58.1.0", + "arrow-data 58.1.0", + "arrow-ipc 58.1.0", + "arrow-schema 58.1.0", + "arrow-select 58.1.0", "base64", "brotli", "bytes", @@ -6761,11 +6671,11 @@ dependencies = [ "futures", "half", "hashbrown 0.16.1", - "lz4_flex 0.12.1", + "lz4_flex 0.13.0", "num-bigint", "num-integer", "num-traits", - "object_store 0.13.1", + "object_store 0.13.2", "paste", "seq-macro", "simdutf8", @@ -6778,11 +6688,11 @@ dependencies = [ [[package]] name = "parquet-variant" -version = "58.0.0" +version = "58.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "00ba4e5dcbc8ad65882b7337a95c12a0f9cbb6add237c53d93b803b7d7f70f02" +checksum = "2bf493f3c9ddd984d0efb019f67343e4aa4bab893931f6a14b82083065dc3d28" dependencies = [ - "arrow-schema 58.0.0", + "arrow-schema 58.1.0", "chrono", "half", "indexmap", @@ -6792,12 +6702,12 @@ dependencies = [ [[package]] name = "parquet-variant-compute" -version = "58.0.0" +version = "58.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9ec4cfb8da15565c8d211b6bc51e8eb481ea65d19132462af3f948b150ac8efe" +checksum = "6ac038d46a503a7d563b4f5df5802c4315d5343d009feab195d15ac512b4cb27" dependencies = [ - "arrow 58.0.0", - "arrow-schema 58.0.0", + "arrow 58.1.0", + "arrow-schema 58.1.0", "chrono", "half", "indexmap", @@ -6809,11 +6719,11 @@ dependencies = [ [[package]] name = "parquet-variant-json" -version = "58.0.0" +version = "58.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3668ff00a6aeb29d172ba15f9d8fedf1675d79bff7d1916daa333efdeaa13e46" +checksum = "015a09c2ffe5108766c7c1235c307b8a3c2ea64eca38455ba1a7f3a7f32f16e2" dependencies = [ - "arrow-schema 58.0.0", + "arrow-schema 58.1.0", "base64", "chrono", "parquet-variant", @@ -6878,11 +6788,11 @@ dependencies = [ [[package]] name = "pbkdf2" -version = "0.12.2" +version = "0.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8ed6a7761f76e3b9f92dfb0a60a6a6477c61024b775147ff0973a02653abaf2" +checksum = "112d82ceb8c5bf524d9af484d4e4970c9fd5a0cc15ba14ad93dccd28873b0629" dependencies = [ - "digest", + "digest 0.11.2", "hmac", ] @@ -6966,21 +6876,15 @@ version = "0.2.17" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" -[[package]] -name = "pin-utils" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b870d8c151b6f2fb93e84a13146138f05d02ed11c7e7c54f8826aaaf7c9f184" - [[package]] name = "ping" version = "0.7.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "044b1fa4f259f4df9ad5078e587b208f5d288a25407575fcddb9face30c7c692" dependencies = [ - "rand 0.9.2", + "rand 0.9.4", "socket2", - "thiserror 2.0.18", + "thiserror 1.0.69", ] [[package]] @@ -6996,9 +6900,9 @@ dependencies = [ [[package]] name = "pkg-config" -version = "0.3.32" +version = "0.3.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7edddbd0b52d732b21ad9a5fab5c704c14cd949e5e9a1ec5929a24fded1b904c" +checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" [[package]] name = "plain" @@ -7062,18 +6966,18 @@ checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49" [[package]] name = "portable-atomic-util" -version = "0.2.6" +version = "0.2.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "091397be61a01d4be58e7841595bd4bfedb15f1cd54977d79b8271e94ed799a3" +checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" dependencies = [ "portable-atomic", ] [[package]] name = "potential_utf" -version = "0.1.4" +version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b73949432f5e2a09657003c25bca5e19a0e9c84f8058ca374f49e0ebe605af77" +checksum = "0103b1cef7ec0cf76490e969665504990193874ea05c85ff9bab8b911d0a0564" dependencies = [ "zerovec", ] @@ -7236,9 +7140,9 @@ dependencies = [ [[package]] name = "psm" -version = "0.1.30" +version = "0.1.31" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3852766467df634d74f0b2d7819bf8dc483a0eb2e3b0f50f756f9cfe8b0d18d8" +checksum = "645dbe486e346d9b5de3ef16ede18c26e6c70ad97418f4874b8b1889d6e761ea" dependencies = [ "ar_archive_writer", "cc", @@ -7267,8 +7171,7 @@ dependencies = [ [[package]] name = "public-api" version = "0.51.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b4597eaaa646a5df48e5f37924b58bd4b2e2a3746d4665195d03538846784858" +source = "git+https://github.com/vortex-data/cargo-public-api.git?branch=main#ca28f3faff19f9c4cd838fb121fd7212953fdb19" dependencies = [ "hashbag", "rustdoc-types", @@ -7280,9 +7183,9 @@ dependencies = [ [[package]] name = "pyo3" -version = "0.28.2" +version = "0.28.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf85e27e86080aafd5a22eae58a162e133a589551542b3e5cee4beb27e54f8e1" +checksum = "91fd8e38a3b50ed1167fb981cd6fd60147e091784c427b8f7183a7ee32c31c12" dependencies = [ "chrono", "indexmap", @@ -7310,9 +7213,9 @@ dependencies = [ [[package]] name = "pyo3-build-config" -version = "0.28.2" +version = "0.28.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8bf94ee265674bf76c09fa430b0e99c26e319c945d96ca0d5a8215f31bf81cf7" +checksum = "e368e7ddfdeb98c9bca7f8383be1648fd84ab466bf2bc015e94008db6d35611e" dependencies = [ "target-lexicon", ] @@ -7329,9 +7232,9 @@ dependencies = [ [[package]] name = "pyo3-ffi" -version = "0.28.2" +version = "0.28.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "491aa5fc66d8059dd44a75f4580a2962c1862a1c2945359db36f6c2818b748dc" +checksum = "7f29e10af80b1f7ccaf7f69eace800a03ecd13e883acfacc1e5d0988605f651e" dependencies = [ "libc", "pyo3-build-config", @@ -7350,9 +7253,9 @@ dependencies = [ [[package]] name = "pyo3-macros" -version = "0.28.2" +version = "0.28.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f5d671734e9d7a43449f8480f8b38115df67bef8d21f76837fa75ee7aaa5e52e" +checksum = "df6e520eff47c45997d2fc7dd8214b25dd1310918bbb2642156ef66a67f29813" dependencies = [ "proc-macro2", "pyo3-macros-backend", @@ -7362,9 +7265,9 @@ dependencies = [ [[package]] name = "pyo3-macros-backend" -version = "0.28.2" +version = "0.28.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "22faaa1ce6c430a1f71658760497291065e6450d7b5dc2bcf254d49f66ee700a" +checksum = "c4cdc218d835738f81c2338f822078af45b4afdf8b2e33cbb5916f108b813acb" dependencies = [ "heck", "proc-macro2", @@ -7386,7 +7289,7 @@ dependencies = [ "http", "humantime", "itertools 0.14.0", - "object_store 0.13.1", + "object_store 0.13.2", "percent-encoding", "pyo3", "pyo3-async-runtimes", @@ -7404,9 +7307,9 @@ checksum = "5a651516ddc9168ebd67b24afd085a718be02f8858fe406591b013d101ce2f40" [[package]] name = "quick-xml" -version = "0.38.4" +version = "0.39.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b66c2058c55a409d601666cffe35f04333cf1013010882cec174a7467cd4e21c" +checksum = "958f21e8e7ceb5a1aa7fa87fab28e7c75976e0bfe7e23ff069e0a260f894067d" dependencies = [ "memchr", "serde", @@ -7442,7 +7345,7 @@ dependencies = [ "bytes", "getrandom 0.3.4", "lru-slab", - "rand 0.9.2", + "rand 0.9.4", "ring", "rustc-hash", "rustls", @@ -7497,9 +7400,9 @@ checksum = "dc33ff2d4973d518d823d61aa239014831e521c75da58e3df4840d3f47749d09" [[package]] name = "rand" -version = "0.8.5" +version = "0.8.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "34af8d1a0e25924bc5b7c43c079c942339d8f0a8b57c39049bef581b46327404" +checksum = "5ca0ecfa931c29007047d1bc58e623ab12e5590e8c7cc53200d5202b69266d8a" dependencies = [ "libc", "rand_chacha 0.3.1", @@ -7508,9 +7411,9 @@ dependencies = [ [[package]] name = "rand" -version = "0.9.2" +version = "0.9.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6db2770f06117d490610c7488547d543617b21bfa07796d7a12f6f1bd53850d1" +checksum = "44c5af06bb1b7d3216d91932aed5265164bf384dc89cd6ba05cf59a35f5f76ea" dependencies = [ "rand_chacha 0.9.0", "rand_core 0.9.5", @@ -7518,13 +7421,13 @@ dependencies = [ [[package]] name = "rand" -version = "0.10.0" +version = "0.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bc266eb313df6c5c09c1c7b1fbe2510961e5bcd3add930c1e31f7ed9da0feff8" +checksum = "d2e8e8bcc7961af1fdac401278c6a831614941f6164ee3bf4ce61b7edb162207" dependencies = [ "chacha20", "getrandom 0.4.2", - "rand_core 0.10.0", + "rand_core 0.10.1", ] [[package]] @@ -7567,9 +7470,9 @@ dependencies = [ [[package]] name = "rand_core" -version = "0.10.0" +version = "0.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c8d0fd677905edcbeedbf2edb6494d676f0e98d54d5cf9bda0b061cb8fb8aba" +checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" [[package]] name = "rand_distr" @@ -7578,7 +7481,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "32cb0b9bc82b0a0876c2dd994a7e7a2683d3e7390ca40e6886785ef0c7e3ee31" dependencies = [ "num-traits", - "rand 0.8.5", + "rand 0.8.6", ] [[package]] @@ -7588,7 +7491,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6a8615d50dcf34fa31f7ab52692afec947c4dd0ab803cc87cb3b0b4570ff7463" dependencies = [ "num-traits", - "rand 0.9.2", + "rand 0.9.4", ] [[package]] @@ -7598,7 +7501,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4d431c2703ccf129de4d45253c03f49ebb22b97d6ad79ee3ecfc7e3f4862c1d8" dependencies = [ "num-traits", - "rand 0.10.0", + "rand 0.10.1", ] [[package]] @@ -7627,7 +7530,7 @@ dependencies = [ "clap", "indicatif", "lance-bench", - "rand 0.10.0", + "rand 0.10.1", "rand_distr 0.6.0", "tokio", "vortex-bench", @@ -7642,7 +7545,7 @@ dependencies = [ "ahash 0.8.12", "brotli", "paste", - "rand 0.9.2", + "rand 0.9.4", "unicase", ] @@ -7676,7 +7579,7 @@ dependencies = [ "indoc", "itertools 0.14.0", "kasuari", - "lru 0.16.3", + "lru 0.16.4", "strum 0.27.2", "thiserror 2.0.18", "unicode-segmentation", @@ -7739,9 +7642,9 @@ checksum = "60a357793950651c4ed0f3f52338f53b2f809f32d83a07f72909fa13e4c6c1e3" [[package]] name = "rayon" -version = "1.11.0" +version = "1.12.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "368f01d005bf8fd9b1206fb6fa653e6c4a81ceb1466406b81792d87c5677a58f" +checksum = "fb39b166781f92d482534ef4b4b1b2568f42613b53e5b6c160e24cfbfa30926d" dependencies = [ "either", "rayon-core", @@ -7788,9 +7691,9 @@ dependencies = [ [[package]] name = "redox_syscall" -version = "0.7.3" +version = "0.7.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6ce70a74e890531977d37e532c34d45e9055d2409ed08ddba14529471ed0be16" +checksum = "f450ad9c3b1da563fb6948a8e0fb0fb9269711c9c73d9ea1de5058c79c8d643a" dependencies = [ "bitflags", ] @@ -7928,9 +7831,8 @@ dependencies = [ "url", "wasm-bindgen", "wasm-bindgen-futures", - "wasm-streams", + "wasm-streams 0.4.2", "web-sys", - "webpki-roots", ] [[package]] @@ -7964,12 +7866,14 @@ dependencies = [ "sync_wrapper", "tokio", "tokio-rustls", + "tokio-util", "tower", "tower-http", "tower-service", "url", "wasm-bindgen", "wasm-bindgen-futures", + "wasm-streams 0.5.0", "web-sys", ] @@ -8018,41 +7922,14 @@ dependencies = [ [[package]] name = "roaring" -version = "0.10.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "19e8d2cfa184d94d0726d650a9f4a1be7f9b76ac9fdb954219878dc00c1c1e7b" -dependencies = [ - "bytemuck", - "byteorder", -] - -[[package]] -name = "roaring" -version = "0.11.3" +version = "0.11.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8ba9ce64a8f45d7fc86358410bb1a82e8c987504c0d4900e9141d69a9f26c885" +checksum = "1dedc5658c6ecb3bdb5ef5f3295bb9253f42dcf3fd1402c03f6b1f7659c3c4a9" dependencies = [ "bytemuck", "byteorder", ] -[[package]] -name = "robust" -version = "1.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4e27ee8bb91ca0adcf0ecb116293afa12d393f9c2b9b9cd54d33e8078fe19839" - -[[package]] -name = "rstar" -version = "0.12.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "421400d13ccfd26dfa5858199c30a5d76f9c54e0dba7575273025b43c5175dbb" -dependencies = [ - "heapless", - "num-traits", - "smallvec", -] - [[package]] name = "rstest" version = "0.26.1" @@ -8089,7 +7966,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b3a8fb4672e840a587a66fc577a5491375df51ddb88f2a2c2a792598c326fe14" dependencies = [ "quote", - "rand 0.8.5", + "rand 0.8.6", "syn 2.0.117", ] @@ -8113,7 +7990,7 @@ dependencies = [ "borsh", "bytes", "num-traits", - "rand 0.8.5", + "rand 0.8.6", "rkyv", "serde", "serde_json", @@ -8122,9 +7999,9 @@ dependencies = [ [[package]] name = "rustc-hash" -version = "2.1.1" +version = "2.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "357703d41365b4b27c590e3ed91eabb1b663f07c4c084095e60cbed4362dff0d" +checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe" [[package]] name = "rustc-hex" @@ -8143,9 +8020,9 @@ dependencies = [ [[package]] name = "rustdoc-types" -version = "0.56.0" +version = "0.57.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "27bf787c529efe523ed9eb6dcdbaa5954d34329f08d5c243fce928441826ca90" +checksum = "7e6919c49bdd9cca072b3b502d16c073d0a4d3b7283ff6e0c86a5f6e6b07bbfe" dependencies = [ "serde", "serde_derive", @@ -8161,7 +8038,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys 0.4.15", - "windows-sys 0.59.0", + "windows-sys 0.52.0", ] [[package]] @@ -8179,9 +8056,9 @@ dependencies = [ [[package]] name = "rustls" -version = "0.23.37" +version = "0.23.38" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "758025cb5fccfd3bc2fd74708fd4682be41d99e5dff73c377c0646c6012c73a4" +checksum = "69f9466fb2c14ea04357e91413efb882e2a6d4a406e625449bc0a5d360d53a21" dependencies = [ "aws-lc-rs", "once_cell", @@ -8222,7 +8099,7 @@ checksum = "1d99feebc72bae7ab76ba994bb5e121b8d83d910ca40b36e0921f53becc41784" dependencies = [ "core-foundation 0.10.1", "core-foundation-sys", - "jni", + "jni 0.21.1", "log", "once_cell", "rustls", @@ -8243,9 +8120,9 @@ checksum = "f87165f0995f63a9fbeea62b64d10b4d9d8e78ec6d7d51fb2125fda7bb36788f" [[package]] name = "rustls-webpki" -version = "0.103.10" +version = "0.103.13" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "df33b2b81ac578cabaf06b89b0631153a3f416b0a886e8a7a1707fb51abbd1ef" +checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e" dependencies = [ "aws-lc-rs", "ring", @@ -8383,9 +8260,9 @@ dependencies = [ [[package]] name = "semver" -version = "1.0.27" +version = "1.0.28" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d767eb0aabc880b29956c35734170f26ed551a859dbd361d140cdbeca61ab1e2" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" dependencies = [ "serde", "serde_core", @@ -8474,9 +8351,9 @@ dependencies = [ [[package]] name = "serde_spanned" -version = "1.0.4" +version = "1.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8bbf91e5a4d6315eee45e704372590b30e260ee83af6639d64557f51b067776" +checksum = "6662b5879511e06e8999a8a235d848113e942c9124f211511b16466ee2995f26" dependencies = [ "serde_core", ] @@ -8561,7 +8438,18 @@ checksum = "e3bf829a2d51ab4a5ddf1352d8470c140cadc8301b2ae1789db023f01cedd6ba" dependencies = [ "cfg-if", "cpufeatures 0.2.17", - "digest", + "digest 0.10.7", +] + +[[package]] +name = "sha1" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "aacc4cc499359472b4abe1bf11d0b12e688af9a805fa5e3016f9a386dc2d0214" +dependencies = [ + "cfg-if", + "cpufeatures 0.3.0", + "digest 0.11.2", ] [[package]] @@ -8572,25 +8460,27 @@ checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283" dependencies = [ "cfg-if", "cpufeatures 0.2.17", - "digest", + "digest 0.10.7", ] [[package]] -name = "sharded-slab" -version = "0.1.7" +name = "sha2" +version = "0.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f40ca3c46823713e0d4209592e8d6e826aa57e928f09752619fc696c499637f6" +checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4" dependencies = [ - "lazy_static", + "cfg-if", + "cpufeatures 0.3.0", + "digest 0.11.2", ] [[package]] -name = "shellexpand" -version = "3.1.2" +name = "sharded-slab" +version = "0.1.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32824fab5e16e6c4d86dc1ba84489390419a39f97699852b66480bb87d297ed8" +checksum = "f40ca3c46823713e0d4209592e8d6e826aa57e928f09752619fc696c499637f6" dependencies = [ - "dirs", + "lazy_static", ] [[package]] @@ -8632,9 +8522,19 @@ dependencies = [ [[package]] name = "simd-adler32" -version = "0.3.8" +version = "0.3.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e320a6c5ad31d271ad523dcf3ad13e2767ad8b1cb8f047f75a8aeaf8da139da2" +checksum = "703d5c7ef118737c72f1af64ad2f6f8c5e1921f818cdcb97b8fe6fc69bf66214" + +[[package]] +name = "simd_cesu8" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94f90157bb87cddf702797c5dadfa0be7d266cdf49e22da2fcaa32eff75b2c33" +dependencies = [ + "rustc_version", + "simdutf8", +] [[package]] name = "simdutf8" @@ -8652,6 +8552,15 @@ dependencies = [ "unicode-segmentation", ] +[[package]] +name = "similar" +version = "3.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "04d93e861ede2e497b47833469b8ec9d5c07fa4c78ce7a00f6eb7dd8168b4b3f" +dependencies = [ + "bstr", +] + [[package]] name = "similar-asserts" version = "1.7.0" @@ -8659,7 +8568,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b5b441962c817e33508847a22bd82f03a30cff43642dc2fae8b050566121eb9a" dependencies = [ "console 0.15.11", - "similar", + "similar 2.7.0", ] [[package]] @@ -8723,18 +8632,18 @@ dependencies = [ [[package]] name = "snafu" -version = "0.8.9" +version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e84b3f4eacbf3a1ce05eac6763b4d629d60cbc94d632e4092c54ade71f1e1a2" +checksum = "d1d4bced6a69f90b2056c03dcff2c4737f98d6fb9e0853493996e1d253ca29c6" dependencies = [ "snafu-derive", ] [[package]] name = "snafu-derive" -version = "0.8.9" +version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c1c97747dbf44bb1ca44a561ece23508e99cb592e862f22222dcf42f51d1e451" +checksum = "54254b8531cafa275c5e096f62d48c81435d1015405a91198ddb11e967301d40" dependencies = [ "heck", "proc-macro2", @@ -8768,18 +8677,6 @@ dependencies = [ "windows-sys 0.61.2", ] -[[package]] -name = "spade" -version = "2.15.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fb313e1c8afee5b5647e00ee0fe6855e3d529eb863a0fdae1d60006c4d1e9990" -dependencies = [ - "hashbrown 0.15.5", - "num-traits", - "robust", - "smallvec", -] - [[package]] name = "sqllogictest" version = "0.29.1" @@ -8796,9 +8693,9 @@ dependencies = [ "libtest-mimic", "md-5", "owo-colors", - "rand 0.8.5", + "rand 0.8.6", "regex", - "similar", + "similar 2.7.0", "subst", "tempfile", "thiserror 2.0.18", @@ -8856,15 +8753,15 @@ checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" [[package]] name = "stacker" -version = "0.1.23" +version = "0.1.24" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "08d74a23609d509411d10e2176dc2a4346e3b4aea2e7b1869f19fdedbc71c013" +checksum = "640c8cdd92b6b12f5bcb1803ca3bbf5ab96e5e6b6b96b9ab77dabe9e880b3190" dependencies = [ "cc", "cfg-if", "libc", "psm", - "windows-sys 0.59.0", + "windows-sys 0.61.2", ] [[package]] @@ -9095,9 +8992,9 @@ dependencies = [ [[package]] name = "taffy" -version = "0.9.2" +version = "0.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41ba83ebaf2954d31d05d67340fd46cebe99da2b7133b0dd68d70c65473a437b" +checksum = "aea22054047c16c3f34d3ac473a2170be1424b1115b2a3adcf28cfb067c88859" dependencies = [ "arrayvec", "grid", @@ -9341,9 +9238,26 @@ checksum = "d4d1330fe7f7f872cd05165130b10602d667b205fd85be09be2814b115d4ced9" [[package]] name = "test-with" -version = "0.15.8" +version = "0.16.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3c292571fb159e65c78c7c88a96e5ddfc326d6819e49b3a040d7b0aa3497925b" +dependencies = [ + "byte-unit", + "chrono", + "num_cpus", + "ping", + "reqwest 0.13.2", + "sysinfo", + "test-with-derive", + "uzers", + "which", +] + +[[package]] +name = "test-with-derive" +version = "0.16.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "27838d769fa9bf364bf4a352ec88862b6e6cb96a8e6705cc78fbb16ff26ee8b9" +checksum = "01b95b557b54c0d50b04688ab86aaeaf16f201a0341936f3bfa19497dee081bd" dependencies = [ "byte-unit", "chrono", @@ -9493,9 +9407,9 @@ dependencies = [ [[package]] name = "tinystr" -version = "0.8.2" +version = "0.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "42d3e9c45c09de15d06dd8acf5f4e0e399e85927b7f00711024eb7ae10fa4869" +checksum = "c8323304221c2a851516f22236c5722a72eaa19749016521d6dff0824447d96d" dependencies = [ "displaydoc", "zerovec", @@ -9528,9 +9442,9 @@ checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20" [[package]] name = "tokio" -version = "1.50.0" +version = "1.52.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "27ad5e34374e03cfffefc301becb44e9dc3c17584f414349ebe29ed26661822d" +checksum = "b67dee974fe86fd92cc45b7a95fdd2f99a36a6d7b0d431a231178d3d670bbcc6" dependencies = [ "bytes", "libc", @@ -9545,9 +9459,9 @@ dependencies = [ [[package]] name = "tokio-macros" -version = "2.6.1" +version = "2.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5c55a2eff8b69ce66c84f85e1da1c233edc36ceb85a2058d11b0d6a3c7e7569c" +checksum = "385a6cb71ab9ab790c5fe8d67f1645e6c450a7ce006a33de03daa956cf70a496" dependencies = [ "proc-macro2", "quote", @@ -9615,39 +9529,39 @@ dependencies = [ [[package]] name = "toml_datetime" -version = "1.0.0+spec-1.1.0" +version = "1.1.1+spec-1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32c2555c699578a4f59f0cc68e5116c8d7cabbd45e1409b989d4be085b53f13e" +checksum = "3165f65f62e28e0115a00b2ebdd37eb6f3b641855f9d636d3cd4103767159ad7" dependencies = [ "serde_core", ] [[package]] name = "toml_edit" -version = "0.25.4+spec-1.1.0" +version = "0.25.11+spec-1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7193cbd0ce53dc966037f54351dbbcf0d5a642c7f0038c382ef9e677ce8c13f2" +checksum = "0b59c4d22ed448339746c59b905d24568fcbb3ab65a500494f7b8c3e97739f2b" dependencies = [ "indexmap", - "toml_datetime 1.0.0+spec-1.1.0", + "toml_datetime 1.1.1+spec-1.1.0", "toml_parser", - "winnow 0.7.15", + "winnow 1.0.2", ] [[package]] name = "toml_parser" -version = "1.0.10+spec-1.1.0" +version = "1.1.2+spec-1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7df25b4befd31c4816df190124375d5a20c6b6921e2cad937316de3fccd63420" +checksum = "a2abe9b86193656635d2411dc43050282ca48aa31c2451210f4202550afb7526" dependencies = [ - "winnow 1.0.0", + "winnow 1.0.2", ] [[package]] name = "toml_writer" -version = "1.0.7+spec-1.1.0" +version = "1.1.1+spec-1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f17aaa1c6e3dc22b1da4b6bba97d066e354c7945cac2f7852d4e4e7ca7a6b56d" +checksum = "756daf9b1013ebe47a8776667b466417e2d4c5679d441c26230efd9ef78692db" [[package]] name = "tonic" @@ -9741,7 +9655,7 @@ name = "tpchgen-arrow" version = "2.0.2" source = "git+https://github.com/clflushopt/tpchgen-rs.git?rev=438e9c2dbc25b2fff82c0efc08b3f13b5707874f#438e9c2dbc25b2fff82c0efc08b3f13b5707874f" dependencies = [ - "arrow 58.0.0", + "arrow 58.1.0", "tpchgen", ] @@ -9799,7 +9713,7 @@ dependencies = [ "bytes", "chrono", "prost 0.12.6", - "rand 0.8.5", + "rand 0.8.6", "thread-id", "tracing", "tracing-subscriber", @@ -9848,7 +9762,7 @@ version = "2.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9ea3136b675547379c4bd395ca6b938e5ad3c3d20fad76e7fe85f9e0d011419c" dependencies = [ - "rand 0.9.2", + "rand 0.9.4", ] [[package]] @@ -9859,9 +9773,9 @@ checksum = "8e28f89b80c87b8fb0cf04ab448d5dd0dd0ade2f8891bae878de66a75a28600e" [[package]] name = "typenum" -version = "1.19.0" +version = "1.20.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "562d481066bde0658276a35467c4af00bdc6ee726305698a55b86e61d7ad82bb" +checksum = "40ce102ab67701b8526c123c1bab5cbe42d7040ccfd0f64af1a385808d2f43de" [[package]] name = "typify" @@ -9937,9 +9851,9 @@ checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" [[package]] name = "unicode-segmentation" -version = "1.12.0" +version = "1.13.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f6ccf251212114b54433ec949fd6a7841275f9ada20dddd2f29e9ceea4501493" +checksum = "9629274872b2bfaf8d66f5f15725007f635594914870f65218920345aa11aa8c" [[package]] name = "unicode-truncate" @@ -10026,9 +9940,9 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" [[package]] name = "uuid" -version = "1.22.0" +version = "1.23.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a68d3c8f01c0cfa54a75291d83601161799e4a89a39e0929f4b0354d88757a37" +checksum = "ddd74a9687298c6858e9b88ec8935ec45d22e8fd5e6394fa1bd4e99a87789c76" dependencies = [ "getrandom 0.4.2", "js-sys", @@ -10052,6 +9966,30 @@ version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ba73ea9cf16a25df0c8caa16c51acb937d5712a8429db78a3ee29d5dcacd3a65" +[[package]] +name = "vector-search-bench" +version = "0.1.0" +dependencies = [ + "anyhow", + "arrow-array 58.1.0", + "arrow-buffer 58.1.0", + "arrow-schema 58.1.0", + "clap", + "futures", + "indicatif", + "parquet 58.1.0", + "rand 0.10.1", + "serde", + "tabled", + "tempfile", + "tokio", + "tracing", + "vortex", + "vortex-bench", + "vortex-btrblocks", + "vortex-tensor", +] + [[package]] name = "version_check" version = "0.9.5" @@ -10063,13 +10001,14 @@ name = "vortex" version = "0.1.0" dependencies = [ "anyhow", - "arrow-array 58.0.0", + "arrow-array 58.1.0", "codspeed-divan-compat", "fastlanes", + "futures", "mimalloc", - "parquet 58.0.0", + "parquet 58.1.0", "paste", - "rand 0.10.0", + "rand 0.10.1", "rand_distr 0.6.0", "serde_json", "tokio", @@ -10078,6 +10017,7 @@ dependencies = [ "vortex", "vortex-alp", "vortex-array", + "vortex-bench", "vortex-btrblocks", "vortex-buffer", "vortex-bytebool", @@ -10114,7 +10054,7 @@ dependencies = [ "itertools 0.14.0", "num-traits", "prost 0.14.3", - "rand 0.10.0", + "rand 0.10.1", "rstest", "rustc-hash", "vortex-array", @@ -10131,16 +10071,17 @@ name = "vortex-array" version = "0.1.0" dependencies = [ "arbitrary", + "arc-swap", "arcref", - "arrow-arith 58.0.0", - "arrow-array 58.0.0", - "arrow-buffer 58.0.0", - "arrow-cast 58.0.0", - "arrow-data 58.0.0", - "arrow-ord 58.0.0", - "arrow-schema 58.0.0", - "arrow-select 58.0.0", - "arrow-string 58.0.0", + "arrow-arith 58.1.0", + "arrow-array 58.1.0", + "arrow-buffer 58.1.0", + "arrow-cast 58.1.0", + "arrow-data 58.1.0", + "arrow-ord 58.1.0", + "arrow-schema 58.1.0", + "arrow-select 58.1.0", + "arrow-string 58.1.0", "async-lock", "bytes", "cfg-if", @@ -10149,7 +10090,6 @@ dependencies = [ "enum-iterator", "flatbuffers", "futures", - "getrandom 0.4.2", "goldenfile", "half", "humansize", @@ -10165,7 +10105,7 @@ dependencies = [ "pin-project-lite", "primitive-types", "prost 0.14.3", - "rand 0.10.0", + "rand 0.10.1", "rand_distr 0.6.0", "rstest", "rstest_reuse", @@ -10204,9 +10144,9 @@ name = "vortex-bench" version = "0.1.0" dependencies = [ "anyhow", - "arrow-array 58.0.0", - "arrow-schema 58.0.0", - "arrow-select 58.0.0", + "arrow-array 58.1.0", + "arrow-schema 58.1.0", + "arrow-select 58.1.0", "async-trait", "bytes", "bzip2", @@ -10221,10 +10161,10 @@ dependencies = [ "noodles-bgzf", "noodles-vcf", "parking_lot", - "parquet 58.0.0", - "rand 0.10.0", + "parquet 58.1.0", + "rand 0.10.1", "regex", - "reqwest 0.12.28", + "reqwest 0.13.2", "serde", "serde_json", "sysinfo", @@ -10241,6 +10181,7 @@ dependencies = [ "url", "uuid", "vortex", + "vortex-tensor", ] [[package]] @@ -10248,11 +10189,10 @@ name = "vortex-btrblocks" version = "0.1.0" dependencies = [ "codspeed-divan-compat", - "getrandom 0.4.2", "itertools 0.14.0", "num-traits", "pco", - "rand 0.10.0", + "rand 0.10.1", "rstest", "rustc-hash", "test-with", @@ -10270,6 +10210,7 @@ dependencies = [ "vortex-pco", "vortex-runend", "vortex-sequence", + "vortex-session", "vortex-sparse", "vortex-tensor", "vortex-utils", @@ -10281,7 +10222,7 @@ dependencies = [ name = "vortex-buffer" version = "0.1.0" dependencies = [ - "arrow-buffer 58.0.0", + "arrow-buffer 58.1.0", "bitvec", "bytes", "codspeed-divan-compat", @@ -10312,16 +10253,17 @@ dependencies = [ name = "vortex-compat" version = "0.1.0" dependencies = [ - "arrow-array 58.0.0", - "arrow-select 58.0.0", + "arrow-array 58.1.0", + "arrow-select 58.1.0", + "base16ct", "bytes", "clap", "futures", - "parquet 58.0.0", - "reqwest 0.12.28", + "parquet 58.1.0", + "reqwest 0.13.2", "serde", "serde_json", - "sha2", + "sha2 0.11.0", "tempfile", "tokio", "tpchgen", @@ -10341,14 +10283,16 @@ dependencies = [ "itertools 0.14.0", "num-traits", "parking_lot", - "rand 0.10.0", + "rand 0.10.1", "rstest", "rustc-hash", "tracing", + "tracing-subscriber", "vortex-array", "vortex-buffer", "vortex-error", "vortex-mask", + "vortex-session", "vortex-utils", ] @@ -10377,7 +10321,8 @@ dependencies = [ "futures", "itertools 0.14.0", "kanal", - "object_store 0.13.1", + "num-traits", + "object_store 0.13.2", "parking_lot", "prost 0.14.3", "rstest", @@ -10405,8 +10350,8 @@ name = "vortex-cxx" version = "0.1.0" dependencies = [ "anyhow", - "arrow-array 58.0.0", - "arrow-schema 58.0.0", + "arrow-array 58.1.0", + "arrow-schema 58.1.0", "async-fs", "cxx", "futures", @@ -10420,25 +10365,25 @@ name = "vortex-datafusion" version = "0.1.0" dependencies = [ "anyhow", - "arrow-schema 58.0.0", + "arrow-schema 58.1.0", "async-trait", - "datafusion 53.0.0", - "datafusion-catalog 53.0.0", - "datafusion-common 53.0.0", - "datafusion-common-runtime 53.0.0", - "datafusion-datasource 53.0.0", - "datafusion-execution 53.0.0", - "datafusion-expr 53.0.0", - "datafusion-functions 53.0.0", - "datafusion-physical-expr 53.0.0", - "datafusion-physical-expr-adapter 53.0.0", - "datafusion-physical-expr-common 53.0.0", - "datafusion-physical-plan 53.0.0", - "datafusion-pruning 53.0.0", + "datafusion 53.1.0", + "datafusion-catalog 53.1.0", + "datafusion-common 53.1.0", + "datafusion-common-runtime 53.1.0", + "datafusion-datasource 53.1.0", + "datafusion-execution 53.1.0", + "datafusion-expr 53.1.0", + "datafusion-functions 53.1.0", + "datafusion-physical-expr 53.1.0", + "datafusion-physical-expr-adapter 53.1.0", + "datafusion-physical-expr-common 53.1.0", + "datafusion-physical-plan 53.1.0", + "datafusion-pruning 53.1.0", "futures", "insta", "itertools 0.14.0", - "object_store 0.13.1", + "object_store 0.13.2", "rstest", "tempfile", "tokio", @@ -10495,10 +10440,10 @@ dependencies = [ "jiff", "kanal", "num-traits", - "object_store 0.13.1", + "object_store 0.13.2", "parking_lot", "paste", - "reqwest 0.12.28", + "reqwest 0.13.2", "rstest", "tempfile", "tracing", @@ -10515,10 +10460,10 @@ dependencies = [ name = "vortex-error" version = "0.1.0" dependencies = [ - "arrow-schema 58.0.0", + "arrow-schema 58.1.0", "flatbuffers", "jiff", - "object_store 0.13.1", + "object_store 0.13.2", "prost 0.14.3", "serial_test", "temp-env", @@ -10535,7 +10480,7 @@ dependencies = [ "lending-iterator", "num-traits", "prost 0.14.3", - "rand 0.10.0", + "rand 0.10.1", "rstest", "vortex-alp", "vortex-array", @@ -10550,14 +10495,17 @@ dependencies = [ name = "vortex-ffi" version = "0.1.0" dependencies = [ + "arrow-array 58.1.0", + "arrow-schema 58.1.0", "async-fs", "cbindgen", "futures", "itertools 0.14.0", "mimalloc", - "object_store 0.13.1", + "object_store 0.13.2", "paste", "prost 0.14.3", + "rand 0.10.1", "tempfile", "tracing", "tracing-subscriber", @@ -10574,17 +10522,15 @@ dependencies = [ "bytes", "flatbuffers", "futures", - "getrandom 0.4.2", "itertools 0.14.0", "kanal", "moka", - "object_store 0.13.1", + "object_store 0.13.2", "oneshot 0.2.1", "parking_lot", "pin-project-lite", "tokio", "tracing", - "uuid", "vortex-alp", "vortex-array", "vortex-btrblocks", @@ -10628,7 +10574,7 @@ dependencies = [ "codspeed-divan-compat", "fsst-rs", "prost 0.14.3", - "rand 0.10.0", + "rand 0.10.1", "rstest", "vortex-array", "vortex-buffer", @@ -10674,13 +10620,11 @@ dependencies = [ "bytes", "custom-labels", "futures", - "getrandom 0.4.2", "glob", - "handle", "itertools 0.14.0", "kanal", "log", - "object_store 0.13.1", + "object_store 0.13.2", "oneshot 0.2.1", "parking_lot", "pin-project-lite", @@ -10694,6 +10638,7 @@ dependencies = [ "vortex-error", "vortex-metrics", "vortex-session", + "vortex-utils", "wasm-bindgen-futures", ] @@ -10718,16 +10663,13 @@ dependencies = [ name = "vortex-jni" version = "0.1.0" dependencies = [ - "arrow-array 58.0.0", - "arrow-ipc 58.0.0", - "arrow-schema 58.0.0", + "arrow-array 58.1.0", + "arrow-schema 58.1.0", "futures", - "jni", - "object_store 0.13.1", + "jni 0.22.4", + "object_store 0.13.2", "parking_lot", - "prost 0.14.3", "thiserror 2.0.18", - "tokio", "tracing", "tracing-subscriber", "url", @@ -10739,8 +10681,8 @@ name = "vortex-layout" version = "0.1.0" dependencies = [ "arcref", - "arrow-array 58.0.0", - "arrow-schema 58.0.0", + "arrow-array 58.1.0", + "arrow-schema 58.1.0", "async-stream", "async-trait", "bit-vec", @@ -10758,6 +10700,7 @@ dependencies = [ "rstest", "rustc-hash", "sketches-ddsketch 0.4.0", + "temp-env", "termtree", "tokio", "tracing", @@ -10769,6 +10712,7 @@ dependencies = [ "vortex-io", "vortex-mask", "vortex-metrics", + "vortex-runend", "vortex-scan", "vortex-sequence", "vortex-session", @@ -10779,7 +10723,7 @@ dependencies = [ name = "vortex-mask" version = "0.1.0" dependencies = [ - "arrow-buffer 58.0.0", + "arrow-buffer 58.1.0", "codspeed-divan-compat", "itertools 0.14.0", "rstest", @@ -10792,7 +10736,6 @@ dependencies = [ name = "vortex-metrics" version = "0.1.0" dependencies = [ - "getrandom 0.4.2", "parking_lot", "sketches-ddsketch 0.4.0", ] @@ -10804,7 +10747,7 @@ dependencies = [ "bindgen", "libloading 0.8.9", "liblzma", - "reqwest 0.12.28", + "reqwest 0.13.2", "tar", "vortex-cuda-macros", ] @@ -10813,9 +10756,9 @@ dependencies = [ name = "vortex-parquet-variant" version = "0.1.0" dependencies = [ - "arrow-array 58.0.0", - "arrow-buffer 58.0.0", - "arrow-schema 58.0.0", + "arrow-array 58.1.0", + "arrow-buffer 58.1.0", + "arrow-schema 58.1.0", "chrono", "parquet-variant", "parquet-variant-compute", @@ -10855,14 +10798,14 @@ dependencies = [ name = "vortex-python" version = "0.1.0" dependencies = [ - "arrow-array 58.0.0", - "arrow-data 58.0.0", - "arrow-schema 58.0.0", + "arrow-array 58.1.0", + "arrow-data 58.1.0", + "arrow-schema 58.1.0", "bytes", "itertools 0.14.0", "log", "mimalloc", - "object_store 0.13.1", + "object_store 0.13.2", "parking_lot", "pyo3", "pyo3-bytes", @@ -10879,13 +10822,13 @@ name = "vortex-runend" version = "0.1.0" dependencies = [ "arbitrary", - "arrow-array 58.0.0", - "arrow-schema 58.0.0", + "arrow-array 58.1.0", + "arrow-schema 58.1.0", "codspeed-divan-compat", "itertools 0.14.0", "num-traits", "prost 0.14.3", - "rand 0.10.0", + "rand 0.10.1", "rstest", "vortex-array", "vortex-buffer", @@ -10900,7 +10843,7 @@ version = "0.1.0" dependencies = [ "async-trait", "futures", - "roaring 0.11.3", + "roaring", "tracing", "vortex-array", "vortex-buffer", @@ -10929,8 +10872,8 @@ dependencies = [ name = "vortex-session" version = "0.1.0" dependencies = [ - "arcref", "dashmap", + "lasso", "parking_lot", "vortex-error", "vortex-utils", @@ -10959,7 +10902,7 @@ dependencies = [ "async-trait", "bigdecimal", "clap", - "datafusion 53.0.0", + "datafusion 53.1.0", "datafusion-sqllogictest", "futures", "indicatif", @@ -10976,14 +10919,17 @@ dependencies = [ name = "vortex-tensor" version = "0.1.0" dependencies = [ + "codspeed-divan-compat", "half", "itertools 0.14.0", + "mimalloc", "num-traits", "prost 0.14.3", - "rand 0.10.0", + "rand 0.10.1", "rand_distr 0.6.0", "rstest", "vortex-array", + "vortex-btrblocks", "vortex-buffer", "vortex-compressor", "vortex-error", @@ -10996,8 +10942,8 @@ dependencies = [ name = "vortex-test-e2e-cuda" version = "0.1.0" dependencies = [ - "arrow-array 58.0.0", - "arrow-schema 58.0.0", + "arrow-array 58.1.0", + "arrow-schema 58.1.0", "futures", "vortex", "vortex-cuda", @@ -11008,12 +10954,12 @@ name = "vortex-tui" version = "0.1.0" dependencies = [ "anyhow", - "arrow-array 58.0.0", - "arrow-schema 58.0.0", + "arrow-array 58.1.0", + "arrow-schema 58.1.0", "clap", "console_error_panic_hook", "crossterm", - "datafusion 53.0.0", + "datafusion 53.1.0", "env_logger", "flatbuffers", "futures", @@ -11022,7 +10968,7 @@ dependencies = [ "indicatif", "itertools 0.14.0", "js-sys", - "parquet 58.0.0", + "parquet 58.1.0", "ratatui", "ratzilla", "serde", @@ -11041,7 +10987,7 @@ name = "vortex-utils" version = "0.1.0" dependencies = [ "dashmap", - "hashbrown 0.16.1", + "hashbrown 0.17.0", "parking_lot", "vortex-error", ] @@ -11050,9 +10996,9 @@ dependencies = [ name = "vortex-web-wasm" version = "0.1.0" dependencies = [ - "arrow-array 58.0.0", - "arrow-ipc 58.0.0", - "arrow-schema 58.0.0", + "arrow-array 58.1.0", + "arrow-ipc 58.1.0", + "arrow-schema 58.1.0", "console_error_panic_hook", "futures", "js-sys", @@ -11120,11 +11066,11 @@ checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" [[package]] name = "wasip2" -version = "1.0.2+wasi-0.2.9" +version = "1.0.3+wasi-0.2.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9517f9239f02c069db75e65f174b3da828fe5f5b945c4dd26bd25d89c03ebcf5" +checksum = "20064672db26d7cdc89c7798c48a0fdfac8213434a1186e5ef29fd560ae223d6" dependencies = [ - "wit-bindgen", + "wit-bindgen 0.57.1", ] [[package]] @@ -11133,14 +11079,14 @@ version = "0.4.0+wasi-0.3.0-rc-2026-01-06" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5428f8bf88ea5ddc08faddef2ac4a67e390b88186c703ce6dbd955e1c145aca5" dependencies = [ - "wit-bindgen", + "wit-bindgen 0.51.0", ] [[package]] name = "wasm-bindgen" -version = "0.2.114" +version = "0.2.118" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6532f9a5c1ece3798cb1c2cfdba640b9b3ba884f5db45973a6f442510a87d38e" +checksum = "0bf938a0bacb0469e83c1e148908bd7d5a6010354cf4fb73279b7447422e3a89" dependencies = [ "cfg-if", "once_cell", @@ -11151,23 +11097,19 @@ dependencies = [ [[package]] name = "wasm-bindgen-futures" -version = "0.4.64" +version = "0.4.68" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e9c5522b3a28661442748e09d40924dfb9ca614b21c00d3fd135720e48b67db8" +checksum = "f371d383f2fb139252e0bfac3b81b265689bf45b6874af544ffa4c975ac1ebf8" dependencies = [ - "cfg-if", - "futures-util", "js-sys", - "once_cell", "wasm-bindgen", - "web-sys", ] [[package]] name = "wasm-bindgen-macro" -version = "0.2.114" +version = "0.2.118" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "18a2d50fcf105fb33bb15f00e7a77b772945a2ee45dcf454961fd843e74c18e6" +checksum = "eeff24f84126c0ec2db7a449f0c2ec963c6a49efe0698c4242929da037ca28ed" dependencies = [ "quote", "wasm-bindgen-macro-support", @@ -11175,9 +11117,9 @@ dependencies = [ [[package]] name = "wasm-bindgen-macro-support" -version = "0.2.114" +version = "0.2.118" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "03ce4caeaac547cdf713d280eda22a730824dd11e6b8c3ca9e42247b25c631e3" +checksum = "9d08065faf983b2b80a79fd87d8254c409281cf7de75fc4b773019824196c904" dependencies = [ "bumpalo", "proc-macro2", @@ -11188,9 +11130,9 @@ dependencies = [ [[package]] name = "wasm-bindgen-shared" -version = "0.2.114" +version = "0.2.118" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "75a326b8c223ee17883a4251907455a2431acc2791c98c26279376490c378c16" +checksum = "5fd04d9e306f1907bd13c6361b5c6bfc7b3b3c095ed3f8a9246390f8dbdee129" dependencies = [ "unicode-ident", ] @@ -11230,6 +11172,19 @@ dependencies = [ "web-sys", ] +[[package]] +name = "wasm-streams" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d1ec4f6517c9e11ae630e200b2b65d193279042e28edd4a2cda233e46670bbb" +dependencies = [ + "futures-util", + "js-sys", + "wasm-bindgen", + "wasm-bindgen-futures", + "web-sys", +] + [[package]] name = "wasmparser" version = "0.244.0" @@ -11244,9 +11199,9 @@ dependencies = [ [[package]] name = "web-sys" -version = "0.3.91" +version = "0.3.95" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "854ba17bb104abfb26ba36da9729addc7ce7f06f5c0f90f3c391f8461cca21f9" +checksum = "4f2dfbb17949fa2088e5d39408c48368947b86f7834484e87b73de55bc14d97d" dependencies = [ "js-sys", "wasm-bindgen", @@ -11264,18 +11219,9 @@ dependencies = [ [[package]] name = "webpki-root-certs" -version = "1.0.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "804f18a4ac2676ffb4e8b5b5fa9ae38af06df08162314f96a68d2a363e21a8ca" -dependencies = [ - "rustls-pki-types", -] - -[[package]] -name = "webpki-roots" -version = "1.0.6" +version = "1.0.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "22cfaf3c063993ff62e73cb4311efde4db1efb31ab78a3e5c457939ad5cc0bed" +checksum = "f31141ce3fc3e300ae89b78c0dd67f9708061d1d2eda54b8209346fd6be9a92c" dependencies = [ "rustls-pki-types", ] @@ -11677,15 +11623,15 @@ name = "winnow" version = "0.7.15" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "df79d97927682d2fd8adb29682d1140b343be4ac0f08fd68b7765d9c059d3945" -dependencies = [ - "memchr", -] [[package]] name = "winnow" -version = "1.0.0" +version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a90e88e4667264a994d34e6d1ab2d26d398dcdca8b7f52bec8668957517fc7d8" +checksum = "2ee1708bef14716a11bae175f579062d4554d95be2c6829f518df847b7b3fdd0" +dependencies = [ + "memchr", +] [[package]] name = "wit-bindgen" @@ -11696,6 +11642,12 @@ dependencies = [ "wit-bindgen-rust-macro", ] +[[package]] +name = "wit-bindgen" +version = "0.57.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" + [[package]] name = "wit-bindgen-core" version = "0.51.0" @@ -11775,36 +11727,11 @@ dependencies = [ "wasmparser", ] -[[package]] -name = "wkb" -version = "0.9.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a120b336c7ad17749026d50427c23d838ecb50cd64aaea6254b5030152f890a9" -dependencies = [ - "byteorder", - "geo-traits", - "num_enum", - "thiserror 1.0.69", -] - -[[package]] -name = "wkt" -version = "0.14.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "efb2b923ccc882312e559ffaa832a055ba9d1ac0cc8e86b3e25453247e4b81d7" -dependencies = [ - "geo-traits", - "geo-types", - "log", - "num-traits", - "thiserror 1.0.69", -] - [[package]] name = "writeable" -version = "0.6.2" +version = "0.6.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9edde0db4769d2dc68579893f2306b26c6ecfbe0ef499b013d731b7b9247e0b9" +checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4" [[package]] name = "wyz" @@ -11866,9 +11793,9 @@ checksum = "cfe53a6657fd280eaa890a3bc59152892ffa3e30101319d168b781ed6529b049" [[package]] name = "yoke" -version = "0.8.1" +version = "0.8.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72d6e5c6afb84d73944e5cedb052c4680d5657337201555f9f2a16b7406d4954" +checksum = "abe8c5fda708d9ca3df187cae8bfb9ceda00dd96231bed36e445a1a48e66f9ca" dependencies = [ "stable_deref_trait", "yoke-derive", @@ -11877,9 +11804,9 @@ dependencies = [ [[package]] name = "yoke-derive" -version = "0.8.1" +version = "0.8.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b659052874eb698efe5b9e8cf382204678a0086ebf46982b79d6ca3182927e5d" +checksum = "de844c262c8848816172cef550288e7dc6c7b7814b4ee56b3e1553f275f1858e" dependencies = [ "proc-macro2", "quote", @@ -11889,18 +11816,18 @@ dependencies = [ [[package]] name = "zerocopy" -version = "0.8.42" +version = "0.8.48" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2578b716f8a7a858b7f02d5bd870c14bf4ddbbcf3a4c05414ba6503640505e3" +checksum = "eed437bf9d6692032087e337407a86f04cd8d6a16a37199ed57949d415bd68e9" dependencies = [ "zerocopy-derive", ] [[package]] name = "zerocopy-derive" -version = "0.8.42" +version = "0.8.48" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7e6cc098ea4d3bd6246687de65af3f920c430e236bee1e3bf2e441463f08a02f" +checksum = "70e3cd084b1788766f53af483dd21f93881ff30d7320490ec3ef7526d203bad4" dependencies = [ "proc-macro2", "quote", @@ -11909,18 +11836,18 @@ dependencies = [ [[package]] name = "zerofrom" -version = "0.1.6" +version = "0.1.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "50cc42e0333e05660c3587f3bf9d0478688e15d870fab3346451ce7f8c9fbea5" +checksum = "69faa1f2a1ea75661980b013019ed6687ed0e83d069bc1114e2cc74c6c04c4df" dependencies = [ "zerofrom-derive", ] [[package]] name = "zerofrom-derive" -version = "0.1.6" +version = "0.1.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d71e5d6e06ab090c67b5e44993ec16b72dcbaabc526db883a360057678b48502" +checksum = "11532158c46691caf0f2593ea8358fed6bbf68a0315e80aae9bd41fbade684a1" dependencies = [ "proc-macro2", "quote", @@ -11936,9 +11863,9 @@ checksum = "b97154e67e32c85465826e8bcc1c59429aaaf107c1e4a9e53c8d8ccd5eff88d0" [[package]] name = "zerotrie" -version = "0.2.3" +version = "0.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2a59c17a5562d507e4b54960e8569ebee33bee890c70aa3fe7b97e85a9fd7851" +checksum = "0f9152d31db0792fa83f70fb2f83148effb5c1f5b8c7686c3459e361d9bc20bf" dependencies = [ "displaydoc", "yoke", @@ -11947,9 +11874,9 @@ dependencies = [ [[package]] name = "zerovec" -version = "0.11.5" +version = "0.11.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6c28719294829477f525be0186d13efa9a3c602f7ec202ca9e353d310fb9a002" +checksum = "90f911cbc359ab6af17377d242225f4d75119aec87ea711a880987b18cd7b239" dependencies = [ "yoke", "zerofrom", @@ -11958,9 +11885,9 @@ dependencies = [ [[package]] name = "zerovec-derive" -version = "0.11.2" +version = "0.11.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "eadce39539ca5cb3985590102671f2567e659fca9666581ad3411d59207951f3" +checksum = "625dc425cab0dca6dc3c3319506e6593dcb08a9f387ea3b284dbd52a92c40555" dependencies = [ "proc-macro2", "quote", @@ -11978,9 +11905,9 @@ dependencies = [ [[package]] name = "zip" -version = "8.3.1" +version = "8.6.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5c546feb4481b0fbafb4ef0d79b6204fc41c6f9884b1b73b1d73f82442fc0845" +checksum = "2d04a6b5381502aa6087c94c669499eb1602eb9c5e8198e534de571f7154809b" dependencies = [ "aes", "bzip2", @@ -11995,7 +11922,7 @@ dependencies = [ "memchr", "pbkdf2", "ppmd-rust", - "sha1", + "sha1 0.11.0", "time", "typed-path", "zeroize", diff --git a/Cargo.toml b/Cargo.toml index cc0c7c45a0c..35179561e14 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -59,6 +59,7 @@ members = [ "benchmarks/datafusion-bench", "benchmarks/duckdb-bench", "benchmarks/random-access-bench", + "benchmarks/vector-search-bench", ] exclude = ["java/testfiles", "wasm-test"] resolver = "2" @@ -81,14 +82,14 @@ keywords = ["vortex"] license = "Apache-2.0" readme = "README.md" repository = "https://github.com/spiraldb/vortex" -rust-version = "1.90" +rust-version = "1.91.0" version = "0.1.0" [workspace.dependencies] aho-corasick = "1.1.3" anyhow = "1.0.97" arbitrary = "1.3.2" -arc-swap = "1.8" +arc-swap = "1.9" arcref = "0.2.0" arrow-arith = "58" arrow-array = "58" @@ -104,6 +105,7 @@ async-fs = "2.2.0" async-lock = "3.4" async-stream = "0.3.6" async-trait = "0.1.89" +base16ct = "1.0.0" bigdecimal = "0.4.8" bindgen = "0.72.0" bit-vec = "0.9.0" @@ -118,14 +120,13 @@ chrono = "0.4.42" clap = "4.5" criterion = "0.8" crossterm = "0.29" -cudarc = { version = "0.18.2", features = [ +cudarc = { version = "0.19.0", features = [ # The NSight Compute version available on lambda.ai hosts does not inject a symbol for # several functions that cudarc expects to load when this is set to "cuda-12080". We # don't use any CUDA 12.8 specific features currently so this is fine. "cuda-12050", ] } custom-labels = "0.4.4" -daachorse = "1.0.0" dashmap = "6.1.0" datafusion = { version = "53", default-features = false, features = ["sql"] } datafusion-catalog = { version = "53" } @@ -154,7 +155,7 @@ get_dir = "0.5.0" glob = "0.3.2" goldenfile = "1" half = { version = "2.7.1", features = ["std", "num-traits"] } -hashbrown = "0.16.1" +hashbrown = "0.17.0" humansize = "2.1.3" indicatif = "0.18.0" insta = "1.43" @@ -162,7 +163,9 @@ inventory = "0.3.20" itertools = "0.14.0" jetscii = "0.5.3" jiff = "0.2.0" +jni = { version = "0.22.0" } kanal = "0.1.1" +lasso = { version = "0.7", features = ["multi-threaded"] } lending-iterator = "0.1.7" libfuzzer-sys = "0.4" libloading = "0.8" @@ -175,7 +178,7 @@ mimalloc = "0.1.42" moka = { version = "0.12.10", default-features = false } multiversion = "0.8.0" noodles-bgzf = "0.46.0" -noodles-vcf = { version = "0.86.0", features = ["async"] } +noodles-vcf = { version = "0.87.0", features = ["async"] } num-traits = "0.2.19" num_enum = { version = "0.7.3", default-features = false } object_store = { version = "0.13.1", default-features = false } @@ -196,23 +199,23 @@ proc-macro2 = "1.0.95" prost = "0.14" prost-build = "0.14" prost-types = "0.14" -public-api = "0.51" +public-api = { git = "https://github.com/vortex-data/cargo-public-api.git", branch = "main" } pyo3 = { version = "0.28.0" } pyo3-bytes = "0.6" pyo3-log = "0.13.0" pyo3-object_store = "0.9.0" quote = "1.0.44" -rand = "0.10.0" +rand = "0.10.1" rand_distr = "0.6" ratatui = { version = "0.30", default-features = false } regex = "1.11.0" regex-automata = "0.4" -reqwest = { version = "0.12.4", features = [ +reqwest = { version = "0.13.0", features = [ + "blocking", "charset", "http2", - "macos-system-configuration", - "blocking", - "rustls-tls", + "rustls", + "system-proxy", ], default-features = false } roaring = "0.11.0" rstest = "0.26.1" @@ -221,9 +224,9 @@ rustc-hash = "2.1" serde = "1.0.220" serde_json = "1.0.138" serde_test = "1.0.176" -sha2 = "0.10" +sha2 = "0.11.0" simdutf8 = "0.1.5" -similar = "2.7.0" +similar = "3.0.0" sketches-ddsketch = "0.4.0" smol = "2.0.2" static_assertions = "1.1" @@ -231,13 +234,14 @@ strum = "0.28" syn = { version = "2.0.117", features = ["full"] } sysinfo = "0.38.0" tabled = { version = "0.20.0", default-features = false } -taffy = "0.9.0" +taffy = "0.10.0" take_mut = "0.2.2" tar = "0.4" target-lexicon = "0.13" +temp-env = "0.3" tempfile = "3" termtree = { version = "1.0" } -test-with = "0.15" +test-with = "0.16" thiserror = "2.0.3" tokio = { version = "1.48" } tokio-stream = "0.1.17" @@ -302,11 +306,6 @@ vortex-duckdb = { path = "./vortex-duckdb", default-features = false } vortex-test-e2e-cuda = { path = "./vortex-test/e2e-cuda", default-features = false } vortex-tui = { path = "./vortex-tui" } -[workspace.dependencies.getrandom_v03] -features = ["wasm_js"] -package = "getrandom" -version = "0.4.0" - [workspace.lints.rust] let_underscore_drop = "deny" macro_use_extern_crate = "deny" @@ -316,8 +315,6 @@ unused_lifetimes = "deny" unused_qualifications = "deny" unexpected_cfgs = { level = "deny", check-cfg = [ "cfg(codspeed)", - "cfg(disable_loom)", - "cfg(vortex_nightly)", 'cfg(target_os, values("unknown"))', ] } warnings = "warn" @@ -349,10 +346,17 @@ mem_forget = "deny" multiple_crate_versions = "allow" needless_range_loop = "allow" or_fun_call = "deny" +ref_option = "deny" +ref_option_ref = "deny" panic = "deny" # panic_in_result_fn = "deny" -- we cannot disable this for tests to use assertions clone_on_ref_ptr = "deny" redundant_clone = "deny" +redundant_static_lifetimes = "deny" +ref_binding_to_reference = "deny" +ref_patterns = "deny" +repeat_vec_with_capacity = "deny" +reserve_after_initialization = "deny" same_name_method = "deny" tests_outside_test_module = "deny" # todo = "deny" diff --git a/README.md b/README.md index 92650b0db67..df5a70c09d7 100644 --- a/README.md +++ b/README.md @@ -11,6 +11,8 @@ [Join the community on Slack!](https://vortex.dev/slack) | [Documentation](https://docs.vortex.dev/) | [Performance Benchmarks](https://bench.vortex.dev) +If you are interested in closer collaboration, please email info@vortex.dev + ## Overview Vortex is a next-generation columnar file format and toolkit designed for high-performance data processing. @@ -86,11 +88,14 @@ uv add vortex-data For browsing the structure of Vortex files, you can use the `vx` command-line tool. ```bash -# Install latest release -cargo install vortex-tui --locked +# Install pre-built binary (fast, recommended) +cargo binstall vortex-tui # Or build from source -cargo install --path vortex-tui --locked +cargo install vortex-tui --locked + +# Or run via Python without installing +uvx --from vortex-data vx --help # Usage vx browse diff --git a/REUSE.toml b/REUSE.toml index 37cc868b2c2..161f6e3086a 100644 --- a/REUSE.toml +++ b/REUSE.toml @@ -30,6 +30,11 @@ path = ["**/README.md", "AGENTS.md", "CLAUDE.md", "CONTRIBUTING.md", "STYLE.md", SPDX-FileCopyrightText = "Copyright the Vortex contributors" SPDX-License-Identifier = "CC-BY-4.0" +[[annotations]] +path = ".agents/skills/**" +SPDX-FileCopyrightText = "Copyright the Vortex contributors" +SPDX-License-Identifier = "CC-BY-4.0" + [[annotations]] path = ["**/.gitignore", ".gitmodules", ".python-version", "**/*.lock", "**/*.lockfile", "**/*.toml", "**/*.json", ".idea/**", ".github/**", "codecov.yml", "java/gradle/wrapper/gradle-wrapper.properties"] precedence = "override" diff --git a/bench-orchestrator/README.md b/bench-orchestrator/README.md index 3d9d7ac99c6..23a927b96d3 100644 --- a/bench-orchestrator/README.md +++ b/bench-orchestrator/README.md @@ -41,18 +41,34 @@ vx-bench run [options] **Arguments:** -- `benchmark`: Benchmark suite to run (`tpch`, `tpcds`, `clickbench`, `fineweb`, `gh-archive`, `public-bi`, `statpopgen`) +- `benchmark`: Benchmark suite to run (`tpch`, `tpcds`, `clickbench`, `fineweb`, `gh-archive`, `polarsignals`, `public-bi`, `statpopgen`) **Options:** - `--engine, -e`: Engines to benchmark, comma-separated (default: `datafusion,duckdb`) - `--format, -f`: Formats to benchmark, comma-separated (default: `parquet,vortex`) +- `--targets-json`: Exact benchmark targets as JSON, e.g. `'[{"engine":"datafusion","format":"parquet"}]'` - `--queries, -q`: Specific queries to run (e.g., `1,2,5`) - `--exclude-queries`: Queries to skip - `--iterations, -i`: Iterations per query (default: 5) - `--label, -l`: Label for this run (useful for later reference) - `--track-memory`: Enable memory usage tracking - `--build/--no-build`: Build binaries before running (default: build) +- `--output`: Optional compatibility output path for raw JSON lines + +### `prepare-data` - Generate Benchmark Data + +Generate only the data formats needed for a benchmark run. + +```bash +vx-bench prepare-data [options] +``` + +**Options:** + +- `--format, -f`: Formats to generate, comma-separated +- `--formats-json`: Exact data formats as JSON, e.g. `'["parquet","vortex"]'` +- `--opt`: Benchmark-specific options such as `scale-factor=10.0` ### `compare` - Compare Results @@ -204,10 +220,10 @@ Test performance at different data scales: ```bash # Run at SF1 -vx-bench run tpch -s 1 -l sf1 +vx-bench run tpch --opt scale-factor=1.0 -l sf1 # Run at SF10 -vx-bench run tpch -s 10 -l sf10 +vx-bench run tpch --opt scale-factor=10.0 -l sf10 # Compare scaling behavior vx-bench compare --runs sf1,sf10 @@ -253,10 +269,30 @@ vx-bench clean --older-than "30 days" --no-keep-labeled | Engine | Supported Formats | |------------|-------------------------------------------| -| datafusion | parquet, vortex, vortex-compact, lance | +| datafusion | arrow, parquet, vortex, vortex-compact, lance | | duckdb | parquet, vortex, vortex-compact, duckdb | | lance | lance | +`datafusion:lance` is executed via the `lance-bench` backend while preserving `datafusion:lance` +result labels. + +## CI Usage + +The SQL benchmark workflow uses explicit JSON target selection instead of parsing `engine:format` +strings in shell: + +```bash +uv run --project bench-orchestrator vx-bench prepare-data tpch \ + --formats-json '["parquet","vortex","vortex-compact"]' \ + --opt scale-factor=1.0 + +uv run --project bench-orchestrator vx-bench run tpch \ + --targets-json '[{"engine":"datafusion","format":"parquet"},{"engine":"duckdb","format":"vortex"}]' \ + --opt scale-factor=1.0 \ + --output results.json \ + --no-build +``` + ## Output Format Comparison results are displayed in a pivot table format: @@ -299,5 +335,6 @@ Benchmarks are built with: - Profile: `release_debug` - RUSTFLAGS: `-C target-cpu=native -C force-frame-pointers=yes` +- Features: `unstable_encodings` This enables native CPU optimizations while preserving debug symbols for profiling. diff --git a/bench-orchestrator/bench_orchestrator/cli.py b/bench-orchestrator/bench_orchestrator/cli.py index eba7521fe9f..d497d85ed13 100644 --- a/bench-orchestrator/bench_orchestrator/cli.py +++ b/bench-orchestrator/bench_orchestrator/cli.py @@ -3,7 +3,10 @@ """CLI for benchmark orchestration.""" +import subprocess +from contextlib import contextmanager from datetime import datetime, timedelta +from pathlib import Path from typing import Annotated import pandas as pd @@ -14,12 +17,16 @@ from .comparison import analyzer from .comparison.reporter import pivot_comparison_table from .config import ( - ENGINE_FORMATS, Benchmark, + BenchmarkTarget, BuildConfig, Engine, Format, RunConfig, + group_targets_by_backend, + parse_formats_json, + parse_targets_json, + resolve_axis_targets, ) from .runner.builder import BenchmarkBuilder from .runner.executor import BenchmarkExecutor @@ -43,6 +50,21 @@ def parse_formats(value: str) -> list[Format]: return [Format(f.strip()) for f in value.split(",")] +def parse_options(value: list[str] | None) -> dict[str, str]: + """Parse repeated --opt key=value options into a dictionary.""" + parsed: dict[str, str] = {} + if not value: + return parsed + + for opt in value: + for segment in opt.split(","): + key, raw_value, *rest = segment.split("=") + if rest: + raise ValueError("Options must be key-value pairs separated by =") + parsed[key] = raw_value + return parsed + + def parse_queries(value: str | None) -> list[int] | None: """Parse comma-separated query numbers.""" if not value: @@ -62,6 +84,101 @@ def run_ref_auto_complete() -> list[str]: return list(map(lambda x: x.run_id, ResultStore().list_runs(limit=None))) +def targets_from_axes(engine: str, format: str) -> tuple[list[BenchmarkTarget], list[str]]: + """Resolve legacy engine/format axes into explicit benchmark targets.""" + return resolve_axis_targets(parse_engines(engine), parse_formats(format)) + + +def backends_for_engines(engines: list[Engine]) -> list[Engine]: + """Resolve legacy engine selections to unique execution engines.""" + seed_formats = { + Engine.DATAFUSION: Format.PARQUET, + Engine.DUCKDB: Format.PARQUET, + Engine.LANCE: Format.LANCE, + } + return list( + group_targets_by_backend(BenchmarkTarget(engine=engine, format=seed_formats[engine]) for engine in engines) + ) + + +@contextmanager +def open_results_output(path: Path | None): + """Open an optional compatibility JSONL output file.""" + if path is None: + yield None + return + + if path.parent != Path(): + path.parent.mkdir(parents=True, exist_ok=True) + + with path.open("w", encoding="utf-8") as handle: + yield handle + + +def write_result_line(line: str, store_writer, compatibility_file) -> None: + """Write a raw result line to the run store and optional compatibility output.""" + store_writer(line) + if compatibility_file is None: + return + + line = line.strip() + if line.startswith("{"): + compatibility_file.write(line + "\n") + compatibility_file.flush() + + +@app.command("prepare-data") +def prepare_data( + benchmark: Annotated[Benchmark, typer.Argument(help="Benchmark suite to prepare data for")], + format: Annotated[ + str | None, + typer.Option("--format", "-f", help="Formats to generate (comma-separated)"), + ] = None, + formats_json: Annotated[ + str | None, + typer.Option("--formats-json", help="Exact data formats to generate as a JSON array"), + ] = None, + verbose: Annotated[bool, typer.Option("--verbose", "-v", help="Log underlying commands")] = False, + options: Annotated[list[str] | None, typer.Option("--opt", help="Benchmark-specific options")] = None, +) -> None: + """Generate benchmark data for explicitly requested formats.""" + if format and formats_json: + console.print("[red]Specify only one of --format or --formats-json[/red]") + raise typer.Exit(1) + if not format and not formats_json: + console.print("[red]Must specify one of --format or --formats-json[/red]") + raise typer.Exit(1) + + try: + formats = parse_formats_json(formats_json) if formats_json else parse_formats(format or "") + bench_opts = parse_options(options) + except ValueError as exc: + console.print(f"[red]{exc}[/red]") + raise typer.Exit(1) from exc + + builder = BenchmarkBuilder(verbose=verbose) + binary_path = builder.get_data_generator_path() + if not binary_path.exists(): + console.print(f"[red]Binary not found: {binary_path}[/red]") + console.print("Build benchmark binaries before running prepare-data") + raise typer.Exit(1) + + cmd = [str(binary_path), benchmark.value, "--formats", ",".join(fmt.value for fmt in formats)] + if verbose: + cmd.append("--verbose") + for key, value in bench_opts.items(): + cmd.extend(["--opt", f"{key}={value}"]) + + if verbose: + console.print(f"[dim]$ {' '.join(cmd)}[/dim]") + + try: + subprocess.run(cmd, check=True) + except subprocess.CalledProcessError as exc: + console.print(f"[red]Data generation failed: {exc}[/red]") + raise typer.Exit(1) from exc + + @app.command() def run( benchmark: Annotated[Benchmark, typer.Argument(help="Benchmark suite to run")], @@ -81,27 +198,45 @@ def run( tracing: Annotated[bool, typer.Option("--tracing", help="Record a trace for use with perfetto")] = False, build: Annotated[bool, typer.Option("--build/--no-build", help="Build binaries before running")] = True, verbose: Annotated[bool, typer.Option("--verbose", "-v", help="Log underlying commands")] = False, + targets_json: Annotated[ + str | None, + typer.Option("--targets-json", help="Exact benchmark targets as a JSON array"), + ] = None, + runner: Annotated[ + str | None, + typer.Option("--runner", help="Benchmark runner ID (e.g., ec2_c6id.8xlarge)"), + ] = None, + output: Annotated[ + Path | None, + typer.Option("--output", help="Optional path for compatibility JSONL output"), + ] = None, options: Annotated[list[str] | None, typer.Option("--opt", help="Engine or benchmark specific options")] = None, ) -> None: """Run benchmarks with specified configuration.""" - engines = parse_engines(engine) - formats = parse_formats(format) query_list = parse_queries(queries) exclude_list = parse_queries(exclude_queries) - bench_opts = {} - # Build options dict - if options: - for opt in options: - for s in opt.split(","): - k, v, *rest = s.split("=") - if rest: - raise RuntimeError("Options must be key-value pairs separated by =") - bench_opts[k] = v + strict_failures = targets_json is not None + + try: + bench_opts = parse_options(options) + if targets_json: + targets = parse_targets_json(targets_json) + warnings: list[str] = [] + else: + targets, warnings = targets_from_axes(engine, format) + except ValueError as exc: + console.print(f"[red]{exc}[/red]") + raise typer.Exit(1) from exc + + for warning in warnings: + console.print(f"[yellow]Warning: {warning}[/yellow]") + if not targets: + console.print("[red]No valid benchmark targets selected[/red]") + raise typer.Exit(1) config = RunConfig( benchmark=benchmark, - engines=engines, - formats=formats, + targets=targets, queries=query_list, exclude_queries=exclude_list, iterations=iterations, @@ -110,73 +245,84 @@ def run( track_memory=track_memory, ) - # Validate configuration - warnings = config.validate() - for warning in warnings: - console.print(f"[yellow]Warning: {warning}[/yellow]") + errors = config.validate() + if errors: + for error in errors: + console.print(f"[red]{error}[/red]") + raise typer.Exit(1) build_config = BuildConfig() builder = BenchmarkBuilder(config=build_config, verbose=verbose) store = ResultStore() - # Build binaries if needed if build: - binary_paths = builder.build(engines) + binary_paths = builder.build(config.backends) else: - binary_paths = {e: builder.get_binary_path(e) for e in engines} - # Check binaries exist - for eng, path in binary_paths.items(): + binary_paths = {backend: builder.get_binary_path(backend) for backend in config.backends} + for backend, path in binary_paths.items(): if not path.exists(): console.print(f"[red]Binary not found: {path}[/red]") console.print("Run with --build to build binaries first") raise typer.Exit(1) console.print(f"\n[bold]Running {benchmark.value} benchmark[/bold]") - console.print(f" Engines: {', '.join(e.value for e in engines)}") - console.print(f" Formats: {', '.join(f.value for f in formats)}") + console.print(f" Targets: {', '.join(map(str, config.targets))}") console.print(f" Iterations: {iterations}") if label: console.print(f" Label: {label}") console.print() - # Run benchmarks and store results - with store.create_run(config, build_config) as ctx: - for eng in engines: - # Filter formats to those supported by this engine - supported_formats = ENGINE_FORMATS.get(eng, []) - engine_formats = [f for f in formats if f in supported_formats] + backend_groups = group_targets_by_backend(config.targets) + soft_failures: list[str] = [] - if not engine_formats: - console.print(f"[yellow]Skipping {eng.value}: no supported formats[/yellow]") - continue - - executor = BenchmarkExecutor(binary_paths[eng], eng, verbose=verbose) - - try: - results = executor.run( - benchmark=benchmark, - formats=engine_formats, - queries=query_list, - exclude_queries=exclude_list, - iterations=iterations, - options=bench_opts, - track_memory=track_memory, - samply=samply, - sample_rate=sample_rate, - tracing=tracing, - on_result=ctx.write_raw_json, - ) - console.print(f"[green]{eng.value}: {len(results)} results[/green]") - except RuntimeError as e: - console.print(f"[red]{eng.value} failed: {e}[/red]") - - # Update metadata with binary paths - ctx.metadata.binaries = {e.value: str(p) for e, p in binary_paths.items()} - - console.print(f"\n[green]Results saved to run: {ctx.metadata.run_id}[/green]") + try: + with store.create_run(config, build_config) as ctx, open_results_output(output) as compatibility_file: + for backend, backend_targets in backend_groups.items(): + executor = BenchmarkExecutor(binary_paths[backend], backend, verbose=verbose) + backend_formats = [target.format for target in backend_targets] + + try: + results = executor.run( + benchmark=benchmark, + formats=backend_formats, + queries=query_list, + exclude_queries=exclude_list, + iterations=iterations, + options=bench_opts, + track_memory=track_memory, + samply=samply, + sample_rate=sample_rate, + tracing=tracing, + runner=runner, + on_result=lambda line, store_writer=ctx.write_raw_json, compatibility=compatibility_file: ( + write_result_line( + line, + store_writer, + compatibility, + ) + ), + ) + console.print(f"[green]{backend.value}: {len(results)} results[/green]") + except RuntimeError as exc: + ctx.metadata.partial = True + if strict_failures: + raise + console.print(f"[red]{backend.value} failed: {exc}[/red]") + soft_failures.append(str(exc)) + + ctx.metadata.binaries = {backend.value: str(path) for backend, path in binary_paths.items()} + except RuntimeError as exc: + console.print(f"[red]{exc}[/red]") + raise typer.Exit(1) from exc + + if soft_failures: + console.print(f"[yellow]Completed with {len(soft_failures)} backend failure(s)[/yellow]") + + metadata = ctx.metadata + console.print(f"\n[green]Results saved to run: {metadata.run_id}[/green]") # Show comparison table if we have multiple engine:format combinations - df = store.load_results(ctx.metadata.run_id) + df = store.load_results(metadata.run_id) if not df.empty: try: pivot = analyzer.compare_within_run(df) @@ -400,6 +546,10 @@ def show( console.print(f" Benchmark: {metadata.benchmark}") console.print(f" Engines: {', '.join(metadata.engines)}") console.print(f" Formats: {', '.join(metadata.formats)}") + if metadata.targets: + console.print( + " Targets: " + ", ".join(f"{target['engine']}:{target['format']}" for target in metadata.targets) + ) console.print(f" Iterations: {metadata.iterations}") console.print(f" Git commit: {metadata.git_commit[:8]}") console.print(f" Git branch: {metadata.git_branch}") @@ -439,6 +589,7 @@ def build( engines = parse_engines(engine) else: engines = list(Engine) + backends = backends_for_engines(engines) console.print(f"[bold]Building: {', '.join(e.value for e in engines)}[/bold]") console.print(f" Profile: {builder.config.profile}") @@ -446,10 +597,10 @@ def build( console.print() try: - paths = builder.build(engines) + paths = builder.build(backends) console.print("\n[green]Build complete:[/green]") - for eng, path in paths.items(): - console.print(f" {eng.value}: {path}") + for backend, path in paths.items(): + console.print(f" {backend.value}: {path}") except RuntimeError as e: console.print(f"[red]Build failed: {e}[/red]") raise typer.Exit(1) diff --git a/bench-orchestrator/bench_orchestrator/config.py b/bench-orchestrator/bench_orchestrator/config.py index 5a99f7262e8..bd81ce64fb2 100644 --- a/bench-orchestrator/bench_orchestrator/config.py +++ b/bench-orchestrator/bench_orchestrator/config.py @@ -3,13 +3,16 @@ """Configuration types and enums for benchmark orchestration.""" +import json +from collections.abc import Iterable from dataclasses import dataclass, field from enum import Enum from pathlib import Path +from typing import Any, TypeVar class Engine(Enum): - """Execution engines for benchmarks.""" + """Logical execution engines for benchmark results.""" DUCKDB = "duckdb" DATAFUSION = "datafusion" @@ -17,7 +20,7 @@ class Engine(Enum): @property def binary_name(self) -> str: - """Return the cargo binary name for this engine.""" + """Return the cargo binary name for this engine when used to execute benchmarks.""" return { Engine.DUCKDB: "duckdb-bench", Engine.DATAFUSION: "datafusion-bench", @@ -28,6 +31,7 @@ def binary_name(self) -> str: class Format(Enum): """Data formats for benchmarks.""" + ARROW = "arrow" PARQUET = "parquet" VORTEX = "vortex" VORTEX_COMPACT = "vortex-compact" @@ -43,13 +47,15 @@ class Benchmark(Enum): CLICKBENCH = "clickbench" FINEWEB = "fineweb" GHARCHIVE = "gh-archive" + POLARSIGNALS = "polarsignals" PUBLIC_BI = "public-bi" STATPOPGEN = "statpopgen" -# Engine to supported formats mapping +# Engine to supported formats mapping. ENGINE_FORMATS: dict[Engine, list[Format]] = { Engine.DATAFUSION: [ + Format.ARROW, Format.PARQUET, Format.VORTEX, Format.VORTEX_COMPACT, @@ -64,14 +70,155 @@ class Benchmark(Enum): Engine.LANCE: [Format.LANCE], } +T = TypeVar("T") + + +def _unique_preserve_order(values: Iterable[T]) -> list[T]: + seen: set[T] = set() + unique: list[T] = [] + for value in values: + if value in seen: + continue + seen.add(value) + unique.append(value) + return unique + + +@dataclass(frozen=True) +class BenchmarkTarget: + """An explicit engine/format benchmark target.""" + + engine: Engine + format: Format + + def normalized(self) -> "BenchmarkTarget": + """Normalize legacy lance targets onto the logical CI identity.""" + if self.engine == Engine.LANCE and self.format == Format.LANCE: + return BenchmarkTarget(engine=Engine.DATAFUSION, format=Format.LANCE) + return self + + @property + def backend(self) -> Engine: + """Return the engine whose benchmark binary should execute this target.""" + target = self.normalized() + if target.format == Format.LANCE: + return Engine.LANCE + if target.engine == Engine.DATAFUSION: + return Engine.DATAFUSION + if target.engine == Engine.DUCKDB: + return Engine.DUCKDB + raise ValueError(f"Unsupported benchmark target: {target}") + + def is_supported(self) -> bool: + """Return whether the logical engine supports this format.""" + target = self.normalized() + return target.format in ENGINE_FORMATS.get(target.engine, []) + + def to_dict(self) -> dict[str, str]: + target = self.normalized() + return {"engine": target.engine.value, "format": target.format.value} + + @classmethod + def from_dict(cls, data: dict[str, Any]) -> "BenchmarkTarget": + if not isinstance(data, dict): + raise ValueError("Targets must be JSON objects with engine and format fields") + try: + target = cls( + engine=Engine(data["engine"]), + format=Format(data["format"]), + ) + except KeyError as exc: + raise ValueError("Targets must include engine and format fields") from exc + except ValueError as exc: + raise ValueError(f"Invalid benchmark target: {data}") from exc + return target.normalized() + + def __str__(self) -> str: + target = self.normalized() + return f"{target.engine.value}:{target.format.value}" + + +def parse_targets_json(value: str) -> list[BenchmarkTarget]: + """Parse a JSON array of explicit benchmark targets.""" + try: + raw_targets = json.loads(value) + except json.JSONDecodeError as exc: + raise ValueError("Targets JSON must be a valid JSON array") from exc + + if not isinstance(raw_targets, list): + raise ValueError("Targets JSON must be an array") + + return _unique_preserve_order(BenchmarkTarget.from_dict(item) for item in raw_targets) + + +def parse_formats_json(value: str) -> list[Format]: + """Parse a JSON array of format names.""" + try: + raw_formats = json.loads(value) + except json.JSONDecodeError as exc: + raise ValueError("Formats JSON must be a valid JSON array") from exc + + if not isinstance(raw_formats, list): + raise ValueError("Formats JSON must be an array") + + formats: list[Format] = [] + for item in raw_formats: + if not isinstance(item, str): + raise ValueError("Formats JSON entries must be strings") + try: + formats.append(Format(item)) + except ValueError as exc: + raise ValueError(f"Invalid format: {item}") from exc + return _unique_preserve_order(formats) + + +def resolve_axis_targets( + engines: Iterable[Engine], formats: Iterable[Format] +) -> tuple[list[BenchmarkTarget], list[str]]: + """Expand engine/format axes into supported explicit targets.""" + warnings: list[str] = [] + targets: list[BenchmarkTarget] = [] + + for engine in engines: + for fmt in formats: + target = BenchmarkTarget(engine=engine, format=fmt).normalized() + if not target.is_supported(): + warnings.append(f"Format {fmt.value} is not supported by engine {engine.value}") + continue + targets.append(target) + + return _unique_preserve_order(targets), warnings + + +def group_targets_by_backend(targets: Iterable[BenchmarkTarget]) -> dict[Engine, list[BenchmarkTarget]]: + """Group logical benchmark targets by the backend binary required to run them.""" + groups: dict[Engine, list[BenchmarkTarget]] = {} + for target in _unique_preserve_order(target.normalized() for target in targets): + groups.setdefault(target.backend, []).append(target) + return groups + + +def validate_targets(targets: Iterable[BenchmarkTarget], options: dict[str, str]) -> list[str]: + """Validate explicit targets against benchmark runner constraints.""" + errors: list[str] = [] + + normalized_targets = [target.normalized() for target in targets] + for target in normalized_targets: + if not target.is_supported(): + errors.append(f"Format {target.format.value} is not supported by engine {target.engine.value}") + + if options.get("remote-data-dir") and any(target.format == Format.LANCE for target in normalized_targets): + errors.append("Lance format is not supported for remote storage benchmarks.") + + return _unique_preserve_order(errors) + @dataclass class RunConfig: """Configuration for a benchmark run.""" benchmark: Benchmark - engines: list[Engine] - formats: list[Format] + targets: list[BenchmarkTarget] queries: list[int] | None = None exclude_queries: list[int] | None = None iterations: int = 5 @@ -79,15 +226,21 @@ class RunConfig: options: dict[str, str] = field(default_factory=dict) track_memory: bool = False + @property + def engines(self) -> list[Engine]: + return _unique_preserve_order(target.engine for target in self.targets) + + @property + def formats(self) -> list[Format]: + return _unique_preserve_order(target.format for target in self.targets) + + @property + def backends(self) -> list[Engine]: + return _unique_preserve_order(target.backend for target in self.targets) + def validate(self) -> list[str]: - """Validate the configuration and return any warnings.""" - warnings = [] - for engine in self.engines: - supported = ENGINE_FORMATS.get(engine, []) - for fmt in self.formats: - if fmt not in supported: - warnings.append(f"Format {fmt.value} is not supported by engine {engine.value}") - return warnings + """Validate the configuration and return any errors.""" + return validate_targets(self.targets, self.options) @dataclass @@ -96,6 +249,7 @@ class BuildConfig: profile: str = "release_debug" rustflags: str = "-C target-cpu=native -C force-frame-pointers=yes" + features: tuple[str, ...] = ("unstable_encodings",) def get_workspace_root() -> Path: diff --git a/bench-orchestrator/bench_orchestrator/runner/builder.py b/bench-orchestrator/bench_orchestrator/runner/builder.py index 55e817c9eb2..070100f1660 100644 --- a/bench-orchestrator/bench_orchestrator/runner/builder.py +++ b/bench-orchestrator/bench_orchestrator/runner/builder.py @@ -29,21 +29,24 @@ def __init__( self.config = config or BuildConfig() self.verbose = verbose - def get_binary_path(self, engine: Engine) -> Path: + def get_binary_path(self, backend: Engine) -> Path: """Get path to built binary.""" - binary_name = engine.binary_name + binary_name = backend.binary_name return self.workspace_root / "target" / self.config.profile / binary_name - def build(self, engines: list[Engine]) -> dict[Engine, Path]: + def get_data_generator_path(self) -> Path: + """Get path to the built benchmark data generator binary.""" + return self.workspace_root / "target" / self.config.profile / "data-gen" + + def build(self, backends: list[Engine]) -> dict[Engine, Path]: """Build binaries for specified engines, return paths.""" - results = {} + results: dict[Engine, Path] = {} - # Build each binary env = os.environ.copy() env["RUSTFLAGS"] = self.config.rustflags - for engine in engines: - binary_name = engine.binary_name + for backend in backends: + binary_name = backend.binary_name console.print(f"[blue]Building {binary_name}...[/blue]") cmd = [ @@ -54,6 +57,8 @@ def build(self, engines: list[Engine]) -> dict[Engine, Path]: "--profile", self.config.profile, ] + if self.config.features: + cmd.extend(["--features", ",".join(self.config.features)]) if self.verbose: console.print(f"[dim]$ RUSTFLAGS='{self.config.rustflags}' {' '.join(cmd)}[/dim]") @@ -65,7 +70,7 @@ def build(self, engines: list[Engine]) -> dict[Engine, Path]: env=env, check=True, ) - results[engine] = self.get_binary_path(engine) + results[backend] = self.get_binary_path(backend) console.print(f"[green]Built {binary_name}[/green]") except subprocess.CalledProcessError as e: console.print(f"[red]Failed to build {binary_name}: {e}[/red]") diff --git a/bench-orchestrator/bench_orchestrator/runner/executor.py b/bench-orchestrator/bench_orchestrator/runner/executor.py index a80080d8d5a..b895afdc2e1 100644 --- a/bench-orchestrator/bench_orchestrator/runner/executor.py +++ b/bench-orchestrator/bench_orchestrator/runner/executor.py @@ -3,7 +3,9 @@ """Benchmark binary execution.""" +import selectors import subprocess +from collections import deque from collections.abc import Callable from pathlib import Path from typing import final @@ -20,12 +22,12 @@ class BenchmarkExecutor: """Executes benchmark binaries and captures output.""" - def __init__(self, binary_path: Path, engine: Engine, verbose: bool = False): + def __init__(self, binary_path: Path, backend: Engine, verbose: bool = False): self.binary_path = binary_path - self.engine = engine + self.backend = backend self.verbose = verbose - def run( + def build_command( self, benchmark: Benchmark, formats: list[Format], @@ -37,24 +39,9 @@ def run( samply: bool = False, sample_rate: int | None = None, tracing: bool = False, - on_result: Callable[[str], None] | None = None, + runner: str | None = None, ) -> list[str]: - """ - Run benchmark and return results as JSON lines. - - Args: - benchmark: The benchmark suite to run - formats: Data formats to benchmark - queries: Specific queries to run (None for all) - exclude_queries: Queries to skip - iterations: Number of runs per query - options: Additional options (e.g., scale_factor) - track_memory: Enable memory tracking - on_result: Callback for each result line (for streaming) - - Returns: - List of JSON lines from the benchmark output - """ + """Build the command used to execute a benchmark binary.""" cmd = [ str(self.binary_path), benchmark.value, @@ -62,11 +49,14 @@ def run( "gh-json", "--iterations", str(iterations), - "--formats", - ",".join(f.value for f in formats), "--hide-progress-bar", ] + if self.backend in {Engine.DATAFUSION, Engine.DUCKDB}: + cmd.extend(["--formats", ",".join(fmt.value for fmt in formats)]) + if self.backend == Engine.DUCKDB: + cmd.append("--delete-duckdb-database") + if queries: cmd.extend(["--queries", ",".join(map(str, queries))]) if exclude_queries: @@ -75,26 +65,76 @@ def run( cmd.append("--track-memory") if tracing: cmd.append("--tracing") + if runner: + cmd.extend(["--runner", runner]) if options: - for k, v in options.items(): - cmd.extend(["--opt", f"{k}={v}"]) + for key, value in options.items(): + cmd.extend(["--opt", f"{key}={value}"]) if samply: cmd = ["--"] + cmd cmd_prefix = ["samply", "record"] if sample_rate: - cmd = cmd_prefix + ["--rate", sample_rate] + cmd + cmd = cmd_prefix + ["--rate", str(sample_rate)] + cmd else: cmd = cmd_prefix + cmd - if samply and self.engine == Engine.DUCKDB: - # Re-use the same DuckDB instance across runs make samply output readable - cmd += ["--reuse"] + if samply and self.backend == Engine.DUCKDB: + # Re-use the same DuckDB instance across runs to keep samply output readable. + cmd.append("--reuse") + + return cmd + + def run( + self, + benchmark: Benchmark, + formats: list[Format], + queries: list[int] | None = None, + exclude_queries: list[int] | None = None, + iterations: int = 5, + options: dict[str, str] | None = None, + track_memory: bool = False, + samply: bool = False, + sample_rate: int | None = None, + tracing: bool = False, + runner: str | None = None, + on_result: Callable[[str], None] | None = None, + ) -> list[str]: + """ + Run benchmark and return results as JSON lines. + + Args: + benchmark: The benchmark suite to run + formats: Data formats to benchmark + queries: Specific queries to run (None for all) + exclude_queries: Queries to skip + iterations: Number of runs per query + options: Additional options (e.g., scale_factor) + track_memory: Enable memory tracking + on_result: Callback for each result line (for streaming) + + Returns: + List of JSON lines from the benchmark output + """ + cmd = self.build_command( + benchmark=benchmark, + formats=formats, + queries=queries, + exclude_queries=exclude_queries, + iterations=iterations, + options=options, + track_memory=track_memory, + samply=samply, + sample_rate=sample_rate, + tracing=tracing, + runner=runner, + ) if self.verbose: console.print(f"[dim]$ {' '.join(cmd)}[/dim]") - results = [] + results: list[str] = [] + diagnostic_lines: deque[str] = deque(maxlen=200) with Progress( SpinnerColumn(), @@ -102,28 +142,51 @@ def run( console=console, transient=True, ) as progress: - _task = progress.add_task(f"Running {self.engine.value} {benchmark.value}...", total=None) + _task = progress.add_task(f"Running {self.backend.value} {benchmark.value}...", total=None) + # Merge stderr into stdout so verbose benchmark logs cannot fill a separate pipe and + # block the child process before it emits JSON results. process = subprocess.Popen( cmd, stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, text=True, + bufsize=1, ) - for line in iter(process.stdout.readline, ""): - line = line.strip() - if line: - results.append(line) - if on_result: - on_result(line) + assert process.stdout is not None + selector = selectors.DefaultSelector() + selector.register(process.stdout, selectors.EVENT_READ) + + try: + while selector.get_map(): + for key, _mask in selector.select(timeout=0.1): + line = key.fileobj.readline() + if line == "": + selector.unregister(key.fileobj) + continue + + line = line.rstrip() + if not line: + continue + + if line.startswith("{"): + results.append(line) + if on_result: + on_result(line) + else: + diagnostic_lines.append(line) + console.print(line, markup=False) + finally: + selector.close() ret_code = process.wait() if ret_code != 0: - stderr = process.stderr.read() if process.stderr else "" console.print(f"[red]Benchmark failed with code {process.returncode}[/red]") - if stderr: - console.print(f"[red]{stderr}[/red]") - raise RuntimeError(f"Benchmark {self.engine.value} {benchmark.value} failed: {stderr}") + diagnostics = "\n".join(diagnostic_lines) + if diagnostics: + console.print(f"[red]{diagnostics}[/red]") + raise RuntimeError(f"Benchmark {self.backend.value} {benchmark.value} failed: {diagnostics}") return results diff --git a/bench-orchestrator/bench_orchestrator/storage/schema.py b/bench-orchestrator/bench_orchestrator/storage/schema.py index f9bd9d19cff..a43c16382f4 100644 --- a/bench-orchestrator/bench_orchestrator/storage/schema.py +++ b/bench-orchestrator/bench_orchestrator/storage/schema.py @@ -93,6 +93,7 @@ class RunMetadata: benchmark: str engines: list[str] formats: list[str] + targets: list[dict[str, str]] iterations: int git_commit: str git_branch: str @@ -116,6 +117,7 @@ def to_dict(self) -> dict[str, Any]: "dataset_config": self.dataset_config, "engines": self.engines, "formats": self.formats, + "targets": self.targets, "queries": self.queries, "iterations": self.iterations, "git_commit": self.git_commit, @@ -139,6 +141,7 @@ def from_dict(cls, data: dict[str, Any]) -> "RunMetadata": dataset_config=data.get("dataset_config", {}), engines=data["engines"], formats=data["formats"], + targets=data.get("targets", []), queries=data.get("queries", []), iterations=data["iterations"], git_commit=data["git_commit"], diff --git a/bench-orchestrator/bench_orchestrator/storage/store.py b/bench-orchestrator/bench_orchestrator/storage/store.py index 1703af1ad6f..54c15593a41 100644 --- a/bench-orchestrator/bench_orchestrator/storage/store.py +++ b/bench-orchestrator/bench_orchestrator/storage/store.py @@ -142,6 +142,7 @@ def create_run(self, config: RunConfig, build_config: BuildConfig) -> Iterator[R dataset_config=config.options, engines=[e.value for e in config.engines], formats=[f.value for f in config.formats], + targets=[target.to_dict() for target in config.targets], queries=config.queries or [], iterations=config.iterations, git_commit=git_commit, diff --git a/bench-orchestrator/tests/test_cli.py b/bench-orchestrator/tests/test_cli.py new file mode 100644 index 00000000000..ffb2e2b3ad9 --- /dev/null +++ b/bench-orchestrator/tests/test_cli.py @@ -0,0 +1,107 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright the Vortex contributors + +import json + +from bench_orchestrator import cli as cli_module +from bench_orchestrator.runner.executor import BenchmarkExecutor +from bench_orchestrator.storage.store import ResultStore +from typer.testing import CliRunner + +runner = CliRunner() + + +def test_prepare_data_uses_formats_json(tmp_path, monkeypatch) -> None: + data_gen = tmp_path / "data-gen" + data_gen.write_text("", encoding="utf-8") + + captured: dict[str, list[str]] = {} + + def fake_run(cmd: list[str], check: bool) -> None: + assert check is True + captured["cmd"] = cmd + + monkeypatch.setattr(cli_module.BenchmarkBuilder, "get_data_generator_path", lambda self: data_gen) + monkeypatch.setattr(cli_module.subprocess, "run", fake_run) + + result = runner.invoke( + cli_module.app, + [ + "prepare-data", + "tpch", + "--formats-json", + '["parquet","vortex"]', + "--opt", + "scale-factor=1.0", + ], + ) + + assert result.exit_code == 0 + assert captured["cmd"] == [ + str(data_gen), + "tpch", + "--formats", + "parquet,vortex", + "--opt", + "scale-factor=1.0", + ] + + +def test_run_writes_compatibility_results_output(tmp_path, monkeypatch) -> None: + run_store = ResultStore(base_dir=tmp_path / "runs") + output_path = tmp_path / "artifacts" / "results.json" + binary_path = tmp_path / "datafusion-bench" + binary_path.write_text("", encoding="utf-8") + + sample_line = json.dumps( + { + "name": "tpch_q1/datafusion:parquet", + "storage": "nvme", + "dataset": {"scale_factor": "1.0"}, + "unit": "ns", + "value": 100, + "all_runtimes": [95, 100, 105], + "target": {"engine": "datafusion", "format": "parquet"}, + "commit_id": "deadbeef", + "env_triple": { + "architecture": "x86_64", + "operating_system": "linux", + "environment": "gnu", + }, + } + ) + + monkeypatch.setattr(cli_module, "ResultStore", lambda: run_store) + monkeypatch.setattr(cli_module.BenchmarkBuilder, "get_binary_path", lambda self, backend: binary_path) + + def fake_run(self, **kwargs): + kwargs["on_result"](sample_line) + return [sample_line] + + monkeypatch.setattr(BenchmarkExecutor, "run", fake_run) + + result = runner.invoke( + cli_module.app, + [ + "run", + "tpch", + "--targets-json", + '[{"engine":"datafusion","format":"parquet"}]', + "--no-build", + "--output", + str(output_path), + ], + ) + + assert result.exit_code == 0 + assert output_path.read_text(encoding="utf-8") == sample_line + "\n" + + run_dirs = [path for path in (tmp_path / "runs").iterdir() if path.is_dir() and path.name != "latest"] + assert len(run_dirs) == 1 + + results_path = run_dirs[0] / "results.jsonl" + assert results_path.read_text(encoding="utf-8") == sample_line + "\n" + + metadata = json.loads((run_dirs[0] / "metadata.json").read_text(encoding="utf-8")) + assert metadata["targets"] == [{"engine": "datafusion", "format": "parquet"}] + assert metadata["binaries"] == {"datafusion": str(binary_path)} diff --git a/bench-orchestrator/tests/test_config.py b/bench-orchestrator/tests/test_config.py new file mode 100644 index 00000000000..c7e2d6bb291 --- /dev/null +++ b/bench-orchestrator/tests/test_config.py @@ -0,0 +1,65 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright the Vortex contributors + +from bench_orchestrator.config import ( + BenchmarkTarget, + Engine, + Format, + group_targets_by_backend, + parse_formats_json, + parse_targets_json, + resolve_axis_targets, + validate_targets, +) + + +def test_parse_targets_json_normalizes_and_dedupes_lance_targets() -> None: + targets = parse_targets_json('[{"engine":"lance","format":"lance"},{"engine":"datafusion","format":"lance"}]') + + assert targets == [BenchmarkTarget(engine=Engine.DATAFUSION, format=Format.LANCE)] + + +def test_parse_formats_json_accepts_ci_format_arrays() -> None: + formats = parse_formats_json('["parquet","vortex","duckdb"]') + + assert formats == [Format.PARQUET, Format.VORTEX, Format.DUCKDB] + + +def test_resolve_axis_targets_filters_unsupported_combinations() -> None: + targets, warnings = resolve_axis_targets( + [Engine.DATAFUSION, Engine.DUCKDB], + [Format.ARROW, Format.PARQUET], + ) + + assert targets == [ + BenchmarkTarget(engine=Engine.DATAFUSION, format=Format.ARROW), + BenchmarkTarget(engine=Engine.DATAFUSION, format=Format.PARQUET), + BenchmarkTarget(engine=Engine.DUCKDB, format=Format.PARQUET), + ] + assert warnings == ["Format arrow is not supported by engine duckdb"] + + +def test_validate_targets_rejects_remote_lance() -> None: + errors = validate_targets( + [BenchmarkTarget(engine=Engine.DATAFUSION, format=Format.LANCE)], + {"remote-data-dir": "s3://benchmarks/tpch/"}, + ) + + assert errors == ["Lance format is not supported for remote storage benchmarks."] + + +def test_group_targets_by_backend_routes_lance_to_lance_binary() -> None: + groups = group_targets_by_backend( + [ + BenchmarkTarget(engine=Engine.DATAFUSION, format=Format.PARQUET), + BenchmarkTarget(engine=Engine.DATAFUSION, format=Format.LANCE), + BenchmarkTarget(engine=Engine.DUCKDB, format=Format.VORTEX), + ] + ) + + assert list(groups) == [ + Engine.DATAFUSION, + Engine.LANCE, + Engine.DUCKDB, + ] + assert groups[Engine.LANCE] == [BenchmarkTarget(engine=Engine.DATAFUSION, format=Format.LANCE)] diff --git a/bench-orchestrator/tests/test_executor.py b/bench-orchestrator/tests/test_executor.py new file mode 100644 index 00000000000..ade3dde1a67 --- /dev/null +++ b/bench-orchestrator/tests/test_executor.py @@ -0,0 +1,78 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright the Vortex contributors + +import sys +import textwrap +from pathlib import Path + +from bench_orchestrator.config import Benchmark, Engine, Format +from bench_orchestrator.runner.executor import BenchmarkExecutor + + +def test_build_command_adds_duckdb_cleanup_flag() -> None: + executor = BenchmarkExecutor(Path("/tmp/duckdb-bench"), Engine.DUCKDB) + + cmd = executor.build_command( + benchmark=Benchmark.TPCH, + formats=[Format.PARQUET, Format.VORTEX], + iterations=7, + options={"scale-factor": "1.0"}, + ) + + assert cmd[:5] == [ + "/tmp/duckdb-bench", + "tpch", + "--display-format", + "gh-json", + "--iterations", + ] + assert "--formats" in cmd + assert "parquet,vortex" in cmd + assert "--delete-duckdb-database" in cmd + assert "--opt" in cmd + assert "scale-factor=1.0" in cmd + + +def test_build_command_omits_formats_for_lance_backend() -> None: + executor = BenchmarkExecutor(Path("/tmp/lance-bench"), Engine.LANCE) + + cmd = executor.build_command( + benchmark=Benchmark.TPCH, + formats=[Format.LANCE], + queries=[1, 3], + ) + + assert cmd[0] == "/tmp/lance-bench" + assert "--formats" not in cmd + assert "--queries" in cmd + assert "1,3" in cmd + + +def test_run_streams_logs_without_counting_them(tmp_path: Path) -> None: + script = tmp_path / "fake-bench.py" + script.write_text( + textwrap.dedent( + f"""\ + #!{sys.executable} + import sys + + print("preparing duckdb tables", file=sys.stderr, flush=True) + print('{{"engine":"duckdb","format":"parquet","query":0}}', flush=True) + print("finished query 0", file=sys.stderr, flush=True) + """ + ) + ) + script.chmod(0o755) + + executor = BenchmarkExecutor(script, Engine.DUCKDB) + streamed: list[str] = [] + + results = executor.run( + benchmark=Benchmark.CLICKBENCH, + formats=[Format.PARQUET], + iterations=1, + on_result=streamed.append, + ) + + assert results == ['{"engine":"duckdb","format":"parquet","query":0}'] + assert streamed == results diff --git a/benchmarks-website/package-lock.json b/benchmarks-website/package-lock.json index 9fbe9f01769..72b68db620d 100644 --- a/benchmarks-website/package-lock.json +++ b/benchmarks-website/package-lock.json @@ -8,808 +8,58 @@ "name": "vortex-benchmarks-website", "version": "2.0.0", "dependencies": { - "chart.js": "^4.4.4", - "chartjs-plugin-zoom": "^2.0.1", + "chart.js": "^4.5.1", + "chartjs-plugin-zoom": "^2.2.0", "downsample": "^1.4.0", "hammerjs": "^2.0.8", - "lucide-react": "^0.563.0", - "react": "^18.3.1", - "react-chartjs-2": "^5.2.0", - "react-dom": "^18.3.1" + "lucide-react": "^1.11.0", + "react": "^19.2.5", + "react-chartjs-2": "^5.3.1", + "react-dom": "^19.2.5" }, "devDependencies": { - "@types/react": "^18.3.3", - "@types/react-dom": "^18.3.0", - "@vitejs/plugin-react": "^4.3.1", - "concurrently": "^8.2.2", - "vite": "^6.0.0" + "@types/react": "^19.2.14", + "@types/react-dom": "^19.2.3", + "@vitejs/plugin-react": "^6.0.1", + "concurrently": "^9.2.1", + "vite": "^8.0.10" }, "engines": { "node": ">=18.0.0" } }, - "node_modules/@babel/code-frame": { - "version": "7.29.0", - "resolved": "https://registry.npmjs.org/@babel/code-frame/-/code-frame-7.29.0.tgz", - "integrity": "sha512-9NhCeYjq9+3uxgdtp20LSiJXJvN0FeCtNGpJxuMFZ1Kv3cWUNb6DOhJwUvcVCzKGR66cw4njwM6hrJLqgOwbcw==", - "dev": true, - "license": "MIT", - "dependencies": { - "@babel/helper-validator-identifier": "^7.28.5", - "js-tokens": "^4.0.0", - "picocolors": "^1.1.1" - }, - "engines": { - "node": ">=6.9.0" - } - }, - "node_modules/@babel/compat-data": { - "version": "7.29.0", - "resolved": "https://registry.npmjs.org/@babel/compat-data/-/compat-data-7.29.0.tgz", - "integrity": "sha512-T1NCJqT/j9+cn8fvkt7jtwbLBfLC/1y1c7NtCeXFRgzGTsafi68MRv8yzkYSapBnFA6L3U2VSc02ciDzoAJhJg==", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=6.9.0" - } - }, - "node_modules/@babel/core": { - "version": "7.29.0", - "resolved": "https://registry.npmjs.org/@babel/core/-/core-7.29.0.tgz", - "integrity": "sha512-CGOfOJqWjg2qW/Mb6zNsDm+u5vFQ8DxXfbM09z69p5Z6+mE1ikP2jUXw+j42Pf1XTYED2Rni5f95npYeuwMDQA==", - "dev": true, - "license": "MIT", - "dependencies": { - "@babel/code-frame": "^7.29.0", - "@babel/generator": "^7.29.0", - "@babel/helper-compilation-targets": "^7.28.6", - "@babel/helper-module-transforms": "^7.28.6", - "@babel/helpers": "^7.28.6", - "@babel/parser": "^7.29.0", - "@babel/template": "^7.28.6", - "@babel/traverse": "^7.29.0", - "@babel/types": "^7.29.0", - "@jridgewell/remapping": "^2.3.5", - "convert-source-map": "^2.0.0", - "debug": "^4.1.0", - "gensync": "^1.0.0-beta.2", - "json5": "^2.2.3", - "semver": "^6.3.1" - }, - "engines": { - "node": ">=6.9.0" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/babel" - } - }, - "node_modules/@babel/generator": { - "version": "7.29.1", - "resolved": "https://registry.npmjs.org/@babel/generator/-/generator-7.29.1.tgz", - "integrity": "sha512-qsaF+9Qcm2Qv8SRIMMscAvG4O3lJ0F1GuMo5HR/Bp02LopNgnZBC/EkbevHFeGs4ls/oPz9v+Bsmzbkbe+0dUw==", - "dev": true, - "license": "MIT", - "dependencies": { - "@babel/parser": "^7.29.0", - "@babel/types": "^7.29.0", - "@jridgewell/gen-mapping": "^0.3.12", - "@jridgewell/trace-mapping": "^0.3.28", - "jsesc": "^3.0.2" - }, - "engines": { - "node": ">=6.9.0" - } - }, - "node_modules/@babel/helper-compilation-targets": { - "version": "7.28.6", - "resolved": "https://registry.npmjs.org/@babel/helper-compilation-targets/-/helper-compilation-targets-7.28.6.tgz", - "integrity": "sha512-JYtls3hqi15fcx5GaSNL7SCTJ2MNmjrkHXg4FSpOA/grxK8KwyZ5bubHsCq8FXCkua6xhuaaBit+3b7+VZRfcA==", - "dev": true, - "license": "MIT", - "dependencies": { - "@babel/compat-data": "^7.28.6", - "@babel/helper-validator-option": "^7.27.1", - "browserslist": "^4.24.0", - "lru-cache": "^5.1.1", - "semver": "^6.3.1" - }, - "engines": { - "node": ">=6.9.0" - } - }, - "node_modules/@babel/helper-globals": { - "version": "7.28.0", - "resolved": "https://registry.npmjs.org/@babel/helper-globals/-/helper-globals-7.28.0.tgz", - "integrity": "sha512-+W6cISkXFa1jXsDEdYA8HeevQT/FULhxzR99pxphltZcVaugps53THCeiWA8SguxxpSp3gKPiuYfSWopkLQ4hw==", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=6.9.0" - } - }, - "node_modules/@babel/helper-module-imports": { - "version": "7.28.6", - "resolved": "https://registry.npmjs.org/@babel/helper-module-imports/-/helper-module-imports-7.28.6.tgz", - "integrity": "sha512-l5XkZK7r7wa9LucGw9LwZyyCUscb4x37JWTPz7swwFE/0FMQAGpiWUZn8u9DzkSBWEcK25jmvubfpw2dnAMdbw==", - "dev": true, - "license": "MIT", - "dependencies": { - "@babel/traverse": "^7.28.6", - "@babel/types": "^7.28.6" - }, - "engines": { - "node": ">=6.9.0" - } - }, - "node_modules/@babel/helper-module-transforms": { - "version": "7.28.6", - "resolved": "https://registry.npmjs.org/@babel/helper-module-transforms/-/helper-module-transforms-7.28.6.tgz", - "integrity": "sha512-67oXFAYr2cDLDVGLXTEABjdBJZ6drElUSI7WKp70NrpyISso3plG9SAGEF6y7zbha/wOzUByWWTJvEDVNIUGcA==", - "dev": true, - "license": "MIT", - "dependencies": { - "@babel/helper-module-imports": "^7.28.6", - "@babel/helper-validator-identifier": "^7.28.5", - "@babel/traverse": "^7.28.6" - }, - "engines": { - "node": ">=6.9.0" - }, - "peerDependencies": { - "@babel/core": "^7.0.0" - } - }, - "node_modules/@babel/helper-plugin-utils": { - "version": "7.28.6", - "resolved": "https://registry.npmjs.org/@babel/helper-plugin-utils/-/helper-plugin-utils-7.28.6.tgz", - "integrity": "sha512-S9gzZ/bz83GRysI7gAD4wPT/AI3uCnY+9xn+Mx/KPs2JwHJIz1W8PZkg2cqyt3RNOBM8ejcXhV6y8Og7ly/Dug==", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=6.9.0" - } - }, - "node_modules/@babel/helper-string-parser": { - "version": "7.27.1", - "resolved": "https://registry.npmjs.org/@babel/helper-string-parser/-/helper-string-parser-7.27.1.tgz", - "integrity": "sha512-qMlSxKbpRlAridDExk92nSobyDdpPijUq2DW6oDnUqd0iOGxmQjyqhMIihI9+zv4LPyZdRje2cavWPbCbWm3eA==", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=6.9.0" - } - }, - "node_modules/@babel/helper-validator-identifier": { - "version": "7.28.5", - "resolved": "https://registry.npmjs.org/@babel/helper-validator-identifier/-/helper-validator-identifier-7.28.5.tgz", - "integrity": "sha512-qSs4ifwzKJSV39ucNjsvc6WVHs6b7S03sOh2OcHF9UHfVPqWWALUsNUVzhSBiItjRZoLHx7nIarVjqKVusUZ1Q==", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=6.9.0" - } - }, - "node_modules/@babel/helper-validator-option": { - "version": "7.27.1", - "resolved": "https://registry.npmjs.org/@babel/helper-validator-option/-/helper-validator-option-7.27.1.tgz", - "integrity": "sha512-YvjJow9FxbhFFKDSuFnVCe2WxXk1zWc22fFePVNEaWJEu8IrZVlda6N0uHwzZrUM1il7NC9Mlp4MaJYbYd9JSg==", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=6.9.0" - } - }, - "node_modules/@babel/helpers": { - "version": "7.29.2", - "resolved": "https://registry.npmjs.org/@babel/helpers/-/helpers-7.29.2.tgz", - "integrity": "sha512-HoGuUs4sCZNezVEKdVcwqmZN8GoHirLUcLaYVNBK2J0DadGtdcqgr3BCbvH8+XUo4NGjNl3VOtSjEKNzqfFgKw==", - "dev": true, - "license": "MIT", - "dependencies": { - "@babel/template": "^7.28.6", - "@babel/types": "^7.29.0" - }, - "engines": { - "node": ">=6.9.0" - } - }, - "node_modules/@babel/parser": { - "version": "7.29.2", - "resolved": "https://registry.npmjs.org/@babel/parser/-/parser-7.29.2.tgz", - "integrity": "sha512-4GgRzy/+fsBa72/RZVJmGKPmZu9Byn8o4MoLpmNe1m8ZfYnz5emHLQz3U4gLud6Zwl0RZIcgiLD7Uq7ySFuDLA==", - "dev": true, - "license": "MIT", - "dependencies": { - "@babel/types": "^7.29.0" - }, - "bin": { - "parser": "bin/babel-parser.js" - }, - "engines": { - "node": ">=6.0.0" - } - }, - "node_modules/@babel/plugin-transform-react-jsx-self": { - "version": "7.27.1", - "resolved": "https://registry.npmjs.org/@babel/plugin-transform-react-jsx-self/-/plugin-transform-react-jsx-self-7.27.1.tgz", - "integrity": "sha512-6UzkCs+ejGdZ5mFFC/OCUrv028ab2fp1znZmCZjAOBKiBK2jXD1O+BPSfX8X2qjJ75fZBMSnQn3Rq2mrBJK2mw==", - "dev": true, - "license": "MIT", - "dependencies": { - "@babel/helper-plugin-utils": "^7.27.1" - }, - "engines": { - "node": ">=6.9.0" - }, - "peerDependencies": { - "@babel/core": "^7.0.0-0" - } - }, - "node_modules/@babel/plugin-transform-react-jsx-source": { - "version": "7.27.1", - "resolved": "https://registry.npmjs.org/@babel/plugin-transform-react-jsx-source/-/plugin-transform-react-jsx-source-7.27.1.tgz", - "integrity": "sha512-zbwoTsBruTeKB9hSq73ha66iFeJHuaFkUbwvqElnygoNbj/jHRsSeokowZFN3CZ64IvEqcmmkVe89OPXc7ldAw==", - "dev": true, - "license": "MIT", - "dependencies": { - "@babel/helper-plugin-utils": "^7.27.1" - }, - "engines": { - "node": ">=6.9.0" - }, - "peerDependencies": { - "@babel/core": "^7.0.0-0" - } - }, - "node_modules/@babel/runtime": { - "version": "7.29.2", - "resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.29.2.tgz", - "integrity": "sha512-JiDShH45zKHWyGe4ZNVRrCjBz8Nh9TMmZG1kh4QTK8hCBTWBi8Da+i7s1fJw7/lYpM4ccepSNfqzZ/QvABBi5g==", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=6.9.0" - } - }, - "node_modules/@babel/template": { - "version": "7.28.6", - "resolved": "https://registry.npmjs.org/@babel/template/-/template-7.28.6.tgz", - "integrity": "sha512-YA6Ma2KsCdGb+WC6UpBVFJGXL58MDA6oyONbjyF/+5sBgxY/dwkhLogbMT2GXXyU84/IhRw/2D1Os1B/giz+BQ==", - "dev": true, - "license": "MIT", - "dependencies": { - "@babel/code-frame": "^7.28.6", - "@babel/parser": "^7.28.6", - "@babel/types": "^7.28.6" - }, - "engines": { - "node": ">=6.9.0" - } - }, - "node_modules/@babel/traverse": { - "version": "7.29.0", - "resolved": "https://registry.npmjs.org/@babel/traverse/-/traverse-7.29.0.tgz", - "integrity": "sha512-4HPiQr0X7+waHfyXPZpWPfWL/J7dcN1mx9gL6WdQVMbPnF3+ZhSMs8tCxN7oHddJE9fhNE7+lxdnlyemKfJRuA==", - "dev": true, - "license": "MIT", - "dependencies": { - "@babel/code-frame": "^7.29.0", - "@babel/generator": "^7.29.0", - "@babel/helper-globals": "^7.28.0", - "@babel/parser": "^7.29.0", - "@babel/template": "^7.28.6", - "@babel/types": "^7.29.0", - "debug": "^4.3.1" - }, - "engines": { - "node": ">=6.9.0" - } - }, - "node_modules/@babel/types": { - "version": "7.29.0", - "resolved": "https://registry.npmjs.org/@babel/types/-/types-7.29.0.tgz", - "integrity": "sha512-LwdZHpScM4Qz8Xw2iKSzS+cfglZzJGvofQICy7W7v4caru4EaAmyUuO6BGrbyQ2mYV11W0U8j5mBhd14dd3B0A==", - "dev": true, - "license": "MIT", - "dependencies": { - "@babel/helper-string-parser": "^7.27.1", - "@babel/helper-validator-identifier": "^7.28.5" - }, - "engines": { - "node": ">=6.9.0" - } - }, - "node_modules/@esbuild/aix-ppc64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.25.12.tgz", - "integrity": "sha512-Hhmwd6CInZ3dwpuGTF8fJG6yoWmsToE+vYgD4nytZVxcu1ulHpUQRAB1UJ8+N1Am3Mz4+xOByoQoSZf4D+CpkA==", - "cpu": [ - "ppc64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "aix" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/android-arm": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.25.12.tgz", - "integrity": "sha512-VJ+sKvNA/GE7Ccacc9Cha7bpS8nyzVv0jdVgwNDaR4gDMC/2TTRc33Ip8qrNYUcpkOHUT5OZ0bUcNNVZQ9RLlg==", - "cpu": [ - "arm" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "android" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/android-arm64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.25.12.tgz", - "integrity": "sha512-6AAmLG7zwD1Z159jCKPvAxZd4y/VTO0VkprYy+3N2FtJ8+BQWFXU+OxARIwA46c5tdD9SsKGZ/1ocqBS/gAKHg==", - "cpu": [ - "arm64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "android" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/android-x64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.25.12.tgz", - "integrity": "sha512-5jbb+2hhDHx5phYR2By8GTWEzn6I9UqR11Kwf22iKbNpYrsmRB18aX/9ivc5cabcUiAT/wM+YIZ6SG9QO6a8kg==", - "cpu": [ - "x64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "android" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/darwin-arm64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.25.12.tgz", - "integrity": "sha512-N3zl+lxHCifgIlcMUP5016ESkeQjLj/959RxxNYIthIg+CQHInujFuXeWbWMgnTo4cp5XVHqFPmpyu9J65C1Yg==", - "cpu": [ - "arm64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "darwin" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/darwin-x64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.25.12.tgz", - "integrity": "sha512-HQ9ka4Kx21qHXwtlTUVbKJOAnmG1ipXhdWTmNXiPzPfWKpXqASVcWdnf2bnL73wgjNrFXAa3yYvBSd9pzfEIpA==", - "cpu": [ - "x64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "darwin" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/freebsd-arm64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.25.12.tgz", - "integrity": "sha512-gA0Bx759+7Jve03K1S0vkOu5Lg/85dou3EseOGUes8flVOGxbhDDh/iZaoek11Y8mtyKPGF3vP8XhnkDEAmzeg==", - "cpu": [ - "arm64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "freebsd" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/freebsd-x64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.25.12.tgz", - "integrity": "sha512-TGbO26Yw2xsHzxtbVFGEXBFH0FRAP7gtcPE7P5yP7wGy7cXK2oO7RyOhL5NLiqTlBh47XhmIUXuGciXEqYFfBQ==", - "cpu": [ - "x64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "freebsd" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/linux-arm": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.25.12.tgz", - "integrity": "sha512-lPDGyC1JPDou8kGcywY0YILzWlhhnRjdof3UlcoqYmS9El818LLfJJc3PXXgZHrHCAKs/Z2SeZtDJr5MrkxtOw==", - "cpu": [ - "arm" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/linux-arm64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.25.12.tgz", - "integrity": "sha512-8bwX7a8FghIgrupcxb4aUmYDLp8pX06rGh5HqDT7bB+8Rdells6mHvrFHHW2JAOPZUbnjUpKTLg6ECyzvas2AQ==", - "cpu": [ - "arm64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/linux-ia32": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.25.12.tgz", - "integrity": "sha512-0y9KrdVnbMM2/vG8KfU0byhUN+EFCny9+8g202gYqSSVMonbsCfLjUO+rCci7pM0WBEtz+oK/PIwHkzxkyharA==", - "cpu": [ - "ia32" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/linux-loong64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.25.12.tgz", - "integrity": "sha512-h///Lr5a9rib/v1GGqXVGzjL4TMvVTv+s1DPoxQdz7l/AYv6LDSxdIwzxkrPW438oUXiDtwM10o9PmwS/6Z0Ng==", - "cpu": [ - "loong64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/linux-mips64el": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.25.12.tgz", - "integrity": "sha512-iyRrM1Pzy9GFMDLsXn1iHUm18nhKnNMWscjmp4+hpafcZjrr2WbT//d20xaGljXDBYHqRcl8HnxbX6uaA/eGVw==", - "cpu": [ - "mips64el" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/linux-ppc64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.25.12.tgz", - "integrity": "sha512-9meM/lRXxMi5PSUqEXRCtVjEZBGwB7P/D4yT8UG/mwIdze2aV4Vo6U5gD3+RsoHXKkHCfSxZKzmDssVlRj1QQA==", - "cpu": [ - "ppc64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/linux-riscv64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.25.12.tgz", - "integrity": "sha512-Zr7KR4hgKUpWAwb1f3o5ygT04MzqVrGEGXGLnj15YQDJErYu/BGg+wmFlIDOdJp0PmB0lLvxFIOXZgFRrdjR0w==", - "cpu": [ - "riscv64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/linux-s390x": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.25.12.tgz", - "integrity": "sha512-MsKncOcgTNvdtiISc/jZs/Zf8d0cl/t3gYWX8J9ubBnVOwlk65UIEEvgBORTiljloIWnBzLs4qhzPkJcitIzIg==", - "cpu": [ - "s390x" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/linux-x64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.25.12.tgz", - "integrity": "sha512-uqZMTLr/zR/ed4jIGnwSLkaHmPjOjJvnm6TVVitAa08SLS9Z0VM8wIRx7gWbJB5/J54YuIMInDquWyYvQLZkgw==", - "cpu": [ - "x64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/netbsd-arm64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/netbsd-arm64/-/netbsd-arm64-0.25.12.tgz", - "integrity": "sha512-xXwcTq4GhRM7J9A8Gv5boanHhRa/Q9KLVmcyXHCTaM4wKfIpWkdXiMog/KsnxzJ0A1+nD+zoecuzqPmCRyBGjg==", - "cpu": [ - "arm64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "netbsd" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/netbsd-x64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.25.12.tgz", - "integrity": "sha512-Ld5pTlzPy3YwGec4OuHh1aCVCRvOXdH8DgRjfDy/oumVovmuSzWfnSJg+VtakB9Cm0gxNO9BzWkj6mtO1FMXkQ==", - "cpu": [ - "x64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "netbsd" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/openbsd-arm64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/openbsd-arm64/-/openbsd-arm64-0.25.12.tgz", - "integrity": "sha512-fF96T6KsBo/pkQI950FARU9apGNTSlZGsv1jZBAlcLL1MLjLNIWPBkj5NlSz8aAzYKg+eNqknrUJ24QBybeR5A==", - "cpu": [ - "arm64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "openbsd" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/openbsd-x64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.25.12.tgz", - "integrity": "sha512-MZyXUkZHjQxUvzK7rN8DJ3SRmrVrke8ZyRusHlP+kuwqTcfWLyqMOE3sScPPyeIXN/mDJIfGXvcMqCgYKekoQw==", - "cpu": [ - "x64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "openbsd" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/openharmony-arm64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/openharmony-arm64/-/openharmony-arm64-0.25.12.tgz", - "integrity": "sha512-rm0YWsqUSRrjncSXGA7Zv78Nbnw4XL6/dzr20cyrQf7ZmRcsovpcRBdhD43Nuk3y7XIoW2OxMVvwuRvk9XdASg==", - "cpu": [ - "arm64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "openharmony" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/sunos-x64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.25.12.tgz", - "integrity": "sha512-3wGSCDyuTHQUzt0nV7bocDy72r2lI33QL3gkDNGkod22EsYl04sMf0qLb8luNKTOmgF/eDEDP5BFNwoBKH441w==", - "cpu": [ - "x64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "sunos" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/win32-arm64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.25.12.tgz", - "integrity": "sha512-rMmLrur64A7+DKlnSuwqUdRKyd3UE7oPJZmnljqEptesKM8wx9J8gx5u0+9Pq0fQQW8vqeKebwNXdfOyP+8Bsg==", - "cpu": [ - "arm64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "win32" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/win32-ia32": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.25.12.tgz", - "integrity": "sha512-HkqnmmBoCbCwxUKKNPBixiWDGCpQGVsrQfJoVGYLPT41XWF8lHuE5N6WhVia2n4o5QK5M4tYr21827fNhi4byQ==", - "cpu": [ - "ia32" - ], + "node_modules/@emnapi/core": { + "version": "1.10.0", + "resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.10.0.tgz", + "integrity": "sha512-yq6OkJ4p82CAfPl0u9mQebQHKPJkY7WrIuk205cTYnYe+k2Z8YBh11FrbRG/H6ihirqcacOgl2BIO8oyMQLeXw==", "dev": true, "license": "MIT", "optional": true, - "os": [ - "win32" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@esbuild/win32-x64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.25.12.tgz", - "integrity": "sha512-alJC0uCZpTFrSL0CCDjcgleBXPnCrEAhTBILpeAp7M/OFgoqtAetfBzX0xM00MUsVVPpVjlPuMbREqnZCXaTnA==", - "cpu": [ - "x64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "win32" - ], - "engines": { - "node": ">=18" - } - }, - "node_modules/@jridgewell/gen-mapping": { - "version": "0.3.13", - "resolved": "https://registry.npmjs.org/@jridgewell/gen-mapping/-/gen-mapping-0.3.13.tgz", - "integrity": "sha512-2kkt/7niJ6MgEPxF0bYdQ6etZaA+fQvDcLKckhy1yIQOzaoKjBBjSj63/aLVjYE3qhRt5dvM+uUyfCg6UKCBbA==", - "dev": true, - "license": "MIT", "dependencies": { - "@jridgewell/sourcemap-codec": "^1.5.0", - "@jridgewell/trace-mapping": "^0.3.24" + "@emnapi/wasi-threads": "1.2.1", + "tslib": "^2.4.0" } }, - "node_modules/@jridgewell/remapping": { - "version": "2.3.5", - "resolved": "https://registry.npmjs.org/@jridgewell/remapping/-/remapping-2.3.5.tgz", - "integrity": "sha512-LI9u/+laYG4Ds1TDKSJW2YPrIlcVYOwi2fUC6xB43lueCjgxV4lffOCZCtYFiH6TNOX+tQKXx97T4IKHbhyHEQ==", + "node_modules/@emnapi/runtime": { + "version": "1.10.0", + "resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.10.0.tgz", + "integrity": "sha512-ewvYlk86xUoGI0zQRNq/mC+16R1QeDlKQy21Ki3oSYXNgLb45GV1P6A0M+/s6nyCuNDqe5VpaY84BzXGwVbwFA==", "dev": true, "license": "MIT", + "optional": true, "dependencies": { - "@jridgewell/gen-mapping": "^0.3.5", - "@jridgewell/trace-mapping": "^0.3.24" - } - }, - "node_modules/@jridgewell/resolve-uri": { - "version": "3.1.2", - "resolved": "https://registry.npmjs.org/@jridgewell/resolve-uri/-/resolve-uri-3.1.2.tgz", - "integrity": "sha512-bRISgCIjP20/tbWSPWMEi54QVPRZExkuD9lJL+UIxUKtwVJA8wW1Trb1jMs1RFXo1CBTNZ/5hpC9QvmKWdopKw==", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=6.0.0" + "tslib": "^2.4.0" } }, - "node_modules/@jridgewell/sourcemap-codec": { - "version": "1.5.5", - "resolved": "https://registry.npmjs.org/@jridgewell/sourcemap-codec/-/sourcemap-codec-1.5.5.tgz", - "integrity": "sha512-cYQ9310grqxueWbl+WuIUIaiUaDcj7WOq5fVhEljNVgRfOUhY9fy2zTvfoqWsnebh8Sl70VScFbICvJnLKB0Og==", - "dev": true, - "license": "MIT" - }, - "node_modules/@jridgewell/trace-mapping": { - "version": "0.3.31", - "resolved": "https://registry.npmjs.org/@jridgewell/trace-mapping/-/trace-mapping-0.3.31.tgz", - "integrity": "sha512-zzNR+SdQSDJzc8joaeP8QQoCQr8NuYx2dIIytl1QeBEZHJ9uW6hebsrYgbz8hJwUQao3TWCMtmfV8Nu1twOLAw==", + "node_modules/@emnapi/wasi-threads": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/@emnapi/wasi-threads/-/wasi-threads-1.2.1.tgz", + "integrity": "sha512-uTII7OYF+/Mes/MrcIOYp5yOtSMLBWSIoLPpcgwipoiKbli6k322tcoFsxoIIxPDqW01SQGAgko4EzZi2BNv2w==", "dev": true, "license": "MIT", + "optional": true, "dependencies": { - "@jridgewell/resolve-uri": "^3.1.0", - "@jridgewell/sourcemap-codec": "^1.4.14" + "tslib": "^2.4.0" } }, "node_modules/@kurkle/color": { @@ -818,31 +68,39 @@ "integrity": "sha512-M5UknZPHRu3DEDWoipU6sE8PdkZ6Z/S+v4dD+Ke8IaNlpdSQah50lz1KtcFBa2vsdOnwbbnxJwVM4wty6udA5w==", "license": "MIT" }, - "node_modules/@rolldown/pluginutils": { - "version": "1.0.0-beta.27", - "resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.0-beta.27.tgz", - "integrity": "sha512-+d0F4MKMCbeVUJwG96uQ4SgAznZNSq93I3V+9NHA4OpvqG8mRCpGdKmK8l/dl02h2CCDHwW2FqilnTyDcAnqjA==", + "node_modules/@napi-rs/wasm-runtime": { + "version": "1.1.4", + "resolved": "https://registry.npmjs.org/@napi-rs/wasm-runtime/-/wasm-runtime-1.1.4.tgz", + "integrity": "sha512-3NQNNgA1YSlJb/kMH1ildASP9HW7/7kYnRI2szWJaofaS1hWmbGI4H+d3+22aGzXXN9IJ+n+GiFVcGipJP18ow==", "dev": true, - "license": "MIT" + "license": "MIT", + "optional": true, + "dependencies": { + "@tybys/wasm-util": "^0.10.1" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + }, + "peerDependencies": { + "@emnapi/core": "^1.7.1", + "@emnapi/runtime": "^1.7.1" + } }, - "node_modules/@rollup/rollup-android-arm-eabi": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm-eabi/-/rollup-android-arm-eabi-4.60.0.tgz", - "integrity": "sha512-WOhNW9K8bR3kf4zLxbfg6Pxu2ybOUbB2AjMDHSQx86LIF4rH4Ft7vmMwNt0loO0eonglSNy4cpD3MKXXKQu0/A==", - "cpu": [ - "arm" - ], + "node_modules/@oxc-project/types": { + "version": "0.127.0", + "resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.127.0.tgz", + "integrity": "sha512-aIYXQBo4lCbO4z0R3FHeucQHpF46l2LbMdxRvqvuRuW2OxdnSkcng5B8+K12spgLDj93rtN3+J2Vac/TIO+ciQ==", "dev": true, "license": "MIT", - "optional": true, - "os": [ - "android" - ] + "funding": { + "url": "https://github.com/sponsors/Boshen" + } }, - "node_modules/@rollup/rollup-android-arm64": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm64/-/rollup-android-arm64-4.60.0.tgz", - "integrity": "sha512-u6JHLll5QKRvjciE78bQXDmqRqNs5M/3GVqZeMwvmjaNODJih/WIrJlFVEihvV0MiYFmd+ZyPr9wxOVbPAG2Iw==", + "node_modules/@rolldown/binding-android-arm64": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.0.0-rc.17.tgz", + "integrity": "sha512-s70pVGhw4zqGeFnXWvAzJDlvxhlRollagdCCKRgOsgUOH3N1l0LIxf83AtGzmb5SiVM4Hjl5HyarMRfdfj3DaQ==", "cpu": [ "arm64" ], @@ -851,12 +109,15 @@ "optional": true, "os": [ "android" - ] + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-darwin-arm64": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-arm64/-/rollup-darwin-arm64-4.60.0.tgz", - "integrity": "sha512-qEF7CsKKzSRc20Ciu2Zw1wRrBz4g56F7r/vRwY430UPp/nt1x21Q/fpJ9N5l47WWvJlkNCPJz3QRVw008fi7yA==", + "node_modules/@rolldown/binding-darwin-arm64": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.0.0-rc.17.tgz", + "integrity": "sha512-4ksWc9n0mhlZpZ9PMZgTGjeOPRu8MB1Z3Tz0Mo02eWfWCHMW1zN82Qz/pL/rC+yQa+8ZnutMF0JjJe7PjwasYw==", "cpu": [ "arm64" ], @@ -865,12 +126,15 @@ "optional": true, "os": [ "darwin" - ] + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-darwin-x64": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-x64/-/rollup-darwin-x64-4.60.0.tgz", - "integrity": "sha512-WADYozJ4QCnXCH4wPB+3FuGmDPoFseVCUrANmA5LWwGmC6FL14BWC7pcq+FstOZv3baGX65tZ378uT6WG8ynTw==", + "node_modules/@rolldown/binding-darwin-x64": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.0.0-rc.17.tgz", + "integrity": "sha512-SUSDOI6WwUVNcWxd02QEBjLdY1VPHvlEkw6T/8nYG322iYWCTxRb1vzk4E+mWWYehTp7ERibq54LSJGjmouOsw==", "cpu": [ "x64" ], @@ -879,26 +143,15 @@ "optional": true, "os": [ "darwin" - ] - }, - "node_modules/@rollup/rollup-freebsd-arm64": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-arm64/-/rollup-freebsd-arm64-4.60.0.tgz", - "integrity": "sha512-6b8wGHJlDrGeSE3aH5mGNHBjA0TTkxdoNHik5EkvPHCt351XnigA4pS7Wsj/Eo9Y8RBU6f35cjN9SYmCFBtzxw==", - "cpu": [ - "arm64" ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "freebsd" - ] + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-freebsd-x64": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-x64/-/rollup-freebsd-x64-4.60.0.tgz", - "integrity": "sha512-h25Ga0t4jaylMB8M/JKAyrvvfxGRjnPQIR8lnCayyzEjEOx2EJIlIiMbhpWxDRKGKF8jbNH01NnN663dH638mA==", + "node_modules/@rolldown/binding-freebsd-x64": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.0.0-rc.17.tgz", + "integrity": "sha512-hwnz3nw9dbJ05EDO/PvcjaaewqqDy7Y1rn1UO81l8iIK1GjenME75dl16ajbvSSMfv66WXSRCYKIqfgq2KCfxw==", "cpu": [ "x64" ], @@ -907,26 +160,15 @@ "optional": true, "os": [ "freebsd" - ] - }, - "node_modules/@rollup/rollup-linux-arm-gnueabihf": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-gnueabihf/-/rollup-linux-arm-gnueabihf-4.60.0.tgz", - "integrity": "sha512-RzeBwv0B3qtVBWtcuABtSuCzToo2IEAIQrcyB/b2zMvBWVbjo8bZDjACUpnaafaxhTw2W+imQbP2BD1usasK4g==", - "cpu": [ - "arm" ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ] + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-linux-arm-musleabihf": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-musleabihf/-/rollup-linux-arm-musleabihf-4.60.0.tgz", - "integrity": "sha512-Sf7zusNI2CIU1HLzuu9Tc5YGAHEZs5Lu7N1ssJG4Tkw6e0MEsN7NdjUDDfGNHy2IU+ENyWT+L2obgWiguWibWQ==", + "node_modules/@rolldown/binding-linux-arm-gnueabihf": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.0.0-rc.17.tgz", + "integrity": "sha512-IS+W7epTcwANmFSQFrS1SivEXHtl1JtuQA9wlxrZTcNi6mx+FDOYrakGevvvTwgj2JvWiK8B29/qD9BELZPyXQ==", "cpu": [ "arm" ], @@ -935,26 +177,15 @@ "optional": true, "os": [ "linux" - ] - }, - "node_modules/@rollup/rollup-linux-arm64-gnu": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-gnu/-/rollup-linux-arm64-gnu-4.60.0.tgz", - "integrity": "sha512-DX2x7CMcrJzsE91q7/O02IJQ5/aLkVtYFryqCjduJhUfGKG6yJV8hxaw8pZa93lLEpPTP/ohdN4wFz7yp/ry9A==", - "cpu": [ - "arm64" ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ] + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-linux-arm64-musl": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-musl/-/rollup-linux-arm64-musl-4.60.0.tgz", - "integrity": "sha512-09EL+yFVbJZlhcQfShpswwRZ0Rg+z/CsSELFCnPt3iK+iqwGsI4zht3secj5vLEs957QvFFXnzAT0FFPIxSrkQ==", + "node_modules/@rolldown/binding-linux-arm64-gnu": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.0.0-rc.17.tgz", + "integrity": "sha512-e6usGaHKW5BMNZOymS1UcEYGowQMWcgZ71Z17Sl/h2+ZziNJ1a9n3Zvcz6LdRyIW5572wBCTH/Z+bKuZouGk9Q==", "cpu": [ "arm64" ], @@ -963,54 +194,32 @@ "optional": true, "os": [ "linux" - ] - }, - "node_modules/@rollup/rollup-linux-loong64-gnu": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-loong64-gnu/-/rollup-linux-loong64-gnu-4.60.0.tgz", - "integrity": "sha512-i9IcCMPr3EXm8EQg5jnja0Zyc1iFxJjZWlb4wr7U2Wx/GrddOuEafxRdMPRYVaXjgbhvqalp6np07hN1w9kAKw==", - "cpu": [ - "loong64" ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ] + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-linux-loong64-musl": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-loong64-musl/-/rollup-linux-loong64-musl-4.60.0.tgz", - "integrity": "sha512-DGzdJK9kyJ+B78MCkWeGnpXJ91tK/iKA6HwHxF4TAlPIY7GXEvMe8hBFRgdrR9Ly4qebR/7gfUs9y2IoaVEyog==", + "node_modules/@rolldown/binding-linux-arm64-musl": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.0.0-rc.17.tgz", + "integrity": "sha512-b/CgbwAJpmrRLp02RPfhbudf5tZnN9nsPWK82znefso832etkem8H7FSZwxrOI9djcdTP7U6YfNhbRnh7djErg==", "cpu": [ - "loong64" + "arm64" ], "dev": true, "license": "MIT", "optional": true, "os": [ "linux" - ] - }, - "node_modules/@rollup/rollup-linux-ppc64-gnu": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-ppc64-gnu/-/rollup-linux-ppc64-gnu-4.60.0.tgz", - "integrity": "sha512-RwpnLsqC8qbS8z1H1AxBA1H6qknR4YpPR9w2XX0vo2Sz10miu57PkNcnHVaZkbqyw/kUWfKMI73jhmfi9BRMUQ==", - "cpu": [ - "ppc64" ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ] + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-linux-ppc64-musl": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-ppc64-musl/-/rollup-linux-ppc64-musl-4.60.0.tgz", - "integrity": "sha512-Z8pPf54Ly3aqtdWC3G4rFigZgNvd+qJlOE52fmko3KST9SoGfAdSRCwyoyG05q1HrrAblLbk1/PSIV+80/pxLg==", + "node_modules/@rolldown/binding-linux-ppc64-gnu": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.0.0-rc.17.tgz", + "integrity": "sha512-4EII1iNGRUN5WwGbF/kOh/EIkoDN9HsupgLQoXfY+D1oyJm7/F4t5PYU5n8SWZgG0FEwakyM8pGgwcBYruGTlA==", "cpu": [ "ppc64" ], @@ -1019,40 +228,15 @@ "optional": true, "os": [ "linux" - ] - }, - "node_modules/@rollup/rollup-linux-riscv64-gnu": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-riscv64-gnu/-/rollup-linux-riscv64-gnu-4.60.0.tgz", - "integrity": "sha512-3a3qQustp3COCGvnP4SvrMHnPQ9d1vzCakQVRTliaz8cIp/wULGjiGpbcqrkv0WrHTEp8bQD/B3HBjzujVWLOA==", - "cpu": [ - "riscv64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ] - }, - "node_modules/@rollup/rollup-linux-riscv64-musl": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-riscv64-musl/-/rollup-linux-riscv64-musl-4.60.0.tgz", - "integrity": "sha512-pjZDsVH/1VsghMJ2/kAaxt6dL0psT6ZexQVrijczOf+PeP2BUqTHYejk3l6TlPRydggINOeNRhvpLa0AYpCWSQ==", - "cpu": [ - "riscv64" ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ] + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-linux-s390x-gnu": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-s390x-gnu/-/rollup-linux-s390x-gnu-4.60.0.tgz", - "integrity": "sha512-3ObQs0BhvPgiUVZrN7gqCSvmFuMWvWvsjG5ayJ3Lraqv+2KhOsp+pUbigqbeWqueGIsnn+09HBw27rJ+gYK4VQ==", + "node_modules/@rolldown/binding-linux-s390x-gnu": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.0.0-rc.17.tgz", + "integrity": "sha512-AH8oq3XqQo4IibpVXvPeLDI5pzkpYn0WiZAfT05kFzoJ6tQNzwRdDYQ45M8I/gslbodRZwW8uxLhbSBbkv96rA==", "cpu": [ "s390x" ], @@ -1061,12 +245,15 @@ "optional": true, "os": [ "linux" - ] + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-linux-x64-gnu": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-gnu/-/rollup-linux-x64-gnu-4.60.0.tgz", - "integrity": "sha512-EtylprDtQPdS5rXvAayrNDYoJhIz1/vzN2fEubo3yLE7tfAw+948dO0g4M0vkTVFhKojnF+n6C8bDNe+gDRdTg==", + "node_modules/@rolldown/binding-linux-x64-gnu": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.0.0-rc.17.tgz", + "integrity": "sha512-cLnjV3xfo7KslbU41Z7z8BH/E1y5mzUYzAqih1d1MDaIGZRCMqTijqLv76/P7fyHuvUcfGsIpqCdddbxLLK9rA==", "cpu": [ "x64" ], @@ -1075,12 +262,15 @@ "optional": true, "os": [ "linux" - ] + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-linux-x64-musl": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-musl/-/rollup-linux-x64-musl-4.60.0.tgz", - "integrity": "sha512-k09oiRCi/bHU9UVFqD17r3eJR9bn03TyKraCrlz5ULFJGdJGi7VOmm9jl44vOJvRJ6P7WuBi/s2A97LxxHGIdw==", + "node_modules/@rolldown/binding-linux-x64-musl": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.0.0-rc.17.tgz", + "integrity": "sha512-0phclDw1spsL7dUB37sIARuis2tAgomCJXAHZlpt8PXZ4Ba0dRP1e+66lsRqrfhISeN9bEGNjQs+T/Fbd7oYGw==", "cpu": [ "x64" ], @@ -1089,26 +279,15 @@ "optional": true, "os": [ "linux" - ] - }, - "node_modules/@rollup/rollup-openbsd-x64": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-openbsd-x64/-/rollup-openbsd-x64-4.60.0.tgz", - "integrity": "sha512-1o/0/pIhozoSaDJoDcec+IVLbnRtQmHwPV730+AOD29lHEEo4F5BEUB24H0OBdhbBBDwIOSuf7vgg0Ywxdfiiw==", - "cpu": [ - "x64" ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "openbsd" - ] + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-openharmony-arm64": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-openharmony-arm64/-/rollup-openharmony-arm64-4.60.0.tgz", - "integrity": "sha512-pESDkos/PDzYwtyzB5p/UoNU/8fJo68vcXM9ZW2V0kjYayj1KaaUfi1NmTUTUpMn4UhU4gTuK8gIaFO4UGuMbA==", + "node_modules/@rolldown/binding-openharmony-arm64": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.0.0-rc.17.tgz", + "integrity": "sha512-0ag/hEgXOwgw4t8QyQvUCxvEg+V0KBcA6YuOx9g0r02MprutRF5dyljgm3EmR02O292UX7UeS6HzWHAl6KgyhA==", "cpu": [ "arm64" ], @@ -1117,54 +296,51 @@ "optional": true, "os": [ "openharmony" - ] - }, - "node_modules/@rollup/rollup-win32-arm64-msvc": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-arm64-msvc/-/rollup-win32-arm64-msvc-4.60.0.tgz", - "integrity": "sha512-hj1wFStD7B1YBeYmvY+lWXZ7ey73YGPcViMShYikqKT1GtstIKQAtfUI6yrzPjAy/O7pO0VLXGmUVWXQMaYgTQ==", - "cpu": [ - "arm64" ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "win32" - ] + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-win32-ia32-msvc": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-ia32-msvc/-/rollup-win32-ia32-msvc-4.60.0.tgz", - "integrity": "sha512-SyaIPFoxmUPlNDq5EHkTbiKzmSEmq/gOYFI/3HHJ8iS/v1mbugVa7dXUzcJGQfoytp9DJFLhHH4U3/eTy2Bq4w==", + "node_modules/@rolldown/binding-wasm32-wasi": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-wasm32-wasi/-/binding-wasm32-wasi-1.0.0-rc.17.tgz", + "integrity": "sha512-LEXei6vo0E5wTGwpkJ4KoT3OZJRnglwldt5ziLzOlc6qqb55z4tWNq2A+PFqCJuvWWdP53CVhG1Z9NtToDPJrA==", "cpu": [ - "ia32" + "wasm32" ], "dev": true, "license": "MIT", "optional": true, - "os": [ - "win32" - ] + "dependencies": { + "@emnapi/core": "1.10.0", + "@emnapi/runtime": "1.10.0", + "@napi-rs/wasm-runtime": "^1.1.4" + }, + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-win32-x64-gnu": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-x64-gnu/-/rollup-win32-x64-gnu-4.60.0.tgz", - "integrity": "sha512-RdcryEfzZr+lAr5kRm2ucN9aVlCCa2QNq4hXelZxb8GG0NJSazq44Z3PCCc8wISRuCVnGs0lQJVX5Vp6fKA+IA==", + "node_modules/@rolldown/binding-win32-arm64-msvc": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.0.0-rc.17.tgz", + "integrity": "sha512-gUmyzBl3SPMa6hrqFUth9sVfcLBlYsbMzBx5PlexMroZStgzGqlZ26pYG89rBb45Mnia+oil6YAIFeEWGWhoZA==", "cpu": [ - "x64" + "arm64" ], "dev": true, "license": "MIT", "optional": true, "os": [ "win32" - ] + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-win32-x64-msvc": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-x64-msvc/-/rollup-win32-x64-msvc-4.60.0.tgz", - "integrity": "sha512-PrsWNQ8BuE00O3Xsx3ALh2Df8fAj9+cvvX9AIA6o4KpATR98c9mud4XtDWVvsEuyia5U4tVSTKygawyJkjm60w==", + "node_modules/@rolldown/binding-win32-x64-msvc": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.0.0-rc.17.tgz", + "integrity": "sha512-3hkiolcUAvPB9FLb3UZdfjVVNWherN1f/skkGWJP/fgSQhYUZpSIRr0/I8ZK9TkF3F7kxvJAk0+IcKvPHk9qQg==", "cpu": [ "x64" ], @@ -1173,113 +349,79 @@ "optional": true, "os": [ "win32" - ] - }, - "node_modules/@types/babel__core": { - "version": "7.20.5", - "resolved": "https://registry.npmjs.org/@types/babel__core/-/babel__core-7.20.5.tgz", - "integrity": "sha512-qoQprZvz5wQFJwMDqeseRXWv3rqMvhgpbXFfVyWhbx9X47POIA6i/+dXefEmZKoAgOaTdaIgNSMqMIU61yRyzA==", - "dev": true, - "license": "MIT", - "dependencies": { - "@babel/parser": "^7.20.7", - "@babel/types": "^7.20.7", - "@types/babel__generator": "*", - "@types/babel__template": "*", - "@types/babel__traverse": "*" - } - }, - "node_modules/@types/babel__generator": { - "version": "7.27.0", - "resolved": "https://registry.npmjs.org/@types/babel__generator/-/babel__generator-7.27.0.tgz", - "integrity": "sha512-ufFd2Xi92OAVPYsy+P4n7/U7e68fex0+Ee8gSG9KX7eo084CWiQ4sdxktvdl0bOPupXtVJPY19zk6EwWqUQ8lg==", - "dev": true, - "license": "MIT", - "dependencies": { - "@babel/types": "^7.0.0" + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" } }, - "node_modules/@types/babel__template": { - "version": "7.4.4", - "resolved": "https://registry.npmjs.org/@types/babel__template/-/babel__template-7.4.4.tgz", - "integrity": "sha512-h/NUaSyG5EyxBIp8YRxo4RMe2/qQgvyowRwVMzhYhBCONbW8PUsg4lkFMrhgZhUe5z3L3MiLDuvyJ/CaPa2A8A==", + "node_modules/@rolldown/pluginutils": { + "version": "1.0.0-rc.7", + "resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.0-rc.7.tgz", + "integrity": "sha512-qujRfC8sFVInYSPPMLQByRh7zhwkGFS4+tyMQ83srV1qrxL4g8E2tyxVVyxd0+8QeBM1mIk9KbWxkegRr76XzA==", "dev": true, - "license": "MIT", - "dependencies": { - "@babel/parser": "^7.1.0", - "@babel/types": "^7.0.0" - } + "license": "MIT" }, - "node_modules/@types/babel__traverse": { - "version": "7.28.0", - "resolved": "https://registry.npmjs.org/@types/babel__traverse/-/babel__traverse-7.28.0.tgz", - "integrity": "sha512-8PvcXf70gTDZBgt9ptxJ8elBeBjcLOAcOtoO/mPJjtji1+CdGbHgm77om1GrsPxsiE+uXIpNSK64UYaIwQXd4Q==", + "node_modules/@tybys/wasm-util": { + "version": "0.10.1", + "resolved": "https://registry.npmjs.org/@tybys/wasm-util/-/wasm-util-0.10.1.tgz", + "integrity": "sha512-9tTaPJLSiejZKx+Bmog4uSubteqTvFrVrURwkmHixBo0G4seD0zUxp98E1DzUBJxLQ3NPwXrGKDiVjwx/DpPsg==", "dev": true, "license": "MIT", + "optional": true, "dependencies": { - "@babel/types": "^7.28.2" + "tslib": "^2.4.0" } }, - "node_modules/@types/estree": { - "version": "1.0.8", - "resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.8.tgz", - "integrity": "sha512-dWHzHa2WqEXI/O1E9OjrocMTKJl2mSrEolh1Iomrv6U+JuNwaHXsXx9bLu5gG7BUWFIN0skIQJQ/L1rIex4X6w==", - "dev": true, - "license": "MIT" - }, "node_modules/@types/hammerjs": { "version": "2.0.46", "resolved": "https://registry.npmjs.org/@types/hammerjs/-/hammerjs-2.0.46.tgz", "integrity": "sha512-ynRvcq6wvqexJ9brDMS4BnBLzmr0e14d6ZJTEShTBWKymQiHwlAyGu0ZPEFI2Fh1U53F7tN9ufClWM5KvqkKOw==", "license": "MIT" }, - "node_modules/@types/prop-types": { - "version": "15.7.15", - "resolved": "https://registry.npmjs.org/@types/prop-types/-/prop-types-15.7.15.tgz", - "integrity": "sha512-F6bEyamV9jKGAFBEmlQnesRPGOQqS2+Uwi0Em15xenOxHaf2hv6L8YCVn3rPdPJOiJfPiCnLIRyvwVaqMY3MIw==", - "dev": true, - "license": "MIT" - }, "node_modules/@types/react": { - "version": "18.3.28", - "resolved": "https://registry.npmjs.org/@types/react/-/react-18.3.28.tgz", - "integrity": "sha512-z9VXpC7MWrhfWipitjNdgCauoMLRdIILQsAEV+ZesIzBq/oUlxk0m3ApZuMFCXdnS4U7KrI+l3WRUEGQ8K1QKw==", + "version": "19.2.14", + "resolved": "https://registry.npmjs.org/@types/react/-/react-19.2.14.tgz", + "integrity": "sha512-ilcTH/UniCkMdtexkoCN0bI7pMcJDvmQFPvuPvmEaYA/NSfFTAgdUSLAoVjaRJm7+6PvcM+q1zYOwS4wTYMF9w==", "dev": true, "license": "MIT", "dependencies": { - "@types/prop-types": "*", "csstype": "^3.2.2" } }, "node_modules/@types/react-dom": { - "version": "18.3.7", - "resolved": "https://registry.npmjs.org/@types/react-dom/-/react-dom-18.3.7.tgz", - "integrity": "sha512-MEe3UeoENYVFXzoXEWsvcpg6ZvlrFNlOQ7EOsvhI3CfAXwzPfO8Qwuxd40nepsYKqyyVQnTdEfv68q91yLcKrQ==", + "version": "19.2.3", + "resolved": "https://registry.npmjs.org/@types/react-dom/-/react-dom-19.2.3.tgz", + "integrity": "sha512-jp2L/eY6fn+KgVVQAOqYItbF0VY/YApe5Mz2F0aykSO8gx31bYCZyvSeYxCHKvzHG5eZjc+zyaS5BrBWya2+kQ==", "dev": true, "license": "MIT", "peerDependencies": { - "@types/react": "^18.0.0" + "@types/react": "^19.2.0" } }, "node_modules/@vitejs/plugin-react": { - "version": "4.7.0", - "resolved": "https://registry.npmjs.org/@vitejs/plugin-react/-/plugin-react-4.7.0.tgz", - "integrity": "sha512-gUu9hwfWvvEDBBmgtAowQCojwZmJ5mcLn3aufeCsitijs3+f2NsrPtlAWIR6OPiqljl96GVCUbLe0HyqIpVaoA==", + "version": "6.0.1", + "resolved": "https://registry.npmjs.org/@vitejs/plugin-react/-/plugin-react-6.0.1.tgz", + "integrity": "sha512-l9X/E3cDb+xY3SWzlG1MOGt2usfEHGMNIaegaUGFsLkb3RCn/k8/TOXBcab+OndDI4TBtktT8/9BwwW8Vi9KUQ==", "dev": true, "license": "MIT", "dependencies": { - "@babel/core": "^7.28.0", - "@babel/plugin-transform-react-jsx-self": "^7.27.1", - "@babel/plugin-transform-react-jsx-source": "^7.27.1", - "@rolldown/pluginutils": "1.0.0-beta.27", - "@types/babel__core": "^7.20.5", - "react-refresh": "^0.17.0" + "@rolldown/pluginutils": "1.0.0-rc.7" }, "engines": { - "node": "^14.18.0 || >=16.0.0" + "node": "^20.19.0 || >=22.12.0" }, "peerDependencies": { - "vite": "^4.2.0 || ^5.0.0 || ^6.0.0 || ^7.0.0" + "@rolldown/plugin-babel": "^0.1.7 || ^0.2.0", + "babel-plugin-react-compiler": "^1.0.0", + "vite": "^8.0.0" + }, + "peerDependenciesMeta": { + "@rolldown/plugin-babel": { + "optional": true + }, + "babel-plugin-react-compiler": { + "optional": true + } } }, "node_modules/ansi-regex": { @@ -1308,74 +450,6 @@ "url": "https://github.com/chalk/ansi-styles?sponsor=1" } }, - "node_modules/baseline-browser-mapping": { - "version": "2.10.10", - "resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.10.10.tgz", - "integrity": "sha512-sUoJ3IMxx4AyRqO4MLeHlnGDkyXRoUG0/AI9fjK+vS72ekpV0yWVY7O0BVjmBcRtkNcsAO2QDZ4tdKKGoI6YaQ==", - "dev": true, - "license": "Apache-2.0", - "bin": { - "baseline-browser-mapping": "dist/cli.cjs" - }, - "engines": { - "node": ">=6.0.0" - } - }, - "node_modules/browserslist": { - "version": "4.28.1", - "resolved": "https://registry.npmjs.org/browserslist/-/browserslist-4.28.1.tgz", - "integrity": "sha512-ZC5Bd0LgJXgwGqUknZY/vkUQ04r8NXnJZ3yYi4vDmSiZmC/pdSN0NbNRPxZpbtO4uAfDUAFffO8IZoM3Gj8IkA==", - "dev": true, - "funding": [ - { - "type": "opencollective", - "url": "https://opencollective.com/browserslist" - }, - { - "type": "tidelift", - "url": "https://tidelift.com/funding/github/npm/browserslist" - }, - { - "type": "github", - "url": "https://github.com/sponsors/ai" - } - ], - "license": "MIT", - "dependencies": { - "baseline-browser-mapping": "^2.9.0", - "caniuse-lite": "^1.0.30001759", - "electron-to-chromium": "^1.5.263", - "node-releases": "^2.0.27", - "update-browserslist-db": "^1.2.0" - }, - "bin": { - "browserslist": "cli.js" - }, - "engines": { - "node": "^6 || ^7 || ^8 || ^9 || ^10 || ^11 || ^12 || >=13.7" - } - }, - "node_modules/caniuse-lite": { - "version": "1.0.30001780", - "resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001780.tgz", - "integrity": "sha512-llngX0E7nQci5BPJDqoZSbuZ5Bcs9F5db7EtgfwBerX9XGtkkiO4NwfDDIRzHTTwcYC8vC7bmeUEPGrKlR/TkQ==", - "dev": true, - "funding": [ - { - "type": "opencollective", - "url": "https://opencollective.com/browserslist" - }, - { - "type": "tidelift", - "url": "https://tidelift.com/funding/github/npm/caniuse-lite" - }, - { - "type": "github", - "url": "https://github.com/sponsors/ai" - } - ], - "license": "CC-BY-4.0" - }, "node_modules/chalk": { "version": "4.1.2", "resolved": "https://registry.npmjs.org/chalk/-/chalk-4.1.2.tgz", @@ -1467,39 +541,29 @@ "license": "MIT" }, "node_modules/concurrently": { - "version": "8.2.2", - "resolved": "https://registry.npmjs.org/concurrently/-/concurrently-8.2.2.tgz", - "integrity": "sha512-1dP4gpXFhei8IOtlXRE/T/4H88ElHgTiUzh71YUmtjTEHMSRS2Z/fgOxHSxxusGHogsRfxNq1vyAwxSC+EVyDg==", + "version": "9.2.1", + "resolved": "https://registry.npmjs.org/concurrently/-/concurrently-9.2.1.tgz", + "integrity": "sha512-fsfrO0MxV64Znoy8/l1vVIjjHa29SZyyqPgQBwhiDcaW8wJc2W3XWVOGx4M3oJBnv/zdUZIIp1gDeS98GzP8Ng==", "dev": true, "license": "MIT", "dependencies": { - "chalk": "^4.1.2", - "date-fns": "^2.30.0", - "lodash": "^4.17.21", - "rxjs": "^7.8.1", - "shell-quote": "^1.8.1", - "spawn-command": "0.0.2", - "supports-color": "^8.1.1", - "tree-kill": "^1.2.2", - "yargs": "^17.7.2" + "chalk": "4.1.2", + "rxjs": "7.8.2", + "shell-quote": "1.8.3", + "supports-color": "8.1.1", + "tree-kill": "1.2.2", + "yargs": "17.7.2" }, "bin": { "conc": "dist/bin/concurrently.js", "concurrently": "dist/bin/concurrently.js" }, "engines": { - "node": "^14.13.0 || >=16.0.0" + "node": ">=18" }, "funding": { - "url": "https://github.com/open-cli-tools/concurrently?sponsor=1" - } - }, - "node_modules/convert-source-map": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/convert-source-map/-/convert-source-map-2.0.0.tgz", - "integrity": "sha512-Kvp459HrV2FEJ1CAsi1Ku+MY3kasH19TFykTz2xWmMeq6bk2NU3XXvfJ+Q61m0xktWwt+1HSYf3JZsTms3aRJg==", - "dev": true, - "license": "MIT" + "url": "https://github.com/open-cli-tools/concurrently?sponsor=1" + } }, "node_modules/csstype": { "version": "3.2.3", @@ -1508,39 +572,14 @@ "dev": true, "license": "MIT" }, - "node_modules/date-fns": { - "version": "2.30.0", - "resolved": "https://registry.npmjs.org/date-fns/-/date-fns-2.30.0.tgz", - "integrity": "sha512-fnULvOpxnC5/Vg3NCiWelDsLiUc9bRwAPs/+LfTLNvetFCtCTN+yQz15C/fs4AwX1R9K5GLtLfn8QW+dWisaAw==", - "dev": true, - "license": "MIT", - "dependencies": { - "@babel/runtime": "^7.21.0" - }, - "engines": { - "node": ">=0.11" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/date-fns" - } - }, - "node_modules/debug": { - "version": "4.4.3", - "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", - "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", + "node_modules/detect-libc": { + "version": "2.1.2", + "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz", + "integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==", "dev": true, - "license": "MIT", - "dependencies": { - "ms": "^2.1.3" - }, + "license": "Apache-2.0", "engines": { - "node": ">=6.0" - }, - "peerDependenciesMeta": { - "supports-color": { - "optional": true - } + "node": ">=8" } }, "node_modules/downsample": { @@ -1549,13 +588,6 @@ "integrity": "sha512-teYPhUPxqwtyICt47t1mP/LjhbRV/ghuKb/LmFDbcZ0CjqFD31tn6rVLZoeCEa1xr8+f2skW8UjRiLiGIKQE4w==", "license": "MIT" }, - "node_modules/electron-to-chromium": { - "version": "1.5.321", - "resolved": "https://registry.npmjs.org/electron-to-chromium/-/electron-to-chromium-1.5.321.tgz", - "integrity": "sha512-L2C7Q279W2D/J4PLZLk7sebOILDSWos7bMsMNN06rK482umHUrh/3lM8G7IlHFOYip2oAg5nha1rCMxr/rs6ZQ==", - "dev": true, - "license": "ISC" - }, "node_modules/emoji-regex": { "version": "8.0.0", "resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-8.0.0.tgz", @@ -1563,48 +595,6 @@ "dev": true, "license": "MIT" }, - "node_modules/esbuild": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.25.12.tgz", - "integrity": "sha512-bbPBYYrtZbkt6Os6FiTLCTFxvq4tt3JKall1vRwshA3fdVztsLAatFaZobhkBC8/BrPetoa0oksYoKXoG4ryJg==", - "dev": true, - "hasInstallScript": true, - "license": "MIT", - "bin": { - "esbuild": "bin/esbuild" - }, - "engines": { - "node": ">=18" - }, - "optionalDependencies": { - "@esbuild/aix-ppc64": "0.25.12", - "@esbuild/android-arm": "0.25.12", - "@esbuild/android-arm64": "0.25.12", - "@esbuild/android-x64": "0.25.12", - "@esbuild/darwin-arm64": "0.25.12", - "@esbuild/darwin-x64": "0.25.12", - "@esbuild/freebsd-arm64": "0.25.12", - "@esbuild/freebsd-x64": "0.25.12", - "@esbuild/linux-arm": "0.25.12", - "@esbuild/linux-arm64": "0.25.12", - "@esbuild/linux-ia32": "0.25.12", - "@esbuild/linux-loong64": "0.25.12", - "@esbuild/linux-mips64el": "0.25.12", - "@esbuild/linux-ppc64": "0.25.12", - "@esbuild/linux-riscv64": "0.25.12", - "@esbuild/linux-s390x": "0.25.12", - "@esbuild/linux-x64": "0.25.12", - "@esbuild/netbsd-arm64": "0.25.12", - "@esbuild/netbsd-x64": "0.25.12", - "@esbuild/openbsd-arm64": "0.25.12", - "@esbuild/openbsd-x64": "0.25.12", - "@esbuild/openharmony-arm64": "0.25.12", - "@esbuild/sunos-x64": "0.25.12", - "@esbuild/win32-arm64": "0.25.12", - "@esbuild/win32-ia32": "0.25.12", - "@esbuild/win32-x64": "0.25.12" - } - }, "node_modules/escalade": { "version": "3.2.0", "resolved": "https://registry.npmjs.org/escalade/-/escalade-3.2.0.tgz", @@ -1648,16 +638,6 @@ "node": "^8.16.0 || ^10.6.0 || >=11.0.0" } }, - "node_modules/gensync": { - "version": "1.0.0-beta.2", - "resolved": "https://registry.npmjs.org/gensync/-/gensync-1.0.0-beta.2.tgz", - "integrity": "sha512-3hN7NaskYvMDLQY55gnW3NQ+mesEAepTqlg+VEbj7zzqEMBVNhzcGYYeqFo/TlYz6eQiFcp1HcsCZO+nGgS8zg==", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=6.9.0" - } - }, "node_modules/get-caller-file": { "version": "2.0.5", "resolved": "https://registry.npmjs.org/get-caller-file/-/get-caller-file-2.0.5.tgz", @@ -1697,83 +677,276 @@ "node": ">=8" } }, - "node_modules/js-tokens": { - "version": "4.0.0", - "resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-4.0.0.tgz", - "integrity": "sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ==", - "license": "MIT" + "node_modules/lightningcss": { + "version": "1.32.0", + "resolved": "https://registry.npmjs.org/lightningcss/-/lightningcss-1.32.0.tgz", + "integrity": "sha512-NXYBzinNrblfraPGyrbPoD19C1h9lfI/1mzgWYvXUTe414Gz/X1FD2XBZSZM7rRTrMA8JL3OtAaGifrIKhQ5yQ==", + "dev": true, + "license": "MPL-2.0", + "dependencies": { + "detect-libc": "^2.0.3" + }, + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + }, + "optionalDependencies": { + "lightningcss-android-arm64": "1.32.0", + "lightningcss-darwin-arm64": "1.32.0", + "lightningcss-darwin-x64": "1.32.0", + "lightningcss-freebsd-x64": "1.32.0", + "lightningcss-linux-arm-gnueabihf": "1.32.0", + "lightningcss-linux-arm64-gnu": "1.32.0", + "lightningcss-linux-arm64-musl": "1.32.0", + "lightningcss-linux-x64-gnu": "1.32.0", + "lightningcss-linux-x64-musl": "1.32.0", + "lightningcss-win32-arm64-msvc": "1.32.0", + "lightningcss-win32-x64-msvc": "1.32.0" + } + }, + "node_modules/lightningcss-android-arm64": { + "version": "1.32.0", + "resolved": "https://registry.npmjs.org/lightningcss-android-arm64/-/lightningcss-android-arm64-1.32.0.tgz", + "integrity": "sha512-YK7/ClTt4kAK0vo6w3X+Pnm0D2cf2vPHbhOXdoNti1Ga0al1P4TBZhwjATvjNwLEBCnKvjJc2jQgHXH0NEwlAg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MPL-2.0", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } }, - "node_modules/jsesc": { - "version": "3.1.0", - "resolved": "https://registry.npmjs.org/jsesc/-/jsesc-3.1.0.tgz", - "integrity": "sha512-/sM3dO2FOzXjKQhJuo0Q173wf2KOo8t4I8vHy6lF9poUp7bKT0/NHE8fPX23PwfhnykfqnC2xRxOnVw5XuGIaA==", + "node_modules/lightningcss-darwin-arm64": { + "version": "1.32.0", + "resolved": "https://registry.npmjs.org/lightningcss-darwin-arm64/-/lightningcss-darwin-arm64-1.32.0.tgz", + "integrity": "sha512-RzeG9Ju5bag2Bv1/lwlVJvBE3q6TtXskdZLLCyfg5pt+HLz9BqlICO7LZM7VHNTTn/5PRhHFBSjk5lc4cmscPQ==", + "cpu": [ + "arm64" + ], "dev": true, - "license": "MIT", - "bin": { - "jsesc": "bin/jsesc" + "license": "MPL-2.0", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">= 12.0.0" }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lightningcss-darwin-x64": { + "version": "1.32.0", + "resolved": "https://registry.npmjs.org/lightningcss-darwin-x64/-/lightningcss-darwin-x64-1.32.0.tgz", + "integrity": "sha512-U+QsBp2m/s2wqpUYT/6wnlagdZbtZdndSmut/NJqlCcMLTWp5muCrID+K5UJ6jqD2BFshejCYXniPDbNh73V8w==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MPL-2.0", + "optional": true, + "os": [ + "darwin" + ], "engines": { - "node": ">=6" + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" } }, - "node_modules/json5": { - "version": "2.2.3", - "resolved": "https://registry.npmjs.org/json5/-/json5-2.2.3.tgz", - "integrity": "sha512-XmOWe7eyHYH14cLdVPoyg+GOH3rYX++KpzrylJwSW98t3Nk+U8XOl8FWKOgwtzdb8lXGf6zYwDUzeHMWfxasyg==", + "node_modules/lightningcss-freebsd-x64": { + "version": "1.32.0", + "resolved": "https://registry.npmjs.org/lightningcss-freebsd-x64/-/lightningcss-freebsd-x64-1.32.0.tgz", + "integrity": "sha512-JCTigedEksZk3tHTTthnMdVfGf61Fky8Ji2E4YjUTEQX14xiy/lTzXnu1vwiZe3bYe0q+SpsSH/CTeDXK6WHig==", + "cpu": [ + "x64" + ], "dev": true, - "license": "MIT", - "bin": { - "json5": "lib/cli.js" + "license": "MPL-2.0", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">= 12.0.0" }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lightningcss-linux-arm-gnueabihf": { + "version": "1.32.0", + "resolved": "https://registry.npmjs.org/lightningcss-linux-arm-gnueabihf/-/lightningcss-linux-arm-gnueabihf-1.32.0.tgz", + "integrity": "sha512-x6rnnpRa2GL0zQOkt6rts3YDPzduLpWvwAF6EMhXFVZXD4tPrBkEFqzGowzCsIWsPjqSK+tyNEODUBXeeVHSkw==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MPL-2.0", + "optional": true, + "os": [ + "linux" + ], "engines": { - "node": ">=6" + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" } }, - "node_modules/lodash": { - "version": "4.17.23", - "resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.23.tgz", - "integrity": "sha512-LgVTMpQtIopCi79SJeDiP0TfWi5CNEc/L/aRdTh3yIvmZXTnheWpKjSZhnvMl8iXbC1tFg9gdHHDMLoV7CnG+w==", + "node_modules/lightningcss-linux-arm64-gnu": { + "version": "1.32.0", + "resolved": "https://registry.npmjs.org/lightningcss-linux-arm64-gnu/-/lightningcss-linux-arm64-gnu-1.32.0.tgz", + "integrity": "sha512-0nnMyoyOLRJXfbMOilaSRcLH3Jw5z9HDNGfT/gwCPgaDjnx0i8w7vBzFLFR1f6CMLKF8gVbebmkUN3fa/kQJpQ==", + "cpu": [ + "arm64" + ], "dev": true, - "license": "MIT" + "license": "MPL-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } }, - "node_modules/loose-envify": { - "version": "1.4.0", - "resolved": "https://registry.npmjs.org/loose-envify/-/loose-envify-1.4.0.tgz", - "integrity": "sha512-lyuxPGr/Wfhrlem2CL/UcnUc1zcqKAImBDzukY7Y5F/yQiNdko6+fRLevlw1HgMySw7f611UIY408EtxRSoK3Q==", - "license": "MIT", - "dependencies": { - "js-tokens": "^3.0.0 || ^4.0.0" + "node_modules/lightningcss-linux-arm64-musl": { + "version": "1.32.0", + "resolved": "https://registry.npmjs.org/lightningcss-linux-arm64-musl/-/lightningcss-linux-arm64-musl-1.32.0.tgz", + "integrity": "sha512-UpQkoenr4UJEzgVIYpI80lDFvRmPVg6oqboNHfoH4CQIfNA+HOrZ7Mo7KZP02dC6LjghPQJeBsvXhJod/wnIBg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MPL-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">= 12.0.0" }, - "bin": { - "loose-envify": "cli.js" + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" } }, - "node_modules/lru-cache": { - "version": "5.1.1", - "resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-5.1.1.tgz", - "integrity": "sha512-KpNARQA3Iwv+jTA0utUVVbrh+Jlrr1Fv0e56GGzAFOXN7dk/FviaDW8LHmK52DlcH4WP2n6gI8vN1aesBFgo9w==", + "node_modules/lightningcss-linux-x64-gnu": { + "version": "1.32.0", + "resolved": "https://registry.npmjs.org/lightningcss-linux-x64-gnu/-/lightningcss-linux-x64-gnu-1.32.0.tgz", + "integrity": "sha512-V7Qr52IhZmdKPVr+Vtw8o+WLsQJYCTd8loIfpDaMRWGUZfBOYEJeyJIkqGIDMZPwPx24pUMfwSxxI8phr/MbOA==", + "cpu": [ + "x64" + ], "dev": true, - "license": "ISC", - "dependencies": { - "yallist": "^3.0.2" + "license": "MPL-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lightningcss-linux-x64-musl": { + "version": "1.32.0", + "resolved": "https://registry.npmjs.org/lightningcss-linux-x64-musl/-/lightningcss-linux-x64-musl-1.32.0.tgz", + "integrity": "sha512-bYcLp+Vb0awsiXg/80uCRezCYHNg1/l3mt0gzHnWV9XP1W5sKa5/TCdGWaR/zBM2PeF/HbsQv/j2URNOiVuxWg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MPL-2.0", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lightningcss-win32-arm64-msvc": { + "version": "1.32.0", + "resolved": "https://registry.npmjs.org/lightningcss-win32-arm64-msvc/-/lightningcss-win32-arm64-msvc-1.32.0.tgz", + "integrity": "sha512-8SbC8BR40pS6baCM8sbtYDSwEVQd4JlFTOlaD3gWGHfThTcABnNDBda6eTZeqbofalIJhFx0qKzgHJmcPTnGdw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MPL-2.0", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" + } + }, + "node_modules/lightningcss-win32-x64-msvc": { + "version": "1.32.0", + "resolved": "https://registry.npmjs.org/lightningcss-win32-x64-msvc/-/lightningcss-win32-x64-msvc-1.32.0.tgz", + "integrity": "sha512-Amq9B/SoZYdDi1kFrojnoqPLxYhQ4Wo5XiL8EVJrVsB8ARoC1PWW6VGtT0WKCemjy8aC+louJnjS7U18x3b06Q==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MPL-2.0", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">= 12.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/parcel" } }, "node_modules/lucide-react": { - "version": "0.563.0", - "resolved": "https://registry.npmjs.org/lucide-react/-/lucide-react-0.563.0.tgz", - "integrity": "sha512-8dXPB2GI4dI8jV4MgUDGBeLdGk8ekfqVZ0BdLcrRzocGgG75ltNEmWS+gE7uokKF/0oSUuczNDT+g9hFJ23FkA==", + "version": "1.11.0", + "resolved": "https://registry.npmjs.org/lucide-react/-/lucide-react-1.11.0.tgz", + "integrity": "sha512-UOhjdztXCgdBReRcIhsvz2siIBogfv/lhJEIViCpLt924dO+GDms9T7DNoucI23s6kEPpe988m5N0D2ajnzb2g==", "license": "ISC", "peerDependencies": { "react": "^16.5.1 || ^17.0.0 || ^18.0.0 || ^19.0.0" } }, - "node_modules/ms": { - "version": "2.1.3", - "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", - "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", - "dev": true, - "license": "MIT" - }, "node_modules/nanoid": { "version": "3.3.11", "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.11.tgz", @@ -1793,13 +966,6 @@ "node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1" } }, - "node_modules/node-releases": { - "version": "2.0.36", - "resolved": "https://registry.npmjs.org/node-releases/-/node-releases-2.0.36.tgz", - "integrity": "sha512-TdC8FSgHz8Mwtw9g5L4gR/Sh9XhSP/0DEkQxfEFXOpiul5IiHgHan2VhYYb6agDSfp4KuvltmGApc8HMgUrIkA==", - "dev": true, - "license": "MIT" - }, "node_modules/picocolors": { "version": "1.1.1", "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz", @@ -1821,9 +987,9 @@ } }, "node_modules/postcss": { - "version": "8.5.8", - "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.8.tgz", - "integrity": "sha512-OW/rX8O/jXnm82Ey1k44pObPtdblfiuWnrd8X7GJ7emImCOstunGbXUpp7HdBrFQX6rJzn3sPT397Wp5aCwCHg==", + "version": "8.5.12", + "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.12.tgz", + "integrity": "sha512-W62t/Se6rA0Az3DfCL0AqJwXuKwBeYg6nOaIgzP+xZ7N5BFCI7DYi1qs6ygUYT6rvfi6t9k65UMLJC+PHZpDAA==", "dev": true, "funding": [ { @@ -1850,13 +1016,10 @@ } }, "node_modules/react": { - "version": "18.3.1", - "resolved": "https://registry.npmjs.org/react/-/react-18.3.1.tgz", - "integrity": "sha512-wS+hAgJShR0KhEvPJArfuPVN1+Hz1t0Y6n5jLrGQbkb4urgPE/0Rve+1kMB1v/oWgHgm4WIcV+i7F2pTVj+2iQ==", + "version": "19.2.5", + "resolved": "https://registry.npmjs.org/react/-/react-19.2.5.tgz", + "integrity": "sha512-llUJLzz1zTUBrskt2pwZgLq59AemifIftw4aB7JxOqf1HY2FDaGDxgwpAPVzHU1kdWabH7FauP4i1oEeer2WCA==", "license": "MIT", - "dependencies": { - "loose-envify": "^1.1.0" - }, "engines": { "node": ">=0.10.0" } @@ -1872,26 +1035,15 @@ } }, "node_modules/react-dom": { - "version": "18.3.1", - "resolved": "https://registry.npmjs.org/react-dom/-/react-dom-18.3.1.tgz", - "integrity": "sha512-5m4nQKp+rZRb09LNH59GM4BxTh9251/ylbKIbpe7TpGxfJ+9kv6BLkLBXIjjspbgbnIBNqlI23tRnTWT0snUIw==", + "version": "19.2.5", + "resolved": "https://registry.npmjs.org/react-dom/-/react-dom-19.2.5.tgz", + "integrity": "sha512-J5bAZz+DXMMwW/wV3xzKke59Af6CHY7G4uYLN1OvBcKEsWOs4pQExj86BBKamxl/Ik5bx9whOrvBlSDfWzgSag==", "license": "MIT", "dependencies": { - "loose-envify": "^1.1.0", - "scheduler": "^0.23.2" + "scheduler": "^0.27.0" }, "peerDependencies": { - "react": "^18.3.1" - } - }, - "node_modules/react-refresh": { - "version": "0.17.0", - "resolved": "https://registry.npmjs.org/react-refresh/-/react-refresh-0.17.0.tgz", - "integrity": "sha512-z6F7K9bV85EfseRCp2bzrpyQ0Gkw1uLoCel9XBVWPg/TjRj94SkJzUTGfOa4bs7iJvBWtQG0Wq7wnI0syw3EBQ==", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=0.10.0" + "react": "^19.2.5" } }, "node_modules/require-directory": { @@ -1904,50 +1056,46 @@ "node": ">=0.10.0" } }, - "node_modules/rollup": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/rollup/-/rollup-4.60.0.tgz", - "integrity": "sha512-yqjxruMGBQJ2gG4HtjZtAfXArHomazDHoFwFFmZZl0r7Pdo7qCIXKqKHZc8yeoMgzJJ+pO6pEEHa+V7uzWlrAQ==", + "node_modules/rolldown": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.0.0-rc.17.tgz", + "integrity": "sha512-ZrT53oAKrtA4+YtBWPQbtPOxIbVDbxT0orcYERKd63VJTF13zPcgXTvD4843L8pcsI7M6MErt8QtON6lrB9tyA==", "dev": true, "license": "MIT", "dependencies": { - "@types/estree": "1.0.8" + "@oxc-project/types": "=0.127.0", + "@rolldown/pluginutils": "1.0.0-rc.17" }, "bin": { - "rollup": "dist/bin/rollup" + "rolldown": "bin/cli.mjs" }, "engines": { - "node": ">=18.0.0", - "npm": ">=8.0.0" + "node": "^20.19.0 || >=22.12.0" }, "optionalDependencies": { - "@rollup/rollup-android-arm-eabi": "4.60.0", - "@rollup/rollup-android-arm64": "4.60.0", - "@rollup/rollup-darwin-arm64": "4.60.0", - "@rollup/rollup-darwin-x64": "4.60.0", - "@rollup/rollup-freebsd-arm64": "4.60.0", - "@rollup/rollup-freebsd-x64": "4.60.0", - "@rollup/rollup-linux-arm-gnueabihf": "4.60.0", - "@rollup/rollup-linux-arm-musleabihf": "4.60.0", - "@rollup/rollup-linux-arm64-gnu": "4.60.0", - "@rollup/rollup-linux-arm64-musl": "4.60.0", - "@rollup/rollup-linux-loong64-gnu": "4.60.0", - "@rollup/rollup-linux-loong64-musl": "4.60.0", - "@rollup/rollup-linux-ppc64-gnu": "4.60.0", - "@rollup/rollup-linux-ppc64-musl": "4.60.0", - "@rollup/rollup-linux-riscv64-gnu": "4.60.0", - "@rollup/rollup-linux-riscv64-musl": "4.60.0", - "@rollup/rollup-linux-s390x-gnu": "4.60.0", - "@rollup/rollup-linux-x64-gnu": "4.60.0", - "@rollup/rollup-linux-x64-musl": "4.60.0", - "@rollup/rollup-openbsd-x64": "4.60.0", - "@rollup/rollup-openharmony-arm64": "4.60.0", - "@rollup/rollup-win32-arm64-msvc": "4.60.0", - "@rollup/rollup-win32-ia32-msvc": "4.60.0", - "@rollup/rollup-win32-x64-gnu": "4.60.0", - "@rollup/rollup-win32-x64-msvc": "4.60.0", - "fsevents": "~2.3.2" - } + "@rolldown/binding-android-arm64": "1.0.0-rc.17", + "@rolldown/binding-darwin-arm64": "1.0.0-rc.17", + "@rolldown/binding-darwin-x64": "1.0.0-rc.17", + "@rolldown/binding-freebsd-x64": "1.0.0-rc.17", + "@rolldown/binding-linux-arm-gnueabihf": "1.0.0-rc.17", + "@rolldown/binding-linux-arm64-gnu": "1.0.0-rc.17", + "@rolldown/binding-linux-arm64-musl": "1.0.0-rc.17", + "@rolldown/binding-linux-ppc64-gnu": "1.0.0-rc.17", + "@rolldown/binding-linux-s390x-gnu": "1.0.0-rc.17", + "@rolldown/binding-linux-x64-gnu": "1.0.0-rc.17", + "@rolldown/binding-linux-x64-musl": "1.0.0-rc.17", + "@rolldown/binding-openharmony-arm64": "1.0.0-rc.17", + "@rolldown/binding-wasm32-wasi": "1.0.0-rc.17", + "@rolldown/binding-win32-arm64-msvc": "1.0.0-rc.17", + "@rolldown/binding-win32-x64-msvc": "1.0.0-rc.17" + } + }, + "node_modules/rolldown/node_modules/@rolldown/pluginutils": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.0-rc.17.tgz", + "integrity": "sha512-n8iosDOt6Ig1UhJ2AYqoIhHWh/isz0xpicHTzpKBeotdVsTEcxsSA/i3EVM7gQAj0rU27OLAxCjzlj15IWY7bg==", + "dev": true, + "license": "MIT" }, "node_modules/rxjs": { "version": "7.8.2", @@ -1960,23 +1108,10 @@ } }, "node_modules/scheduler": { - "version": "0.23.2", - "resolved": "https://registry.npmjs.org/scheduler/-/scheduler-0.23.2.tgz", - "integrity": "sha512-UOShsPwz7NrMUqhR6t0hWjFduvOzbtv7toDH1/hIrfRNIDBnnBWd0CwJTGvTpngVlmwGCdP9/Zl/tVrDqcuYzQ==", - "license": "MIT", - "dependencies": { - "loose-envify": "^1.1.0" - } - }, - "node_modules/semver": { - "version": "6.3.1", - "resolved": "https://registry.npmjs.org/semver/-/semver-6.3.1.tgz", - "integrity": "sha512-BR7VvDCVHO+q2xBEWskxS6DJE1qRnb7DxzUrogb71CWoSficBxYsiAGd+Kl0mmq/MprG9yArRkyrQxTO6XjMzA==", - "dev": true, - "license": "ISC", - "bin": { - "semver": "bin/semver.js" - } + "version": "0.27.0", + "resolved": "https://registry.npmjs.org/scheduler/-/scheduler-0.27.0.tgz", + "integrity": "sha512-eNv+WrVbKu1f3vbYJT/xtiF5syA5HPIMtf9IgY/nKg0sWqzAUEvqY/xm7OcZc/qafLx/iO9FgOmeSAp4v5ti/Q==", + "license": "MIT" }, "node_modules/shell-quote": { "version": "1.8.3", @@ -2001,12 +1136,6 @@ "node": ">=0.10.0" } }, - "node_modules/spawn-command": { - "version": "0.0.2", - "resolved": "https://registry.npmjs.org/spawn-command/-/spawn-command-0.0.2.tgz", - "integrity": "sha512-zC8zGoGkmc8J9ndvml8Xksr1Amk9qBujgbF0JAIWO7kXr43w0h/0GJNM/Vustixu+YE8N/MTrQ7N31FvHUACxQ==", - "dev": true - }, "node_modules/string-width": { "version": "4.2.3", "resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz", @@ -2052,14 +1181,14 @@ } }, "node_modules/tinyglobby": { - "version": "0.2.15", - "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.15.tgz", - "integrity": "sha512-j2Zq4NyQYG5XMST4cbs02Ak8iJUdxRM0XI5QyxXuZOzKOINmWurp3smXu3y5wDcJrptwpSjgXHzIQxR0omXljQ==", + "version": "0.2.16", + "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.16.tgz", + "integrity": "sha512-pn99VhoACYR8nFHhxqix+uvsbXineAasWm5ojXoN8xEwK5Kd3/TrhNn1wByuD52UxWRLy8pu+kRMniEi6Eq9Zg==", "dev": true, "license": "MIT", "dependencies": { "fdir": "^6.5.0", - "picomatch": "^4.0.3" + "picomatch": "^4.0.4" }, "engines": { "node": ">=12.0.0" @@ -2085,56 +1214,24 @@ "dev": true, "license": "0BSD" }, - "node_modules/update-browserslist-db": { - "version": "1.2.3", - "resolved": "https://registry.npmjs.org/update-browserslist-db/-/update-browserslist-db-1.2.3.tgz", - "integrity": "sha512-Js0m9cx+qOgDxo0eMiFGEueWztz+d4+M3rGlmKPT+T4IS/jP4ylw3Nwpu6cpTTP8R1MAC1kF4VbdLt3ARf209w==", - "dev": true, - "funding": [ - { - "type": "opencollective", - "url": "https://opencollective.com/browserslist" - }, - { - "type": "tidelift", - "url": "https://tidelift.com/funding/github/npm/browserslist" - }, - { - "type": "github", - "url": "https://github.com/sponsors/ai" - } - ], - "license": "MIT", - "dependencies": { - "escalade": "^3.2.0", - "picocolors": "^1.1.1" - }, - "bin": { - "update-browserslist-db": "cli.js" - }, - "peerDependencies": { - "browserslist": ">= 4.21.0" - } - }, "node_modules/vite": { - "version": "6.4.2", - "resolved": "https://registry.npmjs.org/vite/-/vite-6.4.2.tgz", - "integrity": "sha512-2N/55r4JDJ4gdrCvGgINMy+HH3iRpNIz8K6SFwVsA+JbQScLiC+clmAxBgwiSPgcG9U15QmvqCGWzMbqda5zGQ==", + "version": "8.0.10", + "resolved": "https://registry.npmjs.org/vite/-/vite-8.0.10.tgz", + "integrity": "sha512-rZuUu9j6J5uotLDs+cAA4O5H4K1SfPliUlQwqa6YEwSrWDZzP4rhm00oJR5snMewjxF5V/K3D4kctsUTsIU9Mw==", "dev": true, "license": "MIT", "dependencies": { - "esbuild": "^0.25.0", - "fdir": "^6.4.4", - "picomatch": "^4.0.2", - "postcss": "^8.5.3", - "rollup": "^4.34.9", - "tinyglobby": "^0.2.13" + "lightningcss": "^1.32.0", + "picomatch": "^4.0.4", + "postcss": "^8.5.10", + "rolldown": "1.0.0-rc.17", + "tinyglobby": "^0.2.16" }, "bin": { "vite": "bin/vite.js" }, "engines": { - "node": "^18.0.0 || ^20.0.0 || >=22.0.0" + "node": "^20.19.0 || >=22.12.0" }, "funding": { "url": "https://github.com/vitejs/vite?sponsor=1" @@ -2143,14 +1240,15 @@ "fsevents": "~2.3.3" }, "peerDependencies": { - "@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0", + "@types/node": "^20.19.0 || >=22.12.0", + "@vitejs/devtools": "^0.1.0", + "esbuild": "^0.27.0 || ^0.28.0", "jiti": ">=1.21.0", - "less": "*", - "lightningcss": "^1.21.0", - "sass": "*", - "sass-embedded": "*", - "stylus": "*", - "sugarss": "*", + "less": "^4.0.0", + "sass": "^1.70.0", + "sass-embedded": "^1.70.0", + "stylus": ">=0.54.8", + "sugarss": "^5.0.0", "terser": "^5.16.0", "tsx": "^4.8.1", "yaml": "^2.4.2" @@ -2159,13 +1257,16 @@ "@types/node": { "optional": true }, - "jiti": { + "@vitejs/devtools": { "optional": true }, - "less": { + "esbuild": { + "optional": true + }, + "jiti": { "optional": true }, - "lightningcss": { + "less": { "optional": true }, "sass": { @@ -2219,13 +1320,6 @@ "node": ">=10" } }, - "node_modules/yallist": { - "version": "3.1.1", - "resolved": "https://registry.npmjs.org/yallist/-/yallist-3.1.1.tgz", - "integrity": "sha512-a4UGQaWPH59mOXUYnAG2ewncQS4i4F43Tv3JoAM+s2VDAmS9NsK8GpDMLrCHPksFT7h3K6TOoUNn2pb7RoXx4g==", - "dev": true, - "license": "ISC" - }, "node_modules/yargs": { "version": "17.7.2", "resolved": "https://registry.npmjs.org/yargs/-/yargs-17.7.2.tgz", diff --git a/benchmarks-website/package.json b/benchmarks-website/package.json index 9c1bcc623d2..879aea5857c 100644 --- a/benchmarks-website/package.json +++ b/benchmarks-website/package.json @@ -13,20 +13,20 @@ "node": ">=18.0.0" }, "dependencies": { - "chart.js": "^4.4.4", - "chartjs-plugin-zoom": "^2.0.1", + "chart.js": "^4.5.1", + "chartjs-plugin-zoom": "^2.2.0", "downsample": "^1.4.0", "hammerjs": "^2.0.8", - "lucide-react": "^0.563.0", - "react": "^18.3.1", - "react-chartjs-2": "^5.2.0", - "react-dom": "^18.3.1" + "lucide-react": "^1.11.0", + "react": "^19.2.5", + "react-chartjs-2": "^5.3.1", + "react-dom": "^19.2.5" }, "devDependencies": { - "@types/react": "^18.3.3", - "@types/react-dom": "^18.3.0", - "@vitejs/plugin-react": "^4.3.1", - "concurrently": "^8.2.2", - "vite": "^6.0.0" + "@types/react": "^19.2.14", + "@types/react-dom": "^19.2.3", + "@vitejs/plugin-react": "^6.0.1", + "concurrently": "^9.2.1", + "vite": "^8.0.10" } } diff --git a/benchmarks/compress-bench/Cargo.toml b/benchmarks/compress-bench/Cargo.toml index 4c72a20f712..baff1daad74 100644 --- a/benchmarks/compress-bench/Cargo.toml +++ b/benchmarks/compress-bench/Cargo.toml @@ -34,6 +34,7 @@ vortex-bench = { workspace = true } [features] lance = ["dep:lance-bench"] +unstable_encodings = ["vortex/unstable_encodings"] [lints] workspace = true diff --git a/benchmarks/compress-bench/src/lib.rs b/benchmarks/compress-bench/src/lib.rs index 3705343fe5a..b1692c40975 100644 --- a/benchmarks/compress-bench/src/lib.rs +++ b/benchmarks/compress-bench/src/lib.rs @@ -1,33 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -use std::sync::Arc; - -use ::vortex::array::arrays::ChunkedArray; -use ::vortex::array::arrays::chunked::ChunkedArrayExt; -use ::vortex::array::arrays::listview::recursive_list_from_list_view; -use arrow_array::RecordBatch; -use arrow_schema::Schema; #[cfg(feature = "lance")] pub use lance_bench::compress::LanceCompressor; pub mod parquet; pub mod vortex; - -pub fn chunked_to_vec_record_batch( - chunked: ChunkedArray, -) -> anyhow::Result<(Vec, Arc)> { - assert!(chunked.nchunks() > 0, "empty chunks"); - - let batches = chunked - .iter_chunks() - .map(|array| { - // TODO(connor)[ListView]: The rust Parquet implementation does not support writing - // `ListView` to Parquet files yet. - let converted_array = recursive_list_from_list_view(array.clone())?; - Ok(RecordBatch::try_from(&converted_array)?) - }) - .collect::>>()?; - - let schema = batches[0].schema(); - Ok((batches, schema)) -} diff --git a/benchmarks/compress-bench/src/main.rs b/benchmarks/compress-bench/src/main.rs index 26b71f1abe3..4800d3d6772 100644 --- a/benchmarks/compress-bench/src/main.rs +++ b/benchmarks/compress-bench/src/main.rs @@ -15,6 +15,7 @@ use regex::Regex; use vortex::utils::aliases::hash_map::HashMap; use vortex_bench::Engine; use vortex_bench::Format; +use vortex_bench::LogFormat; use vortex_bench::Target; use vortex_bench::compress::CompressMeasurements; use vortex_bench::compress::CompressOp; @@ -39,7 +40,7 @@ use vortex_bench::public_bi::PBIDataset::CMSprovider; use vortex_bench::public_bi::PBIDataset::Euro2016; use vortex_bench::public_bi::PBIDataset::Food; use vortex_bench::public_bi::PBIDataset::HashTags; -use vortex_bench::setup_logging_and_tracing; +use vortex_bench::setup_logging_and_tracing_with_format; #[derive(Parser, Debug)] #[command(version, about, long_about = None)] @@ -69,13 +70,17 @@ struct Args { output_path: Option, #[arg(long)] tracing: bool, + /// Format for the primary stderr log sink. `text` is the default human-readable format; + /// `json` emits one JSON object per event, suitable for piping into `jq`. + #[arg(long, value_enum, default_value_t = LogFormat::Text)] + log_format: LogFormat, } #[tokio::main] async fn main() -> anyhow::Result<()> { let args = Args::parse(); - setup_logging_and_tracing(args.verbose, args.tracing)?; + setup_logging_and_tracing_with_format(args.verbose, args.tracing, args.log_format)?; run_compress( args.iterations, diff --git a/benchmarks/datafusion-bench/Cargo.toml b/benchmarks/datafusion-bench/Cargo.toml index eea6e70e8aa..d0015dd2995 100644 --- a/benchmarks/datafusion-bench/Cargo.toml +++ b/benchmarks/datafusion-bench/Cargo.toml @@ -46,6 +46,7 @@ custom-labels = { workspace = true } [features] cuda = ["dep:vortex-cuda"] +unstable_encodings = ["vortex/unstable_encodings"] [lints] workspace = true diff --git a/benchmarks/datafusion-bench/src/lib.rs b/benchmarks/datafusion-bench/src/lib.rs index 94ea053ba8f..469c9be6177 100644 --- a/benchmarks/datafusion-bench/src/lib.rs +++ b/benchmarks/datafusion-bench/src/lib.rs @@ -30,7 +30,7 @@ use vortex_datafusion::VortexFormat; use vortex_datafusion::VortexFormatFactory; use vortex_datafusion::VortexTableOptions; -#[allow(clippy::expect_used)] +#[expect(clippy::expect_used)] pub fn get_session_context() -> SessionContext { let mut rt_builder = RuntimeEnvBuilder::new(); diff --git a/benchmarks/datafusion-bench/src/main.rs b/benchmarks/datafusion-bench/src/main.rs index 8d85dd8f4eb..745d8371303 100644 --- a/benchmarks/datafusion-bench/src/main.rs +++ b/benchmarks/datafusion-bench/src/main.rs @@ -94,6 +94,9 @@ struct Args { #[arg(long, default_value_t = false)] track_memory: bool, + #[arg(long, default_value = "unknown")] + runner: String, + #[arg(long, default_value_t = false)] explain: bool, @@ -149,6 +152,7 @@ async fn main() -> anyhow::Result<()> { let mut runner = SqlBenchmarkRunner::new( &*benchmark, Engine::DataFusion, + args.runner.clone(), args.formats.clone(), args.track_memory, args.hide_progress_bar, @@ -156,7 +160,7 @@ async fn main() -> anyhow::Result<()> { // Collect execution plans for metrics if show_metrics is enabled // Structure: (query_idx, format, execution_plan) - #[allow(clippy::type_complexity)] + #[expect(clippy::type_complexity)] let collected_plans: Arc)>>> = Arc::new(Mutex::new(Vec::new())); let show_metrics = args.show_metrics; @@ -317,8 +321,7 @@ async fn register_v2_tables( }; let multi_ds = MultiFileDataSource::new(SESSION.clone()) - .with_filesystem(fs) - .with_glob(glob_pattern) + .with_glob(glob_pattern, Some(fs)) .build() .await?; diff --git a/benchmarks/datafusion-bench/src/metrics.rs b/benchmarks/datafusion-bench/src/metrics.rs index bb29d447469..9f41d75398b 100644 --- a/benchmarks/datafusion-bench/src/metrics.rs +++ b/benchmarks/datafusion-bench/src/metrics.rs @@ -47,7 +47,7 @@ impl MetricsSetExt for MetricsSet { } } - #[allow(clippy::disallowed_types)] + #[expect(clippy::disallowed_types)] fn aggregate(&self) -> Self { use std::collections::HashMap; let mut map = HashMap::new(); diff --git a/benchmarks/duckdb-bench/Cargo.toml b/benchmarks/duckdb-bench/Cargo.toml index ad9ece90fc3..b70375cf963 100644 --- a/benchmarks/duckdb-bench/Cargo.toml +++ b/benchmarks/duckdb-bench/Cargo.toml @@ -27,6 +27,7 @@ vortex-duckdb = { workspace = true } [features] cuda = ["dep:vortex-cuda"] +unstable_encodings = ["vortex/unstable_encodings"] [lints] workspace = true diff --git a/benchmarks/duckdb-bench/build.rs b/benchmarks/duckdb-bench/build.rs index 4232c385bbb..7e2b12e5d18 100644 --- a/benchmarks/duckdb-bench/build.rs +++ b/benchmarks/duckdb-bench/build.rs @@ -1,8 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] -#![allow(clippy::expect_used)] +#![expect(clippy::unwrap_used)] /// Adds a dynamic linker runtime path pointing to the DuckDB dylib dir. /// diff --git a/benchmarks/duckdb-bench/src/main.rs b/benchmarks/duckdb-bench/src/main.rs index 7ab8f1ac7ab..d8a3306b224 100644 --- a/benchmarks/duckdb-bench/src/main.rs +++ b/benchmarks/duckdb-bench/src/main.rs @@ -64,6 +64,9 @@ struct Args { #[arg(long, default_value_t = false)] hide_progress_bar: bool, + #[arg(long, default_value = "unknown")] + runner: String, + #[arg(long, value_delimiter = ',', value_parser = value_parser!(Format))] formats: Vec, @@ -142,6 +145,7 @@ fn main() -> anyhow::Result<()> { let mut runner = SqlBenchmarkRunner::new( &*benchmark, Engine::DuckDB, + args.runner.clone(), args.formats.clone(), args.track_memory, args.hide_progress_bar, diff --git a/benchmarks/lance-bench/Cargo.toml b/benchmarks/lance-bench/Cargo.toml index acfe39ef531..775b051dd9c 100644 --- a/benchmarks/lance-bench/Cargo.toml +++ b/benchmarks/lance-bench/Cargo.toml @@ -15,8 +15,8 @@ version.workspace = true publish = false [dependencies] -lance = { version = "2.0.0", default-features = false } -lance-encoding = { version = "2.0.0" } +lance = { version = "4", default-features = false } +lance-encoding = { version = "4" } anyhow = { workspace = true } arrow-cast = { version = "57" } @@ -29,5 +29,8 @@ tokio = { workspace = true, features = ["full"] } tracing = { workspace = true } vortex-bench = { workspace = true } +[features] +unstable_encodings = ["vortex-bench/unstable_encodings"] + [lints] workspace = true diff --git a/benchmarks/lance-bench/src/main.rs b/benchmarks/lance-bench/src/main.rs index 73fa8426ffe..6cce97d2548 100644 --- a/benchmarks/lance-bench/src/main.rs +++ b/benchmarks/lance-bench/src/main.rs @@ -65,6 +65,9 @@ struct Args { #[arg(long, default_value_t = false)] track_memory: bool, + #[arg(long, default_value = "unknown")] + runner: String, + #[arg(long = "opt", value_delimiter = ',', value_parser = value_parser!(Opt))] options: Vec, } @@ -93,6 +96,7 @@ async fn main() -> anyhow::Result<()> { let mut runner = SqlBenchmarkRunner::new( &*benchmark, Engine::DataFusion, + args.runner.clone(), vec![Format::Lance], args.track_memory, args.hide_progress_bar, diff --git a/benchmarks/random-access-bench/Cargo.toml b/benchmarks/random-access-bench/Cargo.toml index c22736b65d4..cde8fb6f81a 100644 --- a/benchmarks/random-access-bench/Cargo.toml +++ b/benchmarks/random-access-bench/Cargo.toml @@ -26,6 +26,7 @@ vortex-bench = { workspace = true } [features] lance = ["dep:lance-bench"] +unstable_encodings = ["vortex-bench/unstable_encodings"] [lints] workspace = true diff --git a/benchmarks/random-access-bench/src/main.rs b/benchmarks/random-access-bench/src/main.rs index 2fb617e8787..852332528a8 100644 --- a/benchmarks/random-access-bench/src/main.rs +++ b/benchmarks/random-access-bench/src/main.rs @@ -91,14 +91,14 @@ fn generate_indices(dataset: &dyn BenchDataset, pattern: AccessPattern) -> Vec 0, row_count > 0). - #[allow(clippy::unwrap_used)] + #[expect(clippy::unwrap_used)] let exp = Exp::new(rate).unwrap(); let mut indices = Vec::with_capacity(POISSON_EXPECTED_COUNT); let mut pos = 0.0_f64; loop { let gap: f64 = exp.sample(&mut rng); pos += gap; - #[allow(clippy::cast_possible_truncation)] + #[expect(clippy::cast_possible_truncation)] let idx = pos as u64; if idx >= row_count { break; diff --git a/benchmarks/vector-search-bench/Cargo.toml b/benchmarks/vector-search-bench/Cargo.toml new file mode 100644 index 00000000000..126b62d5e2d --- /dev/null +++ b/benchmarks/vector-search-bench/Cargo.toml @@ -0,0 +1,40 @@ +[package] +name = "vector-search-bench" +description = "Vector similarity search benchmarks for Vortex on public embedding datasets" +authors.workspace = true +categories.workspace = true +edition.workspace = true +homepage.workspace = true +include.workspace = true +keywords.workspace = true +license.workspace = true +readme.workspace = true +repository.workspace = true +rust-version.workspace = true +version.workspace = true +publish = false + +[dependencies] +anyhow = { workspace = true } +arrow-array = { workspace = true } +arrow-buffer = { workspace = true } +arrow-schema = { workspace = true } +clap = { workspace = true, features = ["derive"] } +futures = { workspace = true } +indicatif = { workspace = true } +parquet = { workspace = true, features = ["async"] } +rand = { workspace = true } +serde = { workspace = true, features = ["derive"] } +tabled = { workspace = true, features = ["std"] } +tokio = { workspace = true, features = ["full"] } +tracing = { workspace = true } +vortex = { workspace = true, features = ["files", "tokio", "unstable_encodings"] } +vortex-bench = { workspace = true, features = ["unstable_encodings"] } +vortex-btrblocks = { workspace = true, features = ["unstable_encodings"] } +vortex-tensor = { workspace = true } + +[dev-dependencies] +tempfile = { workspace = true } + +[lints] +workspace = true diff --git a/benchmarks/vector-search-bench/src/compression.rs b/benchmarks/vector-search-bench/src/compression.rs new file mode 100644 index 00000000000..9b40cb15544 --- /dev/null +++ b/benchmarks/vector-search-bench/src/compression.rs @@ -0,0 +1,110 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Vector compression flavors exercised by the benchmark. +//! +//! Each [`VectorFlavor`] variant maps to a [`vortex::file::WriteStrategyBuilder`] configuration +//! applied to the same input data. +//! +//! The benchmark writes one `.vortex` file per flavor per data file, then scans them all with the +//! same query so the comparison is apples-to-apples with the Parquet files. + +use clap::ValueEnum; +use vortex::array::ArrayId; +use vortex::array::scalar_fn::ScalarFnVTable; +use vortex::file::ALLOWED_ENCODINGS; +use vortex::file::VortexWriteOptions; +use vortex::file::WriteOptionsSessionExt; +use vortex::file::WriteStrategyBuilder; +use vortex::session::VortexSession; +use vortex::utils::aliases::hash_set::HashSet; +use vortex_bench::Format; +use vortex_btrblocks::BtrBlocksCompressorBuilder; +use vortex_tensor::encodings::l2_denorm::L2DenormScheme; +use vortex_tensor::scalar_fns::l2_denorm::L2Denorm; +use vortex_tensor::scalar_fns::sorf_transform::SorfTransform; + +/// Every [`VectorFlavor`] variant in CLI-help order. +pub const ALL_VECTOR_FLAVORS: &[VectorFlavor] = + &[VectorFlavor::Uncompressed, VectorFlavor::TurboQuant]; + +/// One write-side compression configuration we measure. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, ValueEnum)] +pub enum VectorFlavor { + /// `BtrBlocksCompressorBuilder::empty()` + #[clap(name = "vortex-uncompressed")] + Uncompressed, + /// `BtrBlocksCompressorBuilder::default().with_turboquant()`. + #[clap(name = "vortex-turboquant")] + TurboQuant, + // TODO(connor): We will want to add `Default` here which is just the default compressor. +} + +impl VectorFlavor { + /// Stable kebab-cased label used in CLI args and metric names. + pub fn label(&self) -> &'static str { + match self { + VectorFlavor::Uncompressed => "vortex-uncompressed", + VectorFlavor::TurboQuant => "vortex-turboquant", + } + } + + /// The `target.format` value emitted on measurements for this flavor. Both flavors produce + /// `.vortex` files, so the compression label carries the flavor split. + pub fn as_format(&self) -> Format { + match self { + VectorFlavor::Uncompressed => Format::OnDiskVortex, + VectorFlavor::TurboQuant => Format::OnDiskVortex, + } + } + + /// Subdirectory name under the per-dataset cache root used to store this flavor's `.vortex` + /// files. + pub fn dir_name(&self) -> &'static str { + match self { + VectorFlavor::Uncompressed => "vortex-uncompressed", + VectorFlavor::TurboQuant => "vortex-turboquant", + } + } + + /// Build the [`vortex::file::WriteStrategyBuilder`]-backed write options for this flavor. + /// + /// TurboQuant produces `L2Denorm(SorfTransform(...))` which the default file + /// `ALLOWED_ENCODINGS` set rejects on normalization — we extend the allow-list with the two + /// scalar-fn array IDs the scheme actually emits. + pub fn create_write_options(&self, session: &VortexSession) -> VortexWriteOptions { + let strategy = match self { + VectorFlavor::Uncompressed => { + // Even though this is uncompressed, we still want to denormalize the data first so + // that the results are fair. + let compressor = BtrBlocksCompressorBuilder::empty() + .with_new_scheme(&L2DenormScheme) + .build(); + + let mut allowed: HashSet = ALLOWED_ENCODINGS.clone(); + allowed.insert(L2Denorm.id()); + + WriteStrategyBuilder::default() + .with_compressor(compressor) + .with_allow_encodings(allowed) + .build() + } + VectorFlavor::TurboQuant => { + let compressor = BtrBlocksCompressorBuilder::default() + .with_turboquant() + .build(); + + let mut allowed: HashSet = ALLOWED_ENCODINGS.clone(); + allowed.insert(L2Denorm.id()); + allowed.insert(SorfTransform.id()); + + WriteStrategyBuilder::default() + .with_compressor(compressor) + .with_allow_encodings(allowed) + .build() + } + }; + + session.write_options().with_strategy(strategy) + } +} diff --git a/benchmarks/vector-search-bench/src/display.rs b/benchmarks/vector-search-bench/src/display.rs new file mode 100644 index 00000000000..a74eec46b01 --- /dev/null +++ b/benchmarks/vector-search-bench/src/display.rs @@ -0,0 +1,133 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Local table renderer for the vector-search benchmark. +//! +//! Groups columns by **flavor** (`vortex-uncompressed`, `vortex-turboquant`) rather than by +//! [`vortex_bench::Format`], because the two Vortex flavors share a single +//! `Format::OnDiskVortex`/`Format::VortexLossy` pair and the generic +//! [`vortex_bench::display::render_table`] groups by Format. Local renderer keeps the +//! column-per-flavor invariant intact without introducing a new global Format value. +//! +//! Output rows: +//! +//! ```text +//! Metric | vortex-uncompressed | vortex-turboquant +//! ------------------ + ------------------- + ----------------- +//! scan wall (mean) | 485 ms | 212 ms +//! scan wall (median) | 490 ms | 215 ms +//! matches | 42 | 39 +//! rows scanned | 10,000,000 | 10,000,000 +//! bytes scanned | 30.5 GB | 7.62 GB +//! rows / sec | 5.2e6 | 1.2e7 +//! ``` + +use std::io::Write; + +use anyhow::Result; +use tabled::settings::Style; + +use crate::compression::VectorFlavor; +use crate::prepare::CompressedVortexDataset; +use crate::scan::ScanTiming; + +/// Final column-per-flavor row set for one dataset. +pub struct DatasetReport<'a> { + pub dataset_name: &'a str, + pub vortex_results: &'a [(VectorFlavor, &'a CompressedVortexDataset, &'a ScanTiming)], +} + +/// Render the full report into the given writer as a tabled table. +pub fn render(report: &DatasetReport<'_>, writer: &mut dyn Write) -> Result<()> { + let mut headers: Vec = vec!["metric".to_owned()]; + for &(flavor, ..) in report.vortex_results { + headers.push(flavor.label().to_owned()); + } + + let rows: Vec> = vec![ + make_row("scan wall (mean)", report, |_, _, scan| { + format_duration(scan.mean) + }), + make_row("scan wall (median)", report, |_, _, scan| { + format_duration(scan.median) + }), + make_row("matches", report, |_, _, scan| scan.matches.to_string()), + make_row("rows scanned", report, |_, _, scan| { + scan.rows_scanned.to_string() + }), + make_row("bytes scanned", report, |_, _, scan| { + format_bytes(scan.bytes_scanned) + }), + make_row("rows / sec", report, |_, _, scan| { + format_throughput_rows(scan.rows_scanned, scan.mean) + }), + ]; + + writeln!(writer, "## {}", report.dataset_name)?; + let mut builder = tabled::builder::Builder::new(); + builder.push_record(headers); + for row in rows { + builder.push_record(row); + } + let mut table = builder.build(); + table.with(Style::modern()); + writeln!(writer, "{table}")?; + Ok(()) +} + +fn make_row(metric: &str, report: &DatasetReport<'_>, vortex_cell: F) -> Vec +where + F: Fn(VectorFlavor, &CompressedVortexDataset, &ScanTiming) -> String, +{ + let mut row = vec![metric.to_owned()]; + for &(flavor, prep, scan) in report.vortex_results { + row.push(vortex_cell(flavor, prep, scan)); + } + row +} + +fn format_duration(d: std::time::Duration) -> String { + let secs = d.as_secs_f64(); + if secs >= 1.0 { + format!("{secs:.2} s") + } else if secs >= 1e-3 { + format!("{:.1} ms", secs * 1e3) + } else { + format!("{:.1} µs", secs * 1e6) + } +} + +fn format_bytes(bytes: u64) -> String { + const UNITS: &[&str] = &["B", "KiB", "MiB", "GiB", "TiB"]; + let mut value = bytes as f64; + let mut unit = UNITS[0]; + for next in &UNITS[1..] { + if value < 1024.0 { + break; + } + value /= 1024.0; + unit = next; + } + if unit == "B" { + format!("{bytes} B") + } else { + format!("{value:.2} {unit}") + } +} + +fn format_throughput_rows(rows: u64, wall: std::time::Duration) -> String { + let secs = wall.as_secs_f64(); + if secs <= 0.0 { + return "—".to_owned(); + } + let rps = rows as f64 / secs; + if rps >= 1e9 { + format!("{:.2}G", rps / 1e9) + } else if rps >= 1e6 { + format!("{:.2}M", rps / 1e6) + } else if rps >= 1e3 { + format!("{:.2}K", rps / 1e3) + } else { + format!("{rps:.0}") + } +} diff --git a/benchmarks/vector-search-bench/src/expression.rs b/benchmarks/vector-search-bench/src/expression.rs new file mode 100644 index 00000000000..b1a5b1ef1c3 --- /dev/null +++ b/benchmarks/vector-search-bench/src/expression.rs @@ -0,0 +1,97 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Cosine-similarity filter [`Expression`]s used by the file-scan path. +//! +//! We can easily build a cosine similarity filter by hand: +//! +//! ```text +//! gt( +//! cosine_similarity(col("emb"), lit(query_scalar)), +//! lit(threshold), +//! ) +//! ``` +//! +//! The query is wrapped as `Scalar::extension::(Scalar::fixed_size_list(F32, ...))` so +//! [`CosineSimilarity`] can treat it as a single-row `Vector` value during evaluation. +//! +//! At scan time the literal expands into a `ConstantArray` whose row count matches the chunk batch +//! size. + +use anyhow::Result; +use vortex::array::expr::Expression; +use vortex::array::expr::col; +use vortex::array::expr::gt; +use vortex::array::expr::lit; +use vortex::array::extension::EmptyMetadata; +use vortex::array::scalar::Scalar; +use vortex::array::scalar_fn::EmptyOptions; +use vortex::array::scalar_fn::ScalarFnVTableExt; +use vortex::dtype::DType; +use vortex::dtype::Nullability; +use vortex::dtype::PType; +use vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity; +use vortex_tensor::vector::Vector; + +/// Build the filter `cosine_similarity(emb, query) > threshold`. +pub fn similarity_filter(query: &[f32], threshold: f32) -> Result { + // Empty queries short-circuit to a literal `false`, so scans return no rows instead of trying + // to evaluate cosine similarity on a zero-dimensional vector. + if query.is_empty() { + return Ok(lit(false)); + } + + let query_lit = lit(query_scalar(query)?); + let cosine = CosineSimilarity.new_expr(EmptyOptions, [col("emb"), query_lit]); + Ok(gt(cosine, lit(threshold))) +} + +/// Wrap a query vector as `Scalar::extension::(Scalar::fixed_size_list(F32, ...))`. +pub fn query_scalar(query: &[f32]) -> Result { + let children: Vec = query + .iter() + .map(|&v| Scalar::primitive(v, Nullability::NonNullable)) + .collect(); + + let element_dtype = DType::Primitive(PType::F32, Nullability::NonNullable); + let fsl = Scalar::fixed_size_list(element_dtype, children, Nullability::NonNullable); + + Ok(Scalar::extension::(EmptyMetadata, fsl)) +} + +/// Project just the `emb` column. Used by the throughput-only scan path. +pub fn emb_projection() -> Expression { + col("emb") +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn query_scalar_accepts_empty_query() { + let scalar = query_scalar(&[]).unwrap(); + match scalar.dtype() { + DType::Extension(_) => {} + other => panic!("expected Extension, got {other}"), + } + } + + #[test] + fn query_scalar_builds_extension_dtype() { + let scalar = query_scalar(&[1.0, 0.0, 0.0]).unwrap(); + match scalar.dtype() { + DType::Extension(_) => {} + other => panic!("expected Extension, got {other}"), + } + } + + #[test] + fn similarity_filter_uses_gt_operator() { + let expr = similarity_filter(&[1.0, 0.0, 0.0], 0.5).unwrap(); + // Quick sanity check: the printed form contains the operator and the threshold so + // future refactors that change the structure get caught here. + let printed = format!("{expr:?}"); + assert!(printed.contains("Gt") || printed.contains(">"), "{printed}"); + } +} diff --git a/benchmarks/vector-search-bench/src/ingest.rs b/benchmarks/vector-search-bench/src/ingest.rs new file mode 100644 index 00000000000..bab176b0482 --- /dev/null +++ b/benchmarks/vector-search-bench/src/ingest.rs @@ -0,0 +1,221 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Per-chunk ingest transform. +//! +//! Bridges the parquet record-batch stream and the Vortex file writer: +//! +//! 1. Project the `emb` column out of each struct chunk. +//! 2. Rewrap the `emb` column as `Extension>` via +//! [`vortex_bench::vector_dataset::list_to_vector_ext`]. +//! 3. Detect the FSL element ptype at runtime and cast `f64` -> `f32` when needed. Detection is +//! from the arrow schema rather than a catalog declaration so upstream parquets whose actual +//! precision disagrees with the catalog still ingest correctly. After this point all +//! downstream code (compression, scan, recall) is f32-only. +//! 4. Optionally project the `scalar_labels` column through unchanged so future filtered-search +//! benchmarks have it without re-ingest. +//! 5. Repackage as `Struct { id: i64, emb: Vector, scalar_labels: ??? }`. + +use anyhow::Context; +use anyhow::Result; +use anyhow::bail; +use anyhow::ensure; +use vortex::array::ArrayRef; +use vortex::array::ExecutionCtx; +use vortex::array::IntoArray; +use vortex::array::arrays::ExtensionArray; +use vortex::array::arrays::FixedSizeListArray; +use vortex::array::arrays::PrimitiveArray; +use vortex::array::arrays::Struct; +use vortex::array::arrays::StructArray; +use vortex::array::arrays::extension::ExtensionArrayExt; +use vortex::array::arrays::fixed_size_list::FixedSizeListArrayExt; +use vortex::array::arrays::struct_::StructArrayExt; +use vortex::array::extension::EmptyMetadata; +use vortex::array::validity::Validity; +use vortex::buffer::Buffer; +use vortex::dtype::DType; +use vortex::dtype::PType; +use vortex::dtype::extension::ExtDType; +use vortex_bench::vector_dataset::list_to_vector_ext; +use vortex_tensor::vector::AnyVector; +use vortex_tensor::vector::Vector; + +/// Apply the transform to a single struct chunk and return the rebuilt chunk. +/// +/// `chunk` must be a non-chunked `Struct { id: i64, emb: List }`, where all of the list +/// elements are +/// +/// The returned array is always a `Struct { id: i64, emb: Vector }`. +pub fn transform_chunk(chunk: ArrayRef, ctx: &mut ExecutionCtx) -> Result { + let struct_view = chunk + .as_opt::() + .with_context(|| format!("ingest: expected struct chunk, got dtype {}", chunk.dtype()))?; + + let id = struct_view + .unmasked_field_by_name("id") + .context("ingest: chunk missing `id` column")? + .clone(); + let emb = struct_view + .unmasked_field_by_name("emb") + .context("ingest: chunk missing `emb` column")? + .clone(); + + let emb_ext: ExtensionArray = list_to_vector_ext(emb)?.execute(ctx)?; + + // Detect the actual FSL element ptype from the extension storage dtype. The dataset catalog + // cannot be trusted here: at least one upstream parquet (`sift-medium-5m`) ships f64 + // embeddings despite the catalog advertising f32. + let element_ptype = { + let storage_dtype = emb_ext.storage_array().dtype(); + match storage_dtype { + DType::FixedSizeList(elem, ..) => match elem.as_ref() { + DType::Primitive(ptype, _) => *ptype, + other => bail!("ingest: expected primitive FSL element dtype, got {other}"), + }, + other => bail!("ingest: expected FSL storage dtype, got {other}"), + } + }; + + let f32_vector_array = match element_ptype { + PType::F32 => emb_ext.into_array(), + PType::F64 => convert_f64_to_f32_vectors(&emb_ext, ctx)?, + other => bail!("ingest: unsupported emb element ptype {other}, expected f32 or f64"), + }; + + let fields = [("id", id), ("emb", f32_vector_array)]; + Ok(StructArray::from_fields(&fields)?.into_array()) +} + +/// Convert a `Vector` extension array down to `Vector`. +/// +/// This conversion is lossy, but we are generally ok with this because most vector search +/// operations do not demand a high amount of precision. +fn convert_f64_to_f32_vectors(ext: &ExtensionArray, ctx: &mut ExecutionCtx) -> Result { + ensure!(ext.ext_dtype().is::()); + + let fsl: FixedSizeListArray = ext.storage_array().clone().execute(ctx)?; + let validity = fsl.validity()?; + let elements: PrimitiveArray = fsl.elements().clone().execute(ctx)?; + ensure!(elements.ptype() == PType::F64); + + let dim = match fsl.dtype() { + DType::FixedSizeList(_, dim, _) => *dim, + other => bail!("cast_vector_ext_to_f32: expected FSL dtype, got {other}"), + }; + + let f64_slice = elements.as_slice::(); + + #[expect( + clippy::cast_possible_truncation, + reason = "this is intentionally lossy" + )] + let f32_buf: Buffer = f64_slice + .iter() + .copied() + .map(|double| double as f32) + .collect(); + + let f32_elements = PrimitiveArray::new::(f32_buf, Validity::NonNullable).into_array(); + let new_fsl = FixedSizeListArray::try_new(f32_elements, dim, validity, fsl.len())?; + let ext_dtype = ExtDType::::try_new(EmptyMetadata, new_fsl.dtype().clone())?.erased(); + + Ok(ExtensionArray::new(ext_dtype, new_fsl.into_array()).into_array()) +} + +#[cfg(test)] +mod tests { + use vortex::VortexSessionDefault; + use vortex::array::VortexSessionExecute; + use vortex::array::arrays::List; + use vortex::buffer::BufferMut; + use vortex::dtype::Nullability; + use vortex::session::VortexSession; + + use super::*; + + fn list_chunk_f64(rows: &[&[f64]]) -> ArrayRef { + let mut elements = BufferMut::::with_capacity(rows.iter().map(|r| r.len()).sum()); + let mut offsets = BufferMut::::with_capacity(rows.len() + 1); + offsets.push(0); + for row in rows { + for &v in row.iter() { + elements.push(v); + } + offsets.push(i32::try_from(elements.len()).unwrap()); + } + let elements_array = + PrimitiveArray::new::(elements.freeze(), Validity::NonNullable).into_array(); + let offsets_array = + PrimitiveArray::new::(offsets.freeze(), Validity::NonNullable).into_array(); + vortex::array::Array::::new(elements_array, offsets_array, Validity::NonNullable) + .into_array() + } + + fn id_array(ids: &[i64]) -> ArrayRef { + PrimitiveArray::new::( + BufferMut::from_iter(ids.iter().copied()).freeze(), + Validity::NonNullable, + ) + .into_array() + } + + #[test] + fn f64_chunk_is_cast_to_f32() -> Result<()> { + let session = VortexSession::default(); + let mut ctx = session.create_execution_ctx(); + + let emb = list_chunk_f64(&[&[1.0, 2.0, 3.0], &[4.0, 5.0, 6.0]]); + let chunk = + StructArray::from_fields(&[("id", id_array(&[0, 1])), ("emb", emb)])?.into_array(); + let out = transform_chunk(chunk, &mut ctx)?; + let out_struct = out + .as_opt::() + .context("transform_chunk should return a Struct array")?; + let out_emb = out_struct + .unmasked_field_by_name("emb") + .context("transform_chunk output should contain an emb field")? + .clone(); + let DType::Extension(ext) = out_emb.dtype() else { + panic!("expected extension dtype, got {}", out_emb.dtype()); + }; + match ext.storage_dtype() { + DType::FixedSizeList(elem, 3, Nullability::NonNullable) => { + assert_eq!( + **elem, + DType::Primitive(PType::F32, Nullability::NonNullable) + ); + } + other => panic!("unexpected storage dtype {other}"), + } + Ok(()) + } + + #[test] + fn f32_chunk_passes_through() -> Result<()> { + let session = VortexSession::default(); + let mut ctx = session.create_execution_ctx(); + + let mut elements = BufferMut::::with_capacity(6); + for v in [1.0f32, 2.0, 3.0, 4.0, 5.0, 6.0] { + elements.push(v); + } + let mut offsets = BufferMut::::with_capacity(3); + offsets.push(0); + offsets.push(3); + offsets.push(6); + let emb = vortex::array::Array::::new( + PrimitiveArray::new::(elements.freeze(), Validity::NonNullable).into_array(), + PrimitiveArray::new::(offsets.freeze(), Validity::NonNullable).into_array(), + Validity::NonNullable, + ) + .into_array(); + let chunk = + StructArray::from_fields(&[("id", id_array(&[0, 1])), ("emb", emb)])?.into_array(); + + let out = transform_chunk(chunk, &mut ctx)?; + let out_struct = out.as_opt::().expect("returns Struct"); + assert_eq!(out_struct.len(), 2); + Ok(()) + } +} diff --git a/benchmarks/vector-search-bench/src/lib.rs b/benchmarks/vector-search-bench/src/lib.rs new file mode 100644 index 00000000000..643cbb5bd0d --- /dev/null +++ b/benchmarks/vector-search-bench/src/lib.rs @@ -0,0 +1,28 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! `vector-search-bench` vector similarity-search benchmark over several datasets. + +pub mod compression; +pub mod display; +pub mod expression; +pub mod ingest; +pub mod prepare; +pub mod query; +pub mod scan; + +use std::sync::LazyLock; + +use vortex::VortexSessionDefault; +use vortex::io::session::RuntimeSessionExt; +use vortex::session::VortexSession; + +pub static SESSION: LazyLock = LazyLock::new(|| { + // SAFETY: called from inside the LazyLock initializer, before any other access to + // `SESSION`. The first thread to dereference SESSION runs this once. + unsafe { std::env::set_var(vortex_tensor::SCALAR_FN_ARRAY_TENSOR_PLUGIN_ENV, "1") }; + + let session = VortexSession::default().with_tokio(); + vortex_tensor::initialize(&session); + session +}); diff --git a/benchmarks/vector-search-bench/src/main.rs b/benchmarks/vector-search-bench/src/main.rs new file mode 100644 index 00000000000..626e3bfce50 --- /dev/null +++ b/benchmarks/vector-search-bench/src/main.rs @@ -0,0 +1,188 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! `vector-search-bench` — on-disk cosine-similarity scan benchmark. +//! +//! ```sh +//! cargo run -p vector-search-bench --release -- \ +//! --dataset cohere-large-10m \ +//! --layout partitioned \ +//! --flavors vortex-uncompressed,vortex-turboquant \ +//! --iterations 3 \ +//! --threshold 0.8 +//! ``` + +use std::path::PathBuf; + +use anyhow::Context; +use anyhow::Result; +use clap::Parser; +use vector_search_bench::compression::ALL_VECTOR_FLAVORS; +use vector_search_bench::compression::VectorFlavor; +use vector_search_bench::display::DatasetReport; +use vector_search_bench::display::render; +use vector_search_bench::prepare::CompressedVortexDataset; +use vector_search_bench::prepare::prepare_all; +use vector_search_bench::query::get_random_query_vector; +use vector_search_bench::scan::ScanConfig; +use vector_search_bench::scan::ScanTiming; +use vector_search_bench::scan::run_search_scan; +use vortex_bench::setup_logging_and_tracing; +use vortex_bench::vector_dataset; +use vortex_bench::vector_dataset::TrainLayout; +use vortex_bench::vector_dataset::VectorDataset; + +#[derive(Parser, Debug)] +#[command(version, about, long_about = None)] +struct Args { + /// Dataset to benchmark. Single dataset per CLI invocation by design — large datasets + /// are intentionally babysat one at a time. + #[arg(long, value_enum)] + dataset: VectorDataset, + + /// Train-split layout. Required when the dataset publishes more than one layout. + /// Defaults to the catalog's first hosted layout when omitted. + #[arg(long, value_enum)] + layout: Option, + + /// Comma-separated list of flavors to run. Each Vortex flavor produces one `.vortex` file per + /// train shard. + #[arg( + long, + value_delimiter = ',', + value_enum, + default_values_t = ALL_VECTOR_FLAVORS.to_vec(), + )] + flavors: Vec, + + /// Number of timed scan iterations per flavor. Mean and median are reported. + #[arg(long, default_value_t = 5)] + iterations: usize, + + /// Cosine threshold passed to the filter expression. + #[arg(long, default_value_t = 0.85)] + threshold: f32, + + /// Seed for the test-parquet query sampler. + #[arg(long, default_value_t = 42)] + query_seed: u64, + + /// Optional path to write the rendered table to instead of stdout. + #[arg(long)] + output_path: Option, + + /// Emit verbose tracing. + #[arg(short, long)] + verbose: bool, + + /// Enable perfetto tracing output. + #[arg(long)] + tracing: bool, +} + +#[tokio::main] +async fn main() -> Result<()> { + let args = Args::parse(); + setup_logging_and_tracing(args.verbose, args.tracing)?; + + let dataset = args.dataset; + let layout = resolve_layout(dataset, args.layout)?; + tracing::info!( + "running {} on layout {} ({} dims, {} rows)", + dataset.name(), + layout, + dataset.dim(), + dataset.num_rows() + ); + + if args.flavors.is_empty() { + anyhow::bail!("no flavors selected, please pass at least one to --flavors"); + } + + // Load the source embeddings parquet files. + let datasets_paths = vector_dataset::download(dataset, layout) + .await + .with_context(|| format!("download {}", dataset.name()))?; + + // Load all vortex files needed, compressing new ones if needed. + let prepared = prepare_all(dataset, layout, &datasets_paths, &args.flavors).await?; + + let query_vector = get_random_query_vector( + &datasets_paths.test, + dataset.dim(), + dataset.element_ptype(), + args.query_seed, + ) + .await?; + tracing::info!( + "sampled query id {} (dim={})", + query_vector.id, + query_vector.query.len() + ); + + let scan_config = ScanConfig { + iterations: args.iterations, + threshold: args.threshold, + }; + + // Run all scans and record how long each takes. + let mut scan_timings: Vec = Vec::with_capacity(prepared.len()); + for prep in &prepared { + let timing = run_search_scan(prep, &query_vector.query, &scan_config).await?; + scan_timings.push(timing); + } + + // Collect the benchmark results. + let pairs: Vec<(VectorFlavor, &CompressedVortexDataset, &ScanTiming)> = prepared + .iter() + .zip(scan_timings.iter()) + .map(|(prep, scan)| (prep.flavor, prep, scan)) + .collect(); + let report = DatasetReport { + dataset_name: dataset.name(), + vortex_results: &pairs, + }; + + // Print the results. + if let Some(path) = args.output_path { + let mut file = + std::fs::File::create(&path).with_context(|| format!("create {}", path.display()))?; + render(&report, &mut file)?; + } else { + let stdout = std::io::stdout(); + let mut handle = stdout.lock(); + render(&report, &mut handle)?; + } + + Ok(()) +} + +/// Every benchmark has different sets of possible dataset layouts available. The user **must** +/// provide one if there are multiple layouts. But if a dataset only has 1 layout, we can choose +/// that for them as the default. +fn resolve_layout(dataset: VectorDataset, requested: Option) -> Result { + let layouts = dataset.layouts(); + + match requested { + Some(layout) => { + dataset.validate_layout(layout)?; + Ok(layout) + } + None => { + if layouts.len() == 1 { + Ok(layouts[0].layout()) + } else { + let allowed = layouts + .iter() + .map(|s| s.layout().label()) + .collect::>() + .join(", "); + anyhow::bail!( + "dataset {} hosts multiple layouts ([{}]): pass --layout to pick one", + dataset.name(), + allowed, + ); + } + } + } +} diff --git a/benchmarks/vector-search-bench/src/prepare.rs b/benchmarks/vector-search-bench/src/prepare.rs new file mode 100644 index 00000000000..8cf9d9860ed --- /dev/null +++ b/benchmarks/vector-search-bench/src/prepare.rs @@ -0,0 +1,227 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Per-flavor on-disk ingest. +//! +//! For each `(dataset, layout, flavor)` triple, [`prepare_flavor`] streams every parquet shard +//! and writes one `.vortex` file per shard. The pipeline is idempotent (existing `.vortex` files +//! are skipped) and reports end-to-end wall-clock time, summed input parquet bytes, and total +//! output bytes. + +use std::path::Path; +use std::path::PathBuf; + +use anyhow::Context; +use anyhow::Result; +use futures::StreamExt; +use parquet::arrow::ParquetRecordBatchStreamBuilder; +use tokio::fs::File; +use tokio::io::AsyncWriteExt; +use tracing::info; +use tracing::warn; +use vortex::array::ArrayRef; +use vortex::array::ExecutionCtx; +use vortex::array::VortexSessionExecute; +use vortex::array::stream::ArrayStreamAdapter; +use vortex::array::stream::ArrayStreamExt; +use vortex::error::VortexResult; +use vortex::error::vortex_err; +use vortex_bench::conversions::parquet_to_vortex_stream; +use vortex_bench::data_dir; +use vortex_bench::utils::file::idempotent_async; +use vortex_bench::vector_dataset::DatasetPaths; +use vortex_bench::vector_dataset::TrainLayout; +use vortex_bench::vector_dataset::VectorDataset; + +use crate::SESSION; +use crate::compression::VectorFlavor; +use crate::ingest::transform_chunk; + +/// The paths of the vortex files that result from preparing one `(dataset, layout, flavor)` triple. +#[derive(Debug, Clone)] +pub struct CompressedVortexDataset { + pub dataset: VectorDataset, + pub layout: TrainLayout, + pub flavor: VectorFlavor, + pub vortex_files: Vec, +} + +/// Drive [`prepare_flavor`] across a list of flavors, returning a [`CompressedVortexDataset`] per +/// flavor in input order. +pub async fn prepare_all( + dataset: VectorDataset, + layout: TrainLayout, + paths_for_dataset: &DatasetPaths, + flavors: &[VectorFlavor], +) -> Result> { + let mut results = Vec::with_capacity(flavors.len()); + + for &flavor in flavors { + let r = prepare_flavor(dataset, layout, paths_for_dataset, flavor).await?; + results.push(r); + } + + Ok(results) +} + +/// Prepare one flavor of one dataset by writing one `.vortex` file per train shard. +/// +/// This function is sequential (for now). +pub async fn prepare_flavor( + dataset: VectorDataset, + layout: TrainLayout, + paths_for_dataset: &DatasetPaths, + flavor: VectorFlavor, +) -> Result { + let mut vortex_files = Vec::with_capacity(paths_for_dataset.train_files.len()); + + for parquet_path in &paths_for_dataset.train_files { + let parquet_path = parquet_path.clone(); + let vortex_path = parquet_to_vortex_path(&parquet_path, dataset, layout, flavor)?; + + let already_cached = vortex_path.exists(); + if already_cached { + warn!( + "skipping cached vortex shard {} ({} flavor)", + vortex_path.display(), + flavor.label() + ); + } else { + info!( + "ingesting {} -> {} ({} flavor)", + parquet_path.display(), + vortex_path.display(), + flavor.label(), + ); + } + + let written_path = idempotent_async(vortex_path.as_path(), |tmp| async move { + write_shard_streaming(&parquet_path, &tmp, flavor).await + }) + .await?; + + vortex_files.push(written_path); + } + + Ok(CompressedVortexDataset { + dataset, + layout, + flavor, + vortex_files, + }) +} + +/// Stream one parquet shard through the chunk transform into a Vortex file. +/// +/// The output dtype is derived once from the first transformed chunk so the [`ArrayStreamAdapter`] +/// can declare it ahead of time. +async fn write_shard_streaming( + parquet_path: &Path, + vortex_path: &Path, + flavor: VectorFlavor, +) -> Result<()> { + let file = File::open(parquet_path).await?; + let builder = ParquetRecordBatchStreamBuilder::new(file).await?; + let mut array_stream = parquet_to_vortex_stream(builder.build()?); + + let mut ctx = SESSION.create_execution_ctx(); + + // We need to get the first chunk so that we know what the dtype of the file is. + let first = match array_stream.next().await { + Some(chunk) => transform_chunk_with_error(chunk, &mut ctx, parquet_path, 1)?, + None => { + return Err(vortex_err!( + "ingest: parquet shard {} produced no chunks", + parquet_path.display(), + ) + .into()); + } + }; + let dtype = first.dtype().clone(); + let shard_path = parquet_path.to_path_buf(); + + let transformed = + futures::stream::iter(std::iter::once(Ok(first))).chain(array_stream.enumerate().map( + move |(chunk_offset, chunk_or_err)| { + let mut local_ctx = SESSION.create_execution_ctx(); + transform_chunk_with_error( + chunk_or_err, + &mut local_ctx, + &shard_path, + chunk_offset + 2, + ) + }, + )); + + let stream = ArrayStreamExt::boxed(ArrayStreamAdapter::new(dtype, transformed)); + + let mut output = tokio::fs::OpenOptions::new() + .write(true) + .truncate(true) + .create(true) + .open(vortex_path) + .await?; + + flavor + .create_write_options(&SESSION) + .write(&mut output, stream) + .await?; + output.flush().await?; + + Ok(()) +} + +fn transform_chunk_with_error( + chunk_or_err: VortexResult, + ctx: &mut ExecutionCtx, + parquet_path: &Path, + chunk_idx: usize, +) -> VortexResult { + let chunk = chunk_or_err.map_err(|err| { + vortex_err!( + "ingest: failed to read chunk {} from {}: {err:#}", + chunk_idx, + parquet_path.display(), + ) + })?; + + transform_chunk(chunk, ctx).map_err(|err| { + vortex_err!( + "ingest: failed to transform chunk {} from {}: {err:#}", + chunk_idx, + parquet_path.display(), + ) + }) +} + +/// Translate a parquet shard path to its `.vortex` companion under the flavor directory. +/// +/// Just swaps the file extension and rebases the file name into the per-[`VectorFlavor`] flavor +/// directory. The shard stem is preserved so a directory listing pairs `00-of-10.parquet` with +/// `00-of-10.vortex`. +pub fn parquet_to_vortex_path( + parquet: &Path, + dataset: VectorDataset, + layout: TrainLayout, + flavor: VectorFlavor, +) -> Result { + let stem = parquet + .file_stem() + .with_context(|| format!("parquet path {} has no file stem", parquet.display()))? + .to_owned(); + + // TODO(connor): Is there a better way to do this? + let mut name = stem; + name.push(".vortex"); + + Ok(flavor_dir(dataset, layout, flavor).join(name)) +} + +/// `vortex-bench/data/vector-search////`. +fn flavor_dir(ds: VectorDataset, layout: TrainLayout, flavor: VectorFlavor) -> PathBuf { + data_dir() + .join("vector-search") + .join(ds.name()) + .join(layout.label()) + .join(flavor.dir_name()) +} diff --git a/benchmarks/vector-search-bench/src/query.rs b/benchmarks/vector-search-bench/src/query.rs new file mode 100644 index 00000000000..cbd9d3bf38e --- /dev/null +++ b/benchmarks/vector-search-bench/src/query.rs @@ -0,0 +1,120 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Sample one query vector from `test.parquet`. +//! +//! The vector datasets ship a `test.parquet` alongside the train split: these are the query vectors +//! meant to be issued against the index. +//! +//! The benchmark picks a single random row (seeded for reproducibility) and uses it as the query +//! for the scan. + +use std::path::Path; + +use anyhow::Context; +use anyhow::Result; +use anyhow::bail; +use anyhow::ensure; +use rand::RngExt; +use rand::SeedableRng; +use rand::rngs::StdRng; +use vortex::array::IntoArray; +use vortex::array::VortexSessionExecute; +use vortex::array::arrays::StructArray; +use vortex::array::arrays::struct_::StructArrayExt; +use vortex::dtype::PType; +use vortex::error::VortexExpect; +use vortex::error::vortex_err; +use vortex_bench::conversions::parquet_to_vortex_chunks; + +use crate::SESSION; + +/// One query vector sampled from `test.parquet`. +#[derive(Debug, Clone)] +pub struct QuerySample { + /// The ID of the vector. + pub id: i64, + /// f32 query values, length `dim`. + pub query: Vec, +} + +/// Sample one query row from `test.parquet`. +/// +/// The cast to f32 happens here when the source is f64 (matching the prepare-side cast), so that +/// all downstream code is uniformly f32. +pub async fn get_random_query_vector( + test_parquet: &Path, + expected_dim: u32, + src_ptype: PType, + seed: u64, +) -> Result { + let mut ctx = SESSION.create_execution_ctx(); + + let chunked = parquet_to_vortex_chunks(test_parquet.to_path_buf()) + .await + .with_context(|| format!("read test parquet {}", test_parquet.display()))?; + // The `test.parquet` files are generally small enough that this is not a big deal. + let struct_array: StructArray = chunked.into_array().execute(&mut ctx)?; + + let id = struct_array + .unmasked_field_by_name("id") + .context("test parquet missing `id` column")? + .clone(); + let emb = struct_array + .unmasked_field_by_name("emb") + .context("test parquet missing `emb` column")? + .clone(); + + let mut rng = StdRng::seed_from_u64(seed); + let query_row_idx = rng.random_range(0..id.len()); + + let id_scalar = id.execute_scalar(query_row_idx, &mut ctx)?; + let emb_scalar = emb.execute_scalar(query_row_idx, &mut ctx)?; + + ensure!(emb_scalar.as_list().len() == expected_dim as usize); + + let id = id_scalar + .as_primitive() + .as_::() + .ok_or_else(|| vortex_err!("vector ID was not a i64"))?; + + let query_vector = match src_ptype { + PType::F32 => emb_scalar + .as_list() + .elements() + .vortex_expect("somehow had a null test vector") + .iter() + .map(|element| { + element + .as_primitive() + .as_::() + .vortex_expect("value was not a f32") + }) + .collect(), + PType::F64 => + { + #[expect( + clippy::cast_possible_truncation, + reason = "this is intentionally lossy" + )] + emb_scalar + .as_list() + .elements() + .vortex_expect("somehow had a null test vector") + .iter() + .map(|element| { + element + .as_primitive() + .as_::() + .vortex_expect("value was not a f64") as f32 + }) + .collect() + } + ptype => bail!("source ptype {ptype} was somehow not f32 or f64"), + }; + + Ok(QuerySample { + query: query_vector, + id, + }) +} diff --git a/benchmarks/vector-search-bench/src/scan.rs b/benchmarks/vector-search-bench/src/scan.rs new file mode 100644 index 00000000000..81a054f74b3 --- /dev/null +++ b/benchmarks/vector-search-bench/src/scan.rs @@ -0,0 +1,177 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Per-iteration scan driver. +//! +//! Each iteration re-opens every `.vortex` shard fresh (so the segment cache is re-primed +//! per run), pushes the cosine-similarity filter through the scan, and drains the resulting +//! [`vortex::array::stream::ArrayStream`]. The wall-clock around the entire per-iteration +//! pass is the headline number; we track the mean and median across iterations. + +use std::path::Path; +use std::path::PathBuf; +use std::time::Duration; +use std::time::Instant; + +use anyhow::Context; +use anyhow::Result; +use futures::TryStreamExt; +use vortex::array::ArrayRef; +use vortex::file::OpenOptionsSessionExt; + +use crate::SESSION; +use crate::compression::VectorFlavor; +use crate::expression::similarity_filter; +use crate::prepare::CompressedVortexDataset; + +/// Inputs to a scan run. +#[derive(Debug, Clone)] +pub struct ScanConfig { + /// Number of timed iterations (best-of-N). + pub iterations: usize, + /// Cosine threshold passed to the filter expression. + pub threshold: f32, +} + +/// Aggregate timing + counters for one `(flavor)` scan. +#[derive(Debug, Clone)] +pub struct ScanTiming { + /// Which compression flavor's `.vortex` files were scanned. + pub flavor: VectorFlavor, + /// Arithmetic mean of the per-iteration wall times. + pub mean: Duration, + /// Median of the per-iteration wall times. + pub median: Duration, + /// Per-iteration wall times in run order. + pub all_runs: Vec, + /// Number of rows that survived the filter (constant across iterations because the + /// filter is deterministic). + pub matches: u64, + /// Total rows scanned (sum of file row counts) as a sanity check that the iteration + /// actually walked the files. + pub rows_scanned: u64, + /// Total on-disk size of the scanned `.vortex` files, in bytes. + pub bytes_scanned: u64, +} + +/// Scan every shard in a [`CompressedVortexDataset`] under the given config. +pub async fn run_search_scan( + dataset: &CompressedVortexDataset, + query: &[f32], + config: &ScanConfig, +) -> Result { + anyhow::ensure!( + config.iterations > 0, + "scan iterations must be >= 1, got {}", + config.iterations + ); + + let bytes_scanned = total_file_size(&dataset.vortex_files)?; + + let mut all_runs = Vec::with_capacity(config.iterations); + let mut matches = 0u64; + let mut rows_scanned = 0u64; + + for iter_idx in 0..config.iterations { + let (wall, iter_matches, iter_rows) = + run_one_iteration(&dataset.vortex_files, query, config.threshold).await?; + tracing::debug!( + "{} iter {} -> {:?} ({} matches, {} rows)", + dataset.flavor.label(), + iter_idx, + wall, + iter_matches, + iter_rows, + ); + // Matches and row counts are deterministic across iterations; reset rather than + // accumulate so the reported value matches a single pass. + matches = iter_matches; + rows_scanned = iter_rows; + all_runs.push(wall); + } + + Ok(ScanTiming { + flavor: dataset.flavor, + mean: mean(&all_runs), + median: median(&all_runs), + all_runs, + matches, + rows_scanned, + bytes_scanned, + }) +} + +/// Sum the on-disk sizes of the given files. +fn total_file_size(paths: &[PathBuf]) -> Result { + let mut total = 0u64; + for path in paths { + let meta = + std::fs::metadata(path).with_context(|| format!("stat {} for size", path.display()))?; + total = total.saturating_add(meta.len()); + } + Ok(total) +} + +async fn run_one_iteration( + vortex_files: &[PathBuf], + query: &[f32], + threshold: f32, +) -> Result<(Duration, u64, u64)> { + let mut matches = 0u64; + let mut rows_scanned = 0u64; + + let started = Instant::now(); + for path in vortex_files { + let (m, r) = scan_one_file(path, query, threshold).await?; + matches = matches.saturating_add(m); + rows_scanned = rows_scanned.saturating_add(r); + } + + Ok((started.elapsed(), matches, rows_scanned)) +} + +async fn scan_one_file(path: &Path, query: &[f32], threshold: f32) -> Result<(u64, u64)> { + let file = SESSION + .open_options() + .open_path(path) + .await + .with_context(|| format!("open {}", path.display()))?; + + let total_rows = file.row_count(); + let filter = similarity_filter(query, threshold)?; + let chunks: Vec = file + .scan()? + .with_filter(filter) + .into_array_stream()? + .try_collect() + .await?; + + let matches: u64 = chunks.iter().map(|c| c.len() as u64).sum(); + Ok((matches, total_rows)) +} + +/// Arithmetic mean of a list of [`Duration`]s. Empty lists return [`Duration::ZERO`]. +pub fn mean(runs: &[Duration]) -> Duration { + if runs.is_empty() { + return Duration::ZERO; + } + let total_nanos: u128 = runs.iter().map(|d| d.as_nanos()).sum(); + let avg_nanos = total_nanos / runs.len() as u128; + Duration::from_nanos(u64::try_from(avg_nanos).unwrap_or(u64::MAX)) +} + +/// Median of a list of [`Duration`]s. Empty lists return [`Duration::ZERO`]. +pub fn median(runs: &[Duration]) -> Duration { + if runs.is_empty() { + return Duration::ZERO; + } + let mut sorted = runs.to_vec(); + sorted.sort(); + let mid = sorted.len() / 2; + if sorted.len() % 2 == 1 { + sorted[mid] + } else { + let total_nanos = sorted[mid - 1].as_nanos() + sorted[mid].as_nanos(); + Duration::from_nanos(u64::try_from(total_nanos / 2).unwrap_or(u64::MAX)) + } +} diff --git a/clippy.toml b/clippy.toml index 4b6fc10fe76..dc1b625c4de 100644 --- a/clippy.toml +++ b/clippy.toml @@ -12,5 +12,6 @@ disallowed-types = [ disallowed-methods = [ { path = "itertools::Itertools::counts", reason = "It uses the default hasher which is slow for primitives. Just inline the loop for better performance.", allow-invalid = true }, - { path = "std::result::Result::and", reason = "This method is a footgun, especially when working with `Result`.", allow-invalid = true }, + { path = "std::result::Result::and", reason = "This method is a footgun, especially when working with `Result`." }, + { path = "std::thread::available_parallelism", reason = "This function might do an unbounded amount of work, use `vortex_utils::parallelism::get_available_parallelism instead" }, ] diff --git a/deny.toml b/deny.toml index 31f67e95f77..64814519f91 100644 --- a/deny.toml +++ b/deny.toml @@ -15,9 +15,6 @@ feature-depth = 1 ignore = [ # Paste is no longer maintained because its essentially "finished". "RUSTSEC-2024-0436", - # rustls-pemfile is a hard requirement from object_store crate. - # need to wait for them to release, see https://github.com/apache/arrow-rs-object-store/pull/565 - "RUSTSEC-2025-0134", ] [licenses] diff --git a/docs/concepts/scanning.md b/docs/concepts/scanning.md index 9e21c589f25..393b7d2d83c 100644 --- a/docs/concepts/scanning.md +++ b/docs/concepts/scanning.md @@ -68,10 +68,12 @@ independently. The scan tracks the selectivity of each conjunct using a probabil and dynamically reorders them so that the most selective predicates are evaluated first. This means that as a scan progresses, it learns the most efficient evaluation order for the filter. -Filters are evaluated in two stages. First, pruning evaluation uses statistics stored in -`ZonedLayout` (such as per-zone min/max values) to eliminate entire regions without reading any -data. Second, filter evaluation materializes only the filter-referenced columns and computes a -row mask of matching rows. +Filters are evaluated in two stages. First, pruning evaluation uses statistics stored in a +`ZonedLayout` auxiliary `zones` child to eliminate entire row zones without reading the underlying +data child. These pruning predicates are falsification checks derived from the original filter, for +example by comparing a zone's min/max values against the requested predicate. Second, filter +evaluation materializes only the filter-referenced columns and computes a row mask of matching +rows. ## Projection Pushdown @@ -89,4 +91,3 @@ Query engines integrate with the Scan API by translating their internal plan rep scan requests and consuming the resulting array stream in their preferred format. Integrations exist for DuckDB, DataFusion, Spark, and Trino, with each engine converting its native filter and projection representations into Vortex [expressions](expressions.md). - diff --git a/docs/developer-guide/benchmarking.md b/docs/developer-guide/benchmarking.md index 2e09c727dd7..0034965a056 100644 --- a/docs/developer-guide/benchmarking.md +++ b/docs/developer-guide/benchmarking.md @@ -168,7 +168,7 @@ Lance, DuckDB native). Before running SQL benchmarks, test data must be generated: ```bash -cargo run --release --bin data-gen -- --formats parquet,vortex +uv run --project bench-orchestrator vx-bench prepare-data --format parquet,vortex ``` The data generator creates base Parquet data and converts it to each requested format. Scale @@ -191,6 +191,19 @@ benchmarks across multiple engines, stores results, and provides comparison tool See [`bench-orchestrator/README.md`](https://github.com/vortex-data/vortex/blob/develop/bench-orchestrator/README.md) for installation, commands, and example workflows. +For CI, the reusable SQL workflow now drives `vx-bench` directly: + +```bash +uv run --project bench-orchestrator vx-bench prepare-data tpch \ + --formats-json '["parquet","vortex","vortex-compact"]' \ + --opt scale-factor=1.0 + +uv run --project bench-orchestrator vx-bench run tpch \ + --targets-json '[{"engine":"datafusion","format":"parquet"},{"engine":"duckdb","format":"vortex"}]' \ + --output results.json \ + --no-build +``` + ## CI Benchmarks Benchmarks run automatically on all commits to `develop` and can be run on-demand for PRs: diff --git a/docs/developer-guide/internals/execution.md b/docs/developer-guide/internals/execution.md index 07eb88875b5..be92e29ceef 100644 --- a/docs/developer-guide/internals/execution.md +++ b/docs/developer-guide/internals/execution.md @@ -60,44 +60,31 @@ support constant vectors directly, avoiding unnecessary expansion. ## Execution Overview -The `execute_until` method on `ArrayRef` drives execution. The scheduler is -iterative: it rewrites and executes arrays in small steps until the current array matches the -requested target form. +Execution has two closely related entry points: -At a high level, each iteration works like this: +- `ArrayRef::execute::` is the single-step executor. It tries `reduce`, + `reduce_parent`, `execute_parent`, then `execute` once. +- `ArrayRef::execute_until` is the matcher-driven loop used by `Canonical`, `Columnar`, + and other target executors. It repeatedly interprets `ExecutionStep` until the current + activation matches `M` or no further progress is possible. -1. `optimize(current)` runs metadata-only rewrites to fixpoint: - `reduce` lets an array simplify itself, and `reduce_parent` lets a child rewrite its parent. -2. If optimization does not finish execution, each child gets a chance to `execute_parent`, - meaning "execute my parent's operation using my representation". -3. If no child can do that, the array's own `execute` method returns the next `ExecutionStep`. +`VTable::execute` never recursively descends into children on its own. Instead it returns an +`ExecutionResult` containing an `ExecutionStep` that tells `execute_until` what to do next. -This keeps execution iterative rather than recursive, and it gives optimization rules another -chance to fire after every structural or computational step. +The loop carries three mutable pieces of state: -## The Four Layers - -The execution model has four layers, but they are not all invoked in the same way. Layers 1 and -2 make up `optimize`, which runs to fixpoint before and after execution steps. Layers 3 and 4 -run only after optimization has stalled. - -``` -execute_until(root): - current = optimize(root) # Layers 1-2 to fixpoint +- `current_array: ArrayRef` -- the array currently in focus. +- `current_builder: Option>` -- active only for the builder path. + `AppendChild` appends detached children here, and `Done` finalizes the builder. +- `stack: Vec` -- suspended parents from `ExecuteSlot`, including the detached + slot index, its `DonePredicate`, and the parent builder that was active before focus moved + into the child. - loop: - if current matches target: - return / reattach to parent +## The Four Layers - Layer 3: try execute_parent on each child - if one succeeds: - current = optimize(result) - continue - - Layer 4: call execute(current) - ExecuteChild(i, pred) -> focus child[i], then optimize - Done -> current = optimize(result) -``` +Encodings can contribute logic in four places. The single-step executor can touch all four. +The iterative `execute_until` loop revisits Layers 3 and 4 directly, using `ExecuteSlot`, +`AppendChild`, and `Done` to move focus around the tree. ### Layer 1: `reduce` -- self-rewrite rules @@ -162,66 +149,92 @@ containing an `ExecutionStep` that tells the scheduler what to do next: ```rust pub enum ExecutionStep { - /// Ask the scheduler to execute child[idx] until it matches the predicate, - /// then replace the child and re-enter execution for this array. - ExecuteChild(usize, DonePredicate), + /// Push the parent onto the stack, focus a single child, and resume the + /// parent once that child matches the predicate. + ExecuteSlot(usize, DonePredicate), + + /// Detach a child, append it into the current activation's builder, and + /// keep the parent as current_array for the next iteration. + AppendChild(usize), - /// Execution is complete. The array in the ExecutionResult is the result. + /// Execution is complete. If a builder is active, it is finalized here. Done, } ``` +- `ExecuteSlot(i, pred)` detaches slot `i`, pushes the parent onto `stack`, and makes that + child the new `current_array` until `pred` says it is done. +- `AppendChild(i)` detaches slot `i`, appends that child into `current_builder`, and keeps + the returned parent as `current_array` for the next iteration. +- `Done` finishes the current activation. If `current_builder` is active, the builder is + finalized and its finished array becomes the result of this activation. + ## The Execution Loop -The full `execute_until` loop uses an explicit work stack to manage -parent-child relationships without recursion: +The full `execute_until` loop uses an explicit work stack and an optional builder +to manage parent-child relationships without recursion. ``` execute_until(root): + current_array = root + current_builder = None stack = [] - current = optimize(root) loop: - ┌─────────────────────────────────────────────────────┐ - │ Is current "done"? │ - │ (matches M if at root, or matches the stack │ - │ frame's DonePredicate if inside a child) │ - ├──────────────────────┬──────────────────────────────┘ + ┌──────────────────────────────────────────────────────────────┐ + │ Step 1: is current_array "done"? │ + │ (matches M at the root, or the stack frame's │ + │ DonePredicate inside ExecuteSlot) │ + ├──────────────────────┬───────────────────────────────────────┘ │ yes │ no │ │ - │ stack empty? │ Already canonical? - │ ├─ yes → return │ ├─ yes → pop stack (can't make more progress) - │ └─ no → pop frame, │ └─ no → continue to execution steps - │ replace child, │ - │ optimize, loop │ - │ ▼ - │ ┌────────────────────────────────────┐ - │ │ Try execute_parent on each child │ - │ │ (Layer 3 parent kernels) │ - │ ├────────┬───────────────────────────┘ - │ │ Some │ None - │ │ │ - │ │ ▼ - │ │ ┌─────────────────────────────────┐ - │ │ │ Call execute (Layer 4) │ - │ │ │ Returns ExecutionResult │ - │ │ ├────────┬────────────────────────┘ - │ │ │ │ - │ │ │ ExecuteChild(i, pred)? - │ │ │ ├─ yes → push (array, i, pred) - │ │ │ │ current = child[i] - │ │ │ │ optimize, loop - │ │ │ └─ Done → current = result - │ │ │ loop - │ │ │ - │ ▼ ▼ - │ optimize result, loop - └────────────────────────── + │ stack empty? │ current_builder active? + │ ├─ yes → return │ ├─ yes → skip Step 2a / 2b + │ └─ no → pop frame, │ └─ no + │ reattach child, │ + │ restore builder, │ + │ loop ▼ + │ ┌────────────────────────────────────────────┐ + │ │ Step 2a: current_array.execute_parent( │ + │ │ stack.top.parent_array ) │ + │ │ child looks UP at the suspended parent │ + │ ├────────────┬───────────────────────────────┘ + │ │ Some │ None + │ │ │ + │ │ ▼ + │ │ ┌─────────────────────────────────────────┐ + │ │ │ Step 2b: each child.execute_parent( │ + │ │ │ current_array ) │ + │ │ │ children look UP at current_array │ + │ │ ├──────────┬──────────────────────────────┘ + │ │ │ Some │ None + │ │ │ │ + │ │ │ ▼ + │ │ │ ┌──────────────────────────────────────┐ + │ │ │ │ Step 3: current_array.execute() │ + │ │ │ ├──────────────┬───────────────────────┘ + │ │ │ │ │ + │ │ │ │ ExecuteSlot(i, pred) + │ │ │ │ -> push parent + builder + │ │ │ │ -> current_array = child[i] + │ │ │ │ -> current_builder = None + │ │ │ │ + │ │ │ │ AppendChild(i) + │ │ │ │ -> ensure current_builder + │ │ │ │ -> child.append_to_builder(...) + │ │ │ │ -> current_array = parent + │ │ │ │ + │ │ │ │ Done + │ │ │ │ -> finish current_builder if present + │ │ │ │ -> otherwise use returned array + │ ▼ ▼ ▼ + │ continue loop with rewritten or finished array + └────────────────────────────────────────────────────── ``` -Note that `optimize` runs after every transformation. This is what enables cross-step -optimizations: after a child is decoded, new `reduce_parent` rules may now match that were -previously blocked. +Step 2a and Step 2b are skipped while `current_builder` is active. `AppendChild` partially +consumes `current_array`: some slots already live in the builder, so a parent rewrite would +observe inconsistent state and could discard accumulated builder data. ## Incremental Execution @@ -242,8 +255,9 @@ If execution jumped straight to canonicalizing the dict's children, it would exp codes through the slice, missing the Dict-RLE optimization entirely. Incremental execution avoids this: -1. First iteration: the slice `execute_parent` (parent kernel on RunEnd for Slice) performs a - binary search on run ends, returning a new `RunEndArray` with adjusted offsets. +1. First iteration: the slice `execute` returns `ExecuteSlot` for its `RunEndArray` child. + Once that child is in focus, Step 2a gives it a chance to rewrite the suspended slice + parent before the child is forced toward canonical form. 2. Second iteration: the `RunEndArray` codes child now matches the Dict-RLE pattern. Its `execute_parent` provides a fused kernel that expands runs while performing dictionary @@ -259,44 +273,96 @@ Input: RunEndArray { ends: [3, 7, 10], values: [A, B, C], len: 10 } Goal: Canonical (PrimitiveArray or similar) Iteration 1: - reduce? → None (no self-rewrite rules match) - reduce_parent? → None (no parent, this is root) - execute_parent? → None (no parent) - execute → ends are not Primitive yet? - ExecuteChild(0, Primitive::matches) + Step 1 → not done + Step 2a → skipped (root, no stacked parent) + Step 2b → None + Step 3 → ends are not Primitive yet? + ExecuteSlot(0, Primitive::matches) Stack: [(RunEnd, child_idx=0, Primitive::matches)] Focus on: ends + current_builder = None Iteration 2: - Current: ends array - Already Primitive? → yes, done. - Pop stack → replace child 0 in RunEnd, optimize. + Step 1 → done (ends already match Primitive) + Pop stack → replace child 0 in RunEnd Iteration 3: - reduce? → None - reduce_parent? → None - execute_parent? → None - execute → values are not Canonical yet? - ExecuteChild(1, AnyCanonical::matches) + Step 1 → not done + Step 2a → skipped (root again after the pop) + Step 2b → None + Step 3 → values are not Canonical yet? + ExecuteSlot(1, AnyCanonical::matches) Stack: [(RunEnd, child_idx=1, AnyCanonical::matches)] Focus on: values Iteration 4: - Current: values array - Already Canonical? → yes, done. - Pop stack → replace child 1 in RunEnd, optimize. + Step 1 → done (values already match AnyCanonical) + Pop stack → replace child 1 in RunEnd Iteration 5: - reduce? → None - reduce_parent? → None - execute_parent? → None - execute → all children ready, decode runs: + Step 1 → not done + Step 2a → skipped (root) + Step 2b → None + Step 3 → all children ready, decode runs: [A, A, A, B, B, B, B, C, C, C] Done → return PrimitiveArray → Result: PrimitiveArray [A, A, A, B, B, B, B, C, C, C] ``` +## Walkthrough: Executing a Chunked Bool Array via `AppendChild` + +`Chunked` uses the builder path for most dtypes. Instead of focusing one child as the new +`current_array`, it detaches one chunk at a time, appends it into `current_builder`, and keeps +the `ChunkedArray` itself as the active parent: + +``` +Input: Chunked { + chunks[0] = Bool[true, false], + chunks[1] = Bool[false], + chunks[2] = Bool[true, true], + } +Goal: Canonical BoolArray + +Iteration 1: + Step 1 → not done + Step 2a → skipped (root, no stacked parent) + Step 2b → None + Step 3 → AppendChild(1) + create current_builder = BoolBuilder [] + append chunks[0] + current_array = Chunked(next_builder_slot = 2) + current_builder = BoolBuilder [true, false] + +Iteration 2: + Step 1 → not done + Step 2a / 2b → skipped (builder active; current_array is partially consumed) + Step 3 → AppendChild(2) + append chunks[1] + current_array = Chunked(next_builder_slot = 3) + current_builder = BoolBuilder [true, false, false] + +Iteration 3: + Step 1 → not done + Step 2a / 2b → skipped + Step 3 → AppendChild(3) + append chunks[2] + current_array = Chunked(next_builder_slot = 4) + current_builder = BoolBuilder [true, false, false, true, true] + +Iteration 4: + Step 1 → not done + Step 2a / 2b → skipped + Step 3 → Done + finish current_builder + result = BoolArray [true, false, false, true, true] + +→ Result: BoolArray [true, false, false, true, true] +``` + +When `current_builder` is active, the array returned alongside `Done` is just the signal that +the parent activation has finished. The actual result comes from finalizing the builder. + ## Implementing an Encoding: Where Does My Logic Go? When adding a new encoding or optimizing an existing one, the key question is whether the @@ -317,7 +383,7 @@ Rules of thumb: knows how to handle its parent's operation more efficiently than the parent knows how to handle the child. - Treat `execute` as the fallback. If no reduce rule or parent kernel applies, the encoding - decodes itself and uses `ExecuteChild` to request child execution when needed. + decodes itself and uses `ExecuteSlot` or `AppendChild` to tell the scheduler what to do next. ## Future Work diff --git a/docs/getting-started/install.md b/docs/getting-started/install.md index 59e808f9f75..9ff30a30fcd 100644 --- a/docs/getting-started/install.md +++ b/docs/getting-started/install.md @@ -7,6 +7,13 @@ the terminal. ::::{tab-set} +:::{tab-item} Binstall (recommended) +```bash +cargo binstall vortex-tui +``` +Downloads a pre-built binary. Requires [cargo-binstall](https://github.com/cargo-bins/cargo-binstall). +::: + :::{tab-item} pip ```bash pip install vortex-data @@ -14,10 +21,18 @@ pip install vortex-data This also installs the Python library. See the [Python quickstart](python.rst) for library usage. ::: +:::{tab-item} uvx +```bash +uvx --from vortex-data vx --help +``` +Runs the CLI without installing. Requires [uv](https://docs.astral.sh/uv/). +::: + :::{tab-item} Cargo ```bash cargo install vortex-tui ``` +Builds from source. This can be slow due to the large dependency tree. ::: :::: diff --git a/docs/user-guide/datafusion.md b/docs/user-guide/datafusion.md index 650d5e1c908..0be982575b6 100644 --- a/docs/user-guide/datafusion.md +++ b/docs/user-guide/datafusion.md @@ -14,7 +14,7 @@ vortex-datafusion = "" Register the Vortex format with a `SessionContext`: -:::{literalinclude} ../../vortex-datafusion/src/persistent/mod.rs +:::{literalinclude} ../../vortex-datafusion/src/persistent/tests.rs :language: rust :dedent: :start-after: [setup] @@ -27,7 +27,7 @@ Register the Vortex format with a `SessionContext`: Create an external table and query it: -:::{literalinclude} ../../vortex-datafusion/src/persistent/mod.rs +:::{literalinclude} ../../vortex-datafusion/src/persistent/tests.rs :language: rust :dedent: :start-after: [create] @@ -49,7 +49,7 @@ You can also register a `ListingTable` directly: Write query results to Vortex using `INSERT INTO`: -:::{literalinclude} ../../vortex-datafusion/src/persistent/mod.rs +:::{literalinclude} ../../vortex-datafusion/src/persistent/tests.rs :language: rust :dedent: :start-after: [write] @@ -63,7 +63,7 @@ partition value. Filters and projections are pushed down into the Vortex scan: -:::{literalinclude} ../../vortex-datafusion/src/persistent/mod.rs +:::{literalinclude} ../../vortex-datafusion/src/persistent/tests.rs :language: rust :dedent: :start-after: [query] diff --git a/docs/user-guide/duckdb.md b/docs/user-guide/duckdb.md index 19631816165..3fc0b03be9b 100644 --- a/docs/user-guide/duckdb.md +++ b/docs/user-guide/duckdb.md @@ -46,10 +46,10 @@ COPY (SELECT * FROM my_table) TO 'output.vortex' (FORMAT vortex); Controls which filesystem implementation is used for reading and writing Vortex files. -| Value | Description | -|-------|-------------| -| `'vortex'` (default) | Uses Vortex's built-in object store filesystem. Supports `file://` and `s3://` schemes. | -| `'duckdb'` | Uses DuckDB's built-in filesystem, including any filesystem extensions such as `httpfs`. | +| Value | Description | +| -------------------- | ---------------------------------------------------------------------------------------- | +| `'vortex'` (default) | Uses Vortex's built-in object store filesystem. Supports `file://` and `s3://` schemes. | +| `'duckdb'` | Uses DuckDB's built-in filesystem, including any filesystem extensions such as `httpfs`. | ```sql SET vortex_filesystem = 'duckdb'; diff --git a/encodings/alp/benches/alp_compress.rs b/encodings/alp/benches/alp_compress.rs index 239541533f8..7fbe60d5997 100644 --- a/encodings/alp/benches/alp_compress.rs +++ b/encodings/alp/benches/alp_compress.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] +#![expect(clippy::unwrap_used)] use divan::Bencher; use rand::RngExt; @@ -12,6 +12,8 @@ use vortex_alp::ALPRDFloat; use vortex_alp::RDEncoder; use vortex_alp::alp_encode; use vortex_alp::decompress_into_array; +use vortex_array::Canonical; +use vortex_array::IntoArray; use vortex_array::LEGACY_SESSION; use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; @@ -66,9 +68,14 @@ fn compress_alp(bencher: Bencher, args: (usize, f64, let values = values.freeze(); let array = PrimitiveArray::new(values, validity); - bencher - .with_inputs(|| &array) - .bench_values(|array| alp_encode(array, None).unwrap()) + bencher.with_inputs(|| &array).bench_values(|array| { + alp_encode( + array.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap() + }) } #[divan::bench(types = [f32, f64], args = BENCH_ARGS)] @@ -93,8 +100,9 @@ fn decompress_alp(bencher: Bencher, args: (usize, f64 .with_inputs(|| { ( alp_encode( - &PrimitiveArray::new(Buffer::copy_from(&values), validity.clone()), + PrimitiveArray::new(Buffer::copy_from(&values), validity.clone()).as_view(), None, + &mut LEGACY_SESSION.create_execution_ctx(), ) .unwrap(), LEGACY_SESSION.create_execution_ctx(), @@ -135,8 +143,8 @@ fn compress_rd(bencher: Bencher, args: (usize, f64) let encoder = RDEncoder::new(primitive.as_slice::()); bencher - .with_inputs(|| (&primitive, &encoder)) - .bench_refs(|(primitive, encoder)| encoder.encode(primitive)) + .with_inputs(|| (&primitive, &encoder, LEGACY_SESSION.create_execution_ctx())) + .bench_refs(|(primitive, encoder, ctx)| encoder.encode(primitive.as_view(), ctx)) } #[divan::bench(types = [f32, f64], args = RD_BENCH_ARGS)] @@ -144,9 +152,12 @@ fn decompress_rd(bencher: Bencher, args: (usize, f6 let (n, fraction_patch) = args; let primitive = make_rd_array::(n, fraction_patch); let encoder = RDEncoder::new(primitive.as_slice::()); - let encoded = encoder.encode(&primitive); + let encoded = encoder.encode( + primitive.as_view(), + &mut LEGACY_SESSION.create_execution_ctx(), + ); bencher - .with_inputs(|| &encoded) - .bench_refs(|encoded| encoded.to_canonical()); + .with_inputs(|| (&encoded, LEGACY_SESSION.create_execution_ctx())) + .bench_refs(|(encoded, ctx)| (**encoded).clone().into_array().execute::(ctx)); } diff --git a/encodings/alp/public-api.lock b/encodings/alp/public-api.lock index 0c5987dbae5..02caae83d0b 100644 --- a/encodings/alp/public-api.lock +++ b/encodings/alp/public-api.lock @@ -6,15 +6,11 @@ pub struct vortex_alp::ALP impl vortex_alp::ALP -pub const vortex_alp::ALP::ID: vortex_array::array::ArrayId +pub fn vortex_alp::ALP::new(vortex_array::array::erased::ArrayRef, vortex_alp::Exponents, core::option::Option) -> vortex_alp::ALPArray -impl vortex_alp::ALP - -pub fn vortex_alp::ALP::new(encoded: vortex_array::array::erased::ArrayRef, exponents: vortex_alp::Exponents, patches: core::option::Option) -> vortex_alp::ALPArray +pub unsafe fn vortex_alp::ALP::new_unchecked(vortex_array::array::erased::ArrayRef, vortex_alp::Exponents, core::option::Option) -> vortex_alp::ALPArray -pub unsafe fn vortex_alp::ALP::new_unchecked(encoded: vortex_array::array::erased::ArrayRef, exponents: vortex_alp::Exponents, patches: core::option::Option) -> vortex_alp::ALPArray - -pub fn vortex_alp::ALP::try_new(encoded: vortex_array::array::erased::ArrayRef, exponents: vortex_alp::Exponents, patches: core::option::Option) -> vortex_error::VortexResult +pub fn vortex_alp::ALP::try_new(vortex_array::array::erased::ArrayRef, vortex_alp::Exponents, core::option::Option) -> vortex_error::VortexResult impl core::clone::Clone for vortex_alp::ALP @@ -22,7 +18,7 @@ pub fn vortex_alp::ALP::clone(&self) -> vortex_alp::ALP impl core::fmt::Debug for vortex_alp::ALP -pub fn vortex_alp::ALP::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_alp::ALP::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::array::vtable::VTable for vortex_alp::ALP @@ -32,67 +28,67 @@ pub type vortex_alp::ALP::OperationsVTable = vortex_alp::ALP pub type vortex_alp::ALP::ValidityVTable = vortex_array::array::vtable::validity::ValidityVTableFromChild -pub fn vortex_alp::ALP::buffer(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_alp::ALP::buffer(vortex_array::array::view::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_alp::ALP::buffer_name(_array: vortex_array::array::view::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_alp::ALP::buffer_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_alp::ALP::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_alp::ALP::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_alp::ALP::execute(array: vortex_array::array::typed::Array, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_alp::ALP::execute(vortex_array::array::typed::Array, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_alp::ALP::execute_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_alp::ALP::execute_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_alp::ALP::id(&self) -> vortex_array::array::ArrayId -pub fn vortex_alp::ALP::nbuffers(_array: vortex_array::array::view::ArrayView<'_, Self>) -> usize +pub fn vortex_alp::ALP::nbuffers(vortex_array::array::view::ArrayView<'_, Self>) -> usize -pub fn vortex_alp::ALP::reduce_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_alp::ALP::reduce_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_alp::ALP::serialize(array: vortex_array::array::view::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_alp::ALP::serialize(vortex_array::array::view::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_alp::ALP::slot_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_alp::ALP::slot_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_alp::ALP::validate(&self, data: &vortex_alp::ALPData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_alp::ALP::validate(&self, &vortex_alp::ALPData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::array::vtable::operations::OperationsVTable for vortex_alp::ALP -pub fn vortex_alp::ALP::scalar_at(array: vortex_array::array::view::ArrayView<'_, vortex_alp::ALP>, index: usize, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_alp::ALP::scalar_at(vortex_array::array::view::ArrayView<'_, vortex_alp::ALP>, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::array::vtable::validity::ValidityChild for vortex_alp::ALP -pub fn vortex_alp::ALP::validity_child(array: vortex_array::array::view::ArrayView<'_, vortex_alp::ALP>) -> vortex_array::array::erased::ArrayRef +pub fn vortex_alp::ALP::validity_child(vortex_array::array::view::ArrayView<'_, vortex_alp::ALP>) -> vortex_array::array::erased::ArrayRef impl vortex_array::arrays::dict::take::TakeExecute for vortex_alp::ALP -pub fn vortex_alp::ALP::take(array: vortex_array::array::view::ArrayView<'_, Self>, indices: &vortex_array::array::erased::ArrayRef, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_alp::ALP::take(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::kernel::FilterKernel for vortex_alp::ALP -pub fn vortex_alp::ALP::filter(array: vortex_array::array::view::ArrayView<'_, Self>, mask: &vortex_mask::Mask, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_alp::ALP::filter(vortex_array::array::view::ArrayView<'_, Self>, &vortex_mask::Mask, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceKernel for vortex_alp::ALP -pub fn vortex_alp::ALP::slice(array: vortex_array::array::view::ArrayView<'_, Self>, range: core::ops::range::Range, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_alp::ALP::slice(vortex_array::array::view::ArrayView<'_, Self>, core::ops::range::Range, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::between::kernel::BetweenReduce for vortex_alp::ALP -pub fn vortex_alp::ALP::between(array: vortex_array::array::view::ArrayView<'_, Self>, lower: &vortex_array::array::erased::ArrayRef, upper: &vortex_array::array::erased::ArrayRef, options: &vortex_array::scalar_fn::fns::between::BetweenOptions) -> vortex_error::VortexResult> +pub fn vortex_alp::ALP::between(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, &vortex_array::array::erased::ArrayRef, &vortex_array::scalar_fn::fns::between::BetweenOptions) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::binary::compare::CompareKernel for vortex_alp::ALP -pub fn vortex_alp::ALP::compare(lhs: vortex_array::array::view::ArrayView<'_, Self>, rhs: &vortex_array::array::erased::ArrayRef, operator: vortex_array::scalar_fn::fns::operators::CompareOperator, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_alp::ALP::compare(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, vortex_array::scalar_fn::fns::operators::CompareOperator, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::kernel::CastReduce for vortex_alp::ALP -pub fn vortex_alp::ALP::cast(array: vortex_array::array::view::ArrayView<'_, Self>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_alp::ALP::cast(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::kernel::MaskKernel for vortex_alp::ALP -pub fn vortex_alp::ALP::mask(array: vortex_array::array::view::ArrayView<'_, Self>, mask: &vortex_array::array::erased::ArrayRef, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_alp::ALP::mask(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::kernel::MaskReduce for vortex_alp::ALP -pub fn vortex_alp::ALP::mask(array: vortex_array::array::view::ArrayView<'_, Self>, mask: &vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_alp::ALP::mask(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult> pub struct vortex_alp::ALPData @@ -102,7 +98,7 @@ pub fn vortex_alp::ALPData::exponents(&self) -> vortex_alp::Exponents impl vortex_alp::ALPData -pub fn vortex_alp::ALPData::new(exponents: vortex_alp::Exponents, patches: core::option::Option) -> Self +pub fn vortex_alp::ALPData::new(vortex_alp::Exponents, core::option::Option) -> Self impl core::clone::Clone for vortex_alp::ALPData @@ -110,19 +106,19 @@ pub fn vortex_alp::ALPData::clone(&self) -> vortex_alp::ALPData impl core::fmt::Debug for vortex_alp::ALPData -pub fn vortex_alp::ALPData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_alp::ALPData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_alp::ALPData -pub fn vortex_alp::ALPData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_alp::ALPData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::hash::ArrayEq for vortex_alp::ALPData -pub fn vortex_alp::ALPData::array_eq(&self, other: &Self, _precision: vortex_array::hash::Precision) -> bool +pub fn vortex_alp::ALPData::array_eq(&self, &Self, vortex_array::hash::Precision) -> bool impl vortex_array::hash::ArrayHash for vortex_alp::ALPData -pub fn vortex_alp::ALPData::array_hash(&self, state: &mut H, _precision: vortex_array::hash::Precision) +pub fn vortex_alp::ALPData::array_hash(&self, &mut H, vortex_array::hash::Precision) pub struct vortex_alp::ALPMetadata @@ -136,7 +132,7 @@ pub fn vortex_alp::ALPMetadata::default() -> Self impl core::fmt::Debug for vortex_alp::ALPMetadata -pub fn vortex_alp::ALPMetadata::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_alp::ALPMetadata::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl prost::message::Message for vortex_alp::ALPMetadata @@ -148,11 +144,9 @@ pub struct vortex_alp::ALPRD impl vortex_alp::ALPRD -pub const vortex_alp::ALPRD::ID: vortex_array::array::ArrayId - -pub unsafe fn vortex_alp::ALPRD::new_unchecked(dtype: vortex_array::dtype::DType, left_parts: vortex_array::array::erased::ArrayRef, left_parts_dictionary: vortex_buffer::buffer::Buffer, right_parts: vortex_array::array::erased::ArrayRef, right_bit_width: u8, left_parts_patches: core::option::Option) -> vortex_alp::ALPRDArray +pub unsafe fn vortex_alp::ALPRD::new_unchecked(vortex_array::dtype::DType, vortex_array::array::erased::ArrayRef, vortex_buffer::buffer::Buffer, vortex_array::array::erased::ArrayRef, u8, core::option::Option) -> vortex_alp::ALPRDArray -pub fn vortex_alp::ALPRD::try_new(dtype: vortex_array::dtype::DType, left_parts: vortex_array::array::erased::ArrayRef, left_parts_dictionary: vortex_buffer::buffer::Buffer, right_parts: vortex_array::array::erased::ArrayRef, right_bit_width: u8, left_parts_patches: core::option::Option) -> vortex_error::VortexResult +pub fn vortex_alp::ALPRD::try_new(vortex_array::dtype::DType, vortex_array::array::erased::ArrayRef, vortex_buffer::buffer::Buffer, vortex_array::array::erased::ArrayRef, u8, core::option::Option, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult impl core::clone::Clone for vortex_alp::ALPRD @@ -160,7 +154,7 @@ pub fn vortex_alp::ALPRD::clone(&self) -> vortex_alp::ALPRD impl core::fmt::Debug for vortex_alp::ALPRD -pub fn vortex_alp::ALPRD::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_alp::ALPRD::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::array::vtable::VTable for vortex_alp::ALPRD @@ -170,65 +164,65 @@ pub type vortex_alp::ALPRD::OperationsVTable = vortex_alp::ALPRD pub type vortex_alp::ALPRD::ValidityVTable = vortex_array::array::vtable::validity::ValidityVTableFromChild -pub fn vortex_alp::ALPRD::buffer(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_alp::ALPRD::buffer(vortex_array::array::view::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_alp::ALPRD::buffer_name(_array: vortex_array::array::view::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_alp::ALPRD::buffer_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_alp::ALPRD::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_alp::ALPRD::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_alp::ALPRD::execute(array: vortex_array::array::typed::Array, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_alp::ALPRD::execute(vortex_array::array::typed::Array, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_alp::ALPRD::execute_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_alp::ALPRD::execute_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_alp::ALPRD::id(&self) -> vortex_array::array::ArrayId -pub fn vortex_alp::ALPRD::nbuffers(_array: vortex_array::array::view::ArrayView<'_, Self>) -> usize +pub fn vortex_alp::ALPRD::nbuffers(vortex_array::array::view::ArrayView<'_, Self>) -> usize -pub fn vortex_alp::ALPRD::reduce_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_alp::ALPRD::reduce_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_alp::ALPRD::serialize(array: vortex_array::array::view::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_alp::ALPRD::serialize(vortex_array::array::view::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_alp::ALPRD::slot_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_alp::ALPRD::slot_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_alp::ALPRD::validate(&self, data: &vortex_alp::ALPRDData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_alp::ALPRD::validate(&self, &vortex_alp::ALPRDData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::array::vtable::operations::OperationsVTable for vortex_alp::ALPRD -pub fn vortex_alp::ALPRD::scalar_at(array: vortex_array::array::view::ArrayView<'_, vortex_alp::ALPRD>, index: usize, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_alp::ALPRD::scalar_at(vortex_array::array::view::ArrayView<'_, vortex_alp::ALPRD>, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::array::vtable::validity::ValidityChild for vortex_alp::ALPRD -pub fn vortex_alp::ALPRD::validity_child(array: vortex_array::array::view::ArrayView<'_, vortex_alp::ALPRD>) -> vortex_array::array::erased::ArrayRef +pub fn vortex_alp::ALPRD::validity_child(vortex_array::array::view::ArrayView<'_, vortex_alp::ALPRD>) -> vortex_array::array::erased::ArrayRef impl vortex_array::arrays::dict::take::TakeExecute for vortex_alp::ALPRD -pub fn vortex_alp::ALPRD::take(array: vortex_array::array::view::ArrayView<'_, Self>, indices: &vortex_array::array::erased::ArrayRef, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_alp::ALPRD::take(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::kernel::FilterKernel for vortex_alp::ALPRD -pub fn vortex_alp::ALPRD::filter(array: vortex_array::array::view::ArrayView<'_, Self>, mask: &vortex_mask::Mask, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_alp::ALPRD::filter(vortex_array::array::view::ArrayView<'_, Self>, &vortex_mask::Mask, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceKernel for vortex_alp::ALPRD -pub fn vortex_alp::ALPRD::slice(array: vortex_array::array::view::ArrayView<'_, Self>, range: core::ops::range::Range, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_alp::ALPRD::slice(vortex_array::array::view::ArrayView<'_, Self>, core::ops::range::Range, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::kernel::CastReduce for vortex_alp::ALPRD -pub fn vortex_alp::ALPRD::cast(array: vortex_array::array::view::ArrayView<'_, Self>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_alp::ALPRD::cast(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::kernel::MaskReduce for vortex_alp::ALPRD -pub fn vortex_alp::ALPRD::mask(array: vortex_array::array::view::ArrayView<'_, Self>, mask: &vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_alp::ALPRD::mask(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult> pub struct vortex_alp::ALPRDData impl vortex_alp::ALPRDData -pub fn vortex_alp::ALPRDData::into_parts(self, left_parts: vortex_array::array::erased::ArrayRef, right_parts: vortex_array::array::erased::ArrayRef) -> vortex_alp::ALPRDDataParts +pub fn vortex_alp::ALPRDData::into_parts(self, vortex_array::array::erased::ArrayRef, vortex_array::array::erased::ArrayRef) -> vortex_alp::ALPRDDataParts pub fn vortex_alp::ALPRDData::left_parts_dictionary(&self) -> &vortex_buffer::buffer::Buffer -pub fn vortex_alp::ALPRDData::new(left_parts_dictionary: vortex_buffer::buffer::Buffer, right_bit_width: u8, left_parts_patches: core::option::Option) -> Self +pub fn vortex_alp::ALPRDData::new(vortex_buffer::buffer::Buffer, u8, core::option::Option) -> Self pub fn vortex_alp::ALPRDData::right_bit_width(&self) -> u8 @@ -238,19 +232,19 @@ pub fn vortex_alp::ALPRDData::clone(&self) -> vortex_alp::ALPRDData impl core::fmt::Debug for vortex_alp::ALPRDData -pub fn vortex_alp::ALPRDData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_alp::ALPRDData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_alp::ALPRDData -pub fn vortex_alp::ALPRDData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_alp::ALPRDData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::hash::ArrayEq for vortex_alp::ALPRDData -pub fn vortex_alp::ALPRDData::array_eq(&self, other: &Self, precision: vortex_array::hash::Precision) -> bool +pub fn vortex_alp::ALPRDData::array_eq(&self, &Self, vortex_array::hash::Precision) -> bool impl vortex_array::hash::ArrayHash for vortex_alp::ALPRDData -pub fn vortex_alp::ALPRDData::array_hash(&self, state: &mut H, precision: vortex_array::hash::Precision) +pub fn vortex_alp::ALPRDData::array_hash(&self, &mut H, vortex_array::hash::Precision) pub struct vortex_alp::ALPRDDataParts @@ -268,7 +262,7 @@ pub fn vortex_alp::ALPRDDataParts::clone(&self) -> vortex_alp::ALPRDDataParts impl core::fmt::Debug for vortex_alp::ALPRDDataParts -pub fn vortex_alp::ALPRDDataParts::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_alp::ALPRDDataParts::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_alp::ALPRDMetadata @@ -276,7 +270,7 @@ impl vortex_alp::ALPRDMetadata pub fn vortex_alp::ALPRDMetadata::left_parts_ptype(&self) -> vortex_array::dtype::ptype::PType -pub fn vortex_alp::ALPRDMetadata::set_left_parts_ptype(&mut self, value: vortex_array::dtype::ptype::PType) +pub fn vortex_alp::ALPRDMetadata::set_left_parts_ptype(&mut self, vortex_array::dtype::ptype::PType) impl core::clone::Clone for vortex_alp::ALPRDMetadata @@ -288,7 +282,7 @@ pub fn vortex_alp::ALPRDMetadata::default() -> Self impl core::fmt::Debug for vortex_alp::ALPRDMetadata -pub fn vortex_alp::ALPRDMetadata::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_alp::ALPRDMetadata::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl prost::message::Message for vortex_alp::ALPRDMetadata @@ -320,7 +314,7 @@ pub const vortex_alp::ALPSlots::PATCH_INDICES: usize pub const vortex_alp::ALPSlots::PATCH_VALUES: usize -pub fn vortex_alp::ALPSlots::from_slots(slots: alloc::vec::Vec>) -> Self +pub fn vortex_alp::ALPSlots::from_slots(alloc::vec::Vec>) -> Self pub fn vortex_alp::ALPSlots::into_slots(self) -> alloc::vec::Vec> @@ -336,7 +330,7 @@ pub vortex_alp::ALPSlotsView::patch_values: core::option::Option<&'a vortex_arra impl<'a> vortex_alp::ALPSlotsView<'a> -pub fn vortex_alp::ALPSlotsView<'a>::from_slots(slots: &'a [core::option::Option]) -> Self +pub fn vortex_alp::ALPSlotsView<'a>::from_slots(&'a [core::option::Option]) -> Self pub fn vortex_alp::ALPSlotsView<'a>::to_owned(&self) -> vortex_alp::ALPSlots @@ -346,7 +340,7 @@ pub fn vortex_alp::ALPSlotsView<'a>::clone(&self) -> vortex_alp::ALPSlotsView<'a impl<'a> core::fmt::Debug for vortex_alp::ALPSlotsView<'a> -pub fn vortex_alp::ALPSlotsView<'a>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_alp::ALPSlotsView<'a>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl<'a> core::marker::Copy for vortex_alp::ALPSlotsView<'a> @@ -364,19 +358,19 @@ impl core::cmp::Eq for vortex_alp::Exponents impl core::cmp::PartialEq for vortex_alp::Exponents -pub fn vortex_alp::Exponents::eq(&self, other: &vortex_alp::Exponents) -> bool +pub fn vortex_alp::Exponents::eq(&self, &vortex_alp::Exponents) -> bool impl core::fmt::Debug for vortex_alp::Exponents -pub fn vortex_alp::Exponents::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_alp::Exponents::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_alp::Exponents -pub fn vortex_alp::Exponents::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_alp::Exponents::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_alp::Exponents -pub fn vortex_alp::Exponents::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_alp::Exponents::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_alp::Exponents @@ -386,11 +380,11 @@ pub struct vortex_alp::RDEncoder impl vortex_alp::RDEncoder -pub fn vortex_alp::RDEncoder::encode(&self, array: &vortex_array::arrays::primitive::vtable::PrimitiveArray) -> vortex_alp::ALPRDArray +pub fn vortex_alp::RDEncoder::encode(&self, vortex_array::array::view::ArrayView<'_, vortex_array::arrays::primitive::vtable::Primitive>, &mut vortex_array::executor::ExecutionCtx) -> vortex_alp::ALPRDArray -pub fn vortex_alp::RDEncoder::from_parts(right_bit_width: u8, codes: alloc::vec::Vec) -> Self +pub fn vortex_alp::RDEncoder::from_parts(u8, alloc::vec::Vec) -> Self -pub fn vortex_alp::RDEncoder::new(sample: &[T]) -> Self where T: vortex_alp::ALPRDFloat + vortex_array::dtype::ptype::NativePType, ::UINT: vortex_array::dtype::ptype::NativePType +pub fn vortex_alp::RDEncoder::new(&[T]) -> Self where T: vortex_alp::ALPRDFloat + vortex_array::dtype::ptype::NativePType, ::UINT: vortex_array::dtype::ptype::NativePType pub trait vortex_alp::ALPArrayExt: vortex_alp::ALPArraySlotsExt @@ -452,33 +446,33 @@ pub const vortex_alp::ALPFloat::SWEET: Self pub fn vortex_alp::ALPFloat::as_int(self) -> Self::ALPInt -pub fn vortex_alp::ALPFloat::decode(encoded: &[Self::ALPInt], exponents: vortex_alp::Exponents) -> alloc::vec::Vec +pub fn vortex_alp::ALPFloat::decode(&[Self::ALPInt], vortex_alp::Exponents) -> alloc::vec::Vec -pub fn vortex_alp::ALPFloat::decode_buffer(encoded: vortex_buffer::buffer_mut::BufferMut, exponents: vortex_alp::Exponents) -> vortex_buffer::buffer_mut::BufferMut +pub fn vortex_alp::ALPFloat::decode_buffer(vortex_buffer::buffer_mut::BufferMut, vortex_alp::Exponents) -> vortex_buffer::buffer_mut::BufferMut -pub fn vortex_alp::ALPFloat::decode_into(encoded: &[Self::ALPInt], exponents: vortex_alp::Exponents, output: &mut [Self]) +pub fn vortex_alp::ALPFloat::decode_into(&[Self::ALPInt], vortex_alp::Exponents, &mut [Self]) -pub fn vortex_alp::ALPFloat::decode_single(encoded: Self::ALPInt, exponents: vortex_alp::Exponents) -> Self +pub fn vortex_alp::ALPFloat::decode_single(Self::ALPInt, vortex_alp::Exponents) -> Self -pub fn vortex_alp::ALPFloat::decode_slice_inplace(encoded: &mut [Self::ALPInt], exponents: vortex_alp::Exponents) +pub fn vortex_alp::ALPFloat::decode_slice_inplace(&mut [Self::ALPInt], vortex_alp::Exponents) -pub fn vortex_alp::ALPFloat::encode(values: &[Self], exponents: core::option::Option) -> (vortex_alp::Exponents, vortex_buffer::buffer::Buffer, vortex_buffer::buffer::Buffer, vortex_buffer::buffer::Buffer, vortex_buffer::buffer_mut::BufferMut) +pub fn vortex_alp::ALPFloat::encode(&[Self], core::option::Option) -> (vortex_alp::Exponents, vortex_buffer::buffer::Buffer, vortex_buffer::buffer::Buffer, vortex_buffer::buffer::Buffer, vortex_buffer::buffer_mut::BufferMut) -pub fn vortex_alp::ALPFloat::encode_above(value: Self, exponents: vortex_alp::Exponents) -> Self::ALPInt +pub fn vortex_alp::ALPFloat::encode_above(Self, vortex_alp::Exponents) -> Self::ALPInt -pub fn vortex_alp::ALPFloat::encode_below(value: Self, exponents: vortex_alp::Exponents) -> Self::ALPInt +pub fn vortex_alp::ALPFloat::encode_below(Self, vortex_alp::Exponents) -> Self::ALPInt -pub fn vortex_alp::ALPFloat::encode_single(value: Self, exponents: vortex_alp::Exponents) -> core::option::Option +pub fn vortex_alp::ALPFloat::encode_single(Self, vortex_alp::Exponents) -> core::option::Option -pub fn vortex_alp::ALPFloat::encode_single_unchecked(value: Self, exponents: vortex_alp::Exponents) -> Self::ALPInt +pub fn vortex_alp::ALPFloat::encode_single_unchecked(Self, vortex_alp::Exponents) -> Self::ALPInt -pub fn vortex_alp::ALPFloat::estimate_encoded_size(encoded: &[Self::ALPInt], patches: &[Self]) -> usize +pub fn vortex_alp::ALPFloat::estimate_encoded_size(&[Self::ALPInt], &[Self]) -> usize pub fn vortex_alp::ALPFloat::fast_round(self) -> Self -pub fn vortex_alp::ALPFloat::find_best_exponents(values: &[Self]) -> vortex_alp::Exponents +pub fn vortex_alp::ALPFloat::find_best_exponents(&[Self]) -> vortex_alp::Exponents -pub fn vortex_alp::ALPFloat::from_int(n: Self::ALPInt) -> Self +pub fn vortex_alp::ALPFloat::from_int(Self::ALPInt) -> Self impl vortex_alp::ALPFloat for f32 @@ -496,33 +490,33 @@ pub const f32::SWEET: Self pub fn f32::as_int(self) -> Self::ALPInt -pub fn f32::decode(encoded: &[Self::ALPInt], exponents: vortex_alp::Exponents) -> alloc::vec::Vec +pub fn f32::decode(&[Self::ALPInt], vortex_alp::Exponents) -> alloc::vec::Vec -pub fn f32::decode_buffer(encoded: vortex_buffer::buffer_mut::BufferMut, exponents: vortex_alp::Exponents) -> vortex_buffer::buffer_mut::BufferMut +pub fn f32::decode_buffer(vortex_buffer::buffer_mut::BufferMut, vortex_alp::Exponents) -> vortex_buffer::buffer_mut::BufferMut -pub fn f32::decode_into(encoded: &[Self::ALPInt], exponents: vortex_alp::Exponents, output: &mut [Self]) +pub fn f32::decode_into(&[Self::ALPInt], vortex_alp::Exponents, &mut [Self]) -pub fn f32::decode_single(encoded: Self::ALPInt, exponents: vortex_alp::Exponents) -> Self +pub fn f32::decode_single(Self::ALPInt, vortex_alp::Exponents) -> Self -pub fn f32::decode_slice_inplace(encoded: &mut [Self::ALPInt], exponents: vortex_alp::Exponents) +pub fn f32::decode_slice_inplace(&mut [Self::ALPInt], vortex_alp::Exponents) -pub fn f32::encode(values: &[Self], exponents: core::option::Option) -> (vortex_alp::Exponents, vortex_buffer::buffer::Buffer, vortex_buffer::buffer::Buffer, vortex_buffer::buffer::Buffer, vortex_buffer::buffer_mut::BufferMut) +pub fn f32::encode(&[Self], core::option::Option) -> (vortex_alp::Exponents, vortex_buffer::buffer::Buffer, vortex_buffer::buffer::Buffer, vortex_buffer::buffer::Buffer, vortex_buffer::buffer_mut::BufferMut) -pub fn f32::encode_above(value: Self, exponents: vortex_alp::Exponents) -> Self::ALPInt +pub fn f32::encode_above(Self, vortex_alp::Exponents) -> Self::ALPInt -pub fn f32::encode_below(value: Self, exponents: vortex_alp::Exponents) -> Self::ALPInt +pub fn f32::encode_below(Self, vortex_alp::Exponents) -> Self::ALPInt -pub fn f32::encode_single(value: Self, exponents: vortex_alp::Exponents) -> core::option::Option +pub fn f32::encode_single(Self, vortex_alp::Exponents) -> core::option::Option -pub fn f32::encode_single_unchecked(value: Self, exponents: vortex_alp::Exponents) -> Self::ALPInt +pub fn f32::encode_single_unchecked(Self, vortex_alp::Exponents) -> Self::ALPInt -pub fn f32::estimate_encoded_size(encoded: &[Self::ALPInt], patches: &[Self]) -> usize +pub fn f32::estimate_encoded_size(&[Self::ALPInt], &[Self]) -> usize pub fn f32::fast_round(self) -> Self -pub fn f32::find_best_exponents(values: &[Self]) -> vortex_alp::Exponents +pub fn f32::find_best_exponents(&[Self]) -> vortex_alp::Exponents -pub fn f32::from_int(n: Self::ALPInt) -> Self +pub fn f32::from_int(Self::ALPInt) -> Self impl vortex_alp::ALPFloat for f64 @@ -540,33 +534,33 @@ pub const f64::SWEET: Self pub fn f64::as_int(self) -> Self::ALPInt -pub fn f64::decode(encoded: &[Self::ALPInt], exponents: vortex_alp::Exponents) -> alloc::vec::Vec +pub fn f64::decode(&[Self::ALPInt], vortex_alp::Exponents) -> alloc::vec::Vec -pub fn f64::decode_buffer(encoded: vortex_buffer::buffer_mut::BufferMut, exponents: vortex_alp::Exponents) -> vortex_buffer::buffer_mut::BufferMut +pub fn f64::decode_buffer(vortex_buffer::buffer_mut::BufferMut, vortex_alp::Exponents) -> vortex_buffer::buffer_mut::BufferMut -pub fn f64::decode_into(encoded: &[Self::ALPInt], exponents: vortex_alp::Exponents, output: &mut [Self]) +pub fn f64::decode_into(&[Self::ALPInt], vortex_alp::Exponents, &mut [Self]) -pub fn f64::decode_single(encoded: Self::ALPInt, exponents: vortex_alp::Exponents) -> Self +pub fn f64::decode_single(Self::ALPInt, vortex_alp::Exponents) -> Self -pub fn f64::decode_slice_inplace(encoded: &mut [Self::ALPInt], exponents: vortex_alp::Exponents) +pub fn f64::decode_slice_inplace(&mut [Self::ALPInt], vortex_alp::Exponents) -pub fn f64::encode(values: &[Self], exponents: core::option::Option) -> (vortex_alp::Exponents, vortex_buffer::buffer::Buffer, vortex_buffer::buffer::Buffer, vortex_buffer::buffer::Buffer, vortex_buffer::buffer_mut::BufferMut) +pub fn f64::encode(&[Self], core::option::Option) -> (vortex_alp::Exponents, vortex_buffer::buffer::Buffer, vortex_buffer::buffer::Buffer, vortex_buffer::buffer::Buffer, vortex_buffer::buffer_mut::BufferMut) -pub fn f64::encode_above(value: Self, exponents: vortex_alp::Exponents) -> Self::ALPInt +pub fn f64::encode_above(Self, vortex_alp::Exponents) -> Self::ALPInt -pub fn f64::encode_below(value: Self, exponents: vortex_alp::Exponents) -> Self::ALPInt +pub fn f64::encode_below(Self, vortex_alp::Exponents) -> Self::ALPInt -pub fn f64::encode_single(value: Self, exponents: vortex_alp::Exponents) -> core::option::Option +pub fn f64::encode_single(Self, vortex_alp::Exponents) -> core::option::Option -pub fn f64::encode_single_unchecked(value: Self, exponents: vortex_alp::Exponents) -> Self::ALPInt +pub fn f64::encode_single_unchecked(Self, vortex_alp::Exponents) -> Self::ALPInt -pub fn f64::estimate_encoded_size(encoded: &[Self::ALPInt], patches: &[Self]) -> usize +pub fn f64::estimate_encoded_size(&[Self::ALPInt], &[Self]) -> usize pub fn f64::fast_round(self) -> Self -pub fn f64::find_best_exponents(values: &[Self]) -> vortex_alp::Exponents +pub fn f64::find_best_exponents(&[Self]) -> vortex_alp::Exponents -pub fn f64::from_int(n: Self::ALPInt) -> Self +pub fn f64::from_int(Self::ALPInt) -> Self pub trait vortex_alp::ALPRDArrayExt: vortex_array::array::typed::TypedArrayRef @@ -606,13 +600,13 @@ pub type vortex_alp::ALPRDFloat::UINT: vortex_array::dtype::ptype::NativePType + pub const vortex_alp::ALPRDFloat::BITS: usize -pub fn vortex_alp::ALPRDFloat::from_bits(bits: Self::UINT) -> Self +pub fn vortex_alp::ALPRDFloat::from_bits(Self::UINT) -> Self -pub fn vortex_alp::ALPRDFloat::from_u16(value: u16) -> Self::UINT +pub fn vortex_alp::ALPRDFloat::from_u16(u16) -> Self::UINT -pub fn vortex_alp::ALPRDFloat::to_bits(value: Self) -> Self::UINT +pub fn vortex_alp::ALPRDFloat::to_bits(Self) -> Self::UINT -pub fn vortex_alp::ALPRDFloat::to_u16(bits: Self::UINT) -> u16 +pub fn vortex_alp::ALPRDFloat::to_u16(Self::UINT) -> u16 impl vortex_alp::ALPRDFloat for f32 @@ -620,13 +614,13 @@ pub type f32::UINT = u32 pub const f32::BITS: usize -pub fn f32::from_bits(bits: Self::UINT) -> Self +pub fn f32::from_bits(Self::UINT) -> Self -pub fn f32::from_u16(value: u16) -> Self::UINT +pub fn f32::from_u16(u16) -> Self::UINT -pub fn f32::to_bits(value: Self) -> Self::UINT +pub fn f32::to_bits(Self) -> Self::UINT -pub fn f32::to_u16(bits: Self::UINT) -> u16 +pub fn f32::to_u16(Self::UINT) -> u16 impl vortex_alp::ALPRDFloat for f64 @@ -634,21 +628,21 @@ pub type f64::UINT = u64 pub const f64::BITS: usize -pub fn f64::from_bits(bits: Self::UINT) -> Self +pub fn f64::from_bits(Self::UINT) -> Self -pub fn f64::from_u16(value: u16) -> Self::UINT +pub fn f64::from_u16(u16) -> Self::UINT -pub fn f64::to_bits(value: Self) -> Self::UINT +pub fn f64::to_bits(Self) -> Self::UINT -pub fn f64::to_u16(bits: Self::UINT) -> u16 +pub fn f64::to_u16(Self::UINT) -> u16 -pub fn vortex_alp::alp_encode(parray: &vortex_array::arrays::primitive::vtable::PrimitiveArray, exponents: core::option::Option) -> vortex_error::VortexResult +pub fn vortex_alp::alp_encode(vortex_array::array::view::ArrayView<'_, vortex_array::arrays::primitive::vtable::Primitive>, core::option::Option, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_alp::alp_rd_decode(left_parts: vortex_buffer::buffer_mut::BufferMut, left_parts_dict: &[u16], right_bit_width: u8, right_parts: vortex_buffer::buffer_mut::BufferMut<::UINT>, left_parts_patches: core::option::Option, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_alp::alp_rd_decode(vortex_buffer::buffer_mut::BufferMut, &[u16], u8, vortex_buffer::buffer_mut::BufferMut<::UINT>, core::option::Option, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> -pub fn vortex_alp::decompress_into_array(array: vortex_alp::ALPArray, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_alp::decompress_into_array(vortex_alp::ALPArray, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_alp::initialize(session: &vortex_session::VortexSession) +pub fn vortex_alp::initialize(&vortex_session::VortexSession) pub type vortex_alp::ALPArray = vortex_array::array::typed::Array diff --git a/encodings/alp/src/alp/array.rs b/encodings/alp/src/alp/array.rs index 73852549965..63549dc24f8 100644 --- a/encodings/alp/src/alp/array.rs +++ b/encodings/alp/src/alp/array.rs @@ -39,6 +39,7 @@ use vortex_error::vortex_bail; use vortex_error::vortex_ensure; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::ALPFloat; use crate::alp::Exponents; @@ -72,7 +73,8 @@ impl VTable for ALP { type ValidityVTable = ValidityVTableFromChild; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.alp"); + *ID } fn validate( @@ -158,7 +160,7 @@ impl VTable for ALP { }) .transpose()?; - let slots = ALPData::make_slots(&encoded, &patches); + let slots = ALPData::make_slots(&encoded, patches.as_ref()); let data = ALPData::new( Exponents { e: u8::try_from(metadata.exp_e)?, @@ -237,10 +239,6 @@ impl Display for ALPData { #[derive(Clone, Debug)] pub struct ALP; -impl ALP { - pub const ID: ArrayId = ArrayId::new_ref("vortex.alp"); -} - #[derive(Clone, prost::Message)] pub struct ALPMetadata { #[prost(uint32, tag = "1")] @@ -373,7 +371,7 @@ impl ALP { pub fn new(encoded: ArrayRef, exponents: Exponents, patches: Option) -> ALPArray { let dtype = ALPData::logical_dtype(&encoded).vortex_expect("ALP encoded dtype"); let len = encoded.len(); - let slots = ALPData::make_slots(&encoded, &patches); + let slots = ALPData::make_slots(&encoded, patches.as_ref()); unsafe { Array::from_parts_unchecked( ArrayParts::new(ALP, dtype, len, ALPData::new(exponents, patches)) @@ -389,7 +387,7 @@ impl ALP { ) -> VortexResult { let dtype = ALPData::logical_dtype(&encoded)?; let len = encoded.len(); - let slots = ALPData::make_slots(&encoded, &patches); + let slots = ALPData::make_slots(&encoded, patches.as_ref()); let data = ALPData::new(exponents, patches); Array::try_from_parts(ArrayParts::new(ALP, dtype, len, data).with_slots(slots)) } @@ -403,7 +401,7 @@ impl ALP { ) -> ALPArray { let dtype = ALPData::logical_dtype(&encoded).vortex_expect("ALP encoded dtype"); let len = encoded.len(); - let slots = ALPData::make_slots(&encoded, &patches); + let slots = ALPData::make_slots(&encoded, patches.as_ref()); let data = unsafe { ALPData::new_unchecked(exponents, patches) }; unsafe { Array::from_parts_unchecked(ArrayParts::new(ALP, dtype, len, data).with_slots(slots)) @@ -412,7 +410,7 @@ impl ALP { } impl ALPData { - fn make_slots(encoded: &ArrayRef, patches: &Option) -> Vec> { + fn make_slots(encoded: &ArrayRef, patches: Option<&Patches>) -> Vec> { let (patch_indices, patch_values, patch_chunk_offsets) = match patches { Some(p) => ( Some(p.indices().clone()), @@ -528,7 +526,6 @@ mod tests { use vortex_array::Canonical; use vortex_array::IntoArray; use vortex_array::LEGACY_SESSION; - use vortex_array::ToCanonical; use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; @@ -554,11 +551,11 @@ mod tests { #[case(2048)] #[case(2049)] fn test_execute_f32(#[case] size: usize) { + let mut ctx = SESSION.create_execution_ctx(); let values = PrimitiveArray::from_iter((0..size).map(|i| i as f32)); - let encoded = alp_encode(&values, None).unwrap(); + let encoded = alp_encode(values.as_view(), None, &mut ctx).unwrap(); let result_canonical = { - let mut ctx = SESSION.create_execution_ctx(); encoded .clone() .into_array() @@ -584,7 +581,12 @@ mod tests { #[case(2049)] fn test_execute_f64(#[case] size: usize) { let values = PrimitiveArray::from_iter((0..size).map(|i| i as f64)); - let encoded = alp_encode(&values, None).unwrap(); + let encoded = alp_encode( + values.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); let result_canonical = { let mut ctx = SESSION.create_execution_ctx(); @@ -618,7 +620,12 @@ mod tests { .collect(); let array = PrimitiveArray::from_iter(values); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode( + array.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); assert!(encoded.patches().unwrap().array_len() > 0); let result_canonical = { @@ -652,7 +659,12 @@ mod tests { .collect(); let array = PrimitiveArray::from_option_iter(values); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode( + array.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); let result_canonical = { let mut ctx = SESSION.create_execution_ctx(); @@ -687,7 +699,12 @@ mod tests { .collect(); let array = PrimitiveArray::from_option_iter(values); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode( + array.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); assert!(encoded.patches().unwrap().array_len() > 0); let result_canonical = { @@ -723,7 +740,12 @@ mod tests { .collect(); let array = PrimitiveArray::from_option_iter(values.clone()); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode( + array.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); let slice_end = size - slice_start; let slice_len = slice_end - slice_start; @@ -761,6 +783,7 @@ mod tests { #[case(1000, 200)] #[case(2048, 512)] fn test_sliced_to_primitive(#[case] size: usize, #[case] slice_start: usize) { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values: Vec> = (0..size) .map(|i| { if i % 5 == 0 { @@ -774,18 +797,24 @@ mod tests { .collect(); let array = PrimitiveArray::from_option_iter(values.clone()); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode(array.as_view(), None, &mut ctx).unwrap(); let slice_end = size - slice_start; let slice_len = slice_end - slice_start; let sliced_encoded = encoded.slice(slice_start..slice_end).unwrap(); - let result_primitive = sliced_encoded.to_primitive(); + let result_primitive = sliced_encoded.execute::(&mut ctx).unwrap(); for idx in 0..slice_len { let expected_value = values[slice_start + idx]; - let result_valid = result_primitive.validity_mask().unwrap().value(idx); + let result_valid = result_primitive + .as_ref() + .validity() + .unwrap() + .execute_mask(result_primitive.as_ref().len(), &mut ctx) + .unwrap() + .value(idx); assert_eq!( result_valid, expected_value.is_some(), @@ -815,7 +844,12 @@ mod tests { let original = PrimitiveArray::from_iter(values); // First encode normally to get a properly formed ALPArray with patches. - let normally_encoded = alp_encode(&original, None).unwrap(); + let normally_encoded = alp_encode( + original.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); assert!( normally_encoded.patches().is_some(), "Test requires patches to be present" diff --git a/encodings/alp/src/alp/compress.rs b/encodings/alp/src/alp/compress.rs index b5c259aeef1..2b4546f98de 100644 --- a/encodings/alp/src/alp/compress.rs +++ b/encodings/alp/src/alp/compress.rs @@ -3,7 +3,10 @@ use itertools::Itertools; use vortex_array::ArrayRef; +use vortex_array::ArrayView; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; +use vortex_array::arrays::Primitive; use vortex_array::arrays::PrimitiveArray; use vortex_array::dtype::PType; use vortex_array::patches::Patches; @@ -39,10 +42,14 @@ macro_rules! match_each_alp_float_ptype { }}; } -pub fn alp_encode(parray: &PrimitiveArray, exponents: Option) -> VortexResult { +pub fn alp_encode( + parray: ArrayView<'_, Primitive>, + exponents: Option, + ctx: &mut ExecutionCtx, +) -> VortexResult { let (exponents, encoded, patches) = match parray.ptype() { - PType::F32 => alp_encode_components_typed::(parray, exponents)?, - PType::F64 => alp_encode_components_typed::(parray, exponents)?, + PType::F32 => alp_encode_components_typed::(parray, exponents, ctx)?, + PType::F64 => alp_encode_components_typed::(parray, exponents, ctx)?, _ => vortex_bail!("ALP can only encode f32 and f64"), }; @@ -55,8 +62,9 @@ pub fn alp_encode(parray: &PrimitiveArray, exponents: Option) -> Vort reason = "u64 index cast to usize is safe for reasonable array sizes" )] fn alp_encode_components_typed( - values: &PrimitiveArray, + values: ArrayView<'_, Primitive>, exponents: Option, + ctx: &mut ExecutionCtx, ) -> VortexResult<(Exponents, ArrayRef, Option)> where T: ALPFloat, @@ -68,7 +76,10 @@ where let encoded_array = PrimitiveArray::new(encoded, values.validity()?).into_array(); - let validity = values.validity_mask()?; + let validity = values + .array() + .validity()? + .execute_mask(values.array().len(), ctx)?; // exceptional_positions may contain exceptions at invalid positions (which contain garbage // data). We remove null exceptions in order to keep the Patches small. let (valid_exceptional_positions, valid_exceptional_values): (Buffer, Buffer) = @@ -125,7 +136,6 @@ mod tests { use f64::consts::E; use f64::consts::PI; use vortex_array::LEGACY_SESSION; - use vortex_array::ToCanonical; use vortex_array::VortexSessionExecute; use vortex_array::assert_arrays_eq; use vortex_array::dtype::NativePType; @@ -141,7 +151,12 @@ mod tests { #[test] fn test_compress() { let array = PrimitiveArray::new(buffer![1.234f32; 1025], Validity::NonNullable); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode( + array.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); assert!(encoded.patches().is_none()); let expected_encoded = PrimitiveArray::from_iter(vec![1234i32; 1025]); assert_arrays_eq!(encoded.encoded(), expected_encoded); @@ -155,7 +170,12 @@ mod tests { #[test] fn test_nullable_compress() { let array = PrimitiveArray::from_option_iter([None, Some(1.234f32), None]); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode( + array.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); assert!(encoded.patches().is_none()); let expected_encoded = PrimitiveArray::from_option_iter([None, Some(1234i32), None]); assert_arrays_eq!(encoded.encoded(), expected_encoded); @@ -168,11 +188,16 @@ mod tests { } #[test] - #[allow(clippy::approx_constant)] // Clippy objects to 2.718, an approximation of e, the base of the natural logarithm. + #[expect(clippy::approx_constant)] // Clippy objects to 2.718, an approximation of e, the base of the natural logarithm. fn test_patched_compress() { let values = buffer![1.234f64, 2.718, PI, 4.0]; let array = PrimitiveArray::new(values.clone(), Validity::NonNullable); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode( + array.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); assert!(encoded.patches().is_some()); let expected_encoded = PrimitiveArray::from_iter(vec![1234i64, 2718, 1234, 4000]); assert_arrays_eq!(encoded.encoded(), expected_encoded); @@ -185,11 +210,16 @@ mod tests { } #[test] - #[allow(clippy::approx_constant)] // Clippy objects to 2.718, an approximation of e, the base of the natural logarithm. + #[expect(clippy::approx_constant)] // Clippy objects to 2.718, an approximation of e, the base of the natural logarithm. fn test_compress_ignores_invalid_exceptional_values() { let values = buffer![1.234f64, 2.718, PI, 4.0]; let array = PrimitiveArray::new(values, Validity::from_iter([true, true, false, true])); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode( + array.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); assert!(encoded.patches().is_none()); let expected_encoded = PrimitiveArray::from_option_iter(buffer![Some(1234i64), Some(2718), None, Some(4000)]); @@ -202,7 +232,7 @@ mod tests { } #[test] - #[allow(clippy::approx_constant)] // ALP doesn't like E + #[expect(clippy::approx_constant)] // ALP doesn't like E fn test_nullable_patched_scalar_at() { let array = PrimitiveArray::from_option_iter([ Some(1.234f64), @@ -211,7 +241,12 @@ mod tests { Some(4.0), None, ]); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode( + array.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); assert!(encoded.patches().is_some()); assert_eq!(encoded.exponents(), Exponents { e: 16, f: 13 }); @@ -225,16 +260,25 @@ mod tests { #[test] fn roundtrips_close_fractional() { let original = PrimitiveArray::from_iter([195.26274f32, 195.27837, -48.815685]); - let alp_arr = alp_encode(&original, None).unwrap(); + let alp_arr = alp_encode( + original.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); assert_arrays_eq!(alp_arr, original); } #[test] fn roundtrips_all_null() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let original = PrimitiveArray::new(buffer![195.26274f64, PI, -48.815685], Validity::AllInvalid); - let alp_arr = alp_encode(&original, None).unwrap(); - let decompressed = alp_arr.into_array().to_primitive(); + let alp_arr = alp_encode(original.as_view(), None, &mut ctx).unwrap(); + let decompressed = alp_arr + .into_array() + .execute::(&mut ctx) + .unwrap(); assert_eq!( // The second and third values become exceptions and are replaced @@ -247,12 +291,17 @@ mod tests { #[test] fn non_finite_numbers() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let original = PrimitiveArray::new( buffer![0.0f32, -0.0, f32::NAN, f32::NEG_INFINITY, f32::INFINITY], Validity::NonNullable, ); - let encoded = alp_encode(&original, None).unwrap(); - let decoded = encoded.as_array().to_primitive(); + let encoded = alp_encode(original.as_view(), None, &mut ctx).unwrap(); + let decoded = encoded + .as_array() + .clone() + .execute::(&mut ctx) + .unwrap(); for idx in 0..original.len() { let decoded_val = decoded.as_slice::()[idx]; let original_val = original.as_slice::()[idx]; @@ -265,6 +314,7 @@ mod tests { #[test] fn test_chunk_offsets() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let mut values = vec![1.0f64; 3072]; values[1023] = PI; @@ -272,86 +322,141 @@ mod tests { values[1025] = PI; let array = PrimitiveArray::new(Buffer::from(values), Validity::NonNullable); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode(array.as_view(), None, &mut ctx).unwrap(); let patches = encoded.patches().unwrap(); - let chunk_offsets = patches.chunk_offsets().clone().unwrap().to_primitive(); + let chunk_offsets = patches + .chunk_offsets() + .clone() + .unwrap() + .execute::(&mut ctx) + .unwrap(); let expected_offsets = PrimitiveArray::from_iter(vec![0u64, 1, 3]); assert_arrays_eq!(chunk_offsets, expected_offsets); - let patch_indices = patches.indices().to_primitive(); + let patch_indices = patches + .indices() + .clone() + .execute::(&mut ctx) + .unwrap(); let expected_indices = PrimitiveArray::from_iter(vec![1023u64, 1024, 1025]); assert_arrays_eq!(patch_indices, expected_indices); - let patch_values = patches.values().to_primitive(); + let patch_values = patches + .values() + .clone() + .execute::(&mut ctx) + .unwrap(); let expected_values = PrimitiveArray::from_iter(vec![PI, E, PI]); assert_arrays_eq!(patch_values, expected_values); } #[test] fn test_chunk_offsets_no_patches_in_middle() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let mut values = vec![1.0f64; 3072]; values[0] = PI; values[2048] = E; let array = PrimitiveArray::new(Buffer::from(values), Validity::NonNullable); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode(array.as_view(), None, &mut ctx).unwrap(); let patches = encoded.patches().unwrap(); - let chunk_offsets = patches.chunk_offsets().clone().unwrap().to_primitive(); + let chunk_offsets = patches + .chunk_offsets() + .clone() + .unwrap() + .execute::(&mut ctx) + .unwrap(); let expected_offsets = PrimitiveArray::from_iter(vec![0u64, 1, 1]); assert_arrays_eq!(chunk_offsets, expected_offsets); - let patch_indices = patches.indices().to_primitive(); + let patch_indices = patches + .indices() + .clone() + .execute::(&mut ctx) + .unwrap(); let expected_indices = PrimitiveArray::from_iter(vec![0u64, 2048]); assert_arrays_eq!(patch_indices, expected_indices); - let patch_values = patches.values().to_primitive(); + let patch_values = patches + .values() + .clone() + .execute::(&mut ctx) + .unwrap(); let expected_values = PrimitiveArray::from_iter(vec![PI, E]); assert_arrays_eq!(patch_values, expected_values); } #[test] fn test_chunk_offsets_trailing_empty_chunks() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let mut values = vec![1.0f64; 3072]; values[0] = PI; let array = PrimitiveArray::new(Buffer::from(values), Validity::NonNullable); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode(array.as_view(), None, &mut ctx).unwrap(); let patches = encoded.patches().unwrap(); - let chunk_offsets = patches.chunk_offsets().clone().unwrap().to_primitive(); + let chunk_offsets = patches + .chunk_offsets() + .clone() + .unwrap() + .execute::(&mut ctx) + .unwrap(); let expected_offsets = PrimitiveArray::from_iter(vec![0u64, 1, 1]); assert_arrays_eq!(chunk_offsets, expected_offsets); - let patch_indices = patches.indices().to_primitive(); + let patch_indices = patches + .indices() + .clone() + .execute::(&mut ctx) + .unwrap(); let expected_indices = PrimitiveArray::from_iter(vec![0u64]); assert_arrays_eq!(patch_indices, expected_indices); - let patch_values = patches.values().to_primitive(); + let patch_values = patches + .values() + .clone() + .execute::(&mut ctx) + .unwrap(); let expected_values = PrimitiveArray::from_iter(vec![PI]); assert_arrays_eq!(patch_values, expected_values); } #[test] fn test_chunk_offsets_single_chunk() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let mut values = vec![1.0f64; 512]; values[0] = PI; values[100] = E; let array = PrimitiveArray::new(Buffer::from(values), Validity::NonNullable); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode(array.as_view(), None, &mut ctx).unwrap(); let patches = encoded.patches().unwrap(); - let chunk_offsets = patches.chunk_offsets().clone().unwrap().to_primitive(); + let chunk_offsets = patches + .chunk_offsets() + .clone() + .unwrap() + .execute::(&mut ctx) + .unwrap(); let expected_offsets = PrimitiveArray::from_iter(vec![0u64]); assert_arrays_eq!(chunk_offsets, expected_offsets); - let patch_indices = patches.indices().to_primitive(); + let patch_indices = patches + .indices() + .clone() + .execute::(&mut ctx) + .unwrap(); let expected_indices = PrimitiveArray::from_iter(vec![0u64, 100]); assert_arrays_eq!(patch_indices, expected_indices); - let patch_values = patches.values().to_primitive(); + let patch_values = patches + .values() + .clone() + .execute::(&mut ctx) + .unwrap(); let expected_values = PrimitiveArray::from_iter(vec![PI, E]); assert_arrays_eq!(patch_values, expected_values); } @@ -361,7 +466,12 @@ mod tests { // Create 1024 elements, encode, slice to first 512, then decode let values = vec![1.234f32; 1024]; let original = PrimitiveArray::new(Buffer::from(values), Validity::NonNullable); - let encoded = alp_encode(&original, None).unwrap(); + let encoded = alp_encode( + original.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); let sliced_alp = encoded.slice(512..1024).unwrap(); @@ -373,7 +483,12 @@ mod tests { fn test_slice_half_chunk_f64_roundtrip() { let values = vec![5.678f64; 1024]; let original = PrimitiveArray::new(Buffer::from(values), Validity::NonNullable); - let encoded = alp_encode(&original, None).unwrap(); + let encoded = alp_encode( + original.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); let sliced_alp = encoded.slice(512..1024).unwrap(); @@ -389,7 +504,12 @@ mod tests { values[600] = 42.42; let original = PrimitiveArray::new(Buffer::from(values), Validity::NonNullable); - let encoded = alp_encode(&original, None).unwrap(); + let encoded = alp_encode( + original.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); let sliced_alp = encoded.slice(512..1024).unwrap(); @@ -409,7 +529,12 @@ mod tests { values[1023] = 42.42; let original = PrimitiveArray::new(Buffer::from(values), Validity::NonNullable); - let encoded = alp_encode(&original, None).unwrap(); + let encoded = alp_encode( + original.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); let sliced_alp = encoded.slice(1023..1025).unwrap(); @@ -420,15 +545,16 @@ mod tests { #[test] fn test_slice_half_chunk_nullable_roundtrip() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values = (0..1024) .map(|i| if i % 3 == 0 { None } else { Some(2.5f32) }) .collect::>(); let original = PrimitiveArray::from_option_iter(values); - let encoded = alp_encode(&original, None).unwrap(); + let encoded = alp_encode(original.as_view(), None, &mut ctx).unwrap(); let sliced_alp = encoded.slice(512..1024).unwrap(); - let decoded = sliced_alp.to_primitive(); + let decoded = sliced_alp.execute::(&mut ctx).unwrap(); let expected_slice = original.slice(512..1024).unwrap(); assert_arrays_eq!(decoded, expected_slice); @@ -438,7 +564,12 @@ mod tests { fn test_large_f32_array_uniform_values() { let size = 10_000; let array = PrimitiveArray::new(buffer![42.125f32; size], Validity::NonNullable); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode( + array.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); assert!(encoded.patches().is_none()); let decoded = @@ -450,7 +581,12 @@ mod tests { fn test_large_f64_array_uniform_values() { let size = 50_000; let array = PrimitiveArray::new(buffer![123.456789f64; size], Validity::NonNullable); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode( + array.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); assert!(encoded.patches().is_none()); let decoded = @@ -468,7 +604,12 @@ mod tests { values[4500] = f32::INFINITY; let array = PrimitiveArray::new(Buffer::from(values), Validity::NonNullable); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode( + array.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); assert!(encoded.patches().is_some()); let decoded = @@ -490,7 +631,12 @@ mod tests { values[7000] = 999.999999999; let array = PrimitiveArray::new(Buffer::from(values.clone()), Validity::NonNullable); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode( + array.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); assert!(encoded.patches().is_some()); let decoded = @@ -520,7 +666,12 @@ mod tests { .collect(); let array = PrimitiveArray::from_option_iter(values); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode( + array.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); let decoded = decompress_into_array(encoded, &mut LEGACY_SESSION.create_execution_ctx()).unwrap(); @@ -541,7 +692,12 @@ mod tests { let validity = Validity::from_iter((0..size).map(|i| !matches!(i, 500 | 2500))); let array = PrimitiveArray::new(Buffer::from(values), validity); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode( + array.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); let decoded = decompress_into_array(encoded, &mut LEGACY_SESSION.create_execution_ctx()).unwrap(); @@ -572,7 +728,12 @@ mod tests { values[2900] = PI; let original = PrimitiveArray::new(Buffer::from(values), Validity::NonNullable); - let encoded = alp_encode(&original, None).unwrap(); + let encoded = alp_encode( + original.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); assert!(encoded.patches().is_some()); // Slice ending mid-chunk-2 (element 2500 is inside chunk 2 = 2048..3072). diff --git a/encodings/alp/src/alp/compute/between.rs b/encodings/alp/src/alp/compute/between.rs index c1922a719f9..7f35f67d51c 100644 --- a/encodings/alp/src/alp/compute/between.rs +++ b/encodings/alp/src/alp/compute/between.rs @@ -95,6 +95,8 @@ where #[cfg(test)] mod tests { + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; @@ -123,7 +125,12 @@ mod tests { fn comparison_range() { let value = 0.0605_f32; let array = PrimitiveArray::from_iter([value; 1]); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode( + array.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); assert!(encoded.patches().is_none()); assert_between( diff --git a/encodings/alp/src/alp/compute/cast.rs b/encodings/alp/src/alp/compute/cast.rs index 83ee609f886..71b51ecf9af 100644 --- a/encodings/alp/src/alp/compute/cast.rs +++ b/encodings/alp/src/alp/compute/cast.rs @@ -6,7 +6,6 @@ use vortex_array::ArrayView; use vortex_array::IntoArray; use vortex_array::builtins::ArrayBuiltins; use vortex_array::dtype::DType; -use vortex_array::patches::Patches; use vortex_array::scalar_fn::fns::cast::CastReduce; use vortex_error::VortexResult; @@ -17,41 +16,29 @@ use crate::alp::ALP; impl CastReduce for ALP { fn cast(array: ArrayView<'_, Self>, dtype: &DType) -> VortexResult> { // Check if this is just a nullability change - if array.dtype().eq_ignore_nullability(dtype) { - // For nullability-only changes, we can avoid decoding - // Cast the encoded array (integers) to handle nullability - let new_encoded = array.encoded().cast( - array - .encoded() - .dtype() - .with_nullability(dtype.nullability()), - )?; - - let new_patches = array - .patches() - .map(|p| { - if p.values().dtype() == dtype { - Ok(p) - } else { - Patches::new( - p.array_len(), - p.offset(), - p.indices().clone(), - p.values().cast(dtype.clone())?, - p.chunk_offsets().clone(), - ) - } - }) - .transpose()?; - - // SAFETY: casting nullability doesn't alter the invariants - unsafe { - Ok(Some( - ALP::new_unchecked(new_encoded, array.exponents(), new_patches).into_array(), - )) - } - } else { - Ok(None) + if !array.dtype().eq_ignore_nullability(dtype) { + return Ok(None); + } + + // For nullability-only changes, we can avoid decoding + // Cast the encoded array (integers) to handle nullability + let new_encoded = array.encoded().cast( + array + .encoded() + .dtype() + .with_nullability(dtype.nullability()), + )?; + + let new_patches = array + .patches() + .map(|p| p.map_values(|v| v.cast(dtype.clone()))) + .transpose()?; + + // SAFETY: casting nullability doesn't alter the invariants + unsafe { + Ok(Some( + ALP::new_unchecked(new_encoded, array.exponents(), new_patches).into_array(), + )) } } } @@ -60,7 +47,8 @@ impl CastReduce for ALP { mod tests { use rstest::rstest; use vortex_array::IntoArray; - use vortex_array::ToCanonical; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; use vortex_array::builtins::ArrayBuiltins; @@ -77,8 +65,10 @@ mod tests { #[test] fn issue_5766_test_cast_alp_with_patches_to_nullable() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values = buffer![1.234f32, f32::NAN, 2.345, f32::INFINITY, 3.456].into_array(); - let alp = alp_encode(&values.to_primitive(), None)?; + let values_primitive = values.clone().execute::(&mut ctx)?; + let alp = alp_encode(values_primitive.as_view(), None, &mut ctx)?; assert!( alp.patches().is_some(), @@ -90,15 +80,18 @@ mod tests { let expected = values.cast(nullable_dtype)?; - assert_arrays_eq!(casted.to_canonical()?.into_primitive(), expected); + let casted_prim = casted.execute::(&mut ctx)?; + assert_arrays_eq!(casted_prim, expected); Ok(()) } #[test] fn test_cast_alp_f32_to_f64() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values = buffer![1.5f32, 2.5, 3.5, 4.5].into_array(); - let alp = alp_encode(&values.to_primitive(), None)?; + let values_primitive = values.execute::(&mut ctx)?; + let alp = alp_encode(values_primitive.as_view(), None, &mut ctx)?; let casted = alp .into_array() @@ -108,7 +101,7 @@ mod tests { &DType::Primitive(PType::F64, Nullability::NonNullable) ); - let decoded = casted.to_canonical()?.into_primitive(); + let decoded = casted.execute::(&mut ctx)?; let values = decoded.as_slice::(); assert_eq!(values.len(), 4); assert!((values[0] - 1.5).abs() < f64::EPSILON); @@ -119,8 +112,10 @@ mod tests { #[test] fn test_cast_alp_to_int() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values = buffer![1.0f32, 2.0, 3.0, 4.0].into_array(); - let alp = alp_encode(&values.to_primitive(), None)?; + let values_primitive = values.execute::(&mut ctx)?; + let alp = alp_encode(values_primitive.as_view(), None, &mut ctx)?; let casted = alp .into_array() @@ -130,7 +125,7 @@ mod tests { &DType::Primitive(PType::I32, Nullability::NonNullable) ); - let decoded = casted.to_canonical()?.into_primitive(); + let decoded = casted.execute::(&mut ctx)?; assert_arrays_eq!(decoded, PrimitiveArray::from_iter([1i32, 2, 3, 4])); Ok(()) @@ -143,7 +138,10 @@ mod tests { #[case(buffer![42.42f64].into_array())] #[case(buffer![0.0f32, -1.5, 2.5, -3.5, 4.5].into_array())] fn test_cast_alp_conformance(#[case] array: vortex_array::ArrayRef) -> VortexResult<()> { - let alp = alp_encode(&array.to_primitive(), None).vortex_expect("cannot fail"); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let array_primitive = array.execute::(&mut ctx)?; + let alp = + alp_encode(array_primitive.as_view(), None, &mut ctx).vortex_expect("cannot fail"); test_cast_conformance(&alp.into_array()); Ok(()) diff --git a/encodings/alp/src/alp/compute/compare.rs b/encodings/alp/src/alp/compute/compare.rs index 245715c2a6e..62fc092c284 100644 --- a/encodings/alp/src/alp/compute/compare.rs +++ b/encodings/alp/src/alp/compute/compare.rs @@ -152,7 +152,8 @@ where mod tests { use rstest::rstest; use vortex_array::ArrayRef; - use vortex_array::ToCanonical; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; use vortex_array::arrays::ConstantArray; use vortex_array::arrays::PrimitiveArray; @@ -182,13 +183,16 @@ mod tests { #[test] fn basic_comparison_test() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = PrimitiveArray::from_iter([1.234f32; 1025]); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode(array.as_view(), None, &mut ctx).unwrap(); assert!(encoded.patches().is_none()); - assert_eq!( - encoded.encoded().to_primitive().as_slice::(), - vec![1234; 1025] - ); + let encoded_prim = encoded + .encoded() + .clone() + .execute::(&mut ctx) + .unwrap(); + assert_eq!(encoded_prim.as_slice::(), vec![1234; 1025]); let r = alp_scalar_compare(encoded.as_view(), 1.3_f32, CompareOperator::Eq) .unwrap() @@ -205,22 +209,23 @@ mod tests { #[test] fn comparison_with_unencodable_value() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = PrimitiveArray::from_iter([1.234f32; 1025]); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode(array.as_view(), None, &mut ctx).unwrap(); assert!(encoded.patches().is_none()); - assert_eq!( - encoded.encoded().to_primitive().as_slice::(), - vec![1234; 1025] - ); + let encoded_prim = encoded + .encoded() + .clone() + .execute::(&mut ctx) + .unwrap(); + assert_eq!(encoded_prim.as_slice::(), vec![1234; 1025]); - #[allow(clippy::excessive_precision)] let r_eq = alp_scalar_compare(encoded.as_view(), 1.234444_f32, CompareOperator::Eq) .unwrap() .unwrap(); let expected = BoolArray::from_iter([false; 1025]); assert_arrays_eq!(r_eq, expected); - #[allow(clippy::excessive_precision)] let r_neq = alp_scalar_compare(encoded.as_view(), 1.234444f32, CompareOperator::NotEq) .unwrap() .unwrap(); @@ -230,13 +235,16 @@ mod tests { #[test] fn comparison_range() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = PrimitiveArray::from_iter([0.0605_f32; 10]); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode(array.as_view(), None, &mut ctx).unwrap(); assert!(encoded.patches().is_none()); - assert_eq!( - encoded.encoded().to_primitive().as_slice::(), - vec![605; 10] - ); + let encoded_prim = encoded + .encoded() + .clone() + .execute::(&mut ctx) + .unwrap(); + assert_eq!(encoded_prim.as_slice::(), vec![605; 10]); // !(0.0605_f32 >= 0.06051_f32); let r_gte = alp_scalar_compare(encoded.as_view(), 0.06051_f32, CompareOperator::Gte) @@ -269,13 +277,16 @@ mod tests { #[test] fn comparison_zeroes() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = PrimitiveArray::from_iter([0.0_f32; 10]); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode(array.as_view(), None, &mut ctx).unwrap(); assert!(encoded.patches().is_none()); - assert_eq!( - encoded.encoded().to_primitive().as_slice::(), - vec![0; 10] - ); + let encoded_prim = encoded + .encoded() + .clone() + .execute::(&mut ctx) + .unwrap(); + assert_eq!(encoded_prim.as_slice::(), vec![0; 10]); let r_gte = test_alp_compare(encoded.as_view(), -0.00000001_f32, CompareOperator::Gte).unwrap(); @@ -312,7 +323,12 @@ mod tests { fn compare_with_patches() { let array = PrimitiveArray::from_iter([1.234f32, 1.5, 19.0, std::f32::consts::E, 1_000_000.9]); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode( + array.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); assert!(encoded.patches().is_some()); // Not supported! @@ -326,7 +342,12 @@ mod tests { #[test] fn compare_to_null() { let array = PrimitiveArray::from_iter([1.234f32; 10]); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode( + array.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); let other = ConstantArray::new( Scalar::null(DType::Primitive(PType::F32, Nullability::Nullable)), @@ -349,7 +370,12 @@ mod tests { #[case(f32::NEG_INFINITY, true)] fn compare_to_non_finite_gt(#[case] value: f32, #[case] result: bool) { let array = PrimitiveArray::from_iter([1.234f32; 10]); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode( + array.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); let r = test_alp_compare(encoded.as_view(), value, CompareOperator::Gt).unwrap(); let expected = BoolArray::from_iter([result; 10]); @@ -363,7 +389,12 @@ mod tests { #[case(f32::NEG_INFINITY, false)] fn compare_to_non_finite_lt(#[case] value: f32, #[case] result: bool) { let array = PrimitiveArray::from_iter([1.234f32; 10]); - let encoded = alp_encode(&array, None).unwrap(); + let encoded = alp_encode( + array.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); let r = test_alp_compare(encoded.as_view(), value, CompareOperator::Lt).unwrap(); let expected = BoolArray::from_iter([result; 10]); diff --git a/encodings/alp/src/alp/compute/filter.rs b/encodings/alp/src/alp/compute/filter.rs index 10acc57ab8f..97b5b4af49e 100644 --- a/encodings/alp/src/alp/compute/filter.rs +++ b/encodings/alp/src/alp/compute/filter.rs @@ -44,7 +44,8 @@ mod test { use rstest::rstest; use vortex_array::ArrayRef; use vortex_array::IntoArray; - use vortex_array::ToCanonical; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::compute::conformance::filter::test_filter_conformance; use vortex_buffer::buffer; @@ -61,7 +62,9 @@ mod test { 11.0, 12.0, 13.0, 14.0, 15.0, 16.0, 17.0, 18.0, 19.0, 20.0 ].into_array())] fn test_filter_alp_conformance(#[case] array: ArrayRef) { - let alp = alp_encode(&array.to_primitive(), None).unwrap(); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let array_primitive = array.execute::(&mut ctx).unwrap(); + let alp = alp_encode(array_primitive.as_view(), None, &mut ctx).unwrap(); test_filter_conformance(&alp.into_array()); } } diff --git a/encodings/alp/src/alp/compute/mask.rs b/encodings/alp/src/alp/compute/mask.rs index 82eab0ebfa5..31f131f6e0c 100644 --- a/encodings/alp/src/alp/compute/mask.rs +++ b/encodings/alp/src/alp/compute/mask.rs @@ -57,7 +57,6 @@ mod test { use rstest::rstest; use vortex_array::IntoArray; use vortex_array::LEGACY_SESSION; - use vortex_array::ToCanonical; use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; use vortex_array::arrays::PrimitiveArray; @@ -79,31 +78,34 @@ mod test { 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2.0 ].into_array())] fn test_mask_alp_conformance(#[case] array: vortex_array::ArrayRef) { - let alp = alp_encode(&array.to_primitive(), None).unwrap(); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let array_primitive = array.execute::(&mut ctx).unwrap(); + let alp = alp_encode(array_primitive.as_view(), None, &mut ctx).unwrap(); test_mask_conformance(&alp.into_array()); } #[test] fn test_mask_alp_with_patches() { use std::f64::consts::PI; + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // PI doesn't encode cleanly with ALP, so it creates patches. let values: Vec = (0..100) .map(|i| if i % 4 == 3 { PI } else { 1.0 }) .collect(); let array = PrimitiveArray::from_iter(values); - let alp = alp_encode(&array, None).unwrap(); + let alp = alp_encode(array.as_view(), None, &mut ctx).unwrap(); assert!(alp.patches().is_some(), "expected patches"); test_mask_conformance(&alp.into_array()); } #[test] fn test_mask_alp_with_patches_casts_surviving_patch_values_to_nullable() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values = PrimitiveArray::from_iter([1.234f32, f32::NAN, 2.345, f32::INFINITY, 3.456]); - let alp = alp_encode(&values, None).unwrap(); + let alp = alp_encode(values.as_view(), None, &mut ctx).unwrap(); assert!(alp.patches().is_some(), "expected patches"); let keep_mask = BoolArray::from_iter([false, true, true, true, true]).into_array(); - let mut ctx = LEGACY_SESSION.create_execution_ctx(); let masked = ::mask(alp.as_view(), &keep_mask, &mut ctx) .unwrap() .unwrap(); diff --git a/encodings/alp/src/alp/compute/mod.rs b/encodings/alp/src/alp/compute/mod.rs index 6bbc1483b2b..e5e5b657740 100644 --- a/encodings/alp/src/alp/compute/mod.rs +++ b/encodings/alp/src/alp/compute/mod.rs @@ -14,6 +14,8 @@ mod take; mod tests { use rstest::rstest; use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::compute::conformance::binary_numeric::test_binary_numeric_array; use vortex_array::compute::conformance::consistency::test_array_consistency; @@ -23,27 +25,27 @@ mod tests { #[rstest] // Basic float arrays - #[case::f32_array(alp_encode(&PrimitiveArray::from_iter([1.23f32, 4.56, 7.89, 10.11, 12.13]), None).unwrap())] - #[case::f64_array(alp_encode(&PrimitiveArray::from_iter([100.1f64, 200.2, 300.3, 400.4, 500.5]), None).unwrap())] + #[case::f32_array(alp_encode(PrimitiveArray::from_iter([1.23f32, 4.56, 7.89, 10.11, 12.13]).as_view(), None, &mut LEGACY_SESSION.create_execution_ctx()).unwrap())] + #[case::f64_array(alp_encode(PrimitiveArray::from_iter([100.1f64, 200.2, 300.3, 400.4, 500.5]).as_view(), None, &mut LEGACY_SESSION.create_execution_ctx()).unwrap())] // Nullable arrays - #[case::nullable_f32(alp_encode(&PrimitiveArray::from_option_iter([Some(1.1f32), None, Some(2.2), Some(3.3), None]), None).unwrap())] - #[case::nullable_f64(alp_encode(&PrimitiveArray::from_option_iter([Some(1.1f64), None, Some(2.2), Some(3.3), None]), None).unwrap())] + #[case::nullable_f32(alp_encode(PrimitiveArray::from_option_iter([Some(1.1f32), None, Some(2.2), Some(3.3), None]).as_view(), None, &mut LEGACY_SESSION.create_execution_ctx()).unwrap())] + #[case::nullable_f64(alp_encode(PrimitiveArray::from_option_iter([Some(1.1f64), None, Some(2.2), Some(3.3), None]).as_view(), None, &mut LEGACY_SESSION.create_execution_ctx()).unwrap())] // Edge cases - #[case::single_element(alp_encode(&PrimitiveArray::from_iter([42.42f64]), None).unwrap())] + #[case::single_element(alp_encode(PrimitiveArray::from_iter([42.42f64]).as_view(), None, &mut LEGACY_SESSION.create_execution_ctx()).unwrap())] // Large arrays - #[case::large_f32(alp_encode(&PrimitiveArray::from_iter((0..1000).map(|i| i as f32 * 0.1)), None).unwrap())] + #[case::large_f32(alp_encode(PrimitiveArray::from_iter((0..1000).map(|i| i as f32 * 0.1)).as_view(), None, &mut LEGACY_SESSION.create_execution_ctx()).unwrap())] // Arrays with patterns - #[case::repeating_pattern(alp_encode(&PrimitiveArray::from_iter([1.1f32, 2.2, 3.3, 1.1, 2.2, 3.3, 1.1, 2.2, 3.3]), None).unwrap())] - #[case::close_values(alp_encode(&PrimitiveArray::from_iter([100.001f64, 100.002, 100.003, 100.004, 100.005]), None).unwrap())] + #[case::repeating_pattern(alp_encode(PrimitiveArray::from_iter([1.1f32, 2.2, 3.3, 1.1, 2.2, 3.3, 1.1, 2.2, 3.3]).as_view(), None, &mut LEGACY_SESSION.create_execution_ctx()).unwrap())] + #[case::close_values(alp_encode(PrimitiveArray::from_iter([100.001f64, 100.002, 100.003, 100.004, 100.005]).as_view(), None, &mut LEGACY_SESSION.create_execution_ctx()).unwrap())] fn test_alp_consistency(#[case] array: ALPArray) { test_array_consistency(&array.into_array()); } #[rstest] - #[case::f32_basic(alp_encode(&PrimitiveArray::from_iter([1.23f32, 4.56, 7.89, 10.11, 12.13]), None).unwrap())] - #[case::f64_basic(alp_encode(&PrimitiveArray::from_iter([100.1f64, 200.2, 300.3, 400.4, 500.5]), None).unwrap())] - #[case::f32_large(alp_encode(&PrimitiveArray::from_iter((0..100).map(|i| i as f32 * 1.5)), None).unwrap())] - #[case::f64_large(alp_encode(&PrimitiveArray::from_iter((0..100).map(|i| i as f64 * 2.5)), None).unwrap())] + #[case::f32_basic(alp_encode(PrimitiveArray::from_iter([1.23f32, 4.56, 7.89, 10.11, 12.13]).as_view(), None, &mut LEGACY_SESSION.create_execution_ctx()).unwrap())] + #[case::f64_basic(alp_encode(PrimitiveArray::from_iter([100.1f64, 200.2, 300.3, 400.4, 500.5]).as_view(), None, &mut LEGACY_SESSION.create_execution_ctx()).unwrap())] + #[case::f32_large(alp_encode(PrimitiveArray::from_iter((0..100).map(|i| i as f32 * 1.5)).as_view(), None, &mut LEGACY_SESSION.create_execution_ctx()).unwrap())] + #[case::f64_large(alp_encode(PrimitiveArray::from_iter((0..100).map(|i| i as f64 * 2.5)).as_view(), None, &mut LEGACY_SESSION.create_execution_ctx()).unwrap())] fn test_alp_binary_numeric(#[case] array: ALPArray) { test_binary_numeric_array(array.into_array()); } diff --git a/encodings/alp/src/alp/compute/take.rs b/encodings/alp/src/alp/compute/take.rs index cf0b5301684..3b2b35ceaa9 100644 --- a/encodings/alp/src/alp/compute/take.rs +++ b/encodings/alp/src/alp/compute/take.rs @@ -42,7 +42,8 @@ impl TakeExecute for ALP { mod test { use rstest::rstest; use vortex_array::IntoArray; - use vortex_array::ToCanonical; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::compute::conformance::take::test_take_conformance; use vortex_buffer::buffer; @@ -55,7 +56,9 @@ mod test { #[case(PrimitiveArray::from_option_iter([Some(1.1f32), None, Some(2.2), Some(3.3), None]).into_array())] #[case(buffer![42.42f64].into_array())] fn test_take_alp_conformance(#[case] array: vortex_array::ArrayRef) { - let alp = alp_encode(&array.to_primitive(), None).unwrap(); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let array_primitive = array.execute::(&mut ctx).unwrap(); + let alp = alp_encode(array_primitive.as_view(), None, &mut ctx).unwrap(); test_take_conformance(&alp.into_array()); } } diff --git a/encodings/alp/src/alp/decompress.rs b/encodings/alp/src/alp/decompress.rs index 6bc5b469d42..dcaacfe1541 100644 --- a/encodings/alp/src/alp/decompress.rs +++ b/encodings/alp/src/alp/decompress.rs @@ -31,20 +31,20 @@ pub fn decompress_into_array( ) -> VortexResult { let dtype = array.dtype().clone(); let (encoded, exponents, patches) = ALPArrayOwnedExt::into_parts(array); - if let Some(ref patches) = patches - && let Some(chunk_offsets) = patches.chunk_offsets() + if let Some(p) = &patches + && let Some(chunk_offsets) = p.chunk_offsets() { let prim_encoded = encoded.execute::(ctx)?; let patches_chunk_offsets = chunk_offsets.clone().execute::(ctx)?; - let patches_indices = patches.indices().clone().execute::(ctx)?; - let patches_values = patches.values().clone().execute::(ctx)?; + let patches_indices = p.indices().clone().execute::(ctx)?; + let patches_values = p.values().clone().execute::(ctx)?; Ok(decompress_chunked_core( prim_encoded, exponents, &patches_indices, &patches_values, &patches_chunk_offsets, - patches, + p, dtype, )) } else { @@ -64,21 +64,21 @@ pub fn decompress_into_array( pub fn execute_decompress(array: ALPArray, ctx: &mut ExecutionCtx) -> VortexResult { let dtype = array.dtype().clone(); let (encoded, exponents, patches) = ALPArrayOwnedExt::into_parts(array); - if let Some(ref patches) = patches - && let Some(chunk_offsets) = patches.chunk_offsets() + if let Some(p) = &patches + && let Some(chunk_offsets) = p.chunk_offsets() { // TODO(joe): have into parts. let encoded = encoded.execute::(ctx)?; let patches_chunk_offsets = chunk_offsets.clone().execute::(ctx)?; - let patches_indices = patches.indices().clone().execute::(ctx)?; - let patches_values = patches.values().clone().execute::(ctx)?; + let patches_indices = p.indices().clone().execute::(ctx)?; + let patches_values = p.values().clone().execute::(ctx)?; Ok(decompress_chunked_core( encoded, exponents, &patches_indices, &patches_values, &patches_chunk_offsets, - patches, + p, dtype, )) } else { diff --git a/encodings/alp/src/alp/mod.rs b/encodings/alp/src/alp/mod.rs index 7d5049b59e3..88375fe2ae6 100644 --- a/encodings/alp/src/alp/mod.rs +++ b/encodings/alp/src/alp/mod.rs @@ -18,8 +18,11 @@ mod compress; pub(crate) mod compute; mod decompress; mod ops; +mod plugin; mod rules; +pub(crate) use plugin::ALPPatchedPlugin; + #[cfg(test)] mod tests { use prost::Message; diff --git a/encodings/alp/src/alp/ops.rs b/encodings/alp/src/alp/ops.rs index dcd8cecf1d5..a8850744056 100644 --- a/encodings/alp/src/alp/ops.rs +++ b/encodings/alp/src/alp/ops.rs @@ -18,7 +18,7 @@ impl OperationsVTable for ALP { fn scalar_at( array: ArrayView<'_, ALP>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { if let Some(patches) = array.patches() && let Some(patch) = patches.get_patched(index)? @@ -26,7 +26,7 @@ impl OperationsVTable for ALP { return patch.cast(array.dtype()); } - let encoded_val = array.encoded().scalar_at(index)?; + let encoded_val = array.encoded().execute_scalar(index, ctx)?; Ok(match_each_alp_float_ptype!(array.dtype().as_ptype(), |T| { let encoded_val: ::ALPInt = diff --git a/encodings/alp/src/alp/plugin.rs b/encodings/alp/src/alp/plugin.rs new file mode 100644 index 00000000000..25266bef821 --- /dev/null +++ b/encodings/alp/src/alp/plugin.rs @@ -0,0 +1,235 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! A custom [`ArrayPlugin`] that lets you load in and deserialize an `ALP` array with interior +//! patches as a `PatchedArray` that wraps a patchless `ALP` array. +//! +//! This enables zero-cost backward compatibility with previously written datasets. + +use vortex_array::Array; +use vortex_array::ArrayId; +use vortex_array::ArrayPlugin; +use vortex_array::ArrayRef; +use vortex_array::ArrayVTable; +use vortex_array::IntoArray; +use vortex_array::VortexSessionExecute; +use vortex_array::arrays::Patched; +use vortex_array::buffer::BufferHandle; +use vortex_array::dtype::DType; +use vortex_array::serde::ArrayChildren; +use vortex_error::VortexResult; +use vortex_error::vortex_err; +use vortex_session::VortexSession; + +use crate::ALP; +use crate::ALPArrayExt; +use crate::ALPArrayOwnedExt; + +/// Custom deserialization plugin that converts an ALP array with interior +/// patches into a PatchedArray holding an ALP array. +#[derive(Debug, Clone)] +pub(crate) struct ALPPatchedPlugin; + +impl ArrayPlugin for ALPPatchedPlugin { + fn id(&self) -> ArrayId { + // We reuse the existing `ALP` ID so that we can take over its + // deserialization pathway. + // TODO(joe): dedup method name + ArrayVTable::id(&ALP) + } + + fn serialize( + &self, + array: &ArrayRef, + session: &VortexSession, + ) -> VortexResult>> { + // Delegate to ALP's metadata serde + ALP.serialize(array, session) + } + + fn deserialize( + &self, + dtype: &DType, + len: usize, + metadata: &[u8], + buffers: &[BufferHandle], + children: &dyn ArrayChildren, + session: &VortexSession, + ) -> VortexResult { + let alp_array = Array::::try_from_parts(ArrayVTable::deserialize( + &ALP, dtype, len, metadata, buffers, children, session, + )?) + .map_err(|_| vortex_err!("ALP plugin should only deserialize vortex.alp"))?; + + // Check if there are interior patches to externalize. + let Some(patches) = alp_array.patches() else { + return Ok(alp_array.into_array()); + }; + + // Extract components and create a new ALP array without patches. + let (encoded, exponents, _) = alp_array.into_parts(); + + let alp_without_patches = ALP::try_new(encoded, exponents, None)?.into_array(); + + let patched = Patched::from_array_and_patches( + alp_without_patches, + &patches, + &mut session.create_execution_ctx(), + )?; + + Ok(patched.into_array()) + } + + fn is_supported_encoding(&self, id: &ArrayId) -> bool { + // TODO(joe): dedup method name + id == ArrayVTable::id(&Patched) || id == ArrayVTable::id(&ALP) + } +} + +#[cfg(test)] +mod tests { + use std::f64::consts::PI; + use std::sync::LazyLock; + + use vortex_array::ArrayPlugin; + use vortex_array::IntoArray; + use vortex_array::VortexSessionExecute; + use vortex_array::arrays::PatchedArray; + use vortex_array::arrays::PrimitiveArray; + use vortex_array::arrays::patched::PatchedArraySlotsExt; + use vortex_array::buffer::BufferHandle; + use vortex_array::session::ArraySession; + use vortex_array::session::ArraySessionExt; + use vortex_error::VortexResult; + use vortex_error::vortex_err; + use vortex_session::VortexSession; + + use super::ALPPatchedPlugin; + use crate::ALP; + use crate::ALPArray; + use crate::ALPArrayExt; + use crate::alp_encode; + + static SESSION: LazyLock = LazyLock::new(|| { + let session = VortexSession::empty().with::(); + session.arrays().register(ALPPatchedPlugin); + session + }); + + #[test] + fn test_decode_alp_patches() -> VortexResult<()> { + // Create values where some don't encode cleanly with ALP, causing patches. + // PI doesn't encode cleanly. + let values: Vec = (0..100) + .map(|i| if i % 4 == 3 { PI } else { i as f64 }) + .collect(); + + let parray = PrimitiveArray::from_iter(values); + let alp_encoded = alp_encode(parray.as_view(), None, &mut SESSION.create_execution_ctx())?; + + assert!( + alp_encoded.patches().is_some(), + "Expected ALP array to have patches" + ); + + let array = alp_encoded.as_array(); + + let metadata = SESSION.array_serialize(array)?.unwrap(); + let children = array.children(); + let buffers = array + .buffers() + .into_iter() + .map(BufferHandle::new_host) + .collect::>(); + + let deserialized = ALPPatchedPlugin.deserialize( + array.dtype(), + array.len(), + &metadata, + &buffers, + &children, + &SESSION, + )?; + + let patched: PatchedArray = deserialized + .try_downcast() + .map_err(|a| vortex_err!("Expected Patched, got {}", a.encoding_id()))?; + + let inner_alp: ALPArray = patched + .inner() + .clone() + .try_downcast() + .map_err(|a| vortex_err!("Expected inner ALP, got {}", a.encoding_id()))?; + + assert!( + inner_alp.patches().is_none(), + "Inner ALP should NOT have patches" + ); + + Ok(()) + } + + #[test] + fn alp_without_patches_stays_alp() -> VortexResult<()> { + // Values that encode cleanly without patches. + let values: Vec = (0..100).map(|i| i as f64).collect(); + let parray = PrimitiveArray::from_iter(values); + let alp_encoded = alp_encode(parray.as_view(), None, &mut SESSION.create_execution_ctx())?; + + assert!( + alp_encoded.patches().is_none(), + "Expected ALP array without patches" + ); + + let array = alp_encoded.as_array(); + + let metadata = SESSION.array_serialize(array)?.unwrap(); + let children = array.children(); + let buffers = array + .buffers() + .into_iter() + .map(BufferHandle::new_host) + .collect::>(); + + let deserialized = ALPPatchedPlugin.deserialize( + array.dtype(), + array.len(), + &metadata, + &buffers, + &children, + &SESSION, + )?; + + let result = deserialized + .try_downcast::() + .map_err(|a| vortex_err!("Expected deserialized ALP, got {}", a.encoding_id()))?; + + assert!(result.patches().is_none(), "Result should not have patches"); + + Ok(()) + } + + #[test] + #[should_panic(expected = "index out of bounds")] + fn primitive_array_returns_error() { + let array = PrimitiveArray::from_iter([1.0f64, 2.0, 3.0]).into_array(); + + let metadata = SESSION.array_serialize(&array).unwrap().unwrap(); + let children = array.children(); + let buffers = array + .buffers() + .into_iter() + .map(BufferHandle::new_host) + .collect::>(); + + // This panics because PrimitiveArray has no children and ALP requires encoded child. + let _result = ALPPatchedPlugin.deserialize( + array.dtype(), + array.len(), + &metadata, + &buffers, + &children, + &SESSION, + ); + } +} diff --git a/encodings/alp/src/alp_rd/array.rs b/encodings/alp/src/alp_rd/array.rs index 431b69c125d..0c25717db6f 100644 --- a/encodings/alp/src/alp_rd/array.rs +++ b/encodings/alp/src/alp_rd/array.rs @@ -16,11 +16,14 @@ use vortex_array::ArrayId; use vortex_array::ArrayParts; use vortex_array::ArrayRef; use vortex_array::ArrayView; +use vortex_array::Canonical; use vortex_array::ExecutionCtx; use vortex_array::ExecutionResult; use vortex_array::IntoArray; +use vortex_array::LEGACY_SESSION; use vortex_array::Precision; use vortex_array::TypedArrayRef; +use vortex_array::VortexSessionExecute; use vortex_array::arrays::Primitive; use vortex_array::arrays::PrimitiveArray; use vortex_array::buffer::BufferHandle; @@ -44,6 +47,7 @@ use vortex_error::vortex_ensure; use vortex_error::vortex_err; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::alp_rd::kernel::PARENT_KERNELS; use crate::alp_rd::rules::RULES; @@ -92,7 +96,8 @@ impl VTable for ALPRD { type ValidityVTable = ValidityVTableFromChild; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.alprd"); + *ID } fn validate( @@ -209,8 +214,14 @@ impl VTable for ALPRD { ) }) .transpose()?; - let left_parts_patches = ALPRDData::canonicalize_patches(&left_parts, left_parts_patches)?; - let slots = ALPRDData::make_slots(&left_parts, &right_parts, &left_parts_patches); + // NOTE: `VTable::deserialize` has a fixed trait signature without `ExecutionCtx`, so we + // cannot plumb a ctx in here. We construct a legacy ctx locally at this trait boundary. + let left_parts_patches = ALPRDData::canonicalize_patches( + &left_parts, + left_parts_patches, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; + let slots = ALPRDData::make_slots(&left_parts, &right_parts, left_parts_patches.as_ref()); let data = ALPRDData::new( left_parts_dictionary, u8::try_from(metadata.right_bit_width).map_err(|_| { @@ -259,7 +270,10 @@ impl VTable for ALPRD { // Decode the left_parts using our builtin dictionary. let left_parts_dict = left_parts_dictionary; - let validity = left_parts.validity_mask()?; + let validity = left_parts + .as_ref() + .validity()? + .execute_mask(left_parts.as_ref().len(), ctx)?; let decoded_array = if ptype == PType::F32 { PrimitiveArray::new( @@ -357,8 +371,6 @@ pub struct ALPRDDataParts { pub struct ALPRD; impl ALPRD { - pub const ID: ArrayId = ArrayId::new_ref("vortex.alprd"); - pub fn try_new( dtype: DType, left_parts: ArrayRef, @@ -366,10 +378,12 @@ impl ALPRD { right_parts: ArrayRef, right_bit_width: u8, left_parts_patches: Option, + ctx: &mut ExecutionCtx, ) -> VortexResult { let len = left_parts.len(); - let left_parts_patches = ALPRDData::canonicalize_patches(&left_parts, left_parts_patches)?; - let slots = ALPRDData::make_slots(&left_parts, &right_parts, &left_parts_patches); + let left_parts_patches = + ALPRDData::canonicalize_patches(&left_parts, left_parts_patches, ctx)?; + let slots = ALPRDData::make_slots(&left_parts, &right_parts, left_parts_patches.as_ref()); let data = ALPRDData::new(left_parts_dictionary, right_bit_width, left_parts_patches); Array::try_from_parts(ArrayParts::new(ALPRD, dtype, len, data).with_slots(slots)) } @@ -385,7 +399,7 @@ impl ALPRD { left_parts_patches: Option, ) -> ALPRDArray { let len = left_parts.len(); - let slots = ALPRDData::make_slots(&left_parts, &right_parts, &left_parts_patches); + let slots = ALPRDData::make_slots(&left_parts, &right_parts, left_parts_patches.as_ref()); let data = unsafe { ALPRDData::new_unchecked(left_parts_dictionary, right_bit_width, left_parts_patches) }; @@ -399,10 +413,11 @@ impl ALPRDData { fn canonicalize_patches( left_parts: &ArrayRef, left_parts_patches: Option, + ctx: &mut ExecutionCtx, ) -> VortexResult> { left_parts_patches .map(|patches| { - if !patches.values().all_valid()? { + if !patches.values().all_valid(ctx)? { vortex_bail!("patches must be all valid: {}", patches.values()); } // TODO(ngates): assert the DType, don't cast it. @@ -410,7 +425,8 @@ impl ALPRDData { let mut patches = patches.cast_values(&left_parts.dtype().as_nonnullable())?; // Force execution of the lazy cast so patch values are materialized // before serialization. - *patches.values_mut() = patches.values().to_canonical()?.into_array(); + let canonical = patches.values().clone().execute::(ctx)?; + *patches.values_mut() = canonical.into_array(); Ok(patches) }) .transpose() @@ -448,7 +464,7 @@ impl ALPRDData { fn make_slots( left_parts: &ArrayRef, right_parts: &ArrayRef, - patches: &Option, + patches: Option<&Patches>, ) -> Vec> { let (pi, pv, pco) = match patches { Some(p) => ( @@ -583,7 +599,9 @@ fn validate_parts( left_parts.dtype(), ); vortex_ensure!( - patches.values().all_valid()?, + patches + .values() + .all_valid(&mut LEGACY_SESSION.create_execution_ctx())?, "patches must be all valid: {}", patches.values() ); @@ -650,7 +668,8 @@ impl ValidityChild for ALPRD { mod test { use prost::Message; use rstest::rstest; - use vortex_array::ToCanonical; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; use vortex_array::dtype::PType; @@ -668,6 +687,7 @@ mod test { #[case] reals: Vec, #[case] seed: T, ) { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); assert_eq!(reals.len(), 1024, "test expects 1024-length fixture"); // Null out some of the values. let mut reals: Vec> = reals.into_iter().map(Some).collect(); @@ -681,9 +701,13 @@ mod test { // Pick a seed that we know will trigger lots of patches. let encoder: alp_rd::RDEncoder = alp_rd::RDEncoder::new(&[seed.powi(-2)]); - let rd_array = encoder.encode(&real_array); + let rd_array = encoder.encode(real_array.as_view(), &mut ctx); - let decoded = rd_array.as_array().to_primitive(); + let decoded = rd_array + .as_array() + .clone() + .execute::(&mut ctx) + .unwrap(); assert_arrays_eq!(decoded, PrimitiveArray::from_option_iter(reals)); } diff --git a/encodings/alp/src/alp_rd/compute/cast.rs b/encodings/alp/src/alp_rd/compute/cast.rs index 01eab5d126f..c110475b880 100644 --- a/encodings/alp/src/alp_rd/compute/cast.rs +++ b/encodings/alp/src/alp_rd/compute/cast.rs @@ -14,35 +14,35 @@ use crate::alp_rd::ALPRD; impl CastReduce for ALPRD { fn cast(array: ArrayView<'_, Self>, dtype: &DType) -> VortexResult> { - // ALPRDArray stores floating-point values, so only cast between float types - // or if just changing nullability - // Check if this is just a nullability change - if array.dtype().eq_ignore_nullability(dtype) { - // For nullability-only changes, we need to cast the left_parts array - // since it carries the validity information - let new_left_parts = array.left_parts().cast( - array - .left_parts() - .dtype() - .with_nullability(dtype.nullability()), - )?; + if !array.dtype().eq_ignore_nullability(dtype) { + return Ok(None); + } + + // For nullability-only changes, we need to cast the left_parts array + // since it carries the validity information + let new_left_parts = array.left_parts().cast( + array + .left_parts() + .dtype() + .with_nullability(dtype.nullability()), + )?; - return Ok(Some( - ALPRD::try_new( + // NOTE: `CastReduce::cast` has a fixed trait signature without `ExecutionCtx`, so we + // construct a legacy ctx locally at this trait boundary. + Ok(Some( + unsafe { + ALPRD::new_unchecked( dtype.clone(), new_left_parts, array.left_parts_dictionary().clone(), array.right_parts().clone(), array.right_bit_width(), array.left_parts_patches(), - )? - .into_array(), - )); - } - - // For other casts (e.g., f32 to f64), decode to canonical and let PrimitiveArray handle it - Ok(None) + ) + } + .into_array(), + )) } } @@ -50,7 +50,8 @@ impl CastReduce for ALPRD { mod tests { use rstest::rstest; use vortex_array::IntoArray; - use vortex_array::ToCanonical; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::builtins::ArrayBuiltins; use vortex_array::compute::conformance::cast::test_cast_conformance; @@ -62,10 +63,11 @@ mod tests { #[test] fn test_cast_alprd_f32_to_f64() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values = vec![1.0f32, 1.1, 1.2, 1.3, 1.4]; let arr = PrimitiveArray::from_iter(values.clone()); let encoder = RDEncoder::new(&values); - let alprd = encoder.encode(&arr); + let alprd = encoder.encode(arr.as_view(), &mut ctx); let casted = alprd .into_array() @@ -76,7 +78,7 @@ mod tests { &DType::Primitive(PType::F64, Nullability::NonNullable) ); - let decoded = casted.to_primitive(); + let decoded = casted.execute::(&mut ctx).unwrap(); let f64_values = decoded.as_slice::(); assert_eq!(f64_values.len(), 5); assert!((f64_values[0] - 1.0).abs() < f64::EPSILON); @@ -85,18 +87,24 @@ mod tests { #[test] fn test_cast_alprd_nullable() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let arr = PrimitiveArray::from_option_iter([Some(10.0f64), None, Some(10.1), Some(10.2), None]); let values = vec![10.0f64, 10.1, 10.2]; let encoder = RDEncoder::new(&values); - let alprd = encoder.encode(&arr); + let alprd = encoder.encode(arr.as_view(), &mut ctx); - // Cast to NonNullable should fail since we have nulls + // Cast to NonNullable should fail since we have nulls. The failure surfaces during + // execution since the reduce path defers when the validity stat is not cached. let result = alprd .clone() .into_array() - .cast(DType::Primitive(PType::F64, Nullability::NonNullable)); - assert!(result.is_err()); + .cast(DType::Primitive(PType::F64, Nullability::NonNullable)) + .and_then(|a| { + a.execute::(&mut ctx) + .map(|p| p.into_array()) + }); + assert!(result.is_err(), "Expected error, got: {result:?}"); // Cast to same type with Nullable should succeed let casted = alprd @@ -114,31 +122,31 @@ mod tests { let values = vec![1.23f32, 4.56, 7.89, 10.11, 12.13]; let arr = PrimitiveArray::from_iter(values.clone()); let encoder = RDEncoder::new(&values); - encoder.encode(&arr) + encoder.encode(arr.as_view(), &mut LEGACY_SESSION.create_execution_ctx()) })] #[case::f64({ let values = vec![100.1f64, 200.2, 300.3, 400.4, 500.5]; let arr = PrimitiveArray::from_iter(values.clone()); let encoder = RDEncoder::new(&values); - encoder.encode(&arr) + encoder.encode(arr.as_view(), &mut LEGACY_SESSION.create_execution_ctx()) })] #[case::single({ let values = vec![42.42f64]; let arr = PrimitiveArray::from_iter(values.clone()); let encoder = RDEncoder::new(&values); - encoder.encode(&arr) + encoder.encode(arr.as_view(), &mut LEGACY_SESSION.create_execution_ctx()) })] #[case::negative({ let values = vec![0.0f32, -1.5, 2.5, -3.5, 4.5]; let arr = PrimitiveArray::from_iter(values.clone()); let encoder = RDEncoder::new(&values); - encoder.encode(&arr) + encoder.encode(arr.as_view(), &mut LEGACY_SESSION.create_execution_ctx()) })] #[case::nullable({ let arr = PrimitiveArray::from_option_iter([Some(1.1f32), None, Some(2.2), Some(3.3), None]); let values = vec![1.1f32, 2.2, 3.3]; let encoder = RDEncoder::new(&values); - encoder.encode(&arr) + encoder.encode(arr.as_view(), &mut LEGACY_SESSION.create_execution_ctx()) })] fn test_cast_alprd_conformance(#[case] alprd: crate::alp_rd::ALPRDArray) { test_cast_conformance(&alprd.into_array()); diff --git a/encodings/alp/src/alp_rd/compute/filter.rs b/encodings/alp/src/alp_rd/compute/filter.rs index 8815a049dc6..5deaa7c466b 100644 --- a/encodings/alp/src/alp_rd/compute/filter.rs +++ b/encodings/alp/src/alp_rd/compute/filter.rs @@ -32,6 +32,7 @@ impl FilterKernel for ALPRD { array.right_parts().filter(mask.clone())?, array.right_bit_width(), left_parts_exceptions, + ctx, )? .into_array(), )) @@ -42,6 +43,8 @@ impl FilterKernel for ALPRD { mod test { use rstest::rstest; use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; use vortex_array::compute::conformance::filter::test_filter_conformance; @@ -57,8 +60,9 @@ mod test { #[case(0.1f32, 0.2f32, 3e25f32)] #[case(0.1f64, 0.2f64, 3e100f64)] fn test_filter(#[case] a: T, #[case] b: T, #[case] outlier: T) { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = PrimitiveArray::new(buffer![a, b, outlier], Validity::NonNullable); - let encoded = RDEncoder::new(&[a, b]).encode(&array); + let encoded = RDEncoder::new(&[a, b]).encode(array.as_view(), &mut ctx); // Make sure that we're testing the exception pathway. assert!(encoded.left_parts_patches().is_some()); @@ -74,9 +78,13 @@ mod test { #[case(0.1f32, 0.2f32, 3e25f32)] #[case(0.1f64, 0.2f64, 3e100f64)] fn test_filter_simple(#[case] a: T, #[case] b: T, #[case] outlier: T) { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); test_filter_conformance( &RDEncoder::new(&[a, b]) - .encode(&PrimitiveArray::from_iter([a, b, outlier, b, outlier])) + .encode( + PrimitiveArray::from_iter([a, b, outlier, b, outlier]).as_view(), + &mut ctx, + ) .into_array(), ); } @@ -85,15 +93,14 @@ mod test { #[case(0.1f32, 3e25f32)] #[case(0.5f64, 1e100f64)] fn test_filter_with_nulls(#[case] a: T, #[case] outlier: T) { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); test_filter_conformance( &RDEncoder::new(&[a]) - .encode(&PrimitiveArray::from_option_iter([ - Some(a), - None, - Some(outlier), - Some(a), - None, - ])) + .encode( + PrimitiveArray::from_option_iter([Some(a), None, Some(outlier), Some(a), None]) + .as_view(), + &mut ctx, + ) .into_array(), ); } diff --git a/encodings/alp/src/alp_rd/compute/mask.rs b/encodings/alp/src/alp_rd/compute/mask.rs index 001c44d4825..47a2f2a6b67 100644 --- a/encodings/alp/src/alp_rd/compute/mask.rs +++ b/encodings/alp/src/alp_rd/compute/mask.rs @@ -4,6 +4,8 @@ use vortex_array::ArrayRef; use vortex_array::ArrayView; use vortex_array::IntoArray; +use vortex_array::LEGACY_SESSION; +use vortex_array::VortexSessionExecute; use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; use vortex_array::scalar_fn::EmptyOptions; use vortex_array::scalar_fn::fns::mask::Mask as MaskExpr; @@ -20,6 +22,8 @@ impl MaskReduce for ALPRD { EmptyOptions, [array.left_parts().clone(), mask.clone()], )?; + // NOTE: `MaskReduce::mask` has a fixed trait signature without `ExecutionCtx`, so we + // construct a legacy ctx locally at this trait boundary. Ok(Some( ALPRD::try_new( array.dtype().as_nullable(), @@ -28,6 +32,7 @@ impl MaskReduce for ALPRD { array.right_parts().clone(), array.right_bit_width(), array.left_parts_patches(), + &mut LEGACY_SESSION.create_execution_ctx(), )? .into_array(), )) @@ -38,6 +43,8 @@ impl MaskReduce for ALPRD { mod tests { use rstest::rstest; use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::compute::conformance::mask::test_mask_conformance; @@ -48,9 +55,13 @@ mod tests { #[case(0.1f32, 0.2f32, 3e25f32)] #[case(0.1f64, 0.2f64, 3e100f64)] fn test_mask_simple(#[case] a: T, #[case] b: T, #[case] outlier: T) { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); test_mask_conformance( &RDEncoder::new(&[a, b]) - .encode(&PrimitiveArray::from_iter([a, b, outlier, b, outlier])) + .encode( + PrimitiveArray::from_iter([a, b, outlier, b, outlier]).as_view(), + &mut ctx, + ) .into_array(), ); } @@ -59,15 +70,14 @@ mod tests { #[case(0.1f32, 3e25f32)] #[case(0.5f64, 1e100f64)] fn test_mask_with_nulls(#[case] a: T, #[case] outlier: T) { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); test_mask_conformance( &RDEncoder::new(&[a]) - .encode(&PrimitiveArray::from_option_iter([ - Some(a), - None, - Some(outlier), - Some(a), - None, - ])) + .encode( + PrimitiveArray::from_option_iter([Some(a), None, Some(outlier), Some(a), None]) + .as_view(), + &mut ctx, + ) .into_array(), ); } diff --git a/encodings/alp/src/alp_rd/compute/mod.rs b/encodings/alp/src/alp_rd/compute/mod.rs index 1971dc28b0e..7d03eddaf40 100644 --- a/encodings/alp/src/alp_rd/compute/mod.rs +++ b/encodings/alp/src/alp_rd/compute/mod.rs @@ -10,6 +10,8 @@ mod take; mod tests { use rstest::rstest; use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::compute::conformance::binary_numeric::test_binary_numeric_array; use vortex_array::compute::conformance::consistency::test_array_consistency; @@ -23,47 +25,47 @@ mod tests { let values = vec![1.0f32, 1.1, 1.2, 1.3, 1.4]; let arr = PrimitiveArray::from_iter(values.clone()); let encoder = RDEncoder::new(&values); - encoder.encode(&arr) + encoder.encode(arr.as_view(), &mut LEGACY_SESSION.create_execution_ctx()) })] #[case::f64_array({ let values = vec![100.0f64, 100.01, 100.02, 100.03, 100.04]; let arr = PrimitiveArray::from_iter(values.clone()); let encoder = RDEncoder::new(&values); - encoder.encode(&arr) + encoder.encode(arr.as_view(), &mut LEGACY_SESSION.create_execution_ctx()) })] // Nullable arrays #[case::nullable_f32({ let values = vec![1.0f32, 1.2, 1.3]; let arr = PrimitiveArray::from_option_iter([Some(1.0f32), None, Some(1.2), Some(1.3), None]); let encoder = RDEncoder::new(&values); - encoder.encode(&arr) + encoder.encode(arr.as_view(), &mut LEGACY_SESSION.create_execution_ctx()) })] #[case::nullable_f64({ let values = vec![10.0f64, 10.2, 10.3]; let arr = PrimitiveArray::from_option_iter([Some(10.0f64), None, Some(10.2), Some(10.3), None]); let encoder = RDEncoder::new(&values); - encoder.encode(&arr) + encoder.encode(arr.as_view(), &mut LEGACY_SESSION.create_execution_ctx()) })] // Edge cases #[case::single_element({ let values = vec![42.42f64]; let arr = PrimitiveArray::from_iter(values.clone()); let encoder = RDEncoder::new(&values); - encoder.encode(&arr) + encoder.encode(arr.as_view(), &mut LEGACY_SESSION.create_execution_ctx()) })] // Arrays with small deltas (good for RD encoding) #[case::small_deltas({ let values = vec![1000.0f32, 1000.001, 1000.002, 1000.003, 1000.004]; let arr = PrimitiveArray::from_iter(values.clone()); let encoder = RDEncoder::new(&values); - encoder.encode(&arr) + encoder.encode(arr.as_view(), &mut LEGACY_SESSION.create_execution_ctx()) })] // Large arrays #[case::large_f32({ let values: Vec = (0..1000).map(|i| 100.0 + i as f32 * 0.01).collect(); let arr = PrimitiveArray::from_iter(values.clone()); let encoder = RDEncoder::new(&values); - encoder.encode(&arr) + encoder.encode(arr.as_view(), &mut LEGACY_SESSION.create_execution_ctx()) })] fn test_alp_rd_consistency(#[case] array: ALPRDArray) { test_array_consistency(&array.into_array()); @@ -74,25 +76,25 @@ mod tests { let values = vec![1.0f32, 1.1, 1.2, 1.3, 1.4]; let arr = PrimitiveArray::from_iter(values.clone()); let encoder = RDEncoder::new(&values); - encoder.encode(&arr) + encoder.encode(arr.as_view(), &mut LEGACY_SESSION.create_execution_ctx()) })] #[case::f64_basic({ let values = vec![100.0f64, 100.01, 100.02, 100.03, 100.04]; let arr = PrimitiveArray::from_iter(values.clone()); let encoder = RDEncoder::new(&values); - encoder.encode(&arr) + encoder.encode(arr.as_view(), &mut LEGACY_SESSION.create_execution_ctx()) })] #[case::f32_large({ let values: Vec = (0..100).map(|i| 50.0 + i as f32 * 0.1).collect(); let arr = PrimitiveArray::from_iter(values.clone()); let encoder = RDEncoder::new(&values); - encoder.encode(&arr) + encoder.encode(arr.as_view(), &mut LEGACY_SESSION.create_execution_ctx()) })] #[case::f64_large({ let values: Vec = (0..100).map(|i| 1000.0 + i as f64 * 0.01).collect(); let arr = PrimitiveArray::from_iter(values.clone()); let encoder = RDEncoder::new(&values); - encoder.encode(&arr) + encoder.encode(arr.as_view(), &mut LEGACY_SESSION.create_execution_ctx()) })] fn test_alp_rd_binary_numeric(#[case] array: ALPRDArray) { test_binary_numeric_array(array.into_array()); diff --git a/encodings/alp/src/alp_rd/compute/take.rs b/encodings/alp/src/alp_rd/compute/take.rs index fe12255f063..0d980e0abfc 100644 --- a/encodings/alp/src/alp_rd/compute/take.rs +++ b/encodings/alp/src/alp_rd/compute/take.rs @@ -48,6 +48,7 @@ impl TakeExecute for ALPRD { right_parts, array.right_bit_width(), left_parts_exceptions, + ctx, )? .into_array(), )) @@ -58,7 +59,8 @@ impl TakeExecute for ALPRD { mod test { use rstest::rstest; use vortex_array::IntoArray; - use vortex_array::ToCanonical; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; use vortex_array::compute::conformance::take::test_take_conformance; @@ -74,8 +76,9 @@ mod test { use vortex_array::IntoArray as _; use vortex_buffer::buffer; + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = PrimitiveArray::from_iter([a, b, outlier]); - let encoded = RDEncoder::new(&[a, b]).encode(&array); + let encoded = RDEncoder::new(&[a, b]).encode(array.as_view(), &mut ctx); assert!(encoded.left_parts_patches().is_some()); assert!( @@ -89,7 +92,8 @@ mod test { let taken = encoded .take(buffer![0, 2].into_array()) .unwrap() - .to_primitive(); + .execute::(&mut ctx) + .unwrap(); assert_arrays_eq!(taken, PrimitiveArray::from_iter([a, outlier])); } @@ -98,8 +102,9 @@ mod test { #[case(0.1f32, 0.2f32, 3e25f32)] #[case(0.1f64, 0.2f64, 3e100f64)] fn take_with_nulls(#[case] a: T, #[case] b: T, #[case] outlier: T) { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = PrimitiveArray::from_iter([a, b, outlier]); - let encoded = RDEncoder::new(&[a, b]).encode(&array); + let encoded = RDEncoder::new(&[a, b]).encode(array.as_view(), &mut ctx); assert!(encoded.left_parts_patches().is_some()); assert!( @@ -113,7 +118,8 @@ mod test { let taken = encoded .take(PrimitiveArray::from_option_iter([Some(0), Some(2), None]).into_array()) .unwrap() - .to_primitive(); + .execute::(&mut ctx) + .unwrap(); assert_arrays_eq!( taken, @@ -125,9 +131,13 @@ mod test { #[case(0.1f32, 0.2f32, 3e25f32)] #[case(0.1f64, 0.2f64, 3e100f64)] fn test_take_conformance_alprd(#[case] a: T, #[case] b: T, #[case] outlier: T) { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); test_take_conformance( &RDEncoder::new(&[a, b]) - .encode(&PrimitiveArray::from_iter([a, b, outlier, b, outlier])) + .encode( + PrimitiveArray::from_iter([a, b, outlier, b, outlier]).as_view(), + &mut ctx, + ) .into_array(), ); } @@ -136,15 +146,14 @@ mod test { #[case(0.1f32, 3e25f32)] #[case(0.5f64, 1e100f64)] fn test_take_with_nulls_conformance(#[case] a: T, #[case] outlier: T) { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); test_take_conformance( &RDEncoder::new(&[a]) - .encode(&PrimitiveArray::from_option_iter([ - Some(a), - None, - Some(outlier), - Some(a), - None, - ])) + .encode( + PrimitiveArray::from_option_iter([Some(a), None, Some(outlier), Some(a), None]) + .as_view(), + &mut ctx, + ) .into_array(), ); } diff --git a/encodings/alp/src/alp_rd/mod.rs b/encodings/alp/src/alp_rd/mod.rs index 6202dd799e7..5f2e703cc0a 100644 --- a/encodings/alp/src/alp_rd/mod.rs +++ b/encodings/alp/src/alp_rd/mod.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::cast_possible_truncation)] +#![expect(clippy::cast_possible_truncation)] pub use array::*; use vortex_array::ExecutionCtx; @@ -25,6 +25,8 @@ use num_traits::Float; use num_traits::One; use num_traits::PrimInt; use rustc_hash::FxBuildHasher; +use vortex_array::ArrayView; +use vortex_array::arrays::Primitive; use vortex_array::arrays::PrimitiveArray; use vortex_array::dtype::DType; use vortex_array::dtype::NativePType; @@ -180,11 +182,15 @@ impl RDEncoder { /// /// Each value will be split into a left and right component, which are compressed individually. // TODO(joe): make fallible - pub fn encode(&self, array: &PrimitiveArray) -> ALPRDArray { - match_each_alp_float_ptype!(array.ptype(), |P| { self.encode_generic::

(array) }) + pub fn encode(&self, array: ArrayView<'_, Primitive>, ctx: &mut ExecutionCtx) -> ALPRDArray { + match_each_alp_float_ptype!(array.ptype(), |P| { self.encode_generic::

(array, ctx) }) } - fn encode_generic(&self, array: &PrimitiveArray) -> ALPRDArray + fn encode_generic( + &self, + array: ArrayView<'_, Primitive>, + ctx: &mut ExecutionCtx, + ) -> ALPRDArray where T: ALPRDFloat + NativePType, T::UINT: NativePType, @@ -283,6 +289,7 @@ impl RDEncoder { packed_right, self.right_bit_width, exceptions, + ctx, ) .vortex_expect("ALPRDArray construction in encode") } diff --git a/encodings/alp/src/alp_rd/ops.rs b/encodings/alp/src/alp_rd/ops.rs index ed4dcba2dfd..0351d446424 100644 --- a/encodings/alp/src/alp_rd/ops.rs +++ b/encodings/alp/src/alp_rd/ops.rs @@ -16,7 +16,7 @@ impl OperationsVTable for ALPRD { fn scalar_at( array: ArrayView<'_, ALPRD>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { // The left value can either be a direct value, or an exception. // The exceptions array represents exception positions with non-null values. @@ -32,7 +32,7 @@ impl OperationsVTable for ALPRD { _ => { let left_code: u16 = array .left_parts() - .scalar_at(index)? + .execute_scalar(index, ctx)? .as_primitive() .as_::() .vortex_expect("left_code must be non-null"); @@ -44,7 +44,7 @@ impl OperationsVTable for ALPRD { Ok(if array.dtype().as_ptype() == PType::F32 { let right: u32 = array .right_parts() - .scalar_at(index)? + .execute_scalar(index, ctx)? .as_primitive() .as_::() .vortex_expect("non-null"); @@ -53,7 +53,7 @@ impl OperationsVTable for ALPRD { } else { let right: u64 = array .right_parts() - .scalar_at(index)? + .execute_scalar(index, ctx)? .as_primitive() .as_::() .vortex_expect("non-null"); @@ -66,6 +66,8 @@ impl OperationsVTable for ALPRD { #[cfg(test)] mod test { use rstest::rstest; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; use vortex_array::scalar::Scalar; @@ -78,8 +80,9 @@ mod test { #[case(0.1f32, 0.2f32, 3e25f32)] #[case(0.1f64, 0.2f64, 3e100f64)] fn test_slice(#[case] a: T, #[case] b: T, #[case] outlier: T) { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = PrimitiveArray::from_iter([a, b, outlier]); - let encoded = RDEncoder::new(&[a, b]).encode(&array); + let encoded = RDEncoder::new(&[a, b]).encode(array.as_view(), &mut ctx); assert!(encoded.left_parts_patches().is_some()); assert_arrays_eq!(encoded, array); @@ -93,19 +96,21 @@ mod test { #[case] b: T, #[case] outlier: T, ) { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = PrimitiveArray::from_iter([a, b, outlier]); - let encoded = RDEncoder::new(&[a, b]).encode(&array); + let encoded = RDEncoder::new(&[a, b]).encode(array.as_view(), &mut ctx); assert!(encoded.left_parts_patches().is_some()); assert_arrays_eq!(encoded, array); } #[test] fn nullable_scalar_at() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let a = 0.1f64; let b = 0.2f64; let outlier = 3e100f64; let array = PrimitiveArray::from_option_iter([Some(a), Some(b), Some(outlier)]); - let encoded = RDEncoder::new(&[a, b]).encode(&array); + let encoded = RDEncoder::new(&[a, b]).encode(array.as_view(), &mut ctx); assert!(encoded.left_parts_patches().is_some()); assert_arrays_eq!( encoded, diff --git a/encodings/alp/src/lib.rs b/encodings/alp/src/lib.rs index 3a955f368af..7da676d6c6c 100644 --- a/encodings/alp/src/lib.rs +++ b/encodings/alp/src/lib.rs @@ -18,9 +18,11 @@ pub use alp::*; pub use alp_rd::*; +use vortex_array::ArrayVTable; use vortex_array::aggregate_fn::AggregateFnVTable; use vortex_array::aggregate_fn::fns::nan_count::NanCount; use vortex_array::aggregate_fn::session::AggregateFnSessionExt; +use vortex_array::arrays::patched::use_experimental_patches; use vortex_array::session::ArraySessionExt; use vortex_session::VortexSession; @@ -29,12 +31,18 @@ mod alp_rd; /// Initialize ALP encoding in the given session. pub fn initialize(session: &VortexSession) { - session.arrays().register(ALP); + // If we're using the experimental Patched encoding, register a shim + // for ALP with interior patches to decode as Patched array. + if use_experimental_patches() { + session.arrays().register(ALPPatchedPlugin); + } else { + session.arrays().register(ALP); + } session.arrays().register(ALPRD); // Register the ALP-specific NaN count aggregate kernel. session.aggregate_fns().register_aggregate_kernel( - ALP::ID, + ALP.id(), Some(NanCount.id()), &compute::nan_count::ALPNanCountKernel, ); diff --git a/encodings/bytebool/public-api.lock b/encodings/bytebool/public-api.lock index 63d9c1b9518..afbeea54ac4 100644 --- a/encodings/bytebool/public-api.lock +++ b/encodings/bytebool/public-api.lock @@ -4,13 +4,11 @@ pub struct vortex_bytebool::ByteBool impl vortex_bytebool::ByteBool -pub const vortex_bytebool::ByteBool::ID: vortex_array::array::ArrayId +pub fn vortex_bytebool::ByteBool::from_option_vec(alloc::vec::Vec>) -> vortex_bytebool::ByteBoolArray -pub fn vortex_bytebool::ByteBool::from_option_vec(data: alloc::vec::Vec>) -> vortex_bytebool::ByteBoolArray +pub fn vortex_bytebool::ByteBool::from_vec>(alloc::vec::Vec, V) -> vortex_bytebool::ByteBoolArray -pub fn vortex_bytebool::ByteBool::from_vec>(data: alloc::vec::Vec, validity: V) -> vortex_bytebool::ByteBoolArray - -pub fn vortex_bytebool::ByteBool::new(buffer: vortex_array::buffer::BufferHandle, validity: vortex_array::validity::Validity) -> vortex_bytebool::ByteBoolArray +pub fn vortex_bytebool::ByteBool::new(vortex_array::buffer::BufferHandle, vortex_array::validity::Validity) -> vortex_bytebool::ByteBoolArray impl core::clone::Clone for vortex_bytebool::ByteBool @@ -18,7 +16,7 @@ pub fn vortex_bytebool::ByteBool::clone(&self) -> vortex_bytebool::ByteBool impl core::fmt::Debug for vortex_bytebool::ByteBool -pub fn vortex_bytebool::ByteBool::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_bytebool::ByteBool::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::array::vtable::VTable for vortex_bytebool::ByteBool @@ -28,108 +26,98 @@ pub type vortex_bytebool::ByteBool::OperationsVTable = vortex_bytebool::ByteBool pub type vortex_bytebool::ByteBool::ValidityVTable = vortex_bytebool::ByteBool -pub fn vortex_bytebool::ByteBool::buffer(array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_bytebool::ByteBool::buffer(vortex_array::array::view::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_bytebool::ByteBool::buffer_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_bytebool::ByteBool::buffer_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_bytebool::ByteBool::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_bytebool::ByteBool::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_bytebool::ByteBool::execute(array: vortex_array::array::typed::Array, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_bytebool::ByteBool::execute(vortex_array::array::typed::Array, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_bytebool::ByteBool::execute_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_bytebool::ByteBool::execute_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_bytebool::ByteBool::id(&self) -> vortex_array::array::ArrayId -pub fn vortex_bytebool::ByteBool::nbuffers(_array: vortex_array::array::view::ArrayView<'_, Self>) -> usize +pub fn vortex_bytebool::ByteBool::nbuffers(vortex_array::array::view::ArrayView<'_, Self>) -> usize -pub fn vortex_bytebool::ByteBool::reduce_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_bytebool::ByteBool::reduce_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_bytebool::ByteBool::serialize(_array: vortex_array::array::view::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_bytebool::ByteBool::serialize(vortex_array::array::view::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_bytebool::ByteBool::slot_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_bytebool::ByteBool::slot_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_bytebool::ByteBool::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_bytebool::ByteBool::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::array::vtable::operations::OperationsVTable for vortex_bytebool::ByteBool -pub fn vortex_bytebool::ByteBool::scalar_at(array: vortex_array::array::view::ArrayView<'_, vortex_bytebool::ByteBool>, index: usize, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_bytebool::ByteBool::scalar_at(vortex_array::array::view::ArrayView<'_, vortex_bytebool::ByteBool>, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::array::vtable::validity::ValidityVTable for vortex_bytebool::ByteBool -pub fn vortex_bytebool::ByteBool::validity(array: vortex_array::array::view::ArrayView<'_, vortex_bytebool::ByteBool>) -> vortex_error::VortexResult +pub fn vortex_bytebool::ByteBool::validity(vortex_array::array::view::ArrayView<'_, vortex_bytebool::ByteBool>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::take::TakeExecute for vortex_bytebool::ByteBool -pub fn vortex_bytebool::ByteBool::take(array: vortex_array::array::view::ArrayView<'_, Self>, indices: &vortex_array::array::erased::ArrayRef, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_bytebool::ByteBool::take(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_bytebool::ByteBool -pub fn vortex_bytebool::ByteBool::slice(array: vortex_array::array::view::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_bytebool::ByteBool::slice(vortex_array::array::view::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::kernel::CastKernel for vortex_bytebool::ByteBool + +pub fn vortex_bytebool::ByteBool::cast(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::dtype::DType, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::kernel::CastReduce for vortex_bytebool::ByteBool -pub fn vortex_bytebool::ByteBool::cast(array: vortex_array::array::view::ArrayView<'_, Self>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_bytebool::ByteBool::cast(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::kernel::MaskReduce for vortex_bytebool::ByteBool -pub fn vortex_bytebool::ByteBool::mask(array: vortex_array::array::view::ArrayView<'_, Self>, mask: &vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_bytebool::ByteBool::mask(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult> pub struct vortex_bytebool::ByteBoolData impl vortex_bytebool::ByteBoolData -pub fn vortex_bytebool::ByteBoolData::as_slice(&self) -> &[bool] - pub fn vortex_bytebool::ByteBoolData::buffer(&self) -> &vortex_array::buffer::BufferHandle -pub fn vortex_bytebool::ByteBoolData::from_vec>(data: alloc::vec::Vec, validity: V) -> Self - pub fn vortex_bytebool::ByteBoolData::is_empty(&self) -> bool pub fn vortex_bytebool::ByteBoolData::len(&self) -> usize -pub fn vortex_bytebool::ByteBoolData::new(buffer: vortex_array::buffer::BufferHandle, validity: vortex_array::validity::Validity) -> Self +pub fn vortex_bytebool::ByteBoolData::new(vortex_array::buffer::BufferHandle) -> Self + +pub fn vortex_bytebool::ByteBoolData::truthy_bytes(&self) -> &[u8] -pub fn vortex_bytebool::ByteBoolData::validate(buffer: &vortex_array::buffer::BufferHandle, validity: &vortex_array::validity::Validity, dtype: &vortex_array::dtype::DType, len: usize) -> vortex_error::VortexResult<()> +pub fn vortex_bytebool::ByteBoolData::validate(&vortex_array::buffer::BufferHandle, &vortex_array::validity::Validity, &vortex_array::dtype::DType, usize) -> vortex_error::VortexResult<()> impl core::clone::Clone for vortex_bytebool::ByteBoolData pub fn vortex_bytebool::ByteBoolData::clone(&self) -> vortex_bytebool::ByteBoolData -impl core::convert::From> for vortex_bytebool::ByteBoolData - -pub fn vortex_bytebool::ByteBoolData::from(value: alloc::vec::Vec) -> Self - -impl core::convert::From>> for vortex_bytebool::ByteBoolData - -pub fn vortex_bytebool::ByteBoolData::from(value: alloc::vec::Vec>) -> Self - impl core::fmt::Debug for vortex_bytebool::ByteBoolData -pub fn vortex_bytebool::ByteBoolData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_bytebool::ByteBoolData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_bytebool::ByteBoolData -pub fn vortex_bytebool::ByteBoolData::fmt(&self, _f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_bytebool::ByteBoolData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::hash::ArrayEq for vortex_bytebool::ByteBoolData -pub fn vortex_bytebool::ByteBoolData::array_eq(&self, other: &Self, precision: vortex_array::hash::Precision) -> bool +pub fn vortex_bytebool::ByteBoolData::array_eq(&self, &Self, vortex_array::hash::Precision) -> bool impl vortex_array::hash::ArrayHash for vortex_bytebool::ByteBoolData -pub fn vortex_bytebool::ByteBoolData::array_hash(&self, state: &mut H, precision: vortex_array::hash::Precision) +pub fn vortex_bytebool::ByteBoolData::array_hash(&self, &mut H, vortex_array::hash::Precision) pub trait vortex_bytebool::ByteBoolArrayExt: vortex_array::array::typed::TypedArrayRef pub fn vortex_bytebool::ByteBoolArrayExt::validity(&self) -> vortex_array::validity::Validity -pub fn vortex_bytebool::ByteBoolArrayExt::validity_mask(&self) -> vortex_mask::Mask - impl> vortex_bytebool::ByteBoolArrayExt for T pub fn T::validity(&self) -> vortex_array::validity::Validity -pub fn T::validity_mask(&self) -> vortex_mask::Mask - pub type vortex_bytebool::ByteBoolArray = vortex_array::array::typed::Array diff --git a/encodings/bytebool/src/array.rs b/encodings/bytebool/src/array.rs index c1a2ebf7fce..538c6c079f3 100644 --- a/encodings/bytebool/src/array.rs +++ b/encodings/bytebool/src/array.rs @@ -29,14 +29,14 @@ use vortex_array::vtable::VTable; use vortex_array::vtable::ValidityVTable; use vortex_array::vtable::child_to_validity; use vortex_array::vtable::validity_to_child; -use vortex_buffer::BitBuffer; +use vortex_buffer::BitBufferMut; use vortex_buffer::ByteBuffer; use vortex_error::VortexResult; use vortex_error::vortex_bail; use vortex_error::vortex_ensure; use vortex_error::vortex_panic; -use vortex_mask::Mask; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::kernel::PARENT_KERNELS; @@ -62,7 +62,8 @@ impl VTable for ByteBool { type ValidityVTable = Self; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.bytebool"); + *ID } fn validate( @@ -72,7 +73,7 @@ impl VTable for ByteBool { len: usize, slots: &[Option], ) -> VortexResult<()> { - let validity = child_to_validity(&slots[VALIDITY_SLOT], dtype.nullability()); + let validity = child_to_validity(slots[VALIDITY_SLOT].as_ref(), dtype.nullability()); ByteBoolData::validate(data.buffer(), &validity, dtype, len) } @@ -130,7 +131,7 @@ impl VTable for ByteBool { } let buffer = buffers[0].clone(); - let data = ByteBoolData::new(buffer, validity.clone()); + let data = ByteBoolData::new(buffer); let slots = ByteBoolData::make_slots(&validity, len); Ok(ArrayParts::new(self.clone(), dtype.clone(), len, data).with_slots(slots)) } @@ -148,7 +149,8 @@ impl VTable for ByteBool { } fn execute(array: Array, _ctx: &mut ExecutionCtx) -> VortexResult { - let boolean_buffer = BitBuffer::from(array.as_slice()); + // convert truthy values to set/unset bits + let boolean_buffer = BitBufferMut::from(array.truthy_bytes()).freeze(); let validity = array.validity()?; Ok(ExecutionResult::done( BoolArray::new(boolean_buffer, validity).into_array(), @@ -184,14 +186,10 @@ impl Display for ByteBoolData { pub trait ByteBoolArrayExt: TypedArrayRef { fn validity(&self) -> Validity { child_to_validity( - &self.as_ref().slots()[VALIDITY_SLOT], + self.as_ref().slots()[VALIDITY_SLOT].as_ref(), self.as_ref().dtype().nullability(), ) } - - fn validity_mask(&self) -> Mask { - self.validity().to_mask(self.as_ref().len()) - } } impl> ByteBoolArrayExt for T {} @@ -200,12 +198,18 @@ impl> ByteBoolArrayExt for T {} pub struct ByteBool; impl ByteBool { - pub const ID: ArrayId = ArrayId::new_ref("vortex.bytebool"); - pub fn new(buffer: BufferHandle, validity: Validity) -> ByteBoolArray { + if let Some(len) = validity.maybe_len() { + assert_eq!( + buffer.len(), + len, + "ByteBool validity and bytes must have same length" + ); + } let dtype = DType::Bool(validity.nullability()); + let slots = ByteBoolData::make_slots(&validity, buffer.len()); - let data = ByteBoolData::new(buffer, validity); + let data = ByteBoolData::new(buffer); let len = data.len(); unsafe { Array::from_parts_unchecked( @@ -217,29 +221,22 @@ impl ByteBool { /// Construct a [`ByteBoolArray`] from a `Vec` and validity. pub fn from_vec>(data: Vec, validity: V) -> ByteBoolArray { let validity = validity.into(); - let data = ByteBoolData::from_vec(data, validity.clone()); - let dtype = DType::Bool(validity.nullability()); - let len = data.len(); - let slots = ByteBoolData::make_slots(&validity, len); - unsafe { - Array::from_parts_unchecked( - ArrayParts::new(ByteBool, dtype, len, data).with_slots(slots), - ) - } + // NOTE: this will not cause allocation on release builds + let bytes: Vec = data.into_iter().map(|b| b as u8).collect(); + let handle = BufferHandle::new_host(ByteBuffer::from(bytes)); + ByteBool::new(handle, validity) } /// Construct a [`ByteBoolArray`] from optional bools. pub fn from_option_vec(data: Vec>) -> ByteBoolArray { let validity = Validity::from_iter(data.iter().map(|v| v.is_some())); - let data = ByteBoolData::from(data); - let dtype = DType::Bool(validity.nullability()); - let len = data.len(); - let slots = ByteBoolData::make_slots(&validity, len); - unsafe { - Array::from_parts_unchecked( - ArrayParts::new(ByteBool, dtype, len, data).with_slots(slots), - ) - } + // NOTE: this will not cause allocation on release builds + let bytes: Vec = data + .into_iter() + .map(|b| b.unwrap_or_default() as u8) + .collect(); + let handle = BufferHandle::new_host(ByteBuffer::from(bytes)); + ByteBool::new(handle, validity) } } @@ -270,17 +267,7 @@ impl ByteBoolData { vec![validity_to_child(validity, len)] } - pub fn new(buffer: BufferHandle, validity: Validity) -> Self { - let length = buffer.len(); - if let Some(vlen) = validity.maybe_len() - && length != vlen - { - vortex_panic!( - "Buffer length ({}) does not match validity length ({})", - length, - vlen - ); - } + pub fn new(buffer: BufferHandle) -> Self { Self { buffer } } @@ -294,21 +281,15 @@ impl ByteBoolData { self.buffer.len() == 0 } - // TODO(ngates): deprecate construction from vec - pub fn from_vec>(data: Vec, validity: V) -> Self { - let validity = validity.into(); - // SAFETY: we are transmuting a Vec into a Vec - let data: Vec = unsafe { std::mem::transmute(data) }; - Self::new(BufferHandle::new_host(ByteBuffer::from(data)), validity) - } - pub fn buffer(&self) -> &BufferHandle { &self.buffer } - pub fn as_slice(&self) -> &[bool] { - // Safety: The internal buffer contains byte-sized bools - unsafe { std::mem::transmute(self.buffer().as_host().as_slice()) } + /// Get access to the underlying 8-bit truthy values. + /// + /// The zero byte indicates `false`, and any non-zero byte is a `true`. + pub fn truthy_bytes(&self) -> &[u8] { + self.buffer().as_host().as_slice() } } @@ -331,28 +312,12 @@ impl OperationsVTable for ByteBool { } } -impl From> for ByteBoolData { - fn from(value: Vec) -> Self { - Self::from_vec(value, Validity::AllValid) - } -} - -impl From>> for ByteBoolData { - fn from(value: Vec>) -> Self { - let validity = Validity::from_iter(value.iter().map(|v| v.is_some())); - - // This doesn't reallocate, and the compiler even vectorizes it - let data = value.into_iter().map(Option::unwrap_or_default).collect(); - - Self::from_vec(data, validity) - } -} - #[cfg(test)] mod tests { use vortex_array::ArrayContext; use vortex_array::IntoArray; use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::assert_arrays_eq; use vortex_array::serde::SerializeOptions; use vortex_array::serde::SerializedArray; @@ -372,15 +337,16 @@ mod tests { let arr = ByteBool::from_vec(v, Validity::AllValid); assert_eq!(v_len, arr.len()); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); for idx in 0..arr.len() { - assert!(arr.is_valid(idx).unwrap()); + assert!(arr.is_valid(idx, &mut ctx).unwrap()); } let v = vec![Some(true), None, Some(false)]; let arr = ByteBool::from_option_vec(v); - assert!(arr.is_valid(0).unwrap()); - assert!(!arr.is_valid(1).unwrap()); - assert!(arr.is_valid(2).unwrap()); + assert!(arr.is_valid(0, &mut ctx).unwrap()); + assert!(!arr.is_valid(1, &mut ctx).unwrap()); + assert!(arr.is_valid(2, &mut ctx).unwrap()); assert_eq!(arr.len(), 3); let v: Vec> = vec![None, None]; @@ -390,7 +356,7 @@ mod tests { assert_eq!(v_len, arr.len()); for idx in 0..arr.len() { - assert!(!arr.is_valid(idx).unwrap()); + assert!(!arr.is_valid(idx, &mut ctx).unwrap()); } assert_eq!(arr.len(), 2); } @@ -407,7 +373,7 @@ mod tests { let serialized = array .clone() .into_array() - .serialize(&ctx, &LEGACY_SESSION, &SerializeOptions::default()) + .serialize(&ctx, &session, &SerializeOptions::default()) .unwrap(); let mut concat = ByteBufferMut::empty(); diff --git a/encodings/bytebool/src/compute.rs b/encodings/bytebool/src/compute.rs index 0fd75cb8123..629a7ebe3e2 100644 --- a/encodings/bytebool/src/compute.rs +++ b/encodings/bytebool/src/compute.rs @@ -8,11 +8,14 @@ use vortex_array::ExecutionCtx; use vortex_array::IntoArray; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::dict::TakeExecute; +use vortex_array::buffer::BufferHandle; use vortex_array::dtype::DType; use vortex_array::match_each_integer_ptype; +use vortex_array::scalar_fn::fns::cast::CastKernel; use vortex_array::scalar_fn::fns::cast::CastReduce; use vortex_array::scalar_fn::fns::mask::MaskReduce; use vortex_array::validity::Validity; +use vortex_buffer::ByteBuffer; use vortex_error::VortexResult; use super::ByteBool; @@ -22,20 +25,43 @@ impl CastReduce for ByteBool { // ByteBool is essentially a bool array stored as bytes // The main difference from BoolArray is the storage format // For casting, we can decode to canonical (BoolArray) and let it handle the cast - // If just changing nullability, we can optimize - if array.dtype().eq_ignore_nullability(dtype) { - let new_validity = array - .validity()? - .cast_nullability(dtype.nullability(), array.len())?; + if !dtype.is_boolean() { + return Ok(None); + } + + let Some(new_validity) = array + .validity()? + .trivial_cast_nullability(dtype.nullability(), array.len())? + else { + return Ok(None); + }; + + Ok(Some( + ByteBool::new(array.buffer().clone(), new_validity).into_array(), + )) + } +} - return Ok(Some( - ByteBool::new(array.buffer().clone(), new_validity).into_array(), - )); +impl CastKernel for ByteBool { + fn cast( + array: ArrayView<'_, Self>, + dtype: &DType, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + // Only handle nullability changes here; non-bool targets fall through to canonicalization. + if !dtype.is_boolean() { + return Ok(None); } - // For other casts, decode to canonical and let BoolArray handle it - Ok(None) + let new_validity = + array + .validity()? + .cast_nullability(dtype.nullability(), array.len(), ctx)?; + + Ok(Some( + ByteBool::new(array.buffer().clone(), new_validity).into_array(), + )) } } @@ -58,23 +84,25 @@ impl TakeExecute for ByteBool { ctx: &mut ExecutionCtx, ) -> VortexResult> { let indices = indices.clone().execute::(ctx)?; - let bools = array.as_slice(); + let values = array.truthy_bytes(); // This handles combining validity from both source array and nullable indices let validity = array.validity()?.take(&indices.clone().into_array())?; - let taken_bools = match_each_integer_ptype!(indices.ptype(), |I| { + let taken = match_each_integer_ptype!(indices.ptype(), |I| { indices .as_slice::() .iter() .map(|&idx| { let idx: usize = idx.as_(); - bools[idx] + values[idx] }) - .collect::>() + .collect::() }); - Ok(Some(ByteBool::from_vec(taken_bools, validity).into_array())) + Ok(Some( + ByteBool::new(BufferHandle::new_host(taken), validity).into_array(), + )) } } diff --git a/encodings/bytebool/src/kernel.rs b/encodings/bytebool/src/kernel.rs index 4373520039a..ddf9679b3d5 100644 --- a/encodings/bytebool/src/kernel.rs +++ b/encodings/bytebool/src/kernel.rs @@ -3,8 +3,11 @@ use vortex_array::arrays::dict::TakeExecuteAdaptor; use vortex_array::kernel::ParentKernelSet; +use vortex_array::scalar_fn::fns::cast::CastExecuteAdaptor; use crate::ByteBool; -pub(crate) const PARENT_KERNELS: ParentKernelSet = - ParentKernelSet::new(&[ParentKernelSet::lift(&TakeExecuteAdaptor(ByteBool))]); +pub(crate) const PARENT_KERNELS: ParentKernelSet = ParentKernelSet::new(&[ + ParentKernelSet::lift(&CastExecuteAdaptor(ByteBool)), + ParentKernelSet::lift(&TakeExecuteAdaptor(ByteBool)), +]); diff --git a/encodings/bytebool/src/lib.rs b/encodings/bytebool/src/lib.rs index 35579a89dc8..3258341cdb9 100644 --- a/encodings/bytebool/src/lib.rs +++ b/encodings/bytebool/src/lib.rs @@ -1,6 +1,45 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +//! A Vortex encoding that mirrors Arrow's [8-bit Boolean canonical extension type][spec]. +//! +//! Each element is stored as a single byte. The zero byte represents `false` and any +//! non-zero byte represents `true`, matching the truthy semantics of the Arrow spec. This +//! trades 8x the storage of the bit-packed `Bool` layout for cheaper per-byte access — +//! useful when data arrives from a C ABI or other source that already emits byte-wide +//! booleans. On execution the array materializes into the standard bit-packed +//! [`BoolArray`][vortex_array::arrays::BoolArray]. +//! +//! # Examples +//! +//! Any non-zero byte in the backing buffer is treated as `true` when the array executes +//! to a canonical [`BoolArray`][vortex_array::arrays::BoolArray]: +//! +//! ``` +//! # use vortex_array::{IntoArray, LEGACY_SESSION, VortexSessionExecute}; +//! # use vortex_array::arrays::BoolArray; +//! # use vortex_array::arrays::bool::BoolArrayExt; +//! # use vortex_array::buffer::BufferHandle; +//! # use vortex_array::validity::Validity; +//! # use vortex_buffer::ByteBuffer; +//! # use vortex_bytebool::ByteBool; +//! # use vortex_error::VortexResult; +//! # fn main() -> VortexResult<()> { +//! # let mut ctx = LEGACY_SESSION.create_execution_ctx(); +//! let handle = BufferHandle::new_host(ByteBuffer::from(vec![0u8, 1, 42, 0])); +//! let array = ByteBool::new(handle, Validity::NonNullable); +//! +//! let bits = array.into_array().execute::(&mut ctx)?.to_bit_buffer(); +//! assert!(!bits.value(0)); +//! assert!(bits.value(1)); +//! assert!(bits.value(2)); // byte 42 is truthy +//! assert!(!bits.value(3)); +//! # Ok(()) +//! # } +//! ``` +//! +//! [spec]: https://arrow.apache.org/docs/format/CanonicalExtensions.html#bit-boolean + pub use array::*; mod array; diff --git a/encodings/datetime-parts/public-api.lock b/encodings/datetime-parts/public-api.lock index 3cdc22cc7b7..97aaa5dec69 100644 --- a/encodings/datetime-parts/public-api.lock +++ b/encodings/datetime-parts/public-api.lock @@ -4,11 +4,9 @@ pub struct vortex_datetime_parts::DateTimeParts impl vortex_datetime_parts::DateTimeParts -pub const vortex_datetime_parts::DateTimeParts::ID: vortex_array::array::ArrayId +pub fn vortex_datetime_parts::DateTimeParts::try_from_temporal(vortex_array::arrays::datetime::TemporalArray, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_datetime_parts::DateTimeParts::try_from_temporal(temporal: vortex_array::arrays::datetime::TemporalArray) -> vortex_error::VortexResult - -pub fn vortex_datetime_parts::DateTimeParts::try_new(dtype: vortex_array::dtype::DType, days: vortex_array::array::erased::ArrayRef, seconds: vortex_array::array::erased::ArrayRef, subseconds: vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_datetime_parts::DateTimeParts::try_new(vortex_array::dtype::DType, vortex_array::array::erased::ArrayRef, vortex_array::array::erased::ArrayRef, vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult impl core::clone::Clone for vortex_datetime_parts::DateTimeParts @@ -16,7 +14,7 @@ pub fn vortex_datetime_parts::DateTimeParts::clone(&self) -> vortex_datetime_par impl core::fmt::Debug for vortex_datetime_parts::DateTimeParts -pub fn vortex_datetime_parts::DateTimeParts::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_datetime_parts::DateTimeParts::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::array::vtable::VTable for vortex_datetime_parts::DateTimeParts @@ -26,91 +24,85 @@ pub type vortex_datetime_parts::DateTimeParts::OperationsVTable = vortex_datetim pub type vortex_datetime_parts::DateTimeParts::ValidityVTable = vortex_array::array::vtable::validity::ValidityVTableFromChild -pub fn vortex_datetime_parts::DateTimeParts::buffer(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_datetime_parts::DateTimeParts::buffer(vortex_array::array::view::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_datetime_parts::DateTimeParts::buffer_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_datetime_parts::DateTimeParts::buffer_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_datetime_parts::DateTimeParts::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_datetime_parts::DateTimeParts::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_datetime_parts::DateTimeParts::execute(array: vortex_array::array::typed::Array, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_datetime_parts::DateTimeParts::execute(vortex_array::array::typed::Array, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_datetime_parts::DateTimeParts::execute_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_datetime_parts::DateTimeParts::execute_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_datetime_parts::DateTimeParts::id(&self) -> vortex_array::array::ArrayId -pub fn vortex_datetime_parts::DateTimeParts::nbuffers(_array: vortex_array::array::view::ArrayView<'_, Self>) -> usize +pub fn vortex_datetime_parts::DateTimeParts::nbuffers(vortex_array::array::view::ArrayView<'_, Self>) -> usize -pub fn vortex_datetime_parts::DateTimeParts::reduce_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_datetime_parts::DateTimeParts::reduce_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_datetime_parts::DateTimeParts::serialize(array: vortex_array::array::view::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_datetime_parts::DateTimeParts::serialize(vortex_array::array::view::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_datetime_parts::DateTimeParts::slot_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_datetime_parts::DateTimeParts::slot_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_datetime_parts::DateTimeParts::validate(&self, _data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_datetime_parts::DateTimeParts::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::array::vtable::operations::OperationsVTable for vortex_datetime_parts::DateTimeParts -pub fn vortex_datetime_parts::DateTimeParts::scalar_at(array: vortex_array::array::view::ArrayView<'_, vortex_datetime_parts::DateTimeParts>, index: usize, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_datetime_parts::DateTimeParts::scalar_at(vortex_array::array::view::ArrayView<'_, vortex_datetime_parts::DateTimeParts>, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::array::vtable::validity::ValidityChild for vortex_datetime_parts::DateTimeParts -pub fn vortex_datetime_parts::DateTimeParts::validity_child(array: vortex_array::array::view::ArrayView<'_, vortex_datetime_parts::DateTimeParts>) -> vortex_array::array::erased::ArrayRef +pub fn vortex_datetime_parts::DateTimeParts::validity_child(vortex_array::array::view::ArrayView<'_, vortex_datetime_parts::DateTimeParts>) -> vortex_array::array::erased::ArrayRef impl vortex_array::arrays::dict::take::TakeExecute for vortex_datetime_parts::DateTimeParts -pub fn vortex_datetime_parts::DateTimeParts::take(array: vortex_array::array::view::ArrayView<'_, Self>, indices: &vortex_array::array::erased::ArrayRef, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_datetime_parts::DateTimeParts::take(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::kernel::FilterReduce for vortex_datetime_parts::DateTimeParts -pub fn vortex_datetime_parts::DateTimeParts::filter(array: vortex_array::array::view::ArrayView<'_, Self>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_datetime_parts::DateTimeParts::filter(vortex_array::array::view::ArrayView<'_, Self>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_datetime_parts::DateTimeParts -pub fn vortex_datetime_parts::DateTimeParts::slice(array: vortex_array::array::view::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_datetime_parts::DateTimeParts::slice(vortex_array::array::view::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::binary::compare::CompareKernel for vortex_datetime_parts::DateTimeParts -pub fn vortex_datetime_parts::DateTimeParts::compare(lhs: vortex_array::array::view::ArrayView<'_, Self>, rhs: &vortex_array::array::erased::ArrayRef, operator: vortex_array::scalar_fn::fns::operators::CompareOperator, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_datetime_parts::DateTimeParts::compare(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, vortex_array::scalar_fn::fns::operators::CompareOperator, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::kernel::CastReduce for vortex_datetime_parts::DateTimeParts -pub fn vortex_datetime_parts::DateTimeParts::cast(array: vortex_array::array::view::ArrayView<'_, Self>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_datetime_parts::DateTimeParts::cast(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::kernel::MaskReduce for vortex_datetime_parts::DateTimeParts -pub fn vortex_datetime_parts::DateTimeParts::mask(array: vortex_array::array::view::ArrayView<'_, Self>, mask: &vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_datetime_parts::DateTimeParts::mask(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult> pub struct vortex_datetime_parts::DateTimePartsData impl vortex_datetime_parts::DateTimePartsData -pub fn vortex_datetime_parts::DateTimePartsData::validate(dtype: &vortex_array::dtype::DType, days: &vortex_array::array::erased::ArrayRef, seconds: &vortex_array::array::erased::ArrayRef, subseconds: &vortex_array::array::erased::ArrayRef, len: usize) -> vortex_error::VortexResult<()> +pub fn vortex_datetime_parts::DateTimePartsData::validate(&vortex_array::dtype::DType, &vortex_array::array::erased::ArrayRef, &vortex_array::array::erased::ArrayRef, &vortex_array::array::erased::ArrayRef, usize) -> vortex_error::VortexResult<()> impl core::clone::Clone for vortex_datetime_parts::DateTimePartsData pub fn vortex_datetime_parts::DateTimePartsData::clone(&self) -> vortex_datetime_parts::DateTimePartsData -impl core::convert::TryFrom for vortex_datetime_parts::DateTimePartsData - -pub type vortex_datetime_parts::DateTimePartsData::Error = vortex_error::VortexError - -pub fn vortex_datetime_parts::DateTimePartsData::try_from(array: vortex_array::arrays::datetime::TemporalArray) -> core::result::Result - impl core::fmt::Debug for vortex_datetime_parts::DateTimePartsData -pub fn vortex_datetime_parts::DateTimePartsData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_datetime_parts::DateTimePartsData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_datetime_parts::DateTimePartsData -pub fn vortex_datetime_parts::DateTimePartsData::fmt(&self, _f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_datetime_parts::DateTimePartsData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::hash::ArrayEq for vortex_datetime_parts::DateTimePartsData -pub fn vortex_datetime_parts::DateTimePartsData::array_eq(&self, _other: &Self, _precision: vortex_array::hash::Precision) -> bool +pub fn vortex_datetime_parts::DateTimePartsData::array_eq(&self, &Self, vortex_array::hash::Precision) -> bool impl vortex_array::hash::ArrayHash for vortex_datetime_parts::DateTimePartsData -pub fn vortex_datetime_parts::DateTimePartsData::array_hash(&self, _state: &mut H, _precision: vortex_array::hash::Precision) +pub fn vortex_datetime_parts::DateTimePartsData::array_hash(&self, &mut H, vortex_array::hash::Precision) #[repr(C)] pub struct vortex_datetime_parts::DateTimePartsMetadata @@ -126,11 +118,11 @@ pub fn vortex_datetime_parts::DateTimePartsMetadata::days_ptype(&self) -> vortex pub fn vortex_datetime_parts::DateTimePartsMetadata::seconds_ptype(&self) -> vortex_array::dtype::ptype::PType -pub fn vortex_datetime_parts::DateTimePartsMetadata::set_days_ptype(&mut self, value: vortex_array::dtype::ptype::PType) +pub fn vortex_datetime_parts::DateTimePartsMetadata::set_days_ptype(&mut self, vortex_array::dtype::ptype::PType) -pub fn vortex_datetime_parts::DateTimePartsMetadata::set_seconds_ptype(&mut self, value: vortex_array::dtype::ptype::PType) +pub fn vortex_datetime_parts::DateTimePartsMetadata::set_seconds_ptype(&mut self, vortex_array::dtype::ptype::PType) -pub fn vortex_datetime_parts::DateTimePartsMetadata::set_subseconds_ptype(&mut self, value: vortex_array::dtype::ptype::PType) +pub fn vortex_datetime_parts::DateTimePartsMetadata::set_subseconds_ptype(&mut self, vortex_array::dtype::ptype::PType) pub fn vortex_datetime_parts::DateTimePartsMetadata::subseconds_ptype(&self) -> vortex_array::dtype::ptype::PType @@ -152,7 +144,7 @@ pub fn vortex_datetime_parts::DateTimePartsMetadata::default() -> Self impl core::fmt::Debug for vortex_datetime_parts::DateTimePartsMetadata -pub fn vortex_datetime_parts::DateTimePartsMetadata::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_datetime_parts::DateTimePartsMetadata::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl prost::message::Message for vortex_datetime_parts::DateTimePartsMetadata @@ -160,6 +152,62 @@ pub fn vortex_datetime_parts::DateTimePartsMetadata::clear(&mut self) pub fn vortex_datetime_parts::DateTimePartsMetadata::encoded_len(&self) -> usize +pub struct vortex_datetime_parts::DateTimePartsParts + +pub vortex_datetime_parts::DateTimePartsParts::days: vortex_array::array::erased::ArrayRef + +pub vortex_datetime_parts::DateTimePartsParts::seconds: vortex_array::array::erased::ArrayRef + +pub vortex_datetime_parts::DateTimePartsParts::subseconds: vortex_array::array::erased::ArrayRef + +pub struct vortex_datetime_parts::DateTimePartsSlots + +pub vortex_datetime_parts::DateTimePartsSlots::days: vortex_array::array::erased::ArrayRef + +pub vortex_datetime_parts::DateTimePartsSlots::seconds: vortex_array::array::erased::ArrayRef + +pub vortex_datetime_parts::DateTimePartsSlots::subseconds: vortex_array::array::erased::ArrayRef + +impl vortex_datetime_parts::DateTimePartsSlots + +pub const vortex_datetime_parts::DateTimePartsSlots::COUNT: usize + +pub const vortex_datetime_parts::DateTimePartsSlots::DAYS: usize + +pub const vortex_datetime_parts::DateTimePartsSlots::NAMES: [&'static str; 3] + +pub const vortex_datetime_parts::DateTimePartsSlots::SECONDS: usize + +pub const vortex_datetime_parts::DateTimePartsSlots::SUBSECONDS: usize + +pub fn vortex_datetime_parts::DateTimePartsSlots::from_slots(alloc::vec::Vec>) -> Self + +pub fn vortex_datetime_parts::DateTimePartsSlots::into_slots(self) -> alloc::vec::Vec> + +pub struct vortex_datetime_parts::DateTimePartsSlotsView<'a> + +pub vortex_datetime_parts::DateTimePartsSlotsView::days: &'a vortex_array::array::erased::ArrayRef + +pub vortex_datetime_parts::DateTimePartsSlotsView::seconds: &'a vortex_array::array::erased::ArrayRef + +pub vortex_datetime_parts::DateTimePartsSlotsView::subseconds: &'a vortex_array::array::erased::ArrayRef + +impl<'a> vortex_datetime_parts::DateTimePartsSlotsView<'a> + +pub fn vortex_datetime_parts::DateTimePartsSlotsView<'a>::from_slots(&'a [core::option::Option]) -> Self + +pub fn vortex_datetime_parts::DateTimePartsSlotsView<'a>::to_owned(&self) -> vortex_datetime_parts::DateTimePartsSlots + +impl<'a> core::clone::Clone for vortex_datetime_parts::DateTimePartsSlotsView<'a> + +pub fn vortex_datetime_parts::DateTimePartsSlotsView<'a>::clone(&self) -> vortex_datetime_parts::DateTimePartsSlotsView<'a> + +impl<'a> core::fmt::Debug for vortex_datetime_parts::DateTimePartsSlotsView<'a> + +pub fn vortex_datetime_parts::DateTimePartsSlotsView<'a>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +impl<'a> core::marker::Copy for vortex_datetime_parts::DateTimePartsSlotsView<'a> + pub struct vortex_datetime_parts::TemporalParts pub vortex_datetime_parts::TemporalParts::days: vortex_array::array::erased::ArrayRef @@ -168,24 +216,36 @@ pub vortex_datetime_parts::TemporalParts::seconds: vortex_array::array::erased:: pub vortex_datetime_parts::TemporalParts::subseconds: vortex_array::array::erased::ArrayRef -pub trait vortex_datetime_parts::DateTimePartsArrayExt: vortex_array::array::typed::TypedArrayRef +pub trait vortex_datetime_parts::DateTimePartsArraySlotsExt: vortex_array::array::typed::TypedArrayRef -pub fn vortex_datetime_parts::DateTimePartsArrayExt::days(&self) -> &vortex_array::array::erased::ArrayRef +pub fn vortex_datetime_parts::DateTimePartsArraySlotsExt::days(&self) -> &vortex_array::array::erased::ArrayRef -pub fn vortex_datetime_parts::DateTimePartsArrayExt::seconds(&self) -> &vortex_array::array::erased::ArrayRef +pub fn vortex_datetime_parts::DateTimePartsArraySlotsExt::seconds(&self) -> &vortex_array::array::erased::ArrayRef -pub fn vortex_datetime_parts::DateTimePartsArrayExt::subseconds(&self) -> &vortex_array::array::erased::ArrayRef +pub fn vortex_datetime_parts::DateTimePartsArraySlotsExt::slots_view(&self) -> vortex_datetime_parts::DateTimePartsSlotsView<'_> -impl> vortex_datetime_parts::DateTimePartsArrayExt for T +pub fn vortex_datetime_parts::DateTimePartsArraySlotsExt::subseconds(&self) -> &vortex_array::array::erased::ArrayRef + +impl> vortex_datetime_parts::DateTimePartsArraySlotsExt for T pub fn T::days(&self) -> &vortex_array::array::erased::ArrayRef pub fn T::seconds(&self) -> &vortex_array::array::erased::ArrayRef +pub fn T::slots_view(&self) -> vortex_datetime_parts::DateTimePartsSlotsView<'_> + pub fn T::subseconds(&self) -> &vortex_array::array::erased::ArrayRef -pub fn vortex_datetime_parts::initialize(session: &vortex_session::VortexSession) +pub trait vortex_datetime_parts::DateTimePartsOwnedExt + +pub fn vortex_datetime_parts::DateTimePartsOwnedExt::into_parts(self) -> vortex_datetime_parts::DateTimePartsParts + +impl vortex_datetime_parts::DateTimePartsOwnedExt for vortex_array::array::typed::Array + +pub fn vortex_array::array::typed::Array::into_parts(self) -> vortex_datetime_parts::DateTimePartsParts + +pub fn vortex_datetime_parts::initialize(&vortex_session::VortexSession) -pub fn vortex_datetime_parts::split_temporal(array: vortex_array::arrays::datetime::TemporalArray) -> vortex_error::VortexResult +pub fn vortex_datetime_parts::split_temporal(vortex_array::arrays::datetime::TemporalArray, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub type vortex_datetime_parts::DateTimePartsArray = vortex_array::array::typed::Array diff --git a/encodings/datetime-parts/src/array.rs b/encodings/datetime-parts/src/array.rs index f66bd2b742a..86e1927422e 100644 --- a/encodings/datetime-parts/src/array.rs +++ b/encodings/datetime-parts/src/array.rs @@ -7,6 +7,7 @@ use std::fmt::Formatter; use std::hash::Hasher; use prost::Message; +use vortex_array::AnyCanonical; use vortex_array::Array; use vortex_array::ArrayEq; use vortex_array::ArrayHash; @@ -18,23 +19,25 @@ use vortex_array::ExecutionCtx; use vortex_array::ExecutionResult; use vortex_array::IntoArray; use vortex_array::Precision; -use vortex_array::TypedArrayRef; +use vortex_array::array_slots; +use vortex_array::arrays::Primitive; use vortex_array::arrays::TemporalArray; use vortex_array::buffer::BufferHandle; use vortex_array::dtype::DType; use vortex_array::dtype::Nullability; use vortex_array::dtype::PType; +use vortex_array::require_child; use vortex_array::serde::ArrayChildren; use vortex_array::vtable::VTable; use vortex_array::vtable::ValidityChild; use vortex_array::vtable::ValidityVTableFromChild; -use vortex_error::VortexExpect; use vortex_error::VortexResult; use vortex_error::vortex_bail; use vortex_error::vortex_ensure; use vortex_error::vortex_err; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::TemporalParts; use crate::canonical::decode_to_temporal; @@ -92,7 +95,8 @@ impl VTable for DateTimeParts { type ValidityVTable = ValidityVTableFromChild; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.datetimeparts"); + *ID } fn validate( @@ -102,16 +106,8 @@ impl VTable for DateTimeParts { len: usize, slots: &[Option], ) -> VortexResult<()> { - let days = slots[DAYS_SLOT] - .as_ref() - .vortex_expect("DateTimePartsArray days slot"); - let seconds = slots[SECONDS_SLOT] - .as_ref() - .vortex_expect("DateTimePartsArray seconds slot"); - let subseconds = slots[SUBSECONDS_SLOT] - .as_ref() - .vortex_expect("DateTimePartsArray subseconds slot"); - DateTimePartsData::validate(dtype, days, seconds, subseconds, len) + let slots = DateTimePartsSlotsView::from_slots(slots); + DateTimePartsData::validate(dtype, slots.days, slots.seconds, slots.subseconds, len) } fn nbuffers(_array: ArrayView<'_, Self>) -> usize { @@ -179,12 +175,20 @@ impl VTable for DateTimeParts { } fn slot_name(_array: ArrayView<'_, Self>, idx: usize) -> String { - SLOT_NAMES[idx].to_string() + DateTimePartsSlots::NAMES[idx].to_string() } fn execute(array: Array, ctx: &mut ExecutionCtx) -> VortexResult { + let array = require_child!(array, array.days(), DateTimePartsSlots::DAYS => Primitive); + let array = + require_child!(array, array.seconds(), DateTimePartsSlots::SECONDS => AnyCanonical); + let array = require_child!(array, array.subseconds(), DateTimePartsSlots::SUBSECONDS => AnyCanonical); + + let dtype = array.dtype().clone(); + let parts = array.into_parts(); + Ok(ExecutionResult::done( - decode_to_temporal(&array, ctx)?.into_array(), + decode_to_temporal(parts, &dtype, ctx)?.into_array(), )) } @@ -206,52 +210,62 @@ impl VTable for DateTimeParts { } } -/// The days component of the datetime, stored as an integer array. -pub(super) const DAYS_SLOT: usize = 0; -/// The seconds component of the datetime (within the day). -pub(super) const SECONDS_SLOT: usize = 1; -/// The sub-second component of the datetime. -pub(super) const SUBSECONDS_SLOT: usize = 2; -pub(super) const NUM_SLOTS: usize = 3; -pub(super) const SLOT_NAMES: [&str; NUM_SLOTS] = ["days", "seconds", "subseconds"]; +#[array_slots(DateTimeParts)] +pub struct DateTimePartsSlots { + /// The days component of the datetime, stored as an integer array. + pub days: ArrayRef, + /// The seconds component of the datetime (within the day). + pub seconds: ArrayRef, + /// The sub-second component of the datetime. + pub subseconds: ArrayRef, +} #[derive(Clone, Debug)] pub struct DateTimePartsData {} -impl Display for DateTimePartsData { - fn fmt(&self, _f: &mut Formatter<'_>) -> std::fmt::Result { - Ok(()) - } +pub struct DateTimePartsParts { + pub days: ArrayRef, + pub seconds: ArrayRef, + pub subseconds: ArrayRef, } -pub trait DateTimePartsArrayExt: TypedArrayRef { - fn days(&self) -> &ArrayRef { - self.as_ref().slots()[DAYS_SLOT] - .as_ref() - .vortex_expect("DateTimePartsArray days slot") - } +pub trait DateTimePartsOwnedExt { + fn into_parts(self) -> DateTimePartsParts; +} - fn seconds(&self) -> &ArrayRef { - self.as_ref().slots()[SECONDS_SLOT] - .as_ref() - .vortex_expect("DateTimePartsArray seconds slot") +impl DateTimePartsOwnedExt for Array { + fn into_parts(self) -> DateTimePartsParts { + match self.try_into_parts() { + Ok(parts) => { + let slots = DateTimePartsSlots::from_slots(parts.slots); + DateTimePartsParts { + days: slots.days, + seconds: slots.seconds, + subseconds: slots.subseconds, + } + } + Err(array) => { + let view = DateTimePartsSlotsView::from_slots(array.as_ref().slots()); + DateTimePartsParts { + days: view.days.clone(), + seconds: view.seconds.clone(), + subseconds: view.subseconds.clone(), + } + } + } } +} - fn subseconds(&self) -> &ArrayRef { - self.as_ref().slots()[SUBSECONDS_SLOT] - .as_ref() - .vortex_expect("DateTimePartsArray subseconds slot") +impl Display for DateTimePartsData { + fn fmt(&self, _f: &mut Formatter<'_>) -> std::fmt::Result { + Ok(()) } } -impl> DateTimePartsArrayExt for T {} - #[derive(Clone, Debug)] pub struct DateTimeParts; impl DateTimeParts { - pub const ID: ArrayId = ArrayId::new_ref("vortex.datetimeparts"); - /// Construct a new [`DateTimePartsArray`] from its components. pub fn try_new( dtype: DType, @@ -271,13 +285,16 @@ impl DateTimeParts { } /// Construct a [`DateTimePartsArray`] from a [`TemporalArray`]. - pub fn try_from_temporal(temporal: TemporalArray) -> VortexResult { + pub fn try_from_temporal( + temporal: TemporalArray, + ctx: &mut ExecutionCtx, + ) -> VortexResult { let dtype = temporal.dtype().clone(); let TemporalParts { days, seconds, subseconds, - } = split_temporal(temporal)?; + } = split_temporal(temporal, ctx)?; Self::try_new(dtype, days, seconds, subseconds) } } diff --git a/encodings/datetime-parts/src/canonical.rs b/encodings/datetime-parts/src/canonical.rs index f1d1aaf6914..78dc8e689c1 100644 --- a/encodings/datetime-parts/src/canonical.rs +++ b/encodings/datetime-parts/src/canonical.rs @@ -4,31 +4,28 @@ use num_traits::AsPrimitive; use vortex_array::ExecutionCtx; use vortex_array::IntoArray; +use vortex_array::arrays::Primitive; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::TemporalArray; -use vortex_array::builtins::ArrayBuiltins; +use vortex_array::arrays::primitive::PrimitiveArrayExt; use vortex_array::dtype::DType; -use vortex_array::dtype::PType; use vortex_array::extension::datetime::TimeUnit; use vortex_array::extension::datetime::Timestamp; use vortex_array::match_each_integer_ptype; -use vortex_array::validity::Validity; use vortex_buffer::BufferMut; use vortex_error::VortexExpect as _; use vortex_error::VortexResult; use vortex_error::vortex_panic; -use crate::DateTimePartsArray; -use crate::array::DateTimePartsArrayExt; +use crate::array::DateTimePartsParts; -/// Decode an [Array] into a [TemporalArray]. -/// -/// Enforces that the passed array is actually a [DateTimePartsArray] with proper metadata. +/// Decode [`DateTimePartsParts`] into a [`TemporalArray`]. pub fn decode_to_temporal( - array: &DateTimePartsArray, + parts: DateTimePartsParts, + dtype: &DType, ctx: &mut ExecutionCtx, ) -> VortexResult { - let DType::Extension(ext) = array.dtype().clone() else { + let DType::Extension(ext) = dtype else { vortex_panic!(Compute: "expected dtype to be DType::Extension variant") }; @@ -44,21 +41,19 @@ pub fn decode_to_temporal( TimeUnit::Days => vortex_panic!(InvalidArgument: "cannot decode into TimeUnit::D"), }; - let days_buf = array - .days() - .cast(DType::Primitive(PType::I64, array.dtype().nullability())) - .vortex_expect("must be able to cast days to i64") - .execute::(ctx)?; - - // We start with the days component, which is always present. - // And then add the seconds and subseconds components. - // We split this into separate passes because often the seconds and/org subseconds components - // are constant. - let mut values: BufferMut = days_buf - .into_buffer::() - .map_each_in_place(|d| d * 86_400 * divisor); - - if let Some(seconds) = array.seconds().as_constant() { + // Days is guaranteed Primitive by require_child. + let days = parts.days.as_::(); + let validity = days.validity()?; + + let mut values: BufferMut = match_each_integer_ptype!(days.ptype(), |D| { + BufferMut::from_iter(days.as_slice::().iter().map(|d| { + let d: i64 = d.as_(); + d * 86_400 * divisor + })) + }); + + // Seconds/subseconds may be Constant — handle the fast path. + if let Some(seconds) = parts.seconds.as_constant() { let seconds = seconds .as_primitive() .as_::() @@ -68,7 +63,7 @@ pub fn decode_to_temporal( *v += seconds; } } else { - let seconds_buf = array.seconds().clone().execute::(ctx)?; + let seconds_buf = parts.seconds.execute::(ctx)?; match_each_integer_ptype!(seconds_buf.ptype(), |S| { for (v, second) in values.iter_mut().zip(seconds_buf.as_slice::()) { let second: i64 = second.as_(); @@ -77,7 +72,7 @@ pub fn decode_to_temporal( }); } - if let Some(subseconds) = array.subseconds().as_constant() { + if let Some(subseconds) = parts.subseconds.as_constant() { let subseconds = subseconds .as_primitive() .as_::() @@ -86,21 +81,17 @@ pub fn decode_to_temporal( *v += subseconds; } } else { - let subseconds_buf = array.subseconds().clone().execute::(ctx)?; + let subseconds_buf = parts.subseconds.execute::(ctx)?; match_each_integer_ptype!(subseconds_buf.ptype(), |S| { - for (v, subseconds) in values.iter_mut().zip(subseconds_buf.as_slice::()) { - let subseconds: i64 = subseconds.as_(); - *v += subseconds; + for (v, subsecond) in values.iter_mut().zip(subseconds_buf.as_slice::()) { + let subsecond: i64 = subsecond.as_(); + *v += subsecond; } }); } Ok(TemporalArray::new_timestamp( - PrimitiveArray::new( - values.freeze(), - Validity::copy_from_array(&array.clone().into_array())?, - ) - .into_array(), + PrimitiveArray::new(values.freeze(), validity).into_array(), options.unit, options.tz.clone(), )) @@ -121,6 +112,8 @@ mod test { use vortex_session::VortexSession; use crate::DateTimeParts; + use crate::array::DateTimePartsArraySlotsExt; + use crate::array::DateTimePartsParts; use crate::canonical::decode_to_temporal; #[rstest] @@ -140,13 +133,15 @@ mod test { ], validity.clone(), ); - let date_times = DateTimeParts::try_from_temporal(TemporalArray::new_timestamp( - milliseconds.clone().into_array(), - TimeUnit::Milliseconds, - Some("UTC".into()), - ))?; - let mut ctx = ExecutionCtx::new(VortexSession::empty()); + let date_times = DateTimeParts::try_from_temporal( + TemporalArray::new_timestamp( + milliseconds.clone().into_array(), + TimeUnit::Milliseconds, + Some("UTC".into()), + ), + &mut ctx, + )?; assert!( date_times @@ -155,18 +150,20 @@ mod test { .mask_eq(&validity, &mut ctx)? ); - let primitive_values = decode_to_temporal(&date_times, &mut ctx)? + let dtype = date_times.dtype().clone(); + let parts = DateTimePartsParts { + days: date_times.days().clone(), + seconds: date_times.seconds().clone(), + subseconds: date_times.subseconds().clone(), + }; + + let primitive_values = decode_to_temporal(parts, &dtype, &mut ctx)? .temporal_values() .clone() .execute::(&mut ctx)?; assert_arrays_eq!(primitive_values, milliseconds); - assert!( - primitive_values - .validity() - .unwrap() - .mask_eq(&validity, &mut ctx)? - ); + assert!(primitive_values.validity()?.mask_eq(&validity, &mut ctx)?); Ok(()) } } diff --git a/encodings/datetime-parts/src/compress.rs b/encodings/datetime-parts/src/compress.rs index 2edb6097c7b..676e7bbfd43 100644 --- a/encodings/datetime-parts/src/compress.rs +++ b/encodings/datetime-parts/src/compress.rs @@ -2,18 +2,16 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors use vortex_array::ArrayRef; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; -use vortex_array::ToCanonical; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::TemporalArray; use vortex_array::builtins::ArrayBuiltins; use vortex_array::dtype::DType; use vortex_array::dtype::PType; use vortex_buffer::BufferMut; -use vortex_error::VortexError; use vortex_error::VortexResult; -use crate::DateTimePartsData; use crate::timestamp; pub struct TemporalParts { pub days: ArrayRef, @@ -25,8 +23,11 @@ pub struct TemporalParts { /// /// Splitting the components by granularity creates more small values, which enables better /// cascading compression. -pub fn split_temporal(array: TemporalArray) -> VortexResult { - let temporal_values = array.temporal_values().to_primitive(); +pub fn split_temporal(array: TemporalArray, ctx: &mut ExecutionCtx) -> VortexResult { + let temporal_values = array + .temporal_values() + .clone() + .execute::(ctx)?; // After this operation, timestamps will be a PrimitiveArray let timestamps = temporal_values @@ -36,7 +37,7 @@ pub fn split_temporal(array: TemporalArray) -> VortexResult { PType::I64, temporal_values.dtype().nullability(), ))? - .to_primitive(); + .execute::(ctx)?; let length = timestamps.len(); let mut days = BufferMut::with_capacity(length); @@ -57,33 +58,11 @@ pub fn split_temporal(array: TemporalArray) -> VortexResult { }) } -impl TryFrom for DateTimePartsData { - type Error = VortexError; - - fn try_from(array: TemporalArray) -> Result { - let ext_dtype = array.ext_dtype(); - let TemporalParts { - days, - seconds, - subseconds, - } = split_temporal(array)?; - DateTimePartsData::validate( - &DType::Extension(ext_dtype), - &days, - &seconds, - &subseconds, - days.len(), - )?; - Ok(DateTimePartsData {}) - } -} - #[cfg(test)] mod tests { use rstest::rstest; use vortex_array::IntoArray; use vortex_array::LEGACY_SESSION; - use vortex_array::ToCanonical; use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::TemporalArray; @@ -101,6 +80,7 @@ mod tests { #[case(Validity::AllInvalid)] #[case(Validity::from_iter([true, false, true]))] fn test_split_temporal(#[case] validity: Validity) { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let milliseconds = PrimitiveArray::new( buffer![ 86_400i64, // element with only day component @@ -116,26 +96,26 @@ mod tests { days, seconds, subseconds, - } = split_temporal(temporal_array).unwrap(); + } = split_temporal(temporal_array, &mut ctx).unwrap(); - let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let days_prim = days.execute::(&mut ctx).unwrap(); assert!( - days.to_primitive() + days_prim .validity() .vortex_expect("days validity should be derivable") .mask_eq(&validity, &mut ctx) .unwrap() ); + let seconds_prim = seconds.execute::(&mut ctx).unwrap(); assert!(matches!( - seconds - .to_primitive() + seconds_prim .validity() .vortex_expect("seconds validity should be derivable"), Validity::NonNullable )); + let subseconds_prim = subseconds.execute::(&mut ctx).unwrap(); assert!(matches!( - subseconds - .to_primitive() + subseconds_prim .validity() .vortex_expect("subseconds validity should be derivable"), Validity::NonNullable diff --git a/encodings/datetime-parts/src/compute/cast.rs b/encodings/datetime-parts/src/compute/cast.rs index 0e56d594dcb..ccc3cc0cd91 100644 --- a/encodings/datetime-parts/src/compute/cast.rs +++ b/encodings/datetime-parts/src/compute/cast.rs @@ -10,7 +10,7 @@ use vortex_array::scalar_fn::fns::cast::CastReduce; use vortex_error::VortexResult; use crate::DateTimeParts; -use crate::array::DateTimePartsArrayExt; +use crate::array::DateTimePartsArraySlotsExt; impl CastReduce for DateTimeParts { fn cast(array: ArrayView<'_, Self>, dtype: &DType) -> VortexResult> { if !array.dtype().eq_ignore_nullability(dtype) { @@ -35,7 +35,10 @@ impl CastReduce for DateTimeParts { mod tests { use rstest::rstest; use vortex_array::ArrayRef; + use vortex_array::Canonical; use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::TemporalArray; use vortex_array::builtins::ArrayBuiltins; @@ -49,19 +52,22 @@ mod tests { use crate::DateTimePartsArray; fn date_time_array(validity: Validity) -> ArrayRef { - DateTimeParts::try_from_temporal(TemporalArray::new_timestamp( - PrimitiveArray::new( - buffer![ - 86_400i64, // element with only day component - 86_400i64 + 1000, // element with day + second components - 86_400i64 + 1000 + 1, // element with day + second + sub-second components - ], - validity, - ) - .into_array(), - TimeUnit::Milliseconds, - Some("UTC".into()), - )) + DateTimeParts::try_from_temporal( + TemporalArray::new_timestamp( + PrimitiveArray::new( + buffer![ + 86_400i64, // element with only day component + 86_400i64 + 1000, // element with day + second components + 86_400i64 + 1000 + 1, // element with day + second + sub-second components + ], + validity, + ) + .into_array(), + TimeUnit::Milliseconds, + Some("UTC".into()), + ), + &mut LEGACY_SESSION.create_execution_ctx(), + ) .unwrap() .into_array() } @@ -89,17 +95,18 @@ mod tests { #[case(Validity::AllInvalid)] #[case(Validity::from_iter([true, false, true]))] fn test_bad_cast_fails(#[case] validity: Validity) { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = date_time_array(validity); - // Cast to incompatible type - force evaluation via to_canonical + // Cast to incompatible type - force evaluation via execute:: let result = array .cast(DType::Bool(Nullability::NonNullable)) - .and_then(|a| a.to_canonical().map(|c| c.into_array())); + .and_then(|a| a.execute::(&mut ctx).map(|c| c.into_array())); assert!(result.is_err(), "Expected error, got: {result:?}"); - // Cast nullable with nulls to non-nullable - force evaluation via to_canonical + // Cast nullable with nulls to non-nullable - force evaluation via execute:: let result = array .cast(array.dtype().with_nullability(Nullability::NonNullable)) - .and_then(|a| a.to_canonical().map(|c| c.into_array())); + .and_then(|a| a.execute::(&mut ctx).map(|c| c.into_array())); assert!(result.is_err(), "Expected error, got: {result:?}"); } @@ -114,7 +121,7 @@ mod tests { ].into_array(), TimeUnit::Milliseconds, Some("UTC".into()) - )).unwrap())] + ), &mut LEGACY_SESSION.create_execution_ctx()).unwrap())] #[case(DateTimeParts::try_from_temporal(TemporalArray::new_timestamp( PrimitiveArray::from_option_iter([ Some(0i64), @@ -125,12 +132,12 @@ mod tests { ]).into_array(), TimeUnit::Milliseconds, Some("UTC".into()) - )).unwrap())] + ), &mut LEGACY_SESSION.create_execution_ctx()).unwrap())] #[case(DateTimeParts::try_from_temporal(TemporalArray::new_timestamp( buffer![86_400_000_000_000i64].into_array(), // 1 day in ns TimeUnit::Nanoseconds, Some("UTC".into()) - )).unwrap())] + ), &mut LEGACY_SESSION.create_execution_ctx()).unwrap())] fn test_cast_datetime_parts_conformance(#[case] array: DateTimePartsArray) { use vortex_array::compute::conformance::cast::test_cast_conformance; test_cast_conformance(&array.into_array()); diff --git a/encodings/datetime-parts/src/compute/compare.rs b/encodings/datetime-parts/src/compute/compare.rs index bd50a7a5d0f..faecc8452b3 100644 --- a/encodings/datetime-parts/src/compute/compare.rs +++ b/encodings/datetime-parts/src/compute/compare.rs @@ -17,7 +17,7 @@ use vortex_array::scalar_fn::fns::operators::Operator; use vortex_error::VortexResult; use crate::array::DateTimeParts; -use crate::array::DateTimePartsArrayExt; +use crate::array::DateTimePartsArraySlotsExt; use crate::timestamp; impl CompareKernel for DateTimeParts { @@ -25,7 +25,7 @@ impl CompareKernel for DateTimeParts { lhs: ArrayView<'_, Self>, rhs: &ArrayRef, operator: CompareOperator, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult> { let Some(rhs_const) = rhs.as_constant() else { return Ok(None); @@ -51,17 +51,17 @@ impl CompareKernel for DateTimeParts { let ts_parts = timestamp::split(timestamp, options.unit)?; match operator { - CompareOperator::Eq => compare_eq(lhs, &ts_parts, nullability), - CompareOperator::NotEq => compare_ne(lhs, &ts_parts, nullability), + CompareOperator::Eq => compare_eq(lhs, &ts_parts, nullability, ctx), + CompareOperator::NotEq => compare_ne(lhs, &ts_parts, nullability, ctx), // lt and lte have identical behavior, as we optimize // for the case that all days on the lhs are smaller. // If that special case is not hit, we return `Ok(None)` to // signal that the comparison wasn't handled within dtp. - CompareOperator::Lt => compare_lt(lhs, &ts_parts, nullability), - CompareOperator::Lte => compare_lt(lhs, &ts_parts, nullability), + CompareOperator::Lt => compare_lt(lhs, &ts_parts, nullability, ctx), + CompareOperator::Lte => compare_lt(lhs, &ts_parts, nullability, ctx), // (Like for lt, lte) - CompareOperator::Gt => compare_gt(lhs, &ts_parts, nullability), - CompareOperator::Gte => compare_gt(lhs, &ts_parts, nullability), + CompareOperator::Gt => compare_gt(lhs, &ts_parts, nullability, ctx), + CompareOperator::Gte => compare_gt(lhs, &ts_parts, nullability, ctx), } } } @@ -70,9 +70,10 @@ fn compare_eq( lhs: ArrayView, ts_parts: ×tamp::TimestampParts, nullability: Nullability, + ctx: &mut ExecutionCtx, ) -> VortexResult> { let mut comparison = compare_dtp(lhs.days(), ts_parts.days, CompareOperator::Eq, nullability)?; - if comparison.statistics().compute_max::() == Some(false) { + if comparison.statistics().compute_max::(ctx) == Some(false) { // All values are different. return Ok(Some(comparison)); } @@ -85,7 +86,7 @@ fn compare_eq( )? .binary(comparison, Operator::And)?; - if comparison.statistics().compute_max::() == Some(false) { + if comparison.statistics().compute_max::(ctx) == Some(false) { // All values are different. return Ok(Some(comparison)); } @@ -105,6 +106,7 @@ fn compare_ne( lhs: ArrayView, ts_parts: ×tamp::TimestampParts, nullability: Nullability, + ctx: &mut ExecutionCtx, ) -> VortexResult> { let mut comparison = compare_dtp( lhs.days(), @@ -112,7 +114,7 @@ fn compare_ne( CompareOperator::NotEq, nullability, )?; - if comparison.statistics().compute_min::() == Some(true) { + if comparison.statistics().compute_min::(ctx) == Some(true) { // All values are different. return Ok(Some(comparison)); } @@ -125,7 +127,7 @@ fn compare_ne( )? .binary(comparison, Operator::Or)?; - if comparison.statistics().compute_min::() == Some(true) { + if comparison.statistics().compute_min::(ctx) == Some(true) { // All values are different. return Ok(Some(comparison)); } @@ -145,9 +147,10 @@ fn compare_lt( lhs: ArrayView, ts_parts: ×tamp::TimestampParts, nullability: Nullability, + ctx: &mut ExecutionCtx, ) -> VortexResult> { let days_lt = compare_dtp(lhs.days(), ts_parts.days, CompareOperator::Lt, nullability)?; - if days_lt.statistics().compute_min::() == Some(true) { + if days_lt.statistics().compute_min::(ctx) == Some(true) { // All values on the lhs are smaller. return Ok(Some(days_lt)); } @@ -159,9 +162,10 @@ fn compare_gt( lhs: ArrayView, ts_parts: ×tamp::TimestampParts, nullability: Nullability, + ctx: &mut ExecutionCtx, ) -> VortexResult> { let days_gt = compare_dtp(lhs.days(), ts_parts.days, CompareOperator::Gt, nullability)?; - if days_gt.statistics().compute_min::() == Some(true) { + if days_gt.statistics().compute_min::(ctx) == Some(true) { // All values on the lhs are larger. return Ok(Some(days_gt)); } @@ -198,6 +202,8 @@ fn compare_dtp( #[cfg(test)] mod test { use rstest::rstest; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::TemporalArray; use vortex_array::dtype::IntegerPType; @@ -213,11 +219,14 @@ mod test { value: T, validity: Validity, ) -> DateTimePartsArray { - DateTimeParts::try_from_temporal(TemporalArray::new_timestamp( - PrimitiveArray::new(buffer![value], validity).into_array(), - TimeUnit::Seconds, - Some("UTC".into()), - )) + DateTimeParts::try_from_temporal( + TemporalArray::new_timestamp( + PrimitiveArray::new(buffer![value], validity).into_array(), + TimeUnit::Seconds, + Some("UTC".into()), + ), + &mut LEGACY_SESSION.create_execution_ctx(), + ) .expect("Failed to construct DateTimePartsArray from TemporalArray") } diff --git a/encodings/datetime-parts/src/compute/filter.rs b/encodings/datetime-parts/src/compute/filter.rs index d717d3d3500..149574f53d4 100644 --- a/encodings/datetime-parts/src/compute/filter.rs +++ b/encodings/datetime-parts/src/compute/filter.rs @@ -9,7 +9,7 @@ use vortex_error::VortexResult; use vortex_mask::Mask; use crate::DateTimeParts; -use crate::array::DateTimePartsArrayExt; +use crate::array::DateTimePartsArraySlotsExt; impl FilterReduce for DateTimeParts { fn filter(array: ArrayView<'_, Self>, mask: &Mask) -> VortexResult> { Ok(Some( @@ -27,6 +27,8 @@ impl FilterReduce for DateTimeParts { #[cfg(test)] mod test { use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::TemporalArray; use vortex_array::compute::conformance::filter::test_filter_conformance; @@ -37,6 +39,7 @@ mod test { #[test] fn test_filter_datetime_parts() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Create temporal arrays and convert to DateTimePartsArray let timestamps = buffer![ 0i64, @@ -50,7 +53,7 @@ mod test { let temporal = TemporalArray::new_timestamp(timestamps, TimeUnit::Milliseconds, Some("UTC".into())); - let array = DateTimeParts::try_from_temporal(temporal).unwrap(); + let array = DateTimeParts::try_from_temporal(temporal, &mut ctx).unwrap(); test_filter_conformance(&array.into_array()); // Test with nullable values @@ -66,7 +69,7 @@ mod test { let temporal = TemporalArray::new_timestamp(timestamps, TimeUnit::Milliseconds, Some("UTC".into())); - let array = DateTimeParts::try_from_temporal(temporal).unwrap(); + let array = DateTimeParts::try_from_temporal(temporal, &mut ctx).unwrap(); test_filter_conformance(&array.into_array()); } } diff --git a/encodings/datetime-parts/src/compute/is_constant.rs b/encodings/datetime-parts/src/compute/is_constant.rs index 94e6806e785..cd4ecf2752a 100644 --- a/encodings/datetime-parts/src/compute/is_constant.rs +++ b/encodings/datetime-parts/src/compute/is_constant.rs @@ -11,7 +11,7 @@ use vortex_array::scalar::Scalar; use vortex_error::VortexResult; use crate::DateTimeParts; -use crate::array::DateTimePartsArrayExt; +use crate::array::DateTimePartsArraySlotsExt; /// DateTimeParts-specific is_constant kernel. /// @@ -37,6 +37,6 @@ impl DynAggregateKernel for DateTimePartsIsConstantKernel { let result = is_constant(array.days(), ctx)? && is_constant(array.seconds(), ctx)? && is_constant(array.subseconds(), ctx)?; - Ok(Some(IsConstant::make_partial(batch, result)?)) + Ok(Some(IsConstant::make_partial(batch, result, ctx)?)) } } diff --git a/encodings/datetime-parts/src/compute/mask.rs b/encodings/datetime-parts/src/compute/mask.rs index de737105ffe..bd81605c446 100644 --- a/encodings/datetime-parts/src/compute/mask.rs +++ b/encodings/datetime-parts/src/compute/mask.rs @@ -9,7 +9,7 @@ use vortex_array::scalar_fn::fns::mask::MaskReduce; use vortex_error::VortexResult; use crate::DateTimeParts; -use crate::array::DateTimePartsArrayExt; +use crate::array::DateTimePartsArraySlotsExt; impl MaskReduce for DateTimeParts { fn mask(array: ArrayView<'_, Self>, mask: &ArrayRef) -> VortexResult> { diff --git a/encodings/datetime-parts/src/compute/mod.rs b/encodings/datetime-parts/src/compute/mod.rs index 53717a52b0c..5d25757c20d 100644 --- a/encodings/datetime-parts/src/compute/mod.rs +++ b/encodings/datetime-parts/src/compute/mod.rs @@ -15,6 +15,8 @@ mod take; mod tests { use rstest::rstest; use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::TemporalArray; use vortex_array::compute::conformance::consistency::test_array_consistency; @@ -25,7 +27,8 @@ mod tests { use crate::DateTimePartsArray; fn dtp_from_temporal(temporal: TemporalArray) -> DateTimePartsArray { - DateTimeParts::try_from_temporal(temporal).unwrap() + DateTimeParts::try_from_temporal(temporal, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() } #[rstest] diff --git a/encodings/datetime-parts/src/compute/rules.rs b/encodings/datetime-parts/src/compute/rules.rs index c18d959cd7e..a8b1b2eca83 100644 --- a/encodings/datetime-parts/src/compute/rules.rs +++ b/encodings/datetime-parts/src/compute/rules.rs @@ -10,8 +10,8 @@ use vortex_array::arrays::Filter; use vortex_array::arrays::ScalarFnArray; use vortex_array::arrays::filter::FilterReduceAdaptor; use vortex_array::arrays::scalar_fn::AnyScalarFn; +use vortex_array::arrays::scalar_fn::ScalarFn; use vortex_array::arrays::scalar_fn::ScalarFnArrayExt; -use vortex_array::arrays::scalar_fn::ScalarFnVTable; use vortex_array::arrays::slice::SliceReduceAdaptor; use vortex_array::builtins::ArrayBuiltins; use vortex_array::dtype::DType; @@ -27,7 +27,7 @@ use vortex_error::VortexExpect; use vortex_error::VortexResult; use crate::DateTimeParts; -use crate::array::DateTimePartsArrayExt; +use crate::array::DateTimePartsArraySlotsExt; use crate::timestamp; pub(crate) const PARENT_RULES: ParentRuleSet = ParentRuleSet::new(&[ ParentRuleSet::lift(&DTPFilterPushDownRule), @@ -95,7 +95,7 @@ impl ArrayParentReduceRule for DTPComparisonPushDownRule { fn reduce_parent( &self, child: ArrayView<'_, DateTimeParts>, - parent: ArrayView<'_, ScalarFnVTable>, + parent: ArrayView<'_, ScalarFn>, child_idx: usize, ) -> VortexResult> { // Only handle comparison operations (Binary comparisons or Between) @@ -179,6 +179,8 @@ fn is_constant_zero(array: &ArrayRef) -> bool { #[cfg(test)] mod tests { + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::TemporalArray; use vortex_array::arrays::scalar_fn::ScalarFnFactoryExt; @@ -217,7 +219,7 @@ mod tests { time_unit, None, ); - DateTimeParts::try_from_temporal(temporal) + DateTimeParts::try_from_temporal(temporal, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("TemporalArray must produce valid DateTimeParts") } @@ -350,7 +352,9 @@ mod tests { TimeUnit::Seconds, None, ); - let dtp = DateTimeParts::try_from_temporal(temporal).unwrap(); + let dtp = + DateTimeParts::try_from_temporal(temporal, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); let len = dtp.len(); // Compare against midnight constant diff --git a/encodings/datetime-parts/src/compute/slice.rs b/encodings/datetime-parts/src/compute/slice.rs index 46819a41562..3cfd2fd6b36 100644 --- a/encodings/datetime-parts/src/compute/slice.rs +++ b/encodings/datetime-parts/src/compute/slice.rs @@ -10,7 +10,7 @@ use vortex_array::arrays::slice::SliceReduce; use vortex_error::VortexResult; use crate::DateTimeParts; -use crate::array::DateTimePartsArrayExt; +use crate::array::DateTimePartsArraySlotsExt; impl SliceReduce for DateTimeParts { fn slice(array: ArrayView<'_, Self>, range: Range) -> VortexResult> { diff --git a/encodings/datetime-parts/src/compute/take.rs b/encodings/datetime-parts/src/compute/take.rs index cd15c466b2a..fcdc00c1979 100644 --- a/encodings/datetime-parts/src/compute/take.rs +++ b/encodings/datetime-parts/src/compute/take.rs @@ -5,7 +5,7 @@ use vortex_array::ArrayRef; use vortex_array::ArrayView; use vortex_array::ExecutionCtx; use vortex_array::IntoArray; -use vortex_array::ToCanonical; +use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::dict::TakeExecute; use vortex_array::builtins::ArrayBuiltins; use vortex_array::dtype::Nullability; @@ -16,13 +16,14 @@ use vortex_error::VortexResult; use vortex_error::vortex_panic; use crate::DateTimeParts; -use crate::array::DateTimePartsArrayExt; +use crate::array::DateTimePartsArraySlotsExt; fn take_datetime_parts( array: ArrayView, indices: &ArrayRef, + ctx: &mut ExecutionCtx, ) -> VortexResult { // we go ahead and canonicalize here to avoid worst-case canonicalizing 3 separate times - let indices = indices.to_primitive(); + let indices = indices.clone().execute::(ctx)?; let taken_days = array.days().take(indices.clone().into_array())?; let taken_seconds = array.seconds().take(indices.clone().into_array())?; @@ -85,9 +86,9 @@ impl TakeExecute for DateTimeParts { fn take( array: ArrayView<'_, Self>, indices: &ArrayRef, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult> { - take_datetime_parts(array, indices).map(Some) + take_datetime_parts(array, indices, ctx).map(Some) } } @@ -95,6 +96,8 @@ impl TakeExecute for DateTimeParts { mod tests { use rstest::rstest; use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::TemporalArray; use vortex_array::compute::conformance::take::test_take_conformance; @@ -115,7 +118,7 @@ mod tests { ].into_array(), TimeUnit::Milliseconds, Some("UTC".into()) - )).unwrap())] + ), &mut LEGACY_SESSION.create_execution_ctx()).unwrap())] #[case(DateTimeParts::try_from_temporal(TemporalArray::new_timestamp( PrimitiveArray::from_option_iter([ Some(0i64), @@ -126,12 +129,12 @@ mod tests { ]).into_array(), TimeUnit::Milliseconds, Some("UTC".into()) - )).unwrap())] + ), &mut LEGACY_SESSION.create_execution_ctx()).unwrap())] #[case(DateTimeParts::try_from_temporal(TemporalArray::new_timestamp( buffer![86_400_000i64].into_array(), TimeUnit::Milliseconds, Some("UTC".into()) - )).unwrap())] + ), &mut LEGACY_SESSION.create_execution_ctx()).unwrap())] fn test_take_datetime_parts_conformance(#[case] array: DateTimePartsArray) { test_take_conformance(&array.into_array()); } diff --git a/encodings/datetime-parts/src/lib.rs b/encodings/datetime-parts/src/lib.rs index 1babf4abaab..e061d69bccd 100644 --- a/encodings/datetime-parts/src/lib.rs +++ b/encodings/datetime-parts/src/lib.rs @@ -11,6 +11,7 @@ mod compute; mod ops; mod timestamp; +use vortex_array::ArrayVTable; use vortex_array::aggregate_fn::AggregateFnVTable; use vortex_array::aggregate_fn::fns::is_constant::IsConstant; use vortex_array::aggregate_fn::session::AggregateFnSessionExt; @@ -22,7 +23,7 @@ pub fn initialize(session: &VortexSession) { session.arrays().register(DateTimeParts); session.aggregate_fns().register_aggregate_kernel( - DateTimeParts::ID, + DateTimeParts.id(), Some(IsConstant.id()), &compute::is_constant::DateTimePartsIsConstantKernel, ); diff --git a/encodings/datetime-parts/src/ops.rs b/encodings/datetime-parts/src/ops.rs index 2c224b078d5..57a1f60173f 100644 --- a/encodings/datetime-parts/src/ops.rs +++ b/encodings/datetime-parts/src/ops.rs @@ -12,7 +12,7 @@ use vortex_error::VortexResult; use vortex_error::vortex_panic; use crate::DateTimeParts; -use crate::array::DateTimePartsArrayExt; +use crate::array::DateTimePartsArraySlotsExt; use crate::timestamp; use crate::timestamp::TimestampParts; @@ -20,7 +20,7 @@ impl OperationsVTable for DateTimeParts { fn scalar_at( array: ArrayView<'_, DateTimeParts>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { let DType::Extension(ext) = array.dtype().clone() else { vortex_panic!( @@ -33,25 +33,25 @@ impl OperationsVTable for DateTimeParts { vortex_panic!(Compute: "must decode TemporalMetadata from extension metadata"); }; - if !array.as_ref().is_valid(index)? { + if !array.as_ref().is_valid(index, ctx)? { return Ok(Scalar::null(DType::Extension(ext))); } let days: i64 = array .days() - .scalar_at(index)? + .execute_scalar(index, ctx)? .as_primitive() .as_::() .vortex_expect("days fits in i64"); let seconds: i64 = array .seconds() - .scalar_at(index)? + .execute_scalar(index, ctx)? .as_primitive() .as_::() .vortex_expect("seconds fits in i64"); let subseconds: i64 = array .subseconds() - .scalar_at(index)? + .execute_scalar(index, ctx)? .as_primitive() .as_::() .vortex_expect("subseconds fits in i64"); diff --git a/encodings/decimal-byte-parts/public-api.lock b/encodings/decimal-byte-parts/public-api.lock index 397885e1b45..d2d62792fbd 100644 --- a/encodings/decimal-byte-parts/public-api.lock +++ b/encodings/decimal-byte-parts/public-api.lock @@ -4,9 +4,7 @@ pub struct vortex_decimal_byte_parts::DecimalByteParts impl vortex_decimal_byte_parts::DecimalByteParts -pub const vortex_decimal_byte_parts::DecimalByteParts::ID: vortex_array::array::ArrayId - -pub fn vortex_decimal_byte_parts::DecimalByteParts::try_new(msp: vortex_array::array::erased::ArrayRef, decimal_dtype: vortex_array::dtype::decimal::DecimalDType) -> vortex_error::VortexResult +pub fn vortex_decimal_byte_parts::DecimalByteParts::try_new(vortex_array::array::erased::ArrayRef, vortex_array::dtype::decimal::DecimalDType) -> vortex_error::VortexResult impl core::clone::Clone for vortex_decimal_byte_parts::DecimalByteParts @@ -14,7 +12,7 @@ pub fn vortex_decimal_byte_parts::DecimalByteParts::clone(&self) -> vortex_decim impl core::fmt::Debug for vortex_decimal_byte_parts::DecimalByteParts -pub fn vortex_decimal_byte_parts::DecimalByteParts::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_decimal_byte_parts::DecimalByteParts::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::array::vtable::VTable for vortex_decimal_byte_parts::DecimalByteParts @@ -24,65 +22,65 @@ pub type vortex_decimal_byte_parts::DecimalByteParts::OperationsVTable = vortex_ pub type vortex_decimal_byte_parts::DecimalByteParts::ValidityVTable = vortex_array::array::vtable::validity::ValidityVTableFromChild -pub fn vortex_decimal_byte_parts::DecimalByteParts::buffer(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_decimal_byte_parts::DecimalByteParts::buffer(vortex_array::array::view::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_decimal_byte_parts::DecimalByteParts::buffer_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_decimal_byte_parts::DecimalByteParts::buffer_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_decimal_byte_parts::DecimalByteParts::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_decimal_byte_parts::DecimalByteParts::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_decimal_byte_parts::DecimalByteParts::execute(array: vortex_array::array::typed::Array, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_decimal_byte_parts::DecimalByteParts::execute(vortex_array::array::typed::Array, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_decimal_byte_parts::DecimalByteParts::execute_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_decimal_byte_parts::DecimalByteParts::execute_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_decimal_byte_parts::DecimalByteParts::id(&self) -> vortex_array::array::ArrayId -pub fn vortex_decimal_byte_parts::DecimalByteParts::nbuffers(_array: vortex_array::array::view::ArrayView<'_, Self>) -> usize +pub fn vortex_decimal_byte_parts::DecimalByteParts::nbuffers(vortex_array::array::view::ArrayView<'_, Self>) -> usize -pub fn vortex_decimal_byte_parts::DecimalByteParts::reduce_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_decimal_byte_parts::DecimalByteParts::reduce_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_decimal_byte_parts::DecimalByteParts::serialize(array: vortex_array::array::view::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_decimal_byte_parts::DecimalByteParts::serialize(vortex_array::array::view::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_decimal_byte_parts::DecimalByteParts::slot_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_decimal_byte_parts::DecimalByteParts::slot_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_decimal_byte_parts::DecimalByteParts::validate(&self, _data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_decimal_byte_parts::DecimalByteParts::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::array::vtable::operations::OperationsVTable for vortex_decimal_byte_parts::DecimalByteParts -pub fn vortex_decimal_byte_parts::DecimalByteParts::scalar_at(array: vortex_array::array::view::ArrayView<'_, vortex_decimal_byte_parts::DecimalByteParts>, index: usize, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_decimal_byte_parts::DecimalByteParts::scalar_at(vortex_array::array::view::ArrayView<'_, vortex_decimal_byte_parts::DecimalByteParts>, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::array::vtable::validity::ValidityChild for vortex_decimal_byte_parts::DecimalByteParts -pub fn vortex_decimal_byte_parts::DecimalByteParts::validity_child(array: vortex_array::array::view::ArrayView<'_, vortex_decimal_byte_parts::DecimalByteParts>) -> vortex_array::array::erased::ArrayRef +pub fn vortex_decimal_byte_parts::DecimalByteParts::validity_child(vortex_array::array::view::ArrayView<'_, vortex_decimal_byte_parts::DecimalByteParts>) -> vortex_array::array::erased::ArrayRef impl vortex_array::arrays::dict::take::TakeExecute for vortex_decimal_byte_parts::DecimalByteParts -pub fn vortex_decimal_byte_parts::DecimalByteParts::take(array: vortex_array::array::view::ArrayView<'_, Self>, indices: &vortex_array::array::erased::ArrayRef, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_decimal_byte_parts::DecimalByteParts::take(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::kernel::FilterReduce for vortex_decimal_byte_parts::DecimalByteParts -pub fn vortex_decimal_byte_parts::DecimalByteParts::filter(array: vortex_array::array::view::ArrayView<'_, Self>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_decimal_byte_parts::DecimalByteParts::filter(vortex_array::array::view::ArrayView<'_, Self>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_decimal_byte_parts::DecimalByteParts -pub fn vortex_decimal_byte_parts::DecimalByteParts::slice(array: vortex_array::array::view::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_decimal_byte_parts::DecimalByteParts::slice(vortex_array::array::view::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::binary::compare::CompareKernel for vortex_decimal_byte_parts::DecimalByteParts -pub fn vortex_decimal_byte_parts::DecimalByteParts::compare(lhs: vortex_array::array::view::ArrayView<'_, Self>, rhs: &vortex_array::array::erased::ArrayRef, operator: vortex_array::scalar_fn::fns::operators::CompareOperator, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_decimal_byte_parts::DecimalByteParts::compare(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, vortex_array::scalar_fn::fns::operators::CompareOperator, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::kernel::CastReduce for vortex_decimal_byte_parts::DecimalByteParts -pub fn vortex_decimal_byte_parts::DecimalByteParts::cast(array: vortex_array::array::view::ArrayView<'_, Self>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_decimal_byte_parts::DecimalByteParts::cast(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::kernel::MaskReduce for vortex_decimal_byte_parts::DecimalByteParts -pub fn vortex_decimal_byte_parts::DecimalByteParts::mask(array: vortex_array::array::view::ArrayView<'_, Self>, mask: &vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_decimal_byte_parts::DecimalByteParts::mask(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult> pub struct vortex_decimal_byte_parts::DecimalBytePartsData impl vortex_decimal_byte_parts::DecimalBytePartsData -pub fn vortex_decimal_byte_parts::DecimalBytePartsData::validate(msp: &vortex_array::array::erased::ArrayRef, decimal_dtype: vortex_array::dtype::decimal::DecimalDType, dtype: &vortex_array::dtype::DType, len: usize) -> vortex_error::VortexResult<()> +pub fn vortex_decimal_byte_parts::DecimalBytePartsData::validate(&vortex_array::array::erased::ArrayRef, vortex_array::dtype::decimal::DecimalDType, &vortex_array::dtype::DType, usize) -> vortex_error::VortexResult<()> impl core::clone::Clone for vortex_decimal_byte_parts::DecimalBytePartsData @@ -90,19 +88,19 @@ pub fn vortex_decimal_byte_parts::DecimalBytePartsData::clone(&self) -> vortex_d impl core::fmt::Debug for vortex_decimal_byte_parts::DecimalBytePartsData -pub fn vortex_decimal_byte_parts::DecimalBytePartsData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_decimal_byte_parts::DecimalBytePartsData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_decimal_byte_parts::DecimalBytePartsData -pub fn vortex_decimal_byte_parts::DecimalBytePartsData::fmt(&self, _f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_decimal_byte_parts::DecimalBytePartsData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::hash::ArrayEq for vortex_decimal_byte_parts::DecimalBytePartsData -pub fn vortex_decimal_byte_parts::DecimalBytePartsData::array_eq(&self, _other: &Self, _precision: vortex_array::hash::Precision) -> bool +pub fn vortex_decimal_byte_parts::DecimalBytePartsData::array_eq(&self, &Self, vortex_array::hash::Precision) -> bool impl vortex_array::hash::ArrayHash for vortex_decimal_byte_parts::DecimalBytePartsData -pub fn vortex_decimal_byte_parts::DecimalBytePartsData::array_hash(&self, _state: &mut H, _precision: vortex_array::hash::Precision) +pub fn vortex_decimal_byte_parts::DecimalBytePartsData::array_hash(&self, &mut H, vortex_array::hash::Precision) pub struct vortex_decimal_byte_parts::DecimalBytePartsDataParts @@ -112,7 +110,7 @@ pub struct vortex_decimal_byte_parts::DecimalBytesPartsMetadata impl vortex_decimal_byte_parts::DecimalBytesPartsMetadata -pub fn vortex_decimal_byte_parts::DecimalBytesPartsMetadata::set_zeroth_child_ptype(&mut self, value: vortex_array::dtype::ptype::PType) +pub fn vortex_decimal_byte_parts::DecimalBytesPartsMetadata::set_zeroth_child_ptype(&mut self, vortex_array::dtype::ptype::PType) pub fn vortex_decimal_byte_parts::DecimalBytesPartsMetadata::zeroth_child_ptype(&self) -> vortex_array::dtype::ptype::PType @@ -126,7 +124,7 @@ pub fn vortex_decimal_byte_parts::DecimalBytesPartsMetadata::default() -> Self impl core::fmt::Debug for vortex_decimal_byte_parts::DecimalBytesPartsMetadata -pub fn vortex_decimal_byte_parts::DecimalBytesPartsMetadata::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_decimal_byte_parts::DecimalBytesPartsMetadata::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl prost::message::Message for vortex_decimal_byte_parts::DecimalBytesPartsMetadata @@ -142,6 +140,6 @@ impl &vortex_array::array::erased::ArrayRef -pub fn vortex_decimal_byte_parts::initialize(session: &vortex_session::VortexSession) +pub fn vortex_decimal_byte_parts::initialize(&vortex_session::VortexSession) pub type vortex_decimal_byte_parts::DecimalBytePartsArray = vortex_array::array::typed::Array diff --git a/encodings/decimal-byte-parts/src/decimal_byte_parts/compute/cast.rs b/encodings/decimal-byte-parts/src/decimal_byte_parts/compute/cast.rs index e554bbfae71..882d07bbaef 100644 --- a/encodings/decimal-byte-parts/src/decimal_byte_parts/compute/cast.rs +++ b/encodings/decimal-byte-parts/src/decimal_byte_parts/compute/cast.rs @@ -7,47 +7,42 @@ use vortex_array::IntoArray; use vortex_array::builtins::ArrayBuiltins; use vortex_array::dtype::DType; use vortex_array::scalar_fn::fns::cast::CastReduce; -use vortex_error::VortexExpect; use vortex_error::VortexResult; use crate::DecimalByteParts; use crate::decimal_byte_parts::DecimalBytePartsArrayExt; + impl CastReduce for DecimalByteParts { fn cast(array: ArrayView<'_, Self>, dtype: &DType) -> VortexResult> { + // Check if this is just a nullability change + if !dtype.eq_ignore_nullability(array.dtype()) { + return Ok(None); + } // DecimalBytePartsArray can only have Decimal dtype, so we only handle decimal-to-decimal casts let DType::Decimal(target_decimal, target_nullability) = dtype else { // Cannot cast decimal to non-decimal types - delegate to canonical form return Ok(None); }; - // Check if this is just a nullability change - if array - .dtype() - .as_decimal_opt() - .vortex_expect("must be a decimal dtype") - == target_decimal - && array.dtype().nullability() != *target_nullability - { - // Cast the msp array to handle nullability change - let new_msp = array - .msp() - .cast(array.msp().dtype().with_nullability(*target_nullability))?; - - return Ok(Some( - DecimalByteParts::try_new(new_msp, *target_decimal)?.into_array(), - )); - } + // Cast the msp array to handle nullability change + let new_msp = array + .msp() + .cast(array.msp().dtype().with_nullability(*target_nullability))?; - // For precision/scale changes, decode to canonical and let DecimalArray handle it - Ok(None) + Ok(Some( + DecimalByteParts::try_new(new_msp, *target_decimal)?.into_array(), + )) } } #[cfg(test)] mod tests { use rstest::rstest; + use vortex_array::Canonical; use vortex_array::IntoArray; - use vortex_array::ToCanonical; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; + use vortex_array::arrays::DecimalArray; use vortex_array::arrays::PrimitiveArray; use vortex_array::builtins::ArrayBuiltins; use vortex_array::compute::conformance::cast::test_cast_conformance; @@ -61,6 +56,7 @@ mod tests { #[test] fn test_cast_decimal_byte_parts_nullability() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let decimal_dtype = DecimalDType::new(10, 2); let array = DecimalByteParts::try_new(buffer![100i32, 200, 300, 400].into_array(), decimal_dtype) @@ -77,12 +73,13 @@ mod tests { ); // Verify the values are preserved - let decoded = casted.to_decimal(); + let decoded = casted.execute::(&mut ctx).unwrap(); assert_eq!(decoded.len(), 4); } #[test] fn test_cast_decimal_byte_parts_nullable_to_non_nullable() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let decimal_dtype = DecimalDType::new(10, 2); let array = DecimalByteParts::try_new( PrimitiveArray::from_option_iter([Some(100i32), None, Some(300)]).into_array(), @@ -90,11 +87,11 @@ mod tests { ) .unwrap(); - // Cast to non-nullable should fail due to nulls - force evaluation via to_canonical + // Cast to non-nullable should fail due to nulls - force evaluation via execute:: let result = array .into_array() .cast(DType::Decimal(decimal_dtype, Nullability::NonNullable)) - .and_then(|a| a.to_canonical().map(|c| c.into_array())); + .and_then(|a| a.execute::(&mut ctx).map(|c| c.into_array())); assert!(result.is_err()); } diff --git a/encodings/decimal-byte-parts/src/decimal_byte_parts/compute/compare.rs b/encodings/decimal-byte-parts/src/decimal_byte_parts/compute/compare.rs index 43f06b6e695..e6cc8d4d72d 100644 --- a/encodings/decimal-byte-parts/src/decimal_byte_parts/compute/compare.rs +++ b/encodings/decimal-byte-parts/src/decimal_byte_parts/compute/compare.rs @@ -33,7 +33,7 @@ impl CompareKernel for DecimalByteParts { lhs: ArrayView<'_, Self>, rhs: &ArrayRef, operator: CompareOperator, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult> { let Some(rhs_const) = rhs.as_constant() else { return Ok(None); @@ -65,7 +65,7 @@ impl CompareKernel for DecimalByteParts { // (depending on the `sign`) than all values in MSP. // If the LHS or the RHS contain nulls, then we must fallback to the canonicalized // implementation which does null-checking instead. - if lhs.array().all_valid()? && rhs.all_valid()? { + if lhs.array().all_valid(ctx)? && rhs.all_valid(ctx)? { Ok(Some( ConstantArray::new( unconvertible_value(sign, operator, nullability), diff --git a/encodings/decimal-byte-parts/src/decimal_byte_parts/compute/is_constant.rs b/encodings/decimal-byte-parts/src/decimal_byte_parts/compute/is_constant.rs index 4d271eb80b3..f4da528a9b3 100644 --- a/encodings/decimal-byte-parts/src/decimal_byte_parts/compute/is_constant.rs +++ b/encodings/decimal-byte-parts/src/decimal_byte_parts/compute/is_constant.rs @@ -35,6 +35,6 @@ impl DynAggregateKernel for DecimalBytePartsIsConstantKernel { }; let result = is_constant(array.msp(), ctx)?; - Ok(Some(IsConstant::make_partial(batch, result)?)) + Ok(Some(IsConstant::make_partial(batch, result, ctx)?)) } } diff --git a/encodings/decimal-byte-parts/src/decimal_byte_parts/mod.rs b/encodings/decimal-byte-parts/src/decimal_byte_parts/mod.rs index ea90bf575e0..80a7772d986 100644 --- a/encodings/decimal-byte-parts/src/decimal_byte_parts/mod.rs +++ b/encodings/decimal-byte-parts/src/decimal_byte_parts/mod.rs @@ -43,6 +43,7 @@ use vortex_error::vortex_bail; use vortex_error::vortex_ensure; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::decimal_byte_parts::compute::kernel::PARENT_KERNELS; use crate::decimal_byte_parts::rules::PARENT_RULES; @@ -75,7 +76,8 @@ impl VTable for DecimalByteParts { type ValidityVTable = ValidityVTableFromChild; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.decimal_byte_parts"); + *ID } fn validate( @@ -253,8 +255,6 @@ impl DecimalBytePartsData { pub struct DecimalByteParts; impl DecimalByteParts { - pub const ID: ArrayId = ArrayId::new_ref("vortex.decimal_byte_parts"); - /// Construct a new [`DecimalBytePartsArray`] from an MSP array and decimal dtype. pub fn try_new( msp: ArrayRef, @@ -303,10 +303,10 @@ impl OperationsVTable for DecimalByteParts { fn scalar_at( array: ArrayView<'_, DecimalByteParts>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { // TODO(joe): support parts len != 1 - let scalar = array.msp().scalar_at(index)?; + let scalar = array.msp().execute_scalar(index, ctx)?; // Note. values in msp, can only be signed integers upto size i64. let primitive_scalar = scalar.as_primitive(); @@ -329,6 +329,8 @@ impl ValidityChild for DecimalByteParts { #[cfg(test)] mod tests { use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; use vortex_array::arrays::PrimitiveArray; use vortex_array::dtype::DType; @@ -357,18 +359,27 @@ mod tests { .unwrap() .into_array(); - assert_eq!(Scalar::null(dtype.clone()), array.scalar_at(0).unwrap()); + assert_eq!( + Scalar::null(dtype.clone()), + array + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); assert_eq!( Scalar::try_new( dtype.clone(), Some(ScalarValue::Decimal(DecimalValue::I64(200))) ) .unwrap(), - array.scalar_at(1).unwrap() + array + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() ); assert_eq!( Scalar::try_new(dtype, Some(ScalarValue::Decimal(DecimalValue::I64(400)))).unwrap(), - array.scalar_at(2).unwrap() + array + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() ); } } diff --git a/encodings/decimal-byte-parts/src/lib.rs b/encodings/decimal-byte-parts/src/lib.rs index 58c2d1dc69a..86566027afa 100644 --- a/encodings/decimal-byte-parts/src/lib.rs +++ b/encodings/decimal-byte-parts/src/lib.rs @@ -13,6 +13,7 @@ use decimal_byte_parts::compute::is_constant::DecimalBytePartsIsConstantKernel; /// an i128 decimal could be converted into a [i64, u64] with further narrowing applied to either /// value. pub use decimal_byte_parts::*; +use vortex_array::ArrayVTable; use vortex_array::aggregate_fn::AggregateFnVTable; use vortex_array::aggregate_fn::fns::is_constant::IsConstant; use vortex_array::aggregate_fn::session::AggregateFnSessionExt; @@ -24,7 +25,7 @@ pub fn initialize(session: &VortexSession) { session.arrays().register(DecimalByteParts); session.aggregate_fns().register_aggregate_kernel( - DecimalByteParts::ID, + DecimalByteParts.id(), Some(IsConstant.id()), &DecimalBytePartsIsConstantKernel, ); diff --git a/encodings/fastlanes/benches/bit_transpose.rs b/encodings/fastlanes/benches/bit_transpose.rs index 4bc9027cc28..08c3ffb12e5 100644 --- a/encodings/fastlanes/benches/bit_transpose.rs +++ b/encodings/fastlanes/benches/bit_transpose.rs @@ -1,8 +1,6 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] - use divan::Bencher; use vortex_fastlanes::bit_transpose::scalar::transpose_bits_scalar; use vortex_fastlanes::bit_transpose::scalar::untranspose_bits_scalar; @@ -12,7 +10,7 @@ fn main() { } /// Generate deterministic test data. -#[allow(clippy::cast_possible_truncation)] +#[expect(clippy::cast_possible_truncation)] fn generate_test_data(seed: usize) -> [u8; 128] { let mut data = [0u8; 128]; for (i, byte) in data.iter_mut().enumerate() { diff --git a/encodings/fastlanes/benches/bitpacking_take.rs b/encodings/fastlanes/benches/bitpacking_take.rs index 61bd234c0f1..a5a7803bcaf 100644 --- a/encodings/fastlanes/benches/bitpacking_take.rs +++ b/encodings/fastlanes/benches/bitpacking_take.rs @@ -1,8 +1,8 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] -#![allow(clippy::cast_possible_truncation)] +#![expect(clippy::unwrap_used)] +#![expect(clippy::cast_possible_truncation)] use divan::Bencher; use rand::RngExt; @@ -28,7 +28,9 @@ fn main() { fn take_10_stratified(bencher: Bencher) { let values = fixture(65_536, 8); let uncompressed = PrimitiveArray::new(values, Validity::NonNullable); - let packed = bitpack_to_best_bit_width(&uncompressed).unwrap(); + let packed = + bitpack_to_best_bit_width(&uncompressed, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); let indices = PrimitiveArray::from_iter((0..10).map(|i| i * 6_553)); bencher @@ -46,7 +48,9 @@ fn take_10_stratified(bencher: Bencher) { fn take_10_contiguous(bencher: Bencher) { let values = fixture(65_536, 8); let uncompressed = PrimitiveArray::new(values, Validity::NonNullable); - let packed = bitpack_to_best_bit_width(&uncompressed).unwrap(); + let packed = + bitpack_to_best_bit_width(&uncompressed, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); let indices = buffer![0..10].into_array(); bencher @@ -65,7 +69,9 @@ fn take_10k_random(bencher: Bencher) { let values = fixture(65_536, 8); let range = Uniform::new(0, values.len()).unwrap(); let uncompressed = PrimitiveArray::new(values, Validity::NonNullable); - let packed = bitpack_to_best_bit_width(&uncompressed).unwrap(); + let packed = + bitpack_to_best_bit_width(&uncompressed, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); let rng = StdRng::seed_from_u64(0); let indices = PrimitiveArray::from_iter(rng.sample_iter(range).take(10_000).map(|i| i as u32)); @@ -85,7 +91,9 @@ fn take_10k_random(bencher: Bencher) { fn take_10k_contiguous(bencher: Bencher) { let values = fixture(65_536, 8); let uncompressed = PrimitiveArray::new(values, Validity::NonNullable); - let packed = bitpack_to_best_bit_width(&uncompressed).unwrap(); + let packed = + bitpack_to_best_bit_width(&uncompressed, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); let indices = PrimitiveArray::from_iter(0..10_000); bencher @@ -103,7 +111,9 @@ fn take_10k_contiguous(bencher: Bencher) { fn take_10k_dispersed(bencher: Bencher) { let values = fixture(65_536, 8); let uncompressed = PrimitiveArray::new(values.clone(), Validity::NonNullable); - let packed = bitpack_to_best_bit_width(&uncompressed).unwrap(); + let packed = + bitpack_to_best_bit_width(&uncompressed, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); let indices = PrimitiveArray::from_iter((0..10_000).map(|i| (i * 42) % values.len() as u64)); bencher @@ -121,7 +131,9 @@ fn take_10k_dispersed(bencher: Bencher) { fn take_10k_first_chunk_only(bencher: Bencher) { let values = fixture(65_536, 8); let uncompressed = PrimitiveArray::new(values, Validity::NonNullable); - let packed = bitpack_to_best_bit_width(&uncompressed).unwrap(); + let packed = + bitpack_to_best_bit_width(&uncompressed, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); let indices = PrimitiveArray::from_iter((0..10_000).map(|i| ((i * 42) % 1024) as u64)); bencher @@ -159,7 +171,9 @@ const NUM_EXCEPTIONS: u32 = 1024; fn patched_take_10_stratified(bencher: Bencher) { let values = (0u32..BIG_BASE2 + NUM_EXCEPTIONS).collect::>(); let uncompressed = PrimitiveArray::new(values, Validity::NonNullable); - let packed = bitpack_to_best_bit_width(&uncompressed).unwrap(); + let packed = + bitpack_to_best_bit_width(&uncompressed, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(packed.patches().is_some()); assert_eq!( @@ -184,7 +198,9 @@ fn patched_take_10_stratified(bencher: Bencher) { fn patched_take_10_contiguous(bencher: Bencher) { let values = (0u32..BIG_BASE2 + NUM_EXCEPTIONS).collect::>(); let uncompressed = PrimitiveArray::new(values, Validity::NonNullable); - let packed = bitpack_to_best_bit_width(&uncompressed).unwrap(); + let packed = + bitpack_to_best_bit_width(&uncompressed, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(packed.patches().is_some()); assert_eq!( @@ -209,7 +225,9 @@ fn patched_take_10_contiguous(bencher: Bencher) { fn patched_take_10k_random(bencher: Bencher) { let values = (0u32..BIG_BASE2 + NUM_EXCEPTIONS).collect::>(); let uncompressed = PrimitiveArray::new(values.clone(), Validity::NonNullable); - let packed = bitpack_to_best_bit_width(&uncompressed).unwrap(); + let packed = + bitpack_to_best_bit_width(&uncompressed, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); let rng = StdRng::seed_from_u64(0); let range = Uniform::new(0, values.len()).unwrap(); @@ -230,7 +248,9 @@ fn patched_take_10k_random(bencher: Bencher) { fn patched_take_10k_contiguous_not_patches(bencher: Bencher) { let values = (0u32..BIG_BASE2 + NUM_EXCEPTIONS).collect::>(); let uncompressed = PrimitiveArray::new(values, Validity::NonNullable); - let packed = bitpack_to_best_bit_width(&uncompressed).unwrap(); + let packed = + bitpack_to_best_bit_width(&uncompressed, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); let indices = PrimitiveArray::from_iter((0u32..NUM_EXCEPTIONS).cycle().take(10000)); bencher @@ -248,7 +268,9 @@ fn patched_take_10k_contiguous_not_patches(bencher: Bencher) { fn patched_take_10k_contiguous_patches(bencher: Bencher) { let values = (0u32..BIG_BASE2 + NUM_EXCEPTIONS).collect::>(); let uncompressed = PrimitiveArray::new(values, Validity::NonNullable); - let packed = bitpack_to_best_bit_width(&uncompressed).unwrap(); + let packed = + bitpack_to_best_bit_width(&uncompressed, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(packed.patches().is_some()); assert_eq!( @@ -274,7 +296,9 @@ fn patched_take_10k_contiguous_patches(bencher: Bencher) { fn patched_take_10k_dispersed(bencher: Bencher) { let values = (0u32..BIG_BASE2 + NUM_EXCEPTIONS).collect::>(); let uncompressed = PrimitiveArray::new(values.clone(), Validity::NonNullable); - let packed = bitpack_to_best_bit_width(&uncompressed).unwrap(); + let packed = + bitpack_to_best_bit_width(&uncompressed, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); let indices = PrimitiveArray::from_iter((0..10_000).map(|i| (i * 42) % values.len() as u64)); bencher @@ -292,7 +316,9 @@ fn patched_take_10k_dispersed(bencher: Bencher) { fn patched_take_10k_first_chunk_only(bencher: Bencher) { let values = (0u32..BIG_BASE2 + NUM_EXCEPTIONS).collect::>(); let uncompressed = PrimitiveArray::new(values, Validity::NonNullable); - let packed = bitpack_to_best_bit_width(&uncompressed).unwrap(); + let packed = + bitpack_to_best_bit_width(&uncompressed, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); let indices = PrimitiveArray::from_iter((0..10_000).map(|i| ((i * 42) % 1024) as u64)); bencher @@ -310,7 +336,9 @@ fn patched_take_10k_first_chunk_only(bencher: Bencher) { fn patched_take_10k_adversarial(bencher: Bencher) { let values = (0u32..BIG_BASE2 + NUM_EXCEPTIONS).collect::>(); let uncompressed = PrimitiveArray::new(values, Validity::NonNullable); - let packed = bitpack_to_best_bit_width(&uncompressed).unwrap(); + let packed = + bitpack_to_best_bit_width(&uncompressed, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); let per_chunk_count = 100; let indices = PrimitiveArray::from_iter( (0..(NUM_EXCEPTIONS + 1024) / 1024) diff --git a/encodings/fastlanes/benches/canonicalize_bench.rs b/encodings/fastlanes/benches/canonicalize_bench.rs index e6942067fca..7b0ee2453f5 100644 --- a/encodings/fastlanes/benches/canonicalize_bench.rs +++ b/encodings/fastlanes/benches/canonicalize_bench.rs @@ -48,13 +48,25 @@ fn into_canonical_non_nullable( let chunks = (0..chunk_count) .map(|_| { - make_array(&mut rng, chunk_len, fraction_patched, 0.0).vortex_expect("make_array works") + make_array( + &mut rng, + chunk_len, + fraction_patched, + 0.0, + &mut SESSION.create_execution_ctx(), + ) + .vortex_expect("make_array works") }) .collect::>(); bencher - .with_inputs(|| ChunkedArray::from_iter(chunks.clone()).into_array()) - .bench_refs(|chunked| chunked.to_canonical()); + .with_inputs(|| { + ( + ChunkedArray::from_iter(chunks.clone()).into_array(), + SESSION.create_execution_ctx(), + ) + }) + .bench_refs(|(chunked, ctx)| chunked.clone().execute::(ctx)); } #[cfg(not(codspeed))] @@ -67,7 +79,14 @@ fn canonical_into_non_nullable( let chunks = (0..chunk_count) .map(|_| { - make_array(&mut rng, chunk_len, fraction_patched, 0.0).vortex_expect("make_array works") + make_array( + &mut rng, + chunk_len, + fraction_patched, + 0.0, + &mut SESSION.create_execution_ctx(), + ) + .vortex_expect("make_array works") }) .collect::>(); @@ -78,11 +97,11 @@ fn canonical_into_non_nullable( chunked.dtype().nullability(), chunk_len * chunk_count, ); - (chunked, primitive_builder) + (chunked, primitive_builder, SESSION.create_execution_ctx()) }) - .bench_refs(|(chunked, primitive_builder)| { + .bench_refs(|(chunked, primitive_builder, ctx)| { chunked - .append_to_builder(primitive_builder, &mut SESSION.create_execution_ctx()) + .append_to_builder(primitive_builder, ctx) .vortex_expect("append failed"); primitive_builder.finish() }); @@ -108,14 +127,25 @@ fn into_canonical_nullable( let chunks = (0..chunk_count) .map(|_| { - make_array(&mut rng, chunk_len, fraction_patched, 0.05) - .vortex_expect("make_array works") + make_array( + &mut rng, + chunk_len, + fraction_patched, + 0.05, + &mut SESSION.create_execution_ctx(), + ) + .vortex_expect("make_array works") }) .collect::>(); bencher - .with_inputs(|| ChunkedArray::from_iter(chunks.clone()).into_array()) - .bench_values(|chunked| chunked.execute::(&mut SESSION.create_execution_ctx())); + .with_inputs(|| { + ( + ChunkedArray::from_iter(chunks.clone()).into_array(), + SESSION.create_execution_ctx(), + ) + }) + .bench_values(|(chunked, mut ctx)| chunked.execute::(&mut ctx)); } #[cfg(not(codspeed))] @@ -128,8 +158,14 @@ fn canonical_into_nullable( let chunks = (0..chunk_count) .map(|_| { - make_array(&mut rng, chunk_len, fraction_patched, 0.05) - .vortex_expect("make_array works") + make_array( + &mut rng, + chunk_len, + fraction_patched, + 0.05, + &mut SESSION.create_execution_ctx(), + ) + .vortex_expect("make_array works") }) .collect::>(); @@ -140,11 +176,11 @@ fn canonical_into_nullable( chunked.dtype().nullability(), chunk_len * chunk_count, ); - (chunked, primitive_builder) + (chunked, primitive_builder, SESSION.create_execution_ctx()) }) - .bench_refs(|(chunked, primitive_builder)| { + .bench_refs(|(chunked, primitive_builder, ctx)| { chunked - .append_to_builder(primitive_builder, &mut SESSION.create_execution_ctx()) + .append_to_builder(primitive_builder, ctx) .vortex_expect("append failed"); primitive_builder.finish() }); diff --git a/encodings/fastlanes/benches/compute_between.rs b/encodings/fastlanes/benches/compute_between.rs index a490a7bd544..1630764ff8e 100644 --- a/encodings/fastlanes/benches/compute_between.rs +++ b/encodings/fastlanes/benches/compute_between.rs @@ -11,7 +11,8 @@ use vortex_alp::ALPArraySlotsExt; use vortex_alp::alp_encode; use vortex_array::ArrayRef; use vortex_array::IntoArray; -use vortex_array::ToCanonical; +use vortex_array::LEGACY_SESSION; +use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::dtype::NativePType; use vortex_error::VortexExpect; @@ -38,7 +39,9 @@ fn generate_bit_pack_primitive_array( .map(|_| T::from_usize(rng.random_range(0..10_000)).vortex_expect("")) .collect::(); - bitpack_to_best_bit_width(&a).vortex_expect("").into_array() + bitpack_to_best_bit_width(&a, &mut LEGACY_SESSION.create_execution_ctx()) + .vortex_expect("") + .into_array() } fn generate_alp_bit_pack_primitive_array( @@ -49,11 +52,16 @@ fn generate_alp_bit_pack_primitive_array( .map(|_| T::from_usize(rng.random_range(0..10_000)).vortex_expect("")) .collect::(); - let alp = alp_encode(&a, None).vortex_expect(""); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let alp = alp_encode(a.as_view(), None, &mut ctx).vortex_expect(""); - let encoded = alp.encoded().to_primitive(); + let encoded = alp + .encoded() + .clone() + .execute::(&mut ctx) + .vortex_expect(""); - let bp = bitpack_to_best_bit_width(&encoded) + let bp = bitpack_to_best_bit_width(&encoded, &mut ctx) .vortex_expect("") .into_array(); ALP::new(bp, alp.exponents(), None).into_array() @@ -75,51 +83,11 @@ mod primitive { use vortex_array::dtype::NativePType; use vortex_array::scalar_fn::fns::between::BetweenOptions; use vortex_array::scalar_fn::fns::between::StrictComparison::NonStrict; - use vortex_array::scalar_fn::fns::operators::Operator; use vortex_error::VortexExpect; use crate::BENCH_ARGS; use crate::generate_primitive_array; - #[divan::bench( - types = [i32, i64, u32, u64, f32, f64], - args = BENCH_ARGS, - )] - fn old_raw_prim_test_between(bencher: Bencher, len: usize) - where - T: NumCast + NativePType, - vortex_array::scalar::Scalar: From, - { - let min = T::from_usize(5561).vortex_expect(""); - let max = T::from_usize(6032).vortex_expect(""); - let mut rng = StdRng::seed_from_u64(0); - let arr = generate_primitive_array::(&mut rng, len); - - bencher - .with_inputs(|| (&arr, LEGACY_SESSION.create_execution_ctx())) - .bench_refs(|(arr, ctx)| { - let gte = arr - .clone() - .into_array() - .binary( - ConstantArray::new(min, arr.len()).into_array(), - Operator::Gte, - ) - .vortex_expect(""); - let lt = arr - .clone() - .into_array() - .binary( - ConstantArray::new(max, arr.len()).into_array(), - Operator::Lt, - ) - .vortex_expect(""); - gte.binary(lt, Operator::And) - .vortex_expect("") - .execute::(ctx) - }) - } - #[divan::bench( types = [i32, i64, u32, u64, f32, f64], args = BENCH_ARGS, @@ -168,50 +136,11 @@ mod bitpack { use vortex_array::dtype::NativePType; use vortex_array::scalar_fn::fns::between::BetweenOptions; use vortex_array::scalar_fn::fns::between::StrictComparison::NonStrict; - use vortex_array::scalar_fn::fns::operators::Operator; use vortex_error::VortexExpect; use crate::BENCH_ARGS; use crate::generate_bit_pack_primitive_array; - #[divan::bench( - types = [i16, i32, i64], - args = BENCH_ARGS, - )] - fn old_bp_prim_test_between(bencher: Bencher, len: usize) - where - T: NumCast + NativePType, - vortex_array::scalar::Scalar: From, - { - let min = T::from_usize(5561).vortex_expect(""); - let max = T::from_usize(6032).vortex_expect(""); - let mut rng = StdRng::seed_from_u64(0); - let arr = generate_bit_pack_primitive_array::(&mut rng, len); - - bencher - .with_inputs(|| (&arr, LEGACY_SESSION.create_execution_ctx())) - .bench_refs(|(arr, ctx)| { - let gte = arr - .clone() - .binary( - ConstantArray::new(min, arr.len()).into_array(), - Operator::Gte, - ) - .vortex_expect(""); - let lt = arr - .clone() - .binary( - ConstantArray::new(max, arr.len()).into_array(), - Operator::Lt, - ) - .vortex_expect(""); - gte.binary(lt, Operator::And) - .unwrap() - .execute::(ctx) - .unwrap() - }) - } - #[divan::bench( types = [i16, i32, i64], args = BENCH_ARGS, @@ -259,50 +188,11 @@ mod alp { use vortex_array::dtype::NativePType; use vortex_array::scalar_fn::fns::between::BetweenOptions; use vortex_array::scalar_fn::fns::between::StrictComparison::NonStrict; - use vortex_array::scalar_fn::fns::operators::Operator; use vortex_error::VortexExpect; use crate::BENCH_ARGS; use crate::generate_alp_bit_pack_primitive_array; - #[divan::bench( - types = [f32, f64], - args = BENCH_ARGS, - )] - fn old_alp_prim_test_between(bencher: Bencher, len: usize) - where - T: NumCast + NativePType, - vortex_array::scalar::Scalar: From, - { - let min = T::from_usize(5561).vortex_expect(""); - let max = T::from_usize(6032).vortex_expect(""); - let mut rng = StdRng::seed_from_u64(0); - let arr = generate_alp_bit_pack_primitive_array::(&mut rng, len); - - bencher - .with_inputs(|| (&arr, LEGACY_SESSION.create_execution_ctx())) - .bench_refs(|(arr, ctx)| { - let gte = arr - .clone() - .binary( - ConstantArray::new(min, arr.len()).into_array(), - Operator::Gte, - ) - .vortex_expect(""); - let lt = arr - .clone() - .binary( - ConstantArray::new(max, arr.len()).into_array(), - Operator::Lt, - ) - .vortex_expect(""); - gte.binary(lt, Operator::And) - .unwrap() - .execute::(ctx) - .unwrap() - }) - } - #[divan::bench( types = [f32, f64], args = BENCH_ARGS, diff --git a/encodings/fastlanes/public-api.lock b/encodings/fastlanes/public-api.lock index 15cc889bf9f..bbd1f18ab4c 100644 --- a/encodings/fastlanes/public-api.lock +++ b/encodings/fastlanes/public-api.lock @@ -2,51 +2,51 @@ pub mod vortex_fastlanes pub mod vortex_fastlanes::bit_transpose -pub fn vortex_fastlanes::bit_transpose::transpose_bitbuffer(bits: vortex_buffer::bit::buf::BitBuffer) -> vortex_buffer::bit::buf::BitBuffer +pub fn vortex_fastlanes::bit_transpose::transpose_bitbuffer(vortex_buffer::bit::buf::BitBuffer) -> vortex_buffer::bit::buf::BitBuffer -pub fn vortex_fastlanes::bit_transpose::transpose_bits(input: &[u8; 128], output: &mut [u8; 128]) +pub fn vortex_fastlanes::bit_transpose::transpose_bits(&[u8; 128], &mut [u8; 128]) -pub fn vortex_fastlanes::bit_transpose::transpose_validity(validity: &vortex_array::validity::Validity, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_fastlanes::bit_transpose::transpose_validity(&vortex_array::validity::Validity, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_fastlanes::bit_transpose::untranspose_bitbuffer(bits: vortex_buffer::bit::buf::BitBuffer) -> vortex_buffer::bit::buf::BitBuffer +pub fn vortex_fastlanes::bit_transpose::untranspose_bitbuffer(vortex_buffer::bit::buf::BitBuffer) -> vortex_buffer::bit::buf::BitBuffer -pub fn vortex_fastlanes::bit_transpose::untranspose_bits(input: &[u8; 128], output: &mut [u8; 128]) +pub fn vortex_fastlanes::bit_transpose::untranspose_bits(&[u8; 128], &mut [u8; 128]) -pub fn vortex_fastlanes::bit_transpose::untranspose_validity(validity: &vortex_array::validity::Validity, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_fastlanes::bit_transpose::untranspose_validity(&vortex_array::validity::Validity, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub mod vortex_fastlanes::bitpack_compress -pub fn vortex_fastlanes::bitpack_compress::bit_width_histogram(array: &vortex_array::arrays::primitive::vtable::PrimitiveArray) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::bitpack_compress::bit_width_histogram(vortex_array::array::view::ArrayView<'_, vortex_array::arrays::primitive::vtable::Primitive>, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> -pub fn vortex_fastlanes::bitpack_compress::bitpack_encode(array: &vortex_array::arrays::primitive::vtable::PrimitiveArray, bit_width: u8, bit_width_freq: core::option::Option<&[usize]>) -> vortex_error::VortexResult +pub fn vortex_fastlanes::bitpack_compress::bitpack_encode(&vortex_array::arrays::primitive::vtable::PrimitiveArray, u8, core::option::Option<&[usize]>, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub unsafe fn vortex_fastlanes::bitpack_compress::bitpack_encode_unchecked(array: vortex_array::arrays::primitive::vtable::PrimitiveArray, bit_width: u8) -> vortex_error::VortexResult +pub unsafe fn vortex_fastlanes::bitpack_compress::bitpack_encode_unchecked(vortex_array::arrays::primitive::vtable::PrimitiveArray, u8) -> vortex_error::VortexResult -pub fn vortex_fastlanes::bitpack_compress::bitpack_primitive(array: &[T], bit_width: u8) -> vortex_buffer::buffer::Buffer +pub fn vortex_fastlanes::bitpack_compress::bitpack_primitive(&[T], u8) -> vortex_buffer::buffer::Buffer -pub fn vortex_fastlanes::bitpack_compress::bitpack_to_best_bit_width(array: &vortex_array::arrays::primitive::vtable::PrimitiveArray) -> vortex_error::VortexResult +pub fn vortex_fastlanes::bitpack_compress::bitpack_to_best_bit_width(&vortex_array::arrays::primitive::vtable::PrimitiveArray, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub unsafe fn vortex_fastlanes::bitpack_compress::bitpack_unchecked(parray: &vortex_array::arrays::primitive::vtable::PrimitiveArray, bit_width: u8) -> vortex_buffer::ByteBuffer +pub unsafe fn vortex_fastlanes::bitpack_compress::bitpack_unchecked(&vortex_array::arrays::primitive::vtable::PrimitiveArray, u8) -> vortex_buffer::ByteBuffer -pub fn vortex_fastlanes::bitpack_compress::find_best_bit_width(ptype: vortex_array::dtype::ptype::PType, bit_width_freq: &[usize]) -> vortex_error::VortexResult +pub fn vortex_fastlanes::bitpack_compress::find_best_bit_width(vortex_array::dtype::ptype::PType, &[usize]) -> vortex_error::VortexResult -pub fn vortex_fastlanes::bitpack_compress::gather_patches(parray: &vortex_array::arrays::primitive::vtable::PrimitiveArray, bit_width: u8, num_exceptions_hint: usize) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::bitpack_compress::gather_patches(&vortex_array::arrays::primitive::vtable::PrimitiveArray, u8, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> pub mod vortex_fastlanes::bitpack_decompress -pub fn vortex_fastlanes::bitpack_decompress::apply_patches_to_uninit_range(dst: &mut vortex_array::builders::primitive::UninitRange<'_, T>, patches: &vortex_array::patches::Patches, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_fastlanes::bitpack_decompress::apply_patches_to_uninit_range(&mut vortex_array::builders::primitive::UninitRange<'_, T>, &vortex_array::patches::Patches, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_fastlanes::bitpack_decompress::apply_patches_to_uninit_range_fn T>(dst: &mut vortex_array::builders::primitive::UninitRange<'_, T>, patches: &vortex_array::patches::Patches, ctx: &mut vortex_array::executor::ExecutionCtx, f: F) -> vortex_error::VortexResult<()> +pub fn vortex_fastlanes::bitpack_decompress::apply_patches_to_uninit_range_fn T>(&mut vortex_array::builders::primitive::UninitRange<'_, T>, &vortex_array::patches::Patches, &mut vortex_array::executor::ExecutionCtx, F) -> vortex_error::VortexResult<()> -pub fn vortex_fastlanes::bitpack_decompress::count_exceptions(bit_width: u8, bit_width_freq: &[usize]) -> usize +pub fn vortex_fastlanes::bitpack_decompress::count_exceptions(u8, &[usize]) -> usize -pub fn vortex_fastlanes::bitpack_decompress::unpack_array(array: vortex_array::array::view::ArrayView<'_, vortex_fastlanes::BitPacked>, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_fastlanes::bitpack_decompress::unpack_array(vortex_array::array::view::ArrayView<'_, vortex_fastlanes::BitPacked>, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_fastlanes::bitpack_decompress::unpack_primitive_array(array: vortex_array::array::view::ArrayView<'_, vortex_fastlanes::BitPacked>, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_fastlanes::bitpack_decompress::unpack_primitive_array(vortex_array::array::view::ArrayView<'_, vortex_fastlanes::BitPacked>, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_fastlanes::bitpack_decompress::unpack_single(array: vortex_array::array::view::ArrayView<'_, vortex_fastlanes::BitPacked>, index: usize) -> vortex_array::scalar::Scalar +pub fn vortex_fastlanes::bitpack_decompress::unpack_single(vortex_array::array::view::ArrayView<'_, vortex_fastlanes::BitPacked>, usize) -> vortex_array::scalar::Scalar -pub unsafe fn vortex_fastlanes::bitpack_decompress::unpack_single_primitive(packed: &[T], bit_width: usize, index_to_decode: usize) -> T +pub unsafe fn vortex_fastlanes::bitpack_decompress::unpack_single_primitive(&[T], usize, usize) -> T pub mod vortex_fastlanes::unpack_iter @@ -54,13 +54,13 @@ pub struct vortex_fastlanes::unpack_iter::BitPackingStrategy impl> vortex_fastlanes::unpack_iter::UnpackStrategy for vortex_fastlanes::unpack_iter::BitPackingStrategy -pub unsafe fn vortex_fastlanes::unpack_iter::BitPackingStrategy::unpack_chunk(&self, bit_width: usize, chunk: &[::Physical], dst: &mut [::Physical]) +pub unsafe fn vortex_fastlanes::unpack_iter::BitPackingStrategy::unpack_chunk(&self, usize, &[::Physical], &mut [::Physical]) pub struct vortex_fastlanes::unpack_iter::BitUnpackIterator<'a, T: vortex_fastlanes::unpack_iter::BitPacked + 'a> impl<'a, T: vortex_fastlanes::unpack_iter::BitPacked> vortex_fastlanes::unpack_iter::BitUnpackIterator<'a, T> -pub fn vortex_fastlanes::unpack_iter::BitUnpackIterator<'a, T>::new(packed: &'a [::Physical], buffer: &'a mut [core::mem::maybe_uninit::MaybeUninit; 1024], bit_width: usize, elems_per_chunk: usize, num_chunks: usize, first_chunk_is_sliced: bool) -> Self +pub fn vortex_fastlanes::unpack_iter::BitUnpackIterator<'a, T>::new(&'a [::Physical], &'a mut [core::mem::maybe_uninit::MaybeUninit; 1024], usize, usize, usize, bool) -> Self impl<'a, T: vortex_fastlanes::unpack_iter::BitPacked + 'a> lending_iterator::lending_iterator::LendingIterator for vortex_fastlanes::unpack_iter::BitUnpackIterator<'a, T> @@ -74,19 +74,19 @@ pub struct vortex_fastlanes::unpack_iter::UnpackedChunks> vortex_fastlanes::unpack_iter::UnpackedChunks -pub fn vortex_fastlanes::unpack_iter::UnpackedChunks::decode_into(&mut self, output: &mut [core::mem::maybe_uninit::MaybeUninit]) +pub fn vortex_fastlanes::unpack_iter::UnpackedChunks::decode_into(&mut self, &mut [core::mem::maybe_uninit::MaybeUninit]) pub fn vortex_fastlanes::unpack_iter::UnpackedChunks::initial(&mut self) -> core::option::Option<&mut [T]> pub fn vortex_fastlanes::unpack_iter::UnpackedChunks::trailer(&mut self) -> core::option::Option<&mut [T]> -pub fn vortex_fastlanes::unpack_iter::UnpackedChunks::try_new_with_strategy(strategy: S, packed: vortex_buffer::ByteBuffer, bit_width: usize, offset: usize, len: usize) -> vortex_error::VortexResult +pub fn vortex_fastlanes::unpack_iter::UnpackedChunks::try_new_with_strategy(S, vortex_buffer::ByteBuffer, usize, usize, usize) -> vortex_error::VortexResult impl vortex_fastlanes::unpack_iter::UnpackedChunks pub fn vortex_fastlanes::unpack_iter::UnpackedChunks::full_chunks(&mut self) -> vortex_fastlanes::unpack_iter::BitUnpackIterator<'_, T> -pub fn vortex_fastlanes::unpack_iter::UnpackedChunks::try_new(array: &vortex_fastlanes::BitPackedData, len: usize) -> vortex_error::VortexResult +pub fn vortex_fastlanes::unpack_iter::UnpackedChunks::try_new(&vortex_fastlanes::BitPackedData, usize) -> vortex_error::VortexResult pub trait vortex_fastlanes::unpack_iter::BitPacked: vortex_array::dtype::ptype::PhysicalPType @@ -108,11 +108,11 @@ impl vortex_fastlanes::unpack_iter::BitPacked for u8 pub trait vortex_fastlanes::unpack_iter::UnpackStrategy -pub unsafe fn vortex_fastlanes::unpack_iter::UnpackStrategy::unpack_chunk(&self, bit_width: usize, chunk: &[::Physical], dst: &mut [::Physical]) +pub unsafe fn vortex_fastlanes::unpack_iter::UnpackStrategy::unpack_chunk(&self, usize, &[::Physical], &mut [::Physical]) impl> vortex_fastlanes::unpack_iter::UnpackStrategy for vortex_fastlanes::unpack_iter::BitPackingStrategy -pub unsafe fn vortex_fastlanes::unpack_iter::BitPackingStrategy::unpack_chunk(&self, bit_width: usize, chunk: &[::Physical], dst: &mut [::Physical]) +pub unsafe fn vortex_fastlanes::unpack_iter::BitPackingStrategy::unpack_chunk(&self, usize, &[::Physical], &mut [::Physical]) pub type vortex_fastlanes::unpack_iter::BitUnpackedChunks = vortex_fastlanes::unpack_iter::UnpackedChunks @@ -120,13 +120,11 @@ pub struct vortex_fastlanes::BitPacked impl vortex_fastlanes::BitPacked -pub const vortex_fastlanes::BitPacked::ID: vortex_array::array::ArrayId +pub fn vortex_fastlanes::BitPacked::encode(&vortex_array::array::erased::ArrayRef, u8, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_fastlanes::BitPacked::encode(array: &vortex_array::array::erased::ArrayRef, bit_width: u8) -> vortex_error::VortexResult +pub fn vortex_fastlanes::BitPacked::into_parts(vortex_fastlanes::BitPackedArray) -> vortex_fastlanes::BitPackedDataParts -pub fn vortex_fastlanes::BitPacked::into_parts(array: vortex_fastlanes::BitPackedArray) -> vortex_fastlanes::BitPackedDataParts - -pub fn vortex_fastlanes::BitPacked::try_new(packed: vortex_array::buffer::BufferHandle, ptype: vortex_array::dtype::ptype::PType, validity: vortex_array::validity::Validity, patches: core::option::Option, bit_width: u8, len: usize, offset: u16) -> vortex_error::VortexResult +pub fn vortex_fastlanes::BitPacked::try_new(vortex_array::buffer::BufferHandle, vortex_array::dtype::ptype::PType, vortex_array::validity::Validity, core::option::Option, u8, usize, u16) -> vortex_error::VortexResult impl core::clone::Clone for vortex_fastlanes::BitPacked @@ -134,7 +132,7 @@ pub fn vortex_fastlanes::BitPacked::clone(&self) -> vortex_fastlanes::BitPacked impl core::fmt::Debug for vortex_fastlanes::BitPacked -pub fn vortex_fastlanes::BitPacked::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_fastlanes::BitPacked::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::array::vtable::VTable for vortex_fastlanes::BitPacked @@ -144,53 +142,57 @@ pub type vortex_fastlanes::BitPacked::OperationsVTable = vortex_fastlanes::BitPa pub type vortex_fastlanes::BitPacked::ValidityVTable = vortex_fastlanes::BitPacked -pub fn vortex_fastlanes::BitPacked::append_to_builder(array: vortex_array::array::view::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_fastlanes::BitPacked::append_to_builder(vortex_array::array::view::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_fastlanes::BitPacked::buffer(array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_fastlanes::BitPacked::buffer(vortex_array::array::view::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_fastlanes::BitPacked::buffer_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_fastlanes::BitPacked::buffer_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_fastlanes::BitPacked::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::BitPacked::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_fastlanes::BitPacked::execute(array: vortex_array::array::typed::Array, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_fastlanes::BitPacked::execute(vortex_array::array::typed::Array, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_fastlanes::BitPacked::execute_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::BitPacked::execute_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_fastlanes::BitPacked::id(&self) -> vortex_array::array::ArrayId -pub fn vortex_fastlanes::BitPacked::nbuffers(_array: vortex_array::array::view::ArrayView<'_, Self>) -> usize +pub fn vortex_fastlanes::BitPacked::nbuffers(vortex_array::array::view::ArrayView<'_, Self>) -> usize -pub fn vortex_fastlanes::BitPacked::reduce_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::BitPacked::reduce_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_fastlanes::BitPacked::serialize(array: vortex_array::array::view::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_fastlanes::BitPacked::serialize(vortex_array::array::view::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_fastlanes::BitPacked::slot_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_fastlanes::BitPacked::slot_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_fastlanes::BitPacked::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_fastlanes::BitPacked::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::array::vtable::operations::OperationsVTable for vortex_fastlanes::BitPacked -pub fn vortex_fastlanes::BitPacked::scalar_at(array: vortex_array::array::view::ArrayView<'_, vortex_fastlanes::BitPacked>, index: usize, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_fastlanes::BitPacked::scalar_at(vortex_array::array::view::ArrayView<'_, vortex_fastlanes::BitPacked>, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::array::vtable::validity::ValidityVTable for vortex_fastlanes::BitPacked -pub fn vortex_fastlanes::BitPacked::validity(array: vortex_array::array::view::ArrayView<'_, vortex_fastlanes::BitPacked>) -> vortex_error::VortexResult +pub fn vortex_fastlanes::BitPacked::validity(vortex_array::array::view::ArrayView<'_, vortex_fastlanes::BitPacked>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::take::TakeExecute for vortex_fastlanes::BitPacked -pub fn vortex_fastlanes::BitPacked::take(array: vortex_array::array::view::ArrayView<'_, Self>, indices: &vortex_array::array::erased::ArrayRef, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::BitPacked::take(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::kernel::FilterKernel for vortex_fastlanes::BitPacked -pub fn vortex_fastlanes::BitPacked::filter(array: vortex_array::array::view::ArrayView<'_, Self>, mask: &vortex_mask::Mask, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::BitPacked::filter(vortex_array::array::view::ArrayView<'_, Self>, &vortex_mask::Mask, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_fastlanes::BitPacked -pub fn vortex_fastlanes::BitPacked::slice(array: vortex_array::array::view::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::BitPacked::slice(vortex_array::array::view::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::kernel::CastKernel for vortex_fastlanes::BitPacked + +pub fn vortex_fastlanes::BitPacked::cast(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::dtype::DType, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::kernel::CastReduce for vortex_fastlanes::BitPacked -pub fn vortex_fastlanes::BitPacked::cast(array: vortex_array::array::view::ArrayView<'_, Self>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::BitPacked::cast(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> pub struct vortex_fastlanes::BitPackedData @@ -198,7 +200,7 @@ impl vortex_fastlanes::BitPackedData pub fn vortex_fastlanes::BitPackedData::bit_width(&self) -> u8 -pub fn vortex_fastlanes::BitPackedData::encode(array: &vortex_array::array::erased::ArrayRef, bit_width: u8) -> vortex_error::VortexResult +pub fn vortex_fastlanes::BitPackedData::encode(&vortex_array::array::erased::ArrayRef, u8, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_fastlanes::BitPackedData::max_packed_value(&self) -> usize @@ -208,11 +210,11 @@ pub fn vortex_fastlanes::BitPackedData::packed(&self) -> &vortex_array::buffer:: pub fn vortex_fastlanes::BitPackedData::packed_slice(&self) -> &[T] -pub fn vortex_fastlanes::BitPackedData::ptype(&self, dtype: &vortex_array::dtype::DType) -> vortex_array::dtype::ptype::PType +pub fn vortex_fastlanes::BitPackedData::ptype(&self, &vortex_array::dtype::DType) -> vortex_array::dtype::ptype::PType -pub fn vortex_fastlanes::BitPackedData::try_new(packed: vortex_array::buffer::BufferHandle, patches: core::option::Option, bit_width: u8, offset: u16) -> vortex_error::VortexResult +pub fn vortex_fastlanes::BitPackedData::try_new(vortex_array::buffer::BufferHandle, core::option::Option, u8, u16) -> vortex_error::VortexResult -pub fn vortex_fastlanes::BitPackedData::unpacked_chunks(&self, dtype: &vortex_array::dtype::DType, len: usize) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::BitPackedData::unpacked_chunks(&self, &vortex_array::dtype::DType, usize) -> vortex_error::VortexResult> impl core::clone::Clone for vortex_fastlanes::BitPackedData @@ -220,19 +222,19 @@ pub fn vortex_fastlanes::BitPackedData::clone(&self) -> vortex_fastlanes::BitPac impl core::fmt::Debug for vortex_fastlanes::BitPackedData -pub fn vortex_fastlanes::BitPackedData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_fastlanes::BitPackedData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_fastlanes::BitPackedData -pub fn vortex_fastlanes::BitPackedData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_fastlanes::BitPackedData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::hash::ArrayEq for vortex_fastlanes::BitPackedData -pub fn vortex_fastlanes::BitPackedData::array_eq(&self, other: &Self, precision: vortex_array::hash::Precision) -> bool +pub fn vortex_fastlanes::BitPackedData::array_eq(&self, &Self, vortex_array::hash::Precision) -> bool impl vortex_array::hash::ArrayHash for vortex_fastlanes::BitPackedData -pub fn vortex_fastlanes::BitPackedData::array_hash(&self, state: &mut H, precision: vortex_array::hash::Precision) +pub fn vortex_fastlanes::BitPackedData::array_hash(&self, &mut H, vortex_array::hash::Precision) pub struct vortex_fastlanes::BitPackedDataParts @@ -272,7 +274,7 @@ pub const vortex_fastlanes::BitPackedSlots::PATCH_VALUES: usize pub const vortex_fastlanes::BitPackedSlots::VALIDITY_CHILD: usize -pub fn vortex_fastlanes::BitPackedSlots::from_slots(slots: alloc::vec::Vec>) -> Self +pub fn vortex_fastlanes::BitPackedSlots::from_slots(alloc::vec::Vec>) -> Self pub fn vortex_fastlanes::BitPackedSlots::into_slots(self) -> alloc::vec::Vec> @@ -280,11 +282,9 @@ pub struct vortex_fastlanes::Delta impl vortex_fastlanes::Delta -pub const vortex_fastlanes::Delta::ID: vortex_array::array::ArrayId +pub fn vortex_fastlanes::Delta::try_from_primitive_array(&vortex_array::arrays::primitive::vtable::PrimitiveArray, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_fastlanes::Delta::try_from_primitive_array(array: &vortex_array::arrays::primitive::vtable::PrimitiveArray, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult - -pub fn vortex_fastlanes::Delta::try_new(bases: vortex_array::array::erased::ArrayRef, deltas: vortex_array::array::erased::ArrayRef, offset: usize, len: usize) -> vortex_error::VortexResult +pub fn vortex_fastlanes::Delta::try_new(vortex_array::array::erased::ArrayRef, vortex_array::array::erased::ArrayRef, usize, usize) -> vortex_error::VortexResult impl core::clone::Clone for vortex_fastlanes::Delta @@ -292,7 +292,7 @@ pub fn vortex_fastlanes::Delta::clone(&self) -> vortex_fastlanes::Delta impl core::fmt::Debug for vortex_fastlanes::Delta -pub fn vortex_fastlanes::Delta::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_fastlanes::Delta::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::array::vtable::VTable for vortex_fastlanes::Delta @@ -302,47 +302,47 @@ pub type vortex_fastlanes::Delta::OperationsVTable = vortex_fastlanes::Delta pub type vortex_fastlanes::Delta::ValidityVTable = vortex_fastlanes::Delta -pub fn vortex_fastlanes::Delta::buffer(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_fastlanes::Delta::buffer(vortex_array::array::view::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_fastlanes::Delta::buffer_name(_array: vortex_array::array::view::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_fastlanes::Delta::buffer_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_fastlanes::Delta::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::Delta::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_fastlanes::Delta::execute(array: vortex_array::array::typed::Array, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_fastlanes::Delta::execute(vortex_array::array::typed::Array, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_fastlanes::Delta::id(&self) -> vortex_array::array::ArrayId -pub fn vortex_fastlanes::Delta::nbuffers(_array: vortex_array::array::view::ArrayView<'_, Self>) -> usize +pub fn vortex_fastlanes::Delta::nbuffers(vortex_array::array::view::ArrayView<'_, Self>) -> usize -pub fn vortex_fastlanes::Delta::reduce_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::Delta::reduce_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_fastlanes::Delta::serialize(array: vortex_array::array::view::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_fastlanes::Delta::serialize(vortex_array::array::view::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_fastlanes::Delta::slot_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_fastlanes::Delta::slot_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_fastlanes::Delta::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_fastlanes::Delta::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::array::vtable::operations::OperationsVTable for vortex_fastlanes::Delta -pub fn vortex_fastlanes::Delta::scalar_at(array: vortex_array::array::view::ArrayView<'_, vortex_fastlanes::Delta>, index: usize, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_fastlanes::Delta::scalar_at(vortex_array::array::view::ArrayView<'_, vortex_fastlanes::Delta>, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::array::vtable::validity::ValidityVTable for vortex_fastlanes::Delta -pub fn vortex_fastlanes::Delta::validity(array: vortex_array::array::view::ArrayView<'_, vortex_fastlanes::Delta>) -> vortex_error::VortexResult +pub fn vortex_fastlanes::Delta::validity(vortex_array::array::view::ArrayView<'_, vortex_fastlanes::Delta>) -> vortex_error::VortexResult impl vortex_array::arrays::slice::SliceReduce for vortex_fastlanes::Delta -pub fn vortex_fastlanes::Delta::slice(array: vortex_array::array::view::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::Delta::slice(vortex_array::array::view::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::kernel::CastReduce for vortex_fastlanes::Delta -pub fn vortex_fastlanes::Delta::cast(array: vortex_array::array::view::ArrayView<'_, Self>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::Delta::cast(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> pub struct vortex_fastlanes::DeltaData impl vortex_fastlanes::DeltaData -pub fn vortex_fastlanes::DeltaData::try_new(offset: usize) -> vortex_error::VortexResult +pub fn vortex_fastlanes::DeltaData::try_new(usize) -> vortex_error::VortexResult impl core::clone::Clone for vortex_fastlanes::DeltaData @@ -350,29 +350,27 @@ pub fn vortex_fastlanes::DeltaData::clone(&self) -> vortex_fastlanes::DeltaData impl core::fmt::Debug for vortex_fastlanes::DeltaData -pub fn vortex_fastlanes::DeltaData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_fastlanes::DeltaData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_fastlanes::DeltaData -pub fn vortex_fastlanes::DeltaData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_fastlanes::DeltaData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::hash::ArrayEq for vortex_fastlanes::DeltaData -pub fn vortex_fastlanes::DeltaData::array_eq(&self, other: &Self, _precision: vortex_array::hash::Precision) -> bool +pub fn vortex_fastlanes::DeltaData::array_eq(&self, &Self, vortex_array::hash::Precision) -> bool impl vortex_array::hash::ArrayHash for vortex_fastlanes::DeltaData -pub fn vortex_fastlanes::DeltaData::array_hash(&self, state: &mut H, _precision: vortex_array::hash::Precision) +pub fn vortex_fastlanes::DeltaData::array_hash(&self, &mut H, vortex_array::hash::Precision) pub struct vortex_fastlanes::FoR impl vortex_fastlanes::FoR -pub const vortex_fastlanes::FoR::ID: vortex_array::array::ArrayId - -pub fn vortex_fastlanes::FoR::encode(array: vortex_array::arrays::primitive::vtable::PrimitiveArray) -> vortex_error::VortexResult +pub fn vortex_fastlanes::FoR::encode(vortex_array::arrays::primitive::vtable::PrimitiveArray) -> vortex_error::VortexResult -pub fn vortex_fastlanes::FoR::try_new(encoded: vortex_array::array::erased::ArrayRef, reference: vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn vortex_fastlanes::FoR::try_new(vortex_array::array::erased::ArrayRef, vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::clone::Clone for vortex_fastlanes::FoR @@ -380,7 +378,7 @@ pub fn vortex_fastlanes::FoR::clone(&self) -> vortex_fastlanes::FoR impl core::fmt::Debug for vortex_fastlanes::FoR -pub fn vortex_fastlanes::FoR::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_fastlanes::FoR::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::array::vtable::VTable for vortex_fastlanes::FoR @@ -390,61 +388,61 @@ pub type vortex_fastlanes::FoR::OperationsVTable = vortex_fastlanes::FoR pub type vortex_fastlanes::FoR::ValidityVTable = vortex_array::array::vtable::validity::ValidityVTableFromChild -pub fn vortex_fastlanes::FoR::buffer(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_fastlanes::FoR::buffer(vortex_array::array::view::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_fastlanes::FoR::buffer_name(_array: vortex_array::array::view::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_fastlanes::FoR::buffer_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_fastlanes::FoR::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::FoR::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_fastlanes::FoR::execute(array: vortex_array::array::typed::Array, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_fastlanes::FoR::execute(vortex_array::array::typed::Array, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_fastlanes::FoR::execute_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::FoR::execute_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_fastlanes::FoR::id(&self) -> vortex_array::array::ArrayId -pub fn vortex_fastlanes::FoR::nbuffers(_array: vortex_array::array::view::ArrayView<'_, Self>) -> usize +pub fn vortex_fastlanes::FoR::nbuffers(vortex_array::array::view::ArrayView<'_, Self>) -> usize -pub fn vortex_fastlanes::FoR::reduce_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::FoR::reduce_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_fastlanes::FoR::serialize(array: vortex_array::array::view::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_fastlanes::FoR::serialize(vortex_array::array::view::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_fastlanes::FoR::slot_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_fastlanes::FoR::slot_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_fastlanes::FoR::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_fastlanes::FoR::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::array::vtable::operations::OperationsVTable for vortex_fastlanes::FoR -pub fn vortex_fastlanes::FoR::scalar_at(array: vortex_array::array::view::ArrayView<'_, vortex_fastlanes::FoR>, index: usize, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_fastlanes::FoR::scalar_at(vortex_array::array::view::ArrayView<'_, vortex_fastlanes::FoR>, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::array::vtable::validity::ValidityChild for vortex_fastlanes::FoR -pub fn vortex_fastlanes::FoR::validity_child(array: vortex_array::array::view::ArrayView<'_, vortex_fastlanes::FoR>) -> vortex_array::array::erased::ArrayRef +pub fn vortex_fastlanes::FoR::validity_child(vortex_array::array::view::ArrayView<'_, vortex_fastlanes::FoR>) -> vortex_array::array::erased::ArrayRef impl vortex_array::arrays::dict::take::TakeExecute for vortex_fastlanes::FoR -pub fn vortex_fastlanes::FoR::take(array: vortex_array::array::view::ArrayView<'_, Self>, indices: &vortex_array::array::erased::ArrayRef, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::FoR::take(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::kernel::FilterReduce for vortex_fastlanes::FoR -pub fn vortex_fastlanes::FoR::filter(array: vortex_array::array::view::ArrayView<'_, Self>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::FoR::filter(vortex_array::array::view::ArrayView<'_, Self>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_fastlanes::FoR -pub fn vortex_fastlanes::FoR::slice(array: vortex_array::array::view::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::FoR::slice(vortex_array::array::view::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::binary::compare::CompareKernel for vortex_fastlanes::FoR -pub fn vortex_fastlanes::FoR::compare(lhs: vortex_array::array::view::ArrayView<'_, Self>, rhs: &vortex_array::array::erased::ArrayRef, operator: vortex_array::scalar_fn::fns::operators::CompareOperator, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::FoR::compare(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, vortex_array::scalar_fn::fns::operators::CompareOperator, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::kernel::CastReduce for vortex_fastlanes::FoR -pub fn vortex_fastlanes::FoR::cast(array: vortex_array::array::view::ArrayView<'_, Self>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::FoR::cast(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> pub struct vortex_fastlanes::FoRData impl vortex_fastlanes::FoRData -pub fn vortex_fastlanes::FoRData::encode(array: vortex_array::arrays::primitive::vtable::PrimitiveArray) -> vortex_error::VortexResult +pub fn vortex_fastlanes::FoRData::encode(vortex_array::arrays::primitive::vtable::PrimitiveArray) -> vortex_error::VortexResult impl vortex_fastlanes::FoRData @@ -456,31 +454,29 @@ pub fn vortex_fastlanes::FoRData::clone(&self) -> vortex_fastlanes::FoRData impl core::fmt::Debug for vortex_fastlanes::FoRData -pub fn vortex_fastlanes::FoRData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_fastlanes::FoRData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_fastlanes::FoRData -pub fn vortex_fastlanes::FoRData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_fastlanes::FoRData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::hash::ArrayEq for vortex_fastlanes::FoRData -pub fn vortex_fastlanes::FoRData::array_eq(&self, other: &Self, _precision: vortex_array::hash::Precision) -> bool +pub fn vortex_fastlanes::FoRData::array_eq(&self, &Self, vortex_array::hash::Precision) -> bool impl vortex_array::hash::ArrayHash for vortex_fastlanes::FoRData -pub fn vortex_fastlanes::FoRData::array_hash(&self, state: &mut H, _precision: vortex_array::hash::Precision) +pub fn vortex_fastlanes::FoRData::array_hash(&self, &mut H, vortex_array::hash::Precision) pub struct vortex_fastlanes::RLE impl vortex_fastlanes::RLE -pub const vortex_fastlanes::RLE::ID: vortex_array::array::ArrayId +pub fn vortex_fastlanes::RLE::encode(vortex_array::array::view::ArrayView<'_, vortex_array::arrays::primitive::vtable::Primitive>, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_fastlanes::RLE::encode(array: &vortex_array::arrays::primitive::vtable::PrimitiveArray) -> vortex_error::VortexResult +pub unsafe fn vortex_fastlanes::RLE::new_unchecked(vortex_array::array::erased::ArrayRef, vortex_array::array::erased::ArrayRef, vortex_array::array::erased::ArrayRef, usize, usize) -> vortex_fastlanes::RLEArray -pub unsafe fn vortex_fastlanes::RLE::new_unchecked(values: vortex_array::array::erased::ArrayRef, indices: vortex_array::array::erased::ArrayRef, values_idx_offsets: vortex_array::array::erased::ArrayRef, offset: usize, length: usize) -> vortex_fastlanes::RLEArray - -pub fn vortex_fastlanes::RLE::try_new(values: vortex_array::array::erased::ArrayRef, indices: vortex_array::array::erased::ArrayRef, values_idx_offsets: vortex_array::array::erased::ArrayRef, offset: usize, length: usize) -> vortex_error::VortexResult +pub fn vortex_fastlanes::RLE::try_new(vortex_array::array::erased::ArrayRef, vortex_array::array::erased::ArrayRef, vortex_array::array::erased::ArrayRef, usize, usize) -> vortex_error::VortexResult impl core::clone::Clone for vortex_fastlanes::RLE @@ -488,7 +484,7 @@ pub fn vortex_fastlanes::RLE::clone(&self) -> vortex_fastlanes::RLE impl core::fmt::Debug for vortex_fastlanes::RLE -pub fn vortex_fastlanes::RLE::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_fastlanes::RLE::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::array::vtable::VTable for vortex_fastlanes::RLE @@ -498,57 +494,57 @@ pub type vortex_fastlanes::RLE::OperationsVTable = vortex_fastlanes::RLE pub type vortex_fastlanes::RLE::ValidityVTable = vortex_fastlanes::RLE -pub fn vortex_fastlanes::RLE::buffer(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_fastlanes::RLE::buffer(vortex_array::array::view::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_fastlanes::RLE::buffer_name(_array: vortex_array::array::view::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_fastlanes::RLE::buffer_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_fastlanes::RLE::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::RLE::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_fastlanes::RLE::execute(array: vortex_array::array::typed::Array, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_fastlanes::RLE::execute(vortex_array::array::typed::Array, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_fastlanes::RLE::execute_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::RLE::execute_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_fastlanes::RLE::id(&self) -> vortex_array::array::ArrayId -pub fn vortex_fastlanes::RLE::nbuffers(_array: vortex_array::array::view::ArrayView<'_, Self>) -> usize +pub fn vortex_fastlanes::RLE::nbuffers(vortex_array::array::view::ArrayView<'_, Self>) -> usize -pub fn vortex_fastlanes::RLE::reduce_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::RLE::reduce_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_fastlanes::RLE::serialize(array: vortex_array::array::view::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_fastlanes::RLE::serialize(vortex_array::array::view::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_fastlanes::RLE::slot_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_fastlanes::RLE::slot_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_fastlanes::RLE::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_fastlanes::RLE::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::array::vtable::operations::OperationsVTable for vortex_fastlanes::RLE -pub fn vortex_fastlanes::RLE::scalar_at(array: vortex_array::array::view::ArrayView<'_, vortex_fastlanes::RLE>, index: usize, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_fastlanes::RLE::scalar_at(vortex_array::array::view::ArrayView<'_, vortex_fastlanes::RLE>, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::array::vtable::validity::ValidityVTable for vortex_fastlanes::RLE -pub fn vortex_fastlanes::RLE::validity(array: vortex_array::array::view::ArrayView<'_, vortex_fastlanes::RLE>) -> vortex_error::VortexResult +pub fn vortex_fastlanes::RLE::validity(vortex_array::array::view::ArrayView<'_, vortex_fastlanes::RLE>) -> vortex_error::VortexResult impl vortex_array::arrays::slice::SliceKernel for vortex_fastlanes::RLE -pub fn vortex_fastlanes::RLE::slice(array: vortex_array::array::view::ArrayView<'_, Self>, range: core::ops::range::Range, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::RLE::slice(vortex_array::array::view::ArrayView<'_, Self>, core::ops::range::Range, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::kernel::CastReduce for vortex_fastlanes::RLE -pub fn vortex_fastlanes::RLE::cast(array: vortex_array::array::view::ArrayView<'_, Self>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_fastlanes::RLE::cast(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> pub struct vortex_fastlanes::RLEData impl vortex_fastlanes::RLEData -pub fn vortex_fastlanes::RLEData::encode(array: &vortex_array::arrays::primitive::vtable::PrimitiveArray) -> vortex_error::VortexResult +pub fn vortex_fastlanes::RLEData::encode(vortex_array::array::view::ArrayView<'_, vortex_array::arrays::primitive::vtable::Primitive>, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult impl vortex_fastlanes::RLEData -pub unsafe fn vortex_fastlanes::RLEData::new_unchecked(offset: usize) -> Self +pub unsafe fn vortex_fastlanes::RLEData::new_unchecked(usize) -> Self pub fn vortex_fastlanes::RLEData::offset(&self) -> usize -pub fn vortex_fastlanes::RLEData::try_new(offset: usize) -> vortex_error::VortexResult +pub fn vortex_fastlanes::RLEData::try_new(usize) -> vortex_error::VortexResult impl core::clone::Clone for vortex_fastlanes::RLEData @@ -556,21 +552,19 @@ pub fn vortex_fastlanes::RLEData::clone(&self) -> vortex_fastlanes::RLEData impl core::fmt::Debug for vortex_fastlanes::RLEData -pub fn vortex_fastlanes::RLEData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_fastlanes::RLEData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_fastlanes::RLEData -pub fn vortex_fastlanes::RLEData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_fastlanes::RLEData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::hash::ArrayEq for vortex_fastlanes::RLEData -pub fn vortex_fastlanes::RLEData::array_eq(&self, other: &Self, _precision: vortex_array::hash::Precision) -> bool +pub fn vortex_fastlanes::RLEData::array_eq(&self, &Self, vortex_array::hash::Precision) -> bool impl vortex_array::hash::ArrayHash for vortex_fastlanes::RLEData -pub fn vortex_fastlanes::RLEData::array_hash(&self, state: &mut H, _precision: vortex_array::hash::Precision) - -pub static vortex_fastlanes::USE_EXPERIMENTAL_PATCHES: std::sync::lazy_lock::LazyLock +pub fn vortex_fastlanes::RLEData::array_hash(&self, &mut H, vortex_array::hash::Precision) pub trait vortex_fastlanes::BitPackedArrayExt: vortex_fastlanes::BitPackedArraySlotsExt @@ -588,8 +582,6 @@ pub fn vortex_fastlanes::BitPackedArrayExt::unpacked_chunks vortex_array::validity::Validity -pub fn vortex_fastlanes::BitPackedArrayExt::validity_mask(&self) -> vortex_mask::Mask - impl> vortex_fastlanes::BitPackedArrayExt for T pub fn T::bit_width(&self) -> u8 @@ -606,8 +598,6 @@ pub fn T::unpacked_chunks(&self) -> pub fn T::validity(&self) -> vortex_array::validity::Validity -pub fn T::validity_mask(&self) -> vortex_mask::Mask - pub trait vortex_fastlanes::BitPackedArraySlotsExt: vortex_array::array::typed::TypedArrayRef pub fn vortex_fastlanes::BitPackedArraySlotsExt::patch_chunk_offsets(&self) -> core::option::Option<&vortex_array::array::erased::ArrayRef> @@ -652,7 +642,7 @@ pub fn vortex_fastlanes::RLEArrayExt::offset(&self) -> usize pub fn vortex_fastlanes::RLEArrayExt::values(&self) -> &vortex_array::array::erased::ArrayRef -pub fn vortex_fastlanes::RLEArrayExt::values_idx_offset(&self, chunk_idx: usize) -> usize +pub fn vortex_fastlanes::RLEArrayExt::values_idx_offset(&self, usize, &mut vortex_array::executor::ExecutionCtx) -> usize pub fn vortex_fastlanes::RLEArrayExt::values_idx_offsets(&self) -> &vortex_array::array::erased::ArrayRef @@ -664,13 +654,13 @@ pub fn T::offset(&self) -> usize pub fn T::values(&self) -> &vortex_array::array::erased::ArrayRef -pub fn T::values_idx_offset(&self, chunk_idx: usize) -> usize +pub fn T::values_idx_offset(&self, usize, &mut vortex_array::executor::ExecutionCtx) -> usize pub fn T::values_idx_offsets(&self) -> &vortex_array::array::erased::ArrayRef -pub fn vortex_fastlanes::delta_compress(array: &vortex_array::arrays::primitive::vtable::PrimitiveArray, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult<(vortex_array::arrays::primitive::vtable::PrimitiveArray, vortex_array::arrays::primitive::vtable::PrimitiveArray)> +pub fn vortex_fastlanes::delta_compress(&vortex_array::arrays::primitive::vtable::PrimitiveArray, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult<(vortex_array::arrays::primitive::vtable::PrimitiveArray, vortex_array::arrays::primitive::vtable::PrimitiveArray)> -pub fn vortex_fastlanes::initialize(session: &vortex_session::VortexSession) +pub fn vortex_fastlanes::initialize(&vortex_session::VortexSession) pub type vortex_fastlanes::BitPackedArray = vortex_array::array::typed::Array diff --git a/encodings/fastlanes/src/bit_transpose/aarch64.rs b/encodings/fastlanes/src/bit_transpose/aarch64.rs index 7a854b43a3f..85dbb1e0690 100644 --- a/encodings/fastlanes/src/bit_transpose/aarch64.rs +++ b/encodings/fastlanes/src/bit_transpose/aarch64.rs @@ -113,7 +113,7 @@ static SCATTER_8X8_NEON: [[u8; 16]; 4] = [ ]; /// Perform 8x8 bit transpose on two u64s packed in a `uint64x2_t`. -#[allow(unsafe_op_in_unsafe_fn)] +#[expect(unsafe_op_in_unsafe_fn)] unsafe fn bit_transpose_8x8_neon(mut v: uint64x2_t) -> uint64x2_t { let mask1 = vdupq_n_u64(TRANSPOSE_2X2); let t = vandq_u64(veorq_u64(v, vshrq_n_u64::<7>(v)), mask1); @@ -137,7 +137,7 @@ unsafe fn bit_transpose_8x8_neon(mut v: uint64x2_t) -> uint64x2_t { /// /// # Safety /// Requires `AArch64` with NEON (always available on `AArch64`). -#[allow(unsafe_op_in_unsafe_fn)] +#[expect(unsafe_op_in_unsafe_fn)] #[inline(never)] pub unsafe fn transpose_bits_neon(input: &[u8; 128], output: &mut [u8; 128]) { // Load all 128 input bytes into two uint8x16x4_t tables (64 bytes each) @@ -197,7 +197,7 @@ pub unsafe fn transpose_bits_neon(input: &[u8; 128], output: &mut [u8; 128]) { /// /// # Safety /// Requires `AArch64` with NEON (always available on `AArch64`). -#[allow(unsafe_op_in_unsafe_fn)] +#[expect(unsafe_op_in_unsafe_fn)] #[inline(never)] pub unsafe fn untranspose_bits_neon(input: &[u8; 128], output: &mut [u8; 128]) { // Load scatter indices (SCATTER_8X8 is self-inverse, so same table un-scatters) diff --git a/encodings/fastlanes/src/bit_transpose/mod.rs b/encodings/fastlanes/src/bit_transpose/mod.rs index 01e5d4a0b1c..99b01bc242c 100644 --- a/encodings/fastlanes/src/bit_transpose/mod.rs +++ b/encodings/fastlanes/src/bit_transpose/mod.rs @@ -98,7 +98,7 @@ pub fn untranspose_bits(input: &[u8; 128], output: &mut [u8; 128]) { } #[cfg(test)] -#[allow(clippy::cast_possible_truncation)] +#[expect(clippy::cast_possible_truncation)] fn generate_test_data(seed: u8) -> [u8; 128] { let mut data = [0u8; 128]; for (i, byte) in data.iter_mut().enumerate() { diff --git a/encodings/fastlanes/src/bit_transpose/x86.rs b/encodings/fastlanes/src/bit_transpose/x86.rs index 6277d76137f..9eaef248d28 100644 --- a/encodings/fastlanes/src/bit_transpose/x86.rs +++ b/encodings/fastlanes/src/bit_transpose/x86.rs @@ -45,8 +45,7 @@ pub fn has_vbmi() -> bool { /// Requires BMI2 support. Check with `has_bmi2()` before calling. #[target_feature(enable = "bmi2")] #[inline(never)] -#[allow(clippy::too_many_lines)] -#[allow(unsafe_op_in_unsafe_fn)] +#[expect(clippy::too_many_lines)] pub unsafe fn transpose_bits_bmi2(input: &[u8; 128], output: &mut [u8; 128]) { // Helper to gather 8 bytes at stride 16 into a u64 fn gather(input: &[u8; 128], base: usize) -> u64 { @@ -233,8 +232,7 @@ pub unsafe fn transpose_bits_bmi2(input: &[u8; 128], output: &mut [u8; 128]) { /// Requires BMI2 support. Check with `has_bmi2()` before calling. #[target_feature(enable = "bmi2")] #[inline(never)] -#[allow(clippy::too_many_lines)] -#[allow(unsafe_op_in_unsafe_fn)] +#[expect(clippy::too_many_lines)] pub unsafe fn untranspose_bits_bmi2(input: &[u8; 128], output: &mut [u8; 128]) { // Helper: scatter a u64 to 8 output bytes at stride 16 fn scatter(output: &mut [u8; 128], base: usize, val: u64) { @@ -492,9 +490,9 @@ static SCATTER_8X8: [u8; 64] = [ /// Requires AVX-512F, AVX-512BW, and AVX-512VBMI support. #[target_feature(enable = "avx512f", enable = "avx512bw", enable = "avx512vbmi")] #[inline(never)] -#[allow(clippy::cast_possible_wrap)] -#[allow(clippy::cast_ptr_alignment)] -#[allow(unsafe_op_in_unsafe_fn)] +#[expect(clippy::cast_possible_wrap)] +#[expect(clippy::cast_ptr_alignment)] +#[expect(unsafe_op_in_unsafe_fn)] pub unsafe fn transpose_bits_vbmi(input: &[u8; 128], output: &mut [u8; 128]) { // Load all 128 input bytes into two ZMM registers let in_lo = _mm512_loadu_si512(input.as_ptr().cast::<__m512i>()); @@ -547,9 +545,9 @@ pub unsafe fn transpose_bits_vbmi(input: &[u8; 128], output: &mut [u8; 128]) { /// Requires AVX-512F, AVX-512BW, and AVX-512VBMI support. #[target_feature(enable = "avx512f", enable = "avx512bw", enable = "avx512vbmi")] #[inline(never)] -#[allow(clippy::cast_possible_wrap)] -#[allow(clippy::cast_ptr_alignment)] -#[allow(unsafe_op_in_unsafe_fn)] +#[expect(clippy::cast_possible_wrap)] +#[expect(clippy::cast_ptr_alignment)] +#[expect(unsafe_op_in_unsafe_fn)] pub unsafe fn untranspose_bits_vbmi(input: &[u8; 128], output: &mut [u8; 128]) { // For untranspose, we gather consecutive bytes from transposed layout, // then scatter back to stride-16 positions diff --git a/encodings/fastlanes/src/bitpacking/array/bitpack_compress.rs b/encodings/fastlanes/src/bitpacking/array/bitpack_compress.rs index dfbb0ed7b17..2e517819059 100644 --- a/encodings/fastlanes/src/bitpacking/array/bitpack_compress.rs +++ b/encodings/fastlanes/src/bitpacking/array/bitpack_compress.rs @@ -4,7 +4,10 @@ use fastlanes::BitPacking; use itertools::Itertools; use num_traits::PrimInt; +use vortex_array::ArrayView; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; +use vortex_array::arrays::Primitive; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::primitive::PrimitiveArrayExt; use vortex_array::buffer::BufferHandle; @@ -28,27 +31,31 @@ use crate::BitPacked; use crate::BitPackedArray; use crate::bitpack_decompress; -pub fn bitpack_to_best_bit_width(array: &PrimitiveArray) -> VortexResult { - let bit_width_freq = bit_width_histogram(array)?; +pub fn bitpack_to_best_bit_width( + array: &PrimitiveArray, + ctx: &mut ExecutionCtx, +) -> VortexResult { + let bit_width_freq = bit_width_histogram(array.as_view(), ctx)?; let best_bit_width = find_best_bit_width(array.ptype(), &bit_width_freq)?; - bitpack_encode(array, best_bit_width, Some(&bit_width_freq)) + bitpack_encode(array, best_bit_width, Some(&bit_width_freq), ctx) } -#[allow(unused_comparisons, clippy::absurd_extreme_comparisons)] +#[expect(unused_comparisons, clippy::absurd_extreme_comparisons)] pub fn bitpack_encode( array: &PrimitiveArray, bit_width: u8, bit_width_freq: Option<&[usize]>, + ctx: &mut ExecutionCtx, ) -> VortexResult { let bit_width_freq = match bit_width_freq { Some(freq) => freq, - None => &bit_width_histogram(array)?, + None => &bit_width_histogram(array.as_view(), ctx)?, }; // Check array contains no negative values. if array.ptype().is_signed_int() { let has_negative_values = match_each_integer_ptype!(array.ptype(), |P| { - array.statistics().compute_min::

().unwrap_or_default() < 0 + array.statistics().compute_min::

(ctx).unwrap_or_default() < 0 }); if has_negative_values { vortex_bail!(InvalidArgument: "cannot bitpack_encode array containing negative integers") @@ -68,7 +75,7 @@ pub fn bitpack_encode( // SAFETY: we check that array only contains non-negative values. let packed = unsafe { bitpack_unchecked(array, bit_width) }; let patches = (num_exceptions > 0) - .then(|| gather_patches(array, bit_width, num_exceptions)) + .then(|| gather_patches(array, bit_width, num_exceptions, ctx)) .transpose()? .flatten(); @@ -191,6 +198,7 @@ pub fn gather_patches( parray: &PrimitiveArray, bit_width: u8, num_exceptions_hint: usize, + ctx: &mut ExecutionCtx, ) -> VortexResult> { let patch_validity = match parray.validity()? { Validity::NonNullable => Validity::NonNullable, @@ -198,7 +206,10 @@ pub fn gather_patches( }; let array_len = parray.len(); - let validity_mask = parray.validity_mask()?; + let validity_mask = parray + .as_ref() + .validity()? + .execute_mask(parray.len(), ctx)?; let patches = if array_len < u8::MAX as usize { match_each_integer_ptype!(parray.ptype(), |T| { @@ -289,18 +300,28 @@ where } } -pub fn bit_width_histogram(array: &PrimitiveArray) -> VortexResult> { - match_each_integer_ptype!(array.ptype(), |P| { bit_width_histogram_typed::

(array) }) +pub fn bit_width_histogram( + array: ArrayView<'_, Primitive>, + ctx: &mut ExecutionCtx, +) -> VortexResult> { + match_each_integer_ptype!(array.ptype(), |P| { + bit_width_histogram_typed::

(array, ctx) + }) } fn bit_width_histogram_typed( - array: &PrimitiveArray, + array: ArrayView<'_, Primitive>, + ctx: &mut ExecutionCtx, ) -> VortexResult> { let bit_width: fn(T) -> usize = |v: T| (8 * size_of::()) - (PrimInt::leading_zeros(v) as usize); let mut bit_widths = vec![0usize; size_of::() * 8 + 1]; - match array.validity_mask()?.bit_buffer() { + match array + .validity()? + .execute_mask(array.as_ref().len(), ctx)? + .bit_buffer() + { AllOr::All => { // All values are valid. for v in array.as_slice::() { @@ -370,8 +391,8 @@ pub mod test_harness { use rand::RngExt; use rand::rngs::StdRng; use vortex_array::ArrayRef; + use vortex_array::ExecutionCtx; use vortex_array::IntoArray; - use vortex_array::ToCanonical; use vortex_array::arrays::PrimitiveArray; use vortex_array::validity::Validity; use vortex_buffer::BufferMut; @@ -384,6 +405,7 @@ pub mod test_harness { len: usize, fraction_patches: f64, fraction_null: f64, + ctx: &mut ExecutionCtx, ) -> VortexResult { let values = (0..len) .map(|_| { @@ -396,13 +418,13 @@ pub mod test_harness { .collect::>(); let values = if fraction_null == 0.0 { - values.into_array().to_primitive() + values.into_array().execute::(ctx)? } else { let validity = Validity::from_iter((0..len).map(|_| !rng.random_bool(fraction_null))); PrimitiveArray::new(values, validity) }; - bitpack_encode(&values, 12, None).map(|a| a.into_array()) + bitpack_encode(&values, 12, None, ctx).map(|a| a.into_array()) } } @@ -412,7 +434,6 @@ mod test { use rand::SeedableRng; use rand::rngs::StdRng; - use vortex_array::ToCanonical; use vortex_array::VortexSessionExecute; use vortex_array::arrays::ChunkedArray; use vortex_array::assert_arrays_eq; @@ -421,6 +442,7 @@ mod test { use vortex_array::session::ArraySession; use vortex_buffer::Buffer; use vortex_error::VortexError; + use vortex_error::vortex_err; use vortex_session::VortexSession; use super::*; @@ -444,18 +466,22 @@ mod test { #[test] fn null_patches() { + let mut ctx = SESSION.create_execution_ctx(); let valid_values = (0..24).map(|v| v < 1 << 4).collect::>(); let values = PrimitiveArray::new( (0u32..24).collect::>(), Validity::from_iter(valid_values), ); assert!(values.ptype().is_unsigned_int()); - let compressed = BitPackedData::encode(&values.into_array(), 4).unwrap(); + let compressed = BitPackedData::encode(&values.into_array(), 4, &mut ctx).unwrap(); assert!(compressed.patches().is_none()); assert_eq!( (0..(1 << 4)).collect::>(), compressed - .validity_mask() + .as_ref() + .validity() + .unwrap() + .execute_mask(compressed.as_ref().len(), &mut ctx) .unwrap() .to_bit_buffer() .set_indices() @@ -465,27 +491,30 @@ mod test { #[test] fn compress_signed_fails() { + let mut ctx = SESSION.create_execution_ctx(); let values: Buffer = (-500..500).collect(); let array = PrimitiveArray::new(values, Validity::AllValid); assert!(array.ptype().is_signed_int()); - let err = BitPackedData::encode(&array.into_array(), 1024u32.ilog2() as u8).unwrap_err(); + let err = BitPackedData::encode(&array.into_array(), 1024u32.ilog2() as u8, &mut ctx) + .unwrap_err(); assert!(matches!(err, VortexError::InvalidArgument(_, _))); } #[test] fn canonicalize_chunked_of_bitpacked() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); let mut rng = StdRng::seed_from_u64(0); let chunks = (0..10) - .map(|_| make_array(&mut rng, 100, 0.25, 0.25).unwrap()) + .map(|_| make_array(&mut rng, 100, 0.25, 0.25, &mut ctx).unwrap()) .collect::>(); let chunked = ChunkedArray::from_iter(chunks).into_array(); - let into_ca = chunked.to_primitive(); + let into_ca = chunked.clone().execute::(&mut ctx)?; let mut primitive_builder = PrimitiveBuilder::::with_capacity(chunked.dtype().nullability(), 10 * 100); - chunked.append_to_builder(&mut primitive_builder, &mut SESSION.create_execution_ctx())?; + chunked.append_to_builder(&mut primitive_builder, &mut ctx)?; let ca_into = primitive_builder.finish(); assert_arrays_eq!(into_ca, ca_into); @@ -501,7 +530,8 @@ mod test { } #[test] - fn test_chunk_offsets() { + fn test_chunk_offsets() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); let patch_value = 1u32 << 20; let patch_indices = [100usize, 200, 3000, 3100]; let mut values = vec![0u32; 4096usize]; @@ -511,20 +541,29 @@ mod test { .for_each(|&idx| values[idx] = patch_value); let array = PrimitiveArray::from_iter(values); - let bitpacked = bitpack_encode(&array, 4, None).unwrap(); - - let patches = bitpacked.patches().unwrap(); - let chunk_offsets = patches.chunk_offsets().as_ref().unwrap().to_primitive(); + let bitpacked = bitpack_encode(&array, 4, None, &mut ctx)?; + + let patches = bitpacked + .patches() + .ok_or_else(|| vortex_err!("expected patches"))?; + let chunk_offsets = patches + .chunk_offsets() + .as_ref() + .ok_or_else(|| vortex_err!("expected chunk offsets"))? + .clone() + .execute::(&mut ctx)?; // chunk 0 (0-1023): patches at 100, 200 -> starts at patch index 0 // chunk 1 (1024-2047): no patches -> points to patch index 2 // chunk 2 (2048-3071): patch at 3000 -> starts at patch index 2 // chunk 3 (3072-4095): patch at 3100 -> starts at patch index 3 assert_arrays_eq!(chunk_offsets, PrimitiveArray::from_iter([0u64, 2, 2, 3])); + Ok(()) } #[test] - fn test_chunk_offsets_no_patches_in_middle() { + fn test_chunk_offsets_no_patches_in_middle() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); let patch_value = 1u32 << 20; let patch_indices = [100usize, 200, 2500]; let mut values = vec![0u32; 3072usize]; @@ -534,16 +573,25 @@ mod test { .for_each(|&idx| values[idx] = patch_value); let array = PrimitiveArray::from_iter(values); - let bitpacked = bitpack_encode(&array, 4, None).unwrap(); - - let patches = bitpacked.patches().unwrap(); - let chunk_offsets = patches.chunk_offsets().as_ref().unwrap().to_primitive(); + let bitpacked = bitpack_encode(&array, 4, None, &mut ctx)?; + + let patches = bitpacked + .patches() + .ok_or_else(|| vortex_err!("expected patches"))?; + let chunk_offsets = patches + .chunk_offsets() + .as_ref() + .ok_or_else(|| vortex_err!("expected chunk offsets"))? + .clone() + .execute::(&mut ctx)?; assert_arrays_eq!(chunk_offsets, PrimitiveArray::from_iter([0u64, 2, 2])); + Ok(()) } #[test] - fn test_chunk_offsets_trailing_empty_chunks() { + fn test_chunk_offsets_trailing_empty_chunks() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); let patch_value = 1u32 << 20; let patch_indices = [100usize, 200, 1500]; let mut values = vec![0u32; 5120usize]; @@ -553,10 +601,17 @@ mod test { .for_each(|&idx| values[idx] = patch_value); let array = PrimitiveArray::from_iter(values); - let bitpacked = bitpack_encode(&array, 4, None).unwrap(); - - let patches = bitpacked.patches().unwrap(); - let chunk_offsets = patches.chunk_offsets().as_ref().unwrap().to_primitive(); + let bitpacked = bitpack_encode(&array, 4, None, &mut ctx)?; + + let patches = bitpacked + .patches() + .ok_or_else(|| vortex_err!("expected patches"))?; + let chunk_offsets = patches + .chunk_offsets() + .as_ref() + .ok_or_else(|| vortex_err!("expected chunk offsets"))? + .clone() + .execute::(&mut ctx)?; // chunk 0 (0-1023): patches at 100, 200 -> starts at patch index 0 // chunk 1 (1024-2047): patch at 1500 -> starts at patch index 2 @@ -564,10 +619,12 @@ mod test { // chunk 3 (3072-4095): no patches -> points to patch index 3 (remaining chunks filled) // chunk 4 (4096-5119): no patches -> points to patch index 3 (remaining chunks filled) assert_arrays_eq!(chunk_offsets, PrimitiveArray::from_iter([0u64, 2, 3, 3, 3])); + Ok(()) } #[test] - fn test_chunk_offsets_single_chunk() { + fn test_chunk_offsets_single_chunk() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); let patch_value = 1u32 << 20; let patch_indices = [100usize, 200]; let mut values = vec![0u32; 500usize]; @@ -577,12 +634,20 @@ mod test { .for_each(|&idx| values[idx] = patch_value); let array = PrimitiveArray::from_iter(values); - let bitpacked = bitpack_encode(&array, 4, None).unwrap(); - - let patches = bitpacked.patches().unwrap(); - let chunk_offsets = patches.chunk_offsets().as_ref().unwrap().to_primitive(); + let bitpacked = bitpack_encode(&array, 4, None, &mut ctx)?; + + let patches = bitpacked + .patches() + .ok_or_else(|| vortex_err!("expected patches"))?; + let chunk_offsets = patches + .chunk_offsets() + .as_ref() + .ok_or_else(|| vortex_err!("expected chunk offsets"))? + .clone() + .execute::(&mut ctx)?; // Single chunk starting at patch index 0. assert_arrays_eq!(chunk_offsets, PrimitiveArray::from_iter([0u64])); + Ok(()) } } diff --git a/encodings/fastlanes/src/bitpacking/array/bitpack_decompress.rs b/encodings/fastlanes/src/bitpacking/array/bitpack_decompress.rs index 5ed8d6b89fa..6570d26bbc9 100644 --- a/encodings/fastlanes/src/bitpacking/array/bitpack_decompress.rs +++ b/encodings/fastlanes/src/bitpacking/array/bitpack_decompress.rs @@ -58,7 +58,7 @@ pub(crate) fn unpack_into_primitive_builder( // SAFETY: We later initialize the the uninitialized range of values with `copy_from_slice`. unsafe { // Append a dense null Mask. - uninit_range.append_mask(array.validity_mask()); + uninit_range.append_mask(array.validity()?.execute_mask(array.as_ref().len(), ctx)?); } // SAFETY: `decode_into` will initialize all values in this range. @@ -67,8 +67,8 @@ pub(crate) fn unpack_into_primitive_builder( let mut bit_packed_iter = array.unpacked_chunks()?; bit_packed_iter.decode_into(uninit_slice); - if let Some(ref patches) = array.patches() { - apply_patches_to_uninit_range(&mut uninit_range, patches, ctx)?; + if let Some(patches) = array.patches() { + apply_patches_to_uninit_range(&mut uninit_range, &patches, ctx)?; }; // SAFETY: We have set a correct validity mask via `append_mask` with `array.len()` values and @@ -97,7 +97,7 @@ pub fn apply_patches_to_uninit_range_fn T>( let indices = patches.indices().clone().execute::(ctx)?; let values = patches.values().clone().execute::(ctx)?; - assert!(values.all_valid()?, "Patch values must be all valid"); + assert!(values.all_valid(ctx)?, "Patch values must be all valid"); let values = values.as_slice::(); match_each_unsigned_integer_ptype!(indices.ptype(), |P| { @@ -160,7 +160,6 @@ mod tests { use vortex_array::Canonical; use vortex_array::IntoArray; - use vortex_array::ToCanonical; use vortex_array::VortexSessionExecute; use vortex_array::assert_arrays_eq; use vortex_array::dtype::Nullability; @@ -177,7 +176,7 @@ mod tests { use crate::bitpack_compress::bitpack_encode; fn encode(array: &PrimitiveArray, bit_width: u8) -> BitPackedArray { - bitpack_encode(array, bit_width, None).unwrap() + bitpack_encode(array, bit_width, None, &mut SESSION.create_execution_ctx()).unwrap() } static SESSION: LazyLock = @@ -188,8 +187,9 @@ mod tests { } fn compression_roundtrip(n: usize) { + let mut ctx = SESSION.create_execution_ctx(); let values = PrimitiveArray::from_iter((0..n).map(|i| (i % 2047) as u16)); - let compressed = BitPackedData::encode(&values.clone().into_array(), 11).unwrap(); + let compressed = BitPackedData::encode(&values.clone().into_array(), 11, &mut ctx).unwrap(); assert_arrays_eq!(compressed, values); values @@ -219,7 +219,10 @@ mod tests { #[test] fn test_all_zeros() -> VortexResult<()> { - let zeros = buffer![0u16, 0, 0, 0].into_array().to_primitive(); + let mut ctx = SESSION.create_execution_ctx(); + let zeros = buffer![0u16, 0, 0, 0] + .into_array() + .execute::(&mut ctx)?; let bitpacked = encode(&zeros, 0); let actual = unpack(&bitpacked)?; assert_arrays_eq!(actual, PrimitiveArray::from_iter([0u16, 0, 0, 0])); @@ -228,7 +231,10 @@ mod tests { #[test] fn test_simple_patches() -> VortexResult<()> { - let zeros = buffer![0u16, 1, 0, 1].into_array().to_primitive(); + let mut ctx = SESSION.create_execution_ctx(); + let zeros = buffer![0u16, 1, 0, 1] + .into_array() + .execute::(&mut ctx)?; let bitpacked = encode(&zeros, 0); let actual = unpack(&bitpacked)?; assert_arrays_eq!(actual, PrimitiveArray::from_iter([0u16, 1, 0, 1])); @@ -237,7 +243,10 @@ mod tests { #[test] fn test_one_full_chunk() -> VortexResult<()> { - let zeros = BufferMut::from_iter(0u16..1024).into_array().to_primitive(); + let mut ctx = SESSION.create_execution_ctx(); + let zeros = BufferMut::from_iter(0u16..1024) + .into_array() + .execute::(&mut ctx)?; let bitpacked = encode(&zeros, 10); let actual = unpack(&bitpacked)?; assert_arrays_eq!(actual, PrimitiveArray::from_iter(0u16..1024)); @@ -246,9 +255,10 @@ mod tests { #[test] fn test_three_full_chunks_with_patches() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); let zeros = BufferMut::from_iter((5u16..1029).chain(5u16..1029).chain(5u16..1029)) .into_array() - .to_primitive(); + .execute::(&mut ctx)?; let bitpacked = encode(&zeros, 10); assert!(bitpacked.patches().is_some()); let actual = unpack(&bitpacked)?; @@ -261,7 +271,10 @@ mod tests { #[test] fn test_one_full_chunk_and_one_short_chunk_no_patch() -> VortexResult<()> { - let zeros = BufferMut::from_iter(0u16..1025).into_array().to_primitive(); + let mut ctx = SESSION.create_execution_ctx(); + let zeros = BufferMut::from_iter(0u16..1025) + .into_array() + .execute::(&mut ctx)?; let bitpacked = encode(&zeros, 11); assert!(bitpacked.patches().is_none()); let actual = unpack(&bitpacked)?; @@ -271,9 +284,10 @@ mod tests { #[test] fn test_one_full_chunk_and_one_short_chunk_with_patches() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); let zeros = BufferMut::from_iter(512u16..1537) .into_array() - .to_primitive(); + .execute::(&mut ctx)?; let bitpacked = encode(&zeros, 10); assert_eq!(bitpacked.len(), 1025); assert!(bitpacked.patches().is_some()); @@ -284,40 +298,30 @@ mod tests { #[test] fn test_offset_and_short_chunk_and_patches() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); let zeros = BufferMut::from_iter(512u16..1537) .into_array() - .to_primitive(); + .execute::(&mut ctx)?; let bitpacked = encode(&zeros, 10); assert_eq!(bitpacked.len(), 1025); assert!(bitpacked.patches().is_some()); - let slice_ref = bitpacked.into_array().slice(1023..1025).unwrap(); - let actual = { - let mut ctx = SESSION.create_execution_ctx(); - slice_ref - .execute::(&mut ctx) - .unwrap() - .into_primitive() - }; + let slice_ref = bitpacked.into_array().slice(1023..1025)?; + let actual = slice_ref.execute::(&mut ctx)?.into_primitive(); assert_arrays_eq!(actual, PrimitiveArray::from_iter([1535u16, 1536])); Ok(()) } #[test] fn test_offset_and_short_chunk_with_chunks_between_and_patches() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); let zeros = BufferMut::from_iter(512u16..2741) .into_array() - .to_primitive(); + .execute::(&mut ctx)?; let bitpacked = encode(&zeros, 10); assert_eq!(bitpacked.len(), 2229); assert!(bitpacked.patches().is_some()); - let slice_ref = bitpacked.into_array().slice(1023..2049).unwrap(); - let actual = { - let mut ctx = SESSION.create_execution_ctx(); - slice_ref - .execute::(&mut ctx) - .unwrap() - .into_primitive() - }; + let slice_ref = bitpacked.into_array().slice(1023..2049)?; + let actual = slice_ref.execute::(&mut ctx)?.into_primitive(); assert_arrays_eq!( actual, PrimitiveArray::from_iter((1023u16..2049).map(|x| x + 512)) @@ -369,11 +373,12 @@ mod tests { // Verify the validity mask was correctly applied. assert_eq!(result.len(), 5); - assert!(!result.scalar_at(0).unwrap().is_null()); - assert!(result.scalar_at(1).unwrap().is_null()); - assert!(!result.scalar_at(2).unwrap().is_null()); - assert!(!result.scalar_at(3).unwrap().is_null()); - assert!(result.scalar_at(4).unwrap().is_null()); + let mut ctx = SESSION.create_execution_ctx(); + assert!(!result.execute_scalar(0, &mut ctx)?.is_null()); + assert!(result.execute_scalar(1, &mut ctx)?.is_null()); + assert!(!result.execute_scalar(2, &mut ctx)?.is_null()); + assert!(!result.execute_scalar(3, &mut ctx)?.is_null()); + assert!(result.execute_scalar(4, &mut ctx)?.is_null()); Ok(()) } @@ -456,9 +461,10 @@ mod tests { // Verify length. assert_eq!(result.len(), 7); // Validity should be preserved when unpacking. - assert!(!result.scalar_at(0).unwrap().is_null()); - assert!(result.scalar_at(1).unwrap().is_null()); - assert!(!result.scalar_at(2).unwrap().is_null()); + let mut ctx = SESSION.create_execution_ctx(); + assert!(!result.execute_scalar(0, &mut ctx).unwrap().is_null()); + assert!(result.execute_scalar(1, &mut ctx).unwrap().is_null()); + assert!(!result.execute_scalar(2, &mut ctx).unwrap().is_null()); // Test combining patches with nullability. let patch_values = Buffer::from_iter([10u16, 0, 2000, 0, 30, 3000, 0]); @@ -492,10 +498,7 @@ mod tests { let executed = { let mut ctx = SESSION.create_execution_ctx(); - bitpacked - .into_array() - .execute::(&mut ctx) - .unwrap() + bitpacked.into_array().execute::(&mut ctx)? }; assert_eq!( @@ -518,8 +521,7 @@ mod tests { let mut ctx = SESSION.create_execution_ctx(); unpacked_array .into_array() - .execute::(&mut ctx) - .unwrap() + .execute::(&mut ctx)? .into_primitive() }; assert_eq!( @@ -548,13 +550,12 @@ mod tests { // Test with sliced array (offset > 0). let values = PrimitiveArray::from_iter(0u32..2048); let bitpacked = encode(&values, 11); - let slice_ref = bitpacked.into_array().slice(500..1500).unwrap(); + let slice_ref = bitpacked.into_array().slice(500..1500)?; let sliced = { let mut ctx = SESSION.create_execution_ctx(); slice_ref .clone() - .execute::(&mut ctx) - .unwrap() + .execute::(&mut ctx)? .into_primitive() }; @@ -563,7 +564,7 @@ mod tests { let unpacked_array = sliced; let executed = { let mut ctx = SESSION.create_execution_ctx(); - slice_ref.execute::(&mut ctx).unwrap() + slice_ref.execute::(&mut ctx)? }; assert_eq!( diff --git a/encodings/fastlanes/src/bitpacking/array/mod.rs b/encodings/fastlanes/src/bitpacking/array/mod.rs index ca86e78e148..75b75a51e99 100644 --- a/encodings/fastlanes/src/bitpacking/array/mod.rs +++ b/encodings/fastlanes/src/bitpacking/array/mod.rs @@ -6,6 +6,7 @@ use std::fmt::Formatter; use fastlanes::BitPacking; use vortex_array::ArrayRef; +use vortex_array::ExecutionCtx; use vortex_array::TypedArrayRef; use vortex_array::array_slots; use vortex_array::arrays::Primitive; @@ -254,12 +255,16 @@ impl BitPackedData { /// /// If the requested bit-width for packing is larger than the array's native width, an /// error will be returned. - pub fn encode(array: &ArrayRef, bit_width: u8) -> VortexResult { + pub fn encode( + array: &ArrayRef, + bit_width: u8, + ctx: &mut ExecutionCtx, + ) -> VortexResult { let parray: PrimitiveArray = array .clone() .try_downcast::() .map_err(|a| vortex_err!(InvalidArgument: "Bitpacking can only encode primitive arrays, got {}", a.encoding_id()))?; - bitpack_encode(&parray, bit_width, None) + bitpack_encode(&parray, bit_width, None, ctx) } /// Calculate the maximum value that **can** be contained by this array, given its bit-width. @@ -311,15 +316,7 @@ pub trait BitPackedArrayExt: BitPackedArraySlotsExt { #[inline] fn validity(&self) -> Validity { - child_to_validity( - &self.validity_child().cloned(), - self.as_ref().dtype().nullability(), - ) - } - - #[inline] - fn validity_mask(&self) -> vortex_mask::Mask { - self.validity().to_mask(self.as_ref().len()) + child_to_validity(self.validity_child(), self.as_ref().dtype().nullability()) } #[inline] @@ -337,17 +334,25 @@ impl> BitPackedArrayExt for T {} #[cfg(test)] mod test { + use std::sync::LazyLock; + use vortex_array::IntoArray; - use vortex_array::ToCanonical; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; + use vortex_array::session::ArraySession; use vortex_buffer::Buffer; + use vortex_session::VortexSession; use crate::BitPackedData; use crate::bitpacking::array::BitPackedArrayExt; + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); + #[test] fn test_encode() { + let mut ctx = SESSION.create_execution_ctx(); let values = [ Some(1u64), None, @@ -358,30 +363,42 @@ mod test { Some(u64::MAX), ]; let uncompressed = PrimitiveArray::from_option_iter(values); - let packed = BitPackedData::encode(&uncompressed.into_array(), 1).unwrap(); + let packed = BitPackedData::encode(&uncompressed.into_array(), 1, &mut ctx).unwrap(); let expected = PrimitiveArray::from_option_iter(values); - assert_arrays_eq!(packed.as_array().to_primitive(), expected); + let packed_primitive = packed + .as_array() + .clone() + .execute::(&mut ctx) + .unwrap(); + assert_arrays_eq!(packed_primitive, expected); } #[test] fn test_encode_too_wide() { + let mut ctx = SESSION.create_execution_ctx(); let values = [Some(1u8), None, Some(1), None, Some(1), None]; let uncompressed = PrimitiveArray::from_option_iter(values); - let _packed = BitPackedData::encode(&uncompressed.clone().into_array(), 8) + let _packed = BitPackedData::encode(&uncompressed.clone().into_array(), 8, &mut ctx) .expect_err("Cannot pack value into the same width"); - let _packed = BitPackedData::encode(&uncompressed.into_array(), 9) + let _packed = BitPackedData::encode(&uncompressed.into_array(), 9, &mut ctx) .expect_err("Cannot pack value into larger width"); } #[test] fn signed_with_patches() { + let mut ctx = SESSION.create_execution_ctx(); let values: Buffer = (0i32..=512).collect(); let parray = values.clone().into_array(); - let packed_with_patches = BitPackedData::encode(&parray, 9).unwrap(); + let packed_with_patches = BitPackedData::encode(&parray, 9, &mut ctx).unwrap(); assert!(packed_with_patches.patches().is_some()); + let packed_primitive = packed_with_patches + .as_array() + .clone() + .execute::(&mut ctx) + .unwrap(); assert_arrays_eq!( - packed_with_patches.as_array().to_primitive(), + packed_primitive, PrimitiveArray::new(values, vortex_array::validity::Validity::NonNullable) ); } diff --git a/encodings/fastlanes/src/bitpacking/array/unpack_iter.rs b/encodings/fastlanes/src/bitpacking/array/unpack_iter.rs index 80a73f82678..2f7187d26f1 100644 --- a/encodings/fastlanes/src/bitpacking/array/unpack_iter.rs +++ b/encodings/fastlanes/src/bitpacking/array/unpack_iter.rs @@ -55,12 +55,14 @@ impl> UnpackStrategy for BitPackingStr /// #[gat(Item)] /// use lending_iterator::prelude::LendingIterator; /// use vortex_array::IntoArray; +/// use vortex_array::VortexSessionExecute; /// use vortex_buffer::buffer; /// use vortex_fastlanes::BitPackedData; /// use vortex_fastlanes::BitPackedArrayExt; /// use vortex_fastlanes::unpack_iter::BitUnpackedChunks; /// -/// let array = BitPackedData::encode(&buffer![2, 3, 4, 5].into_array(), 2).unwrap(); +/// let mut ctx = vortex_array::LEGACY_SESSION.create_execution_ctx(); +/// let array = BitPackedData::encode(&buffer![2, 3, 4, 5].into_array(), 2, &mut ctx).unwrap(); /// let mut unpacked_chunks: BitUnpackedChunks = array.unpacked_chunks().unwrap(); /// /// if let Some(header) = unpacked_chunks.initial() { diff --git a/encodings/fastlanes/src/bitpacking/compute/cast.rs b/encodings/fastlanes/src/bitpacking/compute/cast.rs index fbcd48ce75b..86bd701e2ee 100644 --- a/encodings/fastlanes/src/bitpacking/compute/cast.rs +++ b/encodings/fastlanes/src/bitpacking/compute/cast.rs @@ -3,48 +3,67 @@ use vortex_array::ArrayRef; use vortex_array::ArrayView; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; use vortex_array::builtins::ArrayBuiltins; use vortex_array::dtype::DType; -use vortex_array::patches::Patches; +use vortex_array::scalar_fn::fns::cast::CastKernel; use vortex_array::scalar_fn::fns::cast::CastReduce; +use vortex_array::validity::Validity; use vortex_error::VortexResult; use crate::bitpacking::BitPacked; use crate::bitpacking::array::BitPackedArrayExt; + +fn build_with_validity( + array: ArrayView<'_, BitPacked>, + dtype: &DType, + new_validity: Validity, +) -> VortexResult { + Ok(BitPacked::try_new( + array.packed().clone(), + dtype.as_ptype(), + new_validity, + array + .patches() + .map(|patches| patches.map_values(|values| values.cast(dtype.clone()))) + .transpose()?, + array.bit_width(), + array.len(), + array.offset(), + )? + .into_array()) +} + impl CastReduce for BitPacked { fn cast(array: ArrayView<'_, Self>, dtype: &DType) -> VortexResult> { - if array.dtype().eq_ignore_nullability(dtype) { - let new_validity = array - .validity()? - .cast_nullability(dtype.nullability(), array.len())?; - return Ok(Some( - BitPacked::try_new( - array.packed().clone(), - dtype.as_ptype(), - new_validity, - array - .patches() - .map(|patches| { - let new_values = patches.values().cast(dtype.clone())?; - Patches::new( - patches.array_len(), - patches.offset(), - patches.indices().clone(), - new_values, - patches.chunk_offsets().clone(), - ) - }) - .transpose()?, - array.bit_width(), - array.len(), - array.offset(), - )? - .into_array(), - )); + if !array.dtype().eq_ignore_nullability(dtype) { + return Ok(None); } + let Some(new_validity) = array + .validity()? + .trivial_cast_nullability(dtype.nullability(), array.len())? + else { + return Ok(None); + }; + build_with_validity(array, dtype, new_validity).map(Some) + } +} - Ok(None) +impl CastKernel for BitPacked { + fn cast( + array: ArrayView<'_, Self>, + dtype: &DType, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + if !array.dtype().eq_ignore_nullability(dtype) { + return Ok(None); + } + let new_validity = + array + .validity()? + .cast_nullability(dtype.nullability(), array.len(), ctx)?; + build_with_validity(array, dtype, new_validity).map(Some) } } @@ -53,6 +72,8 @@ mod tests { use rstest::rstest; use vortex_array::ArrayRef; use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; use vortex_array::builtins::ArrayBuiltins; @@ -66,7 +87,7 @@ mod tests { use crate::BitPackedData; fn bp(array: &ArrayRef, bit_width: u8) -> BitPackedArray { - BitPackedData::encode(array, bit_width).unwrap() + BitPackedData::encode(array, bit_width, &mut LEGACY_SESSION.create_execution_ctx()).unwrap() } #[test] diff --git a/encodings/fastlanes/src/bitpacking/compute/filter.rs b/encodings/fastlanes/src/bitpacking/compute/filter.rs index aa3ca07d0b0..aab597e9ff3 100644 --- a/encodings/fastlanes/src/bitpacking/compute/filter.rs +++ b/encodings/fastlanes/src/bitpacking/compute/filter.rs @@ -178,7 +178,8 @@ fn filter_with_indices( #[cfg(test)] mod test { use vortex_array::IntoArray as _; - use vortex_array::ToCanonical; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; use vortex_array::compute::conformance::filter::test_filter_conformance; @@ -192,9 +193,10 @@ mod test { #[test] fn take_indices() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Create a u8 array modulo 63. let unpacked = PrimitiveArray::from_iter((0..4096).map(|i| (i % 63) as u8)); - let bitpacked = BitPackedData::encode(&unpacked.into_array(), 6).unwrap(); + let bitpacked = BitPackedData::encode(&unpacked.into_array(), 6, &mut ctx).unwrap(); let mask = Mask::from_indices(bitpacked.len(), vec![0, 125, 2047, 2049, 2151, 2790]); @@ -207,9 +209,10 @@ mod test { #[test] fn take_sliced_indices() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Create a u8 array modulo 63. let unpacked = PrimitiveArray::from_iter((0..4096).map(|i| (i % 63) as u8)); - let bitpacked = BitPackedData::encode(&unpacked.into_array(), 6).unwrap(); + let bitpacked = BitPackedData::encode(&unpacked.into_array(), 6, &mut ctx).unwrap(); let sliced = bitpacked.slice(128..2050).unwrap(); let mask = Mask::from_indices(sliced.len(), vec![1919, 1921]); @@ -220,26 +223,30 @@ mod test { #[test] fn filter_bitpacked() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let unpacked = PrimitiveArray::from_iter((0..4096).map(|i| (i % 63) as u8)); - let bitpacked = BitPackedData::encode(&unpacked.into_array(), 6).unwrap(); + let bitpacked = BitPackedData::encode(&unpacked.into_array(), 6, &mut ctx).unwrap(); let filtered = bitpacked .filter(Mask::from_indices(4096, (0..1024).collect())) .unwrap(); + let filtered_prim = filtered.execute::(&mut ctx).unwrap(); assert_arrays_eq!( - filtered.to_primitive(), + filtered_prim, PrimitiveArray::from_iter((0..1024).map(|i| (i % 63) as u8)) ); } #[test] fn filter_bitpacked_signed() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values: Buffer = (0..500).collect(); let unpacked = PrimitiveArray::new(values.clone(), Validity::NonNullable); - let bitpacked = BitPackedData::encode(&unpacked.into_array(), 9).unwrap(); + let bitpacked = BitPackedData::encode(&unpacked.into_array(), 9, &mut ctx).unwrap(); let filtered = bitpacked .filter(Mask::from_indices(values.len(), (0..250).collect())) .unwrap() - .to_primitive(); + .execute::(&mut ctx) + .unwrap(); assert_arrays_eq!( filtered, @@ -249,19 +256,20 @@ mod test { #[test] fn test_filter_bitpacked_conformance() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Test with u8 values let unpacked = buffer![1u8, 2, 3, 4, 5].into_array(); - let bitpacked = BitPackedData::encode(&unpacked, 3).unwrap(); + let bitpacked = BitPackedData::encode(&unpacked, 3, &mut ctx).unwrap(); test_filter_conformance(&bitpacked.into_array()); // Test with u32 values let unpacked = buffer![100u32, 200, 300, 400, 500].into_array(); - let bitpacked = BitPackedData::encode(&unpacked, 9).unwrap(); + let bitpacked = BitPackedData::encode(&unpacked, 9, &mut ctx).unwrap(); test_filter_conformance(&bitpacked.into_array()); // Test with nullable values let unpacked = PrimitiveArray::from_option_iter([Some(1u16), None, Some(3), Some(4), None]); - let bitpacked = BitPackedData::encode(&unpacked.into_array(), 3).unwrap(); + let bitpacked = BitPackedData::encode(&unpacked.into_array(), 3, &mut ctx).unwrap(); test_filter_conformance(&bitpacked.into_array()); } @@ -272,11 +280,12 @@ mod test { /// This test ensures that the type handling is correct. #[test] fn filter_bitpacked_signed_with_patches() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Create signed integer values where some exceed the bit width (causing patches). // Values 0-127 fit in 7 bits, but 1000 and 2000 do not. let values: Vec = vec![0, 10, 1000, 20, 30, 2000, 40, 50, 60, 70]; let unpacked = PrimitiveArray::from_iter(values.clone()); - let bitpacked = BitPackedData::encode(&unpacked.into_array(), 7).unwrap(); + let bitpacked = BitPackedData::encode(&unpacked.into_array(), 7, &mut ctx).unwrap(); assert!( bitpacked.patches().is_some(), "Expected patches for values exceeding bit width" @@ -286,7 +295,8 @@ mod test { let filtered = bitpacked .filter(Mask::from_indices(values.len(), vec![0, 2, 5, 9])) .unwrap() - .to_primitive(); + .execute::(&mut ctx) + .unwrap(); assert_arrays_eq!(filtered, PrimitiveArray::from_iter([0i32, 1000, 2000, 70])); } @@ -297,6 +307,7 @@ mod test { /// that doesn't fully decompress the array first. #[test] fn filter_bitpacked_signed_with_patches_low_selectivity() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Create a larger array with signed integers and some patches. let values: Vec = (0..1000) .map(|i| { @@ -308,7 +319,7 @@ mod test { }) .collect(); let unpacked = PrimitiveArray::from_iter(values.clone()); - let bitpacked = BitPackedData::encode(&unpacked.into_array(), 7).unwrap(); + let bitpacked = BitPackedData::encode(&unpacked.into_array(), 7, &mut ctx).unwrap(); assert!( bitpacked.patches().is_some(), "Expected patches for values exceeding bit width" @@ -319,7 +330,8 @@ mod test { let filtered = bitpacked .filter(Mask::from_indices(values.len(), indices)) .unwrap() - .to_primitive(); + .execute::(&mut ctx) + .unwrap(); let expected: Vec = values[0..20].to_vec(); assert_arrays_eq!(filtered, PrimitiveArray::from_iter(expected)); diff --git a/encodings/fastlanes/src/bitpacking/compute/is_constant.rs b/encodings/fastlanes/src/bitpacking/compute/is_constant.rs index 900284e926d..0a743f9269b 100644 --- a/encodings/fastlanes/src/bitpacking/compute/is_constant.rs +++ b/encodings/fastlanes/src/bitpacking/compute/is_constant.rs @@ -8,7 +8,6 @@ use lending_iterator::LendingIterator; use vortex_array::ArrayRef; use vortex_array::ArrayView; use vortex_array::ExecutionCtx; -use vortex_array::ToCanonical; use vortex_array::aggregate_fn::AggregateFnRef; use vortex_array::aggregate_fn::fns::is_constant::IsConstant; use vortex_array::aggregate_fn::fns::is_constant::primitive::IS_CONST_LANE_WIDTH; @@ -34,7 +33,7 @@ impl DynAggregateKernel for BitPackedIsConstantKernel { &self, aggregate_fn: &AggregateFnRef, batch: &ArrayRef, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult> { if !aggregate_fn.is::() { return Ok(None); @@ -45,23 +44,27 @@ impl DynAggregateKernel for BitPackedIsConstantKernel { }; let result = match_each_integer_ptype!(array.dtype().as_ptype(), |P| { - bitpacked_is_constant::() }>(array)? + bitpacked_is_constant::() }>(array, ctx)? }); - Ok(Some(IsConstant::make_partial(batch, result)?)) + Ok(Some(IsConstant::make_partial(batch, result, ctx)?)) } } fn bitpacked_is_constant( array: ArrayView<'_, BitPacked>, + ctx: &mut ExecutionCtx, ) -> VortexResult { let mut bit_unpack_iterator = array.unpacked_chunks::()?; - let patches = array.patches().map(|p| { - let values = p.values().to_primitive(); - let indices = p.indices().to_primitive(); - let offset = p.offset(); - (indices, values, offset) - }); + let patches = array + .patches() + .map(|p| -> VortexResult<_> { + let values = p.values().clone().execute::(ctx)?; + let indices = p.indices().clone().execute::(ctx)?; + let offset = p.offset(); + Ok((indices, values, offset)) + }) + .transpose()?; let mut header_constant_value = None; let mut current_idx = 0; @@ -192,8 +195,8 @@ mod tests { #[test] fn is_constant_with_patches() -> VortexResult<()> { - let array = BitPackedData::encode(&buffer![4; 1025].into_array(), 2)?; let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let array = BitPackedData::encode(&buffer![4; 1025].into_array(), 2, &mut ctx)?; assert!(is_constant(&array.into_array(), &mut ctx)?); Ok(()) } diff --git a/encodings/fastlanes/src/bitpacking/compute/mod.rs b/encodings/fastlanes/src/bitpacking/compute/mod.rs index f404102c019..2501d952356 100644 --- a/encodings/fastlanes/src/bitpacking/compute/mod.rs +++ b/encodings/fastlanes/src/bitpacking/compute/mod.rs @@ -42,6 +42,8 @@ fn chunked_indices( mod tests { use rstest::rstest; use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::compute::conformance::binary_numeric::test_binary_numeric_array; use vortex_array::compute::conformance::consistency::test_array_consistency; @@ -51,7 +53,13 @@ mod tests { use crate::bitpacking::compute::chunked_indices; fn bp(array: &PrimitiveArray, bit_width: u8) -> BitPackedArray { - bitpack_encode(array, bit_width, None).unwrap() + bitpack_encode( + array, + bit_width, + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap() } #[test] diff --git a/encodings/fastlanes/src/bitpacking/compute/slice.rs b/encodings/fastlanes/src/bitpacking/compute/slice.rs index 93cb60353c6..e6e51c57591 100644 --- a/encodings/fastlanes/src/bitpacking/compute/slice.rs +++ b/encodings/fastlanes/src/bitpacking/compute/slice.rs @@ -46,6 +46,8 @@ impl SliceReduce for BitPacked { #[cfg(test)] mod tests { use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::SliceArray; use vortex_error::VortexResult; @@ -55,8 +57,9 @@ mod tests { #[test] fn test_reduce_parent_returns_bitpacked_slice() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values = PrimitiveArray::from_iter(0u32..2048); - let bitpacked = bitpack_encode(&values, 11, None)?; + let bitpacked = bitpack_encode(&values, 11, None, &mut ctx)?; let slice_array = SliceArray::new(bitpacked.clone().into_array(), 500..1500); diff --git a/encodings/fastlanes/src/bitpacking/compute/take.rs b/encodings/fastlanes/src/bitpacking/compute/take.rs index 9422b44be1b..a4c148c5717 100644 --- a/encodings/fastlanes/src/bitpacking/compute/take.rs +++ b/encodings/fastlanes/src/bitpacking/compute/take.rs @@ -158,7 +158,7 @@ fn take_primitive( } #[cfg(test)] -#[allow(clippy::cast_possible_truncation)] +#[expect(clippy::cast_possible_truncation)] mod test { use rand::RngExt; use rand::distr::Uniform; @@ -166,7 +166,6 @@ mod test { use rstest::rstest; use vortex_array::IntoArray; use vortex_array::LEGACY_SESSION; - use vortex_array::ToCanonical; use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; @@ -181,11 +180,12 @@ mod test { #[test] fn take_indices() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let indices = buffer![0, 125, 2047, 2049, 2151, 2790].into_array(); // Create a u8 array modulo 63. let unpacked = PrimitiveArray::from_iter((0..4096).map(|i| (i % 63) as u8)); - let bitpacked = BitPackedData::encode(&unpacked.into_array(), 6).unwrap(); + let bitpacked = BitPackedData::encode(&unpacked.into_array(), 6, &mut ctx).unwrap(); let primitive_result = bitpacked.take(indices).unwrap(); assert_arrays_eq!( @@ -196,8 +196,9 @@ mod test { #[test] fn take_with_patches() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let unpacked = Buffer::from_iter(0u32..1024).into_array(); - let bitpacked = BitPackedData::encode(&unpacked, 2).unwrap(); + let bitpacked = BitPackedData::encode(&unpacked, 2, &mut ctx).unwrap(); let indices = buffer![0, 2, 4, 6].into_array(); @@ -207,11 +208,12 @@ mod test { #[test] fn take_sliced_indices() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let indices = buffer![1919, 1921].into_array(); // Create a u8 array modulo 63. let unpacked = PrimitiveArray::from_iter((0..4096).map(|i| (i % 63) as u8)); - let bitpacked = BitPackedData::encode(&unpacked.into_array(), 6).unwrap(); + let bitpacked = BitPackedData::encode(&unpacked.into_array(), 6, &mut ctx).unwrap(); let sliced = bitpacked.slice(128..2050).unwrap(); let primitive_result = sliced.take(indices).unwrap(); @@ -221,10 +223,11 @@ mod test { #[test] #[cfg_attr(miri, ignore)] // This test is too slow on miri fn take_random_indices() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let num_patches: usize = 128; let values = (0..u16::MAX as u32 + num_patches as u32).collect::>(); let uncompressed = PrimitiveArray::new(values.clone(), Validity::NonNullable); - let packed = BitPackedData::encode(&uncompressed.into_array(), 16).unwrap(); + let packed = BitPackedData::encode(&uncompressed.into_array(), 16, &mut ctx).unwrap(); assert!(packed.patches().is_some()); let rng = rng(); @@ -240,11 +243,11 @@ mod test { .enumerate() .for_each(|(ti, i)| { assert_eq!( - u32::try_from(&packed.scalar_at(*i as usize).unwrap()).unwrap(), + u32::try_from(&packed.execute_scalar(*i as usize, &mut ctx).unwrap()).unwrap(), values[*i as usize] ); assert_eq!( - u32::try_from(&taken.scalar_at(ti).unwrap()).unwrap(), + u32::try_from(&taken.execute_scalar(ti, &mut ctx).unwrap()).unwrap(), values[*i as usize] ); }); @@ -253,14 +256,16 @@ mod test { #[test] #[cfg_attr(miri, ignore)] fn take_signed_with_patches() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let start = - BitPackedData::encode(&buffer![1i32, 2i32, 3i32, 4i32].into_array(), 1).unwrap(); + BitPackedData::encode(&buffer![1i32, 2i32, 3i32, 4i32].into_array(), 1, &mut ctx) + .unwrap(); let taken_primitive = take_primitive::( start.as_view(), &PrimitiveArray::from_iter([0u64, 1, 2, 3]), Validity::NonNullable, - &mut LEGACY_SESSION.create_execution_ctx(), + &mut ctx, ) .unwrap(); assert_arrays_eq!(taken_primitive, PrimitiveArray::from_iter([1i32, 2, 3, 4])); @@ -268,8 +273,10 @@ mod test { #[test] fn take_nullable_with_nullables() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let start = - BitPackedData::encode(&buffer![1i32, 2i32, 3i32, 4i32].into_array(), 1).unwrap(); + BitPackedData::encode(&buffer![1i32, 2i32, 3i32, 4i32].into_array(), 1, &mut ctx) + .unwrap(); let taken_primitive = start .take( @@ -280,19 +287,29 @@ mod test { taken_primitive, PrimitiveArray::from_option_iter([Some(1i32), Some(2), None, Some(4)]) ); - assert_eq!(taken_primitive.to_primitive().invalid_count().unwrap(), 1); + let taken_primitive_prim = taken_primitive.execute::(&mut ctx).unwrap(); + assert_eq!(taken_primitive_prim.invalid_count(&mut ctx).unwrap(), 1); + } + + fn bp(array: vortex_array::ArrayRef, bit_width: u8) -> BitPackedArray { + BitPackedData::encode( + &array, + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap() } #[rstest] - #[case(BitPackedData::encode(&PrimitiveArray::from_iter((0..100).map(|i| (i % 63) as u8)).into_array(), 6).unwrap())] - #[case(BitPackedData::encode(&PrimitiveArray::from_iter((0..256).map(|i| i as u32)).into_array(), 8).unwrap())] - #[case(BitPackedData::encode(&buffer![1i32, 2, 3, 4, 5, 6, 7, 8].into_array(), 3).unwrap())] - #[case(BitPackedData::encode( - &PrimitiveArray::from_option_iter([Some(10u16), None, Some(20), Some(30), None]).into_array(), + #[case(bp(PrimitiveArray::from_iter((0..100).map(|i| (i % 63) as u8)).into_array(), 6))] + #[case(bp(PrimitiveArray::from_iter((0..256).map(|i| i as u32)).into_array(), 8))] + #[case(bp(buffer![1i32, 2, 3, 4, 5, 6, 7, 8].into_array(), 3))] + #[case(bp( + PrimitiveArray::from_option_iter([Some(10u16), None, Some(20), Some(30), None]).into_array(), 5 - ).unwrap())] - #[case(BitPackedData::encode(&buffer![42u32].into_array(), 6).unwrap())] - #[case(BitPackedData::encode(&PrimitiveArray::from_iter((0..1024).map(|i| i as u32)).into_array(), 8).unwrap())] + ))] + #[case(bp(buffer![42u32].into_array(), 6))] + #[case(bp(PrimitiveArray::from_iter((0..1024).map(|i| i as u32)).into_array(), 8))] fn test_take_bitpacked_conformance(#[case] bitpacked: BitPackedArray) { use vortex_array::compute::conformance::take::test_take_conformance; test_take_conformance(&bitpacked.into_array()); diff --git a/encodings/fastlanes/src/bitpacking/plugin.rs b/encodings/fastlanes/src/bitpacking/plugin.rs index fc67714fedd..d5ecdefd0c7 100644 --- a/encodings/fastlanes/src/bitpacking/plugin.rs +++ b/encodings/fastlanes/src/bitpacking/plugin.rs @@ -6,9 +6,11 @@ //! //! This enables zero-cost backward compatibility with previously written datasets. +use vortex_array::Array; use vortex_array::ArrayId; use vortex_array::ArrayPlugin; use vortex_array::ArrayRef; +use vortex_array::ArrayVTable; use vortex_array::IntoArray; use vortex_array::VortexSessionExecute; use vortex_array::arrays::Patched; @@ -31,7 +33,8 @@ impl ArrayPlugin for BitPackedPatchedPlugin { fn id(&self) -> ArrayId { // We reuse the existing `BitPacked` ID so that we can take over its // deserialization pathway. - BitPacked::ID + // TODO(joe): dedup method name + ArrayVTable::id(&BitPacked) } fn serialize( @@ -52,12 +55,10 @@ impl ArrayPlugin for BitPackedPatchedPlugin { children: &dyn ArrayChildren, session: &VortexSession, ) -> VortexResult { - let bitpacked = BitPacked - .deserialize(dtype, len, metadata, buffers, children, session)? - .try_downcast::() - .map_err(|_| { - vortex_err!("BitPacked plugin should only deserialize fastlanes.bitpacked") - })?; + let bitpacked = Array::::try_from_parts(ArrayVTable::deserialize( + &BitPacked, dtype, len, metadata, buffers, children, session, + )?) + .map_err(|_| vortex_err!("BitPacked plugin should only deserialize fastlanes.bitpacked"))?; // Create a new BitPackedArray without the interior patches installed. let Some(patches) = bitpacked.patches() else { @@ -82,6 +83,10 @@ impl ArrayPlugin for BitPackedPatchedPlugin { Ok(patched.into_array()) } + + fn is_supported_encoding(&self, id: &ArrayId) -> bool { + id == ArrayVTable::id(&BitPacked) || id == ArrayVTable::id(&Patched) + } } #[cfg(test)] @@ -90,6 +95,7 @@ mod tests { use vortex_array::ArrayPlugin; use vortex_array::IntoArray; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PatchedArray; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::patched::PatchedArraySlotsExt; @@ -115,11 +121,12 @@ mod tests { #[test] fn test_decode_bitpacked_patches() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); // Create values where some exceed the bit width, causing patches. // With bit_width=9, max value is 511. Values >=512 become patches. let values: Buffer = (0i32..=512).collect(); let parray = values.into_array(); - let bitpacked = BitPackedData::encode(&parray, 9)?; + let bitpacked = BitPackedData::encode(&parray, 9, &mut ctx)?; assert!( bitpacked.patches().is_some(), @@ -128,7 +135,7 @@ mod tests { let array = bitpacked.as_array(); - let metadata = array.metadata(&SESSION)?.unwrap_or_default(); + let metadata = SESSION.array_serialize(array)?.unwrap(); let children = array.children(); let buffers = array .buffers() @@ -165,10 +172,11 @@ mod tests { #[test] fn bitpacked_without_patches_stays_bitpacked() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); // With bit_width=16, max value is 65535. All values 0..100 fit. let values: Buffer = (0i32..100).collect(); let parray = values.into_array(); - let bitpacked = BitPackedData::encode(&parray, 16)?; + let bitpacked = BitPackedData::encode(&parray, 16, &mut ctx)?; assert!( bitpacked.patches().is_none(), @@ -177,7 +185,7 @@ mod tests { let array = bitpacked.as_array(); - let metadata = array.metadata(&SESSION)?.unwrap_or_default(); + let metadata = SESSION.array_serialize(array)?.unwrap(); let children = array.children(); let buffers = array .buffers() @@ -207,7 +215,7 @@ mod tests { fn primitive_array_returns_error() -> VortexResult<()> { let array = PrimitiveArray::from_iter([1i32, 2, 3]).into_array(); - let metadata = array.metadata(&SESSION)?.unwrap_or_default(); + let metadata = SESSION.array_serialize(&array)?.unwrap(); let children = array.children(); let buffers = array .buffers() diff --git a/encodings/fastlanes/src/bitpacking/vtable/kernels.rs b/encodings/fastlanes/src/bitpacking/vtable/kernels.rs index 128ff77d99c..cf975602179 100644 --- a/encodings/fastlanes/src/bitpacking/vtable/kernels.rs +++ b/encodings/fastlanes/src/bitpacking/vtable/kernels.rs @@ -4,10 +4,12 @@ use vortex_array::arrays::dict::TakeExecuteAdaptor; use vortex_array::arrays::filter::FilterExecuteAdaptor; use vortex_array::kernel::ParentKernelSet; +use vortex_array::scalar_fn::fns::cast::CastExecuteAdaptor; use crate::BitPacked; pub(crate) const PARENT_KERNELS: ParentKernelSet = ParentKernelSet::new(&[ + ParentKernelSet::lift(&CastExecuteAdaptor(BitPacked)), ParentKernelSet::lift(&FilterExecuteAdaptor(BitPacked)), ParentKernelSet::lift(&TakeExecuteAdaptor(BitPacked)), ]); diff --git a/encodings/fastlanes/src/bitpacking/vtable/mod.rs b/encodings/fastlanes/src/bitpacking/vtable/mod.rs index 93c5d116fdc..c11a6c4ec51 100644 --- a/encodings/fastlanes/src/bitpacking/vtable/mod.rs +++ b/encodings/fastlanes/src/bitpacking/vtable/mod.rs @@ -36,6 +36,7 @@ use vortex_error::vortex_bail; use vortex_error::vortex_err; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::BitPackedArrayExt; use crate::BitPackedData; @@ -91,7 +92,8 @@ impl VTable for BitPacked { type ValidityVTable = Self; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("fastlanes.bitpacked"); + *ID } fn validate( @@ -103,7 +105,7 @@ impl VTable for BitPacked { ) -> VortexResult<()> { let slots = BitPackedSlotsView::from_slots(slots); - let validity = child_to_validity(&slots.validity_child.cloned(), dtype.nullability()); + let validity = child_to_validity(slots.validity_child, dtype.nullability()); let patches = match (slots.patch_indices, slots.patch_values) { (Some(indices), Some(values)) => { let patch_offset = data @@ -151,18 +153,6 @@ impl VTable for BitPacked { } } - fn reduce_parent( - array: ArrayView<'_, Self>, - parent: &ArrayRef, - child_idx: usize, - ) -> VortexResult> { - RULES.evaluate(array, parent, child_idx) - } - - fn slot_name(_array: ArrayView<'_, Self>, idx: usize) -> String { - BitPackedSlots::NAMES[idx].to_string() - } - fn serialize( array: ArrayView<'_, Self>, _session: &VortexSession, @@ -281,6 +271,10 @@ impl VTable for BitPacked { }) } + fn slot_name(_array: ArrayView<'_, Self>, idx: usize) -> String { + BitPackedSlots::NAMES[idx].to_string() + } + fn execute(array: Array, ctx: &mut ExecutionCtx) -> VortexResult { require_patches!( array, @@ -303,14 +297,20 @@ impl VTable for BitPacked { ) -> VortexResult> { PARENT_KERNELS.execute(array, parent, child_idx, ctx) } + + fn reduce_parent( + array: ArrayView<'_, Self>, + parent: &ArrayRef, + child_idx: usize, + ) -> VortexResult> { + RULES.evaluate(array, parent, child_idx) + } } #[derive(Clone, Debug)] pub struct BitPacked; impl BitPacked { - pub const ID: ArrayId = ArrayId::new_ref("fastlanes.bitpacked"); - pub fn try_new( packed: BufferHandle, ptype: PType, @@ -353,7 +353,11 @@ impl BitPacked { } /// Encode an array into a bitpacked representation with the given bit width. - pub fn encode(array: &ArrayRef, bit_width: u8) -> VortexResult { - BitPackedData::encode(array, bit_width) + pub fn encode( + array: &ArrayRef, + bit_width: u8, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + BitPackedData::encode(array, bit_width, ctx) } } diff --git a/encodings/fastlanes/src/bitpacking/vtable/operations.rs b/encodings/fastlanes/src/bitpacking/vtable/operations.rs index 13e22a2f11c..6b82343d318 100644 --- a/encodings/fastlanes/src/bitpacking/vtable/operations.rs +++ b/encodings/fastlanes/src/bitpacking/vtable/operations.rs @@ -34,6 +34,8 @@ mod test { use vortex_array::ArrayRef; use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::SliceArray; use vortex_array::assert_arrays_eq; @@ -56,7 +58,7 @@ mod test { use crate::bitpacking::array::BitPackedArrayExt; fn bp(array: &ArrayRef, bit_width: u8) -> BitPackedArray { - BitPackedData::encode(array, bit_width).unwrap() + BitPackedData::encode(array, bit_width, &mut LEGACY_SESSION.create_execution_ctx()).unwrap() } fn slice_via_reduce(array: &BitPackedArray, range: Range) -> BitPackedArray { @@ -139,8 +141,9 @@ mod test { #[test] fn slice_empty_patches() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // We create an array that has 1 element that does not fit in the 6-bit range. - let array = BitPackedData::encode(&buffer![0u32..=64].into_array(), 6).unwrap(); + let array = BitPackedData::encode(&buffer![0u32..=64].into_array(), 6, &mut ctx).unwrap(); assert!(array.patches().is_some()); @@ -202,21 +205,29 @@ mod test { .unwrap() .into_array(); assert_eq!( - packed_array.scalar_at(1).unwrap(), + packed_array + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::null(DType::Primitive(PType::U32, Nullability::Nullable)) ); } #[test] fn scalar_at() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values = (0u32..257).collect::>(); let uncompressed = values.clone().into_array(); - let packed = BitPackedData::encode(&uncompressed, 8).unwrap(); + let packed = BitPackedData::encode(&uncompressed, 8, &mut ctx).unwrap(); assert!(packed.patches().is_some()); let patches = packed.patches().unwrap().indices().clone(); assert_eq!( - usize::try_from(&patches.scalar_at(0).unwrap()).unwrap(), + usize::try_from( + &patches + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ) + .unwrap(), 256 ); diff --git a/encodings/fastlanes/src/delta/array/delta_compress.rs b/encodings/fastlanes/src/delta/array/delta_compress.rs index 105004c0c71..d51cef72b49 100644 --- a/encodings/fastlanes/src/delta/array/delta_compress.rs +++ b/encodings/fastlanes/src/delta/array/delta_compress.rs @@ -27,7 +27,7 @@ pub fn delta_compress( // Fill-forward null values so that transposed deltas at null positions remain // small. Without this, bitpacking may skip patches for null positions, and the // corrupted delta values propagate through the cumulative sum during decompression. - let filled = fill_forward_nulls(array.to_buffer::(), &validity); + let filled = fill_forward_nulls(array.to_buffer::(), &validity, ctx)?; let (bases, deltas) = compress_primitive::(&filled); // TODO(robert): This can be avoided if we add TransposedBoolArray that performs index translation when necessary. let validity = transpose_validity(&validity, ctx)?; @@ -96,7 +96,6 @@ mod tests { use rstest::rstest; use vortex_array::IntoArray; - use vortex_array::ToCanonical; use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; @@ -131,12 +130,13 @@ mod tests { /// where null positions contain arbitrary values. Without fill-forward, the delta cumulative /// sum propagates corrupted values from null positions. #[test] - fn delta_bitpacked_trailing_nulls() { + fn delta_bitpacked_trailing_nulls() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); let array = PrimitiveArray::from_option_iter( (0u8..200).map(|i| (!(50..100).contains(&i)).then_some(i)), ); - let (bases, deltas) = delta_compress(&array, &mut SESSION.create_execution_ctx()).unwrap(); - let bitpacked_deltas = bitpack_encode(&deltas, 1, None).unwrap(); + let (bases, deltas) = delta_compress(&array, &mut ctx)?; + let bitpacked_deltas = bitpack_encode(&deltas, 1, None, &mut ctx)?; let packed_delta = Delta::try_new( bases.into_array(), bitpacked_deltas.into_array(), @@ -144,6 +144,11 @@ mod tests { array.len(), ) .vortex_expect("Delta array construction should succeed"); - assert_arrays_eq!(packed_delta.as_array().to_primitive(), array); + let packed_delta_prim = packed_delta + .as_array() + .clone() + .execute::(&mut ctx)?; + assert_arrays_eq!(packed_delta_prim, array); + Ok(()) } } diff --git a/encodings/fastlanes/src/delta/compute/cast.rs b/encodings/fastlanes/src/delta/compute/cast.rs index 4beb558dca0..324e2fc9c22 100644 --- a/encodings/fastlanes/src/delta/compute/cast.rs +++ b/encodings/fastlanes/src/delta/compute/cast.rs @@ -9,24 +9,17 @@ use vortex_array::dtype::DType; use vortex_array::dtype::Nullability::NonNullable; use vortex_array::scalar_fn::fns::cast::CastReduce; use vortex_error::VortexResult; -use vortex_error::vortex_panic; use crate::delta::Delta; use crate::delta::array::DeltaArrayExt; + impl CastReduce for Delta { fn cast(array: ArrayView<'_, Self>, dtype: &DType) -> VortexResult> { - // Delta encoding stores differences between consecutive values, which requires - // unsigned integers to avoid overflow issues. Signed integers could produce - // negative deltas that wouldn't fit in the unsigned delta representation. - // This encoding is optimized for monotonically increasing sequences. let DType::Primitive(target_ptype, _) = dtype else { return Ok(None); }; - let DType::Primitive(source_ptype, _) = array.dtype() else { - vortex_panic!("delta should be primitive typed"); - }; - + let source_ptype = array.dtype().as_ptype(); // TODO(DK): narrows can be safe but we must decompress to compute the maximum value. if target_ptype.is_signed_int() || source_ptype.bit_width() > target_ptype.bit_width() { return Ok(None); diff --git a/encodings/fastlanes/src/delta/vtable/mod.rs b/encodings/fastlanes/src/delta/vtable/mod.rs index e74243b4050..17e571ba4f6 100644 --- a/encodings/fastlanes/src/delta/vtable/mod.rs +++ b/encodings/fastlanes/src/delta/vtable/mod.rs @@ -30,6 +30,7 @@ use vortex_error::vortex_ensure; use vortex_error::vortex_err; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::DeltaData; use crate::delta::array::BASES_SLOT; @@ -38,6 +39,7 @@ use crate::delta::array::DeltaArrayExt; use crate::delta::array::SLOT_NAMES; use crate::delta::array::delta_decompress::delta_decompress; use crate::delta::array::lane_count; +use crate::delta_compress; mod operations; mod rules; @@ -75,7 +77,8 @@ impl VTable for Delta { type ValidityVTable = Self; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("fastlanes.delta"); + *ID } fn validate( @@ -180,8 +183,6 @@ impl VTable for Delta { pub struct Delta; impl Delta { - pub const ID: ArrayId = ArrayId::new_ref("fastlanes.delta"); - pub fn try_new( bases: ArrayRef, deltas: ArrayRef, @@ -200,7 +201,7 @@ impl Delta { ctx: &mut ExecutionCtx, ) -> VortexResult { let logical_len = array.len(); - let (bases, deltas) = crate::delta::array::delta_compress::delta_compress(array, ctx)?; + let (bases, deltas) = delta_compress(array, ctx)?; Self::try_new(bases.into_array(), deltas.into_array(), 0, logical_len) } } diff --git a/encodings/fastlanes/src/delta/vtable/operations.rs b/encodings/fastlanes/src/delta/vtable/operations.rs index 731588a67bb..943f11379ac 100644 --- a/encodings/fastlanes/src/delta/vtable/operations.rs +++ b/encodings/fastlanes/src/delta/vtable/operations.rs @@ -4,7 +4,7 @@ use vortex_array::ArrayView; use vortex_array::ExecutionCtx; use vortex_array::IntoArray; -use vortex_array::ToCanonical; +use vortex_array::arrays::PrimitiveArray; use vortex_array::scalar::Scalar; use vortex_array::vtable::OperationsVTable; use vortex_error::VortexResult; @@ -14,10 +14,13 @@ impl OperationsVTable for Delta { fn scalar_at( array: ArrayView<'_, Delta>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { - let decompressed = array.array().slice(index..index + 1)?.to_primitive(); - decompressed.into_array().scalar_at(0) + let decompressed = array + .array() + .slice(index..index + 1)? + .execute::(ctx)?; + decompressed.into_array().execute_scalar(0, ctx) } } @@ -215,7 +218,9 @@ mod tests { #[should_panic] fn test_scalar_at_non_jagged_array_oob() { let delta = da(&(0u32..2048).collect()).into_array(); - delta.scalar_at(2048).unwrap(); + delta + .execute_scalar(2048, &mut SESSION.create_execution_ctx()) + .unwrap(); } #[test] fn test_scalar_at_jagged_array() { @@ -229,7 +234,9 @@ mod tests { #[should_panic] fn test_scalar_at_jagged_array_oob() { let delta = da(&(0u32..2000).collect()).into_array(); - delta.scalar_at(2000).unwrap(); + delta + .execute_scalar(2000, &mut SESSION.create_execution_ctx()) + .unwrap(); } #[rstest] diff --git a/encodings/fastlanes/src/for/array/for_compress.rs b/encodings/fastlanes/src/for/array/for_compress.rs index 266e6306a9c..ad9566027d2 100644 --- a/encodings/fastlanes/src/for/array/for_compress.rs +++ b/encodings/fastlanes/src/for/array/for_compress.rs @@ -4,6 +4,8 @@ use num_traits::PrimInt; use num_traits::WrappingSub; use vortex_array::IntoArray; +use vortex_array::LEGACY_SESSION; +use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::dtype::NativePType; use vortex_array::expr::stats::Stat; @@ -19,7 +21,7 @@ impl FoRData { let array_ref = array.clone().into_array(); let min = array_ref .statistics() - .compute_stat(Stat::Min)? + .compute_stat(Stat::Min, &mut LEGACY_SESSION.create_execution_ctx())? .ok_or_else(|| vortex_err!("Min stat not found"))?; let encoded = match_each_integer_ptype!(array.ptype(), |T| { @@ -50,7 +52,6 @@ mod test { use std::sync::LazyLock; use itertools::Itertools; - use vortex_array::ToCanonical; use vortex_array::VortexSessionExecute; use vortex_array::arrays::primitive::PrimitiveArrayExt; use vortex_array::assert_arrays_eq; @@ -110,7 +111,10 @@ mod test { assert!(compressed.reference_scalar().dtype().is_signed_int()); assert!(compressed.encoded().dtype().is_signed_int()); - let encoded = compressed.encoded().scalar_at(0).unwrap(); + let encoded = compressed + .encoded() + .execute_scalar(0, &mut SESSION.create_execution_ctx()) + .unwrap(); assert_eq!(encoded, Scalar::from(0i32)); } @@ -124,34 +128,33 @@ mod test { #[test] fn test_decompress_fused() { + let mut ctx = SESSION.create_execution_ctx(); // Create a range offset by a million. let expect = PrimitiveArray::from_iter((0u32..1024).map(|x| x % 7 + 10)); let array = PrimitiveArray::from_iter((0u32..1024).map(|x| x % 7)); - let bp = BitPackedData::encode(&array.into_array(), 3).unwrap(); + let bp = BitPackedData::encode(&array.into_array(), 3, &mut ctx).unwrap(); let compressed = FoR::try_new(bp.into_array(), 10u32.into()).unwrap(); assert_arrays_eq!(compressed, expect); } #[test] fn test_decompress_fused_patches() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); // Create a range offset by a million. let expect = PrimitiveArray::from_iter((0u32..1024).map(|x| x % 7 + 10)); let array = PrimitiveArray::from_iter((0u32..1024).map(|x| x % 7)); - let bp = BitPackedData::encode(&array.into_array(), 2).unwrap(); + let bp = BitPackedData::encode(&array.into_array(), 2, &mut ctx)?; let compressed = FoR::try_new(bp.clone().into_array(), 10u32.into())?; - let decompressed = fused_decompress::( - &compressed, - bp.as_view(), - &mut SESSION.create_execution_ctx(), - )?; + let decompressed = fused_decompress::(&compressed, bp.as_view(), &mut ctx)?; assert_arrays_eq!(decompressed, expect); Ok(()) } #[test] fn test_overflow() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); let array = PrimitiveArray::from_iter(i8::MIN..=i8::MAX); - let compressed = FoRData::encode(array.clone()).unwrap(); + let compressed = FoRData::encode(array.clone())?; assert_eq!( i8::MIN, compressed @@ -163,19 +166,23 @@ mod test { let encoded = compressed .encoded() - .to_primitive() + .clone() + .execute::(&mut ctx)? .reinterpret_cast(PType::U8); let unsigned: Vec = (0..=u8::MAX).collect_vec(); let expected_unsigned = PrimitiveArray::from_iter(unsigned); assert_eq!(encoded.as_slice::(), expected_unsigned.as_slice::()); - let decompressed = decompress(&compressed, &mut SESSION.create_execution_ctx())?; + let decompressed = decompress(&compressed, &mut ctx)?; array .as_slice::() .iter() .enumerate() .for_each(|(i, v)| { - assert_eq!(*v, i8::try_from(&compressed.scalar_at(i).unwrap()).unwrap()); + assert_eq!( + *v, + i8::try_from(&compressed.execute_scalar(i, &mut ctx).unwrap()).unwrap() + ); }); assert_arrays_eq!(decompressed, array); Ok(()) diff --git a/encodings/fastlanes/src/for/array/for_decompress.rs b/encodings/fastlanes/src/for/array/for_decompress.rs index e19ef8d0ff3..a26d6e9053e 100644 --- a/encodings/fastlanes/src/for/array/for_decompress.rs +++ b/encodings/fastlanes/src/for/array/for_decompress.rs @@ -109,7 +109,7 @@ pub(crate) fn fused_decompress< let mut uninit_range = builder.uninit_range(bp.len()); unsafe { // Append a dense null Mask. - uninit_range.append_mask(bp.validity_mask()); + uninit_range.append_mask(bp.validity()?.execute_mask(bp.as_ref().len(), ctx)?); } // SAFETY: `decode_into` will initialize all values in this range. @@ -118,10 +118,10 @@ pub(crate) fn fused_decompress< // Decode all chunks (initial, full, and trailer) in one call. unpacked.decode_into(uninit_slice); - if let Some(ref patches) = bp.patches() { + if let Some(patches) = bp.patches() { bitpack_decompress::apply_patches_to_uninit_range_fn( &mut uninit_range, - patches, + &patches, ctx, |v| v.wrapping_add(&ref_), )?; diff --git a/encodings/fastlanes/src/for/compute/compare.rs b/encodings/fastlanes/src/for/compute/compare.rs index e5eae1b90b0..8cd2fb9c011 100644 --- a/encodings/fastlanes/src/for/compute/compare.rs +++ b/encodings/fastlanes/src/for/compute/compare.rs @@ -220,7 +220,6 @@ mod tests { #[test] fn compare_large_constant() { let reference = Scalar::from(-9219218377546224477i64); - #[allow(clippy::cast_possible_truncation)] let lhs = for_arr( PrimitiveArray::new( buffer![0i64, 9654309310445864926u64 as i64], diff --git a/encodings/fastlanes/src/for/compute/is_constant.rs b/encodings/fastlanes/src/for/compute/is_constant.rs index 246a9c93226..29393884e95 100644 --- a/encodings/fastlanes/src/for/compute/is_constant.rs +++ b/encodings/fastlanes/src/for/compute/is_constant.rs @@ -35,6 +35,6 @@ impl DynAggregateKernel for FoRIsConstantKernel { }; let result = is_constant(array.encoded(), ctx)?; - Ok(Some(IsConstant::make_partial(batch, result)?)) + Ok(Some(IsConstant::make_partial(batch, result, ctx)?)) } } diff --git a/encodings/fastlanes/src/for/compute/is_sorted.rs b/encodings/fastlanes/src/for/compute/is_sorted.rs index 3b81c45c03d..6b72af5902b 100644 --- a/encodings/fastlanes/src/for/compute/is_sorted.rs +++ b/encodings/fastlanes/src/for/compute/is_sorted.rs @@ -4,7 +4,6 @@ use vortex_array::ArrayRef; use vortex_array::ExecutionCtx; use vortex_array::IntoArray; -use vortex_array::ToCanonical; use vortex_array::aggregate_fn::AggregateFnRef; use vortex_array::aggregate_fn::fns::is_sorted::IsSorted; use vortex_array::aggregate_fn::fns::is_sorted::is_sorted; @@ -35,7 +34,7 @@ impl DynAggregateKernel for FoRIsSortedKernel { return Ok(None); }; - let encoded = array.encoded().to_primitive(); + let encoded = array.encoded().clone().execute::(ctx)?; let unsigned_array = PrimitiveArray::from_buffer_handle( encoded.buffer_handle().clone(), encoded.ptype().to_unsigned(), @@ -49,7 +48,12 @@ impl DynAggregateKernel for FoRIsSortedKernel { is_sorted(&unsigned_array, ctx)? }; - Ok(Some(IsSorted::make_partial(batch, result, options.strict)?)) + Ok(Some(IsSorted::make_partial( + batch, + result, + options.strict, + ctx, + )?)) } } diff --git a/encodings/fastlanes/src/for/vtable/mod.rs b/encodings/fastlanes/src/for/vtable/mod.rs index 3f3207a8e24..85f7cf2429b 100644 --- a/encodings/fastlanes/src/for/vtable/mod.rs +++ b/encodings/fastlanes/src/for/vtable/mod.rs @@ -30,6 +30,7 @@ use vortex_error::vortex_bail; use vortex_error::vortex_ensure; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::FoRData; use crate::r#for::array::FoRArrayExt; @@ -66,7 +67,8 @@ impl VTable for FoR { type ValidityVTable = ValidityVTableFromChild; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("fastlanes.for"); + *ID } fn validate( @@ -162,8 +164,6 @@ impl VTable for FoR { pub struct FoR; impl FoR { - pub const ID: ArrayId = ArrayId::new_ref("fastlanes.for"); - /// Construct a new FoR array from an encoded array and a reference scalar. pub fn try_new(encoded: ArrayRef, reference: Scalar) -> VortexResult { vortex_ensure!(!reference.is_null(), "Reference value cannot be null"); diff --git a/encodings/fastlanes/src/for/vtable/operations.rs b/encodings/fastlanes/src/for/vtable/operations.rs index e90ff7f4ad2..5fea6f4bdaa 100644 --- a/encodings/fastlanes/src/for/vtable/operations.rs +++ b/encodings/fastlanes/src/for/vtable/operations.rs @@ -15,9 +15,9 @@ impl OperationsVTable for FoR { fn scalar_at( array: ArrayView<'_, FoR>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { - let encoded_pvalue = array.encoded().scalar_at(index)?; + let encoded_pvalue = array.encoded().execute_scalar(index, ctx)?; let encoded_pvalue = encoded_pvalue.as_primitive(); let reference = array.reference_scalar(); let reference = reference.as_primitive(); diff --git a/encodings/fastlanes/src/lib.rs b/encodings/fastlanes/src/lib.rs index 9a9f37d9af8..9022b7c4e2b 100644 --- a/encodings/fastlanes/src/lib.rs +++ b/encodings/fastlanes/src/lib.rs @@ -1,20 +1,19 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::cast_possible_truncation)] - -use std::env; -use std::sync::LazyLock; +#![expect(clippy::cast_possible_truncation)] pub use bitpacking::*; pub use delta::*; pub use r#for::*; pub use rle::*; -use vortex_array::ToCanonical; +use vortex_array::ExecutionCtx; +use vortex_array::arrays::BoolArray; use vortex_array::arrays::bool::BoolArrayExt; use vortex_array::validity::Validity; use vortex_buffer::Buffer; use vortex_buffer::BufferMut; +use vortex_error::VortexResult; pub mod bit_transpose; mod bitpacking; @@ -27,28 +26,20 @@ pub(crate) const FL_CHUNK_SIZE: usize = 1024; use bitpacking::compute::is_constant::BitPackedIsConstantKernel; use r#for::compute::is_constant::FoRIsConstantKernel; use r#for::compute::is_sorted::FoRIsSortedKernel; +use vortex_array::ArrayVTable; use vortex_array::aggregate_fn::AggregateFnVTable; use vortex_array::aggregate_fn::fns::is_constant::IsConstant; use vortex_array::aggregate_fn::fns::is_sorted::IsSorted; use vortex_array::aggregate_fn::session::AggregateFnSessionExt; +use vortex_array::arrays::patched::use_experimental_patches; use vortex_array::session::ArraySessionExt; use vortex_session::VortexSession; -/// Flag indicating if experimental patched array support is enabled. -/// -/// This is set using the environment variable `VORTEX_EXPERIMENTAL_PATCHED_ARRAY`. -/// -/// When this is true, any BitPacked array with interior patches will be read as a `Patched` -/// array, and the builtin compressor will use Patched array with BitPacked instead of -/// BitPacked array with interior patches. -pub static USE_EXPERIMENTAL_PATCHES: LazyLock = - LazyLock::new(|| env::var("VORTEX_EXPERIMENTAL_PATCHED_ARRAY").is_ok()); - /// Initialize fastlanes encodings in the given session. pub fn initialize(session: &VortexSession) { // If we're using the experimental Patched encoding, register a shim // for BitPacked with interior patches decode as Patched array. - if *USE_EXPERIMENTAL_PATCHES { + if use_experimental_patches() { session.arrays().register(BitPackedPatchedPlugin); } else { session.arrays().register(BitPacked); @@ -59,17 +50,17 @@ pub fn initialize(session: &VortexSession) { // Register the encoding-specific aggregate kernels. session.aggregate_fns().register_aggregate_kernel( - BitPacked::ID, + BitPacked.id(), Some(IsConstant.id()), &BitPackedIsConstantKernel, ); session.aggregate_fns().register_aggregate_kernel( - FoR::ID, + FoR.id(), Some(IsConstant.id()), &FoRIsConstantKernel, ); session.aggregate_fns().register_aggregate_kernel( - FoR::ID, + FoR.id(), Some(IsSorted.id()), &FoRIsSortedKernel, ); @@ -88,12 +79,16 @@ pub fn initialize(session: &VortexSession) { pub(crate) fn fill_forward_nulls( values: Buffer, validity: &Validity, -) -> Buffer { + ctx: &mut ExecutionCtx, +) -> VortexResult> { match validity { - Validity::NonNullable | Validity::AllValid => values, - Validity::AllInvalid => Buffer::zeroed(values.len()), + Validity::NonNullable | Validity::AllValid => Ok(values), + Validity::AllInvalid => Ok(Buffer::zeroed(values.len())), Validity::Array(validity_array) => { - let bit_buffer = validity_array.to_bool().to_bit_buffer(); + let bit_buffer = validity_array + .clone() + .execute::(ctx)? + .to_bit_buffer(); let mut last_valid = T::default(); match values.try_into_mut() { Ok(mut to_fill_mut) => { @@ -108,7 +103,7 @@ pub(crate) fn fill_forward_nulls( *v = last_valid; } } - to_fill_mut.freeze() + Ok(to_fill_mut.freeze()) } Err(to_fill) => { let mut to_fill_mut = BufferMut::::with_capacity(to_fill.len()); @@ -130,7 +125,7 @@ pub(crate) fn fill_forward_nulls( out.write(last_valid); } unsafe { to_fill_mut.set_len(to_fill.len()) }; - to_fill_mut.freeze() + Ok(to_fill_mut.freeze()) } } } @@ -141,6 +136,7 @@ pub(crate) fn fill_forward_nulls( mod test { use std::sync::LazyLock; + use vortex_array::VortexSessionExecute; use vortex_array::session::ArraySessionExt; use vortex_buffer::BitBufferMut; use vortex_session::VortexSession; @@ -157,7 +153,8 @@ mod test { }); #[test] - fn fill_forward_nulls_resets_at_chunk_boundary() { + fn fill_forward_nulls_resets_at_chunk_boundary() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); // Build a buffer spanning two chunks where the last valid value in chunk 0 // is non-zero. Null positions at the start of chunk 1 must get T::default() // (0), not the carry-over from chunk 0. @@ -169,7 +166,7 @@ mod test { validity_bits.set(FL_CHUNK_SIZE - 1); // only this position is valid let validity = Validity::from(validity_bits.freeze()); - let result = fill_forward_nulls(values.freeze(), &validity); + let result = fill_forward_nulls(values.freeze(), &validity, &mut ctx)?; // Within chunk 0, nulls before the valid element get 0 (default), and the // valid element itself is 42. @@ -183,5 +180,6 @@ mod test { "position {i} should be 0, not carried from chunk 0" ); } + Ok(()) } } diff --git a/encodings/fastlanes/src/rle/array/mod.rs b/encodings/fastlanes/src/rle/array/mod.rs index cf418ea114e..aeaba0ef42e 100644 --- a/encodings/fastlanes/src/rle/array/mod.rs +++ b/encodings/fastlanes/src/rle/array/mod.rs @@ -5,6 +5,7 @@ use std::fmt::Display; use std::fmt::Formatter; use vortex_array::ArrayRef; +use vortex_array::ExecutionCtx; use vortex_array::TypedArrayRef; use vortex_error::VortexExpect as _; use vortex_error::VortexResult; @@ -109,16 +110,16 @@ pub trait RLEArrayExt: TypedArrayRef { clippy::expect_used, reason = "expect is safe here as scalar_at returns a valid primitive" )] - fn values_idx_offset(&self, chunk_idx: usize) -> usize { + fn values_idx_offset(&self, chunk_idx: usize, ctx: &mut ExecutionCtx) -> usize { self.values_idx_offsets() - .scalar_at(chunk_idx) + .execute_scalar(chunk_idx, ctx) .expect("index must be in bounds") .as_primitive() .as_::() .expect("index must be of type usize") - self .values_idx_offsets() - .scalar_at(0) + .execute_scalar(0, ctx) .expect("index must be in bounds") .as_primitive() .as_::() @@ -137,9 +138,9 @@ impl> RLEArrayExt for T {} #[cfg(test)] mod tests { use vortex_array::ArrayContext; + use vortex_array::Canonical; use vortex_array::IntoArray; use vortex_array::LEGACY_SESSION; - use vortex_array::ToCanonical; use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::primitive::PrimitiveArrayExt; @@ -204,9 +205,10 @@ mod tests { assert_eq!(rle_array.len(), 3); assert_eq!(rle_array.values().len(), 2); - assert!(rle_array.is_valid(0).unwrap()); - assert!(!rle_array.is_valid(1).unwrap()); - assert!(rle_array.is_valid(2).unwrap()); + let mut ctx = SESSION.create_execution_ctx(); + assert!(rle_array.is_valid(0, &mut ctx).unwrap()); + assert!(!rle_array.is_valid(1, &mut ctx).unwrap()); + assert!(rle_array.is_valid(2, &mut ctx).unwrap()); } #[test] @@ -232,12 +234,17 @@ mod tests { let rle_array = RLE::try_new(values, indices_with_validity, values_idx_offsets, 0, 5) .vortex_expect("RLEData is always valid"); - let valid_slice = rle_array.slice(0..3).unwrap().to_primitive(); + let mut ctx = SESSION.create_execution_ctx(); + let valid_slice = rle_array + .slice(0..3) + .unwrap() + .execute::(&mut ctx) + .unwrap(); // TODO(joe): replace with compute null count - assert!(valid_slice.all_valid().unwrap()); + assert!(valid_slice.all_valid(&mut ctx).unwrap()); let mixed_slice = rle_array.slice(1..5).unwrap(); - assert!(!mixed_slice.all_valid().unwrap()); + assert!(!mixed_slice.all_valid(&mut ctx).unwrap()); } #[test] @@ -267,13 +274,14 @@ mod tests { let invalid_slice = rle_array .slice(2..5) .unwrap() - .to_canonical() + .execute::(&mut LEGACY_SESSION.create_execution_ctx()) .unwrap() .into_primitive(); - assert!(invalid_slice.all_invalid().unwrap()); + let mut ctx = SESSION.create_execution_ctx(); + assert!(invalid_slice.all_invalid(&mut ctx).unwrap()); let mixed_slice = rle_array.slice(1..4).unwrap(); - assert!(!mixed_slice.all_invalid().unwrap()); + assert!(!mixed_slice.all_invalid(&mut ctx).unwrap()); } #[test] @@ -300,7 +308,14 @@ mod tests { .vortex_expect("RLEData is always valid"); let sliced_array = rle_array.slice(1..4).unwrap(); - let validity_mask = sliced_array.validity_mask().unwrap(); + let validity_mask = sliced_array + .validity() + .unwrap() + .execute_mask( + sliced_array.len(), + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); let mut ctx = LEGACY_SESSION.create_execution_ctx(); let expected_mask = Validity::from_iter([false, true, false]) @@ -332,6 +347,7 @@ mod tests { #[test] fn test_multi_chunk_two_chunks() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values = PrimitiveArray::from_iter([10u32, 20, 30, 40]).into_array(); let indices = PrimitiveArray::from_iter([0u16, 1].repeat(1024)).into_array(); let values_idx_offsets = PrimitiveArray::from_iter([0u64, 2]).into_array(); @@ -341,23 +357,27 @@ mod tests { assert_eq!(rle_array.len(), 2048); assert_eq!(rle_array.values().len(), 4); - assert_eq!(rle_array.values_idx_offset(0), 0); - assert_eq!(rle_array.values_idx_offset(1), 2); + assert_eq!(rle_array.values_idx_offset(0, &mut ctx), 0); + assert_eq!(rle_array.values_idx_offset(1, &mut ctx), 2); } #[test] - fn test_rle_serialization() { + fn test_rle_serialization() -> VortexResult<()> { + let mut exec_ctx = SESSION.create_execution_ctx(); let primitive = PrimitiveArray::from_iter((0..2048).map(|i| (i / 100) as u32)); - let rle_array = RLEData::encode(&primitive).unwrap(); + let rle_array = RLEData::encode(primitive.as_view(), &mut exec_ctx)?; assert_eq!(rle_array.len(), 2048); - let original_data = rle_array.as_array().to_primitive(); + let original_data = rle_array + .as_array() + .clone() + .execute::(&mut exec_ctx)?; let ctx = ArrayContext::empty(); - let serialized = rle_array - .into_array() - .serialize(&ctx, &SESSION, &SerializeOptions::default()) - .unwrap(); + let serialized = + rle_array + .into_array() + .serialize(&ctx, &SESSION, &SerializeOptions::default())?; let mut concat = ByteBufferMut::empty(); for buf in serialized { @@ -365,25 +385,25 @@ mod tests { } let concat = concat.freeze(); - let parts = SerializedArray::try_from(concat).unwrap(); - let decoded = parts - .decode( - &DType::Primitive(PType::U32, Nullability::NonNullable), - 2048, - &ReadContext::new(ctx.to_ids()), - &SESSION, - ) - .unwrap(); + let parts = SerializedArray::try_from(concat)?; + let decoded = parts.decode( + &DType::Primitive(PType::U32, Nullability::NonNullable), + 2048, + &ReadContext::new(ctx.to_ids()), + &SESSION, + )?; - let decoded_data = decoded.to_primitive(); + let decoded_data = decoded.execute::(&mut exec_ctx)?; assert_arrays_eq!(original_data, decoded_data); + Ok(()) } #[test] - fn test_rle_serialization_slice() { + fn test_rle_serialization_slice() -> VortexResult<()> { + let mut exec_ctx = SESSION.create_execution_ctx(); let primitive = PrimitiveArray::from_iter((0..2048).map(|i| (i / 100) as u32)); - let rle_array = RLEData::encode(&primitive).unwrap(); + let rle_array = RLEData::encode(primitive.as_view(), &mut exec_ctx)?; let sliced = RLE::try_new( rle_array.values().clone(), @@ -396,11 +416,11 @@ mod tests { assert_eq!(sliced.len(), 100); let ctx = ArrayContext::empty(); - let serialized = sliced - .clone() - .into_array() - .serialize(&ctx, &SESSION, &SerializeOptions::default()) - .unwrap(); + let serialized = + sliced + .clone() + .into_array() + .serialize(&ctx, &SESSION, &SerializeOptions::default())?; let mut concat = ByteBufferMut::empty(); for buf in serialized { @@ -408,20 +428,22 @@ mod tests { } let concat = concat.freeze(); - let parts = SerializedArray::try_from(concat).unwrap(); - let decoded = parts - .decode( - sliced.dtype(), - sliced.len(), - &ReadContext::new(ctx.to_ids()), - &SESSION, - ) - .unwrap(); + let parts = SerializedArray::try_from(concat)?; + let decoded = parts.decode( + sliced.dtype(), + sliced.len(), + &ReadContext::new(ctx.to_ids()), + &SESSION, + )?; - let original_data = sliced.as_array().to_primitive(); - let decoded_data = decoded.to_primitive(); + let original_data = sliced + .as_array() + .clone() + .execute::(&mut exec_ctx)?; + let decoded_data = decoded.execute::(&mut exec_ctx)?; assert_arrays_eq!(original_data, decoded_data); + Ok(()) } /// Regression test: re-encoding RLE indices with RLE must not corrupt @@ -434,6 +456,7 @@ mod tests { /// because chunk 1 only had 1 unique value and index 1 was out of bounds. #[test] fn test_recompress_indices_no_cross_chunk_leak() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); let len = FL_CHUNK_SIZE + 100; let mut values: Vec> = vec![None; len]; // Two distinct values in chunk 0 → indices 0 and 1. @@ -442,12 +465,16 @@ mod tests { // Chunk 1 (positions 1024..) is all-null. let original = PrimitiveArray::from_option_iter(values); - let rle = RLEData::encode(&original)?; + let rle = RLEData::encode(original.as_view(), &mut ctx)?; - // Simulate cascading compression: narrow u16→u8 then re-encode with RLE, + // Simulate cascading compression: narrow u16->u8 then re-encode with RLE, // matching the path taken by the BtrBlocks compressor. - let indices_prim = rle.indices().to_primitive().narrow()?; - let re_encoded = RLEData::encode(&indices_prim)?; + let indices_prim = rle + .indices() + .clone() + .execute::(&mut ctx)? + .narrow()?; + let re_encoded = RLEData::encode(indices_prim.as_view(), &mut ctx)?; // Reconstruct the outer RLE with re-encoded indices. // SAFETY: we only replace the indices child; all other invariants hold. @@ -462,7 +489,10 @@ mod tests { }; // Decompress — panicked before the fill_forward_nulls chunk-boundary fix. - let decoded = reconstructed.as_array().to_primitive(); + let decoded = reconstructed + .as_array() + .clone() + .execute::(&mut ctx)?; assert_arrays_eq!(decoded, original); Ok(()) } diff --git a/encodings/fastlanes/src/rle/array/rle_compress.rs b/encodings/fastlanes/src/rle/array/rle_compress.rs index 36ec3a22a03..e48f31f51da 100644 --- a/encodings/fastlanes/src/rle/array/rle_compress.rs +++ b/encodings/fastlanes/src/rle/array/rle_compress.rs @@ -4,8 +4,11 @@ use std::mem; use fastlanes::RLE as FastLanesRLE; +use vortex_array::ArrayView; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; -use vortex_array::ToCanonical; +use vortex_array::arrays::BoolArray; +use vortex_array::arrays::Primitive; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::bool::BoolArrayExt; use vortex_array::arrays::primitive::NativeValue; @@ -25,22 +28,26 @@ use crate::fill_forward_nulls; impl RLEData { /// Encodes a primitive array of unsigned integers using FastLanes RLE. - pub fn encode(array: &PrimitiveArray) -> VortexResult { - match_each_native_ptype!(array.ptype(), |T| { rle_encode_typed::(array) }) + pub fn encode( + array: ArrayView<'_, Primitive>, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + let array = array.into_owned(); + match_each_native_ptype!(array.ptype(), |T| { rle_encode_typed::(&array, ctx) }) } } /// Encodes a primitive array of unsigned integers using FastLanes RLE. /// /// In case the input array length is % 1024 != 0, the last chunk is padded. -fn rle_encode_typed(array: &PrimitiveArray) -> VortexResult +fn rle_encode_typed(array: &PrimitiveArray, ctx: &mut ExecutionCtx) -> VortexResult where T: NativePType + FastLanesRLE, NativeValue: FastLanesRLE, { // Fill-forward null values so the RLE encoder doesn't see garbage at null positions, // which would create spurious run boundaries and inflate the dictionary. - let values = fill_forward_nulls(array.to_buffer::(), &array.validity()?); + let values = fill_forward_nulls(array.to_buffer::(), &array.validity()?, ctx)?; let len = values.len(); let padded_len = len.next_multiple_of(FL_CHUNK_SIZE); @@ -109,7 +116,7 @@ where RLE::try_new( values_buf.into_array(), - PrimitiveArray::new(indices_buf.freeze(), padded_validity(array)).into_array(), + PrimitiveArray::new(indices_buf.freeze(), padded_validity(array, ctx)?).into_array(), values_idx_offsets.into_array(), 0, array.len(), @@ -117,29 +124,29 @@ where } /// Returns validity padded to the next 1024 chunk for a given array. -fn padded_validity(array: &PrimitiveArray) -> Validity { +fn padded_validity(array: &PrimitiveArray, ctx: &mut ExecutionCtx) -> VortexResult { match array .validity() .vortex_expect("RLE validity should be derivable") { - Validity::NonNullable => Validity::NonNullable, - Validity::AllValid => Validity::AllValid, - Validity::AllInvalid => Validity::AllInvalid, + Validity::NonNullable => Ok(Validity::NonNullable), + Validity::AllValid => Ok(Validity::AllValid), + Validity::AllInvalid => Ok(Validity::AllInvalid), Validity::Array(validity_array) => { let len = array.len(); let padded_len = len.next_multiple_of(FL_CHUNK_SIZE); if len == padded_len { - return Validity::Array(validity_array); + return Ok(Validity::Array(validity_array)); } let mut builder = BitBufferMut::with_capacity(padded_len); - let bool_array = validity_array.to_bool(); + let bool_array = validity_array.execute::(ctx)?; builder.append_buffer(&bool_array.to_bit_buffer()); builder.append_n(false, padded_len - len); - Validity::from(builder.freeze()) + Ok(Validity::from(builder.freeze())) } } } @@ -147,8 +154,10 @@ fn padded_validity(array: &PrimitiveArray) -> Validity { #[cfg(test)] mod tests { use rstest::rstest; + use vortex_array::ExecutionCtx; use vortex_array::IntoArray; - use vortex_array::ToCanonical; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::ConstantArray; use vortex_array::arrays::MaskedArray; use vortex_array::arrays::PrimitiveArray; @@ -162,79 +171,123 @@ mod tests { use crate::rle::array::RLEArrayExt; #[test] - fn test_encode_decode() { + fn test_encode_decode() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // u8 let array_u8: Buffer = buffer![1, 1, 2, 2, 3, 3]; - let encoded_u8 = - RLEData::encode(&PrimitiveArray::new(array_u8, Validity::NonNullable)).unwrap(); - let decoded_u8 = encoded_u8.as_array().to_primitive(); + let encoded_u8 = RLEData::encode( + PrimitiveArray::new(array_u8, Validity::NonNullable).as_view(), + &mut ctx, + )?; + let decoded_u8 = encoded_u8 + .as_array() + .clone() + .execute::(&mut ctx)?; let expected_u8 = PrimitiveArray::from_iter(vec![1u8, 1, 2, 2, 3, 3]); assert_arrays_eq!(decoded_u8, expected_u8); // u16 let array_u16: Buffer = buffer![100, 100, 200, 200]; - let encoded_u16 = - RLEData::encode(&PrimitiveArray::new(array_u16, Validity::NonNullable)).unwrap(); - let decoded_u16 = encoded_u16.as_array().to_primitive(); + let encoded_u16 = RLEData::encode( + PrimitiveArray::new(array_u16, Validity::NonNullable).as_view(), + &mut ctx, + )?; + let decoded_u16 = encoded_u16 + .as_array() + .clone() + .execute::(&mut ctx)?; let expected_u16 = PrimitiveArray::from_iter(vec![100u16, 100, 200, 200]); assert_arrays_eq!(decoded_u16, expected_u16); // u64 let array_u64: Buffer = buffer![1000, 1000, 2000]; - let encoded_u64 = - RLEData::encode(&PrimitiveArray::new(array_u64, Validity::NonNullable)).unwrap(); - let decoded_u64 = encoded_u64.as_array().to_primitive(); + let encoded_u64 = RLEData::encode( + PrimitiveArray::new(array_u64, Validity::NonNullable).as_view(), + &mut ctx, + )?; + let decoded_u64 = encoded_u64 + .as_array() + .clone() + .execute::(&mut ctx)?; let expected_u64 = PrimitiveArray::from_iter(vec![1000u64, 1000, 2000]); assert_arrays_eq!(decoded_u64, expected_u64); + Ok(()) } #[test] fn test_length() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values: Buffer = buffer![1, 1, 2, 2, 2, 3]; - let encoded = RLEData::encode(&PrimitiveArray::new(values, Validity::NonNullable)).unwrap(); + let encoded = RLEData::encode( + PrimitiveArray::new(values, Validity::NonNullable).as_view(), + &mut ctx, + ) + .unwrap(); assert_eq!(encoded.len(), 6); } #[test] fn test_empty_length() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values: Buffer = Buffer::empty(); - let encoded = RLEData::encode(&PrimitiveArray::new(values, Validity::NonNullable)).unwrap(); + let encoded = RLEData::encode( + PrimitiveArray::new(values, Validity::NonNullable).as_view(), + &mut ctx, + ) + .unwrap(); assert_eq!(encoded.len(), 0); assert_eq!(encoded.values().len(), 0); } #[test] - fn test_single_value() { + fn test_single_value() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values: Buffer = vec![42; 2000].into_iter().collect(); - let encoded = RLEData::encode(&PrimitiveArray::new(values, Validity::NonNullable)).unwrap(); + let encoded = RLEData::encode( + PrimitiveArray::new(values, Validity::NonNullable).as_view(), + &mut ctx, + )?; assert_eq!(encoded.values().len(), 2); // 2 chunks, each storing value 42 - let decoded = encoded.as_array().to_primitive(); // Verify round-trip + let decoded = encoded + .as_array() + .clone() + .execute::(&mut ctx)?; // Verify round-trip let expected = PrimitiveArray::from_iter(vec![42u16; 2000]); assert_arrays_eq!(decoded, expected); + Ok(()) } #[test] - fn test_all_different() { + fn test_all_different() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values: Buffer = (0u8..=255).collect(); - let encoded = RLEData::encode(&PrimitiveArray::new(values, Validity::NonNullable)).unwrap(); + let encoded = RLEData::encode( + PrimitiveArray::new(values, Validity::NonNullable).as_view(), + &mut ctx, + )?; assert_eq!(encoded.values().len(), 256); - let decoded = encoded.as_array().to_primitive(); // Verify round-trip + let decoded = encoded + .as_array() + .clone() + .execute::(&mut ctx)?; // Verify round-trip let expected = PrimitiveArray::from_iter((0u8..=255).collect::>()); assert_arrays_eq!(decoded, expected); + Ok(()) } #[test] fn test_partial_last_chunk() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Test array with partial last chunk (not divisible by 1024) let values: Buffer = (0..1500).map(|i| (i / 100) as u32).collect(); let array = PrimitiveArray::new(values, Validity::NonNullable); - let encoded = RLEData::encode(&array).unwrap(); + let encoded = RLEData::encode(array.as_view(), &mut ctx).unwrap(); assert_eq!(encoded.len(), 1500); assert_arrays_eq!(encoded, array); @@ -244,11 +297,12 @@ mod tests { #[test] fn test_two_full_chunks() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Array that spans exactly 2 chunks (2048 elements) let values: Buffer = (0..2048).map(|i| (i / 100) as u32).collect(); let array = PrimitiveArray::new(values, Validity::NonNullable); - let encoded = RLEData::encode(&array).unwrap(); + let encoded = RLEData::encode(array.as_view(), &mut ctx).unwrap(); assert_eq!(encoded.len(), 2048); assert_arrays_eq!(encoded, array); @@ -267,17 +321,22 @@ mod tests { #[case::f16((-2000..2000).map(|i| f16::from_f32(i as f32)).collect::>())] #[case::f32((-2000..2000).map(|i| i as f32).collect::>())] #[case::f64((-2000..2000).map(|i| i as f64).collect::>())] - fn test_roundtrip_primitive_types(#[case] values: Buffer) { - let primitive = values.clone().into_array().to_primitive(); - let result = RLEData::encode(&primitive).unwrap(); - let decoded = result.as_array().to_primitive(); - let expected = PrimitiveArray::new( - values, - primitive - .validity() - .vortex_expect("primitive validity should be derivable"), - ); + fn test_roundtrip_primitive_types( + #[case] values: Buffer, + ) -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let primitive = values + .clone() + .into_array() + .execute::(&mut ctx)?; + let result = RLEData::encode(primitive.as_view(), &mut ctx)?; + let decoded = result + .as_array() + .clone() + .execute::(&mut ctx)?; + let expected = PrimitiveArray::new(values, primitive.validity()?); assert_arrays_eq!(decoded, expected); + Ok(()) } /// Replaces the indices of an RLE array with MaskedArray(ConstantArray(1u16), validity). @@ -286,8 +345,11 @@ mod tests { /// Valid when every chunk has at least two RLE dictionary entries (the /// fill-forward default at index 0 and the actual value at index 1), which /// holds whenever the first position of each chunk is null. - fn with_masked_constant_indices(rle: &RLEArray) -> VortexResult { - let indices_prim = rle.indices().to_primitive(); + fn with_masked_constant_indices( + rle: &RLEArray, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + let indices_prim = rle.indices().clone().execute::(ctx)?; let masked_indices = MaskedArray::try_new( ConstantArray::new(1u16, indices_prim.len()).into_array(), indices_prim.validity()?, @@ -304,40 +366,44 @@ mod tests { #[test] fn test_encode_all_null_chunk() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values: Vec> = vec![None; FL_CHUNK_SIZE]; let original = PrimitiveArray::from_option_iter(values); - let rle = RLEData::encode(&original)?; - let decoded = with_masked_constant_indices(&rle)?; + let rle = RLEData::encode(original.as_view(), &mut ctx)?; + let decoded = with_masked_constant_indices(&rle, &mut ctx)?; assert_arrays_eq!(decoded, original); Ok(()) } #[test] fn test_encode_all_null_chunk_then_value_chunk() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // First chunk is entirely null, second chunk has a value preceded by nulls. let mut values: Vec> = vec![None; 2 * FL_CHUNK_SIZE]; values[FL_CHUNK_SIZE + 100] = Some(42); let original = PrimitiveArray::from_option_iter(values); - let rle = RLEData::encode(&original)?; - let decoded = with_masked_constant_indices(&rle)?; + let rle = RLEData::encode(original.as_view(), &mut ctx)?; + let decoded = with_masked_constant_indices(&rle, &mut ctx)?; assert_arrays_eq!(decoded, original); Ok(()) } #[test] fn test_encode_one_value_near_end() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Single distinct value near the end of the chunk. let mut values: Vec> = vec![None; FL_CHUNK_SIZE]; values[1000] = Some(42); let original = PrimitiveArray::from_option_iter(values); - let rle = RLEData::encode(&original)?; - let decoded = with_masked_constant_indices(&rle)?; + let rle = RLEData::encode(original.as_view(), &mut ctx)?; + let decoded = with_masked_constant_indices(&rle, &mut ctx)?; assert_arrays_eq!(decoded, original); Ok(()) } #[test] fn test_encode_value_chunk_then_all_null_remainder() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // 1085 elements (2 chunks: 1024 + 61 padded to 1024). // Chunk 0 has -1i16 at scattered positions (273..=366), rest null. // Chunk 1 (the remainder) is entirely null. @@ -352,8 +418,8 @@ mod tests { values[pos] = Some(-1); } let original = PrimitiveArray::from_option_iter(values); - let rle = RLEData::encode(&original)?; - let decoded = with_masked_constant_indices(&rle)?; + let rle = RLEData::encode(original.as_view(), &mut ctx)?; + let decoded = with_masked_constant_indices(&rle, &mut ctx)?; assert_arrays_eq!(decoded, original); Ok(()) } @@ -363,8 +429,11 @@ mod tests { /// This simulates a compressor that doesn't preserve index values at null /// positions, which can happen when indices are further compressed and the /// compressor clobbers invalid entries with arbitrary data. - fn with_random_invalid_indices(rle: &RLEArray) -> VortexResult { - let indices_prim = rle.indices().to_primitive(); + fn with_random_invalid_indices( + rle: &RLEArray, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + let indices_prim = rle.indices().clone().execute::(ctx)?; let mut indices_data: Vec = indices_prim.as_slice::().to_vec(); // Use a simple deterministic "random" sequence. @@ -394,43 +463,47 @@ mod tests { #[test] fn test_random_invalid_indices_all_null_chunk() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values: Vec> = vec![None; FL_CHUNK_SIZE]; let original = PrimitiveArray::from_option_iter(values); - let rle = RLEData::encode(&original)?; - let clobbered = with_random_invalid_indices(&rle)?; + let rle = RLEData::encode(original.as_view(), &mut ctx)?; + let clobbered = with_random_invalid_indices(&rle, &mut ctx)?; assert_arrays_eq!(clobbered, original); Ok(()) } #[test] fn test_random_invalid_indices_sparse_values() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let mut values: Vec> = vec![None; FL_CHUNK_SIZE]; values[0] = Some(10); values[500] = Some(20); values[1000] = Some(30); let original = PrimitiveArray::from_option_iter(values); - let rle = RLEData::encode(&original)?; - let clobbered = with_random_invalid_indices(&rle)?; + let rle = RLEData::encode(original.as_view(), &mut ctx)?; + let clobbered = with_random_invalid_indices(&rle, &mut ctx)?; assert_arrays_eq!(clobbered, original); Ok(()) } #[test] fn test_random_invalid_indices_multi_chunk() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Two chunks: first has scattered values, second is all null. let mut values: Vec> = vec![None; 2 * FL_CHUNK_SIZE]; values[0] = Some(10); values[500] = Some(20); values[FL_CHUNK_SIZE + 100] = Some(42); let original = PrimitiveArray::from_option_iter(values); - let rle = RLEData::encode(&original)?; - let clobbered = with_random_invalid_indices(&rle)?; + let rle = RLEData::encode(original.as_view(), &mut ctx)?; + let clobbered = with_random_invalid_indices(&rle, &mut ctx)?; assert_arrays_eq!(clobbered, original); Ok(()) } #[test] fn test_random_invalid_indices_partial_last_chunk() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // 1085 elements: chunk 0 has values at scattered positions, chunk 1 is // a partial (61 elements padded to 1024) that is entirely null. let mut values: Vec> = vec![None; 1085]; @@ -438,14 +511,15 @@ mod tests { values[i] = Some(i as u32); } let original = PrimitiveArray::from_option_iter(values); - let rle = RLEData::encode(&original)?; - let clobbered = with_random_invalid_indices(&rle)?; + let rle = RLEData::encode(original.as_view(), &mut ctx)?; + let clobbered = with_random_invalid_indices(&rle, &mut ctx)?; assert_arrays_eq!(clobbered, original); Ok(()) } #[test] fn test_random_invalid_indices_mostly_valid() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Most positions are valid, only a few are null with garbage indices. let mut values: Vec> = (0..FL_CHUNK_SIZE).map(|i| Some((i / 100) as u64)).collect(); @@ -454,8 +528,8 @@ mod tests { values[i] = None; } let original = PrimitiveArray::from_option_iter(values); - let rle = RLEData::encode(&original)?; - let clobbered = with_random_invalid_indices(&rle)?; + let rle = RLEData::encode(original.as_view(), &mut ctx)?; + let clobbered = with_random_invalid_indices(&rle, &mut ctx)?; assert_arrays_eq!(clobbered, original); Ok(()) } @@ -466,10 +540,14 @@ mod tests { #[case(vec![f16::ZERO, f16::NEG_ZERO])] #[case(vec![0f32, -0f32])] #[case(vec![0f64, -0f64])] - fn test_float_zeros(#[case] values: Vec) { + fn test_float_zeros( + #[case] values: Vec, + ) -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let primitive = PrimitiveArray::from_iter(values); - let rle = RLEData::encode(&primitive).unwrap(); - let decoded = rle.as_array().to_primitive(); + let rle = RLEData::encode(primitive.as_view(), &mut ctx)?; + let decoded = rle.as_array().clone().execute::(&mut ctx)?; assert_arrays_eq!(primitive, decoded); + Ok(()) } } diff --git a/encodings/fastlanes/src/rle/array/rle_decompress.rs b/encodings/fastlanes/src/rle/array/rle_decompress.rs index d9c31b5de60..7240f3ff71f 100644 --- a/encodings/fastlanes/src/rle/array/rle_decompress.rs +++ b/encodings/fastlanes/src/rle/array/rle_decompress.rs @@ -5,12 +5,10 @@ use fastlanes::RLE; use num_traits::AsPrimitive; use num_traits::NumCast; use vortex_array::ExecutionCtx; -use vortex_array::IntoArray; use vortex_array::arrays::PrimitiveArray; use vortex_array::dtype::NativePType; use vortex_array::match_each_native_ptype; use vortex_array::match_each_unsigned_integer_ptype; -use vortex_array::validity::Validity; use vortex_buffer::BufferMut; use vortex_error::VortexExpect; use vortex_error::VortexResult; @@ -56,7 +54,7 @@ where let indices = array.indices().clone().execute::(ctx)?; assert!(indices.len().is_multiple_of(FL_CHUNK_SIZE)); - let has_invalid = !indices.all_valid()?; + let has_invalid = !indices.all_valid(ctx)?; let (indices_sl, _) = indices.as_slice::().as_chunks::(); let chunk_start_idx = array.offset() / FL_CHUNK_SIZE; @@ -124,6 +122,6 @@ where buffer .freeze() .slice(offset_within_chunk..(offset_within_chunk + array.len())), - Validity::copy_from_array(&array.clone().into_array())?, + array.validity()?, )) } diff --git a/encodings/fastlanes/src/rle/compute/cast.rs b/encodings/fastlanes/src/rle/compute/cast.rs index 95286d2f3e7..e01d359e59b 100644 --- a/encodings/fastlanes/src/rle/compute/cast.rs +++ b/encodings/fastlanes/src/rle/compute/cast.rs @@ -12,6 +12,7 @@ use vortex_error::VortexResult; use crate::rle::RLE; use crate::rle::RLEArrayExt; + impl CastReduce for RLE { fn cast(array: ArrayView<'_, Self>, dtype: &DType) -> VortexResult> { // Cast RLE values. @@ -20,14 +21,12 @@ impl CastReduce for RLE { .cast(DType::Primitive(dtype.as_ptype(), Nullability::NonNullable))?; // Cast RLE indices such that validity matches the target dtype. - let casted_indices = if array.indices().dtype().nullability() != dtype.nullability() { - array.indices().cast(DType::Primitive( - array.indices().dtype().as_ptype(), - dtype.nullability(), - ))? - } else { - array.indices().clone() - }; + let casted_indices = array.indices().cast( + array + .indices() + .dtype() + .with_nullability(dtype.nullability()), + )?; Ok(Some( RLE::try_new( @@ -44,8 +43,14 @@ impl CastReduce for RLE { #[cfg(test)] mod tests { + use std::sync::LazyLock; + use rstest::rstest; + use vortex_array::Canonical; + use vortex_array::ExecutionCtx; use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; use vortex_array::builtins::ArrayBuiltins; @@ -53,23 +58,29 @@ mod tests { use vortex_array::dtype::DType; use vortex_array::dtype::Nullability; use vortex_array::dtype::PType; + use vortex_array::session::ArraySession; use vortex_array::validity::Validity; use vortex_buffer::Buffer; + use vortex_session::VortexSession; use crate::RLEData; use crate::rle::RLEArray; - fn rle(primitive: &PrimitiveArray) -> RLEArray { - RLEData::encode(primitive).unwrap() + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); + + fn rle(primitive: &PrimitiveArray, ctx: &mut ExecutionCtx) -> RLEArray { + RLEData::encode(primitive.as_view(), ctx).unwrap() } #[test] fn try_cast_rle_success() { + let mut ctx = SESSION.create_execution_ctx(); let primitive = PrimitiveArray::new( Buffer::from_iter([10u8, 20, 30, 40, 50]), Validity::from_iter([true, true, true, true, true]), ); - let encoded = rle(&primitive); + let encoded = rle(&primitive, &mut ctx); let casted = encoded .into_array() @@ -81,16 +92,17 @@ mod tests { #[test] #[should_panic] fn try_cast_rle_fail() { + let mut ctx = SESSION.create_execution_ctx(); let primitive = PrimitiveArray::new( Buffer::from_iter([10u8, 20, 30, 40, 50]), Validity::from_iter([true, false, true, true, false]), ); - let encoded = rle(&primitive); - encoded + let encoded = rle(&primitive, &mut ctx); + let result = encoded .into_array() .cast(DType::Primitive(PType::U8, Nullability::NonNullable)) - .and_then(|a| a.to_canonical().map(|c| c.into_array())) - .unwrap(); + .and_then(|a| a.execute::(&mut ctx).map(|c| c.into_array())); + result.unwrap(); } #[rstest] @@ -143,7 +155,8 @@ mod tests { ) )] fn test_cast_rle_conformance(#[case] primitive: PrimitiveArray) { - let rle_array = rle(&primitive); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let rle_array = rle(&primitive, &mut ctx); test_cast_conformance(&rle_array.into_array()); } } diff --git a/encodings/fastlanes/src/rle/kernel.rs b/encodings/fastlanes/src/rle/kernel.rs index 14eaf6dda3d..f5b654706cd 100644 --- a/encodings/fastlanes/src/rle/kernel.rs +++ b/encodings/fastlanes/src/rle/kernel.rs @@ -23,15 +23,15 @@ impl SliceKernel for RLE { fn slice( array: ArrayView<'_, Self>, range: Range, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult> { let offset_in_chunk = array.offset(); let chunk_start_idx = (offset_in_chunk + range.start) / FL_CHUNK_SIZE; let chunk_end_idx = (offset_in_chunk + range.end).div_ceil(FL_CHUNK_SIZE); - let values_start_idx = array.values_idx_offset(chunk_start_idx); + let values_start_idx = array.values_idx_offset(chunk_start_idx, ctx); let values_end_idx = if chunk_end_idx < array.values_idx_offsets().len() { - array.values_idx_offset(chunk_end_idx) + array.values_idx_offset(chunk_end_idx, ctx) } else { array.values().len() }; diff --git a/encodings/fastlanes/src/rle/vtable/mod.rs b/encodings/fastlanes/src/rle/vtable/mod.rs index c0e19efd3af..ed549301433 100644 --- a/encodings/fastlanes/src/rle/vtable/mod.rs +++ b/encodings/fastlanes/src/rle/vtable/mod.rs @@ -16,7 +16,7 @@ use vortex_array::ExecutionCtx; use vortex_array::ExecutionResult; use vortex_array::IntoArray; use vortex_array::Precision; -use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::Primitive; use vortex_array::buffer::BufferHandle; use vortex_array::dtype::DType; use vortex_array::dtype::Nullability; @@ -28,6 +28,7 @@ use vortex_error::VortexResult; use vortex_error::vortex_ensure; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::RLEData; use crate::rle::array::INDICES_SLOT; @@ -80,7 +81,8 @@ impl VTable for RLE { type ValidityVTable = Self; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("fastlanes.rle"); + *ID } fn validate( @@ -209,8 +211,6 @@ impl VTable for RLE { pub struct RLE; impl RLE { - pub const ID: ArrayId = ArrayId::new_ref("fastlanes.rle"); - pub fn try_new( values: ArrayRef, indices: ArrayRef, @@ -244,8 +244,11 @@ impl RLE { } /// Encode a primitive array using FastLanes RLE. - pub fn encode(array: &PrimitiveArray) -> VortexResult { - RLEData::encode(array) + pub fn encode( + array: ArrayView<'_, Primitive>, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + RLEData::encode(array, ctx) } } diff --git a/encodings/fastlanes/src/rle/vtable/operations.rs b/encodings/fastlanes/src/rle/vtable/operations.rs index 25069ecccc1..0441a1e61b9 100644 --- a/encodings/fastlanes/src/rle/vtable/operations.rs +++ b/encodings/fastlanes/src/rle/vtable/operations.rs @@ -16,10 +16,12 @@ impl OperationsVTable for RLE { fn scalar_at( array: ArrayView<'_, RLE>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { let offset_in_chunk = array.offset(); - let chunk_relative_idx = array.indices().scalar_at(offset_in_chunk + index)?; + let chunk_relative_idx = array + .indices() + .execute_scalar(offset_in_chunk + index, ctx)?; let chunk_relative_idx = chunk_relative_idx .as_primitive() @@ -27,11 +29,11 @@ impl OperationsVTable for RLE { .vortex_expect("Index must not be null"); let chunk_id = (offset_in_chunk + index) / FL_CHUNK_SIZE; - let value_idx_offset = array.values_idx_offset(chunk_id); + let value_idx_offset = array.values_idx_offset(chunk_id, ctx); let scalar = array .values() - .scalar_at(value_idx_offset + chunk_relative_idx)?; + .execute_scalar(value_idx_offset + chunk_relative_idx, ctx)?; Scalar::try_new(array.dtype().clone(), scalar.into_value()) } @@ -40,7 +42,8 @@ impl OperationsVTable for RLE { #[cfg(test)] mod tests { use vortex_array::IntoArray; - use vortex_array::ToCanonical; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; use vortex_array::validity::Validity; @@ -165,19 +168,21 @@ mod tests { #[test] fn test_scalar_at_multiple_chunks() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Test accessing elements around chunk boundaries let values: Buffer = (0..3000).map(|i| (i / 50) as u16).collect(); let expected: Vec = (0..3000).map(|i| (i / 50) as u16).collect(); let array = values.into_array(); - let encoded = RLEData::encode(&array.to_primitive()).unwrap(); + let primitive = array.execute::(&mut ctx).unwrap(); + let encoded = RLEData::encode(primitive.as_view(), &mut ctx).unwrap(); // Access scalars from multiple chunks. for &idx in &[1023, 1024, 1025, 2047, 2048, 2049] { if idx < encoded.len() { let original_value = expected[idx]; let encoded_value = encoded - .scalar_at(idx) + .execute_scalar(idx, &mut ctx) .unwrap() .as_primitive() .as_::() @@ -191,14 +196,18 @@ mod tests { #[should_panic] fn test_scalar_at_out_of_bounds() { let array = fixture::rle_array(); - array.scalar_at(1025).unwrap(); + array + .execute_scalar(1025, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); } #[test] #[should_panic] fn test_scalar_at_slice_out_of_bounds() { let array = fixture::rle_array().slice(0..1).unwrap(); - array.scalar_at(1).unwrap(); + array + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); } #[test] @@ -255,8 +264,13 @@ mod tests { #[test] fn test_slice_decode_with_nulls() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = fixture::rle_array_with_nulls(); - let sliced = array.slice(1..4).unwrap().to_primitive(); // [null, 20, 20] + let sliced = array + .slice(1..4) + .unwrap() + .execute::(&mut ctx) + .unwrap(); // [null, 20, 20] let expected = PrimitiveArray::from_option_iter([Option::::None, Some(20), Some(20)]); assert_arrays_eq!(sliced.into_array(), expected.into_array()); @@ -272,11 +286,13 @@ mod tests { #[test] fn test_slice_across_chunk_boundaries() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values: Buffer = (0..2100).map(|i| (i / 100) as u32).collect(); let expected: Vec = (0..2100).map(|i| (i / 100) as u32).collect(); let array = values.into_array(); - let encoded = RLEData::encode(&array.to_primitive()).unwrap(); + let primitive = array.execute::(&mut ctx).unwrap(); + let encoded = RLEData::encode(primitive.as_view(), &mut ctx).unwrap(); // Slice across first and second chunk. let slice = encoded.slice(500..1500).unwrap(); diff --git a/encodings/fsst/benches/chunked_dict_fsst_builder.rs b/encodings/fsst/benches/chunked_dict_fsst_builder.rs index 50e2488d360..22131631a6e 100644 --- a/encodings/fsst/benches/chunked_dict_fsst_builder.rs +++ b/encodings/fsst/benches/chunked_dict_fsst_builder.rs @@ -5,6 +5,7 @@ use std::sync::LazyLock; use divan::Bencher; use vortex_array::ArrayRef; +use vortex_array::Canonical; use vortex_array::IntoArray; use vortex_array::VortexSessionExecute; use vortex_array::arrays::ChunkedArray; @@ -36,8 +37,9 @@ fn make_dict_fsst_chunks( unique_values: usize, chunk_count: usize, ) -> ArrayRef { + let mut ctx = SESSION.create_execution_ctx(); (0..chunk_count) - .map(|_| gen_dict_fsst_test_data::(len, unique_values, 20, 30).into_array()) + .map(|_| gen_dict_fsst_test_data::(len, unique_values, 20, 30, &mut ctx).into_array()) .collect::() .into_array() } @@ -49,13 +51,15 @@ fn chunked_dict_fsst_canonical_into( ) { let chunk = make_dict_fsst_chunks::(len, unique_values, chunk_count); - bencher.with_inputs(|| &chunk).bench_refs(|chunk| { - let mut builder = builder_with_capacity(chunk.dtype(), len * chunk_count); - chunk - .append_to_builder(builder.as_mut(), &mut SESSION.create_execution_ctx()) - .vortex_expect("append failed"); - builder.finish() - }) + bencher + .with_inputs(|| (&chunk, SESSION.create_execution_ctx())) + .bench_refs(|(chunk, ctx)| { + let mut builder = builder_with_capacity(chunk.dtype(), len * chunk_count); + chunk + .append_to_builder(builder.as_mut(), ctx) + .vortex_expect("append failed"); + builder.finish() + }) } #[divan::bench(args = BENCH_ARGS)] @@ -66,6 +70,6 @@ fn chunked_dict_fsst_into_canonical( let chunk = make_dict_fsst_chunks::(len, unique_values, chunk_count); bencher - .with_inputs(|| &chunk) - .bench_refs(|chunk| chunk.to_canonical()) + .with_inputs(|| (&chunk, SESSION.create_execution_ctx())) + .bench_refs(|(chunk, ctx)| (**chunk).clone().execute::(ctx)) } diff --git a/encodings/fsst/benches/fsst_compress.rs b/encodings/fsst/benches/fsst_compress.rs index e84eef9cbce..45aedfa464f 100644 --- a/encodings/fsst/benches/fsst_compress.rs +++ b/encodings/fsst/benches/fsst_compress.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] +#![expect(clippy::unwrap_used)] use std::sync::LazyLock; @@ -9,6 +9,7 @@ use divan::Bencher; use rand::RngExt; use rand::SeedableRng; use rand::rngs::StdRng; +use vortex_array::Canonical; use vortex_array::IntoArray; use vortex_array::LEGACY_SESSION; use vortex_array::RecursiveCanonical; @@ -56,9 +57,9 @@ fn compress_fsst(bencher: Bencher, (string_count, avg_len, unique_chars): (usize let array = generate_test_data(string_count, avg_len, unique_chars); let compressor = fsst_train_compressor(&array); bencher - .with_inputs(|| (&array, &compressor)) - .bench_refs(|(array, compressor)| { - fsst_compress(*array, array.len(), array.dtype(), compressor) + .with_inputs(|| (&array, &compressor, LEGACY_SESSION.create_execution_ctx())) + .bench_refs(|(array, compressor, ctx)| { + fsst_compress(*array, array.len(), array.dtype(), compressor, ctx) }) } @@ -68,11 +69,17 @@ fn decompress_fsst(bencher: Bencher, (string_count, avg_len, unique_chars): (usi let compressor = fsst_train_compressor(&array); let len = array.len(); let dtype = array.dtype().clone(); - let encoded = fsst_compress(array, len, &dtype, &compressor); + let encoded = fsst_compress( + array, + len, + &dtype, + &compressor, + &mut LEGACY_SESSION.create_execution_ctx(), + ); bencher - .with_inputs(|| &encoded) - .bench_refs(|encoded| encoded.to_canonical()) + .with_inputs(|| (&encoded, LEGACY_SESSION.create_execution_ctx())) + .bench_refs(|(encoded, ctx)| (**encoded).clone().into_array().execute::(ctx)) } #[divan::bench(args = BENCH_ARGS)] @@ -87,7 +94,13 @@ fn train_compressor(bencher: Bencher, (string_count, avg_len, unique_chars): (us fn pushdown_compare(bencher: Bencher, (string_count, avg_len, unique_chars): (usize, usize, u8)) { let array = generate_test_data(string_count, avg_len, unique_chars); let compressor = fsst_train_compressor(&array); - let fsst_array = fsst_compress(&array, array.len(), array.dtype(), &compressor); + let fsst_array = fsst_compress( + &array, + array.len(), + array.dtype(), + &compressor, + &mut LEGACY_SESSION.create_execution_ctx(), + ); let constant = ConstantArray::new(Scalar::from(&b"const"[..]), array.len()); bencher @@ -116,7 +129,13 @@ fn canonicalize_compare( ) { let array = generate_test_data(string_count, avg_len, unique_chars); let compressor = fsst_train_compressor(&array); - let fsst_array = fsst_compress(&array, array.len(), array.dtype(), &compressor); + let fsst_array = fsst_compress( + &array, + array.len(), + array.dtype(), + &compressor, + &mut LEGACY_SESSION.create_execution_ctx(), + ); let constant = ConstantArray::new(Scalar::from(&b"const"[..]), array.len()); bencher @@ -128,8 +147,10 @@ fn canonicalize_compare( ) }) .bench_refs(|(fsst_array, constant, ctx)| { - fsst_array - .to_canonical() + (*fsst_array) + .clone() + .into_array() + .execute::(ctx) .unwrap() .into_array() .binary(constant.clone().into_array(), Operator::Eq) @@ -179,8 +200,8 @@ fn chunked_into_canonical( let array = generate_chunked_test_data(chunk_size, string_count, avg_len, unique_chars); bencher - .with_inputs(|| &array) - .bench_refs(|array| array.to_canonical()); + .with_inputs(|| (&array, SESSION.create_execution_ctx())) + .bench_refs(|(array, ctx)| (**array).clone().into_array().execute::(ctx)); } /// Helper function to generate random string data. @@ -214,13 +235,14 @@ fn generate_chunked_test_data( avg_len: usize, unique_chars: u8, ) -> ChunkedArray { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); (0..chunk_size) .map(|_| { let array = generate_test_data(string_count, avg_len, unique_chars); let compressor = fsst_train_compressor(&array); let len = array.len(); let dtype = array.dtype().clone(); - fsst_compress(array, len, &dtype, &compressor).into_array() + fsst_compress(array, len, &dtype, &compressor, &mut ctx).into_array() }) .collect::() } diff --git a/encodings/fsst/benches/fsst_like.rs b/encodings/fsst/benches/fsst_like.rs index b4b0e959351..28aa3b109ae 100644 --- a/encodings/fsst/benches/fsst_like.rs +++ b/encodings/fsst/benches/fsst_like.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] +#![expect(clippy::unwrap_used)] use std::fmt; use std::sync::LazyLock; @@ -35,13 +35,20 @@ static SESSION: LazyLock = const N: usize = NUM_STRINGS; -static FSST_URLS: LazyLock = LazyLock::new(|| make_fsst_short_urls(N)); -static FSST_CB_URLS: LazyLock = LazyLock::new(|| make_fsst_clickbench_urls(N)); -static FSST_LOG_LINES: LazyLock = LazyLock::new(|| make_fsst_log_lines(N)); -static FSST_JSON_STRINGS: LazyLock = LazyLock::new(|| make_fsst_json_strings(N)); -static FSST_FILE_PATHS: LazyLock = LazyLock::new(|| make_fsst_file_paths(N)); -static FSST_EMAILS: LazyLock = LazyLock::new(|| make_fsst_emails(N)); -static FSST_RARE_MATCH: LazyLock = LazyLock::new(|| make_fsst_rare_match(N)); +static FSST_URLS: LazyLock = + LazyLock::new(|| make_fsst_short_urls(N, &mut SESSION.create_execution_ctx())); +static FSST_CB_URLS: LazyLock = + LazyLock::new(|| make_fsst_clickbench_urls(N, &mut SESSION.create_execution_ctx())); +static FSST_LOG_LINES: LazyLock = + LazyLock::new(|| make_fsst_log_lines(N, &mut SESSION.create_execution_ctx())); +static FSST_JSON_STRINGS: LazyLock = + LazyLock::new(|| make_fsst_json_strings(N, &mut SESSION.create_execution_ctx())); +static FSST_FILE_PATHS: LazyLock = + LazyLock::new(|| make_fsst_file_paths(N, &mut SESSION.create_execution_ctx())); +static FSST_EMAILS: LazyLock = + LazyLock::new(|| make_fsst_emails(N, &mut SESSION.create_execution_ctx())); +static FSST_RARE_MATCH: LazyLock = + LazyLock::new(|| make_fsst_rare_match(N, &mut SESSION.create_execution_ctx())); enum Dataset { Urls, @@ -109,13 +116,15 @@ fn bench_like(bencher: Bencher, fsst: &FSSTArray, pattern: &str) { let len = fsst.len(); let arr = fsst.clone().into_array(); let pattern = ConstantArray::new(pattern, len).into_array(); - bencher.bench_local(|| { - Like.try_new_array(len, LikeOptions::default(), [arr.clone(), pattern.clone()]) - .unwrap() - .into_array() - .execute::(&mut SESSION.create_execution_ctx()) - .unwrap() - }); + bencher + .with_inputs(|| SESSION.create_execution_ctx()) + .bench_refs(|ctx| { + Like.try_new_array(len, LikeOptions::default(), [arr.clone(), pattern.clone()]) + .unwrap() + .into_array() + .execute::(ctx) + .unwrap() + }); } #[divan::bench(args = [ diff --git a/encodings/fsst/benches/fsst_url_compare.rs b/encodings/fsst/benches/fsst_url_compare.rs index a8572650efb..656cd9f1866 100644 --- a/encodings/fsst/benches/fsst_url_compare.rs +++ b/encodings/fsst/benches/fsst_url_compare.rs @@ -1,11 +1,12 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] +#![expect(clippy::unwrap_used)] use std::sync::LazyLock; use divan::Bencher; +use vortex_array::Canonical; use vortex_array::IntoArray; use vortex_array::RecursiveCanonical; use vortex_array::VortexSessionExecute; @@ -56,7 +57,13 @@ fn pick_url_with_domain(data: &VarBinArray, domain: &str) -> String { fn eq_pushdown_high_match(bencher: Bencher) { let data = &*URL_DATA; let compressor = fsst_train_compressor(data); - let fsst_array = fsst_compress(data, data.len(), data.dtype(), &compressor); + let fsst_array = fsst_compress( + data, + data.len(), + data.dtype(), + &compressor, + &mut SESSION.create_execution_ctx(), + ); let match_url = pick_url_with_domain(data, HIGH_MATCH_DOMAIN); let constant = ConstantArray::new(Scalar::from(match_url.as_str()), NUM_URLS); @@ -77,7 +84,13 @@ fn eq_pushdown_high_match(bencher: Bencher) { fn eq_pushdown_low_match(bencher: Bencher) { let data = &*URL_DATA; let compressor = fsst_train_compressor(data); - let fsst_array = fsst_compress(data, data.len(), data.dtype(), &compressor); + let fsst_array = fsst_compress( + data, + data.len(), + data.dtype(), + &compressor, + &mut SESSION.create_execution_ctx(), + ); let match_url = pick_url_with_domain(data, LOW_MATCH_DOMAIN); let constant = ConstantArray::new(Scalar::from(match_url.as_str()), NUM_URLS); @@ -98,15 +111,23 @@ fn eq_pushdown_low_match(bencher: Bencher) { fn eq_canonicalize_high_match(bencher: Bencher) { let data = &*URL_DATA; let compressor = fsst_train_compressor(data); - let fsst_array = fsst_compress(data, data.len(), data.dtype(), &compressor); + let fsst_array = fsst_compress( + data, + data.len(), + data.dtype(), + &compressor, + &mut SESSION.create_execution_ctx(), + ); let match_url = pick_url_with_domain(data, HIGH_MATCH_DOMAIN); let constant = ConstantArray::new(Scalar::from(match_url.as_str()), NUM_URLS); bencher .with_inputs(|| (&fsst_array, &constant, SESSION.create_execution_ctx())) .bench_refs(|(fsst_array, constant, ctx)| { - fsst_array - .to_canonical() + (*fsst_array) + .clone() + .into_array() + .execute::(ctx) .unwrap() .into_array() .binary(constant.clone().into_array(), Operator::Eq) @@ -120,15 +141,23 @@ fn eq_canonicalize_high_match(bencher: Bencher) { fn eq_canonicalize_low_match(bencher: Bencher) { let data = &*URL_DATA; let compressor = fsst_train_compressor(data); - let fsst_array = fsst_compress(data, data.len(), data.dtype(), &compressor); + let fsst_array = fsst_compress( + data, + data.len(), + data.dtype(), + &compressor, + &mut SESSION.create_execution_ctx(), + ); let match_url = pick_url_with_domain(data, LOW_MATCH_DOMAIN); let constant = ConstantArray::new(Scalar::from(match_url.as_str()), NUM_URLS); bencher .with_inputs(|| (&fsst_array, &constant, SESSION.create_execution_ctx())) .bench_refs(|(fsst_array, constant, ctx)| { - fsst_array - .to_canonical() + (*fsst_array) + .clone() + .into_array() + .execute::(ctx) .unwrap() .into_array() .binary(constant.clone().into_array(), Operator::Eq) @@ -146,7 +175,13 @@ fn eq_canonicalize_low_match(bencher: Bencher) { fn like_substr_high_match(bencher: Bencher) { let data = &*URL_DATA; let compressor = fsst_train_compressor(data); - let fsst_array = fsst_compress(data, data.len(), data.dtype(), &compressor); + let fsst_array = fsst_compress( + data, + data.len(), + data.dtype(), + &compressor, + &mut SESSION.create_execution_ctx(), + ); let pattern = format!("%{HIGH_MATCH_DOMAIN}%"); let expr = like(root(), lit(pattern.as_str())); @@ -167,7 +202,13 @@ fn like_substr_high_match(bencher: Bencher) { fn like_substr_low_match(bencher: Bencher) { let data = &*URL_DATA; let compressor = fsst_train_compressor(data); - let fsst_array = fsst_compress(data, data.len(), data.dtype(), &compressor); + let fsst_array = fsst_compress( + data, + data.len(), + data.dtype(), + &compressor, + &mut SESSION.create_execution_ctx(), + ); let pattern = format!("%{LOW_MATCH_DOMAIN}%"); let expr = like(root(), lit(pattern.as_str())); diff --git a/encodings/fsst/public-api.lock b/encodings/fsst/public-api.lock index 341a2fd8222..bc2400e19d6 100644 --- a/encodings/fsst/public-api.lock +++ b/encodings/fsst/public-api.lock @@ -4,9 +4,7 @@ pub struct vortex_fsst::FSST impl vortex_fsst::FSST -pub const vortex_fsst::FSST::ID: vortex_array::array::ArrayId - -pub fn vortex_fsst::FSST::try_new(dtype: vortex_array::dtype::DType, symbols: vortex_buffer::buffer::Buffer, symbol_lengths: vortex_buffer::buffer::Buffer, codes: vortex_array::arrays::varbin::vtable::VarBinArray, uncompressed_lengths: vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_fsst::FSST::try_new(vortex_array::dtype::DType, vortex_buffer::buffer::Buffer, vortex_buffer::buffer::Buffer, vortex_array::arrays::varbin::vtable::VarBinArray, vortex_array::array::erased::ArrayRef, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult impl core::clone::Clone for vortex_fsst::FSST @@ -14,7 +12,7 @@ pub fn vortex_fsst::FSST::clone(&self) -> vortex_fsst::FSST impl core::fmt::Debug for vortex_fsst::FSST -pub fn vortex_fsst::FSST::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_fsst::FSST::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::array::vtable::VTable for vortex_fsst::FSST @@ -24,61 +22,65 @@ pub type vortex_fsst::FSST::OperationsVTable = vortex_fsst::FSST pub type vortex_fsst::FSST::ValidityVTable = vortex_fsst::FSST -pub fn vortex_fsst::FSST::append_to_builder(array: vortex_array::array::view::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_fsst::FSST::append_to_builder(vortex_array::array::view::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_fsst::FSST::buffer(array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_fsst::FSST::buffer(vortex_array::array::view::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_fsst::FSST::buffer_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_fsst::FSST::buffer_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_fsst::FSST::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_fsst::FSST::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_fsst::FSST::execute(array: vortex_array::array::typed::Array, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_fsst::FSST::execute(vortex_array::array::typed::Array, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_fsst::FSST::execute_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_fsst::FSST::execute_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_fsst::FSST::id(&self) -> vortex_array::array::ArrayId -pub fn vortex_fsst::FSST::nbuffers(_array: vortex_array::array::view::ArrayView<'_, Self>) -> usize +pub fn vortex_fsst::FSST::nbuffers(vortex_array::array::view::ArrayView<'_, Self>) -> usize -pub fn vortex_fsst::FSST::reduce_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_fsst::FSST::reduce_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_fsst::FSST::serialize(array: vortex_array::array::view::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_fsst::FSST::serialize(vortex_array::array::view::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_fsst::FSST::slot_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_fsst::FSST::slot_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_fsst::FSST::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_fsst::FSST::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::array::vtable::operations::OperationsVTable for vortex_fsst::FSST -pub fn vortex_fsst::FSST::scalar_at(array: vortex_array::array::view::ArrayView<'_, vortex_fsst::FSST>, index: usize, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_fsst::FSST::scalar_at(vortex_array::array::view::ArrayView<'_, vortex_fsst::FSST>, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::array::vtable::validity::ValidityVTable for vortex_fsst::FSST -pub fn vortex_fsst::FSST::validity(array: vortex_array::array::view::ArrayView<'_, vortex_fsst::FSST>) -> vortex_error::VortexResult +pub fn vortex_fsst::FSST::validity(vortex_array::array::view::ArrayView<'_, vortex_fsst::FSST>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::take::TakeExecute for vortex_fsst::FSST -pub fn vortex_fsst::FSST::take(array: vortex_array::array::view::ArrayView<'_, Self>, indices: &vortex_array::array::erased::ArrayRef, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_fsst::FSST::take(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::kernel::FilterKernel for vortex_fsst::FSST -pub fn vortex_fsst::FSST::filter(array: vortex_array::array::view::ArrayView<'_, Self>, mask: &vortex_mask::Mask, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_fsst::FSST::filter(vortex_array::array::view::ArrayView<'_, Self>, &vortex_mask::Mask, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_fsst::FSST -pub fn vortex_fsst::FSST::slice(array: vortex_array::array::view::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_fsst::FSST::slice(vortex_array::array::view::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::binary::compare::CompareKernel for vortex_fsst::FSST -pub fn vortex_fsst::FSST::compare(lhs: vortex_array::array::view::ArrayView<'_, Self>, rhs: &vortex_array::array::erased::ArrayRef, operator: vortex_array::scalar_fn::fns::operators::CompareOperator, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_fsst::FSST::compare(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, vortex_array::scalar_fn::fns::operators::CompareOperator, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::kernel::CastKernel for vortex_fsst::FSST + +pub fn vortex_fsst::FSST::cast(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::dtype::DType, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::kernel::CastReduce for vortex_fsst::FSST -pub fn vortex_fsst::FSST::cast(array: vortex_array::array::view::ArrayView<'_, Self>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_fsst::FSST::cast(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::like::kernel::LikeKernel for vortex_fsst::FSST -pub fn vortex_fsst::FSST::like(array: vortex_array::array::view::ArrayView<'_, Self>, pattern: &vortex_array::array::erased::ArrayRef, options: vortex_array::scalar_fn::fns::like::LikeOptions, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_fsst::FSST::like(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, vortex_array::scalar_fn::fns::like::LikeOptions, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> pub struct vortex_fsst::FSSTData @@ -100,9 +102,9 @@ pub fn vortex_fsst::FSSTData::symbol_lengths(&self) -> &vortex_buffer::buffer::B pub fn vortex_fsst::FSSTData::symbols(&self) -> &vortex_buffer::buffer::Buffer -pub fn vortex_fsst::FSSTData::try_new(symbols: vortex_buffer::buffer::Buffer, symbol_lengths: vortex_buffer::buffer::Buffer, codes_bytes: vortex_array::buffer::BufferHandle, len: usize) -> vortex_error::VortexResult +pub fn vortex_fsst::FSSTData::try_new(vortex_buffer::buffer::Buffer, vortex_buffer::buffer::Buffer, vortex_array::buffer::BufferHandle, usize) -> vortex_error::VortexResult -pub fn vortex_fsst::FSSTData::validate(&self, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_fsst::FSSTData::validate(&self, &vortex_array::dtype::DType, usize, &[core::option::Option], &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult<()> impl core::clone::Clone for vortex_fsst::FSSTData @@ -110,19 +112,19 @@ pub fn vortex_fsst::FSSTData::clone(&self) -> vortex_fsst::FSSTData impl core::fmt::Debug for vortex_fsst::FSSTData -pub fn vortex_fsst::FSSTData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_fsst::FSSTData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_fsst::FSSTData -pub fn vortex_fsst::FSSTData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_fsst::FSSTData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::hash::ArrayEq for vortex_fsst::FSSTData -pub fn vortex_fsst::FSSTData::array_eq(&self, other: &Self, precision: vortex_array::hash::Precision) -> bool +pub fn vortex_fsst::FSSTData::array_eq(&self, &Self, vortex_array::hash::Precision) -> bool impl vortex_array::hash::ArrayHash for vortex_fsst::FSSTData -pub fn vortex_fsst::FSSTData::array_hash(&self, state: &mut H, precision: vortex_array::hash::Precision) +pub fn vortex_fsst::FSSTData::array_hash(&self, &mut H, vortex_array::hash::Precision) pub struct vortex_fsst::FSSTMetadata @@ -130,9 +132,9 @@ impl vortex_fsst::FSSTMetadata pub fn vortex_fsst::FSSTMetadata::codes_offsets_ptype(&self) -> vortex_array::dtype::ptype::PType -pub fn vortex_fsst::FSSTMetadata::set_codes_offsets_ptype(&mut self, value: vortex_array::dtype::ptype::PType) +pub fn vortex_fsst::FSSTMetadata::set_codes_offsets_ptype(&mut self, vortex_array::dtype::ptype::PType) -pub fn vortex_fsst::FSSTMetadata::set_uncompressed_lengths_ptype(&mut self, value: vortex_array::dtype::ptype::PType) +pub fn vortex_fsst::FSSTMetadata::set_uncompressed_lengths_ptype(&mut self, vortex_array::dtype::ptype::PType) pub fn vortex_fsst::FSSTMetadata::uncompressed_lengths_ptype(&self) -> vortex_array::dtype::ptype::PType @@ -150,7 +152,7 @@ pub fn vortex_fsst::FSSTMetadata::default() -> Self impl core::fmt::Debug for vortex_fsst::FSSTMetadata -pub fn vortex_fsst::FSSTMetadata::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_fsst::FSSTMetadata::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl prost::message::Message for vortex_fsst::FSSTMetadata @@ -178,10 +180,10 @@ pub fn T::uncompressed_lengths(&self) -> &vortex_array::array::erased::ArrayRef pub fn T::uncompressed_lengths_dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_fsst::fsst_compress>(strings: A, len: usize, dtype: &vortex_array::dtype::DType, compressor: &fsst::Compressor) -> vortex_fsst::FSSTArray +pub fn vortex_fsst::fsst_compress>(A, usize, &vortex_array::dtype::DType, &fsst::Compressor, &mut vortex_array::executor::ExecutionCtx) -> vortex_fsst::FSSTArray -pub fn vortex_fsst::fsst_compress_iter<'a, I>(iter: I, len: usize, dtype: vortex_array::dtype::DType, compressor: &fsst::Compressor) -> vortex_fsst::FSSTArray where I: core::iter::traits::iterator::Iterator> +pub fn vortex_fsst::fsst_compress_iter<'a, I>(I, usize, vortex_array::dtype::DType, &fsst::Compressor, &mut vortex_array::executor::ExecutionCtx) -> vortex_fsst::FSSTArray where I: core::iter::traits::iterator::Iterator> -pub fn vortex_fsst::fsst_train_compressor>(array: &A) -> fsst::Compressor +pub fn vortex_fsst::fsst_train_compressor>(&A) -> fsst::Compressor pub type vortex_fsst::FSSTArray = vortex_array::array::typed::Array diff --git a/encodings/fsst/src/array.rs b/encodings/fsst/src/array.rs index 1a5cc6ef02b..36e57c51239 100644 --- a/encodings/fsst/src/array.rs +++ b/encodings/fsst/src/array.rs @@ -23,8 +23,10 @@ use vortex_array::Canonical; use vortex_array::ExecutionCtx; use vortex_array::ExecutionResult; use vortex_array::IntoArray; +use vortex_array::LEGACY_SESSION; use vortex_array::Precision; use vortex_array::TypedArrayRef; +use vortex_array::VortexSessionExecute; use vortex_array::arrays::VarBin; use vortex_array::arrays::VarBinArray; use vortex_array::arrays::varbin::VarBinArrayExt; @@ -49,6 +51,7 @@ use vortex_error::vortex_ensure; use vortex_error::vortex_err; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::canonical::canonicalize_fsst; use crate::canonical::fsst_decode_views; @@ -101,7 +104,8 @@ impl VTable for FSST { type ValidityVTable = Self; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.fsst"); + *ID } fn validate( @@ -111,7 +115,9 @@ impl VTable for FSST { len: usize, slots: &[Option], ) -> VortexResult<()> { - data.validate(dtype, len, slots) + // TODO(ctx): trait fixes - VTable::validate has a fixed signature. + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + data.validate(dtype, len, slots, &mut ctx) } fn nbuffers(_array: ArrayView<'_, Self>) -> usize { @@ -183,12 +189,13 @@ impl VTable for FSST { metadata: &[u8], buffers: &[BufferHandle], children: &dyn ArrayChildren, - _session: &VortexSession, + session: &VortexSession, ) -> VortexResult> { let metadata = FSSTMetadata::decode(metadata)?; let symbols = Buffer::::from_byte_buffer(buffers[0].clone().try_to_host_sync()?); let symbol_lengths = Buffer::::from_byte_buffer(buffers[1].clone().try_to_host_sync()?); + let mut ctx = session.create_execution_ctx(); if buffers.len() == 2 { return Self::deserialize_legacy( self, @@ -198,6 +205,7 @@ impl VTable for FSST { &symbols, &symbol_lengths, children, + &mut ctx, ); } @@ -240,6 +248,7 @@ impl VTable for FSST { &uncompressed_lengths, dtype, len, + &mut ctx, )?; let slots = vec![ Some(uncompressed_lengths), @@ -286,7 +295,14 @@ impl VTable for FSST { let next_buffer_index = builder.completed_block_count() + u32::from(builder.in_progress()); let (buffers, views) = fsst_decode_views(array, next_buffer_index, ctx)?; - builder.push_buffer_and_adjusted_views(&buffers, &views, array.array().validity_mask()?); + builder.push_buffer_and_adjusted_views( + &buffers, + &views, + array + .array() + .validity()? + .execute_mask(array.array().len(), ctx)?, + ); Ok(()) } @@ -311,10 +327,8 @@ impl VTable for FSST { /// Lengths of the original values before compression, can be compressed. pub(crate) const UNCOMPRESSED_LENGTHS_SLOT: usize = 0; /// The offsets array for the FSST-compressed codes. -#[allow(dead_code, reason = "reserved for back-compat slot numbering")] pub(crate) const CODES_OFFSETS_SLOT: usize = 1; /// The validity bitmap for the compressed codes. -#[allow(dead_code, reason = "reserved for back-compat slot numbering")] pub(crate) const CODES_VALIDITY_SLOT: usize = 2; pub(crate) const NUM_SLOTS: usize = 3; pub(crate) const SLOT_NAMES: [&str; NUM_SLOTS] = @@ -365,8 +379,6 @@ impl Debug for FSSTData { pub struct FSST; impl FSST { - pub const ID: ArrayId = ArrayId::new_ref("vortex.fsst"); - /// Build an FSST array from a set of `symbols` and `codes`. /// /// The `codes` VarBinArray is decomposed: its bytes are stored in [`FSSTData`], while @@ -378,6 +390,7 @@ impl FSST { symbol_lengths: Buffer, codes: VarBinArray, uncompressed_lengths: ArrayRef, + ctx: &mut ExecutionCtx, ) -> VortexResult { let len = codes.len(); FSSTData::validate_parts_from_codes( @@ -387,6 +400,7 @@ impl FSST { &uncompressed_lengths, &dtype, len, + ctx, )?; let slots = FSSTData::make_slots(&codes, &uncompressed_lengths); let codes_bytes = codes.bytes_handle().clone(); @@ -399,6 +413,7 @@ impl FSST { /// Legacy deserialization path (2 buffers): the codes were stored as a full /// `VarBinArray` child. We decompose the VarBinArray into its bytes (stored in /// FSSTData) and offsets/validity (stored in slots). + #[allow(clippy::too_many_arguments)] fn deserialize_legacy( &self, dtype: &DType, @@ -407,6 +422,7 @@ impl FSST { symbols: &Buffer, symbol_lengths: &Buffer, children: &dyn ArrayChildren, + ctx: &mut ExecutionCtx, ) -> VortexResult> { if children.len() != 2 { vortex_bail!(InvalidArgument: "Expected 2 children, got {}", children.len()); @@ -437,6 +453,7 @@ impl FSST { &uncompressed_lengths, dtype, len, + ctx, )?; let slots = FSSTData::make_slots(&codes, &uncompressed_lengths); let codes_bytes = codes.bytes_handle().clone(); @@ -509,6 +526,7 @@ impl FSSTData { dtype: &DType, len: usize, slots: &[Option], + ctx: &mut ExecutionCtx, ) -> VortexResult<()> { let codes_offsets = slots[CODES_OFFSETS_SLOT] .as_ref() @@ -522,11 +540,12 @@ impl FSSTData { uncompressed_lengths_from_slots(slots), dtype, len, + ctx, ) } /// Validate using the decomposed components (codes bytes + offsets + nullability). - #[allow(clippy::too_many_arguments)] + #[expect(clippy::too_many_arguments)] fn validate_parts( symbols: &Buffer, symbol_lengths: &Buffer, @@ -536,6 +555,7 @@ impl FSSTData { uncompressed_lengths: &ArrayRef, dtype: &DType, len: usize, + ctx: &mut ExecutionCtx, ) -> VortexResult<()> { vortex_ensure!( matches!(dtype, DType::Binary(_) | DType::Utf8(_)), @@ -575,7 +595,7 @@ impl FSSTData { // Validate that last offset doesn't exceed bytes length (when host-resident). if codes_bytes.is_on_host() && codes_offsets.is_host() && !codes_offsets.is_empty() { let last_offset: usize = (&codes_offsets - .scalar_at(codes_offsets.len() - 1) + .execute_scalar(codes_offsets.len() - 1, ctx) .vortex_expect("offsets must support scalar_at")) .try_into() .vortex_expect("Failed to convert offset to usize"); @@ -598,6 +618,7 @@ impl FSSTData { uncompressed_lengths: &ArrayRef, dtype: &DType, len: usize, + ctx: &mut ExecutionCtx, ) -> VortexResult<()> { Self::validate_parts( symbols, @@ -608,6 +629,7 @@ impl FSSTData { uncompressed_lengths, dtype, len, + ctx, ) } @@ -697,7 +719,7 @@ pub trait FSSTArrayExt: TypedArrayRef { .vortex_expect("FSSTArray codes_offsets slot") .clone(); let validity = child_to_validity( - &self.as_ref().slots()[CODES_VALIDITY_SLOT], + self.as_ref().slots()[CODES_VALIDITY_SLOT].as_ref(), self.as_ref().dtype().nullability(), ); let codes_bytes = self.codes_bytes_handle().clone(); @@ -723,7 +745,7 @@ impl> FSSTArrayExt for T {} impl ValidityVTable for FSST { fn validity(array: ArrayView<'_, FSST>) -> VortexResult { Ok(child_to_validity( - &array.slots()[CODES_VALIDITY_SLOT], + array.slots()[CODES_VALIDITY_SLOT].as_ref(), array.dtype().nullability(), )) } @@ -782,11 +804,13 @@ mod test { let symbol_lengths = Buffer::::copy_from([3, 8]); let compressor = Compressor::rebuild_from(symbols.as_slice(), symbol_lengths.as_slice()); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let fsst_array = fsst_compress_iter( [Some(b"abcabcab".as_ref()), Some(b"defghijk".as_ref())].into_iter(), 2, DType::Utf8(Nullability::NonNullable), &compressor, + &mut ctx, ); let compressed_codes = fsst_array.codes(); diff --git a/encodings/fsst/src/canonical.rs b/encodings/fsst/src/canonical.rs index 14d06de20f4..a8b8171b043 100644 --- a/encodings/fsst/src/canonical.rs +++ b/encodings/fsst/src/canonical.rs @@ -59,7 +59,7 @@ pub(crate) fn fsst_decode_views( .clone() .execute::(ctx)?; - #[allow(clippy::cast_possible_truncation)] + #[expect(clippy::cast_possible_truncation)] let total_size: usize = match_each_integer_ptype!(uncompressed_lens_array.ptype(), |P| { uncompressed_lens_array .as_slice::

() @@ -95,11 +95,11 @@ mod tests { use rand::prelude::StdRng; use vortex_array::ArrayRef; use vortex_array::IntoArray; - use vortex_array::ToCanonical; use vortex_array::VortexSessionExecute; use vortex_array::accessor::ArrayAccessor; use vortex_array::arrays::ChunkedArray; use vortex_array::arrays::VarBinArray; + use vortex_array::arrays::VarBinViewArray; use vortex_array::builders::ArrayBuilder; use vortex_array::builders::VarBinViewBuilder; use vortex_array::dtype::DType; @@ -148,13 +148,15 @@ mod tests { } fn make_data_chunked() -> (ChunkedArray, Vec>>) { - #[allow(clippy::type_complexity)] + let mut ctx = SESSION.create_execution_ctx(); + #[expect(clippy::type_complexity)] let (arr_vec, data_vec): (Vec, Vec>>>) = (0..10) .map(|_| { let (array, data) = make_data(); let compressor = fsst_train_compressor(&array); ( - fsst_compress(&array, array.len(), array.dtype(), &compressor).into_array(), + fsst_compress(&array, array.len(), array.dtype(), &compressor, &mut ctx) + .into_array(), data, ) }) @@ -168,6 +170,7 @@ mod tests { #[test] fn test_to_canonical() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); let (chunked_arr, data) = make_data_chunked(); let mut builder = @@ -175,7 +178,7 @@ mod tests { chunked_arr .clone() .into_array() - .append_to_builder(&mut builder, &mut SESSION.create_execution_ctx())?; + .append_to_builder(&mut builder, &mut ctx)?; { let arr = builder.finish_into_canonical().into_varbinview(); @@ -185,7 +188,10 @@ mod tests { }; { - let arr2 = chunked_arr.as_array().to_varbinview(); + let arr2 = chunked_arr + .as_array() + .clone() + .execute::(&mut ctx)?; let res2 = arr2.with_iterator(|iter| iter.map(|b| b.map(|v| v.to_vec())).collect::>()); assert_eq!(data, res2) @@ -203,14 +209,16 @@ mod tests { [Some(b"long enough too".to_vec().into_boxed_slice())], dtype, ); + let mut ctx = SESSION.create_execution_ctx(); let fsst_array = fsst_compress( &varbin, varbin.len(), varbin.dtype(), &fsst_train_compressor(&varbin), + &mut ctx, ) .into_array(); - fsst_array.append_to_builder(&mut builder, &mut SESSION.create_execution_ctx())?; + fsst_array.append_to_builder(&mut builder, &mut ctx)?; let _result = builder.finish_into_varbinview(); Ok(()) diff --git a/encodings/fsst/src/compress.rs b/encodings/fsst/src/compress.rs index 53772e82636..872dcbc494d 100644 --- a/encodings/fsst/src/compress.rs +++ b/encodings/fsst/src/compress.rs @@ -5,6 +5,7 @@ use fsst::Compressor; use fsst::Symbol; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; use vortex_array::accessor::ArrayAccessor; use vortex_array::arrays::varbin::builder::VarBinBuilder; @@ -21,8 +22,9 @@ pub fn fsst_compress>( len: usize, dtype: &DType, compressor: &Compressor, + ctx: &mut ExecutionCtx, ) -> FSSTArray { - strings.with_iterator(|iter| fsst_compress_iter(iter, len, dtype.clone(), compressor)) + strings.with_iterator(|iter| fsst_compress_iter(iter, len, dtype.clone(), compressor, ctx)) } /// Train a compressor from an array. @@ -61,6 +63,7 @@ pub fn fsst_compress_iter<'a, I>( len: usize, dtype: DType, compressor: &Compressor, + ctx: &mut ExecutionCtx, ) -> FSSTArray where I: Iterator>, @@ -102,13 +105,22 @@ where let uncompressed_lengths = uncompressed_lengths.into_array(); - FSST::try_new(dtype, symbols, symbol_lengths, codes, uncompressed_lengths) - .vortex_expect("FSST parts must be valid") + FSST::try_new( + dtype, + symbols, + symbol_lengths, + codes, + uncompressed_lengths, + ctx, + ) + .vortex_expect("FSST parts must be valid") } #[cfg(test)] mod tests { use fsst::CompressorBuilder; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::dtype::DType; use vortex_array::dtype::Nullability; use vortex_array::scalar::Scalar; @@ -126,14 +138,16 @@ mod tests { let compressor = CompressorBuilder::default().build(); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let compressed = fsst_compress_iter( [Some(big_string.as_bytes())].into_iter(), 1, DType::Utf8(Nullability::NonNullable), &compressor, + &mut ctx, ); - let decoded = compressed.scalar_at(0).unwrap(); + let decoded = compressed.execute_scalar(0, &mut ctx).unwrap(); let expected = Scalar::utf8(big_string, Nullability::NonNullable); diff --git a/encodings/fsst/src/compute/cast.rs b/encodings/fsst/src/compute/cast.rs index 9553cb4d8a9..cd8c5e3a9c3 100644 --- a/encodings/fsst/src/compute/cast.rs +++ b/encodings/fsst/src/compute/cast.rs @@ -3,57 +3,114 @@ use vortex_array::ArrayRef; use vortex_array::ArrayView; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; -use vortex_array::arrays::VarBin; -use vortex_array::builtins::ArrayBuiltins; +use vortex_array::arrays::VarBinArray; +use vortex_array::arrays::varbin::VarBinArrayExt; use vortex_array::dtype::DType; +use vortex_array::scalar_fn::fns::cast::CastKernel; use vortex_array::scalar_fn::fns::cast::CastReduce; +use vortex_array::validity::Validity; use vortex_error::VortexResult; use crate::FSST; use crate::FSSTArrayExt; + +fn build_with_codes_validity( + array: ArrayView<'_, FSST>, + dtype: &DType, + new_codes_validity: Validity, +) -> VortexResult { + let codes = array.codes(); + let new_codes = VarBinArray::try_new( + codes.offsets().clone(), + codes.bytes().clone(), + codes.dtype().with_nullability(dtype.nullability()), + new_codes_validity, + )?; + + Ok(unsafe { + FSST::new_unchecked( + dtype.clone(), + array.symbols().clone(), + array.symbol_lengths().clone(), + new_codes, + array.uncompressed_lengths().clone(), + ) + } + .into_array()) +} + impl CastReduce for FSST { fn cast(array: ArrayView<'_, Self>, dtype: &DType) -> VortexResult> { - // FSST is a string compression encoding. - // For nullability changes, we can cast the codes and symbols arrays - if array.dtype().eq_ignore_nullability(dtype) { - // Cast codes array to handle nullability - let new_codes = array - .codes() - .into_array() - .cast(array.codes_dtype().with_nullability(dtype.nullability()))?; - - Ok(Some( - FSST::try_new( - dtype.clone(), - array.symbols().clone(), - array.symbol_lengths().clone(), - new_codes.as_::().into_owned(), - array.uncompressed_lengths().clone(), - )? - .into_array(), - )) - } else { - Ok(None) + if !array.dtype().eq_ignore_nullability(dtype) { + return Ok(None); } + + let codes = array.codes(); + let Some(new_codes_validity) = codes + .validity()? + .trivial_cast_nullability(dtype.nullability(), codes.len())? + else { + return Ok(None); + }; + + Ok(Some(build_with_codes_validity( + array, + dtype, + new_codes_validity, + )?)) + } +} + +impl CastKernel for FSST { + fn cast( + array: ArrayView<'_, Self>, + dtype: &DType, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + if !array.dtype().eq_ignore_nullability(dtype) { + return Ok(None); + } + + let codes = array.codes(); + let new_codes_validity = + codes + .validity()? + .cast_nullability(dtype.nullability(), codes.len(), ctx)?; + + Ok(Some(build_with_codes_validity( + array, + dtype, + new_codes_validity, + )?)) } } #[cfg(test)] mod tests { + use std::sync::LazyLock; + use rstest::rstest; use vortex_array::IntoArray; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::VarBinArray; use vortex_array::builtins::ArrayBuiltins; use vortex_array::compute::conformance::cast::test_cast_conformance; use vortex_array::dtype::DType; use vortex_array::dtype::Nullability; + use vortex_array::session::ArraySession; + use vortex_session::VortexSession; use crate::fsst_compress; use crate::fsst_train_compressor; + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); + #[test] fn test_cast_fsst_nullability() { + let mut ctx = SESSION.create_execution_ctx(); let strings = VarBinArray::from_iter( vec![Some("hello"), Some("world"), Some("hello world")], DType::Utf8(Nullability::NonNullable), @@ -62,7 +119,7 @@ mod tests { let compressor = fsst_train_compressor(&strings); let len = strings.len(); let dtype = strings.dtype().clone(); - let fsst = fsst_compress(strings, len, &dtype, &compressor); + let fsst = fsst_compress(strings, len, &dtype, &compressor, &mut ctx); // Cast to nullable let casted = fsst @@ -86,8 +143,9 @@ mod tests { DType::Utf8(Nullability::NonNullable) ))] fn test_cast_fsst_conformance(#[case] array: VarBinArray) { + let mut ctx = SESSION.create_execution_ctx(); let compressor = fsst_train_compressor(&array); - let fsst = fsst_compress(&array, array.len(), array.dtype(), &compressor); + let fsst = fsst_compress(&array, array.len(), array.dtype(), &compressor, &mut ctx); test_cast_conformance(&fsst.into_array()); } } diff --git a/encodings/fsst/src/compute/compare.rs b/encodings/fsst/src/compute/compare.rs index 2ae2cf0ff40..553a3608e75 100644 --- a/encodings/fsst/src/compute/compare.rs +++ b/encodings/fsst/src/compute/compare.rs @@ -13,7 +13,6 @@ use vortex_array::scalar::Scalar; use vortex_array::scalar_fn::fns::binary::CompareKernel; use vortex_array::scalar_fn::fns::operators::CompareOperator; use vortex_array::scalar_fn::fns::operators::Operator; -use vortex_array::validity::Validity; use vortex_buffer::BitBuffer; use vortex_buffer::ByteBuffer; use vortex_error::VortexExpect; @@ -77,7 +76,8 @@ fn compare_fsst_constant( return Ok(Some( BoolArray::new( buffer, - Validity::copy_from_array(left.array())? + left.array() + .validity()? .union_nullability(right.dtype().nullability()), ) .into_array(), @@ -123,7 +123,8 @@ fn compare_fsst_constant( #[cfg(test)] mod tests { use vortex_array::IntoArray; - use vortex_array::ToCanonical; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; use vortex_array::arrays::ConstantArray; use vortex_array::arrays::VarBinArray; @@ -140,6 +141,7 @@ mod tests { #[test] #[cfg_attr(miri, ignore)] fn test_compare_fsst() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let lhs = VarBinArray::from_iter( [ Some("hello"), @@ -153,7 +155,7 @@ mod tests { let compressor = fsst_train_compressor(&lhs); let len = lhs.len(); let dtype = lhs.dtype().clone(); - let lhs = fsst_compress(lhs, len, &dtype, &compressor); + let lhs = fsst_compress(lhs, len, &dtype, &compressor, &mut ctx); let rhs = ConstantArray::new("world", lhs.len()); @@ -163,7 +165,8 @@ mod tests { .into_array() .binary(rhs.clone().into_array(), Operator::Eq) .unwrap() - .to_bool(); + .execute::(&mut ctx) + .unwrap(); assert_eq!(equals.dtype(), &DType::Bool(Nullability::Nullable)); @@ -178,7 +181,8 @@ mod tests { .into_array() .binary(rhs.into_array(), Operator::NotEq) .unwrap() - .to_bool(); + .execute::(&mut ctx) + .unwrap(); assert_eq!(not_equals.dtype(), &DType::Bool(Nullability::Nullable)); assert_arrays_eq!( diff --git a/encodings/fsst/src/compute/filter.rs b/encodings/fsst/src/compute/filter.rs index 9f60cb6c596..74e32cfbd02 100644 --- a/encodings/fsst/src/compute/filter.rs +++ b/encodings/fsst/src/compute/filter.rs @@ -37,6 +37,7 @@ impl FilterKernel for FSST { array.symbol_lengths().clone(), filtered_codes, array.uncompressed_lengths().filter(mask.clone())?, + ctx, )? .into_array(), )) diff --git a/encodings/fsst/src/compute/like.rs b/encodings/fsst/src/compute/like.rs index 2e618ef6588..a6b8c40d4c2 100644 --- a/encodings/fsst/src/compute/like.rs +++ b/encodings/fsst/src/compute/like.rs @@ -1,14 +1,12 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::cast_possible_truncation)] - use vortex_array::ArrayRef; use vortex_array::ArrayView; use vortex_array::ExecutionCtx; use vortex_array::IntoArray; -use vortex_array::ToCanonical; use vortex_array::arrays::BoolArray; +use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::varbin::VarBinArrayExt; use vortex_array::match_each_integer_ptype; use vortex_array::scalar_fn::fns::like::LikeKernel; @@ -25,7 +23,7 @@ impl LikeKernel for FSST { array: ArrayView<'_, Self>, pattern: &ArrayRef, options: LikeOptions, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult> { let Some(pattern_scalar) = pattern.as_constant() else { return Ok(None); @@ -60,7 +58,7 @@ impl LikeKernel for FSST { let negated = options.negated; let codes = array.codes(); - let offsets = codes.offsets().to_primitive(); + let offsets = codes.offsets().clone().execute::(ctx)?; let all_bytes = codes.bytes(); let all_bytes = all_bytes.as_slice(); let n = codes.len(); @@ -115,7 +113,13 @@ mod tests { let compressor = fsst_train_compressor(&varbin); let len = varbin.len(); let dtype = varbin.dtype().clone(); - fsst_compress(varbin, len, &dtype, &compressor) + fsst_compress( + varbin, + len, + &dtype, + &compressor, + &mut SESSION.create_execution_ctx(), + ) } fn run_like(array: FSSTArray, pattern: &str, opts: LikeOptions) -> VortexResult { diff --git a/encodings/fsst/src/compute/mod.rs b/encodings/fsst/src/compute/mod.rs index ac350e6186c..02efdf7febc 100644 --- a/encodings/fsst/src/compute/mod.rs +++ b/encodings/fsst/src/compute/mod.rs @@ -25,7 +25,7 @@ impl TakeExecute for FSST { fn take( array: ArrayView<'_, Self>, indices: &ArrayRef, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult> { Ok(Some( FSST::try_new( @@ -38,7 +38,7 @@ impl TakeExecute for FSST { { let codes = array.codes(); let codes = codes.as_view(); - ::take(codes, indices, _ctx)? + ::take(codes, indices, ctx)? .vortex_expect("VarBin take kernel always returns Some") } .try_downcast::() @@ -49,6 +49,7 @@ impl TakeExecute for FSST { .fill_null(Scalar::zero_value( &array.uncompressed_lengths_dtype().clone(), ))?, + ctx, )? .into_array(), )) @@ -58,7 +59,10 @@ impl TakeExecute for FSST { #[cfg(test)] mod tests { use rstest::rstest; + use vortex_array::ExecutionCtx; use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::VarBinArray; use vortex_array::compute::conformance::consistency::test_array_consistency; @@ -72,9 +76,10 @@ mod tests { #[test] fn test_take_null() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let arr = VarBinArray::from_iter([Some("h")], DType::Utf8(Nullability::NonNullable)); let compr = fsst_train_compressor(&arr); - let fsst = fsst_compress(&arr, arr.len(), arr.dtype(), &compr); + let fsst = fsst_compress(&arr, arr.len(), arr.dtype(), &compr, &mut ctx); let idx1: PrimitiveArray = (0..1).collect(); @@ -105,23 +110,26 @@ mod tests { DType::Utf8(Nullability::NonNullable), ))] fn test_take_fsst_conformance(#[case] varbin: VarBinArray) { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let compressor = fsst_train_compressor(&varbin); - let array = fsst_compress(&varbin, varbin.len(), varbin.dtype(), &compressor); + let array = fsst_compress(&varbin, varbin.len(), varbin.dtype(), &compressor, &mut ctx); test_take_conformance(&array.into_array()); } + type FsstBuilder = fn(&mut ExecutionCtx) -> FSSTArray; + #[rstest] // Basic string arrays - #[case::fsst_simple({ + #[case::fsst_simple(|ctx: &mut ExecutionCtx| { let varbin = VarBinArray::from_iter( ["hello world", "testing fsst", "compression test", "data array", "vortex encoding"].map(Some), DType::Utf8(Nullability::NonNullable), ); let compressor = fsst_train_compressor(&varbin); - fsst_compress(&varbin, varbin.len(), varbin.dtype(), &compressor) + fsst_compress(&varbin, varbin.len(), varbin.dtype(), &compressor, ctx) })] // Nullable strings - #[case::fsst_nullable({ + #[case::fsst_nullable(|ctx: &mut ExecutionCtx| { let varbin = VarBinArray::from_iter( [Some("hello"), None, Some("world"), Some("test"), None], DType::Utf8(Nullability::Nullable), @@ -129,27 +137,27 @@ mod tests { let compressor = fsst_train_compressor(&varbin); let len = varbin.len(); let dtype = varbin.dtype().clone(); - fsst_compress(varbin, len, &dtype, &compressor) + fsst_compress(varbin, len, &dtype, &compressor, ctx) })] // Repetitive patterns (good for FSST compression) - #[case::fsst_repetitive({ + #[case::fsst_repetitive(|ctx: &mut ExecutionCtx| { let varbin = VarBinArray::from_iter( ["http://example.com", "http://test.com", "http://vortex.dev", "http://data.org"].map(Some), DType::Utf8(Nullability::NonNullable), ); let compressor = fsst_train_compressor(&varbin); - fsst_compress(&varbin, varbin.len(), varbin.dtype(), &compressor) + fsst_compress(&varbin, varbin.len(), varbin.dtype(), &compressor, ctx) })] // Edge cases - #[case::fsst_single({ + #[case::fsst_single(|ctx: &mut ExecutionCtx| { let varbin = VarBinArray::from_iter( ["single element"].map(Some), DType::Utf8(Nullability::NonNullable), ); let compressor = fsst_train_compressor(&varbin); - fsst_compress(&varbin, varbin.len(), varbin.dtype(), &compressor) + fsst_compress(&varbin, varbin.len(), varbin.dtype(), &compressor, ctx) })] - #[case::fsst_empty_strings({ + #[case::fsst_empty_strings(|ctx: &mut ExecutionCtx| { let varbin = VarBinArray::from_iter( ["", "test", "", "hello", ""].map(Some), DType::Utf8(Nullability::NonNullable), @@ -157,10 +165,10 @@ mod tests { let compressor = fsst_train_compressor(&varbin); let len = varbin.len(); let dtype = varbin.dtype().clone(); - fsst_compress(varbin, len, &dtype, &compressor) + fsst_compress(varbin, len, &dtype, &compressor, ctx) })] // Large arrays - #[case::fsst_large({ + #[case::fsst_large(|ctx: &mut ExecutionCtx| { let data: Vec> = (0..1500) .map(|i| Some(match i % 10 { 0 => "https://www.example.com/page", @@ -179,10 +187,12 @@ mod tests { let compressor = fsst_train_compressor(&varbin); let len = varbin.len(); let dtype = varbin.dtype().clone(); - fsst_compress(varbin, len, &dtype, &compressor) + fsst_compress(varbin, len, &dtype, &compressor, ctx) })] - fn test_fsst_consistency(#[case] array: FSSTArray) { + fn test_fsst_consistency(#[case] build: FsstBuilder) { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let array = build(&mut ctx); test_array_consistency(&array.into_array()); } } diff --git a/encodings/fsst/src/dfa/tests.rs b/encodings/fsst/src/dfa/tests.rs index caf04adb3c6..6ad30ca685d 100644 --- a/encodings/fsst/src/dfa/tests.rs +++ b/encodings/fsst/src/dfa/tests.rs @@ -229,7 +229,13 @@ fn make_fsst_str(strings: &[Option<&str>]) -> FSSTArray { let compressor = fsst_train_compressor(&varbin); let len = varbin.len(); let dtype = varbin.dtype().clone(); - fsst_compress(varbin, len, &dtype, &compressor) + fsst_compress( + varbin, + len, + &dtype, + &compressor, + &mut SESSION.create_execution_ctx(), + ) } fn run_like(array: FSSTArray, pattern_arr: ArrayRef) -> VortexResult { diff --git a/encodings/fsst/src/kernel.rs b/encodings/fsst/src/kernel.rs index 05b58e3215d..7f455a06e16 100644 --- a/encodings/fsst/src/kernel.rs +++ b/encodings/fsst/src/kernel.rs @@ -5,11 +5,13 @@ use vortex_array::arrays::dict::TakeExecuteAdaptor; use vortex_array::arrays::filter::FilterExecuteAdaptor; use vortex_array::kernel::ParentKernelSet; use vortex_array::scalar_fn::fns::binary::CompareExecuteAdaptor; +use vortex_array::scalar_fn::fns::cast::CastExecuteAdaptor; use vortex_array::scalar_fn::fns::like::LikeExecuteAdaptor; use crate::FSST; pub(super) const PARENT_KERNELS: ParentKernelSet = ParentKernelSet::new(&[ + ParentKernelSet::lift(&CastExecuteAdaptor(FSST)), ParentKernelSet::lift(&CompareExecuteAdaptor(FSST)), ParentKernelSet::lift(&FilterExecuteAdaptor(FSST)), ParentKernelSet::lift(&TakeExecuteAdaptor(FSST)), @@ -58,7 +60,8 @@ mod tests { let compressor = fsst_train_compressor(&input); let len = input.len(); let dtype = input.dtype().clone(); - fsst_compress(input, len, &dtype, &compressor).into_array() + let mut ctx = SESSION.create_execution_ctx(); + fsst_compress(input, len, &dtype, &compressor, &mut ctx).into_array() } #[test] @@ -131,8 +134,15 @@ mod tests { let input = builder.finish(DType::Utf8(Nullability::Nullable)); let compressor = fsst_train_compressor(&input); - let fsst_array: ArrayRef = - fsst_compress(input.clone(), input.len(), input.dtype(), &compressor).into_array(); + let mut ctx = SESSION.create_execution_ctx(); + let fsst_array: ArrayRef = fsst_compress( + input.clone(), + input.len(), + input.dtype(), + &compressor, + &mut ctx, + ) + .into_array(); // Filter: only select the last element (index 22) let mut mask = vec![false; 22]; @@ -140,7 +150,6 @@ mod tests { let mask = Mask::from_iter(mask); let filter_array = FilterArray::new(fsst_array, mask.clone()).into_array(); - let mut ctx = SESSION.create_execution_ctx(); let result = filter_array.execute::(&mut ctx)?; let expected = input.filter(mask)?; @@ -160,13 +169,19 @@ mod tests { let input = builder.finish(DType::Utf8(Nullability::Nullable)); let compressor = fsst_train_compressor(&input); - let fsst_array: ArrayRef = - fsst_compress(input.clone(), input.len(), input.dtype(), &compressor).into_array(); + let mut ctx = SESSION.create_execution_ctx(); + let fsst_array: ArrayRef = fsst_compress( + input.clone(), + input.len(), + input.dtype(), + &compressor, + &mut ctx, + ) + .into_array(); let mask = Mask::from_iter([true, false, true]); let filter_array = FilterArray::new(fsst_array, mask.clone()).into_array(); - let mut ctx = SESSION.create_execution_ctx(); let result = filter_array.execute::(&mut ctx)?; let expected = input.filter(mask)?; diff --git a/encodings/fsst/src/ops.rs b/encodings/fsst/src/ops.rs index 86172377f7e..b630508ed9e 100644 --- a/encodings/fsst/src/ops.rs +++ b/encodings/fsst/src/ops.rs @@ -17,9 +17,9 @@ impl OperationsVTable for FSST { fn scalar_at( array: ArrayView<'_, FSST>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { - let compressed = array.codes().scalar_at(index)?; + let compressed = array.codes().execute_scalar(index, ctx)?; let binary_datum = compressed.as_binary().value().vortex_expect("non-null"); let decoded_buffer = ByteBuffer::from(array.decompressor().decompress(binary_datum)); diff --git a/encodings/fsst/src/test_utils.rs b/encodings/fsst/src/test_utils.rs index 421a5d4c3f5..eeca412e377 100644 --- a/encodings/fsst/src/test_utils.rs +++ b/encodings/fsst/src/test_utils.rs @@ -1,12 +1,13 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] +#![expect(clippy::unwrap_used)] use rand::RngExt; use rand::SeedableRng; use rand::prelude::StdRng; use vortex_array::ArrayRef; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; use vortex_array::arrays::DictArray; use vortex_array::arrays::PrimitiveArray; @@ -20,7 +21,12 @@ use crate::FSSTArray; use crate::fsst_compress; use crate::fsst_train_compressor; -pub fn gen_fsst_test_data(len: usize, avg_str_len: usize, unique_chars: u8) -> ArrayRef { +pub fn gen_fsst_test_data( + len: usize, + avg_str_len: usize, + unique_chars: u8, + ctx: &mut ExecutionCtx, +) -> ArrayRef { let mut rng = StdRng::seed_from_u64(0); let mut strings = Vec::with_capacity(len); @@ -45,7 +51,7 @@ pub fn gen_fsst_test_data(len: usize, avg_str_len: usize, unique_chars: u8) -> A let len = varbin.len(); let dtype = varbin.dtype().clone(); - fsst_compress(varbin, len, &dtype, &compressor).into_array() + fsst_compress(varbin, len, &dtype, &compressor, ctx).into_array() } pub fn gen_dict_fsst_test_data( @@ -53,8 +59,9 @@ pub fn gen_dict_fsst_test_data( unique_values: usize, str_len: usize, unique_char_count: u8, + ctx: &mut ExecutionCtx, ) -> DictArray { - let values = gen_fsst_test_data(len, str_len, unique_char_count); + let values = gen_fsst_test_data(len, str_len, unique_char_count, ctx); let mut rng = StdRng::seed_from_u64(0); let codes = (0..len) .map(|_| T::from(rng.random_range(0..unique_values)).unwrap()) @@ -136,12 +143,12 @@ pub fn generate_url_data_n(n: usize) -> VarBinArray { VarBinArray::from_iter(urls, DType::Utf8(Nullability::NonNullable)) } -pub fn make_fsst_urls(n: usize) -> FSSTArray { +pub fn make_fsst_urls(n: usize, ctx: &mut ExecutionCtx) -> FSSTArray { let varbin = generate_url_data_n(n); let compressor = fsst_train_compressor(&varbin); let len = varbin.len(); let dtype = varbin.dtype().clone(); - fsst_compress(varbin, len, &dtype, &compressor) + fsst_compress(varbin, len, &dtype, &compressor, ctx) } // --------------------------------------------------------------------------- @@ -228,7 +235,7 @@ pub fn generate_clickbench_urls(n: usize) -> Vec { .collect() } -pub fn make_fsst_clickbench_urls(n: usize) -> FSSTArray { +pub fn make_fsst_clickbench_urls(n: usize, ctx: &mut ExecutionCtx) -> FSSTArray { let urls = generate_clickbench_urls(n); let varbin = VarBinArray::from_iter( urls.iter().map(|s| Some(s.as_str())), @@ -237,7 +244,7 @@ pub fn make_fsst_clickbench_urls(n: usize) -> FSSTArray { let compressor = fsst_train_compressor(&varbin); let len = varbin.len(); let dtype = varbin.dtype().clone(); - fsst_compress(varbin, len, &dtype, &compressor) + fsst_compress(varbin, len, &dtype, &compressor, ctx) } // --------------------------------------------------------------------------- @@ -296,7 +303,7 @@ pub fn generate_short_urls(n: usize) -> Vec { .collect() } -pub fn make_fsst_short_urls(n: usize) -> FSSTArray { +pub fn make_fsst_short_urls(n: usize, ctx: &mut ExecutionCtx) -> FSSTArray { let urls = generate_short_urls(n); let varbin = VarBinArray::from_iter( urls.iter().map(|s| Some(s.as_str())), @@ -305,7 +312,7 @@ pub fn make_fsst_short_urls(n: usize) -> FSSTArray { let compressor = fsst_train_compressor(&varbin); let len = varbin.len(); let dtype = varbin.dtype().clone(); - fsst_compress(varbin, len, &dtype, &compressor) + fsst_compress(varbin, len, &dtype, &compressor, ctx) } // --------------------------------------------------------------------------- @@ -368,7 +375,7 @@ pub fn generate_log_lines(n: usize) -> Vec { .collect() } -pub fn make_fsst_log_lines(n: usize) -> FSSTArray { +pub fn make_fsst_log_lines(n: usize, ctx: &mut ExecutionCtx) -> FSSTArray { let lines = generate_log_lines(n); let varbin = VarBinArray::from_iter( lines.iter().map(|s| Some(s.as_str())), @@ -377,7 +384,7 @@ pub fn make_fsst_log_lines(n: usize) -> FSSTArray { let compressor = fsst_train_compressor(&varbin); let len = varbin.len(); let dtype = varbin.dtype().clone(); - fsst_compress(varbin, len, &dtype, &compressor) + fsst_compress(varbin, len, &dtype, &compressor, ctx) } // --------------------------------------------------------------------------- @@ -427,7 +434,7 @@ pub fn generate_json_strings(n: usize) -> Vec { .collect() } -pub fn make_fsst_json_strings(n: usize) -> FSSTArray { +pub fn make_fsst_json_strings(n: usize, ctx: &mut ExecutionCtx) -> FSSTArray { let jsons = generate_json_strings(n); let varbin = VarBinArray::from_iter( jsons.iter().map(|s| Some(s.as_str())), @@ -436,7 +443,7 @@ pub fn make_fsst_json_strings(n: usize) -> FSSTArray { let compressor = fsst_train_compressor(&varbin); let len = varbin.len(); let dtype = varbin.dtype().clone(); - fsst_compress(varbin, len, &dtype, &compressor) + fsst_compress(varbin, len, &dtype, &compressor, ctx) } // --------------------------------------------------------------------------- @@ -499,7 +506,7 @@ pub fn generate_file_paths(n: usize) -> Vec { .collect() } -pub fn make_fsst_file_paths(n: usize) -> FSSTArray { +pub fn make_fsst_file_paths(n: usize, ctx: &mut ExecutionCtx) -> FSSTArray { let paths = generate_file_paths(n); let varbin = VarBinArray::from_iter( paths.iter().map(|s| Some(s.as_str())), @@ -508,7 +515,7 @@ pub fn make_fsst_file_paths(n: usize) -> FSSTArray { let compressor = fsst_train_compressor(&varbin); let len = varbin.len(); let dtype = varbin.dtype().clone(); - fsst_compress(varbin, len, &dtype, &compressor) + fsst_compress(varbin, len, &dtype, &compressor, ctx) } // --------------------------------------------------------------------------- @@ -552,7 +559,7 @@ pub fn generate_emails(n: usize) -> Vec { .collect() } -pub fn make_fsst_emails(n: usize) -> FSSTArray { +pub fn make_fsst_emails(n: usize, ctx: &mut ExecutionCtx) -> FSSTArray { let emails = generate_emails(n); let varbin = VarBinArray::from_iter( emails.iter().map(|s| Some(s.as_str())), @@ -561,7 +568,7 @@ pub fn make_fsst_emails(n: usize) -> FSSTArray { let compressor = fsst_train_compressor(&varbin); let len = varbin.len(); let dtype = varbin.dtype().clone(); - fsst_compress(varbin, len, &dtype, &compressor) + fsst_compress(varbin, len, &dtype, &compressor, ctx) } // --------------------------------------------------------------------------- @@ -591,7 +598,7 @@ pub fn generate_rare_match_strings(n: usize, match_rate: f64) -> Vec { .collect() } -pub fn make_fsst_rare_match(n: usize) -> FSSTArray { +pub fn make_fsst_rare_match(n: usize, ctx: &mut ExecutionCtx) -> FSSTArray { let strings = generate_rare_match_strings(n, 0.00001); let varbin = VarBinArray::from_iter( strings.iter().map(|s| Some(s.as_str())), @@ -600,5 +607,5 @@ pub fn make_fsst_rare_match(n: usize) -> FSSTArray { let compressor = fsst_train_compressor(&varbin); let len = varbin.len(); let dtype = varbin.dtype().clone(); - fsst_compress(varbin, len, &dtype, &compressor) + fsst_compress(varbin, len, &dtype, &compressor, ctx) } diff --git a/encodings/fsst/src/tests.rs b/encodings/fsst/src/tests.rs index fb45ed31629..fb4cb8bdc5c 100644 --- a/encodings/fsst/src/tests.rs +++ b/encodings/fsst/src/tests.rs @@ -3,7 +3,9 @@ use vortex_array::ArrayRef; use vortex_array::IntoArray; -use vortex_array::ToCanonical; +use vortex_array::LEGACY_SESSION; +use vortex_array::VortexSessionExecute; +use vortex_array::arrays::VarBinViewArray; use vortex_array::arrays::varbin::builder::VarBinBuilder; use vortex_array::assert_arrays_eq; use vortex_array::assert_nth_scalar; @@ -29,11 +31,13 @@ pub(crate) fn build_fsst_array() -> ArrayRef { let compressor = fsst_train_compressor(&input_array); let len = input_array.len(); let dtype = input_array.dtype().clone(); - fsst_compress(input_array, len, &dtype, &compressor).into_array() + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + fsst_compress(input_array, len, &dtype, &compressor, &mut ctx).into_array() } #[test] fn test_fsst_array_ops() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // first test the scalar_at values let fsst_array = build_fsst_array(); assert_nth_scalar!( @@ -95,7 +99,11 @@ fn test_fsst_array_ops() { ); // test to_canonical - let canonical_array = fsst_array.to_varbinview().into_array(); + let canonical_array = fsst_array + .clone() + .execute::(&mut ctx) + .unwrap() + .into_array(); assert_arrays_eq!(fsst_array, canonical_array); } diff --git a/encodings/parquet-variant/Cargo.toml b/encodings/parquet-variant/Cargo.toml index cc554c85d10..482659a616d 100644 --- a/encodings/parquet-variant/Cargo.toml +++ b/encodings/parquet-variant/Cargo.toml @@ -36,5 +36,3 @@ vortex-session = { workspace = true } rstest = { workspace = true } vortex-array = { workspace = true, features = ["_test-harness"] } -[package.metadata.cargo-machete] -ignored = ["getrandom_v03"] diff --git a/encodings/parquet-variant/src/array.rs b/encodings/parquet-variant/src/array.rs index ba13eb8e808..15c58b894c0 100644 --- a/encodings/parquet-variant/src/array.rs +++ b/encodings/parquet-variant/src/array.rs @@ -221,7 +221,7 @@ pub trait ParquetVariantArrayExt: TypedArrayRef { fn validity(&self) -> Validity { child_to_validity( - &self.as_ref().slots()[VALIDITY_SLOT], + self.as_ref().slots()[VALIDITY_SLOT].as_ref(), self.as_ref().dtype().nullability(), ) } @@ -304,17 +304,22 @@ mod tests { use vortex_array::validity::Validity; use vortex_buffer::buffer; use vortex_error::VortexResult; + use vortex_error::vortex_err; use crate::ParquetVariant; use crate::ParquetVariantData; use crate::array::ParquetVariantArrayExt; fn assert_arrow_variant_storage_roundtrip(struct_array: StructArray) -> VortexResult<()> { - let arrow_variant = ArrowVariantArray::try_new(&struct_array).unwrap(); + let arrow_variant = ArrowVariantArray::try_new(&struct_array)?; let vortex_arr = ParquetVariantData::from_arrow_variant(&arrow_variant)?; - let variant_view = vortex_arr.as_opt::().unwrap(); + let variant_view = vortex_arr + .as_opt::() + .ok_or_else(|| vortex_err!("expected variant array"))?; let child = variant_view.child(); - let inner = child.as_opt::().unwrap(); + let inner = child + .as_opt::() + .ok_or_else(|| vortex_err!("expected parquet variant child"))?; let mut ctx = LEGACY_SESSION.create_execution_ctx(); let roundtripped = inner.to_arrow(&mut ctx)?; @@ -390,10 +395,9 @@ mod tests { ] .into(); let struct_array = - StructArray::try_new(struct_fields, vec![Arc::new(metadata), typed_value], None) - .unwrap(); + StructArray::try_new(struct_fields, vec![Arc::new(metadata), typed_value], None)?; - let arrow_variant = ArrowVariantArray::try_new(&struct_array).unwrap(); + let arrow_variant = ArrowVariantArray::try_new(&struct_array)?; let vortex_arr = ParquetVariantData::from_arrow_variant(&arrow_variant)?; assert_eq!(vortex_arr.len(), 3); @@ -402,8 +406,13 @@ mod tests { &DType::Variant(Nullability::NonNullable) ); - let variant_arr = vortex_arr.as_opt::().unwrap(); - let inner = variant_arr.child().as_opt::().unwrap(); + let variant_arr = vortex_arr + .as_opt::() + .ok_or_else(|| vortex_err!("expected variant array"))?; + let inner = variant_arr + .child() + .as_opt::() + .ok_or_else(|| vortex_err!("expected parquet variant child"))?; assert!(inner.typed_value_array().is_some()); Ok(()) @@ -473,8 +482,7 @@ mod tests { .into(), vec![metadata, typed_value], None, - ) - .unwrap(); + )?; assert_arrow_variant_storage_roundtrip(struct_array) } @@ -494,8 +502,7 @@ mod tests { .into(), vec![metadata, value, typed_value], None, - ) - .unwrap(); + )?; assert_arrow_variant_storage_roundtrip(struct_array) } @@ -512,8 +519,7 @@ mod tests { .into(), vec![metadata, value], Some(NullBuffer::from(vec![true, false, true])), - ) - .unwrap(); + )?; assert_arrow_variant_storage_roundtrip(struct_array) } diff --git a/encodings/parquet-variant/src/kernel.rs b/encodings/parquet-variant/src/kernel.rs index d47d4e31267..48ef4ec1d64 100644 --- a/encodings/parquet-variant/src/kernel.rs +++ b/encodings/parquet-variant/src/kernel.rs @@ -106,6 +106,8 @@ mod tests { use parquet_variant_compute::VariantArrayBuilder; use vortex_array::ArrayRef; use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_error::VortexResult; use vortex_mask::Mask; @@ -133,9 +135,8 @@ mod tests { inner.fields().clone(), inner.columns().to_vec(), Some(NullBuffer::from(vec![true, false, true, false])), - ) - .unwrap(); - let arrow_variant = ArrowVariantArray::try_new(&null_struct).unwrap(); + )?; + let arrow_variant = ArrowVariantArray::try_new(&null_struct)?; ParquetVariantData::from_arrow_variant(&arrow_variant) } @@ -145,8 +146,14 @@ mod tests { let sliced = arr.slice(1..3)?; assert_eq!(sliced.len(), 2); - assert_eq!(sliced.scalar_at(0)?, arr.scalar_at(1)?); - assert_eq!(sliced.scalar_at(1)?, arr.scalar_at(2)?); + assert_eq!( + sliced.execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx())?, + arr.execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx())? + ); + assert_eq!( + sliced.execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx())?, + arr.execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx())? + ); Ok(()) } @@ -157,9 +164,21 @@ mod tests { let sliced = arr.slice(0..3)?; assert_eq!(sliced.len(), 3); - assert!(!sliced.scalar_at(0)?.is_null()); - assert!(sliced.scalar_at(1)?.is_null()); - assert!(!sliced.scalar_at(2)?.is_null()); + assert!( + !sliced + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx())? + .is_null() + ); + assert!( + sliced + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx())? + .is_null() + ); + assert!( + !sliced + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx())? + .is_null() + ); Ok(()) } @@ -171,8 +190,14 @@ mod tests { let filtered = arr.filter(mask)?; assert_eq!(filtered.len(), 2); - assert_eq!(filtered.scalar_at(0)?, arr.scalar_at(0)?); - assert_eq!(filtered.scalar_at(1)?, arr.scalar_at(2)?); + assert_eq!( + filtered.execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx())?, + arr.execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx())? + ); + assert_eq!( + filtered.execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx())?, + arr.execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx())? + ); Ok(()) } @@ -185,9 +210,21 @@ mod tests { let filtered = arr.filter(mask)?; assert_eq!(filtered.len(), 3); - assert!(!filtered.scalar_at(0)?.is_null()); - assert!(filtered.scalar_at(1)?.is_null()); - assert!(filtered.scalar_at(2)?.is_null()); + assert!( + !filtered + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx())? + .is_null() + ); + assert!( + filtered + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx())? + .is_null() + ); + assert!( + filtered + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx())? + .is_null() + ); Ok(()) } @@ -199,9 +236,18 @@ mod tests { let taken = arr.take(indices.into_array())?; assert_eq!(taken.len(), 3); - assert_eq!(taken.scalar_at(0)?, arr.scalar_at(2)?); - assert_eq!(taken.scalar_at(1)?, arr.scalar_at(0)?); - assert_eq!(taken.scalar_at(2)?, arr.scalar_at(3)?); + assert_eq!( + taken.execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx())?, + arr.execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx())? + ); + assert_eq!( + taken.execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx())?, + arr.execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx())? + ); + assert_eq!( + taken.execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx())?, + arr.execute_scalar(3, &mut LEGACY_SESSION.create_execution_ctx())? + ); Ok(()) } @@ -214,10 +260,26 @@ mod tests { let taken = arr.take(indices.into_array())?; assert_eq!(taken.len(), 4); - assert!(!taken.scalar_at(0)?.is_null()); - assert!(taken.scalar_at(1)?.is_null()); - assert!(taken.scalar_at(2)?.is_null()); - assert!(!taken.scalar_at(3)?.is_null()); + assert!( + !taken + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx())? + .is_null() + ); + assert!( + taken + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx())? + .is_null() + ); + assert!( + taken + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx())? + .is_null() + ); + assert!( + !taken + .execute_scalar(3, &mut LEGACY_SESSION.create_execution_ctx())? + .is_null() + ); Ok(()) } @@ -243,15 +305,20 @@ mod tests { .into(), vec![metadata, typed_value], None, - ) - .unwrap(); - let arrow_variant = ArrowVariantArray::try_new(&struct_array).unwrap(); + )?; + let arrow_variant = ArrowVariantArray::try_new(&struct_array)?; let arr = ParquetVariantData::from_arrow_variant(&arrow_variant)?; let sliced = arr.slice(1..3)?; assert_eq!(sliced.len(), 2); - assert_eq!(sliced.scalar_at(0)?, arr.scalar_at(1)?); - assert_eq!(sliced.scalar_at(1)?, arr.scalar_at(2)?); + assert_eq!( + sliced.execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx())?, + arr.execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx())? + ); + assert_eq!( + sliced.execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx())?, + arr.execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx())? + ); Ok(()) } diff --git a/encodings/parquet-variant/src/operations.rs b/encodings/parquet-variant/src/operations.rs index 2efad44ae6e..e812e0fcafa 100644 --- a/encodings/parquet-variant/src/operations.rs +++ b/encodings/parquet-variant/src/operations.rs @@ -41,7 +41,7 @@ impl OperationsVTable for ParquetVariant { fn scalar_at( array: ArrayView<'_, ParquetVariant>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { if array.validity()?.is_null(index)? { return Ok(Scalar::null(DType::Variant(Nullability::Nullable))); @@ -49,7 +49,7 @@ impl OperationsVTable for ParquetVariant { let metadata = array .metadata_array() - .scalar_at(index)? + .execute_scalar(index, ctx)? .as_binary() .value() .cloned() @@ -59,6 +59,7 @@ impl OperationsVTable for ParquetVariant { array.value_array(), array.typed_value_array(), index, + ctx, )?; Scalar::try_new( @@ -73,17 +74,18 @@ fn scalar_from_variant_storage( value: Option<&ArrayRef>, typed_value: Option<&ArrayRef>, index: usize, + ctx: &mut ExecutionCtx, ) -> VortexResult { if let Some(typed_value) = typed_value - && typed_value.is_valid(index)? + && typed_value.is_valid(index, ctx)? { - return scalar_from_typed_value_array(metadata, value, typed_value, index); + return scalar_from_typed_value_array(metadata, value, typed_value, index, ctx); } if let Some(value) = value - && value.is_valid(index)? + && value.is_valid(index, ctx)? { - return scalar_from_unshredded_value(metadata, &value.scalar_at(index)?); + return scalar_from_unshredded_value(metadata, &value.execute_scalar(index, ctx)?); } Ok(Scalar::null(DType::Null)) @@ -94,12 +96,17 @@ fn scalar_from_typed_value_array( value: Option<&ArrayRef>, typed_value: &ArrayRef, index: usize, + ctx: &mut ExecutionCtx, ) -> VortexResult { let value_scalar = match value { - Some(value) if value.is_valid(index)? => Some(value.scalar_at(index)?), + Some(value) if value.is_valid(index, ctx)? => Some(value.execute_scalar(index, ctx)?), _ => None, }; - scalar_from_typed_value_scalar(metadata, value_scalar, typed_value.scalar_at(index)?) + scalar_from_typed_value_scalar( + metadata, + value_scalar, + typed_value.execute_scalar(index, ctx)?, + ) } fn scalar_from_typed_value_scalar( @@ -207,7 +214,7 @@ fn scalar_from_shredded_object_scalar( let fields = StructFields::new(FieldNames::from(names), dtypes); Scalar::try_new( DType::Struct(fields, Nullability::NonNullable), - Some(ScalarValue::List(field_values)), + Some(ScalarValue::Tuple(field_values)), ) } @@ -323,7 +330,7 @@ fn parquet_variant_to_scalar(variant: PqVariant<'_, '_>) -> VortexResult let fields = StructFields::new(FieldNames::from(names), dtypes); Scalar::try_new( DType::Struct(fields, nn), - Some(ScalarValue::List(field_values)), + Some(ScalarValue::Tuple(field_values)), )? } }) @@ -347,6 +354,7 @@ mod tests { use parquet_variant::VariantBuilder; use parquet_variant_compute::VariantArray as ArrowVariantArray; use parquet_variant_compute::VariantArrayBuilder; + use vortex_array::LEGACY_SESSION; use vortex_array::VortexSessionExecute; use vortex_array::arrays::Variant; use vortex_array::arrays::variant::VariantArrayExt; @@ -376,14 +384,15 @@ mod tests { let vortex_arr = ParquetVariantData::from_arrow_variant(arrow_variant)?; for index in rows { - let expected_inner = - parquet_variant_to_scalar(arrow_variant.try_value(index).unwrap())?; + let expected_inner = parquet_variant_to_scalar(arrow_variant.try_value(index)?)?; let expected = Scalar::try_new( vortex_arr.dtype().clone(), Some(ScalarValue::Variant(Box::new(expected_inner))), - ) - .unwrap(); - assert_eq!(vortex_arr.scalar_at(index)?, expected); + )?; + assert_eq!( + vortex_arr.execute_scalar(index, &mut LEGACY_SESSION.create_execution_ctx())?, + expected + ); } Ok(()) @@ -401,10 +410,9 @@ mod tests { inner.fields().clone(), inner.columns().to_vec(), Some(NullBuffer::from(vec![true, false, true])), - ) - .unwrap(); + )?; - let arrow_variant = ArrowVariantArray::try_new(&null_struct).unwrap(); + let arrow_variant = ArrowVariantArray::try_new(&null_struct)?; let vortex_arr = ParquetVariantData::from_arrow_variant(&arrow_variant)?; assert_eq!(vortex_arr.dtype(), &DType::Variant(Nullability::Nullable)); @@ -412,9 +420,21 @@ mod tests { let variant = vortex_arr.as_opt::().unwrap(); assert!(variant.dtype().is_nullable()); - assert!(vortex_arr.scalar_at(1)?.is_null()); - assert!(!vortex_arr.scalar_at(0)?.is_null()); - assert!(!vortex_arr.scalar_at(2)?.is_null()); + assert!( + vortex_arr + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx())? + .is_null() + ); + assert!( + !vortex_arr + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx())? + .is_null() + ); + assert!( + !vortex_arr + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx())? + .is_null() + ); Ok(()) } @@ -430,20 +450,27 @@ mod tests { inner.fields().clone(), inner.columns().to_vec(), Some(NullBuffer::from(vec![false, false])), - ) - .unwrap(); + )?; - let arrow_variant = ArrowVariantArray::try_new(&null_struct).unwrap(); + let arrow_variant = ArrowVariantArray::try_new(&null_struct)?; let vortex_arr = ParquetVariantData::from_arrow_variant(&arrow_variant)?; assert_eq!(vortex_arr.dtype(), &DType::Variant(Nullability::Nullable)); - assert!(vortex_arr.scalar_at(0)?.is_null()); - assert!(vortex_arr.scalar_at(1)?.is_null()); + assert!( + vortex_arr + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx())? + .is_null() + ); + assert!( + vortex_arr + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx())? + .is_null() + ); let variant_view = vortex_arr.as_opt::().unwrap(); let child = variant_view.child(); let inner_pv = child.as_opt::().unwrap(); - let mut ctx = vortex_array::LEGACY_SESSION.create_execution_ctx(); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let roundtripped = inner_pv.to_arrow(&mut ctx)?; assert_eq!(roundtripped.inner().null_count(), 2); @@ -463,8 +490,16 @@ mod tests { vortex_arr.dtype(), &DType::Variant(Nullability::NonNullable) ); - assert!(!vortex_arr.scalar_at(0)?.is_null()); - assert!(!vortex_arr.scalar_at(1)?.is_null()); + assert!( + !vortex_arr + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx())? + .is_null() + ); + assert!( + !vortex_arr + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx())? + .is_null() + ); assert_scalar_at_matches_arrow_try_value(&arrow_variant, [0, 1])?; Ok(()) @@ -494,10 +529,9 @@ mod tests { Arc::new(Int32Array::from(vec![10, 20, 30])), ], None, - ) - .unwrap(); + )?; - let arrow_variant = ArrowVariantArray::try_new(&struct_array).unwrap(); + let arrow_variant = ArrowVariantArray::try_new(&struct_array)?; assert_scalar_at_matches_arrow_try_value(&arrow_variant, [0, 1, 2]) } @@ -528,10 +562,9 @@ mod tests { .into(), vec![metadata, value, typed_value], None, - ) - .unwrap(); + )?; - let arrow_variant = ArrowVariantArray::try_new(&struct_array).unwrap(); + let arrow_variant = ArrowVariantArray::try_new(&struct_array)?; assert_scalar_at_matches_arrow_try_value(&arrow_variant, [0, 1, 2]) } @@ -544,22 +577,18 @@ mod tests { vec![Arc::new(Field::new("typed_value", DataType::Int32, false))].into(), vec![Arc::new(Int32Array::from(vec![10, 20, 30]))], None, - ) - .unwrap(); + )?; - let typed_value: ArrowArrayRef = Arc::new( - ListArray::try_new( - Arc::new(Field::new( - "element", - element_struct.data_type().clone(), - false, - )), - OffsetBuffer::from_lengths([2, 1]), - Arc::new(element_struct), - None, - ) - .unwrap(), - ); + let typed_value: ArrowArrayRef = Arc::new(ListArray::try_new( + Arc::new(Field::new( + "element", + element_struct.data_type().clone(), + false, + )), + OffsetBuffer::from_lengths([2, 1]), + Arc::new(element_struct), + None, + )?); let struct_array = StructArray::try_new( vec![ @@ -573,13 +602,12 @@ mod tests { .into(), vec![binary_view_array(&[b"\x01\x00", b"\x01\x00"]), typed_value], None, - ) - .unwrap(); + )?; - let arrow_variant = ArrowVariantArray::try_new(&struct_array).unwrap(); + let arrow_variant = ArrowVariantArray::try_new(&struct_array)?; let vortex_arr = ParquetVariantData::from_arrow_variant(&arrow_variant)?; - let row0 = vortex_arr.scalar_at(0)?; + let row0 = vortex_arr.execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx())?; let row0 = row0.as_variant().value().unwrap().as_list(); assert_eq!(row0.len(), 2); assert_eq!( @@ -603,7 +631,7 @@ mod tests { Some(20) ); - let row1 = vortex_arr.scalar_at(1)?; + let row1 = vortex_arr.execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx())?; let row1 = row1.as_variant().value().unwrap().as_list(); assert_eq!(row1.len(), 1); assert_eq!( @@ -637,21 +665,17 @@ mod tests { vec![Arc::new(Field::new("typed_value", DataType::Int32, false))].into(), vec![Arc::new(Int32Array::from(vec![7]))], None, - ) - .unwrap(); - let typed_value: ArrowArrayRef = Arc::new( - StructArray::try_new( - vec![Arc::new(Field::new( - "a", - shredded_a.data_type().clone(), - false, - ))] - .into(), - vec![Arc::new(shredded_a)], - None, - ) - .unwrap(), - ); + )?; + let typed_value: ArrowArrayRef = Arc::new(StructArray::try_new( + vec![Arc::new(Field::new( + "a", + shredded_a.data_type().clone(), + false, + ))] + .into(), + vec![Arc::new(shredded_a)], + None, + )?); let struct_array = StructArray::try_new( vec![ @@ -670,12 +694,11 @@ mod tests { typed_value, ], None, - ) - .unwrap(); + )?; - let arrow_variant = ArrowVariantArray::try_new(&struct_array).unwrap(); + let arrow_variant = ArrowVariantArray::try_new(&struct_array)?; let vortex_arr = ParquetVariantData::from_arrow_variant(&arrow_variant)?; - let object = vortex_arr.scalar_at(0)?; + let object = vortex_arr.execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx())?; let object = object.as_variant().value().unwrap().as_struct(); assert_eq!( diff --git a/encodings/parquet-variant/src/vtable.rs b/encodings/parquet-variant/src/vtable.rs index efc501cdac6..7c8dc942bb4 100644 --- a/encodings/parquet-variant/src/vtable.rs +++ b/encodings/parquet-variant/src/vtable.rs @@ -30,6 +30,7 @@ use vortex_error::vortex_err; use vortex_error::vortex_panic; use vortex_proto::dtype as pb; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::array::METADATA_SLOT; use crate::array::ParquetVariantArrayExt; @@ -45,10 +46,6 @@ use crate::kernel::PARENT_KERNELS; #[derive(Debug, Clone)] pub struct ParquetVariant; -impl ParquetVariant { - pub const ID: ArrayId = ArrayId::new_ref("vortex.parquet.variant"); -} - #[derive(Clone, prost::Message)] struct ParquetVariantMetadataProto { /// Whether the un-shredded `value` child is present. @@ -71,7 +68,8 @@ impl VTable for ParquetVariant { type ValidityVTable = Self; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.parquet.variant"); + *ID } fn validate( @@ -82,7 +80,7 @@ impl VTable for ParquetVariant { slots: &[Option], ) -> VortexResult<()> { let _ = data; - let validity = child_to_validity(&slots[VALIDITY_SLOT], dtype.nullability()); + let validity = child_to_validity(slots[VALIDITY_SLOT].as_ref(), dtype.nullability()); let metadata = slots[METADATA_SLOT] .as_ref() .ok_or_else(|| vortex_err!("ParquetVariantArray metadata slot"))?; @@ -246,13 +244,13 @@ mod tests { use vortex_array::IntoArray; use vortex_array::Precision; use vortex_array::arrays::VarBinViewArray; - use vortex_array::arrays::Variant; use vortex_array::arrays::VariantArray; use vortex_array::dtype::DType; use vortex_array::dtype::Nullability; use vortex_array::dtype::PType; use vortex_array::serde::SerializeOptions; use vortex_array::serde::SerializedArray; + use vortex_array::session::ArraySession; use vortex_array::session::ArraySessionExt; use vortex_array::validity::Validity; use vortex_buffer::BitBuffer; @@ -263,11 +261,14 @@ mod tests { use crate::ParquetVariant; use crate::array::ParquetVariantArrayExt; + fn roundtrip(array: ArrayRef) -> ArrayRef { let dtype = array.dtype().clone(); let len = array.len(); - let session = VortexSession::empty().with::(); + let session = VortexSession::empty().with::(); + session.arrays().register(ParquetVariant); + let ctx = ArrayContext::empty(); let serialized = array .serialize(&ctx, &session, &SerializeOptions::default()) @@ -278,8 +279,6 @@ mod tests { concat.extend_from_slice(buf.as_ref()); } let concat = concat.freeze(); - session.arrays().register(ParquetVariant); - session.arrays().register(Variant); let parts = SerializedArray::try_from(concat).unwrap(); parts diff --git a/encodings/pco/Cargo.toml b/encodings/pco/Cargo.toml index bc222bf1dcc..8d560d5f068 100644 --- a/encodings/pco/Cargo.toml +++ b/encodings/pco/Cargo.toml @@ -29,5 +29,3 @@ vortex-session = { workspace = true } rstest = { workspace = true } vortex-array = { workspace = true, features = ["_test-harness"] } -[package.metadata.cargo-machete] -ignored = ["getrandom_v03"] diff --git a/encodings/pco/public-api.lock b/encodings/pco/public-api.lock index 2cf1233c7c2..25544357593 100644 --- a/encodings/pco/public-api.lock +++ b/encodings/pco/public-api.lock @@ -4,9 +4,7 @@ pub struct vortex_pco::Pco impl vortex_pco::Pco -pub const vortex_pco::Pco::ID: vortex_array::array::ArrayId - -pub fn vortex_pco::Pco::from_primitive(parray: &vortex_array::arrays::primitive::vtable::PrimitiveArray, level: usize, values_per_page: usize) -> vortex_error::VortexResult +pub fn vortex_pco::Pco::from_primitive(vortex_array::array::view::ArrayView<'_, vortex_array::arrays::primitive::vtable::Primitive>, usize, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult impl core::clone::Clone for vortex_pco::Pco @@ -14,7 +12,7 @@ pub fn vortex_pco::Pco::clone(&self) -> vortex_pco::Pco impl core::fmt::Debug for vortex_pco::Pco -pub fn vortex_pco::Pco::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_pco::Pco::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::array::vtable::VTable for vortex_pco::Pco @@ -24,41 +22,41 @@ pub type vortex_pco::Pco::OperationsVTable = vortex_pco::Pco pub type vortex_pco::Pco::ValidityVTable = vortex_pco::Pco -pub fn vortex_pco::Pco::buffer(array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_pco::Pco::buffer(vortex_array::array::view::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_pco::Pco::buffer_name(array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_pco::Pco::buffer_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_pco::Pco::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_pco::Pco::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_pco::Pco::execute(array: vortex_array::array::typed::Array, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_pco::Pco::execute(vortex_array::array::typed::Array, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_pco::Pco::id(&self) -> vortex_array::array::ArrayId -pub fn vortex_pco::Pco::nbuffers(array: vortex_array::array::view::ArrayView<'_, Self>) -> usize +pub fn vortex_pco::Pco::nbuffers(vortex_array::array::view::ArrayView<'_, Self>) -> usize -pub fn vortex_pco::Pco::reduce_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_pco::Pco::reduce_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_pco::Pco::serialize(array: vortex_array::array::view::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_pco::Pco::serialize(vortex_array::array::view::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_pco::Pco::slot_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_pco::Pco::slot_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_pco::Pco::validate(&self, data: &vortex_pco::PcoData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_pco::Pco::validate(&self, &vortex_pco::PcoData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::array::vtable::operations::OperationsVTable for vortex_pco::Pco -pub fn vortex_pco::Pco::scalar_at(array: vortex_array::array::view::ArrayView<'_, vortex_pco::Pco>, index: usize, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_pco::Pco::scalar_at(vortex_array::array::view::ArrayView<'_, vortex_pco::Pco>, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::array::vtable::validity::ValidityVTable for vortex_pco::Pco -pub fn vortex_pco::Pco::validity(array: vortex_array::array::view::ArrayView<'_, vortex_pco::Pco>) -> vortex_error::VortexResult +pub fn vortex_pco::Pco::validity(vortex_array::array::view::ArrayView<'_, vortex_pco::Pco>) -> vortex_error::VortexResult impl vortex_array::arrays::slice::SliceReduce for vortex_pco::Pco -pub fn vortex_pco::Pco::slice(array: vortex_array::array::view::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_pco::Pco::slice(vortex_array::array::view::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::kernel::CastReduce for vortex_pco::Pco -pub fn vortex_pco::Pco::cast(array: vortex_array::array::view::ArrayView<'_, Self>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_pco::Pco::cast(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> pub struct vortex_pco::PcoChunkInfo @@ -74,7 +72,7 @@ pub fn vortex_pco::PcoChunkInfo::default() -> Self impl core::fmt::Debug for vortex_pco::PcoChunkInfo -pub fn vortex_pco::PcoChunkInfo::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_pco::PcoChunkInfo::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl prost::message::Message for vortex_pco::PcoChunkInfo @@ -86,19 +84,19 @@ pub struct vortex_pco::PcoData impl vortex_pco::PcoData -pub fn vortex_pco::PcoData::decompress(&self, unsliced_validity: &vortex_array::validity::Validity, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_pco::PcoData::decompress(&self, &vortex_array::validity::Validity, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_pco::PcoData::from_array(array: vortex_array::array::erased::ArrayRef, level: usize, nums_per_page: usize) -> vortex_error::VortexResult +pub fn vortex_pco::PcoData::from_array(vortex_array::array::erased::ArrayRef, usize, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_pco::PcoData::from_primitive(parray: &vortex_array::arrays::primitive::vtable::PrimitiveArray, level: usize, values_per_page: usize) -> vortex_error::VortexResult +pub fn vortex_pco::PcoData::from_primitive(vortex_array::array::view::ArrayView<'_, vortex_array::arrays::primitive::vtable::Primitive>, usize, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_pco::PcoData::is_empty(&self) -> bool pub fn vortex_pco::PcoData::len(&self) -> usize -pub fn vortex_pco::PcoData::new(chunk_metas: alloc::vec::Vec, pages: alloc::vec::Vec, ptype: vortex_array::dtype::ptype::PType, metadata: vortex_pco::PcoMetadata, len: usize) -> Self +pub fn vortex_pco::PcoData::new(alloc::vec::Vec, alloc::vec::Vec, vortex_array::dtype::ptype::PType, vortex_pco::PcoMetadata, usize) -> Self -pub fn vortex_pco::PcoData::validate(&self, dtype: &vortex_array::dtype::DType, len: usize, validity: &vortex_array::validity::Validity) -> vortex_error::VortexResult<()> +pub fn vortex_pco::PcoData::validate(&self, &vortex_array::dtype::DType, usize, &vortex_array::validity::Validity) -> vortex_error::VortexResult<()> impl core::clone::Clone for vortex_pco::PcoData @@ -106,19 +104,19 @@ pub fn vortex_pco::PcoData::clone(&self) -> vortex_pco::PcoData impl core::fmt::Debug for vortex_pco::PcoData -pub fn vortex_pco::PcoData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_pco::PcoData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_pco::PcoData -pub fn vortex_pco::PcoData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_pco::PcoData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::hash::ArrayEq for vortex_pco::PcoData -pub fn vortex_pco::PcoData::array_eq(&self, other: &Self, precision: vortex_array::hash::Precision) -> bool +pub fn vortex_pco::PcoData::array_eq(&self, &Self, vortex_array::hash::Precision) -> bool impl vortex_array::hash::ArrayHash for vortex_pco::PcoData -pub fn vortex_pco::PcoData::array_hash(&self, state: &mut H, precision: vortex_array::hash::Precision) +pub fn vortex_pco::PcoData::array_hash(&self, &mut H, vortex_array::hash::Precision) pub struct vortex_pco::PcoMetadata @@ -136,7 +134,7 @@ pub fn vortex_pco::PcoMetadata::default() -> Self impl core::fmt::Debug for vortex_pco::PcoMetadata -pub fn vortex_pco::PcoMetadata::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_pco::PcoMetadata::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl prost::message::Message for vortex_pco::PcoMetadata @@ -158,7 +156,7 @@ pub fn vortex_pco::PcoPageInfo::default() -> Self impl core::fmt::Debug for vortex_pco::PcoPageInfo -pub fn vortex_pco::PcoPageInfo::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_pco::PcoPageInfo::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl prost::message::Message for vortex_pco::PcoPageInfo diff --git a/encodings/pco/src/array.rs b/encodings/pco/src/array.rs index 861f80cc6eb..21d748927d4 100644 --- a/encodings/pco/src/array.rs +++ b/encodings/pco/src/array.rs @@ -28,10 +28,7 @@ use vortex_array::ArrayView; use vortex_array::ExecutionCtx; use vortex_array::ExecutionResult; use vortex_array::IntoArray; -use vortex_array::LEGACY_SESSION; use vortex_array::Precision; -use vortex_array::ToCanonical; -use vortex_array::VortexSessionExecute; use vortex_array::arrays::Primitive; use vortex_array::arrays::PrimitiveArray; use vortex_array::buffer::BufferHandle; @@ -55,6 +52,7 @@ use vortex_error::vortex_bail; use vortex_error::vortex_ensure; use vortex_error::vortex_err; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::PcoChunkInfo; use crate::PcoMetadata; @@ -129,7 +127,8 @@ impl VTable for Pco { type ValidityVTable = Self; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.pco"); + *ID } fn validate( @@ -139,7 +138,7 @@ impl VTable for Pco { len: usize, slots: &[Option], ) -> VortexResult<()> { - let validity = child_to_validity(&slots[0], dtype.nullability()); + let validity = child_to_validity(slots[0].as_ref(), dtype.nullability()); data.validate(dtype, len, &validity) } @@ -217,8 +216,10 @@ impl VTable for Pco { } fn execute(array: Array, ctx: &mut ExecutionCtx) -> VortexResult { - let unsliced_validity = - child_to_validity(&array.as_ref().slots()[0], array.dtype().nullability()); + let unsliced_validity = child_to_validity( + array.as_ref().slots()[0].as_ref(), + array.dtype().nullability(), + ); Ok(ExecutionResult::done( array .data() @@ -255,9 +256,19 @@ pub(crate) fn number_type_from_ptype(ptype: PType) -> NumberType { } } -fn collect_valid(parray: &PrimitiveArray) -> VortexResult { - let mask = parray.validity_mask()?; - Ok(parray.filter(mask)?.to_primitive()) +fn collect_valid( + parray: ArrayView<'_, Primitive>, + ctx: &mut ExecutionCtx, +) -> VortexResult { + let mask = parray + .array() + .validity()? + .execute_mask(parray.array().len(), ctx)?; + let result = parray + .array() + .filter(mask)? + .execute::(ctx)?; + Ok(result) } pub(crate) fn vortex_err_from_pco(err: PcoError) -> VortexError { @@ -273,8 +284,6 @@ pub(crate) fn vortex_err_from_pco(err: PcoError) -> VortexError { pub struct Pco; impl Pco { - pub const ID: ArrayId = ArrayId::new_ref("vortex.pco"); - pub(crate) fn try_new( dtype: DType, data: PcoData, @@ -290,13 +299,14 @@ impl Pco { /// Compress a primitive array using pcodec. pub fn from_primitive( - parray: &PrimitiveArray, + parray: ArrayView<'_, Primitive>, level: usize, values_per_page: usize, + ctx: &mut ExecutionCtx, ) -> VortexResult { let dtype = parray.dtype().clone(); let validity = parray.validity()?; - let data = PcoData::from_primitive(parray, level, values_per_page)?; + let data = PcoData::from_primitive(parray, level, values_per_page, ctx)?; Self::try_new(dtype, data, validity) } } @@ -399,18 +409,26 @@ impl PcoData { } pub fn from_primitive( - parray: &PrimitiveArray, + parray: ArrayView<'_, Primitive>, level: usize, values_per_page: usize, + ctx: &mut ExecutionCtx, ) -> VortexResult { - Self::from_primitive_with_values_per_chunk(parray, level, VALUES_PER_CHUNK, values_per_page) + Self::from_primitive_with_values_per_chunk( + parray, + level, + VALUES_PER_CHUNK, + values_per_page, + ctx, + ) } pub(crate) fn from_primitive_with_values_per_chunk( - parray: &PrimitiveArray, + parray: ArrayView<'_, Primitive>, level: usize, values_per_chunk: usize, values_per_page: usize, + ctx: &mut ExecutionCtx, ) -> VortexResult { let number_type = number_type_from_dtype(parray.dtype()); let values_per_page = if values_per_page == 0 { @@ -424,7 +442,7 @@ impl PcoData { .with_compression_level(level) .with_paging_spec(PagingSpec::EqualPagesUpTo(values_per_page)); - let values = collect_valid(parray)?; + let values = collect_valid(parray, ctx)?; let n_values = values.len(); let fc = FileCompressor::default(); @@ -478,14 +496,19 @@ impl PcoData { )) } - pub fn from_array(array: ArrayRef, level: usize, nums_per_page: usize) -> VortexResult { + pub fn from_array( + array: ArrayRef, + level: usize, + nums_per_page: usize, + ctx: &mut ExecutionCtx, + ) -> VortexResult { let parray = array.try_downcast::().map_err(|a| { vortex_err!( "Pco can only encode primitive arrays, got {}", a.encoding_id() ) })?; - Self::from_primitive(&parray, level, nums_per_page) + Self::from_primitive(parray.as_view(), level, nums_per_page, ctx) } pub fn decompress( @@ -620,7 +643,8 @@ impl PcoData { impl ValidityVTable for Pco { fn validity(array: ArrayView<'_, Pco>) -> VortexResult { - let unsliced_validity = child_to_validity(&array.slots()[0], array.dtype().nullability()); + let unsliced_validity = + child_to_validity(array.slots()[0].as_ref(), array.dtype().nullability()); unsliced_validity.slice(array.slice_start()..array.slice_stop()) } } @@ -629,21 +653,23 @@ impl OperationsVTable for Pco { fn scalar_at( array: ArrayView<'_, Pco>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { - let mut ctx = LEGACY_SESSION.create_execution_ctx(); - let unsliced_validity = child_to_validity(&array.slots()[0], array.dtype().nullability()); + let unsliced_validity = + child_to_validity(array.slots()[0].as_ref(), array.dtype().nullability()); array ._slice(index, index + 1) - .decompress(&unsliced_validity, &mut ctx)? + .decompress(&unsliced_validity, ctx)? .into_array() - .scalar_at(0) + .execute_scalar(0, ctx) } } #[cfg(test)] mod tests { use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; use vortex_array::validity::Validity; @@ -653,12 +679,13 @@ mod tests { #[test] fn test_slice_nullable() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Create a nullable array with some nulls let values = PrimitiveArray::new( buffer![10u32, 20, 30, 40, 50, 60], Validity::from_iter([false, true, true, true, true, false]), ); - let pco = Pco::from_primitive(&values, 0, 128).unwrap(); + let pco = Pco::from_primitive(values.as_view(), 0, 128, &mut ctx).unwrap(); assert_arrays_eq!( pco, PrimitiveArray::from_option_iter([ diff --git a/encodings/pco/src/compute/cast.rs b/encodings/pco/src/compute/cast.rs index 21598191fec..7cb1a28bb9c 100644 --- a/encodings/pco/src/compute/cast.rs +++ b/encodings/pco/src/compute/cast.rs @@ -11,48 +11,53 @@ use vortex_error::VortexResult; use crate::Pco; use crate::PcoData; + impl CastReduce for Pco { fn cast(array: ArrayView<'_, Self>, dtype: &DType) -> VortexResult> { - if !dtype.is_nullable() || !array.array().all_valid()? { - // TODO(joe): fixme - // We cannot cast to non-nullable since the validity containing nulls is used to decode - // the PCO array, this would require rewriting tables. + // PCO (Pcodec) stores compressed data and uses validity bits to decode (the validity + // tells PCO which logical positions correspond to compressed values). Casting away + // nullability would change the validity-to-compressed-value mapping, so we cannot + // construct a non-nullable Pco without re-encoding — we only handle nullability changes + // toward `Nullable`. Non-nullable targets fall through to canonicalization. + // + // No `CastKernel` is provided for the same reason: even with execution context, we + // cannot cast away nullability on a PCO array in place. + // + // PCO supports: F16, F32, F64, I16, I32, I64, U16, U32, U64. + if !array.dtype().eq_ignore_nullability(dtype) { return Ok(None); } - // PCO (Pcodec) is a compression encoding that stores data in a compressed format. - // It can efficiently handle nullability changes without decompression, but type changes - // require decompression since the compression algorithm is type-specific. - // PCO supports: F16, F32, F64, I16, I32, I64, U16, U32, U64 - if array.dtype().eq_ignore_nullability(dtype) { - // Create a new validity with the target nullability - let unsliced_validity = - child_to_validity(&array.slots()[0], array.dtype().nullability()); - let new_validity = - unsliced_validity.cast_nullability(dtype.nullability(), array.len())?; - - let data = PcoData::new( - array.chunk_metas.clone(), - array.pages.clone(), - dtype.as_ptype(), - array.metadata.clone(), - array.unsliced_n_rows(), - ) - ._slice(array.slice_start(), array.slice_stop()); - - return Ok(Some( - Pco::try_new(dtype.clone(), data, new_validity)?.into_array(), - )); - } - // For other casts (e.g., numeric type changes), decode to canonical and let PrimitiveArray handle it - Ok(None) + let unsliced_validity = + child_to_validity(array.slots()[0].as_ref(), array.dtype().nullability()); + let Some(new_validity) = + unsliced_validity.trivial_cast_nullability(dtype.nullability(), array.len())? + else { + return Ok(None); + }; + + let data = PcoData::new( + array.chunk_metas.clone(), + array.pages.clone(), + dtype.as_ptype(), + array.metadata.clone(), + array.unsliced_n_rows(), + ) + ._slice(array.slice_start(), array.slice_stop()); + + Ok(Some( + Pco::try_new(dtype.clone(), data, new_validity)?.into_array(), + )) } } #[cfg(test)] mod tests { + use std::sync::LazyLock; + use rstest::rstest; use vortex_array::IntoArray; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; use vortex_array::builtins::ArrayBuiltins; @@ -60,15 +65,21 @@ mod tests { use vortex_array::dtype::DType; use vortex_array::dtype::Nullability; use vortex_array::dtype::PType; + use vortex_array::session::ArraySession; use vortex_array::validity::Validity; use vortex_buffer::buffer; + use vortex_session::VortexSession; use crate::Pco; + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); + #[test] fn test_cast_pco_f32_to_f64() { + let mut ctx = SESSION.create_execution_ctx(); let values = PrimitiveArray::from_iter([1.0f32, 2.0, 3.0, 4.0, 5.0]); - let pco = Pco::from_primitive(&values, 0, 128).unwrap(); + let pco = Pco::from_primitive(values.as_view(), 0, 128, &mut ctx).unwrap(); let casted = pco .into_array() @@ -87,9 +98,10 @@ mod tests { #[test] fn test_cast_pco_nullability_change() { + let mut ctx = SESSION.create_execution_ctx(); // Test casting from NonNullable to Nullable let values = PrimitiveArray::from_iter([10u32, 20, 30, 40]); - let pco = Pco::from_primitive(&values, 0, 128).unwrap(); + let pco = Pco::from_primitive(values.as_view(), 0, 128, &mut ctx).unwrap(); let casted = pco .into_array() @@ -103,11 +115,12 @@ mod tests { #[test] fn test_cast_sliced_pco_nullable_to_nonnullable() { + let mut ctx = SESSION.create_execution_ctx(); let values = PrimitiveArray::new( buffer![10u32, 20, 30, 40, 50, 60], Validity::from_iter([true, true, true, true, true, true]), ); - let pco = Pco::from_primitive(&values, 0, 128).unwrap(); + let pco = Pco::from_primitive(values.as_view(), 0, 128, &mut ctx).unwrap(); let sliced = pco.slice(1..5).unwrap(); let casted = sliced .cast(DType::Primitive(PType::U32, Nullability::NonNullable)) @@ -122,6 +135,7 @@ mod tests { #[test] fn test_cast_sliced_pco_part_valid_to_nonnullable() { + let mut ctx = SESSION.create_execution_ctx(); let values = PrimitiveArray::from_option_iter([ None, Some(20u32), @@ -130,7 +144,7 @@ mod tests { Some(50), Some(60), ]); - let pco = Pco::from_primitive(&values, 0, 128).unwrap(); + let pco = Pco::from_primitive(values.as_view(), 0, 128, &mut ctx).unwrap(); let sliced = pco.slice(1..5).unwrap(); let casted = sliced .cast(DType::Primitive(PType::U32, Nullability::NonNullable)) @@ -164,7 +178,8 @@ mod tests { Validity::NonNullable, ))] fn test_cast_pco_conformance(#[case] values: PrimitiveArray) { - let pco = Pco::from_primitive(&values, 0, 128).unwrap(); + let mut ctx = SESSION.create_execution_ctx(); + let pco = Pco::from_primitive(values.as_view(), 0, 128, &mut ctx).unwrap(); test_cast_conformance(&pco.into_array()); } } diff --git a/encodings/pco/src/compute/mod.rs b/encodings/pco/src/compute/mod.rs index 990a54179ae..036a639ae31 100644 --- a/encodings/pco/src/compute/mod.rs +++ b/encodings/pco/src/compute/mod.rs @@ -7,6 +7,8 @@ mod cast; mod tests { use rstest::rstest; use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::compute::conformance::consistency::test_array_consistency; @@ -15,42 +17,90 @@ mod tests { fn pco_f32() -> PcoArray { let values = PrimitiveArray::from_iter([1.23f32, 4.56, 7.89, 10.11, 12.13]); - Pco::from_primitive(&values, 0, 128).unwrap() + Pco::from_primitive( + values.as_view(), + 0, + 128, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap() } fn pco_f64() -> PcoArray { let values = PrimitiveArray::from_iter([100.1f64, 200.2, 300.3, 400.4, 500.5]); - Pco::from_primitive(&values, 0, 128).unwrap() + Pco::from_primitive( + values.as_view(), + 0, + 128, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap() } fn pco_i32() -> PcoArray { let values = PrimitiveArray::from_iter([100i32, 200, 300, 400, 500]); - Pco::from_primitive(&values, 0, 128).unwrap() + Pco::from_primitive( + values.as_view(), + 0, + 128, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap() } fn pco_u64() -> PcoArray { let values = PrimitiveArray::from_iter([1000u64, 2000, 3000, 4000]); - Pco::from_primitive(&values, 0, 128).unwrap() + Pco::from_primitive( + values.as_view(), + 0, + 128, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap() } fn pco_i16() -> PcoArray { let values = PrimitiveArray::from_iter([10i16, 20, 30, 40, 50]); - Pco::from_primitive(&values, 0, 128).unwrap() + Pco::from_primitive( + values.as_view(), + 0, + 128, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap() } fn pco_i32_alt() -> PcoArray { let values = PrimitiveArray::from_iter([1i32, 2, 3, 4, 5]); - Pco::from_primitive(&values, 0, 128).unwrap() + Pco::from_primitive( + values.as_view(), + 0, + 128, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap() } fn pco_single() -> PcoArray { let values = PrimitiveArray::from_iter([42.42f64]); - Pco::from_primitive(&values, 0, 128).unwrap() + Pco::from_primitive( + values.as_view(), + 0, + 128, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap() } fn pco_large() -> PcoArray { let values = PrimitiveArray::from_iter(0u32..1000); - Pco::from_primitive(&values, 3, 128).unwrap() + Pco::from_primitive( + values.as_view(), + 3, + 128, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap() } #[rstest] diff --git a/encodings/pco/src/lib.rs b/encodings/pco/src/lib.rs index 924322c2a7e..fcf9a9397fb 100644 --- a/encodings/pco/src/lib.rs +++ b/encodings/pco/src/lib.rs @@ -5,8 +5,6 @@ mod array; mod compute; mod rules; mod slice; -#[cfg(test)] -mod test; pub use array::*; @@ -35,3 +33,6 @@ pub struct PcoMetadata { #[prost(message, repeated, tag = "2")] pub chunks: Vec, } + +#[cfg(test)] +mod tests; diff --git a/encodings/pco/src/slice.rs b/encodings/pco/src/slice.rs index 2f341f989b7..30dd1ee3e93 100644 --- a/encodings/pco/src/slice.rs +++ b/encodings/pco/src/slice.rs @@ -14,7 +14,8 @@ use crate::Pco; impl SliceReduce for Pco { fn slice(array: ArrayView<'_, Self>, range: Range) -> VortexResult> { - let unsliced_validity = child_to_validity(&array.slots()[0], array.dtype().nullability()); + let unsliced_validity = + child_to_validity(array.slots()[0].as_ref(), array.dtype().nullability()); Ok(Some( Pco::try_new( array.dtype().clone(), diff --git a/encodings/pco/src/test.rs b/encodings/pco/src/tests.rs similarity index 84% rename from encodings/pco/src/test.rs rename to encodings/pco/src/tests.rs index a431037948a..6fbed76df68 100644 --- a/encodings/pco/src/test.rs +++ b/encodings/pco/src/tests.rs @@ -1,13 +1,12 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::cast_possible_truncation)] +#![expect(clippy::cast_possible_truncation)] use std::sync::LazyLock; use vortex_array::ArrayContext; use vortex_array::IntoArray; use vortex_array::LEGACY_SESSION; -use vortex_array::ToCanonical; use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; use vortex_array::arrays::PrimitiveArray; @@ -42,16 +41,16 @@ static SESSION: LazyLock = LazyLock::new(|| { use crate::Pco; #[test] fn test_compress_decompress() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let data: Vec = (0..200).collect(); let array = PrimitiveArray::from_iter(data.clone()); - let compressed = Pco::from_primitive(&array, 3, 0).unwrap(); + let compressed = Pco::from_primitive(array.as_view(), 3, 0, &mut ctx).unwrap(); // this data should be compressible assert!(compressed.pages.len() < array.into_array().nbytes() as usize); // check full decompression works - let mut ctx = LEGACY_SESSION.create_execution_ctx(); let unsliced_validity = child_to_validity( - &compressed.as_ref().slots()[0], + compressed.as_ref().slots()[0].as_ref(), compressed.dtype().nullability(), ); let decompressed = compressed.decompress(&unsliced_validity, &mut ctx).unwrap(); @@ -70,15 +69,15 @@ fn test_compress_decompress() { #[test] fn test_compress_decompress_small() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = PrimitiveArray::from_option_iter([None, Some(1)]); - let compressed = Pco::from_primitive(&array, 3, 0).unwrap(); + let compressed = Pco::from_primitive(array.as_view(), 3, 0, &mut ctx).unwrap(); let expected = array.into_array(); assert_arrays_eq!(compressed, expected); - let mut ctx = LEGACY_SESSION.create_execution_ctx(); let unsliced_validity = child_to_validity( - &compressed.as_ref().slots()[0], + compressed.as_ref().slots()[0].as_ref(), compressed.dtype().nullability(), ); let decompressed = compressed.decompress(&unsliced_validity, &mut ctx).unwrap(); @@ -87,12 +86,12 @@ fn test_compress_decompress_small() { #[test] fn test_empty() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let data: Vec = vec![]; let array = PrimitiveArray::from_iter(data.clone()); - let compressed = Pco::from_primitive(&array, 3, 100).unwrap(); - let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let compressed = Pco::from_primitive(array.as_view(), 3, 100, &mut ctx).unwrap(); let unsliced_validity = child_to_validity( - &compressed.as_ref().slots()[0], + compressed.as_ref().slots()[0].as_ref(), compressed.dtype().nullability(), ); let primitive = compressed.decompress(&unsliced_validity, &mut ctx).unwrap(); @@ -101,6 +100,7 @@ fn test_empty() { #[test] fn test_validity_and_multiple_chunks_and_pages() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let data: Vec = (0..200).collect(); let mut validity: Vec = vec![true; 200]; validity[7..15].fill(false); @@ -116,10 +116,11 @@ fn test_validity_and_multiple_chunks_and_pages() { let compressed = Pco::try_new( array.dtype().clone(), PcoData::from_primitive_with_values_per_chunk( - &array, + array.as_view(), compression_level, values_per_chunk, values_per_page, + &mut ctx, ) .unwrap(), validity, @@ -140,9 +141,8 @@ fn test_validity_and_multiple_chunks_and_pages() { let slice = compressed.slice(100..103).unwrap(); assert_nth_scalar!(slice, 0, 100); assert_nth_scalar!(slice, 2, 102); - let primitive = slice.to_primitive(); + let primitive = slice.execute::(&mut ctx).unwrap(); - let mut ctx = LEGACY_SESSION.create_execution_ctx(); assert!( primitive .validity() @@ -157,34 +157,45 @@ fn test_validity_and_multiple_chunks_and_pages() { #[test] fn test_validity_vtable() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let data: Vec = (0..5).collect(); let mask_bools = vec![false, true, true, false, true]; let array = PrimitiveArray::new( Buffer::from(data), Validity::Array(BoolArray::from_iter(mask_bools.clone()).into_array()), ); - let compressed = Pco::from_primitive(&array, 3, 0).unwrap(); + let compressed = Pco::from_primitive(array.as_view(), 3, 0, &mut ctx).unwrap(); + let arr = compressed.as_array(); assert_eq!( - compressed.as_array().validity_mask().unwrap(), + arr.validity() + .unwrap() + .execute_mask(arr.len(), &mut ctx) + .unwrap(), Mask::from_iter(mask_bools) ); + let sliced = compressed.slice(1..4).unwrap(); assert_eq!( - compressed.slice(1..4).unwrap().validity_mask().unwrap(), + sliced + .validity() + .unwrap() + .execute_mask(sliced.len(), &mut ctx) + .unwrap(), Mask::from_iter(vec![true, true, false]) ); } #[test] fn test_serde() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); let data: PrimitiveArray = (0i32..1_000_000).collect(); - let pco = Pco::from_primitive(&data, 3, 100)?.into_array(); + let pco = Pco::from_primitive(data.as_view(), 3, 100, &mut ctx)?.into_array(); let context = ArrayContext::empty(); let bytes = pco .serialize( &context, - &LEGACY_SESSION, + &SESSION, &SerializeOptions { offset: 0, include_padding: true, @@ -202,7 +213,6 @@ fn test_serde() -> VortexResult<()> { &ReadContext::new(context.to_ids()), &SESSION, )?; - let mut ctx = SESSION.create_execution_ctx(); let data_type = data.dtype().to_arrow_dtype()?; let pco_arrow = pco.execute_arrow(Some(&data_type), &mut ctx)?; let decoded_arrow = decoded.execute_arrow(Some(&data_type), &mut ctx)?; diff --git a/encodings/runend/benches/run_end_compress.rs b/encodings/runend/benches/run_end_compress.rs index 024f3281c36..1c183f3e5fd 100644 --- a/encodings/runend/benches/run_end_compress.rs +++ b/encodings/runend/benches/run_end_compress.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] +#![expect(clippy::unwrap_used)] use divan::Bencher; use itertools::repeat_n; @@ -47,8 +47,8 @@ fn compress(bencher: Bencher, (length, run_step): (usize, usize)) { ); bencher - .with_inputs(|| &values) - .bench_refs(|values| runend_encode(values.as_view())); + .with_inputs(|| (&values, LEGACY_SESSION.create_execution_ctx())) + .bench_refs(|(values, ctx)| runend_encode(values.as_view(), ctx)); } #[divan::bench(types = [u8, u16, u32, u64], args = BENCH_ARGS)] @@ -64,7 +64,7 @@ fn decompress(bencher: Bencher, (length, run_step): (usize, usi .collect::>() .into_array(); - let run_end_array = RunEnd::new(ends, values); + let run_end_array = RunEnd::new(ends, values, &mut LEGACY_SESSION.create_execution_ctx()); let array = run_end_array.into_array(); bencher @@ -77,7 +77,7 @@ fn decompress(bencher: Bencher, (length, run_step): (usize, usi } #[divan::bench(args = BENCH_ARGS)] -#[allow(clippy::cast_possible_truncation)] +#[expect(clippy::cast_possible_truncation)] fn take_indices(bencher: Bencher, (length, run_step): (usize, usize)) { let values = PrimitiveArray::new( (0..length) @@ -88,8 +88,9 @@ fn take_indices(bencher: Bencher, (length, run_step): (usize, usize)) { ); let source_array = PrimitiveArray::from_iter(0..(length as i32)).into_array(); - let (ends, values) = runend_encode(values.as_view()); - let runend_array = RunEnd::try_new(ends.into_array(), values) + let mut encode_ctx = LEGACY_SESSION.create_execution_ctx(); + let (ends, values) = runend_encode(values.as_view(), &mut encode_ctx); + let runend_array = RunEnd::try_new(ends.into_array(), values, &mut encode_ctx) .unwrap() .into_array(); @@ -121,7 +122,7 @@ fn decompress_utf8(bencher: Bencher, (length, run_step): (usize, usize)) { let values = VarBinViewArray::from_iter_str((0..num_runs).map(|i| format!("run_value_{i}"))) .into_array(); - let run_end_array = RunEnd::new(ends, values); + let run_end_array = RunEnd::new(ends, values, &mut LEGACY_SESSION.create_execution_ctx()); let array = run_end_array.into_array(); bencher diff --git a/encodings/runend/benches/run_end_decode.rs b/encodings/runend/benches/run_end_decode.rs index b2ee2aaf3ab..818c8eb4ed6 100644 --- a/encodings/runend/benches/run_end_decode.rs +++ b/encodings/runend/benches/run_end_decode.rs @@ -1,11 +1,13 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used, clippy::cast_possible_truncation)] +#![expect(clippy::cast_possible_truncation)] use std::fmt; use divan::Bencher; +use vortex_array::LEGACY_SESSION; +use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; use vortex_array::arrays::PrimitiveArray; use vortex_array::validity::Validity; @@ -208,9 +210,15 @@ fn decode_bool(bencher: Bencher, args: BoolBenchArgs) { } = args; let (ends, values) = create_bool_test_data(total_length, avg_run_length, distribution); bencher - .with_inputs(|| (ends.clone(), values.clone())) - .bench_refs(|(ends, values)| { - runend_decode_bools(ends.clone(), values.clone(), 0, total_length) + .with_inputs(|| { + ( + ends.clone(), + values.clone(), + LEGACY_SESSION.create_execution_ctx(), + ) + }) + .bench_refs(|(ends, values, ctx)| { + runend_decode_bools(ends.clone(), values.clone(), 0, total_length, ctx) }); } @@ -371,8 +379,14 @@ fn decode_bool_nullable(bencher: Bencher, args: NullableBoolBenchArgs) { let (ends, values) = create_nullable_bool_test_data(total_length, avg_run_length, distribution, validity); bencher - .with_inputs(|| (ends.clone(), values.clone())) - .bench_refs(|(ends, values)| { - runend_decode_bools(ends.clone(), values.clone(), 0, total_length) + .with_inputs(|| { + ( + ends.clone(), + values.clone(), + LEGACY_SESSION.create_execution_ctx(), + ) + }) + .bench_refs(|(ends, values, ctx)| { + runend_decode_bools(ends.clone(), values.clone(), 0, total_length, ctx) }); } diff --git a/encodings/runend/benches/run_end_null_count.rs b/encodings/runend/benches/run_end_null_count.rs index e101abf6799..c60c72c5e0e 100644 --- a/encodings/runend/benches/run_end_null_count.rs +++ b/encodings/runend/benches/run_end_null_count.rs @@ -1,13 +1,15 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] +#![expect(clippy::unwrap_used)] use divan::Bencher; use rand::RngExt; use rand::SeedableRng; use rand::rngs::StdRng; use vortex_array::IntoArray; +use vortex_array::LEGACY_SESSION; +use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_buffer::Buffer; use vortex_runend::RunEnd; @@ -49,9 +51,11 @@ const BENCH_ARGS: &[(usize, usize, f64)] = &[ fn null_count_run_end(bencher: Bencher, (n, run_step, valid_density): (usize, usize, f64)) { let array = fixture(n, run_step, valid_density).into_array(); - bencher - .with_inputs(|| &array) - .bench_refs(|array| array.invalid_count().unwrap()); + bencher.with_inputs(|| &array).bench_refs(|array| { + array + .invalid_count(&mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + }); } fn fixture(n: usize, run_step: usize, valid_density: f64) -> RunEndArray { @@ -68,5 +72,5 @@ fn fixture(n: usize, run_step: usize, valid_density: f64) -> RunEndArray { ) .into_array(); - RunEnd::new(ends, values) + RunEnd::new(ends, values, &mut LEGACY_SESSION.create_execution_ctx()) } diff --git a/encodings/runend/public-api.lock b/encodings/runend/public-api.lock index c746d4f54a4..63c9f9857fe 100644 --- a/encodings/runend/public-api.lock +++ b/encodings/runend/public-api.lock @@ -2,35 +2,33 @@ pub mod vortex_runend pub mod vortex_runend::compress -pub fn vortex_runend::compress::runend_decode_primitive(ends: vortex_array::arrays::primitive::vtable::PrimitiveArray, values: vortex_array::arrays::primitive::vtable::PrimitiveArray, offset: usize, length: usize) -> vortex_error::VortexResult +pub fn vortex_runend::compress::runend_decode_primitive(vortex_array::arrays::primitive::vtable::PrimitiveArray, vortex_array::arrays::primitive::vtable::PrimitiveArray, usize, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_runend::compress::runend_decode_typed_primitive(run_ends: impl core::iter::traits::iterator::Iterator, values: &[T], values_validity: vortex_mask::Mask, values_nullability: vortex_array::dtype::nullability::Nullability, length: usize) -> vortex_array::arrays::primitive::vtable::PrimitiveArray +pub fn vortex_runend::compress::runend_decode_typed_primitive(impl core::iter::traits::iterator::Iterator, &[T], vortex_mask::Mask, vortex_array::dtype::nullability::Nullability, usize) -> vortex_array::arrays::primitive::vtable::PrimitiveArray -pub fn vortex_runend::compress::runend_decode_varbinview(ends: vortex_array::arrays::primitive::vtable::PrimitiveArray, values: vortex_array::arrays::varbinview::vtable::VarBinViewArray, offset: usize, length: usize) -> vortex_error::VortexResult +pub fn vortex_runend::compress::runend_decode_varbinview(vortex_array::arrays::primitive::vtable::PrimitiveArray, vortex_array::arrays::varbinview::vtable::VarBinViewArray, usize, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_runend::compress::runend_encode(array: vortex_array::array::view::ArrayView<'_, vortex_array::arrays::primitive::vtable::Primitive>) -> (vortex_array::arrays::primitive::vtable::PrimitiveArray, vortex_array::array::erased::ArrayRef) +pub fn vortex_runend::compress::runend_encode(vortex_array::array::view::ArrayView<'_, vortex_array::arrays::primitive::vtable::Primitive>, &mut vortex_array::executor::ExecutionCtx) -> (vortex_array::arrays::primitive::vtable::PrimitiveArray, vortex_array::array::erased::ArrayRef) pub mod vortex_runend::decompress_bool -pub fn vortex_runend::decompress_bool::runend_decode_bools(ends: vortex_array::arrays::primitive::vtable::PrimitiveArray, values: vortex_array::arrays::bool::vtable::BoolArray, offset: usize, length: usize) -> vortex_error::VortexResult +pub fn vortex_runend::decompress_bool::runend_decode_bools(vortex_array::arrays::primitive::vtable::PrimitiveArray, vortex_array::arrays::bool::vtable::BoolArray, usize, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_runend::decompress_bool::runend_decode_typed_bool(run_ends: impl core::iter::traits::iterator::Iterator, values: &vortex_buffer::bit::buf::BitBuffer, values_validity: vortex_mask::Mask, values_nullability: vortex_array::dtype::nullability::Nullability, length: usize) -> vortex_array::array::erased::ArrayRef +pub fn vortex_runend::decompress_bool::runend_decode_typed_bool(impl core::iter::traits::iterator::Iterator, &vortex_buffer::bit::buf::BitBuffer, vortex_mask::Mask, vortex_array::dtype::nullability::Nullability, usize) -> vortex_array::array::erased::ArrayRef pub struct vortex_runend::RunEnd impl vortex_runend::RunEnd -pub const vortex_runend::RunEnd::ID: vortex_array::array::ArrayId +pub fn vortex_runend::RunEnd::encode(vortex_array::array::erased::ArrayRef, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_runend::RunEnd::encode(array: vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_runend::RunEnd::new(vortex_array::array::erased::ArrayRef, vortex_array::array::erased::ArrayRef, &mut vortex_array::executor::ExecutionCtx) -> vortex_runend::RunEndArray -pub fn vortex_runend::RunEnd::new(ends: vortex_array::array::erased::ArrayRef, values: vortex_array::array::erased::ArrayRef) -> vortex_runend::RunEndArray +pub unsafe fn vortex_runend::RunEnd::new_unchecked(vortex_array::array::erased::ArrayRef, vortex_array::array::erased::ArrayRef, usize, usize) -> vortex_runend::RunEndArray -pub unsafe fn vortex_runend::RunEnd::new_unchecked(ends: vortex_array::array::erased::ArrayRef, values: vortex_array::array::erased::ArrayRef, offset: usize, length: usize) -> vortex_runend::RunEndArray +pub fn vortex_runend::RunEnd::try_new(vortex_array::array::erased::ArrayRef, vortex_array::array::erased::ArrayRef, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_runend::RunEnd::try_new(ends: vortex_array::array::erased::ArrayRef, values: vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult - -pub fn vortex_runend::RunEnd::try_new_offset_length(ends: vortex_array::array::erased::ArrayRef, values: vortex_array::array::erased::ArrayRef, offset: usize, length: usize) -> vortex_error::VortexResult +pub fn vortex_runend::RunEnd::try_new_offset_length(vortex_array::array::erased::ArrayRef, vortex_array::array::erased::ArrayRef, usize, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult impl core::clone::Clone for vortex_runend::RunEnd @@ -38,7 +36,7 @@ pub fn vortex_runend::RunEnd::clone(&self) -> vortex_runend::RunEnd impl core::fmt::Debug for vortex_runend::RunEnd -pub fn vortex_runend::RunEnd::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_runend::RunEnd::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::array::vtable::VTable for vortex_runend::RunEnd @@ -48,67 +46,67 @@ pub type vortex_runend::RunEnd::OperationsVTable = vortex_runend::RunEnd pub type vortex_runend::RunEnd::ValidityVTable = vortex_runend::RunEnd -pub fn vortex_runend::RunEnd::buffer(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_runend::RunEnd::buffer(vortex_array::array::view::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_runend::RunEnd::buffer_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_runend::RunEnd::buffer_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_runend::RunEnd::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_runend::RunEnd::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_runend::RunEnd::execute(array: vortex_array::array::typed::Array, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_runend::RunEnd::execute(vortex_array::array::typed::Array, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_runend::RunEnd::execute_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_runend::RunEnd::execute_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_runend::RunEnd::id(&self) -> vortex_array::array::ArrayId -pub fn vortex_runend::RunEnd::nbuffers(_array: vortex_array::array::view::ArrayView<'_, Self>) -> usize +pub fn vortex_runend::RunEnd::nbuffers(vortex_array::array::view::ArrayView<'_, Self>) -> usize -pub fn vortex_runend::RunEnd::reduce_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_runend::RunEnd::reduce_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_runend::RunEnd::serialize(array: vortex_array::array::view::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_runend::RunEnd::serialize(vortex_array::array::view::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_runend::RunEnd::slot_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_runend::RunEnd::slot_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_runend::RunEnd::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_runend::RunEnd::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::array::vtable::operations::OperationsVTable for vortex_runend::RunEnd -pub fn vortex_runend::RunEnd::scalar_at(array: vortex_array::array::view::ArrayView<'_, vortex_runend::RunEnd>, index: usize, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_runend::RunEnd::scalar_at(vortex_array::array::view::ArrayView<'_, vortex_runend::RunEnd>, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::array::vtable::validity::ValidityVTable for vortex_runend::RunEnd -pub fn vortex_runend::RunEnd::validity(array: vortex_array::array::view::ArrayView<'_, vortex_runend::RunEnd>) -> vortex_error::VortexResult +pub fn vortex_runend::RunEnd::validity(vortex_array::array::view::ArrayView<'_, vortex_runend::RunEnd>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::take::TakeExecute for vortex_runend::RunEnd -pub fn vortex_runend::RunEnd::take(array: vortex_array::array::view::ArrayView<'_, Self>, indices: &vortex_array::array::erased::ArrayRef, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_runend::RunEnd::take(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::kernel::FilterKernel for vortex_runend::RunEnd -pub fn vortex_runend::RunEnd::filter(array: vortex_array::array::view::ArrayView<'_, Self>, mask: &vortex_mask::Mask, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_runend::RunEnd::filter(vortex_array::array::view::ArrayView<'_, Self>, &vortex_mask::Mask, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::binary::compare::CompareKernel for vortex_runend::RunEnd -pub fn vortex_runend::RunEnd::compare(lhs: vortex_array::array::view::ArrayView<'_, Self>, rhs: &vortex_array::array::erased::ArrayRef, operator: vortex_array::scalar_fn::fns::operators::CompareOperator, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_runend::RunEnd::compare(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, vortex_array::scalar_fn::fns::operators::CompareOperator, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::kernel::CastReduce for vortex_runend::RunEnd -pub fn vortex_runend::RunEnd::cast(array: vortex_array::array::view::ArrayView<'_, Self>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_runend::RunEnd::cast(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::fill_null::kernel::FillNullReduce for vortex_runend::RunEnd -pub fn vortex_runend::RunEnd::fill_null(array: vortex_array::array::view::ArrayView<'_, Self>, fill_value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult> +pub fn vortex_runend::RunEnd::fill_null(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult> pub struct vortex_runend::RunEndData impl vortex_runend::RunEndData -pub fn vortex_runend::RunEndData::encode(array: vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_runend::RunEndData::encode(vortex_array::array::erased::ArrayRef, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_runend::RunEndData::into_parts(self, ends: vortex_array::array::erased::ArrayRef, values: vortex_array::array::erased::ArrayRef) -> vortex_runend::RunEndDataParts +pub fn vortex_runend::RunEndData::into_parts(self, vortex_array::array::erased::ArrayRef, vortex_array::array::erased::ArrayRef) -> vortex_runend::RunEndDataParts -pub fn vortex_runend::RunEndData::new(offset: usize) -> Self +pub fn vortex_runend::RunEndData::new(usize) -> Self -pub unsafe fn vortex_runend::RunEndData::new_unchecked(offset: usize) -> Self +pub unsafe fn vortex_runend::RunEndData::new_unchecked(usize) -> Self impl core::clone::Clone for vortex_runend::RunEndData @@ -116,23 +114,23 @@ pub fn vortex_runend::RunEndData::clone(&self) -> vortex_runend::RunEndData impl core::fmt::Debug for vortex_runend::RunEndData -pub fn vortex_runend::RunEndData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_runend::RunEndData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_runend::RunEndData -pub fn vortex_runend::RunEndData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_runend::RunEndData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::hash::ArrayEq for vortex_runend::RunEndData -pub fn vortex_runend::RunEndData::array_eq(&self, other: &Self, _precision: vortex_array::hash::Precision) -> bool +pub fn vortex_runend::RunEndData::array_eq(&self, &Self, vortex_array::hash::Precision) -> bool impl vortex_array::hash::ArrayHash for vortex_runend::RunEndData -pub fn vortex_runend::RunEndData::array_hash(&self, state: &mut H, _precision: vortex_array::hash::Precision) +pub fn vortex_runend::RunEndData::array_hash(&self, &mut H, vortex_array::hash::Precision) impl vortex_array::arrow::FromArrowArray<&arrow_array::array::run_array::RunArray> for vortex_runend::RunEndData where ::Native: vortex_array::dtype::ptype::NativePType -pub fn vortex_runend::RunEndData::from_arrow(array: &arrow_array::array::run_array::RunArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_runend::RunEndData::from_arrow(&arrow_array::array::run_array::RunArray, bool) -> vortex_error::VortexResult pub struct vortex_runend::RunEndDataParts @@ -154,7 +152,7 @@ impl vortex_runend::RunEndMetadata pub fn vortex_runend::RunEndMetadata::ends_ptype(&self) -> vortex_array::dtype::ptype::PType -pub fn vortex_runend::RunEndMetadata::set_ends_ptype(&mut self, value: vortex_array::dtype::ptype::PType) +pub fn vortex_runend::RunEndMetadata::set_ends_ptype(&mut self, vortex_array::dtype::ptype::PType) impl core::clone::Clone for vortex_runend::RunEndMetadata @@ -166,7 +164,7 @@ pub fn vortex_runend::RunEndMetadata::default() -> Self impl core::fmt::Debug for vortex_runend::RunEndMetadata -pub fn vortex_runend::RunEndMetadata::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_runend::RunEndMetadata::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl prost::message::Message for vortex_runend::RunEndMetadata @@ -180,7 +178,7 @@ pub fn vortex_runend::RunEndArrayExt::dtype(&self) -> &vortex_array::dtype::DTyp pub fn vortex_runend::RunEndArrayExt::ends(&self) -> &vortex_array::array::erased::ArrayRef -pub fn vortex_runend::RunEndArrayExt::find_physical_index(&self, index: usize) -> vortex_error::VortexResult +pub fn vortex_runend::RunEndArrayExt::find_physical_index(&self, usize) -> vortex_error::VortexResult pub fn vortex_runend::RunEndArrayExt::offset(&self) -> usize @@ -192,14 +190,14 @@ pub fn T::dtype(&self) -> &vortex_array::dtype::DType pub fn T::ends(&self) -> &vortex_array::array::erased::ArrayRef -pub fn T::find_physical_index(&self, index: usize) -> vortex_error::VortexResult +pub fn T::find_physical_index(&self, usize) -> vortex_error::VortexResult pub fn T::offset(&self) -> usize pub fn T::values(&self) -> &vortex_array::array::erased::ArrayRef -pub fn vortex_runend::initialize(session: &vortex_session::VortexSession) +pub fn vortex_runend::initialize(&vortex_session::VortexSession) -pub fn vortex_runend::trimmed_ends_iter(run_ends: &[E], offset: usize, length: usize) -> impl core::iter::traits::iterator::Iterator + use<'_, E> +pub fn vortex_runend::trimmed_ends_iter(&[E], usize, usize) -> impl core::iter::traits::iterator::Iterator + use<'_, E> pub type vortex_runend::RunEndArray = vortex_array::array::typed::Array diff --git a/encodings/runend/src/arbitrary.rs b/encodings/runend/src/arbitrary.rs index 5347935c578..baa15e0fd13 100644 --- a/encodings/runend/src/arbitrary.rs +++ b/encodings/runend/src/arbitrary.rs @@ -5,8 +5,12 @@ use arbitrary::Arbitrary; use arbitrary::Result; use arbitrary::Unstructured; use vortex_array::IntoArray; +use vortex_array::LEGACY_SESSION; +use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::arbitrary::ArbitraryArray; +use vortex_array::arrays::arbitrary::ArbitraryArrayConfig; +use vortex_array::arrays::arbitrary::ArbitraryWith; use vortex_array::dtype::DType; use vortex_array::dtype::Nullability; use vortex_array::dtype::PType; @@ -39,23 +43,39 @@ impl ArbitraryRunEndArray { // Number of runs (values/ends pairs) let num_runs = u.int_in_range(0..=20)?; + // TODO(ctx): trait fixes - Arbitrary::arbitrary has a fixed signature. + let mut ctx = LEGACY_SESSION.create_execution_ctx(); if num_runs == 0 { // Empty RunEndArray let ends = PrimitiveArray::from_iter(Vec::::new()).into_array(); - let values = ArbitraryArray::arbitrary_with(u, Some(0), dtype)?.0; - let runend_array = RunEnd::try_new(ends, values) + let values = ArbitraryArray::arbitrary_with_config( + u, + &ArbitraryArrayConfig { + dtype: Some(dtype.clone()), + len: 0..=0, + }, + )? + .0; + let runend_array = RunEnd::try_new(ends, values, &mut ctx) .vortex_expect("Empty RunEndArray creation should succeed"); return Ok(ArbitraryRunEndArray(runend_array)); } // Generate arbitrary values for each run - let values = ArbitraryArray::arbitrary_with(u, Some(num_runs), dtype)?.0; + let values = ArbitraryArray::arbitrary_with_config( + u, + &ArbitraryArrayConfig { + dtype: Some(dtype.clone()), + len: num_runs..=num_runs, + }, + )? + .0; // Generate strictly increasing ends // Each end must be > previous end, and first end must be >= 1 let ends = random_strictly_sorted_ends(u, num_runs, len)?; - let runend_array = RunEnd::try_new(ends, values) + let runend_array = RunEnd::try_new(ends, values, &mut ctx) .vortex_expect("RunEndArray creation should succeed in arbitrary impl"); Ok(ArbitraryRunEndArray(runend_array)) diff --git a/encodings/runend/src/array.rs b/encodings/runend/src/array.rs index f714d09fc70..7fe1c70461d 100644 --- a/encodings/runend/src/array.rs +++ b/encodings/runend/src/array.rs @@ -18,8 +18,10 @@ use vortex_array::ArrayView; use vortex_array::ExecutionCtx; use vortex_array::ExecutionResult; use vortex_array::IntoArray; +use vortex_array::LEGACY_SESSION; use vortex_array::Precision; use vortex_array::TypedArrayRef; +use vortex_array::VortexSessionExecute; use vortex_array::arrays::Primitive; use vortex_array::arrays::VarBinViewArray; use vortex_array::buffer::BufferHandle; @@ -39,6 +41,7 @@ use vortex_error::vortex_bail; use vortex_error::vortex_ensure; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::compress::runend_decode_primitive; use crate::compress::runend_decode_varbinview; @@ -79,7 +82,8 @@ impl VTable for RunEnd { type ValidityVTable = Self; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.runend"); + *ID } fn validate( @@ -95,7 +99,9 @@ impl VTable for RunEnd { let values = slots[VALUES_SLOT] .as_ref() .vortex_expect("RunEndArray values slot"); - RunEndData::validate_parts(ends, values, data.offset, len)?; + // TODO(ctx): trait fixes - VTable::validate has a fixed signature. + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + RunEndData::validate_parts(ends, values, data.offset, len, &mut ctx)?; vortex_ensure!( values.dtype() == dtype, "expected dtype {}, got {}", @@ -241,8 +247,6 @@ impl> RunEndArrayExt for T {} pub struct RunEnd; impl RunEnd { - pub const ID: ArrayId = ArrayId::new_ref("vortex.runend"); - /// Build a new [`RunEndArray`] without validation. /// /// # Safety @@ -254,9 +258,7 @@ impl RunEnd { length: usize, ) -> RunEndArray { let dtype = values.dtype().clone(); - let slots = vec![Some(ends.clone()), Some(values.clone())]; - RunEndData::validate_parts(&ends, &values, offset, length) - .vortex_expect("RunEndArray validation failed"); + let slots = vec![Some(ends), Some(values)]; let data = unsafe { RunEndData::new_unchecked(offset) }; unsafe { Array::from_parts_unchecked( @@ -266,8 +268,13 @@ impl RunEnd { } /// Build a new [`RunEndArray`] from ends and values. - pub fn try_new(ends: ArrayRef, values: ArrayRef) -> VortexResult { - let len = RunEndData::logical_len_from_ends(&ends)?; + pub fn try_new( + ends: ArrayRef, + values: ArrayRef, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + let len = RunEndData::logical_len_from_ends(&ends, ctx)?; + RunEndData::validate_parts(&ends, &values, 0, len, ctx)?; let dtype = values.dtype().clone(); let slots = vec![Some(ends), Some(values)]; let data = RunEndData::new(0); @@ -280,7 +287,9 @@ impl RunEnd { values: ArrayRef, offset: usize, length: usize, + ctx: &mut ExecutionCtx, ) -> VortexResult { + RunEndData::validate_parts(&ends, &values, offset, length, ctx)?; let dtype = values.dtype().clone(); let slots = vec![Some(ends), Some(values)]; let data = RunEndData::new(offset); @@ -288,14 +297,14 @@ impl RunEnd { } /// Build a new [`RunEndArray`] from ends and values (panics on invalid input). - pub fn new(ends: ArrayRef, values: ArrayRef) -> RunEndArray { - Self::try_new(ends, values).vortex_expect("RunEndData is always valid") + pub fn new(ends: ArrayRef, values: ArrayRef, ctx: &mut ExecutionCtx) -> RunEndArray { + Self::try_new(ends, values, ctx).vortex_expect("RunEndData is always valid") } /// Run the array through run-end encoding. - pub fn encode(array: ArrayRef) -> VortexResult { + pub fn encode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { if let Some(parray) = array.as_opt::() { - let (ends, values) = runend_encode(parray); + let (ends, values) = runend_encode(parray, ctx); let ends = ends.into_array(); let len = array.len(); let dtype = values.dtype().clone(); @@ -309,11 +318,11 @@ impl RunEnd { } impl RunEndData { - fn logical_len_from_ends(ends: &ArrayRef) -> VortexResult { + fn logical_len_from_ends(ends: &ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { if ends.is_empty() { Ok(0) } else { - usize::try_from(&ends.scalar_at(ends.len() - 1)?) + usize::try_from(&ends.execute_scalar(ends.len() - 1, ctx)?) } } @@ -322,6 +331,7 @@ impl RunEndData { values: &ArrayRef, offset: usize, length: usize, + ctx: &mut ExecutionCtx, ) -> VortexResult<()> { // DType validation vortex_ensure!( @@ -350,20 +360,21 @@ impl RunEndData { return Ok(()); } - debug_assert!({ + #[cfg(debug_assertions)] + { // Run ends must be strictly sorted for binary search to work correctly. let pre_validation = ends.statistics().to_owned(); let is_sorted = ends .statistics() - .compute_is_strict_sorted() + .compute_is_strict_sorted(ctx) .unwrap_or(false); // Preserve the original statistics since compute_is_strict_sorted may have mutated them. // We don't want to run with different stats in debug mode and outside. ends.statistics().inherit(pre_validation.iter()); - is_sorted - }); + debug_assert!(is_sorted); + } // Skip host-only validation when ends are not host-resident. if !ends.is_host() { @@ -372,13 +383,13 @@ impl RunEndData { // Validate the offset and length are valid for the given ends and values if offset != 0 && length != 0 { - let first_run_end = usize::try_from(&ends.scalar_at(0)?)?; + let first_run_end = usize::try_from(&ends.execute_scalar(0, ctx)?)?; if first_run_end < offset { vortex_bail!("First run end {first_run_end} must be >= offset {offset}"); } } - let last_run_end = usize::try_from(&ends.scalar_at(ends.len() - 1)?)?; + let last_run_end = usize::try_from(&ends.execute_scalar(ends.len() - 1, ctx)?)?; let min_required_end = offset + length; if last_run_end < min_required_end { vortex_bail!("Last run end {last_run_end} must be >= offset+length {min_required_end}"); @@ -399,18 +410,20 @@ impl RunEndData { /// ``` /// # use vortex_array::arrays::BoolArray; /// # use vortex_array::IntoArray; + /// # use vortex_array::{LEGACY_SESSION, VortexSessionExecute}; /// # use vortex_buffer::buffer; /// # use vortex_error::VortexResult; /// # use vortex_runend::RunEnd; /// # fn main() -> VortexResult<()> { + /// let mut ctx = LEGACY_SESSION.create_execution_ctx(); /// let ends = buffer![2u8, 3u8].into_array(); /// let values = BoolArray::from_iter([false, true]).into_array(); - /// let run_end = RunEnd::new(ends, values); + /// let run_end = RunEnd::new(ends, values, &mut ctx); /// /// // Array encodes - /// assert_eq!(run_end.scalar_at(0)?, false.into()); - /// assert_eq!(run_end.scalar_at(1)?, false.into()); - /// assert_eq!(run_end.scalar_at(2)?, true.into()); + /// assert_eq!(run_end.execute_scalar(0, &mut ctx)?, false.into()); + /// assert_eq!(run_end.execute_scalar(1, &mut ctx)?, false.into()); + /// assert_eq!(run_end.execute_scalar(2, &mut ctx)?, true.into()); /// # Ok(()) /// # } /// ``` @@ -432,9 +445,9 @@ impl RunEndData { } /// Run the array through run-end encoding. - pub fn encode(array: ArrayRef) -> VortexResult { + pub fn encode(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { if let Some(parray) = array.as_opt::() { - let (_ends, _values) = runend_encode(parray); + let (_ends, _values) = runend_encode(parray, ctx); // SAFETY: runend_encode handles this unsafe { Ok(Self::new_unchecked(0)) } } else { @@ -478,18 +491,18 @@ pub(super) fn run_end_canonicalize( Ok(match array.dtype() { DType::Bool(_) => { let bools = array.values().clone().execute_as("values", ctx)?; - runend_decode_bools(pends, bools, array.offset(), array.len())? + runend_decode_bools(pends, bools, array.offset(), array.len(), ctx)? } DType::Primitive(..) => { let pvalues = array.values().clone().execute_as("values", ctx)?; - runend_decode_primitive(pends, pvalues, array.offset(), array.len())?.into_array() + runend_decode_primitive(pends, pvalues, array.offset(), array.len(), ctx)?.into_array() } DType::Utf8(_) | DType::Binary(_) => { let values = array .values() .clone() .execute_as::("values", ctx)?; - runend_decode_varbinview(pends, values, array.offset(), array.len())?.into_array() + runend_decode_varbinview(pends, values, array.offset(), array.len(), ctx)?.into_array() } _ => vortex_bail!("Unsupported RunEnd value type: {}", array.dtype()), }) @@ -497,22 +510,32 @@ pub(super) fn run_end_canonicalize( #[cfg(test)] mod tests { + use std::sync::LazyLock; + use vortex_array::IntoArray; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::DictArray; use vortex_array::arrays::VarBinViewArray; use vortex_array::assert_arrays_eq; use vortex_array::dtype::DType; use vortex_array::dtype::Nullability; use vortex_array::dtype::PType; + use vortex_array::session::ArraySession; use vortex_buffer::buffer; + use vortex_session::VortexSession; use crate::RunEnd; + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); + #[test] fn test_runend_constructor() { + let mut ctx = SESSION.create_execution_ctx(); let arr = RunEnd::new( buffer![2u32, 5, 10].into_array(), buffer![1i32, 2, 3].into_array(), + &mut ctx, ); assert_eq!(arr.len(), 10); assert_eq!( @@ -529,8 +552,9 @@ mod tests { #[test] fn test_runend_utf8() { + let mut ctx = SESSION.create_execution_ctx(); let values = VarBinViewArray::from_iter_str(["a", "b", "c"]).into_array(); - let arr = RunEnd::new(buffer![2u32, 5, 10].into_array(), values); + let arr = RunEnd::new(buffer![2u32, 5, 10].into_array(), values, &mut ctx); assert_eq!(arr.len(), 10); assert_eq!(arr.dtype(), &DType::Utf8(Nullability::NonNullable)); @@ -542,11 +566,17 @@ mod tests { #[test] fn test_runend_dict() { + let mut ctx = SESSION.create_execution_ctx(); let dict_values = VarBinViewArray::from_iter_str(["x", "y", "z"]).into_array(); let dict_codes = buffer![0u32, 1, 2].into_array(); let dict = DictArray::try_new(dict_codes, dict_values).unwrap(); - let arr = RunEnd::try_new(buffer![2u32, 5, 10].into_array(), dict.into_array()).unwrap(); + let arr = RunEnd::try_new( + buffer![2u32, 5, 10].into_array(), + dict.into_array(), + &mut ctx, + ) + .unwrap(); assert_eq!(arr.len(), 10); let expected = diff --git a/encodings/runend/src/arrow.rs b/encodings/runend/src/arrow.rs index 564c1d06cb2..0b1d7c2ca3e 100644 --- a/encodings/runend/src/arrow.rs +++ b/encodings/runend/src/arrow.rs @@ -5,6 +5,8 @@ use arrow_array::RunArray; use arrow_array::types::RunEndIndexType; use vortex_array::ArrayRef; use vortex_array::IntoArray; +use vortex_array::LEGACY_SESSION; +use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::primitive::PrimitiveArrayExt; use vortex_array::arrow::FromArrowArray; @@ -18,6 +20,7 @@ use vortex_error::VortexResult; use crate::RunEndData; use crate::ops::find_slice_end_index; + impl FromArrowArray<&RunArray> for RunEndData where R::Native: NativePType, @@ -53,7 +56,9 @@ where }; // SAFETY: arrow-rs enforces the RunEndArray invariants, we inherit their guarantees. - RunEndData::validate_parts(&ends_slice, &values_slice, offset, len)?; + // TODO(ctx): trait fixes - FromArrowArray::from_arrow has a fixed signature. + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + RunEndData::validate_parts(&ends_slice, &values_slice, offset, len, &mut ctx)?; Ok(unsafe { RunEndData::new_unchecked(offset) }) } } @@ -76,6 +81,7 @@ mod tests { use rstest::rstest; use vortex_array::ArrayRef; use vortex_array::IntoArray as _; + use vortex_array::LEGACY_SESSION; use vortex_array::VortexSessionExecute as _; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::primitive::PrimitiveArrayExt; @@ -90,6 +96,7 @@ mod tests { use vortex_array::search_sorted::SearchSorted; use vortex_array::search_sorted::SearchSortedSide; use vortex_array::session::ArraySession; + use vortex_array::session::ArraySessionExt; use vortex_array::validity::Validity; use vortex_buffer::Buffer; use vortex_buffer::buffer; @@ -99,8 +106,11 @@ mod tests { use crate::RunEnd; use crate::ops::find_slice_end_index; - static SESSION: LazyLock = - LazyLock::new(|| VortexSession::empty().with::()); + static SESSION: LazyLock = LazyLock::new(|| { + let session = VortexSession::empty().with::(); + session.arrays().register(RunEnd); + session + }); fn decode_run_array( array: &RunArray, @@ -138,7 +148,13 @@ mod tests { ) }; - RunEnd::try_new_offset_length(ends_slice, values_slice, offset, array.len()) + RunEnd::try_new_offset_length( + ends_slice, + values_slice, + offset, + array.len(), + &mut SESSION.create_execution_ctx(), + ) } #[test] @@ -148,7 +164,7 @@ mod tests { // Values: [10, 20, 30] means values 10, 10, 10, 20, 20, 30, 30, 30 let run_ends = Int32Array::from(vec![3i32, 5, 8]); let values = Int32Array::from(vec![10, 20, 30]); - let arrow_run_array = RunArray::::try_new(&run_ends, &values).unwrap(); + let arrow_run_array = RunArray::::try_new(&run_ends, &values)?; // Convert to Vortex let vortex_array = decode_run_array(&arrow_run_array, false)?; @@ -165,7 +181,7 @@ mod tests { // Create an Arrow RunArray with nullable values let run_ends = Int32Array::from(vec![2i32, 4, 6]); let values = Int32Array::from(vec![Some(100), None, Some(300)]); - let arrow_run_array = RunArray::::try_new(&run_ends, &values).unwrap(); + let arrow_run_array = RunArray::::try_new(&run_ends, &values)?; // Convert to Vortex with nullable=true let vortex_array = decode_run_array(&arrow_run_array, true)?; @@ -189,7 +205,7 @@ mod tests { // Test with UInt64 run ends and Float64 values let run_ends = Int64Array::from(vec![1i64, 3, 4]); let values = Float64Array::from(vec![1.5, 2.5, 3.5]); - let arrow_run_array = RunArray::::try_new(&run_ends, &values).unwrap(); + let arrow_run_array = RunArray::::try_new(&run_ends, &values)?; // Convert to Vortex let vortex_array = decode_run_array(&arrow_run_array, false)?; @@ -205,7 +221,7 @@ mod tests { // Values: [100, 200, 300, 400] means: 100, 100, 200, 200, 200, 300, 300, 300, 400, 400 let run_ends = Int32Array::from(vec![2i32, 5, 8, 10]); let values = Int32Array::from(vec![100, 200, 300, 400]); - let arrow_run_array = RunArray::::try_new(&run_ends, &values).unwrap(); + let arrow_run_array = RunArray::::try_new(&run_ends, &values)?; // Slice the array from index 1 to 7 (length 6) // This should give us: 100, 200, 200, 200, 300, 300 @@ -227,7 +243,7 @@ mod tests { // Values: [Some(10), None, Some(30), Some(40)] let run_ends = Int64Array::from(vec![3i64, 6, 9, 12]); let values = Int64Array::from(vec![Some(10), None, Some(30), Some(40)]); - let arrow_run_array = RunArray::::try_new(&run_ends, &values).unwrap(); + let arrow_run_array = RunArray::::try_new(&run_ends, &values)?; // Slice from index 4 to 10 (length 6) // Original: 10, 10, 10, null, null, null, 30, 30, 30, 40, 40, 40 @@ -258,7 +274,7 @@ mod tests { // Values: [Some(10), None, Some(30), Some(40)] let run_ends = Int64Array::from(vec![3i64, 6, 9, 12]); let values = Int64Array::from(vec![Some(10), None, Some(30), Some(40)]); - let arrow_run_array = RunArray::::try_new(&run_ends, &values).unwrap(); + let arrow_run_array = RunArray::::try_new(&run_ends, &values)?; // Slice from index 4 to 4 (length 0) // Original: 10, 10, 10, null, null, null, 30, 30, 30, 40, 40, 40 @@ -327,7 +343,11 @@ mod tests { #[case] expected_ends: &[i32], #[case] expected_values: &[i32], ) -> VortexResult<()> { - let array = RunEnd::encode(PrimitiveArray::from_iter(input.iter().copied()).into_array())?; + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let array = RunEnd::encode( + PrimitiveArray::from_iter(input.iter().copied()).into_array(), + &mut ctx, + )?; let sliced = array.into_array().slice(slice_range.clone())?; let target = ree_type(DataType::Int32, DataType::Int32); let result = execute(sliced, &target)?; diff --git a/encodings/runend/src/compress.rs b/encodings/runend/src/compress.rs index df86d6cc5fb..b490076c355 100644 --- a/encodings/runend/src/compress.rs +++ b/encodings/runend/src/compress.rs @@ -4,8 +4,8 @@ use itertools::Itertools; use vortex_array::ArrayRef; use vortex_array::ArrayView; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; -use vortex_array::ToCanonical; use vortex_array::arrays::BoolArray; use vortex_array::arrays::ConstantArray; use vortex_array::arrays::Primitive; @@ -34,7 +34,10 @@ use vortex_mask::Mask; use crate::iter::trimmed_ends_iter; /// Run-end encode a `PrimitiveArray`, returning a tuple of `(ends, values)`. -pub fn runend_encode(array: ArrayView) -> (PrimitiveArray, ArrayRef) { +pub fn runend_encode( + array: ArrayView, + ctx: &mut ExecutionCtx, +) -> (PrimitiveArray, ArrayRef) { let validity = match array .validity() .vortex_expect("run-end validity should be derivable") @@ -51,7 +54,12 @@ pub fn runend_encode(array: ArrayView) -> (PrimitiveArray, ArrayRef) ConstantArray::new(Scalar::null(array.dtype().clone()), 1).into_array(), ); } - Validity::Array(a) => Some(a.to_bool().to_bit_buffer()), + Validity::Array(a) => { + let bool_array = a + .execute::(ctx) + .vortex_expect("validity array must be convertible to bool"); + Some(bool_array.to_bit_buffer()) + } }; let (ends, values) = match validity { @@ -176,8 +184,12 @@ pub fn runend_decode_primitive( values: PrimitiveArray, offset: usize, length: usize, + ctx: &mut ExecutionCtx, ) -> VortexResult { - let validity_mask = values.validity_mask()?; + let validity_mask = values + .as_ref() + .validity()? + .execute_mask(values.as_ref().len(), ctx)?; Ok(match_each_native_ptype!(values.ptype(), |P| { match_each_unsigned_integer_ptype!(ends.ptype(), |E| { runend_decode_typed_primitive( @@ -276,8 +288,12 @@ pub fn runend_decode_varbinview( values: VarBinViewArray, offset: usize, length: usize, + ctx: &mut ExecutionCtx, ) -> VortexResult { - let validity_mask = values.validity_mask()?; + let validity_mask = values + .as_ref() + .validity()? + .execute_mask(values.as_ref().len(), ctx)?; let views = values.views(); let (decoded_views, validity) = match_each_unsigned_integer_ptype!(ends.ptype(), |E| { @@ -301,8 +317,9 @@ pub fn runend_decode_varbinview( } #[cfg(test)] -mod test { - use vortex_array::ToCanonical; +mod tests { + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; use vortex_array::validity::Validity; @@ -314,55 +331,62 @@ mod test { use crate::compress::runend_encode; #[test] - fn encode() { + fn encode() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let arr = PrimitiveArray::from_iter([1i32, 1, 2, 2, 2, 3, 3, 3, 3, 3]); - let (ends, values) = runend_encode(arr.as_view()); - let values = values.to_primitive(); + let (ends, values) = runend_encode(arr.as_view(), &mut ctx); + let values = values.execute::(&mut ctx)?; let expected_ends = PrimitiveArray::from_iter(vec![2u8, 5, 10]); assert_arrays_eq!(ends, expected_ends); let expected_values = PrimitiveArray::from_iter(vec![1i32, 2, 3]); assert_arrays_eq!(values, expected_values); + Ok(()) } #[test] - fn encode_nullable() { + fn encode_nullable() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let arr = PrimitiveArray::new( buffer![1i32, 1, 2, 2, 2, 3, 3, 3, 3, 3], Validity::from(BitBuffer::from(vec![ true, true, false, false, true, true, true, true, false, false, ])), ); - let (ends, values) = runend_encode(arr.as_view()); - let values = values.to_primitive(); + let (ends, values) = runend_encode(arr.as_view(), &mut ctx); + let values = values.execute::(&mut ctx)?; let expected_ends = PrimitiveArray::from_iter(vec![2u8, 4, 5, 8, 10]); assert_arrays_eq!(ends, expected_ends); let expected_values = PrimitiveArray::from_option_iter(vec![Some(1i32), None, Some(2), Some(3), None]); assert_arrays_eq!(values, expected_values); + Ok(()) } #[test] - fn encode_all_null() { + fn encode_all_null() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let arr = PrimitiveArray::new( buffer![0, 0, 0, 0, 0], Validity::from(BitBuffer::new_unset(5)), ); - let (ends, values) = runend_encode(arr.as_view()); - let values = values.to_primitive(); + let (ends, values) = runend_encode(arr.as_view(), &mut ctx); + let values = values.execute::(&mut ctx)?; let expected_ends = PrimitiveArray::from_iter(vec![5u64]); assert_arrays_eq!(ends, expected_ends); let expected_values = PrimitiveArray::from_option_iter(vec![Option::::None]); assert_arrays_eq!(values, expected_values); + Ok(()) } #[test] fn decode() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let ends = PrimitiveArray::from_iter([2u32, 5, 10]); let values = PrimitiveArray::from_iter([1i32, 2, 3]); - let decoded = runend_decode_primitive(ends, values, 0, 10)?; + let decoded = runend_decode_primitive(ends, values, 0, 10, &mut ctx)?; let expected = PrimitiveArray::from_iter(vec![1i32, 1, 2, 2, 2, 3, 3, 3, 3, 3]); assert_arrays_eq!(decoded, expected); diff --git a/encodings/runend/src/compute/cast.rs b/encodings/runend/src/compute/cast.rs index 1333a8d076a..f70bb0becb5 100644 --- a/encodings/runend/src/compute/cast.rs +++ b/encodings/runend/src/compute/cast.rs @@ -33,9 +33,11 @@ impl CastReduce for RunEnd { #[cfg(test)] mod tests { + use std::sync::LazyLock; + use rstest::rstest; use vortex_array::IntoArray; - use vortex_array::ToCanonical; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; @@ -44,16 +46,23 @@ mod tests { use vortex_array::dtype::DType; use vortex_array::dtype::Nullability; use vortex_array::dtype::PType; + use vortex_array::session::ArraySession; use vortex_buffer::buffer; + use vortex_session::VortexSession; use crate::RunEnd; use crate::RunEndArray; + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); + #[test] fn test_cast_runend_i32_to_i64() { + let mut ctx = SESSION.create_execution_ctx(); let runend = RunEnd::try_new( buffer![3u64, 5, 8, 10].into_array(), buffer![100i32, 200, 100, 300].into_array(), + &mut ctx, ) .unwrap(); @@ -67,32 +76,34 @@ mod tests { ); // Verify by decoding to canonical form - let decoded = casted.to_primitive(); + let decoded = casted.execute::(&mut ctx).unwrap(); // RunEnd encoding should expand to [100, 100, 100, 200, 200, 100, 100, 100, 300, 300] assert_eq!(decoded.len(), 10); assert_eq!( - TryInto::::try_into(&decoded.scalar_at(0).unwrap()).unwrap(), + TryInto::::try_into(&decoded.execute_scalar(0, &mut ctx).unwrap()).unwrap(), 100i64 ); assert_eq!( - TryInto::::try_into(&decoded.scalar_at(3).unwrap()).unwrap(), + TryInto::::try_into(&decoded.execute_scalar(3, &mut ctx).unwrap()).unwrap(), 200i64 ); assert_eq!( - TryInto::::try_into(&decoded.scalar_at(5).unwrap()).unwrap(), + TryInto::::try_into(&decoded.execute_scalar(5, &mut ctx).unwrap()).unwrap(), 100i64 ); assert_eq!( - TryInto::::try_into(&decoded.scalar_at(8).unwrap()).unwrap(), + TryInto::::try_into(&decoded.execute_scalar(8, &mut ctx).unwrap()).unwrap(), 300i64 ); } #[test] fn test_cast_runend_nullable() { + let mut ctx = SESSION.create_execution_ctx(); let runend = RunEnd::try_new( buffer![2u64, 4, 7].into_array(), PrimitiveArray::from_option_iter([Some(10i32), None, Some(20)]).into_array(), + &mut ctx, ) .unwrap(); @@ -108,10 +119,12 @@ mod tests { #[test] fn test_cast_runend_with_offset() { + let mut ctx = SESSION.create_execution_ctx(); // Create a RunEndArray: [100, 100, 100, 200, 200, 300, 300, 300, 300, 300] let runend = RunEnd::try_new( buffer![3u64, 5, 10].into_array(), buffer![100i32, 200, 300].into_array(), + &mut ctx, ) .unwrap(); @@ -133,28 +146,37 @@ mod tests { ); } + type RunEndBuilder = fn(&mut vortex_array::ExecutionCtx) -> RunEndArray; + #[rstest] - #[case(RunEnd::try_new( + #[case(|ctx: &mut vortex_array::ExecutionCtx| RunEnd::try_new( buffer![3u64, 5, 8].into_array(), - buffer![100i32, 200, 300].into_array() + buffer![100i32, 200, 300].into_array(), + ctx, ).unwrap())] - #[case(RunEnd::try_new( + #[case(|ctx: &mut vortex_array::ExecutionCtx| RunEnd::try_new( buffer![1u64, 4, 10].into_array(), - buffer![1.5f32, 2.5, 3.5].into_array() + buffer![1.5f32, 2.5, 3.5].into_array(), + ctx, ).unwrap())] - #[case(RunEnd::try_new( + #[case(|ctx: &mut vortex_array::ExecutionCtx| RunEnd::try_new( buffer![2u64, 3, 5].into_array(), - PrimitiveArray::from_option_iter([Some(42i32), None, Some(84)]).into_array() + PrimitiveArray::from_option_iter([Some(42i32), None, Some(84)]).into_array(), + ctx, ).unwrap())] - #[case(RunEnd::try_new( + #[case(|ctx: &mut vortex_array::ExecutionCtx| RunEnd::try_new( buffer![10u64].into_array(), - buffer![255u8].into_array() + buffer![255u8].into_array(), + ctx, ).unwrap())] - #[case(RunEnd::try_new( + #[case(|ctx: &mut vortex_array::ExecutionCtx| RunEnd::try_new( buffer![2u64, 4, 6, 8, 10].into_array(), - BoolArray::from_iter(vec![true, false, true, false, true]).into_array() + BoolArray::from_iter(vec![true, false, true, false, true]).into_array(), + ctx, ).unwrap())] - fn test_cast_runend_conformance(#[case] array: RunEndArray) { + fn test_cast_runend_conformance(#[case] build: RunEndBuilder) { + let mut ctx = SESSION.create_execution_ctx(); + let array = build(&mut ctx); test_cast_conformance(&array.into_array()); } } diff --git a/encodings/runend/src/compute/compare.rs b/encodings/runend/src/compute/compare.rs index 8dd253e453b..4da7a453000 100644 --- a/encodings/runend/src/compute/compare.rs +++ b/encodings/runend/src/compute/compare.rs @@ -36,6 +36,7 @@ impl CompareKernel for RunEnd { values.execute::(ctx)?, lhs.offset(), lhs.len(), + ctx, ) .map(Some); } @@ -47,7 +48,10 @@ impl CompareKernel for RunEnd { #[cfg(test)] mod test { + use vortex_array::ExecutionCtx; use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; use vortex_array::arrays::ConstantArray; use vortex_array::arrays::PrimitiveArray; @@ -58,14 +62,18 @@ mod test { use crate::RunEnd; use crate::RunEndArray; - fn ree_array() -> RunEndArray { - RunEnd::encode(PrimitiveArray::from_iter([1, 1, 1, 4, 4, 4, 2, 2, 5, 5, 5, 5]).into_array()) - .unwrap() + fn ree_array(ctx: &mut ExecutionCtx) -> RunEndArray { + RunEnd::encode( + PrimitiveArray::from_iter([1, 1, 1, 4, 4, 4, 2, 2, 5, 5, 5, 5]).into_array(), + ctx, + ) + .unwrap() } #[test] fn compare_run_end() { - let arr = ree_array(); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let arr = ree_array(&mut ctx); let res = arr .into_array() .binary(ConstantArray::new(5, 12).into_array(), Operator::Eq) diff --git a/encodings/runend/src/compute/filter.rs b/encodings/runend/src/compute/filter.rs index 11093d482ac..60644e2bced 100644 --- a/encodings/runend/src/compute/filter.rs +++ b/encodings/runend/src/compute/filter.rs @@ -42,6 +42,7 @@ impl FilterKernel for RunEnd { array, mask_values.indices(), &Validity::NonNullable, + ctx, )?)) } else { let primitive_run_ends = array.ends().clone().execute::(ctx)?; @@ -115,6 +116,8 @@ fn filter_run_end_primitive + AsPrimitiv #[cfg(test)] mod tests { use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; use vortex_error::VortexResult; @@ -124,22 +127,67 @@ mod tests { use crate::RunEndArray; fn ree_array() -> RunEndArray { - RunEnd::encode(PrimitiveArray::from_iter([1, 1, 1, 4, 4, 4, 2, 2, 5, 5, 5, 5]).into_array()) - .unwrap() + RunEnd::encode( + PrimitiveArray::from_iter([1, 1, 1, 4, 4, 4, 2, 2, 5, 5, 5, 5]).into_array(), + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap() } #[test] fn filter_sliced_run_end() -> VortexResult<()> { - let arr = ree_array().slice(2..7).unwrap(); + let arr = ree_array().slice(2..7)?; let filtered = arr.filter(Mask::from_iter([true, false, false, true, true]))?; + let mut ctx = LEGACY_SESSION.create_execution_ctx(); assert_arrays_eq!( filtered, RunEnd::new( PrimitiveArray::from_iter([1u8, 2, 3]).into_array(), - PrimitiveArray::from_iter([1i32, 4, 2]).into_array() + PrimitiveArray::from_iter([1i32, 4, 2]).into_array(), + &mut ctx, ) ); Ok(()) } + + /// Regression: Filter(Slice(RunEnd)) must preserve RunEnd after execution. + /// Previously Filter.execute() forced its child to canonical, decoding + /// Slice(RunEnd) → Primitive and destroying run structure. The fix lets + /// Filter unwrap one layer at a time so RunEnd's FilterKernel can fire. + #[test] + fn filter_sliced_run_end_preserves_encoding() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + + // 4 runs of 32 each = 128 rows. Large enough that FilterKernel takes + // the run-preserving path (true_count >= 25). + let values: Vec = [10, 20, 30, 40] + .iter() + .flat_map(|&v| std::iter::repeat_n(v, 32)) + .collect(); + let arr = RunEnd::encode(PrimitiveArray::from_iter(values).into_array(), &mut ctx)?; + + // Slice off the first 16 rows. Slice(RunEnd), 112 rows, 4 runs. + let sliced = arr.into_array().slice(16..128)?; + + // Keep every other row = 112/2 = 56 rows. + let mask = Mask::from_iter((0..sliced.len()).map(|i| i % 2 == 0)); + let filtered = sliced.filter(mask)?; + + let executed = filtered.execute_until::(&mut ctx)?; + assert_eq!( + executed.encoding_id().as_ref(), + "vortex.runend", + "Filter(Slice(RunEnd)) should preserve RunEnd encoding" + ); + + let expected: Vec = std::iter::repeat_n(10, 8) + .chain(std::iter::repeat_n(20, 16)) + .chain(std::iter::repeat_n(30, 16)) + .chain(std::iter::repeat_n(40, 16)) + .collect(); + assert_arrays_eq!(executed, PrimitiveArray::from_iter(expected)); + + Ok(()) + } } diff --git a/encodings/runend/src/compute/is_constant.rs b/encodings/runend/src/compute/is_constant.rs index 53418f3ef50..d21f2334018 100644 --- a/encodings/runend/src/compute/is_constant.rs +++ b/encodings/runend/src/compute/is_constant.rs @@ -35,6 +35,6 @@ impl DynAggregateKernel for RunEndIsConstantKernel { }; let result = is_constant(array.values(), ctx)?; - Ok(Some(IsConstant::make_partial(batch, result)?)) + Ok(Some(IsConstant::make_partial(batch, result, ctx)?)) } } diff --git a/encodings/runend/src/compute/is_sorted.rs b/encodings/runend/src/compute/is_sorted.rs index e26a3d8cb57..65de2a174f2 100644 --- a/encodings/runend/src/compute/is_sorted.rs +++ b/encodings/runend/src/compute/is_sorted.rs @@ -2,6 +2,7 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors use vortex_array::ArrayRef; +use vortex_array::Canonical; use vortex_array::ExecutionCtx; use vortex_array::IntoArray as _; use vortex_array::aggregate_fn::AggregateFnRef; @@ -40,11 +41,23 @@ impl DynAggregateKernel for RunEndIsSortedKernel { let result = if options.strict { // Strict sort with run-end encoding means we need to canonicalize // since run-end encoding repeats values. - is_strict_sorted(&array.array().to_canonical()?.into_array(), ctx)? + is_strict_sorted( + &array + .array() + .clone() + .execute::(ctx)? + .into_array(), + ctx, + )? } else { is_sorted(array.values(), ctx)? }; - Ok(Some(IsSorted::make_partial(batch, result, options.strict)?)) + Ok(Some(IsSorted::make_partial( + batch, + result, + options.strict, + ctx, + )?)) } } diff --git a/encodings/runend/src/compute/mod.rs b/encodings/runend/src/compute/mod.rs index f1f6dd97593..9bdb6d67c00 100644 --- a/encodings/runend/src/compute/mod.rs +++ b/encodings/runend/src/compute/mod.rs @@ -15,6 +15,8 @@ pub(crate) mod take_from; mod tests { use rstest::rstest; use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::compute::conformance::consistency::test_array_consistency; use vortex_buffer::buffer; @@ -25,24 +27,30 @@ mod tests { #[rstest] // Simple run-end arrays #[case::runend_i32(RunEnd::encode( - buffer![1i32, 1, 1, 2, 2, 3, 3, 3, 3].into_array() + buffer![1i32, 1, 1, 2, 2, 3, 3, 3, 3].into_array(), + &mut LEGACY_SESSION.create_execution_ctx(), ).unwrap())] #[case::runend_single_run(RunEnd::encode( - buffer![5i32, 5, 5, 5, 5].into_array() + buffer![5i32, 5, 5, 5, 5].into_array(), + &mut LEGACY_SESSION.create_execution_ctx(), ).unwrap())] #[case::runend_alternating(RunEnd::encode( - buffer![1i32, 2, 1, 2, 1, 2].into_array() + buffer![1i32, 2, 1, 2, 1, 2].into_array(), + &mut LEGACY_SESSION.create_execution_ctx(), ).unwrap())] // Different types #[case::runend_u64(RunEnd::encode( - buffer![100u64, 100, 200, 200, 200].into_array() + buffer![100u64, 100, 200, 200, 200].into_array(), + &mut LEGACY_SESSION.create_execution_ctx(), ).unwrap())] // Edge cases #[case::runend_single(RunEnd::encode( - buffer![42i32].into_array() + buffer![42i32].into_array(), + &mut LEGACY_SESSION.create_execution_ctx(), ).unwrap())] #[case::runend_large(RunEnd::encode( - PrimitiveArray::from_iter((0..1000).map(|i| i / 10)).into_array() + PrimitiveArray::from_iter((0..1000).map(|i| i / 10)).into_array(), + &mut LEGACY_SESSION.create_execution_ctx(), ).unwrap())] fn test_runend_consistency(#[case] array: RunEndArray) { diff --git a/encodings/runend/src/compute/take.rs b/encodings/runend/src/compute/take.rs index 9bbdbf6784e..7100faf9eac 100644 --- a/encodings/runend/src/compute/take.rs +++ b/encodings/runend/src/compute/take.rs @@ -7,7 +7,6 @@ use vortex_array::ArrayRef; use vortex_array::ArrayView; use vortex_array::ExecutionCtx; use vortex_array::IntoArray; -use vortex_array::ToCanonical; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::dict::TakeExecute; use vortex_array::match_each_integer_ptype; @@ -50,7 +49,7 @@ impl TakeExecute for RunEnd { }); let indices_validity = primitive_indices.validity()?; - take_indices_unchecked(array, &checked_indices, &indices_validity).map(Some) + take_indices_unchecked(array, &checked_indices, &indices_validity, ctx).map(Some) } } @@ -59,8 +58,9 @@ pub fn take_indices_unchecked>( array: ArrayView<'_, RunEnd>, indices: &[T], validity: &Validity, + ctx: &mut ExecutionCtx, ) -> VortexResult { - let ends = array.ends().to_primitive(); + let ends = array.ends().clone().execute::(ctx)?; let ends_len = ends.len(); // TODO(joe): use the validity mask to skip search sorted. @@ -89,7 +89,7 @@ pub fn take_indices_unchecked>( } #[cfg(test)] -mod test { +mod tests { use rstest::rstest; use vortex_array::ArrayRef; use vortex_array::Canonical; @@ -105,7 +105,11 @@ mod test { use crate::RunEndArray; fn ree_array() -> RunEndArray { - RunEnd::encode(buffer![1, 1, 1, 4, 4, 4, 2, 2, 5, 5, 5, 5].into_array()).unwrap() + RunEnd::encode( + buffer![1, 1, 1, 4, 4, 4, 2, 2, 5, 5, 5, 5].into_array(), + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap() } #[test] @@ -155,6 +159,7 @@ mod test { #[case(ree_array())] #[case(RunEnd::encode( buffer![1u8, 1, 2, 2, 2, 3, 3, 3, 3, 4].into_array(), + &mut LEGACY_SESSION.create_execution_ctx(), ).unwrap())] #[case(RunEnd::encode( PrimitiveArray::from_option_iter([ @@ -167,11 +172,14 @@ mod test { Some(20), ]) .into_array(), + &mut LEGACY_SESSION.create_execution_ctx(), ).unwrap())] - #[case(RunEnd::encode(buffer![42i32, 42, 42, 42, 42].into_array()) + #[case(RunEnd::encode(buffer![42i32, 42, 42, 42, 42].into_array(), + &mut LEGACY_SESSION.create_execution_ctx()) .unwrap())] #[case(RunEnd::encode( buffer![1i32, 2, 3, 4, 5, 6, 7, 8, 9, 10].into_array(), + &mut LEGACY_SESSION.create_execution_ctx(), ).unwrap())] #[case({ let mut values = Vec::new(); @@ -180,7 +188,11 @@ mod test { values.push(i); } } - RunEnd::encode(PrimitiveArray::from_iter(values).into_array()).unwrap() + RunEnd::encode( + PrimitiveArray::from_iter(values).into_array(), + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap() })] fn test_take_runend_conformance(#[case] array: RunEndArray) { test_take_conformance(&array.into_array()); @@ -191,6 +203,7 @@ mod test { #[case({ let array = RunEnd::encode( buffer![1i32, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3].into_array(), + &mut LEGACY_SESSION.create_execution_ctx(), ) .unwrap(); array.slice(2..8).unwrap() diff --git a/encodings/runend/src/compute/take_from.rs b/encodings/runend/src/compute/take_from.rs index 4e913f5feae..32b22eb96c5 100644 --- a/encodings/runend/src/compute/take_from.rs +++ b/encodings/runend/src/compute/take_from.rs @@ -46,13 +46,13 @@ impl ExecuteParentKernel for RunEndTakeFrom { array.len(), ) }; - // Ok(Some(ree_array.into_array())) } } #[cfg(test)] mod tests { + use vortex_array::Canonical; use vortex_array::ExecutionCtx; use vortex_array::IntoArray; use vortex_array::arrays::DictArray; @@ -74,8 +74,8 @@ mod tests { /// Dict values: `[2, 3]` /// Codes: `[0, 0, 0, 1, 1, 0, 0]` /// RunEnd encoded codes: ends=`[3, 5, 7]`, values=`[0, 1, 0]` - fn make_dict_with_runend_codes() -> (RunEndArray, DictArray) { - let codes = RunEnd::encode(buffer![0u32, 0, 0, 1, 1, 0, 0].into_array()).unwrap(); + fn make_dict_with_runend_codes(ctx: &mut ExecutionCtx) -> (RunEndArray, DictArray) { + let codes = RunEnd::encode(buffer![0u32, 0, 0, 1, 1, 0, 0].into_array(), ctx).unwrap(); let values = buffer![2i32, 3].into_array(); let dict = DictArray::try_new(codes.clone().into_array(), values).unwrap(); (codes, dict) @@ -83,21 +83,23 @@ mod tests { #[test] fn test_execute_parent_no_offset() -> VortexResult<()> { - let (codes, dict) = make_dict_with_runend_codes(); let mut ctx = ExecutionCtx::new(VortexSession::empty()); + let (codes, dict) = make_dict_with_runend_codes(&mut ctx); let result = RunEndTakeFrom .execute_parent(codes.as_view(), dict.as_view(), 0, &mut ctx)? .expect("kernel should return Some"); let expected = PrimitiveArray::from_iter([2i32, 2, 2, 3, 3, 2, 2]); - assert_arrays_eq!(result.to_canonical()?.into_array(), expected); + let canonical = result.execute::(&mut ctx)?.into_array(); + assert_arrays_eq!(canonical, expected); Ok(()) } #[test] fn test_execute_parent_with_offset() -> VortexResult<()> { - let (codes, dict) = make_dict_with_runend_codes(); + let mut ctx = ExecutionCtx::new(VortexSession::empty()); + let (codes, dict) = make_dict_with_runend_codes(&mut ctx); // Slice codes to positions 2..5 → logical codes [0, 1, 1] → values [2, 3, 3] let sliced_codes = unsafe { RunEnd::new_unchecked( @@ -107,20 +109,21 @@ mod tests { 3, // len ) }; - let mut ctx = ExecutionCtx::new(VortexSession::empty()); let result = RunEndTakeFrom .execute_parent(sliced_codes.as_view(), dict.as_view(), 0, &mut ctx)? .expect("kernel should return Some"); let expected = PrimitiveArray::from_iter([2i32, 3, 3]); - assert_arrays_eq!(result.to_canonical()?.into_array(), expected); + let canonical = result.execute::(&mut ctx)?.into_array(); + assert_arrays_eq!(canonical, expected); Ok(()) } #[test] fn test_execute_parent_offset_at_run_boundary() -> VortexResult<()> { - let (codes, dict) = make_dict_with_runend_codes(); + let mut ctx = ExecutionCtx::new(VortexSession::empty()); + let (codes, dict) = make_dict_with_runend_codes(&mut ctx); // Slice codes to positions 3..7 → logical codes [1, 1, 0, 0] → values [3, 3, 2, 2] let sliced_codes = unsafe { RunEnd::new_unchecked( @@ -130,20 +133,21 @@ mod tests { 4, // len ) }; - let mut ctx = ExecutionCtx::new(VortexSession::empty()); let result = RunEndTakeFrom .execute_parent(sliced_codes.as_view(), dict.as_view(), 0, &mut ctx)? .expect("kernel should return Some"); let expected = PrimitiveArray::from_iter([3i32, 3, 2, 2]); - assert_arrays_eq!(result.to_canonical()?.into_array(), expected); + let canonical = result.execute::(&mut ctx)?.into_array(); + assert_arrays_eq!(canonical, expected); Ok(()) } #[test] fn test_execute_parent_single_element_offset() -> VortexResult<()> { - let (codes, dict) = make_dict_with_runend_codes(); + let mut ctx = ExecutionCtx::new(VortexSession::empty()); + let (codes, dict) = make_dict_with_runend_codes(&mut ctx); // Slice to single element at position 4 → code=1 → value=3 let sliced_codes = unsafe { RunEnd::new_unchecked( @@ -153,21 +157,21 @@ mod tests { 1, // len ) }; - let mut ctx = ExecutionCtx::new(VortexSession::empty()); let result = RunEndTakeFrom .execute_parent(sliced_codes.as_view(), dict.as_view(), 0, &mut ctx)? .expect("kernel should return Some"); let expected = PrimitiveArray::from_iter([3i32]); - assert_arrays_eq!(result.to_canonical()?.into_array(), expected); + let canonical = result.execute::(&mut ctx)?.into_array(); + assert_arrays_eq!(canonical, expected); Ok(()) } #[test] fn test_execute_parent_returns_none_for_non_codes_child() -> VortexResult<()> { - let (codes, dict) = make_dict_with_runend_codes(); let mut ctx = ExecutionCtx::new(VortexSession::empty()); + let (codes, dict) = make_dict_with_runend_codes(&mut ctx); let result = RunEndTakeFrom.execute_parent(codes.as_view(), dict.as_view(), 1, &mut ctx)?; assert!(result.is_none()); diff --git a/encodings/runend/src/decompress_bool.rs b/encodings/runend/src/decompress_bool.rs index 6733f143a8c..68b6d3d5981 100644 --- a/encodings/runend/src/decompress_bool.rs +++ b/encodings/runend/src/decompress_bool.rs @@ -8,6 +8,7 @@ use itertools::Itertools; use vortex_array::ArrayRef; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; use vortex_array::arrays::BoolArray; use vortex_array::arrays::ConstantArray; @@ -35,8 +36,12 @@ pub fn runend_decode_bools( values: BoolArray, offset: usize, length: usize, + ctx: &mut ExecutionCtx, ) -> VortexResult { - let validity = values.validity_mask()?; + let validity = values + .as_ref() + .validity()? + .execute_mask(values.as_ref().len(), ctx)?; let values_buf = values.to_bit_buffer(); let nullability = values.dtype().nullability(); @@ -241,7 +246,8 @@ fn decode_nullable_sequential( #[cfg(test)] mod tests { - use vortex_array::ToCanonical; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::bool::BoolArrayExt; @@ -254,10 +260,11 @@ mod tests { #[test] fn decode_bools_alternating() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Alternating true/false: [T, T, F, F, F, T, T, T, T, T] let ends = PrimitiveArray::from_iter([2u32, 5, 10]); let values = BoolArray::from(BitBuffer::from(vec![true, false, true])); - let decoded = runend_decode_bools(ends, values, 0, 10)?; + let decoded = runend_decode_bools(ends, values, 0, 10, &mut ctx)?; let expected = BoolArray::from(BitBuffer::from(vec![ true, true, false, false, false, true, true, true, true, true, @@ -268,10 +275,11 @@ mod tests { #[test] fn decode_bools_mostly_true() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Mostly true: [T, T, T, T, T, F, T, T, T, T] let ends = PrimitiveArray::from_iter([5u32, 6, 10]); let values = BoolArray::from(BitBuffer::from(vec![true, false, true])); - let decoded = runend_decode_bools(ends, values, 0, 10)?; + let decoded = runend_decode_bools(ends, values, 0, 10, &mut ctx)?; let expected = BoolArray::from(BitBuffer::from(vec![ true, true, true, true, true, false, true, true, true, true, @@ -282,10 +290,11 @@ mod tests { #[test] fn decode_bools_mostly_false() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Mostly false: [F, F, F, F, F, T, F, F, F, F] let ends = PrimitiveArray::from_iter([5u32, 6, 10]); let values = BoolArray::from(BitBuffer::from(vec![false, true, false])); - let decoded = runend_decode_bools(ends, values, 0, 10)?; + let decoded = runend_decode_bools(ends, values, 0, 10, &mut ctx)?; let expected = BoolArray::from(BitBuffer::from(vec![ false, false, false, false, false, true, false, false, false, false, @@ -296,9 +305,10 @@ mod tests { #[test] fn decode_bools_all_true_single_run() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let ends = PrimitiveArray::from_iter([10u32]); let values = BoolArray::from(BitBuffer::from(vec![true])); - let decoded = runend_decode_bools(ends, values, 0, 10)?; + let decoded = runend_decode_bools(ends, values, 0, 10, &mut ctx)?; let expected = BoolArray::from(BitBuffer::from(vec![ true, true, true, true, true, true, true, true, true, true, @@ -309,9 +319,10 @@ mod tests { #[test] fn decode_bools_all_false_single_run() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let ends = PrimitiveArray::from_iter([10u32]); let values = BoolArray::from(BitBuffer::from(vec![false])); - let decoded = runend_decode_bools(ends, values, 0, 10)?; + let decoded = runend_decode_bools(ends, values, 0, 10, &mut ctx)?; let expected = BoolArray::from(BitBuffer::from(vec![ false, false, false, false, false, false, false, false, false, false, @@ -322,10 +333,11 @@ mod tests { #[test] fn decode_bools_with_offset() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Test with offset: [T, T, F, F, F, T, T, T, T, T] -> slice [2..8] = [F, F, F, T, T, T] let ends = PrimitiveArray::from_iter([2u32, 5, 10]); let values = BoolArray::from(BitBuffer::from(vec![true, false, true])); - let decoded = runend_decode_bools(ends, values, 2, 6)?; + let decoded = runend_decode_bools(ends, values, 2, 6, &mut ctx)?; let expected = BoolArray::from(BitBuffer::from(vec![false, false, false, true, true, true])); @@ -337,13 +349,14 @@ mod tests { fn decode_bools_nullable() -> VortexResult<()> { use vortex_array::validity::Validity; + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // 3 runs: T (valid), F (null), T (valid) -> [T, T, null, null, null, T, T, T, T, T] let ends = PrimitiveArray::from_iter([2u32, 5, 10]); let values = BoolArray::new( BitBuffer::from(vec![true, false, true]), Validity::from(BitBuffer::from(vec![true, false, true])), ); - let decoded = runend_decode_bools(ends, values, 0, 10)?; + let decoded = runend_decode_bools(ends, values, 0, 10, &mut ctx)?; // Expected: values=[T, T, F, F, F, T, T, T, T, T], validity=[1, 1, 0, 0, 0, 1, 1, 1, 1, 1] let expected = BoolArray::new( @@ -360,23 +373,43 @@ mod tests { #[test] fn decode_bools_nullable_few_runs() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Test few runs (uses fast path): 5 runs of length 2000 each let ends = PrimitiveArray::from_iter([2000u32, 4000, 6000, 8000, 10000]); let values = BoolArray::new( BitBuffer::from(vec![true, false, true, false, true]), Validity::from(BitBuffer::from(vec![true, false, true, false, true])), ); - let decoded = runend_decode_bools(ends, values, 0, 10000)?.to_bool(); + let decoded = runend_decode_bools(ends, values, 0, 10000, &mut ctx)? + .execute::(&mut ctx)?; // Check length and a few values assert_eq!(decoded.len(), 10000); // First run: valid true - assert!(decoded.validity_mask()?.value(0)); + assert!( + decoded + .as_ref() + .validity()? + .execute_mask(decoded.as_ref().len(), &mut ctx)? + .value(0) + ); assert!(decoded.to_bit_buffer().value(0)); // Second run: null (validity false) - assert!(!decoded.validity_mask()?.value(2000)); + assert!( + !decoded + .as_ref() + .validity()? + .execute_mask(decoded.as_ref().len(), &mut ctx)? + .value(2000) + ); // Third run: valid true - assert!(decoded.validity_mask()?.value(4000)); + assert!( + decoded + .as_ref() + .validity()? + .execute_mask(decoded.as_ref().len(), &mut ctx)? + .value(4000) + ); assert!(decoded.to_bit_buffer().value(4000)); Ok(()) } diff --git a/encodings/runend/src/kernel.rs b/encodings/runend/src/kernel.rs index 432453ee609..73f6ebc524e 100644 --- a/encodings/runend/src/kernel.rs +++ b/encodings/runend/src/kernel.rs @@ -43,13 +43,17 @@ impl ExecuteParentKernel for RunEndSliceKernel { array: ArrayView<'_, RunEnd>, parent: ArrayView<'_, Slice>, _child_idx: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult> { - slice(array, parent.slice_range().clone()).map(Some) + slice(array, parent.slice_range().clone(), ctx).map(Some) } } -fn slice(array: ArrayView<'_, RunEnd>, range: Range) -> VortexResult { +fn slice( + array: ArrayView<'_, RunEnd>, + range: Range, + ctx: &mut ExecutionCtx, +) -> VortexResult { let new_length = range.len(); let slice_begin = array.find_physical_index(range.start)?; @@ -57,7 +61,7 @@ fn slice(array: ArrayView<'_, RunEnd>, range: Range) -> VortexResult for RunEnd { fn scalar_at( array: ArrayView<'_, RunEnd>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { - array.values().scalar_at(array.find_physical_index(index)?) + array + .values() + .execute_scalar(array.find_physical_index(index)?, ctx) } } @@ -63,9 +65,11 @@ mod tests { #[test] fn slice_array() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let arr = RunEnd::try_new( buffer![2u32, 5, 10].into_array(), buffer![1i32, 2, 3].into_array(), + &mut ctx, ) .unwrap() .slice(3..8) @@ -82,9 +86,11 @@ mod tests { #[test] fn double_slice() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let arr = RunEnd::try_new( buffer![2u32, 5, 10].into_array(), buffer![1i32, 2, 3].into_array(), + &mut ctx, ) .unwrap() .slice(3..8) @@ -99,9 +105,11 @@ mod tests { #[test] fn slice_end_inclusive() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let arr = RunEnd::try_new( buffer![2u32, 5, 10].into_array(), buffer![1i32, 2, 3].into_array(), + &mut ctx, ) .unwrap() .slice(4..10) @@ -118,9 +126,11 @@ mod tests { #[test] fn slice_at_end() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let re_array = RunEnd::try_new( buffer![7_u64, 10].into_array(), buffer![2_u64, 3].into_array(), + &mut ctx, ) .unwrap(); @@ -132,9 +142,11 @@ mod tests { #[test] fn slice_single_end() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let re_array = RunEnd::try_new( buffer![7_u64, 10].into_array(), buffer![2_u64, 3].into_array(), + &mut ctx, ) .unwrap(); @@ -142,27 +154,31 @@ mod tests { let sliced_array = re_array.slice(2..5).unwrap(); - let mut ctx = LEGACY_SESSION.create_execution_ctx(); assert!(is_constant(&sliced_array, &mut ctx).unwrap()) } #[test] fn ree_scalar_at_end() { - let scalar = RunEnd::encode(buffer![1, 1, 1, 4, 4, 4, 2, 2, 5, 5, 5, 5].into_array()) - .unwrap() - .scalar_at(11) - .unwrap(); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let scalar = RunEnd::encode( + buffer![1, 1, 1, 4, 4, 4, 2, 2, 5, 5, 5, 5].into_array(), + &mut ctx, + ) + .unwrap() + .execute_scalar(11, &mut ctx) + .unwrap(); assert_eq!(scalar, 5.into()); } #[test] - #[allow(clippy::cognitive_complexity)] fn slice_along_run_boundaries() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Create a runend array with runs: [1, 1, 1] [4, 4, 4] [2, 2] [5, 5, 5, 5] // Run ends at indices: 3, 6, 8, 12 let arr = RunEnd::try_new( buffer![3u32, 6, 8, 12].into_array(), buffer![1i32, 4, 2, 5].into_array(), + &mut ctx, ) .unwrap(); diff --git a/encodings/runend/src/rules.rs b/encodings/runend/src/rules.rs index 73f9ef3cb7a..cd80e42db46 100644 --- a/encodings/runend/src/rules.rs +++ b/encodings/runend/src/rules.rs @@ -8,8 +8,8 @@ use vortex_array::arrays::Constant; use vortex_array::arrays::ConstantArray; use vortex_array::arrays::ScalarFnArray; use vortex_array::arrays::scalar_fn::AnyScalarFn; +use vortex_array::arrays::scalar_fn::ScalarFn; use vortex_array::arrays::scalar_fn::ScalarFnArrayExt; -use vortex_array::arrays::scalar_fn::ScalarFnVTable; use vortex_array::dtype::DType; use vortex_array::optimizer::rules::ArrayParentReduceRule; use vortex_array::optimizer::rules::ParentRuleSet; @@ -41,7 +41,7 @@ impl ArrayParentReduceRule for RunEndScalarFnRule { fn reduce_parent( &self, run_end: ArrayView<'_, RunEnd>, - parent: ArrayView<'_, ScalarFnVTable>, + parent: ArrayView<'_, ScalarFn>, child_idx: usize, ) -> VortexResult> { for (idx, child) in parent.iter_children().enumerate() { diff --git a/encodings/sequence/public-api.lock b/encodings/sequence/public-api.lock index aaacb5d3963..96924559159 100644 --- a/encodings/sequence/public-api.lock +++ b/encodings/sequence/public-api.lock @@ -4,11 +4,9 @@ pub struct vortex_sequence::Sequence impl vortex_sequence::Sequence -pub const vortex_sequence::Sequence::ID: vortex_array::array::ArrayId +pub fn vortex_sequence::Sequence::try_new(vortex_array::scalar::typed_view::primitive::pvalue::PValue, vortex_array::scalar::typed_view::primitive::pvalue::PValue, vortex_array::dtype::ptype::PType, vortex_array::dtype::nullability::Nullability, usize) -> vortex_error::VortexResult -pub fn vortex_sequence::Sequence::try_new(base: vortex_array::scalar::typed_view::primitive::pvalue::PValue, multiplier: vortex_array::scalar::typed_view::primitive::pvalue::PValue, ptype: vortex_array::dtype::ptype::PType, nullability: vortex_array::dtype::nullability::Nullability, length: usize) -> vortex_error::VortexResult - -pub fn vortex_sequence::Sequence::try_new_typed>(base: T, multiplier: T, nullability: vortex_array::dtype::nullability::Nullability, length: usize) -> vortex_error::VortexResult +pub fn vortex_sequence::Sequence::try_new_typed>(T, T, vortex_array::dtype::nullability::Nullability, usize) -> vortex_error::VortexResult impl core::clone::Clone for vortex_sequence::Sequence @@ -16,7 +14,7 @@ pub fn vortex_sequence::Sequence::clone(&self) -> vortex_sequence::Sequence impl core::fmt::Debug for vortex_sequence::Sequence -pub fn vortex_sequence::Sequence::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_sequence::Sequence::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::array::vtable::VTable for vortex_sequence::Sequence @@ -26,59 +24,59 @@ pub type vortex_sequence::Sequence::OperationsVTable = vortex_sequence::Sequence pub type vortex_sequence::Sequence::ValidityVTable = vortex_sequence::Sequence -pub fn vortex_sequence::Sequence::buffer(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_sequence::Sequence::buffer(vortex_array::array::view::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_sequence::Sequence::buffer_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_sequence::Sequence::buffer_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_sequence::Sequence::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_sequence::Sequence::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_sequence::Sequence::execute(array: vortex_array::array::typed::Array, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_sequence::Sequence::execute(vortex_array::array::typed::Array, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_sequence::Sequence::execute_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_sequence::Sequence::execute_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_sequence::Sequence::id(&self) -> vortex_array::array::ArrayId -pub fn vortex_sequence::Sequence::nbuffers(_array: vortex_array::array::view::ArrayView<'_, Self>) -> usize +pub fn vortex_sequence::Sequence::nbuffers(vortex_array::array::view::ArrayView<'_, Self>) -> usize -pub fn vortex_sequence::Sequence::reduce_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_sequence::Sequence::reduce_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_sequence::Sequence::serialize(array: vortex_array::array::view::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_sequence::Sequence::serialize(vortex_array::array::view::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_sequence::Sequence::slot_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_sequence::Sequence::slot_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_sequence::Sequence::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, _slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_sequence::Sequence::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::array::vtable::operations::OperationsVTable for vortex_sequence::Sequence -pub fn vortex_sequence::Sequence::scalar_at(array: vortex_array::array::view::ArrayView<'_, vortex_sequence::Sequence>, index: usize, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_sequence::Sequence::scalar_at(vortex_array::array::view::ArrayView<'_, vortex_sequence::Sequence>, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::array::vtable::validity::ValidityVTable for vortex_sequence::Sequence -pub fn vortex_sequence::Sequence::validity(_array: vortex_array::array::view::ArrayView<'_, vortex_sequence::Sequence>) -> vortex_error::VortexResult +pub fn vortex_sequence::Sequence::validity(vortex_array::array::view::ArrayView<'_, vortex_sequence::Sequence>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::take::TakeExecute for vortex_sequence::Sequence -pub fn vortex_sequence::Sequence::take(array: vortex_array::array::view::ArrayView<'_, Self>, indices: &vortex_array::array::erased::ArrayRef, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_sequence::Sequence::take(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::kernel::FilterKernel for vortex_sequence::Sequence -pub fn vortex_sequence::Sequence::filter(array: vortex_array::array::view::ArrayView<'_, Self>, mask: &vortex_mask::Mask, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_sequence::Sequence::filter(vortex_array::array::view::ArrayView<'_, Self>, &vortex_mask::Mask, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_sequence::Sequence -pub fn vortex_sequence::Sequence::slice(array: vortex_array::array::view::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_sequence::Sequence::slice(vortex_array::array::view::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::binary::compare::CompareKernel for vortex_sequence::Sequence -pub fn vortex_sequence::Sequence::compare(lhs: vortex_array::array::view::ArrayView<'_, Self>, rhs: &vortex_array::array::erased::ArrayRef, operator: vortex_array::scalar_fn::fns::operators::CompareOperator, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_sequence::Sequence::compare(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, vortex_array::scalar_fn::fns::operators::CompareOperator, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::kernel::CastReduce for vortex_sequence::Sequence -pub fn vortex_sequence::Sequence::cast(array: vortex_array::array::view::ArrayView<'_, Self>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_sequence::Sequence::cast(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::list_contains::kernel::ListContainsElementReduce for vortex_sequence::Sequence -pub fn vortex_sequence::Sequence::list_contains(list: &vortex_array::array::erased::ArrayRef, element: vortex_array::array::view::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_sequence::Sequence::list_contains(&vortex_array::array::erased::ArrayRef, vortex_array::array::view::ArrayView<'_, Self>) -> vortex_error::VortexResult> pub struct vortex_sequence::SequenceData @@ -92,7 +90,7 @@ pub fn vortex_sequence::SequenceData::multiplier(&self) -> vortex_array::scalar: pub fn vortex_sequence::SequenceData::ptype(&self) -> vortex_array::dtype::ptype::PType -pub fn vortex_sequence::SequenceData::validate(base: vortex_array::scalar::typed_view::primitive::pvalue::PValue, multiplier: vortex_array::scalar::typed_view::primitive::pvalue::PValue, dtype: &vortex_array::dtype::DType, length: usize) -> vortex_error::VortexResult<()> +pub fn vortex_sequence::SequenceData::validate(vortex_array::scalar::typed_view::primitive::pvalue::PValue, vortex_array::scalar::typed_view::primitive::pvalue::PValue, &vortex_array::dtype::DType, usize) -> vortex_error::VortexResult<()> impl core::clone::Clone for vortex_sequence::SequenceData @@ -100,19 +98,19 @@ pub fn vortex_sequence::SequenceData::clone(&self) -> vortex_sequence::SequenceD impl core::fmt::Debug for vortex_sequence::SequenceData -pub fn vortex_sequence::SequenceData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_sequence::SequenceData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_sequence::SequenceData -pub fn vortex_sequence::SequenceData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_sequence::SequenceData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::hash::ArrayEq for vortex_sequence::SequenceData -pub fn vortex_sequence::SequenceData::array_eq(&self, other: &Self, _precision: vortex_array::hash::Precision) -> bool +pub fn vortex_sequence::SequenceData::array_eq(&self, &Self, vortex_array::hash::Precision) -> bool impl vortex_array::hash::ArrayHash for vortex_sequence::SequenceData -pub fn vortex_sequence::SequenceData::array_hash(&self, state: &mut H, _precision: vortex_array::hash::Precision) +pub fn vortex_sequence::SequenceData::array_hash(&self, &mut H, vortex_array::hash::Precision) pub struct vortex_sequence::SequenceDataParts @@ -122,8 +120,8 @@ pub vortex_sequence::SequenceDataParts::multiplier: vortex_array::scalar::typed_ pub vortex_sequence::SequenceDataParts::ptype: vortex_array::dtype::ptype::PType -pub fn vortex_sequence::initialize(session: &vortex_session::VortexSession) +pub fn vortex_sequence::initialize(&vortex_session::VortexSession) -pub fn vortex_sequence::sequence_encode(primitive_array: &vortex_array::arrays::primitive::vtable::PrimitiveArray) -> vortex_error::VortexResult> +pub fn vortex_sequence::sequence_encode(vortex_array::array::view::ArrayView<'_, vortex_array::arrays::primitive::vtable::Primitive>, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> pub type vortex_sequence::SequenceArray = vortex_array::array::typed::Array diff --git a/encodings/sequence/src/array.rs b/encodings/sequence/src/array.rs index f941d4cf1a6..4a85f3a4c0b 100644 --- a/encodings/sequence/src/array.rs +++ b/encodings/sequence/src/array.rs @@ -45,6 +45,7 @@ use vortex_error::vortex_ensure; use vortex_error::vortex_err; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::compress::sequence_decompress; use crate::kernel::PARENT_KERNELS; @@ -231,7 +232,8 @@ impl VTable for Sequence { type ValidityVTable = Self; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.sequence"); + *ID } fn validate( @@ -369,8 +371,6 @@ impl ValidityVTable for Sequence { pub struct Sequence; impl Sequence { - pub const ID: ArrayId = ArrayId::new_ref("vortex.sequence"); - fn stats(multiplier: PValue) -> StatsSet { // A sequence A[i] = base + i * multiplier is sorted iff multiplier >= 0, // and strictly sorted iff multiplier > 0. @@ -451,6 +451,8 @@ impl Sequence { #[cfg(test)] mod tests { + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; use vortex_array::dtype::Nullability; @@ -488,7 +490,7 @@ mod tests { fn test_sequence_scalar_at() { let scalar = Sequence::try_new_typed(2i64, 3, Nullability::NonNullable, 4) .unwrap() - .scalar_at(2) + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) .unwrap(); assert_eq!( diff --git a/encodings/sequence/src/compress.rs b/encodings/sequence/src/compress.rs index 6321aa3df78..9f5e62a5217 100644 --- a/encodings/sequence/src/compress.rs +++ b/encodings/sequence/src/compress.rs @@ -6,7 +6,10 @@ use std::ops::Add; use num_traits::CheckedAdd; use num_traits::CheckedSub; use vortex_array::ArrayRef; +use vortex_array::ArrayView; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; +use vortex_array::arrays::Primitive; use vortex_array::arrays::PrimitiveArray; use vortex_array::dtype::NativePType; use vortex_array::dtype::Nullability; @@ -84,13 +87,16 @@ pub fn sequence_decompress(array: &SequenceArray) -> VortexResult { /// 3. The array is representable as a sequence `A[i] = base + i * multiplier` for multiplier != 0. /// 4. The sequence has no deviations from the equation, this could be fixed with patches. However, /// we might want a different array for that since sequence provide fast access. -pub fn sequence_encode(primitive_array: &PrimitiveArray) -> VortexResult> { +pub fn sequence_encode( + primitive_array: ArrayView<'_, Primitive>, + ctx: &mut ExecutionCtx, +) -> VortexResult> { if primitive_array.is_empty() { // we cannot encode an empty array return Ok(None); } - if !primitive_array.all_valid()? { + if !primitive_array.array().all_valid(ctx)? { return Ok(None); } @@ -142,9 +148,10 @@ fn encode_primitive_array + CheckedAdd + CheckedSu #[cfg(test)] mod tests { - #[allow(unused_imports)] + #[expect(unused_imports)] use itertools::Itertools; - use vortex_array::ToCanonical; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; @@ -152,44 +159,58 @@ mod tests { #[test] fn test_encode_array_success() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let primitive_array = PrimitiveArray::from_iter([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]); - let encoded = sequence_encode(&primitive_array).unwrap(); + let encoded = sequence_encode(primitive_array.as_view(), &mut ctx).unwrap(); assert!(encoded.is_some()); - let decoded = encoded.unwrap().to_primitive(); + let decoded = encoded + .unwrap() + .execute::(&mut ctx) + .unwrap(); assert_arrays_eq!(decoded, primitive_array); } #[test] fn test_encode_array_1_success() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let primitive_array = PrimitiveArray::from_iter([0]); - let encoded = sequence_encode(&primitive_array).unwrap(); + let encoded = sequence_encode(primitive_array.as_view(), &mut ctx).unwrap(); assert!(encoded.is_some()); - let decoded = encoded.unwrap().to_primitive(); + let decoded = encoded + .unwrap() + .execute::(&mut ctx) + .unwrap(); assert_arrays_eq!(decoded, primitive_array); } #[test] fn test_encode_array_fail() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let primitive_array = PrimitiveArray::from_iter([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0]); - let encoded = sequence_encode(&primitive_array).unwrap(); + let encoded = sequence_encode(primitive_array.as_view(), &mut ctx).unwrap(); assert!(encoded.is_none()); } #[test] fn test_encode_array_fail_oob() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let primitive_array = PrimitiveArray::from_iter(vec![100i8; 1000]); - let encoded = sequence_encode(&primitive_array).unwrap(); + let encoded = sequence_encode(primitive_array.as_view(), &mut ctx).unwrap(); assert!(encoded.is_none()); } #[test] fn test_encode_all_u8_values() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let primitive_array = PrimitiveArray::from_iter(0u8..=255); - let encoded = sequence_encode(&primitive_array).unwrap(); + let encoded = sequence_encode(primitive_array.as_view(), &mut ctx).unwrap(); assert!(encoded.is_some()); - let decoded = encoded.unwrap().to_primitive(); + let decoded = encoded + .unwrap() + .execute::(&mut ctx) + .unwrap(); assert_arrays_eq!(decoded, primitive_array); } } diff --git a/encodings/sequence/src/compute/cast.rs b/encodings/sequence/src/compute/cast.rs index 1eb9ddf47d8..e6d64fdf7c5 100644 --- a/encodings/sequence/src/compute/cast.rs +++ b/encodings/sequence/src/compute/cast.rs @@ -87,9 +87,11 @@ impl CastReduce for Sequence { #[cfg(test)] mod tests { + use std::sync::LazyLock; + use rstest::rstest; use vortex_array::IntoArray; - use vortex_array::ToCanonical; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; use vortex_array::builtins::ArrayBuiltins; @@ -97,10 +99,15 @@ mod tests { use vortex_array::dtype::DType; use vortex_array::dtype::Nullability; use vortex_array::dtype::PType; + use vortex_array::session::ArraySession; + use vortex_session::VortexSession; use crate::Sequence; use crate::SequenceArray; + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); + #[test] fn test_cast_sequence_nullability() { let sequence = Sequence::try_new_typed(0u32, 1u32, Nullability::NonNullable, 4).unwrap(); @@ -118,6 +125,7 @@ mod tests { #[test] fn test_cast_sequence_u32_to_i64() { + let mut ctx = SESSION.create_execution_ctx(); let sequence = Sequence::try_new_typed(100u32, 10u32, Nullability::NonNullable, 4).unwrap(); let casted = sequence @@ -130,12 +138,13 @@ mod tests { ); // Verify the values - let decoded = casted.to_primitive(); + let decoded = casted.execute::(&mut ctx).unwrap(); assert_arrays_eq!(decoded, PrimitiveArray::from_iter([100i64, 110, 120, 130])); } #[test] fn test_cast_sequence_i16_to_i32_nullable() { + let mut ctx = SESSION.create_execution_ctx(); // Test ptype change AND nullability change in one cast let sequence = Sequence::try_new_typed(5i16, 3i16, Nullability::NonNullable, 3).unwrap(); @@ -149,7 +158,7 @@ mod tests { ); // Verify the values - let decoded = casted.to_primitive(); + let decoded = casted.execute::(&mut ctx).unwrap(); assert_arrays_eq!( decoded, PrimitiveArray::from_option_iter([Some(5i32), Some(8), Some(11)]) @@ -158,6 +167,7 @@ mod tests { #[test] fn test_cast_sequence_to_float_delegates_to_canonical() { + let mut ctx = SESSION.create_execution_ctx(); let sequence = Sequence::try_new_typed(0i32, 1i32, Nullability::NonNullable, 5).unwrap(); // Cast to float should delegate to canonical (SequenceArray doesn't support float) @@ -172,7 +182,7 @@ mod tests { ); // Verify the values were correctly converted - let decoded = casted.to_primitive(); + let decoded = casted.execute::(&mut ctx).unwrap(); assert_arrays_eq!( decoded, PrimitiveArray::from_iter([0.0f32, 1.0, 2.0, 3.0, 4.0]) diff --git a/encodings/sequence/src/compute/is_sorted.rs b/encodings/sequence/src/compute/is_sorted.rs index c930246508f..7df01aded17 100644 --- a/encodings/sequence/src/compute/is_sorted.rs +++ b/encodings/sequence/src/compute/is_sorted.rs @@ -27,7 +27,6 @@ impl DynAggregateKernel for SequenceIsSortedKernel { batch: &ArrayRef, ctx: &mut ExecutionCtx, ) -> VortexResult> { - let _ = ctx; let Some(options) = aggregate_fn.as_opt::() else { return Ok(None); }; @@ -47,6 +46,11 @@ impl DynAggregateKernel for SequenceIsSortedKernel { }) })?; - Ok(Some(IsSorted::make_partial(batch, result, options.strict)?)) + Ok(Some(IsSorted::make_partial( + batch, + result, + options.strict, + ctx, + )?)) } } diff --git a/encodings/sequence/src/compute/take.rs b/encodings/sequence/src/compute/take.rs index ab3a4ce5f97..6e1ebe263df 100644 --- a/encodings/sequence/src/compute/take.rs +++ b/encodings/sequence/src/compute/take.rs @@ -77,7 +77,7 @@ impl TakeExecute for Sequence { indices: &ArrayRef, ctx: &mut ExecutionCtx, ) -> VortexResult> { - let mask = indices.validity_mask()?; + let mask = indices.validity()?.execute_mask(indices.len(), ctx)?; let indices = indices.clone().execute::(ctx)?; let result_nullability = array.dtype().nullability() | indices.dtype().nullability(); diff --git a/encodings/sequence/src/lib.rs b/encodings/sequence/src/lib.rs index aec425e0bae..bd6ab2f508c 100644 --- a/encodings/sequence/src/lib.rs +++ b/encodings/sequence/src/lib.rs @@ -16,6 +16,7 @@ pub use array::SequenceArray; pub use array::SequenceData; pub use array::SequenceDataParts; pub use compress::sequence_encode; +use vortex_array::ArrayVTable; use vortex_array::aggregate_fn::AggregateFnVTable; use vortex_array::aggregate_fn::fns::is_sorted::IsSorted; use vortex_array::aggregate_fn::fns::min_max::MinMax; @@ -29,12 +30,12 @@ pub fn initialize(session: &VortexSession) { // Register the Sequence-specific aggregate kernels. session.aggregate_fns().register_aggregate_kernel( - Sequence::ID, + Sequence.id(), Some(MinMax.id()), &compute::min_max::SequenceMinMaxKernel, ); session.aggregate_fns().register_aggregate_kernel( - Sequence::ID, + Sequence.id(), Some(IsSorted.id()), &compute::is_sorted::SequenceIsSortedKernel, ); diff --git a/encodings/sparse/public-api.lock b/encodings/sparse/public-api.lock index 30a2e9a726c..c8e9abb533a 100644 --- a/encodings/sparse/public-api.lock +++ b/encodings/sparse/public-api.lock @@ -4,13 +4,11 @@ pub struct vortex_sparse::Sparse impl vortex_sparse::Sparse -pub const vortex_sparse::Sparse::ID: vortex_array::array::ArrayId +pub fn vortex_sparse::Sparse::encode(&vortex_array::array::erased::ArrayRef, core::option::Option, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_sparse::Sparse::encode(array: &vortex_array::array::erased::ArrayRef, fill_value: core::option::Option) -> vortex_error::VortexResult +pub fn vortex_sparse::Sparse::try_new(vortex_array::array::erased::ArrayRef, vortex_array::array::erased::ArrayRef, usize, vortex_array::scalar::Scalar) -> vortex_error::VortexResult -pub fn vortex_sparse::Sparse::try_new(indices: vortex_array::array::erased::ArrayRef, values: vortex_array::array::erased::ArrayRef, len: usize, fill_value: vortex_array::scalar::Scalar) -> vortex_error::VortexResult - -pub fn vortex_sparse::Sparse::try_new_from_patches(patches: vortex_array::patches::Patches, fill_value: vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn vortex_sparse::Sparse::try_new_from_patches(vortex_array::patches::Patches, vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::clone::Clone for vortex_sparse::Sparse @@ -18,7 +16,7 @@ pub fn vortex_sparse::Sparse::clone(&self) -> vortex_sparse::Sparse impl core::fmt::Debug for vortex_sparse::Sparse -pub fn vortex_sparse::Sparse::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_sparse::Sparse::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::array::vtable::VTable for vortex_sparse::Sparse @@ -28,55 +26,55 @@ pub type vortex_sparse::Sparse::OperationsVTable = vortex_sparse::Sparse pub type vortex_sparse::Sparse::ValidityVTable = vortex_sparse::Sparse -pub fn vortex_sparse::Sparse::buffer(array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_sparse::Sparse::buffer(vortex_array::array::view::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_sparse::Sparse::buffer_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_sparse::Sparse::buffer_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_sparse::Sparse::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_sparse::Sparse::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_sparse::Sparse::execute(array: vortex_array::array::typed::Array, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_sparse::Sparse::execute(vortex_array::array::typed::Array, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_sparse::Sparse::execute_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_sparse::Sparse::execute_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_sparse::Sparse::id(&self) -> vortex_array::array::ArrayId -pub fn vortex_sparse::Sparse::nbuffers(_array: vortex_array::array::view::ArrayView<'_, Self>) -> usize +pub fn vortex_sparse::Sparse::nbuffers(vortex_array::array::view::ArrayView<'_, Self>) -> usize -pub fn vortex_sparse::Sparse::reduce_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_sparse::Sparse::reduce_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_sparse::Sparse::serialize(array: vortex_array::array::view::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_sparse::Sparse::serialize(vortex_array::array::view::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_sparse::Sparse::slot_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_sparse::Sparse::slot_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_sparse::Sparse::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, _slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_sparse::Sparse::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::array::vtable::operations::OperationsVTable for vortex_sparse::Sparse -pub fn vortex_sparse::Sparse::scalar_at(array: vortex_array::array::view::ArrayView<'_, vortex_sparse::Sparse>, index: usize, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_sparse::Sparse::scalar_at(vortex_array::array::view::ArrayView<'_, vortex_sparse::Sparse>, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::array::vtable::validity::ValidityVTable for vortex_sparse::Sparse -pub fn vortex_sparse::Sparse::validity(array: vortex_array::array::view::ArrayView<'_, vortex_sparse::Sparse>) -> vortex_error::VortexResult +pub fn vortex_sparse::Sparse::validity(vortex_array::array::view::ArrayView<'_, vortex_sparse::Sparse>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::take::TakeExecute for vortex_sparse::Sparse -pub fn vortex_sparse::Sparse::take(array: vortex_array::array::view::ArrayView<'_, Self>, indices: &vortex_array::array::erased::ArrayRef, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_sparse::Sparse::take(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::kernel::FilterKernel for vortex_sparse::Sparse -pub fn vortex_sparse::Sparse::filter(array: vortex_array::array::view::ArrayView<'_, Self>, mask: &vortex_mask::Mask, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_sparse::Sparse::filter(vortex_array::array::view::ArrayView<'_, Self>, &vortex_mask::Mask, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceKernel for vortex_sparse::Sparse -pub fn vortex_sparse::Sparse::slice(array: vortex_array::array::view::ArrayView<'_, Self>, range: core::ops::range::Range, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_sparse::Sparse::slice(vortex_array::array::view::ArrayView<'_, Self>, core::ops::range::Range, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::kernel::CastReduce for vortex_sparse::Sparse -pub fn vortex_sparse::Sparse::cast(array: vortex_array::array::view::ArrayView<'_, Self>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_sparse::Sparse::cast(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::not::kernel::NotReduce for vortex_sparse::Sparse -pub fn vortex_sparse::Sparse::invert(array: vortex_array::array::view::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_sparse::Sparse::invert(vortex_array::array::view::ArrayView<'_, Self>) -> vortex_error::VortexResult> pub struct vortex_sparse::SparseData @@ -84,7 +82,7 @@ impl vortex_sparse::SparseData pub fn vortex_sparse::SparseData::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_sparse::SparseData::encode(array: &vortex_array::array::erased::ArrayRef, fill_value: core::option::Option) -> vortex_error::VortexResult +pub fn vortex_sparse::SparseData::encode(&vortex_array::array::erased::ArrayRef, core::option::Option, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_sparse::SparseData::fill_scalar(&self) -> &vortex_array::scalar::Scalar @@ -96,9 +94,9 @@ pub fn vortex_sparse::SparseData::patches(&self) -> &vortex_array::patches::Patc pub fn vortex_sparse::SparseData::resolved_patches(&self) -> vortex_error::VortexResult -pub fn vortex_sparse::SparseData::try_new_from_patches(patches: vortex_array::patches::Patches, fill_value: vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn vortex_sparse::SparseData::try_new_from_patches(vortex_array::patches::Patches, vortex_array::scalar::Scalar) -> vortex_error::VortexResult -pub fn vortex_sparse::SparseData::validate(patches: &vortex_array::patches::Patches, fill_value: &vortex_array::scalar::Scalar, dtype: &vortex_array::dtype::DType, len: usize) -> vortex_error::VortexResult<()> +pub fn vortex_sparse::SparseData::validate(&vortex_array::patches::Patches, &vortex_array::scalar::Scalar, &vortex_array::dtype::DType, usize) -> vortex_error::VortexResult<()> impl core::clone::Clone for vortex_sparse::SparseData @@ -106,19 +104,19 @@ pub fn vortex_sparse::SparseData::clone(&self) -> vortex_sparse::SparseData impl core::fmt::Debug for vortex_sparse::SparseData -pub fn vortex_sparse::SparseData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_sparse::SparseData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_sparse::SparseData -pub fn vortex_sparse::SparseData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_sparse::SparseData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::hash::ArrayEq for vortex_sparse::SparseData -pub fn vortex_sparse::SparseData::array_eq(&self, other: &Self, precision: vortex_array::hash::Precision) -> bool +pub fn vortex_sparse::SparseData::array_eq(&self, &Self, vortex_array::hash::Precision) -> bool impl vortex_array::hash::ArrayHash for vortex_sparse::SparseData -pub fn vortex_sparse::SparseData::array_hash(&self, state: &mut H, precision: vortex_array::hash::Precision) +pub fn vortex_sparse::SparseData::array_hash(&self, &mut H, vortex_array::hash::Precision) #[repr(C)] pub struct vortex_sparse::SparseMetadata @@ -132,7 +130,7 @@ pub fn vortex_sparse::SparseMetadata::default() -> Self impl core::fmt::Debug for vortex_sparse::SparseMetadata -pub fn vortex_sparse::SparseMetadata::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_sparse::SparseMetadata::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl prost::message::Message for vortex_sparse::SparseMetadata diff --git a/encodings/sparse/src/canonical.rs b/encodings/sparse/src/canonical.rs index c53fd8989e8..17455013720 100644 --- a/encodings/sparse/src/canonical.rs +++ b/encodings/sparse/src/canonical.rs @@ -52,7 +52,6 @@ use vortex_error::VortexError; use vortex_error::VortexExpect; use vortex_error::VortexResult; use vortex_error::vortex_bail; -use vortex_error::vortex_panic; use crate::ConstantArray; use crate::Sparse; @@ -83,7 +82,7 @@ pub(super) fn execute_sparse( DType::Struct(struct_fields, ..) => execute_sparse_struct( struct_fields, array.fill_scalar().as_struct(), - array.dtype(), + array.dtype().nullability(), array.patches(), array.len(), ctx, @@ -148,7 +147,10 @@ fn execute_sparse_lists( let n_filled = array.len() - resolved_patches.num_patches(); let total_canonical_values = values.elements().len() + fill_value.len() * n_filled; - let validity = Validity::from_mask(array.as_array().validity_mask()?, nullability); + let validity = { + let arr = array.as_array(); + Validity::from_mask(arr.validity()?.execute_mask(arr.len(), ctx)?, nullability) + }; Ok(match_each_integer_ptype!(indices.ptype(), |I| { match_smallest_offset_type!(total_canonical_values, |O| { @@ -160,11 +162,13 @@ fn execute_sparse_lists( array.len(), total_canonical_values, validity, + ctx, ) }) })) } +#[expect(clippy::too_many_arguments)] fn execute_sparse_lists_inner( patch_indices: &[I], patch_values: ListViewArray, @@ -173,6 +177,7 @@ fn execute_sparse_lists_inner( len: usize, total_canonical_values: usize, validity: Validity, + ctx: &mut ExecutionCtx, ) -> ArrayRef { // Create the builder with appropriate types. It is easy to just use the same type for both // `offsets` and `sizes` since we have no other constraints. @@ -199,7 +204,7 @@ fn execute_sparse_lists_inner( builder .append_value( patch_values - .scalar_at(patch_idx) + .execute_scalar(patch_idx, ctx) .vortex_expect("scalar_at") .as_list(), ) @@ -208,7 +213,7 @@ fn execute_sparse_lists_inner( } else { // Set with the fill value. builder - .append_value(fill_value.clone()) + .append_value(fill_value) .vortex_expect("Failed to append fill value"); } } @@ -233,7 +238,10 @@ fn execute_sparse_fixed_size_list( .execute::(ctx)?; let fill_value = array.fill_scalar().as_list(); - let validity = Validity::from_mask(array.as_array().validity_mask()?, nullability); + let validity = { + let arr = array.as_array(); + Validity::from_mask(arr.validity()?.execute_mask(arr.len(), ctx)?, nullability) + }; Ok(match_each_integer_ptype!(indices.ptype(), |I| { execute_sparse_fixed_size_list_inner::( @@ -242,6 +250,7 @@ fn execute_sparse_fixed_size_list( fill_value, array.len(), validity, + ctx, ) .into_array() })) @@ -259,6 +268,7 @@ fn execute_sparse_fixed_size_list_inner( fill_value: ListScalar, array_len: usize, validity: Validity, + ctx: &mut ExecutionCtx, ) -> FixedSizeListArray { let list_size = values.list_size(); let element_dtype = values.elements().dtype(); @@ -292,7 +302,7 @@ fn execute_sparse_fixed_size_list_inner( .vortex_expect("fixed_size_list_elements_at"); for i in 0..list_size as usize { builder - .append_scalar(&patch_list.scalar_at(i).vortex_expect("scalar_at")) + .append_scalar(&patch_list.execute_scalar(i, ctx).vortex_expect("scalar_at")) .vortex_expect("element dtype must match"); } } else { @@ -392,7 +402,7 @@ fn execute_sparse_primitives TryFrom<&'a Scalar, Error fn execute_sparse_struct( struct_fields: &StructFields, fill_struct: StructScalar, - dtype: &DType, + nullability: Nullability, // Resolution is unnecessary b/c we're just pushing the patches into the fields. unresolved_patches: &Patches, len: usize, @@ -414,25 +424,20 @@ fn execute_sparse_struct( .execute::(ctx)?; let columns_patch_values = patch_values_as_struct.unmasked_fields(); let names = patch_values_as_struct.names(); - let validity = if dtype.is_nullable() { - top_level_fill_validity.patch( - len, - unresolved_patches.offset(), - unresolved_patches.indices(), - &Validity::from_mask( - unresolved_patches - .values() - .validity_mask() - .vortex_expect("validity_mask"), - Nullability::Nullable, - ), - ctx, - )? - } else { - top_level_fill_validity - .into_non_nullable(len) - .unwrap_or_else(|| vortex_panic!("fill validity should match sparse array nullability")) - }; + let validity = top_level_fill_validity.patch( + len, + unresolved_patches.offset(), + unresolved_patches.indices(), + &Validity::from_mask( + patch_values_as_struct + .validity() + .vortex_expect("validity_mask") + .execute_mask(patch_values_as_struct.len(), ctx) + .vortex_expect("Failed to compute validity mask"), + nullability, + ), + ctx, + )?; Ok(StructArray::try_from_iter_with_validity( names.iter().zip_eq( @@ -490,7 +495,13 @@ fn execute_varbin( let patches = array.resolved_patches()?; let indices = patches.indices().clone().execute::(ctx)?; let values = patches.values().clone().execute::(ctx)?; - let validity = Validity::from_mask(array.as_array().validity_mask()?, dtype.nullability()); + let validity = { + let arr = array.as_array(); + Validity::from_mask( + arr.validity()?.execute_mask(arr.len(), ctx)?, + dtype.nullability(), + ) + }; let len = array.len(); Ok(match_each_integer_ptype!(indices.ptype(), |I| { @@ -542,8 +553,10 @@ mod test { use std::sync::Arc; use rstest::rstest; + use vortex_array::Canonical; use vortex_array::IntoArray; - use vortex_array::ToCanonical; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; use vortex_array::arrays::DecimalArray; use vortex_array::arrays::FixedSizeListArray; @@ -554,7 +567,7 @@ mod test { use vortex_array::arrays::VarBinArray; use vortex_array::arrays::VarBinViewArray; use vortex_array::arrays::listview::ListViewArrayExt; - use vortex_array::arrow::IntoArrowArray as _; + use vortex_array::arrow::ArrowArrayExecutor; use vortex_array::assert_arrays_eq; use vortex_array::dtype::DType; use vortex_array::dtype::DecimalDType; @@ -580,10 +593,15 @@ mod test { #[case(Some(false))] #[case(None)] fn test_sparse_bool(#[case] fill_value: Option) { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let indices = buffer![0u64, 1, 7].into_array(); let values = BoolArray::from_iter([Some(true), None, Some(false)]).into_array(); let sparse_bools = Sparse::try_new(indices, values, 10, Scalar::from(fill_value)).unwrap(); - let actual = sparse_bools.as_array().to_bool(); + let actual = sparse_bools + .as_array() + .clone() + .execute::(&mut ctx) + .unwrap(); let expected = BoolArray::from_iter([ Some(true), @@ -606,12 +624,17 @@ mod test { #[case(Some(-1i32))] #[case(None)] fn test_sparse_primitive(#[case] fill_value: Option) { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let indices = buffer![0u64, 1, 7].into_array(); let values = PrimitiveArray::from_option_iter([Some(0i32), None, Some(1)]).into_array(); let sparse_ints = Sparse::try_new(indices, values, 10, Scalar::from(fill_value)).unwrap(); assert_eq!(*sparse_ints.dtype(), DType::Primitive(PType::I32, Nullable)); - let flat_ints = sparse_ints.as_array().to_primitive(); + let flat_ints = sparse_ints + .as_array() + .clone() + .execute::(&mut ctx) + .unwrap(); let expected = PrimitiveArray::from_option_iter([ Some(0i32), None, @@ -630,6 +653,7 @@ mod test { #[test] fn test_sparse_struct_valid_fill() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let field_names = FieldNames::from_iter(["a", "b"]); let field_types = vec![ DType::Primitive(PType::I32, Nullable), @@ -694,12 +718,17 @@ mod test { .unwrap() .into_array(); - let actual = sparse_struct.as_array().to_struct(); + let actual = sparse_struct + .as_array() + .clone() + .execute::(&mut ctx) + .unwrap(); assert_arrays_eq!(actual, expected); } #[test] fn test_sparse_struct_invalid_fill() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let field_names = FieldNames::from_iter(["a", "b"]); let field_types = vec![ DType::Primitive(PType::I32, Nullable), @@ -761,12 +790,17 @@ mod test { .unwrap() .into_array(); - let actual = sparse_struct.as_array().to_struct(); + let actual = sparse_struct + .as_array() + .clone() + .execute::(&mut ctx) + .unwrap(); assert_arrays_eq!(actual, expected); } #[test] fn test_sparse_decimal() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let indices = buffer![0u32, 1u32, 7u32, 8u32].into_array(); let decimal_dtype = DecimalDType::new(3, 2); let patch_values = DecimalArray::new( @@ -786,14 +820,16 @@ mod test { Validity::from_mask(Mask::from_excluded_indices(10, vec![8]), Nullable), ) .into_array() - .into_arrow_preferred() + .execute_arrow(None, &mut ctx) .unwrap(); let actual = sparse_struct .as_array() - .to_decimal() + .clone() + .execute::(&mut ctx) + .unwrap() .into_array() - .into_arrow_preferred() + .execute_arrow(None, &mut ctx) .unwrap(); assert_eq!(expected.data_type(), actual.data_type()); @@ -802,6 +838,7 @@ mod test { #[test] fn test_sparse_utf8_varbinview_non_null_fill() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let strings = >::from_iter([ Some("hello"), Some("goodbye"), @@ -821,7 +858,12 @@ mod test { ) .unwrap(); - let actual = array.as_array().to_varbinview().into_array(); + let actual = array + .as_array() + .clone() + .execute::(&mut ctx) + .unwrap() + .into_array(); let expected = >::from_iter([ Some("hello"), Some("123"), @@ -843,6 +885,7 @@ mod test { #[test] fn test_sparse_utf8_varbinview_null_fill() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let strings = >::from_iter([ Some("hello"), Some("goodbye"), @@ -862,7 +905,12 @@ mod test { ) .unwrap(); - let actual = array.as_array().to_varbinview().into_array(); + let actual = array + .as_array() + .clone() + .execute::(&mut ctx) + .unwrap() + .into_array(); let expected = >::from_iter([ Some("hello"), None, @@ -884,6 +932,7 @@ mod test { #[test] fn test_sparse_utf8_varbinview_non_nullable() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let strings = VarBinViewArray::from_iter_str(["hello", "goodbye", "hello", "bonjour", "你好"]) .into_array(); @@ -896,7 +945,12 @@ mod test { ) .unwrap(); - let actual = array.as_array().to_varbinview().into_array(); + let actual = array + .as_array() + .clone() + .execute::(&mut ctx) + .unwrap() + .into_array(); let expected = VarBinViewArray::from_iter_str([ "hello", "123", "123", "goodbye", "hello", "bonjour", "123", "123", "你好", ]) @@ -907,6 +961,7 @@ mod test { #[test] fn test_sparse_utf8_varbin_null_fill() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let strings = >::from_iter([ Some("hello"), Some("goodbye"), @@ -926,7 +981,12 @@ mod test { ) .unwrap(); - let actual = array.as_array().to_varbinview().into_array(); + let actual = array + .as_array() + .clone() + .execute::(&mut ctx) + .unwrap() + .into_array(); let expected = >::from_iter([ Some("hello"), None, @@ -948,6 +1008,7 @@ mod test { #[test] fn test_sparse_binary_varbinview_non_null_fill() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let binaries = VarBinViewArray::from_iter_nullable_bin([ Some(b"hello" as &[u8]), Some(b"goodbye"), @@ -967,7 +1028,12 @@ mod test { ) .unwrap(); - let actual = array.as_array().to_varbinview().into_array(); + let actual = array + .as_array() + .clone() + .execute::(&mut ctx) + .unwrap() + .into_array(); let expected = VarBinViewArray::from_iter_nullable_bin([ Some(b"hello" as &[u8]), Some(b"123"), @@ -989,6 +1055,7 @@ mod test { #[test] fn test_sparse_list_null_fill() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Use ListViewArray consistently let elements = buffer![1i32, 2, 1, 2].into_array(); // Create ListView with offsets and sizes @@ -1006,12 +1073,10 @@ mod test { let indices = buffer![0u8, 3u8, 4u8, 5u8].into_array(); let fill_value = Scalar::null(lists.dtype().clone()); - let sparse = Sparse::try_new(indices, lists, 6, fill_value) - .unwrap() - .into_array(); + let sparse = Sparse::try_new(indices, lists, 6, fill_value)?.into_array(); - let actual = sparse.to_canonical()?.into_array(); - let result_listview = actual.to_listview(); + let actual = sparse.execute::(&mut ctx)?.into_array(); + let result_listview = actual.execute::(&mut ctx)?; // Check the structure assert_eq!(result_listview.len(), 6); @@ -1025,7 +1090,10 @@ mod test { assert_eq!(result_listview.size_at(5), 1); // [2] // Verify actual values - let elements_array = result_listview.elements().to_primitive(); + let elements_array = result_listview + .elements() + .clone() + .execute::(&mut ctx)?; let elements_slice = elements_array.as_slice::(); let list0_offset = result_listview.offset_at(0); @@ -1045,6 +1113,7 @@ mod test { #[test] fn test_sparse_list_null_fill_sliced_sparse_values() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Create ListViewArray with 8 elements forming 8 single-element lists let elements = buffer![1i32, 2, 1, 2, 1, 2, 1, 2].into_array(); let offsets = buffer![0u32, 1, 2, 3, 4, 5, 6, 7].into_array(); @@ -1064,8 +1133,13 @@ mod test { .unwrap() .into_array(); - let actual = sparse.to_canonical().vortex_expect("no fail").into_array(); - let result_listview = actual.to_listview(); + let actual = sparse + .execute::(&mut ctx) + .vortex_expect("no fail") + .into_array(); + let result_listview = actual + .execute::(&mut ctx) + .vortex_expect("no fail"); // Check the structure assert_eq!(result_listview.len(), 6); @@ -1079,7 +1153,11 @@ mod test { assert_eq!(result_listview.size_at(5), 1); // [2] - extra element beyond original slice // Verify actual values - let elements_array = result_listview.elements().to_primitive(); + let elements_array = result_listview + .elements() + .clone() + .execute::(&mut ctx) + .unwrap(); let elements_slice = elements_array.as_slice::(); let list0_offset = result_listview.offset_at(0); @@ -1091,6 +1169,7 @@ mod test { #[test] fn test_sparse_list_non_null_fill() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Create ListViewArray with 4 single-element lists let elements = buffer![1i32, 2, 1, 2].into_array(); let offsets = buffer![0u32, 1, 2, 3].into_array(); @@ -1103,12 +1182,10 @@ mod test { let indices = buffer![0u8, 3u8, 4u8, 5u8].into_array(); let fill_value = Scalar::from(Some(vec![5i32, 6, 7, 8])); - let sparse = Sparse::try_new(indices, lists, 6, fill_value) - .unwrap() - .into_array(); + let sparse = Sparse::try_new(indices, lists, 6, fill_value)?.into_array(); - let actual = sparse.to_canonical()?.into_array(); - let result_listview = actual.to_listview(); + let actual = sparse.execute::(&mut ctx)?.into_array(); + let result_listview = actual.execute::(&mut ctx)?; // Check the structure assert_eq!(result_listview.len(), 6); @@ -1122,45 +1199,49 @@ mod test { assert_eq!(result_listview.size_at(5), 1); // [2] from sparse // Verify actual values - let elements_array = result_listview.elements().to_primitive(); + let elements_array = result_listview + .elements() + .clone() + .execute::(&mut ctx)?; let elements_slice = elements_array.as_slice::(); // List 0: [1] - let list0_offset = result_listview.offset_at(0) as usize; + let list0_offset = result_listview.offset_at(0); assert_eq!(elements_slice[list0_offset], 1); // List 1: [5,6,7,8] - let list1_offset = result_listview.offset_at(1) as usize; - let list1_size = result_listview.size_at(1) as usize; + let list1_offset = result_listview.offset_at(1); + let list1_size = result_listview.size_at(1); assert_eq!( &elements_slice[list1_offset..list1_offset + list1_size], &[5, 6, 7, 8] ); // List 2: [5,6,7,8] - let list2_offset = result_listview.offset_at(2) as usize; - let list2_size = result_listview.size_at(2) as usize; + let list2_offset = result_listview.offset_at(2); + let list2_size = result_listview.size_at(2); assert_eq!( &elements_slice[list2_offset..list2_offset + list2_size], &[5, 6, 7, 8] ); // List 3: [2] - let list3_offset = result_listview.offset_at(3) as usize; + let list3_offset = result_listview.offset_at(3); assert_eq!(elements_slice[list3_offset], 2); // List 4: [1] - let list4_offset = result_listview.offset_at(4) as usize; + let list4_offset = result_listview.offset_at(4); assert_eq!(elements_slice[list4_offset], 1); // List 5: [2] - let list5_offset = result_listview.offset_at(5) as usize; + let list5_offset = result_listview.offset_at(5); assert_eq!(elements_slice[list5_offset], 2); Ok(()) } #[test] fn test_sparse_binary_varbin_null_fill() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let strings = >::from_iter([ Some(b"hello" as &[u8]), Some(b"goodbye"), @@ -1180,7 +1261,12 @@ mod test { ) .unwrap(); - let actual = array.as_array().to_varbinview().into_array(); + let actual = array + .as_array() + .clone() + .execute::(&mut ctx) + .unwrap() + .into_array(); let expected = VarBinViewArray::from_iter_nullable_bin([ Some(b"hello" as &[u8]), None, @@ -1204,9 +1290,7 @@ mod test { fn test_sparse_fixed_size_list_null_fill() -> VortexResult<()> { // Create a FixedSizeListArray with 3 lists of size 3. let elements = buffer![1i32, 2, 3, 4, 5, 6, 7, 8, 9].into_array(); - let fsl = FixedSizeListArray::try_new(elements, 3, Validity::AllValid, 3) - .unwrap() - .into_array(); + let fsl = FixedSizeListArray::try_new(elements, 3, Validity::AllValid, 3)?.into_array(); let indices = buffer![0u8, 2u8, 3u8].into_array(); let fill_value = Scalar::null(DType::FixedSizeList( @@ -1214,11 +1298,11 @@ mod test { 3, Nullable, )); - let sparse = Sparse::try_new(indices, fsl, 5, fill_value) - .unwrap() - .into_array(); + let sparse = Sparse::try_new(indices, fsl, 5, fill_value)?.into_array(); - let actual = sparse.to_canonical()?.into_array(); + let actual = sparse + .execute::(&mut LEGACY_SESSION.create_execution_ctx())? + .into_array(); // Expected: [1,2,3], null, [4,5,6], [7,8,9], null. let expected_elements = @@ -1228,8 +1312,7 @@ mod test { 3, Validity::Array(BoolArray::from_iter([true, false, true, true, false]).into_array()), 5, - ) - .unwrap() + )? .into_array(); assert_arrays_eq!(actual, expected); @@ -1239,9 +1322,7 @@ mod test { #[test] fn test_sparse_fixed_size_list_non_null_fill() -> VortexResult<()> { let elements = buffer![1i32, 2, 3, 4, 5, 6].into_array(); - let fsl = FixedSizeListArray::try_new(elements, 2, Validity::AllValid, 3) - .unwrap() - .into_array(); + let fsl = FixedSizeListArray::try_new(elements, 2, Validity::AllValid, 3)?.into_array(); let indices = buffer![0u8, 2u8, 4u8].into_array(); let fill_value = Scalar::fixed_size_list( @@ -1252,16 +1333,15 @@ mod test { ], NonNullable, ); - let sparse = Sparse::try_new(indices, fsl, 6, fill_value) - .unwrap() - .into_array(); + let sparse = Sparse::try_new(indices, fsl, 6, fill_value)?.into_array(); - let actual = sparse.to_canonical()?.into_array(); + let actual = sparse + .execute::(&mut LEGACY_SESSION.create_execution_ctx())? + .into_array(); // Expected: [1,2], [99,88], [3,4], [99,88], [5,6], [99,88]. let expected_elements = buffer![1i32, 2, 99, 88, 3, 4, 99, 88, 5, 6, 99, 88].into_array(); - let expected = FixedSizeListArray::try_new(expected_elements, 2, Validity::NonNullable, 6) - .unwrap() + let expected = FixedSizeListArray::try_new(expected_elements, 2, Validity::NonNullable, 6)? .into_array(); assert_arrays_eq!(actual, expected); @@ -1277,8 +1357,7 @@ mod test { 2, Validity::Array(BoolArray::from_iter([true, false, true]).into_array()), 3, - ) - .unwrap() + )? .into_array(); let indices = buffer![1u16, 3u16, 4u16].into_array(); @@ -1290,11 +1369,11 @@ mod test { ], Nullable, ); - let sparse = Sparse::try_new(indices, fsl, 6, fill_value) - .unwrap() - .into_array(); + let sparse = Sparse::try_new(indices, fsl, 6, fill_value)?.into_array(); - let actual = sparse.to_canonical()?.into_array(); + let actual = sparse + .execute::(&mut LEGACY_SESSION.create_execution_ctx())? + .into_array(); // Expected validity: [true, true, true, false, true, true]. // Expected elements: [7,8], [10,20], [7,8], [30,40], [50,60], [7,8]. @@ -1306,8 +1385,7 @@ mod test { BoolArray::from_iter([true, true, true, false, true, true]).into_array(), ), 6, - ) - .unwrap() + )? .into_array(); assert_arrays_eq!(actual, expected); @@ -1321,9 +1399,7 @@ mod test { // Create patch values: only 3 distinct lists out of 100 total positions. let elements = buffer![10i32, 11, 20, 21, 30, 31].into_array(); - let fsl = FixedSizeListArray::try_new(elements, 2, Validity::AllValid, 3) - .unwrap() - .into_array(); + let fsl = FixedSizeListArray::try_new(elements, 2, Validity::AllValid, 3)?.into_array(); // Patches at positions 5, 50, and 95 out of 100. let indices = buffer![5u32, 50, 95].into_array(); @@ -1338,11 +1414,11 @@ mod test { NonNullable, ); - let sparse = Sparse::try_new(indices, fsl, 100, fill_value) - .unwrap() - .into_array(); + let sparse = Sparse::try_new(indices, fsl, 100, fill_value)?.into_array(); - let actual = sparse.to_canonical()?.into_array(); + let actual = sparse + .execute::(&mut LEGACY_SESSION.create_execution_ctx())? + .into_array(); // Build expected: 97 copies of [99,99] with patches at positions 5, 50, 95. let mut expected_elements_vec = Vec::with_capacity(200); @@ -1370,8 +1446,7 @@ mod test { } let expected_elements = PrimitiveArray::from_iter(expected_elements_vec).into_array(); let expected = - FixedSizeListArray::try_new(expected_elements, 2, Validity::NonNullable, 100) - .unwrap() + FixedSizeListArray::try_new(expected_elements, 2, Validity::NonNullable, 100)? .into_array(); assert_arrays_eq!(actual, expected); @@ -1382,9 +1457,7 @@ mod test { fn test_sparse_fixed_size_list_single_element() -> VortexResult<()> { // Test with a single element FSL array. let elements = buffer![42i32, 43].into_array(); - let fsl = FixedSizeListArray::try_new(elements, 2, Validity::AllValid, 1) - .unwrap() - .into_array(); + let fsl = FixedSizeListArray::try_new(elements, 2, Validity::AllValid, 1)?.into_array(); let indices = buffer![0u32].into_array(); let fill_value = Scalar::fixed_size_list( @@ -1395,16 +1468,15 @@ mod test { ], NonNullable, ); - let sparse = Sparse::try_new(indices, fsl, 1, fill_value) - .unwrap() - .into_array(); + let sparse = Sparse::try_new(indices, fsl, 1, fill_value)?.into_array(); - let actual = sparse.to_canonical()?.into_array(); + let actual = sparse + .execute::(&mut LEGACY_SESSION.create_execution_ctx())? + .into_array(); // Expected: just [42, 43]. let expected_elements = buffer![42i32, 43].into_array(); - let expected = FixedSizeListArray::try_new(expected_elements, 2, Validity::NonNullable, 1) - .unwrap() + let expected = FixedSizeListArray::try_new(expected_elements, 2, Validity::NonNullable, 1)? .into_array(); assert_arrays_eq!(actual, expected); @@ -1413,39 +1485,36 @@ mod test { #[test] fn test_sparse_list_grows_offset_type() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let elements = buffer![1i32, 2, 1, 2].into_array(); let offsets = buffer![0u8, 1, 2, 3, 4].into_array(); - let lists = ListArray::try_new(elements, offsets, Validity::AllValid) - .unwrap() - .into_array(); + let lists = ListArray::try_new(elements, offsets, Validity::AllValid)?.into_array(); let indices = buffer![0u8, 1u8, 2u8, 3u8].into_array(); let fill_value = Scalar::from(Some(vec![42i32; 252])); // 252 + 4 elements = 256 > u8::MAX - let sparse = Sparse::try_new(indices, lists, 5, fill_value) - .unwrap() - .into_array(); + let sparse = Sparse::try_new(indices, lists, 5, fill_value)?.into_array(); - let actual = sparse.to_canonical()?.into_array(); + let actual = sparse.execute::(&mut ctx)?.into_array(); let mut expected_elements = buffer_mut![1, 2, 1, 2]; expected_elements.extend(buffer![42i32; 252]); let expected = ListArray::try_new( expected_elements.freeze().into_array(), buffer![0u16, 1, 2, 3, 4, 256].into_array(), Validity::AllValid, - ) - .unwrap() + )? .into_array(); + let actual_listview = actual.clone().execute::(&mut ctx)?; assert_eq!( - actual.to_listview().offsets().dtype(), + actual_listview.offsets().dtype(), &DType::Primitive(PType::U16, NonNullable) ); assert_arrays_eq!(&actual, &expected); // Note that the preferred arrow list representation is `List` (not `ListView`). - let arrow_dtype = expected.dtype().to_arrow_dtype().unwrap(); - let actual = actual.into_arrow(&arrow_dtype).unwrap(); - let expected = expected.into_arrow(&arrow_dtype).unwrap(); + let arrow_dtype = expected.dtype().to_arrow_dtype()?; + let actual = actual.execute_arrow(Some(&arrow_dtype), &mut ctx)?; + let expected = expected.execute_arrow(Some(&arrow_dtype), &mut ctx)?; assert_eq!(actual.data_type(), expected.data_type()); Ok(()) @@ -1453,6 +1522,7 @@ mod test { #[test] fn test_sparse_listview_null_fill_with_gaps() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // This test specifically catches the bug where the old implementation // incorrectly tracked `last_valid_offset` as the START of the last list // instead of properly handling ListView's offset/size pairs. @@ -1488,12 +1558,18 @@ mod test { list_view.into_array(), 10, Scalar::null(list_dtype), - ) - .unwrap(); + )?; // Convert to canonical form - this triggers the function we're testing - let canonical = sparse.to_canonical()?.into_array(); - let result_listview = canonical.to_listview(); + let canonical = sparse + .into_array() + .execute::(&mut ctx)? + .into_array(); + let result_listview = canonical.execute::(&mut ctx)?; + let elements_primitive = result_listview + .elements() + .clone() + .execute::(&mut ctx)?; // Verify the structure assert_eq!(result_listview.len(), 10); @@ -1505,8 +1581,7 @@ mod test { if size == 0 { vec![] // null/empty list } else { - let elements = result_listview.elements().to_primitive(); - let slice = elements.as_slice::(); + let slice = elements_primitive.as_slice::(); slice[offset..offset + size].to_vec() } }; @@ -1529,6 +1604,7 @@ mod test { #[test] fn test_sparse_listview_sliced_values_null_fill() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // This test uses sliced ListView values to ensure proper handling // of non-zero starting offsets in the source data. @@ -1556,7 +1632,7 @@ mod test { // - Index 0: [2, 3, 4] (original list 1) // - Index 1: [5] (original list 2) // - Index 2: [6, 7] (original list 3) - let sliced = full_listview.slice(1..4).unwrap(); + let sliced = full_listview.slice(1..4)?; // Create sparse array with indices [0, 1] and length 5 // Expected result: @@ -1567,12 +1643,18 @@ mod test { // - Index 4: null let indices = buffer![0u8, 1].into_array(); // Extract only the values we need from the sliced array - let values = sliced.slice(0..2).unwrap(); - let sparse = - Sparse::try_new(indices, values, 5, Scalar::null(sliced.dtype().clone())).unwrap(); + let values = sliced.slice(0..2)?; + let sparse = Sparse::try_new(indices, values, 5, Scalar::null(sliced.dtype().clone()))?; - let canonical = sparse.to_canonical()?.into_array(); - let result_listview = canonical.to_listview(); + let canonical = sparse + .into_array() + .execute::(&mut ctx)? + .into_array(); + let result_listview = canonical.execute::(&mut ctx)?; + let elements_primitive = result_listview + .elements() + .clone() + .execute::(&mut ctx)?; assert_eq!(result_listview.len(), 5); @@ -1583,8 +1665,7 @@ mod test { if size == 0 { vec![] // null/empty list } else { - let elements = result_listview.elements().to_primitive(); - let slice = elements.as_slice::(); + let slice = elements_primitive.as_slice::(); slice[offset..offset + size].to_vec() } }; diff --git a/encodings/sparse/src/compute/cast.rs b/encodings/sparse/src/compute/cast.rs index a9fe0046646..663539a6030 100644 --- a/encodings/sparse/src/compute/cast.rs +++ b/encodings/sparse/src/compute/cast.rs @@ -11,6 +11,7 @@ use vortex_array::scalar_fn::fns::cast::CastReduce; use vortex_error::VortexResult; use crate::Sparse; + impl CastReduce for Sparse { fn cast(array: ArrayView<'_, Self>, dtype: &DType) -> VortexResult> { let casted_patches = array @@ -34,9 +35,11 @@ impl CastReduce for Sparse { #[cfg(test)] mod tests { + use std::sync::LazyLock; + use rstest::rstest; use vortex_array::IntoArray; - use vortex_array::ToCanonical; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; use vortex_array::builtins::ArrayBuiltins; @@ -45,13 +48,19 @@ mod tests { use vortex_array::dtype::Nullability; use vortex_array::dtype::PType; use vortex_array::scalar::Scalar; + use vortex_array::session::ArraySession; use vortex_buffer::buffer; + use vortex_session::VortexSession; use crate::Sparse; use crate::SparseArray; + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); + #[test] fn test_cast_sparse_i32_to_i64() { + let mut ctx = SESSION.create_execution_ctx(); let sparse = Sparse::try_new( buffer![2u64, 5, 8].into_array(), buffer![100i32, 200, 300].into_array(), @@ -70,7 +79,8 @@ mod tests { ); let expected = PrimitiveArray::from_iter([0i64, 0, 100, 0, 0, 200, 0, 0, 300, 0]); - assert_arrays_eq!(casted.to_primitive(), expected); + let casted_primitive = casted.execute::(&mut ctx).unwrap(); + assert_arrays_eq!(casted_primitive, expected); } #[test] @@ -124,6 +134,7 @@ mod tests { #[test] fn test_cast_sparse_null_fill_all_patched_to_non_nullable() -> vortex_error::VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); // Regression test for https://github.com/vortex-data/vortex/issues/6932 // // When all positions are patched the null fill is unused, so a cast to @@ -146,7 +157,8 @@ mod tests { ); let expected = PrimitiveArray::from_iter([10u64, 20, 30, 40, 50]); - assert_arrays_eq!(casted.to_primitive(), expected); + let casted_primitive = casted.execute::(&mut ctx)?; + assert_arrays_eq!(casted_primitive, expected); Ok(()) } diff --git a/encodings/sparse/src/lib.rs b/encodings/sparse/src/lib.rs index 87fd640fd1c..411bc9c7543 100644 --- a/encodings/sparse/src/lib.rs +++ b/encodings/sparse/src/lib.rs @@ -16,12 +16,14 @@ use vortex_array::ArrayId; use vortex_array::ArrayParts; use vortex_array::ArrayRef; use vortex_array::ArrayView; +use vortex_array::Canonical; use vortex_array::ExecutionCtx; use vortex_array::ExecutionResult; use vortex_array::IntoArray; use vortex_array::Precision; -use vortex_array::ToCanonical; +use vortex_array::arrays::BoolArray; use vortex_array::arrays::ConstantArray; +use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::bool::BoolArrayExt; use vortex_array::buffer::BufferHandle; use vortex_array::builtins::ArrayBuiltins; @@ -47,6 +49,7 @@ use vortex_error::vortex_panic; use vortex_mask::AllOr; use vortex_mask::Mask; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::canonical::execute_sparse; use crate::rules::RULES; @@ -88,7 +91,8 @@ impl VTable for Sparse { type ValidityVTable = Self; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.sparse"); + *ID } fn validate( @@ -227,8 +231,6 @@ impl Display for SparseData { pub struct Sparse; impl Sparse { - pub const ID: ArrayId = ArrayId::new_ref("vortex.sparse"); - /// Construct a new [`SparseArray`] from indices, values, length, and fill value. pub fn try_new( indices: ArrayRef, @@ -266,8 +268,12 @@ impl Sparse { } /// Encode the given array as a [`SparseArray`]. - pub fn encode(array: &ArrayRef, fill_value: Option) -> VortexResult { - SparseData::encode(array, fill_value) + pub fn encode( + array: &ArrayRef, + fill_value: Option, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + SparseData::encode(array, fill_value, ctx) } } @@ -392,7 +398,11 @@ impl SparseData { /// Encode given array as a SparseArray. /// /// Optionally provided fill value will be respected if the array is less than 90% null. - pub fn encode(array: &ArrayRef, fill_value: Option) -> VortexResult { + pub fn encode( + array: &ArrayRef, + fill_value: Option, + ctx: &mut ExecutionCtx, + ) -> VortexResult { if let Some(fill_value) = fill_value.as_ref() && !array.dtype().eq_ignore_nullability(fill_value.dtype()) { @@ -402,7 +412,7 @@ impl SparseData { fill_value.dtype() ) } - let mask = array.validity_mask()?; + let mask = array.validity()?.execute_mask(array.len(), ctx)?; if mask.all_false() { // Array is constant NULL @@ -411,8 +421,10 @@ impl SparseData { ); } else if mask.false_count() as f64 > (0.9 * mask.len() as f64) { // Array is dominated by NULL but has non-NULL values - // TODO(joe): use exe ctx? - let non_null_values = array.filter(mask.clone())?.to_canonical()?.into_array(); + let non_null_values = array + .filter(mask.clone())? + .execute::(ctx)? + .into_array(); let non_null_indices = match mask.indices() { AllOr::All => { // We already know that the mask is 90%+ false @@ -445,8 +457,8 @@ impl SparseData { fill.cast(array.dtype())? } else { // TODO(robert): Support other dtypes, only thing missing is getting most common value out of the array - let (top_pvalue, _) = array - .to_primitive() + let primitive = array.clone().execute::(ctx)?; + let (top_pvalue, _) = primitive .top_value()? .vortex_expect("Non empty or all null array"); @@ -454,17 +466,15 @@ impl SparseData { }; let fill_array = ConstantArray::new(fill.clone(), array.len()).into_array(); - let non_top_mask = Mask::from_buffer( - array - .binary(fill_array.clone(), Operator::NotEq)? - .fill_null(Scalar::bool(true, Nullability::NonNullable))? - .to_bool() - .to_bit_buffer(), - ); + let non_top_bool = array + .binary(fill_array.clone(), Operator::NotEq)? + .fill_null(Scalar::bool(true, Nullability::NonNullable))? + .execute::(ctx)?; + let non_top_mask = Mask::from_buffer(non_top_bool.to_bit_buffer()); let non_top_values = array .filter(non_top_mask.clone())? - .to_canonical()? + .execute::(ctx)? .into_array(); let indices: Buffer = match non_top_mask { @@ -512,6 +522,8 @@ impl ValidityVTable for Sparse { mod test { use itertools::Itertools; use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::ConstantArray; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; @@ -549,16 +561,33 @@ mod test { pub fn test_scalar_at() { let array = sparse_array(nullable_fill()); - assert_eq!(array.scalar_at(0).unwrap(), nullable_fill()); - assert_eq!(array.scalar_at(2).unwrap(), Scalar::from(Some(100_i32))); - assert_eq!(array.scalar_at(5).unwrap(), Scalar::from(Some(200_i32))); + assert_eq!( + array + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + nullable_fill() + ); + assert_eq!( + array + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + Scalar::from(Some(100_i32)) + ); + assert_eq!( + array + .execute_scalar(5, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + Scalar::from(Some(200_i32)) + ); } #[test] #[should_panic(expected = "out of bounds")] fn test_scalar_at_oob() { let array = sparse_array(nullable_fill()); - array.scalar_at(10).unwrap(); + array + .execute_scalar(10, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); } #[test] @@ -572,27 +601,47 @@ mod test { .unwrap(); assert_eq!( - arr.scalar_at(10) + arr.execute_scalar(10, &mut LEGACY_SESSION.create_execution_ctx()) .unwrap() .as_primitive() .typed_value::(), Some(1234) ); - assert!(arr.scalar_at(0).unwrap().is_null()); - assert!(arr.scalar_at(99).unwrap().is_null()); + assert!( + arr.execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); + assert!( + arr.execute_scalar(99, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); } #[test] pub fn scalar_at_sliced() { let sliced = sparse_array(nullable_fill()).slice(2..7).unwrap(); - assert_eq!(usize::try_from(&sliced.scalar_at(0).unwrap()).unwrap(), 100); + assert_eq!( + usize::try_from( + &sliced + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ) + .unwrap(), + 100 + ); } #[test] pub fn validity_mask_sliced_null_fill() { let sliced = sparse_array(nullable_fill()).slice(2..7).unwrap(); assert_eq!( - sliced.validity_mask().unwrap(), + sliced + .validity() + .unwrap() + .execute_mask(sliced.len(), &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Mask::from_iter(vec![true, false, false, true, false]) ); } @@ -614,7 +663,11 @@ mod test { .unwrap(); assert_eq!( - sliced.validity_mask().unwrap(), + sliced + .validity() + .unwrap() + .execute_mask(sliced.len(), &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Mask::from_iter(vec![false, true, true, false, true]) ); } @@ -623,13 +676,23 @@ mod test { pub fn scalar_at_sliced_twice() { let sliced_once = sparse_array(nullable_fill()).slice(1..8).unwrap(); assert_eq!( - usize::try_from(&sliced_once.scalar_at(1).unwrap()).unwrap(), + usize::try_from( + &sliced_once + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ) + .unwrap(), 100 ); let sliced_twice = sliced_once.slice(1..6).unwrap(); assert_eq!( - usize::try_from(&sliced_twice.scalar_at(3).unwrap()).unwrap(), + usize::try_from( + &sliced_twice + .execute_scalar(3, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ) + .unwrap(), 200 ); } @@ -639,7 +702,9 @@ mod test { let array = sparse_array(nullable_fill()); assert_eq!( array - .validity_mask() + .validity() + .unwrap() + .execute_mask(array.len(), &mut LEGACY_SESSION.create_execution_ctx()) .unwrap() .to_bit_buffer() .iter() @@ -653,7 +718,14 @@ mod test { #[test] fn sparse_validity_mask_non_null_fill() { let array = sparse_array(non_nullable_fill()); - assert!(array.validity_mask().unwrap().all_true()); + assert!( + array + .validity() + .unwrap() + .execute_mask(array.len(), &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .all_true() + ); } #[test] @@ -675,21 +747,27 @@ mod test { #[test] fn encode_with_nulls() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let original = PrimitiveArray::new( buffer![0i32, 1, 2, 3, 3, 3, 3, 3, 3, 3, 4, 4], Validity::from_iter(vec![ true, true, false, true, false, true, false, true, true, false, true, false, ]), ); - let sparse = Sparse::encode(&original.clone().into_array(), None) + let sparse = Sparse::encode(&original.clone().into_array(), None, &mut ctx) .vortex_expect("Sparse::encode should succeed for test data"); assert_eq!( - sparse.validity_mask().unwrap(), + sparse + .validity() + .unwrap() + .execute_mask(sparse.len(), &mut ctx) + .unwrap(), Mask::from_iter(vec![ true, true, false, true, false, true, false, true, true, false, true, false, ]) ); - assert_arrays_eq!(sparse.to_primitive(), original); + let sparse_primitive = sparse.execute::(&mut ctx).unwrap(); + assert_arrays_eq!(sparse_primitive, original); } #[test] @@ -698,7 +776,11 @@ mod test { let values = PrimitiveArray::from_option_iter([Some(0i16), Some(1), None, None, Some(4)]) .into_array(); let array = Sparse::try_new(indices, values, 10, Scalar::null_native::()).unwrap(); - let actual = array.validity_mask().unwrap(); + let actual = array + .validity() + .unwrap() + .execute_mask(array.len(), &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); let expected = Mask::from_iter([ true, false, true, false, false, false, false, false, true, false, ]); diff --git a/encodings/sparse/src/ops.rs b/encodings/sparse/src/ops.rs index 552dcf5fa6e..cfa89c0a733 100644 --- a/encodings/sparse/src/ops.rs +++ b/encodings/sparse/src/ops.rs @@ -25,7 +25,8 @@ impl OperationsVTable for Sparse { #[cfg(test)] mod tests { use vortex_array::IntoArray; - use vortex_array::ToCanonical; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; use vortex_buffer::buffer; @@ -34,6 +35,7 @@ mod tests { #[test] fn slice_partially_invalid() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values = buffer![0u64].into_array(); let indices = buffer![0u8].into_array(); @@ -42,7 +44,7 @@ mod tests { let mut expected = vec![999u64; 1000]; expected[0] = 0; - let values = sliced.to_primitive(); + let values = sliced.execute::(&mut ctx).unwrap(); assert_arrays_eq!(values, PrimitiveArray::from_iter(expected)); } } diff --git a/encodings/zigzag/public-api.lock b/encodings/zigzag/public-api.lock index a366ce1d631..f38988cd680 100644 --- a/encodings/zigzag/public-api.lock +++ b/encodings/zigzag/public-api.lock @@ -4,9 +4,7 @@ pub struct vortex_zigzag::ZigZag impl vortex_zigzag::ZigZag -pub const vortex_zigzag::ZigZag::ID: vortex_array::array::ArrayId - -pub fn vortex_zigzag::ZigZag::try_new(encoded: vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_zigzag::ZigZag::try_new(vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult impl core::clone::Clone for vortex_zigzag::ZigZag @@ -14,7 +12,7 @@ pub fn vortex_zigzag::ZigZag::clone(&self) -> vortex_zigzag::ZigZag impl core::fmt::Debug for vortex_zigzag::ZigZag -pub fn vortex_zigzag::ZigZag::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_zigzag::ZigZag::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::array::vtable::VTable for vortex_zigzag::ZigZag @@ -24,55 +22,55 @@ pub type vortex_zigzag::ZigZag::OperationsVTable = vortex_zigzag::ZigZag pub type vortex_zigzag::ZigZag::ValidityVTable = vortex_array::array::vtable::validity::ValidityVTableFromChild -pub fn vortex_zigzag::ZigZag::buffer(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_zigzag::ZigZag::buffer(vortex_array::array::view::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_zigzag::ZigZag::buffer_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_zigzag::ZigZag::buffer_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_zigzag::ZigZag::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_zigzag::ZigZag::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_zigzag::ZigZag::execute(array: vortex_array::array::typed::Array, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_zigzag::ZigZag::execute(vortex_array::array::typed::Array, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_zigzag::ZigZag::execute_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_zigzag::ZigZag::execute_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_zigzag::ZigZag::id(&self) -> vortex_array::array::ArrayId -pub fn vortex_zigzag::ZigZag::nbuffers(_array: vortex_array::array::view::ArrayView<'_, Self>) -> usize +pub fn vortex_zigzag::ZigZag::nbuffers(vortex_array::array::view::ArrayView<'_, Self>) -> usize -pub fn vortex_zigzag::ZigZag::reduce_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_zigzag::ZigZag::reduce_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_zigzag::ZigZag::serialize(_array: vortex_array::array::view::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_zigzag::ZigZag::serialize(vortex_array::array::view::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_zigzag::ZigZag::slot_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_zigzag::ZigZag::slot_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_zigzag::ZigZag::validate(&self, _data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_zigzag::ZigZag::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::array::vtable::operations::OperationsVTable for vortex_zigzag::ZigZag -pub fn vortex_zigzag::ZigZag::scalar_at(array: vortex_array::array::view::ArrayView<'_, vortex_zigzag::ZigZag>, index: usize, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_zigzag::ZigZag::scalar_at(vortex_array::array::view::ArrayView<'_, vortex_zigzag::ZigZag>, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::array::vtable::validity::ValidityChild for vortex_zigzag::ZigZag -pub fn vortex_zigzag::ZigZag::validity_child(array: vortex_array::array::view::ArrayView<'_, vortex_zigzag::ZigZag>) -> vortex_array::array::erased::ArrayRef +pub fn vortex_zigzag::ZigZag::validity_child(vortex_array::array::view::ArrayView<'_, vortex_zigzag::ZigZag>) -> vortex_array::array::erased::ArrayRef impl vortex_array::arrays::dict::take::TakeExecute for vortex_zigzag::ZigZag -pub fn vortex_zigzag::ZigZag::take(array: vortex_array::array::view::ArrayView<'_, Self>, indices: &vortex_array::array::erased::ArrayRef, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_zigzag::ZigZag::take(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::kernel::FilterReduce for vortex_zigzag::ZigZag -pub fn vortex_zigzag::ZigZag::filter(array: vortex_array::array::view::ArrayView<'_, Self>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_zigzag::ZigZag::filter(vortex_array::array::view::ArrayView<'_, Self>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_zigzag::ZigZag -pub fn vortex_zigzag::ZigZag::slice(array: vortex_array::array::view::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_zigzag::ZigZag::slice(vortex_array::array::view::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::kernel::CastReduce for vortex_zigzag::ZigZag -pub fn vortex_zigzag::ZigZag::cast(array: vortex_array::array::view::ArrayView<'_, Self>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_zigzag::ZigZag::cast(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::kernel::MaskReduce for vortex_zigzag::ZigZag -pub fn vortex_zigzag::ZigZag::mask(array: vortex_array::array::view::ArrayView<'_, Self>, mask: &vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_zigzag::ZigZag::mask(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult> pub struct vortex_zigzag::ZigZagData @@ -80,7 +78,7 @@ impl vortex_zigzag::ZigZagData pub fn vortex_zigzag::ZigZagData::new() -> Self -pub fn vortex_zigzag::ZigZagData::try_new(encoded_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_zigzag::ZigZagData::try_new(&vortex_array::dtype::DType) -> vortex_error::VortexResult impl core::clone::Clone for vortex_zigzag::ZigZagData @@ -92,19 +90,19 @@ pub fn vortex_zigzag::ZigZagData::default() -> Self impl core::fmt::Debug for vortex_zigzag::ZigZagData -pub fn vortex_zigzag::ZigZagData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_zigzag::ZigZagData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_zigzag::ZigZagData -pub fn vortex_zigzag::ZigZagData::fmt(&self, _f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_zigzag::ZigZagData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::hash::ArrayEq for vortex_zigzag::ZigZagData -pub fn vortex_zigzag::ZigZagData::array_eq(&self, _other: &Self, _precision: vortex_array::hash::Precision) -> bool +pub fn vortex_zigzag::ZigZagData::array_eq(&self, &Self, vortex_array::hash::Precision) -> bool impl vortex_array::hash::ArrayHash for vortex_zigzag::ZigZagData -pub fn vortex_zigzag::ZigZagData::array_hash(&self, _state: &mut H, _precision: vortex_array::hash::Precision) +pub fn vortex_zigzag::ZigZagData::array_hash(&self, &mut H, vortex_array::hash::Precision) pub trait vortex_zigzag::ZigZagArrayExt: vortex_array::array::typed::TypedArrayRef @@ -118,8 +116,8 @@ pub fn T::encoded(&self) -> &vortex_array::array::erased::ArrayRef pub fn T::ptype(&self) -> vortex_array::dtype::ptype::PType -pub fn vortex_zigzag::zigzag_decode(parray: vortex_array::arrays::primitive::vtable::PrimitiveArray) -> vortex_array::arrays::primitive::vtable::PrimitiveArray +pub fn vortex_zigzag::zigzag_decode(vortex_array::arrays::primitive::vtable::PrimitiveArray) -> vortex_array::arrays::primitive::vtable::PrimitiveArray -pub fn vortex_zigzag::zigzag_encode(parray: vortex_array::arrays::primitive::vtable::PrimitiveArray) -> vortex_error::VortexResult +pub fn vortex_zigzag::zigzag_encode(vortex_array::array::view::ArrayView<'_, vortex_array::arrays::primitive::vtable::Primitive>) -> vortex_error::VortexResult pub type vortex_zigzag::ZigZagArray = vortex_array::array::typed::Array diff --git a/encodings/zigzag/src/array.rs b/encodings/zigzag/src/array.rs index 34b1d2d19f6..52658e53961 100644 --- a/encodings/zigzag/src/array.rs +++ b/encodings/zigzag/src/array.rs @@ -33,6 +33,7 @@ use vortex_error::vortex_bail; use vortex_error::vortex_ensure; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use zigzag::ZigZag as ExternalZigZag; use crate::compute::ZigZagEncoded; @@ -50,7 +51,8 @@ impl VTable for ZigZag { type ValidityVTable = ValidityVTableFromChild; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.zigzag"); + *ID } fn validate( @@ -195,8 +197,6 @@ impl> ZigZagArrayExt for T {} pub struct ZigZag; impl ZigZag { - pub const ID: ArrayId = ArrayId::new_ref("vortex.zigzag"); - /// Construct a new [`ZigZagArray`] from an encoded unsigned integer array. pub fn try_new(encoded: ArrayRef) -> VortexResult { let dtype = ZigZagData::dtype_from_encoded_dtype(encoded.dtype())?; @@ -240,9 +240,9 @@ impl OperationsVTable for ZigZag { fn scalar_at( array: ArrayView<'_, ZigZag>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { - let scalar = array.encoded().scalar_at(index)?; + let scalar = array.encoded().execute_scalar(index, ctx)?; if scalar.is_null() { return scalar.primitive_reinterpret_cast(ZigZagArrayExt::ptype(&array)); } @@ -270,7 +270,9 @@ impl ValidityChild for ZigZag { #[cfg(test)] mod test { use vortex_array::IntoArray; - use vortex_array::ToCanonical; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; + use vortex_array::arrays::PrimitiveArray; use vortex_array::scalar::Scalar; use vortex_buffer::buffer; @@ -279,42 +281,43 @@ mod test { #[test] fn test_compute_statistics() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = buffer![1i32, -5i32, 2, 3, 4, 5, 6, 7, 8, 9, 10] .into_array() - .to_primitive(); - let zigzag = zigzag_encode(array.clone())?; + .execute::(&mut ctx)?; + let zigzag = zigzag_encode(array.as_view())?; assert_eq!( - zigzag.statistics().compute_max::(), - array.statistics().compute_max::() + zigzag.statistics().compute_max::(&mut ctx), + array.statistics().compute_max::(&mut ctx) ); assert_eq!( - zigzag.statistics().compute_null_count(), - array.statistics().compute_null_count() + zigzag.statistics().compute_null_count(&mut ctx), + array.statistics().compute_null_count(&mut ctx) ); assert_eq!( - zigzag.statistics().compute_is_constant(), - array.statistics().compute_is_constant() + zigzag.statistics().compute_is_constant(&mut ctx), + array.statistics().compute_is_constant(&mut ctx) ); - let sliced = zigzag.slice(0..2).unwrap(); + let sliced = zigzag.slice(0..2)?; let sliced = sliced.as_::(); assert_eq!( - sliced.array().scalar_at(sliced.len() - 1).unwrap(), + sliced.array().execute_scalar(sliced.len() - 1, &mut ctx,)?, Scalar::from(-5i32) ); assert_eq!( - sliced.statistics().compute_min::(), - array.statistics().compute_min::() + sliced.statistics().compute_min::(&mut ctx), + array.statistics().compute_min::(&mut ctx) ); assert_eq!( - sliced.statistics().compute_null_count(), - array.statistics().compute_null_count() + sliced.statistics().compute_null_count(&mut ctx), + array.statistics().compute_null_count(&mut ctx) ); assert_eq!( - sliced.statistics().compute_is_constant(), - array.statistics().compute_is_constant() + sliced.statistics().compute_is_constant(&mut ctx), + array.statistics().compute_is_constant(&mut ctx) ); Ok(()) } diff --git a/encodings/zigzag/src/compress.rs b/encodings/zigzag/src/compress.rs index 998a39f2b49..2f52bd89d24 100644 --- a/encodings/zigzag/src/compress.rs +++ b/encodings/zigzag/src/compress.rs @@ -1,7 +1,9 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +use vortex_array::ArrayView; use vortex_array::IntoArray; +use vortex_array::arrays::Primitive; use vortex_array::arrays::PrimitiveArray; use vortex_array::dtype::NativePType; use vortex_array::dtype::PType; @@ -15,7 +17,8 @@ use zigzag::ZigZag as ExternalZigZag; use crate::ZigZag; use crate::ZigZagArray; -pub fn zigzag_encode(parray: PrimitiveArray) -> VortexResult { +pub fn zigzag_encode(parray: ArrayView<'_, Primitive>) -> VortexResult { + let parray = parray.into_owned(); let validity = parray.validity()?; let encoded = match parray.ptype() { PType::I8 => zigzag_encode_primitive::(parray.into_buffer_mut(), validity), @@ -75,7 +78,8 @@ where #[cfg(test)] mod test { use vortex_array::IntoArray; - use vortex_array::ToCanonical; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::assert_arrays_eq; use super::*; @@ -83,46 +87,42 @@ mod test { #[test] fn test_compress_i8() { - let compressed = zigzag_encode(PrimitiveArray::from_iter(-100_i8..100)) + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let compressed = zigzag_encode(PrimitiveArray::from_iter(-100_i8..100).as_view()) .unwrap() .into_array(); assert!(compressed.is::()); - assert_arrays_eq!( - compressed.to_primitive(), - PrimitiveArray::from_iter(-100_i8..100) - ); + let decompressed = compressed.execute::(&mut ctx).unwrap(); + assert_arrays_eq!(decompressed, PrimitiveArray::from_iter(-100_i8..100)); } #[test] fn test_compress_i16() { - let compressed = zigzag_encode(PrimitiveArray::from_iter(-100_i16..100)) + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let compressed = zigzag_encode(PrimitiveArray::from_iter(-100_i16..100).as_view()) .unwrap() .into_array(); assert!(compressed.is::()); - assert_arrays_eq!( - compressed.to_primitive(), - PrimitiveArray::from_iter(-100_i16..100) - ); + let decompressed = compressed.execute::(&mut ctx).unwrap(); + assert_arrays_eq!(decompressed, PrimitiveArray::from_iter(-100_i16..100)); } #[test] fn test_compress_i32() { - let compressed = zigzag_encode(PrimitiveArray::from_iter(-100_i32..100)) + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let compressed = zigzag_encode(PrimitiveArray::from_iter(-100_i32..100).as_view()) .unwrap() .into_array(); assert!(compressed.is::()); - assert_arrays_eq!( - compressed.to_primitive(), - PrimitiveArray::from_iter(-100_i32..100) - ); + let decompressed = compressed.execute::(&mut ctx).unwrap(); + assert_arrays_eq!(decompressed, PrimitiveArray::from_iter(-100_i32..100)); } #[test] fn test_compress_i64() { - let compressed = zigzag_encode(PrimitiveArray::from_iter(-100_i64..100)) + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let compressed = zigzag_encode(PrimitiveArray::from_iter(-100_i64..100).as_view()) .unwrap() .into_array(); assert!(compressed.is::()); - assert_arrays_eq!( - compressed.to_primitive(), - PrimitiveArray::from_iter(-100_i64..100) - ); + let decompressed = compressed.execute::(&mut ctx).unwrap(); + assert_arrays_eq!(decompressed, PrimitiveArray::from_iter(-100_i64..100)); } } diff --git a/encodings/zigzag/src/compute/cast.rs b/encodings/zigzag/src/compute/cast.rs index 7f6741d9aa1..a71eee7a838 100644 --- a/encodings/zigzag/src/compute/cast.rs +++ b/encodings/zigzag/src/compute/cast.rs @@ -42,7 +42,7 @@ mod tests { #[test] fn test_cast_zigzag_i32_to_i64() { let values = PrimitiveArray::from_iter([-100i32, -1, 0, 1, 100]); - let zigzag = zigzag_encode(values).unwrap(); + let zigzag = zigzag_encode(values.as_view()).unwrap(); let casted = zigzag .into_array() @@ -68,7 +68,7 @@ mod tests { fn test_cast_zigzag_width_changes() { // Test i32 to i16 (narrowing) let values = PrimitiveArray::from_iter([100i32, -50, 0, 25, -100]); - let zigzag = zigzag_encode(values).unwrap(); + let zigzag = zigzag_encode(values.as_view()).unwrap(); let casted = zigzag .into_array() @@ -87,7 +87,7 @@ mod tests { // Test i16 to i64 (widening) let values16 = PrimitiveArray::from_iter([1000i16, -500, 0, 250, -1000]); - let zigzag16 = zigzag_encode(values16).unwrap(); + let zigzag16 = zigzag_encode(values16.as_view()).unwrap(); let casted64 = zigzag16 .into_array() @@ -109,7 +109,7 @@ mod tests { fn test_cast_zigzag_nullable() { let values = PrimitiveArray::from_option_iter([Some(-10i32), None, Some(0), Some(10), None]); - let zigzag = zigzag_encode(values).unwrap(); + let zigzag = zigzag_encode(values.as_view()).unwrap(); let casted = zigzag .into_array() @@ -122,10 +122,10 @@ mod tests { } #[rstest] - #[case(zigzag_encode(PrimitiveArray::from_iter([-100i32, -50, -1, 0, 1, 50, 100])).unwrap())] - #[case(zigzag_encode(PrimitiveArray::from_iter([-1000i64, -1, 0, 1, 1000])).unwrap())] - #[case(zigzag_encode(PrimitiveArray::from_option_iter([Some(-5i16), None, Some(0), Some(5), None])).unwrap())] - #[case(zigzag_encode(PrimitiveArray::from_iter([i32::MIN, -1, 0, 1, i32::MAX])).unwrap())] + #[case(zigzag_encode(PrimitiveArray::from_iter([-100i32, -50, -1, 0, 1, 50, 100]).as_view()).unwrap())] + #[case(zigzag_encode(PrimitiveArray::from_iter([-1000i64, -1, 0, 1, 1000]).as_view()).unwrap())] + #[case(zigzag_encode(PrimitiveArray::from_option_iter([Some(-5i16), None, Some(0), Some(5), None]).as_view()).unwrap())] + #[case(zigzag_encode(PrimitiveArray::from_iter([i32::MIN, -1, 0, 1, i32::MAX]).as_view()).unwrap())] fn test_cast_zigzag_conformance(#[case] array: ZigZagArray) { test_cast_conformance(&array.into_array()); } diff --git a/encodings/zigzag/src/compute/mod.rs b/encodings/zigzag/src/compute/mod.rs index 71938dbcc18..a90db7eacc7 100644 --- a/encodings/zigzag/src/compute/mod.rs +++ b/encodings/zigzag/src/compute/mod.rs @@ -73,7 +73,8 @@ mod tests { use rstest::rstest; use vortex_array::ArrayRef; use vortex_array::IntoArray; - use vortex_array::ToCanonical; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; use vortex_array::compute::conformance::binary_numeric::test_binary_numeric_array; @@ -90,12 +91,11 @@ mod tests { #[test] pub fn nullable_scalar_at() -> VortexResult<()> { - let zigzag = zigzag_encode(PrimitiveArray::new( - buffer![-189, -160, 1], - Validity::AllValid, - ))?; + let zigzag = zigzag_encode( + PrimitiveArray::new(buffer![-189, -160, 1], Validity::AllValid).as_view(), + )?; assert_eq!( - zigzag.scalar_at(1)?, + zigzag.execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx())?, Scalar::primitive(-160, Nullability::Nullable) ); Ok(()) @@ -103,30 +103,30 @@ mod tests { #[test] fn take_zigzag() -> VortexResult<()> { - let zigzag = zigzag_encode(PrimitiveArray::new( - buffer![-189, -160, 1], - Validity::AllValid, - ))?; + let zigzag = zigzag_encode( + PrimitiveArray::new(buffer![-189, -160, 1], Validity::AllValid).as_view(), + )?; let indices = buffer![0, 2].into_array(); - let actual = zigzag.take(indices).unwrap(); + let actual = zigzag.take(indices)?; let expected = - zigzag_encode(PrimitiveArray::new(buffer![-189, 1], Validity::AllValid))?.into_array(); + zigzag_encode(PrimitiveArray::new(buffer![-189, 1], Validity::AllValid).as_view())? + .into_array(); assert_arrays_eq!(actual, expected); Ok(()) } #[test] fn filter_zigzag() -> VortexResult<()> { - let zigzag = zigzag_encode(PrimitiveArray::new( - buffer![-189, -160, 1], - Validity::AllValid, - ))?; + let zigzag = zigzag_encode( + PrimitiveArray::new(buffer![-189, -160, 1], Validity::AllValid).as_view(), + )?; let filter_mask = BitBuffer::from(vec![true, false, true]).into(); - let actual = zigzag.filter(filter_mask).unwrap(); + let actual = zigzag.filter(filter_mask)?; let expected = - zigzag_encode(PrimitiveArray::new(buffer![-189, 1], Validity::AllValid))?.into_array(); + zigzag_encode(PrimitiveArray::new(buffer![-189, 1], Validity::AllValid).as_view())? + .into_array(); assert_arrays_eq!(actual, expected); Ok(()) } @@ -136,23 +136,25 @@ mod tests { use vortex_array::compute::conformance::filter::test_filter_conformance; // Test with i32 values - let zigzag = zigzag_encode(PrimitiveArray::new( - buffer![-189i32, -160, 1, 42, -73], - Validity::AllValid, - ))?; + let zigzag = zigzag_encode( + PrimitiveArray::new(buffer![-189i32, -160, 1, 42, -73], Validity::AllValid).as_view(), + )?; test_filter_conformance(&zigzag.into_array()); // Test with i64 values - let zigzag = zigzag_encode(PrimitiveArray::new( - buffer![1000i64, -2000, 3000, -4000, 5000], - Validity::AllValid, - ))?; + let zigzag = zigzag_encode( + PrimitiveArray::new( + buffer![1000i64, -2000, 3000, -4000, 5000], + Validity::AllValid, + ) + .as_view(), + )?; test_filter_conformance(&zigzag.into_array()); // Test with nullable values let array = PrimitiveArray::from_option_iter([Some(-10i16), None, Some(20), Some(-30), None]); - let zigzag = zigzag_encode(array)?; + let zigzag = zigzag_encode(array.as_view())?; test_filter_conformance(&zigzag.into_array()); Ok(()) } @@ -162,17 +164,16 @@ mod tests { use vortex_array::compute::conformance::mask::test_mask_conformance; // Test with i32 values - let zigzag = zigzag_encode(PrimitiveArray::new( - buffer![-100i32, 200, -300, 400, -500], - Validity::AllValid, - ))?; + let zigzag = zigzag_encode( + PrimitiveArray::new(buffer![-100i32, 200, -300, 400, -500], Validity::AllValid) + .as_view(), + )?; test_mask_conformance(&zigzag.into_array()); // Test with i8 values - let zigzag = zigzag_encode(PrimitiveArray::new( - buffer![-127i8, 0, 127, -1, 1], - Validity::AllValid, - ))?; + let zigzag = zigzag_encode( + PrimitiveArray::new(buffer![-127i8, 0, 127, -1, 1], Validity::AllValid).as_view(), + )?; test_mask_conformance(&zigzag.into_array()); Ok(()) } @@ -186,36 +187,38 @@ mod tests { fn test_take_zigzag_conformance(#[case] array: ArrayRef) -> VortexResult<()> { use vortex_array::compute::conformance::take::test_take_conformance; - let zigzag = zigzag_encode(array.to_primitive())?; + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let array_primitive = array.execute::(&mut ctx)?; + let zigzag = zigzag_encode(array_primitive.as_view())?; test_take_conformance(&zigzag.into_array()); Ok(()) } #[rstest] // Basic ZigZag arrays - #[case::zigzag_i8(zigzag_encode(PrimitiveArray::from_iter([-128i8, -1, 0, 1, 127])).unwrap())] - #[case::zigzag_i16(zigzag_encode(PrimitiveArray::from_iter([-1000i16, -100, 0, 100, 1000])).unwrap())] - #[case::zigzag_i32(zigzag_encode(PrimitiveArray::from_iter([-100000i32, -1000, 0, 1000, 100000])).unwrap())] - #[case::zigzag_i64(zigzag_encode(PrimitiveArray::from_iter([-1000000i64, -10000, 0, 10000, 1000000])).unwrap())] + #[case::zigzag_i8(zigzag_encode(PrimitiveArray::from_iter([-128i8, -1, 0, 1, 127]).as_view()).unwrap())] + #[case::zigzag_i16(zigzag_encode(PrimitiveArray::from_iter([-1000i16, -100, 0, 100, 1000]).as_view()).unwrap())] + #[case::zigzag_i32(zigzag_encode(PrimitiveArray::from_iter([-100000i32, -1000, 0, 1000, 100000]).as_view()).unwrap())] + #[case::zigzag_i64(zigzag_encode(PrimitiveArray::from_iter([-1000000i64, -10000, 0, 10000, 1000000]).as_view()).unwrap())] // Nullable arrays - #[case::zigzag_nullable_i32(zigzag_encode(PrimitiveArray::from_option_iter([Some(-100i32), None, Some(0), Some(100), None])).unwrap())] - #[case::zigzag_nullable_i64(zigzag_encode(PrimitiveArray::from_option_iter([Some(-1000i64), None, Some(0), Some(1000), None])).unwrap())] + #[case::zigzag_nullable_i32(zigzag_encode(PrimitiveArray::from_option_iter([Some(-100i32), None, Some(0), Some(100), None]).as_view()).unwrap())] + #[case::zigzag_nullable_i64(zigzag_encode(PrimitiveArray::from_option_iter([Some(-1000i64), None, Some(0), Some(1000), None]).as_view()).unwrap())] // Edge cases - #[case::zigzag_single(zigzag_encode(PrimitiveArray::from_iter([-42i32])).unwrap())] - #[case::zigzag_alternating(zigzag_encode(PrimitiveArray::from_iter([-1i32, 1, -2, 2, -3, 3])).unwrap())] + #[case::zigzag_single(zigzag_encode(PrimitiveArray::from_iter([-42i32]).as_view()).unwrap())] + #[case::zigzag_alternating(zigzag_encode(PrimitiveArray::from_iter([-1i32, 1, -2, 2, -3, 3]).as_view()).unwrap())] // Large arrays - #[case::zigzag_large_i32(zigzag_encode(PrimitiveArray::from_iter(-500..500)).unwrap())] - #[case::zigzag_large_i64(zigzag_encode(PrimitiveArray::from_iter((-1000..1000).map(|i| i as i64 * 100))).unwrap())] + #[case::zigzag_large_i32(zigzag_encode(PrimitiveArray::from_iter(-500..500).as_view()).unwrap())] + #[case::zigzag_large_i64(zigzag_encode(PrimitiveArray::from_iter((-1000..1000).map(|i| i as i64 * 100)).as_view()).unwrap())] fn test_zigzag_consistency(#[case] array: ZigZagArray) { test_array_consistency(&array.into_array()); } #[rstest] - #[case::zigzag_i8_basic(zigzag_encode(PrimitiveArray::from_iter([-10i8, -5, 0, 5, 10])).unwrap())] - #[case::zigzag_i16_basic(zigzag_encode(PrimitiveArray::from_iter([-100i16, -50, 0, 50, 100])).unwrap())] - #[case::zigzag_i32_basic(zigzag_encode(PrimitiveArray::from_iter([-1000i32, -500, 0, 500, 1000])).unwrap())] - #[case::zigzag_i64_basic(zigzag_encode(PrimitiveArray::from_iter([-10000i64, -5000, 0, 5000, 10000])).unwrap())] - #[case::zigzag_i32_large(zigzag_encode(PrimitiveArray::from_iter((-50..50).map(|i| i * 10))).unwrap())] + #[case::zigzag_i8_basic(zigzag_encode(PrimitiveArray::from_iter([-10i8, -5, 0, 5, 10]).as_view()).unwrap())] + #[case::zigzag_i16_basic(zigzag_encode(PrimitiveArray::from_iter([-100i16, -50, 0, 50, 100]).as_view()).unwrap())] + #[case::zigzag_i32_basic(zigzag_encode(PrimitiveArray::from_iter([-1000i32, -500, 0, 500, 1000]).as_view()).unwrap())] + #[case::zigzag_i64_basic(zigzag_encode(PrimitiveArray::from_iter([-10000i64, -5000, 0, 5000, 10000]).as_view()).unwrap())] + #[case::zigzag_i32_large(zigzag_encode(PrimitiveArray::from_iter((-50..50).map(|i| i * 10)).as_view()).unwrap())] fn test_zigzag_binary_numeric(#[case] array: ZigZagArray) { test_binary_numeric_array(array.into_array()); } diff --git a/encodings/zstd/benches/listview_rebuild.rs b/encodings/zstd/benches/listview_rebuild.rs index 82b16c392fa..4b8ca95eafc 100644 --- a/encodings/zstd/benches/listview_rebuild.rs +++ b/encodings/zstd/benches/listview_rebuild.rs @@ -1,10 +1,12 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] +#![expect(clippy::unwrap_used)] use divan::Bencher; use vortex_array::IntoArray; +use vortex_array::LEGACY_SESSION; +use vortex_array::VortexSessionExecute; use vortex_array::arrays::ListViewArray; use vortex_array::arrays::VarBinViewArray; use vortex_array::arrays::listview::ListViewRebuildMode; @@ -21,7 +23,7 @@ fn rebuild_naive(bencher: Bencher) { let validity = dudes.validity().unwrap(); let dudes = Zstd::try_new( dtype, - ZstdData::from_array(dudes, 9, 1024).unwrap(), + ZstdData::from_array(dudes, 9, 1024, &mut LEGACY_SESSION.create_execution_ctx()).unwrap(), validity, ) .unwrap() diff --git a/encodings/zstd/public-api.lock b/encodings/zstd/public-api.lock index 44365606591..ef3cfff174a 100644 --- a/encodings/zstd/public-api.lock +++ b/encodings/zstd/public-api.lock @@ -4,17 +4,15 @@ pub struct vortex_zstd::Zstd impl vortex_zstd::Zstd -pub const vortex_zstd::Zstd::ID: vortex_array::array::ArrayId +pub fn vortex_zstd::Zstd::decompress(&vortex_zstd::ZstdArray, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_zstd::Zstd::decompress(array: &vortex_zstd::ZstdArray, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_zstd::Zstd::from_primitive(&vortex_array::arrays::primitive::vtable::PrimitiveArray, i32, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_zstd::Zstd::from_primitive(parray: &vortex_array::arrays::primitive::vtable::PrimitiveArray, level: i32, values_per_frame: usize) -> vortex_error::VortexResult +pub fn vortex_zstd::Zstd::from_var_bin_view(&vortex_array::arrays::varbinview::vtable::VarBinViewArray, i32, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_zstd::Zstd::from_var_bin_view(vbv: &vortex_array::arrays::varbinview::vtable::VarBinViewArray, level: i32, values_per_frame: usize) -> vortex_error::VortexResult +pub fn vortex_zstd::Zstd::from_var_bin_view_without_dict(&vortex_array::arrays::varbinview::vtable::VarBinViewArray, i32, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_zstd::Zstd::from_var_bin_view_without_dict(vbv: &vortex_array::arrays::varbinview::vtable::VarBinViewArray, level: i32, values_per_frame: usize) -> vortex_error::VortexResult - -pub fn vortex_zstd::Zstd::try_new(dtype: vortex_array::dtype::DType, data: vortex_zstd::ZstdData, validity: vortex_array::validity::Validity) -> vortex_error::VortexResult +pub fn vortex_zstd::Zstd::try_new(vortex_array::dtype::DType, vortex_zstd::ZstdData, vortex_array::validity::Validity) -> vortex_error::VortexResult impl core::clone::Clone for vortex_zstd::Zstd @@ -22,7 +20,7 @@ pub fn vortex_zstd::Zstd::clone(&self) -> vortex_zstd::Zstd impl core::fmt::Debug for vortex_zstd::Zstd -pub fn vortex_zstd::Zstd::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_zstd::Zstd::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::array::vtable::VTable for vortex_zstd::Zstd @@ -32,41 +30,41 @@ pub type vortex_zstd::Zstd::OperationsVTable = vortex_zstd::Zstd pub type vortex_zstd::Zstd::ValidityVTable = vortex_zstd::Zstd -pub fn vortex_zstd::Zstd::buffer(array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_zstd::Zstd::buffer(vortex_array::array::view::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_zstd::Zstd::buffer_name(array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_zstd::Zstd::buffer_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_zstd::Zstd::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_zstd::Zstd::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_zstd::Zstd::execute(array: vortex_array::array::typed::Array, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_zstd::Zstd::execute(vortex_array::array::typed::Array, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_zstd::Zstd::id(&self) -> vortex_array::array::ArrayId -pub fn vortex_zstd::Zstd::nbuffers(array: vortex_array::array::view::ArrayView<'_, Self>) -> usize +pub fn vortex_zstd::Zstd::nbuffers(vortex_array::array::view::ArrayView<'_, Self>) -> usize -pub fn vortex_zstd::Zstd::reduce_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_zstd::Zstd::reduce_parent(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::array::erased::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_zstd::Zstd::serialize(array: vortex_array::array::view::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_zstd::Zstd::serialize(vortex_array::array::view::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_zstd::Zstd::slot_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_zstd::Zstd::slot_name(vortex_array::array::view::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_zstd::Zstd::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_zstd::Zstd::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::array::vtable::operations::OperationsVTable for vortex_zstd::Zstd -pub fn vortex_zstd::Zstd::scalar_at(array: vortex_array::array::view::ArrayView<'_, vortex_zstd::Zstd>, index: usize, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_zstd::Zstd::scalar_at(vortex_array::array::view::ArrayView<'_, vortex_zstd::Zstd>, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::array::vtable::validity::ValidityVTable for vortex_zstd::Zstd -pub fn vortex_zstd::Zstd::validity(array: vortex_array::array::view::ArrayView<'_, vortex_zstd::Zstd>) -> vortex_error::VortexResult +pub fn vortex_zstd::Zstd::validity(vortex_array::array::view::ArrayView<'_, vortex_zstd::Zstd>) -> vortex_error::VortexResult impl vortex_array::arrays::slice::SliceReduce for vortex_zstd::Zstd -pub fn vortex_zstd::Zstd::slice(array: vortex_array::array::view::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_zstd::Zstd::slice(vortex_array::array::view::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::kernel::CastReduce for vortex_zstd::Zstd -pub fn vortex_zstd::Zstd::cast(array: vortex_array::array::view::ArrayView<'_, Self>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_zstd::Zstd::cast(vortex_array::array::view::ArrayView<'_, Self>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> pub struct vortex_zstd::ZstdBuffersMetadata @@ -88,7 +86,7 @@ pub fn vortex_zstd::ZstdBuffersMetadata::default() -> Self impl core::fmt::Debug for vortex_zstd::ZstdBuffersMetadata -pub fn vortex_zstd::ZstdBuffersMetadata::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_zstd::ZstdBuffersMetadata::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl prost::message::Message for vortex_zstd::ZstdBuffersMetadata @@ -100,27 +98,27 @@ pub struct vortex_zstd::ZstdData impl vortex_zstd::ZstdData -pub fn vortex_zstd::ZstdData::from_array(array: vortex_array::array::erased::ArrayRef, level: i32, values_per_frame: usize) -> vortex_error::VortexResult +pub fn vortex_zstd::ZstdData::from_array(vortex_array::array::erased::ArrayRef, i32, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_zstd::ZstdData::from_canonical(canonical: &vortex_array::canonical::Canonical, level: i32, values_per_frame: usize) -> vortex_error::VortexResult> +pub fn vortex_zstd::ZstdData::from_canonical(&vortex_array::canonical::Canonical, i32, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> -pub fn vortex_zstd::ZstdData::from_primitive(parray: &vortex_array::arrays::primitive::vtable::PrimitiveArray, level: i32, values_per_frame: usize) -> vortex_error::VortexResult +pub fn vortex_zstd::ZstdData::from_primitive(&vortex_array::arrays::primitive::vtable::PrimitiveArray, i32, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_zstd::ZstdData::from_primitive_without_dict(parray: &vortex_array::arrays::primitive::vtable::PrimitiveArray, level: i32, values_per_frame: usize) -> vortex_error::VortexResult +pub fn vortex_zstd::ZstdData::from_primitive_without_dict(&vortex_array::arrays::primitive::vtable::PrimitiveArray, i32, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_zstd::ZstdData::from_var_bin_view(vbv: &vortex_array::arrays::varbinview::vtable::VarBinViewArray, level: i32, values_per_frame: usize) -> vortex_error::VortexResult +pub fn vortex_zstd::ZstdData::from_var_bin_view(&vortex_array::arrays::varbinview::vtable::VarBinViewArray, i32, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_zstd::ZstdData::from_var_bin_view_without_dict(vbv: &vortex_array::arrays::varbinview::vtable::VarBinViewArray, level: i32, values_per_frame: usize) -> vortex_error::VortexResult +pub fn vortex_zstd::ZstdData::from_var_bin_view_without_dict(&vortex_array::arrays::varbinview::vtable::VarBinViewArray, i32, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_zstd::ZstdData::into_parts(self, validity: vortex_array::validity::Validity) -> vortex_zstd::ZstdDataParts +pub fn vortex_zstd::ZstdData::into_parts(self, vortex_array::validity::Validity) -> vortex_zstd::ZstdDataParts pub fn vortex_zstd::ZstdData::is_empty(&self) -> bool pub fn vortex_zstd::ZstdData::len(&self) -> usize -pub fn vortex_zstd::ZstdData::new(dictionary: core::option::Option, frames: alloc::vec::Vec, metadata: vortex_zstd::ZstdMetadata, n_rows: usize) -> Self +pub fn vortex_zstd::ZstdData::new(core::option::Option, alloc::vec::Vec, vortex_zstd::ZstdMetadata, usize) -> Self -pub fn vortex_zstd::ZstdData::validate(&self, dtype: &vortex_array::dtype::DType, len: usize, validity: &vortex_array::validity::Validity) -> vortex_error::VortexResult<()> +pub fn vortex_zstd::ZstdData::validate(&self, &vortex_array::dtype::DType, usize, &vortex_array::validity::Validity) -> vortex_error::VortexResult<()> impl core::clone::Clone for vortex_zstd::ZstdData @@ -128,19 +126,19 @@ pub fn vortex_zstd::ZstdData::clone(&self) -> vortex_zstd::ZstdData impl core::fmt::Debug for vortex_zstd::ZstdData -pub fn vortex_zstd::ZstdData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_zstd::ZstdData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_zstd::ZstdData -pub fn vortex_zstd::ZstdData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_zstd::ZstdData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::hash::ArrayEq for vortex_zstd::ZstdData -pub fn vortex_zstd::ZstdData::array_eq(&self, other: &Self, precision: vortex_array::hash::Precision) -> bool +pub fn vortex_zstd::ZstdData::array_eq(&self, &Self, vortex_array::hash::Precision) -> bool impl vortex_array::hash::ArrayHash for vortex_zstd::ZstdData -pub fn vortex_zstd::ZstdData::array_hash(&self, state: &mut H, precision: vortex_array::hash::Precision) +pub fn vortex_zstd::ZstdData::array_hash(&self, &mut H, vortex_array::hash::Precision) pub struct vortex_zstd::ZstdDataParts @@ -174,7 +172,7 @@ pub fn vortex_zstd::ZstdFrameMetadata::default() -> Self impl core::fmt::Debug for vortex_zstd::ZstdFrameMetadata -pub fn vortex_zstd::ZstdFrameMetadata::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_zstd::ZstdFrameMetadata::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl prost::message::Message for vortex_zstd::ZstdFrameMetadata @@ -198,7 +196,7 @@ pub fn vortex_zstd::ZstdMetadata::default() -> Self impl core::fmt::Debug for vortex_zstd::ZstdMetadata -pub fn vortex_zstd::ZstdMetadata::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_zstd::ZstdMetadata::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl prost::message::Message for vortex_zstd::ZstdMetadata @@ -206,6 +204,6 @@ pub fn vortex_zstd::ZstdMetadata::clear(&mut self) pub fn vortex_zstd::ZstdMetadata::encoded_len(&self) -> usize -pub fn vortex_zstd::reconstruct_views(buffer: &vortex_buffer::ByteBuffer, max_buffer_len: usize) -> (alloc::vec::Vec, vortex_buffer::buffer::Buffer) +pub fn vortex_zstd::reconstruct_views(&vortex_buffer::ByteBuffer, usize) -> (alloc::vec::Vec, vortex_buffer::buffer::Buffer) pub type vortex_zstd::ZstdArray = vortex_array::array::typed::Array diff --git a/encodings/zstd/src/array.rs b/encodings/zstd/src/array.rs index e8a3612b06d..2a685f1ea2a 100644 --- a/encodings/zstd/src/array.rs +++ b/encodings/zstd/src/array.rs @@ -21,10 +21,7 @@ use vortex_array::Canonical; use vortex_array::ExecutionCtx; use vortex_array::ExecutionResult; use vortex_array::IntoArray; -use vortex_array::LEGACY_SESSION; use vortex_array::Precision; -use vortex_array::ToCanonical; -use vortex_array::VortexSessionExecute; use vortex_array::accessor::ArrayAccessor; use vortex_array::arrays::ConstantArray; use vortex_array::arrays::PrimitiveArray; @@ -55,6 +52,7 @@ use vortex_error::vortex_err; use vortex_error::vortex_panic; use vortex_mask::AllOr; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::ZstdFrameMetadata; use crate::ZstdMetadata; @@ -134,7 +132,8 @@ impl VTable for Zstd { type ValidityVTable = Self; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.zstd"); + *ID } fn validate( @@ -144,7 +143,7 @@ impl VTable for Zstd { len: usize, slots: &[Option], ) -> VortexResult<()> { - let validity = child_to_validity(&slots[0], dtype.nullability()); + let validity = child_to_validity(slots[0].as_ref(), dtype.nullability()); data.validate(dtype, len, &validity) } @@ -231,8 +230,10 @@ impl VTable for Zstd { } fn execute(array: Array, ctx: &mut ExecutionCtx) -> VortexResult { - let unsliced_validity = - child_to_validity(&array.as_ref().slots()[0], array.dtype().nullability()); + let unsliced_validity = child_to_validity( + array.as_ref().slots()[0].as_ref(), + array.dtype().nullability(), + ); array .data() .decompress(array.dtype(), &unsliced_validity, ctx)? @@ -253,8 +254,6 @@ impl VTable for Zstd { pub struct Zstd; impl Zstd { - pub const ID: ArrayId = ArrayId::new_ref("vortex.zstd"); - pub fn try_new(dtype: DType, data: ZstdData, validity: Validity) -> VortexResult { let len = data.len(); data.validate(&dtype, len, &validity)?; @@ -269,11 +268,12 @@ impl Zstd { vbv: &VarBinViewArray, level: i32, values_per_frame: usize, + ctx: &mut ExecutionCtx, ) -> VortexResult { let validity = vbv.validity()?; Self::try_new( vbv.dtype().clone(), - ZstdData::from_var_bin_view_without_dict(vbv, level, values_per_frame)?, + ZstdData::from_var_bin_view_without_dict(vbv, level, values_per_frame, ctx)?, validity, ) } @@ -283,11 +283,12 @@ impl Zstd { parray: &PrimitiveArray, level: i32, values_per_frame: usize, + ctx: &mut ExecutionCtx, ) -> VortexResult { let validity = parray.validity()?; Self::try_new( parray.dtype().clone(), - ZstdData::from_primitive(parray, level, values_per_frame)?, + ZstdData::from_primitive(parray, level, values_per_frame, ctx)?, validity, ) } @@ -297,18 +298,21 @@ impl Zstd { vbv: &VarBinViewArray, level: i32, values_per_frame: usize, + ctx: &mut ExecutionCtx, ) -> VortexResult { let validity = vbv.validity()?; Self::try_new( vbv.dtype().clone(), - ZstdData::from_var_bin_view(vbv, level, values_per_frame)?, + ZstdData::from_var_bin_view(vbv, level, values_per_frame, ctx)?, validity, ) } pub fn decompress(array: &ZstdArray, ctx: &mut ExecutionCtx) -> VortexResult { - let unsliced_validity = - child_to_validity(&array.as_ref().slots()[0], array.dtype().nullability()); + let unsliced_validity = child_to_validity( + array.as_ref().slots()[0].as_ref(), + array.dtype().nullability(), + ); array .data() .decompress(array.dtype(), &unsliced_validity, ctx) @@ -365,13 +369,26 @@ fn choose_max_dict_size(uncompressed_size: usize) -> usize { (uncompressed_size / 100).clamp(256, 100 * 1024) } -fn collect_valid_primitive(parray: &PrimitiveArray) -> VortexResult { - let mask = parray.validity_mask()?; - Ok(parray.filter(mask)?.to_primitive()) +fn collect_valid_primitive( + parray: &PrimitiveArray, + ctx: &mut ExecutionCtx, +) -> VortexResult { + let mask = parray + .as_ref() + .validity()? + .execute_mask(parray.as_ref().len(), ctx)?; + let result = parray.filter(mask)?.execute::(ctx)?; + Ok(result) } -fn collect_valid_vbv(vbv: &VarBinViewArray) -> VortexResult<(ByteBuffer, Vec)> { - let mask = vbv.validity_mask()?; +fn collect_valid_vbv( + vbv: &VarBinViewArray, + ctx: &mut ExecutionCtx, +) -> VortexResult<(ByteBuffer, Vec)> { + let mask = vbv + .as_ref() + .validity()? + .execute_mask(vbv.as_ref().len(), ctx)?; let buffer_and_value_byte_indices = match mask.bit_buffer() { AllOr::None => (Buffer::empty(), Vec::new()), _ => { @@ -616,8 +633,9 @@ impl ZstdData { parray: &PrimitiveArray, level: i32, values_per_frame: usize, + ctx: &mut ExecutionCtx, ) -> VortexResult { - Self::from_primitive_impl(parray, level, values_per_frame, true) + Self::from_primitive_impl(parray, level, values_per_frame, true, ctx) } /// Creates a ZstdArray from a primitive array without using a dictionary. @@ -637,8 +655,9 @@ impl ZstdData { parray: &PrimitiveArray, level: i32, values_per_frame: usize, + ctx: &mut ExecutionCtx, ) -> VortexResult { - Self::from_primitive_impl(parray, level, values_per_frame, false) + Self::from_primitive_impl(parray, level, values_per_frame, false, ctx) } fn from_primitive_impl( @@ -646,11 +665,12 @@ impl ZstdData { level: i32, values_per_frame: usize, use_dictionary: bool, + ctx: &mut ExecutionCtx, ) -> VortexResult { let byte_width = parray.ptype().byte_width(); // We compress only the valid elements. - let values = collect_valid_primitive(parray)?; + let values = collect_valid_primitive(parray, ctx)?; let n_values = values.len(); let values_per_frame = if values_per_frame > 0 { values_per_frame @@ -700,8 +720,9 @@ impl ZstdData { vbv: &VarBinViewArray, level: i32, values_per_frame: usize, + ctx: &mut ExecutionCtx, ) -> VortexResult { - Self::from_var_bin_view_impl(vbv, level, values_per_frame, true) + Self::from_var_bin_view_impl(vbv, level, values_per_frame, true, ctx) } /// Creates a ZstdArray from a VarBinView array without using a dictionary. @@ -721,8 +742,9 @@ impl ZstdData { vbv: &VarBinViewArray, level: i32, values_per_frame: usize, + ctx: &mut ExecutionCtx, ) -> VortexResult { - Self::from_var_bin_view_impl(vbv, level, values_per_frame, false) + Self::from_var_bin_view_impl(vbv, level, values_per_frame, false, ctx) } fn from_var_bin_view_impl( @@ -730,6 +752,7 @@ impl ZstdData { level: i32, values_per_frame: usize, use_dictionary: bool, + ctx: &mut ExecutionCtx, ) -> VortexResult { // Approach for strings: we prefix each string with its length as a u32. // This is the same as what Parquet does. In some cases it may be better @@ -737,7 +760,7 @@ impl ZstdData { // this approach is simpler and can be best in cases when there is // mutual information between strings and their lengths. // We compress only the valid elements. - let (value_bytes, value_byte_indices) = collect_valid_vbv(vbv)?; + let (value_bytes, value_byte_indices) = collect_valid_vbv(vbv, ctx)?; let n_values = value_byte_indices.len(); let values_per_frame = if values_per_frame > 0 { values_per_frame @@ -776,24 +799,33 @@ impl ZstdData { canonical: &Canonical, level: i32, values_per_frame: usize, + ctx: &mut ExecutionCtx, ) -> VortexResult> { match canonical { Canonical::Primitive(parray) => Ok(Some(ZstdData::from_primitive( parray, level, values_per_frame, + ctx, )?)), Canonical::VarBinView(vbv) => Ok(Some(ZstdData::from_var_bin_view( vbv, level, values_per_frame, + ctx, )?)), _ => Ok(None), } } - pub fn from_array(array: ArrayRef, level: i32, values_per_frame: usize) -> VortexResult { - Self::from_canonical(&array.to_canonical()?, level, values_per_frame)? + pub fn from_array( + array: ArrayRef, + level: i32, + values_per_frame: usize, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + let canonical = array.execute::(ctx)?; + Self::from_canonical(&canonical, level, values_per_frame, ctx)? .ok_or_else(|| vortex_err!("Zstd can only encode Primitive and VarBinView arrays")) } @@ -1014,7 +1046,8 @@ impl ZstdData { impl ValidityVTable for Zstd { fn validity(array: ArrayView<'_, Zstd>) -> VortexResult { - let unsliced_validity = child_to_validity(&array.slots()[0], array.dtype().nullability()); + let unsliced_validity = + child_to_validity(array.slots()[0].as_ref(), array.dtype().nullability()); unsliced_validity.slice(array.slice_start()..array.slice_stop()) } } @@ -1023,19 +1056,19 @@ impl OperationsVTable for Zstd { fn scalar_at( array: ArrayView<'_, Zstd>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { - let mut ctx = LEGACY_SESSION.create_execution_ctx(); - let unsliced_validity = child_to_validity(&array.slots()[0], array.dtype().nullability()); + let unsliced_validity = + child_to_validity(array.slots()[0].as_ref(), array.dtype().nullability()); let sliced = array.data().with_slice(index, index + 1); sliced - .decompress(array.dtype(), &unsliced_validity, &mut ctx)? - .scalar_at(0) + .decompress(array.dtype(), &unsliced_validity, ctx)? + .execute_scalar(0, ctx) } } #[cfg(test)] -#[allow(clippy::cast_possible_truncation)] +#[expect(clippy::cast_possible_truncation)] mod tests { use vortex_buffer::ByteBuffer; diff --git a/encodings/zstd/src/compute/cast.rs b/encodings/zstd/src/compute/cast.rs index 2beb94a1a4e..e295e556566 100644 --- a/encodings/zstd/src/compute/cast.rs +++ b/encodings/zstd/src/compute/cast.rs @@ -7,12 +7,12 @@ use vortex_array::IntoArray; use vortex_array::dtype::DType; use vortex_array::dtype::Nullability; use vortex_array::scalar_fn::fns::cast::CastReduce; -use vortex_array::validity::Validity; use vortex_array::vtable::child_to_validity; use vortex_error::VortexResult; use crate::Zstd; use crate::ZstdData; + impl CastReduce for Zstd { fn cast(array: ArrayView<'_, Self>, dtype: &DType) -> VortexResult> { if !dtype.eq_ignore_nullability(array.dtype()) { @@ -24,71 +24,58 @@ impl CastReduce for Zstd { let src_nullability = array.dtype().nullability(); let target_nullability = dtype.nullability(); - match (src_nullability, target_nullability) { + let new_validity = match (src_nullability, target_nullability) { // Same type case. This should be handled in the layer above but for // completeness of the match arms we also handle it here. (Nullability::Nullable, Nullability::Nullable) | (Nullability::NonNullable, Nullability::NonNullable) => { - Ok(Some(array.array().clone())) + return Ok(Some(array.array().clone())); } (Nullability::NonNullable, Nullability::Nullable) => { // nonnull => null, trivial cast by altering the validity - let unsliced_validity = - child_to_validity(&array.slots()[0], array.dtype().nullability()); - Ok(Some( - Zstd::try_new( - dtype.clone(), - ZstdData::new( - array.dictionary.clone(), - array.frames.clone(), - array.metadata.clone(), - array.unsliced_n_rows(), - ), - unsliced_validity, - )? - .into_array() - .slice(array.slice_start()..array.slice_stop())?, - )) + child_to_validity(array.slots()[0].as_ref(), array.dtype().nullability()) } (Nullability::Nullable, Nullability::NonNullable) => { // null => non-null works if there are no nulls in the sliced range let unsliced_validity = - child_to_validity(&array.slots()[0], array.dtype().nullability()); - let has_nulls = !matches!( - unsliced_validity.slice(array.slice_start()..array.slice_stop())?, - Validity::AllValid | Validity::NonNullable - ); + child_to_validity(array.slots()[0].as_ref(), array.dtype().nullability()); + let has_nulls = !unsliced_validity + .slice(array.slice_start()..array.slice_stop())? + .no_nulls(); // We don't attempt to handle casting when there are nulls. if has_nulls { return Ok(None); } - - // If there are no nulls, the cast is trivial - Ok(Some( - Zstd::try_new( - dtype.clone(), - ZstdData::new( - array.dictionary.clone(), - array.frames.clone(), - array.metadata.clone(), - array.unsliced_n_rows(), - ), - unsliced_validity, - )? - .into_array() - .slice(array.slice_start()..array.slice_stop())?, - )) + unsliced_validity } - } + }; + + // If there are no nulls, the cast is trivial + Ok(Some( + Zstd::try_new( + dtype.clone(), + ZstdData::new( + array.dictionary.clone(), + array.frames.clone(), + array.metadata.clone(), + array.unsliced_n_rows(), + ), + new_validity, + )? + .into_array() + .slice(array.slice_start()..array.slice_stop())?, + )) } } #[cfg(test)] mod tests { + use std::sync::LazyLock; + use rstest::rstest; use vortex_array::IntoArray; - use vortex_array::ToCanonical; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; use vortex_array::builtins::ArrayBuiltins; @@ -96,15 +83,21 @@ mod tests { use vortex_array::dtype::DType; use vortex_array::dtype::Nullability; use vortex_array::dtype::PType; + use vortex_array::session::ArraySession; use vortex_array::validity::Validity; use vortex_buffer::buffer; + use vortex_session::VortexSession; use crate::Zstd; + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); + #[test] fn test_cast_zstd_i32_to_i64() { + let mut ctx = SESSION.create_execution_ctx(); let values = PrimitiveArray::from_iter([1i32, 2, 3, 4, 5]); - let zstd = Zstd::from_primitive(&values, 0, 0).unwrap(); + let zstd = Zstd::from_primitive(&values, 0, 0, &mut ctx).unwrap(); let casted = zstd .into_array() @@ -115,14 +108,15 @@ mod tests { &DType::Primitive(PType::I64, Nullability::NonNullable) ); - let decoded = casted.to_primitive(); + let decoded = casted.execute::(&mut ctx).unwrap(); assert_arrays_eq!(decoded, PrimitiveArray::from_iter([1i64, 2, 3, 4, 5])); } #[test] fn test_cast_zstd_nullability_change() { + let mut ctx = SESSION.create_execution_ctx(); let values = PrimitiveArray::from_iter([10u32, 20, 30, 40]); - let zstd = Zstd::from_primitive(&values, 0, 0).unwrap(); + let zstd = Zstd::from_primitive(&values, 0, 0, &mut ctx).unwrap(); let casted = zstd .into_array() @@ -136,11 +130,12 @@ mod tests { #[test] fn test_cast_sliced_zstd_nullable_to_nonnullable() { + let mut ctx = SESSION.create_execution_ctx(); let values = PrimitiveArray::new( buffer![10u32, 20, 30, 40, 50, 60], Validity::from_iter([true, true, true, true, true, true]), ); - let zstd = Zstd::from_primitive(&values, 0, 128).unwrap(); + let zstd = Zstd::from_primitive(&values, 0, 128, &mut ctx).unwrap(); let sliced = zstd.slice(1..5).unwrap(); let casted = sliced .cast(DType::Primitive(PType::U32, Nullability::NonNullable)) @@ -150,12 +145,13 @@ mod tests { &DType::Primitive(PType::U32, Nullability::NonNullable) ); // Verify the values are correct - let decoded = casted.to_primitive(); + let decoded = casted.execute::(&mut ctx).unwrap(); assert_arrays_eq!(decoded, PrimitiveArray::from_iter([20u32, 30, 40, 50])); } #[test] fn test_cast_sliced_zstd_part_valid_to_nonnullable() { + let mut ctx = SESSION.create_execution_ctx(); let values = PrimitiveArray::from_option_iter([ None, Some(20u32), @@ -164,7 +160,7 @@ mod tests { Some(50), Some(60), ]); - let zstd = Zstd::from_primitive(&values, 0, 128).unwrap(); + let zstd = Zstd::from_primitive(&values, 0, 128, &mut ctx).unwrap(); let sliced = zstd.slice(1..5).unwrap(); let casted = sliced .cast(DType::Primitive(PType::U32, Nullability::NonNullable)) @@ -173,7 +169,7 @@ mod tests { casted.dtype(), &DType::Primitive(PType::U32, Nullability::NonNullable) ); - let decoded = casted.to_primitive(); + let decoded = casted.execute::(&mut ctx).unwrap(); let expected = PrimitiveArray::from_iter([20u32, 30, 40, 50]); assert_arrays_eq!(decoded, expected); } @@ -196,7 +192,8 @@ mod tests { Validity::NonNullable, ))] fn test_cast_zstd_conformance(#[case] values: PrimitiveArray) { - let zstd = Zstd::from_primitive(&values, 0, 0).unwrap(); + let zstd = + Zstd::from_primitive(&values, 0, 0, &mut SESSION.create_execution_ctx()).unwrap(); test_cast_conformance(&zstd.into_array()); } } diff --git a/encodings/zstd/src/compute/mod.rs b/encodings/zstd/src/compute/mod.rs index adf6937c246..de175eb3154 100644 --- a/encodings/zstd/src/compute/mod.rs +++ b/encodings/zstd/src/compute/mod.rs @@ -7,6 +7,8 @@ mod cast; mod tests { use rstest::rstest; use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::compute::conformance::consistency::test_array_consistency; use vortex_buffer::buffer; @@ -16,23 +18,23 @@ mod tests { fn zstd_i32() -> ZstdArray { let values = PrimitiveArray::from_iter([100i32, 200, 300, 400, 500]); - Zstd::from_primitive(&values, 0, 0).unwrap() + Zstd::from_primitive(&values, 0, 0, &mut LEGACY_SESSION.create_execution_ctx()).unwrap() } fn zstd_f64() -> ZstdArray { let values = PrimitiveArray::from_iter([1.1f64, 2.2, 3.3, 4.4, 5.5]); - Zstd::from_primitive(&values, 0, 0).unwrap() + Zstd::from_primitive(&values, 0, 0, &mut LEGACY_SESSION.create_execution_ctx()).unwrap() } fn zstd_u32() -> ZstdArray { let values = PrimitiveArray::from_iter([10u32, 20, 30, 40, 50]); - Zstd::from_primitive(&values, 0, 0).unwrap() + Zstd::from_primitive(&values, 0, 0, &mut LEGACY_SESSION.create_execution_ctx()).unwrap() } fn zstd_nullable_i64() -> ZstdArray { let values = PrimitiveArray::from_option_iter([Some(1000i64), None, Some(3000), Some(4000), None]); - Zstd::from_primitive(&values, 0, 0).unwrap() + Zstd::from_primitive(&values, 0, 0, &mut LEGACY_SESSION.create_execution_ctx()).unwrap() } fn zstd_single() -> ZstdArray { @@ -40,7 +42,7 @@ mod tests { buffer![42i64], vortex_array::validity::Validity::NonNullable, ); - Zstd::from_primitive(&values, 0, 0).unwrap() + Zstd::from_primitive(&values, 0, 0, &mut LEGACY_SESSION.create_execution_ctx()).unwrap() } fn zstd_large() -> ZstdArray { @@ -48,7 +50,7 @@ mod tests { buffer![0u32..1000], vortex_array::validity::Validity::NonNullable, ); - Zstd::from_primitive(&values, 3, 0).unwrap() + Zstd::from_primitive(&values, 3, 0, &mut LEGACY_SESSION.create_execution_ctx()).unwrap() } fn zstd_all_same() -> ZstdArray { @@ -56,12 +58,12 @@ mod tests { buffer![42i32; 100], vortex_array::validity::Validity::NonNullable, ); - Zstd::from_primitive(&values, 0, 0).unwrap() + Zstd::from_primitive(&values, 0, 0, &mut LEGACY_SESSION.create_execution_ctx()).unwrap() } fn zstd_negative() -> ZstdArray { let values = PrimitiveArray::from_iter([-100i32, -50, 0, 50, 100]); - Zstd::from_primitive(&values, 0, 0).unwrap() + Zstd::from_primitive(&values, 0, 0, &mut LEGACY_SESSION.create_execution_ctx()).unwrap() } #[rstest] diff --git a/encodings/zstd/src/slice.rs b/encodings/zstd/src/slice.rs index 6b6906a1628..54f4441635d 100644 --- a/encodings/zstd/src/slice.rs +++ b/encodings/zstd/src/slice.rs @@ -14,7 +14,8 @@ use crate::Zstd; impl SliceReduce for Zstd { fn slice(array: ArrayView<'_, Self>, range: Range) -> VortexResult> { - let unsliced_validity = child_to_validity(&array.slots()[0], array.dtype().nullability()); + let unsliced_validity = + child_to_validity(array.slots()[0].as_ref(), array.dtype().nullability()); Ok(Some( Zstd::try_new( array.dtype().clone(), diff --git a/encodings/zstd/src/test.rs b/encodings/zstd/src/test.rs index 01800577d7b..7ed22886b82 100644 --- a/encodings/zstd/src/test.rs +++ b/encodings/zstd/src/test.rs @@ -1,10 +1,9 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::cast_possible_truncation)] +#![expect(clippy::cast_possible_truncation)] use vortex_array::IntoArray; use vortex_array::LEGACY_SESSION; -use vortex_array::ToCanonical; use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; use vortex_array::arrays::PrimitiveArray; @@ -22,16 +21,16 @@ use crate::Zstd; #[test] fn test_zstd_compress_decompress() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let data: Vec = (0..200).collect(); let array = PrimitiveArray::from_iter(data.clone()); - let compressed = Zstd::from_primitive(&array, 3, 0).unwrap(); + let compressed = Zstd::from_primitive(&array, 3, 0, &mut ctx).unwrap(); // this data should be compressible assert!(compressed.frames.len() < array.into_array().nbytes() as usize); assert!(compressed.dictionary.is_none()); // check full decompression works - let mut ctx = LEGACY_SESSION.create_execution_ctx(); let decompressed = Zstd::decompress(&compressed, &mut ctx).unwrap(); assert_arrays_eq!(decompressed, PrimitiveArray::from_iter(data)); @@ -48,19 +47,21 @@ fn test_zstd_compress_decompress() { #[test] fn test_zstd_empty() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let data: Vec = vec![]; let array = PrimitiveArray::new( data.iter().cloned().collect::>(), Validity::NonNullable, ); - let compressed = Zstd::from_primitive(&array, 3, 100).unwrap(); + let compressed = Zstd::from_primitive(&array, 3, 100, &mut ctx).unwrap(); assert_arrays_eq!(compressed, PrimitiveArray::from_iter(data)); } #[test] fn test_zstd_with_validity_and_multi_frame() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let data: Vec = (0..200).collect(); let mut validity: Vec = vec![false; 200]; validity[3] = true; @@ -70,17 +71,17 @@ fn test_zstd_with_validity_and_multi_frame() { Validity::Array(BoolArray::from_iter(validity).into_array()), ); - let compressed = Zstd::from_primitive(&array, 0, 30).unwrap(); + let compressed = Zstd::from_primitive(&array, 0, 30, &mut ctx).unwrap(); assert!(compressed.dictionary.is_none()); assert_nth_scalar!(compressed, 0, None::); assert_nth_scalar!(compressed, 3, 3); assert_nth_scalar!(compressed, 10, None::); assert_nth_scalar!(compressed, 177, 177); - let mut ctx = LEGACY_SESSION.create_execution_ctx(); let decompressed = Zstd::decompress(&compressed, &mut ctx) .unwrap() - .to_primitive(); + .execute::(&mut ctx) + .unwrap(); let decompressed_values = decompressed.as_slice::(); assert_eq!(decompressed_values[3], 3); assert_eq!(decompressed_values[177], 177); @@ -94,9 +95,9 @@ fn test_zstd_with_validity_and_multi_frame() { // check slicing works let slice = compressed.slice(176..179).unwrap(); - let primitive = slice.to_primitive(); + let primitive = slice.execute::(&mut ctx).unwrap(); assert_eq!( - i32::try_from(&primitive.scalar_at(1).unwrap()).unwrap(), + i32::try_from(&primitive.execute_scalar(1, &mut ctx).unwrap()).unwrap(), 177 ); assert!( @@ -113,49 +114,61 @@ fn test_zstd_with_validity_and_multi_frame() { #[test] fn test_zstd_with_dict() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let data: Vec = (0..200).collect(); let array = PrimitiveArray::new( data.iter().cloned().collect::>(), Validity::NonNullable, ); - let compressed = Zstd::from_primitive(&array, 0, 16).unwrap(); + let compressed = Zstd::from_primitive(&array, 0, 16, &mut ctx).unwrap(); assert!(compressed.dictionary.is_some()); assert_nth_scalar!(compressed, 0, 0); assert_nth_scalar!(compressed, 199, 199); - let mut ctx = LEGACY_SESSION.create_execution_ctx(); let decompressed = Zstd::decompress(&compressed, &mut ctx) .unwrap() - .to_primitive(); + .execute::(&mut ctx) + .unwrap(); assert_arrays_eq!(decompressed, PrimitiveArray::from_iter(data)); // check slicing works let slice = compressed.slice(176..179).unwrap(); - let primitive = slice.to_primitive(); + let primitive = slice.execute::(&mut ctx).unwrap(); assert_arrays_eq!(primitive, PrimitiveArray::from_iter([176, 177, 178])); } #[test] fn test_validity_vtable() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let mask_bools = vec![false, true, true, false, true]; let array = PrimitiveArray::new( (0..5).collect::>(), Validity::Array(BoolArray::from_iter(mask_bools.clone()).into_array()), ); - let compressed = Zstd::from_primitive(&array, 3, 0).unwrap(); + let compressed = Zstd::from_primitive(&array, 3, 0, &mut ctx).unwrap(); + let arr = compressed.as_array(); assert_eq!( - compressed.as_array().validity_mask().unwrap(), + arr.validity() + .unwrap() + .execute_mask(arr.len(), &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Mask::from_iter(mask_bools) ); + let sliced = compressed.slice(1..4).unwrap(); assert_eq!( - compressed.slice(1..4).unwrap().validity_mask().unwrap(), + sliced + .validity() + .unwrap() + .execute_mask(sliced.len(), &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Mask::from_iter(vec![true, true, false]) ); } #[test] fn test_zstd_var_bin_view() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let data: [Option<&'static [u8]>; 5] = [ Some(b"foo"), Some(b"bar"), @@ -165,7 +178,7 @@ fn test_zstd_var_bin_view() { ]; let array = VarBinViewArray::from_iter(data, DType::Utf8(Nullability::Nullable)); - let compressed = Zstd::from_var_bin_view(&array, 0, 3).unwrap(); + let compressed = Zstd::from_var_bin_view(&array, 0, 3, &mut ctx).unwrap(); assert!(compressed.dictionary.is_none()); assert_nth_scalar!(compressed, 0, "foo"); assert_nth_scalar!(compressed, 1, "bar"); @@ -181,6 +194,7 @@ fn test_zstd_var_bin_view() { #[test] fn test_zstd_decompress_var_bin_view() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let data: [Option<&'static [u8]>; 5] = [ Some(b"foo"), Some(b"bar"), @@ -190,7 +204,7 @@ fn test_zstd_decompress_var_bin_view() { ]; let array = VarBinViewArray::from_iter(data, DType::Utf8(Nullability::Nullable)); - let compressed = Zstd::from_var_bin_view(&array, 0, 3).unwrap(); + let compressed = Zstd::from_var_bin_view(&array, 0, 3, &mut ctx).unwrap(); assert!(compressed.dictionary.is_none()); assert_nth_scalar!(compressed, 0, "foo"); assert_nth_scalar!(compressed, 1, "bar"); @@ -198,10 +212,10 @@ fn test_zstd_decompress_var_bin_view() { assert_nth_scalar!(compressed, 3, "Lorem ipsum dolor sit amet"); assert_nth_scalar!(compressed, 4, "baz"); - let mut ctx = LEGACY_SESSION.create_execution_ctx(); let decompressed = Zstd::decompress(&compressed, &mut ctx) .unwrap() - .to_varbinview(); + .execute::(&mut ctx) + .unwrap(); assert_nth_scalar!(decompressed, 0, "foo"); assert_nth_scalar!(decompressed, 1, "bar"); assert_nth_scalar!(decompressed, 2, None::); @@ -211,8 +225,10 @@ fn test_zstd_decompress_var_bin_view() { #[test] fn test_sliced_array_children() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let data: Vec> = (0..10).map(|v| (v != 5).then_some(v)).collect(); - let compressed = Zstd::from_primitive(&PrimitiveArray::from_option_iter(data), 0, 100).unwrap(); + let compressed = + Zstd::from_primitive(&PrimitiveArray::from_option_iter(data), 0, 100, &mut ctx).unwrap(); let sliced = compressed.slice(0..4).unwrap(); sliced.children(); } @@ -221,11 +237,12 @@ fn test_sliced_array_children() { /// the buffer alignment when compressing primitive arrays. #[test] fn test_zstd_frame_start_buffer_alignment() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let data = vec![0u8; 2]; let aligned_buffer = Buffer::copy_from_aligned(&data, Alignment::new(8)); // u8 array now has a 8-byte alignment. let array = PrimitiveArray::new(aligned_buffer, Validity::NonNullable); - let compressed = Zstd::from_primitive(&array, 0, 1); + let compressed = Zstd::from_primitive(&array, 0, 1, &mut ctx); assert!(compressed.is_ok()); } diff --git a/encodings/zstd/src/zstd_buffers.rs b/encodings/zstd/src/zstd_buffers.rs index e1712b01c2e..f59a536655b 100644 --- a/encodings/zstd/src/zstd_buffers.rs +++ b/encodings/zstd/src/zstd_buffers.rs @@ -34,6 +34,7 @@ use vortex_error::VortexResult; use vortex_error::vortex_ensure_eq; use vortex_error::vortex_err; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::ZstdBuffersMetadata; @@ -44,8 +45,6 @@ pub type ZstdBuffersArray = Array; pub struct ZstdBuffers; impl ZstdBuffers { - pub const ID: ArrayId = ArrayId::new_ref("vortex.zstd_buffers"); - pub fn try_new( dtype: DType, len: usize, @@ -60,9 +59,9 @@ impl ZstdBuffers { session: &VortexSession, ) -> VortexResult { let encoding_id = array.encoding_id(); - let metadata = array - .metadata(session)? - .ok_or_else(|| vortex_err!("Array does not support serialization"))?; + let metadata = session + .array_serialize(array)? + .ok_or_else(|| vortex_err!("[ZstdBuffers]: Array does not support serialization"))?; let buffer_handles = array.buffer_handles(); let children = array.children(); @@ -333,7 +332,7 @@ fn compute_output_layout( } fn array_id_from_string(s: &str) -> ArrayId { - ArrayId::new_arc(Arc::from(s)) + ArrayId::new(s) } impl ArrayHash for ZstdBuffersData { @@ -369,7 +368,8 @@ impl VTable for ZstdBuffers { type ValidityVTable = Self; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.zstd_buffers"); + *ID } fn validate( @@ -456,7 +456,7 @@ impl OperationsVTable for ZstdBuffers { fn scalar_at( array: ArrayView<'_, ZstdBuffers>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { // TODO(os): maybe we should not support scalar_at, it is really slow, and adding a cache // layer here is weird. Valid use of zstd buffers array would be by executing it first into @@ -465,7 +465,7 @@ impl OperationsVTable for ZstdBuffers { &array.into_owned(), &vortex_array::LEGACY_SESSION, )?; - inner_array.scalar_at(index) + inner_array.execute_scalar(index, ctx) } } @@ -570,11 +570,18 @@ mod tests { let input = make_nullable_primitive_array(); let compressed = ZstdBuffers::compress(&input, 3, &LEGACY_SESSION)?.into_array(); - assert_eq!(compressed.all_valid()?, input.all_valid()?); - assert_eq!(compressed.all_invalid()?, input.all_invalid()?); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + assert_eq!(compressed.all_valid(&mut ctx)?, input.all_valid(&mut ctx)?); + assert_eq!( + compressed.all_invalid(&mut ctx)?, + input.all_invalid(&mut ctx)? + ); for i in 0..input.len() { - assert_eq!(compressed.is_valid(i)?, input.is_valid(i)?); + assert_eq!( + compressed.is_valid(i, &mut ctx)?, + input.is_valid(i, &mut ctx)? + ); } Ok(()) diff --git a/fuzz/fuzz_targets/array_ops.rs b/fuzz/fuzz_targets/array_ops.rs index dded332b64c..17324abe3c3 100644 --- a/fuzz/fuzz_targets/array_ops.rs +++ b/fuzz/fuzz_targets/array_ops.rs @@ -2,7 +2,7 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors #![no_main] -#![allow(clippy::unwrap_used, clippy::result_large_err)] +#![expect(clippy::unwrap_used)] use std::str::FromStr; diff --git a/fuzz/fuzz_targets/array_ops_wasm.rs b/fuzz/fuzz_targets/array_ops_wasm.rs index ddbec35e505..35b31a54fd6 100644 --- a/fuzz/fuzz_targets/array_ops_wasm.rs +++ b/fuzz/fuzz_targets/array_ops_wasm.rs @@ -7,8 +7,6 @@ //! - `LLVMFuzzerTestOneInput` - the fuzzing entry point //! - `wasmfuzz_malloc` / `wasmfuzz_free` - memory allocation for wasmfuzz to pass input data -#![allow(clippy::unwrap_used, clippy::result_large_err)] - use std::alloc::Layout; use std::alloc::alloc; use std::alloc::dealloc; diff --git a/fuzz/fuzz_targets/compress_gpu.rs b/fuzz/fuzz_targets/compress_gpu.rs index eea7ff6c5e3..f88df52d2d5 100644 --- a/fuzz/fuzz_targets/compress_gpu.rs +++ b/fuzz/fuzz_targets/compress_gpu.rs @@ -2,7 +2,7 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors #![no_main] -#![allow(clippy::unwrap_used, clippy::result_large_err)] +#![expect(clippy::unwrap_used)] use libfuzzer_sys::Corpus; use libfuzzer_sys::fuzz_target; diff --git a/fuzz/fuzz_targets/compress_roundtrip.rs b/fuzz/fuzz_targets/compress_roundtrip.rs index 22e2ee80a8e..bb5b9e464ce 100644 --- a/fuzz/fuzz_targets/compress_roundtrip.rs +++ b/fuzz/fuzz_targets/compress_roundtrip.rs @@ -2,7 +2,6 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors #![no_main] -#![allow(clippy::unwrap_used, clippy::result_large_err)] use libfuzzer_sys::Corpus; use libfuzzer_sys::fuzz_target; diff --git a/fuzz/fuzz_targets/file_io.rs b/fuzz/fuzz_targets/file_io.rs index 7d728905148..f62693ccc49 100644 --- a/fuzz/fuzz_targets/file_io.rs +++ b/fuzz/fuzz_targets/file_io.rs @@ -2,14 +2,14 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors #![no_main] -#![allow(clippy::result_large_err)] use itertools::Itertools; use libfuzzer_sys::Corpus; use libfuzzer_sys::fuzz_target; use vortex_array::Canonical; use vortex_array::IntoArray; -use vortex_array::ToCanonical; +use vortex_array::VortexSessionExecute; +use vortex_array::arrays::BoolArray; use vortex_array::arrays::ChunkedArray; use vortex_array::arrays::bool::BoolArrayExt; use vortex_array::builtins::ArrayBuiltins; @@ -45,12 +45,16 @@ fuzz_target!(|fuzz: FuzzFileAction| -> Corpus { return Corpus::Reject; } + let mut ctx = SESSION.create_execution_ctx(); let expected_array = { let bool_mask = array_data .clone() .apply(&filter_expr.clone().unwrap_or_else(|| lit(true))) .vortex_expect("filter expression evaluation should succeed in fuzz test"); - let mask = bool_mask.to_bool().to_mask_fill_null_false(); + let bool_mask_bool = bool_mask + .execute::(&mut ctx) + .vortex_expect("execute bool"); + let mask = bool_mask_bool.to_mask_fill_null_false(&mut ctx); let filtered = array_data .filter(mask) .vortex_expect("filter operation should succeed in fuzz test"); @@ -111,14 +115,17 @@ fuzz_target!(|fuzz: FuzzFileAction| -> Corpus { let bool_result = expected_array .binary(output_array.clone(), Operator::Eq) .vortex_expect("compare operation should succeed in fuzz test") - .to_bool(); + .execute::(&mut ctx) + .vortex_expect("execute bool"); let true_count = bool_result.to_bit_buffer().true_count(); if true_count != expected_array.len() && (bool_result .into_array() - .all_valid() + .all_valid(&mut ctx) .vortex_expect("all_valid") - || expected_array.all_valid().vortex_expect("all_valid")) + || expected_array + .all_valid(&mut ctx) + .vortex_expect("all_valid")) { vortex_panic!( "Failed to match original array {}with{}", diff --git a/fuzz/fuzz_targets/fsst_like.rs b/fuzz/fuzz_targets/fsst_like.rs index 8e03badff00..bf61b2430cf 100644 --- a/fuzz/fuzz_targets/fsst_like.rs +++ b/fuzz/fuzz_targets/fsst_like.rs @@ -2,7 +2,7 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors #![no_main] -#![allow(clippy::unwrap_used, clippy::result_large_err)] +#![expect(clippy::unwrap_used)] use std::str::FromStr; diff --git a/fuzz/src/array/cast.rs b/fuzz/src/array/cast.rs index a6112bec0ff..35d7199c4af 100644 --- a/fuzz/src/array/cast.rs +++ b/fuzz/src/array/cast.rs @@ -2,8 +2,8 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors use vortex_array::ArrayRef; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; -use vortex_array::ToCanonical; use vortex_array::arrays::PrimitiveArray; use vortex_array::dtype::DType; use vortex_array::dtype::Nullability::Nullable; @@ -13,7 +13,11 @@ use vortex_array::validity::Validity; use vortex_buffer::Buffer; use vortex_error::VortexResult; -pub fn cast_canonical_array(array: &ArrayRef, target: &DType) -> VortexResult> { +pub fn cast_canonical_array( + array: &ArrayRef, + target: &DType, + ctx: &mut ExecutionCtx, +) -> VortexResult> { // TODO(joe): support more casting options let is_int_to_int = target.is_int() && array.dtype().is_int(); let is_float_to_float = target.is_float() && array.dtype().is_float(); @@ -31,18 +35,21 @@ pub fn cast_canonical_array(array: &ArrayRef, target: &DType) -> VortexResult() - .iter() - .map(|v| *v as Out) - .collect::>(), - Validity::from_mask(array.validity_mask()?, target.nullability()), - ) - .into_array() + { + let prim = array.clone().execute::(ctx)?; + PrimitiveArray::new( + prim.as_slice::() + .iter() + .map(|v| *v as Out) + .collect::>(), + Validity::from_mask( + array.validity()?.execute_mask(array.len(), ctx)?, + target.nullability(), + ), + ) + .into_array() + } }) } ))) @@ -58,30 +65,35 @@ pub fn cast_canonical_array(array: &ArrayRef, target: &DType) -> VortexResult Ok(Some( - PrimitiveArray::new( - array - .to_primitive() - .as_slice::() - .iter() - .map(|v| *v as f64) - .collect::>(), - Validity::from_mask(array.validity_mask()?, target.nullability()), - ) - .into_array(), - )), - (PType::F64, PType::F32) => - { - #[allow(clippy::cast_possible_truncation)] + (PType::F32, PType::F64) => { + let prim = array.clone().execute::(ctx)?; + Ok(Some( + PrimitiveArray::new( + prim.as_slice::() + .iter() + .map(|v| *v as f64) + .collect::>(), + Validity::from_mask( + array.validity()?.execute_mask(array.len(), ctx)?, + target.nullability(), + ), + ) + .into_array(), + )) + } + (PType::F64, PType::F32) => { + let prim = array.clone().execute::(ctx)?; + #[expect(clippy::cast_possible_truncation)] Ok(Some( PrimitiveArray::new( - array - .to_primitive() - .as_slice::() + prim.as_slice::() .iter() .map(|v| *v as f32) .collect::>(), - Validity::from_mask(array.validity_mask()?, target.nullability()), + Validity::from_mask( + array.validity()?.execute_mask(array.len(), ctx)?, + target.nullability(), + ), ) .into_array(), )) diff --git a/fuzz/src/array/compare.rs b/fuzz/src/array/compare.rs index e6d069d47b1..848917b3422 100644 --- a/fuzz/src/array/compare.rs +++ b/fuzz/src/array/compare.rs @@ -2,10 +2,13 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors use vortex_array::ArrayRef; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; -use vortex_array::ToCanonical; use vortex_array::accessor::ArrayAccessor; use vortex_array::arrays::BoolArray; +use vortex_array::arrays::DecimalArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::VarBinViewArray; use vortex_array::arrays::bool::BoolArrayExt; use vortex_array::arrays::primitive::NativeValue; use vortex_array::dtype::DType; @@ -24,6 +27,7 @@ pub fn compare_canonical_array( array: &ArrayRef, value: &Scalar, operator: CompareOperator, + ctx: &mut ExecutionCtx, ) -> ArrayRef { if value.is_null() { return BoolArray::new(BitBuffer::new_unset(array.len()), Validity::AllInvalid) @@ -38,15 +42,20 @@ pub fn compare_canonical_array( .as_bool() .value() .vortex_expect("nulls handled before"); + let bool_array = array + .clone() + .execute::(ctx) + .vortex_expect("to bool"); compare_to( - array - .to_bool() + bool_array .to_bit_buffer() .iter() .zip( array - .validity_mask() + .validity() .vortex_expect("validity_mask") + .execute_mask(array.len(), ctx) + .vortex_expect("Failed to compute validity mask") .to_bit_buffer() .iter(), ) @@ -58,7 +67,10 @@ pub fn compare_canonical_array( } DType::Primitive(p, _) => { let primitive = value.as_primitive(); - let primitive_array = array.to_primitive(); + let primitive_array = array + .clone() + .execute::(ctx) + .vortex_expect("to primitive"); match_each_native_ptype!(p, |P| { let pval = primitive .typed_value::

() @@ -70,8 +82,10 @@ pub fn compare_canonical_array( .copied() .zip( array - .validity_mask() + .validity() .vortex_expect("validity_mask") + .execute_mask(array.len(), ctx) + .vortex_expect("Failed to compute validity mask") .to_bit_buffer() .iter(), ) @@ -84,7 +98,10 @@ pub fn compare_canonical_array( } DType::Decimal(..) => { let decimal = value.as_decimal(); - let decimal_array = array.to_decimal(); + let decimal_array = array + .clone() + .execute::(ctx) + .vortex_expect("to decimal"); match_each_decimal_value_type!(decimal_array.values_type(), |D| { let dval = decimal .decimal_value() @@ -98,8 +115,10 @@ pub fn compare_canonical_array( .copied() .zip( array - .validity_mask() + .validity() .vortex_expect("validity_mask") + .execute_mask(array.len(), ctx) + .vortex_expect("Failed to compute validity mask") .to_bit_buffer() .iter(), ) @@ -110,29 +129,41 @@ pub fn compare_canonical_array( ) }) } - DType::Utf8(_) => array.to_varbinview().with_iterator(|iter| { - let utf8_value = value.as_utf8(); - compare_to( - iter.map(|v| v.map(|b| unsafe { str::from_utf8_unchecked(b) })), - utf8_value.value().vortex_expect("nulls handled before"), - operator, - result_nullability, - ) - }), - DType::Binary(_) => array.to_varbinview().with_iterator(|iter| { - let binary_value = value.as_binary(); - compare_to( - // Don't understand the lifetime problem here but identity map makes it go away - #[allow(clippy::map_identity)] - iter.map(|v| v), - binary_value.value().vortex_expect("nulls handled before"), - operator, - result_nullability, - ) - }), + DType::Utf8(_) => { + let varbinview = array + .clone() + .execute::(ctx) + .vortex_expect("to varbinview"); + varbinview.with_iterator(|iter| { + let utf8_value = value.as_utf8(); + compare_to( + iter.map(|v| v.map(|b| unsafe { str::from_utf8_unchecked(b) })), + utf8_value.value().vortex_expect("nulls handled before"), + operator, + result_nullability, + ) + }) + } + DType::Binary(_) => { + let varbinview = array + .clone() + .execute::(ctx) + .vortex_expect("to varbinview"); + varbinview.with_iterator(|iter| { + let binary_value = value.as_binary(); + compare_to( + // Don't understand the lifetime problem here but identity map makes it go away + #[expect(clippy::map_identity)] + iter.map(|v| v), + binary_value.value().vortex_expect("nulls handled before"), + operator, + result_nullability, + ) + }) + } DType::Struct(..) | DType::List(..) | DType::FixedSizeList(..) => { let scalar_vals: Vec = (0..array.len()) - .map(|i| array.scalar_at(i).vortex_expect("scalar_at")) + .map(|i| array.execute_scalar(i, ctx).vortex_expect("scalar_at")) .collect(); BoolArray::from_iter(scalar_vals.iter().map(|v| { scalar_cmp(v, value, operator) diff --git a/fuzz/src/array/fill_null.rs b/fuzz/src/array/fill_null.rs index 21430ce2b4d..d19a9f2b360 100644 --- a/fuzz/src/array/fill_null.rs +++ b/fuzz/src/array/fill_null.rs @@ -5,8 +5,8 @@ use std::sync::Arc; use vortex_array::ArrayRef; use vortex_array::Canonical; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; -use vortex_array::ToCanonical; use vortex_array::arrays::BoolArray; use vortex_array::arrays::ConstantArray; use vortex_array::arrays::DecimalArray; @@ -31,16 +31,19 @@ use vortex_error::VortexResult; pub fn fill_null_canonical_array( canonical: Canonical, fill_value: &Scalar, + ctx: &mut ExecutionCtx, ) -> VortexResult { let result_nullability = fill_value.dtype().nullability(); Ok(match canonical { Canonical::Null(array) => ConstantArray::new(fill_value.clone(), array.len()).into_array(), - Canonical::Bool(array) => fill_bool_array(array, fill_value, result_nullability), - Canonical::Primitive(array) => fill_primitive_array(array, fill_value, result_nullability), - Canonical::Decimal(array) => fill_decimal_array(array, fill_value, result_nullability), + Canonical::Bool(array) => fill_bool_array(array, fill_value, result_nullability, ctx), + Canonical::Primitive(array) => { + fill_primitive_array(array, fill_value, result_nullability, ctx) + } + Canonical::Decimal(array) => fill_decimal_array(array, fill_value, result_nullability, ctx), Canonical::VarBinView(array) => { - fill_varbinview_array(array, fill_value, result_nullability) + fill_varbinview_array(array, fill_value, result_nullability, ctx) } Canonical::Struct(_) | Canonical::List(_) @@ -54,6 +57,7 @@ fn fill_bool_array( array: BoolArray, fill_value: &Scalar, result_nullability: Nullability, + ctx: &mut ExecutionCtx, ) -> ArrayRef { let fill_bool = fill_value .as_bool() @@ -69,7 +73,10 @@ fn fill_bool_array( } Validity::AllInvalid => ConstantArray::new(fill_value.clone(), array.len()).into_array(), Validity::Array(validity_array) => { - let validity_bits = validity_array.to_bool().into_bit_buffer(); + let validity_bool = validity_array + .execute::(ctx) + .vortex_expect("validity to bool"); + let validity_bits = validity_bool.into_bit_buffer(); let data_bits = array.into_bit_buffer(); let new_bits = match data_bits.try_into_mut() { @@ -98,6 +105,7 @@ fn fill_primitive_array( array: PrimitiveArray, fill_value: &Scalar, result_nullability: Nullability, + ctx: &mut ExecutionCtx, ) -> ArrayRef { match_each_native_ptype!(array.ptype(), |T| { let fill_val = T::try_from(fill_value) @@ -114,7 +122,9 @@ fn fill_primitive_array( ConstantArray::new(fill_value.clone(), array.len()).into_array() } Validity::Array(validity_array) => { - let validity_bool_array = validity_array.to_bool(); + let validity_bool_array = validity_array + .execute::(ctx) + .vortex_expect("validity to bool"); let validity_bits = validity_bool_array.to_bit_buffer(); let data_slice = array.as_slice::(); @@ -138,6 +148,7 @@ fn fill_decimal_array( array: DecimalArray, fill_value: &Scalar, result_nullability: Nullability, + ctx: &mut ExecutionCtx, ) -> ArrayRef { let decimal_dtype = array.decimal_dtype(); let decimal_scalar = fill_value.as_decimal(); @@ -160,7 +171,9 @@ fn fill_decimal_array( ConstantArray::new(fill_value.clone(), array.len()).into_array() } Validity::Array(validity_array) => { - let validity_bool_array = validity_array.to_bool(); + let validity_bool_array = validity_array + .execute::(ctx) + .vortex_expect("validity to bool"); let validity_bits = validity_bool_array.to_bit_buffer(); let data_buffer = array.buffer::(); @@ -185,6 +198,7 @@ fn fill_varbinview_array( array: VarBinViewArray, fill_value: &Scalar, result_nullability: Nullability, + ctx: &mut ExecutionCtx, ) -> ArrayRef { let array_ref = array.clone().into_array(); match array @@ -194,7 +208,9 @@ fn fill_varbinview_array( Validity::NonNullable | Validity::AllValid => array.into_array(), Validity::AllInvalid => ConstantArray::new(fill_value.clone(), array.len()).into_array(), Validity::Array(validity_array) => { - let validity_bool_array = validity_array.to_bool(); + let validity_bool_array = validity_array + .execute::(ctx) + .vortex_expect("validity to bool"); let validity_bits = validity_bool_array.to_bit_buffer(); match array.dtype() { @@ -207,7 +223,7 @@ fn fill_varbinview_array( .map(|i| { if validity_bits.value(i) { array_ref - .scalar_at(i) + .execute_scalar(i, ctx) .vortex_expect("scalar_at") .as_utf8() .value() @@ -221,9 +237,13 @@ fn fill_varbinview_array( let string_refs: Vec<&str> = strings.iter().map(|s| s.as_str()).collect(); let result = VarBinViewArray::from_iter_str(string_refs).into_array(); if result_nullability == Nullability::Nullable { + let result_vbv = result + .clone() + .execute::(ctx) + .vortex_expect("to varbinview"); VarBinViewArray::new_handle( - result.to_varbinview().views_handle().clone(), - Arc::clone(result.to_varbinview().data_buffers()), + result_vbv.views_handle().clone(), + Arc::clone(result_vbv.data_buffers()), result.dtype().as_nullable(), result_nullability.into(), ) @@ -241,7 +261,7 @@ fn fill_varbinview_array( .map(|i| { if validity_bits.value(i) { array_ref - .scalar_at(i) + .execute_scalar(i, ctx) .vortex_expect("scalar_at") .as_binary() .value() @@ -255,9 +275,13 @@ fn fill_varbinview_array( let binary_refs: Vec<&[u8]> = binaries.iter().map(|b| b.as_slice()).collect(); let result = VarBinViewArray::from_iter_bin(binary_refs).into_array(); if result_nullability == Nullability::Nullable { + let result_vbv = result + .clone() + .execute::(ctx) + .vortex_expect("to varbinview"); VarBinViewArray::new_handle( - result.to_varbinview().views_handle().clone(), - Arc::clone(result.to_varbinview().data_buffers()), + result_vbv.views_handle().clone(), + Arc::clone(result_vbv.data_buffers()), result.dtype().as_nullable(), result_nullability.into(), ) @@ -274,7 +298,10 @@ fn fill_varbinview_array( #[cfg(test)] mod tests { + use vortex_array::Canonical; use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; use vortex_array::arrays::DecimalArray; use vortex_array::arrays::PrimitiveArray; @@ -292,12 +319,20 @@ mod tests { use super::fill_null_canonical_array; + fn canonical(array: impl IntoArray) -> Canonical { + array + .into_array() + .execute::(&mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + } + #[test] fn test_fill_null_primitive() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = PrimitiveArray::from_option_iter([Some(1i32), None, Some(3), None, Some(5)]); let fill_value = Scalar::from(42i32); - let result = fill_null_canonical_array(array.to_canonical().unwrap(), &fill_value).unwrap(); + let result = fill_null_canonical_array(canonical(array), &fill_value, &mut ctx).unwrap(); let expected = PrimitiveArray::from_iter([1i32, 42, 3, 42, 5]); assert_arrays_eq!(expected, result); @@ -305,12 +340,13 @@ mod tests { #[test] fn test_fill_null_bool() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let data_buffer = BitBuffer::from(vec![true, false, false, false]); let validity_buffer = BitBuffer::from(vec![true, false, true, false]); let array = BoolArray::new(data_buffer, Validity::from(validity_buffer)); let fill_value = Scalar::from(true); - let result = fill_null_canonical_array(array.to_canonical().unwrap(), &fill_value).unwrap(); + let result = fill_null_canonical_array(canonical(array), &fill_value, &mut ctx).unwrap(); let expected = BoolArray::from(BitBuffer::from(vec![true, true, false, true])); assert_arrays_eq!(expected, result); @@ -318,13 +354,14 @@ mod tests { #[test] fn test_fill_null_string() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = VarBinViewArray::from_iter( [Some("hello"), None, Some("world")].iter().copied(), DType::Utf8(Nullability::Nullable), ); let fill_value = Scalar::utf8("default", Nullability::NonNullable); - let result = fill_null_canonical_array(array.to_canonical().unwrap(), &fill_value).unwrap(); + let result = fill_null_canonical_array(canonical(array), &fill_value, &mut ctx).unwrap(); let expected = VarBinViewArray::from_iter_str(["hello", "default", "world"]); assert_arrays_eq!(expected, result); @@ -332,10 +369,11 @@ mod tests { #[test] fn test_fill_null_all_invalid() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = PrimitiveArray::from_option_iter([None::, None, None]); let fill_value = Scalar::from(100i32); - let result = fill_null_canonical_array(array.to_canonical().unwrap(), &fill_value).unwrap(); + let result = fill_null_canonical_array(canonical(array), &fill_value, &mut ctx).unwrap(); let expected = PrimitiveArray::from_iter([100i32, 100, 100]); assert_arrays_eq!(expected, result); @@ -343,10 +381,11 @@ mod tests { #[test] fn test_fill_null_no_nulls() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = PrimitiveArray::from_iter([1i32, 2, 3]); let fill_value = Scalar::from(42i32); - let result = fill_null_canonical_array(array.to_canonical().unwrap(), &fill_value).unwrap(); + let result = fill_null_canonical_array(canonical(array), &fill_value, &mut ctx).unwrap(); let expected = PrimitiveArray::from_iter([1i32, 2, 3]); assert_arrays_eq!(expected, result); @@ -355,10 +394,11 @@ mod tests { #[test] #[should_panic] fn test_fill_null_with_null_value_errors() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = PrimitiveArray::from_option_iter([Some(1i32), None, Some(3)]); let fill_value = Scalar::null(DType::Primitive(PType::I32, Nullability::Nullable)); - let result = fill_null_canonical_array(array.to_canonical().unwrap(), &fill_value); + let result = fill_null_canonical_array(canonical(array), &fill_value, &mut ctx); assert!(result.is_err()); assert!( @@ -371,6 +411,7 @@ mod tests { #[test] fn test_fill_null_decimal_i32() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = DecimalArray::from_option_iter( [Some(100i32), None, Some(300i32), None, Some(500i32)], DecimalDType::new(10, 2), @@ -381,7 +422,7 @@ mod tests { Nullability::NonNullable, ); - let result = fill_null_canonical_array(array.to_canonical().unwrap(), &fill_value).unwrap(); + let result = fill_null_canonical_array(canonical(array), &fill_value, &mut ctx).unwrap(); let expected = DecimalArray::from_iter( [100i32, 999i32, 300i32, 999i32, 500i32], @@ -392,6 +433,7 @@ mod tests { #[test] fn test_fill_null_decimal_i64() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = DecimalArray::from_option_iter( [Some(1000i64), None, Some(3000i64)], DecimalDType::new(15, 3), @@ -402,7 +444,7 @@ mod tests { Nullability::NonNullable, ); - let result = fill_null_canonical_array(array.to_canonical().unwrap(), &fill_value).unwrap(); + let result = fill_null_canonical_array(canonical(array), &fill_value, &mut ctx).unwrap(); let expected = DecimalArray::from_iter([1000i64, 9999i64, 3000i64], DecimalDType::new(15, 3)); @@ -411,6 +453,7 @@ mod tests { #[test] fn test_fill_null_decimal_i128() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = DecimalArray::from_option_iter( [Some(10000i128), None, Some(30000i128), None], DecimalDType::new(20, 4), @@ -421,7 +464,7 @@ mod tests { Nullability::NonNullable, ); - let result = fill_null_canonical_array(array.to_canonical().unwrap(), &fill_value).unwrap(); + let result = fill_null_canonical_array(canonical(array), &fill_value, &mut ctx).unwrap(); let expected = DecimalArray::from_iter( [10000i128, 99999i128, 30000i128, 99999i128], @@ -432,6 +475,7 @@ mod tests { #[test] fn test_fill_null_decimal_all_invalid() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = DecimalArray::from_option_iter([None::, None, None], DecimalDType::new(10, 2)); let fill_value = Scalar::decimal( @@ -440,7 +484,7 @@ mod tests { Nullability::NonNullable, ); - let result = fill_null_canonical_array(array.to_canonical().unwrap(), &fill_value).unwrap(); + let result = fill_null_canonical_array(canonical(array), &fill_value, &mut ctx).unwrap(); let expected = DecimalArray::from_option_iter( [Some(777i64), Some(777i64), Some(777i64)], @@ -454,6 +498,7 @@ mod tests { #[test] fn test_fill_null_decimal_no_nulls() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = DecimalArray::from_option_iter( [Some(100i32), Some(200i32), Some(300i32)], DecimalDType::new(10, 2), @@ -464,7 +509,7 @@ mod tests { Nullability::NonNullable, ); - let result = fill_null_canonical_array(array.to_canonical().unwrap(), &fill_value).unwrap(); + let result = fill_null_canonical_array(canonical(array), &fill_value, &mut ctx).unwrap(); let expected = DecimalArray::from_option_iter( [Some(100i32), Some(200i32), Some(300i32)], diff --git a/fuzz/src/array/filter.rs b/fuzz/src/array/filter.rs index fbcc2856e61..c6ffbdb374e 100644 --- a/fuzz/src/array/filter.rs +++ b/fuzz/src/array/filter.rs @@ -2,8 +2,8 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors use vortex_array::ArrayRef; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; -use vortex_array::ToCanonical; use vortex_array::accessor::ArrayAccessor; use vortex_array::arrays::BoolArray; use vortex_array::arrays::DecimalArray; @@ -22,9 +22,16 @@ use vortex_error::VortexResult; use crate::array::take_canonical_array_non_nullable_indices; -pub fn filter_canonical_array(array: &ArrayRef, filter: &[bool]) -> VortexResult { +pub fn filter_canonical_array( + array: &ArrayRef, + filter: &[bool], + ctx: &mut ExecutionCtx, +) -> VortexResult { let validity = if array.dtype().is_nullable() { - let validity_buff = array.validity_mask()?.to_bit_buffer(); + let validity_buff = array + .validity()? + .execute_mask(array.len(), ctx)? + .to_bit_buffer(); Validity::from_iter( filter .iter() @@ -38,7 +45,7 @@ pub fn filter_canonical_array(array: &ArrayRef, filter: &[bool]) -> VortexResult match array.dtype() { DType::Bool(_) => { - let bool_array = array.to_bool(); + let bool_array = array.clone().execute::(ctx)?; Ok(BoolArray::new( BitBuffer::from_iter( filter @@ -52,7 +59,7 @@ pub fn filter_canonical_array(array: &ArrayRef, filter: &[bool]) -> VortexResult .into_array()) } DType::Primitive(p, _) => match_each_native_ptype!(p, |P| { - let primitive_array = array.to_primitive(); + let primitive_array = array.clone().execute::(ctx)?; Ok(PrimitiveArray::new( filter .iter() @@ -65,7 +72,7 @@ pub fn filter_canonical_array(array: &ArrayRef, filter: &[bool]) -> VortexResult .into_array()) }), DType::Decimal(d, _) => { - let decimal_array = array.to_decimal(); + let decimal_array = array.clone().execute::(ctx)?; match_each_decimal_value_type!(decimal_array.values_type(), |D| { let buf = decimal_array.buffer::(); Ok(DecimalArray::new( @@ -82,7 +89,7 @@ pub fn filter_canonical_array(array: &ArrayRef, filter: &[bool]) -> VortexResult }) } DType::Utf8(_) | DType::Binary(_) => { - let utf8 = array.to_varbinview(); + let utf8 = array.clone().execute::(ctx)?; let values = utf8.with_iterator(|iter| { iter.zip(filter.iter()) .filter(|(_, f)| **f) @@ -92,10 +99,10 @@ pub fn filter_canonical_array(array: &ArrayRef, filter: &[bool]) -> VortexResult Ok(VarBinViewArray::from_iter(values, array.dtype().clone()).into_array()) } DType::Struct(..) => { - let struct_array = array.to_struct(); + let struct_array = array.clone().execute::(ctx)?; let filtered_children = struct_array .iter_unmasked_fields() - .map(|c| filter_canonical_array(c, filter)) + .map(|c| filter_canonical_array(c, filter, ctx)) .collect::>>()?; StructArray::try_new_with_dtype( @@ -113,7 +120,7 @@ pub fn filter_canonical_array(array: &ArrayRef, filter: &[bool]) -> VortexResult indices.push(idx); } } - take_canonical_array_non_nullable_indices(array, indices.as_slice()) + take_canonical_array_non_nullable_indices(array, indices.as_slice(), ctx) } d @ (DType::Null | DType::Extension(_) | DType::Variant(_)) => { unreachable!("DType {d} not supported for fuzzing") diff --git a/fuzz/src/array/mask.rs b/fuzz/src/array/mask.rs index 4c97200fa2d..49fc2ad40f9 100644 --- a/fuzz/src/array/mask.rs +++ b/fuzz/src/array/mask.rs @@ -5,8 +5,8 @@ use std::sync::Arc; use vortex_array::ArrayRef; use vortex_array::Canonical; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; -use vortex_array::ToCanonical; use vortex_array::arrays::BoolArray; use vortex_array::arrays::DecimalArray; use vortex_array::arrays::ExtensionArray; @@ -32,7 +32,7 @@ use vortex_mask::Mask; /// This needs to be coherent with applications of Mask. /// The result is always nullable. The result has the same length as self. #[inline] -pub fn mask_validity(validity: &Validity, mask: &Mask) -> Validity { +pub fn mask_validity(validity: &Validity, mask: &Mask, ctx: &mut ExecutionCtx) -> Validity { let out = match mask.bit_buffer() { AllOr::All => validity.clone().into_nullable(), AllOr::None => Validity::AllInvalid, @@ -42,7 +42,10 @@ pub fn mask_validity(validity: &Validity, mask: &Mask) -> Validity { Validity::from_bit_buffer(make_valid.clone(), Nullability::Nullable) } Validity::Array(is_valid) => { - let is_valid = is_valid.to_bool(); + let is_valid = is_valid + .clone() + .execute::(ctx) + .vortex_expect("validity to bool"); Validity::from_bit_buffer( is_valid.to_bit_buffer() & make_valid, Nullability::Nullable, @@ -58,18 +61,22 @@ pub fn mask_validity(validity: &Validity, mask: &Mask) -> Validity { /// Apply mask on the canonical form of the array to get a consistent baseline. /// This implementation manually applies the mask to each canonical type /// without using the mask_fn method, to serve as an independent baseline for testing. -pub fn mask_canonical_array(canonical: Canonical, mask: &Mask) -> VortexResult { +pub fn mask_canonical_array( + canonical: Canonical, + mask: &Mask, + ctx: &mut ExecutionCtx, +) -> VortexResult { Ok(match canonical { Canonical::Null(array) => { // Null arrays are already all invalid, masking has no effect array.into_array() } Canonical::Bool(array) => { - let new_validity = mask_validity(&array.validity()?, mask); + let new_validity = mask_validity(&array.validity()?, mask, ctx); BoolArray::new(array.to_bit_buffer(), new_validity).into_array() } Canonical::Primitive(array) => { - let new_validity = mask_validity(&array.validity()?, mask); + let new_validity = mask_validity(&array.validity()?, mask, ctx); PrimitiveArray::from_buffer_handle( array.buffer_handle().clone(), array.ptype(), @@ -78,14 +85,14 @@ pub fn mask_canonical_array(canonical: Canonical, mask: &Mask) -> VortexResult { - let new_validity = mask_validity(&array.validity()?, mask); + let new_validity = mask_validity(&array.validity()?, mask, ctx); match_each_decimal_value_type!(array.values_type(), |D| { DecimalArray::new(array.buffer::(), array.decimal_dtype(), new_validity) .into_array() }) } Canonical::VarBinView(array) => { - let new_validity = mask_validity(&array.validity()?, mask); + let new_validity = mask_validity(&array.validity()?, mask, ctx); VarBinViewArray::new_handle( array.views_handle().clone(), Arc::clone(array.data_buffers()), @@ -95,7 +102,7 @@ pub fn mask_canonical_array(canonical: Canonical, mask: &Mask) -> VortexResult { - let new_validity = mask_validity(&array.validity()?, mask); + let new_validity = mask_validity(&array.validity()?, mask, ctx); // SAFETY: Since we are only masking the validity and everything else comes from an // already valid `ListViewArray`, all of the invariants are still upheld. @@ -111,7 +118,7 @@ pub fn mask_canonical_array(canonical: Canonical, mask: &Mask) -> VortexResult { - let new_validity = mask_validity(&array.validity()?, mask); + let new_validity = mask_validity(&array.validity()?, mask, ctx); FixedSizeListArray::new( array.elements().clone(), array.list_size(), @@ -121,7 +128,7 @@ pub fn mask_canonical_array(canonical: Canonical, mask: &Mask) -> VortexResult { - let new_validity = mask_validity(&array.validity()?, mask); + let new_validity = mask_validity(&array.validity()?, mask, ctx); StructArray::try_new_with_dtype( array.unmasked_fields(), array.struct_fields().clone(), @@ -133,7 +140,8 @@ pub fn mask_canonical_array(canonical: Canonical, mask: &Mask) -> VortexResult { // Recursively mask the storage array - let masked_storage = mask_canonical_array(array.storage_array().to_canonical()?, mask) + let storage_canonical = array.storage_array().clone().execute::(ctx)?; + let masked_storage = mask_canonical_array(storage_canonical, mask, ctx) .vortex_expect("mask_canonical_array should succeed in fuzz test"); let ext_dtype = array @@ -147,7 +155,10 @@ pub fn mask_canonical_array(canonical: Canonical, mask: &Mask) -> VortexResult Canonical { + array + .into_array() + .execute::(&mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + } + #[test] fn test_mask_null_array() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = NullArray::new(5); let mask = Mask::from_iter([true, false, true, false, true]); - let result = mask_canonical_array(array.to_canonical().unwrap(), &mask).unwrap(); + let result = mask_canonical_array(canonical(array), &mask, &mut ctx).unwrap(); assert_eq!(result.len(), 5); // All values should still be null for i in 0..5 { - assert!(!result.is_valid(i).unwrap()); + assert!(!result.is_valid(i, &mut ctx).unwrap()); } } #[test] fn test_mask_bool_array() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = BoolArray::from_iter([true, false, true, false, true]); let mask = Mask::from_iter([false, true, true, false, true]); - let result = mask_canonical_array(array.to_canonical().unwrap(), &mask).unwrap(); + let result = mask_canonical_array(canonical(array), &mask, &mut ctx).unwrap(); let expected = BoolArray::from_iter([None, Some(false), Some(true), None, Some(true)]); assert_arrays_eq!(result, expected); @@ -191,10 +211,11 @@ mod tests { #[test] fn test_mask_primitive_array() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = PrimitiveArray::from_iter([1i32, 2, 3, 4, 5]); let mask = Mask::from_iter([true, false, true, false, true]); - let result = mask_canonical_array(array.to_canonical().unwrap(), &mask).unwrap(); + let result = mask_canonical_array(canonical(array), &mask, &mut ctx).unwrap(); let expected = PrimitiveArray::from_option_iter([Some(1i32), None, Some(3), None, Some(5)]); assert_arrays_eq!(result, expected); @@ -202,10 +223,11 @@ mod tests { #[test] fn test_mask_primitive_array_with_nulls() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = PrimitiveArray::from_option_iter([Some(1i32), None, Some(3), Some(4), None]); let mask = Mask::from_iter([false, true, true, false, true]); - let result = mask_canonical_array(array.to_canonical().unwrap(), &mask).unwrap(); + let result = mask_canonical_array(canonical(array), &mask, &mut ctx).unwrap(); let expected = PrimitiveArray::from_option_iter([None, None, Some(3i32), None, None]); assert_arrays_eq!(result, expected); @@ -213,6 +235,7 @@ mod tests { #[test] fn test_mask_decimal_array() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let dtype = DecimalDType::new(10, 2); let array = DecimalArray::from_option_iter( [Some(1i128), Some(2), Some(3), Some(4), Some(5)], @@ -220,7 +243,7 @@ mod tests { ); let mask = Mask::from_iter([true, true, false, true, true]); - let result = mask_canonical_array(array.to_canonical().unwrap(), &mask).unwrap(); + let result = mask_canonical_array(canonical(array), &mask, &mut ctx).unwrap(); let expected = DecimalArray::from_option_iter([Some(1i128), Some(2), None, Some(4), Some(5)], dtype); @@ -229,10 +252,11 @@ mod tests { #[test] fn test_mask_varbinview_array() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = VarBinViewArray::from_iter_str(["one", "two", "three", "four", "five"]); let mask = Mask::from_iter([false, true, false, true, false]); - let result = mask_canonical_array(array.to_canonical().unwrap(), &mask).unwrap(); + let result = mask_canonical_array(canonical(array), &mask, &mut ctx).unwrap(); let expected = VarBinViewArray::from_iter_nullable_str([None, Some("two"), None, Some("four"), None]); @@ -241,6 +265,7 @@ mod tests { #[test] fn test_mask_list_array() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let elements = PrimitiveArray::from_iter([1i32, 2, 3, 4, 5, 6]).into_array(); let offsets = PrimitiveArray::from_iter([0i32, 2, 4]).into_array(); let sizes = PrimitiveArray::from_iter([2i32, 2, 2]).into_array(); @@ -251,32 +276,34 @@ mod tests { let mask = Mask::from_iter([true, false, true]); - let result = mask_canonical_array(array.to_canonical().unwrap(), &mask).unwrap(); + let result = mask_canonical_array(canonical(array), &mask, &mut ctx).unwrap(); assert_eq!(result.len(), 3); - assert!(result.is_valid(0).unwrap()); - assert!(!result.is_valid(1).unwrap()); - assert!(result.is_valid(2).unwrap()); + assert!(result.is_valid(0, &mut ctx).unwrap()); + assert!(!result.is_valid(1, &mut ctx).unwrap()); + assert!(result.is_valid(2, &mut ctx).unwrap()); } #[test] fn test_mask_fixed_size_list_array() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let elements = PrimitiveArray::from_iter([1i32, 2, 3, 4, 5, 6]).into_array(); let array = FixedSizeListArray::try_new(elements, 2, Nullability::NonNullable.into(), 3).unwrap(); let mask = Mask::from_iter([false, true, false]); - let result = mask_canonical_array(array.to_canonical().unwrap(), &mask).unwrap(); + let result = mask_canonical_array(canonical(array), &mask, &mut ctx).unwrap(); assert_eq!(result.len(), 3); - assert!(!result.is_valid(0).unwrap()); - assert!(result.is_valid(1).unwrap()); - assert!(!result.is_valid(2).unwrap()); + assert!(!result.is_valid(0, &mut ctx).unwrap()); + assert!(result.is_valid(1, &mut ctx).unwrap()); + assert!(!result.is_valid(2, &mut ctx).unwrap()); } #[test] fn test_mask_struct_array() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let field1 = PrimitiveArray::from_iter([1i32, 2, 3]).into_array(); let field2 = PrimitiveArray::from_iter([4i32, 5, 6]).into_array(); let fields = vec![field1, field2]; @@ -291,20 +318,21 @@ mod tests { let mask = Mask::from_iter([true, false, true]); - let result = mask_canonical_array(array.to_canonical().unwrap(), &mask).unwrap(); + let result = mask_canonical_array(canonical(array), &mask, &mut ctx).unwrap(); assert_eq!(result.len(), 3); - assert!(result.is_valid(0).unwrap()); - assert!(!result.is_valid(1).unwrap()); - assert!(result.is_valid(2).unwrap()); + assert!(result.is_valid(0, &mut ctx).unwrap()); + assert!(!result.is_valid(1, &mut ctx).unwrap()); + assert!(result.is_valid(2, &mut ctx).unwrap()); } #[test] fn test_mask_all_false() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = PrimitiveArray::from_iter([1i32, 2, 3, 4, 5]); let mask = Mask::AllFalse(5); - let result = mask_canonical_array(array.to_canonical().unwrap(), &mask).unwrap(); + let result = mask_canonical_array(canonical(array), &mask, &mut ctx).unwrap(); let expected = PrimitiveArray::from_option_iter([None, None, None, None, None::]); assert_arrays_eq!(result, expected); @@ -312,10 +340,11 @@ mod tests { #[test] fn test_mask_all_true() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = PrimitiveArray::from_iter([1i32, 2, 3, 4, 5]); let mask = Mask::AllTrue(5); - let result = mask_canonical_array(array.to_canonical().unwrap(), &mask).unwrap(); + let result = mask_canonical_array(canonical(array), &mask, &mut ctx).unwrap(); let expected = PrimitiveArray::from_option_iter([Some(1i32), Some(2), Some(3), Some(4), Some(5)]); @@ -324,9 +353,10 @@ mod tests { #[test] fn test_mask_empty_array() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = PrimitiveArray::from_iter(Vec::::new()); for mask in [Mask::AllFalse(0), Mask::AllTrue(0)] { - let result = mask_canonical_array(array.to_canonical().unwrap(), &mask).unwrap(); + let result = mask_canonical_array(canonical(array.clone()), &mask, &mut ctx).unwrap(); assert_eq!(result.len(), 0); } } diff --git a/fuzz/src/array/mod.rs b/fuzz/src/array/mod.rs index b0dd2c2b242..3509ce4be5a 100644 --- a/fuzz/src/array/mod.rs +++ b/fuzz/src/array/mod.rs @@ -40,6 +40,7 @@ use strum::EnumIter; use strum::IntoEnumIterator; use tracing::debug; use vortex_array::ArrayRef; +use vortex_array::Canonical; use vortex_array::IntoArray; use vortex_array::VortexSessionExecute; use vortex_array::aggregate_fn::fns::min_max::MinMaxResult; @@ -48,6 +49,8 @@ use vortex_array::aggregate_fn::fns::sum::sum; use vortex_array::arrays::ConstantArray; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::arbitrary::ArbitraryArray; +use vortex_array::arrays::arbitrary::ArbitraryArrayConfig; +use vortex_array::arrays::arbitrary::ArbitraryWith; use vortex_array::builtins::ArrayBuiltins; use vortex_array::dtype::DType; use vortex_array::dtype::Nullability; @@ -66,6 +69,7 @@ use vortex_error::vortex_panic; use vortex_mask::Mask; use vortex_utils::aliases::hash_set::HashSet; +use crate::FUZZ_ARRAY_MAX_LEN; use crate::SESSION; use crate::error::Backtrace; use crate::error::VortexFuzzError; @@ -169,7 +173,14 @@ impl ExpectedValue { impl<'a> Arbitrary<'a> for FuzzArrayAction { fn arbitrary(u: &mut Unstructured<'a>) -> arbitrary::Result { - let array = ArbitraryArray::arbitrary(u)?.0; + let array = ArbitraryArray::arbitrary_with_config( + u, + &ArbitraryArrayConfig { + dtype: None, + len: 0..=FUZZ_ARRAY_MAX_LEN, + }, + )? + .0; let mut current_array = array.clone(); let mut ctx = SESSION.create_execution_ctx(); @@ -202,7 +213,7 @@ impl<'a> Arbitrary<'a> for FuzzArrayAction { ActionType::Slice => { let start = u.choose_index(current_array.len())?; let stop = u.int_in_range(start..=current_array.len())?; - current_array = slice_canonical_array(¤t_array, start, stop) + current_array = slice_canonical_array(¤t_array, start, stop, &mut ctx) .vortex_expect("slice_canonical_array should succeed in fuzz test"); ( @@ -218,7 +229,7 @@ impl<'a> Arbitrary<'a> for FuzzArrayAction { let indices = random_vec_in_range(u, 0, current_array.len() - 1)?; let nullable = indices.contains(&None); - current_array = take_canonical_array(¤t_array, &indices) + current_array = take_canonical_array(¤t_array, &indices, &mut ctx) .vortex_expect("take_canonical_array should succeed in fuzz test"); let indices_array = if nullable { PrimitiveArray::from_option_iter( @@ -236,7 +247,7 @@ impl<'a> Arbitrary<'a> for FuzzArrayAction { }; let compressed = BtrBlocksCompressor::default() - .compress(&indices_array) + .compress(&indices_array, &mut ctx) .vortex_expect("BtrBlocksCompressor compress should succeed in fuzz test"); ( Action::Take(compressed), @@ -250,7 +261,7 @@ impl<'a> Arbitrary<'a> for FuzzArrayAction { let scalar = if u.arbitrary()? { current_array - .scalar_at(u.choose_index(current_array.len())?) + .execute_scalar(u.choose_index(current_array.len())?, &mut ctx) .vortex_expect("scalar_at") } else { random_scalar(u, current_array.dtype())? @@ -260,7 +271,7 @@ impl<'a> Arbitrary<'a> for FuzzArrayAction { return Err(EmptyChoose); } - let sorted = sort_canonical_array(¤t_array) + let sorted = sort_canonical_array(¤t_array, &mut ctx) .vortex_expect("sort_canonical_array should succeed in fuzz test"); let side = if u.arbitrary()? { @@ -271,9 +282,10 @@ impl<'a> Arbitrary<'a> for FuzzArrayAction { ( Action::SearchSorted(scalar.clone(), side), ExpectedValue::Search( - search_sorted_canonical_array(&sorted, &scalar, side).vortex_expect( - "search_sorted_canonical_array should succeed in fuzz test", - ), + search_sorted_canonical_array(&sorted, &scalar, side, &mut ctx) + .vortex_expect( + "search_sorted_canonical_array should succeed in fuzz test", + ), ), ) } @@ -281,7 +293,7 @@ impl<'a> Arbitrary<'a> for FuzzArrayAction { let mask = (0..current_array.len()) .map(|_| bool::arbitrary(u)) .collect::>>()?; - current_array = filter_canonical_array(¤t_array, &mask) + current_array = filter_canonical_array(¤t_array, &mask, &mut ctx) .vortex_expect("filter_canonical_array should succeed in fuzz test"); ( Action::Filter(Mask::from_iter(mask)), @@ -291,7 +303,7 @@ impl<'a> Arbitrary<'a> for FuzzArrayAction { ActionType::Compare => { let scalar = if u.arbitrary()? { current_array - .scalar_at(u.choose_index(current_array.len())?) + .execute_scalar(u.choose_index(current_array.len())?, &mut ctx) .vortex_expect("scalar_at") } else { // We can compare arrays with different nullability @@ -300,7 +312,7 @@ impl<'a> Arbitrary<'a> for FuzzArrayAction { }; let op = u.arbitrary()?; - current_array = compare_canonical_array(¤t_array, &scalar, op); + current_array = compare_canonical_array(¤t_array, &scalar, op, &mut ctx); ( Action::Compare(scalar, op), ExpectedValue::Array(current_array.clone()), @@ -312,7 +324,7 @@ impl<'a> Arbitrary<'a> for FuzzArrayAction { { return Err(EmptyChoose); } - let Some(result) = cast_canonical_array(¤t_array, &to) + let Some(result) = cast_canonical_array(¤t_array, &to, &mut ctx) .vortex_expect("should fail to create array") else { return Err(EmptyChoose); @@ -327,24 +339,22 @@ impl<'a> Arbitrary<'a> for FuzzArrayAction { } // Sum - returns a scalar, does NOT update current_array (terminal operation) - let sum_result = sum_canonical_array( - current_array - .to_canonical() - .vortex_expect("to_canonical should succeed in fuzz test"), - &mut ctx, - ) - .vortex_expect("sum_canonical_array should succeed in fuzz test"); + let current_array_canonical = current_array + .clone() + .execute::(&mut ctx) + .vortex_expect("execute canonical should succeed in fuzz test"); + let sum_result = sum_canonical_array(current_array_canonical, &mut ctx) + .vortex_expect("sum_canonical_array should succeed in fuzz test"); (Action::Sum, ExpectedValue::Scalar(sum_result)) } ActionType::MinMax => { // MinMax - returns a scalar, does NOT update current_array (terminal operation) - let min_max_result = min_max_canonical_array( - current_array - .to_canonical() - .vortex_expect("to_canonical should succeed in fuzz test"), - &mut ctx, - ) - .vortex_expect("min_max_canonical_array should succeed in fuzz test"); + let current_array_canonical = current_array + .clone() + .execute::(&mut ctx) + .vortex_expect("execute canonical should succeed in fuzz test"); + let min_max_result = min_max_canonical_array(current_array_canonical, &mut ctx) + .vortex_expect("min_max_canonical_array should succeed in fuzz test"); (Action::MinMax, ExpectedValue::MinMax(min_max_result)) } ActionType::FillNull => { @@ -354,7 +364,7 @@ impl<'a> Arbitrary<'a> for FuzzArrayAction { } let fill_value = if u.arbitrary()? && !current_array.is_empty() { current_array - .scalar_at(u.choose_index(current_array.len())?) + .execute_scalar(u.choose_index(current_array.len())?, &mut ctx) .vortex_expect("scalar_at") } else { random_scalar( @@ -370,13 +380,13 @@ impl<'a> Arbitrary<'a> for FuzzArrayAction { } // Compute expected result on canonical form - let expected_result = fill_null_canonical_array( - current_array - .to_canonical() - .vortex_expect("to_canonical should succeed in fuzz test"), - &fill_value, - ) - .vortex_expect("fill_null_canonical_array should succeed in fuzz test"); + let current_array_canonical = current_array + .clone() + .execute::(&mut ctx) + .vortex_expect("execute canonical should succeed in fuzz test"); + let expected_result = + fill_null_canonical_array(current_array_canonical, &fill_value, &mut ctx) + .vortex_expect("fill_null_canonical_array should succeed in fuzz test"); // Update current_array to the result for chaining current_array = expected_result.clone(); ( @@ -391,11 +401,14 @@ impl<'a> Arbitrary<'a> for FuzzArrayAction { .collect::>>()?; // Compute expected result on canonical form + let current_array_canonical = current_array + .clone() + .execute::(&mut ctx) + .vortex_expect("execute canonical should succeed in fuzz test"); let expected_result = mask_canonical_array( - current_array - .to_canonical() - .vortex_expect("to_canonical should succeed in fuzz test"), + current_array_canonical, &Mask::from_iter(mask.clone()), + &mut ctx, ) .vortex_expect("mask_canonical_array should succeed in fuzz test"); // Update current_array to the result for chaining @@ -424,13 +437,13 @@ impl<'a> Arbitrary<'a> for FuzzArrayAction { let expected_scalars: Vec = indices_vec .iter() .map(|&idx| { - scalar_at_canonical_array( - current_array - .to_canonical() - .vortex_expect("to_canonical should succeed in fuzz test"), - idx, + let canonical = current_array + .clone() + .execute::(&mut ctx) + .vortex_expect("execute canonical should succeed in fuzz test"); + scalar_at_canonical_array(canonical, idx, &mut ctx).vortex_expect( + "scalar_at_canonical_array should succeed in fuzz test", ) - .vortex_expect("scalar_at_canonical_array should succeed in fuzz test") }) .collect(); @@ -538,14 +551,15 @@ fn random_action_from_list( /// Compress an array using the given strategy. #[cfg(feature = "zstd")] pub fn compress_array(array: &ArrayRef, strategy: CompressorStrategy) -> ArrayRef { + let mut ctx = SESSION.create_execution_ctx(); match strategy { CompressorStrategy::Default => BtrBlocksCompressor::default() - .compress(array) + .compress(array, &mut ctx) .vortex_expect("BtrBlocksCompressor compress should succeed in fuzz test"), CompressorStrategy::Compact => BtrBlocksCompressorBuilder::default() .with_compact() .build() - .compress(array) + .compress(array, &mut ctx) .vortex_expect("Compact compress should succeed in fuzz test"), } } @@ -554,7 +568,7 @@ pub fn compress_array(array: &ArrayRef, strategy: CompressorStrategy) -> ArrayRe #[cfg(not(feature = "zstd"))] pub fn compress_array(array: &ArrayRef, _strategy: CompressorStrategy) -> ArrayRef { BtrBlocksCompressor::default() - .compress(array) + .compress(array, &mut SESSION.create_execution_ctx()) .vortex_expect("BtrBlocksCompressor compress should succeed in fuzz test") } @@ -564,7 +578,7 @@ pub fn compress_array(array: &ArrayRef, _strategy: CompressorStrategy) -> ArrayR /// - `Ok(true)` - keep in corpus /// - `Ok(false)` - reject from corpus /// - `Err(_)` - a bug was found -#[allow(clippy::result_large_err)] +#[expect(clippy::result_large_err)] pub fn run_fuzz_action(fuzz_action: FuzzArrayAction) -> VortexFuzzResult { let FuzzArrayAction { array, actions } = fuzz_action; let mut current_array = array; @@ -582,8 +596,9 @@ pub fn run_fuzz_action(fuzz_action: FuzzArrayAction) -> VortexFuzzResult { match action { Action::Compress(strategy) => { let canonical = current_array - .to_canonical() - .vortex_expect("to_canonical should succeed in fuzz test"); + .clone() + .execute::(&mut ctx) + .vortex_expect("execute canonical should succeed in fuzz test"); current_array = compress_array(&canonical.into_array(), strategy); assert_array_eq(&expected.array(), ¤t_array, i)?; } @@ -603,7 +618,7 @@ pub fn run_fuzz_action(fuzz_action: FuzzArrayAction) -> VortexFuzzResult { assert_array_eq(&expected.array(), ¤t_array, i)?; } Action::SearchSorted(s, side) => { - let mut sorted = sort_canonical_array(¤t_array) + let mut sorted = sort_canonical_array(¤t_array, &mut ctx) .vortex_expect("sort_canonical_array should succeed in fuzz test"); if !current_array.is_canonical() { @@ -652,7 +667,7 @@ pub fn run_fuzz_action(fuzz_action: FuzzArrayAction) -> VortexFuzzResult { Action::MinMax => { let min_max_result = min_max(¤t_array, &mut ctx) .vortex_expect("min_max operation should succeed in fuzz test"); - assert_min_max_eq(&expected.min_max(), &min_max_result, i)?; + assert_min_max_eq(expected.min_max().as_ref(), min_max_result.as_ref(), i)?; } Action::FillNull(fill_value) => { current_array = current_array @@ -669,7 +684,9 @@ pub fn run_fuzz_action(fuzz_action: FuzzArrayAction) -> VortexFuzzResult { Action::ScalarAt(indices) => { let expected_scalars = expected.scalar_vec(); for (j, &idx) in indices.iter().enumerate() { - let scalar = current_array.scalar_at(idx).vortex_expect("scalar_at"); + let scalar = current_array + .execute_scalar(idx, &mut ctx) + .vortex_expect("scalar_at"); assert_scalar_eq(&expected_scalars[j], &scalar, i)?; } } @@ -678,7 +695,7 @@ pub fn run_fuzz_action(fuzz_action: FuzzArrayAction) -> VortexFuzzResult { Ok(true) // Keep in corpus } -#[allow(clippy::result_large_err)] +#[expect(clippy::result_large_err)] fn assert_search_sorted( array: ArrayRef, s: Scalar, @@ -705,7 +722,7 @@ fn assert_search_sorted( } /// Assert two arrays are equal. -#[allow(clippy::result_large_err)] +#[expect(clippy::result_large_err)] pub fn assert_array_eq(lhs: &ArrayRef, rhs: &ArrayRef, step: usize) -> VortexFuzzResult<()> { if lhs.dtype() != rhs.dtype() { return Err(VortexFuzzError::DTypeMismatch( @@ -726,9 +743,10 @@ pub fn assert_array_eq(lhs: &ArrayRef, rhs: &ArrayRef, step: usize) -> VortexFuz Backtrace::capture(), )); } + let mut ctx = SESSION.create_execution_ctx(); for idx in 0..lhs.len() { - let l = lhs.scalar_at(idx).vortex_expect("scalar_at"); - let r = rhs.scalar_at(idx).vortex_expect("scalar_at"); + let l = lhs.execute_scalar(idx, &mut ctx).vortex_expect("scalar_at"); + let r = rhs.execute_scalar(idx, &mut ctx).vortex_expect("scalar_at"); if l != r { return Err(VortexFuzzError::ArrayNotEqual( @@ -746,7 +764,7 @@ pub fn assert_array_eq(lhs: &ArrayRef, rhs: &ArrayRef, step: usize) -> VortexFuz } /// Assert two scalars are equal. -#[allow(clippy::result_large_err)] +#[expect(clippy::result_large_err)] pub fn assert_scalar_eq(lhs: &Scalar, rhs: &Scalar, step: usize) -> VortexFuzzResult<()> { if lhs != rhs { return Err(VortexFuzzError::ScalarMismatch( @@ -760,16 +778,16 @@ pub fn assert_scalar_eq(lhs: &Scalar, rhs: &Scalar, step: usize) -> VortexFuzzRe } /// Assert two min/max results are equal. -#[allow(clippy::result_large_err)] +#[expect(clippy::result_large_err)] pub fn assert_min_max_eq( - lhs: &Option, - rhs: &Option, + lhs: Option<&MinMaxResult>, + rhs: Option<&MinMaxResult>, step: usize, ) -> VortexFuzzResult<()> { if lhs != rhs { return Err(VortexFuzzError::MinMaxMismatch( - lhs.clone(), - rhs.clone(), + lhs.cloned(), + rhs.cloned(), step, Backtrace::capture(), )); diff --git a/fuzz/src/array/scalar_at.rs b/fuzz/src/array/scalar_at.rs index 91898390df6..e13ad75343d 100644 --- a/fuzz/src/array/scalar_at.rs +++ b/fuzz/src/array/scalar_at.rs @@ -4,6 +4,7 @@ use std::sync::Arc; use vortex_array::Canonical; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; use vortex_array::arrays::bool::BoolArrayExt; use vortex_array::arrays::extension::ExtensionArrayExt; @@ -22,9 +23,13 @@ use vortex_error::VortexResult; /// Baseline implementation of scalar_at that works on canonical arrays. /// This implementation manually extracts the scalar value from each canonical type /// without using the scalar_at method, to serve as an independent baseline for testing. -pub fn scalar_at_canonical_array(canonical: Canonical, index: usize) -> VortexResult { +pub fn scalar_at_canonical_array( + canonical: Canonical, + index: usize, + ctx: &mut ExecutionCtx, +) -> VortexResult { let canonical_ref = canonical.clone().into_array(); - if canonical_ref.is_invalid(index)? { + if canonical_ref.is_invalid(index, ctx)? { return Ok(Scalar::null(canonical_ref.dtype().clone())); } Ok(match canonical { @@ -52,12 +57,12 @@ pub fn scalar_at_canonical_array(canonical: Canonical, index: usize) -> VortexRe let list = array.list_elements_at(index)?; let children: Vec = (0..list.len()) .map(|i| { - scalar_at_canonical_array( - list.to_canonical() - .vortex_expect("to_canonical should succeed in fuzz test"), - i, - ) - .vortex_expect("scalar_at_canonical_array should succeed in fuzz test") + let canonical = list + .clone() + .execute::(ctx) + .vortex_expect("to_canonical should succeed in fuzz test"); + scalar_at_canonical_array(canonical, i, ctx) + .vortex_expect("scalar_at_canonical_array should succeed in fuzz test") }) .collect(); Scalar::list( @@ -70,12 +75,12 @@ pub fn scalar_at_canonical_array(canonical: Canonical, index: usize) -> VortexRe let list = array.fixed_size_list_elements_at(index)?; let children: Vec = (0..list.len()) .map(|i| { - scalar_at_canonical_array( - list.to_canonical() - .vortex_expect("to_canonical should succeed in fuzz test"), - i, - ) - .vortex_expect("scalar_at_canonical_array should succeed in fuzz test") + let canonical = list + .clone() + .execute::(ctx) + .vortex_expect("to_canonical should succeed in fuzz test"); + scalar_at_canonical_array(canonical, i, ctx) + .vortex_expect("scalar_at_canonical_array should succeed in fuzz test") }) .collect(); Scalar::fixed_size_list(list.dtype().clone(), children, array.dtype().nullability()) @@ -84,20 +89,19 @@ pub fn scalar_at_canonical_array(canonical: Canonical, index: usize) -> VortexRe let field_scalars: Vec = array .iter_unmasked_fields() .map(|field| { - scalar_at_canonical_array( - field - .to_canonical() - .vortex_expect("to_canonical should succeed in fuzz test"), - index, - ) - .vortex_expect("scalar_at_canonical_array should succeed in fuzz test") + let canonical = field + .clone() + .execute::(ctx) + .vortex_expect("to_canonical should succeed in fuzz test"); + scalar_at_canonical_array(canonical, index, ctx) + .vortex_expect("scalar_at_canonical_array should succeed in fuzz test") }) .collect(); Scalar::struct_(array.dtype().clone(), field_scalars) } Canonical::Extension(array) => { - let storage_scalar = - scalar_at_canonical_array(array.storage_array().to_canonical()?, index)?; + let storage_canonical = array.storage_array().clone().execute::(ctx)?; + let storage_scalar = scalar_at_canonical_array(storage_canonical, index, ctx)?; Scalar::extension_ref(array.ext_dtype().clone(), storage_scalar) } Canonical::Variant(_) => unreachable!("Variant arrays are not fuzzed"), diff --git a/fuzz/src/array/search_sorted.rs b/fuzz/src/array/search_sorted.rs index d22b583a39b..a35bbc9e4e6 100644 --- a/fuzz/src/array/search_sorted.rs +++ b/fuzz/src/array/search_sorted.rs @@ -5,8 +5,12 @@ use std::cmp::Ordering; use std::fmt::Debug; use vortex_array::ArrayRef; -use vortex_array::ToCanonical; +use vortex_array::ExecutionCtx; use vortex_array::accessor::ArrayAccessor; +use vortex_array::arrays::BoolArray; +use vortex_array::arrays::DecimalArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::VarBinViewArray; use vortex_array::arrays::bool::BoolArrayExt; use vortex_array::dtype::DType; use vortex_array::dtype::NativePType; @@ -60,11 +64,16 @@ pub fn search_sorted_canonical_array( array: &ArrayRef, scalar: &Scalar, side: SearchSortedSide, + ctx: &mut ExecutionCtx, ) -> VortexResult { match array.dtype() { DType::Bool(_) => { - let bool_array = array.to_bool(); - let validity = bool_array.validity_mask()?.to_bit_buffer(); + let bool_array = array.clone().execute::(ctx)?; + let validity = bool_array + .as_ref() + .validity()? + .execute_mask(bool_array.as_ref().len(), ctx)? + .to_bit_buffer(); let opt_values = bool_array .to_bit_buffer() .iter() @@ -75,8 +84,12 @@ pub fn search_sorted_canonical_array( SearchNullableSlice(opt_values).search_sorted(&Some(to_find), side) } DType::Primitive(p, _) => { - let primitive_array = array.to_primitive(); - let validity = primitive_array.validity_mask()?.to_bit_buffer(); + let primitive_array = array.clone().execute::(ctx)?; + let validity = primitive_array + .as_ref() + .validity()? + .execute_mask(primitive_array.as_ref().len(), ctx)? + .to_bit_buffer(); match_each_native_ptype!(p, |P| { let opt_values = primitive_array .as_slice::

() @@ -90,8 +103,12 @@ pub fn search_sorted_canonical_array( }) } DType::Decimal(d, _) => { - let decimal_array = array.to_decimal(); - let validity = decimal_array.validity_mask()?.to_bit_buffer(); + let decimal_array = array.clone().execute::(ctx)?; + let validity = decimal_array + .as_ref() + .validity()? + .execute_mask(decimal_array.as_ref().len(), ctx)? + .to_bit_buffer(); match_each_decimal_value_type!(decimal_array.values_type(), |D| { let buf = decimal_array.buffer::(); let opt_values = buf @@ -115,7 +132,7 @@ pub fn search_sorted_canonical_array( }) } DType::Utf8(_) | DType::Binary(_) => { - let utf8 = array.to_varbinview(); + let utf8 = array.clone().execute::(ctx)?; let opt_values = utf8.with_iterator(|iter| iter.map(|v| v.map(|u| u.to_vec())).collect::>()); let to_find = if matches!(array.dtype(), DType::Utf8(_)) { @@ -127,7 +144,7 @@ pub fn search_sorted_canonical_array( } DType::Struct(..) | DType::List(..) | DType::FixedSizeList(..) => { let scalar_vals = (0..array.len()) - .map(|i| array.scalar_at(i)) + .map(|i| array.execute_scalar(i, ctx)) .collect::>>()?; scalar_vals.search_sorted(&scalar.cast(array.dtype())?, side) } diff --git a/fuzz/src/array/slice.rs b/fuzz/src/array/slice.rs index 65952ee49da..3138d644147 100644 --- a/fuzz/src/array/slice.rs +++ b/fuzz/src/array/slice.rs @@ -2,8 +2,8 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors use vortex_array::ArrayRef; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; -use vortex_array::ToCanonical; use vortex_array::accessor::ArrayAccessor; use vortex_array::arrays::BoolArray; use vortex_array::arrays::DecimalArray; @@ -22,14 +22,17 @@ use vortex_array::match_each_native_ptype; use vortex_array::validity::Validity; use vortex_error::VortexResult; -#[allow(clippy::unnecessary_fallible_conversions)] pub fn slice_canonical_array( array: &ArrayRef, start: usize, stop: usize, + ctx: &mut ExecutionCtx, ) -> VortexResult { let validity = if array.dtype().is_nullable() { - let bool_buff = array.validity_mask()?.to_bit_buffer(); + let bool_buff = array + .validity()? + .execute_mask(array.len(), ctx)? + .to_bit_buffer(); Validity::from(bool_buff.slice(start..stop)) } else { Validity::NonNullable @@ -37,12 +40,12 @@ pub fn slice_canonical_array( match array.dtype() { DType::Bool(_) => { - let bool_array = array.to_bool(); + let bool_array = array.clone().execute::(ctx)?; let sliced_bools = bool_array.to_bit_buffer().slice(start..stop); Ok(BoolArray::new(sliced_bools, validity).into_array()) } DType::Primitive(p, _) => { - let primitive_array = array.to_primitive(); + let primitive_array = array.clone().execute::(ctx)?; match_each_native_ptype!(p, |P| { Ok(PrimitiveArray::new( primitive_array.to_buffer::

().slice(start..stop), @@ -52,7 +55,7 @@ pub fn slice_canonical_array( }) } DType::Utf8(_) | DType::Binary(_) => { - let utf8 = array.to_varbinview(); + let utf8 = array.clone().execute::(ctx)?; let values = utf8.with_iterator(|iter| iter.map(|v| v.map(|u| u.to_vec())).collect::>()); Ok(VarBinViewArray::from_iter( @@ -62,10 +65,10 @@ pub fn slice_canonical_array( .into_array()) } DType::Struct(..) => { - let struct_array = array.to_struct(); + let struct_array = array.clone().execute::(ctx)?; let sliced_children = struct_array .iter_unmasked_fields() - .map(|c| slice_canonical_array(c, start, stop)) + .map(|c| slice_canonical_array(c, start, stop, ctx)) .collect::>>()?; StructArray::try_new_with_dtype( sliced_children, @@ -76,10 +79,10 @@ pub fn slice_canonical_array( .map(|a| a.into_array()) } DType::List(..) => { - let list_array = array.to_listview(); + let list_array = array.clone().execute::(ctx)?; - let offsets = slice_canonical_array(list_array.offsets(), start, stop)?; - let sizes = slice_canonical_array(list_array.sizes(), start, stop)?; + let offsets = slice_canonical_array(list_array.offsets(), start, stop, ctx)?; + let sizes = slice_canonical_array(list_array.sizes(), start, stop, ctx)?; // Since the list view elements can be stored out of order, we cannot slice it. let elements = list_array.elements().clone(); @@ -94,17 +97,21 @@ pub fn slice_canonical_array( .into_array()) } DType::FixedSizeList(..) => { - let fsl_array = array.to_fixed_size_list(); + let fsl_array = array.clone().execute::(ctx)?; let list_size = fsl_array.list_size() as usize; - let elements = - slice_canonical_array(fsl_array.elements(), start * list_size, stop * list_size)?; + let elements = slice_canonical_array( + fsl_array.elements(), + start * list_size, + stop * list_size, + ctx, + )?; let new_len = stop - start; FixedSizeListArray::try_new(elements, fsl_array.list_size(), validity, new_len) .map(|a| a.into_array()) } DType::Decimal(decimal_dtype, _) => { - let decimal_array = array.to_decimal(); + let decimal_array = array.clone().execute::(ctx)?; Ok( match_each_decimal_value_type!(decimal_array.values_type(), |D| { DecimalArray::new( diff --git a/fuzz/src/array/sort.rs b/fuzz/src/array/sort.rs index 6c07c76c20d..a2a70c8b167 100644 --- a/fuzz/src/array/sort.rs +++ b/fuzz/src/array/sort.rs @@ -4,8 +4,8 @@ use std::cmp::Ordering; use vortex_array::ArrayRef; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; -use vortex_array::ToCanonical; use vortex_array::accessor::ArrayAccessor; use vortex_array::arrays::BoolArray; use vortex_array::arrays::DecimalArray; @@ -21,27 +21,41 @@ use vortex_error::VortexResult; use crate::array::take_canonical_array_non_nullable_indices; -pub fn sort_canonical_array(array: &ArrayRef) -> VortexResult { +pub fn sort_canonical_array(array: &ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { match array.dtype() { DType::Bool(_) => { - let bool_array = array.to_bool(); + let bool_array = array.clone().execute::(ctx)?; let mut opt_values = bool_array .to_bit_buffer() .iter() - .zip(bool_array.validity_mask()?.to_bit_buffer().iter()) + .zip( + bool_array + .as_ref() + .validity()? + .execute_mask(bool_array.as_ref().len(), ctx)? + .to_bit_buffer() + .iter(), + ) .map(|(b, v)| v.then_some(b)) .collect::>(); opt_values.sort(); Ok(BoolArray::from_iter(opt_values).into_array()) } DType::Primitive(p, _) => { - let primitive_array = array.to_primitive(); + let primitive_array = array.clone().execute::(ctx)?; match_each_native_ptype!(p, |P| { let mut opt_values = primitive_array .as_slice::

() .iter() .copied() - .zip(primitive_array.validity_mask()?.to_bit_buffer().iter()) + .zip( + primitive_array + .as_ref() + .validity()? + .execute_mask(primitive_array.as_ref().len(), ctx)? + .to_bit_buffer() + .iter(), + ) .map(|(p, v)| v.then_some(p)) .collect::>(); sort_primitive_slice(&mut opt_values); @@ -49,14 +63,21 @@ pub fn sort_canonical_array(array: &ArrayRef) -> VortexResult { }) } DType::Decimal(d, _) => { - let decimal_array = array.to_decimal(); + let decimal_array = array.clone().execute::(ctx)?; match_each_decimal_value_type!(decimal_array.values_type(), |D| { let buf = decimal_array.buffer::(); let mut opt_values = buf .as_slice() .iter() .copied() - .zip(decimal_array.validity_mask()?.to_bit_buffer().iter()) + .zip( + decimal_array + .as_ref() + .validity()? + .execute_mask(decimal_array.as_ref().len(), ctx)? + .to_bit_buffer() + .iter(), + ) .map(|(p, v)| v.then_some(p)) .collect::>(); opt_values.sort(); @@ -64,7 +85,7 @@ pub fn sort_canonical_array(array: &ArrayRef) -> VortexResult { }) } DType::Utf8(_) | DType::Binary(_) => { - let utf8 = array.to_varbinview(); + let utf8 = array.clone().execute::(ctx)?; let mut opt_values = utf8.with_iterator(|iter| iter.map(|v| v.map(|u| u.to_vec())).collect::>()); opt_values.sort(); @@ -73,13 +94,12 @@ pub fn sort_canonical_array(array: &ArrayRef) -> VortexResult { DType::Struct(..) | DType::List(..) | DType::FixedSizeList(..) => { let mut sort_indices = (0..array.len()).collect::>(); sort_indices.sort_by(|a, b| { - array - .scalar_at(*a) - .vortex_expect("scalar_at") - .partial_cmp(&array.scalar_at(*b).vortex_expect("scalar_at")) + let lhs = array.execute_scalar(*a, ctx).vortex_expect("scalar_at"); + let rhs = array.execute_scalar(*b, ctx).vortex_expect("scalar_at"); + lhs.partial_cmp(&rhs) .vortex_expect("must be a valid comparison") }); - take_canonical_array_non_nullable_indices(array, &sort_indices) + take_canonical_array_non_nullable_indices(array, &sort_indices, ctx) } d @ (DType::Null | DType::Extension(_) | DType::Variant(_)) => { unreachable!("DType {d} not supported for fuzzing") diff --git a/fuzz/src/array/take.rs b/fuzz/src/array/take.rs index a113829077b..ea1c8b8071c 100644 --- a/fuzz/src/array/take.rs +++ b/fuzz/src/array/take.rs @@ -2,8 +2,8 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors use vortex_array::ArrayRef; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; -use vortex_array::ToCanonical; use vortex_array::accessor::ArrayAccessor; use vortex_array::arrays::BoolArray; use vortex_array::arrays::DecimalArray; @@ -28,6 +28,7 @@ use vortex_error::VortexResult; pub fn take_canonical_array_non_nullable_indices( array: &ArrayRef, indices: &[usize], + ctx: &mut ExecutionCtx, ) -> VortexResult { take_canonical_array( array, @@ -36,14 +37,22 @@ pub fn take_canonical_array_non_nullable_indices( .map(|i| Some(*i)) .collect::>() .as_slice(), + ctx, ) } -pub fn take_canonical_array(array: &ArrayRef, indices: &[Option]) -> VortexResult { +pub fn take_canonical_array( + array: &ArrayRef, + indices: &[Option], + ctx: &mut ExecutionCtx, +) -> VortexResult { let nullable: Nullability = indices.contains(&None).into(); let validity = if array.dtype().is_nullable() || nullable == Nullability::Nullable { - let validity_idx = array.validity_mask()?.to_bit_buffer(); + let validity_idx = array + .validity()? + .execute_mask(array.len(), ctx)? + .to_bit_buffer(); Validity::from_iter( indices @@ -59,7 +68,7 @@ pub fn take_canonical_array(array: &ArrayRef, indices: &[Option]) -> Vort match array.dtype() { DType::Bool(_) => { - let bool_array = array.to_bool(); + let bool_array = array.clone().execute::(ctx)?; let vec_values = bool_array.to_bit_buffer().iter().collect::>(); Ok(BoolArray::new( indices_slice_non_opt @@ -71,7 +80,7 @@ pub fn take_canonical_array(array: &ArrayRef, indices: &[Option]) -> Vort .into_array()) } DType::Primitive(p, _) => { - let primitive_array = array.to_primitive(); + let primitive_array = array.clone().execute::(ctx)?; match_each_native_ptype!(p, |P| { Ok(take_primitive::

( primitive_array, @@ -81,7 +90,7 @@ pub fn take_canonical_array(array: &ArrayRef, indices: &[Option]) -> Vort }) } DType::Decimal(d, _) => { - let decimal_array = array.to_decimal(); + let decimal_array = array.clone().execute::(ctx)?; match_each_decimal_value_type!(decimal_array.values_type(), |D| { Ok(take_decimal::( @@ -93,7 +102,7 @@ pub fn take_canonical_array(array: &ArrayRef, indices: &[Option]) -> Vort }) } DType::Utf8(_) | DType::Binary(_) => { - let utf8 = array.to_varbinview(); + let utf8 = array.clone().execute::(ctx)?; let values = utf8.with_iterator(|iter| iter.map(|v| v.map(|u| u.to_vec())).collect::>()); Ok(VarBinViewArray::from_iter( @@ -105,10 +114,10 @@ pub fn take_canonical_array(array: &ArrayRef, indices: &[Option]) -> Vort .into_array()) } DType::Struct(..) => { - let struct_array = array.to_struct(); + let struct_array = array.clone().execute::(ctx)?; let taken_children = struct_array .iter_unmasked_fields() - .map(|c| take_canonical_array_non_nullable_indices(c, indices_slice_non_opt)) + .map(|c| take_canonical_array_non_nullable_indices(c, indices_slice_non_opt, ctx)) .collect::>>()?; StructArray::try_new( @@ -128,7 +137,7 @@ pub fn take_canonical_array(array: &ArrayRef, indices: &[Option]) -> Vort if let Some(idx) = idx { builder.append_scalar( &array - .scalar_at(*idx)? + .execute_scalar(*idx, ctx)? .cast(&array.dtype().union_nullability(nullable)) .vortex_expect("cannot cast scalar nullability"), )?; diff --git a/fuzz/src/compress.rs b/fuzz/src/compress.rs index 9ca6fadb348..1284a7be45e 100644 --- a/fuzz/src/compress.rs +++ b/fuzz/src/compress.rs @@ -9,11 +9,15 @@ use arbitrary::Arbitrary; use arbitrary::Unstructured; use vortex_array::ArrayRef; +use vortex_array::Canonical; use vortex_array::IntoArray; +use vortex_array::VortexSessionExecute; use vortex_array::arrays::constant::ArbitraryConstantArray; use vortex_array::arrays::dict::ArbitraryDictArray; use vortex_runend::ArbitraryRunEndArray; +use crate::SESSION; + /// Which compressed encoding to generate. #[derive(Debug, Clone, Copy)] pub enum EncodingKind { @@ -59,7 +63,7 @@ impl<'a> Arbitrary<'a> for FuzzCompressRoundtrip { /// - `Ok(true)` - keep in corpus /// - `Ok(false)` - reject from corpus /// - `Err(_)` - a bug was found -#[allow(clippy::result_large_err)] +#[expect(clippy::result_large_err)] pub fn run_compress_roundtrip(fuzz: FuzzCompressRoundtrip) -> crate::error::VortexFuzzResult { use crate::error::Backtrace; use crate::error::VortexFuzzError; @@ -70,8 +74,10 @@ pub fn run_compress_roundtrip(fuzz: FuzzCompressRoundtrip) -> crate::error::Vort let original_len = array.len(); let original_dtype = array.dtype().clone(); + let mut ctx = SESSION.create_execution_ctx(); + // Try to canonicalize - this is the main thing we're testing - let canonical = match array.to_canonical() { + let canonical = match array.clone().execute::(&mut ctx) { Ok(c) => c, Err(e) => { // Canonicalization failed - this is a bug diff --git a/fuzz/src/file/mod.rs b/fuzz/src/file/mod.rs index bcd6ac0c704..f4fdc6d6ce1 100644 --- a/fuzz/src/file/mod.rs +++ b/fuzz/src/file/mod.rs @@ -5,10 +5,13 @@ use arbitrary::Arbitrary; use arbitrary::Unstructured; use vortex_array::ArrayRef; use vortex_array::arrays::arbitrary::ArbitraryArray; +use vortex_array::arrays::arbitrary::ArbitraryArrayConfig; +use vortex_array::arrays::arbitrary::ArbitraryWith; use vortex_array::expr::Expression; use vortex_array::expr::arbitrary::filter_expr; use vortex_array::expr::arbitrary::projection_expr; +use crate::FUZZ_FILE_ARRAY_MAX_LEN; use crate::array::CompressorStrategy; #[derive(Debug)] @@ -21,7 +24,14 @@ pub struct FuzzFileAction { impl<'a> Arbitrary<'a> for FuzzFileAction { fn arbitrary(u: &mut Unstructured<'a>) -> arbitrary::Result { - let array = ArbitraryArray::arbitrary(u)?.0; + let array = ArbitraryArray::arbitrary_with_config( + u, + &ArbitraryArrayConfig { + dtype: None, + len: 0..=FUZZ_FILE_ARRAY_MAX_LEN, + }, + )? + .0; let dtype = array.dtype().clone(); Ok(FuzzFileAction { array, diff --git a/fuzz/src/fsst_like.rs b/fuzz/src/fsst_like.rs index b962c4244c8..5ca10af310a 100644 --- a/fuzz/src/fsst_like.rs +++ b/fuzz/src/fsst_like.rs @@ -91,7 +91,7 @@ impl<'a> Arbitrary<'a> for FuzzFsstLike { /// - `Ok(true)` — keep in corpus /// - `Ok(false)` — reject (e.g. too few strings) /// - `Err(_)` — mismatch found (bug) -#[allow(clippy::result_large_err)] +#[expect(clippy::result_large_err)] pub fn run_fsst_like_fuzz(fuzz: FuzzFsstLike) -> VortexFuzzResult { let FuzzFsstLike { strings, @@ -113,8 +113,14 @@ pub fn run_fsst_like_fuzz(fuzz: FuzzFsstLike) -> VortexFuzzResult { // Train FSST compressor and compress. let compressor = fsst_train_compressor(&varbin); - let fsst_array: FSSTArray = - fsst_compress(varbin.clone(), varbin.len(), varbin.dtype(), &compressor); + let mut ctx = SESSION.create_execution_ctx(); + let fsst_array: FSSTArray = fsst_compress( + varbin.clone(), + varbin.len(), + varbin.dtype(), + &compressor, + &mut ctx, + ); let opts = LikeOptions { negated, diff --git a/fuzz/src/gpu/mod.rs b/fuzz/src/gpu/mod.rs index 6d762813000..fa6ef110759 100644 --- a/fuzz/src/gpu/mod.rs +++ b/fuzz/src/gpu/mod.rs @@ -10,7 +10,10 @@ use arbitrary::Arbitrary; use arbitrary::Result; use arbitrary::Unstructured; use vortex_array::ArrayRef; +use vortex_array::Canonical; use vortex_array::IntoArray; +use vortex_array::LEGACY_SESSION; +use vortex_array::VortexSessionExecute; use vortex_array::arrays::dict::ArbitraryDictArray; use vortex_array::dtype::Nullability; use vortex_array::dtype::PType; @@ -93,7 +96,6 @@ fn arbitrary_gpu_primitive_dtype(u: &mut Unstructured) -> Result VortexFuzzResult { use vortex_cuda::CanonicalCudaExt; use vortex_cuda::CudaSession; @@ -111,7 +113,10 @@ pub async fn run_compress_gpu(fuzz: FuzzCompressGpu) -> VortexFuzzResult { let original_len = array.len(); - let cpu_canonical = match array.to_canonical() { + let cpu_canonical = match array + .clone() + .execute::(&mut LEGACY_SESSION.create_execution_ctx()) + { Ok(c) => c, Err(e) => { return Err(VortexFuzzError::VortexError(e, Backtrace::capture())); @@ -160,10 +165,10 @@ pub async fn run_compress_gpu(fuzz: FuzzCompressGpu) -> VortexFuzzResult { for i in 0..original_len { let cpu_scalar = cpu_array - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .map_err(|e| VortexFuzzError::VortexError(e, Backtrace::capture()))?; let gpu_scalar = gpu_array - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .map_err(|e| VortexFuzzError::VortexError(e, Backtrace::capture()))?; if cpu_scalar != gpu_scalar { diff --git a/fuzz/src/lib.rs b/fuzz/src/lib.rs index f6712e3540c..b0abf660045 100644 --- a/fuzz/src/lib.rs +++ b/fuzz/src/lib.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::use_debug)] +#![expect(clippy::use_debug)] mod array; pub mod compress; @@ -32,6 +32,9 @@ pub use gpu::FuzzCompressGpu; #[cfg(feature = "cuda")] pub use gpu::run_compress_gpu; +pub const FUZZ_ARRAY_MAX_LEN: usize = 2048; +pub const FUZZ_FILE_ARRAY_MAX_LEN: usize = 16_384; + // Runtime initialization - platform-specific #[cfg(not(target_arch = "wasm32"))] mod native_runtime { diff --git a/java/build.gradle.kts b/java/build.gradle.kts index 48c1ed3349f..bb73c481776 100644 --- a/java/build.gradle.kts +++ b/java/build.gradle.kts @@ -6,8 +6,9 @@ import net.ltgt.gradle.errorprone.errorprone plugins { id("com.diffplug.spotless") version "8.4.0" id("com.palantir.git-version") version "5.0.0" + id("com.palantir.java-format") version "2.90.0" id("net.ltgt.errorprone") version "5.1.0" apply false - id("com.google.protobuf") version "0.9.6" apply false + id("com.google.protobuf") version "0.10.0" apply false id("com.vanniktech.maven.publish") version "0.36.0" apply false } @@ -39,8 +40,13 @@ allprojects { spotless { java { - palantirJavaFormat() + palantirJavaFormat().formatJavadoc(true) licenseHeaderFile("${rootProject.projectDir}/.spotless/java-license-header.txt") + removeUnusedImports() + forbidWildcardImports() + importOrder("") + trimTrailingWhitespace() + leadingTabsToSpaces(4) targetExclude("**/generated/**") targetExcludeIfContentContains("// spotless:disabled") } @@ -52,6 +58,7 @@ allprojects { // ignore protobuf generated files options.errorprone.excludedPaths = ".*/build/generated/.*" options.release = 17 + options.compilerArgs.add("-Werror") options.generatedSourceOutputDirectory = projectDir.resolve("generated_src") } diff --git a/java/gradle/libs.versions.toml b/java/gradle/libs.versions.toml index c881e141de9..67b3155c80c 100644 --- a/java/gradle/libs.versions.toml +++ b/java/gradle/libs.versions.toml @@ -2,20 +2,20 @@ # SPDX-FileCopyrightText: Copyright the Vortex contributors [versions] -arrow = "18.3.0" +arrow = "19.0.0" errorprone = "2.36.0" -guava = "33.5.0-jre" +guava = "33.6.0-jre" immutables = "2.12.1" junit-jupiter = "6.0.3" logback = "1.5.32" -netty = "4.2.7.Final" +netty = "4.2.12.Final" nopen = "1.0.1" -protobuf = "4.33.5" +protobuf = "4.34.1" slf4j = "2.0.17" spark3 = "3.5.8" spark4 = "4.1.1" -s3mock = "4.11.0" -testcontainers-jupiter = "1.20.4" +s3mock = "5.0.0" +testcontainers-jupiter = "1.21.4" [libraries] arrow-c-data = { module = "org.apache.arrow:arrow-c-data", version.ref = "arrow" } diff --git a/java/gradle/wrapper/gradle-wrapper.properties b/java/gradle/wrapper/gradle-wrapper.properties index c61a118f7dd..cbcbf0b9fae 100644 --- a/java/gradle/wrapper/gradle-wrapper.properties +++ b/java/gradle/wrapper/gradle-wrapper.properties @@ -5,3 +5,4 @@ networkTimeout=10000 validateDistributionUrl=true zipStoreBase=GRADLE_USER_HOME zipStorePath=wrapper/dists + diff --git a/java/testfiles/Cargo.lock b/java/testfiles/Cargo.lock deleted file mode 100644 index 190904c651f..00000000000 --- a/java/testfiles/Cargo.lock +++ /dev/null @@ -1,2864 +0,0 @@ -# This file is automatically @generated by Cargo. -# It is not intended for manual editing. -version = 4 - -[[package]] -name = "ahash" -version = "0.8.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75" -dependencies = [ - "cfg-if", - "const-random", - "getrandom 0.3.4", - "once_cell", - "version_check", - "zerocopy", -] - -[[package]] -name = "aho-corasick" -version = "1.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301" -dependencies = [ - "memchr", -] - -[[package]] -name = "allocator-api2" -version = "0.2.21" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "683d7910e743518b0e34f1186f92494becacb047c7b6bf616c96772180fef923" - -[[package]] -name = "android_system_properties" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311" -dependencies = [ - "libc", -] - -[[package]] -name = "anyhow" -version = "1.0.102" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c" - -[[package]] -name = "arcref" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "28f6098a1e8ab66ff91324cce8fea6643101882cf7d09c85acdb1485ecf61e29" - -[[package]] -name = "arrayref" -version = "0.3.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "76a2e8124351fda1ef8aaaa3bbd7ebbcb486bbcd4225aca0aa0d84bb2db8fecb" - -[[package]] -name = "arrow-arith" -version = "58.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ced5406f8b720cc0bc3aa9cf5758f93e8593cda5490677aa194e4b4b383f9a59" -dependencies = [ - "arrow-array", - "arrow-buffer", - "arrow-data", - "arrow-schema", - "chrono", - "num-traits", -] - -[[package]] -name = "arrow-array" -version = "58.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "772bd34cacdda8baec9418d80d23d0fb4d50ef0735685bd45158b83dfeb6e62d" -dependencies = [ - "ahash", - "arrow-buffer", - "arrow-data", - "arrow-schema", - "chrono", - "half", - "hashbrown 0.16.1", - "num-complex", - "num-integer", - "num-traits", -] - -[[package]] -name = "arrow-buffer" -version = "58.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "898f4cf1e9598fdb77f356fdf2134feedfd0ee8d5a4e0a5f573e7d0aec16baa4" -dependencies = [ - "bytes", - "half", - "num-bigint", - "num-traits", -] - -[[package]] -name = "arrow-cast" -version = "58.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b0127816c96533d20fc938729f48c52d3e48f99717e7a0b5ade77d742510736d" -dependencies = [ - "arrow-array", - "arrow-buffer", - "arrow-data", - "arrow-ord", - "arrow-schema", - "arrow-select", - "atoi", - "base64", - "chrono", - "half", - "lexical-core", - "num-traits", - "ryu", -] - -[[package]] -name = "arrow-data" -version = "58.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "42d10beeab2b1c3bb0b53a00f7c944a178b622173a5c7bcabc3cb45d90238df4" -dependencies = [ - "arrow-buffer", - "arrow-schema", - "half", - "num-integer", - "num-traits", -] - -[[package]] -name = "arrow-ord" -version = "58.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "763a7ba279b20b52dad300e68cfc37c17efa65e68623169076855b3a9e941ca5" -dependencies = [ - "arrow-array", - "arrow-buffer", - "arrow-data", - "arrow-schema", - "arrow-select", -] - -[[package]] -name = "arrow-schema" -version = "58.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c30a1365d7a7dc50cc847e54154e6af49e4c4b0fddc9f607b687f29212082743" -dependencies = [ - "bitflags", -] - -[[package]] -name = "arrow-select" -version = "58.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "78694888660a9e8ac949853db393af2a8b8fc82c19ce333132dfa2e72cc1a7fe" -dependencies = [ - "ahash", - "arrow-array", - "arrow-buffer", - "arrow-data", - "arrow-schema", - "num-traits", -] - -[[package]] -name = "arrow-string" -version = "58.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "61e04a01f8bb73ce54437514c5fd3ee2aa3e8abe4c777ee5cc55853b1652f79e" -dependencies = [ - "arrow-array", - "arrow-buffer", - "arrow-data", - "arrow-schema", - "arrow-select", - "memchr", - "num-traits", - "regex", - "regex-syntax", -] - -[[package]] -name = "async-channel" -version = "2.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "924ed96dd52d1b75e9c1a3e6275715fd320f5f9439fb5a4a11fa51f4221158d2" -dependencies = [ - "concurrent-queue", - "event-listener-strategy", - "futures-core", - "pin-project-lite", -] - -[[package]] -name = "async-executor" -version = "1.14.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c96bf972d85afc50bf5ab8fe2d54d1586b4e0b46c97c50a0c9e71e2f7bcd812a" -dependencies = [ - "async-task", - "concurrent-queue", - "fastrand", - "futures-lite", - "pin-project-lite", - "slab", -] - -[[package]] -name = "async-fs" -version = "2.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8034a681df4aed8b8edbd7fbe472401ecf009251c8b40556b304567052e294c5" -dependencies = [ - "async-lock", - "blocking", - "futures-lite", -] - -[[package]] -name = "async-io" -version = "2.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "456b8a8feb6f42d237746d4b3e9a178494627745c3c56c6ea55d92ba50d026fc" -dependencies = [ - "autocfg", - "cfg-if", - "concurrent-queue", - "futures-io", - "futures-lite", - "parking", - "polling", - "rustix", - "slab", - "windows-sys", -] - -[[package]] -name = "async-lock" -version = "3.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "290f7f2596bd5b78a9fec8088ccd89180d7f9f55b94b0576823bbbdc72ee8311" -dependencies = [ - "event-listener", - "event-listener-strategy", - "pin-project-lite", -] - -[[package]] -name = "async-net" -version = "2.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b948000fad4873c1c9339d60f2623323a0cfd3816e5181033c6a5cb68b2accf7" -dependencies = [ - "async-io", - "blocking", - "futures-lite", -] - -[[package]] -name = "async-process" -version = "2.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fc50921ec0055cdd8a16de48773bfeec5c972598674347252c0399676be7da75" -dependencies = [ - "async-channel", - "async-io", - "async-lock", - "async-signal", - "async-task", - "blocking", - "cfg-if", - "event-listener", - "futures-lite", - "rustix", -] - -[[package]] -name = "async-signal" -version = "0.2.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "43c070bbf59cd3570b6b2dd54cd772527c7c3620fce8be898406dd3ed6adc64c" -dependencies = [ - "async-io", - "async-lock", - "atomic-waker", - "cfg-if", - "futures-core", - "futures-io", - "rustix", - "signal-hook-registry", - "slab", - "windows-sys", -] - -[[package]] -name = "async-stream" -version = "0.3.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0b5a71a6f37880a80d1d7f19efd781e4b5de42c88f0722cc13bcb6cc2cfe8476" -dependencies = [ - "async-stream-impl", - "futures-core", - "pin-project-lite", -] - -[[package]] -name = "async-stream-impl" -version = "0.3.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c7c24de15d275a1ecfd47a380fb4d5ec9bfe0933f309ed5e705b775596a3574d" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.117", -] - -[[package]] -name = "async-task" -version = "4.7.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b75356056920673b02621b35afd0f7dda9306d03c79a30f5c56c44cf256e3de" - -[[package]] -name = "async-trait" -version = "0.1.89" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9035ad2d096bed7955a320ee7e2230574d28fd3c3a0f186cbea1ff3c7eed5dbb" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.117", -] - -[[package]] -name = "atoi" -version = "2.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f28d99ec8bfea296261ca1af174f24225171fea9664ba9003cbebee704810528" -dependencies = [ - "num-traits", -] - -[[package]] -name = "atomic-waker" -version = "1.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0" - -[[package]] -name = "autocfg" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c08606f8c3cbf4ce6ec8e28fb0014a2c086708fe954eaa885384a6165172e7e8" - -[[package]] -name = "base64" -version = "0.22.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" - -[[package]] -name = "better_io" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ef0a3155e943e341e557863e69a708999c94ede624e37865c8e2a91b94efa78f" - -[[package]] -name = "bindgen" -version = "0.72.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "993776b509cfb49c750f11b8f07a46fa23e0a1386ffc01fb1e7d343efc387895" -dependencies = [ - "bitflags", - "cexpr", - "clang-sys", - "itertools 0.13.0", - "proc-macro2", - "quote", - "regex", - "rustc-hash", - "shlex", - "syn 2.0.117", -] - -[[package]] -name = "bit-vec" -version = "0.9.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b71798fca2c1fe1086445a7258a4bc81e6e49dcd24c8d0dd9a1e57395b603f51" -dependencies = [ - "serde", -] - -[[package]] -name = "bitflags" -version = "2.11.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "843867be96c8daad0d758b57df9392b6d8d271134fce549de6ce169ff98a92af" - -[[package]] -name = "bitvec" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1bc2832c24239b0141d5674bb9174f9d68a8b5b3f2753311927c172ca46f7e9c" -dependencies = [ - "funty", - "radium", - "tap", - "wyz", -] - -[[package]] -name = "blocking" -version = "1.6.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e83f8d02be6967315521be875afa792a316e28d57b5a2d401897e2a7921b7f21" -dependencies = [ - "async-channel", - "async-task", - "futures-io", - "futures-lite", - "piper", -] - -[[package]] -name = "bumpalo" -version = "3.20.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d20789868f4b01b2f2caec9f5c4e0213b41e3e5702a50157d699ae31ced2fcb" - -[[package]] -name = "bytemuck" -version = "1.25.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8efb64bd706a16a1bdde310ae86b351e4d21550d98d056f22f8a7f7a2183fec" - -[[package]] -name = "byteorder" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" - -[[package]] -name = "bytes" -version = "1.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33" - -[[package]] -name = "cc" -version = "1.2.57" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7a0dd1ca384932ff3641c8718a02769f1698e7563dc6974ffd03346116310423" -dependencies = [ - "find-msvc-tools", - "jobserver", - "libc", - "shlex", -] - -[[package]] -name = "cexpr" -version = "0.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6fac387a98bb7c37292057cffc56d62ecb629900026402633ae9160df93a8766" -dependencies = [ - "nom", -] - -[[package]] -name = "cfg-if" -version = "1.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" - -[[package]] -name = "chacha20" -version = "0.10.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6f8d983286843e49675a4b7a2d174efe136dc93a18d69130dd18198a6c167601" -dependencies = [ - "cfg-if", - "cpufeatures", - "rand_core 0.10.0", -] - -[[package]] -name = "chrono" -version = "0.4.44" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c673075a2e0e5f4a1dde27ce9dee1ea4558c7ffe648f576438a20ca1d2acc4b0" -dependencies = [ - "iana-time-zone", - "num-traits", - "windows-link", -] - -[[package]] -name = "clang-sys" -version = "1.8.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0b023947811758c97c59bf9d1c188fd619ad4718dcaa767947df1cadb14f39f4" -dependencies = [ - "glob", - "libc", - "libloading", -] - -[[package]] -name = "concurrent-queue" -version = "2.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4ca0197aee26d1ae37445ee532fefce43251d24cc7c166799f4d46817f1d3973" -dependencies = [ - "crossbeam-utils", -] - -[[package]] -name = "const-random" -version = "0.1.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "87e00182fe74b066627d63b85fd550ac2998d4b0bd86bfed477a0ae4c7c71359" -dependencies = [ - "const-random-macro", -] - -[[package]] -name = "const-random-macro" -version = "0.1.16" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f9d839f2a20b0aee515dc581a6172f2321f96cab76c1a38a4c584a194955390e" -dependencies = [ - "getrandom 0.2.17", - "once_cell", - "tiny-keccak", -] - -[[package]] -name = "const_for" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d9c50fcfdf972929aff202c16b80086aa3cfc6a3a820af714096c58c7c1d0582" - -[[package]] -name = "core-foundation-sys" -version = "0.8.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" - -[[package]] -name = "cpufeatures" -version = "0.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201" -dependencies = [ - "libc", -] - -[[package]] -name = "crossbeam-channel" -version = "0.5.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "82b8f8f868b36967f9606790d1903570de9ceaf870a7bf9fbbd3016d636a2cb2" -dependencies = [ - "crossbeam-utils", -] - -[[package]] -name = "crossbeam-epoch" -version = "0.9.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5b82ac4a3c2ca9c3460964f020e1402edd5753411d7737aa39c3714ad1b5420e" -dependencies = [ - "crossbeam-utils", -] - -[[package]] -name = "crossbeam-utils" -version = "0.8.21" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d0a5c400df2834b80a4c3327b3aad3a4c4cd4de0629063962b03235697506a28" - -[[package]] -name = "crunchy" -version = "0.2.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5" - -[[package]] -name = "custom-labels" -version = "0.4.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a750ea4bdb7dbf9584b5d5c668bfa3835f88275781a947b5ea0212945bbdd41f" -dependencies = [ - "bindgen", - "cc", - "libc", - "pin-project-lite", -] - -[[package]] -name = "dashmap" -version = "6.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5041cc499144891f3790297212f32a74fb938e5136a14943f338ef9e0ae276cf" -dependencies = [ - "cfg-if", - "crossbeam-utils", - "hashbrown 0.14.5", - "lock_api", - "once_cell", - "parking_lot_core", -] - -[[package]] -name = "dtype_dispatch" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ab23e69df104e2fd85ee63a533a22d2132ef5975dc6b36f9f3e5a7305e4a8ed7" - -[[package]] -name = "either" -version = "1.15.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "48c757948c5ede0e46177b7add2e67155f70e33c07fea8284df6576da70b3719" - -[[package]] -name = "enum-iterator" -version = "2.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a4549325971814bda7a44061bf3fe7e487d447cba01e4220a4b454d630d7a016" -dependencies = [ - "enum-iterator-derive", -] - -[[package]] -name = "enum-iterator-derive" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "685adfa4d6f3d765a26bc5dbc936577de9abf756c1feeb3089b01dd395034842" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.117", -] - -[[package]] -name = "equivalent" -version = "1.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" - -[[package]] -name = "errno" -version = "0.3.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" -dependencies = [ - "libc", - "windows-sys", -] - -[[package]] -name = "event-listener" -version = "5.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e13b66accf52311f30a0db42147dadea9850cb48cd070028831ae5f5d4b856ab" -dependencies = [ - "concurrent-queue", - "parking", - "pin-project-lite", -] - -[[package]] -name = "event-listener-strategy" -version = "0.5.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8be9f3dfaaffdae2972880079a491a1a8bb7cbed0b8dd7a347f668b4150a3b93" -dependencies = [ - "event-listener", - "pin-project-lite", -] - -[[package]] -name = "ext-trait" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d772df1c1a777963712fb68e014235e80863d6a91a85c4e06ba2d16243a310e5" -dependencies = [ - "ext-trait-proc_macros", -] - -[[package]] -name = "ext-trait-proc_macros" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1ab7934152eaf26aa5aa9f7371408ad5af4c31357073c9e84c3b9d7f11ad639a" -dependencies = [ - "proc-macro2", - "quote", - "syn 1.0.109", -] - -[[package]] -name = "extension-traits" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a296e5a895621edf9fa8329c83aa1cb69a964643e36cf54d8d7a69b789089537" -dependencies = [ - "ext-trait", -] - -[[package]] -name = "fastlanes" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "414cb755aee48ff7b0907995d2949c68c8c17900970076dff6a808e18e592d71" -dependencies = [ - "arrayref", - "const_for", - "num-traits", - "paste", - "seq-macro", -] - -[[package]] -name = "fastrand" -version = "2.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "37909eebbb50d72f9059c3b6d82c0463f2ff062c9e95845c43a6c9c0355411be" - -[[package]] -name = "find-msvc-tools" -version = "0.1.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" - -[[package]] -name = "flatbuffers" -version = "25.12.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "35f6839d7b3b98adde531effaf34f0c2badc6f4735d26fe74709d8e513a96ef3" -dependencies = [ - "bitflags", - "rustc_version", -] - -[[package]] -name = "foldhash" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2" - -[[package]] -name = "foldhash" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb" - -[[package]] -name = "fsst-rs" -version = "0.5.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cbfc42bd5e2b76f850ba4ff6dd6cb170389bc3c1e64dc45238a742d79bbfb2ca" -dependencies = [ - "rustc-hash", -] - -[[package]] -name = "funty" -version = "2.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e6d5a32815ae3f33302d95fdcb2ce17862f8c65363dcfd29360480ba1001fc9c" - -[[package]] -name = "futures" -version = "0.3.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b147ee9d1f6d097cef9ce628cd2ee62288d963e16fb287bd9286455b241382d" -dependencies = [ - "futures-channel", - "futures-core", - "futures-executor", - "futures-io", - "futures-sink", - "futures-task", - "futures-util", -] - -[[package]] -name = "futures-channel" -version = "0.3.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "07bbe89c50d7a535e539b8c17bc0b49bdb77747034daa8087407d655f3f7cc1d" -dependencies = [ - "futures-core", - "futures-sink", -] - -[[package]] -name = "futures-core" -version = "0.3.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7e3450815272ef58cec6d564423f6e755e25379b217b0bc688e295ba24df6b1d" - -[[package]] -name = "futures-executor" -version = "0.3.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "baf29c38818342a3b26b5b923639e7b1f4a61fc5e76102d4b1981c6dc7a7579d" -dependencies = [ - "futures-core", - "futures-task", - "futures-util", -] - -[[package]] -name = "futures-io" -version = "0.3.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cecba35d7ad927e23624b22ad55235f2239cfa44fd10428eecbeba6d6a717718" - -[[package]] -name = "futures-lite" -version = "2.6.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f78e10609fe0e0b3f4157ffab1876319b5b0db102a2c60dc4626306dc46b44ad" -dependencies = [ - "fastrand", - "futures-core", - "futures-io", - "parking", - "pin-project-lite", -] - -[[package]] -name = "futures-macro" -version = "0.3.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e835b70203e41293343137df5c0664546da5745f82ec9b84d40be8336958447b" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.117", -] - -[[package]] -name = "futures-sink" -version = "0.3.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c39754e157331b013978ec91992bde1ac089843443c49cbc7f46150b0fad0893" - -[[package]] -name = "futures-task" -version = "0.3.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "037711b3d59c33004d3856fbdc83b99d4ff37a24768fa1be9ce3538a1cde4393" - -[[package]] -name = "futures-util" -version = "0.3.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "389ca41296e6190b48053de0321d02a77f32f8a5d2461dd38762c0593805c6d6" -dependencies = [ - "futures-channel", - "futures-core", - "futures-io", - "futures-macro", - "futures-sink", - "futures-task", - "memchr", - "pin-project-lite", - "slab", -] - -[[package]] -name = "getrandom" -version = "0.2.17" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" -dependencies = [ - "cfg-if", - "libc", - "wasi", -] - -[[package]] -name = "getrandom" -version = "0.3.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" -dependencies = [ - "cfg-if", - "libc", - "r-efi 5.3.0", - "wasip2", -] - -[[package]] -name = "getrandom" -version = "0.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0de51e6874e94e7bf76d726fc5d13ba782deca734ff60d5bb2fb2607c7406555" -dependencies = [ - "cfg-if", - "js-sys", - "libc", - "r-efi 6.0.0", - "rand_core 0.10.0", - "wasip2", - "wasip3", - "wasm-bindgen", -] - -[[package]] -name = "glob" -version = "0.3.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0cc23270f6e1808e30a928bdc84dea0b9b4136a8bc82338574f23baf47bbd280" - -[[package]] -name = "half" -version = "2.7.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6ea2d84b969582b4b1864a92dc5d27cd2b77b622a8d79306834f1be5ba20d84b" -dependencies = [ - "cfg-if", - "crunchy", - "num-traits", - "zerocopy", -] - -[[package]] -name = "handle" -version = "1.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "52ed72152641d513a6084a3907bfcba3f35ae2d3335c22ce2242969c25ff8e46" - -[[package]] -name = "hashbrown" -version = "0.14.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1" - -[[package]] -name = "hashbrown" -version = "0.15.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1" -dependencies = [ - "foldhash 0.1.5", -] - -[[package]] -name = "hashbrown" -version = "0.16.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100" -dependencies = [ - "allocator-api2", - "equivalent", - "foldhash 0.2.0", -] - -[[package]] -name = "heck" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" - -[[package]] -name = "hermit-abi" -version = "0.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fc0fef456e4baa96da950455cd02c081ca953b141298e41db3fc7e36b1da849c" - -[[package]] -name = "humansize" -version = "2.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6cb51c9a029ddc91b07a787f1d86b53ccfa49b0e86688c946ebe8d3555685dd7" -dependencies = [ - "libm", -] - -[[package]] -name = "iana-time-zone" -version = "0.1.65" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470" -dependencies = [ - "android_system_properties", - "core-foundation-sys", - "iana-time-zone-haiku", - "js-sys", - "log", - "wasm-bindgen", - "windows-core", -] - -[[package]] -name = "iana-time-zone-haiku" -version = "0.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f" -dependencies = [ - "cc", -] - -[[package]] -name = "id-arena" -version = "2.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3d3067d79b975e8844ca9eb072e16b31c3c1c36928edf9c6789548c524d0d954" - -[[package]] -name = "indexmap" -version = "2.13.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7714e70437a7dc3ac8eb7e6f8df75fd8eb422675fc7678aff7364301092b1017" -dependencies = [ - "equivalent", - "hashbrown 0.16.1", - "serde", - "serde_core", -] - -[[package]] -name = "inventory" -version = "0.3.22" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "009ae045c87e7082cb72dab0ccd01ae075dd00141ddc108f43a0ea150a9e7227" -dependencies = [ - "rustversion", -] - -[[package]] -name = "itertools" -version = "0.13.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "413ee7dfc52ee1a4949ceeb7dbc8a33f2d6c088194d9f922fb8318faf1f01186" -dependencies = [ - "either", -] - -[[package]] -name = "itertools" -version = "0.14.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285" -dependencies = [ - "either", -] - -[[package]] -name = "itoa" -version = "1.0.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" - -[[package]] -name = "jiff" -version = "0.2.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1a3546dc96b6d42c5f24902af9e2538e82e39ad350b0c766eb3fbf2d8f3d8359" -dependencies = [ - "jiff-static", - "jiff-tzdb-platform", - "log", - "portable-atomic", - "portable-atomic-util", - "serde_core", - "windows-sys", -] - -[[package]] -name = "jiff-static" -version = "0.2.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2a8c8b344124222efd714b73bb41f8b5120b27a7cc1c75593a6ff768d9d05aa4" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.117", -] - -[[package]] -name = "jiff-tzdb" -version = "0.1.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c900ef84826f1338a557697dc8fc601df9ca9af4ac137c7fb61d4c6f2dfd3076" - -[[package]] -name = "jiff-tzdb-platform" -version = "0.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "875a5a69ac2bab1a891711cf5eccbec1ce0341ea805560dcd90b7a2e925132e8" -dependencies = [ - "jiff-tzdb", -] - -[[package]] -name = "jobserver" -version = "0.1.34" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9afb3de4395d6b3e67a780b6de64b51c978ecf11cb9a462c66be7d4ca9039d33" -dependencies = [ - "getrandom 0.3.4", - "libc", -] - -[[package]] -name = "js-sys" -version = "0.3.91" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b49715b7073f385ba4bc528e5747d02e66cb39c6146efb66b781f131f0fb399c" -dependencies = [ - "once_cell", - "wasm-bindgen", -] - -[[package]] -name = "kanal" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9e3953adf0cd667798b396c2fa13552d6d9b3269d7dd1154c4c416442d1ff574" -dependencies = [ - "futures-core", - "lock_api", -] - -[[package]] -name = "leb128fmt" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09edd9e8b54e49e587e4f6295a7d29c3ea94d469cb40ab8ca70b288248a81db2" - -[[package]] -name = "lending-iterator" -version = "0.1.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bc07588c853b50689205fb5c00498aa681d89828e0ce8cbd965ebc7a5d8ae260" -dependencies = [ - "extension-traits", - "lending-iterator-proc_macros", - "macro_rules_attribute", - "never-say-never", - "nougat", - "polonius-the-crab", -] - -[[package]] -name = "lending-iterator-proc_macros" -version = "0.1.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b5445dd1c0deb1e97b8a16561d17fc686ca83e8411128fb036e9668a72d51b1d" -dependencies = [ - "proc-macro2", - "quote", - "syn 1.0.109", -] - -[[package]] -name = "lexical-core" -version = "1.0.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7d8d125a277f807e55a77304455eb7b1cb52f2b18c143b60e766c120bd64a594" -dependencies = [ - "lexical-parse-float", - "lexical-parse-integer", - "lexical-util", - "lexical-write-float", - "lexical-write-integer", -] - -[[package]] -name = "lexical-parse-float" -version = "1.0.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "52a9f232fbd6f550bc0137dcb5f99ab674071ac2d690ac69704593cb4abbea56" -dependencies = [ - "lexical-parse-integer", - "lexical-util", -] - -[[package]] -name = "lexical-parse-integer" -version = "1.0.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9a7a039f8fb9c19c996cd7b2fcce303c1b2874fe1aca544edc85c4a5f8489b34" -dependencies = [ - "lexical-util", -] - -[[package]] -name = "lexical-util" -version = "1.0.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2604dd126bb14f13fb5d1bd6a66155079cb9fa655b37f875b3a742c705dbed17" - -[[package]] -name = "lexical-write-float" -version = "1.0.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "50c438c87c013188d415fbabbb1dceb44249ab81664efbd31b14ae55dabb6361" -dependencies = [ - "lexical-util", - "lexical-write-integer", -] - -[[package]] -name = "lexical-write-integer" -version = "1.0.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "409851a618475d2d5796377cad353802345cba92c867d9fbcde9cf4eac4e14df" -dependencies = [ - "lexical-util", -] - -[[package]] -name = "libc" -version = "0.2.183" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b5b646652bf6661599e1da8901b3b9522896f01e736bad5f723fe7a3a27f899d" - -[[package]] -name = "libloading" -version = "0.8.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55" -dependencies = [ - "cfg-if", - "windows-link", -] - -[[package]] -name = "libm" -version = "0.2.16" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981" - -[[package]] -name = "linux-raw-sys" -version = "0.12.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" - -[[package]] -name = "lock_api" -version = "0.4.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" -dependencies = [ - "scopeguard", -] - -[[package]] -name = "log" -version = "0.4.29" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5e5032e24019045c762d3c0f28f5b6b8bbf38563a65908389bf7978758920897" - -[[package]] -name = "macro_rules_attribute" -version = "0.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf0c9b980bf4f3a37fd7b1c066941dd1b1d0152ce6ee6e8fe8c49b9f6810d862" -dependencies = [ - "macro_rules_attribute-proc_macro", - "paste", -] - -[[package]] -name = "macro_rules_attribute-proc_macro" -version = "0.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "58093314a45e00c77d5c508f76e77c3396afbbc0d01506e7fae47b018bac2b1d" - -[[package]] -name = "memchr" -version = "2.8.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8ca58f447f06ed17d5fc4043ce1b10dd205e060fb3ce5b979b8ed8e59ff3f79" - -[[package]] -name = "minimal-lexical" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a" - -[[package]] -name = "moka" -version = "0.12.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "957228ad12042ee839f93c8f257b62b4c0ab5eaae1d4fa60de53b27c9d7c5046" -dependencies = [ - "async-lock", - "crossbeam-channel", - "crossbeam-epoch", - "crossbeam-utils", - "equivalent", - "event-listener", - "futures-util", - "parking_lot", - "portable-atomic", - "smallvec", - "tagptr", - "uuid", -] - -[[package]] -name = "multiversion" -version = "0.8.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7edb7f0ff51249dfda9ab96b5823695e15a052dc15074c9dbf3d118afaf2c201" -dependencies = [ - "multiversion-macros", - "target-features", -] - -[[package]] -name = "multiversion-macros" -version = "0.8.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b093064383341eb3271f42e381cb8f10a01459478446953953c75d24bd339fc0" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.117", - "target-features", -] - -[[package]] -name = "never-say-never" -version = "6.6.666" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cf5a574dadd7941adeaa71823ecba5e28331b8313fb2e1c6a5c7e5981ea53ad6" - -[[package]] -name = "nom" -version = "7.1.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a" -dependencies = [ - "memchr", - "minimal-lexical", -] - -[[package]] -name = "nougat" -version = "0.2.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "97b57b9ced431322f054fc673f1d3c7fa52d80efd9df74ad2fc759f044742510" -dependencies = [ - "macro_rules_attribute", - "nougat-proc_macros", -] - -[[package]] -name = "nougat-proc_macros" -version = "0.2.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c84f77a45e99a2f9b492695d99e1c23844619caa5f3e57647cffacad773ca257" -dependencies = [ - "proc-macro2", - "quote", - "syn 1.0.109", -] - -[[package]] -name = "num-bigint" -version = "0.4.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a5e44f723f1133c9deac646763579fdb3ac745e418f2a7af9cd0c431da1f20b9" -dependencies = [ - "num-integer", - "num-traits", -] - -[[package]] -name = "num-complex" -version = "0.4.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "73f88a1307638156682bada9d7604135552957b7818057dcef22705b4d509495" -dependencies = [ - "num-traits", -] - -[[package]] -name = "num-integer" -version = "0.1.46" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7969661fd2958a5cb096e56c8e1ad0444ac2bbcd0061bd28660485a44879858f" -dependencies = [ - "num-traits", -] - -[[package]] -name = "num-traits" -version = "0.2.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" -dependencies = [ - "autocfg", - "libm", -] - -[[package]] -name = "num_enum" -version = "0.7.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d0bca838442ec211fa11de3a8b0e0e8f3a4522575b5c4c06ed722e005036f26" -dependencies = [ - "num_enum_derive", - "rustversion", -] - -[[package]] -name = "num_enum_derive" -version = "0.7.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "680998035259dcfcafe653688bf2aa6d3e2dc05e98be6ab46afb089dc84f1df8" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.117", -] - -[[package]] -name = "once_cell" -version = "1.21.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" -dependencies = [ - "parking_lot_core", -] - -[[package]] -name = "oneshot" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cfe21416a02c693fb9f980befcb230ecc70b0b3d1cc4abf88b9675c4c1457f0c" - -[[package]] -name = "parking" -version = "2.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f38d5652c16fde515bb1ecef450ab0f6a219d619a7274976324d5e377f7dceba" - -[[package]] -name = "parking_lot" -version = "0.12.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "93857453250e3077bd71ff98b6a65ea6621a19bb0f559a85248955ac12c45a1a" -dependencies = [ - "lock_api", - "parking_lot_core", -] - -[[package]] -name = "parking_lot_core" -version = "0.9.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2621685985a2ebf1c516881c026032ac7deafcda1a2c9b7850dc81e3dfcb64c1" -dependencies = [ - "cfg-if", - "libc", - "redox_syscall", - "smallvec", - "windows-link", -] - -[[package]] -name = "paste" -version = "1.0.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a" - -[[package]] -name = "pco" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e89d71ab3c07ed898defa4915bdc2a963131d811a1eab0eeacfac65c94cdeae8" -dependencies = [ - "better_io", - "dtype_dispatch", - "half", - "rand_xoshiro", -] - -[[package]] -name = "pin-project-lite" -version = "0.2.17" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" - -[[package]] -name = "piper" -version = "0.2.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c835479a4443ded371d6c535cbfd8d31ad92c5d23ae9770a61bc155e4992a3c1" -dependencies = [ - "atomic-waker", - "fastrand", - "futures-io", -] - -[[package]] -name = "pkg-config" -version = "0.3.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7edddbd0b52d732b21ad9a5fab5c704c14cd949e5e9a1ec5929a24fded1b904c" - -[[package]] -name = "polling" -version = "3.11.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d0e4f59085d47d8241c88ead0f274e8a0cb551f3625263c05eb8dd897c34218" -dependencies = [ - "cfg-if", - "concurrent-queue", - "hermit-abi", - "pin-project-lite", - "rustix", - "windows-sys", -] - -[[package]] -name = "polonius-the-crab" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2a69ee997a6282f8462abf1e0d8c38c965e968799e912b3bed8c9e8a28c2f9f" - -[[package]] -name = "portable-atomic" -version = "1.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49" - -[[package]] -name = "portable-atomic-util" -version = "0.2.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "091397be61a01d4be58e7841595bd4bfedb15f1cd54977d79b8271e94ed799a3" -dependencies = [ - "portable-atomic", -] - -[[package]] -name = "prettyplease" -version = "0.2.37" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" -dependencies = [ - "proc-macro2", - "syn 2.0.117", -] - -[[package]] -name = "proc-macro2" -version = "1.0.106" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" -dependencies = [ - "unicode-ident", -] - -[[package]] -name = "prost" -version = "0.14.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d2ea70524a2f82d518bce41317d0fae74151505651af45faf1ffbd6fd33f0568" -dependencies = [ - "bytes", - "prost-derive", -] - -[[package]] -name = "prost-derive" -version = "0.14.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "27c6023962132f4b30eb4c172c91ce92d933da334c59c23cddee82358ddafb0b" -dependencies = [ - "anyhow", - "itertools 0.14.0", - "proc-macro2", - "quote", - "syn 2.0.117", -] - -[[package]] -name = "prost-types" -version = "0.14.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8991c4cbdb8bc5b11f0b074ffe286c30e523de90fee5ba8132f1399f23cb3dd7" -dependencies = [ - "prost", -] - -[[package]] -name = "quote" -version = "1.0.45" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924" -dependencies = [ - "proc-macro2", -] - -[[package]] -name = "r-efi" -version = "5.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" - -[[package]] -name = "r-efi" -version = "6.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" - -[[package]] -name = "radium" -version = "0.7.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc33ff2d4973d518d823d61aa239014831e521c75da58e3df4840d3f47749d09" - -[[package]] -name = "rand" -version = "0.10.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bc266eb313df6c5c09c1c7b1fbe2510961e5bcd3add930c1e31f7ed9da0feff8" -dependencies = [ - "chacha20", - "getrandom 0.4.2", - "rand_core 0.10.0", -] - -[[package]] -name = "rand_core" -version = "0.6.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c" - -[[package]] -name = "rand_core" -version = "0.10.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c8d0fd677905edcbeedbf2edb6494d676f0e98d54d5cf9bda0b061cb8fb8aba" - -[[package]] -name = "rand_xoshiro" -version = "0.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6f97cdb2a36ed4183de61b2f824cc45c9f1037f28afe0a322e9fff4c108b5aaa" -dependencies = [ - "rand_core 0.6.4", -] - -[[package]] -name = "redox_syscall" -version = "0.5.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d" -dependencies = [ - "bitflags", -] - -[[package]] -name = "regex" -version = "1.12.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e10754a14b9137dd7b1e3e5b0493cc9171fdd105e0ab477f51b72e7f3ac0e276" -dependencies = [ - "aho-corasick", - "memchr", - "regex-automata", - "regex-syntax", -] - -[[package]] -name = "regex-automata" -version = "0.4.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e1dd4122fc1595e8162618945476892eefca7b88c52820e74af6262213cae8f" -dependencies = [ - "aho-corasick", - "memchr", - "regex-syntax", -] - -[[package]] -name = "regex-syntax" -version = "0.8.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "dc897dd8d9e8bd1ed8cdad82b5966c3e0ecae09fb1907d58efaa013543185d0a" - -[[package]] -name = "roaring" -version = "0.11.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8ba9ce64a8f45d7fc86358410bb1a82e8c987504c0d4900e9141d69a9f26c885" -dependencies = [ - "bytemuck", - "byteorder", -] - -[[package]] -name = "rustc-hash" -version = "2.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "357703d41365b4b27c590e3ed91eabb1b663f07c4c084095e60cbed4362dff0d" - -[[package]] -name = "rustc_version" -version = "0.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" -dependencies = [ - "semver", -] - -[[package]] -name = "rustix" -version = "1.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" -dependencies = [ - "bitflags", - "errno", - "libc", - "linux-raw-sys", - "windows-sys", -] - -[[package]] -name = "rustversion" -version = "1.0.22" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d" - -[[package]] -name = "ryu" -version = "1.0.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" - -[[package]] -name = "scopeguard" -version = "1.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" - -[[package]] -name = "semver" -version = "1.0.27" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d767eb0aabc880b29956c35734170f26ed551a859dbd361d140cdbeca61ab1e2" - -[[package]] -name = "seq-macro" -version = "0.3.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1bc711410fbe7399f390ca1c3b60ad0f53f80e95c5eb935e52268a0e2cd49acc" - -[[package]] -name = "serde" -version = "1.0.228" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" -dependencies = [ - "serde_core", - "serde_derive", -] - -[[package]] -name = "serde_core" -version = "1.0.228" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" -dependencies = [ - "serde_derive", -] - -[[package]] -name = "serde_derive" -version = "1.0.228" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.117", -] - -[[package]] -name = "serde_json" -version = "1.0.149" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "83fc039473c5595ace860d8c4fafa220ff474b3fc6bfdb4293327f1a37e94d86" -dependencies = [ - "itoa", - "memchr", - "serde", - "serde_core", - "zmij", -] - -[[package]] -name = "shlex" -version = "1.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64" - -[[package]] -name = "signal-hook-registry" -version = "1.4.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c4db69cba1110affc0e9f7bcd48bbf87b3f4fc7c61fc9155afd4c469eb3d6c1b" -dependencies = [ - "errno", - "libc", -] - -[[package]] -name = "simdutf8" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e" - -[[package]] -name = "sketches-ddsketch" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "05e40b6cf54d988dc1a2223531b969c9a9e30906ad90ef64890c27b4bfbb46ea" - -[[package]] -name = "slab" -version = "0.4.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" - -[[package]] -name = "smallvec" -version = "1.15.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "67b1b7a3b5fe4f1376887184045fcf45c69e92af734b7aaddc05fb777b6fbd03" - -[[package]] -name = "smol" -version = "2.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a33bd3e260892199c3ccfc487c88b2da2265080acb316cd920da72fdfd7c599f" -dependencies = [ - "async-channel", - "async-executor", - "async-fs", - "async-io", - "async-lock", - "async-net", - "async-process", - "blocking", - "futures-lite", -] - -[[package]] -name = "static_assertions" -version = "1.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a2eb9349b6444b326872e140eb1cf5e7c522154d69e7a0ffb0fb81c06b37543f" - -[[package]] -name = "syn" -version = "1.0.109" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "syn" -version = "2.0.117" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "tagptr" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7b2093cf4c8eb1e67749a6762251bc9cd836b6fc171623bd0a9d324d37af2417" - -[[package]] -name = "tap" -version = "1.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "55937e1799185b12863d447f42597ed69d9928686b8d88a1df17376a097d8369" - -[[package]] -name = "target-features" -version = "0.1.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c1bbb9f3c5c463a01705937a24fdabc5047929ac764b2d5b9cf681c1f5041ed5" - -[[package]] -name = "termtree" -version = "1.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d4d1330fe7f7f872cd05165130b10602d667b205fd85be09be2814b115d4ced9" - -[[package]] -name = "testfiles" -version = "0.1.0" -dependencies = [ - "vortex", -] - -[[package]] -name = "tiny-keccak" -version = "2.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2c9d3793400a45f954c52e73d068316d76b6f4e36977e3fcebb13a2721e80237" -dependencies = [ - "crunchy", -] - -[[package]] -name = "tokio" -version = "1.50.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "27ad5e34374e03cfffefc301becb44e9dc3c17584f414349ebe29ed26661822d" -dependencies = [ - "bytes", - "pin-project-lite", -] - -[[package]] -name = "tracing" -version = "0.1.44" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100" -dependencies = [ - "pin-project-lite", - "tracing-core", -] - -[[package]] -name = "tracing-core" -version = "0.1.36" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a" - -[[package]] -name = "unicode-ident" -version = "1.0.24" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" - -[[package]] -name = "unicode-xid" -version = "0.2.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853" - -[[package]] -name = "uuid" -version = "1.22.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a68d3c8f01c0cfa54a75291d83601161799e4a89a39e0929f4b0354d88757a37" -dependencies = [ - "getrandom 0.4.2", - "js-sys", - "wasm-bindgen", -] - -[[package]] -name = "version_check" -version = "0.9.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" - -[[package]] -name = "vortex" -version = "0.1.0" -dependencies = [ - "vortex-alp", - "vortex-array", - "vortex-btrblocks", - "vortex-buffer", - "vortex-bytebool", - "vortex-datetime-parts", - "vortex-decimal-byte-parts", - "vortex-error", - "vortex-fastlanes", - "vortex-file", - "vortex-flatbuffers", - "vortex-fsst", - "vortex-io", - "vortex-ipc", - "vortex-layout", - "vortex-mask", - "vortex-metrics", - "vortex-pco", - "vortex-proto", - "vortex-runend", - "vortex-scan", - "vortex-sequence", - "vortex-session", - "vortex-sparse", - "vortex-utils", - "vortex-zigzag", - "vortex-zstd", -] - -[[package]] -name = "vortex-alp" -version = "0.1.0" -dependencies = [ - "itertools 0.14.0", - "num-traits", - "prost", - "rustc-hash", - "vortex-array", - "vortex-buffer", - "vortex-error", - "vortex-fastlanes", - "vortex-mask", - "vortex-session", - "vortex-utils", -] - -[[package]] -name = "vortex-array" -version = "0.1.0" -dependencies = [ - "arcref", - "arrow-arith", - "arrow-array", - "arrow-buffer", - "arrow-cast", - "arrow-data", - "arrow-ord", - "arrow-schema", - "arrow-select", - "arrow-string", - "async-lock", - "bytes", - "cfg-if", - "enum-iterator", - "flatbuffers", - "futures", - "getrandom 0.4.2", - "half", - "humansize", - "inventory", - "itertools 0.14.0", - "jiff", - "multiversion", - "num-traits", - "num_enum", - "parking_lot", - "paste", - "pin-project-lite", - "prost", - "rand", - "rustc-hash", - "simdutf8", - "static_assertions", - "termtree", - "tracing", - "uuid", - "vortex-buffer", - "vortex-error", - "vortex-flatbuffers", - "vortex-mask", - "vortex-proto", - "vortex-session", - "vortex-utils", -] - -[[package]] -name = "vortex-btrblocks" -version = "0.1.0" -dependencies = [ - "getrandom 0.4.2", - "itertools 0.14.0", - "num-traits", - "pco", - "rand", - "rustc-hash", - "tracing", - "vortex-alp", - "vortex-array", - "vortex-buffer", - "vortex-compressor", - "vortex-datetime-parts", - "vortex-decimal-byte-parts", - "vortex-error", - "vortex-fastlanes", - "vortex-fsst", - "vortex-mask", - "vortex-pco", - "vortex-runend", - "vortex-sequence", - "vortex-sparse", - "vortex-utils", - "vortex-zigzag", - "vortex-zstd", -] - -[[package]] -name = "vortex-buffer" -version = "0.1.0" -dependencies = [ - "arrow-buffer", - "bitvec", - "bytes", - "itertools 0.14.0", - "simdutf8", - "vortex-error", -] - -[[package]] -name = "vortex-bytebool" -version = "0.1.0" -dependencies = [ - "num-traits", - "vortex-array", - "vortex-buffer", - "vortex-error", - "vortex-mask", - "vortex-session", -] - -[[package]] -name = "vortex-compressor" -version = "0.1.0" -dependencies = [ - "itertools 0.14.0", - "num-traits", - "parking_lot", - "rand", - "rustc-hash", - "tracing", - "vortex-array", - "vortex-buffer", - "vortex-error", - "vortex-mask", - "vortex-utils", -] - -[[package]] -name = "vortex-datetime-parts" -version = "0.1.0" -dependencies = [ - "num-traits", - "prost", - "vortex-array", - "vortex-buffer", - "vortex-error", - "vortex-mask", - "vortex-session", -] - -[[package]] -name = "vortex-decimal-byte-parts" -version = "0.1.0" -dependencies = [ - "num-traits", - "prost", - "vortex-array", - "vortex-buffer", - "vortex-error", - "vortex-mask", - "vortex-session", -] - -[[package]] -name = "vortex-error" -version = "0.1.0" -dependencies = [ - "arrow-schema", - "flatbuffers", - "jiff", - "prost", -] - -[[package]] -name = "vortex-fastlanes" -version = "0.1.0" -dependencies = [ - "fastlanes", - "itertools 0.14.0", - "lending-iterator", - "num-traits", - "prost", - "vortex-array", - "vortex-buffer", - "vortex-error", - "vortex-mask", - "vortex-session", -] - -[[package]] -name = "vortex-file" -version = "0.1.0" -dependencies = [ - "async-trait", - "bytes", - "flatbuffers", - "futures", - "getrandom 0.4.2", - "itertools 0.14.0", - "kanal", - "moka", - "oneshot", - "parking_lot", - "pin-project-lite", - "tracing", - "uuid", - "vortex-alp", - "vortex-array", - "vortex-btrblocks", - "vortex-buffer", - "vortex-bytebool", - "vortex-datetime-parts", - "vortex-decimal-byte-parts", - "vortex-error", - "vortex-fastlanes", - "vortex-flatbuffers", - "vortex-fsst", - "vortex-io", - "vortex-layout", - "vortex-mask", - "vortex-metrics", - "vortex-pco", - "vortex-runend", - "vortex-scan", - "vortex-sequence", - "vortex-session", - "vortex-sparse", - "vortex-utils", - "vortex-zigzag", - "vortex-zstd", -] - -[[package]] -name = "vortex-flatbuffers" -version = "0.1.0" -dependencies = [ - "flatbuffers", - "vortex-buffer", - "vortex-error", -] - -[[package]] -name = "vortex-fsst" -version = "0.1.0" -dependencies = [ - "fsst-rs", - "prost", - "vortex-array", - "vortex-buffer", - "vortex-error", - "vortex-mask", - "vortex-session", -] - -[[package]] -name = "vortex-io" -version = "0.1.0" -dependencies = [ - "async-fs", - "async-stream", - "async-trait", - "bytes", - "custom-labels", - "futures", - "getrandom 0.4.2", - "glob", - "handle", - "kanal", - "oneshot", - "parking_lot", - "pin-project-lite", - "smol", - "tokio", - "tracing", - "vortex-array", - "vortex-buffer", - "vortex-error", - "vortex-metrics", - "vortex-session", - "wasm-bindgen-futures", -] - -[[package]] -name = "vortex-ipc" -version = "0.1.0" -dependencies = [ - "bytes", - "flatbuffers", - "futures", - "itertools 0.14.0", - "pin-project-lite", - "vortex-array", - "vortex-buffer", - "vortex-error", - "vortex-flatbuffers", - "vortex-session", -] - -[[package]] -name = "vortex-layout" -version = "0.1.0" -dependencies = [ - "arcref", - "arrow-array", - "arrow-schema", - "async-stream", - "async-trait", - "bit-vec", - "flatbuffers", - "futures", - "itertools 0.14.0", - "kanal", - "moka", - "once_cell", - "oneshot", - "parking_lot", - "paste", - "pin-project-lite", - "prost", - "rustc-hash", - "sketches-ddsketch", - "termtree", - "tracing", - "vortex-array", - "vortex-btrblocks", - "vortex-buffer", - "vortex-error", - "vortex-flatbuffers", - "vortex-io", - "vortex-mask", - "vortex-metrics", - "vortex-scan", - "vortex-sequence", - "vortex-session", - "vortex-utils", -] - -[[package]] -name = "vortex-mask" -version = "0.1.0" -dependencies = [ - "itertools 0.14.0", - "vortex-buffer", - "vortex-error", -] - -[[package]] -name = "vortex-metrics" -version = "0.1.0" -dependencies = [ - "getrandom 0.4.2", - "parking_lot", - "sketches-ddsketch", -] - -[[package]] -name = "vortex-pco" -version = "0.1.0" -dependencies = [ - "pco", - "prost", - "vortex-array", - "vortex-buffer", - "vortex-error", - "vortex-mask", - "vortex-session", -] - -[[package]] -name = "vortex-proto" -version = "0.1.0" -dependencies = [ - "prost", - "prost-types", -] - -[[package]] -name = "vortex-runend" -version = "0.1.0" -dependencies = [ - "arrow-array", - "itertools 0.14.0", - "num-traits", - "prost", - "vortex-array", - "vortex-buffer", - "vortex-error", - "vortex-mask", - "vortex-session", -] - -[[package]] -name = "vortex-scan" -version = "0.1.0" -dependencies = [ - "async-trait", - "futures", - "roaring", - "tracing", - "vortex-array", - "vortex-buffer", - "vortex-error", - "vortex-mask", - "vortex-session", -] - -[[package]] -name = "vortex-sequence" -version = "0.1.0" -dependencies = [ - "num-traits", - "prost", - "vortex-array", - "vortex-buffer", - "vortex-error", - "vortex-mask", - "vortex-proto", - "vortex-session", -] - -[[package]] -name = "vortex-session" -version = "0.1.0" -dependencies = [ - "arcref", - "dashmap", - "parking_lot", - "vortex-error", - "vortex-utils", -] - -[[package]] -name = "vortex-sparse" -version = "0.1.0" -dependencies = [ - "itertools 0.14.0", - "num-traits", - "prost", - "vortex-array", - "vortex-buffer", - "vortex-error", - "vortex-mask", - "vortex-session", -] - -[[package]] -name = "vortex-utils" -version = "0.1.0" -dependencies = [ - "dashmap", - "hashbrown 0.16.1", - "vortex-error", -] - -[[package]] -name = "vortex-zigzag" -version = "0.1.0" -dependencies = [ - "vortex-array", - "vortex-buffer", - "vortex-error", - "vortex-mask", - "vortex-session", - "zigzag", -] - -[[package]] -name = "vortex-zstd" -version = "0.1.0" -dependencies = [ - "itertools 0.14.0", - "prost", - "vortex-array", - "vortex-buffer", - "vortex-error", - "vortex-mask", - "vortex-session", - "zstd", -] - -[[package]] -name = "wasi" -version = "0.11.1+wasi-snapshot-preview1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" - -[[package]] -name = "wasip2" -version = "1.0.2+wasi-0.2.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9517f9239f02c069db75e65f174b3da828fe5f5b945c4dd26bd25d89c03ebcf5" -dependencies = [ - "wit-bindgen", -] - -[[package]] -name = "wasip3" -version = "0.4.0+wasi-0.3.0-rc-2026-01-06" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5428f8bf88ea5ddc08faddef2ac4a67e390b88186c703ce6dbd955e1c145aca5" -dependencies = [ - "wit-bindgen", -] - -[[package]] -name = "wasm-bindgen" -version = "0.2.114" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6532f9a5c1ece3798cb1c2cfdba640b9b3ba884f5db45973a6f442510a87d38e" -dependencies = [ - "cfg-if", - "once_cell", - "rustversion", - "wasm-bindgen-macro", - "wasm-bindgen-shared", -] - -[[package]] -name = "wasm-bindgen-futures" -version = "0.4.64" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e9c5522b3a28661442748e09d40924dfb9ca614b21c00d3fd135720e48b67db8" -dependencies = [ - "cfg-if", - "futures-util", - "js-sys", - "once_cell", - "wasm-bindgen", - "web-sys", -] - -[[package]] -name = "wasm-bindgen-macro" -version = "0.2.114" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "18a2d50fcf105fb33bb15f00e7a77b772945a2ee45dcf454961fd843e74c18e6" -dependencies = [ - "quote", - "wasm-bindgen-macro-support", -] - -[[package]] -name = "wasm-bindgen-macro-support" -version = "0.2.114" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "03ce4caeaac547cdf713d280eda22a730824dd11e6b8c3ca9e42247b25c631e3" -dependencies = [ - "bumpalo", - "proc-macro2", - "quote", - "syn 2.0.117", - "wasm-bindgen-shared", -] - -[[package]] -name = "wasm-bindgen-shared" -version = "0.2.114" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "75a326b8c223ee17883a4251907455a2431acc2791c98c26279376490c378c16" -dependencies = [ - "unicode-ident", -] - -[[package]] -name = "wasm-encoder" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "990065f2fe63003fe337b932cfb5e3b80e0b4d0f5ff650e6985b1048f62c8319" -dependencies = [ - "leb128fmt", - "wasmparser", -] - -[[package]] -name = "wasm-metadata" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bb0e353e6a2fbdc176932bbaab493762eb1255a7900fe0fea1a2f96c296cc909" -dependencies = [ - "anyhow", - "indexmap", - "wasm-encoder", - "wasmparser", -] - -[[package]] -name = "wasmparser" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "47b807c72e1bac69382b3a6fb3dbe8ea4c0ed87ff5629b8685ae6b9a611028fe" -dependencies = [ - "bitflags", - "hashbrown 0.15.5", - "indexmap", - "semver", -] - -[[package]] -name = "web-sys" -version = "0.3.91" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "854ba17bb104abfb26ba36da9729addc7ce7f06f5c0f90f3c391f8461cca21f9" -dependencies = [ - "js-sys", - "wasm-bindgen", -] - -[[package]] -name = "windows-core" -version = "0.62.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb" -dependencies = [ - "windows-implement", - "windows-interface", - "windows-link", - "windows-result", - "windows-strings", -] - -[[package]] -name = "windows-implement" -version = "0.60.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.117", -] - -[[package]] -name = "windows-interface" -version = "0.59.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.117", -] - -[[package]] -name = "windows-link" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" - -[[package]] -name = "windows-result" -version = "0.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5" -dependencies = [ - "windows-link", -] - -[[package]] -name = "windows-strings" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091" -dependencies = [ - "windows-link", -] - -[[package]] -name = "windows-sys" -version = "0.61.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" -dependencies = [ - "windows-link", -] - -[[package]] -name = "wit-bindgen" -version = "0.51.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d7249219f66ced02969388cf2bb044a09756a083d0fab1e566056b04d9fbcaa5" -dependencies = [ - "wit-bindgen-rust-macro", -] - -[[package]] -name = "wit-bindgen-core" -version = "0.51.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ea61de684c3ea68cb082b7a88508a8b27fcc8b797d738bfc99a82facf1d752dc" -dependencies = [ - "anyhow", - "heck", - "wit-parser", -] - -[[package]] -name = "wit-bindgen-rust" -version = "0.51.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b7c566e0f4b284dd6561c786d9cb0142da491f46a9fbed79ea69cdad5db17f21" -dependencies = [ - "anyhow", - "heck", - "indexmap", - "prettyplease", - "syn 2.0.117", - "wasm-metadata", - "wit-bindgen-core", - "wit-component", -] - -[[package]] -name = "wit-bindgen-rust-macro" -version = "0.51.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c0f9bfd77e6a48eccf51359e3ae77140a7f50b1e2ebfe62422d8afdaffab17a" -dependencies = [ - "anyhow", - "prettyplease", - "proc-macro2", - "quote", - "syn 2.0.117", - "wit-bindgen-core", - "wit-bindgen-rust", -] - -[[package]] -name = "wit-component" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9d66ea20e9553b30172b5e831994e35fbde2d165325bec84fc43dbf6f4eb9cb2" -dependencies = [ - "anyhow", - "bitflags", - "indexmap", - "log", - "serde", - "serde_derive", - "serde_json", - "wasm-encoder", - "wasm-metadata", - "wasmparser", - "wit-parser", -] - -[[package]] -name = "wit-parser" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ecc8ac4bc1dc3381b7f59c34f00b67e18f910c2c0f50015669dde7def656a736" -dependencies = [ - "anyhow", - "id-arena", - "indexmap", - "log", - "semver", - "serde", - "serde_derive", - "serde_json", - "unicode-xid", - "wasmparser", -] - -[[package]] -name = "wyz" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "05f360fc0b24296329c78fda852a1e9ae82de9cf7b27dae4b7f62f118f77b9ed" -dependencies = [ - "tap", -] - -[[package]] -name = "zerocopy" -version = "0.8.47" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "efbb2a062be311f2ba113ce66f697a4dc589f85e78a4aea276200804cea0ed87" -dependencies = [ - "zerocopy-derive", -] - -[[package]] -name = "zerocopy-derive" -version = "0.8.47" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0e8bc7269b54418e7aeeef514aa68f8690b8c0489a06b0136e5f57c4c5ccab89" -dependencies = [ - "proc-macro2", - "quote", - "syn 2.0.117", -] - -[[package]] -name = "zigzag" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "70b40401a28d86ce16a330b863b86fd7dbee4d7c940587ab09ab8c019f9e3fdf" -dependencies = [ - "num-traits", -] - -[[package]] -name = "zmij" -version = "1.0.21" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" - -[[package]] -name = "zstd" -version = "0.13.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e91ee311a569c327171651566e07972200e76fcfe2242a4fa446149a3881c08a" -dependencies = [ - "zstd-safe", -] - -[[package]] -name = "zstd-safe" -version = "7.2.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8f49c4d5f0abb602a93fb8736af2a4f4dd9512e36f7f570d66e65ff867ed3b9d" -dependencies = [ - "zstd-sys", -] - -[[package]] -name = "zstd-sys" -version = "2.0.16+zstd.1.5.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "91e19ebc2adc8f83e43039e79776e3fda8ca919132d68a1fed6a5faca2683748" -dependencies = [ - "cc", - "pkg-config", -] diff --git a/java/testfiles/Cargo.toml b/java/testfiles/Cargo.toml deleted file mode 100644 index 9362f735c1d..00000000000 --- a/java/testfiles/Cargo.toml +++ /dev/null @@ -1,11 +0,0 @@ -[package] -name = "testfiles" -version = "0.1.0" -description = "Generate Vortex files for Java integration tests" -authors = ["Vortex Authors "] -license = "Apache-2.0" -repository = "https://github.com/vortex-data/vortex" -edition = "2024" - -[dependencies] -vortex = { path = "../../vortex", features = ["files"] } diff --git a/java/testfiles/src/main.rs b/java/testfiles/src/main.rs deleted file mode 100644 index 1db61c7f08b..00000000000 --- a/java/testfiles/src/main.rs +++ /dev/null @@ -1,96 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -#![allow(clippy::expect_used)] - -use std::path::Path; - -use vortex::VortexSessionDefault; -use vortex::array::IntoArray; -use vortex::array::arrays::StructArray; -use vortex::array::builders::ArrayBuilder; -use vortex::array::builders::DecimalBuilder; -use vortex::array::builders::VarBinViewBuilder; -use vortex::array::validity::Validity; -use vortex::dtype::DType; -use vortex::dtype::DecimalDType; -use vortex::dtype::Nullability; -use vortex::file::WriteOptionsSessionExt; -use vortex::io::runtime::BlockingRuntime; -use vortex::io::runtime::current::CurrentThreadRuntime; -use vortex::io::session::RuntimeSessionExt; -use vortex::session::VortexSession; - -/// Generate a test dataset with the following small set of rows: -/// -/// | Name | Salary | State | -/// |--------|--------|-------| -/// | Alice | 1000 | CA | -/// | Bob | 2000 | NY | -/// | Carol | 3000 | TX | -/// | Dan | 4000 | CA | -/// | Edward | 5000 | NY | -/// | Frida | 6000 | TX | -/// | George | 7000 | CA | -/// | Henry | 8000 | NY | -/// | Ida | 9000 | TX | -/// | John | 10000 | VA | -fn main() { - let runtime = CurrentThreadRuntime::new(); - let session = VortexSession::default().with_handle(runtime.handle()); - - let mut names = VarBinViewBuilder::with_capacity(DType::Utf8(Nullability::NonNullable), 10); - names.append_value("Alice"); - names.append_value("Bob"); - names.append_value("Carol"); - names.append_value("Dan"); - names.append_value("Edward"); - names.append_value("Frida"); - names.append_value("George"); - names.append_value("Henry"); - names.append_value("Ida"); - names.append_value("John"); - let names = names.finish(); - - let mut salary = - DecimalBuilder::with_capacity::(10, DecimalDType::new(9, 2), Nullability::Nullable); - for i in 1..=10 { - salary.append_value(1_000 * i); - } - let salary = salary.finish(); - - let mut states = VarBinViewBuilder::with_capacity(DType::Utf8(Nullability::NonNullable), 10); - states.append_value("CA"); - states.append_value("NY"); - states.append_value("TX"); - states.append_value("CA"); - states.append_value("NY"); - states.append_value("TX"); - states.append_value("CA"); - states.append_value("NY"); - states.append_value("TX"); - states.append_value("VA"); - let states = states.finish(); - - // Make the struct array - let rows = StructArray::try_new( - ["Name", "Salary", "State"].into(), - vec![names, salary, states], - 10, - Validity::NonNullable, - ) - .expect("Could not create struct array") - .into_array(); - - // Save to file - let minimal_path = Path::new(env!("CARGO_MANIFEST_DIR")) - .join("../vortex-jni/src/test/resources/minimal.vortex"); - let mut file = std::fs::File::create(&minimal_path).expect("opening Vortex file"); - session - .write_options() - .blocking(&runtime) - .write(&mut file, rows.to_array_iterator()) - .expect("writing Vortex file"); - - println!("Wrote Vortex file to {}", minimal_path.display()); -} diff --git a/java/vortex-jni/bin/main/dev/vortex/DateTimeUtil.class b/java/vortex-jni/bin/main/dev/vortex/DateTimeUtil.class new file mode 100644 index 00000000000..61331568fe0 Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/DateTimeUtil.class differ diff --git a/java/vortex-jni/bin/main/dev/vortex/VortexCleaner.class b/java/vortex-jni/bin/main/dev/vortex/VortexCleaner.class new file mode 100644 index 00000000000..162348d3d7b Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/VortexCleaner.class differ diff --git a/java/vortex-jni/bin/main/dev/vortex/api/DataSource$RowCount$Estimate.class b/java/vortex-jni/bin/main/dev/vortex/api/DataSource$RowCount$Estimate.class new file mode 100644 index 00000000000..f024edfd951 Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/api/DataSource$RowCount$Estimate.class differ diff --git a/java/vortex-jni/bin/main/dev/vortex/api/DataSource$RowCount$Exact.class b/java/vortex-jni/bin/main/dev/vortex/api/DataSource$RowCount$Exact.class new file mode 100644 index 00000000000..9aea31e8dbc Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/api/DataSource$RowCount$Exact.class differ diff --git a/java/vortex-jni/bin/main/dev/vortex/api/DataSource$RowCount$Unknown.class b/java/vortex-jni/bin/main/dev/vortex/api/DataSource$RowCount$Unknown.class new file mode 100644 index 00000000000..a5f7888ffc7 Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/api/DataSource$RowCount$Unknown.class differ diff --git a/java/vortex-jni/bin/main/dev/vortex/api/DataSource$RowCount.class b/java/vortex-jni/bin/main/dev/vortex/api/DataSource$RowCount.class new file mode 100644 index 00000000000..b06b83518c5 Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/api/DataSource$RowCount.class differ diff --git a/java/vortex-jni/bin/main/dev/vortex/api/DataSource.class b/java/vortex-jni/bin/main/dev/vortex/api/DataSource.class new file mode 100644 index 00000000000..4914cc746d0 Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/api/DataSource.class differ diff --git a/java/vortex-jni/bin/main/dev/vortex/api/Expression$BinaryOp.class b/java/vortex-jni/bin/main/dev/vortex/api/Expression$BinaryOp.class new file mode 100644 index 00000000000..82a2a5dcbab Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/api/Expression$BinaryOp.class differ diff --git a/java/vortex-jni/bin/main/dev/vortex/api/Expression.class b/java/vortex-jni/bin/main/dev/vortex/api/Expression.class new file mode 100644 index 00000000000..3753c2f6bea Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/api/Expression.class differ diff --git a/java/vortex-jni/bin/main/dev/vortex/api/Partition.class b/java/vortex-jni/bin/main/dev/vortex/api/Partition.class new file mode 100644 index 00000000000..4db31fe7c83 Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/api/Partition.class differ diff --git a/java/vortex-jni/bin/main/dev/vortex/api/Scan.class b/java/vortex-jni/bin/main/dev/vortex/api/Scan.class new file mode 100644 index 00000000000..a8e2c5b5a16 Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/api/Scan.class differ diff --git a/java/vortex-jni/bin/main/dev/vortex/api/ScanOptions$SelectionMode.class b/java/vortex-jni/bin/main/dev/vortex/api/ScanOptions$SelectionMode.class new file mode 100644 index 00000000000..5fbcaf75f76 Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/api/ScanOptions$SelectionMode.class differ diff --git a/java/vortex-jni/bin/main/dev/vortex/api/ScanOptions.class b/java/vortex-jni/bin/main/dev/vortex/api/ScanOptions.class new file mode 100644 index 00000000000..f95697e450f Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/api/ScanOptions.class differ diff --git a/java/vortex-jni/bin/main/dev/vortex/api/Session.class b/java/vortex-jni/bin/main/dev/vortex/api/Session.class new file mode 100644 index 00000000000..0cfcb0e9380 Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/api/Session.class differ diff --git a/java/vortex-jni/bin/main/dev/vortex/api/VortexWriter.class b/java/vortex-jni/bin/main/dev/vortex/api/VortexWriter.class new file mode 100644 index 00000000000..6cb3670637e Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/api/VortexWriter.class differ diff --git a/java/vortex-jni/bin/main/dev/vortex/arrow/ArrowAllocation.class b/java/vortex-jni/bin/main/dev/vortex/arrow/ArrowAllocation.class new file mode 100644 index 00000000000..985a3b61f66 Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/arrow/ArrowAllocation.class differ diff --git a/java/vortex-jni/bin/main/dev/vortex/jni/NativeDataSource.class b/java/vortex-jni/bin/main/dev/vortex/jni/NativeDataSource.class new file mode 100644 index 00000000000..978d7217cb2 Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/jni/NativeDataSource.class differ diff --git a/java/vortex-jni/bin/main/dev/vortex/jni/NativeExpression.class b/java/vortex-jni/bin/main/dev/vortex/jni/NativeExpression.class new file mode 100644 index 00000000000..fc501ccd23e Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/jni/NativeExpression.class differ diff --git a/java/vortex-jni/bin/main/dev/vortex/jni/NativeFiles.class b/java/vortex-jni/bin/main/dev/vortex/jni/NativeFiles.class new file mode 100644 index 00000000000..4526cf0674c Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/jni/NativeFiles.class differ diff --git a/java/vortex-jni/bin/main/dev/vortex/jni/NativeLoader.class b/java/vortex-jni/bin/main/dev/vortex/jni/NativeLoader.class new file mode 100644 index 00000000000..d332a38d540 Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/jni/NativeLoader.class differ diff --git a/java/vortex-jni/bin/main/dev/vortex/jni/NativeLogging.class b/java/vortex-jni/bin/main/dev/vortex/jni/NativeLogging.class new file mode 100644 index 00000000000..4e23ae8e227 Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/jni/NativeLogging.class differ diff --git a/java/vortex-jni/bin/main/dev/vortex/jni/NativePartition.class b/java/vortex-jni/bin/main/dev/vortex/jni/NativePartition.class new file mode 100644 index 00000000000..5a5f8e046d0 Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/jni/NativePartition.class differ diff --git a/java/vortex-jni/bin/main/dev/vortex/jni/NativeRuntime.class b/java/vortex-jni/bin/main/dev/vortex/jni/NativeRuntime.class new file mode 100644 index 00000000000..cf22dd34d15 Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/jni/NativeRuntime.class differ diff --git a/java/vortex-jni/bin/main/dev/vortex/jni/NativeScan.class b/java/vortex-jni/bin/main/dev/vortex/jni/NativeScan.class new file mode 100644 index 00000000000..b276fecf36e Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/jni/NativeScan.class differ diff --git a/java/vortex-jni/bin/main/dev/vortex/jni/NativeSession.class b/java/vortex-jni/bin/main/dev/vortex/jni/NativeSession.class new file mode 100644 index 00000000000..1bf0add4635 Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/jni/NativeSession.class differ diff --git a/java/vortex-jni/bin/main/dev/vortex/jni/NativeWriter.class b/java/vortex-jni/bin/main/dev/vortex/jni/NativeWriter.class new file mode 100644 index 00000000000..6ec08d70fa4 Binary files /dev/null and b/java/vortex-jni/bin/main/dev/vortex/jni/NativeWriter.class differ diff --git a/java/vortex-jni/bin/test/dev/vortex/api/TestMinimal$BatchReader.class b/java/vortex-jni/bin/test/dev/vortex/api/TestMinimal$BatchReader.class new file mode 100644 index 00000000000..77dc713ce10 Binary files /dev/null and b/java/vortex-jni/bin/test/dev/vortex/api/TestMinimal$BatchReader.class differ diff --git a/java/vortex-jni/bin/test/dev/vortex/api/TestMinimal$Person.class b/java/vortex-jni/bin/test/dev/vortex/api/TestMinimal$Person.class new file mode 100644 index 00000000000..5d892e3552a Binary files /dev/null and b/java/vortex-jni/bin/test/dev/vortex/api/TestMinimal$Person.class differ diff --git a/java/vortex-jni/bin/test/dev/vortex/api/TestMinimal.class b/java/vortex-jni/bin/test/dev/vortex/api/TestMinimal.class new file mode 100644 index 00000000000..aac8261d63b Binary files /dev/null and b/java/vortex-jni/bin/test/dev/vortex/api/TestMinimal.class differ diff --git a/java/vortex-jni/bin/test/dev/vortex/jni/JNIWriterTest.class b/java/vortex-jni/bin/test/dev/vortex/jni/JNIWriterTest.class new file mode 100644 index 00000000000..cc770bf4a2e Binary files /dev/null and b/java/vortex-jni/bin/test/dev/vortex/jni/JNIWriterTest.class differ diff --git a/java/vortex-jni/build.gradle.kts b/java/vortex-jni/build.gradle.kts index 7bf389683c9..c056184aada 100644 --- a/java/vortex-jni/build.gradle.kts +++ b/java/vortex-jni/build.gradle.kts @@ -8,7 +8,7 @@ plugins { `java-library` `jvm-test-suite` id("com.google.protobuf") - id("com.gradleup.shadow") version "9.4.0" + id("com.gradleup.shadow") version "9.4.1" } dependencies { @@ -139,14 +139,13 @@ tasks.register("makeTestFiles") { } copy { - from("${rootProject.projectDir.absoluteFile.parentFile}/target/debug/libvortex_jni.dylib") - into("$projectDir/src/main/resources/native/darwin-aarch64") + from("${rootProject.projectDir.absoluteFile.parentFile}/target/debug/libvortex_jni.so") + into("$projectDir/src/main/resources/native/linux-aarch64") } - execOps.exec { - workingDir = rootProject.projectDir.absoluteFile.parentFile - executable = "cargo" - args("xtask", "java-test-files") + copy { + from("${rootProject.projectDir.absoluteFile.parentFile}/target/debug/libvortex_jni.dylib") + into("$projectDir/src/main/resources/native/darwin-aarch64") } } } diff --git a/java/vortex-jni/src/main/java/dev/vortex/DateTimeUtil.java b/java/vortex-jni/src/main/java/dev/vortex/DateTimeUtil.java index 56d218f8ab2..d397ad526b5 100644 --- a/java/vortex-jni/src/main/java/dev/vortex/DateTimeUtil.java +++ b/java/vortex-jni/src/main/java/dev/vortex/DateTimeUtil.java @@ -12,23 +12,20 @@ /** * Utility class for date and time conversions in Vortex. * - *

This class provides helper functions for converting Java date/time objects - * to nanoseconds since the Unix epoch, which is the internal representation - * used by Vortex for temporal data.

+ *

This class provides helper functions for converting Java date/time objects to nanoseconds since the Unix epoch, + * which is the internal representation used by Vortex for temporal data. * - *

Helpful functions borrowed from Iceberg class with same name.

+ *

Helpful functions borrowed from Iceberg class with same name. */ public final class DateTimeUtil { - /** - * The Unix epoch as an OffsetDateTime (1970-01-01T00:00:00Z). - */ + /** The Unix epoch as an OffsetDateTime (1970-01-01T00:00:00Z). */ public static final OffsetDateTime EPOCH = Instant.ofEpochSecond(0).atOffset(ZoneOffset.UTC); /** * Converts a LocalDateTime to nanoseconds since the Unix epoch. * - *

The LocalDateTime is assumed to be in UTC timezone for conversion purposes. - * When decoded back to LocalDateTime, the timezone information will not be preserved.

+ *

The LocalDateTime is assumed to be in UTC timezone for conversion purposes. When decoded back to + * LocalDateTime, the timezone information will not be preserved. * * @param dateTime the LocalDateTime to convert * @return nanoseconds since the Unix epoch @@ -40,7 +37,7 @@ public static long nanosFromTimestamp(LocalDateTime dateTime) { /** * Converts an OffsetDateTime to nanoseconds since the Unix epoch. * - *

The timezone information in the OffsetDateTime is preserved during conversion.

+ *

The timezone information in the OffsetDateTime is preserved during conversion. * * @param dateTime the OffsetDateTime to convert * @return nanoseconds since the Unix epoch diff --git a/java/vortex-jni/src/main/java/dev/vortex/VortexCleaner.java b/java/vortex-jni/src/main/java/dev/vortex/VortexCleaner.java new file mode 100644 index 00000000000..2e11f90197d --- /dev/null +++ b/java/vortex-jni/src/main/java/dev/vortex/VortexCleaner.java @@ -0,0 +1,14 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +package dev.vortex; + +import java.lang.ref.Cleaner; + +public final class VortexCleaner { + private static final Cleaner cleaner = Cleaner.create(); + + public static Cleaner.Cleanable register(Object obj, Runnable r) { + return VortexCleaner.cleaner.register(obj, r); + } +} diff --git a/java/vortex-jni/src/main/java/dev/vortex/api/Array.java b/java/vortex-jni/src/main/java/dev/vortex/api/Array.java deleted file mode 100644 index 06069c611de..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/api/Array.java +++ /dev/null @@ -1,191 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.api; - -import java.math.BigDecimal; -import org.apache.arrow.memory.BufferAllocator; -import org.apache.arrow.vector.VectorSchemaRoot; - -public interface Array extends AutoCloseable { - /** - * Returns the number of elements in this array. - * - * @return the length of the array - */ - long getLen(); - - /** - * Returns the total size in bytes of this array including all buffers. - * - * @return the size in bytes - */ - long nbytes(); - - /** - * Export to an ArrowVector. The data will now be owned by the VectorSchemaRoot after this operation. - */ - VectorSchemaRoot exportToArrow(BufferAllocator allocator, VectorSchemaRoot reuse); - - /** - * Returns the data type of this array. - * - * @return the DType describing the logical type of this array - */ - DType getDataType(); - - /** - * Returns a child array at the given field index. - * - *

This is used for accessing fields in struct arrays or elements in list arrays.

- * - * @param index the field index - * @return the child array at the specified index - * @throws IndexOutOfBoundsException if index is out of bounds - */ - Array getField(int index); - - /** - * Returns a slice of this array from start (inclusive) to stop (exclusive). - * - * @param start the starting index (inclusive) - * @param stop the ending index (exclusive) - * @return a new Array containing the sliced elements - * @throws IndexOutOfBoundsException if start or stop are out of bounds - */ - Array slice(int start, int stop); - - /** - * Returns true if the value at the given index is null. - * - * @param index the element index - * @return true if the value is null, false otherwise - * @throws IndexOutOfBoundsException if index is out of bounds - */ - boolean getNull(int index); - - /** - * Returns the total number of null values in this array. - * - * @return the null count - */ - int getNullCount(); - - /** - * Returns the byte value at the given index. - * - * @param index the element index - * @return the byte value - * @throws IndexOutOfBoundsException if index is out of bounds - * @throws ClassCastException if the array type is not compatible with byte - */ - byte getByte(int index); - - /** - * Returns the short value at the given index. - * - * @param index the element index - * @return the short value - * @throws IndexOutOfBoundsException if index is out of bounds - * @throws ClassCastException if the array type is not compatible with short - */ - short getShort(int index); - - /** - * Returns the int value at the given index. - * - * @param index the element index - * @return the int value - * @throws IndexOutOfBoundsException if index is out of bounds - * @throws ClassCastException if the array type is not compatible with int - */ - int getInt(int index); - - /** - * Returns the long value at the given index. - * - * @param index the element index - * @return the long value - * @throws IndexOutOfBoundsException if index is out of bounds - * @throws ClassCastException if the array type is not compatible with long - */ - long getLong(int index); - - /** - * Returns the boolean value at the given index. - * - * @param index the element index - * @return the boolean value - * @throws IndexOutOfBoundsException if index is out of bounds - * @throws ClassCastException if the array type is not compatible with boolean - */ - boolean getBool(int index); - - /** - * Returns the float value at the given index. - * - * @param index the element index - * @return the float value - * @throws IndexOutOfBoundsException if index is out of bounds - * @throws ClassCastException if the array type is not compatible with float - */ - float getFloat(int index); - - /** - * Returns the double value at the given index. - * - * @param index the element index - * @return the double value - * @throws IndexOutOfBoundsException if index is out of bounds - * @throws ClassCastException if the array type is not compatible with double - */ - double getDouble(int index); - - /** - * Returns the BigDecimal value at the given index. - * - * @param index the element index - * @return the BigDecimal value - * @throws IndexOutOfBoundsException if index is out of bounds - * @throws ClassCastException if the array type is not compatible with decimal - */ - BigDecimal getBigDecimal(int index); - - /** - * Returns the UTF-8 string value at the given index. - * - * @param index the element index - * @return the string value - * @throws IndexOutOfBoundsException if index is out of bounds - * @throws ClassCastException if the array type is not compatible with string - */ - String getUTF8(int index); - - /** - * Returns the UTF-8 string value at the given index as a pointer and length. - * - *

This is a low-level method that provides direct access to the underlying - * string data without copying. The pointer and length are written to the - * provided arrays.

- * - * @param index the element index - * @param ptr array to store the pointer (first element will be set) - * @param len array to store the length (first element will be set) - * @throws IndexOutOfBoundsException if index is out of bounds - * @throws ClassCastException if the array type is not compatible with string - */ - void getUTF8_ptr_len(int index, long[] ptr, int[] len); - - /** - * Returns the binary value at the given index as a byte array. - * - * @param index the element index - * @return the binary value as a byte array - * @throws IndexOutOfBoundsException if index is out of bounds - * @throws ClassCastException if the array type is not compatible with binary - */ - byte[] getBinary(int index); - - @Override - void close(); -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/api/ArrayIterator.java b/java/vortex-jni/src/main/java/dev/vortex/api/ArrayIterator.java deleted file mode 100644 index b6d29acbc62..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/api/ArrayIterator.java +++ /dev/null @@ -1,69 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.api; - -import java.util.Iterator; - -/** - * An iterator interface for traversing Vortex arrays while providing type information - * and resource management capabilities. - * - *

This interface extends both {@link Iterator} and {@link AutoCloseable}, allowing - * for efficient iteration over array elements while ensuring proper cleanup of any - * underlying resources. The iterator provides access to the data type of the arrays - * being iterated over, which is useful for type-safe processing.

- * - *

Example usage:

- *
{@code
- * try (ArrayIterator iterator = array.getIterator()) {
- *     DType dataType = iterator.getDataType();
- *     while (iterator.hasNext()) {
- *         Array element = iterator.next();
- *         // Process element based on dataType
- *     }
- * }
- * }
- * - * @see Array - * @see DType - * @see Iterator - * @see AutoCloseable - */ -public interface ArrayIterator extends AutoCloseable, Iterator { - /** - * Returns the data type of the arrays that this iterator produces. - * - *

This method provides type information about the arrays that will be - * returned by subsequent calls to {@link #next()}. The data type remains - * constant throughout the lifetime of the iterator and can be used for - * type-safe processing and validation.

- * - * @return the {@link DType} representing the data type of arrays produced by this iterator - */ - DType getDataType(); - - /** - * Closes this iterator and releases any underlying resources. - * - *

This method should be called when the iterator is no longer needed to - * ensure proper cleanup of any native resources or memory allocations. - * After calling this method, the iterator should not be used for further - * operations.

- * - *

It is recommended to use this iterator within a try-with-resources - * statement to ensure automatic cleanup:

- *
{@code
-     * try (ArrayIterator iterator = array.getIterator()) {
-     *     // Use iterator
-     * } // close() is called automatically
-     * }
- * - *

This method overrides {@link AutoCloseable#close()} and does not - * throw any checked exceptions, making it safe to use in any context.

- * - * @see AutoCloseable#close() - */ - @Override - void close(); -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/api/DType.java b/java/vortex-jni/src/main/java/dev/vortex/api/DType.java deleted file mode 100644 index 2c190b07be8..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/api/DType.java +++ /dev/null @@ -1,549 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.api; - -import dev.vortex.jni.JNIDType; -import dev.vortex.jni.NativeDTypeMethods; -import java.util.List; -import java.util.Optional; - -/** - * Vortex logical type interface representing the schema and metadata for array data. - * - *

DType defines the logical type system used by Vortex arrays, including primitive types, - * complex types like structs and lists, and temporal types with associated metadata. - * This interface provides methods to inspect type variants, nullability, temporal properties, - * decimal precision, and structural information for complex types. - * - *

Implementations of this interface are typically obtained from Vortex arrays and - * should be properly closed when no longer needed to free native resources. - */ -public interface DType extends AutoCloseable { - - /** - * Returns the variant of this data type. - * - * @return the {@link Variant} enum value representing the specific type category - */ - Variant getVariant(); - - /** - * Checks if this data type allows null values. - * - * @return {@code true} if the type is nullable, {@code false} otherwise - */ - boolean isNullable(); - - /** - * Get the field names for a STRUCT type. - */ - List getFieldNames(); - - /** - * Get the field types for a STRUCT type. - */ - List getFieldTypes(); - - /** - * Get the element type for a LIST or FIXED_SIZE_LIST type. - */ - DType getElementType(); - - /** - * Get the fixed size for a FIXED_SIZE_LIST type. - */ - int getFixedSizeListSize(); - - /** - * Checks if this data type represents a date. - * - * @return {@code true} if this is a date type, {@code false} otherwise - */ - boolean isDate(); - - /** - * Checks if this data type represents a time. - * - * @return {@code true} if this is a time type, {@code false} otherwise - */ - boolean isTime(); - - /** - * Checks if this data type represents a timestamp. - * - * @return {@code true} if this is a timestamp type, {@code false} otherwise - */ - boolean isTimestamp(); - - /** - * Returns the time unit for temporal data types. - * - * @return the {@link TimeUnit} for this temporal type - * @throws IllegalStateException if this is not a temporal type - */ - TimeUnit getTimeUnit(); - - /** - * Returns the timezone for timestamp data types. - * - * @return an {@link Optional} containing the timezone string if present, - * or empty if no timezone is specified - */ - Optional getTimeZone(); - - /** - * Checks if this data type represents a decimal number. - * - * @return {@code true} if this is a decimal type, {@code false} otherwise - */ - boolean isDecimal(); - - /** - * Returns the precision for decimal data types. - * - * @return the precision (total number of digits) for decimal types - * @throws IllegalStateException if this is not a decimal type - */ - int getPrecision(); - - /** - * Returns the scale for decimal data types. - * - * @return the scale (number of digits after the decimal point) for decimal types - * @throws IllegalStateException if this is not a decimal type - */ - byte getScale(); - - /** - * Closes this DType and releases any associated native resources. - * - *

After calling this method, the DType should not be used again. - * This method is idempotent and can be called multiple times safely. - */ - @Override - void close(); - - /** - * Create a new signed INT8 data type. - * - * @param isNullable True if the values can be null - * @return The new DType instance, allocated in native heap memory - */ - static DType newByte(boolean isNullable) { - return new JNIDType(NativeDTypeMethods.newInt(isNullable), true); - } - - /** - * Create a new signed INT16 data type. - * - * @param isNullable True if the values can be null - * @return The new DType instance, allocated in native heap memory - */ - static DType newShort(boolean isNullable) { - return new JNIDType(NativeDTypeMethods.newShort(isNullable), true); - } - - /** - * Create a new signed INT32 data type. - * - * @param isNullable True if the values can be null - * @return The new DType instance, allocated in native heap memory - */ - static DType newInt(boolean isNullable) { - return new JNIDType(NativeDTypeMethods.newInt(isNullable), true); - } - - /** - * Create a new signed INT64 data type. - * - * @param isNullable True if the values can be null - * @return The new DType instance, allocated in native heap memory - */ - static DType newLong(boolean isNullable) { - return new JNIDType(NativeDTypeMethods.newLong(isNullable), true); - } - - /** - * Create a new signed FLOAT32 data type. - * - * @param isNullable True if the values can be null - * @return The new DType instance, allocated in native heap memory - */ - static DType newFloat(boolean isNullable) { - return new JNIDType(NativeDTypeMethods.newFloat(isNullable), true); - } - - /** - * Create a new signed FLOAT64 data type. - * - * @param isNullable True if the values can be null - * @return The new DType instance, allocated in native heap memory - */ - static DType newDouble(boolean isNullable) { - return new JNIDType(NativeDTypeMethods.newDouble(isNullable), true); - } - - /** - * Create a new Decimal data type. - * - * @param precision Decimal values precision - * @param scale Decimal values scale - * @param isNullable True if the values can be null - * @return The new DType instance, allocated in native heap memory - */ - static DType newDecimal(int precision, int scale, boolean isNullable) { - return new JNIDType(NativeDTypeMethods.newDecimal(precision, scale, isNullable), true); - } - - /** - * Create a new UTF-8 string data type. - * - * @param isNullable True if the values can be null - * @return The new DType instance, allocated in native heap memory - */ - static DType newUtf8(boolean isNullable) { - return new JNIDType(NativeDTypeMethods.newUtf8(isNullable), true); - } - - /** - * Create a new Binary data type. - * - * @param isNullable True if the values can be null - * @return The new DType instance, allocated in native heap memory - */ - static DType newBinary(boolean isNullable) { - return new JNIDType(NativeDTypeMethods.newBinary(isNullable), true); - } - - /** - * Create a new Binary data type. - * - * @param isNullable True if the values can be null - * @return The new DType instance, allocated in native heap memory - */ - static DType newBool(boolean isNullable) { - return new JNIDType(NativeDTypeMethods.newBool(isNullable), true); - } - - /** - * Create a new List data type. - * - * @param element DType of the list elements - * @param isNullable True if the values can be null - * @return The new DType instance, allocated in native heap memory - */ - static DType newList(DType element, boolean isNullable) { - // Get the pointer - JNIDType jniType = (JNIDType) element; - return new JNIDType(NativeDTypeMethods.newList(jniType.getPointer(), isNullable), true); - } - - /** - * Create a new FixedSizeList data type. - * - * @param element DType of the list elements - * @param size The fixed size of each list - * @param isNullable True if the values can be null - * @return The new DType instance, allocated in native heap memory - */ - static DType newFixedSizeList(DType element, int size, boolean isNullable) { - JNIDType jniType = (JNIDType) element; - return new JNIDType(NativeDTypeMethods.newFixedSizeList(jniType.getPointer(), size, isNullable), true); - } - - /** - * Create a new Struct data type. - * - * @param fieldNames Name of each of the fields - * @param fieldTypes DType for each field - * @param isNullable True if the values can be null - * @return The new DType instance, allocated in native heap memory - */ - static DType newStruct(String[] fieldNames, DType[] fieldTypes, boolean isNullable) { - long[] ptrs = new long[fieldTypes.length]; - for (int i = 0; i < fieldTypes.length; i++) { - ptrs[i] = ((JNIDType) fieldTypes[i]).getPointer(); - } - return new JNIDType(NativeDTypeMethods.newStruct(fieldNames, ptrs, isNullable), true); - } - - /** - * Create a new Timestamp data type. - * - * @param unit Time unit for the timestamp values - * @param timeZone Optional time zone for the timestamp values - * @param isNullable True if the values can be null - * @return The new DType instance, allocated in native heap memory - */ - static DType newTimestamp(TimeUnit unit, Optional timeZone, boolean isNullable) { - byte timeUnit = unit.asByte(); - return new JNIDType(NativeDTypeMethods.newTimestamp(timeUnit, timeZone.orElse(null), isNullable), true); - } - - /** - * Create a new Date data type. - * - * @param unit Time unit for the value - * @param isNullable True if the values can be null - * @return The new DType instance, allocated in native heap memory - */ - static DType newDate(TimeUnit unit, boolean isNullable) { - byte timeUnit = unit.asByte(); - return new JNIDType(NativeDTypeMethods.newDate(timeUnit, isNullable), true); - } - - /** - * Create a new Time data type. - * - * @param unit Time unit for the value - * @param isNullable True if the values can be null - * @return The new DType instance, allocated in native heap memory - */ - static DType newTime(TimeUnit unit, boolean isNullable) { - byte timeUnit = unit.asByte(); - return new JNIDType(NativeDTypeMethods.newTime(timeUnit, isNullable), true); - } - - /** - * Enumeration of time units supported by Vortex temporal data types. - * - *

Time units define the granularity of temporal values and are used - * by date, time, and timestamp data types to specify their precision. - */ - enum TimeUnit { - /** - * Nanosecond precision (10^-9 seconds) - */ - NANOSECONDS, - - /** - * Microsecond precision (10^-6 seconds) - */ - MICROSECONDS, - - /** - * Millisecond precision (10^-3 seconds) - */ - MILLISECONDS, - - /** - * Second precision - */ - SECONDS, - - /** - * Day precision (24-hour periods) - */ - DAYS, - ; - - /** - * Converts a byte value to the corresponding TimeUnit enum. - * - * @param unit the byte value representing the time unit (0-4) - * @return the corresponding {@link TimeUnit} enum value - * @throws RuntimeException if the unit value is not recognized - */ - public static TimeUnit from(byte unit) { - switch (unit) { - case 0: - return NANOSECONDS; - case 1: - return MICROSECONDS; - case 2: - return MILLISECONDS; - case 3: - return SECONDS; - case 4: - return DAYS; - default: - throw new IllegalArgumentException("Unknown TimeUnit: " + unit); - } - } - - /** - * Get the byte value of this TimeUnit. - */ - public byte asByte() { - switch (this) { - case NANOSECONDS: - return 0; - case MICROSECONDS: - return 1; - case MILLISECONDS: - return 2; - case SECONDS: - return 3; - case DAYS: - return 4; - default: - throw new IllegalArgumentException("Unknown TimeUnit: " + this); - } - } - } - - /** - * Enumeration of all supported data type variants in Vortex. - * - *

Each variant represents a different category of data type, from primitive - * numeric types to complex structured types. This enum provides a way to - * categorize and identify the specific type of data stored in a Vortex array. - */ - enum Variant { - /** - * Null type representing absence of value - */ - NULL, - - /** - * Boolean type for true/false values - */ - BOOL, - - /** - * Unsigned 8-bit integer type - */ - PRIMITIVE_U8, - - /** - * Unsigned 16-bit integer type - */ - PRIMITIVE_U16, - - /** - * Unsigned 32-bit integer type - */ - PRIMITIVE_U32, - - /** - * Unsigned 64-bit integer type - */ - PRIMITIVE_U64, - - /** - * Signed 8-bit integer type - */ - PRIMITIVE_I8, - - /** - * Signed 16-bit integer type - */ - PRIMITIVE_I16, - - /** - * Signed 32-bit integer type - */ - PRIMITIVE_I32, - - /** - * Signed 64-bit integer type - */ - PRIMITIVE_I64, - - /** - * 16-bit floating point type - */ - PRIMITIVE_F16, - - /** - * 32-bit floating point type - */ - PRIMITIVE_F32, - - /** - * 64-bit floating point type - */ - PRIMITIVE_F64, - - /** - * UTF-8 encoded string type - */ - UTF8, - - /** - * Binary data type for arbitrary byte sequences - */ - BINARY, - - /** - * Structured type containing named fields - */ - STRUCT, - - /** - * List type containing elements of a single type - */ - LIST, - - /** - * Extension type for custom or domain-specific types - */ - EXTENSION, - - /** - * Decimal type for precise numeric values - */ - DECIMAL, - - /** - * Fixed-size list type containing a fixed number of elements of a single type - */ - FIXED_SIZE_LIST, - ; - - /** - * Converts a byte value to the corresponding Variant enum. - * - * @param variant the byte value representing the variant (0-19) - * @return the corresponding {@link Variant} enum value - * @throws RuntimeException if the variant value is not recognized - */ - public static Variant from(byte variant) { - switch (variant) { - case 0: - return NULL; - case 1: - return BOOL; - case 2: - return PRIMITIVE_U8; - case 3: - return PRIMITIVE_U16; - case 4: - return PRIMITIVE_U32; - case 5: - return PRIMITIVE_U64; - case 6: - return PRIMITIVE_I8; - case 7: - return PRIMITIVE_I16; - case 8: - return PRIMITIVE_I32; - case 9: - return PRIMITIVE_I64; - case 10: - return PRIMITIVE_F16; - case 11: - return PRIMITIVE_F32; - case 12: - return PRIMITIVE_F64; - case 13: - return UTF8; - case 14: - return BINARY; - case 15: - return STRUCT; - case 16: - return LIST; - case 17: - return EXTENSION; - case 18: - return DECIMAL; - case 19: - return FIXED_SIZE_LIST; - default: - throw new IllegalArgumentException("Unknown DType variant: " + variant); - } - } - } -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/api/DataSource.java b/java/vortex-jni/src/main/java/dev/vortex/api/DataSource.java new file mode 100644 index 00000000000..ab812849c55 --- /dev/null +++ b/java/vortex-jni/src/main/java/dev/vortex/api/DataSource.java @@ -0,0 +1,148 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +package dev.vortex.api; + +import com.google.common.base.Preconditions; +import dev.vortex.VortexCleaner; +import dev.vortex.jni.NativeDataSource; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.OptionalLong; +import org.apache.arrow.c.ArrowSchema; +import org.apache.arrow.c.Data; +import org.apache.arrow.memory.BufferAllocator; +import org.apache.arrow.vector.types.pojo.Schema; + +/** + * A set of Vortex files opened through a {@link Session}. Data sources are cheap to open (only the first file is read + * eagerly, to determine the schema) and can be scanned multiple times. + * + *

Native resources are released automatically via {@link VortexCleaner} when the data source becomes unreachable. + */ +public final class DataSource { + private final Session session; + private final long pointer; + + private DataSource(Session session, long pointer) { + Preconditions.checkArgument(pointer != 0, "invalid data source pointer"); + this.session = Objects.requireNonNull(session, "session"); + this.pointer = pointer; + VortexCleaner.register(this, () -> NativeDataSource.free(pointer)); + } + + /** Open a single URI. */ + public static DataSource open(Session session, String uri) { + return open(session, uri, Collections.emptyMap()); + } + + /** + * Open one or more URIs or globs. When a glob is used, the first match is opened eagerly; subsequent matches are + * opened lazily on scan. + * + * @param session open session + * @param uri single URI or glob + * @param properties object-store credentials / options + */ + public static DataSource open(Session session, String uri, Map properties) { + return open(session, List.of(uri), properties); + } + + /** + * Open one or more URIs or globs. When a glob is used, the first match is opened eagerly; subsequent matches are + * opened lazily on scan. + * + * @param session open session + * @param uris URIs or globs to scan + * @param properties object-store credentials / options + */ + public static DataSource open(Session session, List uris, Map properties) { + Objects.requireNonNull(session, "session"); + Objects.requireNonNull(uris, "uris"); + Preconditions.checkArgument(!uris.isEmpty(), "at least one uri is required"); + String[] uriArray = uris.toArray(String[]::new); + Preconditions.checkArgument( + Arrays.stream(uriArray).allMatch(Objects::nonNull), "uris must not contain null values"); + long sessionPointer = session.nativePointer(); + long pointer = NativeDataSource.open(sessionPointer, uriArray, properties); + return new DataSource(session, pointer); + } + + /** Arrow schema of the data source (and of scans produced from it). */ + public Schema arrowSchema(BufferAllocator allocator) { + try (ArrowSchema schema = ArrowSchema.allocateNew(allocator)) { + NativeDataSource.arrowSchema(pointer, schema.memoryAddress()); + return Data.importSchema(allocator, schema, null); + } + } + + /** + * Row count along with the precision of that estimate. Mirrors the Rust {@code Option>} returned by + * {@code DataSource::row_count}: {@link RowCount.Unknown} when no estimate is available, {@link RowCount.Estimate} + * for an inexact hint, {@link RowCount.Exact} when the count is authoritative. + */ + public RowCount rowCount() { + long[] out = new long[2]; + NativeDataSource.rowCount(pointer, out); + return switch ((int) out[1]) { + case 1 -> new RowCount.Estimate(out[0]); + case 2 -> new RowCount.Exact(out[0]); + default -> RowCount.Unknown.INSTANCE; + }; + } + + /** Precision-aware row count. See {@link #rowCount()}. */ + public sealed interface RowCount { + /** Returns the row count as a long, or {@code OptionalLong.empty()} when unknown. */ + OptionalLong asOptional(); + + /** Row count is not known. */ + final class Unknown implements RowCount { + public static final Unknown INSTANCE = new Unknown(); + + private Unknown() {} + + @Override + public OptionalLong asOptional() { + return OptionalLong.empty(); + } + } + + /** Estimated row count; the actual value may differ. */ + record Estimate(long value) implements RowCount { + @Override + public OptionalLong asOptional() { + return OptionalLong.of(value); + } + } + + /** Exact row count. */ + record Exact(long value) implements RowCount { + @Override + public OptionalLong asOptional() { + return OptionalLong.of(value); + } + } + } + + /** Submit a scan. */ + public Scan scan(ScanOptions options) { + Objects.requireNonNull(options, "options"); + + long projectionPtr = options.projection().map(Expression::nativePointer).orElse(0L); + long filterPtr = options.filter().map(Expression::nativePointer).orElse(0L); + long begin = options.rowRangeBegin().orElse(0L); + long end = options.rowRangeEnd().orElse(0L); + long[] selectionIndices = options.selectionIndices().orElse(null); + byte selectionMode = options.selectionMode().code(); + long limit = options.limit().orElse(0L); + boolean ordered = options.ordered(); + + long scanPtr = dev.vortex.jni.NativeScan.create( + pointer, projectionPtr, filterPtr, begin, end, selectionIndices, selectionMode, limit, ordered); + return Scan.fromPointer(session, scanPtr); + } +} diff --git a/java/vortex-jni/src/main/java/dev/vortex/api/Expression.java b/java/vortex-jni/src/main/java/dev/vortex/api/Expression.java index 8dc4ec881b4..e4d8978e112 100644 --- a/java/vortex-jni/src/main/java/dev/vortex/api/Expression.java +++ b/java/vortex-jni/src/main/java/dev/vortex/api/Expression.java @@ -3,101 +3,139 @@ package dev.vortex.api; -import dev.vortex.api.expressions.*; -import java.util.List; -import java.util.Optional; +import com.google.common.base.Preconditions; +import dev.vortex.VortexCleaner; +import dev.vortex.jni.NativeExpression; +import java.util.Arrays; /** - * Vortex expression language. + * A Vortex expression node backed by a native pointer. + * + *

Expressions are composed via the static factories ({@link #root()}, {@link #getItem(String, Expression)}, etc.). + * Each returned {@code Expression} owns its native pointer; the pointer is released automatically when the + * {@code Expression} is no longer reachable. Passing an expression as an input to a builder does not transfer + * ownership — the resulting expression is an independent copy on the native side. */ -public interface Expression { - /** - * The globally unique identifier for this type of expression. - */ - String id(); - - /** - * Returns the children of this expression. - */ - List children(); - - /** - * Returns the serialized metadata for this expression, or empty if serialization is not supported. - */ - Optional metadata(); - - /** - * Accepts a visitor and dispatches to the appropriate visit method based on the expression type. - * This method implements the visitor pattern, allowing different operations to be performed - * on expressions without modifying the expression classes themselves. - * - * @param the return type of the visitor - * @param visitor the visitor to accept - * @return the result of the visitor's operation on this expression - */ - default T accept(Visitor visitor) { - return visitor.visitOther(this); - } - - /** - * Visitor interface for implementing the visitor pattern on expressions. - * This interface defines methods for visiting different types of expressions, - * allowing for type-safe operations across the expression hierarchy. - * - * @param the return type of the visitor methods - */ - interface Visitor { - /** - * Visits a literal expression. - * - * @param literal the literal expression to visit - * @return the result of visiting the literal expression - */ - T visitLiteral(Literal literal); - - /** - * Visits a root expression. - * - * @param root the root expression to visit - * @return the result of visiting the root expression - */ - T visitRoot(Root root); - - /** - * Visits a binary expression. - * - * @param binary the binary expression to visit - * @return the result of visiting the binary expression - */ - T visitBinary(Binary binary); - - /** - * Visits a not expression (logical negation). - * - * @param not the not expression to visit - * @return the result of visiting the not expression - */ - T visitNot(Not not); - - /** - * Visits a get item expression (array/object indexing). - * - * @param getItem the get item expression to visit - * @return the result of visiting the get item expression - */ - T visitGetItem(GetItem getItem); - - /** - * Visits an is null expression (null check). - * - * @param isNull the is null expression to visit - * @return the result of visiting the is null expression - */ - T visitIsNull(IsNull isNull); - - /** - * For expressions that do not have a specific visitor method. - */ - T visitOther(Expression expression); +public final class Expression { + private final long pointer; + + private Expression(long pointer) { + Preconditions.checkArgument(pointer != 0, "invalid expression pointer"); + this.pointer = pointer; + VortexCleaner.register(this, () -> NativeExpression.free(pointer)); + } + + long nativePointer() { + return pointer; + } + + /** The root expression: applying it to an array yields the array itself. */ + public static Expression root() { + return new Expression(NativeExpression.root()); + } + + /** Access a named field from a struct expression. */ + public static Expression getItem(String fieldName, Expression child) { + return new Expression(NativeExpression.getItem(fieldName, child.nativePointer())); + } + + /** Shortcut for {@code getItem(fieldName, root())}. */ + public static Expression column(String fieldName) { + return getItem(fieldName, root()); + } + + /** Project a subset of fields out of a struct expression. */ + public static Expression select(String[] fieldNames, Expression child) { + return new Expression(NativeExpression.select(fieldNames, child.nativePointer())); + } + + /** Logical AND. Requires at least one operand. */ + public static Expression and(Expression... operands) { + Preconditions.checkArgument(operands.length > 0, "and requires at least one operand"); + return new Expression(NativeExpression.and(nativePointers(operands))); + } + + /** Logical OR. Requires at least one operand. */ + public static Expression or(Expression... operands) { + Preconditions.checkArgument(operands.length > 0, "or requires at least one operand"); + return new Expression(NativeExpression.or(nativePointers(operands))); + } + + public static Expression binary(BinaryOp op, Expression lhs, Expression rhs) { + return new Expression(NativeExpression.binary(op.code(), lhs.nativePointer(), rhs.nativePointer())); + } + + public static Expression not(Expression child) { + return new Expression(NativeExpression.not(child.nativePointer())); + } + + public static Expression isNull(Expression child) { + return new Expression(NativeExpression.isNull(child.nativePointer())); + } + + public static Expression literal(boolean value) { + return new Expression(NativeExpression.literalBool(value, false)); + } + + public static Expression nullLiteralBool() { + return new Expression(NativeExpression.literalBool(false, true)); + } + + public static Expression literal(byte value) { + return new Expression(NativeExpression.literalI8(value, false)); + } + + public static Expression literal(short value) { + return new Expression(NativeExpression.literalI16(value, false)); + } + + public static Expression literal(int value) { + return new Expression(NativeExpression.literalI32(value, false)); + } + + public static Expression literal(long value) { + return new Expression(NativeExpression.literalI64(value, false)); + } + + public static Expression literal(float value) { + return new Expression(NativeExpression.literalF32(value, false)); + } + + public static Expression literal(double value) { + return new Expression(NativeExpression.literalF64(value, false)); + } + + public static Expression literal(String value) { + return new Expression(NativeExpression.literalString(value)); + } + + private static long[] nativePointers(Expression[] exprs) { + return Arrays.stream(exprs).mapToLong(Expression::nativePointer).toArray(); + } + + /** Binary operator codes; must match the Rust {@code parse_op} table. */ + public enum BinaryOp { + EQ((byte) 0), + NOT_EQ((byte) 1), + GT((byte) 2), + GTE((byte) 3), + LT((byte) 4), + LTE((byte) 5), + AND((byte) 6), + OR((byte) 7), + ADD((byte) 8), + SUB((byte) 9), + MUL((byte) 10), + DIV((byte) 11); + + private final byte code; + + BinaryOp(byte code) { + this.code = code; + } + + public byte code() { + return code; + } } } diff --git a/java/vortex-jni/src/main/java/dev/vortex/api/File.java b/java/vortex-jni/src/main/java/dev/vortex/api/File.java deleted file mode 100644 index 414f0a47de0..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/api/File.java +++ /dev/null @@ -1,106 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.api; - -/** - * Interface for reading Vortex format files, providing access to schema information, - * row metadata, and configurable scanning capabilities. - * - *

A {@code File} represents a Vortex format file that has been opened for reading. - * It provides methods to inspect the file's schema, count rows, and create iterators - * for scanning the data with various filtering and projection options. This interface - * extends {@link AutoCloseable} to ensure proper resource cleanup when the file - * is no longer needed.

- * - *

Example usage:

- *
{@code
- * try (File file = VortexReader.open(path)) {
- *     DType schema = file.getDType();
- *     long totalRows = file.rowCount();
- *
- *     ScanOptions options = ScanOptions.builder()
- *         .columns(List.of("name", "age"))
- *         .build();
- *
- *     try (ArrayIterator iterator = file.newScan(options)) {
- *         while (iterator.hasNext()) {
- *             Array batch = iterator.next();
- *             // Process batch
- *         }
- *     }
- * }
- * }
- * - * @see ScanOptions - * @see ArrayIterator - * @see DType - * @see AutoCloseable - */ -public interface File extends AutoCloseable { - /** - * Returns the data type (schema) of this Vortex file. - * - *

The returned {@link DType} describes the logical structure and types - * of the data contained in this file. For structured data, this will typically - * be a {@link DType.Variant#STRUCT} containing field names and their corresponding - * data types. The schema remains constant for the lifetime of the file.

- * - * @return the {@link DType} representing the schema of this file - */ - DType getDType(); - - /** - * Returns the total number of rows in this Vortex file. - * - *

This method provides the count of logical rows contained in the file, - * which represents the number of records or tuples that can be read. This - * count is independent of any filtering or projection that may be applied - * during scanning operations.

- * - * @return the total number of rows as a non-negative long value - */ - long rowCount(); - - /** - * Creates a new iterator for scanning this file with the specified options. - * - *

This method returns an {@link ArrayIterator} that can be used to traverse - * the data in this file according to the provided {@link ScanOptions}. The - * scan options allow for column projection, row filtering via predicates, - * and row range or index selection. Each call to this method creates a new - * independent iterator.

- * - *

The returned iterator must be properly closed when no longer needed to - * release any underlying resources. It is recommended to use the iterator - * within a try-with-resources statement.

- * - * @param options the {@link ScanOptions} configuring the scan behavior, - * including column selection, filtering, and row selection - * @return a new {@link ArrayIterator} for scanning the file data - * @throws RuntimeException if the scan options contain invalid - * column names or conflicting row selection criteria - * @see ScanOptions - * @see ArrayIterator - */ - ArrayIterator newScan(ScanOptions options); - - /** - * Closes this file and releases any associated resources. - * - *

This method should be called when the file is no longer needed to ensure - * proper cleanup of any underlying file handles, native memory, or other resources. - * After calling this method, the file should not be used for any further operations. - * This method is idempotent and can be called multiple times safely.

- * - *

It is recommended to use this file within a try-with-resources statement - * to ensure automatic cleanup:

- *
{@code
-     * try (File file = VortexReader.open(path)) {
-     *     // Use file
-     * } // close() is called automatically
-     * }
- */ - @Override - void close(); -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/api/Files.java b/java/vortex-jni/src/main/java/dev/vortex/api/Files.java deleted file mode 100644 index 8483c26be56..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/api/Files.java +++ /dev/null @@ -1,91 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.api; - -import com.google.common.base.Preconditions; -import dev.vortex.jni.JNIFile; -import dev.vortex.jni.NativeFileMethods; -import java.net.URI; -import java.nio.file.Paths; -import java.util.Map; - -/** - * Utility class for opening and accessing Vortex files. - * - *

This class provides static methods to open Vortex files from various sources, - * including local file system paths and URIs. It supports both simple path-based - * access and more advanced configuration through properties. - * - *

All methods in this class are static and the class cannot be instantiated. - * - * @since 1.0 - */ -public final class Files { - - /** - * Private constructor to prevent instantiation of this utility class. - */ - private Files() {} - - /** - * Opens a Vortex file from the specified path string. - * - * @see #open(String, Map) - */ - public static File open(String path) { - return open(path, Map.of()); - } - - /** - * Opens a Vortex file from the specified path string and format parameters. - * - *

This method provides a convenient way to open Vortex files using either - * absolute file system paths or URI strings. If the path starts with "/", - * it is treated as an absolute file system path and converted to a file URI. - * Otherwise, it is parsed as a URI directly. - * - *

This is equivalent to calling {@code open(uri, Map.of())} with no additional - * properties. - * - * @param path the path to the Vortex file, either as an absolute file system path - * (starting with "/") or as a URI string - * @return a {@link File} instance representing the opened Vortex file - * @throws RuntimeException if the file cannot be opened or the path is invalid - * @throws NullPointerException if path is null - * @see #open(URI, Map) - */ - public static File open(String path, Map properties) { - if (path.startsWith("/")) { - return open(Paths.get(path).toUri(), properties); - } - return open(URI.create(path), properties); - } - - /** - * Opens a Vortex file from the specified URI with additional properties. - * - *

This method provides the most flexible way to open Vortex files, allowing - * you to specify both the location via a URI and additional configuration - * properties. The URI can point to local files, remote resources, or other - * supported storage systems. - * - *

The properties map can be used to pass configuration options to the - * underlying file system or storage layer. The specific properties supported - * depend on the URI scheme and storage backend being used. - * - * @param uri the URI pointing to the Vortex file to open - * @param properties a map of configuration properties for opening the file; - * may be empty but must not be null - * @return a {@link File} instance representing the opened Vortex file - * @throws RuntimeException if the file cannot be opened, the URI is invalid, - * or the returned native pointer is invalid - * @throws NullPointerException if uri or properties is null - * @see #open(String) - */ - public static File open(URI uri, Map properties) { - long ptr = NativeFileMethods.open(uri.toString(), properties); - Preconditions.checkArgument(ptr > 0, "Failed to open file: %s", uri); - return new JNIFile(ptr); - } -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/api/Partition.java b/java/vortex-jni/src/main/java/dev/vortex/api/Partition.java new file mode 100644 index 00000000000..09ca58b4e87 --- /dev/null +++ b/java/vortex-jni/src/main/java/dev/vortex/api/Partition.java @@ -0,0 +1,74 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +package dev.vortex.api; + +import com.google.common.base.Preconditions; +import dev.vortex.VortexCleaner; +import dev.vortex.jni.NativePartition; +import java.util.OptionalLong; +import java.util.concurrent.atomic.AtomicBoolean; +import org.apache.arrow.c.ArrowArrayStream; +import org.apache.arrow.c.Data; +import org.apache.arrow.memory.BufferAllocator; +import org.apache.arrow.vector.ipc.ArrowReader; + +/** + * A unit of scan work that materializes into an Arrow stream. Partitions are single-pass: calling + * {@link #scanArrow(BufferAllocator)} consumes the partition and transfers ownership of its native memory to the + * returned {@link ArrowReader}. If the partition is never consumed, its native memory is released automatically via + * {@link VortexCleaner}. + */ +public final class Partition { + private final Session session; + private final long pointer; + private final AtomicBoolean consumed = new AtomicBoolean(false); + + private Partition(Session session, long pointer) { + Preconditions.checkArgument(pointer != 0, "invalid partition pointer"); + this.session = session; + this.pointer = pointer; + AtomicBoolean consumedRef = this.consumed; + VortexCleaner.register(this, () -> { + if (!consumedRef.get()) { + NativePartition.free(pointer); + } + }); + } + + static Partition fromPointer(Session session, long pointer) { + return new Partition(session, pointer); + } + + /** Estimated row count of the partition. Empty when unknown. */ + public OptionalLong rowCount() { + Preconditions.checkState(!consumed.get(), "partition already consumed"); + long[] out = new long[2]; + NativePartition.rowCount(pointer, out); + if (out[1] == 0) { + return OptionalLong.empty(); + } + return OptionalLong.of(out[0]); + } + + /** + * Consume the partition and return an {@link ArrowReader} that yields record batches. The caller must close the + * reader when finished; doing so releases the native partition resources as well. + */ + public ArrowReader scanArrow(BufferAllocator allocator) { + if (!consumed.compareAndSet(false, true)) { + throw new IllegalStateException("partition already consumed"); + } + // Native side unconditionally takes ownership of the partition, regardless of + // whether the call subsequently throws, so it is correct to flip `consumed` + // before invoking JNI. The cleaner then always skips free for this handle. + ArrowArrayStream stream = ArrowArrayStream.allocateNew(allocator); + try { + NativePartition.scanArrow(session.nativePointer(), pointer, stream.memoryAddress()); + } catch (RuntimeException ex) { + stream.close(); + throw ex; + } + return Data.importArrayStream(allocator, stream); + } +} diff --git a/java/vortex-jni/src/main/java/dev/vortex/api/Scan.java b/java/vortex-jni/src/main/java/dev/vortex/api/Scan.java new file mode 100644 index 00000000000..0e1cf94349a --- /dev/null +++ b/java/vortex-jni/src/main/java/dev/vortex/api/Scan.java @@ -0,0 +1,70 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +package dev.vortex.api; + +import com.google.common.base.Preconditions; +import dev.vortex.VortexCleaner; +import dev.vortex.jni.NativeScan; +import java.util.Iterator; +import java.util.NoSuchElementException; +import org.apache.arrow.c.ArrowSchema; +import org.apache.arrow.c.Data; +import org.apache.arrow.memory.BufferAllocator; +import org.apache.arrow.vector.types.pojo.Schema; + +/** + * A lazy handle to a set of {@link Partition partitions}. + * + *

Once a scan has produced its last partition it is effectively exhausted; native resources are released + * automatically via {@link VortexCleaner} when the scan becomes unreachable. + */ +public final class Scan implements Iterator { + private final Session session; + private final long pointer; + + private long nextPartitionPointer; + private boolean primed; + + private Scan(Session session, long pointer) { + Preconditions.checkArgument(pointer != 0, "invalid scan pointer"); + this.session = session; + this.pointer = pointer; + VortexCleaner.register(this, () -> NativeScan.free(pointer)); + } + + static Scan fromPointer(Session session, long pointer) { + return new Scan(session, pointer); + } + + /** + * Arrow schema produced by this scan. Must be called before the first call to {@link #hasNext()}/{@link #next()}. + */ + public Schema arrowSchema(BufferAllocator allocator) { + try (ArrowSchema schema = ArrowSchema.allocateNew(allocator)) { + NativeScan.arrowSchema(pointer, schema.memoryAddress()); + return Data.importSchema(allocator, schema, null); + } + } + + @Override + public boolean hasNext() { + if (primed) { + return nextPartitionPointer != 0; + } + nextPartitionPointer = NativeScan.nextPartition(pointer); + primed = true; + return nextPartitionPointer != 0; + } + + @Override + public Partition next() { + if (!hasNext()) { + throw new NoSuchElementException(); + } + long ptr = nextPartitionPointer; + nextPartitionPointer = 0; + primed = false; + return Partition.fromPointer(session, ptr); + } +} diff --git a/java/vortex-jni/src/main/java/dev/vortex/api/ScanOptions.java b/java/vortex-jni/src/main/java/dev/vortex/api/ScanOptions.java index 68a6e7f9c9a..2431f9e5722 100644 --- a/java/vortex-jni/src/main/java/dev/vortex/api/ScanOptions.java +++ b/java/vortex-jni/src/main/java/dev/vortex/api/ScanOptions.java @@ -3,52 +3,76 @@ package dev.vortex.api; -import java.util.List; import java.util.Optional; +import java.util.OptionalLong; import org.immutables.value.Value; /** - * Create a new set of options for configuring the scan. + * Scan configuration passed to {@link DataSource#scan(ScanOptions)}. + * + *

All fields are optional. A call to {@link #of()} returns a default that reads every row and column. */ @Value.Immutable public interface ScanOptions { - /** - * Columns to project out. - */ - List columns(); - /** - * Optional pruning expression that is pushed down to the scan. - */ - Optional predicate(); + /** Projection expression. If empty, all columns are returned. */ + Optional projection(); - /** - * Optional start (inclusive) and end (exclusive) row indices to select a range of rows - * in the scan. - */ - Optional rowRange(); + /** Filter expression applied before returning rows. */ + Optional filter(); - /** - * Optional row indices to select specific rows. - * These must be sorted in ascending order. - */ - Optional rowIndices(); + /** Inclusive start of the row range to read. */ + OptionalLong rowRangeBegin(); + + /** Exclusive end of the row range to read. */ + OptionalLong rowRangeEnd(); /** - * Creates a new ScanOptions instance with default values. - * - * @return a ScanOptions instance with empty columns list, no predicate, no row range, and no row indices + * Sorted ascending row indices that should be included in (or excluded from) the scan, depending on + * {@link #selectionMode()}. */ + Optional selectionIndices(); + + /** Meaning of {@link #selectionIndices()}. */ + @Value.Default + default SelectionMode selectionMode() { + return SelectionMode.INCLUDE_ALL; + } + + /** Maximum row count to return. Absent means "no limit". */ + OptionalLong limit(); + + /** If {@code true}, the scan preserves the original row order across partitions. */ + @Value.Default + default boolean ordered() { + return false; + } + static ScanOptions of() { return ImmutableScanOptions.builder().build(); } - /** - * Creates a new builder for constructing ScanOptions instances. - * - * @return a new builder instance that can be used to configure and build ScanOptions - */ static ImmutableScanOptions.Builder builder() { return ImmutableScanOptions.builder(); } + + /** How to interpret {@link #selectionIndices()}. */ + enum SelectionMode { + /** Ignore {@link #selectionIndices()}. */ + INCLUDE_ALL((byte) 0), + /** Return only rows at the indices. */ + INCLUDE((byte) 1), + /** Return rows except those at the indices. */ + EXCLUDE((byte) 2); + + private final byte code; + + SelectionMode(byte code) { + this.code = code; + } + + public byte code() { + return code; + } + } } diff --git a/java/vortex-jni/src/main/java/dev/vortex/api/Session.java b/java/vortex-jni/src/main/java/dev/vortex/api/Session.java new file mode 100644 index 00000000000..843e93516b2 --- /dev/null +++ b/java/vortex-jni/src/main/java/dev/vortex/api/Session.java @@ -0,0 +1,37 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +package dev.vortex.api; + +import com.google.common.base.Preconditions; +import dev.vortex.VortexCleaner; +import dev.vortex.jni.NativeSession; + +/** + * Handle to a native Vortex session. The session owns a current-thread async runtime and is the entry point for opening + * {@link DataSource data sources} and {@link VortexWriter writers}. + * + *

Sessions are safe to share across threads within a process, though concrete operations (scans, writes) remain + * single-threaded on the session's runtime thread. + * + *

Native resources are released automatically when the session becomes unreachable, via {@link VortexCleaner}. + */ +public final class Session { + private final long pointer; + + private Session(long pointer) { + Preconditions.checkArgument(pointer != 0, "invalid session pointer"); + this.pointer = pointer; + VortexCleaner.register(this, () -> NativeSession.free(pointer)); + } + + /** Create a new session. */ + public static Session create() { + return new Session(NativeSession.newSession()); + } + + /** Internal: returns the native pointer. Do not free directly. */ + public long nativePointer() { + return pointer; + } +} diff --git a/java/vortex-jni/src/main/java/dev/vortex/api/VortexWriter.java b/java/vortex-jni/src/main/java/dev/vortex/api/VortexWriter.java index 9bc5dfd0cf3..2f452526348 100644 --- a/java/vortex-jni/src/main/java/dev/vortex/api/VortexWriter.java +++ b/java/vortex-jni/src/main/java/dev/vortex/api/VortexWriter.java @@ -3,65 +3,91 @@ package dev.vortex.api; -import dev.vortex.jni.JNIDType; -import dev.vortex.jni.JNIWriter; -import dev.vortex.jni.NativeWriterMethods; +import com.google.common.base.Preconditions; +import dev.vortex.VortexCleaner; +import dev.vortex.jni.NativeWriter; import java.io.IOException; import java.util.Map; +import java.util.Objects; +import java.util.concurrent.atomic.AtomicBoolean; +import org.apache.arrow.c.ArrowSchema; +import org.apache.arrow.c.Data; +import org.apache.arrow.memory.BufferAllocator; +import org.apache.arrow.vector.types.pojo.Schema; /** - * Writer for creating Vortex files from Arrow data. - *

- * This class provides methods to write Arrow VectorSchemaRoot batches - * to Vortex format files. + * Writer for Vortex files. + * + *

Batches are accepted via the Arrow C Data Interface: callers export an Arrow record batch to an {@code ArrowArray} + * / {@code ArrowSchema} pair and pass the addresses to {@link #writeBatch(long, long)}. The writer accepts up to four + * in-flight batches on the session's runtime thread before back-pressuring the caller. + * + *

Call {@link #close()} to flush remaining batches and finalize the file. If the writer becomes unreachable without + * an explicit {@code close()}, {@link VortexCleaner} will flush and release native resources as a backstop — but + * callers should always finalize explicitly so that I/O errors surface through the normal call path. */ -public interface VortexWriter extends AutoCloseable { +public final class VortexWriter implements AutoCloseable { + private final long pointer; + private final AtomicBoolean closed = new AtomicBoolean(false); - /** - * Creates a new VortexWriter for writing to the specified file path. - * - * @param uri The URI for where the file is opened - * @param dtype The Vortex DType for data that gets written - * @param options additional writer options - * @return a new VortexWriter instance - * @throws IOException if the writer cannot be created - */ - static VortexWriter create(String uri, DType dtype, Map options) throws IOException { - long ptr = NativeWriterMethods.create(uri, ((JNIDType) dtype).getPointer(), options); - if (ptr <= 0) { - throw new IOException("Failed to create Vortex writer for: " + uri + " (got ptr=" + ptr + ")"); - } - return new JNIWriter(ptr); + private VortexWriter(long pointer) { + Preconditions.checkArgument(pointer != 0, "invalid writer pointer"); + this.pointer = pointer; + AtomicBoolean closedRef = this.closed; + VortexCleaner.register(this, () -> { + if (closedRef.compareAndSet(false, true)) { + NativeWriter.close(pointer); + } + }); } /** - * Writes a batch of Arrow data to the Vortex file. - * - * @param arrowData the Arrow data in IPC format as byte array - * @throws IOException if writing fails + * Create a writer that streams records into the file at {@code uri}. The Arrow schema describes the exact layout of + * every batch written. */ - void writeBatch(byte[] arrowData) throws IOException; + public static VortexWriter create( + Session session, String uri, Schema arrowSchema, Map options, BufferAllocator allocator) + throws IOException { + Objects.requireNonNull(session, "session"); + Objects.requireNonNull(uri, "uri"); + Objects.requireNonNull(arrowSchema, "arrowSchema"); + Objects.requireNonNull(allocator, "allocator"); + ArrowSchema ffi = ArrowSchema.allocateNew(allocator); + try { + Data.exportSchema(allocator, arrowSchema, null, ffi); + long ptr = NativeWriter.create(session.nativePointer(), uri, ffi.memoryAddress(), options); + if (ptr <= 0) { + throw new IOException("failed to create writer for uri " + uri + " (ptr=" + ptr + ")"); + } + return new VortexWriter(ptr); + } finally { + ffi.close(); + } + } - /** - * Writes a batch of Arrow data directly from Arrow C Data Interface pointers. - *

- * This avoids the IPC serialization overhead by accepting raw memory addresses - * of ArrowArray and ArrowSchema structs. - * - * @param arrowArrayAddr memory address of the ArrowArray struct - * @param arrowSchemaAddr memory address of the ArrowSchema struct - * @throws IOException if writing fails - */ - void writeBatchFfi(long arrowArrayAddr, long arrowSchemaAddr) throws IOException; + /** Write a batch directly from Arrow C Data Interface addresses. */ + public void writeBatch(long arrowArrayAddr, long arrowSchemaAddr) throws IOException { + Preconditions.checkState(!closed.get(), "writer already closed"); + final boolean ok; + try { + ok = NativeWriter.writeBatch(pointer, arrowArrayAddr, arrowSchemaAddr); + } catch (RuntimeException e) { + throw new IOException("failed to write batch", e); + } + if (!ok) { + throw new IOException("failed to write batch"); + } + } - /** - * Closes the writer and finalizes the Vortex file. - *

- * This method must be called to ensure the file is properly written - * with all necessary metadata and footers. - * - * @throws IOException if closing fails - */ + /** Flush any pending batches and finalize the file. Idempotent. */ @Override - void close() throws IOException; + public void close() throws IOException { + if (closed.compareAndSet(false, true)) { + try { + NativeWriter.close(pointer); + } catch (RuntimeException e) { + throw new IOException("failed to close writer", e); + } + } + } } diff --git a/java/vortex-jni/src/main/java/dev/vortex/api/expressions/Binary.java b/java/vortex-jni/src/main/java/dev/vortex/api/expressions/Binary.java deleted file mode 100644 index 54e3b09c7df..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/api/expressions/Binary.java +++ /dev/null @@ -1,333 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.api.expressions; - -import com.google.protobuf.InvalidProtocolBufferException; -import dev.vortex.api.Expression; -import dev.vortex.proto.ExprProtos; -import java.util.List; -import java.util.Objects; -import java.util.Optional; -import java.util.stream.Stream; - -/** - * Represents a binary expression that operates on two child expressions using a binary operator. - * Binary expressions support comparison operations (equality, inequality, relational comparisons) - * and boolean algebra operations (AND, OR). - * - *

This class is immutable and implements the {@link Expression} interface, making it suitable - * for use in expression trees and query processing pipelines.

- * - *

Example usage:

- *
- * // Create equality comparison: left == right
- * Binary equalExpr = Binary.eq(leftExpr, rightExpr);
- *
- * // Create logical AND: expr1 && expr2 && expr3
- * Binary andExpr = Binary.and(expr1, expr2, expr3);
- *
- * // Create greater than comparison: left > right
- * Binary gtExpr = Binary.gt(leftExpr, rightExpr);
- * 
- */ -public final class Binary implements Expression { - private final BinaryOp operator; - private final Expression left; - private final Expression right; - - private Binary(BinaryOp operator, Expression left, Expression right) { - this.operator = operator; - this.left = left; - this.right = right; - } - - /** - * Parses a Binary expression from protobuf metadata and child expressions. - * - * @param metadata the serialized protobuf metadata containing the binary operator - * @param children the list of child expressions, must contain exactly 2 expressions - * @return a new Binary expression instance - * @throws RuntimeException if children size is not 2 or if metadata parsing fails - */ - public static Binary parse(byte[] metadata, List children) { - if (children.size() != 2) { - throw new IllegalArgumentException( - "Binary expression must have exactly two children, found: " + children.size()); - } - try { - ExprProtos.BinaryOpts opts = ExprProtos.BinaryOpts.parseFrom(metadata); - BinaryOp operator = BinaryOp.fromProto(opts.getOp()); - return new Binary(operator, children.get(0), children.get(1)); - } catch (InvalidProtocolBufferException e) { - throw new IllegalArgumentException("Failed to parse Binary metadata", e); - } - } - - /** - * Creates a new Binary expression with the specified operator and operands. - * - * @param operator the binary operator to apply - * @param left the left operand expression - * @param right the right operand expression - * @return a new Binary expression instance - */ - public static Binary of(BinaryOp operator, Expression left, Expression right) { - return new Binary(operator, left, right); - } - - /** - * Creates a logical AND expression combining multiple expressions. - * If only one expression is provided, it returns an AND with a literal true. - * Multiple expressions are right-associated: {@code first && (rest[0] && rest[1] && ...)}. - * - * @param first the first expression (left operand) - * @param rest additional expressions to AND together (right operands) - * @return a new Binary expression representing the logical AND operation - */ - public static Binary and(Expression first, Expression... rest) { - Expression rhs = Stream.of(rest).reduce(Binary::and).orElse(Literal.bool(true)); - return new Binary(BinaryOp.AND, first, rhs); - } - - /** - * Creates a logical OR expression combining multiple expressions. - * If only one expression is provided, it returns an OR with a literal false. - * Multiple expressions are right-associated: {@code first || (rest[0] || rest[1] || ...)}. - * - * @param first the first expression (left operand) - * @param rest additional expressions to OR together (right operands) - * @return a new Binary expression representing the logical OR operation - */ - public static Binary or(Expression first, Expression... rest) { - Expression rhs = Stream.of(rest).reduce(Binary::or).orElse(Literal.bool(false)); - return new Binary(BinaryOp.OR, first, rhs); - } - - /** - * Creates an equality comparison expression (==). - * - * @param left the left operand expression - * @param right the right operand expression - * @return a new Binary expression representing left == right - */ - public static Binary eq(Expression left, Expression right) { - return new Binary(BinaryOp.EQ, left, right); - } - - /** - * Creates an inequality comparison expression (!=). - * - * @param left the left operand expression - * @param right the right operand expression - * @return a new Binary expression representing left != right - */ - public static Binary notEq(Expression left, Expression right) { - return new Binary(BinaryOp.NOT_EQ, left, right); - } - - /** - * Creates a greater-than comparison expression (>). - * - * @param left the left operand expression - * @param right the right operand expression - * @return a new Binary expression representing left > right - */ - public static Binary gt(Expression left, Expression right) { - return new Binary(BinaryOp.GT, left, right); - } - - /** - * Creates a greater-than-or-equal comparison expression (>=). - * - * @param left the left operand expression - * @param right the right operand expression - * @return a new Binary expression representing left >= right - */ - public static Binary gtEq(Expression left, Expression right) { - return new Binary(BinaryOp.GT_EQ, left, right); - } - - /** - * Creates a less-than comparison expression (<). - * - * @param left the left operand expression - * @param right the right operand expression - * @return a new Binary expression representing left < right - */ - public static Binary lt(Expression left, Expression right) { - return new Binary(BinaryOp.LT, left, right); - } - - /** - * Creates a less-than-or-equal comparison expression (<=). - * - * @param left the left operand expression - * @param right the right operand expression - * @return a new Binary expression representing left <= right - */ - public static Binary ltEq(Expression left, Expression right) { - return new Binary(BinaryOp.LT_EQ, left, right); - } - - @Override - public String id() { - return "vortex.binary"; - } - - @Override - public List children() { - return List.of(left, right); - } - - @Override - public Optional metadata() { - return Optional.of(ExprProtos.BinaryOpts.newBuilder() - .setOp(operator.toProto()) - .build() - .toByteArray()); - } - - @Override - public String toString() { - return "(" + left + " " + operator + " " + right + ")"; - } - - @Override - public boolean equals(Object o) { - if (o == null || getClass() != o.getClass()) return false; - Binary binary = (Binary) o; - return operator == binary.operator && Objects.equals(left, binary.left) && Objects.equals(right, binary.right); - } - - @Override - public int hashCode() { - return Objects.hash(operator, left, right); - } - - @Override - public T accept(Visitor visitor) { - return visitor.visitBinary(this); - } - - /** - * Returns the binary operator used in this expression. - * - * @return the binary operator - */ - public BinaryOp getOperator() { - return operator; - } - - /** - * Returns the left operand expression. - * - * @return the left operand expression - */ - public Expression getLeft() { - return left; - } - - /** - * Returns the right operand expression. - * - * @return the right operand expression - */ - public Expression getRight() { - return right; - } - - /** - * Enumeration of binary operators supported by binary expressions. - * Includes comparison operators and boolean algebra operators. - */ - public enum BinaryOp { - /** Equality comparison operator (==) */ - EQ, - /** Inequality comparison operator (!=) */ - NOT_EQ, - /** Greater-than comparison operator (>) */ - GT, - /** Greater-than-or-equal comparison operator (>=) */ - GT_EQ, - /** Less-than comparison operator (<) */ - LT, - /** Less-than-or-equal comparison operator (<=) */ - LT_EQ, - /** Logical AND operator (&&) */ - AND, - /** Logical OR operator (||) */ - OR, - ; - - @Override - public String toString() { - switch (this) { - case EQ: - return "=="; - case NOT_EQ: - return "!="; - case GT: - return ">"; - case GT_EQ: - return ">="; - case LT: - return "<"; - case LT_EQ: - return "<="; - case AND: - return "&&"; - case OR: - return "||"; - default: - throw new IllegalStateException("Unknown Operator: " + this); - } - } - - static BinaryOp fromProto(ExprProtos.BinaryOpts.BinaryOp proto) { - switch (proto) { - case Eq: - return EQ; - case NotEq: - return NOT_EQ; - case Gt: - return GT; - case Gte: - return GT_EQ; - case Lt: - return LT; - case Lte: - return LT_EQ; - case And: - return AND; - case Or: - return OR; - default: - throw new IllegalArgumentException("Unsupported binary operator proto: " + proto); - } - } - - ExprProtos.BinaryOpts.BinaryOp toProto() { - switch (this) { - case EQ: - return ExprProtos.BinaryOpts.BinaryOp.Eq; - case NOT_EQ: - return ExprProtos.BinaryOpts.BinaryOp.NotEq; - case GT: - return ExprProtos.BinaryOpts.BinaryOp.Gt; - case GT_EQ: - return ExprProtos.BinaryOpts.BinaryOp.Gte; - case LT: - return ExprProtos.BinaryOpts.BinaryOp.Lt; - case LT_EQ: - return ExprProtos.BinaryOpts.BinaryOp.Lte; - case AND: - return ExprProtos.BinaryOpts.BinaryOp.And; - case OR: - return ExprProtos.BinaryOpts.BinaryOp.Or; - default: - throw new IllegalArgumentException("Unsupported binary operator: " + this); - } - } - } -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/api/expressions/GetItem.java b/java/vortex-jni/src/main/java/dev/vortex/api/expressions/GetItem.java deleted file mode 100644 index b150b78a7e2..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/api/expressions/GetItem.java +++ /dev/null @@ -1,111 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.api.expressions; - -import com.google.protobuf.InvalidProtocolBufferException; -import dev.vortex.api.Expression; -import dev.vortex.proto.ExprProtos; -import java.util.List; -import java.util.Objects; -import java.util.Optional; - -/** - * Represents a "get item" expression that extracts a field or property from a parent expression. - * This expression is used to access nested fields in complex data structures such as structs, - * lists, or other composite types by specifying a path to the desired field. - */ -public final class GetItem implements Expression { - private final String path; - private final Expression child; - - private GetItem(Expression child, String path) { - this.child = child; - this.path = path; - } - - /** - * Creates a new GetItem expression that extracts the specified field from the given child expression. - * - * @param child the parent expression from which to extract the field - * @param path the path or name of the field to extract - * @return a new GetItem expression - */ - public static GetItem of(Expression child, String path) { - return new GetItem(child, path); - } - - /** - * Parses a GetItem expression from serialized metadata and child expressions. - * This method is used during deserialization of Vortex expressions. - * - * @param metadata the serialized metadata containing the field path information - * @param children the child expressions, must contain exactly one element - * @return a new GetItem expression parsed from the provided data - * @throws RuntimeException if the number of children is not exactly one, - * or if the metadata cannot be parsed - */ - public static GetItem parse(byte[] metadata, List children) { - if (children.size() != 1) { - throw new IllegalArgumentException( - "GetItem expression must have exactly one child, found: " + children.size()); - } - try { - ExprProtos.GetItemOpts opts = ExprProtos.GetItemOpts.parseFrom(metadata); - return new GetItem(children.get(0), opts.getPath()); - } catch (InvalidProtocolBufferException e) { - throw new IllegalArgumentException("Failed to parse GetItem metadata", e); - } - } - - /** - * Returns the child expression from which the field is being extracted. - * - * @return the child expression - */ - public Expression getChild() { - return child; - } - - /** - * Returns the path or name of the field being extracted from the child expression. - * - * @return the field path - */ - public String getPath() { - return path; - } - - @Override - public String id() { - return "vortex.get_item"; - } - - @Override - public List children() { - return List.of(child); - } - - @Override - public Optional metadata() { - return Optional.of( - ExprProtos.GetItemOpts.newBuilder().setPath(path).build().toByteArray()); - } - - @Override - public T accept(Visitor visitor) { - return visitor.visitGetItem(this); - } - - @Override - public boolean equals(Object o) { - if (!(o instanceof GetItem)) return false; - GetItem getItem = (GetItem) o; - return Objects.equals(path, getItem.path); - } - - @Override - public int hashCode() { - return Objects.hashCode(path); - } -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/api/expressions/IsNull.java b/java/vortex-jni/src/main/java/dev/vortex/api/expressions/IsNull.java deleted file mode 100644 index 834cc6d5494..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/api/expressions/IsNull.java +++ /dev/null @@ -1,99 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.api.expressions; - -import dev.vortex.api.Expression; -import java.util.List; -import java.util.Objects; -import java.util.Optional; - -/** - * Represents an IS NULL expression that checks whether values are null. - * This expression returns true for null values and false for non-null values. - */ -public final class IsNull implements Expression { - private final Expression child; - - private IsNull(Expression child) { - this.child = child; - } - - /** - * Parses an IsNull expression from serialized metadata and child expressions. - * This method is used during deserialization of Vortex expressions. - * - * @param metadata the serialized metadata, must be empty for IsNull expressions - * @param children the child expressions, must contain exactly one element - * @return a new IsNull expression parsed from the provided data - * @throws IllegalArgumentException if the number of children is not exactly one, - * or if metadata is not empty - */ - public static IsNull parse(byte[] metadata, List children) { - if (children.size() != 1) { - throw new IllegalArgumentException( - "IsNull expression must have exactly one child, found: " + children.size()); - } - if (metadata.length > 0) { - throw new IllegalArgumentException( - "IsNull expression must not have metadata, found: " + metadata.length); - } - return new IsNull(children.get(0)); - } - - /** - * Creates a new IsNull expression that checks nullity of the given child expression. - * - * @param child the expression to check for null values - * @return a new IsNull expression - */ - public static IsNull of(Expression child) { - return new IsNull(child); - } - - @Override - public boolean equals(Object o) { - if (o == null || getClass() != o.getClass()) return false; - IsNull other = (IsNull) o; - return Objects.equals(child, other.child); - } - - @Override - public int hashCode() { - return Objects.hash(child); - } - - @Override - public String id() { - return "vortex.is_null"; - } - - @Override - public List children() { - return List.of(child); - } - - @Override - public Optional metadata() { - return Optional.of(new byte[] {}); - } - - @Override - public String toString() { - return "vortex.is_null(" + child + ")"; - } - - /** - * Returns the child expression that will be checked for null values. - * - * @return the child expression - */ - public Expression getChild() { - return child; - } - - @Override - public T accept(Visitor visitor) { - return visitor.visitIsNull(this); - } -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/api/expressions/Literal.java b/java/vortex-jni/src/main/java/dev/vortex/api/expressions/Literal.java deleted file mode 100644 index dac9476b5a5..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/api/expressions/Literal.java +++ /dev/null @@ -1,1061 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.api.expressions; - -import com.google.common.base.Preconditions; -import com.google.protobuf.ByteString; -import com.google.protobuf.InvalidProtocolBufferException; -import dev.vortex.api.Expression; -import dev.vortex.api.proto.EndianUtils; -import dev.vortex.api.proto.Scalars; -import dev.vortex.api.proto.TemporalMetadatas; -import dev.vortex.proto.DTypeProtos; -import dev.vortex.proto.ExprProtos; -import dev.vortex.proto.ScalarProtos; -import java.math.BigDecimal; -import java.math.BigInteger; -import java.util.List; -import java.util.Objects; -import java.util.Optional; - -/** - * Represents a literal value expression in the Vortex query system. - *

- * A literal is a constant value of a specific type that can be used in expressions. - * This class provides factory methods for creating literals of various types including - * primitives (boolean, integers, floats), temporal types (dates, times, timestamps), - * and complex types (strings, bytes, decimals). - *

- * Literals are immutable and implement the visitor pattern for type-safe processing. - * - * @param the Java type of the literal value - */ -public abstract class Literal implements Expression { - private final T value; - - private Literal(T value) { - this.value = value; - } - - /** - * Parses a literal from serialized metadata and child expressions. - *

- * This method deserializes a literal expression from its protocol buffer - * representation. Literal expressions must have no children. - * - * @param metadata the serialized literal metadata as bytes - * @param children the list of child expressions (must be empty for literals) - * @return the parsed literal expression - * @throws RuntimeException if children is not empty or if metadata cannot be parsed - */ - public static Literal parse(byte[] metadata, List children) { - if (!children.isEmpty()) { - throw new IllegalArgumentException("Literal expression must have no children, found: " + children.size()); - } - try { - ExprProtos.LiteralOpts opts = ExprProtos.LiteralOpts.parseFrom(metadata); - return deserializeLiteral(opts, children); - } catch (InvalidProtocolBufferException e) { - throw new IllegalArgumentException("Failed to parse literal metadata", e); - } - } - - /** - * Returns the value of this literal. - * - * @return the literal value, may be null for null literals - */ - public T getValue() { - return this.value; - } - - @Override - public String id() { - return "vortex.literal"; - } - - @Override - public List children() { - return List.of(); - } - - @Override - public Optional metadata() { - return Optional.of(ExprProtos.LiteralOpts.newBuilder() - .setValue(this.acceptLiteralVisitor(LiteralToScalar.INSTANCE)) - .build() - .toByteArray()); - } - - @Override - public int hashCode() { - return Objects.hashCode(getValue()); - } - - @Override - public boolean equals(Object o) { - if (!(o instanceof Literal)) return false; - Literal literal = (Literal) o; - return Objects.equals(value, literal.value); - } - - /** - * Creates a null literal. - * - * @return a literal representing null - */ - public static Literal nullLit() { - return NullLiteral.INSTANCE; - } - - /** - * Creates a boolean literal. - * - * @param value the boolean value, may be null - * @return a boolean literal - */ - public static Literal bool(Boolean value) { - return new BooleanLiteral(value); - } - - /** - * Creates an 8-bit signed integer literal. - * - * @param value the byte value, may be null - * @return an int8 literal - */ - public static Literal int8(Byte value) { - return new Int8Literal(value); - } - - /** - * Creates a 16-bit signed integer literal. - * - * @param value the short value, may be null - * @return an int16 literal - */ - public static Literal int16(Short value) { - return new Int16Literal(value); - } - - /** - * Creates a 32-bit signed integer literal. - * - * @param value the integer value, may be null - * @return an int32 literal - */ - public static Literal int32(Integer value) { - return new Int32Literal(value); - } - - /** - * Creates a 64-bit signed integer literal. - * - * @param value the long value, may be null - * @return an int64 literal - */ - public static Literal int64(Long value) { - return new Int64Literal(value); - } - - /** - * Creates a 32-bit floating-point literal. - * - * @param value the float value, may be null - * @return a float32 literal - */ - public static Literal float32(Float value) { - return new Float32Literal(value); - } - - /** - * Creates a 64-bit floating-point literal. - * - * @param value the double value, may be null - * @return a float64 literal - */ - public static Literal float64(Double value) { - return new Float64Literal(value); - } - - /** - * Creates a decimal literal with specified precision and scale. - * - * @param value the decimal value, may be null - * @param precision the total number of digits - * @param scale the number of digits after the decimal point - * @return a decimal literal - * @throws RuntimeException if the value's scale doesn't match the specified scale - */ - public static Literal decimal(BigDecimal value, int precision, int scale) { - return new DecimalLiteral(value, precision, scale); - } - - /** - * Creates a string literal. - * - * @param value the string value, may be null - * @return a string literal - */ - public static Literal string(String value) { - return new StringLiteral(value); - } - - /** - * Creates a byte array literal. - * - * @param value the byte array value, may be null - * @return a bytes literal - */ - public static Literal bytes(byte[] value) { - return new BytesLiteral(value); - } - - /** - * Creates a time literal representing time of day in seconds. - * - * @param value the time value in seconds since midnight, may be null - * @return a time literal with second precision - */ - public static Literal timeSeconds(Integer value) { - return new TimeSeconds(value); - } - - /** - * Creates a time literal representing time of day in milliseconds. - * - * @param value the time value in milliseconds since midnight, may be null - * @return a time literal with millisecond precision - */ - public static Literal timeMillis(Integer value) { - return new TimeMillis(value); - } - - /** - * Creates a time literal representing time of day in microseconds. - * - * @param value the time value in microseconds since midnight, may be null - * @return a time literal with microsecond precision - */ - public static Literal timeMicros(Long value) { - return new TimeMicros(value); - } - - /** - * Creates a time literal representing time of day in nanoseconds. - * - * @param value the time value in nanoseconds since midnight, may be null - * @return a time literal with nanosecond precision - */ - public static Literal timeNanos(Long value) { - return new TimeNanos(value); - } - - /** - * Creates a date literal representing date as days since epoch. - * - * @param value the number of days since Unix epoch (1970-01-01), may be null - * @return a date literal with day precision - */ - public static Literal dateDays(Integer value) { - return new DateDays(value); - } - - /** - * Creates a date literal representing date as milliseconds since epoch. - * - * @param value the number of milliseconds since Unix epoch, may be null - * @return a date literal with millisecond precision - */ - public static Literal dateMillis(Long value) { - return new DateMillis(value); - } - - /** - * Creates a timestamp literal with millisecond precision. - * - * @param value the timestamp value in milliseconds since Unix epoch, may be null - * @param timeZone the time zone identifier (e.g., "UTC", "America/New_York"), optional - * @return a timestamp literal with millisecond precision - */ - public static Literal timestampMillis(Long value, Optional timeZone) { - return new TimestampMillis(value, timeZone); - } - - /** - * Creates a timestamp literal with microsecond precision. - * - * @param value the timestamp value in microseconds since Unix epoch, may be null - * @param timeZone the time zone identifier (e.g., "UTC", "America/New_York"), optional - * @return a timestamp literal with microsecond precision - */ - public static Literal timestampMicros(Long value, Optional timeZone) { - return new TimestampMicros(value, timeZone); - } - - /** - * Creates a timestamp literal with nanosecond precision. - * - * @param value the timestamp value in nanoseconds since Unix epoch, may be null - * @param timeZone the time zone identifier (e.g., "UTC", "America/New_York"), optional - * @return a timestamp literal with nanosecond precision - */ - public static Literal timestampNanos(Long value, Optional timeZone) { - return new TimestampNanos(value, timeZone); - } - - @Override - public R accept(Expression.Visitor visitor) { - return visitor.visitLiteral(this); - } - - /** - * Accepts a literal visitor to process this literal in a type-safe manner. - *

- * This method implements the visitor pattern, allowing different operations - * to be performed on literals based on their specific type. - * - * @param the return type of the visitor operation - * @param visitor the visitor to accept - * @return the result of the visitor operation - */ - public abstract U acceptLiteralVisitor(LiteralVisitor visitor); - - /** - * Visitor interface for processing literals in a type-safe manner. - *

- * This interface defines methods for visiting each type of literal that can - * be represented in the Vortex system. Implementations can provide type-specific - * processing logic while maintaining type safety. - * - * @param the return type of visitor methods - */ - public interface LiteralVisitor { - /** - * Visits a null literal. - * - * @return the result of processing the null literal - */ - U visitNull(); - - /** - * Visits a boolean literal. - * - * @param literal the boolean value, may be null - * @return the result of processing the boolean literal - */ - U visitBoolean(Boolean literal); - - /** - * Visits an 8-bit signed integer literal. - * - * @param literal the byte value, may be null - * @return the result of processing the int8 literal - */ - U visitInt8(Byte literal); - - /** - * Visits a 16-bit signed integer literal. - * - * @param literal the short value, may be null - * @return the result of processing the int16 literal - */ - U visitInt16(Short literal); - - /** - * Visits a 32-bit signed integer literal. - * - * @param literal the integer value, may be null - * @return the result of processing the int32 literal - */ - U visitInt32(Integer literal); - - /** - * Visits a 64-bit signed integer literal. - * - * @param literal the long value, may be null - * @return the result of processing the int64 literal - */ - U visitInt64(Long literal); - - /** - * Visits a date literal with day precision. - * - * @param days the number of days since Unix epoch, may be null - * @return the result of processing the date literal - */ - U visitDateDays(Integer days); - - /** - * Visits a date literal with millisecond precision. - * - * @param millis the number of milliseconds since Unix epoch, may be null - * @return the result of processing the date literal - */ - U visitDateMillis(Long millis); - - /** - * Visits a time literal with second precision. - * - * @param seconds the time in seconds since midnight, may be null - * @return the result of processing the time literal - */ - U visitTimeSeconds(Integer seconds); - - /** - * Visits a time literal with millisecond precision. - * - * @param seconds the time in milliseconds since midnight, may be null - * @return the result of processing the time literal - */ - U visitTimeMillis(Integer seconds); - - /** - * Visits a time literal with microsecond precision. - * - * @param seconds the time in microseconds since midnight, may be null - * @return the result of processing the time literal - */ - U visitTimeMicros(Long seconds); - - /** - * Visits a time literal with nanosecond precision. - * - * @param seconds the time in nanoseconds since midnight, may be null - * @return the result of processing the time literal - */ - U visitTimeNanos(Long seconds); - - /** - * Visits a timestamp literal with millisecond precision. - * - * @param epochMillis the timestamp in milliseconds since Unix epoch, may be null - * @param timeZone the time zone identifier, optional - * @return the result of processing the timestamp literal - */ - U visitTimestampMillis(Long epochMillis, Optional timeZone); - - /** - * Visits a timestamp literal with microsecond precision. - * - * @param epochMicros the timestamp in microseconds since Unix epoch, may be null - * @param timeZone the time zone identifier, optional - * @return the result of processing the timestamp literal - */ - U visitTimestampMicros(Long epochMicros, Optional timeZone); - - /** - * Visits a timestamp literal with nanosecond precision. - * - * @param epochNanos the timestamp in nanoseconds since Unix epoch, may be null - * @param timeZone the time zone identifier, optional - * @return the result of processing the timestamp literal - */ - U visitTimestampNanos(Long epochNanos, Optional timeZone); - - /** - * Visits a 32-bit floating-point literal. - * - * @param literal the float value, may be null - * @return the result of processing the float32 literal - */ - U visitFloat32(Float literal); - - /** - * Visits a 64-bit floating-point literal. - * - * @param literal the double value, may be null - * @return the result of processing the float64 literal - */ - U visitFloat64(Double literal); - - /** - * Visits a decimal literal with specified precision and scale. - * - * @param decimal the decimal value, may be null - * @param precision the total number of digits - * @param scale the number of digits after the decimal point - * @return the result of processing the decimal literal - */ - U visitDecimal(BigDecimal decimal, int precision, int scale); - - /** - * Visits a string literal. - * - * @param literal the string value, may be null - * @return the result of processing the string literal - */ - U visitString(String literal); - - /** - * Visits a byte array literal. - * - * @param literal the byte array value, may be null - * @return the result of processing the bytes literal - */ - U visitBytes(byte[] literal); - } - - static final class NullLiteral extends Literal { - static final NullLiteral INSTANCE = new NullLiteral(); - - private NullLiteral() { - super(null); - } - - @Override - public U acceptLiteralVisitor(LiteralVisitor visitor) { - return visitor.visitNull(); - } - } - - static final class BooleanLiteral extends Literal { - BooleanLiteral(Boolean value) { - super(value); - } - - @Override - public U acceptLiteralVisitor(LiteralVisitor visitor) { - return visitor.visitBoolean(getValue()); - } - } - - static final class Int8Literal extends Literal { - Int8Literal(Byte value) { - super(value); - } - - @Override - public U acceptLiteralVisitor(LiteralVisitor visitor) { - return visitor.visitInt8(getValue()); - } - } - - static final class Int16Literal extends Literal { - Int16Literal(Short value) { - super(value); - } - - @Override - public U acceptLiteralVisitor(LiteralVisitor visitor) { - return visitor.visitInt16(getValue()); - } - } - - static final class Int32Literal extends Literal { - Int32Literal(Integer value) { - super(value); - } - - @Override - public U acceptLiteralVisitor(LiteralVisitor visitor) { - return visitor.visitInt32(getValue()); - } - } - - static final class Int64Literal extends Literal { - Int64Literal(Long value) { - super(value); - } - - @Override - public U acceptLiteralVisitor(LiteralVisitor visitor) { - return visitor.visitInt64(getValue()); - } - } - - static final class Float32Literal extends Literal { - Float32Literal(Float value) { - super(value); - } - - @Override - public U acceptLiteralVisitor(LiteralVisitor visitor) { - return visitor.visitFloat32(getValue()); - } - } - - static final class Float64Literal extends Literal { - Float64Literal(Double value) { - super(value); - } - - @Override - public U acceptLiteralVisitor(LiteralVisitor visitor) { - return visitor.visitFloat64(getValue()); - } - } - - static final class DecimalLiteral extends Literal { - private final int precision; - private final int scale; - - DecimalLiteral(BigDecimal value, int precision, int scale) { - super(value); - if (!Objects.isNull(value)) { - Preconditions.checkArgument(scale == value.scale(), "scale %s ≠ value scale %s", scale, value.scale()); - } - this.precision = precision; - this.scale = scale; - } - - @Override - public U acceptLiteralVisitor(LiteralVisitor visitor) { - return visitor.visitDecimal(getValue(), precision, scale); - } - } - - static final class StringLiteral extends Literal { - StringLiteral(String value) { - super(value); - } - - @Override - public U acceptLiteralVisitor(LiteralVisitor visitor) { - return visitor.visitString(getValue()); - } - } - - static final class BytesLiteral extends Literal { - BytesLiteral(byte[] value) { - super(value); - } - - @Override - public U acceptLiteralVisitor(LiteralVisitor visitor) { - return visitor.visitBytes(getValue()); - } - } - - static final class TimeSeconds extends Literal { - TimeSeconds(Integer value) { - super(value); - } - - @Override - public U acceptLiteralVisitor(LiteralVisitor visitor) { - return visitor.visitTimeSeconds(getValue()); - } - } - - static final class TimeMillis extends Literal { - TimeMillis(Integer value) { - super(value); - } - - @Override - public U acceptLiteralVisitor(LiteralVisitor visitor) { - return visitor.visitTimeMillis(getValue()); - } - } - - static final class TimeMicros extends Literal { - TimeMicros(Long value) { - super(value); - } - - @Override - public U acceptLiteralVisitor(LiteralVisitor visitor) { - return visitor.visitTimeMicros(getValue()); - } - } - - static final class TimeNanos extends Literal { - TimeNanos(Long value) { - super(value); - } - - @Override - public U acceptLiteralVisitor(LiteralVisitor visitor) { - return visitor.visitTimeNanos(getValue()); - } - } - - static final class DateDays extends Literal { - DateDays(Integer value) { - super(value); - } - - @Override - public U acceptLiteralVisitor(LiteralVisitor visitor) { - return visitor.visitDateDays(getValue()); - } - } - - static final class DateMillis extends Literal { - DateMillis(Long value) { - super(value); - } - - @Override - public U acceptLiteralVisitor(LiteralVisitor visitor) { - return visitor.visitDateMillis(getValue()); - } - } - - static final class TimestampMillis extends Literal { - private final Optional timeZone; - - TimestampMillis(Long value, Optional timeZone) { - super(value); - this.timeZone = timeZone; - } - - @Override - public U acceptLiteralVisitor(LiteralVisitor visitor) { - return visitor.visitTimestampMillis(getValue(), timeZone); - } - } - - static final class TimestampMicros extends Literal { - private final Optional timeZone; - - TimestampMicros(Long value, Optional timeZone) { - super(value); - this.timeZone = timeZone; - } - - @Override - public U acceptLiteralVisitor(LiteralVisitor visitor) { - return visitor.visitTimestampMicros(getValue(), timeZone); - } - } - - static final class TimestampNanos extends Literal { - private final Optional timeZone; - - TimestampNanos(Long value, Optional timeZone) { - super(value); - this.timeZone = timeZone; - } - - @Override - public U acceptLiteralVisitor(LiteralVisitor visitor) { - return visitor.visitTimestampNanos(getValue(), timeZone); - } - } - - static final class LiteralToScalar implements LiteralVisitor { - static final LiteralToScalar INSTANCE = new LiteralToScalar(); - - private LiteralToScalar() {} - - @Override - public ScalarProtos.Scalar visitNull() { - return Scalars.nullNull(); - } - - @Override - public ScalarProtos.Scalar visitBoolean(Boolean literal) { - if (Objects.isNull(literal)) { - return Scalars.nullBool(); - } else { - return Scalars.bool(literal); - } - } - - @Override - public ScalarProtos.Scalar visitInt8(Byte literal) { - if (Objects.isNull(literal)) { - return Scalars.nullInt8(); - } else { - return Scalars.int8(literal); - } - } - - @Override - public ScalarProtos.Scalar visitInt16(Short literal) { - if (Objects.isNull(literal)) { - return Scalars.nullInt16(); - } else { - return Scalars.int16(literal); - } - } - - @Override - public ScalarProtos.Scalar visitInt32(Integer literal) { - if (Objects.isNull(literal)) { - return Scalars.nullInt32(); - } else { - return Scalars.int32(literal); - } - } - - @Override - public ScalarProtos.Scalar visitInt64(Long literal) { - if (Objects.isNull(literal)) { - return Scalars.nullInt64(); - } else { - return Scalars.int64(literal); - } - } - - @Override - public ScalarProtos.Scalar visitDateDays(Integer days) { - if (Objects.isNull(days)) { - return Scalars.nullDateDays(); - } else { - return Scalars.dateDays(days); - } - } - - @Override - public ScalarProtos.Scalar visitDateMillis(Long millis) { - if (Objects.isNull(millis)) { - return Scalars.nullDateMillis(); - } else { - return Scalars.dateMillis(millis); - } - } - - @Override - public ScalarProtos.Scalar visitTimeSeconds(Integer seconds) { - if (Objects.isNull(seconds)) { - return Scalars.nullTimeSeconds(); - } else { - return Scalars.timeSeconds(seconds); - } - } - - @Override - public ScalarProtos.Scalar visitTimeMillis(Integer seconds) { - if (Objects.isNull(seconds)) { - return Scalars.nullTimeMillis(); - } else { - return Scalars.timeMillis(seconds); - } - } - - @Override - public ScalarProtos.Scalar visitTimeMicros(Long seconds) { - if (Objects.isNull(seconds)) { - return Scalars.nullTimeMicros(); - } else { - return Scalars.timeMicros(seconds); - } - } - - @Override - public ScalarProtos.Scalar visitTimeNanos(Long seconds) { - if (Objects.isNull(seconds)) { - return Scalars.nullTimeNanos(); - } else { - return Scalars.timeNanos(seconds); - } - } - - @Override - public ScalarProtos.Scalar visitTimestampMillis(Long epochMillis, Optional timeZone) { - if (Objects.isNull(epochMillis)) { - return Scalars.nullTimestampMillis(timeZone); - } else { - return Scalars.timestampMillis(epochMillis, timeZone); - } - } - - @Override - public ScalarProtos.Scalar visitTimestampMicros(Long epochMicros, Optional timeZone) { - if (Objects.isNull(epochMicros)) { - return Scalars.nullTimestampMicros(timeZone); - } else { - return Scalars.timestampMicros(epochMicros, timeZone); - } - } - - @Override - public ScalarProtos.Scalar visitTimestampNanos(Long epochNanos, Optional timeZone) { - if (Objects.isNull(epochNanos)) { - return Scalars.nullTimestampNanos(timeZone); - } else { - return Scalars.timestampNanos(epochNanos, timeZone); - } - } - - @Override - public ScalarProtos.Scalar visitFloat32(Float literal) { - if (Objects.isNull(literal)) { - return Scalars.nullFloat32(); - } else { - return Scalars.float32(literal); - } - } - - @Override - public ScalarProtos.Scalar visitFloat64(Double literal) { - if (Objects.isNull(literal)) { - return Scalars.nullFloat64(); - } else { - return Scalars.float64(literal); - } - } - - @Override - public ScalarProtos.Scalar visitDecimal(BigDecimal decimal, int precision, int scale) { - if (Objects.isNull(decimal)) { - return Scalars.nullDecimal(precision, scale); - } else { - return Scalars.decimal(decimal, precision, scale); - } - } - - @Override - public ScalarProtos.Scalar visitString(String literal) { - if (Objects.isNull(literal)) { - return Scalars.nullString(); - } else { - return Scalars.string(literal); - } - } - - @Override - public ScalarProtos.Scalar visitBytes(byte[] literal) { - if (Objects.isNull(literal)) { - return Scalars.nullBytes(); - } else { - return Scalars.bytes(literal); - } - } - } - - private static Literal deserializeLiteral(ExprProtos.LiteralOpts literal, List children) { - ScalarProtos.Scalar literalScalar = literal.getValue(); - DTypeProtos.DType dtype = literalScalar.getDtype(); - - // Special handling of extension types - if (dtype.hasExtension()) { - return deserializeExtensionLiteral(literal); - } - - ScalarProtos.ScalarValue scalarValue = literalScalar.getValue(); - - switch (scalarValue.getKindCase()) { - case NULL_VALUE: - return nullLiteral(dtype); - case BOOL_VALUE: - return Literal.bool(scalarValue.getBoolValue()); - case INT64_VALUE: - return Literal.int64(scalarValue.getInt64Value()); - case UINT64_VALUE: - return Literal.int64(scalarValue.getUint64Value()); - case F32_VALUE: - return Literal.float32(scalarValue.getF32Value()); - case F64_VALUE: - return Literal.float64(scalarValue.getF64Value()); - case STRING_VALUE: - return Literal.string(scalarValue.getStringValue()); - case BYTES_VALUE: - if (dtype.hasDecimal()) { - ByteString littleEndian = scalarValue.getBytesValue(); - byte[] bigEndian = EndianUtils.reverse(littleEndian); - BigDecimal value = new BigDecimal( - new BigInteger(bigEndian), dtype.getDecimal().getScale()); - return Literal.decimal( - value, - dtype.getDecimal().getPrecision(), - dtype.getDecimal().getScale()); - } else { - return Literal.bytes(scalarValue.getBytesValue().toByteArray()); - } - default: - throw new UnsupportedOperationException("Unsupported ScalarValue type encountered: " + scalarValue); - } - } - - private static Literal deserializeExtensionLiteral(ExprProtos.LiteralOpts literal) { - ScalarProtos.Scalar scalar = literal.getValue(); - DTypeProtos.DType scalarType = scalar.getDtype(); - - Preconditions.checkArgument(scalarType.hasExtension()); - - DTypeProtos.Extension extType = scalarType.getExtension(); - String extId = scalarType.getExtension().getId(); - - switch (extId) { - case "vortex.time": { - byte timeUnit = - TemporalMetadatas.getTimeUnit(extType.getMetadata().toByteArray()); - if (timeUnit == TemporalMetadatas.TIME_UNIT_SECONDS) { - return Literal.timeSeconds(Math.toIntExact(scalar.getValue().getInt64Value())); - } else if (timeUnit == TemporalMetadatas.TIME_UNIT_MILLIS) { - return Literal.timeMillis(Math.toIntExact(scalar.getValue().getInt64Value())); - } else if (timeUnit == TemporalMetadatas.TIME_UNIT_MICROS) { - return Literal.timeMicros(scalar.getValue().getInt64Value()); - } else if (timeUnit == TemporalMetadatas.TIME_UNIT_NANOS) { - return Literal.timeNanos(scalar.getValue().getInt64Value()); - } else { - throw new UnsupportedOperationException("Unsupported TIME time unit: " + timeUnit); - } - } - case "vortex.date": { - byte timeUnit = - TemporalMetadatas.getTimeUnit(extType.getMetadata().toByteArray()); - if (timeUnit == TemporalMetadatas.TIME_UNIT_DAYS) { - return Literal.dateDays(Math.toIntExact(scalar.getValue().getInt64Value())); - } else if (timeUnit == TemporalMetadatas.TIME_UNIT_MILLIS) { - return Literal.dateMillis(scalar.getValue().getInt64Value()); - } else { - throw new UnsupportedOperationException("Unsupported DATE time unit: " + timeUnit); - } - } - case "vortex.timestamp": { - byte timeUnit = - TemporalMetadatas.getTimeUnit(extType.getMetadata().toByteArray()); - Optional timeZone = - TemporalMetadatas.getTimeZone(extType.getMetadata().toByteArray()); - if (timeUnit == TemporalMetadatas.TIME_UNIT_MILLIS) { - return Literal.timestampMillis(scalar.getValue().getInt64Value(), timeZone); - } else if (timeUnit == TemporalMetadatas.TIME_UNIT_MICROS) { - return Literal.timestampMicros(scalar.getValue().getInt64Value(), timeZone); - } else if (timeUnit == TemporalMetadatas.TIME_UNIT_NANOS) { - return Literal.timestampNanos(scalar.getValue().getInt64Value(), timeZone); - } else { - throw new UnsupportedOperationException("Unsupported TIMESTAMP time unit: " + timeUnit); - } - } - default: - throw new UnsupportedOperationException("Unsupported extension type: " + extId); - } - } - - private static Literal nullLiteral(DTypeProtos.DType type) { - switch (type.getDtypeTypeCase()) { - case NULL: - return Literal.nullLit(); - case BOOL: - return Literal.bool(null); - case PRIMITIVE: - switch (type.getPrimitive().getType()) { - case U8: - case I8: - return Literal.int8(null); - case U16: - case I16: - return Literal.int16(null); - case U32: - case I32: - return Literal.int32(null); - case U64: - case I64: - return Literal.int64(null); - case F32: - return Literal.float32(null); - case F64: - return Literal.float64(null); - default: - throw new UnsupportedOperationException("Unsupported ScalarValue type encountered: " + type); - } - case DECIMAL: - return Literal.decimal( - null, - type.getDecimal().getPrecision(), - type.getDecimal().getScale()); - case UTF8: - return Literal.string(null); - case BINARY: - return Literal.bytes(null); - default: - throw new UnsupportedOperationException("Unsupported ScalarValue type encountered: " + type); - } - } -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/api/expressions/Not.java b/java/vortex-jni/src/main/java/dev/vortex/api/expressions/Not.java deleted file mode 100644 index 9d2783ff5af..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/api/expressions/Not.java +++ /dev/null @@ -1,97 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.api.expressions; - -import dev.vortex.api.Expression; -import java.util.List; -import java.util.Objects; -import java.util.Optional; - -/** - * Represents a logical NOT expression that negates the boolean result of its child expression. - * This expression applies the logical NOT operation to the result of evaluating its single child expression. - */ -public final class Not implements Expression { - private final Expression child; - - private Not(Expression child) { - this.child = child; - } - - /** - * Parses a Not expression from serialized metadata and child expressions. - * This method is used during deserialization of Vortex expressions. - * - * @param metadata the serialized metadata, must be empty for Not expressions - * @param children the child expressions, must contain exactly one element - * @return a new Not expression parsed from the provided data - * @throws RuntimeException if the number of children is not exactly one, - * or if metadata is not empty - */ - public static Not parse(byte[] metadata, List children) { - if (children.size() != 1) { - throw new IllegalArgumentException("Not expression must have exactly one child, found: " + children.size()); - } - if (metadata.length > 0) { - throw new IllegalArgumentException("Not expression must not have metadata, found: " + metadata.length); - } - return new Not(children.get(0)); - } - - /** - * Creates a new Not expression that negates the given child expression. - * - * @param child the expression to negate - * @return a new Not expression - */ - public static Not of(Expression child) { - return new Not(child); - } - - @Override - public boolean equals(Object o) { - if (o == null || getClass() != o.getClass()) return false; - Not other = (Not) o; - return Objects.equals(child, other.child); - } - - @Override - public int hashCode() { - return Objects.hash(child); - } - - @Override - public String id() { - return "vortex.not"; - } - - @Override - public List children() { - return List.of(child); - } - - @Override - public Optional metadata() { - return Optional.of(new byte[] {}); - } - - @Override - public String toString() { - return "not(" + child + ")"; - } - - /** - * Returns the child expression that will be negated by this Not expression. - * - * @return the child expression - */ - public Expression getChild() { - return child; - } - - @Override - public T accept(Visitor visitor) { - return visitor.visitNot(this); - } -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/api/expressions/Root.java b/java/vortex-jni/src/main/java/dev/vortex/api/expressions/Root.java deleted file mode 100644 index ba8c0a28a59..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/api/expressions/Root.java +++ /dev/null @@ -1,66 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.api.expressions; - -import dev.vortex.api.Expression; -import java.util.List; -import java.util.Optional; - -/** - * Represents the root expression in a Vortex expression tree. - * This is a singleton expression that serves as the starting point for expression evaluation, - * typically representing the root context or input data from which other expressions derive values. - */ -public final class Root implements Expression { - /** - * The singleton instance of the Root expression. - * Since Root expressions have no state or parameters, a single instance is shared. - */ - public static final Root INSTANCE = new Root(); - - private Root() {} - - /** - * Parses a Root expression from serialized metadata and child expressions. - * This method is used during deserialization of Vortex expressions. - * - * @param _metadata the serialized metadata (ignored for Root expressions) - * @param children the child expressions, must be empty for Root expressions - * @return the singleton Root instance - * @throws RuntimeException if any children are provided - */ - public static Root parse(byte[] _metadata, List children) { - if (!children.isEmpty()) { - throw new IllegalArgumentException("Root expression must have no children, found: " + children.size()); - } - return INSTANCE; - } - - @Override - public String id() { - return "vortex.root"; - } - - @Override - public List children() { - return List.of(); - } - - @Override - public Optional metadata() { - return Optional.of(new byte[] {}); - } - - @Override - public String toString() { - return "$"; - } - - // equals and hashCode depend on address equality to INSTANCE. - - @Override - public T accept(Visitor visitor) { - return visitor.visitRoot(this); - } -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/api/expressions/Unknown.java b/java/vortex-jni/src/main/java/dev/vortex/api/expressions/Unknown.java deleted file mode 100644 index 8beedd06e5a..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/api/expressions/Unknown.java +++ /dev/null @@ -1,47 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.api.expressions; - -import dev.vortex.api.Expression; -import java.util.List; -import java.util.Optional; - -/** - * Represents a generic expression deserialized from a Vortex expression without a concrete Java type. - */ -public final class Unknown implements Expression { - private final String id; - private final List children; - private final byte[] metadata; - - /** - * Creates a new Unknown expression with the specified identifier, children, and metadata. - * This constructor is typically used when deserializing expressions that don't have - * a specific Java implementation, allowing them to be preserved as generic expressions. - * - * @param id the unique identifier for this expression type - * @param children the list of child expressions - * @param metadata the serialized metadata associated with this expression - */ - public Unknown(String id, List children, byte[] metadata) { - this.id = id; - this.children = children; - this.metadata = metadata; - } - - @Override - public String id() { - return id; - } - - @Override - public List children() { - return children; - } - - @Override - public Optional metadata() { - return Optional.of(metadata); - } -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/api/proto/DTypes.java b/java/vortex-jni/src/main/java/dev/vortex/api/proto/DTypes.java deleted file mode 100644 index 6ef609f9514..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/api/proto/DTypes.java +++ /dev/null @@ -1,208 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.api.proto; - -import static dev.vortex.api.proto.TemporalMetadatas.TIME_UNIT_MICROS; -import static dev.vortex.api.proto.TemporalMetadatas.TIME_UNIT_NANOS; - -import com.google.protobuf.ByteString; -import dev.vortex.proto.DTypeProtos; -import java.util.Optional; - -/** - * Factory class for creating Vortex data type definitions. - *

- * This class provides static factory methods to create {@link DTypeProtos.DType} instances - * for all supported Vortex data types. Data types in Vortex can be primitive types - * (like integers, floats, booleans), complex types (like strings, decimals, binary data), - * or extension types (like temporal types with specific metadata). - *

- *

- * Each data type can optionally be nullable, which affects how null values are handled - * during processing. The factory methods follow a consistent pattern where the boolean - * {@code nullable} parameter determines whether the type can contain null values. - *

- *

- * Extension types like dates, times, and timestamps are implemented as extensions - * over primitive storage types with additional metadata that defines their semantic - * meaning and encoding parameters. - *

- *

- * This class is primarily used internally by the {@link Scalars} factory class - * and other parts of the Vortex API to ensure consistent type definitions. - *

- */ -public final class DTypes { - private DTypes() {} - - static DTypeProtos.DType nullType() { - return DTypeProtos.DType.newBuilder() - .setNull(DTypeProtos.Null.newBuilder().build()) - .build(); - } - - static DTypeProtos.DType bool(boolean nullable) { - return DTypeProtos.DType.newBuilder() - .setBool(DTypeProtos.Bool.newBuilder().setNullable(nullable).build()) - .build(); - } - - static DTypeProtos.DType int8(boolean nullable) { - return DTypeProtos.DType.newBuilder() - .setPrimitive(DTypeProtos.Primitive.newBuilder() - .setType(DTypeProtos.PType.I8) - .setNullable(nullable) - .build()) - .build(); - } - - static DTypeProtos.DType int16(boolean nullable) { - return DTypeProtos.DType.newBuilder() - .setPrimitive(DTypeProtos.Primitive.newBuilder() - .setType(DTypeProtos.PType.I16) - .setNullable(nullable) - .build()) - .build(); - } - - static DTypeProtos.DType int32(boolean nullable) { - return DTypeProtos.DType.newBuilder() - .setPrimitive(DTypeProtos.Primitive.newBuilder() - .setType(DTypeProtos.PType.I32) - .setNullable(nullable) - .build()) - .build(); - } - - static DTypeProtos.DType int64(boolean nullable) { - return DTypeProtos.DType.newBuilder() - .setPrimitive(DTypeProtos.Primitive.newBuilder() - .setType(DTypeProtos.PType.I64) - .setNullable(nullable) - .build()) - .build(); - } - - static DTypeProtos.DType float32(boolean nullable) { - return DTypeProtos.DType.newBuilder() - .setPrimitive(DTypeProtos.Primitive.newBuilder() - .setType(DTypeProtos.PType.F32) - .setNullable(nullable) - .build()) - .build(); - } - - static DTypeProtos.DType float64(boolean nullable) { - return DTypeProtos.DType.newBuilder() - .setPrimitive(DTypeProtos.Primitive.newBuilder() - .setType(DTypeProtos.PType.F64) - .setNullable(nullable) - .build()) - .build(); - } - - static DTypeProtos.DType decimal(boolean nullable, int precision, int scale) { - return DTypeProtos.DType.newBuilder() - .setDecimal(DTypeProtos.Decimal.newBuilder() - .setNullable(nullable) - .setPrecision(precision) - .setScale(scale) - .build()) - .build(); - } - - static DTypeProtos.DType string(boolean nullable) { - return DTypeProtos.DType.newBuilder() - .setUtf8(DTypeProtos.Utf8.newBuilder().setNullable(nullable).build()) - .build(); - } - - static DTypeProtos.DType binary(boolean nullable) { - return DTypeProtos.DType.newBuilder() - .setBinary(DTypeProtos.Binary.newBuilder().setNullable(nullable).build()) - .build(); - } - - static DTypeProtos.DType dateDays(boolean nullable) { - return DTypeProtos.DType.newBuilder() - .setExtension(DTypeProtos.Extension.newBuilder() - .setId("vortex.date") - .setStorageDtype(DTypes.int32(nullable)) - .setMetadata(ByteString.copyFrom(TemporalMetadatas.DATE_DAYS.get())) - .build()) - .build(); - } - - static DTypeProtos.DType dateMillis(boolean nullable) { - return DTypeProtos.DType.newBuilder() - .setExtension(DTypeProtos.Extension.newBuilder() - .setId("vortex.date") - .setStorageDtype(DTypes.int64(nullable)) - .setMetadata(ByteString.copyFrom(TemporalMetadatas.DATE_MILLIS.get())) - .build()) - .build(); - } - - static DTypeProtos.DType timeSeconds(boolean nullable) { - return DTypeProtos.DType.newBuilder() - .setExtension(DTypeProtos.Extension.newBuilder() - .setId("vortex.time") - .setStorageDtype(DTypes.int32(nullable)) - .setMetadata(ByteString.copyFrom(TemporalMetadatas.TIME_SECONDS.get())) - .build()) - .build(); - } - - static DTypeProtos.DType timeMillis(boolean nullable) { - return DTypeProtos.DType.newBuilder() - .setExtension(DTypeProtos.Extension.newBuilder() - .setId("vortex.time") - .setStorageDtype(DTypes.int32(nullable)) - .setMetadata(ByteString.copyFrom(TemporalMetadatas.TIME_MILLIS.get())) - .build()) - .build(); - } - - static DTypeProtos.DType timeMicros(boolean nullable) { - return DTypeProtos.DType.newBuilder() - .setExtension(DTypeProtos.Extension.newBuilder() - .setId("vortex.time") - .setStorageDtype(DTypes.int64(nullable)) - .setMetadata(ByteString.copyFrom(TemporalMetadatas.TIME_MICROS.get())) - .build()) - .build(); - } - - static DTypeProtos.DType timeNanos(boolean nullable) { - return DTypeProtos.DType.newBuilder() - .setExtension(DTypeProtos.Extension.newBuilder() - .setId("vortex.time") - .setStorageDtype(DTypes.int64(nullable)) - .setMetadata(ByteString.copyFrom(TemporalMetadatas.TIME_NANOS.get())) - .build()) - .build(); - } - - static DTypeProtos.DType timestampMillis(Optional timeZone, boolean nullable) { - return timestamp(TemporalMetadatas.TIME_UNIT_MILLIS, timeZone, nullable); - } - - static DTypeProtos.DType timestampMicros(Optional timeZone, boolean nullable) { - return timestamp(TIME_UNIT_MICROS, timeZone, nullable); - } - - static DTypeProtos.DType timestampNanos(Optional timeZone, boolean nullable) { - return timestamp(TIME_UNIT_NANOS, timeZone, nullable); - } - - private static DTypeProtos.DType timestamp(byte timeUnit, Optional timeZone, boolean nullable) { - return DTypeProtos.DType.newBuilder() - .setExtension(DTypeProtos.Extension.newBuilder() - .setId("vortex.timestamp") - .setStorageDtype(DTypes.int64(nullable)) - .setMetadata(ByteString.copyFrom(TemporalMetadatas.timestamp(timeUnit, timeZone))) - .build()) - .build(); - } -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/api/proto/EndianUtils.java b/java/vortex-jni/src/main/java/dev/vortex/api/proto/EndianUtils.java deleted file mode 100644 index 6855d92e454..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/api/proto/EndianUtils.java +++ /dev/null @@ -1,78 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.api.proto; - -import com.google.protobuf.ByteString; -import java.math.BigDecimal; -import java.math.BigInteger; - -/** - * Utility class for handling endianness conversions in Vortex protocol buffers. - * Provides methods for converting between big-endian and little-endian byte representations, - * particularly for decimal values that need to be serialized to the Vortex format. - */ -public final class EndianUtils { - /** - * Reverses the byte order of a ByteString, converting from one endianness to another. - * - * @param src the source ByteString to reverse - * @return a new byte array with bytes in reverse order - */ - public static byte[] reverse(ByteString src) { - byte[] dst = new byte[src.size()]; - for (int i = 0; i < dst.length; i++) { - dst[i] = src.byteAt(dst.length - 1 - i); - } - return dst; - } - - /** - * Converts a BigDecimal to a little-endian byte array representation suitable for Vortex decimals. - * The method extracts the unscaled value of the decimal, converts it from big-endian to little-endian, - * and pads to a standard size (1, 2, 4, 8, 16, or 32 bytes) with proper sign extension for negative values. - * - * @param decimal the BigDecimal value to convert - * @return a little-endian byte array representation of the decimal's unscaled value - * @throws RuntimeException if the BigDecimal is too large for Arrow - */ - public static byte[] littleEndianDecimal(BigDecimal decimal) { - BigInteger unscaled = decimal.unscaledValue(); - byte[] bigEndianBytes = unscaled.toByteArray(); - - // Determine target size (1, 2, 4, 8, 16, or 32 bytes) - int targetSize; - if (bigEndianBytes.length <= 1) { - targetSize = 1; - } else if (bigEndianBytes.length <= 2) { - targetSize = 2; - } else if (bigEndianBytes.length <= 4) { - targetSize = 4; - } else if (bigEndianBytes.length <= 8) { - targetSize = 8; - } else if (bigEndianBytes.length <= 16) { - targetSize = 16; - } else if (bigEndianBytes.length <= 32) { - targetSize = 32; - } else { - throw new IllegalArgumentException( - "BigDecimal with " + bigEndianBytes.length + " bytes overflows maximum Vortex decimal size"); - } - - byte[] result = new byte[targetSize]; - - // Copy bytes in reverse order (big endian to little endian) - for (int i = 0; i < bigEndianBytes.length; i++) { - result[i] = bigEndianBytes[bigEndianBytes.length - 1 - i]; - } - - // Sign extend if negative - if (unscaled.signum() < 0) { - for (int i = bigEndianBytes.length; i < targetSize; i++) { - result[i] = (byte) 0xFF; - } - } - - return result; - } -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/api/proto/Expressions.java b/java/vortex-jni/src/main/java/dev/vortex/api/proto/Expressions.java deleted file mode 100644 index dc2464851f2..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/api/proto/Expressions.java +++ /dev/null @@ -1,67 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.api.proto; - -import com.google.protobuf.ByteString; -import dev.vortex.api.Expression; -import dev.vortex.api.expressions.*; -import dev.vortex.proto.ExprProtos; -import java.util.List; -import java.util.stream.Collectors; - -/** - * Generate a protocol buffers representation of an {@link Expression}. - */ -public final class Expressions { - /** - * Serialize an {@link Expression} to a protocol buffer. - */ - public static ExprProtos.Expr serialize(Expression expression) { - ByteString metadata = ByteString.copyFrom(expression - .metadata() - .orElseThrow(() -> new IllegalArgumentException("Expression is not serializable: " + expression.id()))); - - return ExprProtos.Expr.newBuilder() - .setId(expression.id()) - .addAllChildren(expression.children().stream() - .map(Expressions::serialize) - .collect(Collectors.toList())) - .setMetadata(metadata) - .build(); - } - - /** - * Deserialize a protocol buffer representation back into an {@link Expression} object. - * The method examines the expression ID and creates the appropriate concrete expression type - * based on the registered expression types (binary, get_item, root, literal, not). - * If the expression ID is not recognized, an {@link Unknown} expression is created. - * - * @param expr the protocol buffer expression to deserialize - * @return the deserialized Expression object - */ - public static Expression deserialize(ExprProtos.Expr expr) { - byte[] metadata = expr.getMetadata().toByteArray(); - List children = - expr.getChildrenList().stream().map(Expressions::deserialize).collect(Collectors.toList()); - - switch (expr.getId()) { - case "vortex.binary": - return Binary.parse(metadata, children); - case "vortex.get_item": - return GetItem.parse(metadata, children); - case "vortex.root": - return Root.parse(metadata, children); - case "vortex.literal": - return Literal.parse(metadata, children); - case "vortex.not": - return Not.parse(metadata, children); - case "vortex.is_null": - return IsNull.parse(metadata, children); - default: - return new Unknown(expr.getId(), children, expr.getMetadata().toByteArray()); - } - } - - private Expressions() {} -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/api/proto/Scalars.java b/java/vortex-jni/src/main/java/dev/vortex/api/proto/Scalars.java deleted file mode 100644 index 40ea7b09880..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/api/proto/Scalars.java +++ /dev/null @@ -1,443 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.api.proto; - -import com.google.protobuf.ByteString; -import com.google.protobuf.NullValue; -import dev.vortex.proto.ScalarProtos; -import java.math.BigDecimal; -import java.util.Optional; - -/** - * Factory class for creating Vortex scalar values with their associated data types. - *

- * This class provides static factory methods to create {@link ScalarProtos.Scalar} instances - * for all supported Vortex data types. Each scalar consists of a value and its corresponding - * data type definition. The class supports both nullable and non-nullable variants for most types. - *

- *

- * Factory methods follow a consistent naming pattern: - *

- *
    - *
  • {@code typeName(value)} - Creates a non-nullable scalar with the given value
  • - *
  • {@code nullTypeName()} - Creates a nullable scalar with a null value
  • - *
- *

- * Example usage: - *

- *
{@code
- * // Create a non-nullable integer
- * ScalarProtos.Scalar intScalar = Scalars.int32(42);
- *
- * // Create a nullable string with null value
- * ScalarProtos.Scalar nullString = Scalars.nullString();
- *
- * // Create a string with value
- * ScalarProtos.Scalar stringScalar = Scalars.string("hello");
- * }
- */ -public final class Scalars { - private Scalars() {} - - /** - * Creates a null scalar value. - *

- * This represents a scalar that is explicitly null with null data type. - *

- * - * @return a {@link ScalarProtos.Scalar} representing a null value - */ - public static ScalarProtos.Scalar nullNull() { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setNullValue(NullValue.NULL_VALUE) - .build()) - .setDtype(DTypes.nullType()) - .build(); - } - - /** - * Creates a non-nullable boolean scalar with the specified value. - * - * @param value the boolean value - * @return a {@link ScalarProtos.Scalar} containing the boolean value - */ - public static ScalarProtos.Scalar bool(boolean value) { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setBoolValue(value) - .build()) - .setDtype(DTypes.bool(false)) - .build(); - } - - /** - * Creates a nullable boolean scalar with a null value. - * - * @return a {@link ScalarProtos.Scalar} representing a null boolean - */ - public static ScalarProtos.Scalar nullBool() { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setNullValue(NullValue.NULL_VALUE) - .build()) - .setDtype(DTypes.bool(true)) - .build(); - } - - public static ScalarProtos.Scalar int8(byte value) { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setInt64Value(value) - .build()) - .setDtype(DTypes.int8(false)) - .build(); - } - - public static ScalarProtos.Scalar nullInt8() { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setNullValue(NullValue.NULL_VALUE) - .build()) - .setDtype(DTypes.int8(true)) - .build(); - } - - public static ScalarProtos.Scalar int16(short value) { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setInt64Value(value) - .build()) - .setDtype(DTypes.int16(false)) - .build(); - } - - public static ScalarProtos.Scalar nullInt16() { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setNullValue(NullValue.NULL_VALUE) - .build()) - .setDtype(DTypes.int16(true)) - .build(); - } - - /** - * Creates a non-nullable 32-bit integer scalar with the specified value. - * - * @param value the integer value - * @return a {@link ScalarProtos.Scalar} containing the 32-bit integer value - */ - public static ScalarProtos.Scalar int32(int value) { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setInt64Value(value) - .build()) - .setDtype(DTypes.int32(false)) - .build(); - } - - public static ScalarProtos.Scalar nullInt32() { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setNullValue(NullValue.NULL_VALUE) - .build()) - .setDtype(DTypes.int32(true)) - .build(); - } - - public static ScalarProtos.Scalar int64(long value) { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setInt64Value(value) - .build()) - .setDtype(DTypes.int64(false)) - .build(); - } - - public static ScalarProtos.Scalar nullInt64() { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setNullValue(NullValue.NULL_VALUE) - .build()) - .setDtype(DTypes.int64(true)) - .build(); - } - - public static ScalarProtos.Scalar float32(float value) { - return ScalarProtos.Scalar.newBuilder() - .setValue( - ScalarProtos.ScalarValue.newBuilder().setF32Value(value).build()) - .setDtype(DTypes.float32(false)) - .build(); - } - - public static ScalarProtos.Scalar nullFloat32() { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setNullValue(NullValue.NULL_VALUE) - .build()) - .setDtype(DTypes.float32(true)) - .build(); - } - - public static ScalarProtos.Scalar float64(double value) { - return ScalarProtos.Scalar.newBuilder() - .setValue( - ScalarProtos.ScalarValue.newBuilder().setF64Value(value).build()) - .setDtype(DTypes.float64(false)) - .build(); - } - - public static ScalarProtos.Scalar nullFloat64() { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setNullValue(NullValue.NULL_VALUE) - .build()) - .setDtype(DTypes.float64(true)) - .build(); - } - - /** - * Creates a non-nullable string scalar with the specified value. - * - * @param value the string value - * @return a {@link ScalarProtos.Scalar} containing the string value - */ - public static ScalarProtos.Scalar string(String value) { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setStringValue(value) - .build()) - .setDtype(DTypes.string(false)) - .build(); - } - - public static ScalarProtos.Scalar nullString() { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setNullValue(NullValue.NULL_VALUE) - .build()) - .setDtype(DTypes.string(true)) - .build(); - } - - /** - * Creates a non-nullable decimal scalar with the specified value, precision, and scale. - *

- * The decimal value is converted to little-endian byte representation for storage. - *

- * - * @param decimal the decimal value as a {@link BigDecimal} - * @param precision the total number of digits in the decimal - * @param scale the number of digits after the decimal point - * @return a {@link ScalarProtos.Scalar} containing the decimal value - */ - public static ScalarProtos.Scalar decimal(BigDecimal decimal, int precision, int scale) { - var littleEndian = EndianUtils.littleEndianDecimal(decimal); - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setBytesValue(ByteString.copyFrom(littleEndian)) - .build()) - .setDtype(DTypes.decimal(false, precision, scale)) - .build(); - } - - public static ScalarProtos.Scalar nullDecimal(int precision, int scale) { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder().setNullValue(NullValue.NULL_VALUE)) - .setDtype(DTypes.decimal(true, precision, scale)) - .build(); - } - - public static ScalarProtos.Scalar bytes(byte[] value) { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setBytesValue(ByteString.copyFrom(value)) - .build()) - .setDtype(DTypes.binary(false)) - .build(); - } - - public static ScalarProtos.Scalar nullBytes() { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setNullValue(NullValue.NULL_VALUE) - .build()) - .setDtype(DTypes.binary(true)) - .build(); - } - - public static ScalarProtos.Scalar dateDays(int value) { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setInt64Value(value) - .build()) - .setDtype(DTypes.dateDays(false)) - .build(); - } - - public static ScalarProtos.Scalar nullDateDays() { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setNullValue(NullValue.NULL_VALUE) - .build()) - .setDtype(DTypes.dateDays(true)) - .build(); - } - - public static ScalarProtos.Scalar dateMillis(long value) { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setInt64Value(value) - .build()) - .setDtype(DTypes.dateMillis(false)) - .build(); - } - - public static ScalarProtos.Scalar nullDateMillis() { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setNullValue(NullValue.NULL_VALUE) - .build()) - .setDtype(DTypes.dateMillis(true)) - .build(); - } - - public static ScalarProtos.Scalar timeSeconds(int value) { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setInt64Value(value) - .build()) - .setDtype(DTypes.timeSeconds(false)) - .build(); - } - - public static ScalarProtos.Scalar nullTimeSeconds() { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setNullValue(NullValue.NULL_VALUE) - .build()) - .setDtype(DTypes.timeSeconds(true)) - .build(); - } - - public static ScalarProtos.Scalar timeMillis(int value) { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setInt64Value(value) - .build()) - .setDtype(DTypes.timeMillis(false)) - .build(); - } - - public static ScalarProtos.Scalar nullTimeMillis() { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setNullValue(NullValue.NULL_VALUE) - .build()) - .setDtype(DTypes.timeMillis(true)) - .build(); - } - - public static ScalarProtos.Scalar timeMicros(long value) { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setInt64Value(value) - .build()) - .setDtype(DTypes.timeMicros(false)) - .build(); - } - - public static ScalarProtos.Scalar nullTimeMicros() { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setNullValue(NullValue.NULL_VALUE) - .build()) - .setDtype(DTypes.timeMicros(true)) - .build(); - } - - public static ScalarProtos.Scalar timeNanos(long value) { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setInt64Value(value) - .build()) - .setDtype(DTypes.timeNanos(false)) - .build(); - } - - public static ScalarProtos.Scalar nullTimeNanos() { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setNullValue(NullValue.NULL_VALUE) - .build()) - .setDtype(DTypes.timeNanos(true)) - .build(); - } - - /** - * Creates a non-nullable timestamp scalar with millisecond precision. - *

- * The timestamp represents the number of milliseconds since Unix epoch - * (1970-01-01 00:00:00 UTC), optionally adjusted for the specified time zone. - *

- * - * @param value the timestamp value in milliseconds since Unix epoch - * @param timeZone optional time zone identifier (e.g., "UTC", "America/New_York") - * @return a {@link ScalarProtos.Scalar} containing the timestamp value - */ - public static ScalarProtos.Scalar timestampMillis(long value, Optional timeZone) { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setInt64Value(value) - .build()) - .setDtype(DTypes.timestampMillis(timeZone, false)) - .build(); - } - - public static ScalarProtos.Scalar nullTimestampMillis(Optional timeZone) { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setNullValue(NullValue.NULL_VALUE) - .build()) - .setDtype(DTypes.timestampMillis(timeZone, true)) - .build(); - } - - public static ScalarProtos.Scalar timestampMicros(long value, Optional timeZone) { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setInt64Value(value) - .build()) - .setDtype(DTypes.timestampMicros(timeZone, false)) - .build(); - } - - public static ScalarProtos.Scalar nullTimestampMicros(Optional timeZone) { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setNullValue(NullValue.NULL_VALUE) - .build()) - .setDtype(DTypes.timestampMicros(timeZone, true)) - .build(); - } - - public static ScalarProtos.Scalar timestampNanos(long value, Optional timeZone) { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setInt64Value(value) - .build()) - .setDtype(DTypes.timestampNanos(timeZone, false)) - .build(); - } - - public static ScalarProtos.Scalar nullTimestampNanos(Optional timeZone) { - return ScalarProtos.Scalar.newBuilder() - .setValue(ScalarProtos.ScalarValue.newBuilder() - .setNullValue(NullValue.NULL_VALUE) - .build()) - .setDtype(DTypes.timestampNanos(timeZone, true)) - .build(); - } -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/api/proto/TemporalMetadatas.java b/java/vortex-jni/src/main/java/dev/vortex/api/proto/TemporalMetadatas.java deleted file mode 100644 index d5d5756f98e..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/api/proto/TemporalMetadatas.java +++ /dev/null @@ -1,114 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.api.proto; - -import com.google.common.base.Preconditions; -import com.google.common.base.Supplier; -import com.google.common.base.Suppliers; -import java.io.ByteArrayOutputStream; -import java.nio.charset.StandardCharsets; -import java.util.Optional; - -/** - * Utility class for creating and parsing temporal metadata in Vortex protocol buffers. - * Provides constants for time units and methods for serializing/deserializing timestamp - * metadata including time zone information. This class handles the binary format used - * to encode temporal type information in the Vortex data format. - */ -public final class TemporalMetadatas { - private TemporalMetadatas() {} - - /** Time unit constant representing nanoseconds precision. */ - public static byte TIME_UNIT_NANOS = 0; - /** Time unit constant representing microseconds precision. */ - public static byte TIME_UNIT_MICROS = 1; - /** Time unit constant representing milliseconds precision. */ - public static byte TIME_UNIT_MILLIS = 2; - /** Time unit constant representing seconds precision. */ - public static byte TIME_UNIT_SECONDS = 3; - /** Time unit constant representing days precision. */ - public static byte TIME_UNIT_DAYS = 4; - - /** Supplier for date metadata with days precision. */ - public static final Supplier DATE_DAYS = Suppliers.memoize(() -> new byte[] {TIME_UNIT_DAYS}); - /** Supplier for date metadata with milliseconds precision. */ - public static final Supplier DATE_MILLIS = Suppliers.memoize(() -> new byte[] {TIME_UNIT_MILLIS}); - /** Supplier for time metadata with seconds precision. */ - public static final Supplier TIME_SECONDS = Suppliers.memoize(() -> new byte[] {TIME_UNIT_SECONDS}); - /** Supplier for time metadata with milliseconds precision. */ - public static final Supplier TIME_MILLIS = Suppliers.memoize(() -> new byte[] {TIME_UNIT_MILLIS}); - /** Supplier for time metadata with microseconds precision. */ - public static final Supplier TIME_MICROS = Suppliers.memoize(() -> new byte[] {TIME_UNIT_MICROS}); - /** Supplier for time metadata with nanoseconds precision. */ - public static final Supplier TIME_NANOS = Suppliers.memoize(() -> new byte[] {TIME_UNIT_NANOS}); - - /** - * Creates serialized timestamp metadata with the specified time unit and optional time zone. - * The resulting byte array contains the time unit as the first byte, followed by a little-endian - * uint16 length field, and then the UTF-8 encoded time zone string (if present). - * - * @param timeUnit the time unit for the timestamp, must be between TIME_UNIT_NANOS and TIME_UNIT_SECONDS (exclusive of TIME_UNIT_DAYS) - * @param timeZone optional time zone identifier string - * @return serialized timestamp metadata as a byte array - * @throws RuntimeException if timeUnit is invalid for timestamps - */ - public static byte[] timestamp(byte timeUnit, Optional timeZone) { - Preconditions.checkArgument( - timeUnit >= TIME_UNIT_NANOS && timeUnit < TIME_UNIT_DAYS, "invalid timeUnit for Timestamp:" + timeUnit); - - ByteArrayOutputStream baos = new ByteArrayOutputStream(); - baos.write(timeUnit); - if (timeZone.isPresent()) { - byte[] timeZoneBytes = timeZone.get().getBytes(StandardCharsets.UTF_8); - // Write length as little-endian uint16. - int lenLow = timeZoneBytes.length & 0xFF; - int lenHigh = (timeZoneBytes.length >> 8) & 0xFF; - baos.write(lenLow); - baos.write(lenHigh); - baos.writeBytes(timeZoneBytes); - } else { - // write uint16 zero value - baos.write(0); - baos.write(0); - } - return baos.toByteArray(); - } - - /** - * Extract the time unit byte from the serialized metadata. - * - * @param serializedMetadata the serialized temporal metadata byte array - * @return the time unit byte from the first position of the metadata - * @throws RuntimeException if the time unit byte is invalid - */ - public static byte getTimeUnit(byte[] serializedMetadata) { - byte timeUnit = serializedMetadata[0]; - Preconditions.checkArgument( - timeUnit >= TIME_UNIT_NANOS && timeUnit <= TIME_UNIT_DAYS, "invalid timeUnit byte: " + timeUnit); - - return timeUnit; - } - - /** - * Read from a serialized metadata representation into a time zone string. - * Parses the time zone information from the serialized metadata, reading the length - * as a little-endian uint16 and then decoding the UTF-8 time zone string. - * - * @param serializedMetadata the serialized temporal metadata byte array - * @return Optional containing the time zone string, or empty if no time zone was specified - */ - public static Optional getTimeZone(byte[] serializedMetadata) { - byte _timeUnit = serializedMetadata[0]; - byte lenLow = serializedMetadata[1]; - byte lenHigh = serializedMetadata[2]; - int len = ((lenHigh & 0xFF) << 8) | (lenLow & 0xFF); - if (len == 0) { - return Optional.empty(); - } else { - byte[] timeZoneBytes = new byte[len]; - System.arraycopy(serializedMetadata, 3, timeZoneBytes, 0, len); - return Optional.of(new String(timeZoneBytes, StandardCharsets.UTF_8)); - } - } -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/arrow/ArrowAllocation.java b/java/vortex-jni/src/main/java/dev/vortex/arrow/ArrowAllocation.java index 6e0c043f60a..0f014b9178e 100644 --- a/java/vortex-jni/src/main/java/dev/vortex/arrow/ArrowAllocation.java +++ b/java/vortex-jni/src/main/java/dev/vortex/arrow/ArrowAllocation.java @@ -8,11 +8,9 @@ /** * Utility class for managing Apache Arrow memory allocation. - *

- * This class provides a global shared root allocator for Arrow memory operations - * used throughout the Vortex JNI layer. The allocator is configured with the maximum - * available heap size to allow for efficient memory management. - *

+ * + *

This class provides a global shared root allocator for Arrow memory operations used throughout the Vortex JNI + * layer. The allocator is configured with the maximum available heap size to allow for efficient memory management. */ public final class ArrowAllocation { private static final RootAllocator ROOT_ALLOCATOR = new RootAllocator(Long.MAX_VALUE); @@ -21,11 +19,9 @@ private ArrowAllocation() {} /** * Returns the shared root allocator instance for Apache Arrow operations. - *

- * This allocator is shared across all Arrow operations in the JVM and is configured - * to use the maximum available memory. It should be used as the parent allocator - * for all Arrow memory operations within the Vortex system. - *

+ * + *

This allocator is shared across all Arrow operations in the JVM and is configured to use the maximum available + * memory. It should be used as the parent allocator for all Arrow memory operations within the Vortex system. * * @return the shared {@link BufferAllocator} instance */ diff --git a/java/vortex-jni/src/main/java/dev/vortex/jni/JNIArray.java b/java/vortex-jni/src/main/java/dev/vortex/jni/JNIArray.java deleted file mode 100644 index db019339dac..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/jni/JNIArray.java +++ /dev/null @@ -1,151 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.jni; - -import com.google.common.base.Preconditions; -import dev.vortex.api.Array; -import dev.vortex.api.DType; -import java.math.BigDecimal; -import java.util.OptionalLong; -import org.apache.arrow.c.ArrowArray; -import org.apache.arrow.c.ArrowSchema; -import org.apache.arrow.c.CDataDictionaryProvider; -import org.apache.arrow.c.Data; -import org.apache.arrow.memory.BufferAllocator; -import org.apache.arrow.vector.VectorSchemaRoot; - -public final class JNIArray implements Array { - static { - NativeLoader.loadJni(); - } - - private final ThreadLocal schemaPtr = ThreadLocal.withInitial(() -> new long[1]); - private final ThreadLocal arrayPtr = ThreadLocal.withInitial(() -> new long[1]); - - private OptionalLong pointer; - - public JNIArray(long pointer) { - Preconditions.checkArgument(pointer > 0, "Invalid pointer address: " + pointer); - this.pointer = OptionalLong.of(pointer); - } - - @Override - public long getLen() { - return NativeArrayMethods.getLen(pointer.getAsLong()); - } - - @Override - public long nbytes() { - return NativeArrayMethods.nbytes(pointer.getAsLong()); - } - - @Override - public VectorSchemaRoot exportToArrow(BufferAllocator allocator, VectorSchemaRoot reuse) { - // Export the dataset to Arrow over C Data Interface. - NativeArrayMethods.exportToArrow(pointer.getAsLong(), schemaPtr.get(), arrayPtr.get()); - try (ArrowSchema arrowSchema = ArrowSchema.wrap(schemaPtr.get()[0]); - ArrowArray arrowArray = ArrowArray.wrap(arrayPtr.get()[0]); - CDataDictionaryProvider provider = new CDataDictionaryProvider()) { - if (reuse != null) { - Data.importIntoVectorSchemaRoot(allocator, arrowArray, reuse, provider); - return reuse; - } else { - return Data.importVectorSchemaRoot(allocator, arrowArray, arrowSchema, new CDataDictionaryProvider()); - } - } finally { - NativeArrayMethods.dropArrowSchema(schemaPtr.get()[0]); - NativeArrayMethods.dropArrowArray(arrayPtr.get()[0]); - } - } - - @Override - public DType getDataType() { - return new JNIDType(NativeArrayMethods.getDataType(pointer.getAsLong())); - } - - @Override - public Array getField(int index) { - return new JNIArray(NativeArrayMethods.getField(pointer.getAsLong(), index)); - } - - @Override - public Array slice(int start, int stop) { - return new JNIArray(NativeArrayMethods.slice(pointer.getAsLong(), start, stop)); - } - - @Override - public boolean getNull(int index) { - return NativeArrayMethods.getNull(pointer.getAsLong(), index); - } - - @Override - public int getNullCount() { - return NativeArrayMethods.getNullCount(pointer.getAsLong()); - } - - @Override - public byte getByte(int index) { - return NativeArrayMethods.getByte(pointer.getAsLong(), index); - } - - @Override - public short getShort(int index) { - return NativeArrayMethods.getShort(pointer.getAsLong(), index); - } - - @Override - public int getInt(int index) { - return NativeArrayMethods.getInt(pointer.getAsLong(), index); - } - - @Override - public long getLong(int index) { - return NativeArrayMethods.getLong(pointer.getAsLong(), index); - } - - @Override - public boolean getBool(int index) { - return NativeArrayMethods.getBool(pointer.getAsLong(), index); - } - - @Override - public float getFloat(int index) { - return NativeArrayMethods.getFloat(pointer.getAsLong(), index); - } - - @Override - public double getDouble(int index) { - return NativeArrayMethods.getDouble(pointer.getAsLong(), index); - } - - @Override - public BigDecimal getBigDecimal(int index) { - return NativeArrayMethods.getBigDecimal(pointer.getAsLong(), index); - } - - @Override - public String getUTF8(int index) { - return NativeArrayMethods.getUTF8(pointer.getAsLong(), index); - } - - @Override - public void getUTF8_ptr_len(int index, long[] ptr, int[] len) { - NativeArrayMethods.getUTF8_ptr_len(pointer.getAsLong(), index, ptr, len); - } - - @Override - public byte[] getBinary(int index) { - return NativeArrayMethods.getBinary(pointer.getAsLong(), index); - } - - @Override - public void close() { - if (pointer.isEmpty()) { - return; - } - - NativeArrayMethods.free(pointer.getAsLong()); - pointer = OptionalLong.empty(); - } -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/jni/JNIArrayIterator.java b/java/vortex-jni/src/main/java/dev/vortex/jni/JNIArrayIterator.java deleted file mode 100644 index 33f0661e9cc..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/jni/JNIArrayIterator.java +++ /dev/null @@ -1,59 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.jni; - -import com.google.common.base.Preconditions; -import dev.vortex.api.Array; -import dev.vortex.api.ArrayIterator; -import dev.vortex.api.DType; -import java.util.Optional; -import java.util.OptionalLong; - -public final class JNIArrayIterator implements ArrayIterator { - private OptionalLong pointer; - private Optional next; - - public JNIArrayIterator(long pointer) { - Preconditions.checkArgument(pointer > 0, "Invalid pointer address: " + pointer); - this.pointer = OptionalLong.of(pointer); - advance(); - } - - @Override - public boolean hasNext() { - return next.isPresent(); - } - - @Override - public Array next() { - Array array = this.next.get(); - advance(); - return array; - } - - @Override - public DType getDataType() { - return new JNIDType(NativeArrayIteratorMethods.getDType(pointer.getAsLong())); - } - - @Override - public void close() { - if (pointer.isEmpty()) { - return; - } - - NativeArrayIteratorMethods.free(pointer.getAsLong()); - pointer = OptionalLong.empty(); - next = Optional.empty(); - } - - private void advance() { - long next = NativeArrayIteratorMethods.take(pointer.getAsLong()); - if (next <= 0) { - this.next = Optional.empty(); - } else { - this.next = Optional.of(new JNIArray(next)); - } - } -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/jni/JNIDType.java b/java/vortex-jni/src/main/java/dev/vortex/jni/JNIDType.java deleted file mode 100644 index d19dc0624e7..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/jni/JNIDType.java +++ /dev/null @@ -1,127 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.jni; - -import com.google.common.base.Preconditions; -import com.google.common.collect.Lists; -import dev.vortex.api.DType; -import java.util.List; -import java.util.Optional; -import java.util.OptionalLong; - -public final class JNIDType implements DType { - OptionalLong pointer; - final boolean isOwned; // True if this object owns the native memory - - /** - * Creates a JNIDType that borrows a native pointer. - * The caller is responsible for ensuring the pointer remains valid. - */ - public JNIDType(long pointer) { - this(pointer, false); - } - - public long getPointer() { - return pointer.getAsLong(); - } - - /** - * Creates a JNIDType with explicit ownership. - * - * @param pointer the native pointer - * @param isOwned true if this object owns the native memory and should free it - */ - public JNIDType(long pointer, boolean isOwned) { - Preconditions.checkArgument(pointer > 0, "Invalid pointer address: " + pointer); - this.pointer = OptionalLong.of(pointer); - this.isOwned = isOwned; - } - - @Override - public Variant getVariant() { - return Variant.from(NativeDTypeMethods.getVariant(pointer.getAsLong())); - } - - @Override - public boolean isNullable() { - return NativeDTypeMethods.isNullable(pointer.getAsLong()); - } - - @Override - public List getFieldNames() { - return NativeDTypeMethods.getFieldNames(pointer.getAsLong()); - } - - @Override - public List getFieldTypes() { - return Lists.transform(NativeDTypeMethods.getFieldTypes(pointer.getAsLong()), JNIDType::new); - } - - @Override - public DType getElementType() { - // Returns a borrowed reference - the parent DType owns this memory - return new JNIDType(NativeDTypeMethods.getElementType(pointer.getAsLong())); - } - - @Override - public int getFixedSizeListSize() { - return NativeDTypeMethods.getFixedSizeListSize(pointer.getAsLong()); - } - - @Override - public boolean isDate() { - return NativeDTypeMethods.isDate(pointer.getAsLong()); - } - - @Override - public boolean isTime() { - return NativeDTypeMethods.isTime(pointer.getAsLong()); - } - - @Override - public boolean isTimestamp() { - return NativeDTypeMethods.isTimestamp(pointer.getAsLong()); - } - - @Override - public TimeUnit getTimeUnit() { - return TimeUnit.from(NativeDTypeMethods.getTimeUnit(pointer.getAsLong())); - } - - @Override - public Optional getTimeZone() { - return Optional.ofNullable(NativeDTypeMethods.getTimeZone(pointer.getAsLong())); - } - - @Override - public boolean isDecimal() { - return NativeDTypeMethods.isDecimal(pointer.getAsLong()); - } - - @Override - public int getPrecision() { - return NativeDTypeMethods.getDecimalPrecision(pointer.getAsLong()); - } - - @Override - public byte getScale() { - return NativeDTypeMethods.getDecimalScale(pointer.getAsLong()); - } - - @Override - public void close() { - if (isOwned && pointer.isPresent()) { - NativeDTypeMethods.free(pointer.getAsLong()); - pointer = OptionalLong.empty(); - } - } - - /** - * Creates a JNIDType that owns its native memory. - * This should only be used when receiving a newly allocated pointer from Rust. - */ - public static JNIDType ownedDType(long pointer) { - return new JNIDType(pointer, true); - } -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/jni/JNIFile.java b/java/vortex-jni/src/main/java/dev/vortex/jni/JNIFile.java deleted file mode 100644 index d26c20bd18b..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/jni/JNIFile.java +++ /dev/null @@ -1,55 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.jni; - -import com.google.common.base.Preconditions; -import dev.vortex.api.ArrayIterator; -import dev.vortex.api.DType; -import dev.vortex.api.File; -import dev.vortex.api.ScanOptions; -import dev.vortex.api.proto.Expressions; -import java.util.OptionalLong; - -public final class JNIFile implements File { - private OptionalLong pointer; - - public JNIFile(long pointer) { - Preconditions.checkArgument(pointer > 0, "Invalid pointer address: " + pointer); - this.pointer = OptionalLong.of(pointer); - } - - @Override - public DType getDType() { - return new JNIDType(NativeFileMethods.dtype(pointer.getAsLong())); - } - - @Override - public long rowCount() { - return NativeFileMethods.rowCount(pointer.getAsLong()); - } - - @Override - public ArrayIterator newScan(ScanOptions options) { - byte[] predicateProto = null; - - if (options.predicate().isPresent()) { - predicateProto = Expressions.serialize(options.predicate().get()).toByteArray(); - } - - long[] rowIndices = options.rowIndices().orElse(null); - long[] rowRange = options.rowRange().orElse(null); - - return new JNIArrayIterator( - NativeFileMethods.scan(pointer.getAsLong(), options.columns(), predicateProto, rowRange, rowIndices)); - } - - @Override - public void close() { - if (pointer.isEmpty()) { - return; - } - NativeFileMethods.close(pointer.getAsLong()); - pointer = OptionalLong.empty(); - } -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/jni/JNIWriter.java b/java/vortex-jni/src/main/java/dev/vortex/jni/JNIWriter.java deleted file mode 100644 index 6b0c62ca091..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/jni/JNIWriter.java +++ /dev/null @@ -1,87 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.jni; - -import dev.vortex.api.VortexWriter; -import java.io.IOException; -import java.util.OptionalLong; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -/** - * JNI implementation of VortexWriter. - *

- * This class implements AutoCloseable to ensure proper resource cleanup - * when used with try-with-resources. - */ -public final class JNIWriter implements VortexWriter, AutoCloseable { - private static final Logger logger = LoggerFactory.getLogger(JNIWriter.class); - - private OptionalLong ptr; - - /** - * Creates a new JNIWriter with the given native pointer. - * - * @param ptr the native writer pointer - */ - public JNIWriter(long ptr) { - this.ptr = OptionalLong.of(ptr); - logger.debug("Created JNIWriter with ptr={}", ptr); - } - - /** - * Writes a batch of Arrow data to the Vortex file. - * - * @param arrowData the Arrow data in IPC format as byte array - * @throws NullPointerException if this is called after the writer has been closed. - */ - @Override - public void writeBatch(byte[] arrowData) throws IOException { - logger.trace("Writing batch with {} bytes", arrowData.length); - - // Write the Arrow data to Vortex through JNI - boolean success = NativeWriterMethods.writeBatch(ptr.getAsLong(), arrowData); - if (!success) { - logger.error("Failed to write batch to Vortex file"); - throw new IOException("Failed to write batch to Vortex file"); - } - } - - /** - * Writes a batch of Arrow data directly from Arrow C Data Interface pointers. - * - * @param arrowArrayAddr memory address of the ArrowArray struct - * @param arrowSchemaAddr memory address of the ArrowSchema struct - * @throws IOException if writing fails - */ - @Override - public void writeBatchFfi(long arrowArrayAddr, long arrowSchemaAddr) throws IOException { - logger.trace("Writing batch via FFI (arrayAddr={}, schemaAddr={})", arrowArrayAddr, arrowSchemaAddr); - - boolean success = NativeWriterMethods.writeBatchFfi(ptr.getAsLong(), arrowArrayAddr, arrowSchemaAddr); - if (!success) { - logger.error("Failed to write FFI batch to Vortex file"); - throw new IOException("Failed to write FFI batch to Vortex file"); - } - } - - /** - * Closes the writer and finalizes the Vortex file. - * - * @throws RuntimeException if closing fails - */ - @Override - public void close() { - if (this.ptr.isEmpty()) { - logger.debug("Attempted to close already closed JNIWriter, skipping"); - return; - } - - long ptr = this.ptr.getAsLong(); - - logger.debug("Closing JNIWriter with ptr={}", ptr); - NativeWriterMethods.close(ptr); - this.ptr = OptionalLong.empty(); - } -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/jni/NativeArrayIteratorMethods.java b/java/vortex-jni/src/main/java/dev/vortex/jni/NativeArrayIteratorMethods.java deleted file mode 100644 index 9fb2a366b75..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/jni/NativeArrayIteratorMethods.java +++ /dev/null @@ -1,30 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.jni; - -public final class NativeArrayIteratorMethods { - static { - NativeLoader.loadJni(); - } - - private NativeArrayIteratorMethods() {} - - /** - * Free all resources associated with the stream behind the pointer. - */ - public static native void free(long pointer); - - /** - * Returns a pointer to the next element in the stream, or -1 if there are no more elements. - *

- * An exception is thrown if the stream is closed, either via free or due to a previous call - * to this method returning -1. - */ - public static native long take(long pointer); - - /** - * Get a pointer to the DType of the elements of the stream. - */ - public static native long getDType(long pointer); -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/jni/NativeArrayMethods.java b/java/vortex-jni/src/main/java/dev/vortex/jni/NativeArrayMethods.java deleted file mode 100644 index d6f19c2ea83..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/jni/NativeArrayMethods.java +++ /dev/null @@ -1,65 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.jni; - -import java.math.BigDecimal; - -public final class NativeArrayMethods { - static { - NativeLoader.loadJni(); - } - - private NativeArrayMethods() {} - - public static native long nbytes(long pointer); - - public static native void exportToArrow(long pointer, long[] schemaPointer, long[] arrayPointer); - - public static native void dropArrowSchema(long arrowSchemaPtr); - - public static native void dropArrowArray(long arrowArrayPtr); - - public static native void free(long pointer); - - public static native long getLen(long pointer); - - public static native long getDataType(long pointer); - - public static native long getField(long pointer, int index); - - public static native long slice(long pointer, int start, int stop); - - public static native boolean getNull(long pointer, int index); - - public static native int getNullCount(long pointer); - - public static native byte getByte(long pointer, int index); - - public static native short getShort(long pointer, int index); - - public static native int getInt(long pointer, int index); - - public static native long getLong(long pointer, int index); - - public static native boolean getBool(long pointer, int index); - - public static native float getFloat(long pointer, int index); - - public static native double getDouble(long pointer, int index); - - public static native BigDecimal getBigDecimal(long pointer, int index); - - public static native String getUTF8(long pointer, int index); - - /** - * Raw-pointer variant of {@link #getUTF8(long, int)} that accepts an array to hold - * a pointer and an output length. - *

- * For Java query engines that use Unsafe to manipulate native memory, this allows working with the string - * inside of the JVM without copying it into Java heap memory. - */ - public static native void getUTF8_ptr_len(long pointer, int index, long[] outPtr, int[] outLen); - - public static native byte[] getBinary(long pointer, int index); -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/jni/NativeDTypeMethods.java b/java/vortex-jni/src/main/java/dev/vortex/jni/NativeDTypeMethods.java deleted file mode 100644 index a59fc9f95d8..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/jni/NativeDTypeMethods.java +++ /dev/null @@ -1,184 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.jni; - -import java.util.List; - -public final class NativeDTypeMethods { - static { - NativeLoader.loadJni(); - } - - private NativeDTypeMethods() {} - - /** - * Create a new native DType for a PType::I8. The created object lives in native memory. - * - * @param isNullable true if the values can be null - * @return Pointer to a new heap-allocated {@code DType}. - */ - public static native long newByte(boolean isNullable); - - /** - * Create a new native DType for a PType::I16. The created object lives in native memory. - * - * @param isNullable true if the values can be null - * @return Pointer to a new heap-allocated {@code DType}. - */ - public static native long newShort(boolean isNullable); - - /** - * Create a new native DType for a PType::I32. The created object lives in native memory. - * - * @param isNullable true if the values can be null - * @return Pointer to a new heap-allocated {@code DType}. - */ - public static native long newInt(boolean isNullable); - - /** - * Create a new native DType for a PType::I64. The created object lives in native memory. - * - * @param isNullable true if the values can be null - * @return Pointer to a new heap-allocated {@code DType}. - */ - public static native long newLong(boolean isNullable); - - /** - * Create a new native DType for a PType::F32. The created object lives in native memory. - * - * @param isNullable true if the values can be null - * @return Pointer to a new heap-allocated {@code DType}. - */ - public static native long newFloat(boolean isNullable); - - /** - * Create a new native DType for a PType::F64. The created object lives in native memory. - * - * @param isNullable true if the values can be null - * @return Pointer to a new heap-allocated {@code DType}. - */ - public static native long newDouble(boolean isNullable); - - /** - * Create a new native DType for a decimal type. The created object lives in native memory. - * - * @param isNullable true if the values can be null - * @param precision decimal precision - * @param scale decimal scale - * @return Pointer to a new heap-allocated {@code DType}. - */ - public static native long newDecimal(int precision, int scale, boolean isNullable); - - /** - * Create a new native DType for a UTF-8 string type. The created object lives in native memory. - * - * @param isNullable true if the values can be null - * @return Pointer to a new heap-allocated {@code DType}. - */ - public static native long newUtf8(boolean isNullable); - - /** - * Create a new native DType for a Binary type. The created object lives in native memory. - * - * @param isNullable true if the values can be null - * @return Pointer to a new heap-allocated {@code DType}. - */ - public static native long newBinary(boolean isNullable); - - /** - * Create a new native DType for a boolean type. The created object lives in native memory. - * - * @param isNullable true if the values can be null - * @return Pointer to a new heap-allocated {@code DType}. - */ - public static native long newBool(boolean isNullable); - - /** - * Create a new native DType for a List type. The created object lives in native memory. - * - * @param elementTypePtr A native pointer to a DType containing the type of the elements - * @param isNullable true if the values can be null - * @return Pointer to a new heap-allocated {@code DType}. - */ - public static native long newList(long elementTypePtr, boolean isNullable); - - /** - * Create a new native DType for a FixedSizeList type. The created object lives in native memory. - * - * @param elementTypePtr A native pointer to a DType containing the type of the elements - * @param size The fixed size of each list - * @param isNullable true if the values can be null - * @return Pointer to a new heap-allocated {@code DType}. - */ - public static native long newFixedSizeList(long elementTypePtr, int size, boolean isNullable); - - /** - * Create a new native DType for a Struct type. The created object lives in native memory. - * - * @param fieldNames An array of field names - * @param fieldTypes An array of native pointers to the DType for each field - * @param isNullable true if the values can be null - * @return Pointer to a new heap-allocated {@code DType}. - */ - public static native long newStruct(String[] fieldNames, long[] fieldTypes, boolean isNullable); - - /** - * Create a new native DType for a List type. The created object lives in native memory. - * - * @param timeUnit A byte that represents a {@link dev.vortex.api.DType.TimeUnit} - * @param zone The time zone or offset string - * @param isNullable true if the values can be null - * @return Pointer to a new heap-allocated {@code DType}. - */ - public static native long newTimestamp(byte timeUnit, String zone, boolean isNullable); - - /** - * Create a new native DType for a List type. The created object lives in native memory. - * - * @param timeUnit A byte that represents a {@link dev.vortex.api.DType.TimeUnit} - * @param isNullable true if the values can be null - * @return Pointer to a new heap-allocated {@code DType}. - */ - public static native long newDate(byte timeUnit, boolean isNullable); - - /** - * Create a new native DType for a List type. The created object lives in native memory. - * - * @param timeUnit A byte that represents a {@link dev.vortex.api.DType.TimeUnit} - * @param isNullable true if the values can be null - * @return Pointer to a new heap-allocated {@code DType}. - */ - public static native long newTime(byte timeUnit, boolean isNullable); - - public static native void free(long pointer); - - public static native byte getVariant(long pointer); - - public static native boolean isNullable(long pointer); - - public static native List getFieldNames(long pointer); - - // Returns a list of DType pointers. - public static native List getFieldTypes(long pointer); - - public static native long getElementType(long pointer); - - public static native int getFixedSizeListSize(long pointer); - - public static native boolean isDate(long pointer); - - public static native boolean isTime(long pointer); - - public static native boolean isTimestamp(long pointer); - - public static native byte getTimeUnit(long pointer); - - public static native String getTimeZone(long pointer); - - public static native boolean isDecimal(long pointer); - - public static native int getDecimalPrecision(long pointer); - - public static native byte getDecimalScale(long pointer); -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/jni/NativeDataSource.java b/java/vortex-jni/src/main/java/dev/vortex/jni/NativeDataSource.java new file mode 100644 index 00000000000..b7e58d2dc21 --- /dev/null +++ b/java/vortex-jni/src/main/java/dev/vortex/jni/NativeDataSource.java @@ -0,0 +1,36 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +package dev.vortex.jni; + +import java.util.Map; + +/** JNI boundary for {@link dev.vortex.api.DataSource}. */ +public final class NativeDataSource { + static { + NativeLoader.loadJni(); + } + + private NativeDataSource() {} + + /** + * Open a data source from one or more URIs or globs. + * + * @param sessionPointer pointer from {@link NativeSession#newSession()} + * @param uris paths or globs (for example {@code ["file:///a.vortex", "file:///b.vortex"]}) + * @param options object-store properties (may be null) + */ + public static native long open(long sessionPointer, String[] uris, Map options); + + /** Free a data source pointer. */ + public static native void free(long pointer); + + /** Export the data source's schema into the Arrow C Data Interface struct at {@code schemaAddress}. */ + public static native void arrowSchema(long pointer, long schemaAddress); + + /** + * Populate {@code out} with {@code [rows, cardinality]}. Cardinality is one of {@code 0=unknown}, + * {@code 1=estimate}, {@code 2=exact}. + */ + public static native void rowCount(long pointer, long[] out); +} diff --git a/java/vortex-jni/src/main/java/dev/vortex/jni/NativeExpression.java b/java/vortex-jni/src/main/java/dev/vortex/jni/NativeExpression.java new file mode 100644 index 00000000000..fc5bedbe7b5 --- /dev/null +++ b/java/vortex-jni/src/main/java/dev/vortex/jni/NativeExpression.java @@ -0,0 +1,47 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +package dev.vortex.jni; + +/** JNI boundary for {@link dev.vortex.api.Expression}. */ +public final class NativeExpression { + static { + NativeLoader.loadJni(); + } + + private NativeExpression() {} + + public static native long root(); + + public static native long getItem(String fieldName, long childPointer); + + public static native long select(String[] fieldNames, long childPointer); + + public static native long and(long[] operandPointers); + + public static native long or(long[] operandPointers); + + public static native long binary(byte operator, long lhs, long rhs); + + public static native long not(long childPointer); + + public static native long isNull(long childPointer); + + public static native long literalBool(boolean value, boolean isNull); + + public static native long literalI8(byte value, boolean isNull); + + public static native long literalI16(short value, boolean isNull); + + public static native long literalI32(int value, boolean isNull); + + public static native long literalI64(long value, boolean isNull); + + public static native long literalF32(float value, boolean isNull); + + public static native long literalF64(double value, boolean isNull); + + public static native long literalString(String value); + + public static native void free(long pointer); +} diff --git a/java/vortex-jni/src/main/java/dev/vortex/jni/NativeFileMethods.java b/java/vortex-jni/src/main/java/dev/vortex/jni/NativeFileMethods.java deleted file mode 100644 index 78f56255fa1..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/jni/NativeFileMethods.java +++ /dev/null @@ -1,68 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.jni; - -import java.util.List; -import java.util.Map; - -public final class NativeFileMethods { - static { - NativeLoader.loadJni(); - } - - private NativeFileMethods() {} - - /** - * List all Vortex files underneath the current file path. - * - * @param uri The URI describing both the object store and the path to the file - * @param options Any options required to initialize the object store client - * @return A list of URIs for the Vortex files below the provided path - */ - public static native List listVortexFiles(String uri, Map options); - - /** - * Delete the files at the provided URIs. Use the options to configure an object store client. - */ - public static native void delete(String[] uris, Map options); - - /** - * Open a file using the native library with the provided URI and options. - * - * @param uri The URI of the file to open. e.g. "file://path/to/file". - * @param options A map of options to provide for opening the file. - * @return A native pointer to the opened file. This will be 0 if the open call failed. - */ - public static native long open(String uri, Map options); - - /** - * Get the total row count contained in the file associated with the given pointer. - * - * @param pointer The native pointer to a file. Must be a value returned by {@link #open(String, Map)}. - * @return The number of rows of data encoded in the file. This includes null values. - */ - public static native long rowCount(long pointer); - - /** - * Get the data type of the file associated with the given pointer. - * - * @param pointer The native pointer to a file. Must be a value returned by {@link #open(String, Map)}. - * @return Native pointer to the DType of the file. This pointer is owned by the file and should not be freed. - */ - public static native long dtype(long pointer); - - /** - * Close the file associated with the given pointer. - * - * @param pointer The native pointer to a file. Must be a value returned by {@link #open(String, Map)}. - */ - public static native void close(long pointer); - - /** - * Build a new native scan operator that will materialize Arrays from the file, pushing down the optional - * predicate, row range or row indices to perform data skipping. - */ - public static native long scan( - long pointer, List columns, byte[] predicateProto, long[] rowRange, long[] rowIndices); -} diff --git a/java/vortex-jni/src/main/java/dev/vortex/jni/NativeFiles.java b/java/vortex-jni/src/main/java/dev/vortex/jni/NativeFiles.java new file mode 100644 index 00000000000..fae03d50186 --- /dev/null +++ b/java/vortex-jni/src/main/java/dev/vortex/jni/NativeFiles.java @@ -0,0 +1,34 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +package dev.vortex.jni; + +import dev.vortex.api.Session; +import java.util.List; +import java.util.Map; + +/** + * Static utilities for discovering and deleting Vortex files on an object store. The caller supplies a {@link Session}; + * its runtime handle is forwarded to the underlying object store. + */ +public final class NativeFiles { + static { + NativeLoader.loadJni(); + } + + private NativeFiles() {} + + /** List all Vortex files reachable under {@code uri}. */ + public static List listFiles(Session session, String uri, Map options) { + return listFiles(session.nativePointer(), uri, options); + } + + /** Delete files at the given URIs. Silently tolerates an empty list. */ + public static void delete(Session session, String[] uris, Map options) { + delete(session.nativePointer(), uris, options); + } + + private static native List listFiles(long sessionPointer, String uri, Map options); + + private static native void delete(long sessionPointer, String[] uris, Map options); +} diff --git a/java/vortex-jni/src/main/java/dev/vortex/jni/NativeLoader.java b/java/vortex-jni/src/main/java/dev/vortex/jni/NativeLoader.java index 415ae6a9bbb..99052c5bf9a 100644 --- a/java/vortex-jni/src/main/java/dev/vortex/jni/NativeLoader.java +++ b/java/vortex-jni/src/main/java/dev/vortex/jni/NativeLoader.java @@ -4,50 +4,26 @@ package dev.vortex.jni; import com.google.common.io.ByteStreams; -import java.io.*; +import java.io.File; +import java.io.FileNotFoundException; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; import java.util.Locale; -/** - * Utility class for loading the native Vortex JNI library. - *

- * This class handles the platform-specific loading of the native Vortex library - * by detecting the operating system and architecture, extracting the appropriate - * native library from the classpath, and loading it into the JVM. - *

- *

- * The loader supports Windows, macOS, and Linux platforms with automatic - * detection of the correct library file format (.dll, .dylib, or .so). - *

- */ +/** Loads the native vortex-jni shared library from the classpath. */ public final class NativeLoader { private static boolean loaded = false; private NativeLoader() {} - /** - * Loads the native Vortex JNI library if it hasn't been loaded already. - *

- * This method performs platform detection, extracts the appropriate native - * library from the classpath to a temporary file, and loads it using - * {@link System#load(String)}. The method is thread-safe and will only - * perform the loading operation once per JVM session. - *

- *

- * The native library is expected to be located at: - * {@code /native/{platform}-{arch}/libvortex_jni.{ext}} - * where platform is one of: win, darwin, linux and ext is the appropriate - * library extension for the platform. - *

- * - * @throws UnsupportedOperationException if the current platform is not supported - * @throws RuntimeException if the library cannot be extracted or loaded - */ + /** Load the native library into the current JVM. Thread-safe and idempotent. */ public static synchronized void loadJni() { if (loaded) { return; } - // Load the native library String osName = System.getProperty("os.name").toLowerCase(Locale.ROOT); String osArch = System.getProperty("os.arch").toLowerCase(Locale.ROOT); String libName = "libvortex_jni"; @@ -70,8 +46,6 @@ public static synchronized void loadJni() { throw new UnsupportedOperationException("Unsupported OS: " + osName); } - // Extract the library from classpath - // This assumes the library is in the same package as this class String libPath = "/native/" + osShortName + "-" + osArch + "/" + libName; try (InputStream in = NativeLoader.class.getResourceAsStream(libPath)) { if (in == null) { @@ -88,7 +62,6 @@ public static synchronized void loadJni() { throw new RuntimeException("Failed to load library: " + e.getMessage(), e); } - // Load the library System.load(libName); loaded = true; } diff --git a/java/vortex-jni/src/main/java/dev/vortex/jni/NativeLogging.java b/java/vortex-jni/src/main/java/dev/vortex/jni/NativeLogging.java index 519c64ca3eb..aae15168d86 100644 --- a/java/vortex-jni/src/main/java/dev/vortex/jni/NativeLogging.java +++ b/java/vortex-jni/src/main/java/dev/vortex/jni/NativeLogging.java @@ -5,12 +5,10 @@ /** * Utility class for configuring native logging levels in the Vortex JNI layer. - *

- * This class provides constants for different logging levels and methods to - * initialize native logging to the desired verbosity level. The logging levels - * correspond to standard logging frameworks with ERROR being the least verbose - * and TRACE being the most verbose. - *

+ * + *

This class provides constants for different logging levels and methods to initialize native logging to the desired + * verbosity level. The logging levels correspond to standard logging frameworks with ERROR being the least verbose and + * TRACE being the most verbose. */ public final class NativeLogging { static { @@ -36,12 +34,13 @@ private NativeLogging() {} /** * Initialize logging to the desired level. Must be one of: + * *

    - *
  • {@link #ERROR}
  • - *
  • {@link #WARN}
  • - *
  • {@link #INFO}
  • - *
  • {@link #DEBUG}
  • - *
  • {@link #TRACE}
  • + *
  • {@link #ERROR} + *
  • {@link #WARN} + *
  • {@link #INFO} + *
  • {@link #DEBUG} + *
  • {@link #TRACE} *
*/ public static native void initLogging(int level); diff --git a/java/vortex-jni/src/main/java/dev/vortex/jni/NativePartition.java b/java/vortex-jni/src/main/java/dev/vortex/jni/NativePartition.java new file mode 100644 index 00000000000..0ddebc3448d --- /dev/null +++ b/java/vortex-jni/src/main/java/dev/vortex/jni/NativePartition.java @@ -0,0 +1,29 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +package dev.vortex.jni; + +/** JNI boundary for {@link dev.vortex.api.Partition}. */ +public final class NativePartition { + static { + NativeLoader.loadJni(); + } + + private NativePartition() {} + + /** Free a partition pointer that was not consumed by {@link #scanArrow}. */ + public static native void free(long pointer); + + /** Fill {@code out} with {@code [rows, cardinality]}. */ + public static native void rowCount(long pointer, long[] out); + + /** + * Consume the partition into the {@code FFI_ArrowArrayStream} at {@code streamAddress}. The partition pointer is + * invalidated by this call. + * + * @param sessionPointer native session pointer used for execution context + * @param partitionPointer partition pointer to consume + * @param streamAddress address of an allocated {@code FFI_ArrowArrayStream} struct + */ + public static native void scanArrow(long sessionPointer, long partitionPointer, long streamAddress); +} diff --git a/java/vortex-jni/src/main/java/dev/vortex/jni/NativeRuntime.java b/java/vortex-jni/src/main/java/dev/vortex/jni/NativeRuntime.java new file mode 100644 index 00000000000..9a0e116668e --- /dev/null +++ b/java/vortex-jni/src/main/java/dev/vortex/jni/NativeRuntime.java @@ -0,0 +1,32 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +package dev.vortex.jni; + +/** + * Controls for the JVM-wide current-thread worker pool that backs every Vortex session. + * + *

By default the pool has zero background threads: nothing happens until a Java thread calls a blocking Vortex API. + * Adding workers via {@link #setWorkerThreads(int)} lets the pool drive Vortex futures on behalf of the caller — useful + * when a single consumer thread cannot keep the executor busy on its own. Spawning Java threads that each poll a + * thread-safe Vortex iterator is an equivalent alternative. + */ +public final class NativeRuntime { + static { + NativeLoader.loadJni(); + } + + private NativeRuntime() {} + + /** + * Set the number of background worker threads driving Vortex futures. {@code 0} disables background execution; any + * positive value adjusts the pool up or down. + */ + public static native void setWorkerThreads(int n); + + /** Size the pool to {@code Runtime.getRuntime().availableProcessors() - 1}. */ + public static native void setWorkerThreadsToAvailableParallelism(); + + /** Returns the current background worker-thread count. */ + public static native int workerCount(); +} diff --git a/java/vortex-jni/src/main/java/dev/vortex/jni/NativeScan.java b/java/vortex-jni/src/main/java/dev/vortex/jni/NativeScan.java new file mode 100644 index 00000000000..fe1a2e7a674 --- /dev/null +++ b/java/vortex-jni/src/main/java/dev/vortex/jni/NativeScan.java @@ -0,0 +1,51 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +package dev.vortex.jni; + +/** JNI boundary for {@link dev.vortex.api.Scan}. */ +public final class NativeScan { + static { + NativeLoader.loadJni(); + } + + private NativeScan() {} + + /** + * Create a new scan from a data source. The scan is lazy: no I/O happens until {@link #nextPartition(long)} is + * called. + * + * @param dataSourcePointer pointer from {@link NativeDataSource#open} + * @param projectionPointer native expression pointer, or 0 for "all columns" + * @param filterPointer native filter expression pointer, or 0 for "no filter" + * @param rowRangeBegin inclusive start of the row range, 0 for "unbounded" + * @param rowRangeEnd exclusive end of the row range, 0 for "unbounded" + * @param selectionIndices sorted row indices; may be null + * @param selectionInclude {@code 0} (all), {@code 1} (include {@code selectionIndices}), {@code 2} (exclude + * {@code selectionIndices}) + * @param limit max rows to return, or {@code 0} for "no limit" + * @param ordered true to preserve row order across partitions + */ + public static native long create( + long dataSourcePointer, + long projectionPointer, + long filterPointer, + long rowRangeBegin, + long rowRangeEnd, + long[] selectionIndices, + byte selectionInclude, + long limit, + boolean ordered); + + /** Free a scan pointer. */ + public static native void free(long pointer); + + /** Export the scan's schema into the Arrow C Data Interface struct at {@code schemaAddress}. */ + public static native void arrowSchema(long pointer, long schemaAddress); + + /** Fill {@code out} with {@code [count, cardinality]}. */ + public static native void partitionCount(long pointer, long[] out); + + /** Advance the scan and return the next partition pointer. Returns {@code 0} when exhausted. */ + public static native long nextPartition(long pointer); +} diff --git a/java/vortex-jni/src/main/java/dev/vortex/jni/NativeSession.java b/java/vortex-jni/src/main/java/dev/vortex/jni/NativeSession.java new file mode 100644 index 00000000000..6ebdcacf237 --- /dev/null +++ b/java/vortex-jni/src/main/java/dev/vortex/jni/NativeSession.java @@ -0,0 +1,19 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +package dev.vortex.jni; + +/** JNI boundary for {@link dev.vortex.api.Session}. */ +public final class NativeSession { + static { + NativeLoader.loadJni(); + } + + private NativeSession() {} + + /** Allocate a fresh native session. Free with {@link #free(long)}. */ + public static native long newSession(); + + /** Free a session previously returned by {@link #newSession()}. */ + public static native void free(long pointer); +} diff --git a/java/vortex-jni/src/main/java/dev/vortex/jni/NativeWriter.java b/java/vortex-jni/src/main/java/dev/vortex/jni/NativeWriter.java new file mode 100644 index 00000000000..d1f2e725f8c --- /dev/null +++ b/java/vortex-jni/src/main/java/dev/vortex/jni/NativeWriter.java @@ -0,0 +1,32 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +package dev.vortex.jni; + +import java.util.Map; + +/** JNI boundary for {@link dev.vortex.api.VortexWriter}. */ +public final class NativeWriter { + static { + NativeLoader.loadJni(); + } + + private NativeWriter() {} + + /** Open a writer at {@code uri} that accepts batches matching the Arrow schema at {@code arrowSchemaAddress}. */ + public static native long create( + long sessionPointer, String uri, long arrowSchemaAddress, Map options); + + /** + * Write a batch directly from Arrow C Data Interface addresses. + * + * @param writerPointer pointer from {@link #create} + * @param arrowArrayAddress address of an {@code ArrowArray} struct + * @param arrowSchemaAddress address of an {@code ArrowSchema} struct + * @return {@code true} on success + */ + public static native boolean writeBatch(long writerPointer, long arrowArrayAddress, long arrowSchemaAddress); + + /** Flush and close the writer. Must be called exactly once. */ + public static native void close(long writerPointer); +} diff --git a/java/vortex-jni/src/main/java/dev/vortex/jni/NativeWriterMethods.java b/java/vortex-jni/src/main/java/dev/vortex/jni/NativeWriterMethods.java deleted file mode 100644 index 8ddc7ce0bb1..00000000000 --- a/java/vortex-jni/src/main/java/dev/vortex/jni/NativeWriterMethods.java +++ /dev/null @@ -1,55 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.jni; - -import java.util.Map; - -/** - * Native JNI methods for writing Vortex files. - */ -public final class NativeWriterMethods { - - static { - NativeLoader.loadJni(); - } - - private NativeWriterMethods() {} - - /** - * Creates a new native Vortex writer. - * - * @param uri the URI to the file, e.g. "file://path/to/file". - * @param dtype native pointer to a writer schema (Vortex DType) - * @param options additional writer options. For cloud storage this includes things like credentials. - * @return a native pointer to the writer, or 0 on failure - */ - public static native long create(String uri, long dtype, Map options); - - /** - * Writes a batch of Arrow data to the Vortex file. - * - * @param writerPtr the native writer pointer - * @param arrowData the Arrow IPC format data - * @return true if successful, false otherwise - */ - public static native boolean writeBatch(long writerPtr, byte[] arrowData); - - /** - * Writes a batch of Arrow data to the Vortex file directly from Arrow C Data Interface pointers. - * - * @param writerPtr the native writer pointer - * @param arrowArrayAddr memory address of the ArrowArray struct - * @param arrowSchemaAddr memory address of the ArrowSchema struct - * @return true if successful, false otherwise - */ - public static native boolean writeBatchFfi(long writerPtr, long arrowArrayAddr, long arrowSchemaAddr); - - /** - * Close and flush the writer, finalizing it to the storage system. - * - * @param writerPtr the native writer pointer - * @throws RuntimeException if the writer fails to close - */ - public static native void close(long writerPtr); -} diff --git a/java/vortex-jni/src/test/java/dev/vortex/api/DTypeTest.java b/java/vortex-jni/src/test/java/dev/vortex/api/DTypeTest.java deleted file mode 100644 index 6946812e1b0..00000000000 --- a/java/vortex-jni/src/test/java/dev/vortex/api/DTypeTest.java +++ /dev/null @@ -1,73 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.api; - -import static org.junit.jupiter.api.Assertions.*; - -import org.junit.jupiter.api.Test; - -public final class DTypeTest { - - @Test - public void testNewFixedSizeListNonNullable() { - var elementType = DType.newInt(false); - var fslType = DType.newFixedSizeList(elementType, 3, false); - assertEquals(DType.Variant.FIXED_SIZE_LIST, fslType.getVariant()); - assertFalse(fslType.isNullable()); - assertEquals(3, fslType.getFixedSizeListSize()); - - var innerType = fslType.getElementType(); - assertEquals(DType.Variant.PRIMITIVE_I32, innerType.getVariant()); - } - - @Test - public void testNewFixedSizeListNullable() { - var elementType = DType.newUtf8(true); - var fslType = DType.newFixedSizeList(elementType, 5, true); - assertEquals(DType.Variant.FIXED_SIZE_LIST, fslType.getVariant()); - assertTrue(fslType.isNullable()); - assertEquals(5, fslType.getFixedSizeListSize()); - - var innerType = fslType.getElementType(); - assertEquals(DType.Variant.UTF8, innerType.getVariant()); - } - - @Test - public void testNewListGetElementType() { - var elementType = DType.newDouble(false); - var listType = DType.newList(elementType, false); - assertEquals(DType.Variant.LIST, listType.getVariant()); - - var innerType = listType.getElementType(); - assertEquals(DType.Variant.PRIMITIVE_F64, innerType.getVariant()); - } - - @Test - public void testNestedFixedSizeList() { - var innerElement = DType.newLong(false); - var innerFsl = DType.newFixedSizeList(innerElement, 2, false); - var outerFsl = DType.newFixedSizeList(innerFsl, 4, true); - assertEquals(DType.Variant.FIXED_SIZE_LIST, outerFsl.getVariant()); - assertTrue(outerFsl.isNullable()); - assertEquals(4, outerFsl.getFixedSizeListSize()); - - var inner = outerFsl.getElementType(); - assertEquals(DType.Variant.FIXED_SIZE_LIST, inner.getVariant()); - } - - @Test - public void testFixedSizeListInStruct() { - var elementType = DType.newFloat(false); - var fslType = DType.newFixedSizeList(elementType, 3, false); - var structType = - DType.newStruct(new String[] {"id", "embedding"}, new DType[] {DType.newInt(false), fslType}, false); - assertEquals(DType.Variant.STRUCT, structType.getVariant()); - - var fieldTypes = structType.getFieldTypes(); - assertEquals(2, fieldTypes.size()); - - var embeddingType = fieldTypes.get(1); - assertEquals(DType.Variant.FIXED_SIZE_LIST, embeddingType.getVariant()); - } -} diff --git a/java/vortex-jni/src/test/java/dev/vortex/api/TestMinimal.java b/java/vortex-jni/src/test/java/dev/vortex/api/TestMinimal.java index abcdd8cf129..b422b767dd3 100644 --- a/java/vortex-jni/src/test/java/dev/vortex/api/TestMinimal.java +++ b/java/vortex-jni/src/test/java/dev/vortex/api/TestMinimal.java @@ -3,21 +3,36 @@ package dev.vortex.api; +import static java.nio.charset.StandardCharsets.UTF_8; import static org.junit.jupiter.api.Assertions.assertEquals; -import dev.vortex.api.expressions.Binary; -import dev.vortex.api.expressions.GetItem; -import dev.vortex.api.expressions.Literal; -import dev.vortex.api.expressions.Root; +import dev.vortex.arrow.ArrowAllocation; +import dev.vortex.jni.NativeLoader; +import java.io.IOException; import java.math.BigDecimal; +import java.nio.file.Path; import java.util.ArrayList; +import java.util.HashMap; import java.util.List; -import java.util.Map; import java.util.Objects; +import org.apache.arrow.c.ArrowArray; +import org.apache.arrow.c.ArrowSchema; +import org.apache.arrow.c.Data; +import org.apache.arrow.memory.BufferAllocator; +import org.apache.arrow.vector.DecimalVector; +import org.apache.arrow.vector.FieldVector; +import org.apache.arrow.vector.VarCharVector; +import org.apache.arrow.vector.VectorSchemaRoot; +import org.apache.arrow.vector.ViewVarCharVector; +import org.apache.arrow.vector.ipc.ArrowReader; +import org.apache.arrow.vector.types.pojo.ArrowType; +import org.apache.arrow.vector.types.pojo.Field; +import org.apache.arrow.vector.types.pojo.Schema; +import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; public final class TestMinimal { - // POJO representing the person data type. static final class Person { public String name; public BigDecimal salary; @@ -29,16 +44,9 @@ public Person(String name, BigDecimal salary, String state) { this.state = state; } - public Person(Map map) { - this.name = (String) map.get("Name"); - this.salary = (BigDecimal) map.get("Salary"); - this.state = (String) map.get("State"); - } - @Override public boolean equals(Object o) { - if (!(o instanceof Person)) return false; - Person person = (Person) o; + if (!(o instanceof Person person)) return false; return Objects.equals(name, person.name) && Objects.equals(salary, person.salary) && Objects.equals(state, person.state); @@ -51,28 +59,58 @@ public int hashCode() { @Override public String toString() { - return "Person{" + "name='" + name + '\'' + ", salary=" + salary + ", state='" + state + '\'' + '}'; + return "Person{name='" + name + "', salary=" + salary + ", state='" + state + "'}"; + } + } + + @TempDir + static Path tempDir; + + static String writePath; + + @BeforeAll + public static void loadLibrary() { + NativeLoader.loadJni(); + } + + @BeforeAll + static void setup() throws IOException { + Path outputPath = tempDir.resolve("minimal.vortex"); + writePath = outputPath.toAbsolutePath().toUri().toString(); + + BufferAllocator allocator = ArrowAllocation.rootAllocator(); + Schema schema = new Schema(List.of( + Field.notNullable("Name", ArrowType.Utf8View.INSTANCE), + Field.notNullable("Salary", ArrowType.Decimal.createDecimal(9, 2, 128)), + Field.nullable("State", ArrowType.Utf8View.INSTANCE))); + Session session = Session.create(); + try (VortexWriter writer = VortexWriter.create(session, writePath, schema, new HashMap<>(), allocator); + VectorSchemaRoot root = VectorSchemaRoot.create(schema, allocator)) { + ViewVarCharVector nameVec = (ViewVarCharVector) root.getVector("Name"); + DecimalVector salaryVec = (DecimalVector) root.getVector("Salary"); + ViewVarCharVector stateVec = (ViewVarCharVector) root.getVector("State"); + + nameVec.allocateNew(10); + salaryVec.allocateNew(10); + stateVec.allocateNew(10); + + for (int i = 0; i < 10; i++) { + var person = MINIMAL_DATA.get(i); + nameVec.setSafe(i, person.name.getBytes(UTF_8)); + salaryVec.setSafe(i, 1_000 * (i + 1)); + stateVec.setSafe(i, person.state.getBytes(UTF_8)); + } + + root.setRowCount(10); + + try (ArrowArray arrowArray = ArrowArray.allocateNew(allocator); + ArrowSchema arrowSchemaFfi = ArrowSchema.allocateNew(allocator)) { + Data.exportVectorSchemaRoot(allocator, root, null, arrowArray, arrowSchemaFfi); + writer.writeBatch(arrowArray.memoryAddress(), arrowSchemaFfi.memoryAddress()); + } } } - private static final String MINIMAL_PATH = - TestMinimal.class.getResource("/minimal.vortex").getPath(); - - // data representing the complete `minimal` test table: - /// ======================= - /// Name | Salary | State - /// ======================= - /// Alice 1000 CA - /// Bob 2000 NY - /// Carol 3000 TX - /// Dan 4000 CA - /// Edward 5000 NY - /// Frida 6000 TX - /// George 7000 CA - /// Henry 8000 NY - /// Ida 9000 TX - /// John 10000 VA - /// ======================= private static final List MINIMAL_DATA = List.of( new Person("Alice", BigDecimal.valueOf(1000L, 2), "CA"), new Person("Bob", BigDecimal.valueOf(2000L, 2), "NY"), @@ -85,128 +123,102 @@ public String toString() { new Person("Ida", BigDecimal.valueOf(9000L, 2), "TX"), new Person("John", BigDecimal.valueOf(10_000L, 2), "VA")); - private static final List PROJECTED_DATA = List.of( - new Person("Alice", null, "CA"), - new Person("Bob", null, "NY"), - new Person("Carol", null, "TX"), - new Person("Dan", null, "CA"), - new Person("Edward", null, "NY"), - new Person("Frida", null, "TX"), - new Person("George", null, "CA"), - new Person("Henry", null, "NY"), - new Person("Ida", null, "TX"), - new Person("John", null, "VA")); - @Test - public void testFullScan() { - try (var file = Files.open(MINIMAL_PATH); - var fullScan = file.newScan(ScanOptions.of())) { - assertEquals(10, file.rowCount()); - - var dtype = fullScan.getDataType(); - assertEquals(DType.Variant.STRUCT, dtype.getVariant()); - assertEquals(dtype.getFieldNames(), List.of("Name", "Salary", "State")); - - // Perform a full scan, check the result. - var people = readToList(fullScan, new TestCase() { - @Override - public Array[] open(Array batch) { - return new Array[] { - batch.getField(0), batch.getField(1), batch.getField(2), - }; - } + public void testFullScan() throws Exception { + BufferAllocator allocator = ArrowAllocation.rootAllocator(); + Session session = Session.create(); + DataSource ds = DataSource.open(session, writePath); - @Override - public Person readRow(Array[] fields, int idx) { - return new Person(fields[0].getUTF8(idx), fields[1].getBigDecimal(idx), fields[2].getUTF8(idx)); - } - }); + assertEquals(new DataSource.RowCount.Exact(10L), ds.rowCount()); - assertEquals(MINIMAL_DATA, people); - } + var schema = ds.arrowSchema(allocator); + assertEquals( + List.of("Name", "Salary", "State"), + schema.getFields().stream().map(Field::getName).toList()); + + List people = readAll(ds, ScanOptions.of(), allocator, TestMinimal::readFullBatch); + assertEquals(MINIMAL_DATA, people); } @Test - public void testProjectedScan() { - var projectOptions = ScanOptions.builder().addColumns("Name", "State").build(); - - try (var file = Files.open(MINIMAL_PATH); - var projectedScan = file.newScan(projectOptions)) { - // Do stuff. - var dtype = projectedScan.getDataType(); - assertEquals(DType.Variant.STRUCT, dtype.getVariant()); - assertEquals(dtype.getFieldNames(), List.of("Name", "State")); - - var people = readToList(projectedScan, new TestCase() { - @Override - public Array[] open(Array batch) { - return new Array[] { - batch.getField(0), batch.getField(1), - }; - } - - @Override - public Person readRow(Array[] fields, int idx) { - return new Person(fields[0].getUTF8(idx), null, fields[1].getUTF8(idx)); - } - }); - - assertEquals(PROJECTED_DATA, people); + public void testProjectedScan() throws Exception { + BufferAllocator allocator = ArrowAllocation.rootAllocator(); + Session session = Session.create(); + DataSource ds = DataSource.open(session, writePath); + Expression projection = Expression.select(new String[] {"Name", "State"}, Expression.root()); + + ScanOptions options = ScanOptions.builder().projection(projection).build(); + + List people = readAll(ds, options, allocator, batch -> { + List results = new ArrayList<>(); + VarCharVector names = (VarCharVector) batch.getVector("Name"); + VarCharVector states = (VarCharVector) batch.getVector("State"); + for (int i = 0; i < batch.getRowCount(); i++) { + String name = names.isNull(i) ? null : new String(names.get(i), UTF_8); + String state = states.isNull(i) ? null : new String(states.get(i), UTF_8); + results.add(new Person(name, null, state)); + } + return results; + }); + assertEquals(MINIMAL_DATA.size(), people.size()); + for (int i = 0; i < MINIMAL_DATA.size(); i++) { + assertEquals(MINIMAL_DATA.get(i).name, people.get(i).name); + assertEquals(MINIMAL_DATA.get(i).state, people.get(i).state); } } @Test - public void testProjectedScanWithFilter() { - var filterOptions = ScanOptions.builder() - .addColumns("State", "Salary", "Name") - .predicate(Binary.eq(GetItem.of(Root.INSTANCE, "State"), Literal.string("VA"))) - .build(); - - try (var file = Files.open(MINIMAL_PATH); - var filteredScan = file.newScan(filterOptions)) { - var dtype = filteredScan.getDataType(); - assertEquals(DType.Variant.STRUCT, dtype.getVariant()); - assertEquals(dtype.getFieldNames(), List.of("State", "Salary", "Name")); - - var people = readToList(filteredScan, new TestCase() { - @Override - public Array[] open(Array batch) { - return new Array[] { - batch.getField(0), // state - batch.getField(1), // salary - batch.getField(2), // name - }; - } - - @Override - public Person readRow(Array[] fields, int idx) { - var state = fields[0].getUTF8(idx); - var salary = fields[1].getBigDecimal(idx); - var name = fields[2].getUTF8(idx); - return new Person(name, salary, state); - } - }); - - assertEquals(List.of(new Person("John", BigDecimal.valueOf(10_000L, 2), "VA")), people); - } + public void testProjectedScanWithFilter() throws Exception { + BufferAllocator allocator = ArrowAllocation.rootAllocator(); + Session session = Session.create(); + DataSource ds = DataSource.open(session, writePath); + Expression filter = + Expression.binary(Expression.BinaryOp.EQ, Expression.column("State"), Expression.literal("VA")); + + ScanOptions options = ScanOptions.builder().filter(filter).build(); + List people = readAll(ds, options, allocator, TestMinimal::readFullBatch); + assertEquals(List.of(new Person("John", BigDecimal.valueOf(10_000L, 2), "VA")), people); } - interface TestCase { - Array[] open(Array batch); - - Person readRow(Array[] fields, int idx); + private interface BatchReader { + List read(VectorSchemaRoot root); } - private List readToList(ArrayIterator scan, TestCase testCase) { - List people = new ArrayList<>(); + private static List readAll( + DataSource ds, ScanOptions options, BufferAllocator allocator, BatchReader reader) throws Exception { + List result = new ArrayList<>(); + Scan scan = ds.scan(options); while (scan.hasNext()) { - var batch = scan.next(); - Array[] fields = testCase.open(batch); + Partition partition = scan.next(); + try (ArrowReader arrowReader = partition.scanArrow(allocator)) { + while (arrowReader.loadNextBatch()) { + result.addAll(reader.read(arrowReader.getVectorSchemaRoot())); + } + } + } + return result; + } - for (int batchIdx = 0; batchIdx < batch.getLen(); batchIdx++) { - people.add(testCase.readRow(fields, batchIdx)); + private static List readFullBatch(VectorSchemaRoot root) { + List result = new ArrayList<>(); + VarCharVector names = (VarCharVector) root.getVector("Name"); + FieldVector salaries = root.getVector("Salary"); + VarCharVector states = (VarCharVector) root.getVector("State"); + + for (int i = 0; i < root.getRowCount(); i++) { + String name = names.isNull(i) ? null : new String(names.get(i), UTF_8); + String state = states.isNull(i) ? null : new String(states.get(i), UTF_8); + BigDecimal salary = null; + if (!salaries.isNull(i)) { + Object v = salaries.getObject(i); + if (v instanceof BigDecimal) { + salary = (BigDecimal) v; + } else { + salary = new BigDecimal(v.toString()); + } } + result.add(new Person(name, salary, state)); } - return people; + return result; } } diff --git a/java/vortex-jni/src/test/java/dev/vortex/api/expressions/LiteralTest.java b/java/vortex-jni/src/test/java/dev/vortex/api/expressions/LiteralTest.java deleted file mode 100644 index 30d91bd99d9..00000000000 --- a/java/vortex-jni/src/test/java/dev/vortex/api/expressions/LiteralTest.java +++ /dev/null @@ -1,23 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.api.expressions; - -import static org.junit.jupiter.api.Assertions.assertEquals; - -import dev.vortex.api.Expression; -import dev.vortex.api.proto.Expressions; -import dev.vortex.proto.ExprProtos; -import java.math.BigDecimal; -import java.math.BigInteger; -import org.junit.jupiter.api.Test; - -public final class LiteralTest { - @Test - public void testLiteral_decimals() { - Literal lit = Literal.decimal(new BigDecimal(BigInteger.valueOf(-1234L), 3), 38, 3); - ExprProtos.Expr serialized = Expressions.serialize(lit); - Expression out = Expressions.deserialize(serialized); - assertEquals(lit, out); - } -} diff --git a/java/vortex-jni/src/test/java/dev/vortex/api/expressions/proto/TestExpressionProtos.java b/java/vortex-jni/src/test/java/dev/vortex/api/expressions/proto/TestExpressionProtos.java deleted file mode 100644 index f3e9fffd081..00000000000 --- a/java/vortex-jni/src/test/java/dev/vortex/api/expressions/proto/TestExpressionProtos.java +++ /dev/null @@ -1,33 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.api.expressions.proto; - -import static org.junit.jupiter.api.Assertions.assertEquals; - -import dev.vortex.api.Expression; -import dev.vortex.api.expressions.*; -import dev.vortex.api.proto.Expressions; -import dev.vortex.proto.ExprProtos; -import org.junit.jupiter.api.Test; - -public final class TestExpressionProtos { - @Test - public void testRoundTrip() { - Expression expression = Binary.and( - GetItem.of(Root.INSTANCE, "a.b.c"), - Binary.or(Root.INSTANCE, Not.of(Literal.bool(null)), Literal.bool(false)), - Binary.eq(Literal.bool(true), Not.of(Literal.bool(false)))); - ExprProtos.Expr proto = Expressions.serialize(expression); - Expression deserialized = Expressions.deserialize(proto); - assertEquals(expression, deserialized); - } - - @Test - public void testIsNullRoundTrip() { - Expression expression = IsNull.of(GetItem.of(Root.INSTANCE, "a.b.c")); - ExprProtos.Expr proto = Expressions.serialize(expression); - Expression deserialized = Expressions.deserialize(proto); - assertEquals(expression, deserialized); - } -} diff --git a/java/vortex-jni/src/test/java/dev/vortex/api/expressions/proto/TestOpenErrors.java b/java/vortex-jni/src/test/java/dev/vortex/api/expressions/proto/TestOpenErrors.java deleted file mode 100644 index 02d824467f0..00000000000 --- a/java/vortex-jni/src/test/java/dev/vortex/api/expressions/proto/TestOpenErrors.java +++ /dev/null @@ -1,20 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.api.expressions.proto; - -import static org.junit.jupiter.api.Assertions.assertThrows; -import static org.junit.jupiter.api.Assertions.assertTrue; - -import dev.vortex.jni.NativeFileMethods; -import org.junit.jupiter.api.Test; - -public final class TestOpenErrors { - @Test - public void testOpenThrows() { - RuntimeException runtimeException = assertThrows(RuntimeException.class, () -> { - NativeFileMethods.open("bad_scheme:///fake-location", null); - }); - assertTrue(runtimeException.getMessage().contains("Invalid URL")); - } -} diff --git a/java/vortex-jni/src/test/java/dev/vortex/jni/JNIWriterTest.java b/java/vortex-jni/src/test/java/dev/vortex/jni/JNIWriterTest.java index 3a9e356d2fa..d7411e38c07 100644 --- a/java/vortex-jni/src/test/java/dev/vortex/jni/JNIWriterTest.java +++ b/java/vortex-jni/src/test/java/dev/vortex/jni/JNIWriterTest.java @@ -8,14 +8,18 @@ import static org.junit.jupiter.api.Assertions.assertNotNull; import static org.junit.jupiter.api.Assertions.assertTrue; -import dev.vortex.api.DType; +import dev.vortex.api.DataSource; +import dev.vortex.api.Partition; +import dev.vortex.api.Scan; import dev.vortex.api.ScanOptions; +import dev.vortex.api.Session; import dev.vortex.api.VortexWriter; import dev.vortex.arrow.ArrowAllocation; import java.io.IOException; import java.nio.file.Files; import java.nio.file.Path; import java.util.HashMap; +import java.util.List; import java.util.Map; import org.apache.arrow.c.ArrowArray; import org.apache.arrow.c.ArrowSchema; @@ -24,17 +28,14 @@ import org.apache.arrow.vector.IntVector; import org.apache.arrow.vector.VarCharVector; import org.apache.arrow.vector.VectorSchemaRoot; +import org.apache.arrow.vector.ipc.ArrowReader; import org.apache.arrow.vector.types.pojo.ArrowType; import org.apache.arrow.vector.types.pojo.Field; -import org.apache.arrow.vector.types.pojo.FieldType; import org.apache.arrow.vector.types.pojo.Schema; import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.io.TempDir; -/** - * Direct test of JNI writer to isolate pointer alignment issue. - */ public final class JNIWriterTest { @TempDir @@ -45,94 +46,81 @@ public static void loadLibrary() { NativeLoader.loadJni(); } + private static Schema personSchema() { + return new Schema(List.of( + Field.notNullable("name", new ArrowType.Utf8()), + Field.notNullable("age", new ArrowType.Int(32, true)))); + } + @Test public void testCreateWriter() throws IOException { - // Test just creating and closing a writer Path outputPath = tempDir.resolve("test_create.vortex"); String writePath = outputPath.toAbsolutePath().toUri().toString(); - // Make a new file writer with a very simple schema. - var writeSchema = DType.newStruct( - new String[] { - "name", "age", - }, - new DType[] {DType.newUtf8(false), DType.newInt(false)}, - false); - - // Minimal Arrow schema + BufferAllocator allocator = ArrowAllocation.rootAllocator(); Map options = new HashMap<>(); - System.err.println("Creating writer for path: " + writePath); - - try (VortexWriter writer = VortexWriter.create(writePath, writeSchema, options)) { + Session session = Session.create(); + try (VortexWriter writer = VortexWriter.create(session, writePath, personSchema(), options, allocator)) { assertNotNull(writer); - System.err.println("Writer created successfully"); } - // Verify file was created - assertTrue(Files.exists(outputPath), "Output file should exist"); - System.err.println("File created at: " + outputPath); + assertTrue(Files.exists(outputPath), "output file should exist"); } @Test - public void testWriteBatchFfi() throws IOException { + public void testWriteBatch() throws IOException { Path outputPath = tempDir.resolve("test_ffi.vortex"); String writePath = outputPath.toAbsolutePath().toUri().toString(); - var writeSchema = DType.newStruct( - new String[] {"name", "age"}, new DType[] {DType.newUtf8(false), DType.newInt(false)}, false); - BufferAllocator allocator = ArrowAllocation.rootAllocator(); - - Schema arrowSchema = new Schema(java.util.List.of( - new Field("name", FieldType.notNullable(new ArrowType.Utf8()), null), - new Field("age", FieldType.notNullable(new ArrowType.Int(32, true)), null))); - - try (VortexWriter writer = VortexWriter.create(writePath, writeSchema, new HashMap<>())) { - // Build a batch with Arrow Java - try (VectorSchemaRoot root = VectorSchemaRoot.create(arrowSchema, allocator)) { - VarCharVector nameVec = (VarCharVector) root.getVector("name"); - IntVector ageVec = (IntVector) root.getVector("age"); - - nameVec.allocateNew(3); - ageVec.allocateNew(3); - - nameVec.setSafe(0, "Alice".getBytes(UTF_8)); - nameVec.setSafe(1, "Bob".getBytes(UTF_8)); - nameVec.setSafe(2, "Carol".getBytes(UTF_8)); - ageVec.setSafe(0, 30); - ageVec.setSafe(1, 25); - ageVec.setSafe(2, 40); - - root.setRowCount(3); - - // Export to C Data Interface - try (ArrowArray arrowArray = ArrowArray.allocateNew(allocator); - ArrowSchema arrowSchemaFfi = ArrowSchema.allocateNew(allocator)) { - Data.exportVectorSchemaRoot(allocator, root, null, arrowArray, arrowSchemaFfi); - - writer.writeBatchFfi(arrowArray.memoryAddress(), arrowSchemaFfi.memoryAddress()); - } + Schema schema = personSchema(); + + Session session = Session.create(); + try (VortexWriter writer = VortexWriter.create(session, writePath, schema, new HashMap<>(), allocator); + VectorSchemaRoot root = VectorSchemaRoot.create(schema, allocator)) { + VarCharVector nameVec = (VarCharVector) root.getVector("name"); + IntVector ageVec = (IntVector) root.getVector("age"); + + nameVec.allocateNew(3); + ageVec.allocateNew(3); + + nameVec.setSafe(0, "Alice".getBytes(UTF_8)); + nameVec.setSafe(1, "Bob".getBytes(UTF_8)); + nameVec.setSafe(2, "Carol".getBytes(UTF_8)); + ageVec.setSafe(0, 30); + ageVec.setSafe(1, 25); + ageVec.setSafe(2, 40); + + root.setRowCount(3); + + try (ArrowArray arrowArray = ArrowArray.allocateNew(allocator); + ArrowSchema arrowSchemaFfi = ArrowSchema.allocateNew(allocator)) { + Data.exportVectorSchemaRoot(allocator, root, null, arrowArray, arrowSchemaFfi); + writer.writeBatch(arrowArray.memoryAddress(), arrowSchemaFfi.memoryAddress()); } } - assertTrue(Files.exists(outputPath), "Output file should exist"); - - // Read back and verify - try (var file = dev.vortex.api.Files.open(outputPath.toAbsolutePath().toString()); - var scan = file.newScan(ScanOptions.of())) { - assertEquals(3, file.rowCount()); - - var batch = scan.next(); - var nameField = batch.getField(0); - var ageField = batch.getField(1); - - assertEquals("Alice", nameField.getUTF8(0)); - assertEquals("Bob", nameField.getUTF8(1)); - assertEquals("Carol", nameField.getUTF8(2)); - assertEquals(30, ageField.getInt(0)); - assertEquals(25, ageField.getInt(1)); - assertEquals(40, ageField.getInt(2)); + assertTrue(Files.exists(outputPath), "output file should exist"); + + DataSource ds = DataSource.open(session, writePath); + assertEquals(new DataSource.RowCount.Exact(3L), ds.rowCount()); + + Scan scan = ds.scan(ScanOptions.of()); + while (scan.hasNext()) { + Partition p = scan.next(); + try (ArrowReader reader = p.scanArrow(allocator)) { + reader.loadNextBatch(); + VectorSchemaRoot resultRoot = reader.getVectorSchemaRoot(); + VarCharVector nameOut = (VarCharVector) resultRoot.getVector("name"); + IntVector ageOut = (IntVector) resultRoot.getVector("age"); + assertEquals("Alice", nameOut.getObject(0).toString()); + assertEquals("Bob", nameOut.getObject(1).toString()); + assertEquals("Carol", nameOut.getObject(2).toString()); + assertEquals(30, ageOut.get(0)); + assertEquals(25, ageOut.get(1)); + assertEquals(40, ageOut.get(2)); + } } } } diff --git a/java/vortex-spark/build.gradle.kts b/java/vortex-spark/build.gradle.kts index f5fb83fe7ba..12cd759ec5e 100644 --- a/java/vortex-spark/build.gradle.kts +++ b/java/vortex-spark/build.gradle.kts @@ -6,7 +6,7 @@ import com.github.jengelman.gradle.plugins.shadow.tasks.ShadowJar plugins { `java-library` `jvm-test-suite` - id("com.gradleup.shadow") version "9.4.0" + id("com.gradleup.shadow") version "9.4.1" } // Derive Scala and Spark versions from the Gradle project name (vortex-spark_2.12 or vortex-spark_2.13) diff --git a/java/vortex-spark/src/main/java/dev/vortex/spark/ArrowUtils.java b/java/vortex-spark/src/main/java/dev/vortex/spark/ArrowUtils.java index 63f5c86185a..6b3c130c467 100644 --- a/java/vortex-spark/src/main/java/dev/vortex/spark/ArrowUtils.java +++ b/java/vortex-spark/src/main/java/dev/vortex/spark/ArrowUtils.java @@ -16,19 +16,19 @@ /** * Utility class for converting Arrow types to Spark SQL data types. - *

- * This class provides static methods to convert Arrow field definitions and type definitions - * into their corresponding Spark SQL DataType representations. It handles the mapping between - * Arrow's type system and Spark's type system, including complex types like structs and arrays. + * + *

This class provides static methods to convert Arrow field definitions and type definitions into their + * corresponding Spark SQL DataType representations. It handles the mapping between Arrow's type system and Spark's type + * system, including complex types like structs and arrays. */ public final class ArrowUtils { private ArrowUtils() {} /** * Converts an Arrow Field to a Spark SQL DataType. - *

- * This method handles complex types like structs and arrays by recursively converting - * their child fields. For primitive types, it delegates to {@link #fromArrowType(ArrowType)}. + * + *

This method handles complex types like structs and arrays by recursively converting their child fields. For + * primitive types, it delegates to {@link #fromArrowType(ArrowType)}. * * @param field the Arrow field to convert * @return the corresponding Spark SQL DataType @@ -55,10 +55,9 @@ public static DataType fromArrowField(Field field) { /** * Converts an Arrow type to a Spark SQL DataType. - *

- * This method maps primitive Arrow types to their corresponding Spark SQL types. - * It supports most common Arrow types including integers, floating point numbers, - * strings, binary data, dates, timestamps, decimals, and nulls. + * + *

This method maps primitive Arrow types to their corresponding Spark SQL types. It supports most common Arrow + * types including integers, floating point numbers, strings, binary data, dates, timestamps, decimals, and nulls. * * @param dt the Arrow type to convert * @return the corresponding Spark SQL DataType @@ -114,11 +113,7 @@ public static DataType fromArrowType(ArrowType dt) { case Timestamp: { ArrowType.Timestamp ts = (ArrowType.Timestamp) dt; if (ts.getUnit() == TimeUnit.MICROSECOND) { - if (ts.getTimezone() != null) { - return DataTypes.TimestampNTZType; - } else { - return DataTypes.TimestampType; - } + return ts.getTimezone() != null ? DataTypes.TimestampType : DataTypes.TimestampNTZType; } else { throw new UnsupportedOperationException("Unsupported Arrow type: " + dt); } diff --git a/java/vortex-spark/src/main/java/dev/vortex/spark/SparkTypes.java b/java/vortex-spark/src/main/java/dev/vortex/spark/SparkTypes.java deleted file mode 100644 index bcd4aed6219..00000000000 --- a/java/vortex-spark/src/main/java/dev/vortex/spark/SparkTypes.java +++ /dev/null @@ -1,174 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.spark; - -import dev.vortex.api.DType; -import java.util.Optional; -import org.apache.spark.sql.connector.catalog.Column; -import org.apache.spark.sql.types.*; - -/** - * Helpers for converting between Spark and Vortex type systems. - */ -public final class SparkTypes { - private SparkTypes() {} - - public static DType toDType(StructType schema) { - - String[] fieldNames = new String[schema.length()]; - DType[] fieldTypes = new DType[schema.length()]; - - for (int i = 0; i < schema.length(); i++) { - StructField field = schema.fields()[i]; - fieldNames[i] = field.name(); - fieldTypes[i] = convertField(field.dataType(), field.nullable()); - } - - return DType.newStruct(fieldNames, fieldTypes, false); - } - - // Convert field type to Vortex type. - static DType convertField(DataType dataType, boolean isNullable) { - if (dataType instanceof ByteType) { - return DType.newByte(isNullable); - } else if (dataType instanceof ShortType) { - return DType.newShort(isNullable); - } else if (dataType instanceof IntegerType) { - return DType.newInt(isNullable); - } else if (dataType instanceof LongType) { - return DType.newLong(isNullable); - } else if (dataType instanceof FloatType) { - return DType.newFloat(isNullable); - } else if (dataType instanceof DoubleType) { - return DType.newDouble(isNullable); - } else if (dataType instanceof DecimalType) { - DecimalType decimalType = (DecimalType) dataType; - return DType.newDecimal(decimalType.precision(), decimalType.scale(), isNullable); - } else if (dataType instanceof BooleanType) { - return DType.newBinary(isNullable); - } else if (dataType instanceof StringType) { - return DType.newUtf8(isNullable); - } else if (dataType instanceof BinaryType) { - return DType.newBinary(isNullable); - } else if (dataType instanceof ArrayType) { - ArrayType arrayType = ((ArrayType) dataType); - DType elementType = convertField(arrayType.elementType(), arrayType.containsNull()); - return DType.newList(elementType, isNullable); - } else if (dataType instanceof StructType) { - StructType structType = (StructType) dataType; - return toDType(structType); - } else if (dataType instanceof TimestampType) { - // Spark emits timestamps with UTC timezone and microsecond precision. - return DType.newTimestamp(DType.TimeUnit.MICROSECONDS, Optional.of("UTC"), isNullable); - } else if (dataType instanceof TimestampNTZType) { - // TimestampNTZ is microsecond timestamp without zone - return DType.newTimestamp(DType.TimeUnit.MICROSECONDS, Optional.empty(), isNullable); - } else if (dataType instanceof DateType) { - // TODO(aduffy): any problems with the date values since they're refed to - // gregorian proleptic? - return DType.newDate(DType.TimeUnit.DAYS, isNullable); - } else { - throw new IllegalArgumentException("Unsupported data type for Vortex: " + dataType); - } - } - - /** - * Convert a STRUCT Vortex type to a Spark {@link DataType}. - */ - public static DataType toDataType(DType dType) { - switch (dType.getVariant()) { - case NULL: - return DataTypes.NullType; - case BOOL: - return DataTypes.BooleanType; - case PRIMITIVE_U8: - case PRIMITIVE_I8: - return DataTypes.ByteType; - case PRIMITIVE_U16: - case PRIMITIVE_I16: - return DataTypes.ShortType; - case PRIMITIVE_U32: - case PRIMITIVE_I32: - return DataTypes.IntegerType; - case PRIMITIVE_U64: - case PRIMITIVE_I64: - return DataTypes.LongType; - case PRIMITIVE_F16: - throw new IllegalArgumentException("Spark does not support f16"); - case PRIMITIVE_F32: - return DataTypes.FloatType; - case PRIMITIVE_F64: - return DataTypes.DoubleType; - case UTF8: - return DataTypes.StringType; - case BINARY: - return DataTypes.BinaryType; - case STRUCT: - // For each of the inner struct fields, we capture them together here. - var fieldNames = dType.getFieldNames(); - var fieldTypes = dType.getFieldTypes(); - - // NOTE: it's very important we do this with a for loop. Using the streams API can easily - // lead to StackOverflowError being thrown. - var fields = new StructField[fieldNames.size()]; - for (int i = 0; i < fieldNames.size(); i++) { - var name = fieldNames.get(i); - try (var type = fieldTypes.get(i)) { - fields[i] = new StructField(name, toDataType(type), dType.isNullable(), Metadata.empty()); - } - } - - return DataTypes.createStructType(fields); - case LIST: - case FIXED_SIZE_LIST: - return DataTypes.createArrayType(toDataType(dType.getElementType()), dType.isNullable()); - case EXTENSION: - /* - * Spark does not have a direct equivalent for many of the temporal types we support in Vortex or Arrow. - * Notably, there is no DATE type, and timestamps can have at most µs-level precision. - * This means that we need to "cheat" a little in how we convert into Spark's type system. We support - * the following conversions: - * 1. Vortex DATE -> Spark TIMESTAMP (with 00:00:00 time and local timezone) - * 2. Vortex TIMESTAMP -> Spark TIMESTAMP, with precision truncated to µs - * 3. Vortex TIME -> not supported - */ - if (dType.isTime()) { - throw new IllegalArgumentException("Spark does not support Vortex TIME data type"); - } - - if (dType.isDate()) { - return DateType$.MODULE$; - } - - if (dType.isTimestamp()) { - return TimestampType$.MODULE$; - } - - // TODO(aduffy): other extension types - throw new IllegalArgumentException("Unsupported non-temporal extension type"); - case DECIMAL: - return DataTypes.createDecimalType(dType.getPrecision(), dType.getScale()); - default: - throw new IllegalArgumentException("unreachable"); - } - } - - /** - * Convert a STRUCT Vortex type to a Spark {@link Column}. - */ - public static Column[] toColumns(DType dType) { - var fieldNames = dType.getFieldNames(); - var fieldTypes = dType.getFieldTypes(); - var columns = new Column[fieldNames.size()]; - - for (int i = 0; i < columns.length; i++) { - var name = fieldNames.get(i); - try (var type = fieldTypes.get(i)) { - columns[i] = Column.create(name, toDataType(type), type.isNullable()); - } - } - - return columns; - } -} diff --git a/java/vortex-spark/src/main/java/dev/vortex/spark/VortexDataSourceV2.java b/java/vortex-spark/src/main/java/dev/vortex/spark/VortexDataSourceV2.java index 66f3e5f001c..1d564b8c0e0 100644 --- a/java/vortex-spark/src/main/java/dev/vortex/spark/VortexDataSourceV2.java +++ b/java/vortex-spark/src/main/java/dev/vortex/spark/VortexDataSourceV2.java @@ -7,9 +7,8 @@ import com.google.common.collect.ImmutableList; import com.google.common.collect.ImmutableMap; import com.google.common.collect.Iterables; -import dev.vortex.api.File; -import dev.vortex.api.Files; -import dev.vortex.jni.NativeFileMethods; +import dev.vortex.api.DataSource; +import dev.vortex.jni.NativeFiles; import dev.vortex.spark.config.HadoopUtils; import dev.vortex.spark.read.PartitionPathUtils; import java.util.Map; @@ -19,22 +18,23 @@ import java.util.stream.Collectors; import java.util.stream.Stream; import org.apache.spark.sql.SparkSession; -import org.apache.spark.sql.connector.catalog.CatalogV2Util; import org.apache.spark.sql.connector.catalog.Table; import org.apache.spark.sql.connector.catalog.TableProvider; import org.apache.spark.sql.connector.expressions.Transform; import org.apache.spark.sql.sources.DataSourceRegister; import org.apache.spark.sql.types.DataType; +import org.apache.spark.sql.types.Metadata; +import org.apache.spark.sql.types.StructField; import org.apache.spark.sql.types.StructType; import org.apache.spark.sql.util.CaseInsensitiveStringMap; import scala.Option; /** * Spark V2 data source for reading and writing Vortex files. - *

- * This class is automatically registered so it can be discovered by the Spark runtime. - * For reading: {@link org.apache.spark.sql.SparkSession#read} and specify the format as "vortex". - * For writing: {@link org.apache.spark.sql.Dataset#write} and specify the format as "vortex". + * + *

This class is automatically registered so it can be discovered by the Spark runtime. For reading: + * {@link org.apache.spark.sql.SparkSession#read} and specify the format as "vortex". For writing: + * {@link org.apache.spark.sql.Dataset#write} and specify the format as "vortex". */ public final class VortexDataSourceV2 implements TableProvider, DataSourceRegister { private static final ObjectMapper MAPPER = new ObjectMapper(); @@ -46,9 +46,8 @@ public final class VortexDataSourceV2 implements TableProvider, DataSourceRegist /** * Creates a new instance of the Vortex data source. - *

- * This no-argument constructor is required for Spark to instantiate the data source - * through reflection. + * + *

This no-argument constructor is required for Spark to instantiate the data source through reflection. */ public VortexDataSourceV2() { this.sparkSession = SparkSession.getActiveSession(); @@ -56,9 +55,9 @@ public VortexDataSourceV2() { /** * Infers the schema of the Vortex files specified in the options. - *

- * This method examines the last file in the provided paths to determine the schema. - * Currently, schema evolution and merging across multiple files is not supported. + * + *

This method examines the last file in the provided paths to determine the schema. Currently, schema evolution + * and merging across multiple files is not supported. * * @param options the data source options containing file paths * @return the inferred Spark SQL schema @@ -82,20 +81,26 @@ public StructType inferSchema(CaseInsensitiveStringMap options) { var pathToInfer = Objects.requireNonNull(Iterables.getLast(paths)); // If the path is a directory, scan the directory for a file and use that file if (!pathToInfer.endsWith(".vortex")) { - Optional firstFile = NativeFileMethods.listVortexFiles(pathToInfer, formatOptions).stream() - .findFirst(); + Optional firstFile = + NativeFiles.listFiles(VortexSparkSession.get(formatOptions), pathToInfer, formatOptions).stream() + .findFirst(); if (firstFile.isEmpty()) { - return new StructType(); + throw new RuntimeException(String.format("UNABLE_TO_INFER_SCHEMA format: %s", shortName())); } else { pathToInfer = firstFile.get(); } } StructType dataSchema; - try (File file = Files.open(pathToInfer, formatOptions)) { - var columns = SparkTypes.toColumns(file.getDType()); - dataSchema = CatalogV2Util.v2ColumnsToStructType(columns); + { + DataSource ds = DataSource.open(VortexSparkSession.get(formatOptions), pathToInfer, formatOptions); + var arrowSchema = ds.arrowSchema(dev.vortex.arrow.ArrowAllocation.rootAllocator()); + StructField[] fields = arrowSchema.getFields().stream() + .map(f -> new StructField( + f.getName(), ArrowUtils.fromArrowField(f), f.isNullable(), Metadata.empty())) + .toArray(StructField[]::new); + dataSchema = new StructType(fields); } // Discover partition columns from Hive-style directory paths and append them. @@ -115,13 +120,13 @@ public StructType inferSchema(CaseInsensitiveStringMap options) { /** * Creates a Vortex table instance with the given schema and properties. - *

- * This method creates a VortexWritableTable that can be used to both read from and write to - * Vortex files. The partitioning parameter is currently ignored. * - * @param schema the table schema + *

This method creates a VortexWritableTable that can be used to both read from and write to Vortex files. The + * partitioning parameter is currently ignored. + * + * @param schema the table schema * @param partitioning table partitioning transforms - * @param properties the table properties containing file paths and other options + * @param properties the table properties containing file paths and other options * @return a VortexTable instance for reading and writing data * @throws RuntimeException if required path properties are missing */ @@ -134,9 +139,9 @@ public Table getTable(StructType schema, Transform[] partitioning, Map - * Returns true to indicate that this data source accepts external schemas, - * which is necessary for write operations where the DataFrame provides the schema. + * + *

Returns true to indicate that this data source accepts external schemas, which is necessary for write + * operations where the DataFrame provides the schema. * * @return true to accept external schemas */ @@ -147,9 +152,9 @@ public boolean supportsExternalMetadata() { /** * Returns the short name identifier for this data source. - *

- * This name is used by Spark when registering the data source and can be used - * in SQL queries and DataFrame read operations to specify this format. + * + *

This name is used by Spark when registering the data source and can be used in SQL queries and DataFrame read + * operations to specify this format. * * @return the short name "vortex" */ diff --git a/java/vortex-spark/src/main/java/dev/vortex/spark/VortexFilePartition.java b/java/vortex-spark/src/main/java/dev/vortex/spark/VortexFilePartition.java index 7d88f8469c4..4247788a887 100644 --- a/java/vortex-spark/src/main/java/dev/vortex/spark/VortexFilePartition.java +++ b/java/vortex-spark/src/main/java/dev/vortex/spark/VortexFilePartition.java @@ -3,74 +3,30 @@ package dev.vortex.spark; -import com.google.common.collect.ImmutableList; -import com.google.common.collect.ImmutableMap; import java.io.Serializable; +import java.util.List; import java.util.Map; -import org.apache.spark.sql.connector.catalog.Column; import org.apache.spark.sql.connector.read.InputPartition; +import org.apache.spark.sql.types.StructType; /** - * An {@link InputPartition} for reading a whole Vortex file. - *

- * This class represents a partition that corresponds to a single Vortex file. - * It contains the file path and the columns to be read from that file. - * Each partition can be processed independently by Spark executors. + * An {@link InputPartition} describing a group of Vortex files that a single reader should handle together. + * + *

Each executor opens a single Vortex {@code Session}, {@code DataSource} and {@code Scan} over the partition's + * {@link #paths()} and consumes every Vortex partition produced by that scan before moving on to the next Spark + * {@code InputPartition}. + * + *

The requested output schema is carried as a {@link StructType} rather than a list of {@code Column} objects: + * {@code StructType} is the stable serialization surface in Spark and survives shipping to executors reliably. + * + * @param paths the Vortex file paths (or globs) belonging to this input partition + * @param readSchema the requested output schema (data columns + partition columns) + * @param formatOptions object-store properties used to open the files + * @param partitionValues Hive-style partition column values shared by all {@link #paths()} */ -public final class VortexFilePartition implements InputPartition, Serializable { - private final String path; - private final ImmutableList columns; - private final ImmutableMap formatOptions; - private final ImmutableMap partitionValues; - - /** - * Creates a new Vortex file partition. - * - * @param path the file system path to the Vortex file - * @param columns the list of columns to read from the file - * @param formatOptions options for accessing the file (S3/Azure credentials, etc.) - * @param partitionValues Hive-style partition column values extracted from the file path - */ - public VortexFilePartition( - String path, - ImmutableList columns, - ImmutableMap formatOptions, - ImmutableMap partitionValues) { - this.path = path; - this.columns = columns; - this.formatOptions = formatOptions; - this.partitionValues = partitionValues; - } - - /** - * Returns the file system path to the Vortex file for this partition. - * - * @return the file path - */ - public String getPath() { - return path; - } - - /** - * Returns the list of columns to be read from this partition. - * - * @return the immutable list of columns - */ - public ImmutableList getColumns() { - return columns; - } - - public Map getFormatOptions() { - return formatOptions; - } - - /** - * Returns the partition column values parsed from this file's Hive-style directory path. - * Keys are column names, values are the string-encoded partition values. - * - * @return the partition values, empty if the file is not in a partitioned directory - */ - public ImmutableMap getPartitionValues() { - return partitionValues; - } -} +public record VortexFilePartition( + List paths, + StructType readSchema, + Map formatOptions, + Map partitionValues) + implements InputPartition, Serializable {} diff --git a/java/vortex-spark/src/main/java/dev/vortex/spark/VortexSessionProvider.java b/java/vortex-spark/src/main/java/dev/vortex/spark/VortexSessionProvider.java new file mode 100644 index 00000000000..fe3102d4c55 --- /dev/null +++ b/java/vortex-spark/src/main/java/dev/vortex/spark/VortexSessionProvider.java @@ -0,0 +1,24 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +package dev.vortex.spark; + +import dev.vortex.api.Session; + +/** + * User hook for supplying a custom {@link Session} to Vortex Spark readers and writers. + * + *

Implementations must have a public no-argument constructor. They are instantiated exactly once per JVM (driver or + * executor) the first time a scan or write references the provider's class name through the + * {@code vortex.session.provider} option. The returned {@link Session} is then shared across every Vortex task on that + * JVM. + * + *

Typical use: install custom encodings, scalar functions, or layouts on a {@link Session} before returning it. + */ +public interface VortexSessionProvider { + /** + * Construct (or return a cached) {@link Session}. Called at most once per JVM per provider class. The returned + * session is retained for the JVM's lifetime. + */ + Session get(); +} diff --git a/java/vortex-spark/src/main/java/dev/vortex/spark/VortexSparkSession.java b/java/vortex-spark/src/main/java/dev/vortex/spark/VortexSparkSession.java new file mode 100644 index 00000000000..1d3d07d3b25 --- /dev/null +++ b/java/vortex-spark/src/main/java/dev/vortex/spark/VortexSparkSession.java @@ -0,0 +1,91 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +package dev.vortex.spark; + +import dev.vortex.api.Session; +import java.util.Map; +import java.util.Objects; +import java.util.concurrent.ConcurrentHashMap; + +/** + * JVM-wide holder for one or more Vortex {@link Session}s used by Spark readers and writers. The Rust side multiplexes + * every session onto one shared current-thread runtime, so sharing a single Java handle per JVM amortises session + * construction across every Spark task. + * + *

Three levels of customisation: + * + *

    + *
  1. Default: call {@link #get()} — returns a lazily-initialised singleton that lives for the life of the + * JVM. No configuration required. + *
  2. Driver-side override: call {@link #setDefault(Session)} with a custom session before any Spark action. + * Effective on the driver JVM only. + *
  3. Executor-friendly override: pass option {@code vortex.session.provider} with the fully-qualified name of + * a {@link VortexSessionProvider} implementation (no-arg constructor). Spark ships the class name to every + * executor, which instantiates the provider once per JVM and caches the returned session. Use this when you need + * the same custom session on the driver and on executors. + *
+ * + *

Native resources are released by {@code VortexCleaner} when the JVM shuts down; sessions held here are strongly + * referenced for the JVM's lifetime. + */ +public final class VortexSparkSession { + /** Options key used to select a {@link VortexSessionProvider} by class name. */ + public static final String PROVIDER_OPTION = "vortex.session.provider"; + + private static final ConcurrentHashMap providerCache = new ConcurrentHashMap<>(); + private static volatile Session defaultSession; + + private VortexSparkSession() {} + + /** Returns the default JVM-wide session, creating it on first use. */ + public static Session get() { + Session s = defaultSession; + if (s != null) { + return s; + } + synchronized (VortexSparkSession.class) { + if (defaultSession == null) { + defaultSession = Session.create(); + } + return defaultSession; + } + } + + /** + * Resolve the session to use for a given set of Spark format options. Honours the {@value #PROVIDER_OPTION} key; + * falls back to {@link #get()} otherwise. + */ + public static Session get(Map options) { + String providerClass = options == null ? null : options.get(PROVIDER_OPTION); + if (providerClass == null || providerClass.isEmpty()) { + return get(); + } + return providerCache.computeIfAbsent(providerClass, VortexSparkSession::loadProvider); + } + + /** + * Replace the default session. Intended for driver-side customisation before any Spark action runs. Does not + * propagate to executors — use {@link VortexSessionProvider} for that. + */ + public static void setDefault(Session session) { + Objects.requireNonNull(session, "session"); + synchronized (VortexSparkSession.class) { + defaultSession = session; + } + } + + private static Session loadProvider(String className) { + try { + Class cls = Class.forName(className, true, Thread.currentThread().getContextClassLoader()); + Object instance = cls.getDeclaredConstructor().newInstance(); + if (!(instance instanceof VortexSessionProvider provider)) { + throw new IllegalArgumentException( + className + " does not implement " + VortexSessionProvider.class.getName()); + } + return Objects.requireNonNull(provider.get(), className + ".get() returned null"); + } catch (ReflectiveOperationException e) { + throw new IllegalArgumentException("Failed to load Vortex session provider " + className, e); + } + } +} diff --git a/java/vortex-spark/src/main/java/dev/vortex/spark/VortexTable.java b/java/vortex-spark/src/main/java/dev/vortex/spark/VortexTable.java index d650923ee7f..92cc55ff211 100644 --- a/java/vortex-spark/src/main/java/dev/vortex/spark/VortexTable.java +++ b/java/vortex-spark/src/main/java/dev/vortex/spark/VortexTable.java @@ -6,11 +6,17 @@ import com.google.common.collect.ImmutableList; import com.google.common.collect.ImmutableSet; import com.google.common.collect.Iterables; +import com.google.common.collect.Maps; import dev.vortex.spark.read.VortexScanBuilder; import dev.vortex.spark.write.VortexWriteBuilder; +import java.util.Arrays; import java.util.Map; import java.util.Set; -import org.apache.spark.sql.connector.catalog.*; +import org.apache.spark.sql.connector.catalog.CatalogV2Util; +import org.apache.spark.sql.connector.catalog.SupportsRead; +import org.apache.spark.sql.connector.catalog.SupportsWrite; +import org.apache.spark.sql.connector.catalog.Table; +import org.apache.spark.sql.connector.catalog.TableCapability; import org.apache.spark.sql.connector.expressions.Transform; import org.apache.spark.sql.connector.read.ScanBuilder; import org.apache.spark.sql.connector.write.LogicalWriteInfo; @@ -18,9 +24,7 @@ import org.apache.spark.sql.types.StructType; import org.apache.spark.sql.util.CaseInsensitiveStringMap; -/** - * Spark V2 {@link Table} of Vortex files that supports both reading and writing. - */ +/** Spark V2 {@link Table} of Vortex files that supports both reading and writing. */ public final class VortexTable implements Table, SupportsRead, SupportsWrite { private static final String SHORT_NAME = "vortex"; @@ -29,9 +33,7 @@ public final class VortexTable implements Table, SupportsRead, SupportsWrite { private final Map formatOptions; private final Transform[] partitionTransforms; - /** - * Creates a new VortexTable with read/write support. - */ + /** Creates a new VortexTable with read/write support. */ public VortexTable( ImmutableList paths, StructType schema, @@ -45,27 +47,26 @@ public VortexTable( /** * Creates a new ScanBuilder for this table. - *

- * The scan builder is pre-configured with all the file paths and columns - * from this table. The options parameter is currently unused but reserved - * for future use (e.g., S3 credentials). * - * @param _options scan options (currently unused) + *

The scan builder is pre-configured with all the file paths and columns from this table. + * + * @param options scan options * @return a new VortexScanBuilder configured for this table */ @Override - public ScanBuilder newScanBuilder(CaseInsensitiveStringMap _options) { - ImmutableList readColumns = ImmutableList.builder() - .add(CatalogV2Util.structTypeToV2Columns(schema)) - .build(); - return new VortexScanBuilder(formatOptions).addAllPaths(paths).addAllColumns(readColumns); + public ScanBuilder newScanBuilder(CaseInsensitiveStringMap options) { + Map opts = Maps.newHashMap(); + opts.putAll(formatOptions); + opts.putAll(options); + return new VortexScanBuilder(opts) + .addAllPaths(paths) + .addAllColumns(Arrays.asList(CatalogV2Util.structTypeToV2Columns(schema))); } /** * Returns the name of this table. - *

- * The name includes the "vortex" prefix and a comma-separated list - * of all file paths that comprise this table. + * + *

The name includes the "vortex" prefix and a comma-separated list of all file paths that comprise this table. * * @return the table name in the format: vortex."path1,path2,..." */ @@ -76,9 +77,9 @@ public String name() { /** * Returns the schema of this table. - *

- * The schema is derived from the columns available for reading, or from - * the explicit write schema if this table is being used for writing. + * + *

The schema is derived from the columns available for reading, or from the explicit write schema if this table + * is being used for writing. * * @return the StructType representing the table schema */ @@ -89,9 +90,8 @@ public StructType schema() { /** * Creates a new WriteBuilder for writing data to this table. - *

- * The WriteBuilder is responsible for configuring and executing write operations - * to create new Vortex files. + * + *

The WriteBuilder is responsible for configuring and executing write operations to create new Vortex files. * * @param info logical information about the write operation * @return a new VortexWriteBuilder configured for this table @@ -115,8 +115,8 @@ public Transform[] partitioning() { /** * Returns the capabilities supported by this table. - *

- * Vortex tables support batch reading and batch writing. + * + *

Vortex tables support batch reading and batch writing. * * @return a set containing TableCapability.BATCH_READ and BATCH_WRITE */ diff --git a/java/vortex-spark/src/main/java/dev/vortex/spark/read/PartitionPathUtils.java b/java/vortex-spark/src/main/java/dev/vortex/spark/read/PartitionPathUtils.java index bbd73f6115c..24a5c8aa953 100644 --- a/java/vortex-spark/src/main/java/dev/vortex/spark/read/PartitionPathUtils.java +++ b/java/vortex-spark/src/main/java/dev/vortex/spark/read/PartitionPathUtils.java @@ -3,19 +3,34 @@ package dev.vortex.spark.read; +import com.google.common.base.Splitter; +import com.google.common.primitives.Doubles; +import com.google.common.primitives.Ints; +import com.google.common.primitives.Longs; import java.net.URLDecoder; import java.nio.charset.StandardCharsets; import java.util.LinkedHashMap; import java.util.Map; import org.apache.spark.sql.execution.vectorized.ConstantColumnVector; -import org.apache.spark.sql.types.*; +import org.apache.spark.sql.types.BooleanType; +import org.apache.spark.sql.types.ByteType; +import org.apache.spark.sql.types.DataType; +import org.apache.spark.sql.types.DataTypes; +import org.apache.spark.sql.types.DateType; +import org.apache.spark.sql.types.DoubleType; +import org.apache.spark.sql.types.FloatType; +import org.apache.spark.sql.types.IntegerType; +import org.apache.spark.sql.types.LongType; +import org.apache.spark.sql.types.ShortType; +import org.apache.spark.sql.types.StringType; +import org.apache.spark.sql.types.TimestampNTZType; +import org.apache.spark.sql.types.TimestampType; import org.apache.spark.unsafe.types.UTF8String; -/** - * Utilities for discovering and materializing Hive-style partition columns from file paths. - */ +/** Utilities for discovering and materializing Hive-style partition columns from file paths. */ public final class PartitionPathUtils { private static final String HIVE_DEFAULT_PARTITION = "__HIVE_DEFAULT_PARTITION__"; + private static final Splitter PATH_SPLITTER = Splitter.on('/'); private PartitionPathUtils() {} @@ -26,8 +41,7 @@ private PartitionPathUtils() {} */ public static Map parsePartitionValues(String filePath) { LinkedHashMap values = new LinkedHashMap<>(); - String[] segments = filePath.split("/"); - for (String segment : segments) { + for (String segment : PATH_SPLITTER.split(filePath)) { int eqIdx = segment.indexOf('='); if (eqIdx > 0 && eqIdx < segment.length() - 1) { String key = URLDecoder.decode(segment.substring(0, eqIdx), StandardCharsets.UTF_8); @@ -39,27 +53,21 @@ public static Map parsePartitionValues(String filePath) { } /** - * Infers a Spark {@link DataType} from a partition value string. - * Tries integer, long, double, boolean, and falls back to string. + * Infers a Spark {@link DataType} from a partition value string. Tries integer, long, double, boolean, and falls + * back to string. */ public static DataType inferPartitionColumnType(String value) { if (value == null || HIVE_DEFAULT_PARTITION.equals(value)) { return DataTypes.StringType; } - try { - Integer.parseInt(value); + if (Ints.tryParse(value) != null) { return DataTypes.IntegerType; - } catch (NumberFormatException ignored) { } - try { - Long.parseLong(value); + if (Longs.tryParse(value) != null) { return DataTypes.LongType; - } catch (NumberFormatException ignored) { } - try { - Double.parseDouble(value); + if (Doubles.tryParse(value) != null) { return DataTypes.DoubleType; - } catch (NumberFormatException ignored) { } if ("true".equalsIgnoreCase(value) || "false".equalsIgnoreCase(value)) { return DataTypes.BooleanType; @@ -68,8 +76,8 @@ public static DataType inferPartitionColumnType(String value) { } /** - * Creates a Spark {@link ConstantColumnVector} populated with the given partition value, - * parsed according to the target {@link DataType}. + * Creates a Spark {@link ConstantColumnVector} populated with the given partition value, parsed according to the + * target {@link DataType}. */ public static ConstantColumnVector createConstantVector(int numRows, DataType type, String value) { ConstantColumnVector vec = new ConstantColumnVector(numRows, type); diff --git a/java/vortex-spark/src/main/java/dev/vortex/spark/read/ReaderFactory.java b/java/vortex-spark/src/main/java/dev/vortex/spark/read/ReaderFactory.java deleted file mode 100644 index 044c21dfad3..00000000000 --- a/java/vortex-spark/src/main/java/dev/vortex/spark/read/ReaderFactory.java +++ /dev/null @@ -1,67 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.spark.read; - -import dev.vortex.spark.VortexFilePartition; -import java.io.Serializable; -import org.apache.spark.sql.catalyst.InternalRow; -import org.apache.spark.sql.connector.read.InputPartition; -import org.apache.spark.sql.connector.read.PartitionReader; -import org.apache.spark.sql.connector.read.PartitionReaderFactory; -import org.apache.spark.sql.vectorized.ColumnarBatch; - -/** - * A {@link PartitionReaderFactory} for Vortex file partitions. - *

- * This factory creates partition readers for reading Vortex files. It implements the singleton - * pattern using an enum and only supports columnar reads for optimal performance. - * Row-based reads are not supported as Vortex is designed for columnar data processing. - */ -enum ReaderFactory implements PartitionReaderFactory, Serializable { - INSTANCE; - - private static final boolean SUPPORTS_COLUMNAR_READS = true; - - /** - * Creates a row-based partition reader. - *

- * This method is not supported as Vortex only supports columnar reads for performance reasons. - * - * @param partition the input partition to read from - * @return never returns, always throws an exception - * @throws UnsupportedOperationException always, as row-based reading is not supported - */ - @Override - public PartitionReader createReader(InputPartition partition) { - throw new UnsupportedOperationException("ReaderFactory only supports columnar reads"); - } - - /** - * Creates a columnar partition reader for the given partition. - *

- * This method creates a VortexPartitionReader that can efficiently read columnar data - * from a Vortex file partition. - * - * @param partition the input partition to read from, must be a VortexFilePartition - * @return a partition reader that produces ColumnarBatch objects - * @throws ClassCastException if the partition is not a VortexFilePartition - */ - @Override - public PartitionReader createColumnarReader(InputPartition partition) { - return new VortexPartitionReader((VortexFilePartition) partition); - } - - /** - * Indicates whether this factory supports columnar reads for the given partition. - *

- * Vortex always supports and prefers columnar reads for optimal performance. - * - * @param partition the input partition to check (parameter is ignored) - * @return always true, indicating columnar reads are supported - */ - @Override - public boolean supportColumnarReads(InputPartition partition) { - return SUPPORTS_COLUMNAR_READS; - } -} diff --git a/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexArrowColumnVector.java b/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexArrowColumnVector.java index aabbe36031d..99e45feab72 100644 --- a/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexArrowColumnVector.java +++ b/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexArrowColumnVector.java @@ -67,22 +67,12 @@ public int numNulls() { } /** - * Closes this column vector and releases any associated resources. - *

- * This method recursively closes any child columns (for complex types like structs) - * and then closes the underlying Arrow vector accessor. + * No-op: the underlying Arrow {@link ValueVector}s are owned by the + * {@link dev.vortex.relocated.org.apache.arrow.vector.ipc.ArrowReader} that produced + * this batch and are released when that reader is closed. */ @Override - public void close() { - if (childColumns != null) { - for (int i = 0; i < childColumns.length; i++) { - childColumns[i].close(); - childColumns[i] = null; - } - childColumns = null; - } - accessor.close(); - } + public void close() {} /** * Returns whether the value at the specified row is null. @@ -359,10 +349,6 @@ final int getNullCount() { return vector.getNullCount(); } - final void close() { - vector.close(); - } - boolean getBoolean(int rowId) { throw new UnsupportedOperationException(getClass().getName()); } diff --git a/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexBatchExec.java b/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexBatchExec.java index 199513f1d8a..e32d26f5643 100644 --- a/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexBatchExec.java +++ b/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexBatchExec.java @@ -3,75 +3,89 @@ package dev.vortex.spark.read; -import com.google.common.collect.ImmutableList; import com.google.common.collect.ImmutableMap; -import dev.vortex.jni.NativeFileMethods; +import dev.vortex.jni.NativeFiles; import dev.vortex.spark.VortexFilePartition; +import dev.vortex.spark.VortexSparkSession; +import java.util.Arrays; +import java.util.HashSet; +import java.util.List; import java.util.Map; +import java.util.Set; +import java.util.stream.Collectors; import java.util.stream.Stream; +import org.apache.spark.sql.connector.catalog.CatalogV2Util; import org.apache.spark.sql.connector.catalog.Column; import org.apache.spark.sql.connector.read.Batch; import org.apache.spark.sql.connector.read.InputPartition; import org.apache.spark.sql.connector.read.PartitionReaderFactory; +import org.apache.spark.sql.types.StructType; -/** - * Execution source for batch scans of Vortex file tables. - */ +/** Execution source for batch scans of Vortex file tables. */ public final class VortexBatchExec implements Batch { - private final ImmutableList paths; - private final ImmutableList columns; - private final ImmutableMap formatOptions; + private final List paths; + private final StructType readSchema; + private final Map formatOptions; + private List resolvedPaths; /** * Creates a new VortexBatchExec for scanning the specified Vortex files. * - * @param paths the list of file paths to scan + * @param paths the list of file paths to scan * @param columns the list of columns to read from the files */ - public VortexBatchExec( - ImmutableList paths, ImmutableList columns, ImmutableMap formatOptions) { - this.paths = paths; - this.columns = columns; - this.formatOptions = formatOptions; + public VortexBatchExec(List paths, List columns, Map formatOptions) { + this.paths = List.copyOf(paths); + this.readSchema = CatalogV2Util.v2ColumnsToStructType(columns.toArray(new Column[0])); + this.formatOptions = Map.copyOf(formatOptions); } /** * Plans the input partitions for this batch scan. - *

- * Creates one partition per file path, where each partition is responsible - * for reading data from a single Vortex file. * - * @return an array of InputPartition objects, one per file path + *

Directory-like entries are expanded to concrete {@code .vortex} files. Each resolved file becomes its own + * {@link VortexFilePartition}; the partition carries the paths the reader should open, the requested schema, and + * any Hive-style partition values parsed out of the path. */ @Override public InputPartition[] planInputPartitions() { - // Scan all paths and assign each file its own partition. - // For each discovered file, parse Hive-style partition values from the path. - return paths.stream() - .flatMap(path -> { - if (path.endsWith(".vortex")) { - return Stream.of(path); - } else { - return NativeFileMethods.listVortexFiles(path, formatOptions).stream(); - } - }) + resolvedPaths = resolvePaths(); + return resolvedPaths.stream() .map(path -> { Map partVals = PartitionPathUtils.parsePartitionValues(path); - return new VortexFilePartition(path, columns, formatOptions, ImmutableMap.copyOf(partVals)); + return new VortexFilePartition( + List.of(path), readSchema, formatOptions, ImmutableMap.copyOf(partVals)); }) .toArray(InputPartition[]::new); } - /** - * Creates a factory for creating partition readers. - *

- * Returns a singleton ReaderFactory instance that can create readers - * capable of reading Vortex file partitions. - * - * @return the PartitionReaderFactory for Vortex files - */ @Override public PartitionReaderFactory createReaderFactory() { - return ReaderFactory.INSTANCE; + List files = resolvedPaths != null ? resolvedPaths : resolvePaths(); + Set partitionColumns = collectPartitionColumnNames(files); + List dataColumnNames = Arrays.stream(readSchema.fieldNames()) + .filter(name -> !partitionColumns.contains(name)) + .collect(Collectors.toList()); + return new VortexPartitionReaderFactory(dataColumnNames, formatOptions); + } + + private List resolvePaths() { + var session = VortexSparkSession.get(formatOptions); + return paths.stream() + .flatMap(path -> { + if (path.endsWith(".vortex")) { + return Stream.of(path); + } + return NativeFiles.listFiles(session, path, formatOptions).stream(); + }) + .collect(Collectors.toList()); + } + + private static Set collectPartitionColumnNames(List files) { + Set all = new HashSet<>(); + for (String path : files) { + all.addAll(PartitionPathUtils.parsePartitionValues(path).keySet()); + } + return all; } } diff --git a/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexColumnarBatch.java b/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexColumnarBatch.java deleted file mode 100644 index 8f89423fa03..00000000000 --- a/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexColumnarBatch.java +++ /dev/null @@ -1,59 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.spark.read; - -import dev.vortex.api.Array; -import org.apache.spark.sql.vectorized.ColumnVector; -import org.apache.spark.sql.vectorized.ColumnarBatch; - -/** - * A {@link ColumnarBatch} that returns Vortex-managed memory with Arrow format, shared over the C Data Interface. - */ -public final class VortexColumnarBatch extends ColumnarBatch { - private Array backingArray; - - /** - * Creates a new VortexColumnarBatch with the specified backing array and column vectors. - *

- * The backing array holds the native memory that contains the actual data, - * while the column vectors provide the Spark API for accessing that data. - * - * @param backingArray the Vortex Array that holds the native memory - * @param columns the array of ColumnVector objects for data access - * @param numRows the number of rows in this batch - */ - public VortexColumnarBatch(Array backingArray, ColumnVector[] columns, int numRows) { - super(columns, numRows); - this.backingArray = backingArray; - } - - /** - * Closes this columnar batch and releases all associated resources. - *

- * This method frees the native memory held by the backing Vortex array - * and then delegates to the parent class to close the column vectors. - */ - @Override - public void close() { - freeNativeMemory(); - super.close(); - } - - /** - * Closes this columnar batch if it is freeable and releases all associated resources. - *

- * This method frees the native memory held by the backing Vortex array - * and then delegates to the parent class to close the column vectors if freeable. - */ - @Override - public void closeIfFreeable() { - freeNativeMemory(); - super.closeIfFreeable(); - } - - private void freeNativeMemory() { - backingArray.close(); - backingArray = null; - } -} diff --git a/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexColumnarBatchIterator.java b/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexColumnarBatchIterator.java deleted file mode 100644 index 06ba3237e14..00000000000 --- a/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexColumnarBatchIterator.java +++ /dev/null @@ -1,108 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.spark.read; - -import dev.vortex.api.Array; -import dev.vortex.api.ArrayIterator; -import dev.vortex.arrow.ArrowAllocation; -import dev.vortex.relocated.org.apache.arrow.vector.VectorSchemaRoot; -import java.util.Iterator; -import org.apache.spark.sql.vectorized.ColumnVector; -import org.apache.spark.sql.vectorized.ColumnarBatch; - -/** - * Iterator that converts Vortex Arrays into Spark ColumnarBatch objects. - *

- * This iterator wraps a Vortex ArrayIterator and converts each Array into a Spark ColumnarBatch - * by exporting the data to Arrow format and wrapping it with VortexArrowColumnVector instances. - * The iterator uses prefetching to optimize memory usage and performance by batching arrays - * up to a maximum buffer size. - *

- * The iterator maintains a reusable VectorSchemaRoot to minimize allocation overhead when - * converting between Vortex and Arrow formats. - * - * @see ArrayIterator - * @see ColumnarBatch - * @see VortexArrowColumnVector - */ -public final class VortexColumnarBatchIterator implements Iterator, AutoCloseable { - /** - * Maximum buffer size in bytes for prefetching arrays. - *

- * The iterator will prefetch and batch arrays until this size limit is reached, - * which helps optimize memory usage and reduces the overhead of converting - * small arrays individually. - */ - public static final long MAX_BUFFER_BYTES = 16 * 1024 * 1024; // 16MB - - private final ArrayIterator backing; - private final PrefetchingIterator prefetching; - - // Reusable root - private VectorSchemaRoot root = null; - - /** - * Creates a new VortexColumnarBatchIterator that wraps the given ArrayIterator. - *

- * The iterator will use prefetching to batch arrays up to MAX_BUFFER_BYTES - * to optimize memory usage and conversion performance. - * - * @param backing the underlying ArrayIterator to wrap - */ - public VortexColumnarBatchIterator(ArrayIterator backing) { - this.backing = backing; - this.prefetching = new PrefetchingIterator<>(backing, MAX_BUFFER_BYTES, Array::nbytes); - } - - /** - * Returns whether there are more columnar batches available. - * - * @return true if there are more batches to iterate over, false otherwise - */ - @Override - public boolean hasNext() { - return prefetching.hasNext(); - } - - /** - * Returns the next columnar batch from the iterator. - *

- * This method retrieves the next Array from the prefetching iterator, - * exports it to Arrow format using a reusable VectorSchemaRoot, - * and wraps each field vector in a VortexArrowColumnVector to create - * a VortexColumnarBatch. - * - * @return the next ColumnarBatch containing the data from the next Array - * @throws java.util.NoSuchElementException if there are no more elements - */ - @Override - public ColumnarBatch next() { - Array next = prefetching.next(); - - root = next.exportToArrow(ArrowAllocation.rootAllocator(), root); - - int rowCount = root.getRowCount(); - ColumnVector[] vectors = new ColumnVector[root.getFieldVectors().size()]; - for (int i = 0; i < root.getFieldVectors().size(); i++) { - vectors[i] = new VortexArrowColumnVector(root.getFieldVectors().get(i)); - } - return new VortexColumnarBatch(next, vectors, rowCount); - } - - /** - * Closes this iterator and releases all associated resources. - *

- * This method closes the prefetching iterator, the backing ArrayIterator, - * and the reusable VectorSchemaRoot if it exists. - */ - @Override - public void close() { - this.prefetching.close(); - this.backing.close(); - if (root != null) { - root.close(); - root = null; - } - } -} diff --git a/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexPartitionReader.java b/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexPartitionReader.java index 3904e0272f6..9df44c07e6c 100644 --- a/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexPartitionReader.java +++ b/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexPartitionReader.java @@ -3,157 +3,148 @@ package dev.vortex.spark.read; -import static com.google.common.base.Preconditions.checkNotNull; - -import dev.vortex.api.File; -import dev.vortex.api.Files; +import dev.vortex.api.DataSource; +import dev.vortex.api.Expression; +import dev.vortex.api.Partition; +import dev.vortex.api.Scan; import dev.vortex.api.ScanOptions; +import dev.vortex.api.Session; +import dev.vortex.arrow.ArrowAllocation; +import dev.vortex.relocated.org.apache.arrow.memory.BufferAllocator; +import dev.vortex.relocated.org.apache.arrow.vector.VectorSchemaRoot; +import dev.vortex.relocated.org.apache.arrow.vector.ipc.ArrowReader; import dev.vortex.spark.VortexFilePartition; -import java.util.*; -import org.apache.spark.sql.connector.catalog.Column; +import dev.vortex.spark.VortexSparkSession; +import java.io.IOException; +import java.util.List; +import java.util.Map; import org.apache.spark.sql.connector.read.PartitionReader; +import org.apache.spark.sql.types.StructField; import org.apache.spark.sql.vectorized.ColumnVector; import org.apache.spark.sql.vectorized.ColumnarBatch; /** - * A {@link PartitionReader} that reads columnar batches out of a Vortex file into - * Vortex memory format. - *

- * When reading from partitioned directories, partition column values are extracted from the - * Hive-style file path and materialized as Spark - * {@link org.apache.spark.sql.execution.vectorized.ConstantColumnVector} instances that are - * spliced into each output batch. + * Per-{@link VortexFilePartition} columnar reader. + * + *

Opens a single Vortex {@link Session}, {@link DataSource} and {@link Scan} spanning all of + * {@link VortexFilePartition#paths()} and streams every Vortex partition's record batches through the + * {@link PartitionReader} interface. */ final class VortexPartitionReader implements PartitionReader { - private final VortexFilePartition partition; - - private File file; - private VortexColumnarBatchIterator batches; - - /** Names of columns whose values come from the partition path rather than the data file. */ - private Set partitionColumnNames; - - /** Tracks the last data batch so its native memory can be freed properly. */ - private ColumnarBatch lastDataBatch; - - VortexPartitionReader(VortexFilePartition partition) { - this.partition = partition; - initNativeResources(); + private final VortexFilePartition spark; + private final BufferAllocator allocator; + + // Held so the DataSource/Scan stay reachable even if the JVM-wide singleton is + // ever reset during a task; the actual native session is owned by + // {@link VortexSparkSession} and is not released when this reader closes. + private Session session; + private DataSource dataSource; + private Scan scan; + + private Partition currentPartition; + private ArrowReader currentReader; + private boolean currentBatchLoaded; + private boolean exhausted; + + VortexPartitionReader(VortexFilePartition spark, List dataColumnNames, Map formatOptions) { + this.spark = spark; + this.allocator = ArrowAllocation.rootAllocator(); + + session = VortexSparkSession.get(formatOptions); + dataSource = DataSource.open(session, spark.paths(), formatOptions); + + var options = ScanOptions.builder(); + if (!dataColumnNames.isEmpty()) { + Expression projection = Expression.select(dataColumnNames.toArray(new String[0]), Expression.root()); + options.projection(projection); + } + scan = dataSource.scan(options.build()); } @Override public boolean next() { - checkNotNull(batches, "batches"); - return batches.hasNext(); + if (exhausted) { + return false; + } + while (true) { + if (currentReader != null) { + try { + if (currentReader.loadNextBatch()) { + currentBatchLoaded = true; + return true; + } + } catch (IOException e) { + throw new RuntimeException(e); + } + closeCurrentReader(); + } + if (!scan.hasNext()) { + exhausted = true; + return false; + } + currentPartition = scan.next(); + currentReader = currentPartition.scanArrow(allocator); + } } @Override public ColumnarBatch get() { - checkNotNull(batches, "closed ArrayStream"); - - // Free previous data batch native memory - if (lastDataBatch != null) { - lastDataBatch.close(); - lastDataBatch = null; + if (!currentBatchLoaded) { + throw new IllegalStateException("no batch loaded; call next() first"); } + currentBatchLoaded = false; - ColumnarBatch dataBatch = batches.next(); - - if (partitionColumnNames.isEmpty()) { - return dataBatch; + VectorSchemaRoot root; + try { + root = currentReader.getVectorSchemaRoot(); + } catch (IOException e) { + throw new RuntimeException(e); } - // Track the data batch for lifecycle management - lastDataBatch = dataBatch; - return buildCombinedBatch(dataBatch); - } - - /** - * Builds a combined batch with data columns from the file and constant partition columns - * in the order expected by the full table schema. - */ - private ColumnarBatch buildCombinedBatch(ColumnarBatch dataBatch) { - int rowCount = dataBatch.numRows(); - Map partVals = partition.getPartitionValues(); - List allColumns = partition.getColumns(); - ColumnVector[] combined = new ColumnVector[allColumns.size()]; - - int dataIdx = 0; - for (int i = 0; i < allColumns.size(); i++) { - Column col = allColumns.get(i); - String partValue = partVals.get(col.name()); - if (partValue != null) { - combined[i] = PartitionPathUtils.createConstantVector(rowCount, col.dataType(), partValue); - } else { - combined[i] = dataBatch.column(dataIdx++); + int rowCount = root.getRowCount(); + Map partVals = spark.partitionValues(); + if (partVals.isEmpty()) { + ColumnVector[] vectors = new ColumnVector[root.getFieldVectors().size()]; + for (int i = 0; i < vectors.length; i++) { + vectors[i] = new VortexArrowColumnVector(root.getFieldVectors().get(i)); } + return new ColumnarBatch(vectors, rowCount); } - return new CombinedColumnarBatch(combined, rowCount); - } - - /** - * Initialize the Vortex File and ArrayStream resources. - *

- * Partition columns are identified by matching requested column names against the - * partition values from the file path. Only non-partition columns are pushed down - * to the Vortex scan. - */ - void initNativeResources() { - Map partVals = partition.getPartitionValues(); - this.partitionColumnNames = new HashSet<>(); - - List dataColumnNames = new ArrayList<>(); - for (Column col : partition.getColumns()) { - if (partVals.containsKey(col.name())) { - partitionColumnNames.add(col.name()); + StructField[] fields = spark.readSchema().fields(); + ColumnVector[] combined = new ColumnVector[fields.length]; + int dataIdx = 0; + for (int i = 0; i < fields.length; i++) { + StructField field = fields[i]; + String partValue = partVals.get(field.name()); + if (partValue != null) { + combined[i] = PartitionPathUtils.createConstantVector(rowCount, field.dataType(), partValue); } else { - dataColumnNames.add(col.name()); + combined[i] = new VortexArrowColumnVector(root.getFieldVectors().get(dataIdx++)); } } - - file = Files.open(partition.getPath(), partition.getFormatOptions()); - batches = new VortexColumnarBatchIterator( - file.newScan(ScanOptions.builder().columns(dataColumnNames).build())); + return new ColumnarBatch(combined, rowCount); } @Override public void close() { - if (lastDataBatch != null) { - lastDataBatch.close(); - lastDataBatch = null; - } - - checkNotNull(file, "File was closed"); - checkNotNull(batches, "ArrayStream was closed"); - - batches.close(); - batches = null; - - file.close(); - file = null; + closeCurrentReader(); + // Scan and DataSource native resources are released by VortexCleaner once + // references are dropped. Session is the JVM-wide singleton and outlives this reader. + scan = null; + dataSource = null; + session = null; } - /** - * A ColumnarBatch that does not close its column vectors on {@link #close()}. - *

- * The data column vectors are owned by the underlying {@link VortexColumnarBatch} - * (tracked via {@link #lastDataBatch}), and the constant partition vectors have trivial - * lifecycle. Neither should be closed by this wrapper. - */ - private static final class CombinedColumnarBatch extends ColumnarBatch { - CombinedColumnarBatch(ColumnVector[] columns, int numRows) { - super(columns, numRows); - } - - @Override - public void close() { - // Intentionally empty: lifecycle is managed by VortexPartitionReader - } - - @Override - public void closeIfFreeable() { - // Intentionally empty + private void closeCurrentReader() { + if (currentReader != null) { + try { + currentReader.close(); + } catch (IOException e) { + throw new RuntimeException(e); + } + currentReader = null; } + currentPartition = null; } } diff --git a/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexPartitionReaderFactory.java b/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexPartitionReaderFactory.java new file mode 100644 index 00000000000..9ffbfcc3cfb --- /dev/null +++ b/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexPartitionReaderFactory.java @@ -0,0 +1,51 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +package dev.vortex.spark.read; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import dev.vortex.spark.VortexFilePartition; +import java.io.Serializable; +import java.util.List; +import org.apache.spark.sql.catalyst.InternalRow; +import org.apache.spark.sql.connector.read.InputPartition; +import org.apache.spark.sql.connector.read.PartitionReader; +import org.apache.spark.sql.connector.read.PartitionReaderFactory; +import org.apache.spark.sql.vectorized.ColumnarBatch; + +/** + * Factory that produces columnar readers for Vortex files. + * + *

The set of paths belongs to each {@link VortexFilePartition} — the factory itself is stateless across partitions. + * For every input partition, {@link VortexPartitionReader} opens a single {@code Session}, {@code DataSource} and + * {@code Scan} spanning that partition's paths and consumes every Vortex partition produced by that scan before + * returning. + */ +public final class VortexPartitionReaderFactory implements PartitionReaderFactory, Serializable { + private static final long serialVersionUID = 1L; + + private final ImmutableList dataColumnNames; + private final ImmutableMap formatOptions; + + public VortexPartitionReaderFactory(List dataColumnNames, java.util.Map formatOptions) { + this.dataColumnNames = ImmutableList.copyOf(dataColumnNames); + this.formatOptions = ImmutableMap.copyOf(formatOptions); + } + + @Override + public PartitionReader createReader(InputPartition partition) { + throw new UnsupportedOperationException("row-based reads are not supported"); + } + + @Override + public PartitionReader createColumnarReader(InputPartition partition) { + VortexFilePartition spark = (VortexFilePartition) partition; + return new VortexPartitionReader(spark, dataColumnNames, formatOptions); + } + + @Override + public boolean supportColumnarReads(InputPartition partition) { + return true; + } +} diff --git a/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexScan.java b/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexScan.java index 1ec1a0aeb7d..c6ec03eef80 100644 --- a/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexScan.java +++ b/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexScan.java @@ -3,33 +3,29 @@ package dev.vortex.spark.read; -import com.google.common.collect.ImmutableList; -import com.google.common.collect.ImmutableMap; +import java.util.List; +import java.util.Map; import org.apache.spark.sql.connector.catalog.CatalogV2Util; import org.apache.spark.sql.connector.catalog.Column; import org.apache.spark.sql.connector.read.Batch; import org.apache.spark.sql.connector.read.Scan; import org.apache.spark.sql.types.StructType; -/** - * Spark V2 {@link Scan} over a table of Vortex files. - */ +/** Spark V2 {@link Scan} over a table of Vortex files. */ public final class VortexScan implements Scan { - private final ImmutableList paths; - private final ImmutableList readColumns; - private final ImmutableMap formatOptions; + private final List paths; + private final List readColumns; + private final Map formatOptions; /** - * Creates a new VortexScan for the specified file paths and columns. + * Creates a new VortexScan for the specified file paths and columns. The caller is responsible for passing + * immutable collections; the constructor does not copy. * * @param paths the list of Vortex file paths to scan * @param readColumns the list of columns to read from the files */ - public VortexScan( - ImmutableList paths, - ImmutableList readColumns, - ImmutableMap formatOptions) { + public VortexScan(List paths, List readColumns, Map formatOptions) { this.paths = paths; this.readColumns = readColumns; this.formatOptions = formatOptions; @@ -37,9 +33,8 @@ public VortexScan( /** * Returns the schema for the data that will be read by this scan. - *

- * The schema is constructed from the read columns that were specified - * when this scan was created. + * + *

The schema is constructed from the read columns that were specified when this scan was created. * * @return the StructType representing the schema of the read data */ @@ -48,9 +43,7 @@ public StructType readSchema() { return CatalogV2Util.v2ColumnsToStructType(readColumns.toArray(new Column[0])); } - /** - * Logging-friendly readable description of the scan source. - */ + /** Logging-friendly readable description of the scan source. */ @Override public String description() { return String.format("VortexScan{paths=%s, columns=%s}", paths, readColumns); @@ -58,9 +51,8 @@ public String description() { /** * Converts this scan to a Batch for execution. - *

- * Creates a VortexBatchExec that will handle the actual reading - * of the specified files and columns. + * + *

Creates a VortexBatchExec that will handle the actual reading of the specified files and columns. * * @return a Batch implementation for executing this scan */ @@ -71,9 +63,8 @@ public Batch toBatch() { /** * Returns the columnar support mode for this scan. - *

- * Vortex always provides columnar data access, so this method - * always returns SUPPORTED. + * + *

Vortex always provides columnar data access, so this method always returns SUPPORTED. * * @return ColumnarSupportMode.SUPPORTED */ diff --git a/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexScanBuilder.java b/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexScanBuilder.java index 0ea690b6406..a53472bc33b 100644 --- a/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexScanBuilder.java +++ b/java/vortex-spark/src/main/java/dev/vortex/spark/read/VortexScanBuilder.java @@ -6,7 +6,6 @@ import static com.google.common.base.Preconditions.checkState; import com.google.common.collect.ImmutableList; -import com.google.common.collect.ImmutableMap; import java.util.ArrayList; import java.util.List; import java.util.Map; @@ -17,21 +16,17 @@ import org.apache.spark.sql.types.StructField; import org.apache.spark.sql.types.StructType; -/** - * Spark V2 {@link ScanBuilder} for table scans over Vortex files. - */ +/** Spark V2 {@link ScanBuilder} for table scans over Vortex files. */ public final class VortexScanBuilder implements ScanBuilder, SupportsPushDownRequiredColumns { private final ImmutableList.Builder paths; private final List columns; private final Map formatOptions; - /** - * Creates a new VortexScanBuilder with empty paths and columns. - */ + /** Creates a new VortexScanBuilder with empty paths and columns. */ public VortexScanBuilder(Map formatOptions) { this.paths = ImmutableList.builder(); this.columns = new ArrayList<>(); - this.formatOptions = formatOptions; + this.formatOptions = Map.copyOf(formatOptions); } /** @@ -89,22 +84,19 @@ public VortexScanBuilder addAllColumns(Iterable columns) { @Override public Scan build() { var paths = this.paths.build(); - var columns = ImmutableList.copyOf(this.columns); - var formatOptions = ImmutableMap.copyOf(this.formatOptions); checkState(!paths.isEmpty(), "paths cannot be empty"); // Allow empty columns for operations like count() that don't need actual column data // If no columns are specified, we'll read the minimal schema needed - return new VortexScan(paths, columns, formatOptions); + return new VortexScan(paths, List.copyOf(this.columns), this.formatOptions); } /** * Prunes the columns to only include those specified in the required schema. - *

- * This method clears the current column list and replaces it with columns - * derived from the required schema. Currently only supports top-level schema - * pruning - deeply nested schema pruning is not yet implemented. + * + *

This method clears the current column list and replaces it with columns derived from the required schema. + * Currently only supports top-level schema pruning - deeply nested schema pruning is not yet implemented. * * @param requiredSchema the schema specifying which columns are required */ diff --git a/java/vortex-spark/src/main/java/dev/vortex/spark/write/PartitionedVortexDataWriter.java b/java/vortex-spark/src/main/java/dev/vortex/spark/write/PartitionedVortexDataWriter.java index aa27381242d..01ed570e4fc 100644 --- a/java/vortex-spark/src/main/java/dev/vortex/spark/write/PartitionedVortexDataWriter.java +++ b/java/vortex-spark/src/main/java/dev/vortex/spark/write/PartitionedVortexDataWriter.java @@ -13,7 +13,13 @@ import java.time.LocalDate; import java.time.LocalDateTime; import java.time.ZoneOffset; -import java.util.*; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; import java.util.stream.Collectors; import org.apache.hadoop.shaded.com.google.common.collect.Streams; import org.apache.spark.sql.catalyst.InternalRow; @@ -25,7 +31,21 @@ import org.apache.spark.sql.connector.expressions.Transform; import org.apache.spark.sql.connector.write.DataWriter; import org.apache.spark.sql.connector.write.WriterCommitMessage; -import org.apache.spark.sql.types.*; +import org.apache.spark.sql.types.BinaryType; +import org.apache.spark.sql.types.BooleanType; +import org.apache.spark.sql.types.ByteType; +import org.apache.spark.sql.types.DataType; +import org.apache.spark.sql.types.DateType; +import org.apache.spark.sql.types.DoubleType; +import org.apache.spark.sql.types.FloatType; +import org.apache.spark.sql.types.IntegerType; +import org.apache.spark.sql.types.LongType; +import org.apache.spark.sql.types.ShortType; +import org.apache.spark.sql.types.StringType; +import org.apache.spark.sql.types.StructField; +import org.apache.spark.sql.types.StructType; +import org.apache.spark.sql.types.TimestampNTZType; +import org.apache.spark.sql.types.TimestampType; import org.apache.spark.sql.util.CaseInsensitiveStringMap; import org.apache.spark.unsafe.types.UTF8String; import org.slf4j.Logger; @@ -33,11 +53,10 @@ /** * Writes Spark InternalRow data to Vortex files organized in Hive-style partition directories. - *

- * Supports the standard Spark partition transforms: {@code identity}, {@code years}, - * {@code months}, {@code days}, {@code hours}, and {@code bucket}. For each unique combination - * of evaluated transform values, a separate subdirectory is created and a dedicated - * {@link VortexDataWriter} writes data within it. + * + *

Supports the standard Spark partition transforms: {@code identity}, {@code years}, {@code months}, {@code days}, + * {@code hours}, and {@code bucket}. For each unique combination of evaluated transform values, a separate subdirectory + * is created and a dedicated {@link VortexDataWriter} writes data within it. */ public final class PartitionedVortexDataWriter implements DataWriter, AutoCloseable { private static final Logger logger = LoggerFactory.getLogger(PartitionedVortexDataWriter.class); @@ -57,12 +76,12 @@ public final class PartitionedVortexDataWriter implements DataWriter partitionMessages; @@ -449,9 +465,7 @@ public static final class PartitionedWriterCommitMessage implements WriterCommit this.partitionMessages = partitionMessages; } - /** - * Returns the commit messages from each individual partition writer. - */ + /** Returns the commit messages from each individual partition writer. */ public List getPartitionMessages() { return partitionMessages; } diff --git a/java/vortex-spark/src/main/java/dev/vortex/spark/write/SparkToArrowSchema.java b/java/vortex-spark/src/main/java/dev/vortex/spark/write/SparkToArrowSchema.java index ca6a90384b1..3ed5a56e52f 100644 --- a/java/vortex-spark/src/main/java/dev/vortex/spark/write/SparkToArrowSchema.java +++ b/java/vortex-spark/src/main/java/dev/vortex/spark/write/SparkToArrowSchema.java @@ -12,13 +12,29 @@ import dev.vortex.relocated.org.apache.arrow.vector.types.pojo.Schema; import java.util.ArrayList; import java.util.List; -import org.apache.spark.sql.types.*; +import org.apache.spark.sql.types.ArrayType; +import org.apache.spark.sql.types.BinaryType; +import org.apache.spark.sql.types.BooleanType; +import org.apache.spark.sql.types.ByteType; +import org.apache.spark.sql.types.DataType; +import org.apache.spark.sql.types.DateType; +import org.apache.spark.sql.types.DecimalType; +import org.apache.spark.sql.types.DoubleType; +import org.apache.spark.sql.types.FloatType; +import org.apache.spark.sql.types.IntegerType; +import org.apache.spark.sql.types.LongType; +import org.apache.spark.sql.types.MapType; +import org.apache.spark.sql.types.ShortType; +import org.apache.spark.sql.types.StringType; +import org.apache.spark.sql.types.StructField; +import org.apache.spark.sql.types.StructType; +import org.apache.spark.sql.types.TimestampNTZType; +import org.apache.spark.sql.types.TimestampType; /** * Utility class for converting Spark SQL schemas to Arrow schemas. * - * This enables the conversion of Spark DataFrames to Arrow format - * for writing to Vortex files. + *

This enables the conversion of Spark DataFrames to Arrow format for writing to Vortex files. */ public final class SparkToArrowSchema { @@ -100,6 +116,8 @@ private static ArrowType convertType(DataType sparkType) { return new ArrowType.Date(DateUnit.DAY); } else if (sparkType instanceof TimestampType) { return new ArrowType.Timestamp(TimeUnit.MICROSECOND, "UTC"); + } else if (sparkType instanceof TimestampNTZType) { + return new ArrowType.Timestamp(TimeUnit.MICROSECOND, null); } else if (sparkType instanceof DecimalType) { DecimalType decimal = (DecimalType) sparkType; return new ArrowType.Decimal(decimal.precision(), decimal.scale(), 128); diff --git a/java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexBatchWrite.java b/java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexBatchWrite.java index be0b516e81e..7f96bac4404 100644 --- a/java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexBatchWrite.java +++ b/java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexBatchWrite.java @@ -3,7 +3,8 @@ package dev.vortex.spark.write; -import dev.vortex.jni.NativeFileMethods; +import dev.vortex.jni.NativeFiles; +import dev.vortex.spark.VortexSparkSession; import java.io.IOException; import java.io.Serializable; import java.nio.file.Files; @@ -15,16 +16,20 @@ import java.util.stream.Collectors; import java.util.stream.Stream; import org.apache.spark.sql.connector.expressions.Transform; -import org.apache.spark.sql.connector.write.*; +import org.apache.spark.sql.connector.write.BatchWrite; +import org.apache.spark.sql.connector.write.DataWriterFactory; +import org.apache.spark.sql.connector.write.PhysicalWriteInfo; +import org.apache.spark.sql.connector.write.Write; +import org.apache.spark.sql.connector.write.WriterCommitMessage; import org.apache.spark.sql.types.StructType; import org.slf4j.Logger; import org.slf4j.LoggerFactory; /** * Manages the batch write operation for creating Vortex files. - *

- * This class coordinates the distributed write operation across Spark executors, - * handling the creation of data writers and managing commits/aborts. + * + *

This class coordinates the distributed write operation across Spark executors, handling the creation of data + * writers and managing commits/aborts. */ public final class VortexBatchWrite implements Write, BatchWrite, Serializable { @@ -40,10 +45,10 @@ public final class VortexBatchWrite implements Write, BatchWrite, Serializable { /** * Creates a new VortexBatchWrite. * - * @param outputPath the base path where Vortex files will be written - * @param schema the schema of the data to write - * @param options additional write options - * @param overwrite whether to overwrite existing files + * @param outputPath the base path where Vortex files will be written + * @param schema the schema of the data to write + * @param options additional write options + * @param overwrite whether to overwrite existing files * @param partitionTransforms partition transforms (may be empty) */ VortexBatchWrite( @@ -61,8 +66,8 @@ public final class VortexBatchWrite implements Write, BatchWrite, Serializable { /** * Returns this object as a BatchWrite. - *

- * This method is required by the Write interface to support batch writes. + * + *

This method is required by the Write interface to support batch writes. * * @return this object */ @@ -73,9 +78,9 @@ public BatchWrite toBatch() { /** * Creates a DataWriterFactory for producing data writers on executors. - *

- * This method is called once at the start of the write operation, - * making it the right place to handle overwrite cleanup. + * + *

This method is called once at the start of the write operation, making it the right place to handle overwrite + * cleanup. * * @return a new VortexDataWriterFactory */ @@ -83,9 +88,10 @@ public BatchWrite toBatch() { public DataWriterFactory createBatchWriterFactory(PhysicalWriteInfo info) { // Handle overwrite cleanup BEFORE writing starts if (overwrite) { - var uris = NativeFileMethods.listVortexFiles(outputPath, options); + var session = VortexSparkSession.get(options); + var uris = NativeFiles.listFiles(session, outputPath, options); log.info("truncating table with {} files", uris.size()); - NativeFileMethods.delete(uris.toArray(new String[0]), options); + NativeFiles.delete(session, uris.toArray(new String[0]), options); log.warn("overwrite currently does not do anything for vortex format"); } @@ -94,9 +100,8 @@ public DataWriterFactory createBatchWriterFactory(PhysicalWriteInfo info) { /** * Called when a single data writer task completes successfully. - *

- * This is called for each successful task but individual file commits - * are handled in the data writer itself. + * + *

This is called for each successful task but individual file commits are handled in the data writer itself. * * @param message commit message from a successful data writer task */ @@ -109,9 +114,8 @@ public void onDataWriterCommit(WriterCommitMessage message) { /** * Commits the entire write job after all tasks complete successfully. - *

- * This finalizes the write operation and ensures all Vortex files - * are properly written. + * + *

This finalizes the write operation and ensures all Vortex files are properly written. * * @param messages commit messages from all successful write tasks */ @@ -126,8 +130,8 @@ public void commit(WriterCommitMessage[] messages) { /** * Aborts the write job due to failures. - *

- * This method cleans up any partially written files. + * + *

This method cleans up any partially written files. * * @param messages commit messages from write tasks (may include failures) */ diff --git a/java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexDataWriter.java b/java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexDataWriter.java index ed1c986b9c1..3a4390cb8b6 100644 --- a/java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexDataWriter.java +++ b/java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexDataWriter.java @@ -3,16 +3,31 @@ package dev.vortex.spark.write; +import dev.vortex.api.Session; import dev.vortex.api.VortexWriter; import dev.vortex.relocated.org.apache.arrow.c.ArrowArray; import dev.vortex.relocated.org.apache.arrow.c.ArrowSchema; import dev.vortex.relocated.org.apache.arrow.c.Data; import dev.vortex.relocated.org.apache.arrow.memory.BufferAllocator; import dev.vortex.relocated.org.apache.arrow.memory.RootAllocator; -import dev.vortex.relocated.org.apache.arrow.vector.*; +import dev.vortex.relocated.org.apache.arrow.vector.BigIntVector; +import dev.vortex.relocated.org.apache.arrow.vector.BitVector; +import dev.vortex.relocated.org.apache.arrow.vector.DateDayVector; +import dev.vortex.relocated.org.apache.arrow.vector.DecimalVector; +import dev.vortex.relocated.org.apache.arrow.vector.FieldVector; +import dev.vortex.relocated.org.apache.arrow.vector.Float4Vector; +import dev.vortex.relocated.org.apache.arrow.vector.Float8Vector; +import dev.vortex.relocated.org.apache.arrow.vector.IntVector; +import dev.vortex.relocated.org.apache.arrow.vector.SmallIntVector; +import dev.vortex.relocated.org.apache.arrow.vector.TimeStampMicroTZVector; +import dev.vortex.relocated.org.apache.arrow.vector.TimeStampMicroVector; +import dev.vortex.relocated.org.apache.arrow.vector.TinyIntVector; +import dev.vortex.relocated.org.apache.arrow.vector.VarBinaryVector; +import dev.vortex.relocated.org.apache.arrow.vector.VarCharVector; import dev.vortex.relocated.org.apache.arrow.vector.VectorSchemaRoot; import dev.vortex.relocated.org.apache.arrow.vector.complex.ListVector; -import dev.vortex.spark.SparkTypes; +import dev.vortex.relocated.org.apache.arrow.vector.complex.StructVector; +import dev.vortex.spark.VortexSparkSession; import java.io.IOException; import java.nio.file.Files; import java.nio.file.Paths; @@ -23,7 +38,23 @@ import org.apache.spark.sql.catalyst.util.ArrayData; import org.apache.spark.sql.connector.write.DataWriter; import org.apache.spark.sql.connector.write.WriterCommitMessage; -import org.apache.spark.sql.types.*; +import org.apache.spark.sql.types.ArrayType; +import org.apache.spark.sql.types.BinaryType; +import org.apache.spark.sql.types.BooleanType; +import org.apache.spark.sql.types.ByteType; +import org.apache.spark.sql.types.DataType; +import org.apache.spark.sql.types.DateType; +import org.apache.spark.sql.types.DecimalType; +import org.apache.spark.sql.types.DoubleType; +import org.apache.spark.sql.types.FloatType; +import org.apache.spark.sql.types.IntegerType; +import org.apache.spark.sql.types.LongType; +import org.apache.spark.sql.types.ShortType; +import org.apache.spark.sql.types.StringType; +import org.apache.spark.sql.types.StructField; +import org.apache.spark.sql.types.StructType; +import org.apache.spark.sql.types.TimestampNTZType; +import org.apache.spark.sql.types.TimestampType; import org.apache.spark.sql.util.CaseInsensitiveStringMap; import org.apache.spark.unsafe.types.UTF8String; import org.slf4j.Logger; @@ -31,9 +62,9 @@ /** * Writes Spark InternalRow data to a Vortex file. - *

- * This writer converts Spark's internal row format to Arrow vectors - * and writes them to a Vortex file using the Vortex writer API. + * + *

This writer converts Spark's internal row format to Arrow vectors and writes them to a Vortex file using the + * Vortex writer API. */ public final class VortexDataWriter implements DataWriter, AutoCloseable { private static final Logger logger = LoggerFactory.getLogger(VortexDataWriter.class); @@ -47,6 +78,7 @@ public final class VortexDataWriter implements DataWriter, AutoClos private final CaseInsensitiveStringMap options; private final int batchSize; + private Session session; private VortexWriter vortexWriter; private BufferAllocator allocator; private VectorSchemaRoot vectorSchemaRoot; @@ -59,8 +91,8 @@ public final class VortexDataWriter implements DataWriter, AutoClos * Creates a new VortexDataWriter. * * @param filePath the path where the Vortex file will be written - * @param schema the schema of the data to write - * @param options additional write options + * @param schema the schema of the data to write + * @param options additional write options */ VortexDataWriter(String filePath, StructType schema, CaseInsensitiveStringMap options) { this.filePath = filePath; @@ -87,17 +119,12 @@ public final class VortexDataWriter implements DataWriter, AutoClos } try { - // Initialize Arrow components this.allocator = new RootAllocator(); - - // Convert Spark schema to Vortex and Arrow schemas. - var writeSchema = SparkTypes.toDType(schema); var arrowSchema = SparkToArrowSchema.convert(schema); - // Create Vortex writer - this.vortexWriter = VortexWriter.create(filePath, writeSchema, options.asCaseSensitiveMap()); - - // Create VectorSchemaRoot for batching rows + this.session = VortexSparkSession.get(options.asCaseSensitiveMap()); + this.vortexWriter = + VortexWriter.create(session, filePath, arrowSchema, options.asCaseSensitiveMap(), allocator); this.vectorSchemaRoot = VectorSchemaRoot.create(arrowSchema, allocator); logger.debug("Initialized VortexDataWriter for {}", filePath); @@ -110,8 +137,8 @@ public final class VortexDataWriter implements DataWriter, AutoClos /** * Writes a single row to the Vortex file. - *

- * Rows are batched and converted to Arrow format before writing. + * + *

Rows are batched and converted to Arrow format before writing. * * @param row the row to write * @throws IOException if writing fails @@ -128,9 +155,7 @@ public void write(InternalRow row) throws IOException { } } - /** - * Writes the current batch of rows to the Vortex file. - */ + /** Writes the current batch of rows to the Vortex file. */ private void writeBatch() throws IOException { if (batchRows.isEmpty()) { return; @@ -169,16 +194,14 @@ private void writeBatch() throws IOException { try (ArrowArray arrowArray = ArrowArray.allocateNew(allocator); ArrowSchema arrowSchema = ArrowSchema.allocateNew(allocator)) { Data.exportVectorSchemaRoot(allocator, vectorSchemaRoot, null, arrowArray, arrowSchema); - vortexWriter.writeBatchFfi(arrowArray.memoryAddress(), arrowSchema.memoryAddress()); + vortexWriter.writeBatch(arrowArray.memoryAddress(), arrowSchema.memoryAddress()); } vectorSchemaRoot.clear(); batchRows.clear(); } - /** - * Populates an Arrow vector with a value from an InternalRow. - */ + /** Populates an Arrow vector with a value from an InternalRow. */ private void populateVector( FieldVector vector, DataType dataType, SpecializedGetters row, int fieldIndex, int rowIndex) { if (dataType instanceof BooleanType) { @@ -205,16 +228,23 @@ private void populateVector( if (bytes != null) { ((VarBinaryVector) vector).setSafe(rowIndex, bytes); } - } else if (dataType instanceof DecimalType) { - DecimalType decType = (DecimalType) dataType; + } else if (dataType instanceof DateType) { + ((DateDayVector) vector).setSafe(rowIndex, row.getInt(fieldIndex)); + } else if (dataType instanceof TimestampType) { + ((TimeStampMicroTZVector) vector).setSafe(rowIndex, row.getLong(fieldIndex)); + } else if (dataType instanceof TimestampNTZType) { + ((TimeStampMicroVector) vector).setSafe(rowIndex, row.getLong(fieldIndex)); + } else if (dataType instanceof DecimalType decType) { if (decType.precision() <= 38) { // Use Decimal type from InternalRow java.math.BigDecimal decimal = row.getDecimal(fieldIndex, decType.precision(), decType.scale()) .toJavaBigDecimal(); ((DecimalVector) vector).setSafe(rowIndex, decimal); } - } else if (dataType instanceof ArrayType) { - ArrayType arrayType = (ArrayType) dataType; + } else if (dataType instanceof StructType structType) { + populateStructVector( + (StructVector) vector, structType, row.getStruct(fieldIndex, structType.fields().length), rowIndex); + } else if (dataType instanceof ArrayType arrayType) { ArrayData data = row.getArray(fieldIndex); ListVector listVector = ((ListVector) vector); int writtenElements = listVector.getElementEndIndex(listVector.getLastSet()); @@ -229,10 +259,24 @@ private void populateVector( } } + private void populateStructVector(StructVector vector, StructType dataType, InternalRow row, int rowIndex) { + vector.setIndexDefined(rowIndex); + + StructField[] fields = dataType.fields(); + for (int fieldIndex = 0; fieldIndex < fields.length; fieldIndex++) { + FieldVector childVector = (FieldVector) vector.getVectorById(fieldIndex); + if (row.isNullAt(fieldIndex)) { + childVector.setNull(rowIndex); + continue; + } + populateVector(childVector, fields[fieldIndex].dataType(), row, fieldIndex, rowIndex); + } + } + /** * Commits the write operation and returns a commit message. - *

- * This flushes any remaining rows and closes the Vortex writer. + * + *

This flushes any remaining rows and closes the Vortex writer. * * @return a commit message with file information * @throws IOException if commit fails @@ -275,7 +319,7 @@ public WriterCommitMessage commit() throws IOException { } // The Arrow C Data Interface export (Data.exportVectorSchemaRoot) creates structural - // allocations from this allocator. When writeBatchFfi passes the ArrowArray to Rust, + // allocations from this allocator. When writeBatch passes the ArrowArray to Rust, // FFI_ArrowArray::from_raw() takes ownership and nullifies the release callback on // the Java side. The Rust side calls release asynchronously on its own thread, so // small structural allocations may still be outstanding when the allocator is closed. @@ -289,6 +333,10 @@ public WriterCommitMessage commit() throws IOException { allocator = null; } + // Session is the JVM-wide singleton held by VortexSparkSession; we just + // drop our local handle to it here. + session = null; + closed = true; // Throw any exception that occurred during cleanup @@ -302,8 +350,8 @@ public WriterCommitMessage commit() throws IOException { /** * Aborts the write operation and cleans up resources. - *

- * This deletes any partially written file. + * + *

This deletes any partially written file. * * @throws IOException if abort fails */ @@ -335,6 +383,10 @@ public void abort() throws IOException { allocator = null; } + // Session is the JVM-wide singleton held by VortexSparkSession; we just + // drop our local handle to it here. + session = null; + // Delete the partial file if it exists try { Files.deleteIfExists(Paths.get(filePath)); @@ -348,9 +400,9 @@ public void abort() throws IOException { /** * Closes the writer and releases resources. - *

- * This method ensures resources are cleaned up even if commit() or abort() - * were not called, making the class safe for use with try-with-resources. + * + *

This method ensures resources are cleaned up even if commit() or abort() were not called, making the class + * safe for use with try-with-resources. */ @Override public void close() throws IOException { diff --git a/java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexDataWriterFactory.java b/java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexDataWriterFactory.java index 4cc68736a51..e8237bde309 100644 --- a/java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexDataWriterFactory.java +++ b/java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexDataWriterFactory.java @@ -15,11 +15,9 @@ /** * Factory for creating VortexDataWriter instances on Spark executors. - *

- * This factory is serialized and sent to executors where it creates - * data writers for each task. When partition transforms are specified, - * it creates partitioned writers that organize output into Hive-style - * partition directories. + * + *

This factory is serialized and sent to executors where it creates data writers for each task. When partition + * transforms are specified, it creates partitioned writers that organize output into Hive-style partition directories. */ public final class VortexDataWriterFactory implements DataWriterFactory, Serializable { @@ -34,9 +32,9 @@ public final class VortexDataWriterFactory implements DataWriterFactory, Seriali /** * Creates a new VortexDataWriterFactory. * - * @param outputUri the base path where Vortex files will be written - * @param schema the schema of the data to write - * @param options additional write options + * @param outputUri the base path where Vortex files will be written + * @param schema the schema of the data to write + * @param options additional write options * @param resolvedTransforms pre-resolved partition transforms (may be empty) */ VortexDataWriterFactory( @@ -52,13 +50,12 @@ public final class VortexDataWriterFactory implements DataWriterFactory, Seriali /** * Creates a new data writer for a specific partition and task. - *

- * Each task writes its data to a separate Vortex file to avoid conflicts. - * When partition transforms are configured, returns a {@link PartitionedVortexDataWriter} - * that creates Hive-style partition directories. + * + *

Each task writes its data to a separate Vortex file to avoid conflicts. When partition transforms are + * configured, returns a {@link PartitionedVortexDataWriter} that creates Hive-style partition directories. * * @param partitionId the partition ID - * @param taskId the task ID + * @param taskId the task ID * @return a new DataWriter instance */ @Override diff --git a/java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexWriteBuilder.java b/java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexWriteBuilder.java index 8fe910ad370..921e586a910 100644 --- a/java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexWriteBuilder.java +++ b/java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexWriteBuilder.java @@ -12,9 +12,9 @@ /** * Builder for configuring Vortex write operations. - *

- * This class is responsible for creating BatchWrite instances that execute - * the actual write operations to create Vortex files from Spark DataFrames. + * + *

This class is responsible for creating BatchWrite instances that execute the actual write operations to create + * Vortex files from Spark DataFrames. */ public final class VortexWriteBuilder implements WriteBuilder, SupportsTruncate { @@ -27,9 +27,9 @@ public final class VortexWriteBuilder implements WriteBuilder, SupportsTruncate /** * Creates a new VortexWriteBuilder. * - * @param paths root path for write - * @param writeInfo logical information about the write operation - * @param options additional write options + * @param paths root path for write + * @param writeInfo logical information about the write operation + * @param options additional write options * @param partitionTransforms partition transforms (may be empty) */ public VortexWriteBuilder( @@ -52,9 +52,8 @@ public Write build() { /** * Configures the write operation to truncate existing data. - *

- * When truncate is enabled, existing Vortex files at the output path - * will be removed before writing new data. + * + *

When truncate is enabled, existing Vortex files at the output path will be removed before writing new data. * * @return this builder for method chaining */ diff --git a/java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexWriterCommitMessage.java b/java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexWriterCommitMessage.java index 911819106e7..c081d261dc3 100644 --- a/java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexWriterCommitMessage.java +++ b/java/vortex-spark/src/main/java/dev/vortex/spark/write/VortexWriterCommitMessage.java @@ -8,9 +8,8 @@ /** * Commit message containing information about a successfully written Vortex file. - *

- * This message is passed from executors back to the driver to coordinate - * the commit phase of the write operation. + * + *

This message is passed from executors back to the driver to coordinate the commit phase of the write operation. */ public record VortexWriterCommitMessage(String filePath, long recordCount, long bytesWritten) implements WriterCommitMessage, Serializable { @@ -18,8 +17,8 @@ public record VortexWriterCommitMessage(String filePath, long recordCount, long /** * Creates a new commit message for a written Vortex file. * - * @param filePath the path to the written file - * @param recordCount the number of records written + * @param filePath the path to the written file + * @param recordCount the number of records written * @param bytesWritten the number of bytes written */ public VortexWriterCommitMessage {} diff --git a/java/vortex-spark/src/test/java/dev/vortex/spark/SparkTypesTest.java b/java/vortex-spark/src/test/java/dev/vortex/spark/SparkTypesTest.java deleted file mode 100644 index 6c80ab6b694..00000000000 --- a/java/vortex-spark/src/test/java/dev/vortex/spark/SparkTypesTest.java +++ /dev/null @@ -1,44 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -package dev.vortex.spark; - -import static org.junit.jupiter.api.Assertions.*; - -import dev.vortex.api.DType; -import dev.vortex.jni.NativeLoader; -import org.apache.spark.sql.types.ArrayType; -import org.apache.spark.sql.types.DataTypes; -import org.junit.jupiter.api.BeforeAll; -import org.junit.jupiter.api.DisplayName; -import org.junit.jupiter.api.Test; - -public final class SparkTypesTest { - - @BeforeAll - public static void loadLibrary() { - NativeLoader.loadJni(); - } - - @Test - @DisplayName("toDataType should convert FIXED_SIZE_LIST to Spark ArrayType") - public void testFixedSizeListToDataType() { - var elementType = DType.newInt(false); - var fslType = DType.newFixedSizeList(elementType, 3, true); - var sparkType = SparkTypes.toDataType(fslType); - assertInstanceOf(ArrayType.class, sparkType); - ArrayType arrayType = (ArrayType) sparkType; - assertEquals(DataTypes.IntegerType, arrayType.elementType()); - } - - @Test - @DisplayName("toDataType should convert LIST to Spark ArrayType") - public void testListToDataType() { - var elementType = DType.newDouble(false); - var listType = DType.newList(elementType, true); - var sparkType = SparkTypes.toDataType(listType); - assertInstanceOf(ArrayType.class, sparkType); - ArrayType arrayType = (ArrayType) sparkType; - assertEquals(DataTypes.DoubleType, arrayType.elementType()); - } -} diff --git a/java/vortex-spark/src/test/java/dev/vortex/spark/VortexDataSourceBasicTest.java b/java/vortex-spark/src/test/java/dev/vortex/spark/VortexDataSourceBasicTest.java index 81b0ae3613a..f22246d43f2 100644 --- a/java/vortex-spark/src/test/java/dev/vortex/spark/VortexDataSourceBasicTest.java +++ b/java/vortex-spark/src/test/java/dev/vortex/spark/VortexDataSourceBasicTest.java @@ -3,18 +3,20 @@ package dev.vortex.spark; -import static org.junit.jupiter.api.Assertions.*; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; +import static org.junit.jupiter.api.Assertions.assertNotNull; import dev.vortex.relocated.org.apache.arrow.vector.types.pojo.ArrowType; +import dev.vortex.spark.write.SparkToArrowSchema; +import dev.vortex.spark.write.VortexWriterCommitMessage; import org.apache.spark.sql.types.DataTypes; import org.apache.spark.sql.types.StructField; import org.apache.spark.sql.types.StructType; import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; -/** - * Unit tests for VortexDataSourceV2 basic functionality. - */ +/** Unit tests for VortexDataSourceV2 basic functionality. */ public final class VortexDataSourceBasicTest { @Test @@ -36,7 +38,7 @@ public void testSparkToArrowSchemaConversion() { }); // Convert to Arrow schema - var arrowSchema = dev.vortex.spark.write.SparkToArrowSchema.convert(sparkSchema); + var arrowSchema = SparkToArrowSchema.convert(sparkSchema); // Verify conversion assertNotNull(arrowSchema, "Arrow schema should not be null"); @@ -66,7 +68,7 @@ public void testNestedSparkToArrowSchemaConversion() { }); // Convert to Arrow schema - var arrowSchema = dev.vortex.spark.write.SparkToArrowSchema.convert(sparkSchema); + var arrowSchema = SparkToArrowSchema.convert(sparkSchema); // Verify conversion assertNotNull(arrowSchema, "Arrow schema should not be null"); @@ -98,7 +100,7 @@ public void testWriterCommitMessage() { long recordCount = 1000; long bytesWritten = 50000; - var message = new dev.vortex.spark.write.VortexWriterCommitMessage(testPath, recordCount, bytesWritten); + var message = new VortexWriterCommitMessage(testPath, recordCount, bytesWritten); assertEquals(testPath, message.filePath()); assertEquals(recordCount, message.recordCount()); diff --git a/java/vortex-spark/src/test/java/dev/vortex/spark/VortexDataSourceS3MockTest.java b/java/vortex-spark/src/test/java/dev/vortex/spark/VortexDataSourceS3MockTest.java index 04c94546013..a324e64478f 100644 --- a/java/vortex-spark/src/test/java/dev/vortex/spark/VortexDataSourceS3MockTest.java +++ b/java/vortex-spark/src/test/java/dev/vortex/spark/VortexDataSourceS3MockTest.java @@ -13,15 +13,19 @@ import org.apache.spark.sql.SparkSession; import org.apache.spark.sql.types.StructField; import org.apache.spark.sql.types.StructType; -import org.junit.jupiter.api.*; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.TestInstance; import org.testcontainers.junit.jupiter.Container; import org.testcontainers.junit.jupiter.Testcontainers; /** * Integration test for Vortex DataSource with mocked S3 using Adobe S3Mock. * - *

This test verifies that Vortex can correctly read and write files from S3-compatible - * storage by using S3Mock running as a Testcontainer. + *

This test verifies that Vortex can correctly read and write files from S3-compatible storage by using S3Mock + * running as a Testcontainer. */ @Testcontainers @TestInstance(TestInstance.Lifecycle.PER_CLASS) @@ -45,7 +49,9 @@ public void setUp() { .master("local[2]") .config("spark.sql.shuffle.partitions", "2") .config("spark.sql.adaptive.enabled", "false") + .config("spark.driver.host", "127.0.0.1") .config("spark.ui.enabled", "false") + .config("spark.driver.host", "127.0.0.1") // S3A configuration for S3Mock. // This should be propagated into our reader .config("spark.hadoop.fs.s3a.endpoint", s3Endpoint) @@ -123,10 +129,7 @@ public void testWriteAndReadWithFormatOptionsFromS3Mock() { verifyDataContent(originalDf, readDf); } - /** - * Creates a test DataFrame with monotonically increasing integers - * and their string representations. - */ + /** Creates a test DataFrame with monotonically increasing integers and their string representations. */ private Dataset createTestDataFrame(int numRows) { return spark.range(0, numRows) .selectExpr( @@ -135,9 +138,7 @@ private Dataset createTestDataFrame(int numRows) { "array('Alpha', 'Bravo', 'Charlie') AS elements"); } - /** - * Verifies that two schemas are equal. - */ + /** Verifies that two schemas are equal. */ private void assertSchemaEquals(StructType expected, StructType actual) { assertEquals(expected.fields().length, actual.fields().length, "Schemas should have same number of fields"); @@ -157,9 +158,7 @@ private void assertSchemaEquals(StructType expected, StructType actual) { } } - /** - * Verifies that the data content of two DataFrames is identical. - */ + /** Verifies that the data content of two DataFrames is identical. */ private void verifyDataContent(Dataset expected, Dataset actual) { // Sort both DataFrames by id to ensure consistent ordering Dataset expectedSorted = expected.orderBy("id"); diff --git a/java/vortex-spark/src/test/java/dev/vortex/spark/VortexDataSourceWriteTest.java b/java/vortex-spark/src/test/java/dev/vortex/spark/VortexDataSourceWriteTest.java index 6c5686fa93c..325da0bc98c 100644 --- a/java/vortex-spark/src/test/java/dev/vortex/spark/VortexDataSourceWriteTest.java +++ b/java/vortex-spark/src/test/java/dev/vortex/spark/VortexDataSourceWriteTest.java @@ -3,7 +3,10 @@ package dev.vortex.spark; +import static org.junit.jupiter.api.Assertions.assertDoesNotThrow; import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertInstanceOf; import static org.junit.jupiter.api.Assertions.assertTrue; import java.io.IOException; @@ -14,21 +17,27 @@ import java.util.List; import java.util.stream.Collectors; import java.util.stream.Stream; -import org.apache.spark.sql.*; +import org.apache.spark.sql.Dataset; +import org.apache.spark.sql.Row; +import org.apache.spark.sql.RowFactory; +import org.apache.spark.sql.SaveMode; +import org.apache.spark.sql.SparkSession; import org.apache.spark.sql.types.DataTypes; import org.apache.spark.sql.types.StructField; import org.apache.spark.sql.types.StructType; -import org.junit.jupiter.api.*; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.TestInstance; import org.junit.jupiter.api.io.TempDir; /** * Integration test for Vortex DataSource write and read functionality. - *

- * This test verifies that: - * 1. Spark DataFrames can be written as Vortex files - * 2. Multiple partitions create multiple files - * 3. Data can be read back correctly - * 4. Schema is preserved during write/read + * + *

This test verifies that: 1. Spark DataFrames can be written as Vortex files 2. Multiple partitions create multiple + * files 3. Data can be read back correctly 4. Schema is preserved during write/read */ @TestInstance(TestInstance.Lifecycle.PER_CLASS) public final class VortexDataSourceWriteTest { @@ -44,6 +53,7 @@ public void setUp() { spark = SparkSession.builder() .appName("VortexWriteTest") .master("local[2]") // Use 2 threads + .config("spark.driver.host", "127.0.0.1") .config("spark.sql.shuffle.partitions", "2") .config("spark.sql.adaptive.enabled", "false") // Disable AQE for predictable partitioning .config("spark.ui.enabled", "false") // Disable UI for tests @@ -212,8 +222,8 @@ public void testPartitionedWrite() throws IOException { // Verify vortex files inside partition directories List filesA = findVortexFiles(outputPath.resolve("group=A")); List filesB = findVortexFiles(outputPath.resolve("group=B")); - assertTrue(!filesA.isEmpty(), "Partition A should have vortex files"); - assertTrue(!filesB.isEmpty(), "Partition B should have vortex files"); + assertFalse(filesA.isEmpty(), "Partition A should have vortex files"); + assertFalse(filesB.isEmpty(), "Partition B should have vortex files"); // When: read back Dataset readDf = spark.read() @@ -324,10 +334,74 @@ public void testSpecialCharactersAndNulls() throws IOException { assertEquals("special!@#$%^&*()", specialRows.first().getString(1)); } - /** - * Creates a test DataFrame with monotonically increasing integers - * and their string representations. - */ + @Test + @DisplayName("Write and read date, timestamp, and nested struct columns") + public void testWriteAndReadTemporalAndStructColumns() throws IOException { + Dataset originalDf = spark.range(0, 2) + .selectExpr( + "cast(id as int) as id", + "CASE WHEN id = 0 THEN CAST('2024-01-02' AS DATE) ELSE CAST('2024-02-03' AS DATE) END AS event_date", + """ + CASE WHEN id = 0 THEN CAST('2024-01-02 03:04:05.123456' AS TIMESTAMP) + ELSE CAST('2024-02-03 04:05:06.654321' AS TIMESTAMP) END AS event_ts""", + """ + named_struct( + 'event_date', CASE WHEN id = 0 THEN CAST('2024-01-02' AS DATE) ELSE CAST('2024-02-03' AS DATE) END, + 'event_ts', CASE WHEN id = 0 THEN CAST('2024-01-02 03:04:05.123456' AS TIMESTAMP) + ELSE CAST('2024-02-03 04:05:06.654321' AS TIMESTAMP) END, + 'label', CASE WHEN id = 0 THEN 'alpha' ELSE 'beta' END + ) AS payload"""); + + Path outputPath = tempDir.resolve("temporal_struct_output"); + originalDf + .write() + .format("vortex") + .option("path", outputPath.toUri().toString()) + .mode(SaveMode.Overwrite) + .save(); + + Dataset readDf = spark.read() + .format("vortex") + .option("path", outputPath.toUri().toString()) + .load(); + + List expectedRows = List.of( + "{\"id\":0,\"event_date\":\"2024-01-02\",\"event_ts\":\"2024-01-02 03:04:05.123456\"," + + "\"payload_event_date\":\"2024-01-02\",\"payload_event_ts\":\"2024-01-02 03:04:05.123456\"," + + "\"payload_label\":\"alpha\"}", + "{\"id\":1,\"event_date\":\"2024-02-03\",\"event_ts\":\"2024-02-03 04:05:06.654321\"," + + "\"payload_event_date\":\"2024-02-03\",\"payload_event_ts\":\"2024-02-03 04:05:06.654321\"," + + "\"payload_label\":\"beta\"}"); + + assertEquals(DataTypes.DateType, readDf.schema().fields()[1].dataType()); + assertEquals(DataTypes.TimestampType, readDf.schema().fields()[2].dataType()); + assertInstanceOf(StructType.class, readDf.schema().fields()[3].dataType()); + assertEquals(expectedRows, projectTemporalAndStructRows(readDf)); + } + + @Test + @DisplayName("Write TimestampNTZ columns and nested structs") + public void testWriteTimestampNtzColumns() throws IOException { + Dataset timestampNtzDf = spark.range(0, 2).selectExpr("cast(id as int) as id", """ + CASE WHEN id = 0 THEN CAST('2024-01-02 03:04:05.123456' AS TIMESTAMP_NTZ) + ELSE CAST(NULL AS TIMESTAMP_NTZ) END AS event_ntz""", """ + named_struct( + 'event_ntz', CASE WHEN id = 0 THEN CAST('2024-01-02 03:04:05.123456' AS TIMESTAMP_NTZ) + ELSE CAST('2024-02-03 04:05:06.654321' AS TIMESTAMP_NTZ) END + ) AS payload"""); + + Path outputPath = tempDir.resolve("timestamp_ntz_output"); + assertDoesNotThrow(() -> timestampNtzDf + .write() + .format("vortex") + .option("path", outputPath.toUri().toString()) + .mode(SaveMode.Overwrite) + .save()); + + assertFalse(findVortexFiles(outputPath).isEmpty(), "TimestampNTZ write should create Vortex files"); + } + + /** Creates a test DataFrame with monotonically increasing integers and their string representations. */ private Dataset createTestDataFrame(int numRows) { // Create DataFrame with monotonically increasing integers return spark.range(0, numRows) @@ -337,12 +411,24 @@ private Dataset createTestDataFrame(int numRows) { "array('Alpha', 'Bravo', 'Charlie') AS elements"); } - /** - * Finds all Vortex files in the given directory. - */ + private List projectTemporalAndStructRows(Dataset df) { + return df.orderBy("id").selectExpr(""" + to_json(named_struct( + 'id', id, + 'event_date', cast(event_date as string), + 'event_ts', date_format(event_ts, 'yyyy-MM-dd HH:mm:ss.SSSSSS'), + 'payload_event_date', cast(payload.event_date as string), + 'payload_event_ts', date_format(payload.event_ts, 'yyyy-MM-dd HH:mm:ss.SSSSSS'), + 'payload_label', payload.label + )) as json""").collectAsList().stream() + .map(row -> row.getString(0)) + .collect(Collectors.toList()); + } + + /** Finds all Vortex files in the given directory. */ private List findVortexFiles(Path directory) throws IOException { if (!Files.exists(directory)) { - return Arrays.asList(); + return List.of(); } try (Stream paths = Files.walk(directory)) { @@ -353,9 +439,7 @@ private List findVortexFiles(Path directory) throws IOException { } } - /** - * Verifies that two schemas are equal. - */ + /** Verifies that two schemas are equal. */ private void assertSchemaEquals(StructType expected, StructType actual) { assertEquals(expected.fields().length, actual.fields().length, "Schemas should have same number of fields"); @@ -375,9 +459,7 @@ private void assertSchemaEquals(StructType expected, StructType actual) { } } - /** - * Verifies that the data content of two DataFrames is identical. - */ + /** Verifies that the data content of two DataFrames is identical. */ private void verifyDataContent(Dataset expected, Dataset actual) { // Sort both DataFrames by id to ensure consistent ordering Dataset expectedSorted = expected.orderBy("id"); diff --git a/renovate.json b/renovate.json index 9f3ceea4caf..80fb4c20238 100644 --- a/renovate.json +++ b/renovate.json @@ -2,8 +2,7 @@ "$schema": "https://docs.renovatebot.com/renovate-schema.json", "extends": [ "config:recommended", - ":automergePatch", - ":automergeMinor", + ":automergeStableNonMajor", ":automergePr", ":automergeRequireAllStatusChecks", ":combinePatchMinorReleases", @@ -11,6 +10,7 @@ ":separateMultipleMajorReleases", ":configMigration", "group:rust-futuresMonorepo", + "helpers:pinGitHubActionDigests", "schedule:earlyMondays", "docker:disable" ], @@ -19,6 +19,8 @@ }, "autoApprove": true, "automergeStrategy": "squash", + "prHourlyLimit": 0, + "prConcurrentLimit": 20, "rebaseWhen": "conflicted", "cloneSubmodules": true, "platformAutomerge": true, @@ -58,6 +60,36 @@ "matchSourceUrls": [ "https://github.com/hyperium/tonic" ] + }, + { + "groupName": "Rust lock file maintenance", + "groupSlug": "rust-lock-file-maintenance", + "matchManagers": [ + "cargo" + ], + "matchUpdateTypes": [ + "lockFileMaintenance" + ] + }, + { + "groupName": "Python lock file maintenance", + "groupSlug": "python-lock-file-maintenance", + "matchManagers": [ + "pep621" + ], + "matchUpdateTypes": [ + "lockFileMaintenance" + ] + }, + { + "groupName": "JS lock file maintenance", + "groupSlug": "js-lock-file-maintenance", + "matchManagers": [ + "npm" + ], + "matchUpdateTypes": [ + "lockFileMaintenance" + ] } ], "customManagers": [ @@ -75,4 +107,4 @@ "extractVersionTemplate": "^v?(?.*)$" } ] -} \ No newline at end of file +} diff --git a/results.json b/results.json new file mode 100644 index 00000000000..e69de29bb2d diff --git a/rust-toolchain.toml b/rust-toolchain.toml index c8eba8175ae..dbcc5a2f04e 100644 --- a/rust-toolchain.toml +++ b/rust-toolchain.toml @@ -1,4 +1,4 @@ [toolchain] -channel = "1.90" +channel = "1.91.0" components = ["rust-src", "rustfmt", "clippy", "rust-analyzer"] -profile = "minimal" \ No newline at end of file +profile = "minimal" diff --git a/scripts/bench-taskset.sh b/scripts/bench-taskset.sh index a86f6818ab3..4b8d362dd9c 100644 --- a/scripts/bench-taskset.sh +++ b/scripts/bench-taskset.sh @@ -18,12 +18,17 @@ fi if [[ -z "${BENCH_CPUS:-}" ]]; then - cpu_count="$(nproc)" - BENCH_CPUS="2-$((cpu_count - 1))" + if command -v numactl >/dev/null 2>&1; then + # All CPUs on NUMA node 0, skipping CPUs 0-1 to avoid OS interference + BENCH_CPUS=$(numactl --hardware | awk '/^node 0 cpus:/{sep=""; for(i=4;i<=NF;i++){if($i+0>1){printf "%s%s",sep,$i; sep=","}}}') + else + cpu_count="$(nproc)" + BENCH_CPUS="2-$((cpu_count - 1))" + fi fi if command -v numactl >/dev/null 2>&1; then - exec numactl --physcpubind="$BENCH_CPUS" --localalloc "$@" + exec numactl --physcpubind="$BENCH_CPUS" --membind=0 "$@" fi exec taskset -c "$BENCH_CPUS" "$@" diff --git a/uv.lock b/uv.lock index c5ec5b0d865..1c06ece4294 100644 --- a/uv.lock +++ b/uv.lock @@ -597,23 +597,23 @@ wheels = [ [[package]] name = "maturin" -version = "1.9.6" +version = "1.13.1" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/9a/35/c3370188492f4c139c7a318f438d01b8185c216303c49c4bc885c98b6afb/maturin-1.9.6.tar.gz", hash = "sha256:2c2ae37144811d365509889ed7220b0598487f1278c2441829c3abf56cc6324a", size = 214846, upload-time = "2025-10-07T12:45:08.408Z" } +sdist = { url = "https://files.pythonhosted.org/packages/39/16/b284a7bc4af3dd87717c784278c1b8cb18606ad1f6f7a671c47bfd9c3df0/maturin-1.13.1.tar.gz", hash = "sha256:9a87ff3b8e4d1c6eac33ebfe8e261e8236516d98d45c0323550621819b5a1a2f", size = 340369, upload-time = "2026-04-09T15:14:07.026Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/55/5c/b435418ba4ba2647a1f7a95d53314991b1e556e656ae276dea993c3bce1d/maturin-1.9.6-py3-none-linux_armv6l.whl", hash = "sha256:26e3ab1a42a7145824210e9d763f6958f2c46afb1245ddd0bab7d78b1f59bb3f", size = 8134483, upload-time = "2025-10-07T12:44:44.274Z" }, - { url = "https://files.pythonhosted.org/packages/4d/1c/8e58eda6601f328b412cdeeaa88a9b6a10e591e2a73f313e8c0154d68385/maturin-1.9.6-py3-none-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:5263dda3f71feef2e4122baf5c4620e4b3710dbb7f2121f85a337182de214369", size = 15776470, upload-time = "2025-10-07T12:44:47.476Z" }, - { url = "https://files.pythonhosted.org/packages/6c/33/8c967cce6848cdd87a2e442c86120ac644b80c5ed4c32e3291bde6a17df8/maturin-1.9.6-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:fe78262c2800c92f67d1ce3c0f6463f958a692cc67bfb572e5dbf5b4b696a8ba", size = 8226557, upload-time = "2025-10-07T12:44:49.844Z" }, - { url = "https://files.pythonhosted.org/packages/58/bd/3e2675cdc8b7270700ba30c663c852a35694441732a107ac30ebd6878bd8/maturin-1.9.6-py3-none-manylinux_2_12_i686.manylinux2010_i686.musllinux_1_1_i686.whl", hash = "sha256:7ab827c6e8c022eb2e1e7fb6deede54549c8460b20ccc2e9268cc6e8cde957a8", size = 8166544, upload-time = "2025-10-07T12:44:51.396Z" }, - { url = "https://files.pythonhosted.org/packages/58/1f/a2047ddf2230e700d5f8a13dd4b9af5ce806ad380c32e58105888205926e/maturin-1.9.6-py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.musllinux_1_1_x86_64.whl", hash = "sha256:0246202377c49449315305209f45c8ecef6e2d6bd27a04b5b6f1ab3e4ea47238", size = 8641010, upload-time = "2025-10-07T12:44:53.658Z" }, - { url = "https://files.pythonhosted.org/packages/be/1f/265d63c7aa6faf363d4a3f23396f51bc6b4d5c7680a4190ae68dba25dea2/maturin-1.9.6-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.musllinux_1_1_aarch64.whl", hash = "sha256:f5bac167700fbb6f8c8ed1a97b494522554b4432d7578e11403b894b6a91d99f", size = 7965945, upload-time = "2025-10-07T12:44:55.248Z" }, - { url = "https://files.pythonhosted.org/packages/4c/ca/a8e61979ccfe080948bcc1bddd79356157aee687134df7fb013050cec783/maturin-1.9.6-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.musllinux_1_1_armv7l.whl", hash = "sha256:7f53d3b1d8396d3fea3e1ee5fd37558bca5719090f3d194ba1c02b0b56327ae3", size = 7978820, upload-time = "2025-10-07T12:44:56.919Z" }, - { url = "https://files.pythonhosted.org/packages/bf/4a/81b412f8ad02a99801ef19ec059fba0822d1d28fb44cb6a92e722f05f278/maturin-1.9.6-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.musllinux_1_1_ppc64le.whl", hash = "sha256:7f506eb358386d94d6ec3208c003130cf4b69cab26034fc0cbbf8bf83afa4c2e", size = 10452064, upload-time = "2025-10-07T12:44:58.232Z" }, - { url = "https://files.pythonhosted.org/packages/5b/12/cc96c7a8cb51d8dcc9badd886c361caa1526fba7fa69d1e7892e613b71d4/maturin-1.9.6-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:f2d6984ab690af509f525dbd2b130714207c06ebb14a5814edbe1e42b17ae0de", size = 8852401, upload-time = "2025-10-07T12:44:59.8Z" }, - { url = "https://files.pythonhosted.org/packages/51/8e/653ac3c9f2c25cdd81aefb0a2d17ff140ca5a14504f5e3c7f94dcfe4dbb7/maturin-1.9.6-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:5c2252b0956bb331460ac750c805ddf0d9b44442449fc1f16e3b66941689d0bc", size = 8425057, upload-time = "2025-10-07T12:45:01.711Z" }, - { url = "https://files.pythonhosted.org/packages/db/29/f13490328764ae9bfc1da55afc5b707cebe4fa75ad7a1573bfa82cfae0c6/maturin-1.9.6-py3-none-win32.whl", hash = "sha256:f2c58d29ebdd4346fd004e6be213d071fdd94a77a16aa91474a21a4f9dbf6309", size = 7165956, upload-time = "2025-10-07T12:45:03.766Z" }, - { url = "https://files.pythonhosted.org/packages/db/9f/dd51e5ac1fce47581b8efa03d77a03f928c0ef85b6e48a61dfa37b6b85a2/maturin-1.9.6-py3-none-win_amd64.whl", hash = "sha256:1b39a5d82572c240d20d9e8be024d722dfb311d330c5e28ddeb615211755941a", size = 8145722, upload-time = "2025-10-07T12:45:05.487Z" }, - { url = "https://files.pythonhosted.org/packages/65/f2/e97aaba6d0d78c5871771bf9dd71d4eb8dac15df9109cf452748d2207412/maturin-1.9.6-py3-none-win_arm64.whl", hash = "sha256:ac02a30083553d2a781c10cd6f5480119bf6692fd177e743267406cad2ad198c", size = 6857006, upload-time = "2025-10-07T12:45:06.813Z" }, + { url = "https://files.pythonhosted.org/packages/43/4d/a23fc95be881aa8c7a6ea353410417872e4d7065df03d7f3db8f0dbed4a7/maturin-1.13.1-py3-none-linux_armv6l.whl", hash = "sha256:416e4e01cb88b798e606ee43929df897e42c1647b722ef68283816cca99a8742", size = 10102444, upload-time = "2026-04-09T15:13:48.393Z" }, + { url = "https://files.pythonhosted.org/packages/a6/1e/65c385d65bae95cf04895d52f39dbed8b1453ae55da2903d252ade40a774/maturin-1.13.1-py3-none-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:72888e87819ce546d0d2df900e4b385e4ef299077d92ee37b48923a5602dae94", size = 19576043, upload-time = "2026-04-09T15:14:08.685Z" }, + { url = "https://files.pythonhosted.org/packages/8f/13/f6bc868d0bfecd9314870b97f530a167e31f7878ac4945c78245c6eef69c/maturin-1.13.1-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:98b5fcf1a186c217830a8295ecc2989c6b1cf50945417adfc15252107b9475b7", size = 10117339, upload-time = "2026-04-09T15:13:40.559Z" }, + { url = "https://files.pythonhosted.org/packages/51/58/279e081305c11c1c1c4fccacf77df8959646c5d4de7a57ec7e787653e270/maturin-1.13.1-py3-none-manylinux_2_12_i686.manylinux2010_i686.musllinux_1_1_i686.whl", hash = "sha256:3da18cccf2f683c0977bff9146a0908d6ffce836d600665736ac01679f588cb9", size = 10139689, upload-time = "2026-04-09T15:13:38.291Z" }, + { url = "https://files.pythonhosted.org/packages/00/94/69391af5396c6aab723932240803f49e5f3de3dd7c57d32f02d237a0ce32/maturin-1.13.1-py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.musllinux_1_1_x86_64.whl", hash = "sha256:6b1e5916a253243e8f5f9e847b62bbc98420eec48c9ce2e2e8724c6da89d359b", size = 10551141, upload-time = "2026-04-09T15:13:42.887Z" }, + { url = "https://files.pythonhosted.org/packages/9e/bf/4edac2667b49e3733438062ae416413b8fc8d42e1bd499ba15e1fb02fc55/maturin-1.13.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.musllinux_1_1_aarch64.whl", hash = "sha256:dc91031e0619c1e28730279ef9ee5f106c9b9ec806b013f888676b242f892eb7", size = 9983094, upload-time = "2026-04-09T15:13:56.868Z" }, + { url = "https://files.pythonhosted.org/packages/79/94/a6d651cfe8fc6bf2e892c90e3cdbb25c06d81c9115140d03ea1a68a97575/maturin-1.13.1-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.musllinux_1_1_armv7l.whl", hash = "sha256:001741c6cff56aa8ea59a0d78ae990c0550d0e3e82b00b683eedb4158a8ef7e6", size = 9949980, upload-time = "2026-04-09T15:13:59.185Z" }, + { url = "https://files.pythonhosted.org/packages/b5/d1/82c067464f848e38af9910bce55eb54302b1c1284a279d515dbfcf5994f5/maturin-1.13.1-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.musllinux_1_1_ppc64le.whl", hash = "sha256:01c845825c917c07c1d0b2c9032c59c16a7d383d1e649a46481d3e5693c2750f", size = 13186276, upload-time = "2026-04-09T15:13:45.725Z" }, + { url = "https://files.pythonhosted.org/packages/7c/f4/25367baf1025580f047f9b37598bb3fadc416e24536afd4f28e190335c73/maturin-1.13.1-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:f69093ed4a0e6464e52a7fc26d714f859ce15630ec8070743398c6bf41f38a9e", size = 10891837, upload-time = "2026-04-09T15:13:35.68Z" }, + { url = "https://files.pythonhosted.org/packages/af/be/caafad8ce74974b7deafdf144d12f758993dfea4c66c9905b138f51a7792/maturin-1.13.1-py3-none-manylinux_2_31_riscv64.musllinux_1_1_riscv64.whl", hash = "sha256:c1490584f3c70af45466ee99065b49e6657ebdccac6b10571bb44681309c9396", size = 10351032, upload-time = "2026-04-09T15:14:01.632Z" }, + { url = "https://files.pythonhosted.org/packages/66/0e/970a721d27cfa410e8bfa0a1e32e6ef52cb8169692110a5fdabe1af3f570/maturin-1.13.1-py3-none-win32.whl", hash = "sha256:c6a720b252c99de072922dbe4432ab19662b6f80045b0355fec23bdfccb450da", size = 8855465, upload-time = "2026-04-09T15:13:51.122Z" }, + { url = "https://files.pythonhosted.org/packages/88/70/7c1e0d65fa147d5479055a171541c82b8cdfc1c825d85a82240470f14176/maturin-1.13.1-py3-none-win_amd64.whl", hash = "sha256:a2017d2281203d0c6570240e7d746564d766d756105823b7de68bda6ae722711", size = 10230471, upload-time = "2026-04-09T15:13:53.89Z" }, + { url = "https://files.pythonhosted.org/packages/c5/2a/afe0193b673a79ffd2e01ad999511b7e9e6b49af02bb3759d82a78c3043d/maturin-1.13.1-py3-none-win_arm64.whl", hash = "sha256:2839024dcd65776abb4759e5bca29941971e095574162a4d335191da4be9ff24", size = 8905575, upload-time = "2026-04-09T15:14:03.891Z" }, ] [[package]] @@ -1142,7 +1142,7 @@ wheels = [ [[package]] name = "pytest" -version = "8.4.2" +version = "9.0.3" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "colorama", marker = "sys_platform == 'win32'" }, @@ -1151,9 +1151,9 @@ dependencies = [ { name = "pluggy" }, { name = "pygments" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/a3/5c/00a0e072241553e1a7496d638deababa67c5058571567b92a7eaa258397c/pytest-8.4.2.tar.gz", hash = "sha256:86c0d0b93306b961d58d62a4db4879f27fe25513d4b969df351abdddb3c30e01", size = 1519618, upload-time = "2025-09-04T14:34:22.711Z" } +sdist = { url = "https://files.pythonhosted.org/packages/7d/0d/549bd94f1a0a402dc8cf64563a117c0f3765662e2e668477624baeec44d5/pytest-9.0.3.tar.gz", hash = "sha256:b86ada508af81d19edeb213c681b1d48246c1a91d304c6c81a427674c17eb91c", size = 1572165, upload-time = "2026-04-07T17:16:18.027Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/a8/a4/20da314d277121d6534b3a980b29035dcd51e6744bd79075a6ce8fa4eb8d/pytest-8.4.2-py3-none-any.whl", hash = "sha256:872f880de3fc3a5bdc88a11b39c9710c3497a547cfa9320bc3c5e62fbf272e79", size = 365750, upload-time = "2025-09-04T14:34:20.226Z" }, + { url = "https://files.pythonhosted.org/packages/d4/24/a372aaf5c9b7208e7112038812994107bc65a84cd00e0354a88c2c77a617/pytest-9.0.3-py3-none-any.whl", hash = "sha256:2c5efc453d45394fdd706ade797c0a81091eccd1d6e4bccfcd476e2b8e0ab5d9", size = 375249, upload-time = "2026-04-07T17:16:16.13Z" }, ] [[package]] @@ -1260,7 +1260,7 @@ wheels = [ [[package]] name = "ray" -version = "2.54.0" +version = "2.55.1" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "click" }, @@ -1273,17 +1273,20 @@ dependencies = [ { name = "requests" }, ] wheels = [ - { url = "https://files.pythonhosted.org/packages/08/58/6209b2231947f3c8df09ce1436f1c76c4a11fcafd57c8def852dcbb6d8ef/ray-2.54.0-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:8e39dd56b47a0a1820d5a5a54385bbe54d1d67e1093736d12d8ed4e99d0fa455", size = 70098998, upload-time = "2026-02-18T04:04:58.801Z" }, - { url = "https://files.pythonhosted.org/packages/ac/29/7871f4206e6b00a9bb784c16dad32ccd01e9df5a93545db92de220eb2871/ray-2.54.0-cp311-cp311-manylinux2014_aarch64.whl", hash = "sha256:491ae56ab80d8822c4eaf4d5bb96dcf32a6231d8d7b76eb8034400eb9be1bb18", size = 72066630, upload-time = "2026-02-18T04:05:04.957Z" }, - { url = "https://files.pythonhosted.org/packages/1d/e8/d2c8ebd9cd945abc817b01ad02a29df78cdb86cd07d764587e16977389d0/ray-2.54.0-cp311-cp311-manylinux2014_x86_64.whl", hash = "sha256:928bb09245a3c6f7c3c113ba8eafc69f948da9602d7f33e8251ecdf97c157615", size = 72895723, upload-time = "2026-02-18T04:05:10.686Z" }, - { url = "https://files.pythonhosted.org/packages/7e/96/a5ea3a149a943475cda1d68fdcdb14c86251826c652c232ae853600ad7e7/ray-2.54.0-cp311-cp311-win_amd64.whl", hash = "sha256:1e786330de55b3ba2228e36ec305381a9b86f0b01a8b6072c5811c3bc4dd9a3d", size = 27448371, upload-time = "2026-02-18T04:05:16.34Z" }, - { url = "https://files.pythonhosted.org/packages/0e/16/45eefb51eb1767342a6dbf41af0b432279e422e56160705fcd1098a7ec53/ray-2.54.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:cf5c33b4b13850ec24a5bd5f9d9e0a8161f8e586bfd297e52913d170dec447fe", size = 70084880, upload-time = "2026-02-18T04:05:22.007Z" }, - { url = "https://files.pythonhosted.org/packages/60/ad/e07aca3637e9c3ec4857ec4366208099cf8488ece8061a9925ba29b66382/ray-2.54.0-cp312-cp312-manylinux2014_aarch64.whl", hash = "sha256:795ae21d6b764245d3f521bc5833446d58569e7dfde9c5777417eb285d87450f", size = 72107346, upload-time = "2026-02-18T04:05:27.999Z" }, - { url = "https://files.pythonhosted.org/packages/9e/b9/cc5ea8460c3dc602e6b7198277a7c59ba2b8929374ab22efa8df9f3deac8/ray-2.54.0-cp312-cp312-manylinux2014_x86_64.whl", hash = "sha256:a972afd5aa3dda99d0b2f369b5f62e5dd95865ab7d37bf2e0a0e0d2cfbd9b325", size = 72967230, upload-time = "2026-02-18T04:05:33.771Z" }, - { url = "https://files.pythonhosted.org/packages/de/d7/744de3b1bb881701330ddcbb2f6efaccd65915d564ece899a3838f9fb105/ray-2.54.0-cp312-cp312-win_amd64.whl", hash = "sha256:2ee074ede491d0aacfa339c003f5d7a15826e1e2a72ce873234ccbc0446e19b3", size = 27427353, upload-time = "2026-02-18T04:05:38.853Z" }, - { url = "https://files.pythonhosted.org/packages/7f/f2/5c0161d10445e703b7d01413ab54ec1cc5e27032555279d296df89b9c4ee/ray-2.54.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:5ad77961fea16c697a0fb0e51216dd39c0bec28868cde54ac668edd58d12b8ae", size = 70030991, upload-time = "2026-02-18T04:05:43.966Z" }, - { url = "https://files.pythonhosted.org/packages/fd/8c/4a4a38eaec6e9614076a96967f58540f4f8d4aa0c793f43150c5df23cb9a/ray-2.54.0-cp313-cp313-manylinux2014_aarch64.whl", hash = "sha256:8952c23a8aa94f10728c2d16e0dc3732d09aa0e6254801757ff494984a214f45", size = 72013826, upload-time = "2026-02-18T04:05:49.866Z" }, - { url = "https://files.pythonhosted.org/packages/42/ac/e7ec2a406bd755f61c7090460fa5ab3f09b00c3c2d8db6d0b559f78a30eb/ray-2.54.0-cp313-cp313-manylinux2014_x86_64.whl", hash = "sha256:ab89e6089abb6e46fb98fdd96d399b31a852d79127cd8ac00746c61d93defa2c", size = 72880209, upload-time = "2026-02-18T04:05:55.498Z" }, + { url = "https://files.pythonhosted.org/packages/88/7d/48ba2f49b40a34b0071ee27c0144a2573d8836094eaca213d59cef12c271/ray-2.55.1-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:0053fd5b400f7ac56263aa1bbd3d68fb79341b08b8dc697c88782d5aca7b3ed4", size = 65835271, upload-time = "2026-04-22T20:09:34.984Z" }, + { url = "https://files.pythonhosted.org/packages/8f/a3/d6db3a428e4ea17cc72e79f747cfe11e90e63e36e1705bb8324e45f334b7/ray-2.55.1-cp311-cp311-manylinux2014_aarch64.whl", hash = "sha256:0ea2f670a7725833ad2333a8c46ab69865ad06c8e5de9f65695e0f8f35331cec", size = 72879783, upload-time = "2026-04-22T20:09:40.986Z" }, + { url = "https://files.pythonhosted.org/packages/46/59/41da0e72a59cd3e8978480ccfeb86ef4235ae5ceb9b8928168a764fa930a/ray-2.55.1-cp311-cp311-manylinux2014_x86_64.whl", hash = "sha256:d5382da181c03ee2f502ef46cf0ae4bbc30157b5bd9a67d7651f6a272528a85a", size = 73706515, upload-time = "2026-04-22T20:09:47.079Z" }, + { url = "https://files.pythonhosted.org/packages/65/52/c16bbdc3e31a5178f97be88966ab56db6f7e04882640c5cf2fee5b87757b/ray-2.55.1-cp311-cp311-win_amd64.whl", hash = "sha256:5e56d2e8f304cafe990c198a2b894f5b813de018998cd7212869201f6dc17cff", size = 27882093, upload-time = "2026-04-22T20:09:52.943Z" }, + { url = "https://files.pythonhosted.org/packages/ac/3a/4d34f471a68b958b7f94c974c19ad6836a61a2dc16393df4294169a2e4b0/ray-2.55.1-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:137f9006eee28caab8260803cca314f37bbda3fc94fdfa31c770b5d019626ad8", size = 65822379, upload-time = "2026-04-22T20:09:58.064Z" }, + { url = "https://files.pythonhosted.org/packages/f1/13/0db535102d0256b350ca116d8987588aca1a1f9ebb4638e1e1ff88bbcef8/ray-2.55.1-cp312-cp312-manylinux2014_aarch64.whl", hash = "sha256:26541f69bb55607ef8335baac75b2ed12ff2ce02d56313219b29eda003039221", size = 72910802, upload-time = "2026-04-22T20:10:04.382Z" }, + { url = "https://files.pythonhosted.org/packages/4c/f8/fffadf3f4285eebd460e4d7f2ed1c0cd641ed89613c3f49eb881ee9fa7e2/ray-2.55.1-cp312-cp312-manylinux2014_x86_64.whl", hash = "sha256:263705f6bab29e7622a94f82da25fd7f9cead76cdf89a07aab28f79cdf8f9d95", size = 73765203, upload-time = "2026-04-22T20:10:10.495Z" }, + { url = "https://files.pythonhosted.org/packages/10/f7/5acb86fc9625a0e6bbc40e1c7d42c60770e78585439a921c32738b6d675a/ray-2.55.1-cp312-cp312-win_amd64.whl", hash = "sha256:9ad56704c8bd7e92130162f9c58e4ef473609515637673d5a36e761f95335206", size = 27865547, upload-time = "2026-04-22T20:10:15.364Z" }, + { url = "https://files.pythonhosted.org/packages/d5/95/898699cc1a6a5f304ea95376d079843b5c05f4c8c1ec7e55a5cc7ffcea50/ray-2.55.1-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:f9844a9272ef2e6eb5771025866072cf4234cf4c7cc1a31e235b7de7111864be", size = 65766823, upload-time = "2026-04-22T20:10:20.786Z" }, + { url = "https://files.pythonhosted.org/packages/c9/13/87deecc090c672e45a0cf6f5eef511de448b93f37ef18fd10eb8e8557a0d/ray-2.55.1-cp313-cp313-manylinux2014_aarch64.whl", hash = "sha256:b415d590e062f248907e0fe42994943f11726b7178fcf4b1cf5546721fb1a5f8", size = 72818676, upload-time = "2026-04-22T20:10:26.705Z" }, + { url = "https://files.pythonhosted.org/packages/71/d7/fc95d3b8824c62105c64aa1b59c59600b581f608d78a2af753e010936dc9/ray-2.55.1-cp313-cp313-manylinux2014_x86_64.whl", hash = "sha256:1380e043eb57cde69b7e9199c6f2558ceeb8f0fc41c97d1d5e50ea042115f302", size = 73678908, upload-time = "2026-04-22T20:10:32.795Z" }, + { url = "https://files.pythonhosted.org/packages/a9/03/7e552325572e067b23a4584bda8dc6a67af8bd7e03c424d2610bfa93112d/ray-2.55.1-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:b062045c64c2bce39a51661624f7292c7bbf30f2a9d878627aae31d46da5712d", size = 65774106, upload-time = "2026-04-22T20:10:39.885Z" }, + { url = "https://files.pythonhosted.org/packages/94/62/607a8859520ce350861425f11f8e15d66c15ee33e6aac812f9e2889b5df4/ray-2.55.1-cp314-cp314-manylinux2014_aarch64.whl", hash = "sha256:4e618d61e1b14b6fde9a586151f3fd9d435b0b85048b997bcaa7f4a533747b2b", size = 72814044, upload-time = "2026-04-22T20:10:46.985Z" }, + { url = "https://files.pythonhosted.org/packages/04/5a/0699bef04a72d7dc54462960d07ef7a19cd8b1e09979880aba2b6d13cca2/ray-2.55.1-cp314-cp314-manylinux2014_x86_64.whl", hash = "sha256:156ed3e72ad95b645d2006cd71a8dddbcc89b56bfc00027f6225adf78bd9cb74", size = 73644244, upload-time = "2026-04-22T20:10:52.973Z" }, ] [[package]] @@ -1934,7 +1937,7 @@ provides-extras = ["duckdb", "numpy", "pandas", "polars", "ray"] [package.metadata.requires-dev] dev = [ { name = "duckdb", specifier = ">=1.1.2" }, - { name = "maturin", specifier = ">=1.7.2" }, + { name = "maturin", specifier = ">=1.10.2" }, { name = "numpy", specifier = ">=2.2.2" }, { name = "pandas", extras = ["output-formatting"], specifier = ">=2.2.3" }, { name = "pandas-stubs", specifier = ">=2.2.3.241126" }, diff --git a/vortex-array/Cargo.toml b/vortex-array/Cargo.toml index 462e85a1364..f8676d76ef0 100644 --- a/vortex-array/Cargo.toml +++ b/vortex-array/Cargo.toml @@ -21,6 +21,7 @@ workspace = true [dependencies] arbitrary = { workspace = true, optional = true } +arc-swap = { workspace = true } arcref = { workspace = true } arrow-arith = { workspace = true } arrow-array = { workspace = true, features = ["ffi"] } @@ -38,7 +39,6 @@ cudarc = { workspace = true, optional = true } enum-iterator = { workspace = true } flatbuffers = { workspace = true } futures = { workspace = true, features = ["alloc", "async-await", "std"] } -getrandom_v03 = { workspace = true } goldenfile = { workspace = true, optional = true } half = { workspace = true, features = ["num-traits"] } humansize = { workspace = true } @@ -96,6 +96,10 @@ serde_json = { workspace = true } serde_test = { workspace = true } vortex-array = { path = ".", features = ["_test-harness", "table-display"] } +[[bench]] +name = "cast_primitive" +harness = false + [[bench]] name = "search_sorted" harness = false @@ -173,6 +177,10 @@ harness = false name = "take_fsl" harness = false +[[bench]] +name = "take_filter" +harness = false + [[bench]] name = "filter_bool" harness = false @@ -181,5 +189,6 @@ harness = false name = "listview_rebuild" harness = false -[package.metadata.cargo-machete] -ignored = ["getrandom_v03"] +[[bench]] +name = "listview_builder_extend" +harness = false diff --git a/vortex-array/benches/cast_primitive.rs b/vortex-array/benches/cast_primitive.rs new file mode 100644 index 00000000000..86895fb2ce7 --- /dev/null +++ b/vortex-array/benches/cast_primitive.rs @@ -0,0 +1,48 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +use divan::Bencher; +use rand::prelude::*; +use vortex_array::Canonical; +use vortex_array::IntoArray; +use vortex_array::LEGACY_SESSION; +use vortex_array::VortexSessionExecute; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::builtins::ArrayBuiltins; +use vortex_array::dtype::DType; +use vortex_array::dtype::Nullability; +use vortex_array::dtype::PType; +use vortex_array::expr::stats::Stat; + +fn main() { + divan::main(); +} + +const N: usize = 100_000; + +#[divan::bench] +fn cast_u16_to_u32(bencher: Bencher) { + let mut rng = StdRng::seed_from_u64(42); + #[expect(clippy::cast_possible_truncation)] + let arr = PrimitiveArray::from_option_iter((0..N).map(|i| { + if rng.random_bool(0.5) { + None + } else { + Some(i as u16) + } + })) + .into_array(); + // Pre-compute min/max so values_fit_in is a cache hit during the benchmark. + arr.statistics() + .compute_all( + &[Stat::Min, Stat::Max], + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .ok(); + bencher.with_inputs(|| arr.clone()).bench_refs(|a| { + #[expect(clippy::unwrap_used)] + a.cast(DType::Primitive(PType::U32, Nullability::Nullable)) + .unwrap() + .execute::(&mut LEGACY_SESSION.create_execution_ctx()) + }); +} diff --git a/vortex-array/benches/chunk_array_builder.rs b/vortex-array/benches/chunk_array_builder.rs index b334b65e9e6..d74fd309453 100644 --- a/vortex-array/benches/chunk_array_builder.rs +++ b/vortex-array/benches/chunk_array_builder.rs @@ -8,6 +8,7 @@ use rand::RngExt; use rand::SeedableRng; use rand::prelude::StdRng; use vortex_array::ArrayRef; +use vortex_array::Canonical; use vortex_array::IntoArray; use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; @@ -39,26 +40,30 @@ static SESSION: LazyLock = fn chunked_bool_canonical_into(bencher: Bencher, (len, chunk_count): (usize, usize)) { let chunk = make_bool_chunks(len, chunk_count); - bencher.with_inputs(|| &chunk).bench_refs(|chunk| { - let mut builder = builder_with_capacity(chunk.dtype(), len * chunk_count); - chunk - .append_to_builder(builder.as_mut(), &mut SESSION.create_execution_ctx()) - .vortex_expect("append failed"); - builder.finish() - }) + bencher + .with_inputs(|| (&chunk, SESSION.create_execution_ctx())) + .bench_refs(|(chunk, ctx)| { + let mut builder = builder_with_capacity(chunk.dtype(), len * chunk_count); + chunk + .append_to_builder(builder.as_mut(), ctx) + .vortex_expect("append failed"); + builder.finish() + }) } #[divan::bench(args = BENCH_ARGS)] fn chunked_opt_bool_canonical_into(bencher: Bencher, (len, chunk_count): (usize, usize)) { let chunk = make_opt_bool_chunks(len, chunk_count); - bencher.with_inputs(|| &chunk).bench_refs(|chunk| { - let mut builder = builder_with_capacity(chunk.dtype(), len * chunk_count); - chunk - .append_to_builder(builder.as_mut(), &mut SESSION.create_execution_ctx()) - .vortex_expect("append failed"); - builder.finish() - }) + bencher + .with_inputs(|| (&chunk, SESSION.create_execution_ctx())) + .bench_refs(|(chunk, ctx)| { + let mut builder = builder_with_capacity(chunk.dtype(), len * chunk_count); + chunk + .append_to_builder(builder.as_mut(), ctx) + .vortex_expect("append failed"); + builder.finish() + }) } #[divan::bench(args = BENCH_ARGS)] @@ -66,24 +71,26 @@ fn chunked_opt_bool_into_canonical(bencher: Bencher, (len, chunk_count): (usize, let chunk = make_opt_bool_chunks(len, chunk_count); bencher - .with_inputs(|| &chunk) - .bench_refs(|chunk| chunk.to_canonical()) + .with_inputs(|| (&chunk, SESSION.create_execution_ctx())) + .bench_refs(|(chunk, ctx)| (**chunk).clone().execute::(ctx)) } #[divan::bench(args = BENCH_ARGS)] fn chunked_varbinview_canonical_into(bencher: Bencher, (len, chunk_count): (usize, usize)) { let chunks = make_string_chunks(false, len, chunk_count); - bencher.with_inputs(|| &chunks).bench_refs(|chunk| { - let mut builder = VarBinViewBuilder::with_capacity( - DType::Utf8(chunk.dtype().nullability()), - len * chunk_count, - ); - chunk - .append_to_builder(&mut builder, &mut SESSION.create_execution_ctx()) - .vortex_expect("append failed"); - builder.finish() - }) + bencher + .with_inputs(|| (&chunks, SESSION.create_execution_ctx())) + .bench_refs(|(chunk, ctx)| { + let mut builder = VarBinViewBuilder::with_capacity( + DType::Utf8(chunk.dtype().nullability()), + len * chunk_count, + ); + chunk + .append_to_builder(&mut builder, ctx) + .vortex_expect("append failed"); + builder.finish() + }) } #[divan::bench(args = BENCH_ARGS)] @@ -91,24 +98,26 @@ fn chunked_varbinview_into_canonical(bencher: Bencher, (len, chunk_count): (usiz let chunks = make_string_chunks(false, len, chunk_count); bencher - .with_inputs(|| &chunks) - .bench_refs(|chunk| chunk.to_canonical()) + .with_inputs(|| (&chunks, SESSION.create_execution_ctx())) + .bench_refs(|(chunk, ctx)| (**chunk).clone().execute::(ctx)) } #[divan::bench(args = BENCH_ARGS)] fn chunked_varbinview_opt_canonical_into(bencher: Bencher, (len, chunk_count): (usize, usize)) { let chunks = make_string_chunks(true, len, chunk_count); - bencher.with_inputs(|| &chunks).bench_refs(|chunk| { - let mut builder = VarBinViewBuilder::with_capacity( - DType::Utf8(chunk.dtype().nullability()), - len * chunk_count, - ); - chunk - .append_to_builder(&mut builder, &mut SESSION.create_execution_ctx()) - .vortex_expect("append failed"); - builder.finish() - }) + bencher + .with_inputs(|| (&chunks, SESSION.create_execution_ctx())) + .bench_refs(|(chunk, ctx)| { + let mut builder = VarBinViewBuilder::with_capacity( + DType::Utf8(chunk.dtype().nullability()), + len * chunk_count, + ); + chunk + .append_to_builder(&mut builder, ctx) + .vortex_expect("append failed"); + builder.finish() + }) } #[divan::bench(args = BENCH_ARGS)] @@ -116,21 +125,23 @@ fn chunked_varbinview_opt_into_canonical(bencher: Bencher, (len, chunk_count): ( let chunks = make_string_chunks(true, len, chunk_count); bencher - .with_inputs(|| &chunks) - .bench_refs(|chunk| chunk.to_canonical()) + .with_inputs(|| (&chunks, SESSION.create_execution_ctx())) + .bench_refs(|(chunk, ctx)| (**chunk).clone().execute::(ctx)) } #[divan::bench(args = BENCH_ARGS)] fn chunked_constant_i32_append_to_builder(bencher: Bencher, (len, chunk_count): (usize, usize)) { let chunk = make_constant_i32_chunks(len, chunk_count); - bencher.with_inputs(|| &chunk).bench_refs(|chunk| { - let mut builder = builder_with_capacity(chunk.dtype(), len * chunk_count); - chunk - .append_to_builder(builder.as_mut(), &mut SESSION.create_execution_ctx()) - .vortex_expect("append failed"); - builder.finish() - }) + bencher + .with_inputs(|| (&chunk, SESSION.create_execution_ctx())) + .bench_refs(|(chunk, ctx)| { + let mut builder = builder_with_capacity(chunk.dtype(), len * chunk_count); + chunk + .append_to_builder(builder.as_mut(), ctx) + .vortex_expect("append failed"); + builder.finish() + }) } const CONSTANT_UTF8_BENCH_ARGS: &[(&str, usize, usize)] = &[ @@ -146,13 +157,15 @@ fn chunked_constant_utf8_append_to_builder( ) { let chunk = make_constant_utf8_chunks(value, len, chunk_count); - bencher.with_inputs(|| &chunk).bench_refs(|chunk| { - let mut builder = builder_with_capacity(chunk.dtype(), len * chunk_count); - chunk - .append_to_builder(builder.as_mut(), &mut SESSION.create_execution_ctx()) - .vortex_expect("append failed"); - builder.finish() - }) + bencher + .with_inputs(|| (&chunk, SESSION.create_execution_ctx())) + .bench_refs(|(chunk, ctx)| { + let mut builder = builder_with_capacity(chunk.dtype(), len * chunk_count); + chunk + .append_to_builder(builder.as_mut(), ctx) + .vortex_expect("append failed"); + builder.finish() + }) } fn make_constant_utf8_chunks(value: &str, len: usize, chunk_count: usize) -> ArrayRef { diff --git a/vortex-array/benches/chunked_dict_builder.rs b/vortex-array/benches/chunked_dict_builder.rs index b42514fcf19..85c8737356f 100644 --- a/vortex-array/benches/chunked_dict_builder.rs +++ b/vortex-array/benches/chunked_dict_builder.rs @@ -40,13 +40,15 @@ fn chunked_dict_primitive_canonical_into( { let chunk = gen_dict_primitive_chunks::(len, unique_values, chunk_count); - bencher.with_inputs(|| &chunk).bench_refs(|chunk| { - let mut builder = builder_with_capacity(chunk.dtype(), len * chunk_count); - chunk - .append_to_builder(builder.as_mut(), &mut SESSION.create_execution_ctx()) - .vortex_expect("append failed"); - builder.finish() - }) + bencher + .with_inputs(|| (&chunk, SESSION.create_execution_ctx())) + .bench_refs(|(chunk, ctx)| { + let mut builder = builder_with_capacity(chunk.dtype(), len * chunk_count); + chunk + .append_to_builder(builder.as_mut(), ctx) + .vortex_expect("append failed"); + builder.finish() + }) } #[divan::bench(types = [u32, u64, f32, f64], args = BENCH_ARGS)] @@ -59,6 +61,6 @@ fn chunked_dict_primitive_into_canonical( let chunk = gen_dict_primitive_chunks::(len, unique_values, chunk_count); bencher - .with_inputs(|| chunk.clone()) - .bench_values(|chunk| chunk.execute::(&mut SESSION.create_execution_ctx())) + .with_inputs(|| (chunk.clone(), SESSION.create_execution_ctx())) + .bench_values(|(chunk, mut ctx)| chunk.execute::(&mut ctx)) } diff --git a/vortex-array/benches/compare.rs b/vortex-array/benches/compare.rs index 4aceffd6de1..6fe5be688e9 100644 --- a/vortex-array/benches/compare.rs +++ b/vortex-array/benches/compare.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] +#![expect(clippy::unwrap_used)] use divan::Bencher; use rand::RngExt; diff --git a/vortex-array/benches/dict_compare.rs b/vortex-array/benches/dict_compare.rs index b361ba1006e..602ad8638f7 100644 --- a/vortex-array/benches/dict_compare.rs +++ b/vortex-array/benches/dict_compare.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] +#![expect(clippy::unwrap_used)] use std::str::from_utf8; diff --git a/vortex-array/benches/dict_compress.rs b/vortex-array/benches/dict_compress.rs index 5ecf06378cb..4f220ccaa23 100644 --- a/vortex-array/benches/dict_compress.rs +++ b/vortex-array/benches/dict_compress.rs @@ -1,12 +1,15 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] +#![expect(clippy::unwrap_used)] use divan::Bencher; use rand::distr::Distribution; use rand::distr::StandardUniform; +use vortex_array::Canonical; use vortex_array::IntoArray; +use vortex_array::LEGACY_SESSION; +use vortex_array::VortexSessionExecute; use vortex_array::arrays::VarBinArray; use vortex_array::arrays::VarBinViewArray; use vortex_array::arrays::dict_test::gen_primitive_for_dict; @@ -75,8 +78,8 @@ where .into_array(); bencher - .with_inputs(|| &dict) - .bench_refs(|dict| dict.to_canonical()); + .with_inputs(|| (&dict, LEGACY_SESSION.create_execution_ctx())) + .bench_refs(|(dict, ctx)| (**dict).clone().execute::(ctx)); } #[divan::bench(args = BENCH_ARGS)] @@ -85,8 +88,8 @@ fn decode_varbin(bencher: Bencher, (len, unique_values): (usize, usize)) { let dict = dict_encode(&varbin_arr.into_array()).unwrap().into_array(); bencher - .with_inputs(|| &dict) - .bench_refs(|dict| dict.to_canonical()); + .with_inputs(|| (&dict, LEGACY_SESSION.create_execution_ctx())) + .bench_refs(|(dict, ctx)| (**dict).clone().execute::(ctx)); } #[divan::bench(args = BENCH_ARGS)] @@ -97,6 +100,6 @@ fn decode_varbinview(bencher: Bencher, (len, unique_values): (usize, usize)) { .into_array(); bencher - .with_inputs(|| &dict) - .bench_refs(|dict| dict.to_canonical()); + .with_inputs(|| (&dict, LEGACY_SESSION.create_execution_ctx())) + .bench_refs(|(dict, ctx)| (**dict).clone().execute::(ctx)); } diff --git a/vortex-array/benches/dict_mask.rs b/vortex-array/benches/dict_mask.rs index 9f0be5e585d..1951e610a51 100644 --- a/vortex-array/benches/dict_mask.rs +++ b/vortex-array/benches/dict_mask.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] +#![expect(clippy::unwrap_used)] use divan::Bencher; use rand::RngExt; @@ -62,14 +62,13 @@ fn bench_dict_mask(bencher: Bencher, (fraction_valid, fraction_masked): (f64, f6 let filter_mask = filter_mask(len, fraction_masked, &mut rng); let session = VortexSession::empty(); bencher - .with_inputs(|| (&array, &filter_mask)) - .bench_refs(|(array, filter_mask)| { - let mut ctx = session.create_execution_ctx(); + .with_inputs(|| (&array, &filter_mask, session.create_execution_ctx())) + .bench_refs(|(array, filter_mask, ctx)| { array .clone() .mask(filter_mask.clone().into_array()) .unwrap() - .execute::(&mut ctx) + .execute::(ctx) .unwrap() }); } diff --git a/vortex-array/benches/dict_unreferenced_mask.rs b/vortex-array/benches/dict_unreferenced_mask.rs index 24cf366f692..1ec143682af 100644 --- a/vortex-array/benches/dict_unreferenced_mask.rs +++ b/vortex-array/benches/dict_unreferenced_mask.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] +#![expect(clippy::unwrap_used)] use divan::Bencher; use rand::RngExt; @@ -32,7 +32,7 @@ fn bench_many_codes_few_values(bencher: Bencher, num_values: i32) { let values = PrimitiveArray::from_iter(0..num_values).into_array(); // Create codes that randomly reference the values - #[allow(clippy::cast_possible_truncation, clippy::cast_sign_loss)] + #[expect(clippy::cast_possible_truncation, clippy::cast_sign_loss)] let codes = PrimitiveArray::from_iter( (0..num_codes).map(|_| rng.random_range(0..num_values as usize) as u32), ) @@ -63,7 +63,7 @@ fn bench_many_nulls(bencher: Bencher, fraction_valid: f64) { let values = PrimitiveArray::from_iter(0..num_values).into_array(); // Create codes with many nulls based on fraction_valid - #[allow(clippy::cast_possible_truncation, clippy::cast_sign_loss)] + #[expect(clippy::cast_possible_truncation, clippy::cast_sign_loss)] let codes = PrimitiveArray::from_option_iter((0..num_codes).map(|_| { rng.random_bool(fraction_valid) .then(|| rng.random_range(0..num_values as usize) as u32) @@ -94,11 +94,11 @@ fn bench_sparse_coverage(bencher: Bencher, fraction_coverage: f64) { let values = PrimitiveArray::from_iter(0..num_values).into_array(); // Calculate how many unique values we'll actually reference - #[allow(clippy::cast_possible_truncation, clippy::cast_sign_loss)] + #[expect(clippy::cast_possible_truncation, clippy::cast_sign_loss)] let num_referenced = (num_values as f64 * fraction_coverage).max(1.0) as usize; // Create codes that only reference a subset of values - #[allow(clippy::cast_possible_truncation)] + #[expect(clippy::cast_possible_truncation)] let codes = PrimitiveArray::from_iter( (0..num_codes).map(|_| rng.random_range(0..num_referenced) as u32), ) diff --git a/vortex-array/benches/expr/case_when_bench.rs b/vortex-array/benches/expr/case_when_bench.rs index b49ba4944a0..f37daa2b304 100644 --- a/vortex-array/benches/expr/case_when_bench.rs +++ b/vortex-array/benches/expr/case_when_bench.rs @@ -1,8 +1,8 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] -#![allow(clippy::cast_possible_truncation)] +#![expect(clippy::unwrap_used)] +#![expect(clippy::cast_possible_truncation)] use std::sync::LazyLock; @@ -70,14 +70,13 @@ fn case_when_simple(bencher: Bencher, size: usize) { ); bencher - .with_inputs(|| (&expr, &array)) - .bench_refs(|(expr, array)| { - let mut ctx = SESSION.create_execution_ctx(); + .with_inputs(|| (&expr, &array, SESSION.create_execution_ctx())) + .bench_refs(|(expr, array, ctx)| { array .clone() .apply(expr) .unwrap() - .execute::(&mut ctx) + .execute::(ctx) .unwrap() }); } @@ -98,14 +97,13 @@ fn case_when_nary_3_conditions(bencher: Bencher, size: usize) { ); bencher - .with_inputs(|| (&expr, &array)) - .bench_refs(|(expr, array)| { - let mut ctx = SESSION.create_execution_ctx(); + .with_inputs(|| (&expr, &array, SESSION.create_execution_ctx())) + .bench_refs(|(expr, array, ctx)| { array .clone() .apply(expr) .unwrap() - .execute::(&mut ctx) + .execute::(ctx) .unwrap() }); } @@ -127,14 +125,13 @@ fn case_when_nary_10_conditions(bencher: Bencher, size: usize) { let expr = nested_case_when(pairs, Some(lit(0i32))); bencher - .with_inputs(|| (&expr, &array)) - .bench_refs(|(expr, array)| { - let mut ctx = SESSION.create_execution_ctx(); + .with_inputs(|| (&expr, &array, SESSION.create_execution_ctx())) + .bench_refs(|(expr, array, ctx)| { array .clone() .apply(expr) .unwrap() - .execute::(&mut ctx) + .execute::(ctx) .unwrap() }); } @@ -151,14 +148,13 @@ fn case_when_nary_equality_lookup(bencher: Bencher, size: usize) { let expr = nested_case_when(pairs, Some(lit(-1i32))); bencher - .with_inputs(|| (&expr, &array)) - .bench_refs(|(expr, array)| { - let mut ctx = SESSION.create_execution_ctx(); + .with_inputs(|| (&expr, &array, SESSION.create_execution_ctx())) + .bench_refs(|(expr, array, ctx)| { array .clone() .apply(expr) .unwrap() - .execute::(&mut ctx) + .execute::(ctx) .unwrap() }); } @@ -172,14 +168,13 @@ fn case_when_without_else(bencher: Bencher, size: usize) { let expr = case_when_no_else(gt(get_item("value", root()), lit(500i32)), lit(100i32)); bencher - .with_inputs(|| (&expr, &array)) - .bench_refs(|(expr, array)| { - let mut ctx = SESSION.create_execution_ctx(); + .with_inputs(|| (&expr, &array, SESSION.create_execution_ctx())) + .bench_refs(|(expr, array, ctx)| { array .clone() .apply(expr) .unwrap() - .execute::(&mut ctx) + .execute::(ctx) .unwrap() }); } @@ -197,14 +192,13 @@ fn case_when_all_true(bencher: Bencher, size: usize) { ); bencher - .with_inputs(|| (&expr, &array)) - .bench_refs(|(expr, array)| { - let mut ctx = SESSION.create_execution_ctx(); + .with_inputs(|| (&expr, &array, SESSION.create_execution_ctx())) + .bench_refs(|(expr, array, ctx)| { array .clone() .apply(expr) .unwrap() - .execute::(&mut ctx) + .execute::(ctx) .unwrap() }); } @@ -231,14 +225,13 @@ fn case_when_nary_early_dominant(bencher: Bencher, size: usize) { ); bencher - .with_inputs(|| (&expr, &array)) - .bench_refs(|(expr, array)| { - let mut ctx = SESSION.create_execution_ctx(); + .with_inputs(|| (&expr, &array, SESSION.create_execution_ctx())) + .bench_refs(|(expr, array, ctx)| { array .clone() .apply(expr) .unwrap() - .execute::(&mut ctx) + .execute::(ctx) .unwrap() }); } @@ -256,14 +249,13 @@ fn case_when_all_false(bencher: Bencher, size: usize) { ); bencher - .with_inputs(|| (&expr, &array)) - .bench_refs(|(expr, array)| { - let mut ctx = SESSION.create_execution_ctx(); + .with_inputs(|| (&expr, &array, SESSION.create_execution_ctx())) + .bench_refs(|(expr, array, ctx)| { array .clone() .apply(expr) .unwrap() - .execute::(&mut ctx) + .execute::(ctx) .unwrap() }); } @@ -284,14 +276,13 @@ fn case_when_fragmented(bencher: Bencher, size: usize) { ); bencher - .with_inputs(|| (&expr, &array)) - .bench_refs(|(expr, array)| { - let mut ctx = SESSION.create_execution_ctx(); + .with_inputs(|| (&expr, &array, SESSION.create_execution_ctx())) + .bench_refs(|(expr, array, ctx)| { array .clone() .apply(expr) .unwrap() - .execute::(&mut ctx) + .execute::(ctx) .unwrap() }); } diff --git a/vortex-array/benches/expr/large_struct_pack.rs b/vortex-array/benches/expr/large_struct_pack.rs index c114581bce0..31471629cd9 100644 --- a/vortex-array/benches/expr/large_struct_pack.rs +++ b/vortex-array/benches/expr/large_struct_pack.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] +#![expect(clippy::unwrap_used)] use divan::Bencher; use vortex_array::dtype::DType; diff --git a/vortex-array/benches/filter_bool.rs b/vortex-array/benches/filter_bool.rs index c346f653814..f947518d6cd 100644 --- a/vortex-array/benches/filter_bool.rs +++ b/vortex-array/benches/filter_bool.rs @@ -6,10 +6,10 @@ //! Tests multiple mask patterns (mostly-true, mostly-false, random, correlated runs) //! with both uniform-random and power-law distributions, across array sizes from 1K to 250K. -#![allow(clippy::unwrap_used)] -#![allow(clippy::cast_possible_truncation)] -#![allow(clippy::cast_sign_loss)] -#![allow(clippy::cast_precision_loss)] +#![expect(clippy::unwrap_used)] +#![expect(clippy::cast_possible_truncation)] +#![expect(clippy::cast_sign_loss)] +#![expect(clippy::cast_precision_loss)] use divan::Bencher; use rand::prelude::*; diff --git a/vortex-array/benches/listview_builder_extend.rs b/vortex-array/benches/listview_builder_extend.rs new file mode 100644 index 00000000000..75487564ea8 --- /dev/null +++ b/vortex-array/benches/listview_builder_extend.rs @@ -0,0 +1,97 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +#![expect(clippy::cast_possible_truncation)] +#![expect(clippy::cast_possible_wrap)] + +use std::sync::Arc; + +use divan::Bencher; +use vortex_array::IntoArray; +use vortex_array::arrays::ListViewArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::builders::ArrayBuilder; +use vortex_array::builders::ListViewBuilder; +use vortex_array::dtype::DType; +use vortex_array::dtype::Nullability::NonNullable; +use vortex_array::dtype::Nullability::Nullable; +use vortex_array::dtype::PType::I32; +use vortex_array::validity::Validity; +use vortex_buffer::Buffer; + +fn main() { + divan::main(); +} + +const ZCTL_ARGS: &[(usize, usize)] = &[ + // num_lists, list_size + (1_000, 8), + (1_000, 64), + (10_000, 8), +]; + +const NON_ZCTL_ARGS: &[(usize, usize)] = &[ + // num_lists, list_size + (1_000, 8), + (1_000, 32), + (10_000, 8), +]; + +fn make_listview( + num_lists: usize, + list_size: usize, + step: usize, + with_nulls: bool, +) -> ListViewArray { + let element_count = step * num_lists + list_size; + let elements = PrimitiveArray::from_iter(0..element_count as i32).into_array(); + let offsets: Buffer = (0..num_lists).map(|i| (i * step) as u32).collect(); + let sizes: Buffer = std::iter::repeat_n(list_size as u16, num_lists).collect(); + let validity = if with_nulls { + Validity::from_iter((0..num_lists).map(|i| i % 5 != 0)) + } else { + Validity::NonNullable + }; + + ListViewArray::new(elements, offsets.into_array(), sizes.into_array(), validity) +} + +#[divan::bench(args = ZCTL_ARGS)] +fn extend_from_array_zctl(bencher: Bencher, (num_lists, list_size): (usize, usize)) { + let source = make_listview(num_lists, list_size, list_size, false); + debug_assert!(source.is_zero_copy_to_list()); + let source = source.into_array(); + + bencher.with_inputs(|| &source).bench_refs(|source| { + let mut builder = ListViewBuilder::::with_capacity( + Arc::new(DType::Primitive(I32, NonNullable)), + NonNullable, + num_lists * list_size, + num_lists, + ); + builder.extend_from_array(source); + divan::black_box(builder.finish_into_listview()) + }); +} + +#[divan::bench(args = NON_ZCTL_ARGS)] +fn extend_from_array_non_zctl_overlapping( + bencher: Bencher, + (num_lists, list_size): (usize, usize), +) { + // `step = 1` creates heavily overlapping lists, which forces the non-ZCTL extend path. + let source = make_listview(num_lists, list_size, 1, true); + debug_assert!(!source.is_zero_copy_to_list()); + let source = source.into_array(); + + bencher.with_inputs(|| &source).bench_refs(|source| { + let mut builder = ListViewBuilder::::with_capacity( + Arc::new(DType::Primitive(I32, NonNullable)), + Nullable, + num_lists * list_size, + num_lists, + ); + builder.extend_from_array(source); + divan::black_box(builder.finish_into_listview()) + }); +} diff --git a/vortex-array/benches/listview_rebuild.rs b/vortex-array/benches/listview_rebuild.rs index 6affedca45f..00781b1a534 100644 --- a/vortex-array/benches/listview_rebuild.rs +++ b/vortex-array/benches/listview_rebuild.rs @@ -3,11 +3,14 @@ //! Benchmarks for ListView rebuild across different element types and scenarios. -#![allow(clippy::unwrap_used)] -#![allow(clippy::cast_possible_truncation)] +#![expect(clippy::unwrap_used)] +#![expect(clippy::cast_possible_truncation)] use divan::Bencher; +use vortex_array::Canonical; use vortex_array::IntoArray; +use vortex_array::LEGACY_SESSION; +use vortex_array::VortexSessionExecute; use vortex_array::arrays::FixedSizeListArray; use vortex_array::arrays::ListArray; use vortex_array::arrays::ListViewArray; @@ -109,7 +112,11 @@ fn i32_small(bencher: Bencher) { let lv = make_primitive_lv(50, 32, 32); bencher.with_inputs(|| &lv).bench_refs(|lv| { let rebuilt = lv.rebuild(ListViewRebuildMode::MakeZeroCopyToList).unwrap(); - rebuilt.elements().to_canonical().unwrap() + rebuilt + .elements() + .clone() + .execute::(&mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() }); } @@ -118,7 +125,11 @@ fn i32_small_overlapping(bencher: Bencher) { let lv = make_primitive_lv(50, 8, 1); bencher.with_inputs(|| &lv).bench_refs(|lv| { let rebuilt = lv.rebuild(ListViewRebuildMode::MakeZeroCopyToList).unwrap(); - rebuilt.elements().to_canonical().unwrap() + rebuilt + .elements() + .clone() + .execute::(&mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() }); } @@ -127,7 +138,11 @@ fn varbinview_small(bencher: Bencher) { let lv = make_varbinview_lv(50, 32, 32); bencher.with_inputs(|| &lv).bench_refs(|lv| { let rebuilt = lv.rebuild(ListViewRebuildMode::MakeZeroCopyToList).unwrap(); - rebuilt.elements().to_canonical().unwrap() + rebuilt + .elements() + .clone() + .execute::(&mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() }); } @@ -136,7 +151,11 @@ fn struct_small(bencher: Bencher) { let lv = make_struct_lv(50, 32, 32); bencher.with_inputs(|| &lv).bench_refs(|lv| { let rebuilt = lv.rebuild(ListViewRebuildMode::MakeZeroCopyToList).unwrap(); - rebuilt.elements().to_canonical().unwrap() + rebuilt + .elements() + .clone() + .execute::(&mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() }); } @@ -145,7 +164,11 @@ fn i32_large(bencher: Bencher) { let lv = make_primitive_lv(50, 1_024, 1_024); bencher.with_inputs(|| &lv).bench_refs(|lv| { let rebuilt = lv.rebuild(ListViewRebuildMode::MakeZeroCopyToList).unwrap(); - rebuilt.elements().to_canonical().unwrap() + rebuilt + .elements() + .clone() + .execute::(&mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() }); } @@ -154,7 +177,11 @@ fn varbinview_large(bencher: Bencher) { let lv = make_varbinview_lv(5, 1_024, 1_024); bencher.with_inputs(|| &lv).bench_refs(|lv| { let rebuilt = lv.rebuild(ListViewRebuildMode::MakeZeroCopyToList).unwrap(); - rebuilt.elements().to_canonical().unwrap() + rebuilt + .elements() + .clone() + .execute::(&mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() }); } @@ -163,7 +190,11 @@ fn struct_large(bencher: Bencher) { let lv = make_struct_lv(25, 1_024, 1_024); bencher.with_inputs(|| &lv).bench_refs(|lv| { let rebuilt = lv.rebuild(ListViewRebuildMode::MakeZeroCopyToList).unwrap(); - rebuilt.elements().to_canonical().unwrap() + rebuilt + .elements() + .clone() + .execute::(&mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() }); } @@ -185,7 +216,11 @@ fn fsl_large(bencher: Bencher) { ); bencher.with_inputs(|| &lv).bench_refs(|lv| { let rebuilt = lv.rebuild(ListViewRebuildMode::MakeZeroCopyToList).unwrap(); - rebuilt.elements().to_canonical().unwrap() + rebuilt + .elements() + .clone() + .execute::(&mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() }); } @@ -194,6 +229,10 @@ fn list_i32_large(bencher: Bencher) { let lv = make_nested_list_lv(2, 512, 2); bencher.with_inputs(|| &lv).bench_refs(|lv| { let rebuilt = lv.rebuild(ListViewRebuildMode::MakeZeroCopyToList).unwrap(); - rebuilt.elements().to_canonical().unwrap() + rebuilt + .elements() + .clone() + .execute::(&mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() }); } diff --git a/vortex-array/benches/scalar_at_struct.rs b/vortex-array/benches/scalar_at_struct.rs index a4ab5c32df8..757d6252bb9 100644 --- a/vortex-array/benches/scalar_at_struct.rs +++ b/vortex-array/benches/scalar_at_struct.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] +#![expect(clippy::unwrap_used)] use divan::Bencher; use rand::RngExt; @@ -10,6 +10,8 @@ use rand::distr::Uniform; use rand::rngs::StdRng; use vortex_array::ArrayRef; use vortex_array::IntoArray; +use vortex_array::LEGACY_SESSION; +use vortex_array::VortexSessionExecute; use vortex_array::arrays::StructArray; use vortex_array::dtype::FieldNames; use vortex_array::validity::Validity; @@ -23,7 +25,7 @@ const ARRAY_SIZE: usize = 100_000; const NUM_ACCESSES: usize = 1000; #[divan::bench] -fn scalar_at_struct_simple(bencher: Bencher) { +fn execute_scalar_struct_simple(bencher: Bencher) { let mut rng = StdRng::seed_from_u64(0); let range = Uniform::new(0i64, 100_000_000).unwrap(); @@ -49,14 +51,15 @@ fn scalar_at_struct_simple(bencher: Bencher) { bencher .with_inputs(|| (&struct_array, &indices)) .bench_refs(|(array, indices)| { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); for &idx in indices.iter() { - divan::black_box(array.scalar_at(idx).unwrap()); + divan::black_box(array.execute_scalar(idx, &mut ctx).unwrap()); } }); } #[divan::bench] -fn scalar_at_struct_wide(bencher: Bencher) { +fn execute_scalar_struct_wide(bencher: Bencher) { let mut rng = StdRng::seed_from_u64(0); let range = Uniform::new(0i64, 100_000_000).unwrap(); @@ -86,8 +89,9 @@ fn scalar_at_struct_wide(bencher: Bencher) { bencher .with_inputs(|| (&struct_array, &indices)) .bench_refs(|(array, indices)| { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); for &idx in indices.iter() { - divan::black_box(array.scalar_at(idx).unwrap()); + divan::black_box(array.execute_scalar(idx, &mut ctx).unwrap()); } }); } diff --git a/vortex-array/benches/scalar_subtract.rs b/vortex-array/benches/scalar_subtract.rs index 0d026db5d3e..bdbcdd36f3c 100644 --- a/vortex-array/benches/scalar_subtract.rs +++ b/vortex-array/benches/scalar_subtract.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] +#![expect(clippy::unwrap_used)] use divan::Bencher; use rand::RngExt; diff --git a/vortex-array/benches/search_sorted.rs b/vortex-array/benches/search_sorted.rs index 31ed741765a..2b5f7fccbac 100644 --- a/vortex-array/benches/search_sorted.rs +++ b/vortex-array/benches/search_sorted.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] +#![expect(clippy::unwrap_used)] use divan::Bencher; use rand::RngExt; diff --git a/vortex-array/benches/take_filter.rs b/vortex-array/benches/take_filter.rs new file mode 100644 index 00000000000..cfe837d37bf --- /dev/null +++ b/vortex-array/benches/take_filter.rs @@ -0,0 +1,140 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Benchmarks for taking from a lazy [`FilterArray`]. +//! +//! Parameterized over: +//! - Number of indices to take +//! - Number of rows retained by the filter +//! - Filter mask layout (single contiguous slice vs random positions) +//! - Take index layout (sequential vs random ranks) + +#![expect(clippy::unwrap_used)] +#![expect(clippy::cast_possible_truncation)] + +use divan::Bencher; +use rand::RngExt; +use rand::SeedableRng; +use rand::rngs::StdRng; +use rand::seq::SliceRandom; +use vortex_array::ArrayRef; +use vortex_array::IntoArray; +use vortex_array::LEGACY_SESSION; +use vortex_array::RecursiveCanonical; +use vortex_array::VortexSessionExecute; +use vortex_array::arrays::PrimitiveArray; +use vortex_buffer::Buffer; +use vortex_mask::Mask; + +fn main() { + divan::main(); +} + +const ARRAY_LEN: usize = 100_000; +const FILTERED_LENS: &[usize] = &[10_000, 50_000, 90_000]; +const NUM_INDICES: &[usize] = &[1_000, 10_000]; +const MASK_SEED: u64 = 42; +const INDEX_SEED: u64 = 43; + +fn primitive_array() -> ArrayRef { + PrimitiveArray::from_iter(0..ARRAY_LEN as u32).into_array() +} + +fn slice_mask(filtered_len: usize) -> Mask { + let start = (ARRAY_LEN - filtered_len) / 2; + Mask::from_slices(ARRAY_LEN, vec![(start, start + filtered_len)]) +} + +fn random_mask(filtered_len: usize) -> Mask { + let mut indices: Vec = (0..ARRAY_LEN).collect(); + indices.shuffle(&mut StdRng::seed_from_u64(MASK_SEED)); + indices.truncate(filtered_len); + indices.sort_unstable(); + Mask::from_indices(ARRAY_LEN, indices) +} + +fn sequential_indices(num_indices: usize) -> ArrayRef { + Buffer::from_iter(0..num_indices as u64).into_array() +} + +fn random_indices(num_indices: usize, filtered_len: usize) -> ArrayRef { + let mut rng = StdRng::seed_from_u64(INDEX_SEED); + Buffer::from_iter((0..num_indices).map(|_| rng.random_range(0..filtered_len as u64))) + .into_array() +} + +#[divan::bench(args = NUM_INDICES, consts = FILTERED_LENS)] +fn take_filter_slice_mask_sequential_indices( + bencher: Bencher, + num_indices: usize, +) { + let array = primitive_array().filter(slice_mask(FILTERED_LEN)).unwrap(); + let indices = sequential_indices(num_indices); + + bencher + .with_inputs(|| (&array, &indices, LEGACY_SESSION.create_execution_ctx())) + .bench_refs(|(array, indices, ctx)| { + array + .take(indices.clone()) + .unwrap() + .execute::(ctx) + .unwrap() + }); +} + +#[divan::bench(args = NUM_INDICES, consts = FILTERED_LENS)] +fn take_filter_slice_mask_random_indices( + bencher: Bencher, + num_indices: usize, +) { + let array = primitive_array().filter(slice_mask(FILTERED_LEN)).unwrap(); + let indices = random_indices(num_indices, FILTERED_LEN); + + bencher + .with_inputs(|| (&array, &indices, LEGACY_SESSION.create_execution_ctx())) + .bench_refs(|(array, indices, ctx)| { + array + .take(indices.clone()) + .unwrap() + .execute::(ctx) + .unwrap() + }); +} + +#[divan::bench(args = NUM_INDICES, consts = FILTERED_LENS)] +fn take_filter_random_mask_sequential_indices( + bencher: Bencher, + num_indices: usize, +) { + let array = primitive_array().filter(random_mask(FILTERED_LEN)).unwrap(); + let indices = sequential_indices(num_indices); + + bencher + .with_inputs(|| (&array, &indices, LEGACY_SESSION.create_execution_ctx())) + .bench_refs(|(array, indices, ctx)| { + array + .take(indices.clone()) + .unwrap() + .execute::(ctx) + .unwrap() + }); +} + +#[divan::bench(args = NUM_INDICES, consts = FILTERED_LENS)] +fn take_filter_random_mask_random_indices( + bencher: Bencher, + num_indices: usize, +) { + let array = primitive_array().filter(random_mask(FILTERED_LEN)).unwrap(); + let indices = random_indices(num_indices, FILTERED_LEN); + + bencher + .with_inputs(|| (&array, &indices, LEGACY_SESSION.create_execution_ctx())) + .bench_refs(|(array, indices, ctx)| { + array + .take(indices.clone()) + .unwrap() + .execute::(ctx) + .unwrap() + }); +} diff --git a/vortex-array/benches/take_fsl.rs b/vortex-array/benches/take_fsl.rs index 5c66847669e..baf2d4ede37 100644 --- a/vortex-array/benches/take_fsl.rs +++ b/vortex-array/benches/take_fsl.rs @@ -7,8 +7,8 @@ //! - Number of indices to take //! - Fixed size list length (elements per list) -#![allow(clippy::cast_possible_truncation)] -#![allow(clippy::unwrap_used)] +#![expect(clippy::cast_possible_truncation)] +#![expect(clippy::unwrap_used)] use divan::Bencher; use rand::RngExt; diff --git a/vortex-array/benches/take_patches.rs b/vortex-array/benches/take_patches.rs index 5360aa57c7e..f929cbd3b69 100644 --- a/vortex-array/benches/take_patches.rs +++ b/vortex-array/benches/take_patches.rs @@ -1,8 +1,8 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] -#![allow(clippy::cast_possible_truncation)] +#![expect(clippy::unwrap_used)] +#![expect(clippy::cast_possible_truncation)] use divan::Bencher; use rand::RngExt; @@ -11,7 +11,8 @@ use rand::rngs::StdRng; use vortex_array::ArrayRef; use vortex_array::IntoArray; use vortex_array::LEGACY_SESSION; -use vortex_array::ToCanonical; +#[expect(deprecated)] +use vortex_array::ToCanonical as _; use vortex_array::VortexSessionExecute; use vortex_array::patches::Patches; use vortex_buffer::Buffer; @@ -49,7 +50,9 @@ fn take_search(bencher: Bencher, (patches_sparsity, index_multiple): (f64, f64)) bencher .with_inputs(|| (&patches, &indices, LEGACY_SESSION.create_execution_ctx())) .bench_refs(|(patches, indices, ctx)| { - patches.take_search(indices.to_primitive(), false, ctx) + #[expect(deprecated)] + let prim = indices.to_primitive(); + patches.take_search(prim, false, ctx) }); } @@ -66,7 +69,9 @@ fn take_search_chunked(bencher: Bencher, (patches_sparsity, index_multiple): (f6 bencher .with_inputs(|| (&patches, &indices, LEGACY_SESSION.create_execution_ctx())) .bench_refs(|(patches, indices, ctx)| { - patches.take_search(indices.to_primitive(), false, ctx) + #[expect(deprecated)] + let prim = indices.to_primitive(); + patches.take_search(prim, false, ctx) }); } @@ -82,7 +87,11 @@ fn take_map(bencher: Bencher, (patches_sparsity, index_multiple): (f64, f64)) { bencher .with_inputs(|| (&patches, &indices, LEGACY_SESSION.create_execution_ctx())) - .bench_refs(|(patches, indices, ctx)| patches.take_map(indices.to_primitive(), false, ctx)); + .bench_refs(|(patches, indices, ctx)| { + #[expect(deprecated)] + let prim = indices.to_primitive(); + patches.take_map(prim, false, ctx) + }); } fn fixture(len: usize, sparsity: f64, rng: &mut StdRng) -> Patches { diff --git a/vortex-array/benches/take_primitive.rs b/vortex-array/benches/take_primitive.rs index f5f3ce16c75..4411cf1ac79 100644 --- a/vortex-array/benches/take_primitive.rs +++ b/vortex-array/benches/take_primitive.rs @@ -5,14 +5,17 @@ //! //! Both are tracked by number of indices/codes for fair comparison. -#![allow(clippy::cast_possible_truncation)] -#![allow(clippy::unwrap_used)] +#![expect(clippy::cast_possible_truncation)] +#![expect(clippy::unwrap_used)] use divan::Bencher; use rand::distr::Uniform; use rand::prelude::*; use rand_distr::Zipf; +use vortex_array::Canonical; use vortex_array::IntoArray; +use vortex_array::LEGACY_SESSION; +use vortex_array::VortexSessionExecute; use vortex_array::arrays::DictArray; use vortex_array::arrays::PrimitiveArray; @@ -38,9 +41,12 @@ fn dict_canonicalize_uniform(bencher: Bencher, num_indi let dict = DictArray::try_new(codes.into_array(), values.into_array()).unwrap(); - bencher - .with_inputs(|| &dict) - .bench_refs(|dict| dict.to_canonical()); + bencher.with_inputs(|| &dict).bench_refs(|dict| { + (*dict) + .clone() + .into_array() + .execute::(&mut LEGACY_SESSION.create_execution_ctx()) + }); } #[divan::bench(args = NUM_INDICES, consts = VECTOR_SIZE, sample_count = 100_000)] @@ -57,7 +63,10 @@ fn dict_canonicalize_zipfian(bencher: Bencher, num_indi let dict = DictArray::try_new(codes.into_array(), values.into_array()).unwrap(); - bencher - .with_inputs(|| &dict) - .bench_refs(|dict| dict.to_canonical()); + bencher.with_inputs(|| &dict).bench_refs(|dict| { + (*dict) + .clone() + .into_array() + .execute::(&mut LEGACY_SESSION.create_execution_ctx()) + }); } diff --git a/vortex-array/benches/take_struct.rs b/vortex-array/benches/take_struct.rs index 2eabaab62dc..6346bb705ac 100644 --- a/vortex-array/benches/take_struct.rs +++ b/vortex-array/benches/take_struct.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] +#![expect(clippy::unwrap_used)] use divan::Bencher; use rand::RngExt; diff --git a/vortex-array/benches/varbinview_compact.rs b/vortex-array/benches/varbinview_compact.rs index ae763055db1..99243ce3606 100644 --- a/vortex-array/benches/varbinview_compact.rs +++ b/vortex-array/benches/varbinview_compact.rs @@ -7,7 +7,8 @@ use rand::SeedableRng; use rand::rngs::StdRng; use vortex_array::ArrayRef; use vortex_array::IntoArray; -use vortex_array::ToCanonical; +#[expect(deprecated)] +use vortex_array::ToCanonical as _; use vortex_array::arrays::VarBinViewArray; use vortex_array::builders::VarBinViewBuilder; use vortex_array::dtype::DType; @@ -45,6 +46,7 @@ fn compact_impl(bencher: Bencher, (output_size, utilization_pct): (usize, usize) .into_array() .take(indices) .vortex_expect("operation should succeed in benchmark"); + #[expect(deprecated)] let array = taken.to_varbinview(); bencher.with_inputs(|| &array).bench_refs(|array| { @@ -61,6 +63,7 @@ fn compact_sliced_impl(bencher: Bencher, (output_size, utilization_pct): (usize, .into_array() .slice(0..output_size) .vortex_expect("slice should succeed"); + #[expect(deprecated)] let array = sliced.to_varbinview(); bencher.with_inputs(|| &array).bench_refs(|array| { diff --git a/vortex-array/benches/varbinview_zip.rs b/vortex-array/benches/varbinview_zip.rs index b3ce220f558..b7283bd7c52 100644 --- a/vortex-array/benches/varbinview_zip.rs +++ b/vortex-array/benches/varbinview_zip.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] +#![expect(clippy::unwrap_used)] use divan::Bencher; use vortex_array::IntoArray; diff --git a/vortex-array/public-api.lock b/vortex-array/public-api.lock index e836da72d9d..2db059ce90a 100644 --- a/vortex-array/public-api.lock +++ b/vortex-array/public-api.lock @@ -6,30 +6,170 @@ pub mod vortex_array::accessor pub trait vortex_array::accessor::ArrayAccessor -pub fn vortex_array::accessor::ArrayAccessor::with_iterator(&self, f: F) -> R where F: for<'a> core::ops::function::FnOnce(&mut dyn core::iter::traits::iterator::Iterator>) -> R +pub fn vortex_array::accessor::ArrayAccessor::with_iterator(&self, F) -> R where F: for<'a> core::ops::function::FnOnce(&mut dyn core::iter::traits::iterator::Iterator>) -> R impl vortex_array::accessor::ArrayAccessor<[u8]> for &vortex_array::arrays::VarBinArray -pub fn &vortex_array::arrays::VarBinArray::with_iterator(&self, f: F) -> R where F: for<'a> core::ops::function::FnOnce(&mut dyn core::iter::traits::iterator::Iterator>) -> R +pub fn &vortex_array::arrays::VarBinArray::with_iterator(&self, F) -> R where F: for<'a> core::ops::function::FnOnce(&mut dyn core::iter::traits::iterator::Iterator>) -> R impl vortex_array::accessor::ArrayAccessor<[u8]> for &vortex_array::arrays::VarBinViewArray -pub fn &vortex_array::arrays::VarBinViewArray::with_iterator(&self, f: F) -> R where F: for<'a> core::ops::function::FnOnce(&mut dyn core::iter::traits::iterator::Iterator>) -> R +pub fn &vortex_array::arrays::VarBinViewArray::with_iterator(&self, F) -> R where F: for<'a> core::ops::function::FnOnce(&mut dyn core::iter::traits::iterator::Iterator>) -> R impl vortex_array::accessor::ArrayAccessor<[u8]> for vortex_array::arrays::VarBinArray -pub fn vortex_array::arrays::VarBinArray::with_iterator(&self, f: F) -> R where F: for<'a> core::ops::function::FnOnce(&mut dyn core::iter::traits::iterator::Iterator>) -> R +pub fn vortex_array::arrays::VarBinArray::with_iterator(&self, F) -> R where F: for<'a> core::ops::function::FnOnce(&mut dyn core::iter::traits::iterator::Iterator>) -> R impl vortex_array::accessor::ArrayAccessor<[u8]> for vortex_array::arrays::VarBinViewArray -pub fn vortex_array::arrays::VarBinViewArray::with_iterator core::ops::function::FnOnce(&mut dyn core::iter::traits::iterator::Iterator>) -> R, R>(&self, f: F) -> R +pub fn vortex_array::arrays::VarBinViewArray::with_iterator core::ops::function::FnOnce(&mut dyn core::iter::traits::iterator::Iterator>) -> R, R>(&self, F) -> R impl vortex_array::accessor::ArrayAccessor for vortex_array::arrays::PrimitiveArray -pub fn vortex_array::arrays::PrimitiveArray::with_iterator(&self, f: F) -> R where F: for<'a> core::ops::function::FnOnce(&mut dyn core::iter::traits::iterator::Iterator>) -> R +pub fn vortex_array::arrays::PrimitiveArray::with_iterator(&self, F) -> R where F: for<'a> core::ops::function::FnOnce(&mut dyn core::iter::traits::iterator::Iterator>) -> R pub mod vortex_array::aggregate_fn +pub mod vortex_array::aggregate_fn::combined + +pub struct vortex_array::aggregate_fn::combined::Combined(pub T) + +impl vortex_array::aggregate_fn::combined::Combined + +pub fn vortex_array::aggregate_fn::combined::Combined::new(T) -> Self + +impl core::clone::Clone for vortex_array::aggregate_fn::combined::Combined + +pub fn vortex_array::aggregate_fn::combined::Combined::clone(&self) -> vortex_array::aggregate_fn::combined::Combined + +impl core::fmt::Debug for vortex_array::aggregate_fn::combined::Combined + +pub fn vortex_array::aggregate_fn::combined::Combined::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +impl vortex_array::aggregate_fn::AggregateFnVTable for vortex_array::aggregate_fn::combined::Combined + +pub type vortex_array::aggregate_fn::combined::Combined::Options = vortex_array::aggregate_fn::combined::PairOptions<<::Left as vortex_array::aggregate_fn::AggregateFnVTable>::Options, <::Right as vortex_array::aggregate_fn::AggregateFnVTable>::Options> + +pub type vortex_array::aggregate_fn::combined::Combined::Partial = (<::Left as vortex_array::aggregate_fn::AggregateFnVTable>::Partial, <::Right as vortex_array::aggregate_fn::AggregateFnVTable>::Partial) + +pub fn vortex_array::aggregate_fn::combined::Combined::accumulate(&self, &mut Self::Partial, &vortex_array::Columnar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> + +pub fn vortex_array::aggregate_fn::combined::Combined::coerce_args(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult + +pub fn vortex_array::aggregate_fn::combined::Combined::combine_partials(&self, &mut Self::Partial, vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> + +pub fn vortex_array::aggregate_fn::combined::Combined::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult + +pub fn vortex_array::aggregate_fn::combined::Combined::empty_partial(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult + +pub fn vortex_array::aggregate_fn::combined::Combined::finalize(&self, vortex_array::ArrayRef) -> vortex_error::VortexResult + +pub fn vortex_array::aggregate_fn::combined::Combined::finalize_scalar(&self, &Self::Partial) -> vortex_error::VortexResult + +pub fn vortex_array::aggregate_fn::combined::Combined::id(&self) -> vortex_array::aggregate_fn::AggregateFnId + +pub fn vortex_array::aggregate_fn::combined::Combined::is_saturated(&self, &Self::Partial) -> bool + +pub fn vortex_array::aggregate_fn::combined::Combined::partial_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option + +pub fn vortex_array::aggregate_fn::combined::Combined::reset(&self, &mut Self::Partial) + +pub fn vortex_array::aggregate_fn::combined::Combined::return_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option + +pub fn vortex_array::aggregate_fn::combined::Combined::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> + +pub fn vortex_array::aggregate_fn::combined::Combined::to_scalar(&self, &Self::Partial) -> vortex_error::VortexResult + +pub fn vortex_array::aggregate_fn::combined::Combined::try_accumulate(&self, &mut Self::Partial, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult + +pub struct vortex_array::aggregate_fn::combined::PairOptions(pub L, pub R) + +impl core::marker::StructuralPartialEq for vortex_array::aggregate_fn::combined::PairOptions + +impl core::clone::Clone for vortex_array::aggregate_fn::combined::PairOptions + +pub fn vortex_array::aggregate_fn::combined::PairOptions::clone(&self) -> vortex_array::aggregate_fn::combined::PairOptions + +impl core::cmp::Eq for vortex_array::aggregate_fn::combined::PairOptions + +impl core::cmp::PartialEq for vortex_array::aggregate_fn::combined::PairOptions + +pub fn vortex_array::aggregate_fn::combined::PairOptions::eq(&self, &vortex_array::aggregate_fn::combined::PairOptions) -> bool + +impl core::fmt::Debug for vortex_array::aggregate_fn::combined::PairOptions + +pub fn vortex_array::aggregate_fn::combined::PairOptions::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +impl core::fmt::Display for vortex_array::aggregate_fn::combined::PairOptions + +pub fn vortex_array::aggregate_fn::combined::PairOptions::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +impl core::hash::Hash for vortex_array::aggregate_fn::combined::PairOptions + +pub fn vortex_array::aggregate_fn::combined::PairOptions::hash<__H: core::hash::Hasher>(&self, &mut __H) + +pub trait vortex_array::aggregate_fn::combined::BinaryCombined: 'static + core::marker::Send + core::marker::Sync + core::clone::Clone + +pub type vortex_array::aggregate_fn::combined::BinaryCombined::Left: vortex_array::aggregate_fn::AggregateFnVTable + +pub type vortex_array::aggregate_fn::combined::BinaryCombined::Right: vortex_array::aggregate_fn::AggregateFnVTable + +pub fn vortex_array::aggregate_fn::combined::BinaryCombined::coerce_args(&self, &vortex_array::aggregate_fn::combined::CombinedOptions, &vortex_array::dtype::DType) -> vortex_error::VortexResult + +pub fn vortex_array::aggregate_fn::combined::BinaryCombined::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult> + +pub fn vortex_array::aggregate_fn::combined::BinaryCombined::finalize(&self, vortex_array::ArrayRef, vortex_array::ArrayRef) -> vortex_error::VortexResult + +pub fn vortex_array::aggregate_fn::combined::BinaryCombined::finalize_scalar(&self, vortex_array::scalar::Scalar, vortex_array::scalar::Scalar) -> vortex_error::VortexResult + +pub fn vortex_array::aggregate_fn::combined::BinaryCombined::id(&self) -> vortex_array::aggregate_fn::AggregateFnId + +pub fn vortex_array::aggregate_fn::combined::BinaryCombined::left(&self) -> Self::Left + +pub fn vortex_array::aggregate_fn::combined::BinaryCombined::left_name(&self) -> &'static str + +pub fn vortex_array::aggregate_fn::combined::BinaryCombined::partial_struct_dtype(&self, vortex_array::dtype::DType, vortex_array::dtype::DType) -> vortex_array::dtype::DType + +pub fn vortex_array::aggregate_fn::combined::BinaryCombined::return_dtype(&self, &vortex_array::dtype::DType) -> core::option::Option + +pub fn vortex_array::aggregate_fn::combined::BinaryCombined::right(&self) -> Self::Right + +pub fn vortex_array::aggregate_fn::combined::BinaryCombined::right_name(&self) -> &'static str + +pub fn vortex_array::aggregate_fn::combined::BinaryCombined::serialize(&self, &vortex_array::aggregate_fn::combined::CombinedOptions) -> vortex_error::VortexResult>> + +impl vortex_array::aggregate_fn::combined::BinaryCombined for vortex_array::aggregate_fn::fns::mean::Mean + +pub type vortex_array::aggregate_fn::fns::mean::Mean::Left = vortex_array::aggregate_fn::fns::sum::Sum + +pub type vortex_array::aggregate_fn::fns::mean::Mean::Right = vortex_array::aggregate_fn::fns::count::Count + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::coerce_args(&self, &vortex_array::aggregate_fn::combined::PairOptions<::Options, ::Options>, &vortex_array::dtype::DType) -> vortex_error::VortexResult + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult> + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::finalize(&self, vortex_array::ArrayRef, vortex_array::ArrayRef) -> vortex_error::VortexResult + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::finalize_scalar(&self, vortex_array::scalar::Scalar, vortex_array::scalar::Scalar) -> vortex_error::VortexResult + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::id(&self) -> vortex_array::aggregate_fn::AggregateFnId + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::left(&self) -> vortex_array::aggregate_fn::fns::sum::Sum + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::left_name(&self) -> &'static str + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::partial_struct_dtype(&self, vortex_array::dtype::DType, vortex_array::dtype::DType) -> vortex_array::dtype::DType + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::return_dtype(&self, &vortex_array::dtype::DType) -> core::option::Option + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::right(&self) -> vortex_array::aggregate_fn::fns::count::Count + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::right_name(&self) -> &'static str + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::serialize(&self, &vortex_array::aggregate_fn::combined::CombinedOptions) -> vortex_error::VortexResult>> + +pub type vortex_array::aggregate_fn::combined::CombinedOptions = vortex_array::aggregate_fn::combined::PairOptions<<::Left as vortex_array::aggregate_fn::AggregateFnVTable>::Options, <::Right as vortex_array::aggregate_fn::AggregateFnVTable>::Options> + pub mod vortex_array::aggregate_fn::fns pub mod vortex_array::aggregate_fn::fns::count @@ -42,7 +182,7 @@ pub fn vortex_array::aggregate_fn::fns::count::Count::clone(&self) -> vortex_arr impl core::fmt::Debug for vortex_array::aggregate_fn::fns::count::Count -pub fn vortex_array::aggregate_fn::fns::count::Count::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::aggregate_fn::fns::count::Count::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::aggregate_fn::AggregateFnVTable for vortex_array::aggregate_fn::fns::count::Count @@ -50,35 +190,35 @@ pub type vortex_array::aggregate_fn::fns::count::Count::Options = vortex_array:: pub type vortex_array::aggregate_fn::fns::count::Count::Partial = u64 -pub fn vortex_array::aggregate_fn::fns::count::Count::accumulate(&self, _partial: &mut Self::Partial, _batch: &vortex_array::Columnar, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::count::Count::accumulate(&self, &mut Self::Partial, &vortex_array::Columnar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::count::Count::coerce_args(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::count::Count::coerce_args(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::count::Count::combine_partials(&self, partial: &mut Self::Partial, other: vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::count::Count::combine_partials(&self, &mut Self::Partial, vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::count::Count::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::count::Count::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::count::Count::empty_partial(&self, _options: &Self::Options, _input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::count::Count::empty_partial(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::count::Count::finalize(&self, partials: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::count::Count::finalize(&self, vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::count::Count::finalize_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::count::Count::finalize_scalar(&self, &Self::Partial) -> vortex_error::VortexResult pub fn vortex_array::aggregate_fn::fns::count::Count::id(&self) -> vortex_array::aggregate_fn::AggregateFnId -pub fn vortex_array::aggregate_fn::fns::count::Count::is_saturated(&self, _partial: &Self::Partial) -> bool +pub fn vortex_array::aggregate_fn::fns::count::Count::is_saturated(&self, &Self::Partial) -> bool -pub fn vortex_array::aggregate_fn::fns::count::Count::partial_dtype(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::count::Count::partial_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::count::Count::reset(&self, partial: &mut Self::Partial) +pub fn vortex_array::aggregate_fn::fns::count::Count::reset(&self, &mut Self::Partial) -pub fn vortex_array::aggregate_fn::fns::count::Count::return_dtype(&self, _options: &Self::Options, _input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::count::Count::return_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::count::Count::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::aggregate_fn::fns::count::Count::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::aggregate_fn::fns::count::Count::to_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::count::Count::to_scalar(&self, &Self::Partial) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::count::Count::try_accumulate(&self, state: &mut Self::Partial, batch: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::count::Count::try_accumulate(&self, &mut Self::Partial, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub mod vortex_array::aggregate_fn::fns::first @@ -90,7 +230,7 @@ pub fn vortex_array::aggregate_fn::fns::first::First::clone(&self) -> vortex_arr impl core::fmt::Debug for vortex_array::aggregate_fn::fns::first::First -pub fn vortex_array::aggregate_fn::fns::first::First::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::aggregate_fn::fns::first::First::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::aggregate_fn::AggregateFnVTable for vortex_array::aggregate_fn::fns::first::First @@ -98,39 +238,39 @@ pub type vortex_array::aggregate_fn::fns::first::First::Options = vortex_array:: pub type vortex_array::aggregate_fn::fns::first::First::Partial = vortex_array::aggregate_fn::fns::first::FirstPartial -pub fn vortex_array::aggregate_fn::fns::first::First::accumulate(&self, _partial: &mut Self::Partial, _batch: &vortex_array::Columnar, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::first::First::accumulate(&self, &mut Self::Partial, &vortex_array::Columnar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::first::First::coerce_args(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::first::First::coerce_args(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::first::First::combine_partials(&self, partial: &mut Self::Partial, other: vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::first::First::combine_partials(&self, &mut Self::Partial, vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::first::First::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::first::First::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::first::First::empty_partial(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::first::First::empty_partial(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::first::First::finalize(&self, partials: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::first::First::finalize(&self, vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::first::First::finalize_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::first::First::finalize_scalar(&self, &Self::Partial) -> vortex_error::VortexResult pub fn vortex_array::aggregate_fn::fns::first::First::id(&self) -> vortex_array::aggregate_fn::AggregateFnId -pub fn vortex_array::aggregate_fn::fns::first::First::is_saturated(&self, partial: &Self::Partial) -> bool +pub fn vortex_array::aggregate_fn::fns::first::First::is_saturated(&self, &Self::Partial) -> bool -pub fn vortex_array::aggregate_fn::fns::first::First::partial_dtype(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::first::First::partial_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::first::First::reset(&self, partial: &mut Self::Partial) +pub fn vortex_array::aggregate_fn::fns::first::First::reset(&self, &mut Self::Partial) -pub fn vortex_array::aggregate_fn::fns::first::First::return_dtype(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::first::First::return_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::first::First::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::aggregate_fn::fns::first::First::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::aggregate_fn::fns::first::First::to_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::first::First::to_scalar(&self, &Self::Partial) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::first::First::try_accumulate(&self, partial: &mut Self::Partial, batch: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::first::First::try_accumulate(&self, &mut Self::Partial, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub struct vortex_array::aggregate_fn::fns::first::FirstPartial -pub fn vortex_array::aggregate_fn::fns::first::first(array: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::first::first(&vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub mod vortex_array::aggregate_fn::fns::is_constant @@ -138,13 +278,13 @@ pub mod vortex_array::aggregate_fn::fns::is_constant::primitive pub const vortex_array::aggregate_fn::fns::is_constant::primitive::IS_CONST_LANE_WIDTH: usize -pub fn vortex_array::aggregate_fn::fns::is_constant::primitive::compute_is_constant(values: &[T]) -> bool +pub fn vortex_array::aggregate_fn::fns::is_constant::primitive::compute_is_constant(&[T]) -> bool pub struct vortex_array::aggregate_fn::fns::is_constant::IsConstant impl vortex_array::aggregate_fn::fns::is_constant::IsConstant -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::make_partial(batch: &vortex_array::ArrayRef, is_constant: bool) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::make_partial(&vortex_array::ArrayRef, bool, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl core::clone::Clone for vortex_array::aggregate_fn::fns::is_constant::IsConstant @@ -152,7 +292,7 @@ pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::clone(&self) -> impl core::fmt::Debug for vortex_array::aggregate_fn::fns::is_constant::IsConstant -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::aggregate_fn::AggregateFnVTable for vortex_array::aggregate_fn::fns::is_constant::IsConstant @@ -160,41 +300,41 @@ pub type vortex_array::aggregate_fn::fns::is_constant::IsConstant::Options = vor pub type vortex_array::aggregate_fn::fns::is_constant::IsConstant::Partial = vortex_array::aggregate_fn::fns::is_constant::IsConstantPartial -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::accumulate(&self, partial: &mut Self::Partial, batch: &vortex_array::Columnar, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::accumulate(&self, &mut Self::Partial, &vortex_array::Columnar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::coerce_args(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::coerce_args(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::combine_partials(&self, partial: &mut Self::Partial, other: vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::combine_partials(&self, &mut Self::Partial, vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::empty_partial(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::empty_partial(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::finalize(&self, partials: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::finalize(&self, vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::finalize_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::finalize_scalar(&self, &Self::Partial) -> vortex_error::VortexResult pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::id(&self) -> vortex_array::aggregate_fn::AggregateFnId -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::is_saturated(&self, partial: &Self::Partial) -> bool +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::is_saturated(&self, &Self::Partial) -> bool -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::partial_dtype(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::partial_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::reset(&self, partial: &mut Self::Partial) +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::reset(&self, &mut Self::Partial) -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::return_dtype(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::return_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::to_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::to_scalar(&self, &Self::Partial) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::try_accumulate(&self, _state: &mut Self::Partial, _batch: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::try_accumulate(&self, &mut Self::Partial, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub struct vortex_array::aggregate_fn::fns::is_constant::IsConstantPartial -pub fn vortex_array::aggregate_fn::fns::is_constant::is_constant(array: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_constant::is_constant(&vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::is_constant::make_is_constant_partial_dtype(element_dtype: &vortex_array::dtype::DType) -> vortex_array::dtype::DType +pub fn vortex_array::aggregate_fn::fns::is_constant::make_is_constant_partial_dtype(&vortex_array::dtype::DType) -> vortex_array::dtype::DType pub mod vortex_array::aggregate_fn::fns::is_sorted @@ -202,7 +342,7 @@ pub struct vortex_array::aggregate_fn::fns::is_sorted::IsSorted impl vortex_array::aggregate_fn::fns::is_sorted::IsSorted -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::make_partial(batch: &vortex_array::ArrayRef, is_sorted: bool, strict: bool) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::make_partial(&vortex_array::ArrayRef, bool, bool, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl core::clone::Clone for vortex_array::aggregate_fn::fns::is_sorted::IsSorted @@ -210,7 +350,7 @@ pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::clone(&self) -> vor impl core::fmt::Debug for vortex_array::aggregate_fn::fns::is_sorted::IsSorted -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::aggregate_fn::AggregateFnVTable for vortex_array::aggregate_fn::fns::is_sorted::IsSorted @@ -218,35 +358,35 @@ pub type vortex_array::aggregate_fn::fns::is_sorted::IsSorted::Options = vortex_ pub type vortex_array::aggregate_fn::fns::is_sorted::IsSorted::Partial = vortex_array::aggregate_fn::fns::is_sorted::IsSortedPartial -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::accumulate(&self, partial: &mut Self::Partial, batch: &vortex_array::Columnar, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::accumulate(&self, &mut Self::Partial, &vortex_array::Columnar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::coerce_args(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::coerce_args(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::combine_partials(&self, partial: &mut Self::Partial, other: vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::combine_partials(&self, &mut Self::Partial, vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::empty_partial(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::empty_partial(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::finalize(&self, partials: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::finalize(&self, vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::finalize_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::finalize_scalar(&self, &Self::Partial) -> vortex_error::VortexResult pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::id(&self) -> vortex_array::aggregate_fn::AggregateFnId -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::is_saturated(&self, partial: &Self::Partial) -> bool +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::is_saturated(&self, &Self::Partial) -> bool -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::partial_dtype(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::partial_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::reset(&self, partial: &mut Self::Partial) +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::reset(&self, &mut Self::Partial) -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::return_dtype(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::return_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::to_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::to_scalar(&self, &Self::Partial) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::try_accumulate(&self, _state: &mut Self::Partial, _batch: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::try_accumulate(&self, &mut Self::Partial, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub struct vortex_array::aggregate_fn::fns::is_sorted::IsSortedOptions @@ -260,19 +400,19 @@ impl core::cmp::Eq for vortex_array::aggregate_fn::fns::is_sorted::IsSortedOptio impl core::cmp::PartialEq for vortex_array::aggregate_fn::fns::is_sorted::IsSortedOptions -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSortedOptions::eq(&self, other: &vortex_array::aggregate_fn::fns::is_sorted::IsSortedOptions) -> bool +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSortedOptions::eq(&self, &vortex_array::aggregate_fn::fns::is_sorted::IsSortedOptions) -> bool impl core::fmt::Debug for vortex_array::aggregate_fn::fns::is_sorted::IsSortedOptions -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSortedOptions::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSortedOptions::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::aggregate_fn::fns::is_sorted::IsSortedOptions -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSortedOptions::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSortedOptions::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::aggregate_fn::fns::is_sorted::IsSortedOptions -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSortedOptions::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSortedOptions::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_array::aggregate_fn::fns::is_sorted::IsSortedOptions @@ -286,11 +426,11 @@ impl vortex_array::aggregate_fn::fns::is_sorted::IsSortedIteratorExt for T wh pub fn T::is_strict_sorted(self) -> bool where Self: core::marker::Sized, Self::Item: core::cmp::PartialOrd -pub fn vortex_array::aggregate_fn::fns::is_sorted::is_sorted(array: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_sorted::is_sorted(&vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::is_sorted::is_strict_sorted(array: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_sorted::is_strict_sorted(&vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::is_sorted::make_is_sorted_partial_dtype(element_dtype: &vortex_array::dtype::DType) -> vortex_array::dtype::DType +pub fn vortex_array::aggregate_fn::fns::is_sorted::make_is_sorted_partial_dtype(&vortex_array::dtype::DType) -> vortex_array::dtype::DType pub mod vortex_array::aggregate_fn::fns::last @@ -302,7 +442,7 @@ pub fn vortex_array::aggregate_fn::fns::last::Last::clone(&self) -> vortex_array impl core::fmt::Debug for vortex_array::aggregate_fn::fns::last::Last -pub fn vortex_array::aggregate_fn::fns::last::Last::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::aggregate_fn::fns::last::Last::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::aggregate_fn::AggregateFnVTable for vortex_array::aggregate_fn::fns::last::Last @@ -310,39 +450,87 @@ pub type vortex_array::aggregate_fn::fns::last::Last::Options = vortex_array::ag pub type vortex_array::aggregate_fn::fns::last::Last::Partial = vortex_array::aggregate_fn::fns::last::LastPartial -pub fn vortex_array::aggregate_fn::fns::last::Last::accumulate(&self, _partial: &mut Self::Partial, _batch: &vortex_array::Columnar, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::last::Last::accumulate(&self, &mut Self::Partial, &vortex_array::Columnar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::last::Last::coerce_args(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::last::Last::coerce_args(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::last::Last::combine_partials(&self, partial: &mut Self::Partial, other: vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::last::Last::combine_partials(&self, &mut Self::Partial, vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::last::Last::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::last::Last::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::last::Last::empty_partial(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::last::Last::empty_partial(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::last::Last::finalize(&self, partials: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::last::Last::finalize(&self, vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::last::Last::finalize_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::last::Last::finalize_scalar(&self, &Self::Partial) -> vortex_error::VortexResult pub fn vortex_array::aggregate_fn::fns::last::Last::id(&self) -> vortex_array::aggregate_fn::AggregateFnId -pub fn vortex_array::aggregate_fn::fns::last::Last::is_saturated(&self, _partial: &Self::Partial) -> bool +pub fn vortex_array::aggregate_fn::fns::last::Last::is_saturated(&self, &Self::Partial) -> bool -pub fn vortex_array::aggregate_fn::fns::last::Last::partial_dtype(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::last::Last::partial_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::last::Last::reset(&self, partial: &mut Self::Partial) +pub fn vortex_array::aggregate_fn::fns::last::Last::reset(&self, &mut Self::Partial) -pub fn vortex_array::aggregate_fn::fns::last::Last::return_dtype(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::last::Last::return_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::last::Last::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::aggregate_fn::fns::last::Last::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::aggregate_fn::fns::last::Last::to_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::last::Last::to_scalar(&self, &Self::Partial) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::last::Last::try_accumulate(&self, partial: &mut Self::Partial, batch: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::last::Last::try_accumulate(&self, &mut Self::Partial, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub struct vortex_array::aggregate_fn::fns::last::LastPartial -pub fn vortex_array::aggregate_fn::fns::last::last(array: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::last::last(&vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult + +pub mod vortex_array::aggregate_fn::fns::mean + +pub struct vortex_array::aggregate_fn::fns::mean::Mean + +impl vortex_array::aggregate_fn::fns::mean::Mean + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::combined() -> vortex_array::aggregate_fn::combined::Combined + +impl core::clone::Clone for vortex_array::aggregate_fn::fns::mean::Mean + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::clone(&self) -> vortex_array::aggregate_fn::fns::mean::Mean + +impl core::fmt::Debug for vortex_array::aggregate_fn::fns::mean::Mean + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +impl vortex_array::aggregate_fn::combined::BinaryCombined for vortex_array::aggregate_fn::fns::mean::Mean + +pub type vortex_array::aggregate_fn::fns::mean::Mean::Left = vortex_array::aggregate_fn::fns::sum::Sum + +pub type vortex_array::aggregate_fn::fns::mean::Mean::Right = vortex_array::aggregate_fn::fns::count::Count + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::coerce_args(&self, &vortex_array::aggregate_fn::combined::PairOptions<::Options, ::Options>, &vortex_array::dtype::DType) -> vortex_error::VortexResult + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult> + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::finalize(&self, vortex_array::ArrayRef, vortex_array::ArrayRef) -> vortex_error::VortexResult + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::finalize_scalar(&self, vortex_array::scalar::Scalar, vortex_array::scalar::Scalar) -> vortex_error::VortexResult + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::id(&self) -> vortex_array::aggregate_fn::AggregateFnId + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::left(&self) -> vortex_array::aggregate_fn::fns::sum::Sum + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::left_name(&self) -> &'static str + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::partial_struct_dtype(&self, vortex_array::dtype::DType, vortex_array::dtype::DType) -> vortex_array::dtype::DType + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::return_dtype(&self, &vortex_array::dtype::DType) -> core::option::Option + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::right(&self) -> vortex_array::aggregate_fn::fns::count::Count + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::right_name(&self) -> &'static str + +pub fn vortex_array::aggregate_fn::fns::mean::Mean::serialize(&self, &vortex_array::aggregate_fn::combined::CombinedOptions) -> vortex_error::VortexResult>> + +pub fn vortex_array::aggregate_fn::fns::mean::mean(&vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub mod vortex_array::aggregate_fn::fns::min_max @@ -354,7 +542,7 @@ pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::clone(&self) -> vortex_ impl core::fmt::Debug for vortex_array::aggregate_fn::fns::min_max::MinMax -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::aggregate_fn::AggregateFnVTable for vortex_array::aggregate_fn::fns::min_max::MinMax @@ -362,35 +550,35 @@ pub type vortex_array::aggregate_fn::fns::min_max::MinMax::Options = vortex_arra pub type vortex_array::aggregate_fn::fns::min_max::MinMax::Partial = vortex_array::aggregate_fn::fns::min_max::MinMaxPartial -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::accumulate(&self, partial: &mut Self::Partial, batch: &vortex_array::Columnar, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::accumulate(&self, &mut Self::Partial, &vortex_array::Columnar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::coerce_args(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::coerce_args(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::combine_partials(&self, partial: &mut Self::Partial, other: vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::combine_partials(&self, &mut Self::Partial, vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::empty_partial(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::empty_partial(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::finalize(&self, partials: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::finalize(&self, vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::finalize_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::finalize_scalar(&self, &Self::Partial) -> vortex_error::VortexResult pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::id(&self) -> vortex_array::aggregate_fn::AggregateFnId -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::is_saturated(&self, _partial: &Self::Partial) -> bool +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::is_saturated(&self, &Self::Partial) -> bool -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::partial_dtype(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::partial_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::reset(&self, partial: &mut Self::Partial) +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::reset(&self, &mut Self::Partial) -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::return_dtype(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::return_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::to_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::to_scalar(&self, &Self::Partial) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::try_accumulate(&self, _state: &mut Self::Partial, _batch: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::try_accumulate(&self, &mut Self::Partial, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub struct vortex_array::aggregate_fn::fns::min_max::MinMaxPartial @@ -402,7 +590,7 @@ pub vortex_array::aggregate_fn::fns::min_max::MinMaxResult::min: vortex_array::s impl vortex_array::aggregate_fn::fns::min_max::MinMaxResult -pub fn vortex_array::aggregate_fn::fns::min_max::MinMaxResult::from_scalar(scalar: vortex_array::scalar::Scalar) -> vortex_error::VortexResult> +pub fn vortex_array::aggregate_fn::fns::min_max::MinMaxResult::from_scalar(vortex_array::scalar::Scalar) -> vortex_error::VortexResult> impl core::clone::Clone for vortex_array::aggregate_fn::fns::min_max::MinMaxResult @@ -412,17 +600,17 @@ impl core::cmp::Eq for vortex_array::aggregate_fn::fns::min_max::MinMaxResult impl core::cmp::PartialEq for vortex_array::aggregate_fn::fns::min_max::MinMaxResult -pub fn vortex_array::aggregate_fn::fns::min_max::MinMaxResult::eq(&self, other: &vortex_array::aggregate_fn::fns::min_max::MinMaxResult) -> bool +pub fn vortex_array::aggregate_fn::fns::min_max::MinMaxResult::eq(&self, &vortex_array::aggregate_fn::fns::min_max::MinMaxResult) -> bool impl core::fmt::Debug for vortex_array::aggregate_fn::fns::min_max::MinMaxResult -pub fn vortex_array::aggregate_fn::fns::min_max::MinMaxResult::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::aggregate_fn::fns::min_max::MinMaxResult::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::StructuralPartialEq for vortex_array::aggregate_fn::fns::min_max::MinMaxResult -pub fn vortex_array::aggregate_fn::fns::min_max::make_minmax_dtype(element_dtype: &vortex_array::dtype::DType) -> vortex_array::dtype::DType +pub fn vortex_array::aggregate_fn::fns::min_max::make_minmax_dtype(&vortex_array::dtype::DType) -> vortex_array::dtype::DType -pub fn vortex_array::aggregate_fn::fns::min_max::min_max(array: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::aggregate_fn::fns::min_max::min_max(&vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub mod vortex_array::aggregate_fn::fns::nan_count @@ -434,7 +622,7 @@ pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::clone(&self) -> vor impl core::fmt::Debug for vortex_array::aggregate_fn::fns::nan_count::NanCount -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::aggregate_fn::AggregateFnVTable for vortex_array::aggregate_fn::fns::nan_count::NanCount @@ -442,37 +630,37 @@ pub type vortex_array::aggregate_fn::fns::nan_count::NanCount::Options = vortex_ pub type vortex_array::aggregate_fn::fns::nan_count::NanCount::Partial = u64 -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::accumulate(&self, partial: &mut Self::Partial, batch: &vortex_array::Columnar, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::accumulate(&self, &mut Self::Partial, &vortex_array::Columnar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::coerce_args(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::coerce_args(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::combine_partials(&self, partial: &mut Self::Partial, other: vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::combine_partials(&self, &mut Self::Partial, vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::empty_partial(&self, _options: &Self::Options, _input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::empty_partial(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::finalize(&self, partials: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::finalize(&self, vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::finalize_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::finalize_scalar(&self, &Self::Partial) -> vortex_error::VortexResult pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::id(&self) -> vortex_array::aggregate_fn::AggregateFnId -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::is_saturated(&self, _partial: &Self::Partial) -> bool +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::is_saturated(&self, &Self::Partial) -> bool -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::partial_dtype(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::partial_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::reset(&self, partial: &mut Self::Partial) +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::reset(&self, &mut Self::Partial) -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::return_dtype(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::return_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::to_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::to_scalar(&self, &Self::Partial) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::try_accumulate(&self, _state: &mut Self::Partial, _batch: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::try_accumulate(&self, &mut Self::Partial, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::nan_count::nan_count(array: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::nan_count::nan_count(&vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub mod vortex_array::aggregate_fn::fns::sum @@ -498,7 +686,7 @@ pub fn vortex_array::aggregate_fn::fns::sum::Sum::clone(&self) -> vortex_array:: impl core::fmt::Debug for vortex_array::aggregate_fn::fns::sum::Sum -pub fn vortex_array::aggregate_fn::fns::sum::Sum::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::aggregate_fn::fns::sum::Sum::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::aggregate_fn::AggregateFnVTable for vortex_array::aggregate_fn::fns::sum::Sum @@ -506,51 +694,51 @@ pub type vortex_array::aggregate_fn::fns::sum::Sum::Options = vortex_array::aggr pub type vortex_array::aggregate_fn::fns::sum::Sum::Partial = vortex_array::aggregate_fn::fns::sum::SumPartial -pub fn vortex_array::aggregate_fn::fns::sum::Sum::accumulate(&self, partial: &mut Self::Partial, batch: &vortex_array::Columnar, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::sum::Sum::accumulate(&self, &mut Self::Partial, &vortex_array::Columnar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::sum::Sum::coerce_args(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::sum::Sum::coerce_args(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::sum::Sum::combine_partials(&self, partial: &mut Self::Partial, other: vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::sum::Sum::combine_partials(&self, &mut Self::Partial, vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::sum::Sum::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::sum::Sum::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::sum::Sum::empty_partial(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::sum::Sum::empty_partial(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::sum::Sum::finalize(&self, partials: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::sum::Sum::finalize(&self, vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::sum::Sum::finalize_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::sum::Sum::finalize_scalar(&self, &Self::Partial) -> vortex_error::VortexResult pub fn vortex_array::aggregate_fn::fns::sum::Sum::id(&self) -> vortex_array::aggregate_fn::AggregateFnId -pub fn vortex_array::aggregate_fn::fns::sum::Sum::is_saturated(&self, partial: &Self::Partial) -> bool +pub fn vortex_array::aggregate_fn::fns::sum::Sum::is_saturated(&self, &Self::Partial) -> bool -pub fn vortex_array::aggregate_fn::fns::sum::Sum::partial_dtype(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::sum::Sum::partial_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::sum::Sum::reset(&self, partial: &mut Self::Partial) +pub fn vortex_array::aggregate_fn::fns::sum::Sum::reset(&self, &mut Self::Partial) -pub fn vortex_array::aggregate_fn::fns::sum::Sum::return_dtype(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::sum::Sum::return_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::sum::Sum::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::aggregate_fn::fns::sum::Sum::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::aggregate_fn::fns::sum::Sum::to_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::sum::Sum::to_scalar(&self, &Self::Partial) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::sum::Sum::try_accumulate(&self, _state: &mut Self::Partial, _batch: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::sum::Sum::try_accumulate(&self, &mut Self::Partial, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub struct vortex_array::aggregate_fn::fns::sum::SumPartial -pub fn vortex_array::aggregate_fn::fns::sum::sum(array: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::sum::sum(&vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub mod vortex_array::aggregate_fn::kernels pub trait vortex_array::aggregate_fn::kernels::DynAggregateKernel: 'static + core::marker::Send + core::marker::Sync + core::fmt::Debug -pub fn vortex_array::aggregate_fn::kernels::DynAggregateKernel::aggregate(&self, aggregate_fn: &vortex_array::aggregate_fn::AggregateFnRef, batch: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::aggregate_fn::kernels::DynAggregateKernel::aggregate(&self, &vortex_array::aggregate_fn::AggregateFnRef, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub trait vortex_array::aggregate_fn::kernels::DynGroupedAggregateKernel: 'static + core::marker::Send + core::marker::Sync + core::fmt::Debug -pub fn vortex_array::aggregate_fn::kernels::DynGroupedAggregateKernel::grouped_aggregate(&self, aggregate_fn: &vortex_array::aggregate_fn::AggregateFnRef, groups: &vortex_array::arrays::ListViewArray) -> vortex_error::VortexResult> +pub fn vortex_array::aggregate_fn::kernels::DynGroupedAggregateKernel::grouped_aggregate(&self, &vortex_array::aggregate_fn::AggregateFnRef, &vortex_array::arrays::ListViewArray) -> vortex_error::VortexResult> -pub fn vortex_array::aggregate_fn::kernels::DynGroupedAggregateKernel::grouped_aggregate_fixed_size(&self, aggregate_fn: &vortex_array::aggregate_fn::AggregateFnRef, groups: &vortex_array::arrays::FixedSizeListArray) -> vortex_error::VortexResult> +pub fn vortex_array::aggregate_fn::kernels::DynGroupedAggregateKernel::grouped_aggregate_fixed_size(&self, &vortex_array::aggregate_fn::AggregateFnRef, &vortex_array::arrays::FixedSizeListArray) -> vortex_error::VortexResult> pub mod vortex_array::aggregate_fn::proto @@ -560,9 +748,9 @@ pub struct vortex_array::aggregate_fn::session::AggregateFnSession impl vortex_array::aggregate_fn::session::AggregateFnSession -pub fn vortex_array::aggregate_fn::session::AggregateFnSession::register(&self, vtable: V) +pub fn vortex_array::aggregate_fn::session::AggregateFnSession::register(&self, V) -pub fn vortex_array::aggregate_fn::session::AggregateFnSession::register_aggregate_kernel(&self, array_id: vortex_array::ArrayId, agg_fn_id: core::option::Option, kernel: &'static dyn vortex_array::aggregate_fn::kernels::DynAggregateKernel) +pub fn vortex_array::aggregate_fn::session::AggregateFnSession::register_aggregate_kernel(&self, impl core::convert::Into, core::option::Option>, &'static dyn vortex_array::aggregate_fn::kernels::DynAggregateKernel) pub fn vortex_array::aggregate_fn::session::AggregateFnSession::registry(&self) -> &vortex_array::aggregate_fn::session::AggregateFnRegistry @@ -572,7 +760,13 @@ pub fn vortex_array::aggregate_fn::session::AggregateFnSession::default() -> Sel impl core::fmt::Debug for vortex_array::aggregate_fn::session::AggregateFnSession -pub fn vortex_array::aggregate_fn::session::AggregateFnSession::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::aggregate_fn::session::AggregateFnSession::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +impl vortex_session::SessionVar for vortex_array::aggregate_fn::session::AggregateFnSession + +pub fn vortex_array::aggregate_fn::session::AggregateFnSession::as_any(&self) -> &dyn core::any::Any + +pub fn vortex_array::aggregate_fn::session::AggregateFnSession::as_any_mut(&mut self) -> &mut dyn core::any::Any pub trait vortex_array::aggregate_fn::session::AggregateFnSessionExt: vortex_session::SessionExt @@ -588,11 +782,11 @@ pub struct vortex_array::aggregate_fn::Accumulator vortex_array::aggregate_fn::Accumulator -pub fn vortex_array::aggregate_fn::Accumulator::try_new(vtable: V, options: ::Options, dtype: vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::Accumulator::try_new(V, ::Options, vortex_array::dtype::DType) -> vortex_error::VortexResult impl vortex_array::aggregate_fn::DynAccumulator for vortex_array::aggregate_fn::Accumulator -pub fn vortex_array::aggregate_fn::Accumulator::accumulate(&mut self, batch: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::Accumulator::accumulate(&mut self, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> pub fn vortex_array::aggregate_fn::Accumulator::finish(&mut self) -> vortex_error::VortexResult @@ -606,7 +800,7 @@ impl vortex_array::aggregate_f pub fn vortex_array::aggregate_fn::AggregateFn::erased(self) -> vortex_array::aggregate_fn::AggregateFnRef -pub fn vortex_array::aggregate_fn::AggregateFn::new(vtable: V, options: ::Options) -> Self +pub fn vortex_array::aggregate_fn::AggregateFn::new(V, ::Options) -> Self pub fn vortex_array::aggregate_fn::AggregateFn::options(&self) -> &::Options @@ -626,33 +820,33 @@ impl core::cmp::Eq for vortex_array::aggregate_fn::AggregateFnOptions<'_> impl core::cmp::PartialEq for vortex_array::aggregate_fn::AggregateFnOptions<'_> -pub fn vortex_array::aggregate_fn::AggregateFnOptions<'_>::eq(&self, other: &Self) -> bool +pub fn vortex_array::aggregate_fn::AggregateFnOptions<'_>::eq(&self, &Self) -> bool impl core::fmt::Debug for vortex_array::aggregate_fn::AggregateFnOptions<'_> -pub fn vortex_array::aggregate_fn::AggregateFnOptions<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::aggregate_fn::AggregateFnOptions<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::aggregate_fn::AggregateFnOptions<'_> -pub fn vortex_array::aggregate_fn::AggregateFnOptions<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::aggregate_fn::AggregateFnOptions<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::aggregate_fn::AggregateFnOptions<'_> -pub fn vortex_array::aggregate_fn::AggregateFnOptions<'_>::hash(&self, state: &mut H) +pub fn vortex_array::aggregate_fn::AggregateFnOptions<'_>::hash(&self, &mut H) pub struct vortex_array::aggregate_fn::AggregateFnRef(_) impl vortex_array::aggregate_fn::AggregateFnRef -pub fn vortex_array::aggregate_fn::AggregateFnRef::accumulator(&self, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::AggregateFnRef::accumulator(&self, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::AggregateFnRef::accumulator_grouped(&self, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::AggregateFnRef::accumulator_grouped(&self, &vortex_array::dtype::DType) -> vortex_error::VortexResult pub fn vortex_array::aggregate_fn::AggregateFnRef::as_(&self) -> &::Options pub fn vortex_array::aggregate_fn::AggregateFnRef::as_opt(&self) -> core::option::Option<&::Options> -pub fn vortex_array::aggregate_fn::AggregateFnRef::coerce_args(&self, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::AggregateFnRef::coerce_args(&self, &vortex_array::dtype::DType) -> vortex_error::VortexResult pub fn vortex_array::aggregate_fn::AggregateFnRef::id(&self) -> vortex_array::aggregate_fn::AggregateFnId @@ -660,15 +854,15 @@ pub fn vortex_array::aggregate_fn::AggregateFnRef::is vortex_array::aggregate_fn::AggregateFnOptions<'_> -pub fn vortex_array::aggregate_fn::AggregateFnRef::return_dtype(&self, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::AggregateFnRef::return_dtype(&self, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::AggregateFnRef::state_dtype(&self, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::AggregateFnRef::state_dtype(&self, &vortex_array::dtype::DType) -> core::option::Option pub fn vortex_array::aggregate_fn::AggregateFnRef::vtable_ref(&self) -> core::option::Option<&V> impl vortex_array::aggregate_fn::AggregateFnRef -pub fn vortex_array::aggregate_fn::AggregateFnRef::from_proto(proto: &vortex_proto::expr::AggregateFn, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::AggregateFnRef::from_proto(&vortex_proto::expr::AggregateFn, &vortex_session::VortexSession) -> vortex_error::VortexResult pub fn vortex_array::aggregate_fn::AggregateFnRef::serialize_proto(&self) -> vortex_error::VortexResult @@ -680,19 +874,19 @@ impl core::cmp::Eq for vortex_array::aggregate_fn::AggregateFnRef impl core::cmp::PartialEq for vortex_array::aggregate_fn::AggregateFnRef -pub fn vortex_array::aggregate_fn::AggregateFnRef::eq(&self, other: &Self) -> bool +pub fn vortex_array::aggregate_fn::AggregateFnRef::eq(&self, &Self) -> bool impl core::fmt::Debug for vortex_array::aggregate_fn::AggregateFnRef -pub fn vortex_array::aggregate_fn::AggregateFnRef::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::aggregate_fn::AggregateFnRef::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::aggregate_fn::AggregateFnRef -pub fn vortex_array::aggregate_fn::AggregateFnRef::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::aggregate_fn::AggregateFnRef::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::aggregate_fn::AggregateFnRef -pub fn vortex_array::aggregate_fn::AggregateFnRef::hash(&self, state: &mut H) +pub fn vortex_array::aggregate_fn::AggregateFnRef::hash(&self, &mut H) pub struct vortex_array::aggregate_fn::EmptyOptions @@ -704,19 +898,19 @@ impl core::cmp::Eq for vortex_array::aggregate_fn::EmptyOptions impl core::cmp::PartialEq for vortex_array::aggregate_fn::EmptyOptions -pub fn vortex_array::aggregate_fn::EmptyOptions::eq(&self, other: &vortex_array::aggregate_fn::EmptyOptions) -> bool +pub fn vortex_array::aggregate_fn::EmptyOptions::eq(&self, &vortex_array::aggregate_fn::EmptyOptions) -> bool impl core::fmt::Debug for vortex_array::aggregate_fn::EmptyOptions -pub fn vortex_array::aggregate_fn::EmptyOptions::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::aggregate_fn::EmptyOptions::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::aggregate_fn::EmptyOptions -pub fn vortex_array::aggregate_fn::EmptyOptions::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::aggregate_fn::EmptyOptions::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::aggregate_fn::EmptyOptions -pub fn vortex_array::aggregate_fn::EmptyOptions::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::aggregate_fn::EmptyOptions::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_array::aggregate_fn::EmptyOptions @@ -724,11 +918,11 @@ pub struct vortex_array::aggregate_fn::GroupedAccumulator vortex_array::aggregate_fn::GroupedAccumulator -pub fn vortex_array::aggregate_fn::GroupedAccumulator::try_new(vtable: V, options: ::Options, dtype: vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::GroupedAccumulator::try_new(V, ::Options, vortex_array::dtype::DType) -> vortex_error::VortexResult impl vortex_array::aggregate_fn::DynGroupedAccumulator for vortex_array::aggregate_fn::GroupedAccumulator -pub fn vortex_array::aggregate_fn::GroupedAccumulator::accumulate_list(&mut self, groups: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::GroupedAccumulator::accumulate_list(&mut self, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> pub fn vortex_array::aggregate_fn::GroupedAccumulator::finish(&mut self) -> vortex_error::VortexResult @@ -736,15 +930,15 @@ pub fn vortex_array::aggregate_fn::GroupedAccumulator::flush(&mut self) -> vo pub trait vortex_array::aggregate_fn::AggregateFnPlugin: 'static + core::marker::Send + core::marker::Sync -pub fn vortex_array::aggregate_fn::AggregateFnPlugin::deserialize(&self, metadata: &[u8], session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::AggregateFnPlugin::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult pub fn vortex_array::aggregate_fn::AggregateFnPlugin::id(&self) -> vortex_array::aggregate_fn::AggregateFnId impl vortex_array::aggregate_fn::AggregateFnPlugin for V -pub fn V::deserialize(&self, metadata: &[u8], session: &vortex_session::VortexSession) -> core::result::Result +pub fn V::deserialize(&self, &[u8], &vortex_session::VortexSession) -> core::result::Result -pub fn V::id(&self) -> arcref::ArcRef +pub fn V::id(&self) -> vortex_session::registry::Id pub trait vortex_array::aggregate_fn::AggregateFnVTable: 'static + core::marker::Sized + core::clone::Clone + core::marker::Send + core::marker::Sync @@ -752,35 +946,35 @@ pub type vortex_array::aggregate_fn::AggregateFnVTable::Options: 'static + core: pub type vortex_array::aggregate_fn::AggregateFnVTable::Partial: 'static + core::marker::Send -pub fn vortex_array::aggregate_fn::AggregateFnVTable::accumulate(&self, state: &mut Self::Partial, batch: &vortex_array::Columnar, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::AggregateFnVTable::accumulate(&self, &mut Self::Partial, &vortex_array::Columnar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::AggregateFnVTable::coerce_args(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::AggregateFnVTable::coerce_args(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::AggregateFnVTable::combine_partials(&self, partial: &mut Self::Partial, other: vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::AggregateFnVTable::combine_partials(&self, &mut Self::Partial, vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::AggregateFnVTable::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::AggregateFnVTable::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::AggregateFnVTable::empty_partial(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::AggregateFnVTable::empty_partial(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::AggregateFnVTable::finalize(&self, states: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::AggregateFnVTable::finalize(&self, vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::AggregateFnVTable::finalize_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::AggregateFnVTable::finalize_scalar(&self, &Self::Partial) -> vortex_error::VortexResult pub fn vortex_array::aggregate_fn::AggregateFnVTable::id(&self) -> vortex_array::aggregate_fn::AggregateFnId -pub fn vortex_array::aggregate_fn::AggregateFnVTable::is_saturated(&self, state: &Self::Partial) -> bool +pub fn vortex_array::aggregate_fn::AggregateFnVTable::is_saturated(&self, &Self::Partial) -> bool -pub fn vortex_array::aggregate_fn::AggregateFnVTable::partial_dtype(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::AggregateFnVTable::partial_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::AggregateFnVTable::reset(&self, partial: &mut Self::Partial) +pub fn vortex_array::aggregate_fn::AggregateFnVTable::reset(&self, &mut Self::Partial) -pub fn vortex_array::aggregate_fn::AggregateFnVTable::return_dtype(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::AggregateFnVTable::return_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::AggregateFnVTable::serialize(&self, options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::aggregate_fn::AggregateFnVTable::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::aggregate_fn::AggregateFnVTable::to_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::AggregateFnVTable::to_scalar(&self, &Self::Partial) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::AggregateFnVTable::try_accumulate(&self, _state: &mut Self::Partial, _batch: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::AggregateFnVTable::try_accumulate(&self, &mut Self::Partial, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::aggregate_fn::AggregateFnVTable for vortex_array::aggregate_fn::fns::count::Count @@ -788,35 +982,35 @@ pub type vortex_array::aggregate_fn::fns::count::Count::Options = vortex_array:: pub type vortex_array::aggregate_fn::fns::count::Count::Partial = u64 -pub fn vortex_array::aggregate_fn::fns::count::Count::accumulate(&self, _partial: &mut Self::Partial, _batch: &vortex_array::Columnar, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::count::Count::accumulate(&self, &mut Self::Partial, &vortex_array::Columnar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::count::Count::coerce_args(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::count::Count::coerce_args(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::count::Count::combine_partials(&self, partial: &mut Self::Partial, other: vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::count::Count::combine_partials(&self, &mut Self::Partial, vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::count::Count::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::count::Count::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::count::Count::empty_partial(&self, _options: &Self::Options, _input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::count::Count::empty_partial(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::count::Count::finalize(&self, partials: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::count::Count::finalize(&self, vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::count::Count::finalize_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::count::Count::finalize_scalar(&self, &Self::Partial) -> vortex_error::VortexResult pub fn vortex_array::aggregate_fn::fns::count::Count::id(&self) -> vortex_array::aggregate_fn::AggregateFnId -pub fn vortex_array::aggregate_fn::fns::count::Count::is_saturated(&self, _partial: &Self::Partial) -> bool +pub fn vortex_array::aggregate_fn::fns::count::Count::is_saturated(&self, &Self::Partial) -> bool -pub fn vortex_array::aggregate_fn::fns::count::Count::partial_dtype(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::count::Count::partial_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::count::Count::reset(&self, partial: &mut Self::Partial) +pub fn vortex_array::aggregate_fn::fns::count::Count::reset(&self, &mut Self::Partial) -pub fn vortex_array::aggregate_fn::fns::count::Count::return_dtype(&self, _options: &Self::Options, _input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::count::Count::return_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::count::Count::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::aggregate_fn::fns::count::Count::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::aggregate_fn::fns::count::Count::to_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::count::Count::to_scalar(&self, &Self::Partial) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::count::Count::try_accumulate(&self, state: &mut Self::Partial, batch: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::count::Count::try_accumulate(&self, &mut Self::Partial, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::aggregate_fn::AggregateFnVTable for vortex_array::aggregate_fn::fns::first::First @@ -824,35 +1018,35 @@ pub type vortex_array::aggregate_fn::fns::first::First::Options = vortex_array:: pub type vortex_array::aggregate_fn::fns::first::First::Partial = vortex_array::aggregate_fn::fns::first::FirstPartial -pub fn vortex_array::aggregate_fn::fns::first::First::accumulate(&self, _partial: &mut Self::Partial, _batch: &vortex_array::Columnar, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::first::First::accumulate(&self, &mut Self::Partial, &vortex_array::Columnar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::first::First::coerce_args(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::first::First::coerce_args(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::first::First::combine_partials(&self, partial: &mut Self::Partial, other: vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::first::First::combine_partials(&self, &mut Self::Partial, vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::first::First::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::first::First::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::first::First::empty_partial(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::first::First::empty_partial(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::first::First::finalize(&self, partials: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::first::First::finalize(&self, vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::first::First::finalize_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::first::First::finalize_scalar(&self, &Self::Partial) -> vortex_error::VortexResult pub fn vortex_array::aggregate_fn::fns::first::First::id(&self) -> vortex_array::aggregate_fn::AggregateFnId -pub fn vortex_array::aggregate_fn::fns::first::First::is_saturated(&self, partial: &Self::Partial) -> bool +pub fn vortex_array::aggregate_fn::fns::first::First::is_saturated(&self, &Self::Partial) -> bool -pub fn vortex_array::aggregate_fn::fns::first::First::partial_dtype(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::first::First::partial_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::first::First::reset(&self, partial: &mut Self::Partial) +pub fn vortex_array::aggregate_fn::fns::first::First::reset(&self, &mut Self::Partial) -pub fn vortex_array::aggregate_fn::fns::first::First::return_dtype(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::first::First::return_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::first::First::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::aggregate_fn::fns::first::First::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::aggregate_fn::fns::first::First::to_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::first::First::to_scalar(&self, &Self::Partial) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::first::First::try_accumulate(&self, partial: &mut Self::Partial, batch: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::first::First::try_accumulate(&self, &mut Self::Partial, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::aggregate_fn::AggregateFnVTable for vortex_array::aggregate_fn::fns::is_constant::IsConstant @@ -860,35 +1054,35 @@ pub type vortex_array::aggregate_fn::fns::is_constant::IsConstant::Options = vor pub type vortex_array::aggregate_fn::fns::is_constant::IsConstant::Partial = vortex_array::aggregate_fn::fns::is_constant::IsConstantPartial -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::accumulate(&self, partial: &mut Self::Partial, batch: &vortex_array::Columnar, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::accumulate(&self, &mut Self::Partial, &vortex_array::Columnar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::coerce_args(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::coerce_args(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::combine_partials(&self, partial: &mut Self::Partial, other: vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::combine_partials(&self, &mut Self::Partial, vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::empty_partial(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::empty_partial(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::finalize(&self, partials: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::finalize(&self, vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::finalize_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::finalize_scalar(&self, &Self::Partial) -> vortex_error::VortexResult pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::id(&self) -> vortex_array::aggregate_fn::AggregateFnId -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::is_saturated(&self, partial: &Self::Partial) -> bool +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::is_saturated(&self, &Self::Partial) -> bool -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::partial_dtype(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::partial_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::reset(&self, partial: &mut Self::Partial) +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::reset(&self, &mut Self::Partial) -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::return_dtype(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::return_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::to_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::to_scalar(&self, &Self::Partial) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::try_accumulate(&self, _state: &mut Self::Partial, _batch: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_constant::IsConstant::try_accumulate(&self, &mut Self::Partial, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::aggregate_fn::AggregateFnVTable for vortex_array::aggregate_fn::fns::is_sorted::IsSorted @@ -896,35 +1090,35 @@ pub type vortex_array::aggregate_fn::fns::is_sorted::IsSorted::Options = vortex_ pub type vortex_array::aggregate_fn::fns::is_sorted::IsSorted::Partial = vortex_array::aggregate_fn::fns::is_sorted::IsSortedPartial -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::accumulate(&self, partial: &mut Self::Partial, batch: &vortex_array::Columnar, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::accumulate(&self, &mut Self::Partial, &vortex_array::Columnar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::coerce_args(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::coerce_args(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::combine_partials(&self, partial: &mut Self::Partial, other: vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::combine_partials(&self, &mut Self::Partial, vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::empty_partial(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::empty_partial(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::finalize(&self, partials: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::finalize(&self, vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::finalize_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::finalize_scalar(&self, &Self::Partial) -> vortex_error::VortexResult pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::id(&self) -> vortex_array::aggregate_fn::AggregateFnId -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::is_saturated(&self, partial: &Self::Partial) -> bool +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::is_saturated(&self, &Self::Partial) -> bool -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::partial_dtype(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::partial_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::reset(&self, partial: &mut Self::Partial) +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::reset(&self, &mut Self::Partial) -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::return_dtype(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::return_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::to_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::to_scalar(&self, &Self::Partial) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::try_accumulate(&self, _state: &mut Self::Partial, _batch: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::is_sorted::IsSorted::try_accumulate(&self, &mut Self::Partial, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::aggregate_fn::AggregateFnVTable for vortex_array::aggregate_fn::fns::last::Last @@ -932,35 +1126,35 @@ pub type vortex_array::aggregate_fn::fns::last::Last::Options = vortex_array::ag pub type vortex_array::aggregate_fn::fns::last::Last::Partial = vortex_array::aggregate_fn::fns::last::LastPartial -pub fn vortex_array::aggregate_fn::fns::last::Last::accumulate(&self, _partial: &mut Self::Partial, _batch: &vortex_array::Columnar, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::last::Last::accumulate(&self, &mut Self::Partial, &vortex_array::Columnar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::last::Last::coerce_args(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::last::Last::coerce_args(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::last::Last::combine_partials(&self, partial: &mut Self::Partial, other: vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::last::Last::combine_partials(&self, &mut Self::Partial, vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::last::Last::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::last::Last::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::last::Last::empty_partial(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::last::Last::empty_partial(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::last::Last::finalize(&self, partials: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::last::Last::finalize(&self, vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::last::Last::finalize_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::last::Last::finalize_scalar(&self, &Self::Partial) -> vortex_error::VortexResult pub fn vortex_array::aggregate_fn::fns::last::Last::id(&self) -> vortex_array::aggregate_fn::AggregateFnId -pub fn vortex_array::aggregate_fn::fns::last::Last::is_saturated(&self, _partial: &Self::Partial) -> bool +pub fn vortex_array::aggregate_fn::fns::last::Last::is_saturated(&self, &Self::Partial) -> bool -pub fn vortex_array::aggregate_fn::fns::last::Last::partial_dtype(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::last::Last::partial_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::last::Last::reset(&self, partial: &mut Self::Partial) +pub fn vortex_array::aggregate_fn::fns::last::Last::reset(&self, &mut Self::Partial) -pub fn vortex_array::aggregate_fn::fns::last::Last::return_dtype(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::last::Last::return_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::last::Last::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::aggregate_fn::fns::last::Last::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::aggregate_fn::fns::last::Last::to_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::last::Last::to_scalar(&self, &Self::Partial) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::last::Last::try_accumulate(&self, partial: &mut Self::Partial, batch: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::last::Last::try_accumulate(&self, &mut Self::Partial, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::aggregate_fn::AggregateFnVTable for vortex_array::aggregate_fn::fns::min_max::MinMax @@ -968,35 +1162,35 @@ pub type vortex_array::aggregate_fn::fns::min_max::MinMax::Options = vortex_arra pub type vortex_array::aggregate_fn::fns::min_max::MinMax::Partial = vortex_array::aggregate_fn::fns::min_max::MinMaxPartial -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::accumulate(&self, partial: &mut Self::Partial, batch: &vortex_array::Columnar, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::accumulate(&self, &mut Self::Partial, &vortex_array::Columnar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::coerce_args(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::coerce_args(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::combine_partials(&self, partial: &mut Self::Partial, other: vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::combine_partials(&self, &mut Self::Partial, vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::empty_partial(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::empty_partial(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::finalize(&self, partials: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::finalize(&self, vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::finalize_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::finalize_scalar(&self, &Self::Partial) -> vortex_error::VortexResult pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::id(&self) -> vortex_array::aggregate_fn::AggregateFnId -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::is_saturated(&self, _partial: &Self::Partial) -> bool +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::is_saturated(&self, &Self::Partial) -> bool -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::partial_dtype(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::partial_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::reset(&self, partial: &mut Self::Partial) +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::reset(&self, &mut Self::Partial) -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::return_dtype(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::return_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::to_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::to_scalar(&self, &Self::Partial) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::try_accumulate(&self, _state: &mut Self::Partial, _batch: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::min_max::MinMax::try_accumulate(&self, &mut Self::Partial, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::aggregate_fn::AggregateFnVTable for vortex_array::aggregate_fn::fns::nan_count::NanCount @@ -1004,35 +1198,35 @@ pub type vortex_array::aggregate_fn::fns::nan_count::NanCount::Options = vortex_ pub type vortex_array::aggregate_fn::fns::nan_count::NanCount::Partial = u64 -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::accumulate(&self, partial: &mut Self::Partial, batch: &vortex_array::Columnar, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::accumulate(&self, &mut Self::Partial, &vortex_array::Columnar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::coerce_args(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::coerce_args(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::combine_partials(&self, partial: &mut Self::Partial, other: vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::combine_partials(&self, &mut Self::Partial, vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::empty_partial(&self, _options: &Self::Options, _input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::empty_partial(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::finalize(&self, partials: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::finalize(&self, vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::finalize_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::finalize_scalar(&self, &Self::Partial) -> vortex_error::VortexResult pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::id(&self) -> vortex_array::aggregate_fn::AggregateFnId -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::is_saturated(&self, _partial: &Self::Partial) -> bool +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::is_saturated(&self, &Self::Partial) -> bool -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::partial_dtype(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::partial_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::reset(&self, partial: &mut Self::Partial) +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::reset(&self, &mut Self::Partial) -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::return_dtype(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::return_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::to_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::to_scalar(&self, &Self::Partial) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::try_accumulate(&self, _state: &mut Self::Partial, _batch: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::nan_count::NanCount::try_accumulate(&self, &mut Self::Partial, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::aggregate_fn::AggregateFnVTable for vortex_array::aggregate_fn::fns::sum::Sum @@ -1040,47 +1234,83 @@ pub type vortex_array::aggregate_fn::fns::sum::Sum::Options = vortex_array::aggr pub type vortex_array::aggregate_fn::fns::sum::Sum::Partial = vortex_array::aggregate_fn::fns::sum::SumPartial -pub fn vortex_array::aggregate_fn::fns::sum::Sum::accumulate(&self, partial: &mut Self::Partial, batch: &vortex_array::Columnar, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::sum::Sum::accumulate(&self, &mut Self::Partial, &vortex_array::Columnar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::sum::Sum::coerce_args(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::sum::Sum::coerce_args(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::sum::Sum::combine_partials(&self, partial: &mut Self::Partial, other: vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::fns::sum::Sum::combine_partials(&self, &mut Self::Partial, vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> -pub fn vortex_array::aggregate_fn::fns::sum::Sum::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::sum::Sum::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::sum::Sum::empty_partial(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::sum::Sum::empty_partial(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::sum::Sum::finalize(&self, partials: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::sum::Sum::finalize(&self, vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::aggregate_fn::fns::sum::Sum::finalize_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::fns::sum::Sum::finalize_scalar(&self, &Self::Partial) -> vortex_error::VortexResult pub fn vortex_array::aggregate_fn::fns::sum::Sum::id(&self) -> vortex_array::aggregate_fn::AggregateFnId -pub fn vortex_array::aggregate_fn::fns::sum::Sum::is_saturated(&self, partial: &Self::Partial) -> bool +pub fn vortex_array::aggregate_fn::fns::sum::Sum::is_saturated(&self, &Self::Partial) -> bool + +pub fn vortex_array::aggregate_fn::fns::sum::Sum::partial_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option + +pub fn vortex_array::aggregate_fn::fns::sum::Sum::reset(&self, &mut Self::Partial) + +pub fn vortex_array::aggregate_fn::fns::sum::Sum::return_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option + +pub fn vortex_array::aggregate_fn::fns::sum::Sum::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> + +pub fn vortex_array::aggregate_fn::fns::sum::Sum::to_scalar(&self, &Self::Partial) -> vortex_error::VortexResult + +pub fn vortex_array::aggregate_fn::fns::sum::Sum::try_accumulate(&self, &mut Self::Partial, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult + +impl vortex_array::aggregate_fn::AggregateFnVTable for vortex_array::aggregate_fn::combined::Combined + +pub type vortex_array::aggregate_fn::combined::Combined::Options = vortex_array::aggregate_fn::combined::PairOptions<<::Left as vortex_array::aggregate_fn::AggregateFnVTable>::Options, <::Right as vortex_array::aggregate_fn::AggregateFnVTable>::Options> + +pub type vortex_array::aggregate_fn::combined::Combined::Partial = (<::Left as vortex_array::aggregate_fn::AggregateFnVTable>::Partial, <::Right as vortex_array::aggregate_fn::AggregateFnVTable>::Partial) + +pub fn vortex_array::aggregate_fn::combined::Combined::accumulate(&self, &mut Self::Partial, &vortex_array::Columnar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> + +pub fn vortex_array::aggregate_fn::combined::Combined::coerce_args(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult + +pub fn vortex_array::aggregate_fn::combined::Combined::combine_partials(&self, &mut Self::Partial, vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> + +pub fn vortex_array::aggregate_fn::combined::Combined::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult + +pub fn vortex_array::aggregate_fn::combined::Combined::empty_partial(&self, &Self::Options, &vortex_array::dtype::DType) -> vortex_error::VortexResult + +pub fn vortex_array::aggregate_fn::combined::Combined::finalize(&self, vortex_array::ArrayRef) -> vortex_error::VortexResult + +pub fn vortex_array::aggregate_fn::combined::Combined::finalize_scalar(&self, &Self::Partial) -> vortex_error::VortexResult + +pub fn vortex_array::aggregate_fn::combined::Combined::id(&self) -> vortex_array::aggregate_fn::AggregateFnId -pub fn vortex_array::aggregate_fn::fns::sum::Sum::partial_dtype(&self, options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::combined::Combined::is_saturated(&self, &Self::Partial) -> bool -pub fn vortex_array::aggregate_fn::fns::sum::Sum::reset(&self, partial: &mut Self::Partial) +pub fn vortex_array::aggregate_fn::combined::Combined::partial_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::sum::Sum::return_dtype(&self, _options: &Self::Options, input_dtype: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::aggregate_fn::combined::Combined::reset(&self, &mut Self::Partial) -pub fn vortex_array::aggregate_fn::fns::sum::Sum::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::aggregate_fn::combined::Combined::return_dtype(&self, &Self::Options, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::aggregate_fn::fns::sum::Sum::to_scalar(&self, partial: &Self::Partial) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::combined::Combined::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::aggregate_fn::fns::sum::Sum::try_accumulate(&self, _state: &mut Self::Partial, _batch: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::aggregate_fn::combined::Combined::to_scalar(&self, &Self::Partial) -> vortex_error::VortexResult + +pub fn vortex_array::aggregate_fn::combined::Combined::try_accumulate(&self, &mut Self::Partial, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub trait vortex_array::aggregate_fn::AggregateFnVTableExt: vortex_array::aggregate_fn::AggregateFnVTable -pub fn vortex_array::aggregate_fn::AggregateFnVTableExt::bind(&self, options: Self::Options) -> vortex_array::aggregate_fn::AggregateFnRef +pub fn vortex_array::aggregate_fn::AggregateFnVTableExt::bind(&self, Self::Options) -> vortex_array::aggregate_fn::AggregateFnRef impl vortex_array::aggregate_fn::AggregateFnVTableExt for V -pub fn V::bind(&self, options: Self::Options) -> vortex_array::aggregate_fn::AggregateFnRef +pub fn V::bind(&self, Self::Options) -> vortex_array::aggregate_fn::AggregateFnRef pub trait vortex_array::aggregate_fn::DynAccumulator: 'static + core::marker::Send -pub fn vortex_array::aggregate_fn::DynAccumulator::accumulate(&mut self, batch: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::DynAccumulator::accumulate(&mut self, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> pub fn vortex_array::aggregate_fn::DynAccumulator::finish(&mut self) -> vortex_error::VortexResult @@ -1090,7 +1320,7 @@ pub fn vortex_array::aggregate_fn::DynAccumulator::is_saturated(&self) -> bool impl vortex_array::aggregate_fn::DynAccumulator for vortex_array::aggregate_fn::Accumulator -pub fn vortex_array::aggregate_fn::Accumulator::accumulate(&mut self, batch: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::Accumulator::accumulate(&mut self, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> pub fn vortex_array::aggregate_fn::Accumulator::finish(&mut self) -> vortex_error::VortexResult @@ -1100,7 +1330,7 @@ pub fn vortex_array::aggregate_fn::Accumulator::is_saturated(&self) -> bool pub trait vortex_array::aggregate_fn::DynGroupedAccumulator: 'static + core::marker::Send -pub fn vortex_array::aggregate_fn::DynGroupedAccumulator::accumulate_list(&mut self, groups: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::DynGroupedAccumulator::accumulate_list(&mut self, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> pub fn vortex_array::aggregate_fn::DynGroupedAccumulator::finish(&mut self) -> vortex_error::VortexResult @@ -1108,7 +1338,7 @@ pub fn vortex_array::aggregate_fn::DynGroupedAccumulator::flush(&mut self) -> vo impl vortex_array::aggregate_fn::DynGroupedAccumulator for vortex_array::aggregate_fn::GroupedAccumulator -pub fn vortex_array::aggregate_fn::GroupedAccumulator::accumulate_list(&mut self, groups: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::aggregate_fn::GroupedAccumulator::accumulate_list(&mut self, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> pub fn vortex_array::aggregate_fn::GroupedAccumulator::finish(&mut self) -> vortex_error::VortexResult @@ -1116,7 +1346,7 @@ pub fn vortex_array::aggregate_fn::GroupedAccumulator::flush(&mut self) -> vo pub type vortex_array::aggregate_fn::AccumulatorRef = alloc::boxed::Box -pub type vortex_array::aggregate_fn::AggregateFnId = arcref::ArcRef +pub type vortex_array::aggregate_fn::AggregateFnId = vortex_session::registry::Id pub type vortex_array::aggregate_fn::AggregateFnPluginRef = alloc::sync::Arc @@ -1128,21 +1358,17 @@ pub mod vortex_array::arrays::bool pub struct vortex_array::arrays::bool::Bool -impl vortex_array::arrays::Bool - -pub const vortex_array::arrays::Bool::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::Bool pub fn vortex_array::arrays::Bool::clone(&self) -> vortex_array::arrays::Bool impl core::fmt::Debug for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::Bool::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Bool::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::Bool @@ -1152,77 +1378,81 @@ pub type vortex_array::arrays::Bool::OperationsVTable = vortex_array::arrays::Bo pub type vortex_array::arrays::Bool::ValidityVTable = vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Bool::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Bool::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Bool::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Bool::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Bool::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Bool::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Bool::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Bool::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Bool::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Bool::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Bool::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Bool::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Bool::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Bool::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Bool::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Bool::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Bool::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Bool::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Bool::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Bool::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Bool::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Bool::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Bool::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Bool::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Bool::validate(&self, data: &vortex_array::arrays::bool::BoolData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Bool::validate(&self, &vortex_array::arrays::bool::BoolData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Bool::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::take(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterReduce for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::filter(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::filter(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::slice(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::slice(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::arrays::bool::BoolMaskedValidityRule pub type vortex_array::arrays::bool::BoolMaskedValidityRule::Parent = vortex_array::arrays::Masked -pub fn vortex_array::arrays::bool::BoolMaskedValidityRule::reduce_parent(&self, array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, parent: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::bool::BoolMaskedValidityRule::reduce_parent(&self, vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, usize) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::Bool + +pub fn vortex_array::arrays::Bool::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::fill_null::FillNullKernel for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::fill_null(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, fill_value: &vortex_array::scalar::Scalar, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::fill_null(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, &vortex_array::scalar::Scalar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::mask(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> pub struct vortex_array::arrays::bool::BoolData impl vortex_array::arrays::bool::BoolData -pub fn vortex_array::arrays::bool::BoolData::into_parts(self, len: usize) -> vortex_array::arrays::bool::BoolDataParts +pub fn vortex_array::arrays::bool::BoolData::into_parts(self, usize) -> vortex_array::arrays::bool::BoolDataParts impl core::clone::Clone for vortex_array::arrays::bool::BoolData @@ -1230,19 +1460,19 @@ pub fn vortex_array::arrays::bool::BoolData::clone(&self) -> vortex_array::array impl core::fmt::Debug for vortex_array::arrays::bool::BoolData -pub fn vortex_array::arrays::bool::BoolData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::bool::BoolData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::arrays::bool::BoolData -pub fn vortex_array::arrays::bool::BoolData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::bool::BoolData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::ArrayEq for vortex_array::arrays::bool::BoolData -pub fn vortex_array::arrays::bool::BoolData::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::bool::BoolData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayHash for vortex_array::arrays::bool::BoolData -pub fn vortex_array::arrays::bool::BoolData::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn vortex_array::arrays::bool::BoolData::array_hash(&self, &mut H, vortex_array::Precision) pub struct vortex_array::arrays::bool::BoolDataParts @@ -1260,43 +1490,39 @@ pub fn vortex_array::arrays::bool::BoolMaskedValidityRule::default() -> vortex_a impl core::fmt::Debug for vortex_array::arrays::bool::BoolMaskedValidityRule -pub fn vortex_array::arrays::bool::BoolMaskedValidityRule::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::bool::BoolMaskedValidityRule::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::arrays::bool::BoolMaskedValidityRule pub type vortex_array::arrays::bool::BoolMaskedValidityRule::Parent = vortex_array::arrays::Masked -pub fn vortex_array::arrays::bool::BoolMaskedValidityRule::reduce_parent(&self, array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, parent: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::bool::BoolMaskedValidityRule::reduce_parent(&self, vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, usize) -> vortex_error::VortexResult> pub trait vortex_array::arrays::bool::BoolArrayExt: vortex_array::TypedArrayRef -pub fn vortex_array::arrays::bool::BoolArrayExt::bool_validity_mask(&self) -> vortex_mask::Mask +pub fn vortex_array::arrays::bool::BoolArrayExt::execute_mask(&self, &mut vortex_array::ExecutionCtx) -> vortex_mask::Mask -pub fn vortex_array::arrays::bool::BoolArrayExt::maybe_to_mask(&self) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::bool::BoolArrayExt::maybe_execute_mask(&self, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::bool::BoolArrayExt::nullability(&self) -> vortex_array::dtype::Nullability pub fn vortex_array::arrays::bool::BoolArrayExt::to_bit_buffer(&self) -> vortex_buffer::bit::buf::BitBuffer -pub fn vortex_array::arrays::bool::BoolArrayExt::to_mask(&self) -> vortex_mask::Mask - -pub fn vortex_array::arrays::bool::BoolArrayExt::to_mask_fill_null_false(&self) -> vortex_mask::Mask +pub fn vortex_array::arrays::bool::BoolArrayExt::to_mask_fill_null_false(&self, &mut vortex_array::ExecutionCtx) -> vortex_mask::Mask pub fn vortex_array::arrays::bool::BoolArrayExt::validity(&self) -> vortex_array::validity::Validity impl> vortex_array::arrays::bool::BoolArrayExt for T -pub fn T::bool_validity_mask(&self) -> vortex_mask::Mask +pub fn T::execute_mask(&self, &mut vortex_array::ExecutionCtx) -> vortex_mask::Mask -pub fn T::maybe_to_mask(&self) -> vortex_error::VortexResult> +pub fn T::maybe_execute_mask(&self, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn T::nullability(&self) -> vortex_array::dtype::Nullability pub fn T::to_bit_buffer(&self) -> vortex_buffer::bit::buf::BitBuffer -pub fn T::to_mask(&self) -> vortex_mask::Mask - -pub fn T::to_mask_fill_null_false(&self) -> vortex_mask::Mask +pub fn T::to_mask_fill_null_false(&self, &mut vortex_array::ExecutionCtx) -> vortex_mask::Mask pub fn T::validity(&self) -> vortex_array::validity::Validity @@ -1306,21 +1532,17 @@ pub mod vortex_array::arrays::chunked pub struct vortex_array::arrays::chunked::Chunked -impl vortex_array::arrays::Chunked - -pub const vortex_array::arrays::Chunked::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::Chunked pub fn vortex_array::arrays::Chunked::clone(&self) -> vortex_array::arrays::Chunked impl core::fmt::Debug for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::Chunked::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Chunked::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::Chunked @@ -1330,75 +1552,75 @@ pub type vortex_array::arrays::Chunked::OperationsVTable = vortex_array::arrays: pub type vortex_array::arrays::Chunked::ValidityVTable = vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Chunked::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Chunked::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Chunked::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Chunked::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Chunked::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Chunked::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Chunked::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Chunked::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Chunked::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Chunked::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Chunked::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Chunked::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Chunked::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Chunked::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Chunked::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Chunked::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Chunked::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Chunked::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Chunked::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Chunked::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Chunked::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Chunked::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Chunked::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Chunked::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Chunked::validate(&self, data: &vortex_array::arrays::chunked::ChunkedData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Chunked::validate(&self, &vortex_array::arrays::chunked::ChunkedData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Chunked::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::take(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterKernel for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::filter(array: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, mask: &vortex_mask::Mask, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::filter(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, &vortex_mask::Mask, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceKernel for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::fill_null::FillNullReduce for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::fill_null(array: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, fill_value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::fill_null(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskKernel for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, mask: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::mask(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::zip::ZipKernel for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::zip(if_true: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, if_false: &vortex_array::ArrayRef, mask: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::zip(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, &vortex_array::ArrayRef, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub struct vortex_array::arrays::chunked::ChunkedData impl vortex_array::arrays::chunked::ChunkedData -pub fn vortex_array::arrays::chunked::ChunkedData::validate(chunks: &[vortex_array::ArrayRef], dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::chunked::ChunkedData::validate(&[vortex_array::ArrayRef], &vortex_array::dtype::DType) -> vortex_error::VortexResult<()> impl core::clone::Clone for vortex_array::arrays::chunked::ChunkedData @@ -1406,19 +1628,19 @@ pub fn vortex_array::arrays::chunked::ChunkedData::clone(&self) -> vortex_array: impl core::fmt::Debug for vortex_array::arrays::chunked::ChunkedData -pub fn vortex_array::arrays::chunked::ChunkedData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::chunked::ChunkedData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::arrays::chunked::ChunkedData -pub fn vortex_array::arrays::chunked::ChunkedData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::chunked::ChunkedData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::ArrayEq for vortex_array::arrays::chunked::ChunkedData -pub fn vortex_array::arrays::chunked::ChunkedData::array_eq(&self, _other: &Self, _precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::chunked::ChunkedData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayHash for vortex_array::arrays::chunked::ChunkedData -pub fn vortex_array::arrays::chunked::ChunkedData::array_hash(&self, _state: &mut H, _precision: vortex_array::Precision) +pub fn vortex_array::arrays::chunked::ChunkedData::array_hash(&self, &mut H, vortex_array::Precision) pub trait vortex_array::arrays::chunked::ChunkedArrayExt: vortex_array::TypedArrayRef @@ -1426,7 +1648,7 @@ pub fn vortex_array::arrays::chunked::ChunkedArrayExt::array_iterator(&self) -> pub fn vortex_array::arrays::chunked::ChunkedArrayExt::array_stream(&self) -> impl vortex_array::stream::ArrayStream + '_ -pub fn vortex_array::arrays::chunked::ChunkedArrayExt::chunk(&self, idx: usize) -> &vortex_array::ArrayRef +pub fn vortex_array::arrays::chunked::ChunkedArrayExt::chunk(&self, usize) -> &vortex_array::ArrayRef pub fn vortex_array::arrays::chunked::ChunkedArrayExt::chunk_offsets(&self) -> &[usize] @@ -1434,7 +1656,7 @@ pub fn vortex_array::arrays::chunked::ChunkedArrayExt::chunk_offsets_array(&self pub fn vortex_array::arrays::chunked::ChunkedArrayExt::chunks(&self) -> alloc::vec::Vec -pub fn vortex_array::arrays::chunked::ChunkedArrayExt::find_chunk_idx(&self, index: usize) -> vortex_error::VortexResult<(usize, usize)> +pub fn vortex_array::arrays::chunked::ChunkedArrayExt::find_chunk_idx(&self, usize) -> vortex_error::VortexResult<(usize, usize)> pub fn vortex_array::arrays::chunked::ChunkedArrayExt::iter_chunks<'a>(&'a self) -> alloc::boxed::Box<(dyn core::iter::traits::iterator::Iterator + 'a)> @@ -1448,7 +1670,7 @@ pub fn T::array_iterator(&self) -> impl vortex_array::iter::ArrayIterator + '_ pub fn T::array_stream(&self) -> impl vortex_array::stream::ArrayStream + '_ -pub fn T::chunk(&self, idx: usize) -> &vortex_array::ArrayRef +pub fn T::chunk(&self, usize) -> &vortex_array::ArrayRef pub fn T::chunk_offsets(&self) -> &[usize] @@ -1456,7 +1678,7 @@ pub fn T::chunk_offsets_array(&self) -> &vortex_array::ArrayRef pub fn T::chunks(&self) -> alloc::vec::Vec -pub fn T::find_chunk_idx(&self, index: usize) -> vortex_error::VortexResult<(usize, usize)> +pub fn T::find_chunk_idx(&self, usize) -> vortex_error::VortexResult<(usize, usize)> pub fn T::iter_chunks<'a>(&'a self) -> alloc::boxed::Box<(dyn core::iter::traits::iterator::Iterator + 'a)> @@ -1472,10 +1694,6 @@ pub struct vortex_array::arrays::constant::Constant impl vortex_array::arrays::Constant -pub const vortex_array::arrays::Constant::ID: vortex_array::ArrayId - -impl vortex_array::arrays::Constant - pub const vortex_array::arrays::Constant::TAKE_RULES: vortex_array::optimizer::rules::ParentRuleSet impl core::clone::Clone for vortex_array::arrays::Constant @@ -1484,11 +1702,11 @@ pub fn vortex_array::arrays::Constant::clone(&self) -> vortex_array::arrays::Con impl core::fmt::Debug for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::Constant::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, _index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Constant::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::Constant @@ -1498,69 +1716,69 @@ pub type vortex_array::arrays::Constant::OperationsVTable = vortex_array::arrays pub type vortex_array::arrays::Constant::ValidityVTable = vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Constant::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Constant::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Constant::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Constant::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Constant::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Constant::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Constant::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Constant::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Constant::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Constant::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, _metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Constant::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Constant::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Constant::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Constant::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Constant::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Constant::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Constant::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Constant::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Constant::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Constant::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Constant::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Constant::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Constant::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Constant::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Constant::validate(&self, data: &vortex_array::arrays::constant::ConstantData, dtype: &vortex_array::dtype::DType, _len: usize, _slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Constant::validate(&self, &vortex_array::arrays::constant::ConstantData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Constant::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeReduce for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, indices: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::take(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterReduce for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::filter(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::filter(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::between::BetweenReduce for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::between(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, lower: &vortex_array::ArrayRef, upper: &vortex_array::ArrayRef, options: &vortex_array::scalar_fn::fns::between::BetweenOptions) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::between(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, &vortex_array::ArrayRef, &vortex_array::ArrayRef, &vortex_array::scalar_fn::fns::between::BetweenOptions) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::fill_null::FillNullReduce for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::fill_null(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, fill_value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::fill_null(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::not::NotReduce for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::invert(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::invert(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>) -> vortex_error::VortexResult> pub struct vortex_array::arrays::constant::ConstantData @@ -1568,7 +1786,7 @@ impl vortex_array::arrays::constant::ConstantData pub fn vortex_array::arrays::constant::ConstantData::into_parts(self) -> vortex_array::scalar::Scalar -pub fn vortex_array::arrays::constant::ConstantData::new(scalar: S) -> Self where S: core::convert::Into +pub fn vortex_array::arrays::constant::ConstantData::new(S) -> Self where S: core::convert::Into pub fn vortex_array::arrays::constant::ConstantData::scalar(&self) -> &vortex_array::scalar::Scalar @@ -1578,19 +1796,19 @@ pub fn vortex_array::arrays::constant::ConstantData::clone(&self) -> vortex_arra impl core::fmt::Debug for vortex_array::arrays::constant::ConstantData -pub fn vortex_array::arrays::constant::ConstantData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::constant::ConstantData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::arrays::constant::ConstantData -pub fn vortex_array::arrays::constant::ConstantData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::constant::ConstantData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::ArrayEq for vortex_array::arrays::constant::ConstantData -pub fn vortex_array::arrays::constant::ConstantData::array_eq(&self, other: &Self, _precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::constant::ConstantData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayHash for vortex_array::arrays::constant::ConstantData -pub fn vortex_array::arrays::constant::ConstantData::array_hash(&self, state: &mut H, _precision: vortex_array::Precision) +pub fn vortex_array::arrays::constant::ConstantData::array_hash(&self, &mut H, vortex_array::Precision) pub type vortex_array::arrays::constant::ConstantArray = vortex_array::Array @@ -1610,11 +1828,11 @@ pub fn vortex_array::arrays::datetime::TemporalData::temporal_values(&self) -> & impl vortex_array::arrays::datetime::TemporalData -pub fn vortex_array::arrays::datetime::TemporalData::new_date(array: vortex_array::ArrayRef, time_unit: vortex_array::extension::datetime::TimeUnit) -> Self +pub fn vortex_array::arrays::datetime::TemporalData::new_date(vortex_array::ArrayRef, vortex_array::extension::datetime::TimeUnit) -> Self -pub fn vortex_array::arrays::datetime::TemporalData::new_time(array: vortex_array::ArrayRef, time_unit: vortex_array::extension::datetime::TimeUnit) -> Self +pub fn vortex_array::arrays::datetime::TemporalData::new_time(vortex_array::ArrayRef, vortex_array::extension::datetime::TimeUnit) -> Self -pub fn vortex_array::arrays::datetime::TemporalData::new_timestamp(array: vortex_array::ArrayRef, time_unit: vortex_array::extension::datetime::TimeUnit, time_zone: core::option::Option>) -> Self +pub fn vortex_array::arrays::datetime::TemporalData::new_timestamp(vortex_array::ArrayRef, vortex_array::extension::datetime::TimeUnit, core::option::Option>) -> Self impl core::clone::Clone for vortex_array::arrays::datetime::TemporalData @@ -1622,31 +1840,31 @@ pub fn vortex_array::arrays::datetime::TemporalData::clone(&self) -> vortex_arra impl core::convert::From<&vortex_array::arrays::datetime::TemporalData> for vortex_array::arrays::ExtensionArray -pub fn vortex_array::arrays::ExtensionArray::from(value: &vortex_array::arrays::datetime::TemporalData) -> Self +pub fn vortex_array::arrays::ExtensionArray::from(&vortex_array::arrays::datetime::TemporalData) -> Self impl core::convert::From for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from(value: vortex_array::arrays::datetime::TemporalData) -> Self +pub fn vortex_array::ArrayRef::from(vortex_array::arrays::datetime::TemporalData) -> Self impl core::convert::From for vortex_array::arrays::ExtensionArray -pub fn vortex_array::arrays::ExtensionArray::from(value: vortex_array::arrays::datetime::TemporalData) -> Self +pub fn vortex_array::arrays::ExtensionArray::from(vortex_array::arrays::datetime::TemporalData) -> Self impl core::convert::TryFrom> for vortex_array::arrays::datetime::TemporalData pub type vortex_array::arrays::datetime::TemporalData::Error = vortex_error::VortexError -pub fn vortex_array::arrays::datetime::TemporalData::try_from(ext: vortex_array::arrays::ExtensionArray) -> core::result::Result +pub fn vortex_array::arrays::datetime::TemporalData::try_from(vortex_array::arrays::ExtensionArray) -> core::result::Result impl core::convert::TryFrom for vortex_array::arrays::datetime::TemporalData pub type vortex_array::arrays::datetime::TemporalData::Error = vortex_error::VortexError -pub fn vortex_array::arrays::datetime::TemporalData::try_from(value: vortex_array::ArrayRef) -> core::result::Result +pub fn vortex_array::arrays::datetime::TemporalData::try_from(vortex_array::ArrayRef) -> core::result::Result impl core::fmt::Debug for vortex_array::arrays::datetime::TemporalData -pub fn vortex_array::arrays::datetime::TemporalData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::datetime::TemporalData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::IntoArray for vortex_array::arrays::datetime::TemporalData @@ -1658,21 +1876,17 @@ pub mod vortex_array::arrays::decimal pub struct vortex_array::arrays::decimal::Decimal -impl vortex_array::arrays::Decimal - -pub const vortex_array::arrays::Decimal::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::Decimal pub fn vortex_array::arrays::Decimal::clone(&self) -> vortex_array::arrays::Decimal impl core::fmt::Debug for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::Decimal::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Decimal::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::Decimal @@ -1682,71 +1896,75 @@ pub type vortex_array::arrays::Decimal::OperationsVTable = vortex_array::arrays: pub type vortex_array::arrays::Decimal::ValidityVTable = vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Decimal::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Decimal::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Decimal::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Decimal::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Decimal::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Decimal::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Decimal::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Decimal::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Decimal::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Decimal::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Decimal::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Decimal::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Decimal::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Decimal::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Decimal::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Decimal::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Decimal::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Decimal::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Decimal::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Decimal::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Decimal::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Decimal::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Decimal::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Decimal::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Decimal::validate(&self, data: &vortex_array::arrays::decimal::DecimalData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Decimal::validate(&self, &vortex_array::arrays::decimal::DecimalData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Decimal::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::take(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::arrays::decimal::DecimalMaskedValidityRule pub type vortex_array::arrays::decimal::DecimalMaskedValidityRule::Parent = vortex_array::arrays::Masked -pub fn vortex_array::arrays::decimal::DecimalMaskedValidityRule::reduce_parent(&self, array: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, parent: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, _child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::decimal::DecimalMaskedValidityRule::reduce_parent(&self, vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, usize) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::between::BetweenKernel for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::between(arr: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, lower: &vortex_array::ArrayRef, upper: &vortex_array::ArrayRef, options: &vortex_array::scalar_fn::fns::between::BetweenOptions, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::between(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, &vortex_array::ArrayRef, &vortex_array::ArrayRef, &vortex_array::scalar_fn::fns::between::BetweenOptions, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, dtype: &vortex_array::dtype::DType, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::Decimal + +pub fn vortex_array::arrays::Decimal::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::fill_null::FillNullKernel for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::fill_null(array: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, fill_value: &vortex_array::scalar::Scalar, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::fill_null(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, &vortex_array::scalar::Scalar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::mask(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> pub struct vortex_array::arrays::decimal::DecimalData @@ -1762,23 +1980,23 @@ pub fn vortex_array::arrays::decimal::DecimalData::is_empty(&self) -> bool pub fn vortex_array::arrays::decimal::DecimalData::len(&self) -> usize -pub fn vortex_array::arrays::decimal::DecimalData::new(buffer: vortex_buffer::buffer::Buffer, decimal_dtype: vortex_array::dtype::DecimalDType) -> Self +pub fn vortex_array::arrays::decimal::DecimalData::new(vortex_buffer::buffer::Buffer, vortex_array::dtype::DecimalDType) -> Self -pub fn vortex_array::arrays::decimal::DecimalData::new_handle(values: vortex_array::buffer::BufferHandle, values_type: vortex_array::dtype::DecimalType, decimal_dtype: vortex_array::dtype::DecimalDType) -> Self +pub fn vortex_array::arrays::decimal::DecimalData::new_handle(vortex_array::buffer::BufferHandle, vortex_array::dtype::DecimalType, vortex_array::dtype::DecimalDType) -> Self -pub unsafe fn vortex_array::arrays::decimal::DecimalData::new_unchecked(buffer: vortex_buffer::buffer::Buffer, decimal_dtype: vortex_array::dtype::DecimalDType) -> Self +pub unsafe fn vortex_array::arrays::decimal::DecimalData::new_unchecked(vortex_buffer::buffer::Buffer, vortex_array::dtype::DecimalDType) -> Self -pub unsafe fn vortex_array::arrays::decimal::DecimalData::new_unchecked_from_byte_buffer(byte_buffer: vortex_buffer::ByteBuffer, values_type: vortex_array::dtype::DecimalType, decimal_dtype: vortex_array::dtype::DecimalDType) -> Self +pub unsafe fn vortex_array::arrays::decimal::DecimalData::new_unchecked_from_byte_buffer(vortex_buffer::ByteBuffer, vortex_array::dtype::DecimalType, vortex_array::dtype::DecimalDType) -> Self -pub unsafe fn vortex_array::arrays::decimal::DecimalData::new_unchecked_handle(values: vortex_array::buffer::BufferHandle, values_type: vortex_array::dtype::DecimalType, decimal_dtype: vortex_array::dtype::DecimalDType) -> Self +pub unsafe fn vortex_array::arrays::decimal::DecimalData::new_unchecked_handle(vortex_array::buffer::BufferHandle, vortex_array::dtype::DecimalType, vortex_array::dtype::DecimalDType) -> Self pub fn vortex_array::arrays::decimal::DecimalData::precision(&self) -> u8 pub fn vortex_array::arrays::decimal::DecimalData::scale(&self) -> i8 -pub fn vortex_array::arrays::decimal::DecimalData::try_new(buffer: vortex_buffer::buffer::Buffer, decimal_dtype: vortex_array::dtype::DecimalDType) -> vortex_error::VortexResult +pub fn vortex_array::arrays::decimal::DecimalData::try_new(vortex_buffer::buffer::Buffer, vortex_array::dtype::DecimalDType) -> vortex_error::VortexResult -pub fn vortex_array::arrays::decimal::DecimalData::try_new_handle(values: vortex_array::buffer::BufferHandle, values_type: vortex_array::dtype::DecimalType, decimal_dtype: vortex_array::dtype::DecimalDType) -> vortex_error::VortexResult +pub fn vortex_array::arrays::decimal::DecimalData::try_new_handle(vortex_array::buffer::BufferHandle, vortex_array::dtype::DecimalType, vortex_array::dtype::DecimalDType) -> vortex_error::VortexResult pub fn vortex_array::arrays::decimal::DecimalData::values_type(&self) -> vortex_array::dtype::DecimalType @@ -1788,19 +2006,19 @@ pub fn vortex_array::arrays::decimal::DecimalData::clone(&self) -> vortex_array: impl core::fmt::Debug for vortex_array::arrays::decimal::DecimalData -pub fn vortex_array::arrays::decimal::DecimalData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::decimal::DecimalData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::arrays::decimal::DecimalData -pub fn vortex_array::arrays::decimal::DecimalData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::decimal::DecimalData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::ArrayEq for vortex_array::arrays::decimal::DecimalData -pub fn vortex_array::arrays::decimal::DecimalData::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::decimal::DecimalData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayHash for vortex_array::arrays::decimal::DecimalData -pub fn vortex_array::arrays::decimal::DecimalData::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn vortex_array::arrays::decimal::DecimalData::array_hash(&self, &mut H, vortex_array::Precision) pub struct vortex_array::arrays::decimal::DecimalDataParts @@ -1820,13 +2038,13 @@ pub fn vortex_array::arrays::decimal::DecimalMaskedValidityRule::default() -> vo impl core::fmt::Debug for vortex_array::arrays::decimal::DecimalMaskedValidityRule -pub fn vortex_array::arrays::decimal::DecimalMaskedValidityRule::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::decimal::DecimalMaskedValidityRule::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::arrays::decimal::DecimalMaskedValidityRule pub type vortex_array::arrays::decimal::DecimalMaskedValidityRule::Parent = vortex_array::arrays::Masked -pub fn vortex_array::arrays::decimal::DecimalMaskedValidityRule::reduce_parent(&self, array: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, parent: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, _child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::decimal::DecimalMaskedValidityRule::reduce_parent(&self, vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, usize) -> vortex_error::VortexResult> pub trait vortex_array::arrays::decimal::DecimalArrayExt: vortex_array::TypedArrayRef @@ -1868,7 +2086,7 @@ pub fn T::validity_child(&self) -> core::option::Option<&vortex_array::ArrayRef> pub fn T::values_type(&self) -> vortex_array::dtype::DecimalType -pub fn vortex_array::arrays::decimal::narrowed_decimal(decimal_array: vortex_array::arrays::DecimalArray) -> vortex_array::arrays::DecimalArray +pub fn vortex_array::arrays::decimal::narrowed_decimal(vortex_array::arrays::DecimalArray) -> vortex_array::arrays::DecimalArray pub type vortex_array::arrays::decimal::DecimalArray = vortex_array::Array @@ -1878,21 +2096,17 @@ pub mod vortex_array::arrays::dict::vtable pub struct vortex_array::arrays::dict::vtable::Dict -impl vortex_array::arrays::dict::Dict - -pub const vortex_array::arrays::dict::Dict::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::dict::Dict pub fn vortex_array::arrays::dict::Dict::clone(&self) -> vortex_array::arrays::dict::Dict impl core::fmt::Debug for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::dict::Dict::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::dict::Dict::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::dict::Dict @@ -1902,93 +2116,89 @@ pub type vortex_array::arrays::dict::Dict::OperationsVTable = vortex_array::arra pub type vortex_array::arrays::dict::Dict::ValidityVTable = vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::dict::Dict::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::dict::Dict::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::dict::Dict::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::dict::Dict::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::dict::Dict::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::dict::Dict::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::dict::Dict::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::dict::Dict::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::dict::Dict::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::dict::Dict::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::dict::Dict::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::dict::Dict::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::dict::Dict::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::dict::Dict::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::dict::Dict::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::dict::Dict::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::dict::Dict::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::dict::Dict::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::dict::Dict::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::dict::Dict::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::dict::Dict::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::dict::Dict::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::dict::Dict::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::dict::Dict::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::dict::Dict::validate(&self, _data: &vortex_array::arrays::dict::DictData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::dict::Dict::validate(&self, &vortex_array::arrays::dict::DictData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::dict::Dict::validity(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, indices: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::take(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterReduce for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::filter(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::filter(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::binary::CompareKernel for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::compare(lhs: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, rhs: &vortex_array::ArrayRef, operator: vortex_array::scalar_fn::fns::operators::CompareOperator, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::compare(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_array::ArrayRef, vortex_array::scalar_fn::fns::operators::CompareOperator, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::cast(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::fill_null::FillNullKernel for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::fill_null(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, fill_value: &vortex_array::scalar::Scalar, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::fill_null(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_array::scalar::Scalar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::like::LikeReduce for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::like(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, pattern: &vortex_array::ArrayRef, options: vortex_array::scalar_fn::fns::like::LikeOptions) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::like(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_array::ArrayRef, vortex_array::scalar_fn::fns::like::LikeOptions) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::mask(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> pub type vortex_array::arrays::dict::vtable::DictArray = vortex_array::Array pub struct vortex_array::arrays::dict::Dict -impl vortex_array::arrays::dict::Dict - -pub const vortex_array::arrays::dict::Dict::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::dict::Dict pub fn vortex_array::arrays::dict::Dict::clone(&self) -> vortex_array::arrays::dict::Dict impl core::fmt::Debug for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::dict::Dict::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::dict::Dict::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::dict::Dict @@ -1998,83 +2208,83 @@ pub type vortex_array::arrays::dict::Dict::OperationsVTable = vortex_array::arra pub type vortex_array::arrays::dict::Dict::ValidityVTable = vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::dict::Dict::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::dict::Dict::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::dict::Dict::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::dict::Dict::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::dict::Dict::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::dict::Dict::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::dict::Dict::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::dict::Dict::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::dict::Dict::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::dict::Dict::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::dict::Dict::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::dict::Dict::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::dict::Dict::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::dict::Dict::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::dict::Dict::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::dict::Dict::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::dict::Dict::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::dict::Dict::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::dict::Dict::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::dict::Dict::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::dict::Dict::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::dict::Dict::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::dict::Dict::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::dict::Dict::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::dict::Dict::validate(&self, _data: &vortex_array::arrays::dict::DictData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::dict::Dict::validate(&self, &vortex_array::arrays::dict::DictData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::dict::Dict::validity(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, indices: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::take(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterReduce for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::filter(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::filter(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::binary::CompareKernel for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::compare(lhs: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, rhs: &vortex_array::ArrayRef, operator: vortex_array::scalar_fn::fns::operators::CompareOperator, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::compare(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_array::ArrayRef, vortex_array::scalar_fn::fns::operators::CompareOperator, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::cast(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::fill_null::FillNullKernel for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::fill_null(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, fill_value: &vortex_array::scalar::Scalar, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::fill_null(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_array::scalar::Scalar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::like::LikeReduce for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::like(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, pattern: &vortex_array::ArrayRef, options: vortex_array::scalar_fn::fns::like::LikeOptions) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::like(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_array::ArrayRef, vortex_array::scalar_fn::fns::like::LikeOptions) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::mask(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> pub struct vortex_array::arrays::dict::DictData impl vortex_array::arrays::dict::DictData -pub fn vortex_array::arrays::dict::DictData::new(codes_dtype: &vortex_array::dtype::DType) -> Self +pub fn vortex_array::arrays::dict::DictData::new(&vortex_array::dtype::DType) -> Self pub unsafe fn vortex_array::arrays::dict::DictData::new_unchecked() -> Self -pub unsafe fn vortex_array::arrays::dict::DictData::set_all_values_referenced(self, all_values_referenced: bool) -> Self +pub unsafe fn vortex_array::arrays::dict::DictData::set_all_values_referenced(self, bool) -> Self impl core::clone::Clone for vortex_array::arrays::dict::DictData @@ -2082,19 +2292,19 @@ pub fn vortex_array::arrays::dict::DictData::clone(&self) -> vortex_array::array impl core::fmt::Debug for vortex_array::arrays::dict::DictData -pub fn vortex_array::arrays::dict::DictData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::dict::DictData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::arrays::dict::DictData -pub fn vortex_array::arrays::dict::DictData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::dict::DictData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::ArrayEq for vortex_array::arrays::dict::DictData -pub fn vortex_array::arrays::dict::DictData::array_eq(&self, _other: &Self, _precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::dict::DictData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayHash for vortex_array::arrays::dict::DictData -pub fn vortex_array::arrays::dict::DictData::array_hash(&self, _state: &mut H, _precision: vortex_array::Precision) +pub fn vortex_array::arrays::dict::DictData::array_hash(&self, &mut H, vortex_array::Precision) pub struct vortex_array::arrays::dict::DictMetadata @@ -2106,7 +2316,7 @@ pub fn vortex_array::arrays::dict::DictMetadata::codes_ptype(&self) -> vortex_ar pub fn vortex_array::arrays::dict::DictMetadata::is_nullable_codes(&self) -> bool -pub fn vortex_array::arrays::dict::DictMetadata::set_codes_ptype(&mut self, value: vortex_array::dtype::PType) +pub fn vortex_array::arrays::dict::DictMetadata::set_codes_ptype(&mut self, vortex_array::dtype::PType) impl core::clone::Clone for vortex_array::arrays::dict::DictMetadata @@ -2118,7 +2328,7 @@ pub fn vortex_array::arrays::dict::DictMetadata::default() -> Self impl core::fmt::Debug for vortex_array::arrays::dict::DictMetadata -pub fn vortex_array::arrays::dict::DictMetadata::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::dict::DictMetadata::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl prost::message::Message for vortex_array::arrays::dict::DictMetadata @@ -2126,6 +2336,14 @@ pub fn vortex_array::arrays::dict::DictMetadata::clear(&mut self) pub fn vortex_array::arrays::dict::DictMetadata::encoded_len(&self) -> usize +pub struct vortex_array::arrays::dict::DictParts + +pub vortex_array::arrays::dict::DictParts::codes: vortex_array::ArrayRef + +pub vortex_array::arrays::dict::DictParts::dtype: vortex_array::dtype::DType + +pub vortex_array::arrays::dict::DictParts::values: vortex_array::ArrayRef + pub struct vortex_array::arrays::dict::DictSlots pub vortex_array::arrays::dict::DictSlots::codes: vortex_array::ArrayRef @@ -2142,7 +2360,7 @@ pub const vortex_array::arrays::dict::DictSlots::NAMES: [&'static str; 2] pub const vortex_array::arrays::dict::DictSlots::VALUES: usize -pub fn vortex_array::arrays::dict::DictSlots::from_slots(slots: alloc::vec::Vec>) -> Self +pub fn vortex_array::arrays::dict::DictSlots::from_slots(alloc::vec::Vec>) -> Self pub fn vortex_array::arrays::dict::DictSlots::into_slots(self) -> alloc::vec::Vec> @@ -2154,7 +2372,7 @@ pub vortex_array::arrays::dict::DictSlotsView::values: &'a vortex_array::ArrayRe impl<'a> vortex_array::arrays::dict::DictSlotsView<'a> -pub fn vortex_array::arrays::dict::DictSlotsView<'a>::from_slots(slots: &'a [core::option::Option]) -> Self +pub fn vortex_array::arrays::dict::DictSlotsView<'a>::from_slots(&'a [core::option::Option]) -> Self pub fn vortex_array::arrays::dict::DictSlotsView<'a>::to_owned(&self) -> vortex_array::arrays::dict::DictSlots @@ -2164,7 +2382,7 @@ pub fn vortex_array::arrays::dict::DictSlotsView<'a>::clone(&self) -> vortex_arr impl<'a> core::fmt::Debug for vortex_array::arrays::dict::DictSlotsView<'a> -pub fn vortex_array::arrays::dict::DictSlotsView<'a>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::dict::DictSlotsView<'a>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl<'a> core::marker::Copy for vortex_array::arrays::dict::DictSlotsView<'a> @@ -2176,13 +2394,13 @@ pub fn vortex_array::arrays::dict::TakeExecuteAdaptor::default() -> vortex_ar impl core::fmt::Debug for vortex_array::arrays::dict::TakeExecuteAdaptor -pub fn vortex_array::arrays::dict::TakeExecuteAdaptor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::dict::TakeExecuteAdaptor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::kernel::ExecuteParentKernel for vortex_array::arrays::dict::TakeExecuteAdaptor where V: vortex_array::arrays::dict::TakeExecute pub type vortex_array::arrays::dict::TakeExecuteAdaptor::Parent = vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::TakeExecuteAdaptor::execute_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: ::Match, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::TakeExecuteAdaptor::execute_parent(&self, vortex_array::ArrayView<'_, V>, ::Match, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub struct vortex_array::arrays::dict::TakeReduceAdaptor(pub V) @@ -2192,17 +2410,17 @@ pub fn vortex_array::arrays::dict::TakeReduceAdaptor::default() -> vortex_arr impl core::fmt::Debug for vortex_array::arrays::dict::TakeReduceAdaptor -pub fn vortex_array::arrays::dict::TakeReduceAdaptor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::dict::TakeReduceAdaptor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::arrays::dict::TakeReduceAdaptor where V: vortex_array::arrays::dict::TakeReduce pub type vortex_array::arrays::dict::TakeReduceAdaptor::Parent = vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::TakeReduceAdaptor::reduce_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::TakeReduceAdaptor::reduce_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, usize) -> vortex_error::VortexResult> pub trait vortex_array::arrays::dict::DictArrayExt: vortex_array::TypedArrayRef + vortex_array::arrays::dict::DictArraySlotsExt -pub fn vortex_array::arrays::dict::DictArrayExt::compute_referenced_values_mask(&self, referenced: bool) -> vortex_error::VortexResult +pub fn vortex_array::arrays::dict::DictArrayExt::compute_referenced_values_mask(&self, bool) -> vortex_error::VortexResult pub fn vortex_array::arrays::dict::DictArrayExt::has_all_values_referenced(&self) -> bool @@ -2210,7 +2428,7 @@ pub fn vortex_array::arrays::dict::DictArrayExt::validate_all_values_referenced( impl> vortex_array::arrays::dict::DictArrayExt for T -pub fn T::compute_referenced_values_mask(&self, referenced: bool) -> vortex_error::VortexResult +pub fn T::compute_referenced_values_mask(&self, bool) -> vortex_error::VortexResult pub fn T::has_all_values_referenced(&self) -> bool @@ -2232,79 +2450,89 @@ pub fn T::slots_view(&self) -> vortex_array::arrays::dict::DictSlotsView<'_> pub fn T::values(&self) -> &vortex_array::ArrayRef +pub trait vortex_array::arrays::dict::DictOwnedExt + +pub fn vortex_array::arrays::dict::DictOwnedExt::into_parts(self) -> vortex_array::arrays::dict::DictParts + +impl vortex_array::arrays::dict::DictOwnedExt for vortex_array::Array + +pub fn vortex_array::Array::into_parts(self) -> vortex_array::arrays::dict::DictParts + pub trait vortex_array::arrays::dict::TakeExecute: vortex_array::VTable -pub fn vortex_array::arrays::dict::TakeExecute::take(array: vortex_array::ArrayView<'_, Self>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::TakeExecute::take(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::take(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::take(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::take(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, indices: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::take(vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::take(vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::List -pub fn vortex_array::arrays::List::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::List>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::take(vortex_array::ArrayView<'_, vortex_array::arrays::List>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> + +impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::ListView + +pub fn vortex_array::arrays::ListView::take(vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::take(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::take(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::take(vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, indices: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::take(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::patched::Patched -pub fn vortex_array::arrays::patched::Patched::take(array: vortex_array::ArrayView<'_, Self>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::take(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub trait vortex_array::arrays::dict::TakeReduce: vortex_array::VTable -pub fn vortex_array::arrays::dict::TakeReduce::take(array: vortex_array::ArrayView<'_, Self>, indices: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::TakeReduce::take(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::arrays::dict::TakeReduce for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, indices: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::take(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::arrays::dict::TakeReduce for vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, indices: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::take(vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::arrays::dict::TakeReduce for vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, indices: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::take(vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::arrays::dict::TakeReduce for vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, indices: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::take(vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::arrays::dict::TakeReduce for vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, indices: &vortex_array::ArrayRef) -> vortex_error::VortexResult> - -pub fn vortex_array::arrays::dict::take_canonical(values: vortex_array::Canonical, codes: &vortex_array::arrays::PrimitiveArray, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::null::Null::take(vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> pub type vortex_array::arrays::dict::DictArray = vortex_array::Array @@ -2312,128 +2540,96 @@ pub mod vortex_array::arrays::extension pub struct vortex_array::arrays::extension::Extension -impl vortex_array::arrays::Extension - -pub const vortex_array::arrays::Extension::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::Extension pub fn vortex_array::arrays::Extension::clone(&self) -> vortex_array::arrays::Extension impl core::fmt::Debug for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::Extension::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Extension::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::Extension -pub type vortex_array::arrays::Extension::ArrayData = vortex_array::arrays::extension::ExtensionData +pub type vortex_array::arrays::Extension::ArrayData = vortex_array::EmptyArrayData pub type vortex_array::arrays::Extension::OperationsVTable = vortex_array::arrays::Extension pub type vortex_array::arrays::Extension::ValidityVTable = vortex_array::ValidityVTableFromChild -pub fn vortex_array::arrays::Extension::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Extension::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Extension::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Extension::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Extension::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Extension::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Extension::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Extension::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Extension::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Extension::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Extension::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Extension::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Extension::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Extension::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Extension::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Extension::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Extension::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Extension::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Extension::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Extension::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Extension::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Extension::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Extension::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Extension::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Extension::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Extension::validate(&self, data: &vortex_array::arrays::extension::ExtensionData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Extension::validate(&self, &vortex_array::EmptyArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityChild for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::validity_child(array: vortex_array::ArrayView<'_, vortex_array::arrays::Extension>) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Extension::validity_child(vortex_array::ArrayView<'_, vortex_array::arrays::Extension>) -> vortex_array::ArrayRef impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, indices: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::take(vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterReduce for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::filter(array: vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::filter(vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::binary::CompareKernel for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::compare(lhs: vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, rhs: &vortex_array::ArrayRef, operator: vortex_array::scalar_fn::fns::operators::CompareOperator, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::compare(vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, &vortex_array::ArrayRef, vortex_array::scalar_fn::fns::operators::CompareOperator, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> - -pub struct vortex_array::arrays::extension::ExtensionData - -impl vortex_array::arrays::extension::ExtensionData - -pub fn vortex_array::arrays::extension::ExtensionData::ext_dtype(&self) -> &vortex_array::dtype::extension::ExtDTypeRef - -pub fn vortex_array::arrays::extension::ExtensionData::new(ext_dtype: vortex_array::dtype::extension::ExtDTypeRef, storage_dtype: &vortex_array::dtype::DType) -> Self - -pub unsafe fn vortex_array::arrays::extension::ExtensionData::new_unchecked(ext_dtype: vortex_array::dtype::extension::ExtDTypeRef, storage_dtype: &vortex_array::dtype::DType) -> Self - -pub fn vortex_array::arrays::extension::ExtensionData::try_new(ext_dtype: vortex_array::dtype::extension::ExtDTypeRef, storage_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult - -impl core::clone::Clone for vortex_array::arrays::extension::ExtensionData - -pub fn vortex_array::arrays::extension::ExtensionData::clone(&self) -> vortex_array::arrays::extension::ExtensionData - -impl core::fmt::Debug for vortex_array::arrays::extension::ExtensionData - -pub fn vortex_array::arrays::extension::ExtensionData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result - -impl core::fmt::Display for vortex_array::arrays::extension::ExtensionData - -pub fn vortex_array::arrays::extension::ExtensionData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result - -impl vortex_array::ArrayEq for vortex_array::arrays::extension::ExtensionData - -pub fn vortex_array::arrays::extension::ExtensionData::array_eq(&self, _other: &Self, _precision: vortex_array::Precision) -> bool - -impl vortex_array::ArrayHash for vortex_array::arrays::extension::ExtensionData - -pub fn vortex_array::arrays::extension::ExtensionData::array_hash(&self, _state: &mut H, _precision: vortex_array::Precision) +pub fn vortex_array::arrays::Extension::mask(vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> pub trait vortex_array::arrays::extension::ExtensionArrayExt: vortex_array::TypedArrayRef +pub fn vortex_array::arrays::extension::ExtensionArrayExt::ext_dtype(&self) -> &vortex_array::dtype::extension::ExtDTypeRef + pub fn vortex_array::arrays::extension::ExtensionArrayExt::storage_array(&self) -> &vortex_array::ArrayRef impl> vortex_array::arrays::extension::ExtensionArrayExt for T +pub fn T::ext_dtype(&self) -> &vortex_array::dtype::extension::ExtDTypeRef + pub fn T::storage_array(&self) -> &vortex_array::ArrayRef pub type vortex_array::arrays::extension::ExtensionArray = vortex_array::Array @@ -2442,21 +2638,17 @@ pub mod vortex_array::arrays::filter pub struct vortex_array::arrays::filter::Filter -impl vortex_array::arrays::Filter - -pub const vortex_array::arrays::Filter::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::Filter pub fn vortex_array::arrays::Filter::clone(&self) -> vortex_array::arrays::Filter impl core::fmt::Debug for vortex_array::arrays::Filter -pub fn vortex_array::arrays::Filter::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::Filter::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::Filter -pub fn vortex_array::arrays::Filter::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Filter>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Filter::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Filter>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::Filter @@ -2466,41 +2658,41 @@ pub type vortex_array::arrays::Filter::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::Filter::ValidityVTable = vortex_array::arrays::Filter -pub fn vortex_array::arrays::Filter::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Filter::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Filter::buffer(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Filter::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Filter::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Filter::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Filter::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Filter::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Filter::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Filter::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Filter::deserialize(&self, _dtype: &vortex_array::dtype::DType, _len: usize, _metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Filter::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Filter::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Filter::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Filter::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Filter::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Filter::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Filter::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Filter::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Filter::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Filter::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Filter::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Filter::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Filter::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Filter::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Filter::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Filter::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Filter::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Filter::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Filter::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Filter::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::Filter -pub fn vortex_array::arrays::Filter::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Filter>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Filter::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Filter>) -> vortex_error::VortexResult pub struct vortex_array::arrays::filter::FilterData @@ -2514,7 +2706,7 @@ pub fn vortex_array::arrays::filter::FilterData::is_empty(&self) -> bool pub fn vortex_array::arrays::filter::FilterData::len(&self) -> usize -pub fn vortex_array::arrays::filter::FilterData::new(mask: vortex_mask::Mask) -> Self +pub fn vortex_array::arrays::filter::FilterData::new(vortex_mask::Mask) -> Self impl core::clone::Clone for vortex_array::arrays::filter::FilterData @@ -2522,19 +2714,19 @@ pub fn vortex_array::arrays::filter::FilterData::clone(&self) -> vortex_array::a impl core::fmt::Debug for vortex_array::arrays::filter::FilterData -pub fn vortex_array::arrays::filter::FilterData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::filter::FilterData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::arrays::filter::FilterData -pub fn vortex_array::arrays::filter::FilterData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::filter::FilterData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::ArrayEq for vortex_array::arrays::filter::FilterData -pub fn vortex_array::arrays::filter::FilterData::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::filter::FilterData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayHash for vortex_array::arrays::filter::FilterData -pub fn vortex_array::arrays::filter::FilterData::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn vortex_array::arrays::filter::FilterData::array_hash(&self, &mut H, vortex_array::Precision) pub struct vortex_array::arrays::filter::FilterDataParts @@ -2548,13 +2740,13 @@ pub fn vortex_array::arrays::filter::FilterExecuteAdaptor::default() -> vorte impl core::fmt::Debug for vortex_array::arrays::filter::FilterExecuteAdaptor -pub fn vortex_array::arrays::filter::FilterExecuteAdaptor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::filter::FilterExecuteAdaptor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::kernel::ExecuteParentKernel for vortex_array::arrays::filter::FilterExecuteAdaptor where V: vortex_array::arrays::filter::FilterKernel pub type vortex_array::arrays::filter::FilterExecuteAdaptor::Parent = vortex_array::arrays::Filter -pub fn vortex_array::arrays::filter::FilterExecuteAdaptor::execute_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: ::Match, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::filter::FilterExecuteAdaptor::execute_parent(&self, vortex_array::ArrayView<'_, V>, ::Match, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub struct vortex_array::arrays::filter::FilterReduceAdaptor(pub V) @@ -2564,13 +2756,13 @@ pub fn vortex_array::arrays::filter::FilterReduceAdaptor::default() -> vortex impl core::fmt::Debug for vortex_array::arrays::filter::FilterReduceAdaptor -pub fn vortex_array::arrays::filter::FilterReduceAdaptor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::filter::FilterReduceAdaptor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::arrays::filter::FilterReduceAdaptor where V: vortex_array::arrays::filter::FilterReduce pub type vortex_array::arrays::filter::FilterReduceAdaptor::Parent = vortex_array::arrays::Filter -pub fn vortex_array::arrays::filter::FilterReduceAdaptor::reduce_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::ArrayView<'_, vortex_array::arrays::Filter>, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::filter::FilterReduceAdaptor::reduce_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::ArrayView<'_, vortex_array::arrays::Filter>, usize) -> vortex_error::VortexResult> pub trait vortex_array::arrays::filter::FilterArrayExt: vortex_array::TypedArrayRef @@ -2582,51 +2774,51 @@ pub fn T::child(&self) -> &vortex_array::ArrayRef pub trait vortex_array::arrays::filter::FilterKernel: vortex_array::VTable -pub fn vortex_array::arrays::filter::FilterKernel::filter(array: vortex_array::ArrayView<'_, Self>, mask: &vortex_mask::Mask, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::filter::FilterKernel::filter(vortex_array::ArrayView<'_, Self>, &vortex_mask::Mask, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterKernel for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::filter(array: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, mask: &vortex_mask::Mask, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::filter(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, &vortex_mask::Mask, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterKernel for vortex_array::arrays::List -pub fn vortex_array::arrays::List::filter(array: vortex_array::ArrayView<'_, vortex_array::arrays::List>, mask: &vortex_mask::Mask, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::filter(vortex_array::ArrayView<'_, vortex_array::arrays::List>, &vortex_mask::Mask, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterKernel for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::filter(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, mask: &vortex_mask::Mask, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::filter(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, &vortex_mask::Mask, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub trait vortex_array::arrays::filter::FilterReduce: vortex_array::VTable -pub fn vortex_array::arrays::filter::FilterReduce::filter(array: vortex_array::ArrayView<'_, Self>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::filter::FilterReduce::filter(vortex_array::ArrayView<'_, Self>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterReduce for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::filter(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::filter(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterReduce for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::filter(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::filter(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterReduce for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::filter(array: vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::filter(vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterReduce for vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::filter(array: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::filter(vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterReduce for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::filter(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::filter(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterReduce for vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::filter(_array: vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::filter(vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterReduce for vortex_array::arrays::patched::Patched -pub fn vortex_array::arrays::patched::Patched::filter(array: vortex_array::ArrayView<'_, Self>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::filter(vortex_array::ArrayView<'_, Self>, &vortex_mask::Mask) -> vortex_error::VortexResult> pub type vortex_array::arrays::filter::FilterArray = vortex_array::Array @@ -2634,21 +2826,17 @@ pub mod vortex_array::arrays::fixed_size_list pub struct vortex_array::arrays::fixed_size_list::FixedSizeList -impl vortex_array::arrays::FixedSizeList - -pub const vortex_array::arrays::FixedSizeList::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::FixedSizeList pub fn vortex_array::arrays::FixedSizeList::clone(&self) -> vortex_array::arrays::FixedSizeList impl core::fmt::Debug for vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::FixedSizeList::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::FixedSizeList::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::FixedSizeList @@ -2658,67 +2846,71 @@ pub type vortex_array::arrays::FixedSizeList::OperationsVTable = vortex_array::a pub type vortex_array::arrays::FixedSizeList::ValidityVTable = vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::FixedSizeList::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::FixedSizeList::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::FixedSizeList::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::FixedSizeList::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::FixedSizeList::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::FixedSizeList::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::FixedSizeList::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::FixedSizeList::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::FixedSizeList::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::FixedSizeList::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::FixedSizeList::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::FixedSizeList::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::FixedSizeList::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::FixedSizeList::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::FixedSizeList::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::FixedSizeList::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::FixedSizeList::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::FixedSizeList::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::FixedSizeList::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::FixedSizeList::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::FixedSizeList::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::FixedSizeList::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::FixedSizeList::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::FixedSizeList::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::FixedSizeList::validate(&self, data: &vortex_array::arrays::fixed_size_list::FixedSizeListData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::FixedSizeList::validate(&self, &vortex_array::arrays::fixed_size_list::FixedSizeListData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::FixedSizeList::validity(vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::take(vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::FixedSizeList + +pub fn vortex_array::arrays::FixedSizeList::cast(vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::cast(vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::mask(vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> pub struct vortex_array::arrays::fixed_size_list::FixedSizeListData impl vortex_array::arrays::fixed_size_list::FixedSizeListData -pub fn vortex_array::arrays::fixed_size_list::FixedSizeListData::build(elements: vortex_array::ArrayRef, list_size: u32, validity: vortex_array::validity::Validity, len: usize) -> Self +pub fn vortex_array::arrays::fixed_size_list::FixedSizeListData::build(vortex_array::ArrayRef, u32, vortex_array::validity::Validity, usize) -> Self -pub unsafe fn vortex_array::arrays::fixed_size_list::FixedSizeListData::new_unchecked(list_size: u32, len: usize) -> Self +pub unsafe fn vortex_array::arrays::fixed_size_list::FixedSizeListData::new_unchecked(u32, usize) -> Self -pub fn vortex_array::arrays::fixed_size_list::FixedSizeListData::validate(elements: &vortex_array::ArrayRef, len: usize, list_size: u32, validity: &vortex_array::validity::Validity) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::fixed_size_list::FixedSizeListData::validate(&vortex_array::ArrayRef, usize, u32, &vortex_array::validity::Validity) -> vortex_error::VortexResult<()> impl core::clone::Clone for vortex_array::arrays::fixed_size_list::FixedSizeListData @@ -2726,19 +2918,19 @@ pub fn vortex_array::arrays::fixed_size_list::FixedSizeListData::clone(&self) -> impl core::fmt::Debug for vortex_array::arrays::fixed_size_list::FixedSizeListData -pub fn vortex_array::arrays::fixed_size_list::FixedSizeListData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::fixed_size_list::FixedSizeListData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::arrays::fixed_size_list::FixedSizeListData -pub fn vortex_array::arrays::fixed_size_list::FixedSizeListData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::fixed_size_list::FixedSizeListData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::ArrayEq for vortex_array::arrays::fixed_size_list::FixedSizeListData -pub fn vortex_array::arrays::fixed_size_list::FixedSizeListData::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::fixed_size_list::FixedSizeListData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayHash for vortex_array::arrays::fixed_size_list::FixedSizeListData -pub fn vortex_array::arrays::fixed_size_list::FixedSizeListData::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn vortex_array::arrays::fixed_size_list::FixedSizeListData::array_hash(&self, &mut H, vortex_array::Precision) pub struct vortex_array::arrays::fixed_size_list::FixedSizeListDataParts @@ -2754,12 +2946,10 @@ pub fn vortex_array::arrays::fixed_size_list::FixedSizeListArrayExt::dtype_parts pub fn vortex_array::arrays::fixed_size_list::FixedSizeListArrayExt::elements(&self) -> &vortex_array::ArrayRef -pub fn vortex_array::arrays::fixed_size_list::FixedSizeListArrayExt::fixed_size_list_elements_at(&self, index: usize) -> vortex_error::VortexResult +pub fn vortex_array::arrays::fixed_size_list::FixedSizeListArrayExt::fixed_size_list_elements_at(&self, usize) -> vortex_error::VortexResult pub fn vortex_array::arrays::fixed_size_list::FixedSizeListArrayExt::fixed_size_list_validity(&self) -> vortex_array::validity::Validity -pub fn vortex_array::arrays::fixed_size_list::FixedSizeListArrayExt::fixed_size_list_validity_mask(&self) -> vortex_mask::Mask - pub fn vortex_array::arrays::fixed_size_list::FixedSizeListArrayExt::list_size(&self) -> u32 impl> vortex_array::arrays::fixed_size_list::FixedSizeListArrayExt for T @@ -2768,12 +2958,10 @@ pub fn T::dtype_parts(&self) -> (&vortex_array::dtype::DType, u32, vortex_array: pub fn T::elements(&self) -> &vortex_array::ArrayRef -pub fn T::fixed_size_list_elements_at(&self, index: usize) -> vortex_error::VortexResult +pub fn T::fixed_size_list_elements_at(&self, usize) -> vortex_error::VortexResult pub fn T::fixed_size_list_validity(&self) -> vortex_array::validity::Validity -pub fn T::fixed_size_list_validity_mask(&self) -> vortex_mask::Mask - pub fn T::list_size(&self) -> u32 pub type vortex_array::arrays::fixed_size_list::FixedSizeListArray = vortex_array::Array @@ -2782,21 +2970,17 @@ pub mod vortex_array::arrays::list pub struct vortex_array::arrays::list::List -impl vortex_array::arrays::List - -pub const vortex_array::arrays::List::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::List pub fn vortex_array::arrays::List::clone(&self) -> vortex_array::arrays::List impl core::fmt::Debug for vortex_array::arrays::List -pub fn vortex_array::arrays::List::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::List::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::List -pub fn vortex_array::arrays::List::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::List>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::List::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::List>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::List @@ -2806,71 +2990,75 @@ pub type vortex_array::arrays::List::OperationsVTable = vortex_array::arrays::Li pub type vortex_array::arrays::List::ValidityVTable = vortex_array::arrays::List -pub fn vortex_array::arrays::List::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::List::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::List::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::List::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::List::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::List::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::List::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::List::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::List::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::List::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::List::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::List::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::List::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::List::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::List::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::List::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::List::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::List::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::List::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::List::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::List::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::List::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::List::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::List::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::List::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::List::validate(&self, _data: &vortex_array::arrays::list::ListData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::List::validate(&self, &vortex_array::arrays::list::ListData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::List -pub fn vortex_array::arrays::List::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::List>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::List::validity(vortex_array::ArrayView<'_, vortex_array::arrays::List>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::List -pub fn vortex_array::arrays::List::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::List>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::take(vortex_array::ArrayView<'_, vortex_array::arrays::List>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterKernel for vortex_array::arrays::List -pub fn vortex_array::arrays::List::filter(array: vortex_array::ArrayView<'_, vortex_array::arrays::List>, mask: &vortex_mask::Mask, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::filter(vortex_array::ArrayView<'_, vortex_array::arrays::List>, &vortex_mask::Mask, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::List -pub fn vortex_array::arrays::List::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::List + +pub fn vortex_array::arrays::List::cast(vortex_array::ArrayView<'_, vortex_array::arrays::List>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::List -pub fn vortex_array::arrays::List::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::List>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::cast(vortex_array::ArrayView<'_, vortex_array::arrays::List>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::List -pub fn vortex_array::arrays::List::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::List>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::mask(vortex_array::ArrayView<'_, vortex_array::arrays::List>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> pub struct vortex_array::arrays::list::ListData impl vortex_array::arrays::list::ListData -pub fn vortex_array::arrays::list::ListData::build(elements: vortex_array::ArrayRef, offsets: vortex_array::ArrayRef, validity: vortex_array::validity::Validity) -> Self +pub fn vortex_array::arrays::list::ListData::build(vortex_array::ArrayRef, vortex_array::ArrayRef, vortex_array::validity::Validity) -> Self pub unsafe fn vortex_array::arrays::list::ListData::new_unchecked() -> Self -pub fn vortex_array::arrays::list::ListData::validate(elements: &vortex_array::ArrayRef, offsets: &vortex_array::ArrayRef, validity: &vortex_array::validity::Validity) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::list::ListData::validate(&vortex_array::ArrayRef, &vortex_array::ArrayRef, &vortex_array::validity::Validity) -> vortex_error::VortexResult<()> impl core::clone::Clone for vortex_array::arrays::list::ListData @@ -2882,19 +3070,19 @@ pub fn vortex_array::arrays::list::ListData::default() -> vortex_array::arrays:: impl core::fmt::Debug for vortex_array::arrays::list::ListData -pub fn vortex_array::arrays::list::ListData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::list::ListData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::arrays::list::ListData -pub fn vortex_array::arrays::list::ListData::fmt(&self, _f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::list::ListData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::ArrayEq for vortex_array::arrays::list::ListData -pub fn vortex_array::arrays::list::ListData::array_eq(&self, _other: &Self, _precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::list::ListData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayHash for vortex_array::arrays::list::ListData -pub fn vortex_array::arrays::list::ListData::array_hash(&self, _state: &mut H, _precision: vortex_array::Precision) +pub fn vortex_array::arrays::list::ListData::array_hash(&self, &mut H, vortex_array::Precision) pub struct vortex_array::arrays::list::ListDataParts @@ -2912,19 +3100,17 @@ pub fn vortex_array::arrays::list::ListArrayExt::element_dtype(&self) -> &vortex pub fn vortex_array::arrays::list::ListArrayExt::elements(&self) -> &vortex_array::ArrayRef -pub fn vortex_array::arrays::list::ListArrayExt::list_elements_at(&self, index: usize) -> vortex_error::VortexResult +pub fn vortex_array::arrays::list::ListArrayExt::list_elements_at(&self, usize) -> vortex_error::VortexResult pub fn vortex_array::arrays::list::ListArrayExt::list_validity(&self) -> vortex_array::validity::Validity -pub fn vortex_array::arrays::list::ListArrayExt::list_validity_mask(&self) -> vortex_mask::Mask - pub fn vortex_array::arrays::list::ListArrayExt::nullability(&self) -> vortex_array::dtype::Nullability -pub fn vortex_array::arrays::list::ListArrayExt::offset_at(&self, index: usize) -> vortex_error::VortexResult +pub fn vortex_array::arrays::list::ListArrayExt::offset_at(&self, usize) -> vortex_error::VortexResult pub fn vortex_array::arrays::list::ListArrayExt::offsets(&self) -> &vortex_array::ArrayRef -pub fn vortex_array::arrays::list::ListArrayExt::reset_offsets(&self, recurse: bool) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::list::ListArrayExt::reset_offsets(&self, bool) -> vortex_error::VortexResult> pub fn vortex_array::arrays::list::ListArrayExt::sliced_elements(&self) -> vortex_error::VortexResult @@ -2934,19 +3120,17 @@ pub fn T::element_dtype(&self) -> &vortex_array::dtype::DType pub fn T::elements(&self) -> &vortex_array::ArrayRef -pub fn T::list_elements_at(&self, index: usize) -> vortex_error::VortexResult +pub fn T::list_elements_at(&self, usize) -> vortex_error::VortexResult pub fn T::list_validity(&self) -> vortex_array::validity::Validity -pub fn T::list_validity_mask(&self) -> vortex_mask::Mask - pub fn T::nullability(&self) -> vortex_array::dtype::Nullability -pub fn T::offset_at(&self, index: usize) -> vortex_error::VortexResult +pub fn T::offset_at(&self, usize) -> vortex_error::VortexResult pub fn T::offsets(&self) -> &vortex_array::ArrayRef -pub fn T::reset_offsets(&self, recurse: bool) -> vortex_error::VortexResult> +pub fn T::reset_offsets(&self, bool) -> vortex_error::VortexResult> pub fn T::sliced_elements(&self) -> vortex_error::VortexResult @@ -2966,21 +3150,17 @@ pub vortex_array::arrays::listview::ListViewRebuildMode::TrimElements pub struct vortex_array::arrays::listview::ListView -impl vortex_array::arrays::ListView - -pub const vortex_array::arrays::ListView::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::ListView pub fn vortex_array::arrays::ListView::clone(&self) -> vortex_array::arrays::ListView impl core::fmt::Debug for vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::ListView::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::ListView::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::ListView @@ -2990,57 +3170,65 @@ pub type vortex_array::arrays::ListView::OperationsVTable = vortex_array::arrays pub type vortex_array::arrays::ListView::ValidityVTable = vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::ListView::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::ListView::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::ListView::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::ListView::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::ListView::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::ListView::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::ListView::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::ListView::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::ListView::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::ListView::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::ListView::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::ListView::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::ListView::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::ListView::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::ListView::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::ListView::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::ListView::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::ListView::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::ListView::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::ListView::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::ListView::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::ListView::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::ListView::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::ListView::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::ListView::validate(&self, _data: &vortex_array::arrays::listview::ListViewData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::ListView::validate(&self, &vortex_array::arrays::listview::ListViewData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::ListView>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::ListView::validity(vortex_array::ArrayView<'_, vortex_array::arrays::ListView>) -> vortex_error::VortexResult + +impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::ListView + +pub fn vortex_array::arrays::ListView::take(vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::dict::TakeReduce for vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, indices: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::take(vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::ListView + +pub fn vortex_array::arrays::ListView::cast(vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::cast(vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::mask(vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> pub struct vortex_array::arrays::listview::ListViewData @@ -3054,9 +3242,9 @@ pub unsafe fn vortex_array::arrays::listview::ListViewData::new_unchecked() -> S pub fn vortex_array::arrays::listview::ListViewData::try_new() -> vortex_error::VortexResult -pub fn vortex_array::arrays::listview::ListViewData::validate(elements: &vortex_array::ArrayRef, offsets: &vortex_array::ArrayRef, sizes: &vortex_array::ArrayRef, validity: &vortex_array::validity::Validity) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::listview::ListViewData::validate(&vortex_array::ArrayRef, &vortex_array::ArrayRef, &vortex_array::ArrayRef, &vortex_array::validity::Validity) -> vortex_error::VortexResult<()> -pub unsafe fn vortex_array::arrays::listview::ListViewData::with_zero_copy_to_list(self, is_zctl: bool) -> Self +pub unsafe fn vortex_array::arrays::listview::ListViewData::with_zero_copy_to_list(self, bool) -> Self impl core::clone::Clone for vortex_array::arrays::listview::ListViewData @@ -3068,19 +3256,19 @@ pub fn vortex_array::arrays::listview::ListViewData::default() -> Self impl core::fmt::Debug for vortex_array::arrays::listview::ListViewData -pub fn vortex_array::arrays::listview::ListViewData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::listview::ListViewData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::arrays::listview::ListViewData -pub fn vortex_array::arrays::listview::ListViewData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::listview::ListViewData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::ArrayEq for vortex_array::arrays::listview::ListViewData -pub fn vortex_array::arrays::listview::ListViewData::array_eq(&self, other: &Self, _precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::listview::ListViewData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayHash for vortex_array::arrays::listview::ListViewData -pub fn vortex_array::arrays::listview::ListViewData::array_hash(&self, state: &mut H, _precision: vortex_array::Precision) +pub fn vortex_array::arrays::listview::ListViewData::array_hash(&self, &mut H, vortex_array::Precision) pub struct vortex_array::arrays::listview::ListViewDataParts @@ -3098,19 +3286,17 @@ pub trait vortex_array::arrays::listview::ListViewArrayExt: vortex_array::TypedA pub fn vortex_array::arrays::listview::ListViewArrayExt::elements(&self) -> &vortex_array::ArrayRef -pub fn vortex_array::arrays::listview::ListViewArrayExt::list_elements_at(&self, index: usize) -> vortex_error::VortexResult +pub fn vortex_array::arrays::listview::ListViewArrayExt::list_elements_at(&self, usize) -> vortex_error::VortexResult pub fn vortex_array::arrays::listview::ListViewArrayExt::listview_validity(&self) -> vortex_array::validity::Validity -pub fn vortex_array::arrays::listview::ListViewArrayExt::listview_validity_mask(&self) -> vortex_mask::Mask - pub fn vortex_array::arrays::listview::ListViewArrayExt::nullability(&self) -> vortex_array::dtype::Nullability -pub fn vortex_array::arrays::listview::ListViewArrayExt::offset_at(&self, index: usize) -> usize +pub fn vortex_array::arrays::listview::ListViewArrayExt::offset_at(&self, usize) -> usize pub fn vortex_array::arrays::listview::ListViewArrayExt::offsets(&self) -> &vortex_array::ArrayRef -pub fn vortex_array::arrays::listview::ListViewArrayExt::size_at(&self, index: usize) -> usize +pub fn vortex_array::arrays::listview::ListViewArrayExt::size_at(&self, usize) -> usize pub fn vortex_array::arrays::listview::ListViewArrayExt::sizes(&self) -> &vortex_array::ArrayRef @@ -3120,29 +3306,27 @@ impl> vortex_arra pub fn T::elements(&self) -> &vortex_array::ArrayRef -pub fn T::list_elements_at(&self, index: usize) -> vortex_error::VortexResult +pub fn T::list_elements_at(&self, usize) -> vortex_error::VortexResult pub fn T::listview_validity(&self) -> vortex_array::validity::Validity -pub fn T::listview_validity_mask(&self) -> vortex_mask::Mask - pub fn T::nullability(&self) -> vortex_array::dtype::Nullability -pub fn T::offset_at(&self, index: usize) -> usize +pub fn T::offset_at(&self, usize) -> usize pub fn T::offsets(&self) -> &vortex_array::ArrayRef -pub fn T::size_at(&self, index: usize) -> usize +pub fn T::size_at(&self, usize) -> usize pub fn T::sizes(&self) -> &vortex_array::ArrayRef pub fn T::verify_is_zero_copy_to_list(&self) -> bool -pub fn vortex_array::arrays::listview::list_from_list_view(list_view: vortex_array::arrays::ListViewArray) -> vortex_error::VortexResult +pub fn vortex_array::arrays::listview::list_from_list_view(vortex_array::arrays::ListViewArray) -> vortex_error::VortexResult -pub fn vortex_array::arrays::listview::list_view_from_list(list: vortex_array::arrays::ListArray, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::listview::list_view_from_list(vortex_array::arrays::ListArray, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::listview::recursive_list_from_list_view(array: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::arrays::listview::recursive_list_from_list_view(vortex_array::ArrayRef) -> vortex_error::VortexResult pub type vortex_array::arrays::listview::ListViewArray = vortex_array::Array @@ -3150,21 +3334,17 @@ pub mod vortex_array::arrays::masked pub struct vortex_array::arrays::masked::Masked -impl vortex_array::arrays::Masked - -pub const vortex_array::arrays::Masked::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::Masked pub fn vortex_array::arrays::Masked::clone(&self) -> vortex_array::arrays::Masked impl core::fmt::Debug for vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::Masked::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Masked::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::Masked @@ -3174,57 +3354,57 @@ pub type vortex_array::arrays::Masked::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::Masked::ValidityVTable = vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Masked::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Masked::buffer(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Masked::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Masked::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Masked::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Masked::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Masked::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Masked::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Masked::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Masked::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Masked::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Masked::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Masked::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Masked::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Masked::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Masked::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Masked::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Masked::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Masked::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Masked::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Masked::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Masked::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Masked::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Masked::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Masked::validate(&self, _data: &vortex_array::arrays::masked::MaskedData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Masked::validate(&self, &vortex_array::arrays::masked::MaskedData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Masked::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Masked>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeReduce for vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, indices: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::take(vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterReduce for vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::filter(array: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::filter(vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::mask(vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> pub struct vortex_array::arrays::masked::MaskedData @@ -3234,41 +3414,65 @@ pub fn vortex_array::arrays::masked::MaskedData::clone(&self) -> vortex_array::a impl core::fmt::Debug for vortex_array::arrays::masked::MaskedData -pub fn vortex_array::arrays::masked::MaskedData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::masked::MaskedData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::arrays::masked::MaskedData -pub fn vortex_array::arrays::masked::MaskedData::fmt(&self, _f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::masked::MaskedData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::ArrayEq for vortex_array::arrays::masked::MaskedData -pub fn vortex_array::arrays::masked::MaskedData::array_eq(&self, _other: &Self, _precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::masked::MaskedData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayHash for vortex_array::arrays::masked::MaskedData -pub fn vortex_array::arrays::masked::MaskedData::array_hash(&self, _state: &mut H, _precision: vortex_array::Precision) +pub fn vortex_array::arrays::masked::MaskedData::array_hash(&self, &mut H, vortex_array::Precision) -pub trait vortex_array::arrays::masked::MaskedArrayExt: vortex_array::TypedArrayRef +pub struct vortex_array::arrays::masked::MaskedSlots -pub fn vortex_array::arrays::masked::MaskedArrayExt::child(&self) -> &vortex_array::ArrayRef +pub vortex_array::arrays::masked::MaskedSlots::child: vortex_array::ArrayRef -pub fn vortex_array::arrays::masked::MaskedArrayExt::masked_validity(&self) -> vortex_array::validity::Validity +pub vortex_array::arrays::masked::MaskedSlots::validity: core::option::Option -pub fn vortex_array::arrays::masked::MaskedArrayExt::masked_validity_mask(&self) -> vortex_mask::Mask +impl vortex_array::arrays::masked::MaskedSlots -pub fn vortex_array::arrays::masked::MaskedArrayExt::validity_child(&self) -> core::option::Option<&vortex_array::ArrayRef> +pub const vortex_array::arrays::masked::MaskedSlots::CHILD: usize -impl> vortex_array::arrays::masked::MaskedArrayExt for T +pub const vortex_array::arrays::masked::MaskedSlots::COUNT: usize -pub fn T::child(&self) -> &vortex_array::ArrayRef +pub const vortex_array::arrays::masked::MaskedSlots::NAMES: [&'static str; 2] + +pub const vortex_array::arrays::masked::MaskedSlots::VALIDITY: usize + +pub fn vortex_array::arrays::masked::MaskedSlots::from_slots(alloc::vec::Vec>) -> Self + +pub fn vortex_array::arrays::masked::MaskedSlots::into_slots(self) -> alloc::vec::Vec> + +pub trait vortex_array::arrays::masked::MaskedArrayExt: vortex_array::TypedArrayRef + vortex_array::arrays::masked::MaskedArraySlotsExt + +pub fn vortex_array::arrays::masked::MaskedArrayExt::masked_validity(&self) -> vortex_array::validity::Validity + +impl> vortex_array::arrays::masked::MaskedArrayExt for T pub fn T::masked_validity(&self) -> vortex_array::validity::Validity -pub fn T::masked_validity_mask(&self) -> vortex_mask::Mask +pub trait vortex_array::arrays::masked::MaskedArraySlotsExt: vortex_array::TypedArrayRef -pub fn T::validity_child(&self) -> core::option::Option<&vortex_array::ArrayRef> +pub fn vortex_array::arrays::masked::MaskedArraySlotsExt::child(&self) -> &vortex_array::ArrayRef + +pub fn vortex_array::arrays::masked::MaskedArraySlotsExt::slots_view(&self) -> vortex_array::arrays::masked::array::MaskedSlotsView<'_> + +pub fn vortex_array::arrays::masked::MaskedArraySlotsExt::validity(&self) -> core::option::Option<&vortex_array::ArrayRef> -pub fn vortex_array::arrays::masked::mask_validity_canonical(canonical: vortex_array::Canonical, validity_mask: &vortex_mask::Mask, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +impl> vortex_array::arrays::masked::MaskedArraySlotsExt for T + +pub fn T::child(&self) -> &vortex_array::ArrayRef + +pub fn T::slots_view(&self) -> vortex_array::arrays::masked::array::MaskedSlotsView<'_> + +pub fn T::validity(&self) -> core::option::Option<&vortex_array::ArrayRef> + +pub fn vortex_array::arrays::masked::mask_validity_canonical(vortex_array::Canonical, vortex_array::validity::Validity, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub type vortex_array::arrays::masked::MaskedArray = vortex_array::Array @@ -3278,10 +3482,6 @@ pub struct vortex_array::arrays::null::Null impl vortex_array::arrays::null::Null -pub const vortex_array::arrays::null::Null::ID: vortex_array::ArrayId - -impl vortex_array::arrays::null::Null - pub const vortex_array::arrays::null::Null::TAKE_RULES: vortex_array::optimizer::rules::ParentRuleSet impl core::clone::Clone for vortex_array::arrays::null::Null @@ -3290,11 +3490,11 @@ pub fn vortex_array::arrays::null::Null::clone(&self) -> vortex_array::arrays::n impl core::fmt::Debug for vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::null::Null::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::scalar_at(_array: vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, _index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::null::Null::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::null::Null @@ -3304,61 +3504,61 @@ pub type vortex_array::arrays::null::Null::OperationsVTable = vortex_array::arra pub type vortex_array::arrays::null::Null::ValidityVTable = vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::null::Null::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::null::Null::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::null::Null::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::null::Null::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::null::Null::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::null::Null::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::null::Null::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::null::Null::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::null::Null::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::null::Null::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::null::Null::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::null::Null::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::null::Null::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::null::Null::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::null::Null::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::null::Null::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::null::Null::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::null::Null::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::null::Null::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::null::Null::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::null::Null::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::null::Null::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::null::Null::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::null::Null::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::null::Null::validate(&self, _data: &vortex_array::EmptyArrayData, dtype: &vortex_array::dtype::DType, _len: usize, _slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::null::Null::validate(&self, &vortex_array::EmptyArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::validity(_array: vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::null::Null::validity(vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeReduce for vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, indices: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::take(vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterReduce for vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::filter(_array: vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::filter(vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::slice(_array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::cast(vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, _mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::mask(vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> pub type vortex_array::arrays::null::NullArray = vortex_array::Array @@ -3368,7 +3568,7 @@ pub struct vortex_array::arrays::patched::Patched impl vortex_array::arrays::patched::Patched -pub fn vortex_array::arrays::patched::Patched::from_array_and_patches(inner: vortex_array::ArrayRef, patches: &vortex_array::patches::Patches, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::from_array_and_patches(vortex_array::ArrayRef, &vortex_array::patches::Patches, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl core::clone::Clone for vortex_array::arrays::patched::Patched @@ -3376,11 +3576,11 @@ pub fn vortex_array::arrays::patched::Patched::clone(&self) -> vortex_array::arr impl core::fmt::Debug for vortex_array::arrays::patched::Patched -pub fn vortex_array::arrays::patched::Patched::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::patched::Patched::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::patched::Patched -pub fn vortex_array::arrays::patched::Patched::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::patched::Patched>, index: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::patched::Patched::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::patched::Patched>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::patched::Patched @@ -3390,57 +3590,57 @@ pub type vortex_array::arrays::patched::Patched::OperationsVTable = vortex_array pub type vortex_array::arrays::patched::Patched::ValidityVTable = vortex_array::ValidityVTableFromChild -pub fn vortex_array::arrays::patched::Patched::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::patched::Patched::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::patched::Patched::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::patched::Patched::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::patched::Patched::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::patched::Patched::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::patched::Patched::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::patched::Patched::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::patched::Patched::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::patched::Patched::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::patched::Patched::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::patched::Patched::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::patched::Patched::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::patched::Patched::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::patched::Patched::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::patched::Patched::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::patched::Patched::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::patched::Patched::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::patched::Patched::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::patched::Patched::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::patched::Patched::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::patched::Patched::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::patched::Patched::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::patched::Patched::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::patched::Patched::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::patched::Patched::validate(&self, data: &vortex_array::arrays::patched::PatchedData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::patched::Patched::validate(&self, &vortex_array::arrays::patched::PatchedData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityChild for vortex_array::arrays::patched::Patched -pub fn vortex_array::arrays::patched::Patched::validity_child(array: vortex_array::ArrayView<'_, vortex_array::arrays::patched::Patched>) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::patched::Patched::validity_child(vortex_array::ArrayView<'_, vortex_array::arrays::patched::Patched>) -> vortex_array::ArrayRef impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::patched::Patched -pub fn vortex_array::arrays::patched::Patched::take(array: vortex_array::ArrayView<'_, Self>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::take(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterReduce for vortex_array::arrays::patched::Patched -pub fn vortex_array::arrays::patched::Patched::filter(array: vortex_array::ArrayView<'_, Self>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::filter(vortex_array::ArrayView<'_, Self>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::patched::Patched -pub fn vortex_array::arrays::patched::Patched::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::binary::CompareKernel for vortex_array::arrays::patched::Patched -pub fn vortex_array::arrays::patched::Patched::compare(lhs: vortex_array::ArrayView<'_, Self>, rhs: &vortex_array::ArrayRef, operator: vortex_array::scalar_fn::fns::operators::CompareOperator, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::compare(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, vortex_array::scalar_fn::fns::operators::CompareOperator, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub struct vortex_array::arrays::patched::PatchedData @@ -3450,19 +3650,19 @@ pub fn vortex_array::arrays::patched::PatchedData::clone(&self) -> vortex_array: impl core::fmt::Debug for vortex_array::arrays::patched::PatchedData -pub fn vortex_array::arrays::patched::PatchedData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::patched::PatchedData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::arrays::patched::PatchedData -pub fn vortex_array::arrays::patched::PatchedData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::patched::PatchedData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::ArrayEq for vortex_array::arrays::patched::PatchedData -pub fn vortex_array::arrays::patched::PatchedData::array_eq(&self, other: &Self, _precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::patched::PatchedData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayHash for vortex_array::arrays::patched::PatchedData -pub fn vortex_array::arrays::patched::PatchedData::array_hash(&self, state: &mut H, _precision: vortex_array::Precision) +pub fn vortex_array::arrays::patched::PatchedData::array_hash(&self, &mut H, vortex_array::Precision) pub struct vortex_array::arrays::patched::PatchedMetadata @@ -3476,7 +3676,7 @@ pub fn vortex_array::arrays::patched::PatchedMetadata::default() -> Self impl core::fmt::Debug for vortex_array::arrays::patched::PatchedMetadata -pub fn vortex_array::arrays::patched::PatchedMetadata::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::patched::PatchedMetadata::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl prost::message::Message for vortex_array::arrays::patched::PatchedMetadata @@ -3508,7 +3708,7 @@ pub const vortex_array::arrays::patched::PatchedSlots::PATCH_INDICES: usize pub const vortex_array::arrays::patched::PatchedSlots::PATCH_VALUES: usize -pub fn vortex_array::arrays::patched::PatchedSlots::from_slots(slots: alloc::vec::Vec>) -> Self +pub fn vortex_array::arrays::patched::PatchedSlots::from_slots(alloc::vec::Vec>) -> Self pub fn vortex_array::arrays::patched::PatchedSlots::into_slots(self) -> alloc::vec::Vec> @@ -3524,7 +3724,7 @@ pub vortex_array::arrays::patched::PatchedSlotsView::patch_values: &'a vortex_ar impl<'a> vortex_array::arrays::patched::PatchedSlotsView<'a> -pub fn vortex_array::arrays::patched::PatchedSlotsView<'a>::from_slots(slots: &'a [core::option::Option]) -> Self +pub fn vortex_array::arrays::patched::PatchedSlotsView<'a>::from_slots(&'a [core::option::Option]) -> Self pub fn vortex_array::arrays::patched::PatchedSlotsView<'a>::to_owned(&self) -> vortex_array::arrays::patched::PatchedSlots @@ -3534,29 +3734,29 @@ pub fn vortex_array::arrays::patched::PatchedSlotsView<'a>::clone(&self) -> vort impl<'a> core::fmt::Debug for vortex_array::arrays::patched::PatchedSlotsView<'a> -pub fn vortex_array::arrays::patched::PatchedSlotsView<'a>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::patched::PatchedSlotsView<'a>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl<'a> core::marker::Copy for vortex_array::arrays::patched::PatchedSlotsView<'a> pub trait vortex_array::arrays::patched::PatchedArrayExt: vortex_array::arrays::patched::PatchedArraySlotsExt -pub fn vortex_array::arrays::patched::PatchedArrayExt::lane_range(&self, chunk: usize, lane: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::PatchedArrayExt::lane_range(&self, usize, usize) -> vortex_error::VortexResult> pub fn vortex_array::arrays::patched::PatchedArrayExt::n_lanes(&self) -> usize pub fn vortex_array::arrays::patched::PatchedArrayExt::offset(&self) -> usize -pub fn vortex_array::arrays::patched::PatchedArrayExt::slice_chunks(&self, chunks: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::PatchedArrayExt::slice_chunks(&self, core::ops::range::Range) -> vortex_error::VortexResult> impl> vortex_array::arrays::patched::PatchedArrayExt for T -pub fn T::lane_range(&self, chunk: usize, lane: usize) -> vortex_error::VortexResult> +pub fn T::lane_range(&self, usize, usize) -> vortex_error::VortexResult> pub fn T::n_lanes(&self) -> usize pub fn T::offset(&self) -> usize -pub fn T::slice_chunks(&self, chunks: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn T::slice_chunks(&self, core::ops::range::Range) -> vortex_error::VortexResult> pub trait vortex_array::arrays::patched::PatchedArraySlotsExt: vortex_array::TypedArrayRef @@ -3582,6 +3782,8 @@ pub fn T::patch_values(&self) -> &vortex_array::ArrayRef pub fn T::slots_view(&self) -> vortex_array::arrays::patched::PatchedSlotsView<'_> +pub fn vortex_array::arrays::patched::use_experimental_patches() -> bool + pub type vortex_array::arrays::patched::PatchedArray = vortex_array::Array pub mod vortex_array::arrays::primitive @@ -3590,47 +3792,47 @@ pub mod vortex_array::arrays::primitive impl core::hash::Hash for vortex_array::arrays::primitive::NativeValue -pub fn vortex_array::arrays::primitive::NativeValue::hash(&self, state: &mut H) +pub fn vortex_array::arrays::primitive::NativeValue::hash(&self, &mut H) impl core::hash::Hash for vortex_array::arrays::primitive::NativeValue -pub fn vortex_array::arrays::primitive::NativeValue::hash(&self, state: &mut H) +pub fn vortex_array::arrays::primitive::NativeValue::hash(&self, &mut H) impl core::hash::Hash for vortex_array::arrays::primitive::NativeValue -pub fn vortex_array::arrays::primitive::NativeValue::hash(&self, state: &mut H) +pub fn vortex_array::arrays::primitive::NativeValue::hash(&self, &mut H) impl core::hash::Hash for vortex_array::arrays::primitive::NativeValue -pub fn vortex_array::arrays::primitive::NativeValue::hash(&self, state: &mut H) +pub fn vortex_array::arrays::primitive::NativeValue::hash(&self, &mut H) impl core::hash::Hash for vortex_array::arrays::primitive::NativeValue -pub fn vortex_array::arrays::primitive::NativeValue::hash(&self, state: &mut H) +pub fn vortex_array::arrays::primitive::NativeValue::hash(&self, &mut H) impl core::hash::Hash for vortex_array::arrays::primitive::NativeValue -pub fn vortex_array::arrays::primitive::NativeValue::hash(&self, state: &mut H) +pub fn vortex_array::arrays::primitive::NativeValue::hash(&self, &mut H) impl core::hash::Hash for vortex_array::arrays::primitive::NativeValue -pub fn vortex_array::arrays::primitive::NativeValue::hash(&self, state: &mut H) +pub fn vortex_array::arrays::primitive::NativeValue::hash(&self, &mut H) impl core::hash::Hash for vortex_array::arrays::primitive::NativeValue -pub fn vortex_array::arrays::primitive::NativeValue::hash(&self, state: &mut H) +pub fn vortex_array::arrays::primitive::NativeValue::hash(&self, &mut H) impl core::hash::Hash for vortex_array::arrays::primitive::NativeValue -pub fn vortex_array::arrays::primitive::NativeValue::hash(&self, state: &mut H) +pub fn vortex_array::arrays::primitive::NativeValue::hash(&self, &mut H) impl core::hash::Hash for vortex_array::arrays::primitive::NativeValue -pub fn vortex_array::arrays::primitive::NativeValue::hash(&self, state: &mut H) +pub fn vortex_array::arrays::primitive::NativeValue::hash(&self, &mut H) impl core::hash::Hash for vortex_array::arrays::primitive::NativeValue -pub fn vortex_array::arrays::primitive::NativeValue::hash(&self, state: &mut H) +pub fn vortex_array::arrays::primitive::NativeValue::hash(&self, &mut H) impl core::clone::Clone for vortex_array::arrays::primitive::NativeValue @@ -3638,7 +3840,7 @@ pub fn vortex_array::arrays::primitive::NativeValue::clone(&self) -> vortex_a impl core::fmt::Debug for vortex_array::arrays::primitive::NativeValue -pub fn vortex_array::arrays::primitive::NativeValue::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::primitive::NativeValue::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_array::arrays::primitive::NativeValue @@ -3646,29 +3848,25 @@ impl core::cmp::Eq for vortex_array::arrays impl core::cmp::PartialEq for vortex_array::arrays::primitive::NativeValue -pub fn vortex_array::arrays::primitive::NativeValue::eq(&self, other: &vortex_array::arrays::primitive::NativeValue) -> bool +pub fn vortex_array::arrays::primitive::NativeValue::eq(&self, &vortex_array::arrays::primitive::NativeValue) -> bool impl core::cmp::PartialOrd for vortex_array::arrays::primitive::NativeValue -pub fn vortex_array::arrays::primitive::NativeValue::partial_cmp(&self, other: &vortex_array::arrays::primitive::NativeValue) -> core::option::Option +pub fn vortex_array::arrays::primitive::NativeValue::partial_cmp(&self, &vortex_array::arrays::primitive::NativeValue) -> core::option::Option pub struct vortex_array::arrays::primitive::Primitive -impl vortex_array::arrays::Primitive - -pub const vortex_array::arrays::Primitive::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::Primitive pub fn vortex_array::arrays::Primitive::clone(&self) -> vortex_array::arrays::Primitive impl core::fmt::Debug for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::Primitive::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Primitive::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::Primitive @@ -3678,71 +3876,75 @@ pub type vortex_array::arrays::Primitive::OperationsVTable = vortex_array::array pub type vortex_array::arrays::Primitive::ValidityVTable = vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Primitive::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Primitive::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Primitive::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Primitive::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Primitive::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Primitive::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Primitive::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Primitive::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Primitive::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Primitive::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Primitive::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Primitive::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Primitive::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Primitive::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Primitive::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Primitive::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Primitive::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Primitive::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Primitive::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Primitive::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Primitive::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Primitive::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Primitive::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Primitive::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Primitive::validate(&self, data: &vortex_array::arrays::primitive::PrimitiveData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Primitive::validate(&self, &vortex_array::arrays::primitive::PrimitiveData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Primitive::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::take(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::arrays::primitive::PrimitiveMaskedValidityRule pub type vortex_array::arrays::primitive::PrimitiveMaskedValidityRule::Parent = vortex_array::arrays::Masked -pub fn vortex_array::arrays::primitive::PrimitiveMaskedValidityRule::reduce_parent(&self, array: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, parent: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, _child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::primitive::PrimitiveMaskedValidityRule::reduce_parent(&self, vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, usize) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::between::BetweenKernel for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::between(arr: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, lower: &vortex_array::ArrayRef, upper: &vortex_array::ArrayRef, options: &vortex_array::scalar_fn::fns::between::BetweenOptions, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::between(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, &vortex_array::ArrayRef, &vortex_array::ArrayRef, &vortex_array::scalar_fn::fns::between::BetweenOptions, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, dtype: &vortex_array::dtype::DType, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::Primitive + +pub fn vortex_array::arrays::Primitive::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::fill_null::FillNullKernel for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::fill_null(array: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, fill_value: &vortex_array::scalar::Scalar, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::fill_null(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, &vortex_array::scalar::Scalar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::mask(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> pub struct vortex_array::arrays::primitive::PrimitiveData @@ -3754,11 +3956,11 @@ impl vortex_array::arrays::primitive::PrimitiveData pub fn vortex_array::arrays::primitive::PrimitiveData::buffer_handle(&self) -> &vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::primitive::PrimitiveData::from_buffer_handle(handle: vortex_array::buffer::BufferHandle, ptype: vortex_array::dtype::PType, _validity: vortex_array::validity::Validity) -> Self +pub fn vortex_array::arrays::primitive::PrimitiveData::from_buffer_handle(vortex_array::buffer::BufferHandle, vortex_array::dtype::PType, vortex_array::validity::Validity) -> Self -pub fn vortex_array::arrays::primitive::PrimitiveData::from_byte_buffer(buffer: vortex_buffer::ByteBuffer, ptype: vortex_array::dtype::PType, validity: vortex_array::validity::Validity) -> Self +pub fn vortex_array::arrays::primitive::PrimitiveData::from_byte_buffer(vortex_buffer::ByteBuffer, vortex_array::dtype::PType, vortex_array::validity::Validity) -> Self -pub fn vortex_array::arrays::primitive::PrimitiveData::from_values_byte_buffer(valid_elems_buffer: vortex_buffer::ByteBuffer, ptype: vortex_array::dtype::PType, validity: vortex_array::validity::Validity, n_rows: usize) -> Self +pub fn vortex_array::arrays::primitive::PrimitiveData::from_values_byte_buffer(vortex_buffer::ByteBuffer, vortex_array::dtype::PType, vortex_array::validity::Validity, usize) -> Self pub fn vortex_array::arrays::primitive::PrimitiveData::into_buffer(self) -> vortex_buffer::buffer::Buffer @@ -3776,17 +3978,17 @@ pub fn vortex_array::arrays::primitive::PrimitiveData::try_into_buffer_mut(nullability: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::arrays::primitive::PrimitiveData::empty(vortex_array::dtype::Nullability) -> Self -pub fn vortex_array::arrays::primitive::PrimitiveData::new(buffer: impl core::convert::Into>, validity: vortex_array::validity::Validity) -> Self +pub fn vortex_array::arrays::primitive::PrimitiveData::new(impl core::convert::Into>, vortex_array::validity::Validity) -> Self -pub unsafe fn vortex_array::arrays::primitive::PrimitiveData::new_unchecked(buffer: vortex_buffer::buffer::Buffer, _validity: vortex_array::validity::Validity) -> Self +pub unsafe fn vortex_array::arrays::primitive::PrimitiveData::new_unchecked(vortex_buffer::buffer::Buffer, vortex_array::validity::Validity) -> Self -pub unsafe fn vortex_array::arrays::primitive::PrimitiveData::new_unchecked_from_handle(handle: vortex_array::buffer::BufferHandle, ptype: vortex_array::dtype::PType, _validity: vortex_array::validity::Validity) -> Self +pub unsafe fn vortex_array::arrays::primitive::PrimitiveData::new_unchecked_from_handle(vortex_array::buffer::BufferHandle, vortex_array::dtype::PType, vortex_array::validity::Validity) -> Self -pub fn vortex_array::arrays::primitive::PrimitiveData::try_new(buffer: vortex_buffer::buffer::Buffer, validity: vortex_array::validity::Validity) -> vortex_error::VortexResult +pub fn vortex_array::arrays::primitive::PrimitiveData::try_new(vortex_buffer::buffer::Buffer, vortex_array::validity::Validity) -> vortex_error::VortexResult -pub fn vortex_array::arrays::primitive::PrimitiveData::validate(buffer: &vortex_buffer::buffer::Buffer, validity: &vortex_array::validity::Validity) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::primitive::PrimitiveData::validate(&vortex_buffer::buffer::Buffer, &vortex_array::validity::Validity) -> vortex_error::VortexResult<()> impl core::clone::Clone for vortex_array::arrays::primitive::PrimitiveData @@ -3794,19 +3996,19 @@ pub fn vortex_array::arrays::primitive::PrimitiveData::clone(&self) -> vortex_ar impl core::fmt::Debug for vortex_array::arrays::primitive::PrimitiveData -pub fn vortex_array::arrays::primitive::PrimitiveData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::primitive::PrimitiveData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::arrays::primitive::PrimitiveData -pub fn vortex_array::arrays::primitive::PrimitiveData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::primitive::PrimitiveData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::ArrayEq for vortex_array::arrays::primitive::PrimitiveData -pub fn vortex_array::arrays::primitive::PrimitiveData::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::primitive::PrimitiveData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayHash for vortex_array::arrays::primitive::PrimitiveData -pub fn vortex_array::arrays::primitive::PrimitiveData::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn vortex_array::arrays::primitive::PrimitiveData::array_hash(&self, &mut H, vortex_array::Precision) pub struct vortex_array::arrays::primitive::PrimitiveDataParts @@ -3824,13 +4026,13 @@ pub fn vortex_array::arrays::primitive::PrimitiveMaskedValidityRule::default() - impl core::fmt::Debug for vortex_array::arrays::primitive::PrimitiveMaskedValidityRule -pub fn vortex_array::arrays::primitive::PrimitiveMaskedValidityRule::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::primitive::PrimitiveMaskedValidityRule::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::arrays::primitive::PrimitiveMaskedValidityRule pub type vortex_array::arrays::primitive::PrimitiveMaskedValidityRule::Parent = vortex_array::arrays::Masked -pub fn vortex_array::arrays::primitive::PrimitiveMaskedValidityRule::reduce_parent(&self, array: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, parent: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, _child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::primitive::PrimitiveMaskedValidityRule::reduce_parent(&self, vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, usize) -> vortex_error::VortexResult> pub trait vortex_array::arrays::primitive::PrimitiveArrayExt: vortex_array::TypedArrayRef @@ -3842,14 +4044,12 @@ pub fn vortex_array::arrays::primitive::PrimitiveArrayExt::nullability(&self) -> pub fn vortex_array::arrays::primitive::PrimitiveArrayExt::ptype(&self) -> vortex_array::dtype::PType -pub fn vortex_array::arrays::primitive::PrimitiveArrayExt::reinterpret_cast(&self, ptype: vortex_array::dtype::PType) -> vortex_array::arrays::PrimitiveArray +pub fn vortex_array::arrays::primitive::PrimitiveArrayExt::reinterpret_cast(&self, vortex_array::dtype::PType) -> vortex_array::arrays::PrimitiveArray pub fn vortex_array::arrays::primitive::PrimitiveArrayExt::validity(&self) -> vortex_array::validity::Validity pub fn vortex_array::arrays::primitive::PrimitiveArrayExt::validity_child(&self) -> core::option::Option<&vortex_array::ArrayRef> -pub fn vortex_array::arrays::primitive::PrimitiveArrayExt::validity_mask(&self) -> vortex_mask::Mask - impl> vortex_array::arrays::primitive::PrimitiveArrayExt for T pub fn T::buffer_handle(&self) -> &vortex_array::buffer::BufferHandle @@ -3860,161 +4060,161 @@ pub fn T::nullability(&self) -> vortex_array::dtype::Nullability pub fn T::ptype(&self) -> vortex_array::dtype::PType -pub fn T::reinterpret_cast(&self, ptype: vortex_array::dtype::PType) -> vortex_array::arrays::PrimitiveArray +pub fn T::reinterpret_cast(&self, vortex_array::dtype::PType) -> vortex_array::arrays::PrimitiveArray pub fn T::validity(&self) -> vortex_array::validity::Validity pub fn T::validity_child(&self) -> core::option::Option<&vortex_array::ArrayRef> -pub fn T::validity_mask(&self) -> vortex_mask::Mask +pub fn vortex_array::arrays::primitive::chunk_range(usize, usize, usize) -> core::ops::range::Range -pub fn vortex_array::arrays::primitive::chunk_range(chunk_idx: usize, offset: usize, array_len: usize) -> core::ops::range::Range - -pub fn vortex_array::arrays::primitive::patch_chunk(decoded_values: &mut [T], patches_indices: &[I], patches_values: &[T], patches_offset: usize, chunk_offsets_slice: &[C], chunk_idx: usize, offset_within_chunk: usize) where T: vortex_array::dtype::NativePType, I: vortex_array::dtype::UnsignedPType, C: vortex_array::dtype::UnsignedPType +pub fn vortex_array::arrays::primitive::patch_chunk(&mut [T], &[I], &[T], usize, &[C], usize, usize) where T: vortex_array::dtype::NativePType, I: vortex_array::dtype::UnsignedPType, C: vortex_array::dtype::UnsignedPType pub type vortex_array::arrays::primitive::PrimitiveArray = vortex_array::Array pub mod vortex_array::arrays::scalar_fn -pub struct vortex_array::arrays::scalar_fn::AnyScalarFn +pub mod vortex_array::arrays::scalar_fn::plugin -impl core::fmt::Debug for vortex_array::arrays::scalar_fn::AnyScalarFn +pub struct vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayParts -pub fn vortex_array::arrays::scalar_fn::AnyScalarFn::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayParts::children: alloc::vec::Vec -impl vortex_array::matcher::Matcher for vortex_array::arrays::scalar_fn::AnyScalarFn +pub vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayParts::options: ::Options -pub type vortex_array::arrays::scalar_fn::AnyScalarFn::Match<'a> = vortex_array::ArrayView<'a, vortex_array::arrays::scalar_fn::ScalarFnVTable> +pub struct vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin(_) -pub fn vortex_array::arrays::scalar_fn::AnyScalarFn::matches(array: &vortex_array::ArrayRef) -> bool +impl vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin -pub fn vortex_array::arrays::scalar_fn::AnyScalarFn::try_match(array: &vortex_array::ArrayRef) -> core::option::Option +pub fn vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin::new(V) -> Self -pub struct vortex_array::arrays::scalar_fn::ExactScalarFn(_) +impl vortex_array::ArrayPlugin for vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin -impl core::default::Default for vortex_array::arrays::scalar_fn::ExactScalarFn +pub fn vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::arrays::scalar_fn::ExactScalarFn::default() -> vortex_array::arrays::scalar_fn::ExactScalarFn +pub fn vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin::id(&self) -> vortex_array::ArrayId -impl core::fmt::Debug for vortex_array::arrays::scalar_fn::ExactScalarFn +pub fn vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin::is_supported_encoding(&self, &vortex_array::ArrayId) -> bool -pub fn vortex_array::arrays::scalar_fn::ExactScalarFn::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin::serialize(&self, &vortex_array::ArrayRef, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -impl vortex_array::matcher::Matcher for vortex_array::arrays::scalar_fn::ExactScalarFn +pub trait vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayVTable: vortex_array::scalar_fn::ScalarFnVTable -pub type vortex_array::arrays::scalar_fn::ExactScalarFn::Match<'a> = vortex_array::arrays::scalar_fn::ScalarFnArrayView<'a, F> +pub fn vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayVTable::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> + +pub fn vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayVTable::serialize(&self, &vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::scalar_fn::ExactScalarFn::matches(array: &vortex_array::ArrayRef) -> bool +pub struct vortex_array::arrays::scalar_fn::AnyScalarFn -pub fn vortex_array::arrays::scalar_fn::ExactScalarFn::try_match(array: &vortex_array::ArrayRef) -> core::option::Option +impl core::fmt::Debug for vortex_array::arrays::scalar_fn::AnyScalarFn -pub struct vortex_array::arrays::scalar_fn::ScalarFnArrayView<'a, F: vortex_array::scalar_fn::ScalarFnVTable> +pub fn vortex_array::arrays::scalar_fn::AnyScalarFn::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result -pub vortex_array::arrays::scalar_fn::ScalarFnArrayView::options: &'a ::Options +impl vortex_array::matcher::Matcher for vortex_array::arrays::scalar_fn::AnyScalarFn -pub vortex_array::arrays::scalar_fn::ScalarFnArrayView::vtable: &'a F +pub type vortex_array::arrays::scalar_fn::AnyScalarFn::Match<'a> = vortex_array::ArrayView<'a, vortex_array::arrays::scalar_fn::ScalarFn> -impl core::ops::deref::Deref for vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, F> +pub fn vortex_array::arrays::scalar_fn::AnyScalarFn::matches(&vortex_array::ArrayRef) -> bool -pub type vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, F>::Target = vortex_array::ArrayRef +pub fn vortex_array::arrays::scalar_fn::AnyScalarFn::try_match(&vortex_array::ArrayRef) -> core::option::Option -pub fn vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, F>::deref(&self) -> &Self::Target +pub struct vortex_array::arrays::scalar_fn::ExactScalarFn(_) -pub struct vortex_array::arrays::scalar_fn::ScalarFnData +impl core::default::Default for vortex_array::arrays::scalar_fn::ExactScalarFn -impl vortex_array::arrays::scalar_fn::ScalarFnData +pub fn vortex_array::arrays::scalar_fn::ExactScalarFn::default() -> vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::arrays::scalar_fn::ScalarFnData::build(scalar_fn: vortex_array::scalar_fn::ScalarFnRef, children: alloc::vec::Vec, len: usize) -> vortex_error::VortexResult +impl core::fmt::Debug for vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::arrays::scalar_fn::ScalarFnData::scalar_fn(&self) -> &vortex_array::scalar_fn::ScalarFnRef +pub fn vortex_array::arrays::scalar_fn::ExactScalarFn::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result -impl core::clone::Clone for vortex_array::arrays::scalar_fn::ScalarFnData +impl vortex_array::matcher::Matcher for vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::arrays::scalar_fn::ScalarFnData::clone(&self) -> vortex_array::arrays::scalar_fn::ScalarFnData +pub type vortex_array::arrays::scalar_fn::ExactScalarFn::Match<'a> = vortex_array::arrays::scalar_fn::ScalarFnArrayView<'a, F> -impl core::fmt::Debug for vortex_array::arrays::scalar_fn::ScalarFnData +pub fn vortex_array::arrays::scalar_fn::ExactScalarFn::matches(&vortex_array::ArrayRef) -> bool -pub fn vortex_array::arrays::scalar_fn::ScalarFnData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::scalar_fn::ExactScalarFn::try_match(&vortex_array::ArrayRef) -> core::option::Option -impl core::fmt::Display for vortex_array::arrays::scalar_fn::ScalarFnData +pub struct vortex_array::arrays::scalar_fn::ScalarFn -pub fn vortex_array::arrays::scalar_fn::ScalarFnData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +impl core::clone::Clone for vortex_array::arrays::scalar_fn::ScalarFn -impl vortex_array::ArrayEq for vortex_array::arrays::scalar_fn::ScalarFnData +pub fn vortex_array::arrays::scalar_fn::ScalarFn::clone(&self) -> vortex_array::arrays::scalar_fn::ScalarFn -pub fn vortex_array::arrays::scalar_fn::ScalarFnData::array_eq(&self, other: &Self, _precision: vortex_array::Precision) -> bool +impl core::fmt::Debug for vortex_array::arrays::scalar_fn::ScalarFn -impl vortex_array::ArrayHash for vortex_array::arrays::scalar_fn::ScalarFnData +pub fn vortex_array::arrays::scalar_fn::ScalarFn::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result -pub fn vortex_array::arrays::scalar_fn::ScalarFnData::array_hash(&self, state: &mut H, _precision: vortex_array::Precision) +impl vortex_array::OperationsVTable for vortex_array::arrays::scalar_fn::ScalarFn -pub struct vortex_array::arrays::scalar_fn::ScalarFnVTable +pub fn vortex_array::arrays::scalar_fn::ScalarFn::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::scalar_fn::ScalarFn>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -impl core::clone::Clone for vortex_array::arrays::scalar_fn::ScalarFnVTable +impl vortex_array::VTable for vortex_array::arrays::scalar_fn::ScalarFn -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::clone(&self) -> vortex_array::arrays::scalar_fn::ScalarFnVTable +pub type vortex_array::arrays::scalar_fn::ScalarFn::ArrayData = vortex_array::arrays::scalar_fn::array::ScalarFnData -impl core::fmt::Debug for vortex_array::arrays::scalar_fn::ScalarFnVTable +pub type vortex_array::arrays::scalar_fn::ScalarFn::OperationsVTable = vortex_array::arrays::scalar_fn::ScalarFn -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub type vortex_array::arrays::scalar_fn::ScalarFn::ValidityVTable = vortex_array::arrays::scalar_fn::ScalarFn -impl vortex_array::OperationsVTable for vortex_array::arrays::scalar_fn::ScalarFnVTable +pub fn vortex_array::arrays::scalar_fn::ScalarFn::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::scalar_fn::ScalarFnVTable>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::scalar_fn::ScalarFn::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -impl vortex_array::VTable for vortex_array::arrays::scalar_fn::ScalarFnVTable +pub fn vortex_array::arrays::scalar_fn::ScalarFn::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub type vortex_array::arrays::scalar_fn::ScalarFnVTable::ArrayData = vortex_array::arrays::scalar_fn::ScalarFnData +pub fn vortex_array::arrays::scalar_fn::ScalarFn::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub type vortex_array::arrays::scalar_fn::ScalarFnVTable::OperationsVTable = vortex_array::arrays::scalar_fn::ScalarFnVTable +pub fn vortex_array::arrays::scalar_fn::ScalarFn::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub type vortex_array::arrays::scalar_fn::ScalarFnVTable::ValidityVTable = vortex_array::arrays::scalar_fn::ScalarFnVTable +pub fn vortex_array::arrays::scalar_fn::ScalarFn::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::scalar_fn::ScalarFn::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::scalar_fn::ScalarFn::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::scalar_fn::ScalarFn::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::scalar_fn::ScalarFn::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::deserialize(&self, _dtype: &vortex_array::dtype::DType, _len: usize, _metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::scalar_fn::ScalarFn::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::id(&self) -> vortex_array::ArrayId +pub fn vortex_array::arrays::scalar_fn::ScalarFn::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::scalar_fn::ScalarFn::validate(&self, &vortex_array::arrays::scalar_fn::array::ScalarFnData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +impl vortex_array::ValidityVTable for vortex_array::arrays::scalar_fn::ScalarFn -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::validity(vortex_array::ArrayView<'_, vortex_array::arrays::scalar_fn::ScalarFn>) -> vortex_error::VortexResult -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub struct vortex_array::arrays::scalar_fn::ScalarFnArrayView<'a, F: vortex_array::scalar_fn::ScalarFnVTable> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub vortex_array::arrays::scalar_fn::ScalarFnArrayView::options: &'a ::Options -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::slot_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub vortex_array::arrays::scalar_fn::ScalarFnArrayView::vtable: &'a F -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::validate(&self, data: &vortex_array::arrays::scalar_fn::ScalarFnData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +impl core::ops::deref::Deref for vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, F> -impl vortex_array::ValidityVTable for vortex_array::arrays::scalar_fn::ScalarFnVTable +pub type vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, F>::Target = vortex_array::ArrayRef -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::scalar_fn::ScalarFnVTable>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, F>::deref(&self) -> &Self::Target -pub trait vortex_array::arrays::scalar_fn::ScalarFnArrayExt: vortex_array::TypedArrayRef +pub trait vortex_array::arrays::scalar_fn::ScalarFnArrayExt: vortex_array::TypedArrayRef -pub fn vortex_array::arrays::scalar_fn::ScalarFnArrayExt::child_at(&self, idx: usize) -> &vortex_array::ArrayRef +pub fn vortex_array::arrays::scalar_fn::ScalarFnArrayExt::child_at(&self, usize) -> &vortex_array::ArrayRef pub fn vortex_array::arrays::scalar_fn::ScalarFnArrayExt::child_count(&self) -> usize pub fn vortex_array::arrays::scalar_fn::ScalarFnArrayExt::children(&self) -> alloc::vec::Vec -pub fn vortex_array::arrays::scalar_fn::ScalarFnArrayExt::get_child(&self, idx: usize) -> &vortex_array::ArrayRef +pub fn vortex_array::arrays::scalar_fn::ScalarFnArrayExt::get_child(&self, usize) -> &vortex_array::ArrayRef pub fn vortex_array::arrays::scalar_fn::ScalarFnArrayExt::iter_children(&self) -> impl core::iter::traits::iterator::Iterator + '_ @@ -4022,15 +4222,15 @@ pub fn vortex_array::arrays::scalar_fn::ScalarFnArrayExt::nchildren(&self) -> us pub fn vortex_array::arrays::scalar_fn::ScalarFnArrayExt::scalar_fn(&self) -> &vortex_array::scalar_fn::ScalarFnRef -impl> vortex_array::arrays::scalar_fn::ScalarFnArrayExt for T +impl> vortex_array::arrays::scalar_fn::ScalarFnArrayExt for T -pub fn T::child_at(&self, idx: usize) -> &vortex_array::ArrayRef +pub fn T::child_at(&self, usize) -> &vortex_array::ArrayRef pub fn T::child_count(&self) -> usize pub fn T::children(&self) -> alloc::vec::Vec -pub fn T::get_child(&self, idx: usize) -> &vortex_array::ArrayRef +pub fn T::get_child(&self, usize) -> &vortex_array::ArrayRef pub fn T::iter_children(&self) -> impl core::iter::traits::iterator::Iterator + '_ @@ -4042,39 +4242,35 @@ pub trait vortex_array::arrays::scalar_fn::ScalarFnFactoryExt: vortex_array::sca pub trait vortex_array::arrays::scalar_fn::ScalarFnFactoryExt: vortex_array::scalar_fn::ScalarFnVTable -pub fn vortex_array::arrays::scalar_fn::ScalarFnFactoryExt::try_new_array(&self, len: usize, options: Self::Options, children: impl core::convert::Into>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::scalar_fn::ScalarFnFactoryExt::try_new_array(&self, usize, Self::Options, impl core::convert::Into>) -> vortex_error::VortexResult -pub fn vortex_array::arrays::scalar_fn::ScalarFnFactoryExt::try_new_array(&self, len: usize, options: Self::Options, children: impl core::convert::Into>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::scalar_fn::ScalarFnFactoryExt::try_new_array(&self, usize, Self::Options, impl core::convert::Into>) -> vortex_error::VortexResult impl vortex_array::arrays::scalar_fn::ScalarFnFactoryExt for V impl vortex_array::arrays::scalar_fn::ScalarFnFactoryExt for V -pub fn V::try_new_array(&self, len: usize, options: Self::Options, children: impl core::convert::Into>) -> vortex_error::VortexResult +pub fn V::try_new_array(&self, usize, Self::Options, impl core::convert::Into>) -> vortex_error::VortexResult -pub fn V::try_new_array(&self, len: usize, options: Self::Options, children: impl core::convert::Into>) -> vortex_error::VortexResult +pub fn V::try_new_array(&self, usize, Self::Options, impl core::convert::Into>) -> vortex_error::VortexResult -pub type vortex_array::arrays::scalar_fn::ScalarFnArray = vortex_array::Array +pub type vortex_array::arrays::scalar_fn::ScalarFnArray = vortex_array::Array pub mod vortex_array::arrays::shared pub struct vortex_array::arrays::shared::Shared -impl vortex_array::arrays::Shared - -pub const vortex_array::arrays::Shared::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::Shared pub fn vortex_array::arrays::Shared::clone(&self) -> vortex_array::arrays::Shared impl core::fmt::Debug for vortex_array::arrays::Shared -pub fn vortex_array::arrays::Shared::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::Shared::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::Shared -pub fn vortex_array::arrays::Shared::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Shared>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Shared::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Shared>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::Shared @@ -4084,41 +4280,41 @@ pub type vortex_array::arrays::Shared::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::Shared::ValidityVTable = vortex_array::arrays::Shared -pub fn vortex_array::arrays::Shared::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Shared::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Shared::buffer(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Shared::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Shared::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Shared::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Shared::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Shared::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Shared::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Shared::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Shared::deserialize(&self, _dtype: &vortex_array::dtype::DType, _len: usize, _metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Shared::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Shared::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Shared::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Shared::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Shared::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Shared::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Shared::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Shared::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Shared::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Shared::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Shared::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Shared::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Shared::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Shared::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Shared::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Shared::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Shared::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Shared::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Shared::validate(&self, _data: &vortex_array::arrays::shared::SharedData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Shared::validate(&self, &vortex_array::arrays::shared::SharedData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::Shared -pub fn vortex_array::arrays::Shared::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Shared>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Shared::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Shared>) -> vortex_error::VortexResult pub struct vortex_array::arrays::shared::SharedData @@ -4136,27 +4332,27 @@ pub fn vortex_array::arrays::shared::SharedData::default() -> Self impl core::fmt::Debug for vortex_array::arrays::shared::SharedData -pub fn vortex_array::arrays::shared::SharedData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::shared::SharedData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::arrays::shared::SharedData -pub fn vortex_array::arrays::shared::SharedData::fmt(&self, _f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::shared::SharedData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::ArrayEq for vortex_array::arrays::shared::SharedData -pub fn vortex_array::arrays::shared::SharedData::array_eq(&self, _other: &Self, _precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::shared::SharedData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayHash for vortex_array::arrays::shared::SharedData -pub fn vortex_array::arrays::shared::SharedData::array_hash(&self, _state: &mut H, _precision: vortex_array::Precision) +pub fn vortex_array::arrays::shared::SharedData::array_hash(&self, &mut H, vortex_array::Precision) pub trait vortex_array::arrays::shared::SharedArrayExt: vortex_array::TypedArrayRef pub fn vortex_array::arrays::shared::SharedArrayExt::current_array_ref(&self) -> &vortex_array::ArrayRef -pub fn vortex_array::arrays::shared::SharedArrayExt::get_or_compute(&self, f: impl core::ops::function::FnOnce(&vortex_array::ArrayRef) -> vortex_error::VortexResult) -> vortex_error::VortexResult +pub fn vortex_array::arrays::shared::SharedArrayExt::get_or_compute(&self, impl core::ops::function::FnOnce(&vortex_array::ArrayRef) -> vortex_error::VortexResult) -> vortex_error::VortexResult -pub async fn vortex_array::arrays::shared::SharedArrayExt::get_or_compute_async(&self, f: F) -> vortex_error::VortexResult where F: core::ops::function::FnOnce(vortex_array::ArrayRef) -> Fut, Fut: core::future::future::Future> +pub async fn vortex_array::arrays::shared::SharedArrayExt::get_or_compute_async(&self, F) -> vortex_error::VortexResult where F: core::ops::function::FnOnce(vortex_array::ArrayRef) -> Fut, Fut: core::future::future::Future> pub fn vortex_array::arrays::shared::SharedArrayExt::source(&self) -> &vortex_array::ArrayRef @@ -4164,9 +4360,9 @@ impl> vortex_array: pub fn T::current_array_ref(&self) -> &vortex_array::ArrayRef -pub fn T::get_or_compute(&self, f: impl core::ops::function::FnOnce(&vortex_array::ArrayRef) -> vortex_error::VortexResult) -> vortex_error::VortexResult +pub fn T::get_or_compute(&self, impl core::ops::function::FnOnce(&vortex_array::ArrayRef) -> vortex_error::VortexResult) -> vortex_error::VortexResult -pub async fn T::get_or_compute_async(&self, f: F) -> vortex_error::VortexResult where F: core::ops::function::FnOnce(vortex_array::ArrayRef) -> Fut, Fut: core::future::future::Future> +pub async fn T::get_or_compute_async(&self, F) -> vortex_error::VortexResult where F: core::ops::function::FnOnce(vortex_array::ArrayRef) -> Fut, Fut: core::future::future::Future> pub fn T::source(&self) -> &vortex_array::ArrayRef @@ -4176,21 +4372,17 @@ pub mod vortex_array::arrays::slice pub struct vortex_array::arrays::slice::Slice -impl vortex_array::arrays::slice::Slice - -pub const vortex_array::arrays::slice::Slice::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::slice::Slice pub fn vortex_array::arrays::slice::Slice::clone(&self) -> vortex_array::arrays::slice::Slice impl core::fmt::Debug for vortex_array::arrays::slice::Slice -pub fn vortex_array::arrays::slice::Slice::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::slice::Slice::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::slice::Slice -pub fn vortex_array::arrays::slice::Slice::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::slice::Slice>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::slice::Slice::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::slice::Slice>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::slice::Slice @@ -4200,45 +4392,45 @@ pub type vortex_array::arrays::slice::Slice::OperationsVTable = vortex_array::ar pub type vortex_array::arrays::slice::Slice::ValidityVTable = vortex_array::arrays::slice::Slice -pub fn vortex_array::arrays::slice::Slice::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::slice::Slice::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::slice::Slice::buffer(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::slice::Slice::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::slice::Slice::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::slice::Slice::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::slice::Slice::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::slice::Slice::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::slice::Slice::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::slice::Slice::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::slice::Slice::deserialize(&self, _dtype: &vortex_array::dtype::DType, _len: usize, _metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::slice::Slice::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::slice::Slice::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::slice::Slice::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::slice::Slice::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::slice::Slice::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::slice::Slice::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::slice::Slice::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::slice::Slice::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::slice::Slice::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::slice::Slice::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::slice::Slice::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::slice::Slice::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::slice::Slice::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::slice::Slice::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::slice::Slice::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::slice::Slice::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::slice::Slice -pub fn vortex_array::arrays::slice::Slice::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::slice::Slice>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::slice::Slice::validity(vortex_array::ArrayView<'_, vortex_array::arrays::slice::Slice>) -> vortex_error::VortexResult impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::slice::Slice -pub fn vortex_array::arrays::slice::Slice::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> pub struct vortex_array::arrays::slice::SliceData @@ -4250,7 +4442,7 @@ pub fn vortex_array::arrays::slice::SliceData::is_empty(&self) -> bool pub fn vortex_array::arrays::slice::SliceData::len(&self) -> usize -pub fn vortex_array::arrays::slice::SliceData::new(range: core::ops::range::Range) -> Self +pub fn vortex_array::arrays::slice::SliceData::new(core::ops::range::Range) -> Self pub fn vortex_array::arrays::slice::SliceData::slice_range(&self) -> &core::ops::range::Range @@ -4260,19 +4452,19 @@ pub fn vortex_array::arrays::slice::SliceData::clone(&self) -> vortex_array::arr impl core::fmt::Debug for vortex_array::arrays::slice::SliceData -pub fn vortex_array::arrays::slice::SliceData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::slice::SliceData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::arrays::slice::SliceData -pub fn vortex_array::arrays::slice::SliceData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::slice::SliceData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::ArrayEq for vortex_array::arrays::slice::SliceData -pub fn vortex_array::arrays::slice::SliceData::array_eq(&self, other: &Self, _precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::slice::SliceData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayHash for vortex_array::arrays::slice::SliceData -pub fn vortex_array::arrays::slice::SliceData::array_hash(&self, state: &mut H, _precision: vortex_array::Precision) +pub fn vortex_array::arrays::slice::SliceData::array_hash(&self, &mut H, vortex_array::Precision) pub struct vortex_array::arrays::slice::SliceDataParts @@ -4286,19 +4478,19 @@ pub fn vortex_array::arrays::slice::SliceExecuteAdaptor::default() -> vortex_ impl core::fmt::Debug for vortex_array::arrays::slice::SliceExecuteAdaptor -pub fn vortex_array::arrays::slice::SliceExecuteAdaptor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::slice::SliceExecuteAdaptor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::kernel::ExecuteParentKernel for vortex_array::arrays::slice::SliceExecuteAdaptor where V: vortex_array::arrays::slice::SliceKernel pub type vortex_array::arrays::slice::SliceExecuteAdaptor::Parent = vortex_array::arrays::slice::Slice -pub fn vortex_array::arrays::slice::SliceExecuteAdaptor::execute_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: ::Match, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::SliceExecuteAdaptor::execute_parent(&self, vortex_array::ArrayView<'_, V>, ::Match, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub struct vortex_array::arrays::slice::SliceMetadata(_) impl core::fmt::Debug for vortex_array::arrays::slice::SliceMetadata -pub fn vortex_array::arrays::slice::SliceMetadata::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::slice::SliceMetadata::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_array::arrays::slice::SliceReduceAdaptor(pub V) @@ -4308,13 +4500,13 @@ pub fn vortex_array::arrays::slice::SliceReduceAdaptor::default() -> vortex_a impl core::fmt::Debug for vortex_array::arrays::slice::SliceReduceAdaptor -pub fn vortex_array::arrays::slice::SliceReduceAdaptor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::slice::SliceReduceAdaptor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::arrays::slice::SliceReduceAdaptor where V: vortex_array::arrays::slice::SliceReduce pub type vortex_array::arrays::slice::SliceReduceAdaptor::Parent = vortex_array::arrays::slice::Slice -pub fn vortex_array::arrays::slice::SliceReduceAdaptor::reduce_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: ::Match, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::SliceReduceAdaptor::reduce_parent(&self, vortex_array::ArrayView<'_, V>, ::Match, usize) -> vortex_error::VortexResult> pub trait vortex_array::arrays::slice::SliceArrayExt: vortex_array::TypedArrayRef @@ -4326,79 +4518,79 @@ pub fn T::child(&self) -> &vortex_array::ArrayRef pub trait vortex_array::arrays::slice::SliceKernel: vortex_array::VTable -pub fn vortex_array::arrays::slice::SliceKernel::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::SliceKernel::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceKernel for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub trait vortex_array::arrays::slice::SliceReduce: vortex_array::VTable -pub fn vortex_array::arrays::slice::SliceReduce::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::SliceReduce::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::slice(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::slice(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::List -pub fn vortex_array::arrays::List::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::slice(_array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::patched::Patched -pub fn vortex_array::arrays::patched::Patched::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::slice::Slice -pub fn vortex_array::arrays::slice::Slice::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> pub type vortex_array::arrays::slice::SliceArray = vortex_array::Array @@ -4406,21 +4598,17 @@ pub mod vortex_array::arrays::struct_ pub struct vortex_array::arrays::struct_::Struct -impl vortex_array::arrays::Struct - -pub const vortex_array::arrays::Struct::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::Struct pub fn vortex_array::arrays::Struct::clone(&self) -> vortex_array::arrays::Struct impl core::fmt::Debug for vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::Struct::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Struct::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::Struct @@ -4430,61 +4618,65 @@ pub type vortex_array::arrays::Struct::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::Struct::ValidityVTable = vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Struct::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Struct::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Struct::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Struct::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Struct::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Struct::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Struct::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Struct::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Struct::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Struct::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Struct::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Struct::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Struct::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Struct::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Struct::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Struct::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Struct::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Struct::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Struct::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Struct::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Struct::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Struct::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Struct::slot_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Struct::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Struct::validate(&self, _data: &vortex_array::EmptyArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Struct::validate(&self, &vortex_array::EmptyArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Struct>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Struct::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Struct>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeReduce for vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, indices: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::take(vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, dtype: &vortex_array::dtype::DType, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::Struct + +pub fn vortex_array::arrays::Struct::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::mask(vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::zip::ZipKernel for vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::zip(if_true: vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, if_false: &vortex_array::ArrayRef, mask: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::zip(vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, &vortex_array::ArrayRef, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub struct vortex_array::arrays::struct_::StructDataParts @@ -4506,11 +4698,11 @@ pub fn vortex_array::arrays::struct_::StructArrayExt::struct_fields(&self) -> &v pub fn vortex_array::arrays::struct_::StructArrayExt::struct_validity(&self) -> vortex_array::validity::Validity -pub fn vortex_array::arrays::struct_::StructArrayExt::unmasked_field(&self, idx: usize) -> &vortex_array::ArrayRef +pub fn vortex_array::arrays::struct_::StructArrayExt::unmasked_field(&self, usize) -> &vortex_array::ArrayRef -pub fn vortex_array::arrays::struct_::StructArrayExt::unmasked_field_by_name(&self, name: impl core::convert::AsRef) -> vortex_error::VortexResult<&vortex_array::ArrayRef> +pub fn vortex_array::arrays::struct_::StructArrayExt::unmasked_field_by_name(&self, impl core::convert::AsRef) -> vortex_error::VortexResult<&vortex_array::ArrayRef> -pub fn vortex_array::arrays::struct_::StructArrayExt::unmasked_field_by_name_opt(&self, name: impl core::convert::AsRef) -> core::option::Option<&vortex_array::ArrayRef> +pub fn vortex_array::arrays::struct_::StructArrayExt::unmasked_field_by_name_opt(&self, impl core::convert::AsRef) -> core::option::Option<&vortex_array::ArrayRef> pub fn vortex_array::arrays::struct_::StructArrayExt::unmasked_fields(&self) -> alloc::sync::Arc<[vortex_array::ArrayRef]> @@ -4526,11 +4718,11 @@ pub fn T::struct_fields(&self) -> &vortex_array::dtype::StructFields pub fn T::struct_validity(&self) -> vortex_array::validity::Validity -pub fn T::unmasked_field(&self, idx: usize) -> &vortex_array::ArrayRef +pub fn T::unmasked_field(&self, usize) -> &vortex_array::ArrayRef -pub fn T::unmasked_field_by_name(&self, name: impl core::convert::AsRef) -> vortex_error::VortexResult<&vortex_array::ArrayRef> +pub fn T::unmasked_field_by_name(&self, impl core::convert::AsRef) -> vortex_error::VortexResult<&vortex_array::ArrayRef> -pub fn T::unmasked_field_by_name_opt(&self, name: impl core::convert::AsRef) -> core::option::Option<&vortex_array::ArrayRef> +pub fn T::unmasked_field_by_name_opt(&self, impl core::convert::AsRef) -> core::option::Option<&vortex_array::ArrayRef> pub fn T::unmasked_fields(&self) -> alloc::sync::Arc<[vortex_array::ArrayRef]> @@ -4544,21 +4736,21 @@ pub struct vortex_array::arrays::varbin::builder::VarBinBuilder vortex_array::arrays::varbin::builder::VarBinBuilder -pub fn vortex_array::arrays::varbin::builder::VarBinBuilder::append(&mut self, value: core::option::Option<&[u8]>) +pub fn vortex_array::arrays::varbin::builder::VarBinBuilder::append(&mut self, core::option::Option<&[u8]>) -pub fn vortex_array::arrays::varbin::builder::VarBinBuilder::append_n_nulls(&mut self, n: usize) +pub fn vortex_array::arrays::varbin::builder::VarBinBuilder::append_n_nulls(&mut self, usize) pub fn vortex_array::arrays::varbin::builder::VarBinBuilder::append_null(&mut self) -pub fn vortex_array::arrays::varbin::builder::VarBinBuilder::append_value(&mut self, value: impl core::convert::AsRef<[u8]>) +pub fn vortex_array::arrays::varbin::builder::VarBinBuilder::append_value(&mut self, impl core::convert::AsRef<[u8]>) -pub fn vortex_array::arrays::varbin::builder::VarBinBuilder::append_values(&mut self, values: &[u8], end_offsets: impl core::iter::traits::iterator::Iterator, num: usize) where O: 'static, usize: num_traits::cast::AsPrimitive +pub fn vortex_array::arrays::varbin::builder::VarBinBuilder::append_values(&mut self, &[u8], impl core::iter::traits::iterator::Iterator, usize) where O: 'static, usize: num_traits::cast::AsPrimitive -pub fn vortex_array::arrays::varbin::builder::VarBinBuilder::finish(self, dtype: vortex_array::dtype::DType) -> vortex_array::arrays::VarBinArray +pub fn vortex_array::arrays::varbin::builder::VarBinBuilder::finish(self, vortex_array::dtype::DType) -> vortex_array::arrays::VarBinArray pub fn vortex_array::arrays::varbin::builder::VarBinBuilder::new() -> Self -pub fn vortex_array::arrays::varbin::builder::VarBinBuilder::with_capacity(len: usize) -> Self +pub fn vortex_array::arrays::varbin::builder::VarBinBuilder::with_capacity(usize) -> Self impl core::default::Default for vortex_array::arrays::varbin::builder::VarBinBuilder @@ -4568,11 +4760,7 @@ pub struct vortex_array::arrays::varbin::VarBin impl vortex_array::arrays::VarBin -pub const vortex_array::arrays::VarBin::ID: vortex_array::ArrayId - -impl vortex_array::arrays::VarBin - -pub fn vortex_array::arrays::VarBin::_slice(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, range: core::ops::range::Range) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBin::_slice(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, core::ops::range::Range) -> vortex_error::VortexResult impl core::clone::Clone for vortex_array::arrays::VarBin @@ -4580,11 +4768,11 @@ pub fn vortex_array::arrays::VarBin::clone(&self) -> vortex_array::arrays::VarBi impl core::fmt::Debug for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::VarBin::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBin::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::VarBin @@ -4594,87 +4782,91 @@ pub type vortex_array::arrays::VarBin::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::VarBin::ValidityVTable = vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::VarBin::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::VarBin::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::VarBin::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::VarBin::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::VarBin::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::VarBin::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::VarBin::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::VarBin::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::VarBin::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::VarBin::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBin::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBin::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::VarBin::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::VarBin::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::VarBin::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::VarBin::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::VarBin::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::VarBin::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::VarBin::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBin::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBin::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::VarBin::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::VarBin::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::VarBin::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::VarBin::validate(&self, _data: &vortex_array::arrays::varbin::VarBinData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::VarBin::validate(&self, &vortex_array::arrays::varbin::VarBinData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBin::validity(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::take(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterKernel for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::filter(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, mask: &vortex_mask::Mask, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::filter(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, &vortex_mask::Mask, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::binary::CompareKernel for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::compare(lhs: vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, rhs: &vortex_array::ArrayRef, operator: vortex_array::scalar_fn::fns::operators::CompareOperator, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::compare(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, &vortex_array::ArrayRef, vortex_array::scalar_fn::fns::operators::CompareOperator, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::VarBin + +pub fn vortex_array::arrays::VarBin::cast(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::cast(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::mask(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> pub struct vortex_array::arrays::varbin::VarBinData impl vortex_array::arrays::varbin::VarBinData -pub fn vortex_array::arrays::varbin::VarBinData::build(offsets: vortex_array::ArrayRef, bytes: vortex_buffer::ByteBuffer, dtype: vortex_array::dtype::DType, validity: vortex_array::validity::Validity) -> Self +pub fn vortex_array::arrays::varbin::VarBinData::build(vortex_array::ArrayRef, vortex_buffer::ByteBuffer, vortex_array::dtype::DType, vortex_array::validity::Validity) -> Self -pub fn vortex_array::arrays::varbin::VarBinData::build_from_handle(offset: vortex_array::ArrayRef, bytes: vortex_array::buffer::BufferHandle, dtype: vortex_array::dtype::DType, validity: vortex_array::validity::Validity) -> Self +pub fn vortex_array::arrays::varbin::VarBinData::build_from_handle(vortex_array::ArrayRef, vortex_array::buffer::BufferHandle, vortex_array::dtype::DType, vortex_array::validity::Validity) -> Self pub fn vortex_array::arrays::varbin::VarBinData::bytes(&self) -> &vortex_buffer::ByteBuffer pub fn vortex_array::arrays::varbin::VarBinData::bytes_handle(&self) -> &vortex_array::buffer::BufferHandle -pub unsafe fn vortex_array::arrays::varbin::VarBinData::new_unchecked(bytes: vortex_buffer::ByteBuffer) -> Self +pub unsafe fn vortex_array::arrays::varbin::VarBinData::new_unchecked(vortex_buffer::ByteBuffer) -> Self -pub unsafe fn vortex_array::arrays::varbin::VarBinData::new_unchecked_from_handle(bytes: vortex_array::buffer::BufferHandle) -> Self +pub unsafe fn vortex_array::arrays::varbin::VarBinData::new_unchecked_from_handle(vortex_array::buffer::BufferHandle) -> Self -pub fn vortex_array::arrays::varbin::VarBinData::try_build(offsets: vortex_array::ArrayRef, bytes: vortex_buffer::ByteBuffer, dtype: vortex_array::dtype::DType, validity: vortex_array::validity::Validity) -> vortex_error::VortexResult +pub fn vortex_array::arrays::varbin::VarBinData::try_build(vortex_array::ArrayRef, vortex_buffer::ByteBuffer, vortex_array::dtype::DType, vortex_array::validity::Validity) -> vortex_error::VortexResult -pub fn vortex_array::arrays::varbin::VarBinData::try_build_from_handle(offsets: vortex_array::ArrayRef, bytes: vortex_array::buffer::BufferHandle, dtype: vortex_array::dtype::DType, validity: vortex_array::validity::Validity) -> vortex_error::VortexResult +pub fn vortex_array::arrays::varbin::VarBinData::try_build_from_handle(vortex_array::ArrayRef, vortex_array::buffer::BufferHandle, vortex_array::dtype::DType, vortex_array::validity::Validity) -> vortex_error::VortexResult -pub fn vortex_array::arrays::varbin::VarBinData::validate(offsets: &vortex_array::ArrayRef, bytes: &vortex_array::buffer::BufferHandle, dtype: &vortex_array::dtype::DType, validity: &vortex_array::validity::Validity) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::varbin::VarBinData::validate(&vortex_array::ArrayRef, &vortex_array::buffer::BufferHandle, &vortex_array::dtype::DType, &vortex_array::validity::Validity) -> vortex_error::VortexResult<()> impl core::clone::Clone for vortex_array::arrays::varbin::VarBinData @@ -4682,19 +4874,19 @@ pub fn vortex_array::arrays::varbin::VarBinData::clone(&self) -> vortex_array::a impl core::fmt::Debug for vortex_array::arrays::varbin::VarBinData -pub fn vortex_array::arrays::varbin::VarBinData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::varbin::VarBinData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::arrays::varbin::VarBinData -pub fn vortex_array::arrays::varbin::VarBinData::fmt(&self, _f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::varbin::VarBinData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::ArrayEq for vortex_array::arrays::varbin::VarBinData -pub fn vortex_array::arrays::varbin::VarBinData::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::varbin::VarBinData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayHash for vortex_array::arrays::varbin::VarBinData -pub fn vortex_array::arrays::varbin::VarBinData::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn vortex_array::arrays::varbin::VarBinData::array_hash(&self, &mut H, vortex_array::Precision) pub struct vortex_array::arrays::varbin::VarBinDataParts @@ -4708,7 +4900,7 @@ pub vortex_array::arrays::varbin::VarBinDataParts::validity: vortex_array::valid pub trait vortex_array::arrays::varbin::VarBinArrayExt: vortex_array::TypedArrayRef -pub fn vortex_array::arrays::varbin::VarBinArrayExt::bytes_at(&self, index: usize) -> vortex_buffer::ByteBuffer +pub fn vortex_array::arrays::varbin::VarBinArrayExt::bytes_at(&self, usize) -> vortex_buffer::ByteBuffer pub fn vortex_array::arrays::varbin::VarBinArrayExt::dtype_parts(&self) -> (bool, vortex_array::dtype::Nullability) @@ -4716,7 +4908,7 @@ pub fn vortex_array::arrays::varbin::VarBinArrayExt::is_utf8(&self) -> bool pub fn vortex_array::arrays::varbin::VarBinArrayExt::nullability(&self) -> vortex_array::dtype::Nullability -pub fn vortex_array::arrays::varbin::VarBinArrayExt::offset_at(&self, index: usize) -> usize +pub fn vortex_array::arrays::varbin::VarBinArrayExt::offset_at(&self, usize) -> usize pub fn vortex_array::arrays::varbin::VarBinArrayExt::offsets(&self) -> &vortex_array::ArrayRef @@ -4726,11 +4918,9 @@ pub fn vortex_array::arrays::varbin::VarBinArrayExt::validity_child(&self) -> co pub fn vortex_array::arrays::varbin::VarBinArrayExt::varbin_validity(&self) -> vortex_array::validity::Validity -pub fn vortex_array::arrays::varbin::VarBinArrayExt::varbin_validity_mask(&self) -> vortex_mask::Mask - impl> vortex_array::arrays::varbin::VarBinArrayExt for T -pub fn T::bytes_at(&self, index: usize) -> vortex_buffer::ByteBuffer +pub fn T::bytes_at(&self, usize) -> vortex_buffer::ByteBuffer pub fn T::dtype_parts(&self) -> (bool, vortex_array::dtype::Nullability) @@ -4738,7 +4928,7 @@ pub fn T::is_utf8(&self) -> bool pub fn T::nullability(&self) -> vortex_array::dtype::Nullability -pub fn T::offset_at(&self, index: usize) -> usize +pub fn T::offset_at(&self, usize) -> usize pub fn T::offsets(&self) -> &vortex_array::ArrayRef @@ -4748,9 +4938,7 @@ pub fn T::validity_child(&self) -> core::option::Option<&vortex_array::ArrayRef> pub fn T::varbin_validity(&self) -> vortex_array::validity::Validity -pub fn T::varbin_validity_mask(&self) -> vortex_mask::Mask - -pub fn vortex_array::arrays::varbin::varbin_scalar(value: vortex_buffer::ByteBuffer, dtype: &vortex_array::dtype::DType) -> vortex_array::scalar::Scalar +pub fn vortex_array::arrays::varbin::varbin_scalar(vortex_buffer::ByteBuffer, &vortex_array::dtype::DType) -> vortex_array::scalar::Scalar pub type vortex_array::arrays::varbin::VarBinArray = vortex_array::Array @@ -4780,9 +4968,9 @@ pub fn vortex_array::arrays::varbinview::BinaryView::is_inlined(&self) -> bool pub fn vortex_array::arrays::varbinview::BinaryView::len(&self) -> u32 -pub fn vortex_array::arrays::varbinview::BinaryView::make_view(value: &[u8], block: u32, offset: u32) -> Self +pub fn vortex_array::arrays::varbinview::BinaryView::make_view(&[u8], u32, u32) -> Self -pub fn vortex_array::arrays::varbinview::BinaryView::new_inlined(value: &[u8]) -> Self +pub fn vortex_array::arrays::varbinview::BinaryView::new_inlined(&[u8]) -> Self impl core::clone::Clone for vortex_array::arrays::varbinview::BinaryView @@ -4792,15 +4980,15 @@ impl core::cmp::Eq for vortex_array::arrays::varbinview::BinaryView impl core::cmp::PartialEq for vortex_array::arrays::varbinview::BinaryView -pub fn vortex_array::arrays::varbinview::BinaryView::eq(&self, other: &Self) -> bool +pub fn vortex_array::arrays::varbinview::BinaryView::eq(&self, &Self) -> bool impl core::convert::From for vortex_array::arrays::varbinview::BinaryView -pub fn vortex_array::arrays::varbinview::BinaryView::from(value: u128) -> Self +pub fn vortex_array::arrays::varbinview::BinaryView::from(u128) -> Self impl core::convert::From for vortex_array::arrays::varbinview::BinaryView -pub fn vortex_array::arrays::varbinview::BinaryView::from(value: vortex_array::arrays::varbinview::Ref) -> Self +pub fn vortex_array::arrays::varbinview::BinaryView::from(vortex_array::arrays::varbinview::Ref) -> Self impl core::default::Default for vortex_array::arrays::varbinview::BinaryView @@ -4808,19 +4996,19 @@ pub fn vortex_array::arrays::varbinview::BinaryView::default() -> Self impl core::fmt::Debug for vortex_array::arrays::varbinview::BinaryView -pub fn vortex_array::arrays::varbinview::BinaryView::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::varbinview::BinaryView::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::arrays::varbinview::BinaryView -pub fn vortex_array::arrays::varbinview::BinaryView::hash(&self, state: &mut H) +pub fn vortex_array::arrays::varbinview::BinaryView::hash(&self, &mut H) impl core::marker::Copy for vortex_array::arrays::varbinview::BinaryView pub const vortex_array::arrays::varbinview::build_views::MAX_BUFFER_LEN: usize -pub fn vortex_array::arrays::varbinview::build_views::build_views>(start_buf_index: u32, max_buffer_len: usize, bytes: vortex_buffer::ByteBufferMut, lens: &[P]) -> (alloc::vec::Vec, vortex_buffer::buffer::Buffer) +pub fn vortex_array::arrays::varbinview::build_views::build_views>(u32, usize, vortex_buffer::ByteBufferMut, &[P]) -> (alloc::vec::Vec, vortex_buffer::buffer::Buffer) -pub fn vortex_array::arrays::varbinview::build_views::offsets_to_lengths(offsets: &[P]) -> vortex_buffer::buffer::Buffer

+pub fn vortex_array::arrays::varbinview::build_views::offsets_to_lengths(&[P]) -> vortex_buffer::buffer::Buffer

(array, strict) }) +pub(super) fn check_primitive_sorted( + array: &PrimitiveArray, + strict: bool, + ctx: &mut ExecutionCtx, +) -> VortexResult { + match_each_native_ptype!(array.ptype(), |P| { + compute_is_sorted::

(array, strict, ctx) + }) } -fn compute_is_sorted(array: &PrimitiveArray, strict: bool) -> VortexResult { - match array.validity_mask()? { +fn compute_is_sorted( + array: &PrimitiveArray, + strict: bool, + ctx: &mut ExecutionCtx, +) -> VortexResult { + match array + .as_ref() + .validity()? + .execute_mask(array.as_ref().len(), ctx)? + { Mask::AllFalse(_) => Ok(!strict), Mask::AllTrue(_) => { let slice = array.as_slice::(); diff --git a/vortex-array/src/aggregate_fn/fns/last/mod.rs b/vortex-array/src/aggregate_fn/fns/last/mod.rs index 009b487ad2d..a0034d0820b 100644 --- a/vortex-array/src/aggregate_fn/fns/last/mod.rs +++ b/vortex-array/src/aggregate_fn/fns/last/mod.rs @@ -40,7 +40,7 @@ impl AggregateFnVTable for Last { type Partial = LastPartial; fn id(&self) -> AggregateFnId { - AggregateFnId::new_ref("vortex.last") + AggregateFnId::new("vortex.last") } fn serialize(&self, _options: &Self::Options) -> VortexResult>> { @@ -95,10 +95,10 @@ impl AggregateFnVTable for Last { &self, partial: &mut Self::Partial, batch: &ArrayRef, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { - if let Some(idx) = batch.validity_mask()?.last() { - let scalar = batch.scalar_at(idx)?; + if let Some(idx) = batch.validity()?.execute_mask(batch.len(), ctx)?.last() { + let scalar = batch.execute_scalar(idx, ctx)?; partial.value = Some(scalar.into_nullable()); } Ok(true) diff --git a/vortex-array/src/aggregate_fn/fns/mean/mod.rs b/vortex-array/src/aggregate_fn/fns/mean/mod.rs new file mode 100644 index 00000000000..f7d471023ee --- /dev/null +++ b/vortex-array/src/aggregate_fn/fns/mean/mod.rs @@ -0,0 +1,262 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +use vortex_error::VortexResult; +use vortex_error::vortex_bail; + +use crate::ArrayRef; +use crate::ExecutionCtx; +use crate::aggregate_fn::Accumulator; +use crate::aggregate_fn::AggregateFnId; +use crate::aggregate_fn::AggregateFnVTable; +use crate::aggregate_fn::DynAccumulator; +use crate::aggregate_fn::EmptyOptions; +use crate::aggregate_fn::combined::BinaryCombined; +use crate::aggregate_fn::combined::Combined; +use crate::aggregate_fn::combined::CombinedOptions; +use crate::aggregate_fn::combined::PairOptions; +use crate::aggregate_fn::fns::count::Count; +use crate::aggregate_fn::fns::sum::Sum; +use crate::builtins::ArrayBuiltins; +use crate::dtype::DType; +use crate::dtype::Nullability; +use crate::dtype::PType; +use crate::scalar::Scalar; +use crate::scalar_fn::fns::operators::Operator; + +/// Compute the arithmetic mean of an array. +/// +/// See [`Mean`] for details. +pub fn mean(array: &ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + let mut acc = Accumulator::try_new( + Mean::combined(), + PairOptions(EmptyOptions, EmptyOptions), + array.dtype().clone(), + )?; + acc.accumulate(array, ctx)?; + acc.finish() +} + +/// Compute the arithmetic mean of an array. +/// +/// Implemented as `Sum / Count` via [`BinaryCombined`]. +/// +/// Coercion / return type: +/// - Booleans and primitive numeric types are coerced to `f64` and the result +/// is a nullable `f64`. +/// - Decimals are kept as decimals but not implemented currently +#[derive(Clone, Debug)] +pub struct Mean; + +impl Mean { + pub fn combined() -> Combined { + Combined(Mean) + } +} + +impl BinaryCombined for Mean { + type Left = Sum; + type Right = Count; + + fn id(&self) -> AggregateFnId { + AggregateFnId::new("vortex.mean") + } + + fn left(&self) -> Sum { + Sum + } + + fn right(&self) -> Count { + Count + } + + fn left_name(&self) -> &'static str { + "sum" + } + + fn right_name(&self) -> &'static str { + "count" + } + + fn return_dtype(&self, input_dtype: &DType) -> Option { + Some(mean_output_dtype(input_dtype)?.with_nullability(Nullability::Nullable)) + } + + fn finalize(&self, sum: ArrayRef, count: ArrayRef) -> VortexResult { + let target = match sum.dtype() { + DType::Decimal(..) => sum.dtype().with_nullability(Nullability::Nullable), + _ => DType::Primitive(PType::F64, Nullability::Nullable), + }; + let sum_cast = sum.cast(target.clone())?; + let count_cast = count.cast(target)?; + sum_cast.binary(count_cast, Operator::Div) + } + + fn finalize_scalar(&self, left_scalar: Scalar, right_scalar: Scalar) -> VortexResult { + if let DType::Decimal(..) = left_scalar.dtype() { + vortex_bail!("mean::finalize_scalar not yet implemented for decimal inputs"); + } + + let target = DType::Primitive(PType::F64, Nullability::Nullable); + let sum_cast = left_scalar.cast(&target)?; + let count_cast = right_scalar.cast(&target)?; + + let sum = sum_cast.as_primitive().typed_value::(); + let count = count_cast.as_primitive().typed_value::(); + let value = match (sum, count) { + (None, _) | (_, None) | (_, Some(0.0)) => return Ok(Scalar::null(target)), // Sum overflowed + (Some(s), Some(c)) => s / c, + }; + Ok(Scalar::primitive(value, Nullability::Nullable)) + } + + fn serialize(&self, _options: &CombinedOptions) -> VortexResult>> { + unimplemented!("mean is not yet serializable"); + } + + fn coerce_args( + &self, + _options: &PairOptions< + ::Options, + ::Options, + >, + input_dtype: &DType, + ) -> VortexResult { + // Advisory hint for query planners: where possible, cast input to the + // type we're going to compute the mean in. + Ok(coerced_input_dtype(input_dtype).unwrap_or_else(|| input_dtype.clone())) + } +} + +/// Hint for callers: what to cast the input to before accumulation. +/// +/// - Bool stays as bool — `Sum` has a native bool path and bool → f64 isn't +/// currently a direct cast in vortex. +/// - Primitive numerics → `f64` so the sum and finalize work without overflow. +fn coerced_input_dtype(input_dtype: &DType) -> Option { + match input_dtype { + DType::Bool(_) => Some(input_dtype.clone()), + DType::Primitive(_, n) => Some(DType::Primitive(PType::F64, *n)), + DType::Decimal(..) => { + unimplemented!("mean is not implemented for decimals yet") + } + _ => None, + } +} + +fn mean_output_dtype(input_dtype: &DType) -> Option { + match input_dtype { + DType::Bool(_) | DType::Primitive(..) => { + Some(DType::Primitive(PType::F64, Nullability::Nullable)) + } + DType::Decimal(..) => { + unimplemented!("mean for decimals is not yet implemented"); + } + _ => None, + } +} + +#[cfg(test)] +mod tests { + use vortex_buffer::buffer; + use vortex_error::VortexResult; + + use super::*; + use crate::IntoArray; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; + use crate::arrays::BoolArray; + use crate::arrays::ChunkedArray; + use crate::arrays::ConstantArray; + use crate::arrays::PrimitiveArray; + use crate::validity::Validity; + + #[test] + fn mean_all_valid() -> VortexResult<()> { + let array = PrimitiveArray::new(buffer![1.0f64, 2.0, 3.0, 4.0, 5.0], Validity::NonNullable) + .into_array(); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let result = mean(&array, &mut ctx)?; + assert_eq!(result.as_primitive().as_::(), Some(3.0)); + Ok(()) + } + + #[test] + fn mean_with_nulls() -> VortexResult<()> { + let array = PrimitiveArray::from_option_iter([Some(2.0f64), None, Some(4.0)]).into_array(); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let result = mean(&array, &mut ctx)?; + assert_eq!(result.as_primitive().as_::(), Some(3.0)); + Ok(()) + } + + #[test] + fn mean_integers() -> VortexResult<()> { + let array = PrimitiveArray::new(buffer![10i32, 20, 30], Validity::NonNullable).into_array(); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let result = mean(&array, &mut ctx)?; + assert_eq!(result.as_primitive().as_::(), Some(20.0)); + Ok(()) + } + + #[test] + fn mean_bool() -> VortexResult<()> { + let array: BoolArray = [true, false, true, true].into_iter().collect(); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let result = mean(&array.into_array(), &mut ctx)?; + assert_eq!(result.as_primitive().as_::(), Some(0.75)); + Ok(()) + } + + #[test] + fn mean_constant_non_null() -> VortexResult<()> { + let array = ConstantArray::new(5.0f64, 4); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let result = mean(&array.into_array(), &mut ctx)?; + assert_eq!(result.as_primitive().as_::(), Some(5.0)); + Ok(()) + } + + #[test] + fn mean_chunked() -> VortexResult<()> { + let chunk1 = PrimitiveArray::from_option_iter([Some(1.0f64), None, Some(3.0)]); + let chunk2 = PrimitiveArray::from_option_iter([Some(5.0f64), None]); + let dtype = chunk1.dtype().clone(); + let chunked = ChunkedArray::try_new(vec![chunk1.into_array(), chunk2.into_array()], dtype)?; + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let result = mean(&chunked.into_array(), &mut ctx)?; + assert_eq!(result.as_primitive().as_::(), Some(3.0)); + Ok(()) + } + + #[test] + fn mean_all_null_returns_null() -> VortexResult<()> { + let array = PrimitiveArray::from_option_iter::([None, None, None]).into_array(); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let result = mean(&array, &mut ctx)?; + assert_eq!(result.as_primitive().as_::(), None); + Ok(()) + } + + #[test] + fn mean_multi_batch() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let dtype = DType::Primitive(PType::F64, Nullability::NonNullable); + let mut acc = Accumulator::try_new( + Mean::combined(), + PairOptions(EmptyOptions, EmptyOptions), + dtype, + )?; + + let batch1 = + PrimitiveArray::new(buffer![1.0f64, 2.0, 3.0], Validity::NonNullable).into_array(); + acc.accumulate(&batch1, &mut ctx)?; + + let batch2 = PrimitiveArray::new(buffer![4.0f64, 5.0], Validity::NonNullable).into_array(); + acc.accumulate(&batch2, &mut ctx)?; + + let result = acc.finish()?; + assert_eq!(result.as_primitive().as_::(), Some(3.0)); + Ok(()) + } +} diff --git a/vortex-array/src/aggregate_fn/fns/min_max/bool.rs b/vortex-array/src/aggregate_fn/fns/min_max/bool.rs index 557598489e6..69471eec556 100644 --- a/vortex-array/src/aggregate_fn/fns/min_max/bool.rs +++ b/vortex-array/src/aggregate_fn/fns/min_max/bool.rs @@ -4,68 +4,53 @@ use std::ops::BitAnd; use vortex_error::VortexResult; -use vortex_mask::Mask; +use vortex_mask::AllOr; use super::MinMaxPartial; use super::MinMaxResult; +use crate::ExecutionCtx; use crate::arrays::BoolArray; use crate::arrays::bool::BoolArrayExt; use crate::dtype::Nullability::NonNullable; use crate::scalar::Scalar; -pub(super) fn accumulate_bool(partial: &mut MinMaxPartial, array: &BoolArray) -> VortexResult<()> { +pub(super) fn accumulate_bool( + partial: &mut MinMaxPartial, + array: &BoolArray, + ctx: &mut ExecutionCtx, +) -> VortexResult<()> { if array.is_empty() { return Ok(()); } - let mask = array.validity_mask()?; - let true_non_null = match &mask { - Mask::AllTrue(_) => array.to_bit_buffer(), - Mask::AllFalse(_) => return Ok(()), - Mask::Values(v) => array.to_bit_buffer().bitand(v.bit_buffer()), + let mask = array + .as_ref() + .validity()? + .execute_mask(array.as_ref().len(), ctx)?; + let (true_count, valid_count) = match mask.bit_buffer() { + AllOr::None => return Ok(()), + AllOr::All => (array.to_bit_buffer().true_count(), array.as_ref().len()), + AllOr::Some(validity) => ( + array.to_bit_buffer().bitand(validity).true_count(), + validity.true_count(), + ), }; - let mut true_slices = true_non_null.set_slices(); - - let Some(slice) = true_slices.next() else { - // all false - partial.merge(Some(MinMaxResult { - min: Scalar::bool(false, NonNullable), - max: Scalar::bool(false, NonNullable), - })); - return Ok(()); - }; - - if slice.0 == 0 && slice.1 == array.len() { - // all true - partial.merge(Some(MinMaxResult { - min: Scalar::bool(true, NonNullable), - max: Scalar::bool(true, NonNullable), - })); + if valid_count == 0 { return Ok(()); } - // Check for valid false values when we have a partial validity mask - match &mask { - Mask::AllTrue(_) | Mask::AllFalse(_) => {} - Mask::Values(v) => { - let false_non_null = (!array.to_bit_buffer()).bitand(v.bit_buffer()); - let mut false_slices = false_non_null.set_slices(); - - if false_slices.next().is_none() { - // No false values, so all valid values are true - partial.merge(Some(MinMaxResult { - min: Scalar::bool(true, NonNullable), - max: Scalar::bool(true, NonNullable), - })); - return Ok(()); - } - } - } + let (min, max) = if true_count == 0 { + (false, false) + } else if true_count == valid_count { + (true, true) + } else { + (false, true) + }; partial.merge(Some(MinMaxResult { - min: Scalar::bool(false, NonNullable), - max: Scalar::bool(true, NonNullable), + min: Scalar::bool(min, NonNullable), + max: Scalar::bool(max, NonNullable), })); Ok(()) } diff --git a/vortex-array/src/aggregate_fn/fns/min_max/decimal.rs b/vortex-array/src/aggregate_fn/fns/min_max/decimal.rs index 93400f37275..019136e2584 100644 --- a/vortex-array/src/aggregate_fn/fns/min_max/decimal.rs +++ b/vortex-array/src/aggregate_fn/fns/min_max/decimal.rs @@ -7,6 +7,7 @@ use vortex_mask::Mask; use super::MinMaxPartial; use super::MinMaxResult; +use crate::ExecutionCtx; use crate::arrays::DecimalArray; use crate::dtype::DecimalDType; use crate::dtype::NativeDecimalType; @@ -18,30 +19,40 @@ use crate::scalar::Scalar; pub(super) fn accumulate_decimal( partial: &mut MinMaxPartial, array: &DecimalArray, + ctx: &mut ExecutionCtx, ) -> VortexResult<()> { match_each_decimal_value_type!(array.values_type(), |T| { - let local = compute_min_max_with_validity::(array)?; + let local = compute_min_max_with_validity::(array, ctx)?; partial.merge(local); Ok(()) }) } -fn compute_min_max_with_validity(array: &DecimalArray) -> VortexResult> +fn compute_min_max_with_validity( + array: &DecimalArray, + ctx: &mut ExecutionCtx, +) -> VortexResult> where D: Into + NativeDecimalType, { - Ok(match array.validity_mask()? { - Mask::AllTrue(_) => compute_min_max(array.buffer::().iter(), array.decimal_dtype()), - Mask::AllFalse(_) => None, - Mask::Values(v) => compute_min_max( - array - .buffer::() - .iter() - .zip(v.bit_buffer().iter()) - .filter_map(|(v, m)| m.then_some(v)), - array.decimal_dtype(), - ), - }) + Ok( + match array + .as_ref() + .validity()? + .execute_mask(array.as_ref().len(), ctx)? + { + Mask::AllTrue(_) => compute_min_max(array.buffer::().iter(), array.decimal_dtype()), + Mask::AllFalse(_) => None, + Mask::Values(v) => compute_min_max( + array + .buffer::() + .iter() + .zip(v.bit_buffer().iter()) + .filter_map(|(v, m)| m.then_some(v)), + array.decimal_dtype(), + ), + }, + ) } fn compute_min_max<'a, T>( diff --git a/vortex-array/src/aggregate_fn/fns/min_max/mod.rs b/vortex-array/src/aggregate_fn/fns/min_max/mod.rs index 91d178a4814..fdd3229d75e 100644 --- a/vortex-array/src/aggregate_fn/fns/min_max/mod.rs +++ b/vortex-array/src/aggregate_fn/fns/min_max/mod.rs @@ -63,7 +63,7 @@ pub fn min_max(array: &ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult VortexResult AggregateFnId { - AggregateFnId::new_ref("vortex.min_max") + AggregateFnId::new("vortex.min_max") } fn serialize(&self, _options: &Self::Options) -> VortexResult>> { @@ -257,10 +257,10 @@ impl AggregateFnVTable for MinMax { Ok(()) } Columnar::Canonical(c) => match c { - Canonical::Primitive(p) => accumulate_primitive(partial, p), - Canonical::Bool(b) => accumulate_bool(partial, b), + Canonical::Primitive(p) => accumulate_primitive(partial, p, ctx), + Canonical::Bool(b) => accumulate_bool(partial, b, ctx), Canonical::VarBinView(v) => accumulate_varbinview(partial, v), - Canonical::Decimal(d) => accumulate_decimal(partial, d), + Canonical::Decimal(d) => accumulate_decimal(partial, d, ctx), Canonical::Extension(e) => accumulate_extension(partial, e, ctx), Canonical::Null(_) => Ok(()), Canonical::Struct(_) diff --git a/vortex-array/src/aggregate_fn/fns/min_max/primitive.rs b/vortex-array/src/aggregate_fn/fns/min_max/primitive.rs index 1332fa7dd88..3470ba3728d 100644 --- a/vortex-array/src/aggregate_fn/fns/min_max/primitive.rs +++ b/vortex-array/src/aggregate_fn/fns/min_max/primitive.rs @@ -7,6 +7,7 @@ use vortex_mask::Mask; use super::MinMaxPartial; use super::MinMaxResult; +use crate::ExecutionCtx; use crate::arrays::PrimitiveArray; use crate::dtype::NativePType; use crate::dtype::Nullability::NonNullable; @@ -17,30 +18,40 @@ use crate::scalar::Scalar; pub(super) fn accumulate_primitive( partial: &mut MinMaxPartial, p: &PrimitiveArray, + ctx: &mut ExecutionCtx, ) -> VortexResult<()> { match_each_native_ptype!(p.ptype(), |T| { - let local = compute_min_max_with_validity::(p)?; + let local = compute_min_max_with_validity::(p, ctx)?; partial.merge(local); Ok(()) }) } -fn compute_min_max_with_validity(array: &PrimitiveArray) -> VortexResult> +fn compute_min_max_with_validity( + array: &PrimitiveArray, + ctx: &mut ExecutionCtx, +) -> VortexResult> where T: NativePType, PValue: From, { - Ok(match array.validity_mask()? { - Mask::AllTrue(_) => compute_min_max(array.as_slice::().iter()), - Mask::AllFalse(_) => None, - Mask::Values(v) => compute_min_max( - array - .as_slice::() - .iter() - .zip(v.bit_buffer().iter()) - .filter_map(|(v, m)| m.then_some(v)), - ), - }) + Ok( + match array + .as_ref() + .validity()? + .execute_mask(array.as_ref().len(), ctx)? + { + Mask::AllTrue(_) => compute_min_max(array.as_slice::().iter()), + Mask::AllFalse(_) => None, + Mask::Values(v) => compute_min_max( + array + .as_slice::() + .iter() + .zip(v.bit_buffer().iter()) + .filter_map(|(v, m)| m.then_some(v)), + ), + }, + ) } fn compute_min_max<'a, T>(iter: impl Iterator) -> Option diff --git a/vortex-array/src/aggregate_fn/fns/mod.rs b/vortex-array/src/aggregate_fn/fns/mod.rs index 38d5340cd1f..f1281c18544 100644 --- a/vortex-array/src/aggregate_fn/fns/mod.rs +++ b/vortex-array/src/aggregate_fn/fns/mod.rs @@ -6,6 +6,7 @@ pub mod first; pub mod is_constant; pub mod is_sorted; pub mod last; +pub mod mean; pub mod min_max; pub mod nan_count; pub mod sum; diff --git a/vortex-array/src/aggregate_fn/fns/nan_count/mod.rs b/vortex-array/src/aggregate_fn/fns/nan_count/mod.rs index 9a1a8e4793b..0e229ed7982 100644 --- a/vortex-array/src/aggregate_fn/fns/nan_count/mod.rs +++ b/vortex-array/src/aggregate_fn/fns/nan_count/mod.rs @@ -46,7 +46,7 @@ pub fn nan_count(array: &ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult AggregateFnId { - AggregateFnId::new_ref("vortex.nan_count") + AggregateFnId::new("vortex.nan_count") } fn serialize(&self, _options: &Self::Options) -> VortexResult>> { @@ -136,7 +136,7 @@ impl AggregateFnVTable for NanCount { &self, partial: &mut Self::Partial, batch: &Columnar, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult<()> { match batch { Columnar::Constant(c) => { @@ -150,7 +150,7 @@ impl AggregateFnVTable for NanCount { Ok(()) } Columnar::Canonical(c) => match c { - Canonical::Primitive(p) => accumulate_primitive(partial, p), + Canonical::Primitive(p) => accumulate_primitive(partial, p, ctx), _ => vortex_bail!( "Unsupported canonical type for nan_count: {}", batch.dtype() diff --git a/vortex-array/src/aggregate_fn/fns/nan_count/primitive.rs b/vortex-array/src/aggregate_fn/fns/nan_count/primitive.rs index 34781bafbd7..8d8358235cc 100644 --- a/vortex-array/src/aggregate_fn/fns/nan_count/primitive.rs +++ b/vortex-array/src/aggregate_fn/fns/nan_count/primitive.rs @@ -4,13 +4,21 @@ use vortex_error::VortexResult; use vortex_mask::Mask; +use crate::ExecutionCtx; use crate::arrays::PrimitiveArray; use crate::dtype::NativePType; use crate::match_each_float_ptype; -pub(super) fn accumulate_primitive(count: &mut u64, p: &PrimitiveArray) -> VortexResult<()> { +pub(super) fn accumulate_primitive( + count: &mut u64, + p: &PrimitiveArray, + ctx: &mut ExecutionCtx, +) -> VortexResult<()> { match_each_float_ptype!(p.ptype(), |F| { - *count += compute_nan_count_with_validity(p.as_slice::(), p.validity_mask()?) as u64; + *count += compute_nan_count_with_validity( + p.as_slice::(), + p.as_ref().validity()?.execute_mask(p.as_ref().len(), ctx)?, + ) as u64; }); Ok(()) } diff --git a/vortex-array/src/aggregate_fn/fns/sum/bool.rs b/vortex-array/src/aggregate_fn/fns/sum/bool.rs index f80023e0b57..a32f3ba72f7 100644 --- a/vortex-array/src/aggregate_fn/fns/sum/bool.rs +++ b/vortex-array/src/aggregate_fn/fns/sum/bool.rs @@ -9,15 +9,20 @@ use vortex_mask::AllOr; use super::SumState; use super::checked_add_u64; +use crate::ExecutionCtx; use crate::arrays::BoolArray; use crate::arrays::bool::BoolArrayExt; -pub(super) fn accumulate_bool(inner: &mut SumState, b: &BoolArray) -> VortexResult { +pub(super) fn accumulate_bool( + inner: &mut SumState, + b: &BoolArray, + ctx: &mut ExecutionCtx, +) -> VortexResult { let SumState::Unsigned(acc) = inner else { vortex_panic!("expected unsigned sum state for bool input"); }; - let mask = b.validity_mask()?; + let mask = b.as_ref().validity()?.execute_mask(b.as_ref().len(), ctx)?; let true_count = match mask.bit_buffer() { AllOr::None => return Ok(false), AllOr::All => b.to_bit_buffer().true_count() as u64, diff --git a/vortex-array/src/aggregate_fn/fns/sum/decimal.rs b/vortex-array/src/aggregate_fn/fns/sum/decimal.rs index 91fd48e73ea..e37b671eb46 100644 --- a/vortex-array/src/aggregate_fn/fns/sum/decimal.rs +++ b/vortex-array/src/aggregate_fn/fns/sum/decimal.rs @@ -13,6 +13,7 @@ use vortex_error::vortex_panic; use vortex_mask::Mask; use super::SumState; +use crate::ExecutionCtx; use crate::arrays::DecimalArray; use crate::dtype::DecimalDType; use crate::dtype::DecimalType; @@ -22,8 +23,12 @@ use crate::scalar::DecimalValue; /// Accumulate a decimal array into the sum state. /// Returns Ok(true) if saturated (overflow), Ok(false) if not. -pub(super) fn accumulate_decimal(inner: &mut SumState, d: &DecimalArray) -> VortexResult { - let mask = d.validity_mask()?; +pub(super) fn accumulate_decimal( + inner: &mut SumState, + d: &DecimalArray, + ctx: &mut ExecutionCtx, +) -> VortexResult { + let mask = d.as_ref().validity()?.execute_mask(d.as_ref().len(), ctx)?; let validity = match &mask { Mask::AllTrue(_) => None, Mask::Values(mask_values) => Some(mask_values.bit_buffer()), diff --git a/vortex-array/src/aggregate_fn/fns/sum/mod.rs b/vortex-array/src/aggregate_fn/fns/sum/mod.rs index 4d09eac4d5b..cad8cedf626 100644 --- a/vortex-array/src/aggregate_fn/fns/sum/mod.rs +++ b/vortex-array/src/aggregate_fn/fns/sum/mod.rs @@ -71,7 +71,7 @@ impl AggregateFnVTable for Sum { type Partial = SumPartial; fn id(&self) -> AggregateFnId { - AggregateFnId::new_ref("vortex.sum") + AggregateFnId::new("vortex.sum") } fn serialize(&self, _options: &Self::Options) -> VortexResult>> { @@ -217,7 +217,7 @@ impl AggregateFnVTable for Sum { &self, partial: &mut Self::Partial, batch: &Columnar, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult<()> { // Constants compute scalar * len and combine via combine_partials. if let Columnar::Constant(c) = batch { @@ -234,9 +234,9 @@ impl AggregateFnVTable for Sum { let result = match batch { Columnar::Canonical(c) => match c { - Canonical::Primitive(p) => accumulate_primitive(&mut inner, p), - Canonical::Bool(b) => accumulate_bool(&mut inner, b), - Canonical::Decimal(d) => accumulate_decimal(&mut inner, d), + Canonical::Primitive(p) => accumulate_primitive(&mut inner, p, ctx), + Canonical::Bool(b) => accumulate_bool(&mut inner, b, ctx), + Canonical::Decimal(d) => accumulate_decimal(&mut inner, d, ctx), _ => vortex_bail!("Unsupported canonical type for sum: {}", batch.dtype()), }, Columnar::Constant(_) => unreachable!(), diff --git a/vortex-array/src/aggregate_fn/fns/sum/primitive.rs b/vortex-array/src/aggregate_fn/fns/sum/primitive.rs index ed8ecd17a6a..19c414b8204 100644 --- a/vortex-array/src/aggregate_fn/fns/sum/primitive.rs +++ b/vortex-array/src/aggregate_fn/fns/sum/primitive.rs @@ -11,11 +11,16 @@ use vortex_mask::AllOr; use super::SumState; use super::checked_add_i64; use super::checked_add_u64; +use crate::ExecutionCtx; use crate::arrays::PrimitiveArray; use crate::match_each_native_ptype; -pub(super) fn accumulate_primitive(inner: &mut SumState, p: &PrimitiveArray) -> VortexResult { - let mask = p.validity_mask()?; +pub(super) fn accumulate_primitive( + inner: &mut SumState, + p: &PrimitiveArray, + ctx: &mut ExecutionCtx, +) -> VortexResult { + let mask = p.as_ref().validity()?.execute_mask(p.as_ref().len(), ctx)?; match mask.bit_buffer() { AllOr::None => Ok(false), AllOr::All => accumulate_primitive_all(inner, p), diff --git a/vortex-array/src/aggregate_fn/foreign.rs b/vortex-array/src/aggregate_fn/foreign.rs index e79d2456493..feb07e47175 100644 --- a/vortex-array/src/aggregate_fn/foreign.rs +++ b/vortex-array/src/aggregate_fn/foreign.rs @@ -54,7 +54,7 @@ impl AggregateFnVTable for ForeignAggregateFnVTable { type Partial = (); fn id(&self) -> AggregateFnId { - self.id.clone() + self.id } fn serialize(&self, options: &Self::Options) -> VortexResult>> { @@ -111,6 +111,10 @@ impl AggregateFnVTable for ForeignAggregateFnVTable { fn finalize(&self, _states: ArrayRef) -> VortexResult { vortex_bail!("Cannot execute unknown aggregate function '{}'", self.id) } + + fn finalize_scalar(&self, _partial: &Self::Partial) -> VortexResult { + vortex_bail!("Cannot execute unknown aggregate function '{}'", self.id) + } } pub fn new_foreign_aggregate_fn(id: AggregateFnId, metadata: Vec) -> AggregateFnRef { diff --git a/vortex-array/src/aggregate_fn/mod.rs b/vortex-array/src/aggregate_fn/mod.rs index a22006cb981..1bcd2cac030 100644 --- a/vortex-array/src/aggregate_fn/mod.rs +++ b/vortex-array/src/aggregate_fn/mod.rs @@ -6,7 +6,7 @@ //! This module contains the [`AggregateFnVTable`] trait, the [`Accumulator`] trait, and the //! type-erasure infrastructure for aggregate functions. -use arcref::ArcRef; +use vortex_session::registry::Id; mod accumulator; pub use accumulator::*; @@ -32,13 +32,14 @@ pub use erased::*; mod options; pub use options::*; +pub mod combined; pub mod fns; pub mod kernels; pub mod proto; pub mod session; /// A unique identifier for an aggregate function. -pub type AggregateFnId = ArcRef; +pub type AggregateFnId = Id; /// Private module to seal [`typed::DynAggregateFn`]. mod sealed { diff --git a/vortex-array/src/aggregate_fn/proto.rs b/vortex-array/src/aggregate_fn/proto.rs index dfb86ad0ed9..9bcfab5818c 100644 --- a/vortex-array/src/aggregate_fn/proto.rs +++ b/vortex-array/src/aggregate_fn/proto.rs @@ -1,9 +1,6 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -use std::sync::Arc; - -use arcref::ArcRef; use vortex_error::VortexResult; use vortex_error::vortex_bail; use vortex_error::vortex_err; @@ -38,11 +35,11 @@ impl AggregateFnRef { /// /// Note: the serialization format is not stable and may change between versions. pub fn from_proto(proto: &pb::AggregateFn, session: &VortexSession) -> VortexResult { - let agg_fn_id: AggregateFnId = ArcRef::new_arc(Arc::from(proto.id.as_str())); + let agg_fn_id: AggregateFnId = AggregateFnId::new(proto.id.as_str()); let agg_fn = if let Some(plugin) = session.aggregate_fns().registry().find(&agg_fn_id) { plugin.deserialize(proto.metadata(), session)? } else if session.allows_unknown() { - new_foreign_aggregate_fn(agg_fn_id.clone(), proto.metadata().to_vec()) + new_foreign_aggregate_fn(agg_fn_id, proto.metadata().to_vec()) } else { return Err(vortex_err!("unknown aggregate function id: {}", proto.id)); }; @@ -89,7 +86,7 @@ mod tests { type Partial = (); fn id(&self) -> AggregateFnId { - AggregateFnId::new_ref("vortex.test.proto") + AggregateFnId::new("vortex.test.proto") } fn serialize(&self, _options: &Self::Options) -> VortexResult>> { @@ -150,6 +147,10 @@ mod tests { fn finalize(&self, partials: ArrayRef) -> VortexResult { Ok(partials) } + + fn finalize_scalar(&self, _partial: &Self::Partial) -> VortexResult { + vortex_panic!("TestAgg is for serde tests only"); + } } #[test] diff --git a/vortex-array/src/aggregate_fn/session.rs b/vortex-array/src/aggregate_fn/session.rs index 64e66195358..17720398f6c 100644 --- a/vortex-array/src/aggregate_fn/session.rs +++ b/vortex-array/src/aggregate_fn/session.rs @@ -1,11 +1,13 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +use std::any::Any; use std::sync::Arc; use parking_lot::RwLock; use vortex_session::Ref; use vortex_session::SessionExt; +use vortex_session::SessionVar; use vortex_session::registry::Registry; use vortex_utils::aliases::hash_map::HashMap; @@ -22,6 +24,7 @@ use crate::aggregate_fn::fns::sum::Sum; use crate::aggregate_fn::kernels::DynAggregateKernel; use crate::aggregate_fn::kernels::DynGroupedAggregateKernel; use crate::array::ArrayId; +use crate::array::VTable; use crate::arrays::Chunked; use crate::arrays::Dict; use crate::arrays::chunked::compute::aggregate::ChunkedArrayAggregate; @@ -41,6 +44,16 @@ pub struct AggregateFnSession { pub(super) grouped_kernels: RwLock>, } +impl SessionVar for AggregateFnSession { + fn as_any(&self) -> &dyn Any { + self + } + + fn as_any_mut(&mut self) -> &mut dyn Any { + self + } +} + type KernelKey = (ArrayId, Option); impl Default for AggregateFnSession { @@ -61,10 +74,10 @@ impl Default for AggregateFnSession { this.register(Sum); // Register the built-in aggregate kernels. - this.register_aggregate_kernel(Chunked::ID, None, &ChunkedArrayAggregate); - this.register_aggregate_kernel(Dict::ID, Some(MinMax.id()), &DictMinMaxKernel); - this.register_aggregate_kernel(Dict::ID, Some(IsConstant.id()), &DictIsConstantKernel); - this.register_aggregate_kernel(Dict::ID, Some(IsSorted.id()), &DictIsSortedKernel); + this.register_aggregate_kernel(Chunked.id(), None::, &ChunkedArrayAggregate); + this.register_aggregate_kernel(Dict.id(), Some(MinMax.id()), &DictMinMaxKernel); + this.register_aggregate_kernel(Dict.id(), Some(IsConstant.id()), &DictIsConstantKernel); + this.register_aggregate_kernel(Dict.id(), Some(IsSorted.id()), &DictIsSortedKernel); this } @@ -86,11 +99,13 @@ impl AggregateFnSession { /// Register an aggregate function kernel for a specific aggregate function and array type. pub fn register_aggregate_kernel( &self, - array_id: ArrayId, - agg_fn_id: Option, + array_id: impl Into, + agg_fn_id: Option>, kernel: &'static dyn DynAggregateKernel, ) { - self.kernels.write().insert((array_id, agg_fn_id), kernel); + self.kernels + .write() + .insert((array_id.into(), agg_fn_id.map(|id| id.into())), kernel); } } diff --git a/vortex-array/src/aggregate_fn/vtable.rs b/vortex-array/src/aggregate_fn/vtable.rs index 73dd2604700..da6fcbc4165 100644 --- a/vortex-array/src/aggregate_fn/vtable.rs +++ b/vortex-array/src/aggregate_fn/vtable.rs @@ -14,11 +14,9 @@ use vortex_session::VortexSession; use crate::ArrayRef; use crate::Columnar; use crate::ExecutionCtx; -use crate::IntoArray; use crate::aggregate_fn::AggregateFn; use crate::aggregate_fn::AggregateFnId; use crate::aggregate_fn::AggregateFnRef; -use crate::arrays::ConstantArray; use crate::dtype::DType; use crate::scalar::Scalar; @@ -138,12 +136,7 @@ pub trait AggregateFnVTable: 'static + Sized + Clone + Send + Sync { /// /// The provided `state` has dtype as specified by `state_dtype`, the result scalar must have /// dtype as specified by `return_dtype`. - fn finalize_scalar(&self, partial: &Self::Partial) -> VortexResult { - let scalar = self.to_scalar(partial)?; - let array = ConstantArray::new(scalar, 1).into_array(); - let result = self.finalize(array)?; - result.scalar_at(0) - } + fn finalize_scalar(&self, partial: &Self::Partial) -> VortexResult; } #[derive(Clone, Debug, PartialEq, Eq, Hash)] diff --git a/vortex-array/src/array/erased.rs b/vortex-array/src/array/erased.rs index c2fea73dd73..f9dea4e0eef 100644 --- a/vortex-array/src/array/erased.rs +++ b/vortex-array/src/array/erased.rs @@ -15,7 +15,6 @@ use vortex_error::vortex_ensure; use vortex_error::vortex_err; use vortex_error::vortex_panic; use vortex_mask::Mask; -use vortex_session::VortexSession; use crate::AnyCanonical; use crate::Array; @@ -24,9 +23,9 @@ use crate::ArrayHash; use crate::ArrayView; use crate::Canonical; use crate::ExecutionCtx; +use crate::ExecutionResult; use crate::IntoArray; use crate::LEGACY_SESSION; -use crate::ToCanonical; use crate::VTable; use crate::VortexSessionExecute; use crate::aggregate_fn::fns::sum::sum; @@ -42,7 +41,6 @@ use crate::arrays::Primitive; use crate::arrays::SliceArray; use crate::arrays::VarBin; use crate::arrays::VarBinView; -use crate::arrays::bool::BoolArrayExt; use crate::buffer::BufferHandle; use crate::builders::ArrayBuilder; use crate::dtype::DType; @@ -96,6 +94,12 @@ impl ArrayRef { &self.0 } + /// Returns a reference to the inner Arc. + #[inline(always)] + pub(crate) fn inner_mut(&mut self) -> &mut Arc { + &mut self.0 + } + /// Consumes the array reference, returning the owned backing allocation. #[inline(always)] pub(crate) fn into_inner(self) -> Arc { @@ -125,8 +129,6 @@ impl ArrayEq for ArrayRef { self.0.dyn_array_eq(other, precision) } } - -#[allow(clippy::same_name_method)] impl ArrayRef { /// Returns the length of the array. #[inline] @@ -211,24 +213,33 @@ impl ArrayRef { } /// Fetch the scalar at the given index. + #[deprecated( + note = "Use `execute_scalar` instead, which allows passing an execution context for more \ + efficient execution when fetching multiple scalars from the same array." + )] pub fn scalar_at(&self, index: usize) -> VortexResult { + self.execute_scalar(index, &mut LEGACY_SESSION.create_execution_ctx()) + } + + /// Execute the array to extract a scalar at the given index. + pub fn execute_scalar(&self, index: usize, ctx: &mut ExecutionCtx) -> VortexResult { vortex_ensure!(index < self.len(), OutOfBounds: index, 0, self.len()); - if self.is_invalid(index)? { + if self.dtype().is_nullable() && self.is_invalid(index, ctx)? { return Ok(Scalar::null(self.dtype().clone())); } - let scalar = self.0.scalar_at(self, index)?; - vortex_ensure!(self.dtype() == scalar.dtype(), "Scalar dtype mismatch"); + let scalar = self.0.execute_scalar(self, index, ctx)?; + debug_assert_eq!(self.dtype(), scalar.dtype(), "Scalar dtype mismatch"); Ok(scalar) } /// Returns whether the item at `index` is valid. - pub fn is_valid(&self, index: usize) -> VortexResult { + pub fn is_valid(&self, index: usize, ctx: &mut ExecutionCtx) -> VortexResult { vortex_ensure!(index < self.len(), OutOfBounds: index, 0, self.len()); match self.validity()? { Validity::NonNullable | Validity::AllValid => Ok(true), Validity::AllInvalid => Ok(false), Validity::Array(a) => a - .scalar_at(index)? + .execute_scalar(index, ctx)? .as_bool() .value() .ok_or_else(|| vortex_err!("validity value at index {} is null", index)), @@ -236,30 +247,30 @@ impl ArrayRef { } /// Returns whether the item at `index` is invalid. - pub fn is_invalid(&self, index: usize) -> VortexResult { - Ok(!self.is_valid(index)?) + pub fn is_invalid(&self, index: usize, ctx: &mut ExecutionCtx) -> VortexResult { + Ok(!self.is_valid(index, ctx)?) } /// Returns whether all items in the array are valid. - pub fn all_valid(&self) -> VortexResult { + pub fn all_valid(&self, ctx: &mut ExecutionCtx) -> VortexResult { match self.validity()? { Validity::NonNullable | Validity::AllValid => Ok(true), Validity::AllInvalid => Ok(false), - Validity::Array(a) => Ok(a.statistics().compute_min::().unwrap_or(false)), + Validity::Array(a) => Ok(a.statistics().compute_min::(ctx).unwrap_or(false)), } } /// Returns whether the array is all invalid. - pub fn all_invalid(&self) -> VortexResult { + pub fn all_invalid(&self, ctx: &mut ExecutionCtx) -> VortexResult { match self.validity()? { Validity::NonNullable | Validity::AllValid => Ok(false), Validity::AllInvalid => Ok(true), - Validity::Array(a) => Ok(!a.statistics().compute_max::().unwrap_or(true)), + Validity::Array(a) => Ok(!a.statistics().compute_max::(ctx).unwrap_or(true)), } } /// Returns the number of valid elements in the array. - pub fn valid_count(&self) -> VortexResult { + pub fn valid_count(&self, ctx: &mut ExecutionCtx) -> VortexResult { let len = self.len(); if let Some(Precision::Exact(invalid_count)) = self.statistics().get_as::(Stat::NullCount) @@ -271,8 +282,7 @@ impl ArrayRef { Validity::NonNullable | Validity::AllValid => len, Validity::AllInvalid => 0, Validity::Array(a) => { - let mut ctx = LEGACY_SESSION.create_execution_ctx(); - let array_sum = sum(&a, &mut ctx)?; + let array_sum = sum(&a, ctx)?; array_sum .as_primitive() .as_::() @@ -288,8 +298,8 @@ impl ArrayRef { } /// Returns the number of invalid elements in the array. - pub fn invalid_count(&self) -> VortexResult { - Ok(self.len() - self.valid_count()?) + pub fn invalid_count(&self, ctx: &mut ExecutionCtx) -> VortexResult { + Ok(self.len() - self.valid_count(ctx)?) } /// Returns the [`Validity`] of the array. @@ -297,23 +307,18 @@ impl ArrayRef { self.0.validity(self) } - /// Returns the canonical validity mask for the array. - pub fn validity_mask(&self) -> VortexResult { - match self.validity()? { - Validity::NonNullable | Validity::AllValid => Ok(Mask::new_true(self.len())), - Validity::AllInvalid => Ok(Mask::new_false(self.len())), - Validity::Array(a) => Ok(a.to_bool().to_mask()), - } - } - /// Returns the canonical representation of the array. + #[deprecated(note = "use `array.execute::(ctx)` instead")] pub fn into_canonical(self) -> VortexResult { self.execute(&mut LEGACY_SESSION.create_execution_ctx()) } /// Returns the canonical representation of the array. + #[deprecated(note = "use `array.execute::(ctx)` instead")] pub fn to_canonical(&self) -> VortexResult { - self.clone().into_canonical() + #[expect(deprecated)] + let result = self.clone().into_canonical(); + result } /// Writes the array into the canonical builder. @@ -433,6 +438,68 @@ impl ArrayRef { self.with_slots(slots) } + /// Take a slot for executor-owned physical rewrites. + /// + /// On return the produced parent has the taken slot set to `None` + /// callers must put the slot back (typically via [`put_slot_unchecked`]) before the parent is + /// returned from the execution loop. + /// + /// When the `Arc` was shared this allocates a fresh parent. + /// + /// # Safety + /// The caller must put back a slot with the same logical dtype and length before exposing the + /// parent array, and must only use this for physical rewrites. + pub(crate) unsafe fn take_slot_unchecked( + mut self, + slot_idx: usize, + ) -> VortexResult<(ArrayRef, ArrayRef)> { + if let Some(inner) = Arc::get_mut(&mut self.0) { + // SAFETY: ensured by the caller. + let child = unsafe { inner.slots_mut()[slot_idx].take() } + .vortex_expect("take_slot_unchecked cannot take an absent slot"); + return Ok((self, child)); + } + + // Arc is shared: clone the child out and build a fresh parent with slot_idx = None, + // bypassing encoding-level validation so the absent slot does not panic `V::validate`. + let child = self.slots()[slot_idx] + .as_ref() + .vortex_expect("take_slot_unchecked cannot take an absent slot") + .clone(); + + let mut new_slots = self.slots().to_vec(); + new_slots[slot_idx] = None; + + // SAFETY: ensured by the caller — the None slot is either put back or driven to completion + // via the builder path before the parent escapes the executor. + let new_parent = unsafe { self.0.with_slots_unchecked(&self, new_slots) }; + Ok((new_parent, child)) + } + + /// Puts an array into `slot_idx` by either, cloning the inner array if the Arc is not exclusive + /// or replacing the slot in this `ArrayRef`. + /// This is the mirror of [`take_slot_unchecked`]. + /// + /// # Safety + /// The replacement must have the same logical dtype and length as the taken slot, and this + /// must only be used for physical rewrites. + pub(crate) unsafe fn put_slot_unchecked( + mut self, + slot_idx: usize, + replacement: ArrayRef, + ) -> VortexResult { + if let Some(inner) = Arc::get_mut(&mut self.0) { + // # Safety: ensured by the caller. + unsafe { inner.slots_mut()[slot_idx] = Some(replacement) }; + return Ok(self); + } + + let mut slots = self.slots().to_vec(); + slots[slot_idx] = Some(replacement); + let inner = Arc::clone(&self.0); + inner.with_slots(self, slots) + } + /// Returns a new array with the provided slots. /// /// This is only valid for physical rewrites: slot count, presence, logical `DType`, and @@ -484,12 +551,28 @@ impl ArrayRef { self.0.reduce_parent(self, parent, child_idx) } - pub(crate) fn execute_encoding( + pub(crate) fn execute_encoding(self, ctx: &mut ExecutionCtx) -> VortexResult { + let inner = Arc::as_ptr(&self.0); + // Safety the Arc outline the DynArray function call + unsafe { (&*inner).execute(self, ctx) } + } + + /// Execute a single encoding step without applying `Done`-result postconditions. + /// + /// This is for the iterative executor only. It may operate on suspended executor-private + /// arrays whose slots temporarily contain `None`, so the executor itself must interpret + /// `Done`, enforce any `len`/`dtype` invariants, and transfer statistics. + pub(crate) fn execute_encoding_unchecked( self, ctx: &mut ExecutionCtx, - ) -> VortexResult { - let inner = Arc::clone(&self.0); - inner.execute(self, ctx) + ) -> VortexResult { + let inner = Arc::as_ptr(&self.0); + // Safety the Arc outline the DynArray function call + let inner = unsafe { &*inner }; + // SAFETY: `inner` points at the allocation owned by `self.0`. `self` stays alive for the + // duration of the call, so the pointee remains valid. Avoiding an extra `Arc` clone here + // preserves uniqueness so execute-time metadata cursors can use `Arc::get_mut`. + unsafe { inner.execute_unchecked(self, ctx) } } pub fn execute_parent( @@ -563,11 +646,6 @@ impl ArrayRef { self.0.slot_name(self, idx) } - /// Returns the serialized metadata of the array. - pub fn metadata(&self, session: &VortexSession) -> VortexResult>> { - self.0.metadata(self, session) - } - /// Formats a human-readable metadata description. pub fn metadata_fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { self.0.metadata_fmt(f) @@ -618,6 +696,7 @@ impl Matcher for V { fn try_match<'a>(array: &'a ArrayRef) -> Option> { let inner = array.0.as_any().downcast_ref::>()?; + // # Safety checked by `downcast_ref`. Some(unsafe { ArrayView::new_unchecked(array, &inner.data) }) } } diff --git a/vortex-array/src/array/foreign.rs b/vortex-array/src/array/foreign.rs index b29d76ba2b8..3233d2f4228 100644 --- a/vortex-array/src/array/foreign.rs +++ b/vortex-array/src/array/foreign.rs @@ -94,7 +94,7 @@ impl VTable for ForeignArray { type ValidityVTable = ForeignValidityVTable; fn id(&self) -> ArrayId { - self.id.clone() + self.id } fn validate( diff --git a/vortex-array/src/array/mod.rs b/vortex-array/src/array/mod.rs index 9ef3e57cab2..1ade2a5a083 100644 --- a/vortex-array/src/array/mod.rs +++ b/vortex-array/src/array/mod.rs @@ -7,18 +7,15 @@ use std::fmt::Formatter; use std::hash::Hash; use std::hash::Hasher; -use arcref::ArcRef; use vortex_buffer::ByteBuffer; use vortex_error::VortexExpect; use vortex_error::VortexResult; use vortex_error::vortex_ensure; use vortex_error::vortex_err; use vortex_error::vortex_panic; -use vortex_session::VortexSession; +use vortex_session::registry::Id; use crate::ExecutionCtx; -use crate::LEGACY_SESSION; -use crate::VortexSessionExecute; use crate::buffer::BufferHandle; use crate::builders::ArrayBuilder; use crate::dtype::DType; @@ -59,6 +56,9 @@ pub(crate) trait DynArray: 'static + private::Sealed + Send + Sync + Debug { /// Returns the array as a reference to a generic [`Any`] trait object. fn as_any(&self) -> &dyn Any; + /// Returns the array as a mutable reference to a generic [`Any`] trait object. + fn as_any_mut(&mut self) -> &mut dyn Any; + /// Converts an owned array allocation into an owned [`Any`] allocation for downcasting. fn into_any_arc(self: std::sync::Arc) -> std::sync::Arc; @@ -71,14 +71,16 @@ pub(crate) trait DynArray: 'static + private::Sealed + Send + Sync + Debug { /// Returns the slots of the array. fn slots(&self) -> &[Option]; + /// Returns mutable slots of the array. + /// + /// # Safety: any slot (Some(child)) that replaces an existing slot must have a compatible + /// DType and length. Currently compatible means equal, but there is no reason why that must + /// be the case. + unsafe fn slots_mut(&mut self) -> &mut [Option]; + /// Returns the encoding ID of the array. fn encoding_id(&self) -> ArrayId; - /// Fetch the scalar at the given index. - /// - /// This method panics if the index is out of bounds for the array. - fn scalar_at(&self, this: &ArrayRef, index: usize) -> VortexResult; - /// Returns the [`Validity`] of the array. fn validity(&self, this: &ArrayRef) -> VortexResult; @@ -132,10 +134,6 @@ pub(crate) trait DynArray: 'static + private::Sealed + Send + Sync + Debug { /// Returns the name of the slot at the given index. fn slot_name(&self, this: &ArrayRef, idx: usize) -> String; - /// Returns the serialized metadata of the array, or `None` if the array does not - /// support serialization. - fn metadata(&self, this: &ArrayRef, session: &VortexSession) -> VortexResult>>; - /// Formats a human-readable metadata description. fn metadata_fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result; @@ -148,6 +146,24 @@ pub(crate) trait DynArray: 'static + private::Sealed + Send + Sync + Debug { /// Returns a new array with the given slots. fn with_slots(&self, this: ArrayRef, slots: Vec>) -> VortexResult; + /// Returns a new array with the given slots, bypassing encoding-level validation. + /// + /// Used by the executor to temporarily carry an array that has had one of its child slots + /// taken out (leaving `None`) without panicking `V::validate`. The caller must ensure the + /// missing slot is filled back in (via `put_slot_unchecked`) or driven to completion by the + /// builder path before the array becomes externally observable. + /// + /// # Safety + /// + /// The array returned may have slots whose content does not match the encoding's normal + /// invariants. Callers must re-establish those invariants before handing the array to + /// anything outside the executor. + unsafe fn with_slots_unchecked( + &self, + this: &ArrayRef, + slots: Vec>, + ) -> ArrayRef; + /// Attempt to reduce the array to a simpler representation. fn reduce(&self, this: &ArrayRef) -> VortexResult>; @@ -160,8 +176,30 @@ pub(crate) trait DynArray: 'static + private::Sealed + Send + Sync + Debug { ) -> VortexResult>; /// Execute the array by taking a single encoding-specific execution step. + /// + /// This is the checked entry point. If the encoding reports + /// [`ExecutionStep::Done`](crate::ExecutionStep::Done), implementations must validate that the + /// returned array preserves this array's logical `len` and `dtype`, and must transfer this + /// array's statistics to the returned array. fn execute(&self, this: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult; + /// Execute the array by taking a single encoding-specific execution step without applying + /// `Done`-result postconditions. + /// + /// This exists for the iterative executor, which may call into `execute` on suspended + /// executor-private arrays whose slots temporarily contain `None`. In that mode the executor + /// itself is responsible for deciding when a `Done` result represents a real logical array, + /// enforcing any `len`/`dtype` invariants, and transferring statistics. + /// + /// # Safety + /// The `array` returned should have it's `DType` and len checked + /// (optionally it should have its stats propagated from `this`). + unsafe fn execute_unchecked( + &self, + this: ArrayRef, + ctx: &mut ExecutionCtx, + ) -> VortexResult; + /// Attempt to execute the parent of this array. fn execute_parent( &self, @@ -170,6 +208,16 @@ pub(crate) trait DynArray: 'static + private::Sealed + Send + Sync + Debug { child_idx: usize, ctx: &mut ExecutionCtx, ) -> VortexResult>; + + /// Execute the scalar at the given index. + /// + /// This method panics if the index is out of bounds for the array. + fn execute_scalar( + &self, + this: &ArrayRef, + index: usize, + ctx: &mut ExecutionCtx, + ) -> VortexResult; } /// Trait for converting a type into a Vortex [`ArrayRef`]. @@ -198,6 +246,10 @@ impl DynArray for ArrayInner { self } + fn as_any_mut(&mut self) -> &mut dyn Any { + self + } + fn into_any_arc(self: std::sync::Arc) -> std::sync::Arc { self } @@ -214,17 +266,12 @@ impl DynArray for ArrayInner { &self.slots } - fn encoding_id(&self) -> ArrayId { - self.vtable.id() + unsafe fn slots_mut(&mut self) -> &mut [Option] { + &mut self.slots } - fn scalar_at(&self, this: &ArrayRef, index: usize) -> VortexResult { - let view = unsafe { ArrayView::new_unchecked(this, &self.data) }; - >::scalar_at( - view, - index, - &mut LEGACY_SESSION.create_execution_ctx(), - ) + fn encoding_id(&self) -> ArrayId { + self.vtable.id() } fn validity(&self, this: &ArrayRef) -> VortexResult { @@ -341,11 +388,6 @@ impl DynArray for ArrayInner { V::slot_name(view, idx) } - fn metadata(&self, this: &ArrayRef, session: &VortexSession) -> VortexResult>> { - let view = unsafe { ArrayView::new_unchecked(this, &self.data) }; - V::serialize(view, session) - } - fn metadata_fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { std::fmt::Display::fmt(&self.data, f) } @@ -392,6 +434,26 @@ impl DynArray for ArrayInner { .into_array()) } + unsafe fn with_slots_unchecked( + &self, + this: &ArrayRef, + slots: Vec>, + ) -> ArrayRef { + // SAFETY: we intentionally skip `V::validate` here. Caller guarantees that the resulting + // array is either repaired or not externally observed. + let inner = unsafe { + ArrayInner::::from_data_unchecked( + self.vtable.clone(), + this.dtype().clone(), + self.len, + self.data.clone(), + slots, + self.stats.clone(), + ) + }; + ArrayRef::from_inner(std::sync::Arc::new(inner)) + } + fn reduce(&self, this: &ArrayRef) -> VortexResult> { let view = unsafe { ArrayView::new_unchecked(this, &self.data) }; let Some(reduced) = V::reduce(view)? else { @@ -442,12 +504,8 @@ impl DynArray for ArrayInner { fn execute(&self, this: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { let len = this.len(); let dtype = this.dtype().clone(); - let stats = this.statistics().to_owned(); - - let typed = Array::::try_from_array_ref(this) - .map_err(|_| vortex_err!("Failed to downcast array for execute")) - .vortex_expect("Failed to downcast array for execute"); - let result = V::execute(typed, ctx)?; + let stats = this.statistics().to_array_stats(); + let result = unsafe { self.execute_unchecked(this, ctx)? }; if matches!(result.step(), ExecutionStep::Done) { if cfg!(debug_assertions) { @@ -463,12 +521,26 @@ impl DynArray for ArrayInner { ); } - result.array().statistics().set_iter(stats.into_iter()); + result + .array() + .statistics() + .set_iter(crate::stats::StatsSet::from(stats).into_iter()); } Ok(result) } + unsafe fn execute_unchecked( + &self, + this: ArrayRef, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + let typed = Array::::try_from_array_ref(this) + .map_err(|_| vortex_err!("Failed to downcast array for execute")) + .vortex_expect("Failed to downcast array for execute"); + V::execute(typed, ctx) + } + fn execute_parent( &self, this: &ArrayRef, @@ -494,6 +566,16 @@ impl DynArray for ArrayInner { Ok(Some(result)) } + + fn execute_scalar( + &self, + this: &ArrayRef, + index: usize, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + let view = unsafe { ArrayView::new_unchecked(this, &self.data) }; + >::scalar_at(view, index, ctx) + } } /// Wrapper around `&mut dyn Hasher` that implements `Hasher` (and is `Sized`). @@ -510,4 +592,4 @@ impl Hasher for HasherWrapper<'_> { } /// ArrayId is a globally unique name for the array's vtable. -pub type ArrayId = ArcRef; +pub type ArrayId = Id; diff --git a/vortex-array/src/array/plugin.rs b/vortex-array/src/array/plugin.rs index bff76a30c59..66845eb9a0a 100644 --- a/vortex-array/src/array/plugin.rs +++ b/vortex-array/src/array/plugin.rs @@ -1,6 +1,9 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +use std::fmt; +use std::fmt::Debug; +use std::fmt::Formatter; use std::sync::Arc; use vortex_error::VortexResult; @@ -27,6 +30,9 @@ pub type ArrayPluginRef = Arc; /// [`deserialize`]: ArrayPlugin::deserialize pub trait ArrayPlugin: 'static + Send + Sync { /// Returns the ID for this array encoding. + /// + /// During serde, this is the key the registry uses to find + /// this plugin instance and call the appropriate method on it. fn id(&self) -> ArrayId; /// Serialize the array metadata. @@ -40,7 +46,6 @@ pub trait ArrayPlugin: 'static + Send + Sync { /// /// The returned array doesn't necessary have to match this plugin's encoding ID. This is /// useful for implementing back-compat logic and deserializing arrays into the new version. - #[allow(clippy::too_many_arguments)] fn deserialize( &self, dtype: &DType, @@ -50,10 +55,18 @@ pub trait ArrayPlugin: 'static + Send + Sync { children: &dyn ArrayChildren, session: &VortexSession, ) -> VortexResult; + + /// Can this plugin emit an array with the given encoding. + /// + /// By default, this is just the [ID][Self::id] of the plugin, but + /// can be overridden if this plugin instance supports reading/writing multiple arrays. + fn is_supported_encoding(&self, id: &ArrayId) -> bool { + self.id() == *id + } } -impl std::fmt::Debug for dyn ArrayPlugin { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { +impl Debug for dyn ArrayPlugin { + fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { f.debug_tuple("ArrayPlugin").field(&self.id()).finish() } } diff --git a/vortex-array/src/array/typed.rs b/vortex-array/src/array/typed.rs index 0d8239af589..ffdbcc6f161 100644 --- a/vortex-array/src/array/typed.rs +++ b/vortex-array/src/array/typed.rs @@ -15,7 +15,10 @@ use std::sync::Arc; use vortex_error::VortexResult; use crate::ArrayRef; +use crate::ExecutionCtx; use crate::IntoArray; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::array::ArrayId; use crate::array::ArrayView; use crate::array::VTable; @@ -192,7 +195,6 @@ pub struct Array { _phantom: PhantomData, } -#[allow(clippy::same_name_method)] impl Array { /// Create a typed array from explicit construction parameters. pub fn try_from_parts(new: ArrayParts) -> VortexResult { @@ -229,7 +231,6 @@ impl Array { /// /// # Safety /// Caller must ensure the `ArrayRef` contains an `ArrayInner`. - #[allow(dead_code)] pub(crate) unsafe fn from_array_ref_unchecked(array: ArrayRef) -> Self { Self { inner: array, @@ -279,6 +280,14 @@ impl Array { &self.downcast_inner().data } + /// Try to fetch a mut ref to the inner ArrayData. + pub fn data_mut(&mut self) -> Option<&mut V::ArrayData> { + let m = self.inner.inner_mut(); + let inner = Arc::get_mut(m)?; + let array_inner = inner.as_any_mut().downcast_mut::>(); + Some(&mut array_inner?.data) + } + /// Returns the full typed array construction parts if this handle owns the allocation. pub fn try_into_parts(self) -> Result, Self> { let Self { inner, _phantom } = self; @@ -342,61 +351,67 @@ impl Array { /// Public API methods that shadow `DynArray` / `ArrayRef` methods. impl Array { - #[allow(clippy::same_name_method)] pub fn slice(&self, range: std::ops::Range) -> VortexResult { self.inner.slice(range) } - #[allow(clippy::same_name_method)] + #[deprecated( + note = "Use `execute_scalar` instead, which allows passing an execution context for more \ + efficient execution when fetching multiple scalars from the same array." + )] pub fn scalar_at(&self, index: usize) -> VortexResult { - self.inner.scalar_at(index) + self.inner + .execute_scalar(index, &mut LEGACY_SESSION.create_execution_ctx()) + } + + /// Execute the array to extract a scalar at the given index. + pub fn execute_scalar( + &self, + index: usize, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + self.inner.execute_scalar(index, ctx) } - #[allow(clippy::same_name_method)] pub fn filter(&self, mask: vortex_mask::Mask) -> VortexResult { self.inner.filter(mask) } - #[allow(clippy::same_name_method)] pub fn take(&self, indices: ArrayRef) -> VortexResult { self.inner.take(indices) } - #[allow(clippy::same_name_method)] pub fn validity(&self) -> VortexResult { self.inner.validity() } - #[allow(clippy::same_name_method)] - pub fn is_valid(&self, index: usize) -> VortexResult { - self.inner.is_valid(index) + pub fn is_valid(&self, index: usize, ctx: &mut ExecutionCtx) -> VortexResult { + self.inner.is_valid(index, ctx) } - #[allow(clippy::same_name_method)] - pub fn is_invalid(&self, index: usize) -> VortexResult { - self.inner.is_invalid(index) + pub fn is_invalid(&self, index: usize, ctx: &mut ExecutionCtx) -> VortexResult { + self.inner.is_invalid(index, ctx) } - #[allow(clippy::same_name_method)] - pub fn all_valid(&self) -> VortexResult { - self.inner.all_valid() + pub fn all_valid(&self, ctx: &mut ExecutionCtx) -> VortexResult { + self.inner.all_valid(ctx) } - #[allow(clippy::same_name_method)] - pub fn all_invalid(&self) -> VortexResult { - self.inner.all_invalid() + pub fn all_invalid(&self, ctx: &mut ExecutionCtx) -> VortexResult { + self.inner.all_invalid(ctx) } - #[allow(clippy::same_name_method)] + #[deprecated(note = "Use Array::::execute::() instead")] pub fn to_canonical(&self) -> VortexResult { - self.inner.to_canonical() + #[expect(deprecated)] + let result = self.inner.to_canonical(); + result } pub fn nbytes(&self) -> u64 { self.inner.nbytes() } - #[allow(clippy::same_name_method)] pub fn nbuffers(&self) -> usize { self.inner.nbuffers() } @@ -405,29 +420,21 @@ impl Array { self.inner.as_constant() } - #[allow(clippy::same_name_method)] - pub fn valid_count(&self) -> VortexResult { - self.inner.valid_count() + pub fn valid_count(&self, ctx: &mut ExecutionCtx) -> VortexResult { + self.inner.valid_count(ctx) } - #[allow(clippy::same_name_method)] - pub fn invalid_count(&self) -> VortexResult { - self.inner.invalid_count() + pub fn invalid_count(&self, ctx: &mut ExecutionCtx) -> VortexResult { + self.inner.invalid_count(ctx) } - #[allow(clippy::same_name_method)] pub fn append_to_builder( &self, builder: &mut dyn crate::builders::ArrayBuilder, - ctx: &mut crate::ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult<()> { self.inner.append_to_builder(builder, ctx) } - - #[allow(clippy::same_name_method)] - pub fn validity_mask(&self) -> VortexResult { - self.inner.validity_mask() - } } impl Deref for Array { diff --git a/vortex-array/src/array/vtable/mod.rs b/vortex-array/src/array/vtable/mod.rs index 2b65853bb88..7006c55b076 100644 --- a/vortex-array/src/array/vtable/mod.rs +++ b/vortex-array/src/array/vtable/mod.rs @@ -254,7 +254,7 @@ pub fn validity_to_child(validity: &Validity, len: usize) -> Option { /// /// This is the inverse of [`validity_to_child`]. #[inline] -pub fn child_to_validity(child: &Option, nullability: Nullability) -> Validity { +pub fn child_to_validity(child: Option<&ArrayRef>, nullability: Nullability) -> Validity { match child { Some(arr) => { // Detect constant bool arrays created by validity_to_child. diff --git a/vortex-array/src/arrays/arbitrary.rs b/vortex-array/src/arrays/arbitrary.rs index 16099d5c2c1..bfe237bc20b 100644 --- a/vortex-array/src/arrays/arbitrary.rs +++ b/vortex-array/src/arrays/arbitrary.rs @@ -2,9 +2,11 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors use std::iter; +use std::ops::RangeInclusive; use std::sync::Arc; use arbitrary::Arbitrary; +use arbitrary::Error::IncorrectFormat; use arbitrary::Result; use arbitrary::Unstructured; use vortex_buffer::BitBuffer; @@ -13,7 +15,8 @@ use vortex_error::VortexExpect; use crate::ArrayRef; use crate::IntoArray; -use crate::ToCanonical; +#[expect(deprecated)] +use crate::ToCanonical as _; use crate::arrays::BoolArray; use crate::arrays::ChunkedArray; use crate::arrays::NullArray; @@ -40,16 +43,37 @@ use crate::validity::Validity; #[derive(Clone, Debug)] pub struct ArbitraryArray(pub ArrayRef); -impl<'a> Arbitrary<'a> for ArbitraryArray { - fn arbitrary(u: &mut Unstructured<'a>) -> Result { - let dtype = u.arbitrary()?; - Self::arbitrary_with(u, None, &dtype) - } +/// Trait for generating arbitrary values with a caller-provided configuration. +pub trait ArbitraryWith<'a, C>: Sized { + /// Generate an arbitrary value using the provided configuration. + fn arbitrary_with_config(u: &mut Unstructured<'a>, config: &C) -> Result; +} + +/// Configuration for arbitrary array generation. +#[derive(Clone, Debug)] +pub struct ArbitraryArrayConfig { + /// Fixed dtype, or `None` to generate one from [`Unstructured`]. + pub dtype: Option, + /// Inclusive range for the total array length. + pub len: RangeInclusive, } -impl ArbitraryArray { - pub fn arbitrary_with(u: &mut Unstructured, len: Option, dtype: &DType) -> Result { - random_array(u, dtype, len).map(ArbitraryArray) +impl<'a> ArbitraryWith<'a, ArbitraryArrayConfig> for ArbitraryArray { + fn arbitrary_with_config( + u: &mut Unstructured<'a>, + config: &ArbitraryArrayConfig, + ) -> Result { + if config.len.is_empty() { + return Err(IncorrectFormat); + } + + let dtype = match &config.dtype { + Some(dtype) => dtype.clone(), + None => u.arbitrary()?, + }; + let len = u.int_in_range(config.len.clone())?; + + random_array(u, &dtype, Some(len)).map(ArbitraryArray) } } @@ -105,10 +129,14 @@ fn random_array_chunk( PType::I16 => random_primitive::(u, *n, chunk_len), PType::I32 => random_primitive::(u, *n, chunk_len), PType::I64 => random_primitive::(u, *n, chunk_len), - PType::F16 => Ok(random_primitive::(u, *n, chunk_len)? - .to_primitive() - .reinterpret_cast(PType::F16) - .into_array()), + PType::F16 => { + #[expect(deprecated)] + let prim = random_primitive::(u, *n, chunk_len)? + .to_primitive() + .reinterpret_cast(PType::F16) + .into_array(); + Ok(prim) + } PType::F32 => random_primitive::(u, *n, chunk_len), PType::F64 => random_primitive::(u, *n, chunk_len), }, diff --git a/vortex-array/src/arrays/assertions.rs b/vortex-array/src/arrays/assertions.rs index 990dde4b37d..389baa23857 100644 --- a/vortex-array/src/arrays/assertions.rs +++ b/vortex-array/src/arrays/assertions.rs @@ -30,8 +30,12 @@ fn execute_to_canonical(array: ArrayRef, ctx: &mut ExecutionCtx) -> ArrayRef { #[expect(clippy::unwrap_used)] fn find_mismatched_indices(left: &ArrayRef, right: &ArrayRef) -> Vec { assert_eq!(left.len(), right.len()); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); (0..left.len()) - .filter(|i| left.scalar_at(*i).unwrap() != right.scalar_at(*i).unwrap()) + .filter(|i| { + left.execute_scalar(*i, &mut ctx).unwrap() + != right.execute_scalar(*i, &mut ctx).unwrap() + }) .collect() } @@ -49,9 +53,13 @@ fn find_mismatched_indices(left: &ArrayRef, right: &ArrayRef) -> Vec { macro_rules! assert_nth_scalar { ($arr:expr, $n:expr, $expected:expr) => {{ use $crate::IntoArray as _; + use $crate::LEGACY_SESSION; + use $crate::VortexSessionExecute as _; let arr_ref: $crate::ArrayRef = $crate::IntoArray::into_array($arr.clone()); assert_eq!( - arr_ref.scalar_at($n).unwrap(), + arr_ref + .execute_scalar($n, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), $expected.try_into().unwrap() ); }}; @@ -68,12 +76,19 @@ macro_rules! assert_nth_scalar { #[macro_export] macro_rules! assert_nth_scalar_is_null { ($arr:expr, $n:expr) => {{ + use $crate::LEGACY_SESSION; + use $crate::VortexSessionExecute as _; let arr_ref: $crate::ArrayRef = $crate::IntoArray::into_array($arr.clone()); assert!( - arr_ref.scalar_at($n).unwrap().is_null(), + arr_ref + .execute_scalar($n, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null(), "expected scalar at index {} to be null, but was {:?}", $n, - arr_ref.scalar_at($n).unwrap() + arr_ref + .execute_scalar($n, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() ); }}; } @@ -105,9 +120,7 @@ macro_rules! assert_arrays_eq { right.display_values() ); - #[allow(deprecated)] let left = left.clone(); - #[allow(deprecated)] let right = right.clone(); $crate::arrays::assert_arrays_eq_impl(&left, &right); }}; @@ -116,7 +129,7 @@ macro_rules! assert_arrays_eq { /// Implementation of `assert_arrays_eq!` — called by the macro after converting inputs to /// `ArrayRef`. #[track_caller] -#[allow(clippy::panic)] +#[expect(clippy::panic)] pub fn assert_arrays_eq_impl(left: &ArrayRef, right: &ArrayRef) { let executed = execute_to_canonical(left.clone(), &mut LEGACY_SESSION.create_execution_ctx()); diff --git a/vortex-array/src/arrays/bool/array.rs b/vortex-array/src/arrays/bool/array.rs index 8a7eed44037..0cfc55869b1 100644 --- a/vortex-array/src/arrays/bool/array.rs +++ b/vortex-array/src/arrays/bool/array.rs @@ -13,6 +13,7 @@ use vortex_error::vortex_ensure; use vortex_mask::Mask; use crate::ArrayRef; +use crate::ExecutionCtx; use crate::IntoArray; use crate::array::Array; use crate::array::ArrayParts; @@ -47,7 +48,7 @@ pub(super) const SLOT_NAMES: [&str; NUM_SLOTS] = ["validity"]; /// ``` /// # fn main() -> vortex_error::VortexResult<()> { /// use vortex_array::arrays::BoolArray; -/// use vortex_array::IntoArray; +/// use vortex_array::{IntoArray, LEGACY_SESSION, VortexSessionExecute}; /// /// // Create from iterator using FromIterator impl /// let array: BoolArray = [true, false, true, false].into_iter().collect(); @@ -57,7 +58,8 @@ pub(super) const SLOT_NAMES: [&str; NUM_SLOTS] = ["validity"]; /// assert_eq!(sliced.len(), 2); /// /// // Access individual values -/// let value = array.scalar_at(0).unwrap(); +/// let mut ctx = LEGACY_SESSION.create_execution_ctx(); +/// let value = array.execute_scalar(0, &mut ctx).unwrap(); /// assert_eq!(value, true.into()); /// # Ok(()) /// # } @@ -89,11 +91,10 @@ pub trait BoolArrayExt: TypedArrayRef { } fn validity(&self) -> Validity { - child_to_validity(&self.as_ref().slots()[VALIDITY_SLOT], self.nullability()) - } - - fn bool_validity_mask(&self) -> Mask { - self.validity().to_mask(self.as_ref().len()) + child_to_validity( + self.as_ref().slots()[VALIDITY_SLOT].as_ref(), + self.nullability(), + ) } fn to_bit_buffer(&self) -> BitBuffer { @@ -101,23 +102,26 @@ pub trait BoolArrayExt: TypedArrayRef { BitBuffer::new_with_offset(buffer, self.as_ref().len(), self.offset) } - fn maybe_to_mask(&self) -> VortexResult> { + fn maybe_execute_mask(&self, ctx: &mut ExecutionCtx) -> VortexResult> { let all_valid = match &self.validity() { Validity::NonNullable | Validity::AllValid => true, Validity::AllInvalid => false, - Validity::Array(a) => a.statistics().compute_min::().unwrap_or(false), + Validity::Array(a) => a.statistics().compute_min::(ctx).unwrap_or(false), }; Ok(all_valid.then(|| Mask::from_buffer(self.to_bit_buffer()))) } - fn to_mask(&self) -> Mask { - self.maybe_to_mask() + fn execute_mask(&self, ctx: &mut ExecutionCtx) -> Mask { + self.maybe_execute_mask(ctx) .vortex_expect("failed to check validity") .vortex_expect("cannot convert nullable boolean array to mask") } - fn to_mask_fill_null_false(&self) -> Mask { - let validity_mask = self.bool_validity_mask(); + fn to_mask_fill_null_false(&self, ctx: &mut ExecutionCtx) -> Mask { + let validity_mask = self + .validity() + .execute_mask(self.as_ref().len(), ctx) + .vortex_expect("Failed to compute validity mask"); let buffer = match validity_mask { Mask::AllTrue(_) => self.to_bit_buffer(), Mask::AllFalse(_) => return Mask::new_false(self.as_ref().len()), @@ -369,7 +373,11 @@ mod tests { #[test] fn bool_array() { let arr = BoolArray::from_iter([true, false, true]); - let scalar = bool::try_from(&arr.scalar_at(0).unwrap()).unwrap(); + let scalar = bool::try_from( + &arr.execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + ) + .unwrap(); assert!(scalar); } @@ -379,9 +387,17 @@ mod tests { assert!(matches!(arr.validity(), Ok(Validity::AllValid))); - let scalar = bool::try_from(&arr.scalar_at(0).unwrap()).unwrap(); + let scalar = bool::try_from( + &arr.execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + ) + .unwrap(); assert!(scalar); - let scalar = bool::try_from(&arr.scalar_at(1).unwrap()).unwrap(); + let scalar = bool::try_from( + &arr.execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + ) + .unwrap(); assert!(!scalar); } @@ -389,19 +405,35 @@ mod tests { fn test_bool_from_iter() { let arr = BoolArray::from_iter([Some(true), Some(true), None, Some(false), None]); - let scalar = bool::try_from(&arr.scalar_at(0).unwrap()).unwrap(); + let scalar = bool::try_from( + &arr.execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + ) + .unwrap(); assert!(scalar); - let scalar = bool::try_from(&arr.scalar_at(1).unwrap()).unwrap(); + let scalar = bool::try_from( + &arr.execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + ) + .unwrap(); assert!(scalar); - let scalar = arr.scalar_at(2).unwrap(); + let scalar = arr + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(scalar.is_null()); - let scalar = bool::try_from(&arr.scalar_at(3).unwrap()).unwrap(); + let scalar = bool::try_from( + &arr.execute_scalar(3, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + ) + .unwrap(); assert!(!scalar); - let scalar = arr.scalar_at(4).unwrap(); + let scalar = arr + .execute_scalar(4, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(scalar.is_null()); } diff --git a/vortex-array/src/arrays/bool/compute/cast.rs b/vortex-array/src/arrays/bool/compute/cast.rs index 3b5fbea7607..6bfaa03b9db 100644 --- a/vortex-array/src/arrays/bool/compute/cast.rs +++ b/vortex-array/src/arrays/bool/compute/cast.rs @@ -4,24 +4,48 @@ use vortex_error::VortexResult; use crate::ArrayRef; +use crate::ExecutionCtx; use crate::IntoArray; use crate::array::ArrayView; use crate::arrays::Bool; use crate::arrays::BoolArray; use crate::arrays::bool::BoolArrayExt; use crate::dtype::DType; +use crate::scalar_fn::fns::cast::CastKernel; use crate::scalar_fn::fns::cast::CastReduce; impl CastReduce for Bool { fn cast(array: ArrayView<'_, Bool>, dtype: &DType) -> VortexResult> { - if !matches!(dtype, DType::Bool(_)) { + if !dtype.is_boolean() { return Ok(None); } - let new_nullability = dtype.nullability(); - let new_validity = array + let Some(new_validity) = array .validity()? - .cast_nullability(new_nullability, array.len())?; + .trivial_cast_nullability(dtype.nullability(), array.len())? + else { + return Ok(None); + }; + Ok(Some( + BoolArray::new(array.to_bit_buffer(), new_validity).into_array(), + )) + } +} + +impl CastKernel for Bool { + fn cast( + array: ArrayView<'_, Bool>, + dtype: &DType, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + if !dtype.is_boolean() { + return Ok(None); + } + + let new_validity = + array + .validity()? + .cast_nullability(dtype.nullability(), array.len(), ctx)?; Ok(Some( BoolArray::new(array.to_bit_buffer(), new_validity).into_array(), )) @@ -30,14 +54,23 @@ impl CastReduce for Bool { #[cfg(test)] mod tests { + use std::sync::LazyLock; + use rstest::rstest; + use vortex_session::VortexSession; + use crate::Canonical; use crate::IntoArray; + use crate::VortexSessionExecute; use crate::arrays::BoolArray; use crate::builtins::ArrayBuiltins; use crate::compute::conformance::cast::test_cast_conformance; use crate::dtype::DType; use crate::dtype::Nullability; + use crate::session::ArraySession; + + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); #[test] fn try_cast_bool_success() { @@ -51,12 +84,16 @@ mod tests { } #[test] - #[should_panic] fn try_cast_bool_fail() { + // When the validity array's min stat is not cached, the reduce rule defers and the + // failure surfaces during execution via the kernel (cast_nullability -> compute_min). let bool = BoolArray::from_iter(vec![Some(true), Some(false), None]); - bool.into_array() + let mut ctx = SESSION.create_execution_ctx(); + let result = bool + .into_array() .cast(DType::Bool(Nullability::NonNullable)) - .unwrap(); + .and_then(|a| a.execute::(&mut ctx).map(|c| c.into_array())); + assert!(result.is_err(), "Expected error, got: {result:?}"); } #[rstest] diff --git a/vortex-array/src/arrays/bool/compute/fill_null.rs b/vortex-array/src/arrays/bool/compute/fill_null.rs index f2489f9d073..59b9ca0b717 100644 --- a/vortex-array/src/arrays/bool/compute/fill_null.rs +++ b/vortex-array/src/arrays/bool/compute/fill_null.rs @@ -51,7 +51,8 @@ mod tests { use crate::arrays::BoolArray; use crate::arrays::bool::BoolArrayExt; use crate::builtins::ArrayBuiltins; - use crate::canonical::ToCanonical; + #[expect(deprecated)] + use crate::canonical::ToCanonical as _; use crate::dtype::DType; use crate::dtype::Nullability; use crate::scalar::Scalar; @@ -65,6 +66,7 @@ mod tests { BitBuffer::from_iter([true, true, false, false]), Validity::from_iter([true, false, true, false]), ); + #[expect(deprecated)] let non_null_array = bool_array .into_array() .fill_null(Scalar::from(fill_value)) diff --git a/vortex-array/src/arrays/bool/compute/take.rs b/vortex-array/src/arrays/bool/compute/take.rs index 19b1508fd84..1f679445254 100644 --- a/vortex-array/src/arrays/bool/compute/take.rs +++ b/vortex-array/src/arrays/bool/compute/take.rs @@ -28,7 +28,7 @@ impl TakeExecute for Bool { indices: &ArrayRef, ctx: &mut ExecutionCtx, ) -> VortexResult> { - let indices_nulls_zeroed = match indices.validity_mask()? { + let indices_nulls_zeroed = match indices.validity()?.execute_mask(indices.len(), ctx)? { Mask::AllTrue(_) => indices.clone(), Mask::AllFalse(_) => { return Ok(Some( @@ -84,7 +84,8 @@ mod test { use vortex_buffer::buffer; use crate::IntoArray as _; - use crate::ToCanonical; + #[expect(deprecated)] + use crate::ToCanonical as _; use crate::arrays::BoolArray; use crate::arrays::PrimitiveArray; use crate::arrays::bool::BoolArrayExt; @@ -102,6 +103,7 @@ mod test { Some(false), ]); + #[expect(deprecated)] let b = reference .take(buffer![0, 3, 4].into_array()) .unwrap() diff --git a/vortex-array/src/arrays/bool/test_harness.rs b/vortex-array/src/arrays/bool/test_harness.rs index 52cffafdbd9..4e13d62e3ef 100644 --- a/vortex-array/src/arrays/bool/test_harness.rs +++ b/vortex-array/src/arrays/bool/test_harness.rs @@ -4,13 +4,20 @@ use vortex_error::VortexExpect; use vortex_error::vortex_panic; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::arrays::BoolArray; use crate::arrays::bool::BoolArrayExt; impl BoolArray { pub fn opt_bool_vec(&self) -> Vec> { - self.validity_mask() - .vortex_expect("validity_mask") + self.validity() + .vortex_expect("failed to get validity") + .execute_mask( + self.as_ref().len(), + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .vortex_expect("Failed to compute validity mask") .to_bit_buffer() .iter() .zip(self.to_bit_buffer().iter()) @@ -19,8 +26,13 @@ impl BoolArray { } pub fn bool_vec(&self) -> Vec { - self.validity_mask() - .vortex_expect("validity_mask") + self.validity() + .vortex_expect("failed to get validity") + .execute_mask( + self.as_ref().len(), + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .vortex_expect("Failed to compute validity mask") .to_bit_buffer() .iter() .zip(self.to_bit_buffer().iter()) diff --git a/vortex-array/src/arrays/bool/vtable/kernel.rs b/vortex-array/src/arrays/bool/vtable/kernel.rs index e8fa5f30e9c..4f1047a5132 100644 --- a/vortex-array/src/arrays/bool/vtable/kernel.rs +++ b/vortex-array/src/arrays/bool/vtable/kernel.rs @@ -4,9 +4,11 @@ use crate::arrays::Bool; use crate::arrays::dict::TakeExecuteAdaptor; use crate::kernel::ParentKernelSet; +use crate::scalar_fn::fns::cast::CastExecuteAdaptor; use crate::scalar_fn::fns::fill_null::FillNullExecuteAdaptor; pub(super) const PARENT_KERNELS: ParentKernelSet = ParentKernelSet::new(&[ + ParentKernelSet::lift(&CastExecuteAdaptor(Bool)), ParentKernelSet::lift(&FillNullExecuteAdaptor(Bool)), ParentKernelSet::lift(&TakeExecuteAdaptor(Bool)), ]); diff --git a/vortex-array/src/arrays/bool/vtable/mod.rs b/vortex-array/src/arrays/bool/vtable/mod.rs index d02a8da22cf..270699ec9b0 100644 --- a/vortex-array/src/arrays/bool/vtable/mod.rs +++ b/vortex-array/src/arrays/bool/vtable/mod.rs @@ -17,8 +17,10 @@ use crate::ArrayRef; use crate::ExecutionCtx; use crate::ExecutionResult; use crate::array::Array; +use crate::array::ArrayParts; use crate::array::ArrayView; use crate::array::VTable; +use crate::array::child_to_validity; use crate::arrays::bool::BoolData; use crate::arrays::bool::array::SLOT_NAMES; use crate::buffer::BufferHandle; @@ -30,6 +32,8 @@ mod kernel; mod operations; mod validity; +use vortex_session::registry::CachedId; + use crate::Precision; use crate::array::ArrayId; use crate::arrays::bool::compute::rules::RULES; @@ -66,7 +70,8 @@ impl VTable for Bool { type ValidityVTable = Self; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.bool"); + *ID } fn nbuffers(_array: ArrayView<'_, Self>) -> usize { @@ -118,7 +123,7 @@ impl VTable for Bool { data.bits.len() * 8 ); - let validity = crate::array::child_to_validity(&slots[0], *nullability); + let validity = child_to_validity(slots[0].as_ref(), *nullability); if let Some(validity_len) = validity.maybe_len() { vortex_ensure!( validity_len == len, @@ -136,11 +141,10 @@ impl VTable for Bool { dtype: &DType, len: usize, metadata: &[u8], - buffers: &[BufferHandle], children: &dyn ArrayChildren, _session: &VortexSession, - ) -> VortexResult> { + ) -> VortexResult> { let metadata = BoolMetadata::decode(metadata)?; if buffers.len() != 1 { vortex_bail!("Expected 1 buffer, got {}", buffers.len()); @@ -158,7 +162,7 @@ impl VTable for Bool { let buffer = buffers[0].clone(); let slots = BoolData::make_slots(&validity, len); let data = BoolData::try_new_from_handle(buffer, metadata.offset as usize, len, validity)?; - Ok(crate::array::ArrayParts::new(self.clone(), dtype.clone(), len, data).with_slots(slots)) + Ok(ArrayParts::new(self.clone(), dtype.clone(), len, data).with_slots(slots)) } fn slot_name(_array: ArrayView<'_, Self>, idx: usize) -> String { @@ -190,10 +194,6 @@ impl VTable for Bool { #[derive(Clone, Debug)] pub struct Bool; -impl Bool { - pub const ID: ArrayId = ArrayId::new_ref("vortex.bool"); -} - #[cfg(test)] mod tests { use vortex_buffer::ByteBufferMut; diff --git a/vortex-array/src/arrays/bool/vtable/operations.rs b/vortex-array/src/arrays/bool/vtable/operations.rs index ece00b038ea..11147b810b2 100644 --- a/vortex-array/src/arrays/bool/vtable/operations.rs +++ b/vortex-array/src/arrays/bool/vtable/operations.rs @@ -28,7 +28,8 @@ mod tests { use std::iter; use crate::IntoArray; - use crate::ToCanonical; + #[expect(deprecated)] + use crate::ToCanonical as _; use crate::arrays::BoolArray; use crate::arrays::bool::BoolArrayExt; use crate::assert_arrays_eq; @@ -36,6 +37,7 @@ mod tests { #[test] fn test_slice_hundred_elements() { let arr = BoolArray::from_iter(iter::repeat_n(Some(true), 100)); + #[expect(deprecated)] let sliced_arr = arr.into_array().slice(8..16).unwrap().to_bool(); assert_eq!(sliced_arr.len(), 8); assert_eq!(sliced_arr.to_bit_buffer().len(), 8); @@ -45,6 +47,7 @@ mod tests { #[test] fn test_slice() { let arr = BoolArray::from_iter([Some(true), Some(true), None, Some(false), None]); + #[expect(deprecated)] let sliced_arr = arr.into_array().slice(1..4).unwrap().to_bool(); assert_arrays_eq!( diff --git a/vortex-array/src/arrays/chunked/array.rs b/vortex-array/src/arrays/chunked/array.rs index 4537b685fa0..6ec93d3aefb 100644 --- a/vortex-array/src/arrays/chunked/array.rs +++ b/vortex-array/src/arrays/chunked/array.rs @@ -37,6 +37,8 @@ pub(super) const CHUNKS_OFFSET: usize = 1; #[derive(Clone, Debug)] pub struct ChunkedData { pub(super) chunk_offsets: Vec, + /// This is used to find the next child to execute when in executing into a builder. + pub(super) next_builder_slot: usize, } impl Display for ChunkedData { @@ -114,6 +116,13 @@ pub trait ChunkedArrayExt: TypedArrayRef { impl> ChunkedArrayExt for T {} impl ChunkedData { + pub(super) fn new(chunk_offsets: Vec) -> Self { + Self { + chunk_offsets, + next_builder_slot: CHUNKS_OFFSET, + } + } + pub(super) fn compute_chunk_offsets(chunks: &[ArrayRef]) -> Vec { let mut chunk_offsets = Vec::with_capacity(chunks.len() + 1); chunk_offsets.push(0); @@ -159,24 +168,31 @@ impl ChunkedData { } impl Array { + pub(super) fn with_next_builder_slot(mut self, next_builder_slot: usize) -> Self { + if let Some(data) = self.data_mut() { + data.next_builder_slot = next_builder_slot; + return self; + } + // This is the slow path that will be hit at most once per execution since the second one + // *MUST* have execlusive access due to this copy. + let stats = self.statistics().to_owned(); + let mut data = self.data().clone(); + data.next_builder_slot = next_builder_slot; + // SAFETY: we only modified next_builder_slot which doesn't affect array invariants. + unsafe { + Array::from_parts_unchecked( + ArrayParts::new(Chunked, self.dtype().clone(), self.len(), data) + .with_slots(self.slots().to_vec()), + ) + } + .with_stats_set(stats) + } + /// Constructs a new `ChunkedArray`. pub fn try_new(chunks: Vec, dtype: DType) -> VortexResult { ChunkedData::validate(&chunks, &dtype)?; - let len = chunks.iter().map(|chunk| chunk.len()).sum(); - let chunk_offsets = ChunkedData::compute_chunk_offsets(&chunks); - Ok(unsafe { - Array::from_parts_unchecked( - ArrayParts::new( - Chunked, - dtype, - len, - ChunkedData { - chunk_offsets: chunk_offsets.clone(), - }, - ) - .with_slots(ChunkedData::make_slots(&chunk_offsets, &chunks)), - ) - }) + // SAFETY just validated on previous line. + Ok(unsafe { Self::new_unchecked(chunks, dtype) }) } pub fn rechunk(&self, target_bytesize: u64, target_rowsize: usize) -> VortexResult { @@ -192,14 +208,14 @@ impl Array { || new_chunk_n_elements + n_elements > target_rowsize) && !chunks_to_combine.is_empty() { - new_chunks.push( - unsafe { - Array::::new_unchecked(chunks_to_combine, self.dtype().clone()) - } - .into_array() - .to_canonical()? - .into_array(), - ); + #[expect(deprecated)] + let canonical = unsafe { + Array::::new_unchecked(chunks_to_combine, self.dtype().clone()) + } + .into_array() + .to_canonical()? + .into_array(); + new_chunks.push(canonical); new_chunk_n_bytes = 0; new_chunk_n_elements = 0; @@ -216,12 +232,13 @@ impl Array { } if !chunks_to_combine.is_empty() { - new_chunks.push( + #[expect(deprecated)] + let canonical = unsafe { Array::::new_unchecked(chunks_to_combine, self.dtype().clone()) } .into_array() .to_canonical()? - .into_array(), - ); + .into_array(); + new_chunks.push(canonical); } unsafe { Ok(Self::new_unchecked(new_chunks, self.dtype().clone())) } @@ -237,15 +254,8 @@ impl Array { let chunk_offsets = ChunkedData::compute_chunk_offsets(&chunks); unsafe { Array::from_parts_unchecked( - ArrayParts::new( - Chunked, - dtype, - len, - ChunkedData { - chunk_offsets: chunk_offsets.clone(), - }, - ) - .with_slots(ChunkedData::make_slots(&chunk_offsets, &chunks)), + ArrayParts::new(Chunked, dtype, len, ChunkedData::new(chunk_offsets.clone())) + .with_slots(ChunkedData::make_slots(&chunk_offsets, &chunks)), ) } } @@ -269,6 +279,8 @@ mod test { use vortex_error::VortexResult; use crate::IntoArray; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::arrays::ChunkedArray; use crate::arrays::PrimitiveArray; use crate::arrays::chunked::ChunkedArrayExt; @@ -358,8 +370,12 @@ mod test { ChunkedArray::try_new(chunks, DType::Primitive(PType::U64, Nullability::Nullable))?; // Should be all_valid since all non-empty chunks are all_valid - assert!(chunked.all_valid().unwrap()); - assert!(!chunked.into_array().all_invalid().unwrap()); + assert!(chunked.all_valid(&mut LEGACY_SESSION.create_execution_ctx())?); + assert!( + !chunked + .into_array() + .all_invalid(&mut LEGACY_SESSION.create_execution_ctx())? + ); Ok(()) } @@ -378,8 +394,12 @@ mod test { ChunkedArray::try_new(chunks, DType::Primitive(PType::U64, Nullability::Nullable))?; // Should be all_invalid since all non-empty chunks are all_invalid - assert!(!chunked.all_valid().unwrap()); - assert!(chunked.into_array().all_invalid().unwrap()); + assert!(!chunked.all_valid(&mut LEGACY_SESSION.create_execution_ctx())?); + assert!( + chunked + .into_array() + .all_invalid(&mut LEGACY_SESSION.create_execution_ctx())? + ); Ok(()) } @@ -398,8 +418,12 @@ mod test { ChunkedArray::try_new(chunks, DType::Primitive(PType::U64, Nullability::Nullable))?; // Should be neither all_valid nor all_invalid - assert!(!chunked.all_valid().unwrap()); - assert!(!chunked.into_array().all_invalid().unwrap()); + assert!(!chunked.all_valid(&mut LEGACY_SESSION.create_execution_ctx())?); + assert!( + !chunked + .into_array() + .all_invalid(&mut LEGACY_SESSION.create_execution_ctx())? + ); Ok(()) } diff --git a/vortex-array/src/arrays/chunked/compute/rules.rs b/vortex-array/src/arrays/chunked/compute/rules.rs index 8ef63a188c3..d8d324a8e86 100644 --- a/vortex-array/src/arrays/chunked/compute/rules.rs +++ b/vortex-array/src/arrays/chunked/compute/rules.rs @@ -11,8 +11,8 @@ use crate::arrays::Chunked; use crate::arrays::ChunkedArray; use crate::arrays::Constant; use crate::arrays::ConstantArray; +use crate::arrays::ScalarFn; use crate::arrays::ScalarFnArray; -use crate::arrays::ScalarFnVTable; use crate::arrays::chunked::ChunkedArrayExt; use crate::arrays::scalar_fn::AnyScalarFn; use crate::arrays::scalar_fn::ScalarFnArrayExt; @@ -38,7 +38,7 @@ impl ArrayParentReduceRule for ChunkedUnaryScalarFnPushDownRule { fn reduce_parent( &self, array: ArrayView<'_, Chunked>, - parent: ArrayView<'_, ScalarFnVTable>, + parent: ArrayView<'_, ScalarFn>, _child_idx: usize, ) -> VortexResult> { if parent.nchildren() != 1 { @@ -73,7 +73,7 @@ impl ArrayParentReduceRule for ChunkedConstantScalarFnPushDownRule { fn reduce_parent( &self, array: ArrayView<'_, Chunked>, - parent: ArrayView<'_, ScalarFnVTable>, + parent: ArrayView<'_, ScalarFn>, child_idx: usize, ) -> VortexResult> { for (idx, child) in parent.iter_children().enumerate() { diff --git a/vortex-array/src/arrays/chunked/compute/take.rs b/vortex-array/src/arrays/chunked/compute/take.rs index 1cd48c57319..7024c2b3875 100644 --- a/vortex-array/src/arrays/chunked/compute/take.rs +++ b/vortex-array/src/arrays/chunked/compute/take.rs @@ -31,7 +31,10 @@ fn take_chunked( .cast(DType::Primitive(PType::U64, indices.dtype().nullability()))? .execute::(ctx)?; - let indices_mask = indices.validity_mask()?; + let indices_mask = indices + .as_ref() + .validity()? + .execute_mask(indices.as_ref().len(), ctx)?; let indices_values = indices.as_slice::(); let n = indices_values.len(); @@ -96,8 +99,13 @@ fn take_chunked( // 4. Single take to restore original order and expand duplicates. // Carry the original index validity so null indices produce null outputs. - let take_validity = - Validity::from_mask(indices.validity_mask()?, indices.dtype().nullability()); + let take_validity = Validity::from_mask( + indices + .as_ref() + .validity()? + .execute_mask(indices.as_ref().len(), ctx)?, + indices.dtype().nullability(), + ); flat.take(PrimitiveArray::new(final_take.freeze(), take_validity).into_array()) } @@ -118,7 +126,8 @@ mod test { use vortex_error::VortexResult; use crate::IntoArray; - use crate::ToCanonical; + #[expect(deprecated)] + use crate::ToCanonical as _; use crate::arrays::BoolArray; use crate::arrays::ChunkedArray; use crate::arrays::PrimitiveArray; @@ -276,6 +285,7 @@ mod test { let result = arr.take(indices_arr.into_array())?; // Verify every element. + #[expect(deprecated)] let result = result.to_primitive(); let result_vals = result.as_slice::(); for (pos, &idx) in indices.iter().enumerate() { diff --git a/vortex-array/src/arrays/chunked/compute/zip.rs b/vortex-array/src/arrays/chunked/compute/zip.rs index 0979cb15031..1d7873888a3 100644 --- a/vortex-array/src/arrays/chunked/compute/zip.rs +++ b/vortex-array/src/arrays/chunked/compute/zip.rs @@ -52,7 +52,8 @@ mod tests { use crate::ArrayRef; use crate::IntoArray; use crate::LEGACY_SESSION; - use crate::ToCanonical; + #[expect(deprecated)] + use crate::ToCanonical as _; use crate::VortexSessionExecute; use crate::arrays::Chunked; use crate::arrays::ChunkedArray; @@ -102,6 +103,7 @@ mod tests { assert_eq!(zipped.nchunks(), 4); let mut values: Vec = Vec::new(); for chunk in zipped.chunks() { + #[expect(deprecated)] let primitive = chunk.to_primitive(); values.extend_from_slice(primitive.as_slice::()); } diff --git a/vortex-array/src/arrays/chunked/paired_chunks.rs b/vortex-array/src/arrays/chunked/paired_chunks.rs index f786d62b9fb..2145c88dbbd 100644 --- a/vortex-array/src/arrays/chunked/paired_chunks.rs +++ b/vortex-array/src/arrays/chunked/paired_chunks.rs @@ -121,6 +121,8 @@ mod tests { use vortex_error::VortexResult; use crate::IntoArray; + #[expect(deprecated)] + use crate::ToCanonical as _; use crate::arrays::ChunkedArray; use crate::arrays::chunked::paired_chunks::PairedChunksExt; use crate::dtype::DType; @@ -131,16 +133,17 @@ mod tests { DType::Primitive(PType::I32, Nullability::NonNullable) } - #[allow(clippy::type_complexity)] + #[expect(clippy::type_complexity)] fn collect_pairs( left: &ChunkedArray, right: &ChunkedArray, ) -> VortexResult, Vec, std::ops::Range)>> { - use crate::ToCanonical; let mut result = Vec::new(); for pair in left.paired_chunks(right) { let pair = pair?; + #[expect(deprecated)] let l: Vec = pair.left.to_primitive().as_slice::().to_vec(); + #[expect(deprecated)] let r: Vec = pair.right.to_primitive().as_slice::().to_vec(); result.push((l, r, pair.pos)); } diff --git a/vortex-array/src/arrays/chunked/tests.rs b/vortex-array/src/arrays/chunked/tests.rs index a20e775cf5c..00d113f273a 100644 --- a/vortex-array/src/arrays/chunked/tests.rs +++ b/vortex-array/src/arrays/chunked/tests.rs @@ -2,11 +2,15 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors use std::sync::Arc; +use std::sync::LazyLock; use vortex_buffer::Buffer; use vortex_buffer::buffer; +use vortex_session::VortexSession; +use crate::Canonical; use crate::IntoArray; +use crate::VortexSessionExecute; use crate::accessor::ArrayAccessor; use crate::arrays::Chunked; use crate::arrays::ChunkedArray; @@ -15,15 +19,23 @@ use crate::arrays::PrimitiveArray; use crate::arrays::StructArray; use crate::arrays::VarBinViewArray; use crate::arrays::chunked::ChunkedArrayExt; +use crate::arrays::dict_test::gen_dict_primitive_chunks; use crate::arrays::struct_::StructArrayExt; use crate::assert_arrays_eq; -use crate::canonical::ToCanonical; +use crate::builders::builder_with_capacity; +#[expect(deprecated)] +use crate::canonical::ToCanonical as _; use crate::dtype::DType; use crate::dtype::Nullability; use crate::dtype::PType; use crate::dtype::PType::I32; +use crate::executor::execute_into_builder; +use crate::session::ArraySession; use crate::validity::Validity; +static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); + fn chunked_array() -> ChunkedArray { ChunkedArray::try_new( vec![ @@ -36,6 +48,156 @@ fn chunked_array() -> ChunkedArray { .unwrap() } +#[test] +fn builder_kernel_path_canonicalizes_primitive_chunks() { + let mut ctx = SESSION.create_execution_ctx(); + + let array = chunked_array().into_array(); + let dtype = array.dtype().clone(); + let len = array.len(); + + let builder = builder_with_capacity(&dtype, len); + // Clone the array into the builder path — the test also holds `array` so refcount > 1 on + // entry, which previously caused `take_slot_unchecked` to silently keep slots populated. + let mut builder = execute_into_builder(array.clone(), builder, &mut ctx).unwrap(); + let output = builder.finish(); + drop(array); + + assert_arrays_eq!( + output, + PrimitiveArray::from_iter([1u64, 2, 3, 4, 5, 6, 7, 8, 9]) + ); +} + +#[test] +fn builder_kernel_nested_chunked_of_chunked() { + let mut ctx = SESSION.create_execution_ctx(); + + let inner_1 = ChunkedArray::try_new( + vec![buffer![1u64, 2].into_array(), buffer![3u64].into_array()], + DType::Primitive(PType::U64, Nullability::NonNullable), + ) + .unwrap() + .into_array(); + let inner_2 = ChunkedArray::try_new( + vec![buffer![4u64, 5, 6].into_array()], + DType::Primitive(PType::U64, Nullability::NonNullable), + ) + .unwrap() + .into_array(); + let outer = ChunkedArray::try_new( + vec![inner_1, inner_2], + DType::Primitive(PType::U64, Nullability::NonNullable), + ) + .unwrap() + .into_array(); + + let dtype = outer.dtype().clone(); + let len = outer.len(); + let builder = builder_with_capacity(&dtype, len); + let mut builder = execute_into_builder(outer, builder, &mut ctx).unwrap(); + let output = builder.finish(); + + assert_arrays_eq!(output, PrimitiveArray::from_iter([1u64, 2, 3, 4, 5, 6])); +} + +#[test] +fn builder_kernel_path_repeated_shared_chunked_dict_execution() { + let mut ctx = SESSION.create_execution_ctx(); + + let array = gen_dict_primitive_chunks::(8, 3, 3); + let keep_alive = array.clone(); + let dtype = array.dtype().clone(); + let len = array.len(); + + let expected = array + .clone() + .execute::(&mut ctx) + .unwrap() + .into_array(); + + let first = { + let builder = builder_with_capacity(&dtype, len); + let mut builder = execute_into_builder(array.clone(), builder, &mut ctx).unwrap(); + builder.finish() + }; + + let second = { + let builder = builder_with_capacity(&dtype, len); + let mut builder = execute_into_builder(array, builder, &mut ctx).unwrap(); + builder.finish() + }; + + drop(keep_alive); + + assert_arrays_eq!(first, expected); + assert_arrays_eq!(second, expected); +} + +#[test] +fn execute_path_repeated_shared_chunked_dict_execution() { + let mut ctx = SESSION.create_execution_ctx(); + let array = gen_dict_primitive_chunks::(8, 3, 3); + let keep_alive = array.clone(); + + let expected_source = gen_dict_primitive_chunks::(8, 3, 3); + let expected = expected_source + .execute::(&mut ctx) + .unwrap() + .into_array(); + + let first = array + .clone() + .execute::(&mut ctx) + .unwrap() + .into_array(); + + let second = array.execute::(&mut ctx).unwrap().into_array(); + + drop(keep_alive); + + assert_arrays_eq!(first, expected); + assert_arrays_eq!(second, expected); +} + +#[test] +fn execute_path_nested_chunked_dict_of_dict_into_canonical() { + let mut ctx = SESSION.create_execution_ctx(); + let inner_1 = gen_dict_primitive_chunks::(8, 3, 2); + let inner_2 = gen_dict_primitive_chunks::(8, 3, 3); + let outer = ChunkedArray::try_new( + vec![inner_1.clone(), inner_2.clone()], + inner_1.dtype().clone(), + ) + .unwrap() + .into_array(); + let keep_alive = outer.clone(); + + let expected = { + let mut builder = builder_with_capacity(outer.dtype(), outer.len()); + inner_1 + .append_to_builder(builder.as_mut(), &mut ctx) + .unwrap(); + inner_2 + .append_to_builder(builder.as_mut(), &mut ctx) + .unwrap(); + builder.finish() + }; + + let first = outer + .clone() + .execute::(&mut ctx) + .unwrap() + .into_array(); + + let second = outer.execute::(&mut ctx).unwrap().into_array(); + + drop(keep_alive); + + assert_arrays_eq!(first, expected); + assert_arrays_eq!(second, expected); +} + #[test] fn with_slot_rewrites_chunk_and_offsets() { let array = chunked_array().into_array(); @@ -188,8 +350,11 @@ pub fn pack_nested_structs() { ) .unwrap() .into_array(); + #[expect(deprecated)] let canonical_struct = chunked.to_struct(); + #[expect(deprecated)] let canonical_varbin = canonical_struct.unmasked_fields()[0].to_varbinview(); + #[expect(deprecated)] let original_varbin = struct_array.unmasked_fields()[0].to_varbinview(); let orig_values = original_varbin.with_iterator(|it| it.map(|a| a.map(|v| v.to_vec())).collect::>()); @@ -200,6 +365,7 @@ pub fn pack_nested_structs() { #[test] pub fn pack_nested_lists() { + let mut ctx = SESSION.create_execution_ctx(); let l1 = ListArray::try_new( buffer![1, 2, 3, 4].into_array(), buffer![0, 3].into_array(), @@ -222,8 +388,15 @@ pub fn pack_nested_lists() { ), ); + #[expect(deprecated)] let canon_values = chunked_list.unwrap().as_array().to_listview(); - assert_eq!(l1.scalar_at(0).unwrap(), canon_values.scalar_at(0).unwrap()); - assert_eq!(l2.scalar_at(0).unwrap(), canon_values.scalar_at(1).unwrap()); + assert_eq!( + l1.execute_scalar(0, &mut ctx).unwrap(), + canon_values.execute_scalar(0, &mut ctx).unwrap() + ); + assert_eq!( + l2.execute_scalar(0, &mut ctx).unwrap(), + canon_values.execute_scalar(1, &mut ctx).unwrap() + ); } diff --git a/vortex-array/src/arrays/chunked/vtable/canonical.rs b/vortex-array/src/arrays/chunked/vtable/canonical.rs index 63bb4132eca..42e0b5d2073 100644 --- a/vortex-array/src/arrays/chunked/vtable/canonical.rs +++ b/vortex-array/src/arrays/chunked/vtable/canonical.rs @@ -1,6 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +use itertools::Itertools as _; use vortex_buffer::Buffer; use vortex_error::VortexExpect; use vortex_error::VortexResult; @@ -18,13 +19,11 @@ use crate::arrays::StructArray; use crate::arrays::chunked::ChunkedArrayExt; use crate::arrays::listview::ListViewArrayExt; use crate::arrays::listview::ListViewRebuildMode; -use crate::arrays::struct_::StructArrayExt; use crate::builders::builder_with_capacity_in; use crate::builtins::ArrayBuiltins; use crate::dtype::DType; use crate::dtype::Nullability; use crate::dtype::PType; -use crate::dtype::StructFields; use crate::memory::HostAllocatorExt; use crate::validity::Validity; @@ -41,18 +40,13 @@ pub(super) fn _canonicalize( let owned_chunks: Vec = array.iter_chunks().cloned().collect(); Ok(match array.dtype() { - DType::Struct(struct_dtype, _) => { - let struct_array = pack_struct_chunks( - &owned_chunks, - Validity::copy_from_array(array.array())?, - struct_dtype, - ctx, - )?; + DType::Struct(..) => { + let struct_array = pack_struct_chunks(owned_chunks, ctx)?; Canonical::Struct(struct_array) } DType::List(elem_dtype, _) => Canonical::List(swizzle_list_chunks( &owned_chunks, - Validity::copy_from_array(array.array())?, + array.array().validity()?, elem_dtype, ctx, )?), @@ -68,36 +62,11 @@ pub(super) fn _canonicalize( /// field is a [`ChunkedArray`]. /// /// The caller guarantees there are at least 2 chunks. -fn pack_struct_chunks( - chunks: &[ArrayRef], - validity: Validity, - struct_dtype: &StructFields, - ctx: &mut ExecutionCtx, -) -> VortexResult { - let len = chunks.iter().map(|chunk| chunk.len()).sum(); - let mut field_arrays = Vec::new(); - - let executed_chunks: Vec = chunks - .iter() - .map(|c| c.clone().execute::(ctx)) - .collect::>()?; - - for (field_idx, field_dtype) in struct_dtype.fields().enumerate() { - let mut field_chunks = Vec::with_capacity(chunks.len()); - for struct_array in &executed_chunks { - let field = struct_array.unmasked_field(field_idx).clone(); - field_chunks.push(field); - } - - // SAFETY: field_chunks are extracted from valid StructArrays with matching dtypes. - // Each chunk's field array is guaranteed to be valid for field_dtype. - let field_array = unsafe { ChunkedArray::new_unchecked(field_chunks, field_dtype.clone()) }; - field_arrays.push(field_array.into_array()); - } - - // SAFETY: field_arrays are built from corresponding chunks of same length, dtypes match by - // construction. - Ok(unsafe { StructArray::new_unchecked(field_arrays, struct_dtype.clone(), len, validity) }) +fn pack_struct_chunks(chunks: Vec, ctx: &mut ExecutionCtx) -> VortexResult { + chunks + .into_iter() + .map(|c| c.execute::(ctx)) + .process_results(|iter| StructArray::try_concat(iter))? } /// Packs [`ListViewArray`]s together into a chunked `ListViewArray`. @@ -219,7 +188,10 @@ mod tests { use crate::Canonical; use crate::ExecutionCtx; use crate::IntoArray; - use crate::ToCanonical; + use crate::LEGACY_SESSION; + #[expect(deprecated)] + use crate::ToCanonical as _; + use crate::VortexSessionExecute; use crate::accessor::ArrayAccessor; use crate::arrays::ChunkedArray; use crate::arrays::ListArray; @@ -272,8 +244,11 @@ mod tests { ) .unwrap() .into_array(); + #[expect(deprecated)] let canonical_struct = chunked.to_struct(); + #[expect(deprecated)] let canonical_varbin = canonical_struct.unmasked_field(0).to_varbinview(); + #[expect(deprecated)] let original_varbin = struct_array.unmasked_field(0).to_varbinview(); let orig_values = original_varbin .with_iterator(|it| it.map(|a| a.map(|v| v.to_vec())).collect::>()); @@ -303,10 +278,23 @@ mod tests { List(Arc::new(Primitive(I32, NonNullable)), NonNullable), ); + #[expect(deprecated)] let canon_values = chunked_list.unwrap().as_array().to_listview(); - assert_eq!(l1.scalar_at(0).unwrap(), canon_values.scalar_at(0).unwrap()); - assert_eq!(l2.scalar_at(0).unwrap(), canon_values.scalar_at(1).unwrap()); + assert_eq!( + l1.execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + canon_values + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); + assert_eq!( + l2.execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + canon_values + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); } #[test] diff --git a/vortex-array/src/arrays/chunked/vtable/mod.rs b/vortex-array/src/arrays/chunked/vtable/mod.rs index 29e4b397d35..046fdea1caa 100644 --- a/vortex-array/src/arrays/chunked/vtable/mod.rs +++ b/vortex-array/src/arrays/chunked/vtable/mod.rs @@ -11,6 +11,7 @@ use vortex_error::vortex_ensure; use vortex_error::vortex_err; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::ArrayEq; use crate::ArrayHash; @@ -20,7 +21,8 @@ use crate::ExecutionCtx; use crate::ExecutionResult; use crate::IntoArray; use crate::Precision; -use crate::ToCanonical; +#[expect(deprecated)] +use crate::ToCanonical as _; use crate::array::Array; use crate::array::ArrayId; use crate::array::ArrayParts; @@ -42,16 +44,13 @@ use crate::serde::ArrayChildren; mod canonical; mod operations; mod validity; + /// A [`Chunked`]-encoded Vortex array. pub type ChunkedArray = Array; #[derive(Clone, Debug)] pub struct Chunked; -impl Chunked { - pub const ID: ArrayId = ArrayId::new_ref("vortex.chunked"); -} - impl ArrayHash for ChunkedData { fn array_hash(&self, _state: &mut H, _precision: Precision) { // Chunk offsets are cached derived data. Slot 0 already stores the logical offsets array, @@ -72,9 +71,9 @@ impl VTable for Chunked { type OperationsVTable = Self; type ValidityVTable = Self; - fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.chunked"); + *ID } fn validate( @@ -189,6 +188,7 @@ impl VTable for Chunked { &DType::Primitive(PType::U64, Nullability::NonNullable), nchunks + 1, )?; + #[expect(deprecated)] let chunk_offsets_buf = chunk_offsets.to_primitive().to_buffer::(); let chunk_offsets_usize = chunk_offsets_buf .iter() @@ -214,9 +214,7 @@ impl VTable for Chunked { self.clone(), dtype.clone(), len, - ChunkedData { - chunk_offsets: chunk_offsets_usize, - }, + ChunkedData::new(chunk_offsets_usize), ) .with_slots(slots)) } @@ -240,7 +238,27 @@ impl VTable for Chunked { } fn execute(array: Array, ctx: &mut ExecutionCtx) -> VortexResult { - Ok(ExecutionResult::done(_canonicalize(array.as_view(), ctx)?)) + match array.dtype() { + // Struct and List need special swizzling logic, use the existing canonicalize path. + DType::Struct(..) | DType::List(..) => { + // TODO(joe)[#7674]: iterative execution here too + Ok(ExecutionResult::done(_canonicalize(array.as_view(), ctx)?)) + } + // For all other types, use the builder path via AppendChild. + _ => { + let slot_idx = array.next_builder_slot.max(CHUNKS_OFFSET); + if slot_idx < array.slots().len() { + Ok(ExecutionResult::append_child( + array.with_next_builder_slot(slot_idx + 1), + slot_idx, + )) + } else { + Ok(ExecutionResult::done( + Canonical::empty(array.dtype()).into_array(), + )) + } + } + } } fn execute_parent( diff --git a/vortex-array/src/arrays/chunked/vtable/operations.rs b/vortex-array/src/arrays/chunked/vtable/operations.rs index 7c7095a05c7..ce44b8231f8 100644 --- a/vortex-array/src/arrays/chunked/vtable/operations.rs +++ b/vortex-array/src/arrays/chunked/vtable/operations.rs @@ -14,10 +14,10 @@ impl OperationsVTable for Chunked { fn scalar_at( array: ArrayView<'_, Chunked>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { let (chunk_index, chunk_offset) = array.find_chunk_idx(index)?; - array.chunk(chunk_index).scalar_at(chunk_offset) + array.chunk(chunk_index).execute_scalar(chunk_offset, ctx) } } diff --git a/vortex-array/src/arrays/chunked/vtable/validity.rs b/vortex-array/src/arrays/chunked/vtable/validity.rs index 5f5c11c7832..265c2843ee1 100644 --- a/vortex-array/src/arrays/chunked/vtable/validity.rs +++ b/vortex-array/src/arrays/chunked/vtable/validity.rs @@ -4,52 +4,24 @@ use itertools::Itertools; use vortex_error::VortexResult; -use crate::IntoArray; use crate::array::ArrayView; use crate::array::ValidityVTable; use crate::arrays::Chunked; -use crate::arrays::ChunkedArray; use crate::arrays::chunked::ChunkedArrayExt; -use crate::dtype::DType; -use crate::dtype::Nullability; use crate::validity::Validity; impl ValidityVTable for Chunked { fn validity(array: ArrayView<'_, Chunked>) -> VortexResult { - let validities: Vec = - array.chunks().iter().map(|c| c.validity()).try_collect()?; + let validities = array + .chunks() + .iter() + .map(|chunk| chunk.validity().map(|v| (v, chunk.len()))) + .try_collect()?; + let Some(validity) = Validity::concat(validities) else { + // If there are no chunks: + return Ok(array.dtype().nullability().into()); + }; - match validities.first() { - // If there are no chunks, return the array's dtype nullability - None => return Ok(array.dtype().nullability().into()), - // If all chunks have the same non-array validity, return that validity directly - // We skip Validity::Array since equality is very expensive. - Some(first) if !matches!(first, Validity::Array(_)) => { - let target = std::mem::discriminant(first); - if validities - .iter() - .all(|v| std::mem::discriminant(v) == target) - { - return Ok(first.clone()); - } - } - _ => { - // Array validity or mixed validities, proceed to build the validity array - } - } - - Ok(Validity::Array( - unsafe { - ChunkedArray::new_unchecked( - validities - .into_iter() - .zip(array.iter_chunks()) - .map(|(v, chunk)| v.to_array(chunk.len())) - .collect(), - DType::Bool(Nullability::NonNullable), - ) - } - .into_array(), - )) + Ok(validity) } } diff --git a/vortex-array/src/arrays/constant/compute/cast.rs b/vortex-array/src/arrays/constant/compute/cast.rs index 68c6dbeb758..2fd9b3ded32 100644 --- a/vortex-array/src/arrays/constant/compute/cast.rs +++ b/vortex-array/src/arrays/constant/compute/cast.rs @@ -15,7 +15,7 @@ impl CastReduce for Constant { fn cast(array: ArrayView<'_, Constant>, dtype: &DType) -> VortexResult> { match array.scalar().cast(dtype) { Ok(scalar) => Ok(Some(ConstantArray::new(scalar, array.len()).into_array())), - Err(_e) => Ok(None), + Err(_) => Ok(None), } } } diff --git a/vortex-array/src/arrays/constant/compute/fill_null.rs b/vortex-array/src/arrays/constant/compute/fill_null.rs index 22fe70c246d..207f908b5cc 100644 --- a/vortex-array/src/arrays/constant/compute/fill_null.rs +++ b/vortex-array/src/arrays/constant/compute/fill_null.rs @@ -22,8 +22,10 @@ impl FillNullReduce for Constant { #[cfg(test)] mod test { use crate::IntoArray as _; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::arrays::ConstantArray; - use crate::arrow::IntoArrowArray as _; + use crate::arrow::ArrowArrayExecutor; use crate::builtins::ArrayBuiltins; use crate::dtype::DType; use crate::dtype::Nullability; @@ -32,6 +34,7 @@ mod test { #[test] fn test_null() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let actual = ConstantArray::new(Scalar::null_native::(), 3) .into_array() .fill_null(Scalar::from(1)) @@ -40,8 +43,8 @@ mod test { assert!(!actual.dtype().is_nullable()); - let actual_arrow = actual.clone().into_arrow_preferred().unwrap(); - let expected_arrow = expected.clone().into_arrow_preferred().unwrap(); + let actual_arrow = actual.clone().execute_arrow(None, &mut ctx).unwrap(); + let expected_arrow = expected.clone().execute_arrow(None, &mut ctx).unwrap(); assert_eq!( &actual_arrow, &expected_arrow, @@ -53,6 +56,7 @@ mod test { #[test] fn test_non_null() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let actual = ConstantArray::new(Scalar::from(Some(1)), 3) .into_array() .fill_null(Scalar::from(1)) @@ -61,8 +65,8 @@ mod test { assert!(!actual.dtype().is_nullable()); - let actual_arrow = actual.clone().into_arrow_preferred().unwrap(); - let expected_arrow = expected.clone().into_arrow_preferred().unwrap(); + let actual_arrow = actual.clone().execute_arrow(None, &mut ctx).unwrap(); + let expected_arrow = expected.clone().execute_arrow(None, &mut ctx).unwrap(); assert_eq!( &actual_arrow, &expected_arrow, @@ -74,6 +78,7 @@ mod test { #[test] fn test_non_nullable_with_nullable() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let actual = ConstantArray::new(Scalar::from(1), 3) .into_array() .fill_null(Scalar::new( @@ -87,8 +92,8 @@ mod test { assert!(actual.dtype().is_nullable()); - let actual_arrow = actual.clone().into_arrow_preferred().unwrap(); - let expected_arrow = expected.clone().into_arrow_preferred().unwrap(); + let actual_arrow = actual.clone().execute_arrow(None, &mut ctx).unwrap(); + let expected_arrow = expected.clone().execute_arrow(None, &mut ctx).unwrap(); assert_eq!( &actual_arrow, &expected_arrow, diff --git a/vortex-array/src/arrays/constant/compute/take.rs b/vortex-array/src/arrays/constant/compute/take.rs index 96aa71fd9ac..1f05dc9db71 100644 --- a/vortex-array/src/arrays/constant/compute/take.rs +++ b/vortex-array/src/arrays/constant/compute/take.rs @@ -6,6 +6,8 @@ use vortex_mask::AllOr; use crate::ArrayRef; use crate::IntoArray; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::array::ArrayView; use crate::arrays::Constant; use crate::arrays::ConstantArray; @@ -18,7 +20,12 @@ use crate::validity::Validity; impl TakeReduce for Constant { fn take(array: ArrayView<'_, Constant>, indices: &ArrayRef) -> VortexResult> { - let result = match indices.validity_mask()?.bit_buffer() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let result = match indices + .validity()? + .execute_mask(indices.len(), &mut ctx)? + .bit_buffer() + { AllOr::All => { let scalar = Scalar::try_new( array @@ -64,7 +71,10 @@ mod tests { use vortex_mask::AllOr; use crate::IntoArray; - use crate::ToCanonical; + use crate::LEGACY_SESSION; + #[expect(deprecated)] + use crate::ToCanonical as _; + use crate::VortexSessionExecute; use crate::arrays::ConstantArray; use crate::arrays::PrimitiveArray; use crate::assert_arrays_eq; @@ -91,6 +101,7 @@ mod tests { taken.dtype() ); assert_arrays_eq!( + #[expect(deprecated)] taken.to_primitive(), PrimitiveArray::new( buffer![42i32, 42, 42], @@ -98,7 +109,12 @@ mod tests { ) ); assert_eq!( - taken.validity_mask().unwrap().indices(), + taken + .validity() + .unwrap() + .execute_mask(taken.len(), &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .indices(), AllOr::Some(valid_indices) ); } @@ -114,10 +130,19 @@ mod tests { taken.dtype() ); assert_arrays_eq!( + #[expect(deprecated)] taken.to_primitive(), PrimitiveArray::new(buffer![42i32, 42, 42], Validity::AllValid) ); - assert_eq!(taken.validity_mask().unwrap().indices(), AllOr::All); + assert_eq!( + taken + .validity() + .unwrap() + .execute_mask(taken.len(), &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .indices(), + AllOr::All + ); } #[rstest] diff --git a/vortex-array/src/arrays/constant/vtable/canonical.rs b/vortex-array/src/arrays/constant/vtable/canonical.rs index 137f48fd8eb..2ca23ae9a66 100644 --- a/vortex-array/src/arrays/constant/vtable/canonical.rs +++ b/vortex-array/src/arrays/constant/vtable/canonical.rs @@ -10,6 +10,7 @@ use vortex_error::VortexExpect; use vortex_error::VortexResult; use crate::Canonical; +use crate::ExecutionCtx; use crate::IntoArray; use crate::array::ArrayView; use crate::arrays::BoolArray; @@ -36,7 +37,10 @@ use crate::scalar::Scalar; use crate::validity::Validity; /// Shared implementation for both `canonicalize` and `execute` methods. -pub(crate) fn constant_canonicalize(array: ArrayView<'_, Constant>) -> VortexResult { +pub(crate) fn constant_canonicalize( + array: ArrayView<'_, Constant>, + ctx: &mut ExecutionCtx, +) -> VortexResult { let scalar = array.scalar(); let validity = match array.dtype().nullability() { @@ -163,7 +167,16 @@ pub(crate) fn constant_canonicalize(array: ArrayView<'_, Constant>) -> VortexRes let s = scalar.as_extension(); let storage_scalar = s.to_storage_scalar(); - let storage_self = ConstantArray::new(storage_scalar, array.len()).into_array(); + + // NB: We need to execute the constant array to be canonical because there is a + // reduction rule that turns `Extension(Constant(..))` into `Constant(Extension(..))`, + // and if we don't do this we create an infinite cycle. + // See `ExtensionConstantRule` for more details. + let storage_self = ConstantArray::new(storage_scalar, array.len()) + .into_array() + .execute::(ctx)? + .into_array(); + Canonical::Extension(ExtensionArray::new(ext_dtype.clone(), storage_self)) } DType::Variant(_) => { @@ -323,6 +336,8 @@ mod tests { use vortex_error::VortexResult; use crate::IntoArray; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::arrays::ConstantArray; use crate::arrays::PrimitiveArray; use crate::arrays::VarBinArray; @@ -331,7 +346,8 @@ mod tests { use crate::arrays::listview::ListViewRebuildMode; use crate::arrays::struct_::StructArrayExt; use crate::assert_arrays_eq; - use crate::canonical::ToCanonical; + #[expect(deprecated)] + use crate::canonical::ToCanonical as _; use crate::dtype::DType; use crate::dtype::Nullability; use crate::dtype::PType; @@ -344,9 +360,15 @@ mod tests { #[test] fn test_canonicalize_null() { let const_null = ConstantArray::new(Scalar::null(DType::Null), 42); + #[expect(deprecated)] let actual = const_null.as_array().to_null(); assert_eq!(actual.len(), 42); - assert_eq!(actual.scalar_at(33).unwrap(), Scalar::null(DType::Null)); + assert_eq!( + actual + .execute_scalar(33, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + Scalar::null(DType::Null) + ); } #[test] @@ -361,10 +383,11 @@ mod tests { fn test_canonicalize_propagates_stats() -> VortexResult<()> { let scalar = Scalar::bool(true, Nullability::NonNullable); let const_array = ConstantArray::new(scalar, 4).into_array(); - let stats = const_array - .statistics() - .compute_all(&all::().collect_vec()) - .unwrap(); + let stats = const_array.statistics().compute_all( + &all::().collect_vec(), + &mut LEGACY_SESSION.create_execution_ctx(), + )?; + #[expect(deprecated)] let canonical = const_array.to_canonical()?.into_array(); let canonical_stats = canonical.statistics(); @@ -390,10 +413,16 @@ mod tests { // Create a ConstantArray with the f16 scalar let const_array = ConstantArray::new(f16_scalar.clone(), 1).into_array(); + #[expect(deprecated)] let canonical_const = const_array.to_primitive(); // Verify the scalar value is preserved through canonicalization - assert_eq!(canonical_const.scalar_at(0).unwrap(), f16_scalar); + assert_eq!( + canonical_const + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + f16_scalar + ); } #[test] @@ -404,17 +433,21 @@ mod tests { Nullability::NonNullable, ); let const_array = ConstantArray::new(list_scalar, 2).into_array(); + #[expect(deprecated)] let canonical_const = const_array.to_listview(); let list_array = canonical_const.rebuild(ListViewRebuildMode::MakeZeroCopyToList)?; assert_arrays_eq!( + #[expect(deprecated)] list_array.elements().to_primitive(), PrimitiveArray::from_iter([1u64, 2, 1, 2]) ); assert_arrays_eq!( + #[expect(deprecated)] list_array.offsets().to_primitive(), PrimitiveArray::from_iter([0u64, 2]) ); assert_arrays_eq!( + #[expect(deprecated)] list_array.sizes().to_primitive(), PrimitiveArray::from_iter([2u64, 2]) ); @@ -429,13 +462,18 @@ mod tests { Nullability::NonNullable, ); let const_array = ConstantArray::new(list_scalar, 2).into_array(); + #[expect(deprecated)] let canonical_const = const_array.to_listview(); - assert!(canonical_const.elements().to_primitive().is_empty()); + #[expect(deprecated)] + let elements_prim = canonical_const.elements().to_primitive(); + assert!(elements_prim.is_empty()); assert_arrays_eq!( + #[expect(deprecated)] canonical_const.offsets().to_primitive(), PrimitiveArray::from_iter([0u64, 0]) ); assert_arrays_eq!( + #[expect(deprecated)] canonical_const.sizes().to_primitive(), PrimitiveArray::from_iter([0u64, 0]) ); @@ -448,13 +486,18 @@ mod tests { Nullability::Nullable, )); let const_array = ConstantArray::new(list_scalar, 2).into_array(); + #[expect(deprecated)] let canonical_const = const_array.to_listview(); - assert!(canonical_const.elements().to_primitive().is_empty()); + #[expect(deprecated)] + let elements_prim = canonical_const.elements().to_primitive(); + assert!(elements_prim.is_empty()); assert_arrays_eq!( + #[expect(deprecated)] canonical_const.offsets().to_primitive(), PrimitiveArray::from_iter([0u64, 0]) ); assert_arrays_eq!( + #[expect(deprecated)] canonical_const.sizes().to_primitive(), PrimitiveArray::from_iter([0u64, 0]) ); @@ -473,9 +516,15 @@ mod tests { 3, ); + #[expect(deprecated)] let struct_array = array.as_array().to_struct(); assert_eq!(struct_array.len(), 3); - assert_eq!(struct_array.valid_count().unwrap(), 0); + assert_eq!( + struct_array + .valid_count(&mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 0 + ); let field = struct_array .unmasked_field_by_name("non_null_field") @@ -501,6 +550,7 @@ mod tests { ); let const_array = ConstantArray::new(fsl_scalar, 4).into_array(); + #[expect(deprecated)] let canonical = const_array.to_fixed_size_list(); assert_eq!(canonical.len(), 4); @@ -510,6 +560,7 @@ mod tests { // Check that each list is [10, 20, 30]. for i in 0..4 { let list = canonical.fixed_size_list_elements_at(i).unwrap(); + #[expect(deprecated)] let list_primitive = list.to_primitive(); assert_arrays_eq!(list_primitive, PrimitiveArray::from_iter([10i32, 20, 30])); } @@ -528,6 +579,7 @@ mod tests { ); let const_array = ConstantArray::new(fsl_scalar, 3).into_array(); + #[expect(deprecated)] let canonical = const_array.to_fixed_size_list(); assert_eq!(canonical.len(), 3); @@ -535,6 +587,7 @@ mod tests { assert!(matches!(canonical.validity(), Ok(Validity::AllValid))); // Check elements. + #[expect(deprecated)] let elements = canonical.elements().to_primitive(); assert_arrays_eq!( elements, @@ -552,6 +605,7 @@ mod tests { )); let const_array = ConstantArray::new(fsl_scalar, 5).into_array(); + #[expect(deprecated)] let canonical = const_array.to_fixed_size_list(); assert_eq!(canonical.len(), 5); @@ -559,6 +613,7 @@ mod tests { assert!(matches!(canonical.validity(), Ok(Validity::AllInvalid))); // Elements should be defaults (zeros). + #[expect(deprecated)] let elements = canonical.elements().to_primitive(); assert_eq!(elements.len(), 20); // 5 lists * 4 elements each assert!(elements.as_slice::().iter().all(|&x| x == 0)); @@ -574,6 +629,7 @@ mod tests { ); let const_array = ConstantArray::new(fsl_scalar, 10).into_array(); + #[expect(deprecated)] let canonical = const_array.to_fixed_size_list(); assert_eq!(canonical.len(), 10); @@ -594,17 +650,39 @@ mod tests { ); let const_array = ConstantArray::new(fsl_scalar, 2).into_array(); + #[expect(deprecated)] let canonical = const_array.to_fixed_size_list(); assert_eq!(canonical.len(), 2); assert_eq!(canonical.list_size(), 2); // Check elements are repeated correctly. + #[expect(deprecated)] let elements = canonical.elements().to_varbinview(); - assert_eq!(elements.scalar_at(0).unwrap(), "hello".into()); - assert_eq!(elements.scalar_at(1).unwrap(), "world".into()); - assert_eq!(elements.scalar_at(2).unwrap(), "hello".into()); - assert_eq!(elements.scalar_at(3).unwrap(), "world".into()); + assert_eq!( + elements + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + "hello".into() + ); + assert_eq!( + elements + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + "world".into() + ); + assert_eq!( + elements + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + "hello".into() + ); + assert_eq!( + elements + .execute_scalar(3, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + "world".into() + ); } #[test] @@ -617,11 +695,13 @@ mod tests { ); let const_array = ConstantArray::new(fsl_scalar, 1).into_array(); + #[expect(deprecated)] let canonical = const_array.to_fixed_size_list(); assert_eq!(canonical.len(), 1); assert_eq!(canonical.list_size(), 1); + #[expect(deprecated)] let elements = canonical.elements().to_primitive(); assert_arrays_eq!(elements, PrimitiveArray::from_iter([42i16])); } @@ -640,6 +720,7 @@ mod tests { ); let const_array = ConstantArray::new(fsl_scalar, 3).into_array(); + #[expect(deprecated)] let canonical = const_array.to_fixed_size_list(); assert_eq!(canonical.len(), 3); @@ -647,13 +728,26 @@ mod tests { assert!(matches!(canonical.validity(), Ok(Validity::NonNullable))); // Check elements including nulls. + #[expect(deprecated)] let elements = canonical.elements().to_primitive(); - assert_eq!(elements.scalar_at(0).unwrap(), Scalar::from(100i32)); assert_eq!( - elements.scalar_at(1).unwrap(), + elements + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + Scalar::from(100i32) + ); + assert_eq!( + elements + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::null(DType::Primitive(PType::I32, Nullability::Nullable)) ); - assert_eq!(elements.scalar_at(2).unwrap(), Scalar::from(200i32)); + assert_eq!( + elements + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + Scalar::from(200i32) + ); // Check element validity. let element_validity = elements @@ -685,22 +779,49 @@ mod tests { ); let const_array = ConstantArray::new(fsl_scalar, 1000).into_array(); + #[expect(deprecated)] let canonical = const_array.to_fixed_size_list(); assert_eq!(canonical.len(), 1000); assert_eq!(canonical.list_size(), 5); + #[expect(deprecated)] let elements = canonical.elements().to_primitive(); assert_eq!(elements.len(), 5000); // Check pattern repeats correctly. for i in 0..1000 { let base = i * 5; - assert_eq!(elements.scalar_at(base).unwrap(), Scalar::from(1u8)); - assert_eq!(elements.scalar_at(base + 1).unwrap(), Scalar::from(2u8)); - assert_eq!(elements.scalar_at(base + 2).unwrap(), Scalar::from(3u8)); - assert_eq!(elements.scalar_at(base + 3).unwrap(), Scalar::from(4u8)); - assert_eq!(elements.scalar_at(base + 4).unwrap(), Scalar::from(5u8)); + assert_eq!( + elements + .execute_scalar(base, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + Scalar::from(1u8) + ); + assert_eq!( + elements + .execute_scalar(base + 1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + Scalar::from(2u8) + ); + assert_eq!( + elements + .execute_scalar(base + 2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + Scalar::from(3u8) + ); + assert_eq!( + elements + .execute_scalar(base + 3, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + Scalar::from(4u8) + ); + assert_eq!( + elements + .execute_scalar(base + 4, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + Scalar::from(5u8) + ); } } } diff --git a/vortex-array/src/arrays/constant/vtable/mod.rs b/vortex-array/src/arrays/constant/vtable/mod.rs index d6f1c6ab2d3..7384505d854 100644 --- a/vortex-array/src/arrays/constant/vtable/mod.rs +++ b/vortex-array/src/arrays/constant/vtable/mod.rs @@ -11,6 +11,7 @@ use vortex_error::VortexResult; use vortex_error::vortex_ensure; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::ArrayEq; use crate::ArrayHash; @@ -51,10 +52,6 @@ pub type ConstantArray = Array; #[derive(Clone, Debug)] pub struct Constant; -impl Constant { - pub const ID: ArrayId = ArrayId::new_ref("vortex.constant"); -} - impl ArrayHash for ConstantData { fn array_hash(&self, state: &mut H, _precision: Precision) { self.scalar.hash(state); @@ -74,7 +71,8 @@ impl VTable for Constant { type ValidityVTable = Self; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.constant"); + *ID } fn validate( @@ -162,9 +160,10 @@ impl VTable for Constant { PARENT_RULES.evaluate(array, parent, child_idx) } - fn execute(array: Array, _ctx: &mut ExecutionCtx) -> VortexResult { + fn execute(array: Array, ctx: &mut ExecutionCtx) -> VortexResult { Ok(ExecutionResult::done(constant_canonicalize( array.as_view(), + ctx, )?)) } @@ -270,10 +269,11 @@ fn append_value_or_nulls( #[cfg(test)] mod tests { use rstest::rstest; + use vortex_error::VortexResult; use vortex_session::VortexSession; - use crate::ExecutionCtx; use crate::IntoArray; + use crate::VortexSessionExecute; use crate::arrays::ConstantArray; use crate::arrays::constant::vtable::canonical::constant_canonicalize; use crate::assert_arrays_eq; @@ -284,34 +284,29 @@ mod tests { use crate::dtype::StructFields; use crate::scalar::Scalar; - fn ctx() -> ExecutionCtx { - ExecutionCtx::new(VortexSession::empty()) - } - /// Appends `array` into a fresh builder and asserts the result matches `constant_canonicalize`. - fn assert_append_matches_canonical(array: ConstantArray) -> vortex_error::VortexResult<()> { - let expected = constant_canonicalize(array.as_view())?.into_array(); + fn assert_append_matches_canonical(array: ConstantArray) -> VortexResult<()> { + let mut ctx = VortexSession::empty().create_execution_ctx(); + + let expected = constant_canonicalize(array.as_view(), &mut ctx)?.into_array(); let mut builder = builder_with_capacity(array.dtype(), array.len()); array .into_array() - .append_to_builder(builder.as_mut(), &mut ctx())?; + .append_to_builder(builder.as_mut(), &mut ctx)?; let result = builder.finish(); assert_arrays_eq!(&result, &expected); Ok(()) } #[test] - fn test_null_constant_append() -> vortex_error::VortexResult<()> { + fn test_null_constant_append() -> VortexResult<()> { assert_append_matches_canonical(ConstantArray::new(Scalar::null(DType::Null), 5)) } #[rstest] #[case::bool_true(true, 5)] #[case::bool_false(false, 3)] - fn test_bool_constant_append( - #[case] value: bool, - #[case] n: usize, - ) -> vortex_error::VortexResult<()> { + fn test_bool_constant_append(#[case] value: bool, #[case] n: usize) -> VortexResult<()> { assert_append_matches_canonical(ConstantArray::new( Scalar::bool(value, Nullability::NonNullable), n, @@ -319,7 +314,7 @@ mod tests { } #[test] - fn test_bool_null_constant_append() -> vortex_error::VortexResult<()> { + fn test_bool_null_constant_append() -> VortexResult<()> { assert_append_matches_canonical(ConstantArray::new( Scalar::null(DType::Bool(Nullability::Nullable)), 4, @@ -334,7 +329,7 @@ mod tests { fn test_primitive_constant_append( #[case] scalar: Scalar, #[case] n: usize, - ) -> vortex_error::VortexResult<()> { + ) -> VortexResult<()> { assert_append_matches_canonical(ConstantArray::new(scalar, n)) } @@ -343,10 +338,7 @@ mod tests { #[case::utf8_noninline("hello world!!", 5)] // >12 bytes: requires buffer block #[case::utf8_empty("", 3)] #[case::utf8_n_zero("hello world!!", 0)] // n=0 with non-inline: must not write orphaned bytes - fn test_utf8_constant_append( - #[case] value: &str, - #[case] n: usize, - ) -> vortex_error::VortexResult<()> { + fn test_utf8_constant_append(#[case] value: &str, #[case] n: usize) -> VortexResult<()> { assert_append_matches_canonical(ConstantArray::new( Scalar::utf8(value, Nullability::NonNullable), n, @@ -354,7 +346,7 @@ mod tests { } #[test] - fn test_utf8_null_constant_append() -> vortex_error::VortexResult<()> { + fn test_utf8_null_constant_append() -> VortexResult<()> { assert_append_matches_canonical(ConstantArray::new( Scalar::null(DType::Utf8(Nullability::Nullable)), 4, @@ -364,10 +356,7 @@ mod tests { #[rstest] #[case::binary_inline(vec![1u8, 2, 3], 5)] // ≤12 bytes: inlined #[case::binary_noninline(vec![0u8; 13], 5)] // >12 bytes: buffer block - fn test_binary_constant_append( - #[case] value: Vec, - #[case] n: usize, - ) -> vortex_error::VortexResult<()> { + fn test_binary_constant_append(#[case] value: Vec, #[case] n: usize) -> VortexResult<()> { assert_append_matches_canonical(ConstantArray::new( Scalar::binary(value, Nullability::NonNullable), n, @@ -375,7 +364,7 @@ mod tests { } #[test] - fn test_binary_null_constant_append() -> vortex_error::VortexResult<()> { + fn test_binary_null_constant_append() -> VortexResult<()> { assert_append_matches_canonical(ConstantArray::new( Scalar::null(DType::Binary(Nullability::Nullable)), 4, @@ -383,7 +372,7 @@ mod tests { } #[test] - fn test_struct_constant_append() -> vortex_error::VortexResult<()> { + fn test_struct_constant_append() -> VortexResult<()> { let fields = StructFields::new( ["x", "y"].into(), vec![ @@ -402,7 +391,7 @@ mod tests { } #[test] - fn test_null_struct_constant_append() -> vortex_error::VortexResult<()> { + fn test_null_struct_constant_append() -> VortexResult<()> { let fields = StructFields::new( ["x"].into(), vec![DType::Primitive(PType::I32, Nullability::Nullable)], diff --git a/vortex-array/src/arrays/datetime/test.rs b/vortex-array/src/arrays/datetime/test.rs index 3d270ec7053..2b1a681e7fb 100644 --- a/vortex-array/src/arrays/datetime/test.rs +++ b/vortex-array/src/arrays/datetime/test.rs @@ -8,7 +8,8 @@ use vortex_error::VortexResult; use crate::IntoArray; use crate::Precision; -use crate::ToCanonical; +#[expect(deprecated)] +use crate::ToCanonical as _; use crate::arrays::PrimitiveArray; use crate::arrays::datetime::TemporalData; use crate::assert_arrays_eq; @@ -195,11 +196,10 @@ fn test_validity_preservation(#[case] validity: Validity) { let temporal_array = TemporalData::new_timestamp(milliseconds, TimeUnit::Milliseconds, Some("UTC".into())); + #[expect(deprecated)] + let prim = temporal_array.temporal_values().to_primitive(); assert!( - temporal_array - .temporal_values() - .to_primitive() - .validity() + prim.validity() .vortex_expect("temporal validity should be derivable") .array_eq(&validity, Precision::Ptr) ); diff --git a/vortex-array/src/arrays/decimal/array.rs b/vortex-array/src/arrays/decimal/array.rs index 7bddbb8afb2..eb461337752 100644 --- a/vortex-array/src/arrays/decimal/array.rs +++ b/vortex-array/src/arrays/decimal/array.rs @@ -145,7 +145,10 @@ pub trait DecimalArrayExt: TypedArrayRef { } fn validity(&self) -> Validity { - child_to_validity(&self.as_ref().slots()[VALIDITY_SLOT], self.nullability()) + child_to_validity( + self.as_ref().slots()[VALIDITY_SLOT].as_ref(), + self.nullability(), + ) } fn values_type(&self) -> DecimalType { @@ -529,7 +532,7 @@ impl Array { } } - #[allow( + #[expect( clippy::cognitive_complexity, reason = "patching depends on both patch and value physical types" )] diff --git a/vortex-array/src/arrays/decimal/compute/cast.rs b/vortex-array/src/arrays/decimal/compute/cast.rs index 80377102b7d..34ccc27d094 100644 --- a/vortex-array/src/arrays/decimal/compute/cast.rs +++ b/vortex-array/src/arrays/decimal/compute/cast.rs @@ -18,12 +18,53 @@ use crate::dtype::DecimalType; use crate::dtype::NativeDecimalType; use crate::match_each_decimal_value_type; use crate::scalar_fn::fns::cast::CastKernel; +use crate::scalar_fn::fns::cast::CastReduce; + +impl CastReduce for Decimal { + fn cast(array: ArrayView<'_, Decimal>, dtype: &DType) -> VortexResult> { + // Only nullability changes within the same decimal dtype are reducible without execution. + // Precision/scale changes need the kernel. + let DType::Decimal(to_decimal_dtype, to_nullability) = dtype else { + return Ok(None); + }; + let DType::Decimal(from_decimal_dtype, _) = array.dtype() else { + vortex_panic!( + "DecimalArray must have decimal dtype, got {:?}", + array.dtype() + ); + }; + + if from_decimal_dtype != to_decimal_dtype { + return Ok(None); + } + + let Some(new_validity) = array + .validity()? + .trivial_cast_nullability(*to_nullability, array.len())? + else { + return Ok(None); + }; + + // SAFETY: validity has the same length, only its nullability tag changes. + unsafe { + Ok(Some( + DecimalArray::new_unchecked_handle( + array.buffer_handle().clone(), + array.values_type(), + *to_decimal_dtype, + new_validity, + ) + .into_array(), + )) + } + } +} impl CastKernel for Decimal { fn cast( array: ArrayView<'_, Decimal>, dtype: &DType, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult> { // Early return if not casting to decimal let DType::Decimal(to_decimal_dtype, to_nullability) = dtype else { @@ -62,7 +103,7 @@ impl CastKernel for Decimal { // Cast the validity to the new nullability let new_validity = array .validity()? - .cast_nullability(*to_nullability, array.len())?; + .cast_nullability(*to_nullability, array.len(), ctx)?; // If the target needs a wider physical type, upcast the values let target_values_type = DecimalType::smallest_decimal_value_type(to_decimal_dtype); @@ -146,9 +187,12 @@ mod tests { use super::upcast_decimal_values; use crate::IntoArray; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::arrays::DecimalArray; use crate::builtins::ArrayBuiltins; - use crate::canonical::ToCanonical; + #[expect(deprecated)] + use crate::canonical::ToCanonical as _; use crate::compute::conformance::cast::test_cast_conformance; use crate::dtype::DType; use crate::dtype::DecimalDType; @@ -167,6 +211,7 @@ mod tests { // Cast to nullable let nullable_dtype = DType::Decimal(decimal_dtype, Nullability::Nullable); + #[expect(deprecated)] let casted = array .into_array() .cast(nullable_dtype.clone()) @@ -187,6 +232,7 @@ mod tests { // Cast to non-nullable let non_nullable_dtype = DType::Decimal(decimal_dtype, Nullability::NonNullable); + #[expect(deprecated)] let casted = array .into_array() .cast(non_nullable_dtype.clone()) @@ -207,11 +253,12 @@ mod tests { // Attempt to cast to non-nullable should fail let non_nullable_dtype = DType::Decimal(decimal_dtype, Nullability::NonNullable); - array + #[expect(deprecated)] + let result = array .into_array() .cast(non_nullable_dtype) - .and_then(|a| a.to_canonical().map(|c| c.into_array())) - .unwrap(); + .and_then(|a| a.to_canonical().map(|c| c.into_array())); + result.unwrap(); } #[test] @@ -224,6 +271,7 @@ mod tests { // Try to cast to different scale - not supported let different_dtype = DType::Decimal(DecimalDType::new(15, 3), Nullability::NonNullable); + #[expect(deprecated)] let result = array .into_array() .cast(different_dtype) @@ -248,6 +296,7 @@ mod tests { // Try to downcast precision - not supported let smaller_dtype = DType::Decimal(DecimalDType::new(10, 2), Nullability::NonNullable); + #[expect(deprecated)] let result = array .into_array() .cast(smaller_dtype) @@ -272,6 +321,7 @@ mod tests { // Cast to higher precision with same scale - should succeed let wider_dtype = DType::Decimal(DecimalDType::new(38, 2), Nullability::NonNullable); + #[expect(deprecated)] let casted = array.into_array().cast(wider_dtype).unwrap().to_decimal(); assert_eq!(casted.precision(), 38); @@ -290,6 +340,7 @@ mod tests { ); // Try to cast to non-decimal type - should fail since no kernel can handle it + #[expect(deprecated)] let result = array .into_array() .cast(DType::Utf8(Nullability::NonNullable)) @@ -383,7 +434,15 @@ mod tests { assert_eq!(casted.len(), 3); // Check validity is preserved - let mask = casted.validity_mask().unwrap(); + let mask = casted + .as_ref() + .validity() + .unwrap() + .execute_mask( + casted.as_ref().len(), + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); assert!(mask.value(0)); assert!(!mask.value(1)); assert!(mask.value(2)); diff --git a/vortex-array/src/arrays/decimal/compute/fill_null.rs b/vortex-array/src/arrays/decimal/compute/fill_null.rs index e697a277cca..144c90c74f9 100644 --- a/vortex-array/src/arrays/decimal/compute/fill_null.rs +++ b/vortex-array/src/arrays/decimal/compute/fill_null.rs @@ -89,10 +89,13 @@ mod tests { use vortex_buffer::buffer; use crate::IntoArray; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::arrays::DecimalArray; use crate::assert_arrays_eq; use crate::builtins::ArrayBuiltins; - use crate::canonical::ToCanonical; + #[expect(deprecated)] + use crate::canonical::ToCanonical as _; use crate::dtype::DecimalDType; use crate::dtype::Nullability; use crate::scalar::DecimalValue; @@ -106,6 +109,7 @@ mod tests { [None, Some(800i128), None, Some(1000i128), None], decimal_dtype, ); + #[expect(deprecated)] let p = arr .into_array() .fill_null(Scalar::decimal( @@ -123,7 +127,14 @@ mod tests { p.buffer::().as_slice(), vec![4200, 800, 4200, 1000, 4200] ); - assert!(p.validity_mask().unwrap().all_true()); + assert!( + p.as_ref() + .validity() + .unwrap() + .execute_mask(p.as_ref().len(), &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .all_true() + ); } #[test] @@ -135,6 +146,7 @@ mod tests { decimal_dtype, ); + #[expect(deprecated)] let p = arr .into_array() .fill_null(Scalar::decimal( @@ -156,6 +168,7 @@ mod tests { let decimal_dtype = DecimalDType::new(3, 0); let arr = DecimalArray::from_option_iter([None, Some(10i8), None], decimal_dtype); // i8 max is 127, so 200 doesn't fit — the array should be widened to i16. + #[expect(deprecated)] let result = arr .into_array() .fill_null(Scalar::decimal( @@ -180,6 +193,7 @@ mod tests { decimal_dtype, Validity::NonNullable, ); + #[expect(deprecated)] let p = arr .into_array() .fill_null(Scalar::decimal( diff --git a/vortex-array/src/arrays/decimal/compute/rules.rs b/vortex-array/src/arrays/decimal/compute/rules.rs index fae0c3dd866..31df664e5c7 100644 --- a/vortex-array/src/arrays/decimal/compute/rules.rs +++ b/vortex-array/src/arrays/decimal/compute/rules.rs @@ -16,10 +16,12 @@ use crate::arrays::slice::SliceReduceAdaptor; use crate::match_each_decimal_value_type; use crate::optimizer::rules::ArrayParentReduceRule; use crate::optimizer::rules::ParentRuleSet; +use crate::scalar_fn::fns::cast::CastReduceAdaptor; use crate::scalar_fn::fns::mask::MaskReduceAdaptor; pub(crate) static RULES: ParentRuleSet = ParentRuleSet::new(&[ ParentRuleSet::lift(&DecimalMaskedValidityRule), + ParentRuleSet::lift(&CastReduceAdaptor(Decimal)), ParentRuleSet::lift(&MaskReduceAdaptor(Decimal)), ParentRuleSet::lift(&SliceReduceAdaptor(Decimal)), ]); diff --git a/vortex-array/src/arrays/decimal/vtable/mod.rs b/vortex-array/src/arrays/decimal/vtable/mod.rs index 1894466ed6b..5360aba5eb0 100644 --- a/vortex-array/src/arrays/decimal/vtable/mod.rs +++ b/vortex-array/src/arrays/decimal/vtable/mod.rs @@ -32,6 +32,8 @@ mod validity; use std::hash::Hash; +use vortex_session::registry::CachedId; + use crate::Precision; use crate::array::ArrayId; use crate::arrays::decimal::array::SLOT_NAMES; @@ -68,7 +70,8 @@ impl VTable for Decimal { type ValidityVTable = Self; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.decimal"); + *ID } fn nbuffers(_array: ArrayView<'_, Self>) -> usize { @@ -118,7 +121,7 @@ impl VTable for Decimal { data.len(), len ); - let validity = crate::array::child_to_validity(&slots[0], *nullability); + let validity = crate::array::child_to_validity(slots[0].as_ref(), *nullability); if let Some(validity_len) = validity.maybe_len() { vortex_ensure!( validity_len == len, @@ -203,10 +206,6 @@ impl VTable for Decimal { #[derive(Clone, Debug)] pub struct Decimal; -impl Decimal { - pub const ID: ArrayId = ArrayId::new_ref("vortex.decimal"); -} - #[cfg(test)] mod tests { use vortex_buffer::ByteBufferMut; diff --git a/vortex-array/src/arrays/decimal/vtable/operations.rs b/vortex-array/src/arrays/decimal/vtable/operations.rs index cfd14d95d63..aca0f912583 100644 --- a/vortex-array/src/arrays/decimal/vtable/operations.rs +++ b/vortex-array/src/arrays/decimal/vtable/operations.rs @@ -32,6 +32,8 @@ mod tests { use vortex_buffer::buffer; use crate::IntoArray; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::arrays::Decimal; use crate::arrays::DecimalArray; use crate::dtype::DecimalDType; @@ -78,7 +80,9 @@ mod tests { ); assert_eq!( - array.scalar_at(0).unwrap(), + array + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::decimal( DecimalValue::I128(100), DecimalDType::new(3, 2), diff --git a/vortex-array/src/arrays/dict/arbitrary.rs b/vortex-array/src/arrays/dict/arbitrary.rs index 0c4f5780874..4eb938181b9 100644 --- a/vortex-array/src/arrays/dict/arbitrary.rs +++ b/vortex-array/src/arrays/dict/arbitrary.rs @@ -13,6 +13,8 @@ use crate::ArrayRef; use crate::IntoArray; use crate::arrays::PrimitiveArray; use crate::arrays::arbitrary::ArbitraryArray; +use crate::arrays::arbitrary::ArbitraryArrayConfig; +use crate::arrays::arbitrary::ArbitraryWith; use crate::arrays::arbitrary::random_validity; use crate::dtype::DType; use crate::dtype::NativePType; @@ -36,7 +38,14 @@ impl ArbitraryDictArray { // Generate the number of unique values (dictionary size) let values_len = u.int_in_range(1..=20)?; // Generate values array with the given dtype - let values = ArbitraryArray::arbitrary_with(u, Some(values_len), dtype)?.0; + let values = ArbitraryArray::arbitrary_with_config( + u, + &ArbitraryArrayConfig { + dtype: Some(dtype.clone()), + len: values_len..=values_len, + }, + )? + .0; // Generate codes that index into the values let codes_len = len.unwrap_or(u.int_in_range(0..=100)?); diff --git a/vortex-array/src/arrays/dict/array.rs b/vortex-array/src/arrays/dict/array.rs index b7d0cc6c0f4..eafbc9f6642 100644 --- a/vortex-array/src/arrays/dict/array.rs +++ b/vortex-array/src/arrays/dict/array.rs @@ -12,7 +12,10 @@ use vortex_error::vortex_ensure; use vortex_mask::AllOr; use crate::ArrayRef; -use crate::ToCanonical; +use crate::LEGACY_SESSION; +#[expect(deprecated)] +use crate::ToCanonical as _; +use crate::VortexSessionExecute; use crate::array::Array; use crate::array::ArrayParts; use crate::array::TypedArrayRef; @@ -138,12 +141,12 @@ pub trait DictArrayExt: TypedArrayRef + DictArraySlotsExt { Ok(()) } - #[allow( - clippy::cognitive_complexity, - reason = "branching depends on validity representation and code type" - )] fn compute_referenced_values_mask(&self, referenced: bool) -> VortexResult { - let codes_validity = self.codes().validity_mask()?; + let codes = self.codes(); + let codes_validity = codes + .validity()? + .execute_mask(codes.len(), &mut LEGACY_SESSION.create_execution_ctx())?; + #[expect(deprecated)] let codes_primitive = self.codes().to_primitive(); let values_len = self.values().len(); @@ -186,6 +189,40 @@ pub trait DictArrayExt: TypedArrayRef + DictArraySlotsExt { } impl> DictArrayExt for T {} +/// Concrete parts of a [`DictArray`](super::DictArray) after iterative execution. +pub struct DictParts { + pub dtype: DType, + pub codes: ArrayRef, + pub values: ArrayRef, +} + +pub trait DictOwnedExt { + fn into_parts(self) -> DictParts; +} + +impl DictOwnedExt for Array { + fn into_parts(self) -> DictParts { + match self.try_into_parts() { + Ok(array_parts) => { + let slots = DictSlots::from_slots(array_parts.slots); + DictParts { + dtype: array_parts.dtype, + codes: slots.codes, + values: slots.values, + } + } + Err(array) => { + let slots = DictSlotsView::from_slots(array.slots()); + DictParts { + dtype: array.dtype().clone(), + codes: slots.codes.clone(), + values: slots.values.clone(), + } + } + } + } +} + impl Array { /// Build a new `DictArray` from its components, `codes` and `values`. pub fn new(codes: ArrayRef, values: ArrayRef) -> Self { @@ -252,8 +289,6 @@ impl Array { #[cfg(test)] mod test { - #[allow(unused_imports)] - use itertools::Itertools; use rand::RngExt; use rand::SeedableRng; use rand::distr::Distribution; @@ -269,7 +304,8 @@ mod test { use crate::ArrayRef; use crate::IntoArray; use crate::LEGACY_SESSION; - use crate::ToCanonical; + #[expect(deprecated)] + use crate::ToCanonical as _; use crate::VortexSessionExecute; use crate::arrays::ChunkedArray; use crate::arrays::DictArray; @@ -294,7 +330,15 @@ mod test { PrimitiveArray::new(buffer![3, 6, 9], Validity::AllValid).into_array(), ) .unwrap(); - let mask = dict.validity_mask().unwrap(); + let mask = dict + .as_ref() + .validity() + .unwrap() + .execute_mask( + dict.as_ref().len(), + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); let AllOr::Some(indices) = mask.indices() else { vortex_panic!("Expected indices from mask") }; @@ -312,7 +356,15 @@ mod test { .into_array(), ) .unwrap(); - let mask = dict.validity_mask().unwrap(); + let mask = dict + .as_ref() + .validity() + .unwrap() + .execute_mask( + dict.as_ref().len(), + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); let AllOr::Some(indices) = mask.indices() else { vortex_panic!("Expected indices from mask") }; @@ -334,7 +386,15 @@ mod test { .into_array(), ) .unwrap(); - let mask = dict.validity_mask().unwrap(); + let mask = dict + .as_ref() + .validity() + .unwrap() + .execute_mask( + dict.as_ref().len(), + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); let AllOr::Some(indices) = mask.indices() else { vortex_panic!("Expected indices from mask") }; @@ -352,7 +412,15 @@ mod test { PrimitiveArray::new(buffer![3, 6, 9], Validity::NonNullable).into_array(), ) .unwrap(); - let mask = dict.validity_mask().unwrap(); + let mask = dict + .as_ref() + .validity() + .unwrap() + .execute_mask( + dict.as_ref().len(), + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); let AllOr::Some(indices) = mask.indices() else { vortex_panic!("Expected indices from mask") }; @@ -400,6 +468,7 @@ mod test { ); array.append_to_builder(builder.as_mut(), &mut LEGACY_SESSION.create_execution_ctx())?; + #[expect(deprecated)] let into_prim = array.to_primitive(); let prim_into = builder.finish_into_canonical().into_primitive(); diff --git a/vortex-array/src/arrays/dict/compute/cast.rs b/vortex-array/src/arrays/dict/compute/cast.rs index 2398ed2c161..ee80a1140b4 100644 --- a/vortex-array/src/arrays/dict/compute/cast.rs +++ b/vortex-array/src/arrays/dict/compute/cast.rs @@ -18,10 +18,7 @@ impl CastReduce for Dict { fn cast(array: ArrayView<'_, Dict>, dtype: &DType) -> VortexResult> { // Can have un-reference null values making the cast of values fail without a possible mask. // TODO(joe): optimize this, could look at accessible values and fill_null not those? - if !dtype.is_nullable() - && array.values().dtype().is_nullable() - && !array.values().all_valid()? - { + if !dtype.is_nullable() && !array.values().validity()?.no_nulls() { return Ok(None); } // Cast the dictionary values to the target type @@ -53,7 +50,8 @@ mod tests { use vortex_buffer::buffer; use crate::IntoArray; - use crate::ToCanonical; + #[expect(deprecated)] + use crate::ToCanonical as _; use crate::arrays::Dict; use crate::arrays::PrimitiveArray; use crate::arrays::dict::DictArraySlotsExt; @@ -79,6 +77,7 @@ mod tests { &DType::Primitive(PType::I64, Nullability::NonNullable) ); + #[expect(deprecated)] let decoded = casted.to_primitive(); assert_arrays_eq!(decoded, PrimitiveArray::from_iter([1i64, 2, 3, 2, 1])); } @@ -163,20 +162,11 @@ mod tests { &DType::Primitive(PType::I32, Nullability::NonNullable) ); - // Check that both codes and values are NonNullable again - let back_dict = back_to_non_nullable.as_::(); - assert_eq!( - back_dict.codes().dtype().nullability(), - Nullability::NonNullable - ); - assert_eq!( - back_dict.values().dtype().nullability(), - Nullability::NonNullable - ); - // Verify values are unchanged + #[expect(deprecated)] let original_values = dict.as_array().to_primitive(); - let final_values = back_dict.array().to_primitive(); + #[expect(deprecated)] + let final_values = back_to_non_nullable.to_primitive(); assert_arrays_eq!(original_values, final_values); } @@ -224,9 +214,8 @@ mod tests { casted.dtype(), &DType::Primitive(PType::F64, Nullability::NonNullable) ); - assert_arrays_eq!( - casted.to_primitive(), - PrimitiveArray::from_iter([1.0f64, 3.0, 1.0]) - ); + #[expect(deprecated)] + let casted_prim = casted.to_primitive(); + assert_arrays_eq!(casted_prim, PrimitiveArray::from_iter([1.0f64, 3.0, 1.0])); } } diff --git a/vortex-array/src/arrays/dict/compute/fill_null.rs b/vortex-array/src/arrays/dict/compute/fill_null.rs index fb1b4034bde..189d8240050 100644 --- a/vortex-array/src/arrays/dict/compute/fill_null.rs +++ b/vortex-array/src/arrays/dict/compute/fill_null.rs @@ -94,7 +94,10 @@ mod tests { use vortex_error::VortexExpect; use crate::IntoArray; - use crate::ToCanonical; + use crate::LEGACY_SESSION; + #[expect(deprecated)] + use crate::ToCanonical as _; + use crate::VortexSessionExecute; use crate::arrays::DictArray; use crate::arrays::PrimitiveArray; use crate::assert_arrays_eq; @@ -119,8 +122,13 @@ mod tests { .into_array() .fill_null(Scalar::primitive(20, Nullability::NonNullable)) .vortex_expect("operation should succeed in test"); + #[expect(deprecated)] let filled_primitive = filled.to_primitive(); assert_arrays_eq!(filled_primitive, PrimitiveArray::from_iter([10, 20, 20])); - assert!(filled_primitive.all_valid().unwrap()); + assert!( + filled_primitive + .all_valid(&mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); } } diff --git a/vortex-array/src/arrays/dict/compute/is_constant.rs b/vortex-array/src/arrays/dict/compute/is_constant.rs index f5a903a126f..82518963106 100644 --- a/vortex-array/src/arrays/dict/compute/is_constant.rs +++ b/vortex-array/src/arrays/dict/compute/is_constant.rs @@ -40,7 +40,7 @@ impl DynAggregateKernel for DictIsConstantKernel { // If codes are constant, only one dictionary value is referenced → constant. if is_constant(dict.codes(), ctx)? { - return Ok(Some(IsConstant::make_partial(batch, true)?)); + return Ok(Some(IsConstant::make_partial(batch, true, ctx)?)); } // Otherwise, check the values array. Filter to only referenced values if needed. @@ -53,6 +53,6 @@ impl DynAggregateKernel for DictIsConstantKernel { is_constant(&filtered_values, ctx)? }; - Ok(Some(IsConstant::make_partial(batch, result)?)) + Ok(Some(IsConstant::make_partial(batch, result, ctx)?)) } } diff --git a/vortex-array/src/arrays/dict/compute/is_sorted.rs b/vortex-array/src/arrays/dict/compute/is_sorted.rs index cb20287ae83..6ab5d5727d5 100644 --- a/vortex-array/src/arrays/dict/compute/is_sorted.rs +++ b/vortex-array/src/arrays/dict/compute/is_sorted.rs @@ -44,7 +44,7 @@ impl DynAggregateKernel for DictIsSortedKernel { }; if result { - Ok(Some(IsSorted::make_partial(batch, true, strict)?)) + Ok(Some(IsSorted::make_partial(batch, true, strict, ctx)?)) } else { // We can't definitively say it's NOT sorted without canonicalizing, // so return None to let the accumulator handle it. diff --git a/vortex-array/src/arrays/dict/compute/min_max.rs b/vortex-array/src/arrays/dict/compute/min_max.rs index 8abd34dbc0a..b390005adcf 100644 --- a/vortex-array/src/arrays/dict/compute/min_max.rs +++ b/vortex-array/src/arrays/dict/compute/min_max.rs @@ -92,42 +92,73 @@ mod tests { Ok(()) } - #[rstest] - #[case::covering( + fn dict_covering() -> DictArray { DictArray::try_new( buffer![0u32, 1, 2, 3, 0, 1].into_array(), buffer![10i32, 20, 30, 40].into_array(), - ).unwrap(), - (10, 40) - )] - #[case::non_covering_duplicates( + ) + .expect("valid test dictionary") + } + + fn dict_non_covering_duplicates() -> DictArray { DictArray::try_new( buffer![1u32, 1, 1, 3, 3].into_array(), buffer![1i32, 2, 3, 4, 5].into_array(), - ).unwrap(), - (2, 4) - )] - #[case::non_covering_gaps( + ) + .expect("valid test dictionary") + } + + fn dict_non_covering_gaps() -> DictArray { DictArray::try_new( buffer![0u32, 2, 4].into_array(), buffer![1i32, 2, 3, 4, 5].into_array(), - ).unwrap(), - (1, 5) - )] - #[case::single(dict_encode(&buffer![42i32].into_array()).unwrap(), (42, 42))] - #[case::nullable_codes( + ) + .expect("valid test dictionary") + } + + fn dict_single() -> DictArray { + dict_encode(&buffer![42i32].into_array()).expect("valid single-value dictionary") + } + + fn dict_nullable_codes() -> DictArray { DictArray::try_new( PrimitiveArray::from_option_iter([Some(0u32), None, Some(1), Some(2)]).into_array(), buffer![10i32, 20, 30].into_array(), - ).unwrap(), - (10, 30) - )] - #[case::nullable_values( + ) + .expect("valid nullable-code dictionary") + } + + fn dict_nullable_values() -> DictArray { dict_encode( - &PrimitiveArray::from_option_iter([Some(1i32), None, Some(2), Some(1), None]).into_array() - ).unwrap(), - (1, 2) - )] + &PrimitiveArray::from_option_iter([Some(1i32), None, Some(2), Some(1), None]) + .into_array(), + ) + .expect("valid nullable-value dictionary") + } + + fn dict_empty() -> DictArray { + DictArray::try_new( + PrimitiveArray::from_iter(Vec::::new()).into_array(), + buffer![10i32, 20, 30].into_array(), + ) + .expect("valid empty dictionary") + } + + fn dict_all_null_codes() -> DictArray { + DictArray::try_new( + PrimitiveArray::from_option_iter([Option::::None, None, None]).into_array(), + buffer![10i32, 20, 30].into_array(), + ) + .expect("valid all-null-code dictionary") + } + + #[rstest] + #[case::covering(dict_covering(), (10, 40))] + #[case::non_covering_duplicates(dict_non_covering_duplicates(), (2, 4))] + #[case::non_covering_gaps(dict_non_covering_gaps(), (1, 5))] + #[case::single(dict_single(), (42, 42))] + #[case::nullable_codes(dict_nullable_codes(), (10, 30))] + #[case::nullable_values(dict_nullable_values(), (1, 2))] fn test_min_max(#[case] dict: DictArray, #[case] expected: (i32, i32)) -> VortexResult<()> { assert_min_max(&dict.into_array(), Some(expected)) } @@ -141,18 +172,8 @@ mod tests { } #[rstest] - #[case::empty( - DictArray::try_new( - PrimitiveArray::from_iter(Vec::::new()).into_array(), - buffer![10i32, 20, 30].into_array(), - ).unwrap() - )] - #[case::all_null_codes( - DictArray::try_new( - PrimitiveArray::from_option_iter([Option::::None, None, None]).into_array(), - buffer![10i32, 20, 30].into_array(), - ).unwrap() - )] + #[case::empty(dict_empty())] + #[case::all_null_codes(dict_all_null_codes())] fn test_min_max_none(#[case] dict: DictArray) -> VortexResult<()> { assert_min_max(&dict.into_array(), None) } diff --git a/vortex-array/src/arrays/dict/compute/mod.rs b/vortex-array/src/arrays/dict/compute/mod.rs index 575dab50f50..c56cc8ef367 100644 --- a/vortex-array/src/arrays/dict/compute/mod.rs +++ b/vortex-array/src/arrays/dict/compute/mod.rs @@ -54,13 +54,14 @@ impl FilterReduce for Dict { #[cfg(test)] mod test { - #[allow(unused_imports)] + #[expect(unused_imports)] use itertools::Itertools; use vortex_buffer::buffer; use crate::ArrayRef; use crate::IntoArray; - use crate::ToCanonical; + #[expect(deprecated)] + use crate::ToCanonical as _; use crate::accessor::ArrayAccessor; use crate::arrays::ConstantArray; use crate::arrays::PrimitiveArray; @@ -89,6 +90,7 @@ mod test { let dict = dict_encode(&PrimitiveArray::from_option_iter(values.clone()).into_array()).unwrap(); + #[expect(deprecated)] let actual = dict.as_array().to_primitive(); let expected = PrimitiveArray::from_option_iter(values); @@ -102,6 +104,7 @@ mod test { let expected = PrimitiveArray::from_iter((0..1000).map(|i| unique_values[i % 32])); let dict = dict_encode(&expected.clone().into_array()).unwrap(); + #[expect(deprecated)] let actual = dict.as_array().to_primitive(); assert_arrays_eq!(actual, expected); @@ -113,6 +116,7 @@ mod test { let expected = PrimitiveArray::from_iter((0..1000).map(|i| unique_values[i % 100])); let dict = dict_encode(&expected.clone().into_array()).unwrap(); + #[expect(deprecated)] let actual = dict.as_array().to_primitive(); assert_arrays_eq!(actual, expected); @@ -126,6 +130,7 @@ mod test { ); assert_eq!(reference.len(), 6); let dict = dict_encode(&reference.clone().into_array()).unwrap(); + #[expect(deprecated)] let flattened_dict = dict.as_array().to_varbinview(); assert_eq!( flattened_dict.with_iterator(|iter| iter diff --git a/vortex-array/src/arrays/dict/compute/rules.rs b/vortex-array/src/arrays/dict/compute/rules.rs index c6817a39d58..f6fe816a6cc 100644 --- a/vortex-array/src/arrays/dict/compute/rules.rs +++ b/vortex-array/src/arrays/dict/compute/rules.rs @@ -8,12 +8,13 @@ use crate::ArrayRef; use crate::IntoArray; use crate::Precision; use crate::array::ArrayView; +use crate::array::VTable; use crate::arrays::Constant; use crate::arrays::ConstantArray; use crate::arrays::Dict; use crate::arrays::DictArray; +use crate::arrays::ScalarFn; use crate::arrays::ScalarFnArray; -use crate::arrays::ScalarFnVTable; use crate::arrays::dict::DictArraySlotsExt; use crate::arrays::filter::FilterReduceAdaptor; use crate::arrays::scalar_fn::AnyScalarFn; @@ -28,6 +29,7 @@ use crate::scalar_fn::fns::cast::CastReduceAdaptor; use crate::scalar_fn::fns::like::LikeReduceAdaptor; use crate::scalar_fn::fns::mask::MaskReduceAdaptor; use crate::scalar_fn::fns::pack::Pack; +use crate::validity::Validity; pub(crate) const PARENT_RULES: ParentRuleSet = ParentRuleSet::new(&[ ParentRuleSet::lift(&FilterReduceAdaptor(Dict)), @@ -49,7 +51,7 @@ impl ArrayParentReduceRule for DictionaryScalarFnValuesPushDownRule { fn reduce_parent( &self, array: ArrayView<'_, Dict>, - parent: ArrayView<'_, ScalarFnVTable>, + parent: ArrayView<'_, ScalarFn>, child_idx: usize, ) -> VortexResult> { // Check that the scalar function can actually be pushed down. @@ -80,7 +82,7 @@ impl ArrayParentReduceRule for DictionaryScalarFnValuesPushDownRule { tracing::trace!( "Not pushing down fallible scalar function {} over dictionary with sparse codes {}", parent.scalar_fn(), - Dict::ID, + Dict.id(), ); return Ok(None); } @@ -98,13 +100,16 @@ impl ArrayParentReduceRule for DictionaryScalarFnValuesPushDownRule { // If the scalar function is null-sensitive, then we cannot push it down to values if // we have any nulls in the codes. if array.codes().dtype().is_nullable() - && !array.codes().all_valid()? + && !matches!( + array.codes().validity()?, + Validity::NonNullable | Validity::AllValid + ) && sig.is_null_sensitive() { tracing::trace!( "Not pushing down null-sensitive scalar function {} over dictionary with null codes {}", parent.scalar_fn(), - Dict::ID, + Dict.id(), ); return Ok(None); } @@ -151,7 +156,7 @@ impl ArrayParentReduceRule for DictionaryScalarFnCodesPullUpRule { fn reduce_parent( &self, array: ArrayView<'_, Dict>, - parent: ArrayView<'_, ScalarFnVTable>, + parent: ArrayView<'_, ScalarFn>, child_idx: usize, ) -> VortexResult> { // Don't attempt to pull up if there are less than 2 siblings. diff --git a/vortex-array/src/arrays/dict/execute.rs b/vortex-array/src/arrays/dict/execute.rs index d95cd239a44..5b73ec39d9e 100644 --- a/vortex-array/src/arrays/dict/execute.rs +++ b/vortex-array/src/arrays/dict/execute.rs @@ -7,6 +7,7 @@ use vortex_error::VortexExpect; use vortex_error::VortexResult; use crate::Canonical; +use crate::CanonicalView; use crate::ExecutionCtx; use crate::IntoArray; use crate::arrays::Bool; @@ -35,18 +36,19 @@ use crate::arrays::variant::VariantArrayExt; /// /// This is the core operation for dictionary decoding - it expands the dictionary /// by looking up each code in the values array. -pub fn take_canonical( - values: Canonical, +pub(crate) fn take_canonical( + values: CanonicalView, codes: &PrimitiveArray, ctx: &mut ExecutionCtx, ) -> VortexResult { + let values = Canonical::from(values); Ok(match values { Canonical::Null(a) => Canonical::Null(take_null(&a, codes)), Canonical::Bool(a) => Canonical::Bool(take_bool(&a, codes, ctx)?), Canonical::Primitive(a) => Canonical::Primitive(take_primitive(&a, codes, ctx)), Canonical::Decimal(a) => Canonical::Decimal(take_decimal(&a, codes, ctx)), Canonical::VarBinView(a) => Canonical::VarBinView(take_varbinview(&a, codes, ctx)), - Canonical::List(a) => Canonical::List(take_listview(&a, codes)), + Canonical::List(a) => Canonical::List(take_listview(&a, codes, ctx)), Canonical::FixedSizeList(a) => { Canonical::FixedSizeList(take_fixed_size_list(&a, codes, ctx)) } @@ -123,12 +125,16 @@ fn take_varbinview( .into_owned() } -fn take_listview(array: &ListViewArray, codes: &PrimitiveArray) -> ListViewArray { +fn take_listview( + array: &ListViewArray, + codes: &PrimitiveArray, + ctx: &mut ExecutionCtx, +) -> ListViewArray { let codes_ref = codes.clone().into_array(); let array = array.as_view(); - ::take(array, &codes_ref) - .vortex_expect("take listview array") - .vortex_expect("take listview should not return None") + ::take(array, &codes_ref, ctx) + .vortex_expect("take listview execute") + .vortex_expect("ListView TakeExecute should not return None") .as_::() .into_owned() } diff --git a/vortex-array/src/arrays/dict/mod.rs b/vortex-array/src/arrays/dict/mod.rs index 7ba509e09e5..0414eea7def 100644 --- a/vortex-array/src/arrays/dict/mod.rs +++ b/vortex-array/src/arrays/dict/mod.rs @@ -17,8 +17,6 @@ pub use array::*; pub(crate) mod compute; mod execute; -pub use execute::take_canonical; - mod take; pub use take::*; diff --git a/vortex-array/src/arrays/dict/take.rs b/vortex-array/src/arrays/dict/take.rs index 7886f0b898a..edec1a3f83c 100644 --- a/vortex-array/src/arrays/dict/take.rs +++ b/vortex-array/src/arrays/dict/take.rs @@ -21,6 +21,7 @@ use crate::matcher::Matcher; use crate::optimizer::rules::ArrayParentReduceRule; use crate::scalar::Scalar; use crate::stats::StatsSet; +use crate::validity::Validity; pub trait TakeReduce: VTable { /// Take elements from an array at the given indices without reading buffers. @@ -99,7 +100,7 @@ where return Ok(Some(result)); } let result = ::take(array, parent.codes())?; - if let Some(ref taken) = result { + if let Some(taken) = &result { propagate_take_stats(array.array(), taken, parent.codes())?; } Ok(result) @@ -130,7 +131,7 @@ where return Ok(Some(result)); } let result = ::take(array, parent.codes(), ctx)?; - if let Some(ref taken) = result { + if let Some(taken) = &result { propagate_take_stats(array.array(), taken, parent.codes())?; } Ok(result) @@ -142,8 +143,12 @@ pub(crate) fn propagate_take_stats( target: &ArrayRef, indices: &ArrayRef, ) -> VortexResult<()> { + let indices_all_valid = matches!( + indices.validity()?, + Validity::NonNullable | Validity::AllValid + ); target.statistics().with_mut_typed_stats_set(|mut st| { - if indices.all_valid().unwrap_or(false) { + if indices_all_valid { let is_constant = source.statistics().get_as::(Stat::IsConstant); if is_constant == Some(Precision::Exact(true)) { // Any combination of elements from a constant array is still const diff --git a/vortex-array/src/arrays/dict/vtable/mod.rs b/vortex-array/src/arrays/dict/vtable/mod.rs index b13374b9d8d..9f64e089bc5 100644 --- a/vortex-array/src/arrays/dict/vtable/mod.rs +++ b/vortex-array/src/arrays/dict/vtable/mod.rs @@ -5,19 +5,20 @@ use std::hash::Hasher; use kernel::PARENT_KERNELS; use prost::Message; -use vortex_error::VortexExpect; use vortex_error::VortexResult; use vortex_error::vortex_bail; use vortex_error::vortex_ensure; use vortex_error::vortex_err; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use super::DictData; use super::DictMetadata; +use super::DictOwnedExt; +use super::DictParts; use super::array::DictSlots; use super::array::DictSlotsView; -use super::take_canonical; use crate::AnyCanonical; use crate::ArrayEq; use crate::ArrayHash; @@ -26,6 +27,7 @@ use crate::Canonical; use crate::Precision; use crate::array::Array; use crate::array::ArrayId; +use crate::array::ArrayParts; use crate::array::ArrayView; use crate::array::VTable; use crate::arrays::ConstantArray; @@ -33,6 +35,7 @@ use crate::arrays::Primitive; use crate::arrays::dict::DictArrayExt; use crate::arrays::dict::DictArraySlotsExt; use crate::arrays::dict::compute::rules::PARENT_RULES; +use crate::arrays::dict::execute::take_canonical; use crate::buffer::BufferHandle; use crate::dtype::DType; use crate::dtype::Nullability; @@ -42,6 +45,8 @@ use crate::executor::ExecutionResult; use crate::require_child; use crate::scalar::Scalar; use crate::serde::ArrayChildren; +use crate::validity::Validity; + mod kernel; mod operations; mod validity; @@ -52,10 +57,6 @@ pub type DictArray = Array; #[derive(Clone, Debug)] pub struct Dict; -impl Dict { - pub const ID: ArrayId = ArrayId::new_ref("vortex.dict"); -} - impl ArrayHash for DictData { fn array_hash(&self, _state: &mut H, _precision: Precision) {} } @@ -73,7 +74,8 @@ impl VTable for Dict { type ValidityVTable = Self; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.dict"); + *ID } fn validate( @@ -134,11 +136,10 @@ impl VTable for Dict { dtype: &DType, len: usize, metadata: &[u8], - _buffers: &[BufferHandle], children: &dyn ArrayChildren, _session: &VortexSession, - ) -> VortexResult> { + ) -> VortexResult> { let metadata = DictMetadata::decode(metadata)?; if children.len() != 2 { vortex_bail!( @@ -157,12 +158,10 @@ impl VTable for Dict { let values = children.get(1, dtype, metadata.values_len as usize)?; let all_values_referenced = metadata.all_values_referenced.unwrap_or(false); - Ok( - crate::array::ArrayParts::new(self.clone(), dtype.clone(), len, unsafe { - DictData::new_unchecked().set_all_values_referenced(all_values_referenced) - }) - .with_slots(vec![Some(codes), Some(values)]), - ) + Ok(ArrayParts::new(self.clone(), dtype.clone(), len, unsafe { + DictData::new_unchecked().set_all_values_referenced(all_values_referenced) + }) + .with_slots(vec![Some(codes), Some(values)])) } fn slot_name(_array: ArrayView<'_, Self>, idx: usize) -> String { @@ -179,10 +178,7 @@ impl VTable for Dict { let array = require_child!(array, array.codes(), DictSlots::CODES => Primitive); - // TODO(joe): use stat get instead computing. - // Also not the check to do here it take value validity using code validity, but this approx - // is correct. - if array.codes().all_invalid()? { + if matches!(array.codes().validity()?, Validity::AllInvalid) { return Ok(ExecutionResult::done(ConstantArray::new( Scalar::null(array.dtype().as_nullable()), array.codes().len(), @@ -191,18 +187,13 @@ impl VTable for Dict { let array = require_child!(array, array.values(), DictSlots::VALUES => AnyCanonical); - let codes = array - .codes() - .clone() - .try_downcast::() - .ok() - .vortex_expect("must be primitive"); - let values = array.values().clone(); - debug_assert!(values.is_canonical()); - // TODO: add canonical owned cast. - let values = values.to_canonical()?; - - Ok(ExecutionResult::done(take_canonical(values, &codes, ctx)?)) + let DictParts { values, codes, .. } = array.into_parts(); + + Ok(ExecutionResult::done(take_canonical( + values.as_::(), + &codes.downcast::(), + ctx, + )?)) } fn reduce_parent( diff --git a/vortex-array/src/arrays/dict/vtable/operations.rs b/vortex-array/src/arrays/dict/vtable/operations.rs index 9c75e4f2924..1982a1e0870 100644 --- a/vortex-array/src/arrays/dict/vtable/operations.rs +++ b/vortex-array/src/arrays/dict/vtable/operations.rs @@ -15,11 +15,11 @@ impl OperationsVTable for Dict { fn scalar_at( array: ArrayView<'_, Dict>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { let Some(dict_index) = array .codes() - .scalar_at(index)? + .execute_scalar(index, ctx)? .as_primitive() .as_::() else { @@ -28,7 +28,7 @@ impl OperationsVTable for Dict { Ok(array .values() - .scalar_at(dict_index)? + .execute_scalar(dict_index, ctx)? .cast(array.dtype()) .vortex_expect("Array dtype will only differ by nullability")) } diff --git a/vortex-array/src/arrays/dict_test.rs b/vortex-array/src/arrays/dict_test.rs index 8402b66c670..49be0744cf0 100644 --- a/vortex-array/src/arrays/dict_test.rs +++ b/vortex-array/src/arrays/dict_test.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] +#![expect(clippy::unwrap_used)] use rand::RngExt; use rand::SeedableRng; diff --git a/vortex-array/src/arrays/extension/array.rs b/vortex-array/src/arrays/extension/array.rs index 4ff2ad1bcfb..b42f7b63603 100644 --- a/vortex-array/src/arrays/extension/array.rs +++ b/vortex-array/src/arrays/extension/array.rs @@ -1,132 +1,34 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -use std::fmt::Display; -use std::fmt::Formatter; - use vortex_error::VortexExpect; use vortex_error::VortexResult; +use vortex_error::vortex_ensure_eq; use crate::ArrayRef; +use crate::EmptyArrayData; use crate::array::Array; use crate::array::ArrayParts; use crate::array::TypedArrayRef; use crate::arrays::Extension; use crate::dtype::DType; +use crate::dtype::extension::ExtDType; use crate::dtype::extension::ExtDTypeRef; +use crate::dtype::extension::ExtVTable; /// The backing storage array for this extension array. pub(super) const STORAGE_SLOT: usize = 0; pub(super) const NUM_SLOTS: usize = 1; pub(super) const SLOT_NAMES: [&str; NUM_SLOTS] = ["storage"]; -/// An extension array that wraps another array with additional type information. -/// -/// **⚠️ Unstable API**: This is an experimental feature that may change significantly -/// in future versions. The extension type system is still evolving. -/// -/// Unlike Apache Arrow's extension arrays, Vortex extension arrays provide a more flexible -/// mechanism for adding semantic meaning to existing array types without requiring -/// changes to the core type system. -/// -/// ## Design Philosophy -/// -/// Extension arrays serve as a type-safe wrapper that: -/// - Preserves the underlying storage format and operations -/// - Adds semantic type information via `ExtDType` -/// - Enables custom serialization and deserialization logic -/// - Allows domain-specific interpretations of generic data -/// -/// ## Storage and Type Relationship -/// -/// The extension array maintains a strict contract: -/// - **Storage array**: Contains the actual data in a standard Vortex encoding -/// - **Extension type**: Defines how to interpret the storage data semantically -/// - **Type safety**: The storage array's dtype must match the extension type's storage dtype -/// -/// ## Use Cases -/// -/// Extension arrays are ideal for: -/// - **Custom numeric types**: Units of measurement, currencies -/// - **Temporal types**: Custom date/time formats, time zones, calendars -/// - **Domain-specific types**: UUIDs, IP addresses, geographic coordinates -/// - **Encoded types**: Base64 strings, compressed data, encrypted values -/// -/// ## Validity and Operations -/// -/// Extension arrays delegate validity and most operations to their storage array: -/// - Validity is inherited from the underlying storage -/// - Slicing preserves the extension type -/// - Scalar access wraps storage scalars with extension metadata -#[derive(Clone, Debug)] -pub struct ExtensionData { - /// The storage dtype. This **must** be a [`Extension::DType`] variant. - pub(super) ext_dtype: ExtDTypeRef, -} - -impl Display for ExtensionData { - fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { - write!(f, "ext_dtype: {}", self.ext_dtype) - } -} - -impl ExtensionData { - /// Constructs a new `ExtensionArray`. - /// - /// # Panics - /// - /// Panics if the storage array in not compatible with the extension dtype. - pub fn new(ext_dtype: ExtDTypeRef, storage_dtype: &DType) -> Self { - Self::try_new(ext_dtype, storage_dtype).vortex_expect("Failed to create `ExtensionArray`") - } - - /// Tries to construct a new `ExtensionArray`. - /// - /// # Errors - /// - /// Returns an error if the storage array in not compatible with the extension dtype. - pub fn try_new(ext_dtype: ExtDTypeRef, storage_dtype: &DType) -> VortexResult { - // TODO(connor): Replace these statements once we add `validate_storage_array`. - // ext_dtype.validate_storage_array(&storage_array)?; - assert_eq!( - ext_dtype.storage_dtype(), - storage_dtype, - "ExtensionArray: storage_dtype must match storage array DType", - ); - - // SAFETY: we validate that the inputs are valid above. - Ok(unsafe { Self::new_unchecked(ext_dtype, storage_dtype) }) - } - - /// Creates a new `ExtensionArray`. - /// - /// # Safety - /// - /// The caller must ensure that the storage array is compatible with the extension dtype. In - /// other words, they must know that `ext_dtype.validate_storage_array(&storage_array)` has been - /// called successfully on this storage array. - pub unsafe fn new_unchecked(ext_dtype: ExtDTypeRef, storage_dtype: &DType) -> Self { - // TODO(connor): Replace these statements once we add `validate_storage_array`. - // #[cfg(debug_assertions)] - // ext_dtype - // .validate_storage_array(&storage_array) - // .vortex_expect("[Debug Assertion]: Invalid storage array for `ExtensionArray`"); - debug_assert_eq!( - ext_dtype.storage_dtype(), - storage_dtype, - "ExtensionArray: storage_dtype must match storage array DType", - ); - - Self { ext_dtype } +pub trait ExtensionArrayExt: TypedArrayRef { + fn ext_dtype(&self) -> &ExtDTypeRef { + self.as_ref() + .dtype() + .as_extension_opt() + .vortex_expect("extension array somehow did not have an extension dtype") } - /// The extension dtype of this array. - pub fn ext_dtype(&self) -> &ExtDTypeRef { - &self.ext_dtype - } -} - -pub trait ExtensionArrayExt: TypedArrayRef { fn storage_array(&self) -> &ArrayRef { self.as_ref().slots()[STORAGE_SLOT] .as_ref() @@ -142,25 +44,37 @@ impl Array { /// /// Panics if the storage array is not compatible with the extension dtype. pub fn new(ext_dtype: ExtDTypeRef, storage_array: ArrayRef) -> Self { - let dtype = DType::Extension(ext_dtype.clone()); - let len = storage_array.len(); - let data = ExtensionData::new(ext_dtype, storage_array.dtype()); - unsafe { - Array::from_parts_unchecked( - ArrayParts::new(Extension, dtype, len, data).with_slots(vec![Some(storage_array)]), - ) - } + Self::try_new(ext_dtype, storage_array).vortex_expect("Unable to create `ExtensionArray`") } /// Tries to construct a new `ExtensionArray`. pub fn try_new(ext_dtype: ExtDTypeRef, storage_array: ArrayRef) -> VortexResult { - let dtype = DType::Extension(ext_dtype.clone()); + vortex_ensure_eq!( + ext_dtype.storage_dtype(), + storage_array.dtype(), + "Tried to create an `ExtensionArray` with an incompatible storage array" + ); + + let dtype = DType::Extension(ext_dtype); let len = storage_array.len(); - let data = ExtensionData::try_new(ext_dtype, storage_array.dtype())?; - Ok(unsafe { - Array::from_parts_unchecked( - ArrayParts::new(Extension, dtype, len, data).with_slots(vec![Some(storage_array)]), - ) - }) + + let parts = ArrayParts::new(Extension, dtype, len, EmptyArrayData) + .with_slots(vec![Some(storage_array)]); + + Ok(unsafe { Array::from_parts_unchecked(parts) }) + } + + /// Creates a new [`ExtensionArray`](crate::arrays::ExtensionArray) from a vtable, metadata, and + /// a storage array. + pub fn try_new_from_vtable( + vtable: V, + metadata: V::Metadata, + storage_array: ArrayRef, + ) -> VortexResult { + let ext_dtype = + ExtDType::::try_with_vtable(vtable, metadata, storage_array.dtype().clone())? + .erased(); + + Self::try_new(ext_dtype, storage_array) } } diff --git a/vortex-array/src/arrays/extension/compute/cast.rs b/vortex-array/src/arrays/extension/compute/cast.rs index aa0394c73a3..0f4746fd4d6 100644 --- a/vortex-array/src/arrays/extension/compute/cast.rs +++ b/vortex-array/src/arrays/extension/compute/cast.rs @@ -4,7 +4,6 @@ use vortex_buffer::BufferMut; use vortex_error::VortexResult; use vortex_error::vortex_bail; -use vortex_mask::AllOr; use crate::ArrayRef; use crate::IntoArray; @@ -21,11 +20,14 @@ use crate::extension::datetime::AnyTemporal; use crate::extension::datetime::TemporalMetadata; use crate::extension::datetime::TimeUnit; use crate::scalar_fn::fns::cast::CastReduce; +use crate::validity::Validity; impl CastReduce for Extension { fn cast(array: ArrayView<'_, Extension>, dtype: &DType) -> VortexResult> { let DType::Extension(ext_dtype) = dtype else { - return Ok(None); + // Target is not an extension type. + // Delegate to the storage array's cast. + return Ok(Some(array.storage_array().cast(dtype.clone())?)); }; if array.ext_dtype().eq_ignore_nullability(ext_dtype) { @@ -98,23 +100,30 @@ fn cast_date_values_to_timestamp( let (multiply, divide) = date_to_timestamp_scale(source_unit, target_unit)?; let input = values.as_slice::(); + let validity = values.validity()?; let mut output = BufferMut::with_capacity(input.len()); - match values.validity_mask()?.bit_buffer() { - AllOr::All => { + + match &validity { + Validity::NonNullable | Validity::AllValid => { for &value in input { // SAFETY: output has sufficient capacity for all pushed values. unsafe { output.push_unchecked(convert_temporal_value(value, multiply, divide)?) }; } } - AllOr::None => { + Validity::AllInvalid => { for _ in 0..input.len() { // SAFETY: output has sufficient capacity for all pushed values. unsafe { output.push_unchecked(0i64) }; } } - AllOr::Some(bits) => { - for (&value, valid) in input.iter().zip(bits.iter()) { - if valid { + Validity::Array(validity_arr) => { + for (i, &value) in input.iter().enumerate() { + let is_valid = validity_arr + .scalar_at(i)? + .as_bool() + .value() + .unwrap_or(false); + if is_valid { // SAFETY: output has sufficient capacity for all pushed values. unsafe { output.push_unchecked(convert_temporal_value(value, multiply, divide)?) @@ -127,7 +136,7 @@ fn cast_date_values_to_timestamp( } } - Ok(PrimitiveArray::new(output.freeze(), values.validity()?)) + Ok(PrimitiveArray::new(output.freeze(), validity)) } fn date_to_timestamp_scale( @@ -185,20 +194,30 @@ fn convert_temporal_value(value: i64, multiply: i64, divide: i64) -> VortexResul #[cfg(test)] mod tests { + use std::sync::LazyLock; use rstest::rstest; use vortex_buffer::Buffer; use vortex_buffer::buffer; + use vortex_session::VortexSession; use super::*; use crate::IntoArray; use crate::arrays::PrimitiveArray; + use crate::assert_arrays_eq; use crate::builtins::ArrayBuiltins; use crate::compute::conformance::cast::test_cast_conformance; + use crate::dtype::DType; use crate::dtype::Nullability; + use crate::dtype::PType; + use crate::executor::VortexSessionExecute; use crate::extension::datetime::Date; use crate::extension::datetime::TimeUnit; use crate::extension::datetime::Timestamp; + use crate::session::ArraySession; + + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); #[test] fn cast_same_ext_dtype() { @@ -294,12 +313,34 @@ mod tests { let storage = buffer![1i64].into_array(); let arr = ExtensionArray::new(original_dtype, storage); - assert!( - arr.into_array() - .cast(DType::Extension(target_dtype)) - .and_then(|a| a.to_canonical().map(|c| c.into_array())) - .is_err() + let result = arr + .into_array() + .cast(DType::Extension(target_dtype)) + .and_then(|a| { + a.execute::(&mut SESSION.create_execution_ctx()) + .map(|c| c.into_array()) + }); + assert!(result.is_err()); + } + + #[test] + fn cast_timestamp_to_i64() -> VortexResult<()> { + let ext_dtype = Timestamp::new_with_tz( + TimeUnit::Nanoseconds, + Some("UTC".into()), + Nullability::NonNullable, + ) + .erased(); + let storage = buffer![1i64, 2, 3].into_array(); + let arr = ExtensionArray::new(ext_dtype, storage).into_array(); + + let result = arr.cast(DType::Primitive(PType::I64, Nullability::NonNullable))?; + assert_eq!( + result.dtype(), + &DType::Primitive(PType::I64, Nullability::NonNullable) ); + assert_arrays_eq!(result, buffer![1i64, 2, 3].into_array()); + Ok(()) } #[rstest] diff --git a/vortex-array/src/arrays/extension/compute/rules.rs b/vortex-array/src/arrays/extension/compute/rules.rs index af1cbc832b9..d7c8469d1dc 100644 --- a/vortex-array/src/arrays/extension/compute/rules.rs +++ b/vortex-array/src/arrays/extension/compute/rules.rs @@ -6,6 +6,8 @@ use vortex_error::VortexResult; use crate::ArrayRef; use crate::IntoArray; use crate::array::ArrayView; +use crate::arrays::Constant; +use crate::arrays::ConstantArray; use crate::arrays::Extension; use crate::arrays::ExtensionArray; use crate::arrays::Filter; @@ -13,10 +15,35 @@ use crate::arrays::extension::ExtensionArrayExt; use crate::arrays::filter::FilterReduceAdaptor; use crate::arrays::slice::SliceReduceAdaptor; use crate::optimizer::rules::ArrayParentReduceRule; +use crate::optimizer::rules::ArrayReduceRule; use crate::optimizer::rules::ParentRuleSet; +use crate::optimizer::rules::ReduceRuleSet; +use crate::scalar::Scalar; use crate::scalar_fn::fns::cast::CastReduceAdaptor; use crate::scalar_fn::fns::mask::MaskReduceAdaptor; +pub(crate) const RULES: ReduceRuleSet = ReduceRuleSet::new(&[&ExtensionConstantRule]); + +/// Normalize `Extension(Constant(storage))` children to `Constant(Extension(storage))`. +#[derive(Debug)] +struct ExtensionConstantRule; + +impl ArrayReduceRule for ExtensionConstantRule { + fn reduce(&self, array: ArrayView<'_, Extension>) -> VortexResult> { + let Some(const_array) = array.storage_array().as_opt::() else { + return Ok(None); + }; + + let storage_scalar = const_array.scalar().clone(); + let ext_scalar = Scalar::extension_ref(array.ext_dtype().clone(), storage_scalar); + + let constant_with_extension_scalar = + ConstantArray::new(ext_scalar, array.len()).into_array(); + + Ok(Some(constant_with_extension_scalar.into_array())) + } +} + pub(crate) const PARENT_RULES: ParentRuleSet = ParentRuleSet::new(&[ ParentRuleSet::lift(&ExtensionFilterPushDownRule), ParentRuleSet::lift(&CastReduceAdaptor(Extension)), @@ -51,17 +78,23 @@ impl ArrayParentReduceRule for ExtensionFilterPushDownRule { #[cfg(test)] mod tests { + use std::sync::LazyLock; + use vortex_buffer::buffer; use vortex_error::VortexResult; use vortex_mask::Mask; + use vortex_session::VortexSession; use crate::IntoArray; - use crate::ToCanonical; + #[expect(deprecated)] + use crate::ToCanonical as _; + use crate::arrays::Constant; use crate::arrays::ConstantArray; use crate::arrays::Extension; use crate::arrays::ExtensionArray; use crate::arrays::FilterArray; use crate::arrays::PrimitiveArray; + use crate::arrays::ScalarFn; use crate::arrays::extension::ExtensionArrayExt; use crate::arrays::scalar_fn::ScalarFnArrayExt; use crate::arrays::scalar_fn::ScalarFnFactoryExt; @@ -78,6 +111,10 @@ mod tests { use crate::scalar::ScalarValue; use crate::scalar_fn::fns::binary::Binary; use crate::scalar_fn::fns::operators::Operator; + use crate::session::ArraySession; + + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); #[derive(Clone, Debug, Default, PartialEq, Eq, Hash)] struct TestExt; @@ -86,7 +123,7 @@ mod tests { type NativeValue<'a> = &'a str; fn id(&self) -> ExtId { - ExtId::new_ref("test_ext") + ExtId::new("test_ext") } fn serialize_metadata(&self, _metadata: &Self::Metadata) -> VortexResult> { @@ -143,7 +180,9 @@ mod tests { assert_eq!(ext_result.ext_dtype(), &ext_dtype); // Check the storage values - let storage_result: &[i64] = &ext_result.storage_array().to_primitive().to_buffer::(); + #[expect(deprecated)] + let storage_prim = ext_result.storage_array().to_primitive(); + let storage_result: &[i64] = &storage_prim.to_buffer::(); assert_eq!(storage_result, &[1, 3, 5]); } @@ -169,10 +208,36 @@ mod tests { assert_eq!(ext_result.len(), 3); // Check values: should be [Some(1), None, None] + #[expect(deprecated)] let canonical = ext_result.storage_array().to_primitive(); assert_eq!(canonical.len(), 3); } + #[test] + fn test_extension_constant_child_normalizes_under_scalar_fn() { + let ext_dtype = test_ext_dtype(); + + let constant_storage = ConstantArray::new(Scalar::from(10i64), 3).into_array(); + let constant_ext = ExtensionArray::new(ext_dtype.clone(), constant_storage).into_array(); + + let storage = buffer![15i64, 25, 35].into_array(); + let ext_array = ExtensionArray::new(ext_dtype, storage).into_array(); + + let scalar_fn_array = Binary + .try_new_array(3, Operator::Lt, [constant_ext, ext_array]) + .unwrap(); + + let optimized = scalar_fn_array.optimize_recursive(&SESSION).unwrap(); + let scalar_fn = optimized.as_opt::().unwrap(); + let children = scalar_fn.children(); + let constant = children[0] + .as_opt::() + .expect("constant extension child should be normalized"); + + assert!(constant.scalar().as_extension_opt().is_some()); + assert_eq!(constant.len(), 3); + } + #[test] fn test_scalar_fn_no_pushdown_different_ext_types() { #[derive(Clone, Debug, Default, PartialEq, Eq, Hash)] @@ -182,7 +247,7 @@ mod tests { type NativeValue<'a> = &'a str; fn id(&self) -> ExtId { - ExtId::new_ref("test_ext_2") + ExtId::new("test_ext_2") } fn serialize_metadata(&self, _metadata: &Self::Metadata) -> VortexResult> { @@ -226,7 +291,7 @@ mod tests { let optimized = scalar_fn_array.optimize().unwrap(); // The first child should still be an ExtensionArray (no pushdown happened) - let scalar_fn = optimized.as_opt::().unwrap(); + let scalar_fn = optimized.as_opt::().unwrap(); assert!( scalar_fn.children()[0].as_opt::().is_some(), "Expected first child to remain ExtensionArray when ext types differ" @@ -251,7 +316,7 @@ mod tests { let optimized = scalar_fn_array.optimize().unwrap(); // No pushdown should happen because sibling is not a constant - let scalar_fn = optimized.as_opt::().unwrap(); + let scalar_fn = optimized.as_opt::().unwrap(); assert!( scalar_fn.children()[0].as_opt::().is_some(), "Expected first child to remain ExtensionArray when sibling is not constant" @@ -274,7 +339,7 @@ mod tests { let optimized = scalar_fn_array.optimize().unwrap(); // No pushdown should happen because constant is not an extension scalar - let scalar_fn = optimized.as_opt::().unwrap(); + let scalar_fn = optimized.as_opt::().unwrap(); assert!( scalar_fn.children()[0].as_opt::().is_some(), "Expected first child to remain ExtensionArray when constant is not extension" diff --git a/vortex-array/src/arrays/extension/mod.rs b/vortex-array/src/arrays/extension/mod.rs index 802545025c1..57bc9df5ead 100644 --- a/vortex-array/src/arrays/extension/mod.rs +++ b/vortex-array/src/arrays/extension/mod.rs @@ -3,7 +3,6 @@ mod array; pub use array::ExtensionArrayExt; -pub use array::ExtensionData; pub use vtable::ExtensionArray; pub(crate) mod compute; diff --git a/vortex-array/src/arrays/extension/vtable/mod.rs b/vortex-array/src/arrays/extension/vtable/mod.rs index ba403403dbe..b048c81fff7 100644 --- a/vortex-array/src/arrays/extension/vtable/mod.rs +++ b/vortex-array/src/arrays/extension/vtable/mod.rs @@ -1,60 +1,122 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -mod canonical; -mod kernel; -mod operations; -mod validity; - -use std::hash::Hasher; use kernel::PARENT_KERNELS; use vortex_error::VortexExpect; use vortex_error::VortexResult; use vortex_error::vortex_bail; -use vortex_error::vortex_ensure; +use vortex_error::vortex_ensure_eq; +use vortex_error::vortex_err; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; -use crate::ArrayEq; -use crate::ArrayHash; use crate::ArrayRef; +use crate::EmptyArrayData; use crate::ExecutionCtx; use crate::ExecutionResult; -use crate::Precision; use crate::array::Array; use crate::array::ArrayId; +use crate::array::ArrayParts; use crate::array::ArrayView; use crate::array::VTable; use crate::array::ValidityVTableFromChild; -use crate::arrays::extension::ExtensionData; use crate::arrays::extension::array::SLOT_NAMES; use crate::arrays::extension::array::STORAGE_SLOT; use crate::arrays::extension::compute::rules::PARENT_RULES; +use crate::arrays::extension::compute::rules::RULES; use crate::buffer::BufferHandle; use crate::dtype::DType; use crate::serde::ArrayChildren; -/// A [`Extension`]-encoded Vortex array. -pub type ExtensionArray = Array; +mod kernel; +mod operations; +mod validity; -impl ArrayHash for ExtensionData { - fn array_hash(&self, _state: &mut H, _precision: Precision) {} -} +/// An extension array that wraps another array with additional type information. +/// +/// **⚠️ Unstable API**: This is an experimental feature that may change significantly +/// in future versions. The extension type system is still evolving. +/// +/// Unlike Apache Arrow's extension arrays, Vortex extension arrays provide a more flexible +/// mechanism for adding semantic meaning to existing array types without requiring +/// changes to the core type system. +/// +/// ## Design Philosophy +/// +/// Extension arrays serve as a type-safe wrapper that: +/// - Preserves the underlying storage format and operations +/// - Adds semantic type information via `ExtDType` +/// - Enables custom serialization and deserialization logic +/// - Allows domain-specific interpretations of generic data +/// +/// ## Storage and Type Relationship +/// +/// The extension array maintains a strict contract: +/// - **Storage array**: Contains the actual data in a standard Vortex encoding +/// - **Extension type**: Defines how to interpret the storage data semantically +/// - **Type safety**: The storage array's dtype must match the extension type's storage dtype +/// +/// ## Use Cases +/// +/// Extension arrays are ideal for: +/// - **Custom numeric types**: Units of measurement, currencies +/// - **Temporal types**: Custom date/time formats, time zones, calendars +/// - **Domain-specific types**: UUIDs, IP addresses, geographic coordinates +/// - **Encoded types**: Base64 strings, compressed data, encrypted values +/// +/// ## Validity and Operations +/// +/// Extension arrays delegate validity and most operations to their storage array: +/// - Validity is inherited from the underlying storage +/// - Slicing preserves the extension type +/// - Scalar access wraps storage scalars with extension metadata +#[derive(Clone, Debug)] +pub struct Extension; -impl ArrayEq for ExtensionData { - fn array_eq(&self, _other: &Self, _precision: Precision) -> bool { - true - } -} +/// A [`Extension`]-encoded Vortex array. +pub type ExtensionArray = Array; impl VTable for Extension { - type ArrayData = ExtensionData; + type ArrayData = EmptyArrayData; type OperationsVTable = Self; type ValidityVTable = ValidityVTableFromChild; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.ext"); + *ID + } + + fn validate( + &self, + _data: &EmptyArrayData, + dtype: &DType, + len: usize, + slots: &[Option], + ) -> VortexResult<()> { + let storage = slots[STORAGE_SLOT] + .as_ref() + .vortex_expect("ExtensionArray storage slot"); + vortex_ensure_eq!( + storage.len(), + len, + "ExtensionArray length {} does not match outer length {len}", + storage.len(), + ); + + let ext_dtype = dtype + .as_extension_opt() + .ok_or_else(|| vortex_err!("not an extension dtype"))?; + + let actual_dtype = DType::Extension(ext_dtype.clone()); + vortex_ensure_eq!( + &actual_dtype, + dtype, + "ExtensionArray dtype {actual_dtype} does not match outer dtype {dtype}", + ); + + Ok(()) } fn nbuffers(_array: ArrayView<'_, Self>) -> usize { @@ -69,10 +131,6 @@ impl VTable for Extension { None } - fn slot_name(_array: ArrayView<'_, Self>, idx: usize) -> String { - SLOT_NAMES[idx].to_string() - } - fn serialize( _array: ArrayView<'_, Self>, _session: &VortexSession, @@ -80,35 +138,6 @@ impl VTable for Extension { Ok(Some(vec![])) } - fn validate( - &self, - data: &ExtensionData, - dtype: &DType, - len: usize, - slots: &[Option], - ) -> VortexResult<()> { - _ = data; - let storage = slots[STORAGE_SLOT] - .as_ref() - .vortex_expect("ExtensionArray storage slot"); - vortex_ensure!( - storage.len() == len, - "ExtensionArray length {} does not match outer length {}", - storage.len(), - len - ); - - let actual_dtype = DType::Extension(data.ext_dtype.clone()); - vortex_ensure!( - &actual_dtype == dtype, - "ExtensionArray dtype {} does not match outer dtype {}", - actual_dtype, - dtype - ); - - Ok(()) - } - fn deserialize( &self, dtype: &DType, @@ -118,7 +147,7 @@ impl VTable for Extension { _buffers: &[BufferHandle], children: &dyn ArrayChildren, _session: &VortexSession, - ) -> VortexResult> { + ) -> VortexResult> { if !metadata.is_empty() { vortex_bail!( "ExtensionArray expects empty metadata, got {} bytes", @@ -132,40 +161,38 @@ impl VTable for Extension { vortex_bail!("Expected 1 child, got {}", children.len()); } let storage = children.get(0, ext_dtype.storage_dtype(), len)?; - Ok(crate::array::ArrayParts::new( - self.clone(), - dtype.clone(), - len, - ExtensionData::new(ext_dtype.clone(), storage.dtype()), + Ok( + ArrayParts::new(self.clone(), dtype.clone(), len, EmptyArrayData) + .with_slots(vec![Some(storage)]), ) - .with_slots(vec![Some(storage)])) + } + + fn slot_name(_array: ArrayView<'_, Self>, idx: usize) -> String { + SLOT_NAMES[idx].to_string() } fn execute(array: Array, _ctx: &mut ExecutionCtx) -> VortexResult { Ok(ExecutionResult::done(array)) } - fn reduce_parent( + fn execute_parent( array: ArrayView<'_, Self>, parent: &ArrayRef, child_idx: usize, + ctx: &mut ExecutionCtx, ) -> VortexResult> { - PARENT_RULES.evaluate(array, parent, child_idx) + PARENT_KERNELS.execute(array, parent, child_idx, ctx) } - fn execute_parent( + fn reduce(array: ArrayView<'_, Self>) -> VortexResult> { + RULES.evaluate(array) + } + + fn reduce_parent( array: ArrayView<'_, Self>, parent: &ArrayRef, child_idx: usize, - ctx: &mut ExecutionCtx, ) -> VortexResult> { - PARENT_KERNELS.execute(array, parent, child_idx, ctx) + PARENT_RULES.evaluate(array, parent, child_idx) } } - -#[derive(Clone, Debug)] -pub struct Extension; - -impl Extension { - pub const ID: ArrayId = ArrayId::new_ref("vortex.ext"); -} diff --git a/vortex-array/src/arrays/extension/vtable/operations.rs b/vortex-array/src/arrays/extension/vtable/operations.rs index d86c7599aef..66de94b596a 100644 --- a/vortex-array/src/arrays/extension/vtable/operations.rs +++ b/vortex-array/src/arrays/extension/vtable/operations.rs @@ -14,11 +14,11 @@ impl OperationsVTable for Extension { fn scalar_at( array: ArrayView<'_, Extension>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { Ok(Scalar::extension_ref( array.ext_dtype().clone(), - array.storage_array().scalar_at(index)?, + array.storage_array().execute_scalar(index, ctx)?, )) } } diff --git a/vortex-array/src/arrays/filter/execute/bitbuffer.rs b/vortex-array/src/arrays/filter/execute/bitbuffer.rs index e1c61b2ebc1..1ad07801967 100644 --- a/vortex-array/src/arrays/filter/execute/bitbuffer.rs +++ b/vortex-array/src/arrays/filter/execute/bitbuffer.rs @@ -32,7 +32,7 @@ fn filter_bitbuffer_by_indices(bb: &BitBuffer, indices: &[usize]) -> BitBuffer { .freeze() } -#[allow(unused)] +#[expect(unused)] fn filter_bitbuffer_by_slices(bb: &BitBuffer, slices: &[(usize, usize)]) -> BitBuffer { let bools = bb.inner().as_ref(); let bit_offset = bb.offset(); diff --git a/vortex-array/src/arrays/filter/execute/bool.rs b/vortex-array/src/arrays/filter/execute/bool.rs index 1a35d1e74aa..a22e55366fc 100644 --- a/vortex-array/src/arrays/filter/execute/bool.rs +++ b/vortex-array/src/arrays/filter/execute/bool.rs @@ -31,7 +31,8 @@ mod test { use crate::IntoArray; use crate::arrays::filter::execute::bool::BoolArray; - use crate::canonical::ToCanonical; + #[expect(deprecated)] + use crate::canonical::ToCanonical as _; use crate::compute::conformance::filter::test_filter_conformance; #[test] @@ -39,6 +40,7 @@ mod test { let arr = BoolArray::from_iter([true, true, false]); let mask = Mask::from_iter([true, false, true]); + #[expect(deprecated)] let filtered = arr.filter(mask).unwrap().to_bool(); assert_eq!(2, filtered.len()); diff --git a/vortex-array/src/arrays/filter/execute/listview.rs b/vortex-array/src/arrays/filter/execute/listview.rs index e6c30416e17..32f8df02f7e 100644 --- a/vortex-array/src/arrays/filter/execute/listview.rs +++ b/vortex-array/src/arrays/filter/execute/listview.rs @@ -4,26 +4,15 @@ use std::sync::Arc; use vortex_error::VortexExpect; +use vortex_mask::Mask; use vortex_mask::MaskValues; use crate::arrays::ListViewArray; use crate::arrays::filter::execute::filter_validity; -use crate::arrays::filter::execute::values_to_mask; +use crate::arrays::listview; use crate::arrays::listview::ListViewArrayExt; use crate::arrays::listview::ListViewRebuildMode; -// TODO(connor)[ListView]: Make use of this threshold after we start migrating operators. -/// The threshold for triggering a rebuild of the [`ListViewArray`]. -/// -/// By default, we will not touch the underlying `elements` array of the [`ListViewArray`] since it -/// can be potentially expensive to reorganize the array based on what views we have into it. -/// -/// However, we also do not want to carry around a large amount of garbage data. Below this -/// threshold of the density of the selection mask, we will rebuild the [`ListViewArray`], removing -/// any garbage data. -#[allow(unused)] -const REBUILD_DENSITY_THRESHOLD: f64 = 0.1; - /// [`ListViewArray`] filter implementation. /// /// This implementation is deliberately simple and read-optimized. We just filter the `offsets` and @@ -54,7 +43,7 @@ pub fn filter_listview(array: &ListViewArray, selection_mask: &Arc) ); // Simply filter the offsets and sizes arrays. - let mask_for_filter = values_to_mask(selection_mask); + let mask_for_filter = Mask::Values(Arc::clone(selection_mask)); let new_offsets = offsets .filter(mask_for_filter.clone()) .vortex_expect("ListViewArray offsets are guaranteed to support filter"); @@ -70,13 +59,14 @@ pub fn filter_listview(array: &ListViewArray, selection_mask: &Arc) ListViewArray::new_unchecked(elements.clone(), new_offsets, new_sizes, new_validity) }; - // TODO(connor)[ListView]: Ideally, we would only rebuild after all `take`s and `filter` - // compute functions have run, at the "top" of the operator tree. However, we cannot do this - // right now, so we will just rebuild every time (similar to `ListArray`). - - new_array - .rebuild(ListViewRebuildMode::MakeZeroCopyToList) - .vortex_expect("ListViewArray rebuild to zero-copy List should always succeed") + let kept_row_fraction = selection_mask.true_count() as f32 / array.sizes().len() as f32; + if kept_row_fraction < listview::compute::REBUILD_DENSITY_THRESHOLD { + new_array + .rebuild(ListViewRebuildMode::MakeZeroCopyToList) + .vortex_expect("ListViewArray rebuild to zero-copy List should always succeed") + } else { + new_array + } } #[cfg(test)] @@ -85,7 +75,10 @@ mod test { use vortex_mask::Mask; use crate::IntoArray; - use crate::ToCanonical; + use crate::LEGACY_SESSION; + #[expect(deprecated)] + use crate::ToCanonical as _; + use crate::VortexSessionExecute; use crate::arrays::ListViewArray; use crate::arrays::PrimitiveArray; use crate::arrays::filter::execute::ConstantArray; @@ -194,6 +187,7 @@ mod test { // Filter to keep only 2 lists. let mask = Mask::from_iter([true, false, false, true, false]); let result = listview.filter(mask).unwrap(); + #[expect(deprecated)] let result_list = result.to_listview(); assert_eq!(result_list.len(), 2, "Wrong number of filtered lists"); @@ -226,6 +220,7 @@ mod test { // Filter to keep lists with gaps and overlaps. let mask = Mask::from_iter([false, true, true, true, false]); let result = listview.filter(mask).unwrap(); + #[expect(deprecated)] let result_list = result.to_listview(); assert_eq!(result_list.len(), 3, "Wrong filter result length"); @@ -269,6 +264,7 @@ mod test { let mask1 = Mask::from_iter([true, false, true, false]); let result1 = const_offset_list.filter(mask1).unwrap(); + #[expect(deprecated)] let result1_list = result1.to_listview(); assert_eq!(result1_list.len(), 2); @@ -292,6 +288,7 @@ mod test { let mask2 = Mask::from_iter([true, false, true]); let result2 = both_const_list.filter(mask2).unwrap(); + #[expect(deprecated)] let result2_list = result2.to_listview(); assert_eq!(result2_list.len(), 2); @@ -318,6 +315,7 @@ mod test { // Filter to keep only 2 lists, demonstrating we keep all 10000 elements. let mask = Mask::from_iter([false, true, false, false, true]); let result = listview.filter(mask).unwrap(); + #[expect(deprecated)] let result_list = result.to_listview(); assert_eq!(result_list.len(), 2); @@ -333,7 +331,7 @@ mod test { let list0 = result_list.list_elements_at(0).unwrap(); assert_eq!( list0 - .scalar_at(0) + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) .unwrap() .as_primitive() .as_::() @@ -342,7 +340,7 @@ mod test { ); assert_eq!( list0 - .scalar_at(1) + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) .unwrap() .as_primitive() .as_::() @@ -353,6 +351,7 @@ mod test { // Test sparse selection from large dataset. let sparse_mask = Mask::from_iter((0..5).map(|i| i == 0 || i == 4)); let sparse_result = listview.filter(sparse_mask).unwrap(); + #[expect(deprecated)] let sparse_list = sparse_result.to_listview(); assert_eq!(sparse_list.len(), 2); diff --git a/vortex-array/src/arrays/filter/execute/mod.rs b/vortex-array/src/arrays/filter/execute/mod.rs index bcbf28c0d96..943238a7e1f 100644 --- a/vortex-array/src/arrays/filter/execute/mod.rs +++ b/vortex-array/src/arrays/filter/execute/mod.rs @@ -39,22 +39,17 @@ mod slice; mod struct_; mod varbinview; -/// Reconstruct a [`Mask`] from an [`Arc`]. -fn values_to_mask(values: &Arc) -> Mask { - Mask::Values(Arc::clone(values)) -} - /// A helper function that lazily filters a [`Validity`] with selection mask values. fn filter_validity(validity: Validity, mask: &Arc) -> Validity { validity - .filter(&values_to_mask(mask)) + .filter(&Mask::Values(Arc::clone(mask))) .vortex_expect("Somehow unable to wrap filter around a validity array") } /// Check for some fast-path execution conditions before calling [`execute_filter`]. pub(super) fn execute_filter_fast_paths( array: ArrayView<'_, Filter>, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult> { let true_count = array.mask.true_count(); @@ -70,7 +65,13 @@ pub(super) fn execute_filter_fast_paths( // Also check if the array itself is completely null, in which case we only care about the total // number of nulls, not the values. - if array.array().validity_mask()?.true_count() == 0 { + let child_arr = array.array(); + if child_arr + .validity()? + .execute_mask(child_arr.len(), ctx)? + .true_count() + == 0 + { return Ok(Some( ConstantArray::new(Scalar::null(array.dtype().clone()), true_count).into_array(), )); @@ -95,14 +96,14 @@ pub(super) fn execute_filter(canonical: Canonical, mask: &Arc) -> Ca Canonical::Extension(a) => { let filtered_storage = a .storage_array() - .filter(values_to_mask(mask)) + .filter(Mask::Values(Arc::clone(mask))) .vortex_expect("ExtensionArray storage type somehow could not be filtered"); Canonical::Extension(ExtensionArray::new(a.ext_dtype().clone(), filtered_storage)) } Canonical::Variant(a) => { let filtered_child = a .child() - .filter(values_to_mask(mask)) + .filter(Mask::Values(Arc::clone(mask))) .vortex_expect("VariantArray child could not be filtered"); Canonical::Variant(VariantArray::new(filtered_child)) } diff --git a/vortex-array/src/arrays/filter/execute/primitive.rs b/vortex-array/src/arrays/filter/execute/primitive.rs index 86ef8ecbbb2..0aae5738da7 100644 --- a/vortex-array/src/arrays/filter/execute/primitive.rs +++ b/vortex-array/src/arrays/filter/execute/primitive.rs @@ -27,7 +27,7 @@ pub fn filter_primitive(array: &PrimitiveArray, mask: &Arc) -> Primi } #[cfg(test)] -#[allow(clippy::cast_possible_truncation)] +#[expect(clippy::cast_possible_truncation)] mod test { use itertools::Itertools; use rstest::rstest; @@ -35,7 +35,8 @@ mod test { use crate::IntoArray; use crate::arrays::PrimitiveArray; - use crate::canonical::ToCanonical; + #[expect(deprecated)] + use crate::canonical::ToCanonical as _; use crate::compute::conformance::filter::LARGE_SIZE; use crate::compute::conformance::filter::MEDIUM_SIZE; use crate::compute::conformance::filter::test_filter_conformance; @@ -45,6 +46,7 @@ mod test { let mask = [true, true, false, true, true, true, false, true]; let arr = PrimitiveArray::from_iter([1u32, 24, 54, 2, 3, 2, 3, 2]); + #[expect(deprecated)] let filtered = arr.filter(Mask::from_iter(mask)).unwrap().to_primitive(); assert_eq!( filtered.len(), diff --git a/vortex-array/src/arrays/filter/execute/struct_.rs b/vortex-array/src/arrays/filter/execute/struct_.rs index c9dac73a8cd..ee209b90fc3 100644 --- a/vortex-array/src/arrays/filter/execute/struct_.rs +++ b/vortex-array/src/arrays/filter/execute/struct_.rs @@ -4,12 +4,12 @@ use std::sync::Arc; use vortex_error::VortexExpect; +use vortex_mask::Mask; use vortex_mask::MaskValues; use crate::ArrayRef; use crate::arrays::StructArray; use crate::arrays::filter::execute::filter_validity; -use crate::arrays::filter::execute::values_to_mask; use crate::arrays::struct_::StructArrayExt; pub fn filter_struct(array: &StructArray, mask: &Arc) -> StructArray { @@ -20,7 +20,7 @@ pub fn filter_struct(array: &StructArray, mask: &Arc) -> StructArray mask, ); - let mask_for_filter = values_to_mask(mask); + let mask_for_filter = Mask::Values(Arc::clone(mask)); let fields: Vec = array .iter_unmasked_fields() .map(|field| { diff --git a/vortex-array/src/arrays/filter/execute/varbinview.rs b/vortex-array/src/arrays/filter/execute/varbinview.rs index b757baa7c77..6c0bb052bc2 100644 --- a/vortex-array/src/arrays/filter/execute/varbinview.rs +++ b/vortex-array/src/arrays/filter/execute/varbinview.rs @@ -10,15 +10,16 @@ use vortex_mask::MaskValues; use crate::ArrayRef; use crate::IntoArray; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::arrays::VarBinView; use crate::arrays::VarBinViewArray; -use crate::arrays::filter::execute::values_to_mask; +use crate::arrow::ArrowArrayExecutor; use crate::arrow::FromArrowArray; -use crate::arrow::IntoArrowArray; pub fn filter_varbinview(array: &VarBinViewArray, mask: &Arc) -> VarBinViewArray { // Delegate to the Arrow implementation of filter over `VarBinView`. - arrow_filter_fn(&array.clone().into_array(), &values_to_mask(mask)) + arrow_filter_fn(&array.clone().into_array(), &Mask::Values(Arc::clone(mask))) .vortex_expect("VarBinViewArray is Arrow-compatible and supports arrow_filter_fn") .as_::() .into_owned() @@ -30,7 +31,9 @@ fn arrow_filter_fn(array: &ArrayRef, mask: &Mask) -> vortex_error::VortexResult< Mask::AllTrue(_) | Mask::AllFalse(_) => unreachable!("check in filter invoke"), }; - let array_ref = array.clone().into_arrow_preferred()?; + let array_ref = array + .clone() + .execute_arrow(None, &mut LEGACY_SESSION.create_execution_ctx())?; let mask_array = BooleanArray::new(values.bit_buffer().clone().into(), None); let filtered = arrow_select::filter::filter(array_ref.as_ref(), &mask_array)?; diff --git a/vortex-array/src/arrays/filter/vtable.rs b/vortex-array/src/arrays/filter/vtable.rs index 8eb7fb62213..f6cd88410af 100644 --- a/vortex-array/src/arrays/filter/vtable.rs +++ b/vortex-array/src/arrays/filter/vtable.rs @@ -11,6 +11,7 @@ use vortex_error::vortex_ensure; use vortex_error::vortex_panic; use vortex_mask::Mask; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::AnyCanonical; use crate::ArrayEq; @@ -48,10 +49,6 @@ pub type FilterArray = Array; #[derive(Clone, Debug)] pub struct Filter; -impl Filter { - pub const ID: ArrayId = ArrayId::new_ref("vortex.filter"); -} - impl ArrayHash for FilterData { fn array_hash(&self, state: &mut H, precision: Precision) { self.mask.array_hash(state, precision); @@ -68,9 +65,9 @@ impl VTable for Filter { type ArrayData = FilterData; type OperationsVTable = Self; type ValidityVTable = Self; - fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.filter"); + *ID } fn validate( @@ -181,10 +178,10 @@ impl OperationsVTable for Filter { fn scalar_at( array: ArrayView<'_, Filter>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { let rank_idx = array.mask.rank(index); - array.child().scalar_at(rank_idx) + array.child().execute_scalar(rank_idx, ctx) } } diff --git a/vortex-array/src/arrays/fixed_size_list/array.rs b/vortex-array/src/arrays/fixed_size_list/array.rs index dcbee9f8f6f..e885ecce0f4 100644 --- a/vortex-array/src/arrays/fixed_size_list/array.rs +++ b/vortex-array/src/arrays/fixed_size_list/array.rs @@ -228,11 +228,7 @@ pub trait FixedSizeListArrayExt: TypedArrayRef { fn fixed_size_list_validity(&self) -> Validity { let (_, _, nullability) = self.dtype_parts(); - child_to_validity(&self.as_ref().slots()[VALIDITY_SLOT], nullability) - } - - fn fixed_size_list_validity_mask(&self) -> vortex_mask::Mask { - self.fixed_size_list_validity().to_mask(self.as_ref().len()) + child_to_validity(self.as_ref().slots()[VALIDITY_SLOT].as_ref(), nullability) } fn fixed_size_list_elements_at(&self, index: usize) -> VortexResult { diff --git a/vortex-array/src/arrays/fixed_size_list/compute/cast.rs b/vortex-array/src/arrays/fixed_size_list/compute/cast.rs index 791462d51c9..a54fe5c7c4d 100644 --- a/vortex-array/src/arrays/fixed_size_list/compute/cast.rs +++ b/vortex-array/src/arrays/fixed_size_list/compute/cast.rs @@ -4,6 +4,7 @@ use vortex_error::VortexResult; use crate::ArrayRef; +use crate::ExecutionCtx; use crate::IntoArray; use crate::array::ArrayView; use crate::arrays::FixedSizeList; @@ -11,7 +12,20 @@ use crate::arrays::FixedSizeListArray; use crate::arrays::fixed_size_list::FixedSizeListArrayExt; use crate::builtins::ArrayBuiltins; use crate::dtype::DType; +use crate::scalar_fn::fns::cast::CastKernel; use crate::scalar_fn::fns::cast::CastReduce; +use crate::validity::Validity; + +fn build_with_validity( + array: ArrayView<'_, FixedSizeList>, + elements: ArrayRef, + validity: Validity, +) -> ArrayRef { + // SAFETY: The only requirements for safety here are related to lengths, and no lengths have + // changed here. So as long as the original array is valid, this is also valid. + unsafe { FixedSizeListArray::new_unchecked(elements, array.list_size(), validity, array.len()) } + .into_array() +} /// Cast implementation for [`FixedSizeListArray`]. /// @@ -23,23 +37,33 @@ impl CastReduce for FixedSizeList { return Ok(None); }; + let Some(validity) = array + .validity()? + .trivial_cast_nullability(dtype.nullability(), array.len())? + else { + return Ok(None); + }; let elements = array.elements().cast((**target_element_type).clone())?; + + Ok(Some(build_with_validity(array, elements, validity))) + } +} + +impl CastKernel for FixedSizeList { + fn cast( + array: ArrayView<'_, FixedSizeList>, + dtype: &DType, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + let Some(target_element_type) = dtype.as_fixed_size_list_element_opt() else { + return Ok(None); + }; + let validity = array .validity()? - .cast_nullability(dtype.nullability(), array.len())?; - - Ok(Some( - // SAFETY: The only requirements for safety here are related to lengths, and no lengths - // have changed here. So as long as the original array is valid, this is also valid. - unsafe { - FixedSizeListArray::new_unchecked( - elements, - array.list_size(), - validity, - array.len(), - ) - } - .into_array(), - )) + .cast_nullability(dtype.nullability(), array.len(), ctx)?; + let elements = array.elements().cast((**target_element_type).clone())?; + + Ok(Some(build_with_validity(array, elements, validity))) } } diff --git a/vortex-array/src/arrays/fixed_size_list/compute/take.rs b/vortex-array/src/arrays/fixed_size_list/compute/take.rs index 259eb382013..3f2cf451c00 100644 --- a/vortex-array/src/arrays/fixed_size_list/compute/take.rs +++ b/vortex-array/src/arrays/fixed_size_list/compute/take.rs @@ -6,17 +6,19 @@ use vortex_buffer::BufferMut; use vortex_error::VortexExpect; use vortex_error::VortexResult; use vortex_error::vortex_panic; +use vortex_mask::Mask; use crate::ArrayRef; use crate::IntoArray; use crate::array::ArrayView; +use crate::arrays::BoolArray; use crate::arrays::FixedSizeList; use crate::arrays::FixedSizeListArray; use crate::arrays::Primitive; use crate::arrays::PrimitiveArray; +use crate::arrays::bool::BoolArrayExt; use crate::arrays::dict::TakeExecute; use crate::arrays::fixed_size_list::FixedSizeListArrayExt; -use crate::arrays::primitive::PrimitiveArrayExt; use crate::dtype::IntegerPType; use crate::executor::ExecutionCtx; use crate::match_each_integer_ptype; @@ -82,7 +84,7 @@ fn take_with_indices( // The result's nullability is the union of the input nullabilities. if array.dtype().is_nullable() || indices_array.dtype().is_nullable() { let indices_array = indices_array.as_view(); - take_nullable_fsl::(array, indices_array) + take_nullable_fsl::(array, indices_array, ctx) } else { let indices_array = indices_array.as_view(); take_non_nullable_fsl::(array, indices_array) @@ -143,13 +145,25 @@ fn take_non_nullable_fsl( fn take_nullable_fsl( array: ArrayView<'_, FixedSizeList>, indices_array: ArrayView<'_, Primitive>, + ctx: &mut ExecutionCtx, ) -> VortexResult { let list_size = array.list_size() as usize; let indices: &[I] = indices_array.as_slice::(); let new_len = indices.len(); - let array_validity = array.fixed_size_list_validity_mask(); - let indices_validity = indices_array.validity_mask(); + let array_validity = array + .fixed_size_list_validity() + .execute_mask(array.as_ref().len(), ctx) + .vortex_expect("Failed to compute validity mask"); + let indices_len = indices_array.as_ref().len(); + let indices_validity = match indices_array + .validity() + .vortex_expect("Failed to compute validity mask") + { + Validity::NonNullable | Validity::AllValid => Mask::new_true(indices_len), + Validity::AllInvalid => Mask::new_false(indices_len), + Validity::Array(a) => a.execute::(ctx)?.execute_mask(ctx), + }; // We must use placeholder zeros for null lists to maintain the array length without // propagating nullability to the element array's take operation. diff --git a/vortex-array/src/arrays/fixed_size_list/tests/basic.rs b/vortex-array/src/arrays/fixed_size_list/tests/basic.rs index 3d4b6e20ce2..f910bd4a05f 100644 --- a/vortex-array/src/arrays/fixed_size_list/tests/basic.rs +++ b/vortex-array/src/arrays/fixed_size_list/tests/basic.rs @@ -6,6 +6,8 @@ use std::sync::Arc; use vortex_buffer::buffer; use crate::IntoArray; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::arrays::FixedSizeListArray; use crate::arrays::fixed_size_list::FixedSizeListArrayExt; use crate::dtype::DType; @@ -36,24 +38,84 @@ fn test_basic_fixed_size_list() { // Check the actual values in each list. let first_list = fsl.fixed_size_list_elements_at(0).unwrap(); - assert_eq!(first_list.scalar_at(0).unwrap(), 1i32.into()); - assert_eq!(first_list.scalar_at(1).unwrap(), 2i32.into()); - assert_eq!(first_list.scalar_at(2).unwrap(), 3i32.into()); + assert_eq!( + first_list + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 1i32.into() + ); + assert_eq!( + first_list + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 2i32.into() + ); + assert_eq!( + first_list + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 3i32.into() + ); let second_list = fsl.fixed_size_list_elements_at(1).unwrap(); - assert_eq!(second_list.scalar_at(0).unwrap(), 4i32.into()); - assert_eq!(second_list.scalar_at(1).unwrap(), 5i32.into()); - assert_eq!(second_list.scalar_at(2).unwrap(), 6i32.into()); + assert_eq!( + second_list + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 4i32.into() + ); + assert_eq!( + second_list + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 5i32.into() + ); + assert_eq!( + second_list + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 6i32.into() + ); let third_list = fsl.fixed_size_list_elements_at(2).unwrap(); - assert_eq!(third_list.scalar_at(0).unwrap(), 7i32.into()); - assert_eq!(third_list.scalar_at(1).unwrap(), 8i32.into()); - assert_eq!(third_list.scalar_at(2).unwrap(), 9i32.into()); + assert_eq!( + third_list + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 7i32.into() + ); + assert_eq!( + third_list + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 8i32.into() + ); + assert_eq!( + third_list + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 9i32.into() + ); let fourth_list = fsl.fixed_size_list_elements_at(3).unwrap(); - assert_eq!(fourth_list.scalar_at(0).unwrap(), 10i32.into()); - assert_eq!(fourth_list.scalar_at(1).unwrap(), 11i32.into()); - assert_eq!(fourth_list.scalar_at(2).unwrap(), 12i32.into()); + assert_eq!( + fourth_list + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 10i32.into() + ); + assert_eq!( + fourth_list + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 11i32.into() + ); + assert_eq!( + fourth_list + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 12i32.into() + ); } #[test] @@ -65,7 +127,9 @@ fn test_scalar_at() { let fsl = FixedSizeListArray::new(elements.into_array(), list_size, Validity::NonNullable, len); // First list: [1, 2, 3]. - let first = fsl.scalar_at(0).unwrap(); + let first = fsl + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert_eq!( first, Scalar::fixed_size_list( @@ -77,12 +141,29 @@ fn test_scalar_at() { // Additionally check individual elements via fixed_size_list_at. let first_list = fsl.fixed_size_list_elements_at(0).unwrap(); - assert_eq!(first_list.scalar_at(0).unwrap(), 1i32.into()); - assert_eq!(first_list.scalar_at(1).unwrap(), 2i32.into()); - assert_eq!(first_list.scalar_at(2).unwrap(), 3i32.into()); + assert_eq!( + first_list + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 1i32.into() + ); + assert_eq!( + first_list + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 2i32.into() + ); + assert_eq!( + first_list + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 3i32.into() + ); // Second list: [4, 5, 6]. - let second = fsl.scalar_at(1).unwrap(); + let second = fsl + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert_eq!( second, Scalar::fixed_size_list( @@ -94,9 +175,24 @@ fn test_scalar_at() { // Additionally check individual elements via fixed_size_list_at. let second_list = fsl.fixed_size_list_elements_at(1).unwrap(); - assert_eq!(second_list.scalar_at(0).unwrap(), 4i32.into()); - assert_eq!(second_list.scalar_at(1).unwrap(), 5i32.into()); - assert_eq!(second_list.scalar_at(2).unwrap(), 6i32.into()); + assert_eq!( + second_list + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 4i32.into() + ); + assert_eq!( + second_list + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 5i32.into() + ); + assert_eq!( + second_list + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 6i32.into() + ); } #[test] @@ -110,14 +206,34 @@ fn test_fixed_size_list_at() { // Get the first list [1.0, 2.0]. let first_list = fsl.fixed_size_list_elements_at(0).unwrap(); assert_eq!(first_list.len(), list_size as usize); - assert_eq!(first_list.scalar_at(0).unwrap(), 1.0f64.into()); - assert_eq!(first_list.scalar_at(1).unwrap(), 2.0f64.into()); + assert_eq!( + first_list + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 1.0f64.into() + ); + assert_eq!( + first_list + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 2.0f64.into() + ); // Get the third list [5.0, 6.0]. let third_list = fsl.fixed_size_list_elements_at(2).unwrap(); assert_eq!(third_list.len(), list_size as usize); - assert_eq!(third_list.scalar_at(0).unwrap(), 5.0f64.into()); - assert_eq!(third_list.scalar_at(1).unwrap(), 6.0f64.into()); + assert_eq!( + third_list + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 5.0f64.into() + ); + assert_eq!( + third_list + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 6.0f64.into() + ); } #[test] diff --git a/vortex-array/src/arrays/fixed_size_list/tests/degenerate.rs b/vortex-array/src/arrays/fixed_size_list/tests/degenerate.rs index a6fb279788d..39a7ada3af3 100644 --- a/vortex-array/src/arrays/fixed_size_list/tests/degenerate.rs +++ b/vortex-array/src/arrays/fixed_size_list/tests/degenerate.rs @@ -6,6 +6,8 @@ use std::sync::Arc; use vortex_buffer::buffer; use crate::IntoArray; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::arrays::FixedSizeListArray; use crate::arrays::PrimitiveArray; use crate::arrays::fixed_size_list::FixedSizeListArrayExt; @@ -54,7 +56,9 @@ fn test_fsl_size_0_length_1_non_nullable() { assert_eq!(fsl.elements().len(), 0); // Get the single empty list. - let scalar = fsl.scalar_at(0).unwrap(); + let scalar = fsl + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(!scalar.is_null()); assert_eq!( scalar, @@ -80,7 +84,9 @@ fn test_fsl_size_0_huge_length_non_nullable() { assert_eq!(fsl.elements().len(), 0); // Spot check a few lists. - let scalar_first = fsl.scalar_at(0).unwrap(); + let scalar_first = fsl + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(!scalar_first.is_null()); assert_eq!( scalar_first, @@ -91,7 +97,9 @@ fn test_fsl_size_0_huge_length_non_nullable() { ) ); - let scalar_middle = fsl.scalar_at(500_000_000_000).unwrap(); + let scalar_middle = fsl + .execute_scalar(500_000_000_000, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(!scalar_middle.is_null()); assert_eq!( scalar_middle, @@ -102,7 +110,9 @@ fn test_fsl_size_0_huge_length_non_nullable() { ) ); - let scalar_end = fsl.scalar_at(999_999_999_999).unwrap(); + let scalar_end = fsl + .execute_scalar(999_999_999_999, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(!scalar_end.is_null()); assert_eq!( scalar_end, @@ -152,7 +162,9 @@ fn test_fsl_size_0_length_1_nullable_valid() { assert_eq!(fsl.elements().len(), 0); // Get the single empty list (should be valid). - let scalar = fsl.scalar_at(0).unwrap(); + let scalar = fsl + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(!scalar.is_null()); assert_eq!( scalar, @@ -175,7 +187,9 @@ fn test_fsl_size_0_length_1_nullable_null() { assert_eq!(fsl.elements().len(), 0); // The single list should be null. - let scalar = fsl.scalar_at(0).unwrap(); + let scalar = fsl + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(scalar.is_null()); } @@ -200,7 +214,9 @@ fn test_fsl_size_0_length_10_nullable_mixed() { true, false, true, true, false, false, true, false, true, true, ]; for i in 0..len { - let scalar = fsl.scalar_at(i).unwrap(); + let scalar = fsl + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); if expected_valid[i] { assert!(!scalar.is_null()); assert_eq!( @@ -239,7 +255,9 @@ fn test_fsl_size_0_nullable_elements() { // All lists should be empty but valid. for i in 0..len { - let scalar = fsl.scalar_at(i).unwrap(); + let scalar = fsl + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(!scalar.is_null()); } } diff --git a/vortex-array/src/arrays/fixed_size_list/tests/nested.rs b/vortex-array/src/arrays/fixed_size_list/tests/nested.rs index 41447881996..efd6f630580 100644 --- a/vortex-array/src/arrays/fixed_size_list/tests/nested.rs +++ b/vortex-array/src/arrays/fixed_size_list/tests/nested.rs @@ -6,7 +6,10 @@ use std::sync::Arc; use vortex_buffer::buffer; use crate::IntoArray; -use crate::ToCanonical; +use crate::LEGACY_SESSION; +#[expect(deprecated)] +use crate::ToCanonical as _; +use crate::VortexSessionExecute; use crate::arrays::FixedSizeListArray; use crate::arrays::PrimitiveArray; use crate::arrays::StructArray; @@ -72,7 +75,9 @@ fn test_fsl_of_fsl_basic() { // The first outer list should contain 3 inner lists. // We can check by slicing and examining scalars. - let first_scalar = outer_fsl.scalar_at(0).unwrap(); + let first_scalar = outer_fsl + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(!first_scalar.is_null()); // Check the actual values in the nested structure. @@ -80,55 +85,121 @@ fn test_fsl_of_fsl_basic() { let first_outer_list = outer_fsl.fixed_size_list_elements_at(0).unwrap(); // Check first inner list [1,2]. + #[expect(deprecated)] let inner_list_0 = first_outer_list .to_fixed_size_list() .fixed_size_list_elements_at(0) .unwrap(); - assert_eq!(inner_list_0.scalar_at(0).unwrap(), 1i32.into()); - assert_eq!(inner_list_0.scalar_at(1).unwrap(), 2i32.into()); + assert_eq!( + inner_list_0 + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 1i32.into() + ); + assert_eq!( + inner_list_0 + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 2i32.into() + ); // Check second inner list [3,4]. + #[expect(deprecated)] let inner_list_1 = first_outer_list .to_fixed_size_list() .fixed_size_list_elements_at(1) .unwrap(); - assert_eq!(inner_list_1.scalar_at(0).unwrap(), 3i32.into()); - assert_eq!(inner_list_1.scalar_at(1).unwrap(), 4i32.into()); + assert_eq!( + inner_list_1 + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 3i32.into() + ); + assert_eq!( + inner_list_1 + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 4i32.into() + ); // Check third inner list [5,6]. + #[expect(deprecated)] let inner_list_2 = first_outer_list .to_fixed_size_list() .fixed_size_list_elements_at(2) .unwrap(); - assert_eq!(inner_list_2.scalar_at(0).unwrap(), 5i32.into()); - assert_eq!(inner_list_2.scalar_at(1).unwrap(), 6i32.into()); + assert_eq!( + inner_list_2 + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 5i32.into() + ); + assert_eq!( + inner_list_2 + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 6i32.into() + ); // Second outer list contains: [[7,8], [9,10], [11,12]]. let second_outer_list = outer_fsl.fixed_size_list_elements_at(1).unwrap(); // Check first inner list [7,8]. + #[expect(deprecated)] let inner_list_0 = second_outer_list .to_fixed_size_list() .fixed_size_list_elements_at(0) .unwrap(); - assert_eq!(inner_list_0.scalar_at(0).unwrap(), 7i32.into()); - assert_eq!(inner_list_0.scalar_at(1).unwrap(), 8i32.into()); + assert_eq!( + inner_list_0 + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 7i32.into() + ); + assert_eq!( + inner_list_0 + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 8i32.into() + ); // Check second inner list [9,10]. + #[expect(deprecated)] let inner_list_1 = second_outer_list .to_fixed_size_list() .fixed_size_list_elements_at(1) .unwrap(); - assert_eq!(inner_list_1.scalar_at(0).unwrap(), 9i32.into()); - assert_eq!(inner_list_1.scalar_at(1).unwrap(), 10i32.into()); + assert_eq!( + inner_list_1 + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 9i32.into() + ); + assert_eq!( + inner_list_1 + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 10i32.into() + ); // Check third inner list [11,12]. + #[expect(deprecated)] let inner_list_2 = second_outer_list .to_fixed_size_list() .fixed_size_list_elements_at(2) .unwrap(); - assert_eq!(inner_list_2.scalar_at(0).unwrap(), 11i32.into()); - assert_eq!(inner_list_2.scalar_at(1).unwrap(), 12i32.into()); + assert_eq!( + inner_list_2 + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 11i32.into() + ); + assert_eq!( + inner_list_2 + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 12i32.into() + ); } #[test] @@ -173,13 +244,28 @@ fn test_fsl_of_fsl_with_nulls() { assert_eq!(outer_fsl.len(), outer_len); // First outer list is valid. - assert!(!outer_fsl.scalar_at(0).unwrap().is_null()); + assert!( + !outer_fsl + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); // Second outer list is null. - assert!(outer_fsl.scalar_at(1).unwrap().is_null()); + assert!( + outer_fsl + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); // Third outer list is valid. - assert!(!outer_fsl.scalar_at(2).unwrap().is_null()); + assert!( + !outer_fsl + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); } #[test] @@ -222,44 +308,90 @@ fn test_deeply_nested_fsl() { // Check the actual deeply nested values. // Structure: [[[1,2],[3,4]],[[5,6],[7,8]]]. let top_level = level3.fixed_size_list_elements_at(0).unwrap(); + #[expect(deprecated)] let level2_0 = top_level .to_fixed_size_list() .fixed_size_list_elements_at(0) .unwrap(); + #[expect(deprecated)] let level2_1 = top_level .to_fixed_size_list() .fixed_size_list_elements_at(1) .unwrap(); // First level-2 list: [[1,2],[3,4]]. + #[expect(deprecated)] let level1_0_0 = level2_0 .to_fixed_size_list() .fixed_size_list_elements_at(0) .unwrap(); - assert_eq!(level1_0_0.scalar_at(0).unwrap(), 1i32.into()); - assert_eq!(level1_0_0.scalar_at(1).unwrap(), 2i32.into()); + assert_eq!( + level1_0_0 + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 1i32.into() + ); + assert_eq!( + level1_0_0 + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 2i32.into() + ); + #[expect(deprecated)] let level1_0_1 = level2_0 .to_fixed_size_list() .fixed_size_list_elements_at(1) .unwrap(); - assert_eq!(level1_0_1.scalar_at(0).unwrap(), 3i32.into()); - assert_eq!(level1_0_1.scalar_at(1).unwrap(), 4i32.into()); + assert_eq!( + level1_0_1 + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 3i32.into() + ); + assert_eq!( + level1_0_1 + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 4i32.into() + ); // Second level-2 list: [[5,6],[7,8]]. + #[expect(deprecated)] let level1_1_0 = level2_1 .to_fixed_size_list() .fixed_size_list_elements_at(0) .unwrap(); - assert_eq!(level1_1_0.scalar_at(0).unwrap(), 5i32.into()); - assert_eq!(level1_1_0.scalar_at(1).unwrap(), 6i32.into()); + assert_eq!( + level1_1_0 + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 5i32.into() + ); + assert_eq!( + level1_1_0 + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 6i32.into() + ); + #[expect(deprecated)] let level1_1_1 = level2_1 .to_fixed_size_list() .fixed_size_list_elements_at(1) .unwrap(); - assert_eq!(level1_1_1.scalar_at(0).unwrap(), 7i32.into()); - assert_eq!(level1_1_1.scalar_at(1).unwrap(), 8i32.into()); + assert_eq!( + level1_1_1 + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 7i32.into() + ); + assert_eq!( + level1_1_1 + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 8i32.into() + ); } //////////////////////////////////////////////////////////////////////////////////////////////////// diff --git a/vortex-array/src/arrays/fixed_size_list/tests/nullability.rs b/vortex-array/src/arrays/fixed_size_list/tests/nullability.rs index 13894ae3ec4..20deef502ed 100644 --- a/vortex-array/src/arrays/fixed_size_list/tests/nullability.rs +++ b/vortex-array/src/arrays/fixed_size_list/tests/nullability.rs @@ -6,6 +6,8 @@ use std::sync::Arc; use vortex_buffer::buffer; use crate::IntoArray; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::arrays::BoolArray; use crate::arrays::FixedSizeListArray; use crate::arrays::PrimitiveArray; @@ -30,7 +32,9 @@ fn test_nullable_fsl_with_nulls() { assert_eq!(fsl.list_size(), list_size); // First list is valid: [1, 2]. - let first = fsl.scalar_at(0).unwrap(); + let first = fsl + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(!first.is_null()); assert_eq!( first, @@ -43,15 +47,29 @@ fn test_nullable_fsl_with_nulls() { // Check individual elements of the first list. let first_list = fsl.fixed_size_list_elements_at(0).unwrap(); - assert_eq!(first_list.scalar_at(0).unwrap(), 1i32.into()); - assert_eq!(first_list.scalar_at(1).unwrap(), 2i32.into()); + assert_eq!( + first_list + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 1i32.into() + ); + assert_eq!( + first_list + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 2i32.into() + ); // Second list is null. - let second = fsl.scalar_at(1).unwrap(); + let second = fsl + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(second.is_null()); // Third list is valid: [5, 6]. - let third = fsl.scalar_at(2).unwrap(); + let third = fsl + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(!third.is_null()); assert_eq!( third, @@ -64,11 +82,23 @@ fn test_nullable_fsl_with_nulls() { // Check individual elements of the third list. let third_list = fsl.fixed_size_list_elements_at(2).unwrap(); - assert_eq!(third_list.scalar_at(0).unwrap(), 5i32.into()); - assert_eq!(third_list.scalar_at(1).unwrap(), 6i32.into()); + assert_eq!( + third_list + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 5i32.into() + ); + assert_eq!( + third_list + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 6i32.into() + ); // Fourth list is null. - let fourth = fsl.scalar_at(3).unwrap(); + let fourth = fsl + .execute_scalar(3, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(fourth.is_null()); } @@ -92,7 +122,9 @@ fn test_nullable_elements_non_nullable_lists() { )); // First list: [Some(1), None, Some(3)]. - let first = fsl.scalar_at(0).unwrap(); + let first = fsl + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(!first.is_null()); assert_eq!( first, @@ -104,7 +136,9 @@ fn test_nullable_elements_non_nullable_lists() { ); // Second list: [Some(4), Some(5), None]. - let second = fsl.scalar_at(1).unwrap(); + let second = fsl + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(!second.is_null()); assert_eq!( second, @@ -130,7 +164,9 @@ fn test_nullable_elements_and_nullable_lists() { assert_eq!(fsl.len(), len); // First list is valid: [Some(10), None]. - let first = fsl.scalar_at(0).unwrap(); + let first = fsl + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(!first.is_null()); assert_eq!( first, @@ -143,15 +179,29 @@ fn test_nullable_elements_and_nullable_lists() { // Check individual elements of the first list. let first_list = fsl.fixed_size_list_elements_at(0).unwrap(); - assert_eq!(first_list.scalar_at(0).unwrap(), Some(10u16).into()); - assert_eq!(first_list.scalar_at(1).unwrap(), None::.into()); + assert_eq!( + first_list + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + Some(10u16).into() + ); + assert_eq!( + first_list + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + None::.into() + ); // Second list is null (but elements would be [Some(20), Some(30)]). - let second = fsl.scalar_at(1).unwrap(); + let second = fsl + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(second.is_null()); // Third list is valid: [None, None]. - let third = fsl.scalar_at(2).unwrap(); + let third = fsl + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(!third.is_null()); assert_eq!( third, @@ -164,8 +214,18 @@ fn test_nullable_elements_and_nullable_lists() { // Check individual elements of the third list. let third_list = fsl.fixed_size_list_elements_at(2).unwrap(); - assert_eq!(third_list.scalar_at(0).unwrap(), None::.into()); - assert_eq!(third_list.scalar_at(1).unwrap(), None::.into()); + assert_eq!( + third_list + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + None::.into() + ); + assert_eq!( + third_list + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + None::.into() + ); } #[test] @@ -182,7 +242,9 @@ fn test_alternating_nulls() { // Check alternating pattern. for i in 0..len { - let scalar = fsl.scalar_at(i).unwrap(); + let scalar = fsl + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); if i % 2 == 0 { assert!(!scalar.is_null()); let expected_value = u8::try_from(i + 1).unwrap(); @@ -212,7 +274,11 @@ fn test_validity_types() { { let fsl = FixedSizeListArray::new(elements.clone(), list_size, Validity::AllInvalid, len); for i in 0..len { - assert!(fsl.scalar_at(i).unwrap().is_null()); + assert!( + fsl.execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); } } @@ -226,10 +292,26 @@ fn test_validity_types() { len, ); - assert!(!fsl.scalar_at(0).unwrap().is_null()); - assert!(!fsl.scalar_at(1).unwrap().is_null()); - assert!(fsl.scalar_at(2).unwrap().is_null()); - assert!(!fsl.scalar_at(3).unwrap().is_null()); + assert!( + !fsl.execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); + assert!( + !fsl.execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); + assert!( + fsl.execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); + assert!( + !fsl.execute_scalar(3, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); } } @@ -255,22 +337,32 @@ fn test_mixed_nullability_patterns() { let fsl = FixedSizeListArray::new(elements.into_array(), list_size, validity, len); // List 0: valid with [Some(1), None]. - let list0 = fsl.scalar_at(0).unwrap(); + let list0 = fsl + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(!list0.is_null()); // List 1: null. - let list1 = fsl.scalar_at(1).unwrap(); + let list1 = fsl + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(list1.is_null()); // List 2: valid with [Some(5), Some(6)]. - let list2 = fsl.scalar_at(2).unwrap(); + let list2 = fsl + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(!list2.is_null()); // List 3: valid with [Some(7), None]. - let list3 = fsl.scalar_at(3).unwrap(); + let list3 = fsl + .execute_scalar(3, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(!list3.is_null()); // List 4: valid with [None, Some(10)]. - let list4 = fsl.scalar_at(4).unwrap(); + let list4 = fsl + .execute_scalar(4, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(!list4.is_null()); } diff --git a/vortex-array/src/arrays/fixed_size_list/tests/take.rs b/vortex-array/src/arrays/fixed_size_list/tests/take.rs index 53b959d22a6..8bccb80c8cb 100644 --- a/vortex-array/src/arrays/fixed_size_list/tests/take.rs +++ b/vortex-array/src/arrays/fixed_size_list/tests/take.rs @@ -11,6 +11,8 @@ use super::common::create_nullable_fsl; use super::common::create_single_element_fsl; use crate::ArrayRef; use crate::IntoArray; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::arrays::FixedSizeListArray; use crate::arrays::PrimitiveArray; use crate::assert_arrays_eq; @@ -93,7 +95,13 @@ fn test_take_degenerate_lists( assert_eq!(result.len(), expected_len); for (i, expected_null) in expected_nulls.iter().enumerate() { - assert_eq!(result.scalar_at(i).unwrap().is_null(), *expected_null); + assert_eq!( + result + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null(), + *expected_null + ); } } @@ -221,6 +229,12 @@ fn test_take_nullable_arrays_fsl_specific( assert_eq!(result.len(), indices.len()); for (i, expected_null) in expected_nulls.iter().enumerate() { - assert_eq!(result.scalar_at(i).unwrap().is_null(), *expected_null); + assert_eq!( + result + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null(), + *expected_null + ); } } diff --git a/vortex-array/src/arrays/fixed_size_list/vtable/kernel.rs b/vortex-array/src/arrays/fixed_size_list/vtable/kernel.rs index c28a3a2a805..319751070de 100644 --- a/vortex-array/src/arrays/fixed_size_list/vtable/kernel.rs +++ b/vortex-array/src/arrays/fixed_size_list/vtable/kernel.rs @@ -4,8 +4,11 @@ use crate::arrays::FixedSizeList; use crate::arrays::dict::TakeExecuteAdaptor; use crate::kernel::ParentKernelSet; +use crate::scalar_fn::fns::cast::CastExecuteAdaptor; impl FixedSizeList { - pub(crate) const PARENT_KERNELS: ParentKernelSet = - ParentKernelSet::new(&[ParentKernelSet::lift(&TakeExecuteAdaptor(FixedSizeList))]); + pub(crate) const PARENT_KERNELS: ParentKernelSet = ParentKernelSet::new(&[ + ParentKernelSet::lift(&CastExecuteAdaptor(FixedSizeList)), + ParentKernelSet::lift(&TakeExecuteAdaptor(FixedSizeList)), + ]); } diff --git a/vortex-array/src/arrays/fixed_size_list/vtable/mod.rs b/vortex-array/src/arrays/fixed_size_list/vtable/mod.rs index b2c02ec6891..a3724b06fca 100644 --- a/vortex-array/src/arrays/fixed_size_list/vtable/mod.rs +++ b/vortex-array/src/arrays/fixed_size_list/vtable/mod.rs @@ -11,6 +11,7 @@ use vortex_error::vortex_bail; use vortex_error::vortex_ensure; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::ArrayEq; use crate::ArrayHash; @@ -40,10 +41,6 @@ pub type FixedSizeListArray = Array; #[derive(Clone, Debug)] pub struct FixedSizeList; -impl FixedSizeList { - pub const ID: ArrayId = ArrayId::new_ref("vortex.fixed_size_list"); -} - impl ArrayHash for FixedSizeListData { fn array_hash(&self, state: &mut H, precision: Precision) { let _precision = precision; @@ -63,9 +60,9 @@ impl VTable for FixedSizeList { type OperationsVTable = Self; type ValidityVTable = Self; - fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.fixed_size_list"); + *ID } fn nbuffers(_array: ArrayView<'_, Self>) -> usize { diff --git a/vortex-array/src/arrays/fixed_size_list/vtable/operations.rs b/vortex-array/src/arrays/fixed_size_list/vtable/operations.rs index ebd2ed463a4..9f4cf02fbf8 100644 --- a/vortex-array/src/arrays/fixed_size_list/vtable/operations.rs +++ b/vortex-array/src/arrays/fixed_size_list/vtable/operations.rs @@ -14,12 +14,12 @@ impl OperationsVTable for FixedSizeList { fn scalar_at( array: ArrayView<'_, FixedSizeList>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { // By the preconditions we know that the list scalar is not null. let list = array.fixed_size_list_elements_at(index)?; let children_elements: Vec = (0..list.len()) - .map(|i| list.scalar_at(i)) + .map(|i| list.execute_scalar(i, ctx)) .collect::>()?; debug_assert_eq!(children_elements.len(), array.list_size() as usize); diff --git a/vortex-array/src/arrays/list/array.rs b/vortex-array/src/arrays/list/array.rs index 75db7d0f08c..140b112ff39 100644 --- a/vortex-array/src/arrays/list/array.rs +++ b/vortex-array/src/arrays/list/array.rs @@ -197,9 +197,10 @@ impl ListData { // We can safely unwrap the DType as primitive now let offsets_ptype = offsets.dtype().as_ptype(); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Offsets must be sorted (but not strictly sorted, zero-length lists are allowed) - if let Some(is_sorted) = offsets.statistics().compute_is_sorted() { + if let Some(is_sorted) = offsets.statistics().compute_is_sorted(&mut ctx) { vortex_ensure!(is_sorted, InvalidArgument: "offsets must be sorted"); } else { vortex_bail!(InvalidArgument: "offsets must report is_sorted statistic"); @@ -207,7 +208,6 @@ impl ListData { // Validate that offsets min is non-negative, and max does not exceed the length of // the elements array. - let mut ctx = LEGACY_SESSION.create_execution_ctx(); if let Some(min_max) = min_max(offsets, &mut ctx)? { match_each_integer_ptype!(offsets_ptype, |P| { #[allow(clippy::absurd_extreme_comparisons, unused_comparisons)] @@ -285,11 +285,10 @@ pub trait ListArrayExt: TypedArrayRef { } fn list_validity(&self) -> Validity { - child_to_validity(&self.as_ref().slots()[VALIDITY_SLOT], self.nullability()) - } - - fn list_validity_mask(&self) -> vortex_mask::Mask { - self.list_validity().to_mask(self.as_ref().len()) + child_to_validity( + self.as_ref().slots()[VALIDITY_SLOT].as_ref(), + self.nullability(), + ) } fn offset_at(&self, index: usize) -> VortexResult { @@ -305,7 +304,7 @@ pub trait ListArrayExt: TypedArrayRef { })) } else { self.offsets() - .scalar_at(index)? + .execute_scalar(index, &mut LEGACY_SESSION.create_execution_ctx())? .as_primitive() .as_::() .ok_or_else(|| vortex_error::vortex_err!("offset value does not fit in usize")) @@ -331,7 +330,9 @@ pub trait ListArrayExt: TypedArrayRef { fn reset_offsets(&self, recurse: bool) -> VortexResult> { let mut elements = self.sliced_elements()?; if recurse && elements.is_canonical() { - elements = elements.to_canonical()?.compact()?.into_array(); + #[expect(deprecated)] + let compacted = elements.to_canonical()?.compact()?.into_array(); + elements = compacted; } else if recurse && let Some(child_list_array) = elements.as_opt::() { elements = child_list_array .into_owned() @@ -340,7 +341,7 @@ pub trait ListArrayExt: TypedArrayRef { } let offsets = self.offsets(); - let first_offset = offsets.scalar_at(0)?; + let first_offset = offsets.execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx())?; let adjusted_offsets = offsets.clone().binary( ConstantArray::new(first_offset, offsets.len()).into_array(), Operator::Sub, diff --git a/vortex-array/src/arrays/list/compute/cast.rs b/vortex-array/src/arrays/list/compute/cast.rs index bac84916390..e9b29b9b145 100644 --- a/vortex-array/src/arrays/list/compute/cast.rs +++ b/vortex-array/src/arrays/list/compute/cast.rs @@ -4,6 +4,7 @@ use vortex_error::VortexResult; use crate::ArrayRef; +use crate::ExecutionCtx; use crate::IntoArray; use crate::array::ArrayView; use crate::arrays::List; @@ -11,6 +12,7 @@ use crate::arrays::ListArray; use crate::arrays::list::ListArrayExt; use crate::builtins::ArrayBuiltins; use crate::dtype::DType; +use crate::scalar_fn::fns::cast::CastKernel; use crate::scalar_fn::fns::cast::CastReduce; impl CastReduce for List { @@ -19,24 +21,56 @@ impl CastReduce for List { return Ok(None); }; + let Some(validity) = array + .validity()? + .trivial_cast_nullability(dtype.nullability(), array.len())? + else { + return Ok(None); + }; + + let new_elements = array.elements().cast((**target_element_type).clone())?; + + Ok(Some( + unsafe { ListArray::new_unchecked(new_elements, array.offsets().clone(), validity) } + .into_array(), + )) + } +} + +impl CastKernel for List { + fn cast( + array: ArrayView<'_, List>, + dtype: &DType, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + let Some(target_element_type) = dtype.as_list_element_opt() else { + return Ok(None); + }; + let validity = array .validity()? - .cast_nullability(dtype.nullability(), array.len())?; + .cast_nullability(dtype.nullability(), array.len(), ctx)?; let new_elements = array.elements().cast((**target_element_type).clone())?; - ListArray::try_new(new_elements, array.offsets().clone(), validity) - .map(|a| Some(a.into_array())) + Ok(Some( + unsafe { ListArray::new_unchecked(new_elements, array.offsets().clone(), validity) } + .into_array(), + )) } } #[cfg(test)] mod tests { use std::sync::Arc; + use std::sync::LazyLock; use rstest::rstest; + use vortex_array::session::ArraySession; use vortex_buffer::buffer; + use vortex_session::VortexSession; + use crate::Canonical; use crate::IntoArray; use crate::LEGACY_SESSION; use crate::RecursiveCanonical; @@ -52,6 +86,9 @@ mod tests { use crate::dtype::PType; use crate::validity::Validity; + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); + #[test] fn test_cast_list_success() { let list = ListArray::try_new( @@ -90,7 +127,8 @@ mod tests { let result = list .into_array() .cast(target_dtype) - .and_then(|a| a.to_canonical().map(|c| c.into_array())); + .and_then(|a| a.execute::(&mut SESSION.create_execution_ctx())) + .map(|c| c.into_array()); assert!(result.is_err()); } @@ -114,7 +152,8 @@ mod tests { let result = list .into_array() .cast(target_dtype) - .and_then(|a| a.to_canonical().map(|c| c.into_array())); + .and_then(|a| a.execute::(&mut SESSION.create_execution_ctx())) + .map(|c| c.into_array()); assert!(result.is_err()); // Nulls in list element array — the inner cast error is deferred until diff --git a/vortex-array/src/arrays/list/compute/kernels.rs b/vortex-array/src/arrays/list/compute/kernels.rs index b4268642e82..188c83c1bf5 100644 --- a/vortex-array/src/arrays/list/compute/kernels.rs +++ b/vortex-array/src/arrays/list/compute/kernels.rs @@ -5,8 +5,10 @@ use crate::arrays::List; use crate::arrays::dict::TakeExecuteAdaptor; use crate::arrays::filter::FilterExecuteAdaptor; use crate::kernel::ParentKernelSet; +use crate::scalar_fn::fns::cast::CastExecuteAdaptor; pub(crate) const PARENT_KERNELS: ParentKernelSet = ParentKernelSet::new(&[ + ParentKernelSet::lift(&CastExecuteAdaptor(List)), ParentKernelSet::lift(&FilterExecuteAdaptor(List)), ParentKernelSet::lift(&TakeExecuteAdaptor(List)), ]); diff --git a/vortex-array/src/arrays/list/compute/take.rs b/vortex-array/src/arrays/list/compute/take.rs index 841fd4ac45b..7cc5ccd78cd 100644 --- a/vortex-array/src/arrays/list/compute/take.rs +++ b/vortex-array/src/arrays/list/compute/take.rs @@ -59,8 +59,13 @@ fn _take( indices_array: ArrayView<'_, Primitive>, ctx: &mut ExecutionCtx, ) -> VortexResult { - let data_validity = array.list_validity_mask(); - let indices_validity = indices_array.validity_mask(); + let data_validity = array + .list_validity() + .execute_mask(array.as_ref().len(), ctx)?; + let indices_validity = indices_array + .validity() + .vortex_expect("Failed to compute validity mask") + .execute_mask(indices_array.as_ref().len(), ctx)?; if !indices_validity.all_true() || !data_validity.all_true() { return _take_nullable::(array, indices_array, ctx); @@ -124,8 +129,13 @@ fn _take_nullable(ctx)?; let offsets: &[O] = offsets_array.as_slice(); let indices: &[I] = indices_array.as_slice(); - let data_validity = array.list_validity_mask(); - let indices_validity = indices_array.validity_mask(); + let data_validity = array + .list_validity() + .execute_mask(array.as_ref().len(), ctx)?; + let indices_validity = indices_array + .validity() + .vortex_expect("Failed to compute validity mask") + .execute_mask(indices_array.as_ref().len(), ctx)?; let mut new_offsets = PrimitiveBuilder::::with_capacity( Nullability::NonNullable, @@ -193,7 +203,10 @@ mod test { use vortex_buffer::buffer; use crate::IntoArray as _; - use crate::ToCanonical; + use crate::LEGACY_SESSION; + #[expect(deprecated)] + use crate::ToCanonical as _; + use crate::VortexSessionExecute; use crate::arrays::BoolArray; use crate::arrays::ListArray; use crate::arrays::PrimitiveArray; @@ -227,15 +240,22 @@ mod test { ) ); + #[expect(deprecated)] let result = result.to_listview(); assert_eq!(result.len(), 4); let element_dtype: Arc = Arc::new(I32.into()); - assert!(result.is_valid(0).unwrap()); + assert!( + result + .is_valid(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); assert_eq!( - result.scalar_at(0).unwrap(), + result + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::list( Arc::clone(&element_dtype), vec![0i32.into(), 5.into()], @@ -243,11 +263,21 @@ mod test { ) ); - assert!(result.is_invalid(1).unwrap()); + assert!( + result + .is_invalid(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); - assert!(result.is_valid(2).unwrap()); + assert!( + result + .is_valid(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); assert_eq!( - result.scalar_at(2).unwrap(), + result + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::list( Arc::clone(&element_dtype), vec![3i32.into()], @@ -255,9 +285,15 @@ mod test { ) ); - assert!(result.is_valid(3).unwrap()); + assert!( + result + .is_valid(3, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); assert_eq!( - result.scalar_at(3).unwrap(), + result + .execute_scalar(3, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::list(element_dtype, vec![], Nullability::Nullable) ); } @@ -307,15 +343,22 @@ mod test { ) ); + #[expect(deprecated)] let result = result.to_listview(); assert_eq!(result.len(), 3); let element_dtype: Arc = Arc::new(I32.into()); - assert!(result.is_valid(0).unwrap()); + assert!( + result + .is_valid(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); assert_eq!( - result.scalar_at(0).unwrap(), + result + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::list( Arc::clone(&element_dtype), vec![3i32.into()], @@ -323,9 +366,15 @@ mod test { ) ); - assert!(result.is_valid(1).unwrap()); + assert!( + result + .is_valid(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); assert_eq!( - result.scalar_at(1).unwrap(), + result + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::list( Arc::clone(&element_dtype), vec![0i32.into(), 5.into()], @@ -333,9 +382,15 @@ mod test { ) ); - assert!(result.is_valid(2).unwrap()); + assert!( + result + .is_valid(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); assert_eq!( - result.scalar_at(2).unwrap(), + result + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::list(element_dtype, vec![], Nullability::NonNullable) ); } @@ -419,10 +474,19 @@ mod test { assert_eq!(result.len(), 2); + #[expect(deprecated)] let result_view = result.to_listview(); assert_eq!(result_view.len(), 2); - assert!(result_view.is_valid(0).unwrap()); - assert!(result_view.is_valid(1).unwrap()); + assert!( + result_view + .is_valid(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); + assert!( + result_view + .is_valid(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); } #[test] @@ -440,11 +504,24 @@ mod test { assert_eq!(result.len(), 3); + #[expect(deprecated)] let result_view = result.to_listview(); assert_eq!(result_view.len(), 3); - assert!(result_view.is_valid(0).unwrap()); - assert!(result_view.is_invalid(1).unwrap()); - assert!(result_view.is_valid(2).unwrap()); + assert!( + result_view + .is_valid(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); + assert!( + result_view + .is_invalid(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); + assert!( + result_view + .is_valid(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); } /// Regression test for validity length mismatch bug. diff --git a/vortex-array/src/arrays/list/tests.rs b/vortex-array/src/arrays/list/tests.rs index e2e7407d111..f27e0a9f305 100644 --- a/vortex-array/src/arrays/list/tests.rs +++ b/vortex-array/src/arrays/list/tests.rs @@ -51,7 +51,8 @@ fn test_simple_list_array() { vec![1.into(), 2.into()], Nullability::Nullable ), - list.scalar_at(0).unwrap() + list.execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() ); assert_eq!( Scalar::list( @@ -59,11 +60,13 @@ fn test_simple_list_array() { vec![3.into(), 4.into()], Nullability::Nullable ), - list.scalar_at(1).unwrap() + list.execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() ); assert_eq!( Scalar::list(Arc::new(I32.into()), vec![5.into()], Nullability::Nullable), - list.scalar_at(2).unwrap() + list.execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() ); } @@ -81,12 +84,18 @@ fn test_simple_list_array_from_iter() { assert_eq!(list.len(), list_from_iter.len()); assert_eq!( - list.scalar_at(0).unwrap(), - list_from_iter.scalar_at(0).unwrap() + list.execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + list_from_iter + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() ); assert_eq!( - list.scalar_at(1).unwrap(), - list_from_iter.scalar_at(1).unwrap() + list.execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + list_from_iter + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() ); } @@ -218,10 +227,30 @@ fn test_list_filter_with_nulls() { assert_eq!(filtered.len(), 4); // Check validity of filtered array using scalar_at (works on any array). - assert!(filtered.scalar_at(0).unwrap().is_valid()); - assert!(!filtered.scalar_at(1).unwrap().is_valid()); // Was null. - assert!(!filtered.scalar_at(2).unwrap().is_valid()); // Was null. - assert!(filtered.scalar_at(3).unwrap().is_valid()); + assert!( + filtered + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_valid() + ); + assert!( + !filtered + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_valid() + ); // Was null. + assert!( + !filtered + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_valid() + ); // Was null. + assert!( + filtered + .execute_scalar(3, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_valid() + ); } #[test] @@ -443,7 +472,7 @@ fn test_offset_to_0() { type OptVec = Vec>; // Helper function to create a list of lists from a 3D vector with Option types. -#[allow(clippy::cast_possible_truncation)] +#[expect(clippy::cast_possible_truncation)] fn create_list_of_lists_nullable(data: OptVec>>) -> ListArray { // Flatten all elements and track offsets and validity. let mut all_elements = Vec::new(); @@ -625,7 +654,9 @@ fn test_list_of_lists() { assert_arrays_eq!(inner, PrimitiveArray::from_iter([7])); // Test scalar conversion. - let scalar = list_of_lists.scalar_at(0).unwrap(); + let scalar = list_of_lists + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(matches!(scalar.dtype(), DType::List(_, _))); let list_scalar = scalar.as_list(); assert_eq!(list_scalar.len(), 2); @@ -670,11 +701,15 @@ fn test_list_of_lists_nullable_outer() { )); // First element should be [[1, 2], [3]]. - let first = list_of_lists.scalar_at(0).unwrap(); + let first = list_of_lists + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(!first.is_null()); // Second element should be null. - let second = list_of_lists.scalar_at(1).unwrap(); + let second = list_of_lists + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(second.is_null()); // Third element should be [[4, 5, 6]]. @@ -727,7 +762,10 @@ fn test_list_of_lists_nullable_inner() { assert_eq!(first_list.len(), 3); // Check that second inner list is null. - let second_inner = first_list.array().scalar_at(1).unwrap(); + let second_inner = first_list + .array() + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(second_inner.is_null()); } @@ -755,7 +793,9 @@ fn test_list_of_lists_both_nullable() { )); // First outer list should have 2 elements, second is null inner list. - let first_outer = list_of_lists.scalar_at(0).unwrap(); + let first_outer = list_of_lists + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(!first_outer.is_null()); let first_outer_array = list_of_lists.list_elements_at(0).unwrap(); let first_list = first_outer_array.as_::(); @@ -766,11 +806,16 @@ fn test_list_of_lists_both_nullable() { assert_eq!(first_inner.len(), 2); // Second inner list should be null. - let second_inner = first_list.array().scalar_at(1).unwrap(); + let second_inner = first_list + .array() + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(second_inner.is_null()); // Second outer list should be null. - let second_outer = list_of_lists.scalar_at(1).unwrap(); + let second_outer = list_of_lists + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(second_outer.is_null()); // Third outer list should have [3]. @@ -784,7 +829,10 @@ fn test_list_of_lists_both_nullable() { let fourth_outer = list_of_lists.list_elements_at(3).unwrap(); let fourth_list = fourth_outer.as_::(); assert_eq!(fourth_list.len(), 1); - let inner = fourth_list.array().scalar_at(0).unwrap(); + let inner = fourth_list + .array() + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(inner.is_null()); } @@ -916,8 +964,12 @@ fn test_recursive_compact_list_of_lists() { let non_recursive_array = non_recursive.into_array(); let recursive_array = recursive.into_array(); assert_eq!( - non_recursive_array.scalar_at(0).unwrap(), - recursive_array.scalar_at(0).unwrap() + non_recursive_array + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + recursive_array + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() ); } diff --git a/vortex-array/src/arrays/list/vtable/mod.rs b/vortex-array/src/arrays/list/vtable/mod.rs index d9624f49560..38d1959219d 100644 --- a/vortex-array/src/arrays/list/vtable/mod.rs +++ b/vortex-array/src/arrays/list/vtable/mod.rs @@ -11,6 +11,7 @@ use vortex_error::vortex_bail; use vortex_error::vortex_ensure; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::ArrayEq; use crate::ArrayHash; @@ -21,6 +22,7 @@ use crate::IntoArray; use crate::Precision; use crate::array::Array; use crate::array::ArrayId; +use crate::array::ArrayParts; use crate::array::ArrayView; use crate::array::VTable; use crate::arrays::list::ListArrayExt; @@ -64,9 +66,9 @@ impl VTable for List { type OperationsVTable = Self; type ValidityVTable = Self; - fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.list"); + *ID } fn nbuffers(_array: ArrayView<'_, Self>) -> usize { @@ -147,7 +149,7 @@ impl VTable for List { _buffers: &[BufferHandle], children: &dyn ArrayChildren, _session: &VortexSession, - ) -> VortexResult> { + ) -> VortexResult> { let metadata = ListMetadata::decode(metadata)?; let validity = if children.len() == 2 { Validity::from(dtype.nullability()) @@ -175,7 +177,7 @@ impl VTable for List { let data = ListData::try_build(elements.clone(), offsets.clone(), validity.clone())?; let slots = ListData::make_slots(&elements, &offsets, &validity, len); - Ok(crate::array::ArrayParts::new(self.clone(), dtype.clone(), len, data).with_slots(slots)) + Ok(ArrayParts::new(self.clone(), dtype.clone(), len, data).with_slots(slots)) } fn slot_name(_array: ArrayView<'_, Self>, idx: usize) -> String { @@ -200,7 +202,3 @@ impl VTable for List { #[derive(Clone, Debug)] pub struct List; - -impl List { - pub const ID: ArrayId = ArrayId::new_ref("vortex.list"); -} diff --git a/vortex-array/src/arrays/list/vtable/operations.rs b/vortex-array/src/arrays/list/vtable/operations.rs index c9fc69c7bbe..02c686cd1f1 100644 --- a/vortex-array/src/arrays/list/vtable/operations.rs +++ b/vortex-array/src/arrays/list/vtable/operations.rs @@ -16,12 +16,12 @@ impl OperationsVTable for List { fn scalar_at( array: ArrayView<'_, List>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { // By the preconditions we know that the list scalar is not null. let elems = array.list_elements_at(index)?; let scalars: Vec = (0..elems.len()) - .map(|i| elems.scalar_at(i)) + .map(|i| elems.execute_scalar(i, ctx)) .collect::>()?; Ok(Scalar::list( diff --git a/vortex-array/src/arrays/listview/array.rs b/vortex-array/src/arrays/listview/array.rs index 72c376b2f44..e22809cf4c7 100644 --- a/vortex-array/src/arrays/listview/array.rs +++ b/vortex-array/src/arrays/listview/array.rs @@ -13,7 +13,10 @@ use vortex_error::vortex_ensure; use vortex_error::vortex_err; use crate::ArrayRef; -use crate::ToCanonical; +use crate::LEGACY_SESSION; +#[expect(deprecated)] +use crate::ToCanonical as _; +use crate::VortexSessionExecute; use crate::array::Array; use crate::array::ArrayParts; use crate::array::TypedArrayRef; @@ -247,7 +250,9 @@ impl ListViewData { // Skip host-only validation when offsets/sizes are not host-resident. if offsets.is_host() && sizes.is_host() { + #[expect(deprecated)] let offsets_primitive = offsets.to_primitive(); + #[expect(deprecated)] let sizes_primitive = sizes.to_primitive(); // Validate the `offsets` and `sizes` arrays. @@ -331,11 +336,10 @@ pub trait ListViewArrayExt: TypedArrayRef { } fn listview_validity(&self) -> Validity { - child_to_validity(&self.as_ref().slots()[VALIDITY_SLOT], self.nullability()) - } - - fn listview_validity_mask(&self) -> vortex_mask::Mask { - self.listview_validity().to_mask(self.as_ref().len()) + child_to_validity( + self.as_ref().slots()[VALIDITY_SLOT].as_ref(), + self.nullability(), + ) } fn offset_at(&self, index: usize) -> usize { @@ -349,8 +353,8 @@ pub trait ListViewArrayExt: TypedArrayRef { .map(|p| match_each_integer_ptype!(p.ptype(), |P| { p.as_slice::

()[index].as_() })) .unwrap_or_else(|| { self.offsets() - .scalar_at(index) - .vortex_expect("offsets must support scalar_at") + .execute_scalar(index, &mut LEGACY_SESSION.create_execution_ctx()) + .vortex_expect("offsets must support execute_scalar") .as_primitive() .as_::() .vortex_expect("offset must fit in usize") @@ -369,8 +373,8 @@ pub trait ListViewArrayExt: TypedArrayRef { .map(|p| match_each_integer_ptype!(p.ptype(), |P| { p.as_slice::

()[index].as_() })) .unwrap_or_else(|| { self.sizes() - .scalar_at(index) - .vortex_expect("sizes must support scalar_at") + .execute_scalar(index, &mut LEGACY_SESSION.create_execution_ctx()) + .vortex_expect("sizes must support execute_scalar") .as_primitive() .as_::() .vortex_expect("size must fit in usize") @@ -384,12 +388,11 @@ pub trait ListViewArrayExt: TypedArrayRef { } fn verify_is_zero_copy_to_list(&self) -> bool { - validate_zctl( - self.elements(), - self.offsets().to_primitive(), - self.sizes().to_primitive(), - ) - .is_ok() + #[expect(deprecated)] + let offsets_primitive = self.offsets().to_primitive(); + #[expect(deprecated)] + let sizes_primitive = self.sizes().to_primitive(); + validate_zctl(self.elements(), offsets_primitive, sizes_primitive).is_ok() } } impl> ListViewArrayExt for T {} @@ -458,12 +461,12 @@ impl Array { /// See [`ListViewData::with_zero_copy_to_list`]. pub unsafe fn with_zero_copy_to_list(self, is_zctl: bool) -> Self { if cfg!(debug_assertions) && is_zctl { - validate_zctl( - self.elements(), - self.offsets().to_primitive(), - self.sizes().to_primitive(), - ) - .vortex_expect("Failed to validate zero-copy to list flag"); + #[expect(deprecated)] + let offsets_primitive = self.offsets().to_primitive(); + #[expect(deprecated)] + let sizes_primitive = self.sizes().to_primitive(); + validate_zctl(self.elements(), offsets_primitive, sizes_primitive) + .vortex_expect("Failed to validate zero-copy to list flag"); } let dtype = self.dtype().clone(); let len = self.len(); @@ -557,7 +560,8 @@ fn validate_zctl( ) -> VortexResult<()> { // Offsets must be sorted (but not strictly sorted, zero-length lists are allowed), even // if there are null views. - if let Some(is_sorted) = offsets_primitive.statistics().compute_is_sorted() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + if let Some(is_sorted) = offsets_primitive.statistics().compute_is_sorted(&mut ctx) { vortex_ensure!(is_sorted, "offsets must be sorted"); } else { vortex_bail!("offsets must report is_sorted statistic"); diff --git a/vortex-array/src/arrays/listview/compute/cast.rs b/vortex-array/src/arrays/listview/compute/cast.rs index 77cd7ab7736..473f76b12ee 100644 --- a/vortex-array/src/arrays/listview/compute/cast.rs +++ b/vortex-array/src/arrays/listview/compute/cast.rs @@ -4,6 +4,7 @@ use vortex_error::VortexResult; use crate::ArrayRef; +use crate::ExecutionCtx; use crate::IntoArray; use crate::array::ArrayView; use crate::arrays::ListView; @@ -11,7 +12,27 @@ use crate::arrays::ListViewArray; use crate::arrays::listview::ListViewArrayExt; use crate::builtins::ArrayBuiltins; use crate::dtype::DType; +use crate::scalar_fn::fns::cast::CastKernel; use crate::scalar_fn::fns::cast::CastReduce; +use crate::validity::Validity; + +fn build_with_validity( + array: ArrayView<'_, ListView>, + new_elements: ArrayRef, + validity: Validity, +) -> ArrayRef { + // SAFETY: Since `cast` is length-preserving, all of the invariants remain the same. + unsafe { + ListViewArray::new_unchecked( + new_elements, + array.offsets().clone(), + array.sizes().clone(), + validity, + ) + .with_zero_copy_to_list(array.is_zero_copy_to_list()) + } + .into_array() +} impl CastReduce for ListView { fn cast(array: ArrayView<'_, ListView>, dtype: &DType) -> VortexResult> { @@ -19,25 +40,34 @@ impl CastReduce for ListView { let Some(target_element_type) = dtype.as_list_element_opt() else { return Ok(None); }; + let Some(validity) = array + .validity()? + .trivial_cast_nullability(dtype.nullability(), array.len())? + else { + return Ok(None); + }; // Cast the elements to the target element type. let new_elements = array.elements().cast((**target_element_type).clone())?; + Ok(Some(build_with_validity(array, new_elements, validity))) + } +} + +impl CastKernel for ListView { + fn cast( + array: ArrayView<'_, ListView>, + dtype: &DType, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + let Some(target_element_type) = dtype.as_list_element_opt() else { + return Ok(None); + }; + let validity = array .validity()? - .cast_nullability(dtype.nullability(), array.len())?; - - // SAFETY: Since `cast` is length-preserving, all of the invariants remain the same. - Ok(Some( - unsafe { - ListViewArray::new_unchecked( - new_elements, - array.offsets().clone(), - array.sizes().clone(), - validity, - ) - .with_zero_copy_to_list(array.is_zero_copy_to_list()) - } - .into_array(), - )) + .cast_nullability(dtype.nullability(), array.len(), ctx)?; + let new_elements = array.elements().cast((**target_element_type).clone())?; + + Ok(Some(build_with_validity(array, new_elements, validity))) } } diff --git a/vortex-array/src/arrays/listview/compute/mod.rs b/vortex-array/src/arrays/listview/compute/mod.rs index 9a43503c4b5..3ea82cafb33 100644 --- a/vortex-array/src/arrays/listview/compute/mod.rs +++ b/vortex-array/src/arrays/listview/compute/mod.rs @@ -6,3 +6,15 @@ mod mask; pub(crate) mod rules; mod slice; mod take; + +/// The threshold below which we rebuild the elements of a listview. +/// +/// We don't touch `elements` on the metadata-only path since reorganizing it can be expensive. +/// However, we also don't want to drag around a large amount of garbage data when the selection +/// is sparse. Below this fraction of list rows retained, the rebuild is worth it. +/// Rebuilding is needed when exporting the ListView's elements. +/// +// TODO(connor)[ListView]: Ideally, we would only rebuild after all `take`s and `filter` +// compute functions have run, at the "top" of the operator tree. However, we cannot do this +// right now, so we will just rebuild every time (similar to [`ListArray`]). +pub(crate) const REBUILD_DENSITY_THRESHOLD: f32 = 0.1; diff --git a/vortex-array/src/arrays/listview/compute/take.rs b/vortex-array/src/arrays/listview/compute/take.rs index 5cfadd40214..04e404a846e 100644 --- a/vortex-array/src/arrays/listview/compute/take.rs +++ b/vortex-array/src/arrays/listview/compute/take.rs @@ -4,11 +4,14 @@ use num_traits::Zero; use vortex_error::VortexResult; +use super::REBUILD_DENSITY_THRESHOLD; use crate::ArrayRef; +use crate::ExecutionCtx; use crate::IntoArray; use crate::array::ArrayView; use crate::arrays::ListView; use crate::arrays::ListViewArray; +use crate::arrays::dict::TakeExecute; use crate::arrays::dict::TakeReduce; use crate::arrays::listview::ListViewArrayExt; use crate::arrays::listview::ListViewRebuildMode; @@ -17,72 +20,79 @@ use crate::dtype::Nullability; use crate::match_each_integer_ptype; use crate::scalar::Scalar; -// TODO(connor)[ListView]: Make use of this threshold after we start migrating operators. -/// The threshold for triggering a rebuild of the [`ListViewArray`]. -/// -/// By default, we will not touch the underlying `elements` array of the [`ListViewArray`] since it -/// can be potentially expensive to reorganize the array based on what views we have into it. -/// -/// However, we also do not want to carry around a large amount of garbage data. Below this -/// threshold of the density of the selection mask, we will rebuild the [`ListViewArray`], removing -/// any garbage data. -#[allow(unused)] -const REBUILD_DENSITY_THRESHOLD: f64 = 0.1; - -/// [`ListViewArray`] take implementation. -/// -/// This implementation is deliberately simple and read-optimized. We just take the `offsets` and -/// `sizes` at the requested indices and reuse the original `elements` array. This works because -/// `ListView` (unlike `List`) allows non-contiguous and out-of-order lists. -/// -/// We don't slice the `elements` array because it would require computing min/max offsets and -/// adjusting all offsets accordingly, which is not really worth the small potential memory we would -/// be able to get back. -/// -/// The trade-off is that we may keep unreferenced elements in memory, but this is acceptable since -/// we're optimizing for read performance and the data isn't being copied. +/// Metadata-only take for [`ListViewArray`]. impl TakeReduce for ListView { fn take(array: ArrayView<'_, ListView>, indices: &ArrayRef) -> VortexResult> { - let elements = array.elements(); - let offsets = array.offsets(); - let sizes = array.sizes(); + // Approximate element density by the fraction of list rows retained. Assumes roughly + // uniform list sizes; good enough to decide whether dragging along the full `elements` + // buffer is worth avoiding a rebuild. + let kept_row_fraction = indices.len() as f32 / array.sizes().len() as f32; + if kept_row_fraction < REBUILD_DENSITY_THRESHOLD { + return Ok(None); + } - // Compute the new validity by combining the array's validity with the indices' validity. - let new_validity = array.validity()?.take(indices)?; + Ok(Some(apply_take(array, indices)?.into_array())) + } +} - // Take the offsets and sizes arrays at the requested indices. - // Take can reorder offsets, create gaps, and may introduce overlaps if the `indices` - // contain duplicates. - let nullable_new_offsets = offsets.take(indices.clone())?; - let nullable_new_sizes = sizes.take(indices.clone())?; +/// Execution-path take for [`ListViewArray`]. +/// +/// This does the same metadata-only take as [`TakeReduce`], but also rebuilds the array if the +/// resulting array will be less dense than `REBUILD_DENSITY_THRESHOLD`. +impl TakeExecute for ListView { + fn take( + array: ArrayView<'_, ListView>, + indices: &ArrayRef, + _ctx: &mut ExecutionCtx, + ) -> VortexResult> { + let kept_row_fraction = indices.len() as f32 / array.sizes().len() as f32; + let taken = apply_take(array, indices)?; - // Since `take` returns nullable arrays, we simply cast it back to non-nullable (filled with - // zeros to represent null lists). - let new_offsets = match_each_integer_ptype!(nullable_new_offsets.dtype().as_ptype(), |O| { - nullable_new_offsets - .fill_null(Scalar::primitive(O::zero(), Nullability::NonNullable))? - }); - let new_sizes = match_each_integer_ptype!(nullable_new_sizes.dtype().as_ptype(), |S| { - nullable_new_sizes.fill_null(Scalar::primitive(S::zero(), Nullability::NonNullable))? - }); - // SAFETY: Take operation maintains all `ListViewArray` invariants: - // - `new_offsets` and `new_sizes` are derived from existing valid child arrays. - // - `new_offsets` and `new_sizes` are non-nullable. - // - `new_offsets` and `new_sizes` have the same length (both taken with the same - // `indices`). - // - Validity correctly reflects the combination of array and indices validity. - let new_array = unsafe { - ListViewArray::new_unchecked(elements.clone(), new_offsets, new_sizes, new_validity) - }; + if kept_row_fraction < REBUILD_DENSITY_THRESHOLD { + // TODO(connor)[ListView]: Ideally, we would only rebuild after all `take`s and `filter` + // compute functions have run, at the "top" of the operator tree. However, we cannot do + // this right now, so we will just rebuild every time (similar to `ListArray`). + Ok(Some( + taken + .rebuild(ListViewRebuildMode::MakeZeroCopyToList)? + .into_array(), + )) + } else { + Ok(Some(taken.into_array())) + } + } +} - // TODO(connor)[ListView]: Ideally, we would only rebuild after all `take`s and `filter` - // compute functions have run, at the "top" of the operator tree. However, we cannot do this - // right now, so we will just rebuild every time (similar to `ListArray`). +/// Shared metadata-only take: take `offsets`, `sizes` and `validity` at `indices` while reusing +/// the original `elements` buffer as-is. +fn apply_take(array: ArrayView<'_, ListView>, indices: &ArrayRef) -> VortexResult { + let elements = array.elements(); + let offsets = array.offsets(); + let sizes = array.sizes(); - Ok(Some( - new_array - .rebuild(ListViewRebuildMode::MakeZeroCopyToList)? - .into_array(), - )) - } + // Combine the array's validity with the indices' validity. + let new_validity = array.validity()?.take(indices)?; + + // Take can reorder offsets, create gaps, and may introduce overlaps if `indices` contain + // duplicates. + let nullable_new_offsets = offsets.take(indices.clone())?; + let nullable_new_sizes = sizes.take(indices.clone())?; + + // `take` returns nullable arrays; cast back to non-nullable (filling with zeros to represent + // the null lists — the validity mask tracks nullness separately). + let new_offsets = match_each_integer_ptype!(nullable_new_offsets.dtype().as_ptype(), |O| { + nullable_new_offsets.fill_null(Scalar::primitive(O::zero(), Nullability::NonNullable))? + }); + let new_sizes = match_each_integer_ptype!(nullable_new_sizes.dtype().as_ptype(), |S| { + nullable_new_sizes.fill_null(Scalar::primitive(S::zero(), Nullability::NonNullable))? + }); + + // SAFETY: Take operation maintains all `ListViewArray` invariants: + // - `new_offsets` and `new_sizes` are derived from existing valid child arrays. + // - `new_offsets` and `new_sizes` are non-nullable. + // - `new_offsets` and `new_sizes` have the same length (both taken with the same `indices`). + // - Validity correctly reflects the combination of array and indices validity. + Ok(unsafe { + ListViewArray::new_unchecked(elements.clone(), new_offsets, new_sizes, new_validity) + }) } diff --git a/vortex-array/src/arrays/listview/conversion.rs b/vortex-array/src/arrays/listview/conversion.rs index 54571ecdf21..54ee8c2705c 100644 --- a/vortex-array/src/arrays/listview/conversion.rs +++ b/vortex-array/src/arrays/listview/conversion.rs @@ -8,7 +8,8 @@ use crate::ArrayRef; use crate::Canonical; use crate::ExecutionCtx; use crate::IntoArray; -use crate::ToCanonical; +#[expect(deprecated)] +use crate::ToCanonical as _; use crate::arrays::ExtensionArray; use crate::arrays::FixedSizeListArray; use crate::arrays::ListArray; @@ -149,6 +150,7 @@ unsafe fn build_list_offsets_from_list_view( // Create uninit range for direct memory access. let mut offsets_range = offsets_builder.uninit_range(len + 1); + #[expect(deprecated)] let offsets = list_view.offsets().to_primitive(); let offsets_slice = offsets.as_slice::(); debug_assert!(offsets_slice.is_sorted()); @@ -187,6 +189,7 @@ pub fn recursive_list_from_list_view(array: ArrayRef) -> VortexResult return Ok(array); } + #[expect(deprecated)] let canonical = array.to_canonical()?; Ok(match canonical { diff --git a/vortex-array/src/arrays/listview/rebuild.rs b/vortex-array/src/arrays/listview/rebuild.rs index 4bcfb8311f8..416d73bb92a 100644 --- a/vortex-array/src/arrays/listview/rebuild.rs +++ b/vortex-array/src/arrays/listview/rebuild.rs @@ -8,7 +8,8 @@ use vortex_error::VortexResult; use crate::IntoArray; use crate::LEGACY_SESSION; -use crate::ToCanonical; +#[expect(deprecated)] +use crate::ToCanonical as _; use crate::VortexSessionExecute; use crate::aggregate_fn::fns::min_max::min_max; use crate::arrays::ConstantArray; @@ -116,6 +117,7 @@ impl ListViewArray { fn naive_rebuild( &self, ) -> VortexResult { + #[expect(deprecated)] let sizes_canonical = self.sizes().to_primitive(); let total: u64 = sizes_canonical .as_slice::() @@ -149,8 +151,10 @@ impl ListViewArray { fn rebuild_with_take( &self, ) -> VortexResult { + #[expect(deprecated)] let offsets_canonical = self.offsets().to_primitive(); let offsets_slice = offsets_canonical.as_slice::(); + #[expect(deprecated)] let sizes_canonical = self.sizes().to_primitive(); let sizes_slice = sizes_canonical.as_slice::(); @@ -202,8 +206,10 @@ impl ListViewArray { .as_list_element_opt() .vortex_expect("somehow had a canonical list that was not a list"); + #[expect(deprecated)] let offsets_canonical = self.offsets().to_primitive(); let offsets_slice = offsets_canonical.as_slice::(); + #[expect(deprecated)] let sizes_canonical = self.sizes().to_primitive(); let sizes_slice = sizes_canonical.as_slice::(); @@ -217,6 +223,7 @@ impl ListViewArray { let mut new_sizes = BufferMut::::with_capacity(len); // Canonicalize the elements up front as we will be slicing the elements quite a lot. + #[expect(deprecated)] let elements_canonical = self .elements() .to_canonical() @@ -285,9 +292,13 @@ impl ListViewArray { // completely fine for us to use this as a lower-bounded start of the `elements`. self.offset_at(0) } else { - self.offsets().statistics().compute_min().vortex_expect( - "[ListViewArray::rebuild]: `offsets` must report min statistic that is a `usize`", - ) + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + self.offsets() + .statistics() + .compute_min(&mut ctx) + .vortex_expect( + "[ListViewArray::rebuild]: `offsets` must report min statistic that is a `usize`", + ) }; let end = if self.is_zero_copy_to_list() { @@ -368,14 +379,16 @@ impl ListViewArray { } #[cfg(test)] -#[allow(clippy::cast_possible_truncation)] mod tests { use vortex_buffer::BitBuffer; use vortex_error::VortexResult; use super::ListViewRebuildMode; use crate::IntoArray; - use crate::ToCanonical; + use crate::LEGACY_SESSION; + #[expect(deprecated)] + use crate::ToCanonical as _; + use crate::VortexSessionExecute; use crate::arrays::ListViewArray; use crate::arrays::PrimitiveArray; use crate::arrays::listview::ListViewArrayExt; @@ -408,12 +421,12 @@ mod tests { // Verify the data is correct assert_arrays_eq!( - flattened.list_elements_at(0).unwrap(), + flattened.list_elements_at(0)?, PrimitiveArray::from_iter([1i32, 2, 3]) ); assert_arrays_eq!( - flattened.list_elements_at(1).unwrap(), + flattened.list_elements_at(1)?, PrimitiveArray::from_iter([2i32, 3]) ); Ok(()) @@ -441,18 +454,18 @@ mod tests { // Verify nullability is preserved assert_eq!(flattened.dtype().nullability(), Nullability::Nullable); - assert!(flattened.validity()?.is_valid(0).unwrap()); - assert!(!flattened.validity()?.is_valid(1).unwrap()); - assert!(flattened.validity()?.is_valid(2).unwrap()); + assert!(flattened.validity()?.is_valid(0)?); + assert!(!flattened.validity()?.is_valid(1)?); + assert!(flattened.validity()?.is_valid(2)?); // Verify valid lists contain correct data assert_arrays_eq!( - flattened.list_elements_at(0).unwrap(), + flattened.list_elements_at(0)?, PrimitiveArray::from_iter([1i32, 2]) ); assert_arrays_eq!( - flattened.list_elements_at(2).unwrap(), + flattened.list_elements_at(2)?, PrimitiveArray::from_iter([3i32]) ); Ok(()) @@ -487,18 +500,22 @@ mod tests { // Verify the data is correct. assert_arrays_eq!( - trimmed.list_elements_at(0).unwrap(), + trimmed.list_elements_at(0)?, PrimitiveArray::from_iter([1i32, 2]) ); assert_arrays_eq!( - trimmed.list_elements_at(1).unwrap(), + trimmed.list_elements_at(1)?, PrimitiveArray::from_iter([3i32, 4]) ); // Note that element at index 2 (97) is preserved as a gap. + #[expect(deprecated)] let all_elements = trimmed.elements().to_primitive(); - assert_eq!(all_elements.scalar_at(2).unwrap(), 97i32.into()); + assert_eq!( + all_elements.execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx())?, + 97i32.into() + ); Ok(()) } @@ -538,19 +555,19 @@ mod tests { let exact = rebuilt.rebuild(ListViewRebuildMode::MakeExact)?; // Verify the result is still valid - assert!(exact.is_valid(0).unwrap()); - assert!(exact.is_valid(1).unwrap()); - assert!(!exact.is_valid(2).unwrap()); - assert!(!exact.is_valid(3).unwrap()); + assert!(exact.is_valid(0, &mut LEGACY_SESSION.create_execution_ctx())?); + assert!(exact.is_valid(1, &mut LEGACY_SESSION.create_execution_ctx())?); + assert!(!exact.is_valid(2, &mut LEGACY_SESSION.create_execution_ctx())?); + assert!(!exact.is_valid(3, &mut LEGACY_SESSION.create_execution_ctx())?); // Verify data is preserved assert_arrays_eq!( - exact.list_elements_at(0).unwrap(), + exact.list_elements_at(0)?, PrimitiveArray::from_iter([1i32, 2]) ); assert_arrays_eq!( - exact.list_elements_at(1).unwrap(), + exact.list_elements_at(1)?, PrimitiveArray::from_iter([3i32, 4]) ); Ok(()) @@ -572,7 +589,7 @@ mod tests { let listview = ListViewArray::new(elements, offsets, sizes, Validity::NonNullable); let trimmed = listview.rebuild(ListViewRebuildMode::TrimElements)?; assert_arrays_eq!( - trimmed.list_elements_at(1).unwrap(), + trimmed.list_elements_at(1)?, PrimitiveArray::from_iter([30i32, 40]) ); Ok(()) @@ -592,7 +609,7 @@ mod tests { let listview = ListViewArray::new(elements, offsets, sizes, Validity::NonNullable); let trimmed = listview.rebuild(ListViewRebuildMode::TrimElements)?; assert_arrays_eq!( - trimmed.list_elements_at(1).unwrap(), + trimmed.list_elements_at(1)?, PrimitiveArray::from_iter([30i32, 40]) ); Ok(()) diff --git a/vortex-array/src/arrays/listview/tests/basic.rs b/vortex-array/src/arrays/listview/tests/basic.rs index 5a1bf09d028..61664c2f608 100644 --- a/vortex-array/src/arrays/listview/tests/basic.rs +++ b/vortex-array/src/arrays/listview/tests/basic.rs @@ -54,7 +54,9 @@ fn test_basic_listview_comprehensive() { ); // Test scalar_at which returns entire lists as Scalar values. - let first_scalar = listview.scalar_at(0).unwrap(); + let first_scalar = listview + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert_eq!( first_scalar, Scalar::list( @@ -125,7 +127,7 @@ fn test_from_list_array() -> VortexResult<()> { let elements = buffer![1i32, 2, 3, 4, 5, 6, 7].into_array(); let validity = Validity::from_iter([true, false, true]); - let list_array = ListArray::try_new(elements, offsets, validity).unwrap(); + let list_array = ListArray::try_new(elements, offsets, validity)?; let mut ctx = LEGACY_SESSION.create_execution_ctx(); let list_view = list_view_from_list(list_array, &mut ctx)?; @@ -133,14 +135,14 @@ fn test_from_list_array() -> VortexResult<()> { // Check first list. assert_arrays_eq!( - list_view.list_elements_at(0).unwrap(), + list_view.list_elements_at(0)?, PrimitiveArray::from_iter([1i32, 2]) ); // Check validity is preserved. - assert!(list_view.is_valid(0).unwrap()); - assert!(list_view.is_invalid(1).unwrap()); - assert!(list_view.is_valid(2).unwrap()); + assert!(list_view.is_valid(0, &mut LEGACY_SESSION.create_execution_ctx())?); + assert!(list_view.is_invalid(1, &mut LEGACY_SESSION.create_execution_ctx())?); + assert!(list_view.is_valid(2, &mut LEGACY_SESSION.create_execution_ctx())?); // Check third list. assert_arrays_eq!( @@ -198,15 +200,27 @@ fn test_listview_with_constant_arrays(#[case] const_sizes: bool, #[case] const_o } else if const_offsets { // All lists start at offset 0, different sizes (overlapping). assert_eq!( - listview.list_elements_at(0).unwrap().scalar_at(0).unwrap(), + listview + .list_elements_at(0) + .unwrap() + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), 1i32.into() ); assert_eq!( - listview.list_elements_at(1).unwrap().scalar_at(0).unwrap(), + listview + .list_elements_at(1) + .unwrap() + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), 1i32.into() ); assert_eq!( - listview.list_elements_at(2).unwrap().scalar_at(0).unwrap(), + listview + .list_elements_at(2) + .unwrap() + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), 1i32.into() ); } diff --git a/vortex-array/src/arrays/listview/tests/filter.rs b/vortex-array/src/arrays/listview/tests/filter.rs index 40eef10d825..26a5190e517 100644 --- a/vortex-array/src/arrays/listview/tests/filter.rs +++ b/vortex-array/src/arrays/listview/tests/filter.rs @@ -11,7 +11,10 @@ use super::common::create_large_listview; use super::common::create_nullable_listview; use super::common::create_overlapping_listview; use crate::IntoArray; -use crate::ToCanonical; +use crate::LEGACY_SESSION; +#[expect(deprecated)] +use crate::ToCanonical as _; +use crate::VortexSessionExecute; use crate::arrays::ConstantArray; use crate::arrays::ListViewArray; use crate::arrays::PrimitiveArray; @@ -47,6 +50,7 @@ fn test_filter_preserves_unreferenced_elements() { // Filter to keep only 2 lists. let mask = Mask::from_iter([true, false, false, true, false]); let result = listview.filter(mask).unwrap(); + #[expect(deprecated)] let result_list = result.to_listview(); assert_eq!(result_list.len(), 2, "Wrong number of filtered lists"); @@ -78,6 +82,7 @@ fn test_filter_with_gaps() { // Filter to keep lists with gaps and overlaps. let mask = Mask::from_iter([false, true, true, true, false]); let result = listview.filter(mask).unwrap(); + #[expect(deprecated)] let result_list = result.to_listview(); assert_eq!(result_list.len(), 3, "Wrong filter result length"); @@ -121,6 +126,7 @@ fn test_filter_constant_arrays() { let mask1 = Mask::from_iter([true, false, true, false]); let result1 = const_offset_list.filter(mask1).unwrap(); + #[expect(deprecated)] let result1_list = result1.to_listview(); assert_eq!(result1_list.len(), 2); @@ -144,6 +150,7 @@ fn test_filter_constant_arrays() { let mask2 = Mask::from_iter([true, false, true]); let result2 = both_const_list.filter(mask2).unwrap(); + #[expect(deprecated)] let result2_list = result2.to_listview(); assert_eq!(result2_list.len(), 2); @@ -169,6 +176,7 @@ fn test_filter_extreme_offsets() { // Filter to keep only 2 lists, demonstrating we keep all 10000 elements. let mask = Mask::from_iter([false, true, false, false, true]); let result = listview.filter(mask).unwrap(); + #[expect(deprecated)] let result_list = result.to_listview(); assert_eq!(result_list.len(), 2); @@ -184,7 +192,7 @@ fn test_filter_extreme_offsets() { let list0 = result_list.list_elements_at(0).unwrap(); assert_eq!( list0 - .scalar_at(0) + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) .unwrap() .as_primitive() .as_::() @@ -193,7 +201,7 @@ fn test_filter_extreme_offsets() { ); assert_eq!( list0 - .scalar_at(1) + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) .unwrap() .as_primitive() .as_::() @@ -204,6 +212,7 @@ fn test_filter_extreme_offsets() { // Test sparse selection from large dataset. let sparse_mask = Mask::from_iter((0..5).map(|i| i == 0 || i == 4)); let sparse_result = listview.filter(sparse_mask).unwrap(); + #[expect(deprecated)] let sparse_list = sparse_result.to_listview(); assert_eq!(sparse_list.len(), 2); diff --git a/vortex-array/src/arrays/listview/tests/nested.rs b/vortex-array/src/arrays/listview/tests/nested.rs index e5ec85a690c..4aa3edd75d5 100644 --- a/vortex-array/src/arrays/listview/tests/nested.rs +++ b/vortex-array/src/arrays/listview/tests/nested.rs @@ -4,6 +4,8 @@ use vortex_buffer::buffer; use crate::IntoArray; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::arrays::ListView; use crate::arrays::ListViewArray; use crate::arrays::PrimitiveArray; @@ -72,7 +74,7 @@ fn test_listview_of_listview_with_overlapping() { // inner[0] should be [1, 2, 3]. assert_eq!( inner0 - .scalar_at(0) + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) .unwrap() .as_primitive() .as_::() @@ -81,7 +83,7 @@ fn test_listview_of_listview_with_overlapping() { ); assert_eq!( inner0 - .scalar_at(2) + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) .unwrap() .as_primitive() .as_::() @@ -92,7 +94,7 @@ fn test_listview_of_listview_with_overlapping() { // inner[1] should be [3, 4, 5] - shares element 3 with inner[0]. assert_eq!( inner1 - .scalar_at(0) + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) .unwrap() .as_primitive() .as_::() @@ -101,7 +103,7 @@ fn test_listview_of_listview_with_overlapping() { ); assert_eq!( inner1 - .scalar_at(1) + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) .unwrap() .as_primitive() .as_::() @@ -285,7 +287,7 @@ fn test_listview_zero_and_overlapping() { assert_eq!(inner1.len(), 3); // [1, 2, 3] assert_eq!( inner1 - .scalar_at(0) + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) .unwrap() .as_primitive() .as_::() @@ -304,7 +306,7 @@ fn test_listview_zero_and_overlapping() { assert_eq!(inner3.len(), 3); // [2, 3, 4] assert_eq!( inner3 - .scalar_at(0) + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) .unwrap() .as_primitive() .as_::() @@ -381,7 +383,12 @@ fn test_listview_of_struct_with_nulls() { assert_eq!(list1.len(), 3); // The middle element (struct[2]) should be null. - assert!(list1.scalar_at(1).unwrap().is_null()); + assert!( + list1 + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); // Test slicing preserves null handling. let sliced = listview.slice(1..3).unwrap(); diff --git a/vortex-array/src/arrays/listview/tests/nullability.rs b/vortex-array/src/arrays/listview/tests/nullability.rs index aa56291e233..ff3f9c44c6c 100644 --- a/vortex-array/src/arrays/listview/tests/nullability.rs +++ b/vortex-array/src/arrays/listview/tests/nullability.rs @@ -7,6 +7,8 @@ use rstest::rstest; use vortex_buffer::buffer; use crate::IntoArray; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::arrays::BoolArray; use crate::arrays::ListViewArray; use crate::arrays::PrimitiveArray; @@ -34,9 +36,21 @@ fn test_nullable_listview_comprehensive() { assert_eq!(listview.len(), 3); // Check validity. - assert!(listview.is_valid(0).unwrap()); - assert!(listview.is_invalid(1).unwrap()); - assert!(listview.is_valid(2).unwrap()); + assert!( + listview + .is_valid(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); + assert!( + listview + .is_invalid(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); + assert!( + listview + .is_valid(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); // Check dtype reflects nullability. assert!(matches!( @@ -45,7 +59,9 @@ fn test_nullable_listview_comprehensive() { )); // Test scalar_at with nulls. - let first = listview.scalar_at(0).unwrap(); + let first = listview + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(!first.is_null()); assert_eq!( first, @@ -56,10 +72,14 @@ fn test_nullable_listview_comprehensive() { ) ); - let second = listview.scalar_at(1).unwrap(); + let second = listview + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(second.is_null()); - let third = listview.scalar_at(2).unwrap(); + let third = listview + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(!third.is_null()); assert_eq!( third, @@ -73,8 +93,18 @@ fn test_nullable_listview_comprehensive() { // list_elements_at still returns data even for null lists. let null_list_data = listview.list_elements_at(1).unwrap(); assert_eq!(null_list_data.len(), 2); - assert_eq!(null_list_data.scalar_at(0).unwrap(), 3i32.into()); - assert_eq!(null_list_data.scalar_at(1).unwrap(), 4i32.into()); + assert_eq!( + null_list_data + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 3i32.into() + ); + assert_eq!( + null_list_data + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 4i32.into() + ); } // Parameterized tests for different null patterns. @@ -91,7 +121,12 @@ fn test_nullable_patterns(#[case] validity: Validity, #[case] expected_validity: let listview = unsafe { ListViewArray::new_unchecked(elements, offsets, sizes, validity) }; for (i, &expected) in expected_validity.iter().enumerate() { - assert_eq!(listview.is_valid(i).unwrap(), expected); + assert_eq!( + listview + .is_valid(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + expected + ); } } @@ -113,22 +148,72 @@ fn test_nullable_elements() { // First list: [Some(1), None]. let first_list = listview.list_elements_at(0).unwrap(); assert_eq!(first_list.len(), 2); - assert!(!first_list.scalar_at(0).unwrap().is_null()); - assert_eq!(first_list.scalar_at(0).unwrap(), 1i32.into()); - assert!(first_list.scalar_at(1).unwrap().is_null()); + assert!( + !first_list + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); + assert_eq!( + first_list + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 1i32.into() + ); + assert!( + first_list + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); // Second list: [Some(3), None]. let second_list = listview.list_elements_at(1).unwrap(); - assert!(!second_list.scalar_at(0).unwrap().is_null()); - assert_eq!(second_list.scalar_at(0).unwrap(), 3i32.into()); - assert!(second_list.scalar_at(1).unwrap().is_null()); + assert!( + !second_list + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); + assert_eq!( + second_list + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 3i32.into() + ); + assert!( + second_list + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); // Third list: [Some(5), Some(6)]. let third_list = listview.list_elements_at(2).unwrap(); - assert!(!third_list.scalar_at(0).unwrap().is_null()); - assert_eq!(third_list.scalar_at(0).unwrap(), 5i32.into()); - assert!(!third_list.scalar_at(1).unwrap().is_null()); - assert_eq!(third_list.scalar_at(1).unwrap(), 6i32.into()); + assert!( + !third_list + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); + assert_eq!( + third_list + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 5i32.into() + ); + assert!( + !third_list + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); + assert_eq!( + third_list + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 6i32.into() + ); // Check dtype of elements. assert!(matches!( diff --git a/vortex-array/src/arrays/listview/tests/operations.rs b/vortex-array/src/arrays/listview/tests/operations.rs index 09cf700bb98..235caf53caa 100644 --- a/vortex-array/src/arrays/listview/tests/operations.rs +++ b/vortex-array/src/arrays/listview/tests/operations.rs @@ -12,7 +12,8 @@ use super::common::create_large_listview; use super::common::create_nullable_listview; use crate::IntoArray; use crate::LEGACY_SESSION; -use crate::ToCanonical; +#[expect(deprecated)] +use crate::ToCanonical as _; use crate::VortexSessionExecute; use crate::aggregate_fn::fns::is_constant::is_constant; use crate::arrays::BoolArray; @@ -59,8 +60,13 @@ fn test_slice_comprehensive() { for i in 0..4 { // Compare the sliced elements assert_eq!( - full_list.array().scalar_at(i).unwrap(), - listview.scalar_at(i).unwrap(), + full_list + .array() + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + listview + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), "Mismatch at index {}", i ); @@ -148,8 +154,18 @@ fn test_slice_with_nulls() { let sliced_list = sliced.as_::(); assert_eq!(sliced_list.len(), 2); - assert!(sliced_list.array().is_invalid(0).unwrap()); // Original index 1 was null. - assert!(sliced_list.array().is_valid(1).unwrap()); // Original index 2 was valid. + assert!( + sliced_list + .array() + .is_invalid(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); // Original index 1 was null. + assert!( + sliced_list + .array() + .is_valid(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); // Original index 2 was valid. // Verify offsets and sizes are preserved. assert_eq!(sliced_list.offset_at(0), 2); @@ -236,6 +252,7 @@ fn test_cast_numeric_types(#[case] from_ptype: PType, #[case] to_ptype: PType) { let result = listview.cast(target_dtype.clone()).unwrap(); assert_eq!(result.dtype(), &target_dtype); + #[expect(deprecated)] let result_list = result.to_listview(); assert!( result_list.len() == 3 || result_list.len() == 2, @@ -272,9 +289,18 @@ fn test_cast_with_nulls() { let result = listview.cast(target_dtype.clone()).unwrap(); assert_eq!(result.dtype(), &target_dtype); + #[expect(deprecated)] let result_list = result.to_listview(); - assert!(result_list.is_valid(0).unwrap()); - assert!(result_list.is_invalid(1).unwrap()); + assert!( + result_list + .is_valid(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); + assert!( + result_list + .is_invalid(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); } #[rstest] @@ -314,6 +340,7 @@ fn test_cast_special_patterns(#[case] expected_sizes: Vec, #[case] list_c }; let result = listview.cast(target_dtype).unwrap(); + #[expect(deprecated)] let result_list = result.to_listview(); assert_eq!(result_list.len(), list_count); @@ -346,6 +373,7 @@ fn test_cast_large_dataset() { ); let result = listview.cast(target_dtype).unwrap(); + #[expect(deprecated)] let result_list = result.to_listview(); assert_eq!(result_list.len(), 20); @@ -514,13 +542,30 @@ fn test_mask_preserves_structure() { let result = listview.mask((!&selection).into_array()).unwrap(); assert_eq!(result.len(), 4); // Length is preserved. + #[expect(deprecated)] let result_list = result.to_listview(); // Check validity: true in selection means null. - assert!(!result_list.is_valid(0).unwrap()); // Masked. - assert!(result_list.is_valid(1).unwrap()); // Not masked. - assert!(!result_list.is_valid(2).unwrap()); // Masked. - assert!(!result_list.is_valid(3).unwrap()); // Masked. + assert!( + !result_list + .is_valid(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); // Masked. + assert!( + result_list + .is_valid(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); // Not masked. + assert!( + !result_list + .is_valid(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); // Masked. + assert!( + !result_list + .is_valid(3, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); // Masked. // Offsets and sizes are preserved. assert_eq!(result_list.offset_at(0), 0); @@ -551,12 +596,25 @@ fn test_mask_with_existing_nulls() { // Mask additional elements. let selection = Mask::from_iter([false, true, true]); let result = listview.mask((!&selection).into_array()).unwrap(); + #[expect(deprecated)] let result_list = result.to_listview(); // Check combined validity: - assert!(result_list.is_valid(0).unwrap()); // Was valid, mask is false -> valid. - assert!(!result_list.is_valid(1).unwrap()); // Was invalid, mask is true -> invalid. - assert!(!result_list.is_valid(2).unwrap()); // Was valid, mask is true -> invalid. + assert!( + result_list + .is_valid(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); // Was valid, mask is false -> valid. + assert!( + !result_list + .is_valid(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); // Was invalid, mask is true -> invalid. + assert!( + !result_list + .is_valid(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); // Was valid, mask is true -> invalid. } #[test] @@ -571,12 +629,25 @@ fn test_mask_with_gaps() { let selection = Mask::from_iter([true, false, false]); let result = listview.mask((!&selection).into_array()).unwrap(); + #[expect(deprecated)] let result_list = result.to_listview(); assert_eq!(result_list.len(), 3); - assert!(!result_list.is_valid(0).unwrap()); // Masked - assert!(result_list.is_valid(1).unwrap()); // Not masked - assert!(result_list.is_valid(2).unwrap()); // Not masked + assert!( + !result_list + .is_valid(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); // Masked + assert!( + result_list + .is_valid(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); // Not masked + assert!( + result_list + .is_valid(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); // Not masked // Offsets and sizes still preserved assert_eq!(result_list.offset_at(1), 4); @@ -603,12 +674,25 @@ fn test_mask_constant_arrays() { let selection = Mask::from_iter([false, true, false]); let result = const_list.mask((!&selection).into_array()).unwrap(); + #[expect(deprecated)] let result_list = result.to_listview(); assert_eq!(result_list.len(), 3); - assert!(result_list.is_valid(0).unwrap()); - assert!(!result_list.is_valid(1).unwrap()); // Masked - assert!(result_list.is_valid(2).unwrap()); + assert!( + result_list + .is_valid(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); + assert!( + !result_list + .is_valid(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); // Masked + assert!( + result_list + .is_valid(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); // All offsets and sizes remain constant assert_eq!(result_list.offset_at(0), 1); diff --git a/vortex-array/src/arrays/listview/tests/take.rs b/vortex-array/src/arrays/listview/tests/take.rs index 042d93dfe59..a162d61b217 100644 --- a/vortex-array/src/arrays/listview/tests/take.rs +++ b/vortex-array/src/arrays/listview/tests/take.rs @@ -10,7 +10,10 @@ use super::common::create_large_listview; use super::common::create_nullable_listview; use super::common::create_overlapping_listview; use crate::IntoArray; -use crate::ToCanonical; +use crate::LEGACY_SESSION; +#[expect(deprecated)] +use crate::ToCanonical as _; +use crate::VortexSessionExecute; use crate::arrays::ConstantArray; use crate::arrays::ListViewArray; use crate::arrays::PrimitiveArray; @@ -46,6 +49,7 @@ fn test_take_preserves_unreferenced_elements() { // Take only 2 lists. let indices = buffer![1u32, 3].into_array(); let result = listview.take(indices).unwrap(); + #[expect(deprecated)] let result_list = result.to_listview(); assert_eq!(result_list.len(), 2); @@ -74,6 +78,7 @@ fn test_take_with_gaps() { let indices = buffer![1u32, 3, 4, 2].into_array(); let result = listview.take(indices).unwrap(); + #[expect(deprecated)] let result_list = result.to_listview(); // Verify the entire elements array is preserved including gaps. @@ -109,6 +114,7 @@ fn test_take_constant_arrays() { let indices = buffer![3u32, 0, 2].into_array(); let result = const_offset_list.take(indices).unwrap(); + #[expect(deprecated)] let result_list = result.to_listview(); assert_eq!(result_list.len(), 3); @@ -133,6 +139,7 @@ fn test_take_constant_arrays() { let indices2 = buffer![2u32, 0].into_array(); let result2 = both_const_list.take(indices2).unwrap(); + #[expect(deprecated)] let result2_list = result2.to_listview(); assert_eq!(result2_list.len(), 2); @@ -158,6 +165,7 @@ fn test_take_extreme_offsets() { // Take only 2 lists, demonstrating we keep all 10000 elements. let indices = buffer![1u32, 4].into_array(); let result = listview.take(indices).unwrap(); + #[expect(deprecated)] let result_list = result.to_listview(); assert_eq!(result_list.len(), 2); @@ -173,7 +181,7 @@ fn test_take_extreme_offsets() { let list0 = result_list.list_elements_at(0).unwrap(); assert_eq!( list0 - .scalar_at(0) + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) .unwrap() .as_primitive() .as_::() @@ -182,7 +190,7 @@ fn test_take_extreme_offsets() { ); assert_eq!( list0 - .scalar_at(1) + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) .unwrap() .as_primitive() .as_::() diff --git a/vortex-array/src/arrays/listview/vtable/kernel.rs b/vortex-array/src/arrays/listview/vtable/kernel.rs new file mode 100644 index 00000000000..f6ceca284bf --- /dev/null +++ b/vortex-array/src/arrays/listview/vtable/kernel.rs @@ -0,0 +1,9 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +use crate::arrays::ListView; +use crate::kernel::ParentKernelSet; +use crate::scalar_fn::fns::cast::CastExecuteAdaptor; + +pub(super) const PARENT_KERNELS: ParentKernelSet = + ParentKernelSet::new(&[ParentKernelSet::lift(&CastExecuteAdaptor(ListView))]); diff --git a/vortex-array/src/arrays/listview/vtable/mod.rs b/vortex-array/src/arrays/listview/vtable/mod.rs index 6338c6504c1..f51fde706a4 100644 --- a/vortex-array/src/arrays/listview/vtable/mod.rs +++ b/vortex-array/src/arrays/listview/vtable/mod.rs @@ -12,6 +12,7 @@ use vortex_error::vortex_bail; use vortex_error::vortex_ensure; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::ArrayEq; use crate::ArrayHash; @@ -28,12 +29,14 @@ use crate::arrays::listview::ListViewData; use crate::arrays::listview::array::NUM_SLOTS; use crate::arrays::listview::array::SLOT_NAMES; use crate::arrays::listview::compute::rules::PARENT_RULES; +use crate::arrays::listview::vtable::kernel::PARENT_KERNELS; use crate::buffer::BufferHandle; use crate::dtype::DType; use crate::dtype::Nullability; use crate::dtype::PType; use crate::serde::ArrayChildren; use crate::validity::Validity; +mod kernel; mod operations; mod validity; /// A [`ListView`]-encoded Vortex array. @@ -42,10 +45,6 @@ pub type ListViewArray = Array; #[derive(Clone, Debug)] pub struct ListView; -impl ListView { - pub const ID: ArrayId = ArrayId::new_ref("vortex.listview"); -} - #[derive(Clone, prost::Message)] pub struct ListViewMetadata { #[prost(uint64, tag = "1")] @@ -73,9 +72,9 @@ impl VTable for ListView { type OperationsVTable = Self; type ValidityVTable = Self; - fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.listview"); + *ID } fn nbuffers(_array: ArrayView<'_, Self>) -> usize { @@ -217,4 +216,13 @@ impl VTable for ListView { ) -> VortexResult> { PARENT_RULES.evaluate(array, parent, child_idx) } + + fn execute_parent( + array: ArrayView<'_, Self>, + parent: &ArrayRef, + child_idx: usize, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + PARENT_KERNELS.execute(array, parent, child_idx, ctx) + } } diff --git a/vortex-array/src/arrays/listview/vtable/operations.rs b/vortex-array/src/arrays/listview/vtable/operations.rs index c899083027a..f0cb9539cc3 100644 --- a/vortex-array/src/arrays/listview/vtable/operations.rs +++ b/vortex-array/src/arrays/listview/vtable/operations.rs @@ -16,12 +16,12 @@ impl OperationsVTable for ListView { fn scalar_at( array: ArrayView<'_, ListView>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { // By the preconditions we know that the list scalar is not null. let list = array.list_elements_at(index)?; let children: Vec = (0..list.len()) - .map(|i| list.scalar_at(i)) + .map(|i| list.execute_scalar(i, ctx)) .collect::>()?; Ok(Scalar::list( diff --git a/vortex-array/src/arrays/masked/array.rs b/vortex-array/src/arrays/masked/array.rs index 4f6d53e5b35..ce828020111 100644 --- a/vortex-array/src/arrays/masked/array.rs +++ b/vortex-array/src/arrays/masked/array.rs @@ -4,25 +4,28 @@ use std::fmt::Display; use std::fmt::Formatter; -use vortex_error::VortexExpect; use vortex_error::VortexResult; use vortex_error::vortex_bail; use crate::ArrayRef; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::array::Array; use crate::array::ArrayParts; use crate::array::TypedArrayRef; use crate::array::child_to_validity; use crate::array::validity_to_child; +use crate::array_slots; use crate::arrays::Masked; use crate::validity::Validity; -/// The underlying child array being masked. -pub(super) const CHILD_SLOT: usize = 0; -/// The validity bitmap defining which elements are non-null. -pub(super) const VALIDITY_SLOT: usize = 1; -pub(super) const NUM_SLOTS: usize = 2; -pub(super) const SLOT_NAMES: [&str; NUM_SLOTS] = ["child", "validity"]; +#[array_slots(Masked)] +pub struct MaskedSlots { + /// The underlying child array being masked. + pub child: ArrayRef, + /// The validity bitmap defining which elements are non-null. + pub validity: Option, +} #[derive(Clone, Debug)] pub struct MaskedData; @@ -33,27 +36,13 @@ impl Display for MaskedData { } } -pub trait MaskedArrayExt: TypedArrayRef { - fn child(&self) -> &ArrayRef { - self.as_ref().slots()[CHILD_SLOT] - .as_ref() - .vortex_expect("validated masked child slot") - } - - fn validity_child(&self) -> Option<&ArrayRef> { - self.as_ref().slots()[VALIDITY_SLOT].as_ref() - } - +pub trait MaskedArrayExt: TypedArrayRef + MaskedArraySlotsExt { fn masked_validity(&self) -> Validity { child_to_validity( - &self.as_ref().slots()[VALIDITY_SLOT], + self.as_ref().slots()[MaskedSlots::VALIDITY].as_ref(), self.as_ref().dtype().nullability(), ) } - - fn masked_validity_mask(&self) -> vortex_mask::Mask { - self.masked_validity().to_mask(self.as_ref().len()) - } } impl> MaskedArrayExt for T {} @@ -89,7 +78,11 @@ impl Array { let dtype = child.dtype().as_nullable(); let len = child.len(); let validity_slot = validity_to_child(&validity, len); - let data = MaskedData::try_new(len, child.all_valid()?, validity)?; + let data = MaskedData::try_new( + len, + child.all_valid(&mut LEGACY_SESSION.create_execution_ctx())?, + validity, + )?; Ok(unsafe { Array::from_parts_unchecked( ArrayParts::new(Masked, dtype, len, data) diff --git a/vortex-array/src/arrays/masked/compute/filter.rs b/vortex-array/src/arrays/masked/compute/filter.rs index fb3969b4bb8..9ea6b168105 100644 --- a/vortex-array/src/arrays/masked/compute/filter.rs +++ b/vortex-array/src/arrays/masked/compute/filter.rs @@ -10,7 +10,7 @@ use crate::array::ArrayView; use crate::arrays::Masked; use crate::arrays::MaskedArray; use crate::arrays::filter::FilterReduce; -use crate::arrays::masked::MaskedArrayExt; +use crate::arrays::masked::MaskedArraySlotsExt; impl FilterReduce for Masked { fn filter(array: ArrayView<'_, Masked>, mask: &Mask) -> VortexResult> { diff --git a/vortex-array/src/arrays/masked/compute/mask.rs b/vortex-array/src/arrays/masked/compute/mask.rs index d8590a68d34..514d0aeaafc 100644 --- a/vortex-array/src/arrays/masked/compute/mask.rs +++ b/vortex-array/src/arrays/masked/compute/mask.rs @@ -6,7 +6,7 @@ use vortex_error::VortexResult; use crate::ArrayRef; use crate::array::ArrayView; use crate::arrays::Masked; -use crate::arrays::masked::MaskedArrayExt; +use crate::arrays::masked::MaskedArraySlotsExt; use crate::arrays::scalar_fn::ScalarFnFactoryExt; use crate::scalar_fn::EmptyOptions; use crate::scalar_fn::fns::mask::Mask as MaskExpr; diff --git a/vortex-array/src/arrays/masked/compute/slice.rs b/vortex-array/src/arrays/masked/compute/slice.rs index fe295bd6826..5a51ca9f681 100644 --- a/vortex-array/src/arrays/masked/compute/slice.rs +++ b/vortex-array/src/arrays/masked/compute/slice.rs @@ -10,7 +10,7 @@ use crate::IntoArray; use crate::array::ArrayView; use crate::arrays::Masked; use crate::arrays::MaskedArray; -use crate::arrays::masked::MaskedArrayExt; +use crate::arrays::masked::MaskedArraySlotsExt; use crate::arrays::slice::SliceReduce; impl SliceReduce for Masked { diff --git a/vortex-array/src/arrays/masked/compute/take.rs b/vortex-array/src/arrays/masked/compute/take.rs index c02a88a08fe..1eff686b4d9 100644 --- a/vortex-array/src/arrays/masked/compute/take.rs +++ b/vortex-array/src/arrays/masked/compute/take.rs @@ -9,13 +9,18 @@ use crate::array::ArrayView; use crate::arrays::Masked; use crate::arrays::MaskedArray; use crate::arrays::dict::TakeReduce; -use crate::arrays::masked::MaskedArrayExt; +use crate::arrays::masked::MaskedArraySlotsExt; use crate::builtins::ArrayBuiltins; use crate::scalar::Scalar; +use crate::validity::Validity; impl TakeReduce for Masked { fn take(array: ArrayView<'_, Masked>, indices: &ArrayRef) -> VortexResult> { - let taken_child = if !indices.all_valid()? { + let indices_all_valid = matches!( + indices.validity()?, + Validity::NonNullable | Validity::AllValid + ); + let taken_child = if !indices_all_valid { // This is safe because we'll mask out these positions in the validity. let fill_scalar = Scalar::zero_value(indices.dtype()); let filled_take_indices = indices.clone().fill_null(fill_scalar)?; diff --git a/vortex-array/src/arrays/masked/execute.rs b/vortex-array/src/arrays/masked/execute.rs index 4bc6449ead3..354b246405c 100644 --- a/vortex-array/src/arrays/masked/execute.rs +++ b/vortex-array/src/arrays/masked/execute.rs @@ -3,11 +3,9 @@ //! Execution logic for MaskedArray - applies a validity mask to canonical arrays. -use std::ops::BitAnd; use std::sync::Arc; use vortex_error::VortexResult; -use vortex_mask::Mask; use crate::Canonical; use crate::IntoArray; @@ -17,7 +15,6 @@ use crate::arrays::ExtensionArray; use crate::arrays::FixedSizeListArray; use crate::arrays::ListViewArray; use crate::arrays::MaskedArray; -use crate::arrays::NullArray; use crate::arrays::PrimitiveArray; use crate::arrays::StructArray; use crate::arrays::VarBinViewArray; @@ -28,9 +25,7 @@ use crate::arrays::fixed_size_list::FixedSizeListArrayExt; use crate::arrays::listview::ListViewArrayExt; use crate::arrays::struct_::StructArrayExt; use crate::arrays::variant::VariantArrayExt; -use crate::dtype::Nullability; use crate::executor::ExecutionCtx; -use crate::match_each_decimal_value_type; use crate::validity::Validity; /// TODO: replace usage of compute fn. @@ -40,64 +35,36 @@ use crate::validity::Validity; /// validity with the provided mask, marking additional positions as invalid. pub fn mask_validity_canonical( canonical: Canonical, - validity_mask: &Mask, + validity: Validity, ctx: &mut ExecutionCtx, ) -> VortexResult { Ok(match canonical { - Canonical::Null(a) => Canonical::Null(mask_validity_null(a, validity_mask)), - Canonical::Bool(a) => Canonical::Bool(mask_validity_bool(a, validity_mask, ctx)?), - Canonical::Primitive(a) => { - Canonical::Primitive(mask_validity_primitive(a, validity_mask, ctx)?) - } - Canonical::Decimal(a) => Canonical::Decimal(mask_validity_decimal(a, validity_mask, ctx)?), - Canonical::VarBinView(a) => { - Canonical::VarBinView(mask_validity_varbinview(a, validity_mask, ctx)?) - } - Canonical::List(a) => Canonical::List(mask_validity_listview(a, validity_mask, ctx)?), + n @ Canonical::Null(_) => n, + Canonical::Bool(a) => Canonical::Bool(mask_validity_bool(a, validity)?), + Canonical::Primitive(a) => Canonical::Primitive(mask_validity_primitive(a, validity)?), + Canonical::Decimal(a) => Canonical::Decimal(mask_validity_decimal(a, validity)?), + Canonical::VarBinView(a) => Canonical::VarBinView(mask_validity_varbinview(a, validity)?), + Canonical::List(a) => Canonical::List(mask_validity_listview(a, validity)?), Canonical::FixedSizeList(a) => { - Canonical::FixedSizeList(mask_validity_fixed_size_list(a, validity_mask, ctx)?) + Canonical::FixedSizeList(mask_validity_fixed_size_list(a, validity)?) } - Canonical::Struct(a) => Canonical::Struct(mask_validity_struct(a, validity_mask, ctx)?), - Canonical::Extension(a) => { - Canonical::Extension(mask_validity_extension(a, validity_mask, ctx)?) - } - Canonical::Variant(a) => Canonical::Variant(mask_validity_variant(a, validity_mask, ctx)?), + Canonical::Struct(a) => Canonical::Struct(mask_validity_struct(a, validity)?), + Canonical::Extension(a) => Canonical::Extension(mask_validity_extension(a, validity, ctx)?), + Canonical::Variant(a) => Canonical::Variant(mask_validity_variant(a, validity)?), }) } -fn combine_validity( - validity: &Validity, - mask: &Mask, - len: usize, - ctx: &mut ExecutionCtx, -) -> VortexResult { - let current_mask = validity.execute_mask(len, ctx)?; - let combined = current_mask.bitand(mask); - Ok(Validity::from_mask(combined, Nullability::Nullable)) -} - -fn mask_validity_null(array: NullArray, _mask: &Mask) -> NullArray { - array -} - -fn mask_validity_bool( - array: BoolArray, - mask: &Mask, - ctx: &mut ExecutionCtx, -) -> VortexResult { - let len = array.len(); - let new_validity = combine_validity(&array.validity()?, mask, len, ctx)?; +fn mask_validity_bool(array: BoolArray, mask: Validity) -> VortexResult { + let new_validity = Validity::and(array.validity()?, mask)?; Ok(BoolArray::new(array.to_bit_buffer(), new_validity)) } fn mask_validity_primitive( array: PrimitiveArray, - mask: &Mask, - ctx: &mut ExecutionCtx, + validity: Validity, ) -> VortexResult { - let len = array.len(); let ptype = array.ptype(); - let new_validity = combine_validity(&array.validity()?, mask, len, ctx)?; + let new_validity = Validity::and(array.validity()?, validity)?; // SAFETY: validity has same length as values Ok(unsafe { PrimitiveArray::new_unchecked_from_handle( @@ -108,31 +75,26 @@ fn mask_validity_primitive( }) } -fn mask_validity_decimal( - array: DecimalArray, - mask: &Mask, - ctx: &mut ExecutionCtx, -) -> VortexResult { - let len = array.len(); - let dec_dtype = array.decimal_dtype(); - let values_type = array.values_type(); - let new_validity = combine_validity(&array.validity()?, mask, len, ctx)?; +fn mask_validity_decimal(array: DecimalArray, validity: Validity) -> VortexResult { + let new_validity = Validity::and(array.validity()?, validity)?; // SAFETY: We're only changing validity, not the data structure - Ok(match_each_decimal_value_type!(values_type, |T| { - let buffer = array.buffer::(); - unsafe { DecimalArray::new_unchecked(buffer, dec_dtype, new_validity) } - })) + Ok(unsafe { + DecimalArray::new_unchecked_handle( + array.buffer_handle().clone(), + array.values_type(), + array.decimal_dtype(), + new_validity, + ) + }) } /// Mask validity for VarBinViewArray. fn mask_validity_varbinview( array: VarBinViewArray, - mask: &Mask, - ctx: &mut ExecutionCtx, + validity: Validity, ) -> VortexResult { - let len = array.len(); let dtype = array.dtype().as_nullable(); - let new_validity = combine_validity(&array.validity()?, mask, len, ctx)?; + let new_validity = Validity::and(array.validity()?, validity)?; // SAFETY: We're only changing validity, not the data structure Ok(unsafe { VarBinViewArray::new_handle_unchecked( @@ -144,13 +106,8 @@ fn mask_validity_varbinview( }) } -fn mask_validity_listview( - array: ListViewArray, - mask: &Mask, - ctx: &mut ExecutionCtx, -) -> VortexResult { - let len = array.len(); - let new_validity = combine_validity(&array.validity()?, mask, len, ctx)?; +fn mask_validity_listview(array: ListViewArray, validity: Validity) -> VortexResult { + let new_validity = Validity::and(array.validity()?, validity)?; // SAFETY: We're only changing validity, not the data structure Ok(unsafe { ListViewArray::new_unchecked( @@ -164,25 +121,20 @@ fn mask_validity_listview( fn mask_validity_fixed_size_list( array: FixedSizeListArray, - mask: &Mask, - ctx: &mut ExecutionCtx, + validity: Validity, ) -> VortexResult { let len = array.len(); let list_size = array.list_size(); - let new_validity = combine_validity(&array.validity()?, mask, len, ctx)?; + let new_validity = Validity::and(array.validity()?, validity)?; // SAFETY: We're only changing validity, not the data structure Ok(unsafe { FixedSizeListArray::new_unchecked(array.elements().clone(), list_size, new_validity, len) }) } -fn mask_validity_struct( - array: StructArray, - mask: &Mask, - ctx: &mut ExecutionCtx, -) -> VortexResult { +fn mask_validity_struct(array: StructArray, validity: Validity) -> VortexResult { let len = array.len(); - let new_validity = combine_validity(&array.validity()?, mask, len, ctx)?; + let new_validity = Validity::and(array.validity()?, validity)?; let fields = array.unmasked_fields(); let struct_fields = array.struct_fields(); // SAFETY: We're only changing validity, not the data structure @@ -191,12 +143,12 @@ fn mask_validity_struct( fn mask_validity_extension( array: ExtensionArray, - mask: &Mask, + validity: Validity, ctx: &mut ExecutionCtx, ) -> VortexResult { // For extension arrays, we need to mask the underlying storage let storage = array.storage_array().clone().execute::(ctx)?; - let masked_storage = mask_validity_canonical(storage, mask, ctx)?; + let masked_storage = mask_validity_canonical(storage, validity, ctx)?; let masked_storage = masked_storage.into_array(); Ok(ExtensionArray::new( array @@ -206,11 +158,7 @@ fn mask_validity_extension( )) } -fn mask_validity_variant( - array: VariantArray, - mask: &Mask, - ctx: &mut ExecutionCtx, -) -> VortexResult { +fn mask_validity_variant(array: VariantArray, validity: Validity) -> VortexResult { let child = array.child().clone(); let len = child.len(); let child_validity = child.validity()?; @@ -218,8 +166,7 @@ fn mask_validity_variant( match child_validity { Validity::NonNullable | Validity::AllValid => { // Child has no nulls — wrap in MaskedArray to apply the mask. - let new_validity = Validity::from_mask(mask.clone(), Nullability::Nullable); - let masked_child = MaskedArray::try_new(child, new_validity)?; + let masked_child = MaskedArray::try_new(child, validity)?; Ok(VariantArray::new(masked_child.into_array())) } Validity::AllInvalid => { @@ -229,7 +176,7 @@ fn mask_validity_variant( Validity::Array(_) => { // Child has an array-backed validity stored as its first child. // Combine with the mask and replace that child via with_children. - let combined = combine_validity(&child_validity, mask, len, ctx)?; + let combined = Validity::and(child_validity, validity)?; let new_child = child.with_slot(0, combined.to_array(len))?; Ok(VariantArray::new(new_child)) } diff --git a/vortex-array/src/arrays/masked/mod.rs b/vortex-array/src/arrays/masked/mod.rs index ac437d30321..7e23653c1c4 100644 --- a/vortex-array/src/arrays/masked/mod.rs +++ b/vortex-array/src/arrays/masked/mod.rs @@ -3,7 +3,9 @@ mod array; pub use array::MaskedArrayExt; +pub use array::MaskedArraySlotsExt; pub use array::MaskedData; +pub use array::MaskedSlots; pub use vtable::MaskedArray; pub(crate) mod compute; diff --git a/vortex-array/src/arrays/masked/tests.rs b/vortex-array/src/arrays/masked/tests.rs index 35d533d4f23..2721ba9b519 100644 --- a/vortex-array/src/arrays/masked/tests.rs +++ b/vortex-array/src/arrays/masked/tests.rs @@ -6,8 +6,10 @@ use vortex_error::VortexExpect; use vortex_error::VortexResult; use super::*; +use crate::Canonical; use crate::IntoArray; use crate::LEGACY_SESSION; +#[expect(deprecated)] use crate::ToCanonical as _; use crate::VortexSessionExecute; use crate::arrays::PrimitiveArray; @@ -44,9 +46,12 @@ fn test_dtype_nullability_with_nullable_child() { fn test_canonical_dtype_matches_array_dtype() -> VortexResult<()> { // The canonical form should have the same nullability as the array's dtype. let child = PrimitiveArray::from_iter([1i32, 2, 3]).into_array(); - let array = MaskedArray::try_new(child, Validity::AllValid).unwrap(); + let array = MaskedArray::try_new(child, Validity::AllValid)?; - let canonical = array.to_canonical()?; + let canonical = array + .clone() + .into_array() + .execute::(&mut LEGACY_SESSION.create_execution_ctx())?; assert_eq!(canonical.dtype(), array.dtype()); Ok(()) } @@ -58,15 +63,34 @@ fn test_masked_child_with_validity() { let array = MaskedArray::try_new(child, Validity::from_iter([true, false, true, false, true])).unwrap(); + #[expect(deprecated)] let prim = array.as_array().to_primitive(); // Positions where validity is false should be null in masked_child. - assert_eq!(prim.valid_count().unwrap(), 3); - assert!(prim.is_valid(0).unwrap()); - assert!(!prim.is_valid(1).unwrap()); - assert!(prim.is_valid(2).unwrap()); - assert!(!prim.is_valid(3).unwrap()); - assert!(prim.is_valid(4).unwrap()); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + assert_eq!(prim.valid_count(&mut ctx).unwrap(), 3); + assert!( + prim.is_valid(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); + assert!( + !prim + .is_valid(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); + assert!( + prim.is_valid(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); + assert!( + !prim + .is_valid(3, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); + assert!( + prim.is_valid(4, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); } #[test] @@ -76,7 +100,12 @@ fn test_masked_child_all_valid() { let array = MaskedArray::try_new(child, Validity::AllValid).unwrap(); assert_eq!(array.len(), 3); - assert_eq!(array.valid_count().unwrap(), 3); + assert_eq!( + array + .valid_count(&mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 3 + ); assert_arrays_eq!( PrimitiveArray::from_option_iter([10i32, 20, 30].map(Some)), array @@ -94,7 +123,7 @@ fn test_masked_child_preserves_length(#[case] validity: Validity) { _ => 3, }; - #[allow(clippy::cast_possible_truncation)] + #[expect(clippy::cast_possible_truncation)] let child = PrimitiveArray::from_iter(0..len as i32).into_array(); let array = MaskedArray::try_new(child, validity.clone()).unwrap(); diff --git a/vortex-array/src/arrays/masked/vtable/canonical.rs b/vortex-array/src/arrays/masked/vtable/canonical.rs index c612f522ef8..90bd999b8ed 100644 --- a/vortex-array/src/arrays/masked/vtable/canonical.rs +++ b/vortex-array/src/arrays/masked/vtable/canonical.rs @@ -6,32 +6,42 @@ mod tests { use rstest::rstest; use vortex_error::VortexResult; + use crate::Canonical; use crate::IntoArray; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::arrays::MaskedArray; use crate::arrays::PrimitiveArray; use crate::dtype::Nullability; use crate::validity::Validity; - #[rstest] - #[case( + fn masked_all_valid() -> MaskedArray { MaskedArray::try_new( PrimitiveArray::from_iter([1i32, 2, 3]).into_array(), - Validity::AllValid - ).unwrap(), - Nullability::Nullable - )] - #[case( + Validity::AllValid, + ) + .expect("valid masked array") + } + + fn masked_with_nulls() -> MaskedArray { MaskedArray::try_new( PrimitiveArray::from_iter([1i32, 2, 3]).into_array(), - Validity::from_iter([true, false, true]) - ).unwrap(), - Nullability::Nullable - )] + Validity::from_iter([true, false, true]), + ) + .expect("valid masked array") + } + + #[rstest] + #[case(masked_all_valid(), Nullability::Nullable)] + #[case(masked_with_nulls(), Nullability::Nullable)] fn test_canonical_nullability( #[case] array: MaskedArray, #[case] expected_nullability: Nullability, ) -> VortexResult<()> { - let canonical = array.to_canonical()?; + let canonical = array + .clone() + .into_array() + .execute::(&mut LEGACY_SESSION.create_execution_ctx())?; assert_eq!(canonical.dtype().nullability(), expected_nullability); assert_eq!(canonical.dtype(), array.dtype()); Ok(()) @@ -42,19 +52,21 @@ mod tests { let array = MaskedArray::try_new( PrimitiveArray::from_iter([1i32, 2, 3, 4, 5]).into_array(), Validity::from_iter([true, false, true, false, true]), - ) - .unwrap(); + )?; - let canonical = array.to_canonical()?; + let canonical = array + .into_array() + .execute::(&mut LEGACY_SESSION.create_execution_ctx())?; let prim = canonical.into_primitive(); // Check that null positions match validity. - assert_eq!(prim.valid_count().unwrap(), 3); - assert!(prim.is_valid(0).unwrap()); - assert!(!prim.is_valid(1).unwrap()); - assert!(prim.is_valid(2).unwrap()); - assert!(!prim.is_valid(3).unwrap()); - assert!(prim.is_valid(4).unwrap()); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + assert_eq!(prim.valid_count(&mut ctx)?, 3); + assert!(prim.is_valid(0, &mut LEGACY_SESSION.create_execution_ctx())?); + assert!(!prim.is_valid(1, &mut LEGACY_SESSION.create_execution_ctx())?); + assert!(prim.is_valid(2, &mut LEGACY_SESSION.create_execution_ctx())?); + assert!(!prim.is_valid(3, &mut LEGACY_SESSION.create_execution_ctx())?); + assert!(prim.is_valid(4, &mut LEGACY_SESSION.create_execution_ctx())?); Ok(()) } @@ -63,12 +75,18 @@ mod tests { let array = MaskedArray::try_new( PrimitiveArray::from_iter([10i32, 20, 30]).into_array(), Validity::AllValid, - ) - .unwrap(); + )?; - let canonical = array.to_canonical()?; + let canonical = array + .into_array() + .execute::(&mut LEGACY_SESSION.create_execution_ctx())?; assert_eq!(canonical.dtype().nullability(), Nullability::Nullable); - assert_eq!(canonical.into_array().valid_count().unwrap(), 3); + assert_eq!( + canonical + .into_array() + .valid_count(&mut LEGACY_SESSION.create_execution_ctx())?, + 3 + ); Ok(()) } } diff --git a/vortex-array/src/arrays/masked/vtable/mod.rs b/vortex-array/src/arrays/masked/vtable/mod.rs index cc5ef29a758..2ef10ba0082 100644 --- a/vortex-array/src/arrays/masked/vtable/mod.rs +++ b/vortex-array/src/arrays/masked/vtable/mod.rs @@ -12,6 +12,7 @@ use vortex_error::vortex_bail; use vortex_error::vortex_ensure; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::AnyCanonical; use crate::ArrayEq; @@ -19,7 +20,9 @@ use crate::ArrayHash; use crate::ArrayRef; use crate::Canonical; use crate::IntoArray; +use crate::LEGACY_SESSION; use crate::Precision; +use crate::VortexSessionExecute; use crate::array::Array; use crate::array::ArrayId; use crate::array::ArrayView; @@ -27,10 +30,9 @@ use crate::array::VTable; use crate::array::validity_to_child; use crate::arrays::ConstantArray; use crate::arrays::masked::MaskedArrayExt; +use crate::arrays::masked::MaskedArraySlotsExt; use crate::arrays::masked::MaskedData; -use crate::arrays::masked::array::CHILD_SLOT; -use crate::arrays::masked::array::SLOT_NAMES; -use crate::arrays::masked::array::VALIDITY_SLOT; +use crate::arrays::masked::array::MaskedSlots; use crate::arrays::masked::compute::rules::PARENT_RULES; use crate::arrays::masked::mask_validity_canonical; use crate::buffer::BufferHandle; @@ -38,7 +40,6 @@ use crate::dtype::DType; use crate::executor::ExecutionCtx; use crate::executor::ExecutionResult; use crate::require_child; -use crate::require_validity; use crate::scalar::Scalar; use crate::serde::ArrayChildren; use crate::validity::Validity; @@ -48,10 +49,6 @@ pub type MaskedArray = Array; #[derive(Clone, Debug)] pub struct Masked; -impl Masked { - pub const ID: ArrayId = ArrayId::new_ref("vortex.masked"); -} - impl ArrayHash for MaskedData { fn array_hash(&self, _state: &mut H, _precision: Precision) {} } @@ -69,7 +66,8 @@ impl VTable for Masked { type ValidityVTable = Self; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.masked"); + *ID } fn validate( @@ -80,10 +78,10 @@ impl VTable for Masked { slots: &[Option], ) -> VortexResult<()> { vortex_ensure!( - slots[CHILD_SLOT].is_some(), + slots[MaskedSlots::CHILD].is_some(), "MaskedArray child slot must be present" ); - let child = slots[CHILD_SLOT] + let child = slots[MaskedSlots::CHILD] .as_ref() .vortex_expect("validated child slot"); vortex_ensure!(child.len() == len, "MaskedArray child length mismatch"); @@ -149,7 +147,11 @@ impl VTable for Masked { }; let validity_slot = validity_to_child(&validity, len); - let data = MaskedData::try_new(len, child.all_valid()?, validity)?; + let data = MaskedData::try_new( + len, + child.all_valid(&mut LEGACY_SESSION.create_execution_ctx())?, + validity, + )?; Ok( crate::array::ArrayParts::new(self.clone(), dtype.clone(), len, data) .with_slots(vec![Some(child), validity_slot]), @@ -157,10 +159,12 @@ impl VTable for Masked { } fn execute(array: Array, ctx: &mut ExecutionCtx) -> VortexResult { - let validity_mask = array.masked_validity_mask(); + let array = require_child!(array, array.child(), MaskedSlots::CHILD => AnyCanonical); + + let validity = array.masked_validity(); // Fast path: all masked means result is all nulls. - if validity_mask.all_false() { + if matches!(validity, Validity::AllInvalid) { return Ok(ExecutionResult::done( ConstantArray::new(Scalar::null(array.dtype().as_nullable()), array.len()) .into_array(), @@ -173,12 +177,9 @@ impl VTable for Masked { // While we could manually convert the dtype, `mask_validity_canonical` is already O(1) for // `AllTrue` masks (no data copying), so there's no benefit. - let array = require_child!(array, array.child(), CHILD_SLOT => AnyCanonical); - require_validity!(array, VALIDITY_SLOT); - let child = Canonical::from(array.child().as_::()); Ok(ExecutionResult::done( - mask_validity_canonical(child, &validity_mask, ctx)?.into_array(), + mask_validity_canonical(child, validity, ctx)?.into_array(), )) } @@ -190,7 +191,7 @@ impl VTable for Masked { PARENT_RULES.evaluate(array, parent, child_idx) } fn slot_name(_array: ArrayView<'_, Self>, idx: usize) -> String { - SLOT_NAMES[idx].to_string() + MaskedSlots::NAMES[idx].to_string() } } diff --git a/vortex-array/src/arrays/masked/vtable/operations.rs b/vortex-array/src/arrays/masked/vtable/operations.rs index eb3132cf957..c82d0bf03ed 100644 --- a/vortex-array/src/arrays/masked/vtable/operations.rs +++ b/vortex-array/src/arrays/masked/vtable/operations.rs @@ -7,16 +7,16 @@ use crate::ExecutionCtx; use crate::array::ArrayView; use crate::array::OperationsVTable; use crate::arrays::Masked; -use crate::arrays::masked::MaskedArrayExt; +use crate::arrays::masked::MaskedArraySlotsExt; use crate::scalar::Scalar; impl OperationsVTable for Masked { fn scalar_at( array: ArrayView<'_, Masked>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { // Invalid indices are handled by the entrypoint function. - Ok(array.child().scalar_at(index)?.into_nullable()) + Ok(array.child().execute_scalar(index, ctx)?.into_nullable()) } } diff --git a/vortex-array/src/arrays/mod.rs b/vortex-array/src/arrays/mod.rs index 2597708919a..212f4bbe619 100644 --- a/vortex-array/src/arrays/mod.rs +++ b/vortex-array/src/arrays/mod.rs @@ -75,8 +75,8 @@ pub use primitive::Primitive; pub use primitive::PrimitiveArray; pub mod scalar_fn; +pub use scalar_fn::ScalarFn; pub use scalar_fn::ScalarFnArray; -pub use scalar_fn::ScalarFnVTable; pub mod shared; pub use shared::Shared; diff --git a/vortex-array/src/arrays/null/compute/cast.rs b/vortex-array/src/arrays/null/compute/cast.rs index b39dea5988b..be6c73ad7d5 100644 --- a/vortex-array/src/arrays/null/compute/cast.rs +++ b/vortex-array/src/arrays/null/compute/cast.rs @@ -32,6 +32,8 @@ mod tests { use rstest::rstest; use crate::IntoArray; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::arrays::NullArray; use crate::builtins::ArrayBuiltins; use crate::compute::conformance::cast::test_cast_conformance; @@ -64,7 +66,12 @@ mod tests { // Verify all values are null for i in 0..5 { - assert!(result.scalar_at(i).unwrap().is_null()); + assert!( + result + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); } } diff --git a/vortex-array/src/arrays/null/compute/mod.rs b/vortex-array/src/arrays/null/compute/mod.rs index ad5558a8807..df2bc1547ae 100644 --- a/vortex-array/src/arrays/null/compute/mod.rs +++ b/vortex-array/src/arrays/null/compute/mod.rs @@ -15,7 +15,10 @@ mod test { use vortex_mask::Mask; use crate::IntoArray; - use crate::ToCanonical; + use crate::LEGACY_SESSION; + #[expect(deprecated)] + use crate::ToCanonical as _; + use crate::VortexSessionExecute; use crate::arrays::NullArray; use crate::compute::conformance::consistency::test_array_consistency; use crate::compute::conformance::filter::test_filter_conformance; @@ -26,11 +29,17 @@ mod test { #[test] fn test_slice_nulls() { let nulls = NullArray::new(10); + #[expect(deprecated)] let sliced = nulls.slice(0..4).unwrap().to_null(); assert_eq!(sliced.len(), 4); + let sliced_arr = sliced.as_array(); assert!(matches!( - sliced.as_array().validity_mask().unwrap(), + sliced_arr + .validity() + .unwrap() + .execute_mask(sliced_arr.len(), &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Mask::AllFalse(4) )); } @@ -38,14 +47,20 @@ mod test { #[test] fn test_take_nulls() { let nulls = NullArray::new(10); + #[expect(deprecated)] let taken = nulls .take(buffer![0u64, 2, 4, 6, 8].into_array()) .unwrap() .to_null(); assert_eq!(taken.len(), 5); + let taken_arr = taken.as_array(); assert!(matches!( - taken.as_array().validity_mask().unwrap(), + taken_arr + .validity() + .unwrap() + .execute_mask(taken_arr.len(), &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Mask::AllFalse(5) )); } @@ -54,7 +69,9 @@ mod test { fn test_scalar_at_nulls() { let nulls = NullArray::new(10); - let scalar = nulls.scalar_at(0).unwrap(); + let scalar = nulls + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!(scalar.is_null()); assert_eq!(scalar.dtype().clone(), DType::Null); } diff --git a/vortex-array/src/arrays/null/compute/take.rs b/vortex-array/src/arrays/null/compute/take.rs index f24b89aea45..5352ae67653 100644 --- a/vortex-array/src/arrays/null/compute/take.rs +++ b/vortex-array/src/arrays/null/compute/take.rs @@ -6,7 +6,8 @@ use vortex_error::vortex_bail; use crate::ArrayRef; use crate::IntoArray; -use crate::ToCanonical; +#[expect(deprecated)] +use crate::ToCanonical as _; use crate::array::ArrayView; use crate::arrays::Null; use crate::arrays::NullArray; @@ -16,8 +17,9 @@ use crate::match_each_integer_ptype; use crate::optimizer::rules::ParentRuleSet; impl TakeReduce for Null { - #[allow(clippy::cast_possible_truncation)] + #[expect(clippy::cast_possible_truncation)] fn take(array: ArrayView<'_, Null>, indices: &ArrayRef) -> VortexResult> { + #[expect(deprecated)] let indices = indices.to_primitive(); // Enforce all indices are valid diff --git a/vortex-array/src/arrays/null/mod.rs b/vortex-array/src/arrays/null/mod.rs index b2bdcc3a840..30f68cecc51 100644 --- a/vortex-array/src/arrays/null/mod.rs +++ b/vortex-array/src/arrays/null/mod.rs @@ -5,6 +5,7 @@ use vortex_error::VortexResult; use vortex_error::vortex_ensure; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::ArrayRef; use crate::ExecutionCtx; @@ -36,7 +37,8 @@ impl VTable for Null { type ValidityVTable = Self; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.null"); + *ID } fn validate( @@ -121,7 +123,7 @@ impl VTable for Null { /// ``` /// # fn main() -> vortex_error::VortexResult<()> { /// use vortex_array::arrays::NullArray; -/// use vortex_array::IntoArray; +/// use vortex_array::{IntoArray, LEGACY_SESSION, VortexSessionExecute}; /// /// // Create a null array with 5 elements /// let array = NullArray::new(5); @@ -131,7 +133,8 @@ impl VTable for Null { /// assert_eq!(sliced.len(), 2); /// /// // All elements are null -/// let scalar = array.scalar_at(0).unwrap(); +/// let mut ctx = LEGACY_SESSION.create_execution_ctx(); +/// let scalar = array.execute_scalar(0, &mut ctx).unwrap(); /// assert!(scalar.is_null()); /// # Ok(()) /// # } @@ -139,10 +142,6 @@ impl VTable for Null { #[derive(Clone, Debug)] pub struct Null; -impl Null { - pub const ID: ArrayId = ArrayId::new_ref("vortex.null"); -} - impl Array { pub fn new(len: usize) -> Self { unsafe { diff --git a/vortex-array/src/arrays/patched/array.rs b/vortex-array/src/arrays/patched/array.rs index 869671d396a..910409cb919 100644 --- a/vortex-array/src/arrays/patched/array.rs +++ b/vortex-array/src/arrays/patched/array.rs @@ -15,6 +15,8 @@ use crate::ArrayRef; use crate::Canonical; use crate::ExecutionCtx; use crate::IntoArray; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::array::Array; use crate::array::ArrayParts; use crate::array::TypedArrayRef; @@ -111,12 +113,14 @@ pub trait PatchedArrayExt: PatchedArraySlotsExt { assert!(chunk * 1024 <= self.as_ref().len() + self.offset()); assert!(lane < self.n_lanes()); - let start = self - .lane_offsets() - .scalar_at(chunk * self.n_lanes() + lane)?; - let stop = self - .lane_offsets() - .scalar_at(chunk * self.n_lanes() + lane + 1)?; + let start = self.lane_offsets().execute_scalar( + chunk * self.n_lanes() + lane, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; + let stop = self.lane_offsets().execute_scalar( + chunk * self.n_lanes() + lane + 1, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; let start = start .as_primitive() @@ -186,7 +190,7 @@ impl Patched { ); vortex_ensure!( - patches.values().all_valid()?, + patches.values().all_valid(ctx)?, "PatchedArray cannot be built from Patches with nulls" ); @@ -247,7 +251,6 @@ impl Patched { } /// Transpose a set of patches from the default sorted layout into the data parallel layout. -#[allow(clippy::cognitive_complexity)] fn transpose_patches(patches: &Patches, ctx: &mut ExecutionCtx) -> VortexResult { let array_len = patches.array_len(); let offset = patches.offset(); @@ -285,7 +288,7 @@ fn transpose_patches(patches: &Patches, ctx: &mut ExecutionCtx) -> VortexResult< }) } -#[allow(clippy::cast_possible_truncation)] +#[expect(clippy::cast_possible_truncation)] fn transpose( indices_in: &[I], values_in: &[V], diff --git a/vortex-array/src/arrays/patched/compute/compare.rs b/vortex-array/src/arrays/patched/compute/compare.rs index abd54574940..c7d879323cb 100644 --- a/vortex-array/src/arrays/patched/compute/compare.rs +++ b/vortex-array/src/arrays/patched/compute/compare.rs @@ -53,7 +53,7 @@ impl CompareKernel for Patched { .execute::(ctx)? .into_bool(); - let validity = child_to_validity(&result.slots()[0], result.dtype().nullability()); + let validity = child_to_validity(result.slots()[0].as_ref(), result.dtype().nullability()); let len = result.len(); let BoolDataParts { bits, offset, len } = result.into_data().into_parts(len); @@ -160,6 +160,7 @@ impl ApplyPatches<'_, V> { mod tests { use vortex_buffer::buffer; use vortex_error::VortexResult; + use vortex_error::vortex_err; use crate::ExecutionCtx; use crate::IntoArray; @@ -261,7 +262,7 @@ mod tests { let lhs = Patched::from_array_and_patches(lhs, &patches, &mut ctx)? .into_array() .try_downcast::() - .unwrap(); + .map_err(|_| vortex_err!("expected patched array"))?; let rhs = ConstantArray::new(subnormal, 512).into_array(); @@ -271,7 +272,7 @@ mod tests { CompareOperator::Eq, &mut ctx, )? - .unwrap(); + .ok_or_else(|| vortex_err!("expected compare result"))?; let expected = BoolArray::from_indices(512, [510], Validity::NonNullable).into_array(); @@ -295,7 +296,7 @@ mod tests { let lhs = Patched::from_array_and_patches(lhs, &patches, &mut ctx)? .into_array() .try_downcast::() - .unwrap(); + .map_err(|_| vortex_err!("expected patched array"))?; let rhs = ConstantArray::new(0.0f32, 10).into_array(); @@ -305,7 +306,7 @@ mod tests { CompareOperator::Eq, &mut ctx, )? - .unwrap(); + .ok_or_else(|| vortex_err!("expected compare result"))?; let expected = BoolArray::from_indices(10, [7], Validity::NonNullable).into_array(); diff --git a/vortex-array/src/arrays/patched/compute/take.rs b/vortex-array/src/arrays/patched/compute/take.rs index 49aa726c51c..893d18db1ac 100644 --- a/vortex-array/src/arrays/patched/compute/take.rs +++ b/vortex-array/src/arrays/patched/compute/take.rs @@ -87,7 +87,7 @@ impl TakeExecute for Patched { /// /// First, builds a hashmap from index to patch value, then uses the hashmap in a loop to collect /// the values. -#[allow(clippy::too_many_arguments)] +#[expect(clippy::too_many_arguments)] fn take_map( output: &mut [V], indices: &[I], @@ -173,6 +173,7 @@ mod tests { // Take indices [0, 1, 2, 3, 4] - should get [0, 10, 0, 30, 0] let indices = buffer![0u32, 1, 2, 3, 4].into_array(); + #[expect(deprecated)] let result = array.take(indices)?.to_canonical()?.into_array(); let expected = PrimitiveArray::from_iter([0u16, 10, 0, 30, 0]).into_array(); @@ -186,6 +187,7 @@ mod tests { let array = make_patched_array(&[0; 10], &[1, 3], &[100, 200], 2..10)?; let indices = buffer![0u32, 1, 2, 3, 7].into_array(); + #[expect(deprecated)] let result = array.take(indices)?.to_canonical()?.into_array(); let expected = PrimitiveArray::from_iter([0u16, 200, 0, 0, 0]).into_array(); @@ -201,6 +203,7 @@ mod tests { // Take indices in reverse order let indices = buffer![4u32, 3, 2, 1, 0].into_array(); + #[expect(deprecated)] let result = array.take(indices)?.to_canonical()?.into_array(); let expected = PrimitiveArray::from_iter([0u16, 30, 0, 10, 0]).into_array(); @@ -216,9 +219,11 @@ mod tests { // Take the same patched index multiple times let indices = buffer![2u32, 2, 0, 2].into_array(); + #[expect(deprecated)] let result = array.take(indices)?.to_canonical()?.into_array(); // execute the array. + #[expect(deprecated)] let _canonical = result.to_canonical()?.into_primitive(); let expected = PrimitiveArray::from_iter([99u16, 99, 0, 99]).into_array(); @@ -250,6 +255,7 @@ mod tests { .into_array(), ), ); + #[expect(deprecated)] let result = array .take(indices.into_array())? .to_canonical()? diff --git a/vortex-array/src/arrays/patched/mod.rs b/vortex-array/src/arrays/patched/mod.rs index 56024f50d8e..0ba25ab7929 100644 --- a/vortex-array/src/arrays/patched/mod.rs +++ b/vortex-array/src/arrays/patched/mod.rs @@ -71,6 +71,9 @@ mod array; mod compute; mod vtable; +use std::env; +use std::sync::LazyLock; + pub use array::*; use vortex_buffer::ByteBuffer; pub use vtable::*; @@ -96,3 +99,17 @@ const fn patch_lanes() -> usize { // from shared to global memory. if size_of::() < 8 { 32 } else { 16 } } + +/// Flag indicating if experimental patched array support is enabled. +/// +/// This is set using the environment variable `VORTEX_EXPERIMENTAL_PATCHED_ARRAY`. +/// +/// When this is true, any arrays with interior `Patches` will be read as a `Patched` +/// array, and eliminate the interior patches. +/// +/// The builtin compressor will also generate Patched arrays. +pub fn use_experimental_patches() -> bool { + static USE_EXPERIMENTAL_PATCHES: LazyLock = + LazyLock::new(|| env::var("VORTEX_EXPERIMENTAL_PATCHED_ARRAY").is_ok_and(|v| v == "1")); + *USE_EXPERIMENTAL_PATCHES +} diff --git a/vortex-array/src/arrays/patched/vtable/mod.rs b/vortex-array/src/arrays/patched/vtable/mod.rs index 4d430277bc8..ad2fc9aadc7 100644 --- a/vortex-array/src/arrays/patched/vtable/mod.rs +++ b/vortex-array/src/arrays/patched/vtable/mod.rs @@ -17,6 +17,7 @@ use vortex_error::VortexExpect; use vortex_error::VortexResult; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::ArrayRef; use crate::Canonical; @@ -99,7 +100,8 @@ impl VTable for Patched { type ValidityVTable = ValidityVTableFromChild; fn id(&self) -> ArrayId { - ArrayId::new_ref("vortex.patched") + static ID: CachedId = CachedId::new("vortex.patched"); + *ID } fn validate( @@ -318,7 +320,6 @@ impl VTable for Patched { } /// Apply patches on top of the existing value types. -#[allow(clippy::too_many_arguments)] fn apply_patches_primitive( output: &mut [V], offset: usize, @@ -372,6 +373,7 @@ mod tests { use crate::patches::Patches; use crate::serde::SerializeOptions; use crate::serde::SerializedArray; + use crate::session::ArraySessionExt; use crate::validity::Validity; #[test] @@ -587,7 +589,9 @@ mod tests { let dtype = array.dtype().clone(); let len = array.len(); - let ctx = ArrayContext::empty(); + LEGACY_SESSION.arrays().register(Patched); + + let ctx = ArrayContext::empty().with_registry(LEGACY_SESSION.arrays().registry().clone()); let serialized = array .serialize(&ctx, &LEGACY_SESSION, &SerializeOptions::default()) .unwrap(); diff --git a/vortex-array/src/arrays/patched/vtable/operations.rs b/vortex-array/src/arrays/patched/vtable/operations.rs index 95beeaaf87f..f0491666e56 100644 --- a/vortex-array/src/arrays/patched/vtable/operations.rs +++ b/vortex-array/src/arrays/patched/vtable/operations.rs @@ -43,12 +43,15 @@ impl OperationsVTable for Patched { // be slower. for (&patch_index, idx) in std::iter::zip(patch_indices.as_slice::(), range) { if patch_index == chunk_index { - return array.patch_values().scalar_at(idx)?.cast(array.dtype()); + return array + .patch_values() + .execute_scalar(idx, ctx)? + .cast(array.dtype()); } } // Otherwise, access the underlying value. - array.inner().scalar_at(index) + array.inner().execute_scalar(index, ctx) } } @@ -59,6 +62,8 @@ mod tests { use crate::ExecutionCtx; use crate::IntoArray; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::arrays::Patched; use crate::dtype::Nullability; use crate::optimizer::ArrayOptimizer; @@ -85,19 +90,27 @@ mod tests { .into_array(); assert_eq!( - array.scalar_at(0).unwrap(), + array + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::primitive(0u16, Nullability::NonNullable) ); assert_eq!( - array.scalar_at(1).unwrap(), + array + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::primitive(1u16, Nullability::NonNullable) ); assert_eq!( - array.scalar_at(2).unwrap(), + array + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::primitive(1u16, Nullability::NonNullable) ); assert_eq!( - array.scalar_at(3).unwrap(), + array + .execute_scalar(3, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::primitive(1u16, Nullability::NonNullable) ); } @@ -122,7 +135,9 @@ mod tests { .into_array(); for index in 0..array.len() { - let value = array.scalar_at(index).unwrap(); + let value = array + .execute_scalar(index, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); if [1, 2, 3].contains(&index) { assert_eq!(value, 1u16.into()); @@ -157,9 +172,19 @@ mod tests { assert!(array.is::()); - assert_eq!(array.scalar_at(0).unwrap(), 1u16.into()); + assert_eq!( + array + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 1u16.into() + ); for index in 1..array.len() { - assert_eq!(array.scalar_at(index).unwrap(), 0u16.into()); + assert_eq!( + array + .execute_scalar(index, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 0u16.into() + ); } } } diff --git a/vortex-array/src/arrays/primitive/array/accessor.rs b/vortex-array/src/arrays/primitive/array/accessor.rs index 5a3665c71ed..f26b201ea2b 100644 --- a/vortex-array/src/arrays/primitive/array/accessor.rs +++ b/vortex-array/src/arrays/primitive/array/accessor.rs @@ -5,7 +5,8 @@ use std::iter; use vortex_error::VortexExpect; -use crate::ToCanonical; +#[expect(deprecated)] +use crate::ToCanonical as _; use crate::accessor::ArrayAccessor; use crate::arrays::PrimitiveArray; use crate::dtype::NativePType; @@ -26,6 +27,7 @@ impl ArrayAccessor for PrimitiveArray { } Validity::AllInvalid => f(&mut iter::repeat_n(None, self.len())), Validity::Array(v) => { + #[expect(deprecated)] let validity = v.to_bool().into_bit_buffer(); let mut iter = self .as_slice::() diff --git a/vortex-array/src/arrays/primitive/array/mod.rs b/vortex-array/src/arrays/primitive/array/mod.rs index e64cc891d4e..254c4b0baee 100644 --- a/vortex-array/src/arrays/primitive/array/mod.rs +++ b/vortex-array/src/arrays/primitive/array/mod.rs @@ -16,7 +16,8 @@ use vortex_error::vortex_err; use vortex_error::vortex_panic; use crate::LEGACY_SESSION; -use crate::ToCanonical; +#[expect(deprecated)] +use crate::ToCanonical as _; use crate::VortexSessionExecute; use crate::array::Array; use crate::array::ArrayParts; @@ -68,6 +69,7 @@ pub(super) const SLOT_NAMES: [&str; NUM_SLOTS] = ["validity"]; /// ``` /// # fn main() -> vortex_error::VortexResult<()> { /// use vortex_array::arrays::PrimitiveArray; +/// use vortex_array::{LEGACY_SESSION, VortexSessionExecute}; /// /// // Create from iterator using FromIterator impl /// let array: PrimitiveArray = [1i32, 2, 3, 4, 5].into_iter().collect(); @@ -76,7 +78,8 @@ pub(super) const SLOT_NAMES: [&str; NUM_SLOTS] = ["validity"]; /// let sliced = array.slice(1..3)?; /// /// // Access individual values -/// let value = sliced.scalar_at(0).unwrap(); +/// let mut ctx = LEGACY_SESSION.create_execution_ctx(); +/// let value = sliced.execute_scalar(0, &mut ctx).unwrap(); /// assert_eq!(value, 2i32.into()); /// /// # Ok(()) @@ -120,11 +123,10 @@ pub trait PrimitiveArrayExt: TypedArrayRef { } fn validity(&self) -> Validity { - child_to_validity(&self.as_ref().slots()[VALIDITY_SLOT], self.nullability()) - } - - fn validity_mask(&self) -> vortex_mask::Mask { - self.validity().to_mask(self.as_ref().len()) + child_to_validity( + self.as_ref().slots()[VALIDITY_SLOT].as_ref(), + self.nullability(), + ) } fn buffer_handle(&self) -> &BufferHandle { @@ -181,46 +183,58 @@ pub trait PrimitiveArrayExt: TypedArrayRef { if min < 0 || max < 0 { // Signed if min >= i8::MIN as i64 && max <= i8::MAX as i64 { - return Ok(self + #[expect(deprecated)] + let result = self .as_ref() .cast(DType::Primitive(PType::I8, nullability))? - .to_primitive()); + .to_primitive(); + return Ok(result); } if min >= i16::MIN as i64 && max <= i16::MAX as i64 { - return Ok(self + #[expect(deprecated)] + let result = self .as_ref() .cast(DType::Primitive(PType::I16, nullability))? - .to_primitive()); + .to_primitive(); + return Ok(result); } if min >= i32::MIN as i64 && max <= i32::MAX as i64 { - return Ok(self + #[expect(deprecated)] + let result = self .as_ref() .cast(DType::Primitive(PType::I32, nullability))? - .to_primitive()); + .to_primitive(); + return Ok(result); } } else { // Unsigned if max <= u8::MAX as i64 { - return Ok(self + #[expect(deprecated)] + let result = self .as_ref() .cast(DType::Primitive(PType::U8, nullability))? - .to_primitive()); + .to_primitive(); + return Ok(result); } if max <= u16::MAX as i64 { - return Ok(self + #[expect(deprecated)] + let result = self .as_ref() .cast(DType::Primitive(PType::U16, nullability))? - .to_primitive()); + .to_primitive(); + return Ok(result); } if max <= u32::MAX as i64 { - return Ok(self + #[expect(deprecated)] + let result = self .as_ref() .cast(DType::Primitive(PType::U32, nullability))? - .to_primitive()); + .to_primitive(); + return Ok(result); } } @@ -485,6 +499,7 @@ impl Array { BufferMut::::from_iter(buf_iter.zip(iter::repeat(false)).map(f)) } Validity::Array(val) => { + #[expect(deprecated)] let val = val.to_bool().into_bit_buffer(); BufferMut::::from_iter(buf_iter.zip(val.iter()).map(f)) } @@ -538,6 +553,7 @@ impl PrimitiveData { Validity::AllValid | Validity::NonNullable => valid_elems_buffer.aligned(alignment), Validity::AllInvalid => ByteBuffer::zeroed_aligned(n_rows * byte_width, alignment), Validity::Array(is_valid) => { + #[expect(deprecated)] let bool_array = is_valid.to_bool(); let bool_buffer = bool_array.to_bit_buffer(); let mut bytes = ByteBufferMut::zeroed_aligned(n_rows * byte_width, alignment); diff --git a/vortex-array/src/arrays/primitive/array/patch.rs b/vortex-array/src/arrays/primitive/array/patch.rs index 51fae5e9cd0..197db4d3854 100644 --- a/vortex-array/src/arrays/primitive/array/patch.rs +++ b/vortex-array/src/arrays/primitive/array/patch.rs @@ -135,7 +135,8 @@ mod tests { use vortex_buffer::buffer; use super::*; - use crate::ToCanonical; + #[expect(deprecated)] + use crate::ToCanonical as _; use crate::assert_arrays_eq; use crate::validity::Validity; @@ -174,8 +175,10 @@ mod tests { fn patch_sliced() { let input = PrimitiveArray::new(buffer![2u32; 10], Validity::AllValid); let sliced = input.slice(2..8).unwrap(); + #[expect(deprecated)] + let sliced_primitive = sliced.to_primitive(); assert_arrays_eq!( - sliced.to_primitive(), + sliced_primitive, PrimitiveArray::new(buffer![2u32; 6], Validity::AllValid) ); } diff --git a/vortex-array/src/arrays/primitive/array/top_value.rs b/vortex-array/src/arrays/primitive/array/top_value.rs index dae36a5a523..d3ee5eb5a65 100644 --- a/vortex-array/src/arrays/primitive/array/top_value.rs +++ b/vortex-array/src/arrays/primitive/array/top_value.rs @@ -10,6 +10,8 @@ use vortex_mask::AllOr; use vortex_mask::Mask; use vortex_utils::aliases::hash_map::HashMap; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::arrays::PrimitiveArray; use crate::arrays::primitive::NativeValue; use crate::dtype::NativePType; @@ -29,7 +31,13 @@ impl PrimitiveArray { } match_each_native_ptype!(self.ptype(), |P| { - let (top, count) = typed_top_value(self.as_slice::

(), self.validity_mask()?); + let (top, count) = typed_top_value( + self.as_slice::

(), + self.as_ref().validity()?.execute_mask( + self.as_ref().len(), + &mut LEGACY_SESSION.create_execution_ctx(), + )?, + ); Ok(Some((top.into(), count))) }) } diff --git a/vortex-array/src/arrays/primitive/compute/cast.rs b/vortex-array/src/arrays/primitive/compute/cast.rs index 9053afaf66f..92140bf9b8b 100644 --- a/vortex-array/src/arrays/primitive/compute/cast.rs +++ b/vortex-array/src/arrays/primitive/compute/cast.rs @@ -1,13 +1,11 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +use num_traits::AsPrimitive; use vortex_buffer::Buffer; use vortex_buffer::BufferMut; use vortex_error::VortexResult; use vortex_error::vortex_bail; -use vortex_error::vortex_err; -use vortex_mask::AllOr; -use vortex_mask::Mask; use crate::ArrayRef; use crate::ExecutionCtx; @@ -23,6 +21,37 @@ use crate::dtype::Nullability; use crate::dtype::PType; use crate::match_each_native_ptype; use crate::scalar_fn::fns::cast::CastKernel; +use crate::scalar_fn::fns::cast::CastReduce; + +impl CastReduce for Primitive { + fn cast(array: ArrayView<'_, Primitive>, dtype: &DType) -> VortexResult> { + // Only the same ptype is reducible without execution; type changes need the kernel + // to verify values fit in the target range. + let DType::Primitive(new_ptype, new_nullability) = dtype else { + return Ok(None); + }; + if *new_ptype != array.ptype() { + return Ok(None); + } + + let Some(new_validity) = array + .validity()? + .trivial_cast_nullability(*new_nullability, array.len())? + else { + return Ok(None); + }; + + // SAFETY: validity and data buffer still have same length. + Ok(Some(unsafe { + PrimitiveArray::new_unchecked_from_handle( + array.buffer_handle().clone(), + array.ptype(), + new_validity, + ) + .into_array() + })) + } +} impl CastKernel for Primitive { fn cast( @@ -38,7 +67,7 @@ impl CastKernel for Primitive { // First, check that the cast is compatible with the source array's validity let new_validity = array .validity()? - .cast_nullability(new_nullability, array.len())?; + .cast_nullability(new_nullability, array.len(), ctx)?; // Same ptype: zero-copy, just update validity. if array.ptype() == new_ptype { @@ -53,6 +82,14 @@ impl CastKernel for Primitive { })); } + if !values_fit_in(array, new_ptype, ctx) { + vortex_bail!( + Compute: "Cannot cast {} to {} — values exceed target range", + array.ptype(), + new_ptype, + ); + } + // Same-width integers have identical bit representations due to 2's // complement. If all values fit in the target range, reinterpret with // no allocation. @@ -60,13 +97,6 @@ impl CastKernel for Primitive { && new_ptype.is_int() && array.ptype().byte_width() == new_ptype.byte_width() { - if !values_fit_in(array, new_ptype, ctx) { - vortex_bail!( - Compute: "Cannot cast {} to {} — values exceed target range", - array.ptype(), - new_ptype, - ); - } // SAFETY: both types are integers with the same size and alignment, and // min/max confirm all valid values are representable in the target type. return Ok(Some(unsafe { @@ -79,13 +109,10 @@ impl CastKernel for Primitive { })); } - let mask = array.validity_mask(); - // Otherwise, we need to cast the values one-by-one. Ok(Some(match_each_native_ptype!(new_ptype, |T| { match_each_native_ptype!(array.ptype(), |F| { - PrimitiveArray::new(cast::(array.as_slice(), mask)?, new_validity) - .into_array() + PrimitiveArray::new(cast::(array.as_slice()), new_validity).into_array() }) }))) } @@ -104,30 +131,11 @@ fn values_fit_in( .is_none_or(|mm| mm.min.cast(&target_dtype).is_ok() && mm.max.cast(&target_dtype).is_ok()) } -fn cast(array: &[F], mask: Mask) -> VortexResult> { - let try_cast = |src: F| -> VortexResult { - T::from(src).ok_or_else(|| vortex_err!(Compute: "Failed to cast {} to {:?}", src, T::PTYPE)) - }; - match mask.bit_buffer() { - AllOr::None => Ok(Buffer::zeroed(array.len())), - AllOr::All => { - let mut buffer = BufferMut::with_capacity(array.len()); - for &src in array { - // SAFETY: we've pre-allocated the required capacity - unsafe { buffer.push_unchecked(try_cast(src)?) } - } - Ok(buffer.freeze()) - } - AllOr::Some(b) => { - let mut buffer = BufferMut::with_capacity(array.len()); - for (&src, valid) in array.iter().zip(b.iter()) { - let dst = if valid { try_cast(src)? } else { T::default() }; - // SAFETY: we've pre-allocated the required capacity - unsafe { buffer.push_unchecked(dst) } - } - Ok(buffer.freeze()) - } - } +/// Caller must ensure all valid values are representable via `values_fit_in`. +/// Out-of-range values at invalid positions are truncated/wrapped by `as`, +/// which is fine because they are masked out by validity. +fn cast, T: NativePType>(array: &[F]) -> Buffer { + BufferMut::from_trusted_len_iter(array.iter().map(|&src| src.as_())).freeze() } #[cfg(test)] @@ -139,27 +147,31 @@ mod test { use vortex_mask::Mask; use crate::IntoArray; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::arrays::PrimitiveArray; use crate::assert_arrays_eq; use crate::builtins::ArrayBuiltins; - use crate::canonical::ToCanonical; + #[expect(deprecated)] + use crate::canonical::ToCanonical as _; use crate::compute::conformance::cast::test_cast_conformance; use crate::dtype::DType; use crate::dtype::Nullability; use crate::dtype::PType; use crate::validity::Validity; - #[allow(clippy::cognitive_complexity)] #[test] fn cast_u32_u8() { let arr = buffer![0u32, 10, 200].into_array(); // cast from u32 to u8 + #[expect(deprecated)] let p = arr.cast(PType::U8.into()).unwrap().to_primitive(); assert_arrays_eq!(p, PrimitiveArray::from_iter([0u8, 10, 200])); assert!(matches!(p.validity(), Ok(Validity::NonNullable))); // to nullable + #[expect(deprecated)] let p = p .into_array() .cast(DType::Primitive(PType::U8, Nullability::Nullable)) @@ -172,6 +184,7 @@ mod test { assert!(matches!(p.validity(), Ok(Validity::AllValid))); // back to non-nullable + #[expect(deprecated)] let p = p .into_array() .cast(DType::Primitive(PType::U8, Nullability::NonNullable)) @@ -181,6 +194,7 @@ mod test { assert!(matches!(p.validity(), Ok(Validity::NonNullable))); // to nullable u32 + #[expect(deprecated)] let p = p .into_array() .cast(DType::Primitive(PType::U32, Nullability::Nullable)) @@ -193,6 +207,7 @@ mod test { assert!(matches!(p.validity(), Ok(Validity::AllValid))); // to non-nullable u8 + #[expect(deprecated)] let p = p .into_array() .cast(DType::Primitive(PType::U8, Nullability::NonNullable)) @@ -205,6 +220,7 @@ mod test { #[test] fn cast_u32_f32() { let arr = buffer![0u32, 10, 200].into_array(); + #[expect(deprecated)] let u8arr = arr.cast(PType::F32.into()).unwrap().to_primitive(); assert_arrays_eq!(u8arr, PrimitiveArray::from_iter([0.0f32, 10., 200.])); } @@ -212,6 +228,7 @@ mod test { #[test] fn cast_i32_u32() { let arr = buffer![-1i32].into_array(); + #[expect(deprecated)] let error = arr .cast(PType::U32.into()) .and_then(|a| a.to_canonical().map(|c| c.into_array())) @@ -223,6 +240,7 @@ mod test { #[test] fn cast_array_with_nulls_to_nonnullable() { let arr = PrimitiveArray::from_option_iter([Some(-1i32), None, Some(10)]); + #[expect(deprecated)] let err = arr .into_array() .cast(PType::I32.into()) @@ -242,6 +260,7 @@ mod test { buffer![-1i32, 0, 10], Validity::from_iter([false, true, true]), ); + #[expect(deprecated)] let p = arr .into_array() .cast(DType::Primitive(PType::U32, Nullability::Nullable)) @@ -252,7 +271,11 @@ mod test { PrimitiveArray::from_option_iter([None, Some(0u32), Some(10)]) ); assert_eq!( - p.validity_mask().unwrap(), + p.as_ref() + .validity() + .unwrap() + .execute_mask(p.as_ref().len(), &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Mask::from(BitBuffer::from(vec![false, true, true])) ); } @@ -264,6 +287,7 @@ mod test { let src = PrimitiveArray::from_iter([0u32, 10, 100]); let src_ptr = src.as_slice::().as_ptr(); + #[expect(deprecated)] let dst = src.into_array().cast(PType::I32.into())?.to_primitive(); let dst_ptr = dst.as_slice::().as_ptr(); @@ -278,6 +302,7 @@ mod test { #[test] fn cast_same_width_int_out_of_range_errors() { let arr = buffer![u32::MAX].into_array(); + #[expect(deprecated)] let err = arr .cast(PType::I32.into()) .and_then(|a| a.to_canonical().map(|c| c.into_array())) @@ -290,6 +315,7 @@ mod test { #[test] fn cast_same_width_all_null() -> vortex_error::VortexResult<()> { let arr = PrimitiveArray::new(buffer![0xFFu8, 0xFF], Validity::AllInvalid); + #[expect(deprecated)] let casted = arr .into_array() .cast(DType::Primitive(PType::I8, Nullability::Nullable))? @@ -309,6 +335,7 @@ mod test { buffer![u32::MAX, 0u32, 42u32], Validity::from_iter([false, true, true]), ); + #[expect(deprecated)] let casted = arr .into_array() .cast(DType::Primitive(PType::I32, Nullability::Nullable))? @@ -320,6 +347,24 @@ mod test { Ok(()) } + #[test] + fn cast_u32_to_u8_with_out_of_range_nulls() -> vortex_error::VortexResult<()> { + let arr = PrimitiveArray::new( + buffer![1000u32, 10u32, 42u32], + Validity::from_iter([false, true, true]), + ); + #[expect(deprecated)] + let casted = arr + .into_array() + .cast(DType::Primitive(PType::U8, Nullability::Nullable))? + .to_primitive(); + assert_arrays_eq!( + casted, + PrimitiveArray::from_option_iter([None, Some(10u8), Some(42)]) + ); + Ok(()) + } + #[rstest] #[case(buffer![0u8, 1, 2, 3, 255].into_array())] #[case(buffer![0u16, 100, 1000, 65535].into_array())] @@ -330,7 +375,9 @@ mod test { #[case(buffer![-1000000i32, -1, 0, 1, 1000000].into_array())] #[case(buffer![-1000000000i64, -1, 0, 1, 1000000000].into_array())] #[case(buffer![0.0f32, 1.5, -2.5, 100.0, 1e6].into_array())] + #[case(buffer![f32::NAN, f32::INFINITY, f32::NEG_INFINITY, 0.0f32].into_array())] #[case(buffer![0.0f64, 1.5, -2.5, 100.0, 1e12].into_array())] + #[case(buffer![f64::NAN, f64::INFINITY, f64::NEG_INFINITY, 0.0f64].into_array())] #[case(PrimitiveArray::from_option_iter([Some(1u8), None, Some(255), Some(0), None]).into_array())] #[case(PrimitiveArray::from_option_iter([Some(1i32), None, Some(-100), Some(0), None]).into_array())] #[case(buffer![42u32].into_array())] diff --git a/vortex-array/src/arrays/primitive/compute/fill_null.rs b/vortex-array/src/arrays/primitive/compute/fill_null.rs index d0de9a81a4f..67089bd534e 100644 --- a/vortex-array/src/arrays/primitive/compute/fill_null.rs +++ b/vortex-array/src/arrays/primitive/compute/fill_null.rs @@ -51,37 +51,56 @@ mod test { use vortex_buffer::buffer; use crate::IntoArray; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::arrays::PrimitiveArray; use crate::arrays::primitive::compute::fill_null::BoolArray; use crate::assert_arrays_eq; use crate::builtins::ArrayBuiltins; - use crate::canonical::ToCanonical; + #[expect(deprecated)] + use crate::canonical::ToCanonical as _; use crate::scalar::Scalar; use crate::validity::Validity; #[test] fn fill_null_leading_none() { let arr = PrimitiveArray::from_option_iter([None, Some(8u8), None, Some(10), None]); + #[expect(deprecated)] let p = arr .into_array() .fill_null(Scalar::from(42u8)) .unwrap() .to_primitive(); assert_arrays_eq!(p, PrimitiveArray::from_iter([42u8, 8, 42, 10, 42])); - assert!(p.validity_mask().unwrap().all_true()); + assert!( + p.as_ref() + .validity() + .unwrap() + .execute_mask(p.as_ref().len(), &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .all_true() + ); } #[test] fn fill_null_all_none() { let arr = PrimitiveArray::from_option_iter([Option::::None, None, None, None, None]); + #[expect(deprecated)] let p = arr .into_array() .fill_null(Scalar::from(255u8)) .unwrap() .to_primitive(); assert_arrays_eq!(p, PrimitiveArray::from_iter([255u8, 255, 255, 255, 255])); - assert!(p.validity_mask().unwrap().all_true()); + assert!( + p.as_ref() + .validity() + .unwrap() + .execute_mask(p.as_ref().len(), &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .all_true() + ); } #[test] @@ -90,20 +109,36 @@ mod test { buffer![8u8, 10, 12, 14, 16], Validity::Array(BoolArray::from_iter([true, true, true, true, true]).into_array()), ); + #[expect(deprecated)] let p = arr .into_array() .fill_null(Scalar::from(255u8)) .unwrap() .to_primitive(); assert_arrays_eq!(p, PrimitiveArray::from_iter([8u8, 10, 12, 14, 16])); - assert!(p.validity_mask().unwrap().all_true()); + assert!( + p.as_ref() + .validity() + .unwrap() + .execute_mask(p.as_ref().len(), &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .all_true() + ); } #[test] fn fill_null_non_nullable() { let arr = buffer![8u8, 10, 12, 14, 16].into_array(); + #[expect(deprecated)] let p = arr.fill_null(Scalar::from(255u8)).unwrap().to_primitive(); assert_arrays_eq!(p, PrimitiveArray::from_iter([8u8, 10, 12, 14, 16])); - assert!(p.validity_mask().unwrap().all_true()); + assert!( + p.as_ref() + .validity() + .unwrap() + .execute_mask(p.as_ref().len(), &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .all_true() + ); } } diff --git a/vortex-array/src/arrays/primitive/compute/rules.rs b/vortex-array/src/arrays/primitive/compute/rules.rs index 99b0a7464d5..528e5e91520 100644 --- a/vortex-array/src/arrays/primitive/compute/rules.rs +++ b/vortex-array/src/arrays/primitive/compute/rules.rs @@ -12,10 +12,12 @@ use crate::arrays::PrimitiveArray; use crate::arrays::slice::SliceReduceAdaptor; use crate::optimizer::rules::ArrayParentReduceRule; use crate::optimizer::rules::ParentRuleSet; +use crate::scalar_fn::fns::cast::CastReduceAdaptor; use crate::scalar_fn::fns::mask::MaskReduceAdaptor; pub(crate) const RULES: ParentRuleSet = ParentRuleSet::new(&[ ParentRuleSet::lift(&PrimitiveMaskedValidityRule), + ParentRuleSet::lift(&CastReduceAdaptor(Primitive)), ParentRuleSet::lift(&MaskReduceAdaptor(Primitive)), ParentRuleSet::lift(&SliceReduceAdaptor(Primitive)), ]); diff --git a/vortex-array/src/arrays/primitive/compute/take/mod.rs b/vortex-array/src/arrays/primitive/compute/take/mod.rs index 230595039fa..4023991c65d 100644 --- a/vortex-array/src/arrays/primitive/compute/take/mod.rs +++ b/vortex-array/src/arrays/primitive/compute/take/mod.rs @@ -4,9 +4,6 @@ #[cfg(any(target_arch = "x86_64", target_arch = "x86"))] mod avx2; -#[cfg(vortex_nightly)] -mod portable; - use std::sync::LazyLock; use vortex_buffer::Buffer; @@ -32,23 +29,19 @@ use crate::validity::Validity; // Kernel selection happens on the first call to `take` and uses a combination of compile-time // and runtime feature detection to infer the best kernel for the platform. static PRIMITIVE_TAKE_KERNEL: LazyLock<&'static dyn TakeImpl> = LazyLock::new(|| { - cfg_if::cfg_if! { - if #[cfg(vortex_nightly)] { - // nightly codepath: use portable_simd kernel - &portable::TakeKernelPortableSimd - } else if #[cfg(target_arch = "x86_64")] { - // stable x86_64 path: use the optimized AVX2 kernel when available, falling - // back to scalar when not. - if is_x86_feature_detected!("avx2") { - &avx2::TakeKernelAVX2 - } else { - &TakeKernelScalar - } + #[cfg(any(target_arch = "x86_64", target_arch = "x86"))] + { + if is_x86_feature_detected!("avx2") { + &avx2::TakeKernelAVX2 } else { - // stable all other platforms: scalar kernel &TakeKernelScalar } } + + #[cfg(not(any(target_arch = "x86_64", target_arch = "x86")))] + { + &TakeKernelScalar + } }); trait TakeImpl: Send + Sync { @@ -60,7 +53,6 @@ trait TakeImpl: Send + Sync { ) -> VortexResult; } -#[allow(unused)] struct TakeKernelScalar; impl TakeImpl for TakeKernelScalar { @@ -113,7 +105,6 @@ impl TakeExecute for Primitive { } // Compiler may see this as unused based on enabled features -#[allow(unused)] #[inline(always)] fn take_primitive_scalar( buffer: &[T], @@ -145,6 +136,8 @@ mod test { use vortex_error::VortexExpect; use crate::IntoArray; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::arrays::BoolArray; use crate::arrays::PrimitiveArray; use crate::arrays::primitive::compute::take::take_primitive_scalar; @@ -171,17 +164,23 @@ mod test { ); let actual = values.take(indices.into_array()).unwrap(); assert_eq!( - actual.scalar_at(0).vortex_expect("no fail"), + actual + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .vortex_expect("no fail"), Scalar::from(Some(1)) ); // position 3 is null assert_eq!( - actual.scalar_at(1).vortex_expect("no fail"), + actual + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .vortex_expect("no fail"), Scalar::null_native::() ); // the third index is null assert_eq!( - actual.scalar_at(2).vortex_expect("no fail"), + actual + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .vortex_expect("no fail"), Scalar::null_native::() ); } diff --git a/vortex-array/src/arrays/primitive/compute/take/portable.rs b/vortex-array/src/arrays/primitive/compute/take/portable.rs deleted file mode 100644 index 2a9fec9cfb8..00000000000 --- a/vortex-array/src/arrays/primitive/compute/take/portable.rs +++ /dev/null @@ -1,318 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -//! An implementation of the Take kernel for primitive Arrays that uses -//! the nightly-only `portable_simd` feature. -//! -//! This is only enabled on non-x86_64 platforms and when using the nightly compiler for builds. - -#![allow(unused)] - -use std::mem::MaybeUninit; -use std::mem::size_of; -use std::mem::transmute; -use std::simd; -use std::simd::num::SimdUint; - -use multiversion::multiversion; -use num_traits::AsPrimitive; -use vortex_buffer::Alignment; -use vortex_buffer::Buffer; -use vortex_buffer::BufferMut; -use vortex_error::VortexResult; - -use crate::ArrayRef; -use crate::IntoArray; -use crate::array::ArrayView; -use crate::arrays::PrimitiveArray; -use crate::arrays::primitive::PrimitiveArrayExt; -use crate::arrays::primitive::compute::take::TakeImpl; -use crate::arrays::primitive::vtable::Primitive; -use crate::dtype::NativePType; -use crate::dtype::PType; -use crate::dtype::UnsignedPType; -use crate::dtype::half::f16; -use crate::match_each_native_simd_ptype; -use crate::match_each_unsigned_integer_ptype; -use crate::validity::Validity; - -pub(super) struct TakeKernelPortableSimd; - -impl TakeImpl for TakeKernelPortableSimd { - fn take( - &self, - array: ArrayView<'_, Primitive>, - unsigned_indices: ArrayView<'_, Primitive>, - validity: Validity, - ) -> VortexResult { - if array.ptype() == PType::F16 { - // Special handling for f16 to treat as opaque u16. - let decoded = match_each_unsigned_integer_ptype!(unsigned_indices.ptype(), |C| { - take_portable_simd::( - array.reinterpret_cast(PType::U16).as_slice(), - unsigned_indices.as_slice(), - ) - }); - Ok(PrimitiveArray::new(decoded, validity) - .reinterpret_cast(PType::F16) - .into_array()) - } else { - match_each_unsigned_integer_ptype!(unsigned_indices.ptype(), |C| { - match_each_native_simd_ptype!(array.ptype(), |V| { - let decoded = take_portable_simd::( - array.as_slice(), - unsigned_indices.as_slice(), - ); - Ok(PrimitiveArray::new(decoded, validity).into_array()) - }) - }) - } - } -} - -// --------------------------------------------------------------------------- -// Portable SIMD take algorithm -// --------------------------------------------------------------------------- - -/// SIMD types larger than the SIMD register size are beneficial for performance as this leads to -/// better instruction level parallelism. -const SIMD_WIDTH: usize = 64; - -/// Takes the specified indices into a new [`Buffer`] using portable SIMD. -/// -/// This function handles the type matching required to satisfy [`simd::SimdElement`] bounds. For -/// `f16` values, it reinterprets them as `u16` since `f16` doesn't implement `SimdElement`. -fn take_portable(buffer: &[T], indices: &[I]) -> Buffer { - if T::PTYPE == PType::F16 { - assert_eq!(size_of::(), size_of::()); - - // Since Rust does not actually support 16-bit floats, we first reinterpret the data as - // `u16` integers. - // SAFETY: We know that f16 has the same bit pattern as u16, so this transmute is fine. - let u16_slice: &[u16] = - unsafe { std::slice::from_raw_parts(buffer.as_ptr().cast(), buffer.len()) }; - return unsafe { take_with_indices(u16_slice, indices).transmute::() }; - } - - match_each_native_simd_ptype!(T::PTYPE, |TC| { - assert_eq!(size_of::(), size_of::()); - - // SAFETY: This is essentially a no-op that tricks the compiler into adding the - // `simd::SimdElement` bound we need to call `take_with_indices`. - let buffer: &[TC] = - unsafe { std::slice::from_raw_parts(buffer.as_ptr().cast::(), buffer.len()) }; - unsafe { take_with_indices(buffer, indices).transmute::() } - }) -} - -/// Helper that matches on index type and calls [`take_portable_simd`]. -/// -/// We separate this code out from above to add the [`simd::SimdElement`] constraint. -fn take_with_indices( - buffer: &[T], - indices: &[I], -) -> Buffer { - match_each_unsigned_integer_ptype!(I::PTYPE, |IC| { - let indices: &[IC] = - unsafe { std::slice::from_raw_parts(indices.as_ptr().cast::(), indices.len()) }; - take_portable_simd::(buffer, indices) - }) -} - -/// Takes elements from an array using SIMD indexing. -/// -/// Performs a gather operation that takes values at specified indices and returns them in a new -/// buffer. Uses SIMD instructions to process `LANE_COUNT` indices in parallel. -/// -/// Returns a [`Buffer`] where each element corresponds to `values[indices[i]]`. -#[multiversion(targets("x86_64+avx2", "x86_64+avx", "aarch64+neon"))] -fn take_portable_simd(values: &[T], indices: &[I]) -> Buffer -where - T: NativePType + simd::SimdElement, - I: UnsignedPType + simd::SimdElement, - simd::Simd: SimdUint = simd::Simd>, -{ - let indices_len = indices.len(); - - let mut buffer = BufferMut::::with_capacity_aligned( - indices_len, - Alignment::of::>(), - ); - - let buf_slice = buffer.spare_capacity_mut(); - - for chunk_idx in 0..(indices_len / LANE_COUNT) { - let offset = chunk_idx * LANE_COUNT; - let mask = simd::Mask::from_bitmask(u64::MAX); - let codes_chunk = simd::Simd::::from_slice(&indices[offset..]); - - let selection = simd::Simd::gather_select( - values, - mask, - codes_chunk.cast::(), - simd::Simd::::default(), - ); - - unsafe { - selection.store_select_unchecked( - transmute::<&mut [MaybeUninit], &mut [T]>(&mut buf_slice[offset..][..64]), - mask.cast(), - ); - } - } - - for idx in ((indices_len / LANE_COUNT) * LANE_COUNT)..indices_len { - unsafe { - buf_slice - .get_unchecked_mut(idx) - .write(values[indices[idx].as_()]); - } - } - - unsafe { - buffer.set_len(indices_len); - } - - // NOTE: if we don't do this, we pass back a Buffer which is over-aligned to the SIMD - // register width. The caller expects that this memory should be aligned to the value type - // so that we can slice it at value boundaries. - buffer = buffer.aligned(Alignment::of::()); - - buffer.freeze() -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn test_take_out_of_bounds() { - let indices = vec![2_000_000u32; 64]; - let values = vec![1i32]; - - let result = take_portable_simd::(&values, &indices); - assert_eq!(result.as_slice(), [0i32; 64]); - } - - /// Tests SIMD gather with a mix of sequential, strided, and repeated indices. This exercises - /// irregular access patterns that stress the gather operation. - #[test] - fn test_take_mixed_access_patterns() { - // Create a values array with distinct elements. - let values: Vec = (0..256).map(|i| i * 100).collect(); - - // Build indices with mixed patterns: - // - Sequential access (0, 1, 2, ...). - // - Strided access (0, 4, 8, ...). - // - Repeated indices (same index multiple times). - // - Reverse order. - let mut indices: Vec = Vec::with_capacity(200); - - // Sequential: indices 0..64. - indices.extend(0u32..64); - // Strided by 4: 0, 4, 8, ..., 252. - indices.extend((0u32..64).map(|i| i * 4)); - // Repeated: index 42 repeated 32 times. - indices.extend(std::iter::repeat(42u32).take(32)); - // Reverse: 255, 254, ..., 216. - indices.extend((216u32..256).rev()); - - let result = take_portable_simd::(&values, &indices); - let result_slice = result.as_slice(); - - // Verify sequential portion. - for i in 0..64 { - assert_eq!(result_slice[i], (i as i64) * 100, "sequential at index {i}"); - } - - // Verify strided portion. - for i in 0..64 { - assert_eq!( - result_slice[64 + i], - (i as i64) * 4 * 100, - "strided at index {i}" - ); - } - - // Verify repeated portion. - for i in 0..32 { - assert_eq!(result_slice[128 + i], 42 * 100, "repeated at index {i}"); - } - - // Verify reverse portion. - for i in 0..40 { - assert_eq!( - result_slice[160 + i], - (255 - i as i64) * 100, - "reverse at index {i}" - ); - } - } - - /// Tests that the scalar remainder path works correctly when the number of indices is not - /// evenly divisible by the SIMD lane count. - #[test] - fn test_take_with_remainder() { - let values: Vec = (0..1000).collect(); - - // Use 64 + 37 = 101 indices to test both the SIMD loop (64 elements) and the scalar - // remainder (37 elements). - let indices: Vec = (0u8..101).collect(); - - let result = take_portable_simd::(&values, &indices); - let result_slice = result.as_slice(); - - assert_eq!(result_slice.len(), 101); - - // Verify all elements. - for i in 0..101 { - assert_eq!(result_slice[i], i as u16, "mismatch at index {i}"); - } - - // Also test with exactly 1 remainder element. - let indices_one_remainder: Vec = (0u8..65).collect(); - let result_one = take_portable_simd::(&values, &indices_one_remainder); - assert_eq!(result_one.as_slice().len(), 65); - assert_eq!(result_one.as_slice()[64], 64); - } - - /// Tests gather with large 64-bit values and various index types to ensure no truncation - /// occurs during the operation. - #[test] - fn test_take_large_values_no_truncation() { - // Create values near the edges of i64 range. - let values: Vec = vec![ - i64::MIN, - i64::MIN + 1, - -1_000_000_000_000i64, - -1, - 0, - 1, - 1_000_000_000_000i64, - i64::MAX - 1, - i64::MAX, - ]; - - // Indices that access each value multiple times in different orders. - let indices: Vec = vec![ - 0, 8, 1, 7, 2, 6, 3, 5, 4, // Forward-backward interleaved. - 8, 8, 8, 0, 0, 0, // Repeated extremes. - 4, 4, 4, 4, 4, 4, 4, 4, // Repeated zero. - 0, 1, 2, 3, 4, 5, 6, 7, 8, // Sequential. - 8, 7, 6, 5, 4, 3, 2, 1, 0, // Reverse. - // Pad to 64 to ensure we hit the SIMD path. - 0, 1, 2, 3, 4, 5, 6, 7, 8, 0, 1, 2, 3, 4, 5, 6, 7, 8, 0, 1, 2, 3, - ]; - - let result = take_portable_simd::(&values, &indices); - let result_slice = result.as_slice(); - - // Verify each result matches the expected value. - for (i, &idx) in indices.iter().enumerate() { - assert_eq!( - result_slice[i], values[idx as usize], - "mismatch at position {i} for index {idx}" - ); - } - } -} diff --git a/vortex-array/src/arrays/primitive/vtable/mod.rs b/vortex-array/src/arrays/primitive/vtable/mod.rs index 806c95b6a8f..cb28f96631e 100644 --- a/vortex-array/src/arrays/primitive/vtable/mod.rs +++ b/vortex-array/src/arrays/primitive/vtable/mod.rs @@ -27,6 +27,7 @@ use std::hash::Hasher; use vortex_buffer::Alignment; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::Precision; use crate::array::ArrayId; @@ -57,7 +58,8 @@ impl VTable for Primitive { type ValidityVTable = Self; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.primitive"); + *ID } fn nbuffers(_array: ArrayView<'_, Self>) -> usize { @@ -101,7 +103,7 @@ impl VTable for Primitive { data.len(), len ); - let validity = crate::array::child_to_validity(&slots[0], *nullability); + let validity = crate::array::child_to_validity(slots[0].as_ref(), *nullability); if let Some(validity_len) = validity.maybe_len() { vortex_ensure!( validity_len == len, @@ -203,10 +205,6 @@ impl VTable for Primitive { #[derive(Clone, Debug)] pub struct Primitive; -impl Primitive { - pub const ID: ArrayId = ArrayId::new_ref("vortex.primitive"); -} - #[cfg(test)] mod tests { use vortex_buffer::ByteBufferMut; diff --git a/vortex-array/src/arrays/scalar_fn/array.rs b/vortex-array/src/arrays/scalar_fn/array.rs index 301713761b5..4a82951ebdb 100644 --- a/vortex-array/src/arrays/scalar_fn/array.rs +++ b/vortex-array/src/arrays/scalar_fn/array.rs @@ -12,7 +12,7 @@ use crate::ArrayRef; use crate::array::Array; use crate::array::ArrayParts; use crate::array::TypedArrayRef; -use crate::arrays::ScalarFnVTable; +use crate::arrays::ScalarFn; use crate::scalar_fn::ScalarFnRef; // ScalarFnArray has a variable number of slots (one per child) @@ -43,14 +43,13 @@ impl ScalarFnData { } /// Get the scalar function bound to this array. - #[allow(clippy::same_name_method)] #[inline(always)] pub fn scalar_fn(&self) -> &ScalarFnRef { &self.scalar_fn } } -pub trait ScalarFnArrayExt: TypedArrayRef { +pub trait ScalarFnArrayExt: TypedArrayRef { fn scalar_fn(&self) -> &ScalarFnRef { &self.scalar_fn } @@ -65,12 +64,10 @@ pub trait ScalarFnArrayExt: TypedArrayRef { self.as_ref().slots().len() } - #[allow(clippy::same_name_method)] fn nchildren(&self) -> usize { self.child_count() } - #[allow(clippy::same_name_method)] fn get_child(&self, idx: usize) -> &ArrayRef { self.child_at(idx) } @@ -83,9 +80,9 @@ pub trait ScalarFnArrayExt: TypedArrayRef { self.iter_children().cloned().collect() } } -impl> ScalarFnArrayExt for T {} +impl> ScalarFnArrayExt for T {} -impl Array { +impl Array { /// Create a new ScalarFnArray from a scalar function and its children. pub fn try_new( scalar_fn: ScalarFnRef, @@ -95,7 +92,7 @@ impl Array { let arg_dtypes: Vec<_> = children.iter().map(|c| c.dtype().clone()).collect(); let dtype = scalar_fn.return_dtype(&arg_dtypes)?; let data = ScalarFnData::build(scalar_fn.clone(), children.clone(), len)?; - let vtable = ScalarFnVTable { scalar_fn }; + let vtable = ScalarFn { id: scalar_fn.id() }; Ok(unsafe { Array::from_parts_unchecked( ArrayParts::new(vtable, dtype, len, data) diff --git a/vortex-array/src/arrays/scalar_fn/mod.rs b/vortex-array/src/arrays/scalar_fn/mod.rs index 9d2e9e66ed8..6c508dcad96 100644 --- a/vortex-array/src/arrays/scalar_fn/mod.rs +++ b/vortex-array/src/arrays/scalar_fn/mod.rs @@ -2,10 +2,10 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors mod array; +pub mod plugin; mod rules; mod vtable; pub use array::ScalarFnArrayExt; -pub use array::ScalarFnData; pub use vtable::ScalarFnFactoryExt; pub use vtable::*; diff --git a/vortex-array/src/arrays/scalar_fn/plugin.rs b/vortex-array/src/arrays/scalar_fn/plugin.rs new file mode 100644 index 00000000000..c685503a370 --- /dev/null +++ b/vortex-array/src/arrays/scalar_fn/plugin.rs @@ -0,0 +1,92 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +use vortex_error::VortexResult; +use vortex_session::VortexSession; + +use crate::ArrayId; +use crate::ArrayPlugin; +use crate::ArrayRef; +use crate::IntoArray; +use crate::arrays::ScalarFnArray; +use crate::arrays::scalar_fn::ExactScalarFn; +use crate::arrays::scalar_fn::ScalarFnArrayView; +use crate::buffer::BufferHandle; +use crate::dtype::DType; +use crate::scalar_fn::ScalarFnVTable; +use crate::scalar_fn::TypedScalarFnInstance; +use crate::serde::ArrayChildren; + +/// An adapter for enabling a scalar function to be serialized as an array. +pub struct ScalarFnArrayPlugin(V); + +impl ScalarFnArrayPlugin { + /// Create a new plugin for the given scalar function vtable. + pub fn new(vtable: V) -> Self { + Self(vtable) + } +} + +pub trait ScalarFnArrayVTable: ScalarFnVTable { + /// Serialize metadata for storing the scalar function as an array. + /// + /// Notably, this metadata needs enough information to reconstruct the child DTypes, as well + /// as the scalar function's own options. + fn serialize( + &self, + view: &ScalarFnArrayView, + session: &VortexSession, + ) -> VortexResult>>; + + /// Deserialize a scalar function array from its serialized components. + fn deserialize( + &self, + dtype: &DType, + len: usize, + metadata: &[u8], + children: &dyn ArrayChildren, + session: &VortexSession, + ) -> VortexResult>; +} + +/// The parts used to construct a ScalarFnArray. +pub struct ScalarFnArrayParts { + pub options: V::Options, + pub children: Vec, +} + +impl ArrayPlugin for ScalarFnArrayPlugin { + fn id(&self) -> ArrayId { + self.0.id() + } + + fn serialize( + &self, + array: &ArrayRef, + session: &VortexSession, + ) -> VortexResult>> { + // We serialize the scalar function options, along with any scalar function array data. + let scalar_fn = array.as_::>(); + ::serialize(&self.0, &scalar_fn, session) + } + + fn deserialize( + &self, + dtype: &DType, + len: usize, + metadata: &[u8], + _buffers: &[BufferHandle], + children: &dyn ArrayChildren, + session: &VortexSession, + ) -> VortexResult { + let parts = ::deserialize( + &self.0, dtype, len, metadata, children, session, + )?; + Ok(ScalarFnArray::try_new( + TypedScalarFnInstance::new(self.0.clone(), parts.options).erased(), + parts.children, + len, + )? + .into_array()) + } +} diff --git a/vortex-array/src/arrays/scalar_fn/rules.rs b/vortex-array/src/arrays/scalar_fn/rules.rs index d0cb0efdbf4..1e9563cf9de 100644 --- a/vortex-array/src/arrays/scalar_fn/rules.rs +++ b/vortex-array/src/arrays/scalar_fn/rules.rs @@ -9,14 +9,13 @@ use vortex_error::VortexExpect; use vortex_error::VortexResult; use crate::ArrayRef; -use crate::Canonical; use crate::IntoArray; use crate::array::ArrayView; use crate::arrays::Constant; use crate::arrays::ConstantArray; use crate::arrays::Filter; +use crate::arrays::ScalarFn; use crate::arrays::ScalarFnArray; -use crate::arrays::ScalarFnVTable; use crate::arrays::Slice; use crate::arrays::StructArray; use crate::arrays::scalar_fn::ScalarFnArrayExt; @@ -32,13 +31,10 @@ use crate::scalar_fn::ScalarFnRef; use crate::scalar_fn::fns::pack::Pack; use crate::validity::Validity; -pub(super) const RULES: ReduceRuleSet = ReduceRuleSet::new(&[ - &ScalarFnPackToStructRule, - &ScalarFnConstantRule, - &ScalarFnAbstractReduceRule, -]); +pub(super) const RULES: ReduceRuleSet = + ReduceRuleSet::new(&[&ScalarFnPackToStructRule, &ScalarFnAbstractReduceRule]); -pub(super) const PARENT_RULES: ParentRuleSet = ParentRuleSet::new(&[ +pub(super) const PARENT_RULES: ParentRuleSet = ParentRuleSet::new(&[ ParentRuleSet::lift(&ScalarFnUnaryFilterPushDownRule), ParentRuleSet::lift(&ScalarFnSliceReduceRule), ]); @@ -46,8 +42,8 @@ pub(super) const PARENT_RULES: ParentRuleSet = ParentRuleSet::ne /// Converts a ScalarFnArray with Pack into a StructArray directly. #[derive(Debug)] struct ScalarFnPackToStructRule; -impl ArrayReduceRule for ScalarFnPackToStructRule { - fn reduce(&self, array: ArrayView<'_, ScalarFnVTable>) -> VortexResult> { +impl ArrayReduceRule for ScalarFnPackToStructRule { + fn reduce(&self, array: ArrayView<'_, ScalarFn>) -> VortexResult> { let Some(pack_options) = array.scalar_fn().as_opt::() else { return Ok(None); }; @@ -69,30 +65,14 @@ impl ArrayReduceRule for ScalarFnPackToStructRule { } } -#[derive(Debug)] -struct ScalarFnConstantRule; -impl ArrayReduceRule for ScalarFnConstantRule { - fn reduce(&self, array: ArrayView<'_, ScalarFnVTable>) -> VortexResult> { - if !array.children().iter().all(|c| c.is::()) { - return Ok(None); - } - if array.is_empty() { - Ok(Some(Canonical::empty(array.dtype()).into_array())) - } else { - let result = array.array().scalar_at(0)?; - Ok(Some(ConstantArray::new(result, array.len()).into_array())) - } - } -} - #[derive(Debug)] struct ScalarFnSliceReduceRule; -impl ArrayParentReduceRule for ScalarFnSliceReduceRule { +impl ArrayParentReduceRule for ScalarFnSliceReduceRule { type Parent = Slice; fn reduce_parent( &self, - array: ArrayView<'_, ScalarFnVTable>, + array: ArrayView<'_, ScalarFn>, parent: ArrayView<'_, Slice>, _child_idx: usize, ) -> VortexResult> { @@ -111,8 +91,8 @@ impl ArrayParentReduceRule for ScalarFnSliceReduceRule { #[derive(Debug)] struct ScalarFnAbstractReduceRule; -impl ArrayReduceRule for ScalarFnAbstractReduceRule { - fn reduce(&self, array: ArrayView<'_, ScalarFnVTable>) -> VortexResult> { +impl ArrayReduceRule for ScalarFnAbstractReduceRule { + fn reduce(&self, array: ArrayView<'_, ScalarFn>) -> VortexResult> { if let Some(reduced) = array .scalar_fn() .reduce(array.as_ref(), &ArrayReduceCtx { len: array.len() })? @@ -139,8 +119,7 @@ impl ReduceNode for ArrayRef { } fn scalar_fn(&self) -> Option<&ScalarFnRef> { - self.as_opt::() - .map(|a| a.data().scalar_fn()) + self.as_opt::().map(|a| a.data().scalar_fn()) } fn child(&self, idx: usize) -> ReduceNodeRef { @@ -184,12 +163,12 @@ impl ReduceCtx for ArrayReduceCtx { #[derive(Debug)] struct ScalarFnUnaryFilterPushDownRule; -impl ArrayParentReduceRule for ScalarFnUnaryFilterPushDownRule { +impl ArrayParentReduceRule for ScalarFnUnaryFilterPushDownRule { type Parent = Filter; fn reduce_parent( &self, - child: ArrayView<'_, ScalarFnVTable>, + child: ArrayView<'_, ScalarFn>, parent: ArrayView<'_, Filter>, _child_idx: usize, ) -> VortexResult> { diff --git a/vortex-array/src/arrays/scalar_fn/vtable/mod.rs b/vortex-array/src/arrays/scalar_fn/vtable/mod.rs index 996fbee1046..34c472eac07 100644 --- a/vortex-array/src/arrays/scalar_fn/vtable/mod.rs +++ b/vortex-array/src/arrays/scalar_fn/vtable/mod.rs @@ -41,17 +41,16 @@ use crate::scalar_fn::Arity; use crate::scalar_fn::ChildName; use crate::scalar_fn::ExecutionArgs; use crate::scalar_fn::ScalarFnId; -use crate::scalar_fn::ScalarFnRef; use crate::scalar_fn::ScalarFnVTableExt; use crate::scalar_fn::VecExecutionArgs; use crate::serde::ArrayChildren; -/// A [`ScalarFnVTable`]-encoded Vortex array. -pub type ScalarFnArray = Array; +/// A [`ScalarFn`]-encoded Vortex array. +pub type ScalarFnArray = Array; #[derive(Clone, Debug)] -pub struct ScalarFnVTable { - pub(super) scalar_fn: ScalarFnRef, +pub struct ScalarFn { + pub(super) id: ScalarFnId, } impl ArrayHash for ScalarFnData { @@ -66,13 +65,13 @@ impl ArrayEq for ScalarFnData { } } -impl VTable for ScalarFnVTable { +impl VTable for ScalarFn { type ArrayData = ScalarFnData; type OperationsVTable = Self; type ValidityVTable = Self; fn id(&self) -> ArrayId { - self.scalar_fn.id() + self.id } fn validate( @@ -83,7 +82,7 @@ impl VTable for ScalarFnVTable { slots: &[Option], ) -> VortexResult<()> { vortex_ensure!( - data.scalar_fn == self.scalar_fn, + data.scalar_fn.id() == self.id, "ScalarFnArray data scalar_fn does not match vtable" ); vortex_ensure!( @@ -97,7 +96,7 @@ impl VTable for ScalarFnVTable { .map(|c| c.dtype().clone()) .collect_vec(); vortex_ensure!( - self.scalar_fn.return_dtype(&child_dtypes)? == *dtype, + data.scalar_fn.return_dtype(&child_dtypes)? == *dtype, "ScalarFnArray dtype does not match scalar function return dtype" ); Ok(()) @@ -128,7 +127,6 @@ impl VTable for ScalarFnVTable { _dtype: &DType, _len: usize, _metadata: &[u8], - _buffers: &[BufferHandle], _children: &dyn ArrayChildren, _session: &VortexSession, @@ -175,7 +173,7 @@ pub trait ScalarFnFactoryExt: scalar_fn::ScalarFnVTable { options: Self::Options, children: impl Into>, ) -> VortexResult { - let scalar_fn = scalar_fn::ScalarFn::new(self.clone(), options).erased(); + let scalar_fn = scalar_fn::TypedScalarFnInstance::new(self.clone(), options).erased(); let children = children.into(); vortex_ensure!( @@ -189,7 +187,7 @@ pub trait ScalarFnFactoryExt: scalar_fn::ScalarFnVTable { let data = ScalarFnData { scalar_fn: scalar_fn.clone(), }; - let vtable = ScalarFnVTable { scalar_fn }; + let vtable = ScalarFn { id: scalar_fn.id() }; Ok(unsafe { Array::from_parts_unchecked( ArrayParts::new(vtable, dtype, len, data) @@ -205,14 +203,14 @@ impl ScalarFnFactoryExt for V {} #[derive(Debug)] pub struct AnyScalarFn; impl Matcher for AnyScalarFn { - type Match<'a> = ArrayView<'a, ScalarFnVTable>; + type Match<'a> = ArrayView<'a, ScalarFn>; fn matches(array: &ArrayRef) -> bool { - array.is::() + array.is::() } fn try_match(array: &ArrayRef) -> Option> { - array.as_opt::() + array.as_opt::() } } @@ -224,7 +222,7 @@ impl Matcher for ExactScalarFn { type Match<'a> = ScalarFnArrayView<'a, F>; fn matches(array: &ArrayRef) -> bool { - if let Some(scalar_fn_array) = array.as_opt::() { + if let Some(scalar_fn_array) = array.as_opt::() { scalar_fn_array.data().scalar_fn().is::() } else { false @@ -232,7 +230,7 @@ impl Matcher for ExactScalarFn { } fn try_match(array: &ArrayRef) -> Option> { - let scalar_fn_array = array.as_opt::()?; + let scalar_fn_array = array.as_opt::()?; let scalar_fn_data = scalar_fn_array.data(); let scalar_fn = scalar_fn_data.scalar_fn().downcast_ref::()?; Some(ScalarFnArrayView { @@ -286,7 +284,7 @@ impl scalar_fn::ScalarFnVTable for ArrayExpr { type Options = FakeEq; fn id(&self) -> ScalarFnId { - ScalarFnId::from("vortex.array") + ScalarFnId::new("vortex.array") } fn arity(&self, _options: &Self::Options) -> Arity { diff --git a/vortex-array/src/arrays/scalar_fn/vtable/operations.rs b/vortex-array/src/arrays/scalar_fn/vtable/operations.rs index bcb0368772a..1f710af3cdf 100644 --- a/vortex-array/src/arrays/scalar_fn/vtable/operations.rs +++ b/vortex-array/src/arrays/scalar_fn/vtable/operations.rs @@ -5,40 +5,37 @@ use vortex_error::VortexResult; use crate::ExecutionCtx; use crate::IntoArray; -use crate::LEGACY_SESSION; -use crate::VortexSessionExecute; use crate::array::ArrayView; use crate::array::OperationsVTable; use crate::arrays::ConstantArray; use crate::arrays::scalar_fn::ScalarFnArrayExt; -use crate::arrays::scalar_fn::vtable::ScalarFnVTable; +use crate::arrays::scalar_fn::vtable::ScalarFn; use crate::columnar::Columnar; use crate::scalar::Scalar; use crate::scalar_fn::VecExecutionArgs; -impl OperationsVTable for ScalarFnVTable { +impl OperationsVTable for ScalarFn { fn scalar_at( - array: ArrayView<'_, ScalarFnVTable>, + array: ArrayView<'_, ScalarFn>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { let inputs: Vec<_> = array .children() .iter() - .map(|child| Ok(ConstantArray::new(child.scalar_at(index)?, 1).into_array())) + .map(|child| Ok(ConstantArray::new(child.execute_scalar(index, ctx)?, 1).into_array())) .collect::>()?; - let mut ctx = LEGACY_SESSION.create_execution_ctx(); let args = VecExecutionArgs::new(inputs, 1); - let result = array.scalar_fn().execute(&args, &mut ctx)?; + let result = array.scalar_fn().execute(&args, ctx)?; - let scalar = match result.execute::(&mut ctx)? { + let scalar = match result.execute::(ctx)? { Columnar::Canonical(arr) => { tracing::info!( "Scalar function {} returned non-constant array from execution over all scalar inputs", array.scalar_fn(), ); - arr.into_array().scalar_at(0)? + arr.into_array().execute_scalar(0, ctx)? } Columnar::Constant(constant) => constant.scalar().clone(), }; @@ -61,12 +58,15 @@ mod tests { use vortex_buffer::buffer; use vortex_error::VortexResult; + use crate::Canonical; use crate::IntoArray; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::arrays::BoolArray; use crate::arrays::PrimitiveArray; use crate::arrays::ScalarFnArray; use crate::assert_arrays_eq; - use crate::scalar_fn::ScalarFn; + use crate::scalar_fn::TypedScalarFnInstance; use crate::scalar_fn::fns::binary::Binary; use crate::scalar_fn::fns::operators::Operator; use crate::validity::Validity; @@ -76,10 +76,13 @@ mod tests { let lhs = buffer![1i32, 2, 3].into_array(); let rhs = buffer![10i32, 20, 30].into_array(); - let scalar_fn = ScalarFn::new(Binary, Operator::Add).erased(); + let scalar_fn = TypedScalarFnInstance::new(Binary, Operator::Add).erased(); let scalar_fn_array = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs], 3)?; - let result = scalar_fn_array.to_canonical()?.into_array(); + let result = scalar_fn_array + .into_array() + .execute::(&mut LEGACY_SESSION.create_execution_ctx())? + .into_array(); let expected = buffer![11i32, 22, 33].into_array(); assert_arrays_eq!(result, expected); @@ -91,10 +94,13 @@ mod tests { let lhs = buffer![2i32, 3, 4].into_array(); let rhs = buffer![5i32, 6, 7].into_array(); - let scalar_fn = ScalarFn::new(Binary, Operator::Mul).erased(); + let scalar_fn = TypedScalarFnInstance::new(Binary, Operator::Mul).erased(); let scalar_fn_array = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs], 3)?; - let result = scalar_fn_array.to_canonical()?.into_array(); + let result = scalar_fn_array + .into_array() + .execute::(&mut LEGACY_SESSION.create_execution_ctx())? + .into_array(); let expected = buffer![10i32, 18, 28].into_array(); assert_arrays_eq!(result, expected); @@ -110,10 +116,13 @@ mod tests { ) .into_array(); - let scalar_fn = ScalarFn::new(Binary, Operator::Add).erased(); + let scalar_fn = TypedScalarFnInstance::new(Binary, Operator::Add).erased(); let scalar_fn_array = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs], 3)?; - let result = scalar_fn_array.to_canonical()?.into_array(); + let result = scalar_fn_array + .into_array() + .execute::(&mut LEGACY_SESSION.create_execution_ctx())? + .into_array(); let expected = PrimitiveArray::new( buffer![11i32, 0, 33], Validity::from_iter([true, false, true]), @@ -129,10 +138,13 @@ mod tests { let lhs = buffer![1i32, 5, 3].into_array(); let rhs = buffer![2i32, 5, 1].into_array(); - let scalar_fn = ScalarFn::new(Binary, Operator::Eq).erased(); + let scalar_fn = TypedScalarFnInstance::new(Binary, Operator::Eq).erased(); let scalar_fn_array = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs], 3)?; - let result = scalar_fn_array.to_canonical()?.into_array(); + let result = scalar_fn_array + .into_array() + .execute::(&mut LEGACY_SESSION.create_execution_ctx())? + .into_array(); let expected = BoolArray::from_iter([false, true, false]).into_array(); assert_arrays_eq!(result, expected); diff --git a/vortex-array/src/arrays/scalar_fn/vtable/validity.rs b/vortex-array/src/arrays/scalar_fn/vtable/validity.rs index 2884530f8ba..2ac376155e3 100644 --- a/vortex-array/src/arrays/scalar_fn/vtable/validity.rs +++ b/vortex-array/src/arrays/scalar_fn/vtable/validity.rs @@ -12,10 +12,10 @@ use crate::array::ValidityVTable; use crate::arrays::scalar_fn::ScalarFnArrayExt; use crate::arrays::scalar_fn::vtable::ArrayExpr; use crate::arrays::scalar_fn::vtable::FakeEq; -use crate::arrays::scalar_fn::vtable::ScalarFnVTable; +use crate::arrays::scalar_fn::vtable::ScalarFn; use crate::expr::Expression; use crate::expr::lit; -use crate::scalar_fn::ScalarFn; +use crate::scalar_fn::TypedScalarFnInstance; use crate::scalar_fn::VecExecutionArgs; use crate::scalar_fn::fns::literal::Literal; use crate::scalar_fn::fns::root::Root; @@ -50,15 +50,18 @@ fn execute_expr(expr: &Expression, row_count: usize) -> VortexResult { Ok(expr.scalar_fn().execute(&args, &mut ctx)?.into_array()) } -impl ValidityVTable for ScalarFnVTable { - fn validity(array: ArrayView<'_, ScalarFnVTable>) -> VortexResult { +impl ValidityVTable for ScalarFn { + fn validity(array: ArrayView<'_, ScalarFn>) -> VortexResult { let inputs: Vec<_> = array .iter_children() .map(|child| { if let Some(scalar) = child.as_constant() { return Ok(lit(scalar)); } - Expression::try_new(ScalarFn::new(ArrayExpr, FakeEq(child.clone())).erased(), []) + Expression::try_new( + TypedScalarFnInstance::new(ArrayExpr, FakeEq(child.clone())).erased(), + [], + ) }) .collect::>()?; diff --git a/vortex-array/src/arrays/shared/array.rs b/vortex-array/src/arrays/shared/array.rs index 37d704db4d1..5e90aece98e 100644 --- a/vortex-array/src/arrays/shared/array.rs +++ b/vortex-array/src/arrays/shared/array.rs @@ -41,7 +41,7 @@ impl Display for SharedData { } } -#[allow(async_fn_in_trait)] +#[expect(async_fn_in_trait)] pub trait SharedArrayExt: TypedArrayRef { fn source(&self) -> &ArrayRef { self.as_ref().slots()[SOURCE_SLOT] diff --git a/vortex-array/src/arrays/shared/vtable.rs b/vortex-array/src/arrays/shared/vtable.rs index 6be759c83eb..52245aeb198 100644 --- a/vortex-array/src/arrays/shared/vtable.rs +++ b/vortex-array/src/arrays/shared/vtable.rs @@ -7,6 +7,7 @@ use vortex_error::VortexExpect; use vortex_error::VortexResult; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::ArrayEq; use crate::ArrayHash; @@ -37,10 +38,6 @@ pub type SharedArray = Array; #[derive(Clone, Debug)] pub struct Shared; -impl Shared { - pub const ID: ArrayId = ArrayId::new_ref("vortex.shared"); -} - impl ArrayHash for SharedData { fn array_hash(&self, _state: &mut H, _precision: Precision) {} } @@ -55,9 +52,9 @@ impl VTable for Shared { type ArrayData = SharedData; type OperationsVTable = Self; type ValidityVTable = Self; - fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.shared"); + *ID } fn validate( @@ -121,9 +118,9 @@ impl OperationsVTable for Shared { fn scalar_at( array: ArrayView<'_, Shared>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { - array.current_array_ref().scalar_at(index) + array.current_array_ref().execute_scalar(index, ctx) } } diff --git a/vortex-array/src/arrays/slice/vtable.rs b/vortex-array/src/arrays/slice/vtable.rs index 2ac41a358b9..56dbcbcd915 100644 --- a/vortex-array/src/arrays/slice/vtable.rs +++ b/vortex-array/src/arrays/slice/vtable.rs @@ -13,6 +13,7 @@ use vortex_error::vortex_bail; use vortex_error::vortex_ensure; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::AnyCanonical; use crate::ArrayEq; @@ -45,10 +46,6 @@ pub type SliceArray = Array; #[derive(Clone, Debug)] pub struct Slice; -impl Slice { - pub const ID: ArrayId = ArrayId::new_ref("vortex.slice"); -} - impl ArrayHash for SliceData { fn array_hash(&self, state: &mut H, _precision: Precision) { self.range.start.hash(state); @@ -66,9 +63,9 @@ impl VTable for Slice { type ArrayData = SliceData; type OperationsVTable = Self; type ValidityVTable = Self; - fn id(&self) -> ArrayId { - Slice::ID + static ID: CachedId = CachedId::new("vortex.slice"); + *ID } fn validate( @@ -166,9 +163,9 @@ impl OperationsVTable for Slice { fn scalar_at( array: ArrayView<'_, Slice>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { - array.child().scalar_at(array.range.start + index) + array.child().execute_scalar(array.range.start + index, ctx) } } diff --git a/vortex-array/src/arrays/struct_/array.rs b/vortex-array/src/arrays/struct_/array.rs index 3d147f9c85e..02df75aa659 100644 --- a/vortex-array/src/arrays/struct_/array.rs +++ b/vortex-array/src/arrays/struct_/array.rs @@ -1,11 +1,14 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +use std::borrow::Borrow; use std::iter::once; use std::sync::Arc; +use itertools::Itertools; use vortex_error::VortexExpect; use vortex_error::VortexResult; +use vortex_error::vortex_bail; use vortex_error::vortex_err; use crate::ArrayRef; @@ -16,6 +19,7 @@ use crate::array::EmptyArrayData; use crate::array::TypedArrayRef; use crate::array::child_to_validity; use crate::array::validity_to_child; +use crate::arrays::ChunkedArray; use crate::arrays::Struct; use crate::dtype::DType; use crate::dtype::FieldName; @@ -52,7 +56,7 @@ pub(super) const FIELDS_OFFSET: usize = 1; /// use vortex_array::arrays::{StructArray, BoolArray}; /// use vortex_array::validity::Validity; /// use vortex_array::dtype::FieldNames; -/// use vortex_array::IntoArray; +/// use vortex_array::{IntoArray, LEGACY_SESSION, VortexSessionExecute}; /// use vortex_buffer::buffer; /// /// // Create struct with all non-null fields but struct-level nulls @@ -66,13 +70,14 @@ pub(super) const FIELDS_OFFSET: usize = 1; /// 2, /// Validity::Array(BoolArray::from_iter([true, false]).into_array()), // row 1 is null /// ).unwrap(); +/// let mut ctx = LEGACY_SESSION.create_execution_ctx(); /// /// // Row 0 is valid - returns a struct scalar with field values -/// let row0 = struct_array.scalar_at(0).unwrap(); +/// let row0 = struct_array.execute_scalar(0, &mut ctx).unwrap(); /// assert!(!row0.is_null()); /// /// // Row 1 is null at struct level - returns null even though fields have values -/// let row1 = struct_array.scalar_at(1).unwrap(); +/// let row1 = struct_array.execute_scalar(1, &mut ctx).unwrap(); /// assert!(row1.is_null()); /// ``` /// @@ -86,7 +91,7 @@ pub(super) const FIELDS_OFFSET: usize = 1; /// use vortex_array::arrays::struct_::StructArrayExt; /// use vortex_array::validity::Validity; /// use vortex_array::dtype::FieldNames; -/// use vortex_array::IntoArray; +/// use vortex_array::{IntoArray, LEGACY_SESSION, VortexSessionExecute}; /// use vortex_buffer::buffer; /// /// // Create struct with duplicate "data" field names @@ -102,7 +107,8 @@ pub(super) const FIELDS_OFFSET: usize = 1; /// /// // field_by_name returns the FIRST "data" field /// let first_data = struct_array.unmasked_field_by_name("data").unwrap(); -/// assert_eq!(first_data.scalar_at(0).unwrap(), 1i32.into()); +/// let mut ctx = LEGACY_SESSION.create_execution_ctx(); +/// assert_eq!(first_data.execute_scalar(0, &mut ctx).unwrap(), 1i32.into()); /// ``` /// /// ## Field Operations @@ -178,7 +184,10 @@ pub trait StructArrayExt: TypedArrayRef { } fn struct_validity(&self) -> Validity { - child_to_validity(&self.as_ref().slots()[VALIDITY_SLOT], self.nullability()) + child_to_validity( + self.as_ref().slots()[VALIDITY_SLOT].as_ref(), + self.nullability(), + ) } fn iter_unmasked_fields(&self) -> impl Iterator + '_ { @@ -428,9 +437,7 @@ impl Array { }; Some((new_array, field)) } -} -impl Array { pub fn with_column(&self, name: impl Into, array: ArrayRef) -> VortexResult { let name = name.into(); let struct_dtype = self.struct_fields(); @@ -451,4 +458,70 @@ impl Array { pub fn remove_column_owned(&self, name: impl Into) -> Option<(Self, ArrayRef)> { self.remove_column(name) } + + pub fn try_concat(chunks: impl IntoIterator) -> VortexResult + where + T: Borrow>, + { + let mut it = chunks.into_iter(); + let Some(first) = it.next() else { + vortex_bail!("cannot concat empty iterator of arrays"); + }; + let first_dtype = first.borrow().dtype().clone(); + let struct_fields = first_dtype.as_struct_fields().clone(); + let names = struct_fields.names(); + + let it = [first].into_iter().chain(it); + let (field_arrays_per_chunk, validities) = it + .map(|chunk| { + let chunk = chunk.borrow(); + if &first_dtype != chunk.dtype() { + vortex_bail!( + "cannot concatenate struct arrays with differing dtypes: {}, {}", + first_dtype, + chunk.dtype(), + ); + } + + let fields = names + .iter() + .map(|name| { + chunk + .unmasked_field_by_name(name) + .vortex_expect("field exists because it is in dtype") + .clone() + }) + .collect::>(); + let validity = chunk.validity()?; + + Ok((fields, (validity, chunk.len()))) + }) + .process_results(|iter| iter.unzip::<_, _, Vec<_>, Vec<_>>())?; + + let field_arrays = struct_fields + .fields() + .enumerate() + .map(|(i, dtype)| { + // SAFETY: We establish above that every array has the same type. + let chunks = field_arrays_per_chunk + .iter() + .map(|x| x[i].clone()) + .collect(); + unsafe { ChunkedArray::new_unchecked(chunks, dtype) }.into_array() + }) + .collect::>(); + let len = validities.iter().map(|(_v, len)| len).sum(); + let validity = Validity::concat(validities).vortex_expect("verified non-empty above"); + + // SAFETY: + // + // 1. The field arrays, by construction, have the type specified in fields. + // + // 2. Each Array has a valid len, therefore the sum of those lens should be valid + // for the concatenation of each field. + // + // 3. Each Array has a valid validity, so the concatenation of those validities has + // the correct length and dtype harmony. + Ok(unsafe { Array::::new_unchecked(field_arrays, struct_fields, len, validity) }) + } } diff --git a/vortex-array/src/arrays/struct_/compute/cast.rs b/vortex-array/src/arrays/struct_/compute/cast.rs index 5e63cdc2092..ddb3cbeadf2 100644 --- a/vortex-array/src/arrays/struct_/compute/cast.rs +++ b/vortex-array/src/arrays/struct_/compute/cast.rs @@ -22,7 +22,7 @@ impl CastKernel for Struct { fn cast( array: ArrayView<'_, Struct>, dtype: &DType, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult> { let Some(target_sdtype) = dtype.as_struct_fields_opt() else { return Ok(None); @@ -73,7 +73,7 @@ impl CastKernel for Struct { let validity = array .validity()? - .cast_nullability(dtype.nullability(), array.len())?; + .cast_nullability(dtype.nullability(), array.len(), ctx)?; StructArray::try_new( target_sdtype.names().clone(), @@ -91,7 +91,8 @@ mod tests { use vortex_buffer::buffer; use crate::IntoArray; - use crate::ToCanonical; + #[expect(deprecated)] + use crate::ToCanonical as _; use crate::arrays::PrimitiveArray; use crate::arrays::StructArray; use crate::arrays::VarBinArray; @@ -209,7 +210,9 @@ mod tests { .unwrap(); assert_eq!(result.dtype(), &target_dtype); assert_eq!(result.len(), 3); - assert_eq!(result.to_struct().struct_fields().nfields(), 2); + #[expect(deprecated)] + let nfields = result.to_struct().struct_fields().nfields(); + assert_eq!(nfields, 2); } #[test] @@ -238,6 +241,8 @@ mod tests { .unwrap(); assert_eq!(result.dtype(), &target_dtype); assert_eq!(result.len(), 3); - assert_eq!(result.to_struct().struct_fields().nfields(), 3); + #[expect(deprecated)] + let nfields = result.to_struct().struct_fields().nfields(); + assert_eq!(nfields, 3); } } diff --git a/vortex-array/src/arrays/struct_/compute/mod.rs b/vortex-array/src/arrays/struct_/compute/mod.rs index 6d6e8c5b788..9ddee9496aa 100644 --- a/vortex-array/src/arrays/struct_/compute/mod.rs +++ b/vortex-array/src/arrays/struct_/compute/mod.rs @@ -70,7 +70,12 @@ mod tests { .execute::(&mut LEGACY_SESSION.create_execution_ctx()) .unwrap(); assert_eq!(taken.len(), 1); - assert!(taken.into_array().all_invalid().unwrap()); + assert!( + taken + .into_array() + .all_invalid(&mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); } #[test] diff --git a/vortex-array/src/arrays/struct_/compute/rules.rs b/vortex-array/src/arrays/struct_/compute/rules.rs index 91518ca7693..6e95e3eaff8 100644 --- a/vortex-array/src/arrays/struct_/compute/rules.rs +++ b/vortex-array/src/arrays/struct_/compute/rules.rs @@ -18,43 +18,46 @@ use crate::arrays::scalar_fn::ScalarFnFactoryExt; use crate::arrays::slice::SliceReduceAdaptor; use crate::arrays::struct_::StructArrayExt; use crate::builtins::ArrayBuiltins; +use crate::dtype::DType; use crate::optimizer::rules::ArrayParentReduceRule; use crate::optimizer::rules::ParentRuleSet; use crate::scalar_fn::EmptyOptions; -use crate::scalar_fn::fns::cast::Cast; +use crate::scalar_fn::fns::cast::CastReduce; +use crate::scalar_fn::fns::cast::CastReduceAdaptor; use crate::scalar_fn::fns::get_item::GetItem; use crate::scalar_fn::fns::mask::Mask; use crate::scalar_fn::fns::mask::MaskReduceAdaptor; use crate::validity::Validity; pub(crate) const PARENT_RULES: ParentRuleSet = ParentRuleSet::new(&[ - ParentRuleSet::lift(&StructCastPushDownRule), + ParentRuleSet::lift(&CastReduceAdaptor(Struct)), ParentRuleSet::lift(&StructGetItemRule), ParentRuleSet::lift(&MaskReduceAdaptor(Struct)), ParentRuleSet::lift(&SliceReduceAdaptor(Struct)), ParentRuleSet::lift(&TakeReduceAdaptor(Struct)), ]); -/// Rule to push down cast into struct fields. +/// Push the cast into struct fields without execution. /// -/// TODO(joe/rob): should be have this in casts. +/// Supports schema evolution by allowing new nullable fields to be added during the cast, +/// filled with null values. For nullability changes, only handles the cheap path +/// (`try_cast_nullability`); when statistics computation is required to determine whether +/// the array contains invalid values, returns `Ok(None)` so [`CastKernel`] can run instead. /// -/// This rule supports schema evolution by allowing new nullable fields to be added -/// at the end of the struct, filled with null values. -#[derive(Debug)] -struct StructCastPushDownRule; -impl ArrayParentReduceRule for StructCastPushDownRule { - type Parent = ExactScalarFn; +/// [`CastKernel`]: crate::scalar_fn::fns::cast::CastKernel +impl CastReduce for Struct { + fn cast(array: ArrayView<'_, Struct>, dtype: &DType) -> VortexResult> { + let Some(target_fields) = dtype.as_struct_fields_opt() else { + return Ok(None); + }; - fn reduce_parent( - &self, - array: ArrayView<'_, Struct>, - parent: ScalarFnArrayView, - _child_idx: usize, - ) -> VortexResult> { - let Some(target_fields) = parent.options.as_struct_fields_opt() else { + let Some(validity) = array + .validity()? + .trivial_cast_nullability(dtype.nullability(), array.len())? + else { return Ok(None); }; + let mut new_fields = Vec::with_capacity(target_fields.nfields()); for (target_name, target_dtype) in target_fields.names().iter().zip(target_fields.fields()) @@ -78,20 +81,12 @@ impl ArrayParentReduceRule for StructCastPushDownRule { } } - let validity = if parent.options.is_nullable() { - array.validity()?.into_nullable() - } else { - array - .validity()? - .into_non_nullable(array.len()) - .ok_or_else(|| vortex_err!("Failed to cast nullable struct to non-nullable"))? - }; - - let new_struct = unsafe { - StructArray::new_unchecked(new_fields, target_fields.clone(), array.len(), validity) - }; - - Ok(Some(new_struct.into_array())) + Ok(Some( + unsafe { + StructArray::new_unchecked(new_fields, target_fields.clone(), array.len(), validity) + } + .into_array(), + )) } } @@ -144,24 +139,31 @@ impl ArrayParentReduceRule for StructGetItemRule { #[cfg(test)] mod tests { + use std::sync::LazyLock; + use vortex_buffer::buffer; + use vortex_session::VortexSession; use crate::IntoArray; + use crate::VortexSessionExecute; use crate::arrays::StructArray; use crate::arrays::VarBinViewArray; use crate::arrays::struct_::StructArrayExt; use crate::arrays::struct_::compute::rules::ConstantArray; use crate::assert_arrays_eq; use crate::builtins::ArrayBuiltins; - use crate::canonical::ToCanonical; use crate::dtype::DType; use crate::dtype::FieldNames; use crate::dtype::Nullability; use crate::dtype::PType; use crate::dtype::StructFields; use crate::scalar::Scalar; + use crate::session::ArraySession; use crate::validity::Validity; + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); + #[test] fn test_struct_cast_field_reorder() { // Source: {a, b}, Target: {c, b, a} - reordered + new null field @@ -187,7 +189,12 @@ mod tests { // Use `ArrayBuiltins::cast` which goes through the optimizer and applies // `StructCastPushDownRule`. - let result = source.into_array().cast(target).unwrap().to_struct(); + let result = source + .into_array() + .cast(target) + .unwrap() + .execute::(&mut SESSION.create_execution_ctx()) + .unwrap(); assert_arrays_eq!( result.unmasked_field_by_name("a").unwrap(), VarBinViewArray::from_iter_nullable_str([Some("A")]) @@ -255,7 +262,12 @@ mod tests { Nullability::NonNullable, ); - let result = source.into_array().cast(target).unwrap().to_struct(); + let result = source + .into_array() + .cast(target) + .unwrap() + .execute::(&mut SESSION.create_execution_ctx()) + .unwrap(); assert_eq!(result.unmasked_fields().len(), 2); assert_arrays_eq!( result.unmasked_field_by_name("a").unwrap(), @@ -286,7 +298,12 @@ mod tests { Nullability::NonNullable, ); - let result = source.into_array().cast(target).unwrap().to_struct(); + let result = source + .into_array() + .cast(target) + .unwrap() + .execute::(&mut SESSION.create_execution_ctx()) + .unwrap(); assert_eq!( result.unmasked_field_by_name("val").unwrap().dtype(), &DType::Primitive(PType::I64, Nullability::NonNullable) diff --git a/vortex-array/src/arrays/struct_/tests.rs b/vortex-array/src/arrays/struct_/tests.rs index 38b12074900..2db9e62f4f5 100644 --- a/vortex-array/src/arrays/struct_/tests.rs +++ b/vortex-array/src/arrays/struct_/tests.rs @@ -4,7 +4,10 @@ use vortex_buffer::buffer; use vortex_error::VortexResult; +use crate::Canonical; use crate::IntoArray; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::arrays::BoolArray; use crate::arrays::ConstantArray; use crate::arrays::PrimitiveArray; @@ -140,10 +143,15 @@ fn test_uncompressed_size_in_bytes() -> VortexResult<()> { Validity::NonNullable, ); - let canonical_size = struct_array.to_canonical()?.into_array().nbytes(); + let canonical_size = struct_array + .clone() + .into_array() + .execute::(&mut LEGACY_SESSION.create_execution_ctx())? + .into_array() + .nbytes(); let uncompressed_size = struct_array .statistics() - .compute_uncompressed_size_in_bytes(); + .compute_uncompressed_size_in_bytes(&mut LEGACY_SESSION.create_execution_ctx()); assert_eq!(canonical_size, 2); assert_eq!(uncompressed_size, Some(4000)); diff --git a/vortex-array/src/arrays/struct_/vtable/mod.rs b/vortex-array/src/arrays/struct_/vtable/mod.rs index f33cea360c3..6a7a2cd8dd8 100644 --- a/vortex-array/src/arrays/struct_/vtable/mod.rs +++ b/vortex-array/src/arrays/struct_/vtable/mod.rs @@ -29,6 +29,8 @@ mod kernel; mod operations; mod validity; +use vortex_session::registry::CachedId; + use crate::array::ArrayId; /// A [`Struct`]-encoded Vortex array. @@ -39,9 +41,9 @@ impl VTable for Struct { type OperationsVTable = Self; type ValidityVTable = Self; - fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.struct"); + *ID } fn nbuffers(_array: ArrayView<'_, Self>) -> usize { @@ -68,7 +70,7 @@ impl VTable for Struct { ); } - let validity = child_to_validity(&slots[VALIDITY_SLOT], *nullability); + let validity = child_to_validity(slots[VALIDITY_SLOT].as_ref(), *nullability); if let Some(validity_len) = validity.maybe_len() && validity_len != len { @@ -205,7 +207,3 @@ impl VTable for Struct { #[derive(Clone, Debug)] pub struct Struct; - -impl Struct { - pub const ID: ArrayId = ArrayId::new_ref("vortex.struct"); -} diff --git a/vortex-array/src/arrays/struct_/vtable/operations.rs b/vortex-array/src/arrays/struct_/vtable/operations.rs index 021d93f3c2f..d59bf7eeca3 100644 --- a/vortex-array/src/arrays/struct_/vtable/operations.rs +++ b/vortex-array/src/arrays/struct_/vtable/operations.rs @@ -14,11 +14,11 @@ impl OperationsVTable for Struct { fn scalar_at( array: ArrayView<'_, Struct>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { let field_scalars: VortexResult> = array .iter_unmasked_fields() - .map(|field| field.scalar_at(index)) + .map(|field| field.execute_scalar(index, ctx)) .collect(); // SAFETY: The vtable guarantees index is in-bounds and non-null before this is called. // Each field's scalar_at returns a scalar with the field's own dtype. diff --git a/vortex-array/src/arrays/varbin/accessor.rs b/vortex-array/src/arrays/varbin/accessor.rs index aa84ac64421..ee68f94bd55 100644 --- a/vortex-array/src/arrays/varbin/accessor.rs +++ b/vortex-array/src/arrays/varbin/accessor.rs @@ -5,7 +5,8 @@ use std::iter; use vortex_error::VortexExpect; -use crate::ToCanonical; +#[expect(deprecated)] +use crate::ToCanonical as _; use crate::accessor::ArrayAccessor; use crate::arrays::VarBinArray; use crate::arrays::varbin::VarBinArrayExt; @@ -17,6 +18,7 @@ impl ArrayAccessor<[u8]> for VarBinArray { where F: for<'a> FnOnce(&mut dyn Iterator>) -> R, { + #[expect(deprecated)] let offsets = self.offsets().to_primitive(); let validity = self .validity() @@ -38,6 +40,7 @@ impl ArrayAccessor<[u8]> for VarBinArray { } Validity::AllInvalid => f(&mut iter::repeat_n(None, self.len())), Validity::Array(v) => { + #[expect(deprecated)] let validity = v.to_bool().into_bit_buffer(); let mut iter = offsets .windows(2) diff --git a/vortex-array/src/arrays/varbin/array.rs b/vortex-array/src/arrays/varbin/array.rs index 7fbbcb27b2c..805eb68aaa2 100644 --- a/vortex-array/src/arrays/varbin/array.rs +++ b/vortex-array/src/arrays/varbin/array.rs @@ -10,10 +10,12 @@ use vortex_error::VortexExpect; use vortex_error::VortexResult; use vortex_error::vortex_ensure; use vortex_error::vortex_err; -use vortex_mask::Mask; use crate::ArrayRef; -use crate::ToCanonical; +use crate::LEGACY_SESSION; +#[expect(deprecated)] +use crate::ToCanonical as _; +use crate::VortexSessionExecute; use crate::array::Array; use crate::array::ArrayParts; use crate::array::TypedArrayRef; @@ -211,7 +213,10 @@ impl VarBinData { // Skip host-only validation when offsets/bytes are not host-resident. if offsets.is_host() && bytes.is_on_host() { let last_offset = offsets - .scalar_at(offsets.len() - 1)? + .execute_scalar( + offsets.len() - 1, + &mut LEGACY_SESSION.create_execution_ctx(), + )? .as_primitive() .as_::() .ok_or_else( @@ -241,6 +246,7 @@ impl VarBinData { && matches!(dtype, DType::Utf8(_)) && let Some(bytes) = bytes.as_host_opt() { + #[expect(deprecated)] let primitive_offsets = offsets.to_primitive(); match_each_integer_ptype!(primitive_offsets.dtype().as_ptype(), |O| { let offsets_slice = primitive_offsets.as_slice::(); @@ -255,7 +261,7 @@ impl VarBinData { let string_bytes = &bytes.as_ref()[start..end]; simdutf8::basic::from_utf8(string_bytes).map_err(|_| { - #[allow(clippy::unwrap_used)] + #[expect(clippy::unwrap_used)] // run validation using `compat` package to get more detailed error message let err = simdutf8::compat::from_utf8(string_bytes).unwrap_err(); vortex_err!("invalid utf-8: {err} at index {i}") @@ -314,11 +320,10 @@ pub trait VarBinArrayExt: TypedArrayRef { } fn varbin_validity(&self) -> Validity { - child_to_validity(&self.as_ref().slots()[VALIDITY_SLOT], self.nullability()) - } - - fn varbin_validity_mask(&self) -> Mask { - self.varbin_validity().to_mask(self.as_ref().len()) + child_to_validity( + self.as_ref().slots()[VALIDITY_SLOT].as_ref(), + self.nullability(), + ) } fn offset_at(&self, index: usize) -> usize { @@ -330,8 +335,8 @@ pub trait VarBinArrayExt: TypedArrayRef { (&self .offsets() - .scalar_at(index) - .vortex_expect("offsets must support scalar_at")) + .execute_scalar(index, &mut LEGACY_SESSION.create_execution_ctx()) + .vortex_expect("offsets must support execute_scalar")) .try_into() .vortex_expect("Failed to convert offset to usize") } diff --git a/vortex-array/src/arrays/varbin/builder.rs b/vortex-array/src/arrays/varbin/builder.rs index 9b63889e70f..2e329d5b9a6 100644 --- a/vortex-array/src/arrays/varbin/builder.rs +++ b/vortex-array/src/arrays/varbin/builder.rs @@ -7,6 +7,10 @@ use vortex_buffer::BufferMut; use vortex_error::vortex_panic; use crate::IntoArray; +#[cfg(debug_assertions)] +use crate::LEGACY_SESSION; +#[cfg(debug_assertions)] +use crate::VortexSessionExecute; use crate::arrays::PrimitiveArray; use crate::arrays::VarBinArray; use crate::dtype::DType; @@ -99,10 +103,14 @@ impl VarBinBuilder { // The builder guarantees offsets are monotonically increasing, so we can set // this stat eagerly. This avoids an O(n) recomputation when the array is // deserialized and VarBinArray::validate checks sortedness. - debug_assert!( - offsets.statistics().compute_is_sorted().unwrap_or(false), - "VarBinBuilder offsets must be sorted" - ); + #[cfg(debug_assertions)] + { + let offsets_are_sorted = offsets + .statistics() + .compute_is_sorted(&mut LEGACY_SESSION.create_execution_ctx()) + .unwrap_or(false); + debug_assert!(offsets_are_sorted, "VarBinBuilder offsets must be sorted"); + } offsets .statistics() .set(Stat::IsSorted, Precision::Exact(true.into())); @@ -122,6 +130,8 @@ impl VarBinBuilder { mod tests { use vortex_error::VortexResult; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::arrays::varbin::VarBinArrayExt; use crate::arrays::varbin::builder::VarBinBuilder; use crate::dtype::DType; @@ -142,10 +152,17 @@ mod tests { assert_eq!(array.len(), 3); assert_eq!(array.dtype().nullability(), Nullable); assert_eq!( - array.scalar_at(0).unwrap(), + array + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::utf8("hello".to_string(), Nullable) ); - assert!(array.scalar_at(1).unwrap().is_null()); + assert!( + array + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); } #[test] diff --git a/vortex-array/src/arrays/varbin/compute/cast.rs b/vortex-array/src/arrays/varbin/compute/cast.rs index 6b02283aec8..e6403535e2d 100644 --- a/vortex-array/src/arrays/varbin/compute/cast.rs +++ b/vortex-array/src/arrays/varbin/compute/cast.rs @@ -4,13 +4,30 @@ use vortex_error::VortexResult; use crate::ArrayRef; +use crate::ExecutionCtx; use crate::IntoArray; use crate::array::ArrayView; use crate::arrays::VarBin; use crate::arrays::VarBinArray; use crate::arrays::varbin::VarBinArrayExt; use crate::dtype::DType; +use crate::scalar_fn::fns::cast::CastKernel; use crate::scalar_fn::fns::cast::CastReduce; +use crate::validity::Validity; + +fn build_with_validity( + array: ArrayView<'_, VarBin>, + new_dtype: DType, + new_validity: Validity, +) -> VortexResult { + Ok(VarBinArray::try_new( + array.offsets().clone(), + array.bytes().clone(), + new_dtype, + new_validity, + )? + .into_array()) +} impl CastReduce for VarBin { fn cast(array: ArrayView<'_, VarBin>, dtype: &DType) -> VortexResult> { @@ -18,33 +35,56 @@ impl CastReduce for VarBin { return Ok(None); } + let new_nullability = dtype.nullability(); + let Some(new_validity) = array + .validity()? + .trivial_cast_nullability(new_nullability, array.len())? + else { + return Ok(None); + }; + let new_dtype = array.dtype().with_nullability(new_nullability); + Ok(Some(build_with_validity(array, new_dtype, new_validity)?)) + } +} + +impl CastKernel for VarBin { + fn cast( + array: ArrayView<'_, VarBin>, + dtype: &DType, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + if !array.dtype().eq_ignore_nullability(dtype) { + return Ok(None); + } + let new_nullability = dtype.nullability(); let new_validity = array .validity()? - .cast_nullability(new_nullability, array.len())?; + .cast_nullability(new_nullability, array.len(), ctx)?; let new_dtype = array.dtype().with_nullability(new_nullability); - Ok(Some( - VarBinArray::try_new( - array.offsets().clone(), - array.bytes().clone(), - new_dtype, - new_validity, - )? - .into_array(), - )) + Ok(Some(build_with_validity(array, new_dtype, new_validity)?)) } } #[cfg(test)] mod tests { + use std::sync::LazyLock; + use rstest::rstest; + use vortex_session::VortexSession; + use crate::Canonical; use crate::IntoArray; + use crate::VortexSessionExecute; use crate::arrays::VarBinArray; use crate::builtins::ArrayBuiltins; use crate::compute::conformance::cast::test_cast_conformance; use crate::dtype::DType; use crate::dtype::Nullability; + use crate::session::ArraySession; + + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); #[rstest] #[case( @@ -71,14 +111,18 @@ mod tests { } #[rstest] - #[should_panic] #[case(DType::Utf8(Nullability::Nullable))] - #[should_panic] #[case(DType::Binary(Nullability::Nullable))] fn try_cast_varbin_fail(#[case] source: DType) { + // Failure surfaces during execution via the kernel. let non_nullable_source = source.as_nonnullable(); let varbin = VarBinArray::from_iter(vec![Some("a"), Some("b"), None], source); - varbin.into_array().cast(non_nullable_source).unwrap(); + let mut ctx = SESSION.create_execution_ctx(); + let result = varbin + .into_array() + .cast(non_nullable_source) + .and_then(|a| a.execute::(&mut ctx).map(|c| c.into_array())); + assert!(result.is_err(), "Expected error, got: {result:?}"); } #[rstest] diff --git a/vortex-array/src/arrays/varbin/compute/compare.rs b/vortex-array/src/arrays/varbin/compute/compare.rs index b4ad470760e..511f930d3d0 100644 --- a/vortex-array/src/arrays/varbin/compute/compare.rs +++ b/vortex-array/src/arrays/varbin/compute/compare.rs @@ -80,10 +80,10 @@ impl CompareKernel for VarBin { )); } - let lhs = Datum::try_new(lhs.array())?; + let lhs = Datum::try_new(lhs.array(), ctx)?; // Use StringViewArray/BinaryViewArray to match the Utf8View/BinaryView types - // produced by Datum::try_new (which uses into_arrow_preferred()) + // produced by Datum::try_new (which uses execute_arrow(None, ctx)) let arrow_rhs: &dyn arrow_array::Datum = match rhs_const.dtype() { DType::Utf8(_) => &rhs_const .as_utf8() @@ -145,7 +145,10 @@ mod test { use vortex_buffer::ByteBuffer; use crate::IntoArray; - use crate::ToCanonical; + use crate::LEGACY_SESSION; + #[expect(deprecated)] + use crate::ToCanonical as _; + use crate::VortexSessionExecute; use crate::arrays::ConstantArray; use crate::arrays::VarBinArray; use crate::arrays::VarBinViewArray; @@ -162,6 +165,7 @@ mod test { [Some(b"abc".to_vec()), None, Some(b"def".to_vec())], DType::Binary(Nullability::Nullable), ); + #[expect(deprecated)] let result = array .into_array() .binary( @@ -176,7 +180,16 @@ mod test { .to_bool(); assert_eq!( - &result.validity_mask().unwrap().to_bit_buffer(), + &result + .as_ref() + .validity() + .unwrap() + .execute_mask( + result.as_ref().len(), + &mut LEGACY_SESSION.create_execution_ctx() + ) + .unwrap() + .to_bit_buffer(), &BitBuffer::from_iter([true, false, true]) ); assert_eq!( @@ -195,6 +208,7 @@ mod test { [None, None, Some(b"def".to_vec())], DType::Binary(Nullability::Nullable), ); + #[expect(deprecated)] let result = array .into_array() .binary(vbv.into_array(), Operator::Eq) @@ -202,7 +216,16 @@ mod test { .to_bool(); assert_eq!( - result.validity_mask().unwrap().to_bit_buffer(), + result + .as_ref() + .validity() + .unwrap() + .execute_mask( + result.as_ref().len(), + &mut LEGACY_SESSION.create_execution_ctx() + ) + .unwrap() + .to_bit_buffer(), BitBuffer::from_iter([false, false, true]) ); assert_eq!( diff --git a/vortex-array/src/arrays/varbin/compute/filter.rs b/vortex-array/src/arrays/varbin/compute/filter.rs index 8f4b6793a72..b3e1aa87a4f 100644 --- a/vortex-array/src/arrays/varbin/compute/filter.rs +++ b/vortex-array/src/arrays/varbin/compute/filter.rs @@ -68,7 +68,10 @@ fn filter_select_var_bin_by_slice( offsets.as_slice::(), values.bytes().as_slice(), mask_slices, - values.varbin_validity_mask(), + values + .varbin_validity() + .execute_mask(values.as_ref().len(), ctx) + .vortex_expect("Failed to compute validity mask"), selection_count, ) }) diff --git a/vortex-array/src/arrays/varbin/compute/take.rs b/vortex-array/src/arrays/varbin/compute/take.rs index de8b1ffff77..7c799bfa54b 100644 --- a/vortex-array/src/arrays/varbin/compute/take.rs +++ b/vortex-array/src/arrays/varbin/compute/take.rs @@ -37,8 +37,13 @@ impl TakeExecute for VarBin { .dtype() .clone() .union_nullability(indices.dtype().nullability()); - let array_validity = array.varbin_validity_mask(); - let indices_validity = indices.validity_mask()?; + let array_validity = array + .varbin_validity() + .execute_mask(array.as_ref().len(), ctx)?; + let indices_validity = indices + .as_ref() + .validity()? + .execute_mask(indices.as_ref().len(), ctx)?; let array = match_each_integer_ptype!(indices.ptype(), |I| { // On take, offsets get widened to either 32- or 64-bit based on the original type, diff --git a/vortex-array/src/arrays/varbin/vtable/canonical.rs b/vortex-array/src/arrays/varbin/vtable/canonical.rs index 726bbfd8dcd..fc30ee2c94e 100644 --- a/vortex-array/src/arrays/varbin/vtable/canonical.rs +++ b/vortex-array/src/arrays/varbin/vtable/canonical.rs @@ -47,11 +47,14 @@ pub(crate) fn varbin_to_canonical( mod tests { use rstest::rstest; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::arrays::VarBinArray; use crate::arrays::VarBinViewArray; use crate::arrays::varbin::builder::VarBinBuilder; use crate::assert_arrays_eq; - use crate::canonical::ToCanonical; + #[expect(deprecated)] + use crate::canonical::ToCanonical as _; use crate::dtype::DType; use crate::dtype::Nullability; @@ -70,10 +73,15 @@ mod tests { let varbin = varbin.slice(1..4).unwrap(); + #[expect(deprecated)] let canonical = varbin.to_varbinview(); assert_eq!(canonical.dtype(), &dtype); - assert!(!canonical.is_valid(0).unwrap()); + assert!( + !canonical + .is_valid(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); // First value is inlined (12 bytes) assert!(canonical.views()[1].is_inlined()); @@ -89,6 +97,7 @@ mod tests { #[case(DType::Binary(Nullability::NonNullable))] fn test_canonical_varbin_unsliced(#[case] dtype: DType) { let varbin = VarBinArray::from_iter_nonnull(["foo", "bar", "baz"], dtype.clone()); + #[expect(deprecated)] let canonical = varbin.as_array().to_varbinview(); let expected = match dtype { DType::Utf8(_) => VarBinViewArray::from_iter_str(["foo", "bar", "baz"]), @@ -102,6 +111,7 @@ mod tests { fn test_canonical_varbin_empty() { let varbin = VarBinArray::from_iter_nonnull([] as [&str; 0], DType::Utf8(Nullability::NonNullable)); + #[expect(deprecated)] let canonical = varbin.as_array().to_varbinview(); assert_eq!(canonical.len(), 0); } diff --git a/vortex-array/src/arrays/varbin/vtable/kernel.rs b/vortex-array/src/arrays/varbin/vtable/kernel.rs index 8cb6a2ca377..9258e677933 100644 --- a/vortex-array/src/arrays/varbin/vtable/kernel.rs +++ b/vortex-array/src/arrays/varbin/vtable/kernel.rs @@ -6,8 +6,10 @@ use crate::arrays::dict::TakeExecuteAdaptor; use crate::arrays::filter::FilterExecuteAdaptor; use crate::kernel::ParentKernelSet; use crate::scalar_fn::fns::binary::CompareExecuteAdaptor; +use crate::scalar_fn::fns::cast::CastExecuteAdaptor; pub(super) const PARENT_KERNELS: ParentKernelSet = ParentKernelSet::new(&[ + ParentKernelSet::lift(&CastExecuteAdaptor(VarBin)), ParentKernelSet::lift(&CompareExecuteAdaptor(VarBin)), ParentKernelSet::lift(&FilterExecuteAdaptor(VarBin)), ParentKernelSet::lift(&TakeExecuteAdaptor(VarBin)), diff --git a/vortex-array/src/arrays/varbin/vtable/mod.rs b/vortex-array/src/arrays/varbin/vtable/mod.rs index ffe44158c27..b72ac5eaa48 100644 --- a/vortex-array/src/arrays/varbin/vtable/mod.rs +++ b/vortex-array/src/arrays/varbin/vtable/mod.rs @@ -9,6 +9,7 @@ use vortex_error::VortexResult; use vortex_error::vortex_bail; use vortex_error::vortex_ensure; use vortex_error::vortex_panic; +use vortex_session::registry::CachedId; use crate::ArrayRef; use crate::ExecutionCtx; @@ -68,9 +69,9 @@ impl VTable for VarBin { type OperationsVTable = Self; type ValidityVTable = Self; - fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.varbin"); + *ID } fn nbuffers(_array: ArrayView<'_, Self>) -> usize { @@ -198,7 +199,3 @@ impl VTable for VarBin { #[derive(Clone, Debug)] pub struct VarBin; - -impl VarBin { - pub const ID: ArrayId = ArrayId::new_ref("vortex.varbin"); -} diff --git a/vortex-array/src/arrays/varbinview/accessor.rs b/vortex-array/src/arrays/varbinview/accessor.rs index ef1470a090d..cd42c3f2b58 100644 --- a/vortex-array/src/arrays/varbinview/accessor.rs +++ b/vortex-array/src/arrays/varbinview/accessor.rs @@ -5,7 +5,8 @@ use std::iter; use vortex_error::VortexExpect; -use crate::ToCanonical; +#[expect(deprecated)] +use crate::ToCanonical as _; use crate::accessor::ArrayAccessor; use crate::arrays::VarBinViewArray; use crate::validity::Validity; @@ -39,6 +40,7 @@ impl ArrayAccessor<[u8]> for VarBinViewArray { } Validity::AllInvalid => f(&mut iter::repeat_n(None, views.len())), Validity::Array(v) => { + #[expect(deprecated)] let validity = v.to_bool().into_bit_buffer(); let mut iter = views.iter().zip(validity.iter()).map(|(view, valid)| { if valid { diff --git a/vortex-array/src/arrays/varbinview/array.rs b/vortex-array/src/arrays/varbinview/array.rs index 37d055107cf..06256bfa77c 100644 --- a/vortex-array/src/arrays/varbinview/array.rs +++ b/vortex-array/src/arrays/varbinview/array.rs @@ -15,7 +15,6 @@ use vortex_error::vortex_bail; use vortex_error::vortex_ensure; use vortex_error::vortex_err; use vortex_error::vortex_panic; -use vortex_mask::Mask; use crate::ArrayRef; use crate::array::Array; @@ -542,11 +541,10 @@ pub trait VarBinViewArrayExt: TypedArrayRef { } fn varbinview_validity(&self) -> Validity { - child_to_validity(&self.as_ref().slots()[VALIDITY_SLOT], self.dtype_parts().1) - } - - fn varbinview_validity_mask(&self) -> Mask { - self.varbinview_validity().to_mask(self.as_ref().len()) + child_to_validity( + self.as_ref().slots()[VALIDITY_SLOT].as_ref(), + self.dtype_parts().1, + ) } } impl> VarBinViewArrayExt for T {} diff --git a/vortex-array/src/arrays/varbinview/compact.rs b/vortex-array/src/arrays/varbinview/compact.rs index 37bae3d9822..6effc7c656a 100644 --- a/vortex-array/src/arrays/varbinview/compact.rs +++ b/vortex-array/src/arrays/varbinview/compact.rs @@ -11,6 +11,8 @@ use vortex_error::VortexResult; use vortex_mask::Mask; use crate::IntoArray; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::arrays::VarBinViewArray; use crate::arrays::varbinview::Ref; use crate::builders::ArrayBuilder; @@ -63,7 +65,10 @@ impl VarBinViewArray { where F: FnMut(&Ref), { - match self.validity_mask()? { + match self.as_ref().validity()?.execute_mask( + self.as_ref().len(), + &mut LEGACY_SESSION.create_execution_ctx(), + )? { Mask::AllTrue(_) => { for &view in self.views().iter() { if !view.is_inlined() { diff --git a/vortex-array/src/arrays/varbinview/compute/cast.rs b/vortex-array/src/arrays/varbinview/compute/cast.rs index a8e649f52e4..9b3d6b45dc7 100644 --- a/vortex-array/src/arrays/varbinview/compute/cast.rs +++ b/vortex-array/src/arrays/varbinview/compute/cast.rs @@ -6,12 +6,32 @@ use std::sync::Arc; use vortex_error::VortexResult; use crate::ArrayRef; +use crate::ExecutionCtx; use crate::IntoArray; use crate::array::ArrayView; use crate::arrays::VarBinView; use crate::arrays::VarBinViewArray; use crate::dtype::DType; +use crate::scalar_fn::fns::cast::CastKernel; use crate::scalar_fn::fns::cast::CastReduce; +use crate::validity::Validity; + +fn build_with_validity( + array: ArrayView<'_, VarBinView>, + new_dtype: DType, + new_validity: Validity, +) -> ArrayRef { + // SAFETY: casting just changes the DType, does not affect invariants on views/buffers. + unsafe { + VarBinViewArray::new_handle_unchecked( + array.views_handle().clone(), + Arc::clone(array.data_buffers()), + new_dtype, + new_validity, + ) + .into_array() + } +} impl CastReduce for VarBinView { fn cast(array: ArrayView<'_, VarBinView>, dtype: &DType) -> VortexResult> { @@ -20,36 +40,55 @@ impl CastReduce for VarBinView { } let new_nullability = dtype.nullability(); - let new_validity = array + let Some(new_validity) = array .validity()? - .cast_nullability(new_nullability, array.len())?; + .trivial_cast_nullability(new_nullability, array.len())? + else { + return Ok(None); + }; let new_dtype = array.dtype().with_nullability(new_nullability); + Ok(Some(build_with_validity(array, new_dtype, new_validity))) + } +} - // SAFETY: casting just changes the DType, does not affect invariants on views/buffers. - unsafe { - Ok(Some( - VarBinViewArray::new_handle_unchecked( - array.views_handle().clone(), - Arc::clone(array.data_buffers()), - new_dtype, - new_validity, - ) - .into_array(), - )) +impl CastKernel for VarBinView { + fn cast( + array: ArrayView<'_, VarBinView>, + dtype: &DType, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + if !array.dtype().eq_ignore_nullability(dtype) { + return Ok(None); } + + let new_nullability = dtype.nullability(); + let new_validity = array + .validity()? + .cast_nullability(new_nullability, array.len(), ctx)?; + let new_dtype = array.dtype().with_nullability(new_nullability); + Ok(Some(build_with_validity(array, new_dtype, new_validity))) } } #[cfg(test)] mod tests { + use std::sync::LazyLock; + use rstest::rstest; + use vortex_session::VortexSession; + use crate::Canonical; use crate::IntoArray; + use crate::VortexSessionExecute; use crate::arrays::VarBinViewArray; use crate::builtins::ArrayBuiltins; use crate::compute::conformance::cast::test_cast_conformance; use crate::dtype::DType; use crate::dtype::Nullability; + use crate::session::ArraySession; + + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); #[rstest] #[case( @@ -76,14 +115,18 @@ mod tests { } #[rstest] - #[should_panic] #[case(DType::Utf8(Nullability::Nullable))] - #[should_panic] #[case(DType::Binary(Nullability::Nullable))] fn try_cast_varbin_fail(#[case] source: DType) { + // Failure surfaces during execution via the kernel. let non_nullable_source = source.as_nonnullable(); let varbin = VarBinViewArray::from_iter(vec![Some("a"), Some("b"), None], source); - varbin.into_array().cast(non_nullable_source).unwrap(); + let mut ctx = SESSION.create_execution_ctx(); + let result = varbin + .into_array() + .cast(non_nullable_source) + .and_then(|a| a.execute::(&mut ctx).map(|c| c.into_array())); + assert!(result.is_err(), "Expected error, got: {result:?}"); } #[rstest] diff --git a/vortex-array/src/arrays/varbinview/compute/mod.rs b/vortex-array/src/arrays/varbinview/compute/mod.rs index 772221f13ef..b69efad1e88 100644 --- a/vortex-array/src/arrays/varbinview/compute/mod.rs +++ b/vortex-array/src/arrays/varbinview/compute/mod.rs @@ -15,7 +15,8 @@ mod tests { use crate::IntoArray; use crate::accessor::ArrayAccessor; use crate::arrays::VarBinViewArray; - use crate::canonical::ToCanonical; + #[expect(deprecated)] + use crate::canonical::ToCanonical as _; #[test] fn take_nullable() { let arr = VarBinViewArray::from_iter_nullable_str([ @@ -30,12 +31,12 @@ mod tests { let taken = arr.take(buffer![0, 3].into_array()).unwrap(); assert!(taken.dtype().is_nullable()); - assert_eq!( - taken.to_varbinview().with_iterator(|it| it - .map(|v| v.map(|b| unsafe { String::from_utf8_unchecked(b.to_vec()) })) - .collect::>()), - [Some("one".to_string()), Some("four".to_string())] - ); + #[expect(deprecated)] + let result = taken.to_varbinview().with_iterator(|it| { + it.map(|v| v.map(|b| unsafe { String::from_utf8_unchecked(b.to_vec()) })) + .collect::>() + }); + assert_eq!(result, [Some("one".to_string()), Some("four".to_string())]); } // Consistency tests use rstest::rstest; diff --git a/vortex-array/src/arrays/varbinview/compute/take.rs b/vortex-array/src/arrays/varbinview/compute/take.rs index 644465ecac3..5c5ff08e2d0 100644 --- a/vortex-array/src/arrays/varbinview/compute/take.rs +++ b/vortex-array/src/arrays/varbinview/compute/take.rs @@ -32,7 +32,10 @@ impl TakeExecute for VarBinView { let validity = array.validity()?.take(indices)?; let indices = indices.clone().execute::(ctx)?; - let indices_mask = indices.validity_mask()?; + let indices_mask = indices + .as_ref() + .validity()? + .execute_mask(indices.as_ref().len(), ctx)?; let views_buffer = match_each_integer_ptype!(indices.ptype(), |I| { take_views(array.views(), indices.as_slice::(), &indices_mask) }); @@ -92,7 +95,8 @@ mod tests { use crate::accessor::ArrayAccessor; use crate::arrays::VarBinViewArray; use crate::arrays::varbinview::compute::take::PrimitiveArray; - use crate::canonical::ToCanonical; + #[expect(deprecated)] + use crate::canonical::ToCanonical as _; use crate::compute::conformance::take::test_take_conformance; use crate::dtype::DType; use crate::dtype::Nullability::NonNullable; @@ -112,12 +116,12 @@ mod tests { let taken = arr.take(buffer![0, 3].into_array()).unwrap(); assert!(taken.dtype().is_nullable()); - assert_eq!( - taken.to_varbinview().with_iterator(|it| it - .map(|v| v.map(|b| unsafe { String::from_utf8_unchecked(b.to_vec()) })) - .collect::>()), - [Some("one".to_string()), Some("four".to_string())] - ); + #[expect(deprecated)] + let result = taken.to_varbinview().with_iterator(|it| { + it.map(|v| v.map(|b| unsafe { String::from_utf8_unchecked(b.to_vec()) })) + .collect::>() + }); + assert_eq!(result, [Some("one".to_string()), Some("four".to_string())]); } #[test] @@ -133,12 +137,12 @@ mod tests { let taken = arr.take(indices.into_array()).unwrap(); assert!(taken.dtype().is_nullable()); - assert_eq!( - taken.to_varbinview().with_iterator(|it| it - .map(|v| v.map(|b| unsafe { String::from_utf8_unchecked(b.to_vec()) })) - .collect::>()), - [Some("two".to_string()), None] - ); + #[expect(deprecated)] + let result = taken.to_varbinview().with_iterator(|it| { + it.map(|v| v.map(|b| unsafe { String::from_utf8_unchecked(b.to_vec()) })) + .collect::>() + }); + assert_eq!(result, [Some("two".to_string()), None]); } #[rstest] diff --git a/vortex-array/src/arrays/varbinview/compute/zip.rs b/vortex-array/src/arrays/varbinview/compute/zip.rs index dfa2ae8e71c..f4a5c26a9ce 100644 --- a/vortex-array/src/arrays/varbinview/compute/zip.rs +++ b/vortex-array/src/arrays/varbinview/compute/zip.rs @@ -54,8 +54,8 @@ impl ZipKernel for VarBinView { let mut views_builder = BufferMut::::with_capacity(len); let mut validity_builder = LazyBitBufferBuilder::new(len); - let true_validity = if_true.varbinview_validity_mask(); - let false_validity = if_false.varbinview_validity_mask(); + let true_validity = if_true.varbinview_validity().execute_mask(len, ctx)?; + let false_validity = if_false.varbinview_validity().execute_mask(len, ctx)?; let mask = mask.try_to_mask_fill_null_false(ctx)?; let if_false_view = if_false; @@ -214,7 +214,8 @@ mod tests { use crate::accessor::ArrayAccessor; use crate::arrays::VarBinViewArray; use crate::builtins::ArrayBuiltins; - use crate::canonical::ToCanonical; + #[expect(deprecated)] + use crate::canonical::ToCanonical as _; use crate::dtype::DType; use crate::dtype::Nullability; @@ -246,6 +247,7 @@ mod tests { let mask = Mask::from_iter([true, false, true, false, false, true]); + #[expect(deprecated)] let zipped = mask .clone() .into_array() diff --git a/vortex-array/src/arrays/varbinview/tests.rs b/vortex-array/src/arrays/varbinview/tests.rs index ab19e33dd59..2892ff16283 100644 --- a/vortex-array/src/arrays/varbinview/tests.rs +++ b/vortex-array/src/arrays/varbinview/tests.rs @@ -1,7 +1,8 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -use crate::ToCanonical; +#[expect(deprecated)] +use crate::ToCanonical as _; use crate::arrays::VarBinViewArray; use crate::arrays::varbinview::BinaryView; use crate::assert_arrays_eq; @@ -31,6 +32,7 @@ pub fn slice_array() { #[test] pub fn flatten_array() { let binary_arr = VarBinViewArray::from_iter_str(["string1", "string2"]); + #[expect(deprecated)] let var_bin = binary_arr.as_array().to_varbinview(); assert_arrays_eq!( var_bin, diff --git a/vortex-array/src/arrays/varbinview/vtable/kernel.rs b/vortex-array/src/arrays/varbinview/vtable/kernel.rs index 8486f0959f7..cd9d68010af 100644 --- a/vortex-array/src/arrays/varbinview/vtable/kernel.rs +++ b/vortex-array/src/arrays/varbinview/vtable/kernel.rs @@ -4,9 +4,11 @@ use crate::arrays::VarBinView; use crate::arrays::dict::TakeExecuteAdaptor; use crate::kernel::ParentKernelSet; +use crate::scalar_fn::fns::cast::CastExecuteAdaptor; use crate::scalar_fn::fns::zip::ZipExecuteAdaptor; pub(super) const PARENT_KERNELS: ParentKernelSet = ParentKernelSet::new(&[ + ParentKernelSet::lift(&CastExecuteAdaptor(VarBinView)), ParentKernelSet::lift(&TakeExecuteAdaptor(VarBinView)), ParentKernelSet::lift(&ZipExecuteAdaptor(VarBinView)), ]); diff --git a/vortex-array/src/arrays/varbinview/vtable/mod.rs b/vortex-array/src/arrays/varbinview/vtable/mod.rs index 453cd7b61d4..3827549744c 100644 --- a/vortex-array/src/arrays/varbinview/vtable/mod.rs +++ b/vortex-array/src/arrays/varbinview/vtable/mod.rs @@ -13,6 +13,7 @@ use vortex_error::vortex_ensure; use vortex_error::vortex_err; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::ArrayRef; use crate::ExecutionCtx; @@ -42,10 +43,6 @@ pub type VarBinViewArray = Array; #[derive(Clone, Debug)] pub struct VarBinView; -impl VarBinView { - pub const ID: ArrayId = ArrayId::new_ref("vortex.varbinview"); -} - impl ArrayHash for VarBinViewData { fn array_hash(&self, state: &mut H, precision: Precision) { for buffer in self.buffers.iter() { @@ -74,7 +71,8 @@ impl VTable for VarBinView { type ValidityVTable = Self; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.varbinview"); + *ID } fn nbuffers(array: ArrayView<'_, Self>) -> usize { diff --git a/vortex-array/src/arrays/variant/vtable/mod.rs b/vortex-array/src/arrays/variant/vtable/mod.rs index 2fdf200b98f..25db7e16077 100644 --- a/vortex-array/src/arrays/variant/vtable/mod.rs +++ b/vortex-array/src/arrays/variant/vtable/mod.rs @@ -9,6 +9,7 @@ use vortex_error::VortexResult; use vortex_error::vortex_ensure; use vortex_error::vortex_panic; use vortex_session::VortexSession; +use vortex_session::registry::CachedId; use crate::ArrayRef; use crate::ExecutionCtx; @@ -29,10 +30,6 @@ pub type VariantArray = Array; #[derive(Clone, Debug)] pub struct Variant; -impl Variant { - pub const ID: ArrayId = ArrayId::new_ref("vortex.variant"); -} - impl VTable for Variant { type ArrayData = EmptyArrayData; @@ -41,7 +38,8 @@ impl VTable for Variant { type ValidityVTable = Self; fn id(&self) -> ArrayId { - Self::ID + static ID: CachedId = CachedId::new("vortex.variant"); + *ID } fn validate( diff --git a/vortex-array/src/arrays/variant/vtable/operations.rs b/vortex-array/src/arrays/variant/vtable/operations.rs index fb6fc0eb499..32e4737e308 100644 --- a/vortex-array/src/arrays/variant/vtable/operations.rs +++ b/vortex-array/src/arrays/variant/vtable/operations.rs @@ -14,8 +14,8 @@ impl OperationsVTable for Variant { fn scalar_at( array: ArrayView<'_, Variant>, index: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { - array.child().scalar_at(index) + array.child().execute_scalar(index, ctx) } } diff --git a/vortex-array/src/arrow/datum.rs b/vortex-array/src/arrow/datum.rs index ce77c3bb159..66c457d3727 100644 --- a/vortex-array/src/arrow/datum.rs +++ b/vortex-array/src/arrow/datum.rs @@ -11,10 +11,13 @@ use vortex_error::vortex_panic; use crate::ArrayRef; use crate::IntoArray; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::arrays::Constant; use crate::arrays::ConstantArray; +use crate::arrow::ArrowArrayExecutor; use crate::arrow::FromArrowArray; -use crate::arrow::IntoArrowArray; +use crate::executor::ExecutionCtx; /// A wrapper around a generic Arrow array that can be used as a Datum in Arrow compute. #[derive(Debug)] @@ -25,15 +28,15 @@ pub struct Datum { impl Datum { /// Create a new [`Datum`] from an [`ArrayRef`], which can then be passed to Arrow compute. - pub fn try_new(array: &ArrayRef) -> VortexResult { + pub fn try_new(array: &ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { if array.is::() { Ok(Self { - array: array.slice(0..1)?.into_arrow_preferred()?, + array: array.slice(0..1)?.execute_arrow(None, ctx)?, is_scalar: true, }) } else { Ok(Self { - array: array.clone().into_arrow_preferred()?, + array: array.clone().execute_arrow(None, ctx)?, is_scalar: false, }) } @@ -41,9 +44,9 @@ impl Datum { /// Create a new [`Datum`] from an `DynArray`, which can then be passed to Arrow compute. /// This not try and convert the array to a scalar if it is constant. - pub fn try_new_array(array: &ArrayRef) -> VortexResult { + pub fn try_new_array(array: &ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { Ok(Self { - array: array.clone().into_arrow_preferred()?, + array: array.clone().execute_arrow(None, ctx)?, is_scalar: false, }) } @@ -51,15 +54,18 @@ impl Datum { pub fn try_new_with_target_datatype( array: &ArrayRef, target_datatype: &DataType, + ctx: &mut ExecutionCtx, ) -> VortexResult { if array.is::() { Ok(Self { - array: array.slice(0..1)?.into_arrow(target_datatype)?, + array: array + .slice(0..1)? + .execute_arrow(Some(target_datatype), ctx)?, is_scalar: true, }) } else { Ok(Self { - array: array.clone().into_arrow(target_datatype)?, + array: array.clone().execute_arrow(Some(target_datatype), ctx)?, is_scalar: false, }) } @@ -103,8 +109,8 @@ where Ok(ConstantArray::new( array - .scalar_at(0) - .vortex_expect("array of length 1 must support scalar_at(0)"), + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .vortex_expect("array of length 1 must support execute_scalar(0)"), len, ) .into_array()) diff --git a/vortex-array/src/arrow/executor/bool.rs b/vortex-array/src/arrow/executor/bool.rs index 28862a150d0..68f2451cd77 100644 --- a/vortex-array/src/arrow/executor/bool.rs +++ b/vortex-array/src/arrow/executor/bool.rs @@ -14,10 +14,18 @@ use crate::arrays::bool::BoolArrayExt; use crate::arrow::null_buffer::to_null_buffer; /// Convert a canonical BoolArray directly to Arrow. -pub fn canonical_bool_to_arrow(array: &BoolArray) -> VortexResult { +pub fn canonical_bool_to_arrow( + array: &BoolArray, + ctx: &mut ExecutionCtx, +) -> VortexResult { Ok(Arc::new(ArrowBooleanArray::new( array.to_bit_buffer().into(), - to_null_buffer(array.validity_mask()?), + to_null_buffer( + array + .as_ref() + .validity()? + .execute_mask(array.as_ref().len(), ctx)?, + ), ))) } @@ -26,5 +34,5 @@ pub(super) fn to_arrow_bool( ctx: &mut ExecutionCtx, ) -> VortexResult { let bool_array = array.execute::(ctx)?; - canonical_bool_to_arrow(&bool_array) + canonical_bool_to_arrow(&bool_array, ctx) } diff --git a/vortex-array/src/arrow/executor/byte_view.rs b/vortex-array/src/arrow/executor/byte_view.rs index 9b19ddcc722..b88b1895d53 100644 --- a/vortex-array/src/arrow/executor/byte_view.rs +++ b/vortex-array/src/arrow/executor/byte_view.rs @@ -22,6 +22,7 @@ use crate::dtype::arrow::FromArrowType; /// Convert a canonical VarBinViewArray directly to Arrow. pub fn canonical_varbinview_to_arrow( array: &VarBinViewArray, + ctx: &mut ExecutionCtx, ) -> VortexResult { let views = ScalarBuffer::::from(array.views_handle().as_host().clone().into_arrow_buffer()); @@ -30,7 +31,12 @@ pub fn canonical_varbinview_to_arrow( .iter() .map(|buffer| buffer.as_host().clone().into_arrow_buffer()) .collect(); - let nulls = to_null_buffer(array.validity_mask()?); + let nulls = to_null_buffer( + array + .as_ref() + .validity()? + .execute_mask(array.as_ref().len(), ctx)?, + ); // SAFETY: our own VarBinView array is considered safe. Ok(Arc::new(unsafe { @@ -68,5 +74,5 @@ pub(super) fn to_arrow_byte_view( let array = array.cast(DType::from_arrow((&T::DATA_TYPE, Nullability::Nullable)))?; let varbinview = array.execute::(ctx)?; - canonical_varbinview_to_arrow::(&varbinview) + canonical_varbinview_to_arrow::(&varbinview, ctx) } diff --git a/vortex-array/src/arrow/executor/decimal.rs b/vortex-array/src/arrow/executor/decimal.rs index 234b253f903..077495354cc 100644 --- a/vortex-array/src/arrow/executor/decimal.rs +++ b/vortex-array/src/arrow/executor/decimal.rs @@ -32,16 +32,21 @@ pub(super) fn to_arrow_decimal( let decimal_array = array.execute::(ctx)?; match data_type { - DataType::Decimal32(..) => to_arrow_decimal32(decimal_array), - DataType::Decimal64(..) => to_arrow_decimal64(decimal_array), - DataType::Decimal128(..) => to_arrow_decimal128(decimal_array), - DataType::Decimal256(..) => to_arrow_decimal256(decimal_array), + DataType::Decimal32(..) => to_arrow_decimal32(decimal_array, ctx), + DataType::Decimal64(..) => to_arrow_decimal64(decimal_array, ctx), + DataType::Decimal128(..) => to_arrow_decimal128(decimal_array, ctx), + DataType::Decimal256(..) => to_arrow_decimal256(decimal_array, ctx), _ => unreachable!("to_arrow_decimal called with non-decimal type"), } } -fn to_arrow_decimal32(array: DecimalArray) -> VortexResult { - let null_buffer = to_null_buffer(array.validity_mask()?); +fn to_arrow_decimal32(array: DecimalArray, ctx: &mut ExecutionCtx) -> VortexResult { + let null_buffer = to_null_buffer( + array + .as_ref() + .validity()? + .execute_mask(array.as_ref().len(), ctx)?, + ); let buffer: Buffer = match array.values_type() { DecimalType::I8 => { Buffer::from_trusted_len_iter(array.buffer::().into_iter().map(|x| x.as_())) @@ -84,8 +89,13 @@ fn to_arrow_decimal32(array: DecimalArray) -> VortexResult { )) } -fn to_arrow_decimal64(array: DecimalArray) -> VortexResult { - let null_buffer = to_null_buffer(array.validity_mask()?); +fn to_arrow_decimal64(array: DecimalArray, ctx: &mut ExecutionCtx) -> VortexResult { + let null_buffer = to_null_buffer( + array + .as_ref() + .validity()? + .execute_mask(array.as_ref().len(), ctx)?, + ); let buffer: Buffer = match array.values_type() { DecimalType::I8 => { Buffer::from_trusted_len_iter(array.buffer::().into_iter().map(|x| x.as_())) @@ -123,8 +133,13 @@ fn to_arrow_decimal64(array: DecimalArray) -> VortexResult { )) } -fn to_arrow_decimal128(array: DecimalArray) -> VortexResult { - let null_buffer = to_null_buffer(array.validity_mask()?); +fn to_arrow_decimal128(array: DecimalArray, ctx: &mut ExecutionCtx) -> VortexResult { + let null_buffer = to_null_buffer( + array + .as_ref() + .validity()? + .execute_mask(array.as_ref().len(), ctx)?, + ); let buffer: Buffer = match array.values_type() { DecimalType::I8 => { Buffer::from_trusted_len_iter(array.buffer::().into_iter().map(|x| x.as_())) @@ -157,8 +172,13 @@ fn to_arrow_decimal128(array: DecimalArray) -> VortexResult { )) } -fn to_arrow_decimal256(array: DecimalArray) -> VortexResult { - let null_buffer = to_null_buffer(array.validity_mask()?); +fn to_arrow_decimal256(array: DecimalArray, ctx: &mut ExecutionCtx) -> VortexResult { + let null_buffer = to_null_buffer( + array + .as_ref() + .validity()? + .execute_mask(array.as_ref().len(), ctx)?, + ); let buffer: Buffer = match array.values_type() { DecimalType::I8 => { Buffer::from_trusted_len_iter(array.buffer::().into_iter().map(|x| x.as_())) @@ -206,7 +226,6 @@ mod tests { use crate::VortexSessionExecute; use crate::array::IntoArray; use crate::arrow::ArrowArrayExecutor; - use crate::arrow::IntoArrowArray; use crate::arrow::executor::decimal::DecimalArray; use crate::builders::ArrayBuilder; use crate::builders::DecimalBuilder; @@ -216,16 +235,16 @@ mod tests { #[test] fn decimal_to_arrow() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Make a very simple i128 and i256 array. let decimal_vortex = DecimalArray::new( buffer![1i128, 2i128, 3i128, 4i128, 5i128], DecimalDType::new(19, 2), Validity::NonNullable, ); - let arrow = decimal_vortex.into_array().execute_arrow( - Some(&DataType::Decimal128(19, 2)), - &mut LEGACY_SESSION.create_execution_ctx(), - )?; + let arrow = decimal_vortex + .into_array() + .execute_arrow(Some(&DataType::Decimal128(19, 2)), &mut ctx)?; assert_eq!(arrow.data_type(), &DataType::Decimal128(19, 2)); let decimal_array = arrow.as_any().downcast_ref::().unwrap(); assert_eq!( @@ -245,13 +264,14 @@ mod tests { fn test_to_arrow_decimal128( #[case] _decimal_type: T, ) -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let mut decimal = DecimalBuilder::new::(DecimalDType::new(2, 1), false.into()); decimal.append_value(10); decimal.append_value(11); decimal.append_value(12); let decimal = decimal.finish(); - let arrow_array = decimal.into_arrow(&DataType::Decimal128(2, 1))?; + let arrow_array = decimal.execute_arrow(Some(&DataType::Decimal128(2, 1)), &mut ctx)?; let arrow_decimal = arrow_array .as_any() .downcast_ref::() @@ -272,13 +292,14 @@ mod tests { fn test_to_arrow_decimal32(#[case] _decimal_type: T) -> VortexResult<()> { use arrow_array::Decimal32Array; + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let mut decimal = DecimalBuilder::new::(DecimalDType::new(2, 1), false.into()); decimal.append_value(10); decimal.append_value(11); decimal.append_value(12); let decimal = decimal.finish(); - let arrow_array = decimal.into_arrow(&DataType::Decimal32(2, 1))?; + let arrow_array = decimal.execute_arrow(Some(&DataType::Decimal32(2, 1)), &mut ctx)?; let arrow_decimal = arrow_array .as_any() .downcast_ref::() @@ -299,13 +320,14 @@ mod tests { fn test_to_arrow_decimal64(#[case] _decimal_type: T) -> VortexResult<()> { use arrow_array::Decimal64Array; + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let mut decimal = DecimalBuilder::new::(DecimalDType::new(2, 1), false.into()); decimal.append_value(10); decimal.append_value(11); decimal.append_value(12); let decimal = decimal.finish(); - let arrow_array = decimal.into_arrow(&DataType::Decimal64(2, 1))?; + let arrow_array = decimal.execute_arrow(Some(&DataType::Decimal64(2, 1)), &mut ctx)?; let arrow_decimal = arrow_array .as_any() .downcast_ref::() @@ -326,13 +348,14 @@ mod tests { fn test_to_arrow_decimal256( #[case] _decimal_type: T, ) -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let mut decimal = DecimalBuilder::new::(DecimalDType::new(2, 1), false.into()); decimal.append_value(10); decimal.append_value(11); decimal.append_value(12); let decimal = decimal.finish(); - let arrow_array = decimal.into_arrow(&DataType::Decimal256(2, 1))?; + let arrow_array = decimal.execute_arrow(Some(&DataType::Decimal256(2, 1)), &mut ctx)?; let arrow_decimal = arrow_array .as_any() .downcast_ref::() diff --git a/vortex-array/src/arrow/executor/dictionary.rs b/vortex-array/src/arrow/executor/dictionary.rs index f0808d88818..99e315bd8f7 100644 --- a/vortex-array/src/arrow/executor/dictionary.rs +++ b/vortex-array/src/arrow/executor/dictionary.rs @@ -168,6 +168,24 @@ mod tests { array.execute_arrow(Some(dt), &mut LEGACY_SESSION.create_execution_ctx()) } + fn dict_basic_input() -> crate::ArrayRef { + DictArray::try_new( + buffer![0u8, 1, 0].into_array(), + VarBinViewArray::from_iter_str(["a", "b"]).into_array(), + ) + .expect("valid dictionary input") + .into_array() + } + + fn dict_with_null_codes_input() -> crate::ArrayRef { + DictArray::try_new( + PrimitiveArray::from_option_iter(vec![Some(0u8), None, Some(1)]).into_array(), + VarBinViewArray::from_iter_str(["a", "b"]).into_array(), + ) + .expect("valid dictionary input with null codes") + .into_array() + } + #[rstest] #[case::constant_null( ConstantArray::new(Scalar::null(DType::Utf8(Nullable)), 4).into_array(), @@ -180,18 +198,12 @@ mod tests { Arc::new(vec![Some("hello"); 5].into_iter().collect::>()) as arrow_array::ArrayRef, )] #[case::dict_basic( - DictArray::try_new( - buffer![0u8, 1, 0].into_array(), - VarBinViewArray::from_iter_str(["a", "b"]).into_array(), - ).unwrap().into_array(), + dict_basic_input(), dict_type(DataType::UInt8, DataType::Utf8), Arc::new(vec![Some("a"), Some("b"), Some("a")].into_iter().collect::>()) as arrow_array::ArrayRef, )] #[case::dict_with_null_codes( - DictArray::try_new( - PrimitiveArray::from_option_iter(vec![Some(0u8), None, Some(1)]).into_array(), - VarBinViewArray::from_iter_str(["a", "b"]).into_array(), - ).unwrap().into_array(), + dict_with_null_codes_input(), dict_type(DataType::UInt8, DataType::Utf8), Arc::new(vec![Some("a"), None, Some("b")].into_iter().collect::>()) as arrow_array::ArrayRef, )] diff --git a/vortex-array/src/arrow/executor/list.rs b/vortex-array/src/arrow/executor/list.rs index 285ad733cfb..6c7c271e933 100644 --- a/vortex-array/src/arrow/executor/list.rs +++ b/vortex-array/src/arrow/executor/list.rs @@ -147,7 +147,7 @@ fn list_view_zctl( // For ZCTL, we know that we only care about the final size. assert!(!sizes.is_empty()); let final_size = sizes - .scalar_at(sizes.len() - 1)? + .execute_scalar(sizes.len() - 1, ctx)? .cast(&DType::Primitive(O::PTYPE, Nullability::NonNullable))?; let final_size = final_size .as_primitive() @@ -206,8 +206,10 @@ mod tests { use crate::Canonical; use crate::IntoArray; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::arrays::PrimitiveArray; - use crate::arrow::IntoArrowArray; + use crate::arrow::ArrowArrayExecutor; use crate::arrow::executor::list::ListViewArray; use crate::dtype::DType; use crate::dtype::Nullability::NonNullable; @@ -215,6 +217,7 @@ mod tests { #[test] fn test_to_arrow_list_i32() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Create a ListViewArray with i32 elements: [[1, 2, 3], [4, 5]] let elements = PrimitiveArray::new(buffer![1i32, 2, 3, 4, 5], Validity::NonNullable); let offsets = PrimitiveArray::new(buffer![0i32, 3], Validity::NonNullable); @@ -233,7 +236,9 @@ mod tests { // Convert to Arrow List with i32 offsets. let field = Field::new("item", DataType::Int32, false); let arrow_dt = DataType::List(field.into()); - let arrow_array = list_array.into_array().into_arrow(&arrow_dt)?; + let arrow_array = list_array + .into_array() + .execute_arrow(Some(&arrow_dt), &mut ctx)?; // Verify the type is correct. assert_eq!(arrow_array.data_type(), &arrow_dt); @@ -267,6 +272,7 @@ mod tests { #[test] fn test_to_arrow_list_i64() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Create a ListViewArray with i64 offsets: [[10, 20], [30]] let elements = PrimitiveArray::new(buffer![10i64, 20, 30], Validity::NonNullable); let offsets = PrimitiveArray::new(buffer![0i64, 2], Validity::NonNullable); @@ -285,7 +291,9 @@ mod tests { // Convert to Arrow LargeList with i64 offsets. let field = Field::new("item", DataType::Int64, false); let arrow_dt = DataType::LargeList(field.into()); - let arrow_array = list_array.into_array().into_arrow(&arrow_dt)?; + let arrow_array = list_array + .into_array() + .execute_arrow(Some(&arrow_dt), &mut ctx)?; // Verify the type is correct. assert_eq!(arrow_array.data_type(), &arrow_dt); @@ -304,6 +312,7 @@ mod tests { #[test] fn test_to_arrow_list_non_zctl() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Overlapping lists are NOT zero-copy-to-list, so this exercises the rebuild path. // Elements: [1, 2, 3, 4], List 0: [1,2,3], List 1: [2,3,4] (overlap at indices 1-2) let elements = PrimitiveArray::new(buffer![1i32, 2, 3, 4], Validity::NonNullable); @@ -320,7 +329,9 @@ mod tests { let field = Field::new("item", DataType::Int32, false); let arrow_dt = DataType::List(field.into()); - let arrow_array = list_array.into_array().into_arrow(&arrow_dt)?; + let arrow_array = list_array + .into_array() + .execute_arrow(Some(&arrow_dt), &mut ctx)?; let list = arrow_array .as_any() @@ -343,6 +354,7 @@ mod tests { #[test] fn test_to_arrow_list_empty_zctl() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let dtype = DType::List( Arc::new(DType::Primitive(crate::dtype::PType::I32, NonNullable)), NonNullable, @@ -354,7 +366,9 @@ mod tests { }; let arrow_dt = DataType::List(Field::new("item", DataType::Int32, false).into()); - let arrow_array = list_array.into_array().into_arrow(&arrow_dt)?; + let arrow_array = list_array + .into_array() + .execute_arrow(Some(&arrow_dt), &mut ctx)?; assert_eq!(arrow_array.len(), 0); Ok(()) } diff --git a/vortex-array/src/arrow/executor/list_view.rs b/vortex-array/src/arrow/executor/list_view.rs index bf6d0da0b8b..a6795b1adaa 100644 --- a/vortex-array/src/arrow/executor/list_view.rs +++ b/vortex-array/src/arrow/executor/list_view.rs @@ -89,13 +89,16 @@ mod tests { use vortex_error::VortexResult; use crate::IntoArray; - use crate::arrow::IntoArrowArray; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; + use crate::arrow::ArrowArrayExecutor; use crate::arrow::executor::list_view::ListViewArray; use crate::arrow::executor::list_view::PrimitiveArray; use crate::validity::Validity; #[test] fn test_to_arrow_listview_i32() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Create a ListViewArray with overlapping views: [[1, 2], [2, 3], [3, 4]] let elements = PrimitiveArray::new(buffer![1i32, 2, 3, 4], Validity::NonNullable); let offsets = PrimitiveArray::new(buffer![0i32, 1, 2], Validity::NonNullable); @@ -111,7 +114,9 @@ mod tests { // Convert to Arrow ListView with i32 offsets. let field = Field::new("item", DataType::Int32, false); let arrow_dt = DataType::ListView(field.into()); - let arrow_array = list_array.into_array().into_arrow(&arrow_dt)?; + let arrow_array = list_array + .into_array() + .execute_arrow(Some(&arrow_dt), &mut ctx)?; // Verify the type is correct. assert_eq!(arrow_array.data_type(), &arrow_dt); @@ -148,6 +153,7 @@ mod tests { #[test] fn test_to_arrow_listview_i64() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Create a ListViewArray with nullable elements: [[100], null, [200, 300]] let elements = PrimitiveArray::new(buffer![100i64, 200, 300], Validity::NonNullable); let offsets = PrimitiveArray::new(buffer![0i64, 1, 1], Validity::NonNullable); @@ -167,7 +173,9 @@ mod tests { // Convert to Arrow LargeListView with i64 offsets. let field = Field::new("item", DataType::Int64, false); let arrow_dt = DataType::LargeListView(field.into()); - let arrow_array = list_array.into_array().into_arrow(&arrow_dt)?; + let arrow_array = list_array + .into_array() + .execute_arrow(Some(&arrow_dt), &mut ctx)?; // Verify the type is correct. assert_eq!(arrow_array.data_type(), &arrow_dt); diff --git a/vortex-array/src/arrow/executor/primitive.rs b/vortex-array/src/arrow/executor/primitive.rs index 7910ed5fa31..0d41176972b 100644 --- a/vortex-array/src/arrow/executor/primitive.rs +++ b/vortex-array/src/arrow/executor/primitive.rs @@ -20,11 +20,15 @@ use crate::dtype::Nullability; /// Convert a canonical PrimitiveArray directly to Arrow. pub fn canonical_primitive_to_arrow( array: PrimitiveArray, + ctx: &mut ExecutionCtx, ) -> VortexResult where T::Native: NativePType, { - let validity = array.validity_mask()?; + let validity = array + .as_ref() + .validity()? + .execute_mask(array.as_ref().len(), ctx)?; let null_buffer = to_null_buffer(validity); let buffer = array.into_buffer::().into_arrow_scalar_buffer(); Ok(Arc::new(ArrowPrimitiveArray::::new(buffer, null_buffer))) @@ -40,5 +44,5 @@ where // We use nullable here so we can essentially ignore nullability during the cast. let array = array.cast(DType::Primitive(T::Native::PTYPE, Nullability::Nullable))?; let primitive = array.execute::(ctx)?; - canonical_primitive_to_arrow::(primitive) + canonical_primitive_to_arrow::(primitive, ctx) } diff --git a/vortex-array/src/arrow/executor/run_end.rs b/vortex-array/src/arrow/executor/run_end.rs index 69aca62a994..b579cab7e4d 100644 --- a/vortex-array/src/arrow/executor/run_end.rs +++ b/vortex-array/src/arrow/executor/run_end.rs @@ -24,6 +24,7 @@ use crate::IntoArray; use crate::arrays::Constant; use crate::arrays::ConstantArray; use crate::arrow::ArrowArrayExecutor; +use crate::session::ArraySessionExt; /// The encoding ID used by `vortex-runend`. We match on this string to avoid a crate dependency. const VORTEX_RUNEND_ID: &str = "vortex.runend"; @@ -79,8 +80,9 @@ fn run_end_to_arrow( ctx: &mut ExecutionCtx, ) -> VortexResult { let length = array.len(); - let metadata_bytes = array - .metadata(ctx.session())? + let metadata_bytes = ctx + .session() + .array_serialize(&array)? .ok_or_else(|| vortex_err!("RunEndArray missing metadata"))?; let metadata = RunEndMetadata::decode(&*metadata_bytes) .map_err(|e| vortex_err!("Failed to decode RunEndMetadata: {e}"))?; @@ -203,6 +205,7 @@ mod tests { use arrow_schema::Field; use rstest::rstest; use vortex_error::VortexResult; + use vortex_error::vortex_err; use vortex_session::VortexSession; use crate::IntoArray; @@ -230,22 +233,36 @@ mod tests { array.execute_arrow(Some(dt), &mut SESSION.create_execution_ctx()) } + fn constant_i32_with_i16_ends() -> arrow_array::ArrayRef { + Arc::new( + RunArray::::try_new( + &Int16Array::from(vec![5i16]), + &Int32Array::from(vec![42]), + ) + .expect("valid run-end test array"), + ) as arrow_array::ArrayRef + } + + fn constant_f64_with_i64_ends() -> arrow_array::ArrayRef { + Arc::new( + RunArray::::try_new( + &Int64Array::from(vec![7i64]), + &arrow_array::Float64Array::from(vec![1.5]), + ) + .expect("valid run-end test array"), + ) as arrow_array::ArrayRef + } + #[rstest] #[case::i32_with_i16_ends( ConstantArray::new(Scalar::from(42i32), 5).into_array(), ree_type(DataType::Int16, DataType::Int32), - Arc::new(RunArray::::try_new( - &Int16Array::from(vec![5i16]), - &Int32Array::from(vec![42]), - ).unwrap()) as arrow_array::ArrayRef, + constant_i32_with_i16_ends(), )] #[case::f64_with_i64_ends( ConstantArray::new(Scalar::from(1.5f64), 7).into_array(), ree_type(DataType::Int64, DataType::Float64), - Arc::new(RunArray::::try_new( - &Int64Array::from(vec![7i64]), - &arrow_array::Float64Array::from(vec![1.5]), - ).unwrap()) as arrow_array::ArrayRef, + constant_f64_with_i64_ends(), )] #[case::null( ConstantArray::new(Scalar::null(DType::Primitive(PType::I32, Nullable)), 4).into_array(), @@ -310,9 +327,13 @@ mod tests { let ree = result .as_any() .downcast_ref::>() - .unwrap(); + .ok_or_else(|| vortex_err!("expected Int32 run-end array"))?; assert_eq!(ree.run_ends().values(), expected_ends); - let values = ree.values().as_any().downcast_ref::().unwrap(); + let values = ree + .values() + .as_any() + .downcast_ref::() + .ok_or_else(|| vortex_err!("expected Int64 values"))?; assert_eq!(values.values(), expected_values); Ok(()) } diff --git a/vortex-array/src/arrow/executor/struct_.rs b/vortex-array/src/arrow/executor/struct_.rs index a85005c0a20..909fd78059d 100644 --- a/vortex-array/src/arrow/executor/struct_.rs +++ b/vortex-array/src/arrow/executor/struct_.rs @@ -16,7 +16,7 @@ use crate::ArrayRef; use crate::ExecutionCtx; use crate::IntoArray; use crate::arrays::Chunked; -use crate::arrays::ScalarFnVTable; +use crate::arrays::ScalarFn; use crate::arrays::Struct; use crate::arrays::StructArray; use crate::arrays::scalar_fn::ScalarFnArrayExt; @@ -69,7 +69,7 @@ pub(super) fn to_arrow_struct( }; // We can also short-circuit if the array is a `pack` scalar function: - if let Some(array) = array.as_opt::() + if let Some(array) = array.as_opt::() && let Some(_pack_options) = array.scalar_fn().as_opt::() { let DType::Struct(struct_fields, _) = array.dtype() else { @@ -190,6 +190,7 @@ fn create_from_fields( mod tests { use std::sync::Arc; + use arrays::varbinview::VarBinViewArray; use arrow_array::ArrayRef; use arrow_array::PrimitiveArray as ArrowPrimitiveArray; use arrow_array::StringViewArray; @@ -210,12 +211,12 @@ mod tests { use crate::arrays::StructArray; use crate::arrow::ArrowArrayExecutor; use crate::arrow::FromArrowArray; - use crate::arrow::IntoArrowArray; use crate::dtype::FieldNames; use crate::validity::Validity; #[test] fn struct_nullable_non_null_to_arrow() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let xs = PrimitiveArray::new(buffer![0i64, 1, 2, 3, 4], Validity::AllValid); let struct_a = StructArray::try_new( @@ -228,12 +229,15 @@ mod tests { let fields = vec![Field::new("xs", DataType::Int64, false)]; let arrow_dt = DataType::Struct(fields.into()); - struct_a.into_array().into_arrow(&arrow_dt)?; + struct_a + .into_array() + .execute_arrow(Some(&arrow_dt), &mut ctx)?; Ok(()) } #[test] fn struct_nullable_with_nulls_to_arrow() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let xs = PrimitiveArray::from_option_iter(vec![Some(0_i64), Some(1), Some(2), None, Some(3)]); @@ -247,12 +251,18 @@ mod tests { let fields = vec![Field::new("xs", DataType::Int64, false)]; let arrow_dt = DataType::Struct(fields.into()); - assert!(struct_a.into_array().into_arrow(&arrow_dt).is_err()); + assert!( + struct_a + .into_array() + .execute_arrow(Some(&arrow_dt), &mut ctx) + .is_err() + ); Ok(()) } #[test] fn struct_to_arrow_with_schema_mismatch() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let xs = PrimitiveArray::new(buffer![0i64, 1, 2, 3, 4], Validity::AllValid); let struct_a = StructArray::try_new( @@ -268,7 +278,11 @@ mod tests { ]; let arrow_dt = DataType::Struct(fields.into()); - let err = struct_a.into_array().into_arrow(&arrow_dt).err().unwrap(); + let err = struct_a + .into_array() + .execute_arrow(Some(&arrow_dt), &mut ctx) + .err() + .unwrap(); assert!( err.to_string() .contains("StructArray has 1 fields, but target Arrow type has 2 fields") @@ -278,6 +292,7 @@ mod tests { #[test] fn test_to_arrow() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = StructArray::from_fields( vec![ ( @@ -286,8 +301,7 @@ mod tests { ), ( "b", - arrays::varbinview::VarBinViewArray::from_iter_str(vec!["a", "b", "c"]) - .into_array(), + VarBinViewArray::from_iter_str(vec!["a", "b", "c"]).into_array(), ), ] .as_slice(), @@ -311,10 +325,9 @@ mod tests { let arrow_dtype = array.dtype().to_arrow_dtype()?; assert_eq!( - &array.into_array().execute_arrow( - Some(&arrow_dtype), - &mut LEGACY_SESSION.create_execution_ctx() - )?, + &array + .into_array() + .execute_arrow(Some(&arrow_dtype), &mut ctx)?, &arrow_array ); Ok(()) @@ -322,6 +335,7 @@ mod tests { #[test] fn to_arrow_with_non_nullable_fields() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = StructArray::from_fields( vec![ ( @@ -330,14 +344,13 @@ mod tests { ), ( "b", - arrays::varbinview::VarBinViewArray::from_iter_str(vec!["a", "b", "c"]) - .into_array(), + VarBinViewArray::from_iter_str(vec!["a", "b", "c"]).into_array(), ), ] .as_slice(), )?; let orig_dtype = array.dtype().clone(); - let arrow_array = array.into_array().into_arrow_preferred()?; + let arrow_array = array.into_array().execute_arrow(None, &mut ctx)?; let from_arrow = array::ArrayRef::from_arrow(arrow_array.as_ref(), false)?; assert_eq!(&orig_dtype, from_arrow.dtype()); Ok(()) diff --git a/vortex-array/src/arrow/executor/temporal.rs b/vortex-array/src/arrow/executor/temporal.rs index 29bcc15763e..c68a4f3ee9f 100644 --- a/vortex-array/src/arrow/executor/temporal.rs +++ b/vortex-array/src/arrow/executor/temporal.rs @@ -86,16 +86,16 @@ pub(super) fn to_arrow_temporal( match (unit, arrow_unit) { (TimeUnit::Seconds, ArrowTimeUnit::Second) => { - to_arrow_timestamp::(array, arrow_tz, ctx) + to_arrow_timestamp::(array, arrow_tz.as_ref(), ctx) } (TimeUnit::Milliseconds, ArrowTimeUnit::Millisecond) => { - to_arrow_timestamp::(array, arrow_tz, ctx) + to_arrow_timestamp::(array, arrow_tz.as_ref(), ctx) } (TimeUnit::Microseconds, ArrowTimeUnit::Microsecond) => { - to_arrow_timestamp::(array, arrow_tz, ctx) + to_arrow_timestamp::(array, arrow_tz.as_ref(), ctx) } (TimeUnit::Nanoseconds, ArrowTimeUnit::Nanosecond) => { - to_arrow_timestamp::(array, arrow_tz, ctx) + to_arrow_timestamp::(array, arrow_tz.as_ref(), ctx) } _ => vortex_bail!( "Cannot convert {} array to Arrow type {}", @@ -125,14 +125,14 @@ where fn to_arrow_timestamp( array: ArrayRef, - arrow_tz: &Option>, + arrow_tz: Option<&Arc>, ctx: &mut ExecutionCtx, ) -> VortexResult where T::Native: NativePType, { Ok(Arc::new( - to_arrow_temporal_primitive::(array, ctx)?.with_timezone_opt(arrow_tz.clone()), + to_arrow_temporal_primitive::(array, ctx)?.with_timezone_opt(arrow_tz.cloned()), )) } @@ -157,7 +157,10 @@ where primitive.ptype() ); - let validity = primitive.validity_mask()?; + let validity = primitive + .as_ref() + .validity()? + .execute_mask(primitive.as_ref().len(), ctx)?; let buffer = primitive.to_buffer::(); let values = buffer.into_arrow_scalar_buffer(); diff --git a/vortex-array/src/arrow/mod.rs b/vortex-array/src/arrow/mod.rs index bcc9e6094a1..efc83aa6af6 100644 --- a/vortex-array/src/arrow/mod.rs +++ b/vortex-array/src/arrow/mod.rs @@ -30,12 +30,16 @@ pub trait FromArrowArray { Self: Sized; } +#[deprecated(note = "Use `execute_arrow(None, ctx)` or `execute_arrow(Some(dt), ctx)` instead")] pub trait IntoArrowArray { + #[deprecated(note = "Use `execute_arrow(None, ctx)` instead")] fn into_arrow_preferred(self) -> VortexResult; + #[deprecated(note = "Use `execute_arrow(Some(data_type), ctx)` instead")] fn into_arrow(self, data_type: &DataType) -> VortexResult; } +#[allow(deprecated)] impl IntoArrowArray for ArrayRef { /// Convert this [`crate::ArrayRef`] into an Arrow [`crate::ArrayRef`] by using the array's /// preferred (cheapest) Arrow [`DataType`]. diff --git a/vortex-array/src/arrow/record_batch.rs b/vortex-array/src/arrow/record_batch.rs index a7596169c15..b57c307aed0 100644 --- a/vortex-array/src/arrow/record_batch.rs +++ b/vortex-array/src/arrow/record_batch.rs @@ -5,40 +5,14 @@ use arrow_array::RecordBatch; use arrow_array::cast::AsArray; use arrow_schema::DataType; use arrow_schema::Schema; -use vortex_error::VortexError; use vortex_error::VortexResult; -use vortex_error::vortex_bail; -use vortex_error::vortex_ensure; -use crate::ArrayRef; -use crate::Canonical; use crate::LEGACY_SESSION; use crate::VortexSessionExecute; use crate::array::IntoArray; use crate::arrays::StructArray; use crate::arrow::ArrowArrayExecutor; -impl TryFrom<&ArrayRef> for RecordBatch { - type Error = VortexError; - - fn try_from(value: &ArrayRef) -> VortexResult { - let Canonical::Struct(struct_array) = value.to_canonical()? else { - vortex_bail!("RecordBatch can only be constructed from ") - }; - - vortex_ensure!( - struct_array.all_valid()?, - "RecordBatch can only be constructed from StructArray with no nulls" - ); - - let data_type = struct_array.dtype().to_arrow_dtype()?; - let array_ref = struct_array - .into_array() - .execute_arrow(Some(&data_type), &mut LEGACY_SESSION.create_execution_ctx())?; - Ok(RecordBatch::from(array_ref.as_struct())) - } -} - impl StructArray { pub fn into_record_batch_with_schema( self, diff --git a/vortex-array/src/buffer.rs b/vortex-array/src/buffer.rs index 07baf843ead..29f9d3582be 100644 --- a/vortex-array/src/buffer.rs +++ b/vortex-array/src/buffer.rs @@ -222,7 +222,7 @@ impl BufferHandle { self.slice(start..end) } - #[allow(clippy::panic)] + #[expect(clippy::panic)] /// Unwraps the handle as host memory. /// /// # Panics @@ -235,7 +235,7 @@ impl BufferHandle { } } - #[allow(clippy::panic)] + #[expect(clippy::panic)] /// Unwraps the handle as device memory. /// /// # Panics diff --git a/vortex-array/src/builders/bool.rs b/vortex-array/src/builders/bool.rs index 4c5eb5d9a61..c09d69a5acd 100644 --- a/vortex-array/src/builders/bool.rs +++ b/vortex-array/src/builders/bool.rs @@ -12,13 +12,16 @@ use vortex_mask::Mask; use crate::ArrayRef; use crate::IntoArray; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::arrays::BoolArray; use crate::arrays::bool::BoolArrayExt; use crate::builders::ArrayBuilder; use crate::builders::DEFAULT_BUILDER_CAPACITY; use crate::builders::LazyBitBufferBuilder; use crate::canonical::Canonical; -use crate::canonical::ToCanonical; +#[expect(deprecated)] +use crate::canonical::ToCanonical as _; use crate::dtype::DType; use crate::dtype::Nullability; use crate::scalar::Scalar; @@ -113,11 +116,21 @@ impl ArrayBuilder for BoolBuilder { } unsafe fn extend_from_array_unchecked(&mut self, array: &ArrayRef) { + #[expect(deprecated)] let bool_array = array.to_bool(); self.inner.append_buffer(&bool_array.to_bit_buffer()); - self.nulls - .append_validity_mask(bool_array.validity_mask().vortex_expect("validity_mask")); + self.nulls.append_validity_mask( + bool_array + .as_ref() + .validity() + .vortex_expect("validity_mask") + .execute_mask( + bool_array.as_ref().len(), + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .vortex_expect("Failed to compute validity mask"), + ); } fn reserve_exact(&mut self, additional: usize) { @@ -157,7 +170,8 @@ mod tests { use crate::builders::BoolBuilder; use crate::builders::bool::BoolArray; use crate::builders::builder_with_capacity; - use crate::canonical::ToCanonical; + #[expect(deprecated)] + use crate::canonical::ToCanonical as _; use crate::dtype::DType; use crate::dtype::Nullability; use crate::scalar::Scalar; @@ -191,7 +205,9 @@ mod tests { .clone() .append_to_builder(builder.as_mut(), &mut ctx)?; + #[expect(deprecated)] let canon_into = builder.finish().to_bool(); + #[expect(deprecated)] let into_canon = chunk.to_bool(); assert!( diff --git a/vortex-array/src/builders/decimal.rs b/vortex-array/src/builders/decimal.rs index d3811314a04..c80f97d9ea2 100644 --- a/vortex-array/src/builders/decimal.rs +++ b/vortex-array/src/builders/decimal.rs @@ -13,7 +13,10 @@ use vortex_mask::Mask; use crate::ArrayRef; use crate::IntoArray; -use crate::ToCanonical; +use crate::LEGACY_SESSION; +#[expect(deprecated)] +use crate::ToCanonical as _; +use crate::VortexSessionExecute; use crate::arrays::DecimalArray; use crate::builders::ArrayBuilder; use crate::builders::DEFAULT_BUILDER_CAPACITY; @@ -193,6 +196,7 @@ impl ArrayBuilder for DecimalBuilder { } unsafe fn extend_from_array_unchecked(&mut self, array: &ArrayRef) { + #[expect(deprecated)] let decimal_array = array.to_decimal(); match_each_decimal_value_type!(decimal_array.values_type(), |D| { @@ -202,8 +206,17 @@ impl ArrayBuilder for DecimalBuilder { .extend(decimal_array.buffer::().iter().copied()); }); - self.nulls - .append_validity_mask(decimal_array.validity_mask().vortex_expect("validity_mask")); + self.nulls.append_validity_mask( + decimal_array + .as_ref() + .validity() + .vortex_expect("validity_mask") + .execute_mask( + decimal_array.as_ref().len(), + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .vortex_expect("Failed to compute validity mask"), + ); } fn reserve_exact(&mut self, additional: usize) { @@ -297,6 +310,8 @@ impl Default for DecimalBuffer { #[cfg(test)] mod tests { + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::assert_arrays_eq; use crate::builders::ArrayBuilder; use crate::builders::DecimalBuilder; @@ -318,7 +333,13 @@ mod tests { let i128s = i128s.finish(); for i in 0..i8s.len() { - assert_eq!(i8s.scalar_at(i).unwrap(), i128s.scalar_at(i).unwrap()); + assert_eq!( + i8s.execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + i128s + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); } } @@ -342,7 +363,9 @@ mod tests { // Test by taking a scalar from the array and appending it to a new builder. let mut builder2 = DecimalBuilder::new::(DecimalDType::new(10, 2), true.into()); for i in 0..array.len() { - let scalar = array.scalar_at(i).unwrap(); + let scalar = array + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); builder2.append_scalar(&scalar).unwrap(); } diff --git a/vortex-array/src/builders/dict/bytes.rs b/vortex-array/src/builders/dict/bytes.rs index 155d0b1db9a..d66870d6ba5 100644 --- a/vortex-array/src/builders/dict/bytes.rs +++ b/vortex-array/src/builders/dict/bytes.rs @@ -25,7 +25,8 @@ use crate::arrays::VarBin; use crate::arrays::VarBinView; use crate::arrays::VarBinViewArray; use crate::arrays::varbinview::build_views::BinaryView; -use crate::canonical::ToCanonical; +#[expect(deprecated)] +use crate::canonical::ToCanonical as _; use crate::dtype::DType; use crate::dtype::PType; use crate::dtype::UnsignedPType; @@ -172,7 +173,9 @@ impl DictEncoder for BytesDictBuilder { } else { // NOTE(aduffy): it is very rare that this path would be taken, only e.g. // if we're performing dictionary encoding downstream of some other compression. - self.encode_bytes(&array.to_varbinview(), len) + #[expect(deprecated)] + let varbinview = array.to_varbinview(); + self.encode_bytes(&varbinview, len) } } @@ -204,7 +207,8 @@ mod test { use std::str; use crate::IntoArray; - use crate::ToCanonical; + #[expect(deprecated)] + use crate::ToCanonical as _; use crate::accessor::ArrayAccessor; use crate::arrays::VarBinArray; use crate::arrays::dict::DictArraySlotsExt; @@ -214,11 +218,12 @@ mod test { fn encode_varbin() { let arr = VarBinArray::from(vec!["hello", "world", "hello", "again", "world"]); let dict = dict_encode(&arr.into_array()).unwrap(); - assert_eq!( - dict.codes().to_primitive().as_slice::(), - &[0, 1, 0, 2, 1] - ); - dict.values().to_varbinview().with_iterator(|iter| { + #[expect(deprecated)] + let codes = dict.codes().to_primitive(); + assert_eq!(codes.as_slice::(), &[0, 1, 0, 2, 1]); + #[expect(deprecated)] + let values = dict.values().to_varbinview(); + values.with_iterator(|iter| { assert_eq!( iter.flatten() .map(|b| unsafe { str::from_utf8_unchecked(b) }) @@ -243,11 +248,12 @@ mod test { .into_iter() .collect(); let dict = dict_encode(&arr.into_array()).unwrap(); - assert_eq!( - dict.codes().to_primitive().as_slice::(), - &[0, 1, 2, 0, 1, 3, 2, 1] - ); - dict.values().to_varbinview().with_iterator(|iter| { + #[expect(deprecated)] + let codes = dict.codes().to_primitive(); + assert_eq!(codes.as_slice::(), &[0, 1, 2, 0, 1, 3, 2, 1]); + #[expect(deprecated)] + let values = dict.values().to_varbinview(); + values.with_iterator(|iter| { assert_eq!( iter.map(|b| b.map(|v| unsafe { str::from_utf8_unchecked(v) })) .collect::>(), @@ -260,7 +266,9 @@ mod test { fn repeated_values() { let arr = VarBinArray::from(vec!["a", "a", "b", "b", "a", "b", "a", "b"]); let dict = dict_encode(&arr.into_array()).unwrap(); - dict.values().to_varbinview().with_iterator(|iter| { + #[expect(deprecated)] + let values = dict.values().to_varbinview(); + values.with_iterator(|iter| { assert_eq!( iter.flatten() .map(|b| unsafe { str::from_utf8_unchecked(b) }) @@ -268,9 +276,8 @@ mod test { vec!["a", "b"] ); }); - assert_eq!( - dict.codes().to_primitive().as_slice::(), - &[0, 0, 1, 1, 0, 1, 0, 1] - ); + #[expect(deprecated)] + let codes = dict.codes().to_primitive(); + assert_eq!(codes.as_slice::(), &[0, 0, 1, 1, 0, 1, 0, 1]); } } diff --git a/vortex-array/src/builders/dict/mod.rs b/vortex-array/src/builders/dict/mod.rs index aa24000ac20..7cfcfbbfb8a 100644 --- a/vortex-array/src/builders/dict/mod.rs +++ b/vortex-array/src/builders/dict/mod.rs @@ -9,7 +9,8 @@ use vortex_error::vortex_panic; use crate::ArrayRef; use crate::IntoArray; -use crate::ToCanonical; +#[expect(deprecated)] +use crate::ToCanonical as _; use crate::arrays::DictArray; use crate::arrays::Primitive; use crate::arrays::VarBin; @@ -66,7 +67,9 @@ pub fn dict_encode_with_constraints( constraints: &DictConstraints, ) -> VortexResult { let mut encoder = dict_encoder(array, constraints); - let codes = encoder.encode(array).to_primitive().narrow()?; + let encoded = encoder.encode(array); + #[expect(deprecated)] + let codes = encoded.to_primitive().narrow()?; // SAFETY: The encoding process will produce a value set of codes and values // All values in the dictionary are guaranteed to be referenced by at least one code // since we build the dictionary from the codes we observe during encoding diff --git a/vortex-array/src/builders/dict/primitive.rs b/vortex-array/src/builders/dict/primitive.rs index 9f41e462734..03d930dbef1 100644 --- a/vortex-array/src/builders/dict/primitive.rs +++ b/vortex-array/src/builders/dict/primitive.rs @@ -15,7 +15,8 @@ use super::DictConstraints; use super::DictEncoder; use crate::ArrayRef; use crate::IntoArray; -use crate::ToCanonical; +#[expect(deprecated)] +use crate::ToCanonical as _; use crate::accessor::ArrayAccessor; use crate::arrays::PrimitiveArray; use crate::arrays::primitive::NativeValue; @@ -125,7 +126,9 @@ where fn encode(&mut self, array: &ArrayRef) -> ArrayRef { let mut codes = BufferMut::::with_capacity(array.len()); - array.to_primitive().with_iterator(|it| { + #[expect(deprecated)] + let prim = array.to_primitive(); + prim.with_iterator(|it| { for value in it { let Some(code) = self.encode_value(value.copied()) else { break; @@ -152,7 +155,7 @@ where #[cfg(test)] mod test { - #[allow(unused_imports)] + #[expect(unused_imports)] use itertools::Itertools; use vortex_buffer::buffer; diff --git a/vortex-array/src/builders/extension.rs b/vortex-array/src/builders/extension.rs index ae540c1a6c2..d947877e297 100644 --- a/vortex-array/src/builders/extension.rs +++ b/vortex-array/src/builders/extension.rs @@ -15,7 +15,8 @@ use crate::builders::ArrayBuilder; use crate::builders::DEFAULT_BUILDER_CAPACITY; use crate::builders::builder_with_capacity; use crate::canonical::Canonical; -use crate::canonical::ToCanonical; +#[expect(deprecated)] +use crate::canonical::ToCanonical as _; use crate::dtype::DType; use crate::dtype::extension::ExtDTypeRef; use crate::scalar::ExtScalar; @@ -99,6 +100,7 @@ impl ArrayBuilder for ExtensionBuilder { } unsafe fn extend_from_array_unchecked(&mut self, array: &ArrayRef) { + #[expect(deprecated)] let ext_array = array.to_extension(); self.storage.extend_from_array(ext_array.storage_array()) } diff --git a/vortex-array/src/builders/fixed_size_list.rs b/vortex-array/src/builders/fixed_size_list.rs index f26281d5363..5937c37b041 100644 --- a/vortex-array/src/builders/fixed_size_list.rs +++ b/vortex-array/src/builders/fixed_size_list.rs @@ -13,6 +13,8 @@ use vortex_mask::Mask; use crate::ArrayRef; use crate::IntoArray; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::arrays::FixedSizeListArray; use crate::arrays::fixed_size_list::FixedSizeListArrayExt; use crate::builders::ArrayBuilder; @@ -20,7 +22,8 @@ use crate::builders::DEFAULT_BUILDER_CAPACITY; use crate::builders::LazyBitBufferBuilder; use crate::builders::builder_with_capacity; use crate::canonical::Canonical; -use crate::canonical::ToCanonical; +#[expect(deprecated)] +use crate::canonical::ToCanonical as _; use crate::dtype::DType; use crate::dtype::Nullability; use crate::scalar::ListScalar; @@ -235,6 +238,7 @@ impl ArrayBuilder for FixedSizeListBuilder { /// This will increase the capacity if extending with this `array` would go past the original /// capacity. unsafe fn extend_from_array_unchecked(&mut self, array: &ArrayRef) { + #[expect(deprecated)] let fsl = array.to_fixed_size_list(); if fsl.is_empty() { return; @@ -243,8 +247,10 @@ impl ArrayBuilder for FixedSizeListBuilder { self.elements_builder.extend_from_array(fsl.elements()); self.nulls.append_validity_mask( array - .validity_mask() - .vortex_expect("validity_mask in extend_from_array_unchecked"), + .validity() + .vortex_expect("validity_mask in extend_from_array_unchecked") + .execute_mask(array.len(), &mut LEGACY_SESSION.create_execution_ctx()) + .vortex_expect("Failed to compute validity mask"), ); } @@ -277,7 +283,10 @@ mod tests { use super::FixedSizeListBuilder; use crate::IntoArray as _; - use crate::ToCanonical; + use crate::LEGACY_SESSION; + #[expect(deprecated)] + use crate::ToCanonical as _; + use crate::VortexSessionExecute; use crate::arrays::PrimitiveArray; use crate::arrays::fixed_size_list::FixedSizeListArrayExt; use crate::builders::ArrayBuilder; @@ -329,6 +338,7 @@ mod tests { let fsl = builder.finish(); assert_eq!(fsl.len(), 2); + #[expect(deprecated)] let fsl_array = fsl.to_fixed_size_list(); assert_eq!(fsl_array.elements().len(), 6); assert_eq!(fsl_array.list_size(), 3); @@ -352,6 +362,7 @@ mod tests { let fsl = builder.finish(); assert_eq!(fsl.len(), 100); + #[expect(deprecated)] let fsl_array = fsl.to_fixed_size_list(); assert_eq!(fsl_array.list_size(), 0); // The elements array should be empty since list_size is 0. @@ -381,6 +392,7 @@ mod tests { let fsl = builder.finish(); assert_eq!(fsl.len(), 100); + #[expect(deprecated)] let fsl_array = fsl.to_fixed_size_list(); assert_eq!(fsl_array.list_size(), 0); assert_eq!(fsl_array.elements().len(), 0); @@ -410,6 +422,7 @@ mod tests { let fsl = builder.finish(); assert_eq!(fsl.len(), 5); + #[expect(deprecated)] let fsl_array = fsl.to_fixed_size_list(); assert_eq!(fsl_array.elements().len(), 10); } @@ -423,6 +436,7 @@ mod tests { let fsl = builder.finish(); assert_eq!(fsl.len(), 0); + #[expect(deprecated)] let fsl_array = fsl.to_fixed_size_list(); assert_eq!(fsl_array.list_size(), 100000000); assert_eq!(fsl_array.elements().len(), 0); @@ -455,6 +469,7 @@ mod tests { let fsl = builder.finish(); assert_eq!(fsl.len(), 3); + #[expect(deprecated)] let fsl_array = fsl.to_fixed_size_list(); assert!( fsl_array @@ -518,6 +533,7 @@ mod tests { let fsl = builder.finish(); assert_eq!(fsl.len(), 2); + #[expect(deprecated)] let fsl_array = fsl.to_fixed_size_list(); assert_eq!(fsl_array.elements().len(), 6); } @@ -532,11 +548,13 @@ mod tests { let fsl = builder.finish(); assert_eq!(fsl.len(), 5); + #[expect(deprecated)] let fsl_array = fsl.to_fixed_size_list(); assert_eq!(fsl_array.list_size(), 3); assert_eq!(fsl_array.elements().len(), 15); // Check that all elements are zeros. + #[expect(deprecated)] let elements_array = fsl_array.elements().to_primitive(); let elements = elements_array.as_slice::(); assert!(elements.iter().all(|&x| x == 0)); @@ -555,6 +573,7 @@ mod tests { let fsl = builder.finish(); assert_eq!(fsl.len(), 3); + #[expect(deprecated)] let fsl_array = fsl.to_fixed_size_list(); assert_eq!(fsl_array.list_size(), 2); @@ -585,6 +604,7 @@ mod tests { let fsl = builder.finish(); assert_eq!(fsl.len(), 1); + #[expect(deprecated)] let fsl_array = fsl.to_fixed_size_list(); assert_eq!(fsl_array.list_size(), 2); @@ -610,6 +630,7 @@ mod tests { let fsl = builder.finish(); assert_eq!(fsl.len(), 1000); + #[expect(deprecated)] let fsl_array = fsl.to_fixed_size_list(); assert_eq!(fsl_array.list_size(), 0); assert_eq!(fsl_array.elements().len(), 0); @@ -661,6 +682,7 @@ mod tests { let fsl = builder.finish(); assert_eq!(fsl.len(), 6); + #[expect(deprecated)] let fsl_array = fsl.to_fixed_size_list(); assert_eq!(fsl_array.elements().len(), 12); @@ -736,6 +758,7 @@ mod tests { let fsl = builder.finish(); assert_eq!(fsl.len(), 5); + #[expect(deprecated)] let fsl_array = fsl.to_fixed_size_list(); assert_eq!(fsl_array.list_size(), 0); assert_eq!(fsl_array.elements().len(), 0); @@ -848,6 +871,7 @@ mod tests { let fsl = builder.finish(); assert_eq!(fsl.len(), 6); + #[expect(deprecated)] let fsl_array = fsl.to_fixed_size_list(); assert_eq!(fsl_array.elements().len(), 12); @@ -919,7 +943,9 @@ mod tests { // Check actual values using scalar_at. - let scalar0 = array.scalar_at(0).unwrap(); + let scalar0 = array + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); let list0 = scalar0.as_list(); assert_eq!(list0.len(), 2); if let Some(list0_items) = list0.elements() { @@ -927,7 +953,9 @@ mod tests { assert_eq!(list0_items[1].as_primitive().typed_value::(), Some(2)); } - let scalar1 = array.scalar_at(1).unwrap(); + let scalar1 = array + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); let list1 = scalar1.as_list(); assert_eq!(list1.len(), 2); if let Some(list1_items) = list1.elements() { @@ -1007,6 +1035,7 @@ mod tests { assert_eq!(fsl.list_size(), 3); // Verify elements array: [1, 2, 3, 10, 11, 12, 4, 5, 6, 20, 21, 22]. + #[expect(deprecated)] let elements = fsl.elements().to_primitive(); assert_eq!( elements.as_slice::(), diff --git a/vortex-array/src/builders/list.rs b/vortex-array/src/builders/list.rs index 5eb6d12223b..ae1bb20dfa9 100644 --- a/vortex-array/src/builders/list.rs +++ b/vortex-array/src/builders/list.rs @@ -14,6 +14,8 @@ use vortex_mask::Mask; use crate::ArrayRef; use crate::Canonical; use crate::IntoArray; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::arrays::ListArray; use crate::arrays::listview::ListViewArrayExt; use crate::builders::ArrayBuilder; @@ -21,7 +23,8 @@ use crate::builders::DEFAULT_BUILDER_CAPACITY; use crate::builders::LazyBitBufferBuilder; use crate::builders::PrimitiveBuilder; use crate::builders::builder_with_capacity; -use crate::canonical::ToCanonical; +#[expect(deprecated)] +use crate::canonical::ToCanonical as _; use crate::dtype::DType; use crate::dtype::IntegerPType; use crate::dtype::Nullability; @@ -215,6 +218,7 @@ impl ArrayBuilder for ListBuilder { } unsafe fn extend_from_array_unchecked(&mut self, array: &ArrayRef) { + #[expect(deprecated)] let list = array.to_listview(); if list.is_empty() { return; @@ -223,13 +227,17 @@ impl ArrayBuilder for ListBuilder { // Append validity information. self.nulls.append_validity_mask( array - .validity_mask() - .vortex_expect("validity_mask in extend_from_array_unchecked"), + .validity() + .vortex_expect("validity_mask in extend_from_array_unchecked") + .execute_mask(array.len(), &mut LEGACY_SESSION.create_execution_ctx()) + .vortex_expect("Failed to compute validity mask"), ); // Note that `ListViewArray` has `n` offsets and sizes, not `n+1` offsets like `ListArray`. let elements = list.elements(); + #[expect(deprecated)] let offsets = list.offsets().to_primitive(); + #[expect(deprecated)] let sizes = list.sizes().to_primitive(); fn extend_inner( @@ -301,7 +309,9 @@ impl ArrayBuilder for ListBuilder { } fn finish_into_canonical(&mut self) -> Canonical { - Canonical::List(self.finish_into_list().into_array().to_listview()) + #[expect(deprecated)] + let listview = self.finish_into_list().into_array().to_listview(); + Canonical::List(listview) } } @@ -316,7 +326,8 @@ mod tests { use crate::IntoArray; use crate::LEGACY_SESSION; - use crate::ToCanonical; + #[expect(deprecated)] + use crate::ToCanonical as _; use crate::arrays::ChunkedArray; use crate::arrays::PrimitiveArray; use crate::arrays::list::ListArrayExt; @@ -372,6 +383,7 @@ mod tests { let list = builder.finish(); assert_eq!(list.len(), 2); + #[expect(deprecated)] let list_array = list.to_listview(); assert_eq!(list_array.list_elements_at(0).unwrap().len(), 3); @@ -424,6 +436,7 @@ mod tests { let list = builder.finish(); assert_eq!(list.len(), 3); + #[expect(deprecated)] let list_array = list.to_listview(); assert_eq!(list_array.list_elements_at(0).unwrap().len(), 3); @@ -447,6 +460,7 @@ mod tests { builder.extend_from_array(&list.slice(0..0).unwrap()); builder.extend_from_array(&list.slice(1..3).unwrap()); + #[expect(deprecated)] let expected = ListArray::from_iter_opt_slow::( [ Some(vec![0, 1, 2]), @@ -520,15 +534,24 @@ mod tests { DType::List(Arc::new(DType::Primitive(I32, NonNullable)), NonNullable), ); + #[expect(deprecated)] let canon_values = chunked_list.unwrap().as_array().to_listview(); assert_eq!( - one_trailing_unused_element.scalar_at(0).unwrap(), - canon_values.scalar_at(0).unwrap() + one_trailing_unused_element + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + canon_values + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() ); assert_eq!( - second_array.scalar_at(0).unwrap(), - canon_values.scalar_at(1).unwrap() + second_array + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + canon_values + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() ); } @@ -559,7 +582,9 @@ mod tests { // Check actual values using scalar_at. - let scalar0 = array.scalar_at(0).unwrap(); + let scalar0 = array + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); let list0 = scalar0.as_list(); assert_eq!(list0.len(), 2); if let Some(list0_items) = list0.elements() { @@ -567,7 +592,9 @@ mod tests { assert_eq!(list0_items[1].as_primitive().typed_value::(), Some(2)); } - let scalar1 = array.scalar_at(1).unwrap(); + let scalar1 = array + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); let list1 = scalar1.as_list(); assert_eq!(list1.len(), 3); if let Some(list1_items) = list1.elements() { @@ -576,7 +603,9 @@ mod tests { assert_eq!(list1_items[2].as_primitive().typed_value::(), Some(5)); } - let scalar2 = array.scalar_at(2).unwrap(); + let scalar2 = array + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); let list2 = scalar2.as_list(); assert!(list2.is_null()); // This should be null. diff --git a/vortex-array/src/builders/listview.rs b/vortex-array/src/builders/listview.rs index b339dc430f7..3c7bb8e8885 100644 --- a/vortex-array/src/builders/listview.rs +++ b/vortex-array/src/builders/listview.rs @@ -20,7 +20,10 @@ use vortex_mask::Mask; use crate::ArrayRef; use crate::Canonical; -use crate::ToCanonical; +use crate::LEGACY_SESSION; +#[expect(deprecated)] +use crate::ToCanonical as _; +use crate::VortexSessionExecute; use crate::array::IntoArray; use crate::arrays::ListViewArray; use crate::arrays::PrimitiveArray; @@ -291,28 +294,14 @@ impl ArrayBuilder for ListViewBuilder { } unsafe fn extend_from_array_unchecked(&mut self, array: &ArrayRef) { + #[expect(deprecated)] let listview = array.to_listview(); if listview.is_empty() { return; } - // If we do not have the guarantee that the array is zero-copyable to a list, then we have - // to manually append each scalar. - if !listview.is_zero_copy_to_list() { - for i in 0..listview.len() { - let list = listview - .scalar_at(i) - .vortex_expect("scalar_at failed in extend_from_array_unchecked"); - - self.append_scalar(&list) - .vortex_expect("was unable to extend the `ListViewBuilder`") - } - - return; - } - - // Otherwise, after removing any leading and trailing elements, we can simply bulk append - // the entire array. + // Normalize to an exact zero-copy-to-list layout and then bulk append. This avoids the + // very expensive scalar_at-per-list path for overlapping / out-of-order list views. let listview = listview .rebuild(ListViewRebuildMode::MakeExact) .vortex_expect("ListViewArray::rebuild(MakeExact) failed in extend_from_array"); @@ -320,8 +309,10 @@ impl ArrayBuilder for ListViewBuilder { self.nulls.append_validity_mask( array - .validity_mask() - .vortex_expect("validity_mask in extend_from_array_unchecked"), + .validity() + .vortex_expect("validity_mask in extend_from_array_unchecked") + .execute_mask(array.len(), &mut LEGACY_SESSION.create_execution_ctx()) + .vortex_expect("Failed to compute validity mask"), ); // Bulk append the new elements (which should have no gaps or overlaps). @@ -352,6 +343,7 @@ impl ArrayBuilder for ListViewBuilder { let uninit_range = self.offsets_builder.uninit_range(extend_length); // This should be cheap because we didn't compress after rebuilding. + #[expect(deprecated)] let new_offsets = listview.offsets().to_primitive(); match_each_integer_ptype!(new_offsets.ptype(), |A| { @@ -429,11 +421,13 @@ fn adjust_and_extend_offsets<'a, O: IntegerPType, A: IntegerPType>( mod tests { use std::sync::Arc; + use vortex_buffer::buffer; use vortex_error::VortexExpect; use super::ListViewBuilder; use crate::IntoArray; use crate::arrays::ListArray; + use crate::arrays::ListViewArray; use crate::arrays::listview::ListViewArrayExt; use crate::assert_arrays_eq; use crate::builders::ArrayBuilder; @@ -443,6 +437,7 @@ mod tests { use crate::dtype::Nullability::Nullable; use crate::dtype::PType::I32; use crate::scalar::Scalar; + use crate::validity::Validity; #[test] fn test_empty() { @@ -696,6 +691,50 @@ mod tests { ); } + #[test] + fn test_extend_from_array_overlapping_listview() { + let dtype: Arc = Arc::new(I32.into()); + + // Non-ZCTL source: + // - List 0: [10, 20] + // - List 1: null (size is intentionally non-zero in source metadata) + // - List 2: [10] + let source = unsafe { + ListViewArray::new_unchecked( + buffer![10i32, 20, 30].into_array(), + buffer![0u32, 1, 0].into_array(), + buffer![2u8, 2, 1].into_array(), + Validity::from_iter([true, false, true]), + ) + }; + assert!(!source.is_zero_copy_to_list()); + + let mut builder = + ListViewBuilder::::with_capacity(Arc::clone(&dtype), Nullable, 0, 0); + builder.extend_from_array(&source.into_array()); + + let listview = builder.finish_into_listview(); + assert_eq!(listview.len(), 3); + assert!(listview.is_zero_copy_to_list()); + + assert_arrays_eq!( + listview.list_elements_at(0).unwrap(), + PrimitiveArray::from_iter([10i32, 20]) + ); + assert!( + !listview + .validity() + .vortex_expect("listview validity should be derivable") + .is_valid(1) + .unwrap() + ); + assert_eq!(listview.list_elements_at(1).unwrap().len(), 0); + assert_arrays_eq!( + listview.list_elements_at(2).unwrap(), + PrimitiveArray::from_iter([10i32]) + ); + } + #[test] fn test_error_append_null_to_non_nullable() { let dtype: Arc = Arc::new(I32.into()); @@ -719,8 +758,6 @@ mod tests { #[test] fn test_append_array_as_list() { - use vortex_buffer::buffer; - let dtype: Arc = Arc::new(I32.into()); let mut builder = ListViewBuilder::::with_capacity(Arc::clone(&dtype), NonNullable, 20, 10); diff --git a/vortex-array/src/builders/mod.rs b/vortex-array/src/builders/mod.rs index 72c6f286e42..60e96a2b806 100644 --- a/vortex-array/src/builders/mod.rs +++ b/vortex-array/src/builders/mod.rs @@ -11,6 +11,7 @@ //! ``` //! use vortex_array::builders::{builder_with_capacity, ArrayBuilder}; //! use vortex_array::dtype::{DType, Nullability}; +//! use vortex_array::{LEGACY_SESSION, VortexSessionExecute}; //! //! // Create a new builder for string data. //! let mut builder = builder_with_capacity(&DType::Utf8(Nullability::NonNullable), 4); @@ -21,11 +22,12 @@ //! builder.append_scalar(&"d".into()).unwrap(); //! //! let strings = builder.finish(); +//! let mut ctx = LEGACY_SESSION.create_execution_ctx(); //! -//! assert_eq!(strings.scalar_at(0).unwrap(), "a".into()); -//! assert_eq!(strings.scalar_at(1).unwrap(), "b".into()); -//! assert_eq!(strings.scalar_at(2).unwrap(), "c".into()); -//! assert_eq!(strings.scalar_at(3).unwrap(), "d".into()); +//! assert_eq!(strings.execute_scalar(0, &mut ctx).unwrap(), "a".into()); +//! assert_eq!(strings.execute_scalar(1, &mut ctx).unwrap(), "b".into()); +//! assert_eq!(strings.execute_scalar(2, &mut ctx).unwrap(), "c".into()); +//! assert_eq!(strings.execute_scalar(3, &mut ctx).unwrap(), "d".into()); //! ``` use std::any::Any; @@ -223,6 +225,7 @@ pub trait ArrayBuilder: Send { /// ``` /// use vortex_array::builders::{builder_with_capacity, ArrayBuilder}; /// use vortex_array::dtype::{DType, Nullability}; +/// use vortex_array::{LEGACY_SESSION, VortexSessionExecute}; /// /// // Create a new builder for string data. /// let mut builder = builder_with_capacity(&DType::Utf8(Nullability::NonNullable), 4); @@ -233,11 +236,12 @@ pub trait ArrayBuilder: Send { /// builder.append_scalar(&"d".into()).unwrap(); /// /// let strings = builder.finish(); +/// let mut ctx = LEGACY_SESSION.create_execution_ctx(); /// -/// assert_eq!(strings.scalar_at(0).unwrap(), "a".into()); -/// assert_eq!(strings.scalar_at(1).unwrap(), "b".into()); -/// assert_eq!(strings.scalar_at(2).unwrap(), "c".into()); -/// assert_eq!(strings.scalar_at(3).unwrap(), "d".into()); +/// assert_eq!(strings.execute_scalar(0, &mut ctx).unwrap(), "a".into()); +/// assert_eq!(strings.execute_scalar(1, &mut ctx).unwrap(), "b".into()); +/// assert_eq!(strings.execute_scalar(2, &mut ctx).unwrap(), "c".into()); +/// assert_eq!(strings.execute_scalar(3, &mut ctx).unwrap(), "d".into()); /// ``` pub fn builder_with_capacity(dtype: &DType, capacity: usize) -> Box { match dtype { diff --git a/vortex-array/src/builders/primitive.rs b/vortex-array/src/builders/primitive.rs index 8321c03500c..59381fc602f 100644 --- a/vortex-array/src/builders/primitive.rs +++ b/vortex-array/src/builders/primitive.rs @@ -12,12 +12,15 @@ use vortex_mask::Mask; use crate::ArrayRef; use crate::IntoArray; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::arrays::PrimitiveArray; use crate::builders::ArrayBuilder; use crate::builders::DEFAULT_BUILDER_CAPACITY; use crate::builders::LazyBitBufferBuilder; use crate::canonical::Canonical; -use crate::canonical::ToCanonical; +#[expect(deprecated)] +use crate::canonical::ToCanonical as _; use crate::dtype::DType; use crate::dtype::NativePType; use crate::dtype::Nullability; @@ -175,6 +178,7 @@ impl ArrayBuilder for PrimitiveBuilder { } unsafe fn extend_from_array_unchecked(&mut self, array: &ArrayRef) { + #[expect(deprecated)] let array = array.to_primitive(); // This should be checked in `extend_from_array` but we can check it again. @@ -185,8 +189,17 @@ impl ArrayBuilder for PrimitiveBuilder { ); self.values.extend_from_slice(array.as_slice::()); - self.nulls - .append_validity_mask(array.validity_mask().vortex_expect("validity_mask")); + self.nulls.append_validity_mask( + array + .as_ref() + .validity() + .vortex_expect("validity_mask") + .execute_mask( + array.as_ref().len(), + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .vortex_expect("Failed to compute validity mask"), + ); } fn reserve_exact(&mut self, additional: usize) { @@ -427,9 +440,24 @@ mod tests { let array = builder.finish_into_primitive(); assert_eq!(array.len(), 3); // Check validity using scalar_at - nulls will return is_null() = true. - assert!(!array.scalar_at(0).unwrap().is_null()); - assert!(array.scalar_at(1).unwrap().is_null()); - assert!(!array.scalar_at(2).unwrap().is_null()); + assert!( + !array + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); + assert!( + array + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); + assert!( + !array + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); } /// REGRESSION TEST: This test verifies that `append_mask` validates the mask length. @@ -517,13 +545,38 @@ mod tests { assert_eq!(array.as_slice::(), &[100, 200, 10, 20, 30]); // Check validity - the first two should be valid (from append_value). - assert!(!array.scalar_at(0).unwrap().is_null()); // initial value 100 - assert!(!array.scalar_at(1).unwrap().is_null()); // initial value 200 + assert!( + !array + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); // initial value 100 + assert!( + !array + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); // initial value 200 // Check the range items with modified validity. - assert!(!array.scalar_at(2).unwrap().is_null()); // range index 0 - set to valid - assert!(array.scalar_at(3).unwrap().is_null()); // range index 1 - left as null - assert!(!array.scalar_at(4).unwrap().is_null()); // range index 2 - set to valid + assert!( + !array + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); // range index 0 - set to valid + assert!( + array + .execute_scalar(3, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); // range index 1 - left as null + assert!( + !array + .execute_scalar(4, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); // range index 2 - set to valid } /// Test that creating a zero-length uninit range panics. diff --git a/vortex-array/src/builders/struct_.rs b/vortex-array/src/builders/struct_.rs index 87f7ecb5e88..b7c737b3f1c 100644 --- a/vortex-array/src/builders/struct_.rs +++ b/vortex-array/src/builders/struct_.rs @@ -13,6 +13,8 @@ use vortex_mask::Mask; use crate::ArrayRef; use crate::IntoArray; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::arrays::StructArray; use crate::arrays::struct_::StructArrayExt; use crate::builders::ArrayBuilder; @@ -20,7 +22,8 @@ use crate::builders::DEFAULT_BUILDER_CAPACITY; use crate::builders::LazyBitBufferBuilder; use crate::builders::builder_with_capacity; use crate::canonical::Canonical; -use crate::canonical::ToCanonical; +#[expect(deprecated)] +use crate::canonical::ToCanonical as _; use crate::dtype::DType; use crate::dtype::Nullability; use crate::dtype::StructFields; @@ -166,6 +169,7 @@ impl ArrayBuilder for StructBuilder { } unsafe fn extend_from_array_unchecked(&mut self, array: &ArrayRef) { + #[expect(deprecated)] let array = array.to_struct(); for (a, builder) in array @@ -175,8 +179,13 @@ impl ArrayBuilder for StructBuilder { builder.extend_from_array(a); } - self.nulls - .append_validity_mask(array.validity_mask().vortex_expect("validity_mask")); + self.nulls.append_validity_mask( + array + .validity() + .vortex_expect("validity_mask") + .execute_mask(array.len(), &mut LEGACY_SESSION.create_execution_ctx()) + .vortex_expect("Failed to compute validity mask"), + ); } fn reserve_exact(&mut self, capacity: usize) { @@ -203,6 +212,8 @@ impl ArrayBuilder for StructBuilder { #[cfg(test)] mod tests { use crate::IntoArray; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::arrays::PrimitiveArray; use crate::arrays::VarBinArray; use crate::assert_arrays_eq; @@ -246,7 +257,12 @@ mod tests { let struct_ = builder.finish(); assert_eq!(struct_.len(), 3); assert_eq!(struct_.dtype(), &dtype); - assert_eq!(struct_.valid_count().unwrap(), 1); + assert_eq!( + struct_ + .valid_count(&mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + 1 + ); } #[test] diff --git a/vortex-array/src/builders/tests.rs b/vortex-array/src/builders/tests.rs index 1a738954058..a46f97f1966 100644 --- a/vortex-array/src/builders/tests.rs +++ b/vortex-array/src/builders/tests.rs @@ -6,6 +6,8 @@ use std::sync::Arc; use rstest::rstest; use vortex_error::VortexExpect; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::builders::ArrayBuilder; use crate::builders::builder_with_capacity; use crate::dtype::DType; @@ -90,8 +92,12 @@ fn test_append_zeros_matches_default_value(#[case] dtype: DType) { // Compare each element. for i in 0..num_elements { - let scalar_zeros = array_zeros.scalar_at(i).unwrap(); - let scalar_manual = array_manual.scalar_at(i).unwrap(); + let scalar_zeros = array_zeros + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); + let scalar_manual = array_manual + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert_eq!( scalar_zeros, scalar_manual, @@ -188,7 +194,9 @@ fn test_append_defaults_behavior(#[case] dtype: DType, #[case] should_be_null: b assert_eq!(array.len(), 3); for i in 0..3 { - let scalar = array.scalar_at(i).unwrap(); + let scalar = array + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); if should_be_null { assert!(scalar.is_null(), "Element at index {} should be null", i); } else { @@ -229,6 +237,7 @@ where // Get canonical arrays using both methods. let canonical_direct = builder1.finish_into_canonical(); + #[expect(deprecated)] let canonical_indirect = builder2 .finish() .to_canonical() @@ -243,8 +252,12 @@ where // Compare each element. for i in 0..array_direct.len() { - let scalar_direct = array_direct.scalar_at(i).unwrap(); - let scalar_indirect = array_indirect.scalar_at(i).unwrap(); + let scalar_direct = array_direct + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); + let scalar_indirect = array_indirect + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert_eq!( scalar_direct, scalar_indirect, @@ -533,13 +546,17 @@ fn test_append_scalar_comprehensive(#[case] dtype: DType) { // Verify each scalar matches. for (i, expected_scalar) in scalars.iter().enumerate() { - let actual_scalar = array.scalar_at(i).unwrap(); + let actual_scalar = array + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert_scalars_equal(&actual_scalar, expected_scalar, &dtype, i); } // If nullable, verify the last element is null. if dtype.is_nullable() { - let null_scalar = array.scalar_at(num_elements).unwrap(); + let null_scalar = array + .execute_scalar(num_elements, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert!( null_scalar.is_null(), "Last element should be null for nullable dtype" @@ -548,7 +565,7 @@ fn test_append_scalar_comprehensive(#[case] dtype: DType) { } /// Helper function to create test scalars for a given dtype. -#[allow(clippy::cast_possible_truncation)] +#[expect(clippy::cast_possible_truncation)] fn create_test_scalars_for_dtype(dtype: &DType, count: usize) -> Vec { let mut scalars = Vec::with_capacity(count); @@ -685,16 +702,62 @@ fn test_append_scalar_mixed_nulls(#[case] dtype: DType) { assert_eq!(array.len(), 5); // Check the pattern. - assert!(!array.scalar_at(0).unwrap().is_null()); - assert!(array.scalar_at(1).unwrap().is_null()); - assert!(!array.scalar_at(2).unwrap().is_null()); - assert!(array.scalar_at(3).unwrap().is_null()); - assert!(!array.scalar_at(4).unwrap().is_null()); + assert!( + !array + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); + assert!( + array + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); + assert!( + !array + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); + assert!( + array + .execute_scalar(3, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); + assert!( + !array + .execute_scalar(4, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_null() + ); // Verify non-null values match. - assert_scalars_equal(&array.scalar_at(0).unwrap(), &test_scalars[0], &dtype, 0); - assert_scalars_equal(&array.scalar_at(2).unwrap(), &test_scalars[1], &dtype, 2); - assert_scalars_equal(&array.scalar_at(4).unwrap(), &test_scalars[2], &dtype, 4); + assert_scalars_equal( + &array + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + &test_scalars[0], + &dtype, + 0, + ); + assert_scalars_equal( + &array + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + &test_scalars[1], + &dtype, + 2, + ); + assert_scalars_equal( + &array + .execute_scalar(4, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + &test_scalars[2], + &dtype, + 4, + ); } /// Test that `append_scalar` correctly rejects scalars with wrong dtype. @@ -745,7 +808,9 @@ fn test_append_scalar_repeated_same_instance() { // All values should be 42. for i in 0..5 { - let actual = array.scalar_at(i).unwrap(); + let actual = array + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert_eq!( actual.as_primitive().typed_value::(), Some(42), diff --git a/vortex-array/src/builders/varbinview.rs b/vortex-array/src/builders/varbinview.rs index 294ac5cd551..9150e8e70e1 100644 --- a/vortex-array/src/builders/varbinview.rs +++ b/vortex-array/src/builders/varbinview.rs @@ -20,13 +20,16 @@ use vortex_utils::aliases::hash_map::HashMap; use crate::ArrayRef; use crate::IntoArray; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::arrays::VarBinViewArray; use crate::arrays::varbinview::build_views::BinaryView; use crate::arrays::varbinview::compact::BufferUtilization; use crate::builders::ArrayBuilder; use crate::builders::LazyBitBufferBuilder; use crate::canonical::Canonical; -use crate::canonical::ToCanonical; +#[expect(deprecated)] +use crate::canonical::ToCanonical as _; use crate::dtype::DType; use crate::scalar::Scalar; @@ -292,10 +295,21 @@ impl ArrayBuilder for VarBinViewBuilder { } unsafe fn extend_from_array_unchecked(&mut self, array: &ArrayRef) { + #[expect(deprecated)] let array = array.to_varbinview(); self.flush_in_progress(); - self.push_only_validity_mask(array.validity_mask().vortex_expect("validity_mask")); + self.push_only_validity_mask( + array + .as_ref() + .validity() + .vortex_expect("validity_mask") + .execute_mask( + array.as_ref().len(), + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .vortex_expect("Failed to compute validity mask"), + ); let view_adjustment = self.completed @@ -312,7 +326,16 @@ impl ArrayBuilder for VarBinViewBuilder { .map(|view| adjustment.adjust_view(view)), ), ViewAdjustment::Rewriting(adjustment) => { - match array.validity_mask().vortex_expect("validity_mask") { + match array + .as_ref() + .validity() + .vortex_expect("validity_mask") + .execute_mask( + array.as_ref().len(), + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .vortex_expect("Failed to compute validity mask") + { Mask::AllTrue(_) => { for (idx, &view) in array.views().iter().enumerate() { let new_view = self.push_view(view, &adjustment, &array, idx); @@ -391,7 +414,7 @@ impl Default for CompletedBuffers { } // Self::push enforces len < u32::max -#[allow(clippy::cast_possible_truncation)] +#[expect(clippy::cast_possible_truncation)] impl CompletedBuffers { fn len(&self) -> u32 { match self { @@ -477,7 +500,7 @@ pub struct DeduplicatedBuffers { impl DeduplicatedBuffers { // Self::push enforces len < u32::max - #[allow(clippy::cast_possible_truncation)] + #[expect(clippy::cast_possible_truncation)] fn len(&self) -> u32 { self.buffers.len() as u32 } @@ -1037,7 +1060,7 @@ mod tests { // Verify the value was stored correctly let retrieved = array - .scalar_at(0) + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) .unwrap() .as_binary() .value() diff --git a/vortex-array/src/builtins.rs b/vortex-array/src/builtins.rs index b4a3eb23984..dcbe934097e 100644 --- a/vortex-array/src/builtins.rs +++ b/vortex-array/src/builtins.rs @@ -28,6 +28,7 @@ use crate::scalar_fn::fns::binary::Binary; use crate::scalar_fn::fns::cast::Cast; use crate::scalar_fn::fns::fill_null::FillNull; use crate::scalar_fn::fns::get_item::GetItem; +use crate::scalar_fn::fns::is_not_null::IsNotNull; use crate::scalar_fn::fns::is_null::IsNull; use crate::scalar_fn::fns::list_contains::ListContains; use crate::scalar_fn::fns::mask::Mask; @@ -49,6 +50,9 @@ pub trait ExprBuiltins: Sized { /// Is null check. fn is_null(&self) -> VortexResult; + /// Is not null check. + fn is_not_null(&self) -> VortexResult; + /// Mask the expression using the given boolean mask. /// The resulting expression's validity is the intersection of the original expression's /// validity. @@ -84,6 +88,10 @@ impl ExprBuiltins for Expression { IsNull.try_new_expr(EmptyOptions, [self.clone()]) } + fn is_not_null(&self) -> VortexResult { + IsNotNull.try_new_expr(EmptyOptions, [self.clone()]) + } + fn mask(&self, mask: Expression) -> VortexResult { Mask.try_new_expr(EmptyOptions, [self.clone(), mask]) } @@ -118,6 +126,9 @@ pub trait ArrayBuiltins: Sized { /// Is null check. fn is_null(&self) -> VortexResult; + /// Is not null check. + fn is_not_null(&self) -> VortexResult; + /// Mask the array using the given boolean mask. /// The resulting array's validity is the intersection of the original array's validity /// and the mask's validity. @@ -182,6 +193,12 @@ impl ArrayBuiltins for ArrayRef { .optimize() } + fn is_not_null(&self) -> VortexResult { + IsNotNull + .try_new_array(self.len(), EmptyOptions, [self.clone()])? + .optimize() + } + fn mask(self, mask: ArrayRef) -> VortexResult { Mask.try_new_array(self.len(), EmptyOptions, [self, mask])? .optimize() diff --git a/vortex-array/src/canonical.rs b/vortex-array/src/canonical.rs index ffae20f09c8..150172b0df8 100644 --- a/vortex-array/src/canonical.rs +++ b/vortex-array/src/canonical.rs @@ -425,94 +425,105 @@ impl IntoArray for Canonical { /// # Canonicalization /// /// This trait has a blanket implementation for all types implementing [ToCanonical]. +#[deprecated(note = "use `array.execute::(ctx)` instead")] pub trait ToCanonical { /// Canonicalize into a [`NullArray`] if the target is [`Null`](DType::Null) typed. + #[deprecated(note = "use `array.execute::(ctx)` instead")] fn to_null(&self) -> NullArray; /// Canonicalize into a [`BoolArray`] if the target is [`Bool`](DType::Bool) typed. + #[deprecated(note = "use `array.execute::(ctx)` instead")] fn to_bool(&self) -> BoolArray; /// Canonicalize into a [`PrimitiveArray`] if the target is [`Primitive`](DType::Primitive) /// typed. + #[deprecated(note = "use `array.execute::(ctx)` instead")] fn to_primitive(&self) -> PrimitiveArray; /// Canonicalize into a [`DecimalArray`] if the target is [`Decimal`](DType::Decimal) /// typed. + #[deprecated(note = "use `array.execute::(ctx)` instead")] fn to_decimal(&self) -> DecimalArray; /// Canonicalize into a [`StructArray`] if the target is [`Struct`](DType::Struct) typed. + #[deprecated(note = "use `array.execute::(ctx)` instead")] fn to_struct(&self) -> StructArray; /// Canonicalize into a [`ListViewArray`] if the target is [`List`](DType::List) typed. + #[deprecated(note = "use `array.execute::(ctx)` instead")] fn to_listview(&self) -> ListViewArray; /// Canonicalize into a [`FixedSizeListArray`] if the target is [`List`](DType::FixedSizeList) /// typed. + #[deprecated(note = "use `array.execute::(ctx)` instead")] fn to_fixed_size_list(&self) -> FixedSizeListArray; /// Canonicalize into a [`VarBinViewArray`] if the target is [`Utf8`](DType::Utf8) /// or [`Binary`](DType::Binary) typed. + #[deprecated(note = "use `array.execute::(ctx)` instead")] fn to_varbinview(&self) -> VarBinViewArray; /// Canonicalize into an [`ExtensionArray`] if the array is [`Extension`](DType::Extension) /// typed. + #[deprecated(note = "use `array.execute::(ctx)` instead")] fn to_extension(&self) -> ExtensionArray; } // Blanket impl for all Array encodings. +#[expect(deprecated)] impl ToCanonical for ArrayRef { fn to_null(&self) -> NullArray { - self.to_canonical() - .vortex_expect("to_canonical failed") - .into_null() + #[expect(deprecated)] + let result = self.to_canonical().vortex_expect("to_canonical failed"); + result.into_null() } fn to_bool(&self) -> BoolArray { - self.to_canonical() - .vortex_expect("to_canonical failed") - .into_bool() + #[expect(deprecated)] + let result = self.to_canonical().vortex_expect("to_canonical failed"); + result.into_bool() } fn to_primitive(&self) -> PrimitiveArray { - self.to_canonical() - .vortex_expect("to_canonical failed") - .into_primitive() + #[expect(deprecated)] + let result = self.to_canonical().vortex_expect("to_canonical failed"); + result.into_primitive() } fn to_decimal(&self) -> DecimalArray { - self.to_canonical() - .vortex_expect("to_canonical failed") - .into_decimal() + #[expect(deprecated)] + let result = self.to_canonical().vortex_expect("to_canonical failed"); + result.into_decimal() } fn to_struct(&self) -> StructArray { - self.to_canonical() - .vortex_expect("to_canonical failed") - .into_struct() + #[expect(deprecated)] + let result = self.to_canonical().vortex_expect("to_canonical failed"); + result.into_struct() } fn to_listview(&self) -> ListViewArray { - self.to_canonical() - .vortex_expect("to_canonical failed") - .into_listview() + #[expect(deprecated)] + let result = self.to_canonical().vortex_expect("to_canonical failed"); + result.into_listview() } fn to_fixed_size_list(&self) -> FixedSizeListArray { - self.to_canonical() - .vortex_expect("to_canonical failed") - .into_fixed_size_list() + #[expect(deprecated)] + let result = self.to_canonical().vortex_expect("to_canonical failed"); + result.into_fixed_size_list() } fn to_varbinview(&self) -> VarBinViewArray { - self.to_canonical() - .vortex_expect("to_canonical failed") - .into_varbinview() + #[expect(deprecated)] + let result = self.to_canonical().vortex_expect("to_canonical failed"); + result.into_varbinview() } fn to_extension(&self) -> ExtensionArray { - self.to_canonical() - .vortex_expect("to_canonical failed") - .into_extension() + #[expect(deprecated)] + let result = self.to_canonical().vortex_expect("to_canonical failed"); + result.into_extension() } } @@ -548,7 +559,7 @@ impl Executable for CanonicalValidity { match array.execute::(ctx)? { n @ Canonical::Null(_) => Ok(CanonicalValidity(n)), Canonical::Bool(b) => { - let validity = child_to_validity(&b.slots()[0], b.dtype().nullability()); + let validity = child_to_validity(b.slots()[0].as_ref(), b.dtype().nullability()); let len = b.len(); let BoolDataParts { bits, offset, len } = b.into_data().into_parts(len); Ok(CanonicalValidity(Canonical::Bool( @@ -667,7 +678,7 @@ impl Executable for RecursiveCanonical { match array.execute::(ctx)? { n @ Canonical::Null(_) => Ok(RecursiveCanonical(n)), Canonical::Bool(b) => { - let validity = child_to_validity(&b.slots()[0], b.dtype().nullability()); + let validity = child_to_validity(b.slots()[0].as_ref(), b.dtype().nullability()); let len = b.len(); let BoolDataParts { bits, offset, len } = b.into_data().into_parts(len); Ok(RecursiveCanonical(Canonical::Bool( @@ -1056,13 +1067,16 @@ mod test { use crate::ArrayRef; use crate::IntoArray; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::arrays::ConstantArray; + use crate::arrow::ArrowArrayExecutor; use crate::arrow::FromArrowArray; - use crate::arrow::IntoArrowArray; use crate::canonical::StructArray; #[test] fn test_canonicalize_nested_struct() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Create a struct array with multiple internal components. let nested_struct_array = StructArray::from_fields(&[ ("a", buffer![1u64].into_array()), @@ -1084,7 +1098,7 @@ mod test { let arrow_struct = nested_struct_array .into_array() - .into_arrow_preferred() + .execute_arrow(None, &mut ctx) .unwrap() .as_any() .downcast_ref::() @@ -1119,6 +1133,7 @@ mod test { #[test] fn roundtrip_struct() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let mut nulls = NullBufferBuilder::new(6); nulls.append_n_non_nulls(4); nulls.append_null(); @@ -1154,12 +1169,16 @@ mod test { assert_eq!( &arrow_struct, - vortex_struct.into_arrow_preferred().unwrap().as_struct() + vortex_struct + .execute_arrow(None, &mut ctx) + .unwrap() + .as_struct() ); } #[test] fn roundtrip_list() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let names = Arc::new(StringArray::from_iter(vec![ Some("Joseph"), Some("Angela"), @@ -1176,7 +1195,9 @@ mod test { let vortex_list = ArrayRef::from_arrow(&arrow_list, true).unwrap(); - let rt_arrow_list = vortex_list.into_arrow(list_data_type).unwrap(); + let rt_arrow_list = vortex_list + .execute_arrow(Some(list_data_type), &mut ctx) + .unwrap(); assert_eq!( (Arc::new(arrow_list.clone()) as ArrowArrayRef).as_ref(), diff --git a/vortex-array/src/compute/conformance/binary_numeric.rs b/vortex-array/src/compute/conformance/binary_numeric.rs index c36367746e7..ce3eba940df 100644 --- a/vortex-array/src/compute/conformance/binary_numeric.rs +++ b/vortex-array/src/compute/conformance/binary_numeric.rs @@ -31,7 +31,8 @@ use crate::ArrayRef; use crate::IntoArray; use crate::LEGACY_SESSION; use crate::RecursiveCanonical; -use crate::ToCanonical; +#[expect(deprecated)] +use crate::ToCanonical as _; use crate::VortexSessionExecute; use crate::arrays::ConstantArray; use crate::builtins::ArrayBuiltins; @@ -47,7 +48,7 @@ fn to_vec_of_scalar(array: &ArrayRef) -> Vec { (0..array.len()) .map(|index| { array - .scalar_at(index) + .execute_scalar(index, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") }) .collect_vec() @@ -92,6 +93,7 @@ fn test_standard_binary_numeric(array: ArrayRef) where Scalar: From, { + #[expect(deprecated)] let canonicalized_array = array.to_primitive(); let original_values = to_vec_of_scalar(&canonicalized_array.into_array()); @@ -328,6 +330,7 @@ where T: NativePType + Num + Copy + std::fmt::Debug, Scalar: From, { + #[expect(deprecated)] let canonicalized_array = array.to_primitive(); let original_values = to_vec_of_scalar(&canonicalized_array.into_array()); diff --git a/vortex-array/src/compute/conformance/cast.rs b/vortex-array/src/compute/conformance/cast.rs index cab73acc521..be5142d8f53 100644 --- a/vortex-array/src/compute/conformance/cast.rs +++ b/vortex-array/src/compute/conformance/cast.rs @@ -74,10 +74,10 @@ fn test_cast_identity(array: &ArrayRef) { for i in 0..array.len().min(10) { assert_eq!( array - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), result - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); } @@ -109,7 +109,7 @@ fn test_cast_from_null(array: &ArrayRef) { for i in 0..array.len().min(10) { assert!( result - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") .is_null() ); @@ -129,7 +129,7 @@ fn test_cast_from_null(array: &ArrayRef) { fn test_cast_to_non_nullable(array: &ArrayRef) { if array - .invalid_count() + .invalid_count(&mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("invalid_count should succeed in conformance test") == 0 { @@ -141,10 +141,10 @@ fn test_cast_to_non_nullable(array: &ArrayRef) { for i in 0..array.len().min(10) { assert_eq!( array - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), non_nullable - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); } @@ -157,10 +157,10 @@ fn test_cast_to_non_nullable(array: &ArrayRef) { for i in 0..array.len().min(10) { assert_eq!( array - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), back_to_nullable - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); } @@ -190,10 +190,10 @@ fn test_cast_to_nullable(array: &ArrayRef) { for i in 0..array.len().min(10) { assert_eq!( array - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), nullable - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); } @@ -206,9 +206,9 @@ fn test_cast_to_nullable(array: &ArrayRef) { for i in 0..array.len().min(10) { assert_eq!( array - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), - back.scalar_at(i) + back.execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); } @@ -286,18 +286,22 @@ fn test_cast_to_primitive(array: &ArrayRef, target_ptype: PType, test_round_trip }); assert_eq!( array - .validity_mask() - .vortex_expect("validity_mask should succeed in conformance test"), + .validity() + .vortex_expect("validity_mask should succeed in conformance test") + .execute_mask(array.len(), &mut LEGACY_SESSION.create_execution_ctx()) + .vortex_expect("Failed to compute validity mask"), casted - .validity_mask() + .validity() .vortex_expect("validity_mask should succeed in conformance test") + .execute_mask(casted.len(), &mut LEGACY_SESSION.create_execution_ctx()) + .vortex_expect("Failed to compute validity mask") ); for i in 0..array.len().min(10) { let original = array - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); let casted = casted - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); assert_eq!( original diff --git a/vortex-array/src/compute/conformance/consistency.rs b/vortex-array/src/compute/conformance/consistency.rs index dec1f8d1d61..a6d2996b091 100644 --- a/vortex-array/src/compute/conformance/consistency.rs +++ b/vortex-array/src/compute/conformance/consistency.rs @@ -89,10 +89,10 @@ fn test_filter_take_consistency(array: &ArrayRef) { for i in 0..filtered.len() { let filtered_val = filtered - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); let taken_val = taken - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); assert_eq!( filtered_val, taken_val, @@ -163,10 +163,10 @@ fn test_double_mask_consistency(array: &ArrayRef) { for i in 0..double_masked.len() { let double_val = double_masked - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); let direct_val = directly_masked - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); assert_eq!( double_val, direct_val, @@ -213,10 +213,10 @@ fn test_filter_identity(array: &ArrayRef) { for i in 0..len { let original_val = array - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); let filtered_val = filtered - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); assert_eq!( filtered_val, original_val, @@ -269,10 +269,10 @@ fn test_mask_identity(array: &ArrayRef) { for i in 0..len { let original_val = array - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); let masked_val = masked - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); let expected_val = original_val.clone().into_nullable(); assert_eq!( @@ -328,10 +328,10 @@ fn test_slice_filter_consistency(array: &ArrayRef) { for i in 0..filtered.len() { let filtered_val = filtered - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); let sliced_val = sliced - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); assert_eq!( filtered_val, sliced_val, @@ -383,10 +383,10 @@ fn test_take_slice_consistency(array: &ArrayRef) { for i in 0..taken.len() { let taken_val = taken - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); let sliced_val = sliced - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); assert_eq!( taken_val, sliced_val, @@ -416,26 +416,26 @@ fn test_filter_preserves_order(array: &ArrayRef) { if len >= 4 { assert_eq!( filtered - .scalar_at(0) + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), array - .scalar_at(0) + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); assert_eq!( filtered - .scalar_at(1) + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), array - .scalar_at(2) + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); assert_eq!( filtered - .scalar_at(2) + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), array - .scalar_at(3) + .execute_scalar(3, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); } @@ -458,10 +458,10 @@ fn test_take_repeated_indices(array: &ArrayRef) { for i in 0..3 { assert_eq!( taken - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), array - .scalar_at(0) + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); } @@ -498,10 +498,10 @@ fn test_mask_filter_null_consistency(array: &ArrayRef) { for i in 0..filtered.len() { assert_eq!( filtered - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), direct_filtered - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); } @@ -555,10 +555,10 @@ fn test_take_preserves_properties(array: &ArrayRef) { for i in 0..len { assert_eq!( taken - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), array - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); } @@ -610,11 +610,11 @@ fn test_nullable_indices_consistency(array: &ArrayRef) { // Check first element (from index 0) let expected_0 = array - .scalar_at(0) + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") .into_nullable(); let actual_0 = taken - .scalar_at(0) + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); assert_eq!( actual_0, expected_0, @@ -624,7 +624,7 @@ fn test_nullable_indices_consistency(array: &ArrayRef) { // Check second element (should be null) let actual_1 = taken - .scalar_at(1) + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); assert!( actual_1.is_null(), @@ -633,11 +633,11 @@ fn test_nullable_indices_consistency(array: &ArrayRef) { // Check third element (from index 2) let expected_2 = array - .scalar_at(2) + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") .into_nullable(); let actual_2 = taken - .scalar_at(2) + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); assert_eq!( actual_2, expected_2, @@ -672,10 +672,10 @@ fn test_large_array_consistency(array: &ArrayRef) { for i in 0..taken.len() { assert_eq!( taken - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), filtered - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); } @@ -714,7 +714,7 @@ fn test_comparison_inverse_consistency(array: &ArrayRef) { return; } else { array - .scalar_at(len / 2) + .execute_scalar(len / 2, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") }; @@ -740,10 +740,10 @@ fn test_comparison_inverse_consistency(array: &ArrayRef) { for i in 0..inverted_eq.len() { let inv_val = inverted_eq - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); let neq_val = neq_result - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); assert_eq!( inv_val, neq_val, @@ -768,10 +768,10 @@ fn test_comparison_inverse_consistency(array: &ArrayRef) { for i in 0..inverted_gt.len() { let inv_val = inverted_gt - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); let lte_val = lte_result - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); assert_eq!( inv_val, lte_val, @@ -796,10 +796,10 @@ fn test_comparison_inverse_consistency(array: &ArrayRef) { for i in 0..inverted_lt.len() { let inv_val = inverted_lt - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); let gte_val = gte_result - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); assert_eq!( inv_val, gte_val, @@ -842,7 +842,7 @@ fn test_comparison_symmetry_consistency(array: &ArrayRef) { return; } else { array - .scalar_at(len / 2) + .execute_scalar(len / 2, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") }; @@ -867,10 +867,10 @@ fn test_comparison_symmetry_consistency(array: &ArrayRef) { for i in 0..arr_gt_scalar.len() { let arr_gt = arr_gt_scalar - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); let scalar_lt = scalar_lt_arr - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); assert_eq!( arr_gt, scalar_lt, @@ -889,10 +889,10 @@ fn test_comparison_symmetry_consistency(array: &ArrayRef) { ) { for i in 0..arr_eq_scalar.len() { let arr_eq = arr_eq_scalar - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); let scalar_eq = scalar_eq_arr - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); assert_eq!( arr_eq, scalar_eq, @@ -951,10 +951,10 @@ fn test_boolean_demorgan_consistency(array: &ArrayRef) { for i in 0..not_a_and_b.len() { let left = not_a_and_b - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); let right = not_a_or_not_b - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); assert_eq!( left, right, @@ -979,10 +979,10 @@ fn test_boolean_demorgan_consistency(array: &ArrayRef) { for i in 0..not_a_or_b.len() { let left = not_a_or_b - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); let right = not_a_and_not_b - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); assert_eq!( left, right, @@ -1029,6 +1029,7 @@ fn test_slice_aggregate_consistency(array: &ArrayRef) { let sliced = array .slice(start..end) .vortex_expect("slice should succeed in conformance test"); + #[expect(deprecated)] let canonical = array.to_canonical().vortex_expect("to_canonical failed"); let canonical_sliced = canonical .into_array() @@ -1037,10 +1038,10 @@ fn test_slice_aggregate_consistency(array: &ArrayRef) { // Test null count through invalid_count let sliced_invalid_count = sliced - .invalid_count() + .invalid_count(&mut ctx) .vortex_expect("invalid_count should succeed in conformance test"); let canonical_invalid_count = canonical_sliced - .invalid_count() + .invalid_count(&mut ctx) .vortex_expect("invalid_count should succeed in conformance test"); assert_eq!( sliced_invalid_count, canonical_invalid_count, @@ -1130,6 +1131,7 @@ fn test_cast_slice_consistency(array: &ArrayRef) { let end = 7.min(len - 2).max(start + 1); // Ensure we have at least 1 element // Get canonical form of the original array + #[expect(deprecated)] let canonical = array.to_canonical().vortex_expect("to_canonical failed"); // Choose appropriate target dtype based on the array's type @@ -1254,10 +1256,10 @@ fn test_cast_slice_consistency(array: &ArrayRef) { .vortex_expect("slice should succeed in conformance test"); // Try to cast the sliced array (force execution via to_canonical) - let slice_then_cast = match sliced - .cast(target_dtype.clone()) - .and_then(|a| a.to_canonical().map(|c| c.into_array())) - { + let slice_then_cast = match sliced.cast(target_dtype.clone()).and_then(|a| { + #[expect(deprecated)] + a.to_canonical().map(|c| c.into_array()) + }) { Ok(result) => result, Err(_) => continue, // Skip if cast fails }; @@ -1274,14 +1276,14 @@ fn test_cast_slice_consistency(array: &ArrayRef) { // Compare each value against the canonical form for i in 0..slice_then_cast.len() { let slice_cast_val = slice_then_cast - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); // Get the corresponding value from the canonical array (adjusted for slice offset) let canonical_val = canonical .clone() .into_array() - .scalar_at(start + i) + .execute_scalar(start + i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); // Cast the canonical scalar to the target dtype @@ -1306,11 +1308,10 @@ fn test_cast_slice_consistency(array: &ArrayRef) { } // Also test the other way: cast then slice - let casted = match array - .clone() - .cast(target_dtype.clone()) - .and_then(|a| a.to_canonical().map(|c| c.into_array())) - { + let casted = match array.clone().cast(target_dtype.clone()).and_then(|a| { + #[expect(deprecated)] + a.to_canonical().map(|c| c.into_array()) + }) { Ok(result) => result, Err(_) => continue, // Skip if cast fails }; @@ -1327,10 +1328,10 @@ fn test_cast_slice_consistency(array: &ArrayRef) { for i in 0..slice_then_cast.len() { let slice_cast_val = slice_then_cast - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); let cast_slice_val = cast_then_slice - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); assert_eq!( slice_cast_val, cast_slice_val, diff --git a/vortex-array/src/compute/conformance/filter.rs b/vortex-array/src/compute/conformance/filter.rs index 6be08dff69a..7d5f4abcad8 100644 --- a/vortex-array/src/compute/conformance/filter.rs +++ b/vortex-array/src/compute/conformance/filter.rs @@ -6,6 +6,8 @@ use vortex_mask::Mask; use crate::ArrayRef; use crate::IntoArray; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::assert_arrays_eq; use crate::dtype::DType; @@ -98,10 +100,10 @@ fn test_selective_filter(array: &ArrayRef) { for (filtered_idx, i) in (0..len).step_by(2).enumerate() { assert_eq!( filtered - .scalar_at(filtered_idx) + .execute_scalar(filtered_idx, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), array - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); } @@ -118,18 +120,18 @@ fn test_selective_filter(array: &ArrayRef) { assert_eq!(filtered.len(), 2); assert_eq!( filtered - .scalar_at(0) + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), array - .scalar_at(0) + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); assert_eq!( filtered - .scalar_at(1) + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), array - .scalar_at(len - 1) + .execute_scalar(len - 1, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); } @@ -151,10 +153,10 @@ fn test_single_element_filter(array: &ArrayRef) { assert_eq!(filtered.len(), 1); assert_eq!( filtered - .scalar_at(0) + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), array - .scalar_at(0) + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); @@ -169,10 +171,10 @@ fn test_single_element_filter(array: &ArrayRef) { assert_eq!(filtered.len(), 1); assert_eq!( filtered - .scalar_at(0) + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), array - .scalar_at(len - 1) + .execute_scalar(len - 1, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); } @@ -235,10 +237,10 @@ fn test_alternating_pattern_filter(array: &ArrayRef) { if keep { assert_eq!( filtered - .scalar_at(filtered_idx) + .execute_scalar(filtered_idx, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), array - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); filtered_idx += 1; diff --git a/vortex-array/src/compute/conformance/mask.rs b/vortex-array/src/compute/conformance/mask.rs index debdff9a806..42d7bb9c494 100644 --- a/vortex-array/src/compute/conformance/mask.rs +++ b/vortex-array/src/compute/conformance/mask.rs @@ -6,6 +6,8 @@ use vortex_mask::Mask; use crate::ArrayRef; use crate::IntoArray; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::arrays::BoolArray; use crate::arrays::bool::BoolArrayExt; use crate::builtins::ArrayBuiltins; @@ -52,16 +54,16 @@ fn test_heterogenous_mask(array: &ArrayRef) { if masked_out { assert!( !masked - .is_valid(i) + .is_valid(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("is_valid should succeed in conformance test") ); } else { assert_eq!( masked - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), array - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") .into_nullable() ); @@ -85,10 +87,10 @@ fn test_empty_mask(array: &ArrayRef) { for i in 0..len { assert_eq!( masked - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), array - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") .into_nullable() ); @@ -111,7 +113,7 @@ fn test_full_mask(array: &ArrayRef) { for i in 0..len { assert!( !masked - .is_valid(i) + .is_valid(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("is_valid should succeed in conformance test") ); } @@ -133,16 +135,16 @@ fn test_alternating_mask(array: &ArrayRef) { if i % 2 == 0 { assert!( !masked - .is_valid(i) + .is_valid(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("is_valid should succeed in conformance test") ); } else { assert_eq!( masked - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), array - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") .into_nullable() ); @@ -171,7 +173,7 @@ fn test_sparse_mask(array: &ArrayRef) { let valid_count = (0..len) .filter(|&i| { masked - .is_valid(i) + .is_valid(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("is_valid should succeed in conformance test") }) .count(); @@ -183,7 +185,7 @@ fn test_sparse_mask(array: &ArrayRef) { .filter(|&i| { pattern[i] || !array - .is_valid(i) + .is_valid(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("is_valid should succeed in conformance test") }) .count(); @@ -206,17 +208,17 @@ fn test_single_element_mask(array: &ArrayRef) { .vortex_expect("mask should succeed in conformance test"); assert!( !masked - .is_valid(0) + .is_valid(0, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("is_valid should succeed in conformance test") ); for i in 1..len { assert_eq!( masked - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), array - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") .into_nullable() ); @@ -247,16 +249,16 @@ fn test_double_mask(array: &ArrayRef) { if mask1_pattern[i] || mask2_pattern[i] { assert!( !double_masked - .is_valid(i) + .is_valid(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("is_valid should succeed in conformance test") ); } else { assert_eq!( double_masked - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), array - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") .into_nullable() ); @@ -279,7 +281,8 @@ fn test_nullable_mask_input(array: &ArrayRef) { let validity = crate::validity::Validity::from_iter(validity_values.clone()); let nullable_mask = BoolArray::new(bool_array.to_bit_buffer(), validity); - let mask_array = nullable_mask.to_mask_fill_null_false(); + let mask_array = + nullable_mask.to_mask_fill_null_false(&mut LEGACY_SESSION.create_execution_ctx()); let masked = array .clone() .mask((!&mask_array).into_array()) @@ -290,16 +293,16 @@ fn test_nullable_mask_input(array: &ArrayRef) { if bool_values[i] && validity_values[i] { assert!( !masked - .is_valid(i) + .is_valid(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("is_valid should succeed in conformance test") ); } else { assert_eq!( masked - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), array - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") .into_nullable() ); diff --git a/vortex-array/src/compute/conformance/search_sorted.rs b/vortex-array/src/compute/conformance/search_sorted.rs index a112b7fe4d1..472dfec9dc1 100644 --- a/vortex-array/src/compute/conformance/search_sorted.rs +++ b/vortex-array/src/compute/conformance/search_sorted.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] +#![expect(clippy::unwrap_used)] pub use rstest::rstest; pub use rstest_reuse; diff --git a/vortex-array/src/compute/conformance/take.rs b/vortex-array/src/compute/conformance/take.rs index 214262e5572..911a2ae7295 100644 --- a/vortex-array/src/compute/conformance/take.rs +++ b/vortex-array/src/compute/conformance/take.rs @@ -7,6 +7,8 @@ use vortex_error::VortexExpect; use crate::ArrayRef; use crate::Canonical; use crate::IntoArray as _; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::arrays::PrimitiveArray; use crate::dtype::Nullability; @@ -62,14 +64,15 @@ fn test_take_all(array: &ArrayRef) { assert_eq!(result.dtype(), array.dtype()); // Verify elements match - match ( - array - .to_canonical() - .vortex_expect("to_canonical failed on array"), - result - .to_canonical() - .vortex_expect("to_canonical failed on result"), - ) { + #[expect(deprecated)] + let array_canonical = array + .to_canonical() + .vortex_expect("to_canonical failed on array"); + #[expect(deprecated)] + let result_canonical = result + .to_canonical() + .vortex_expect("to_canonical failed on result"); + match (array_canonical, result_canonical) { (Canonical::Primitive(orig_prim), Canonical::Primitive(result_prim)) => { assert_eq!( orig_prim.buffer_handle().to_host_sync(), @@ -81,10 +84,10 @@ fn test_take_all(array: &ArrayRef) { for i in 0..len { assert_eq!( array - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), result - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); } @@ -102,7 +105,7 @@ fn test_take_none(array: &ArrayRef) { assert_eq!(result.dtype(), array.dtype()); } -#[allow(clippy::cast_possible_truncation)] +#[expect(clippy::cast_possible_truncation)] fn test_take_selective(array: &ArrayRef) { let len = array.len(); @@ -120,10 +123,13 @@ fn test_take_selective(array: &ArrayRef) { for (result_idx, &original_idx) in indices.iter().enumerate() { assert_eq!( array - .scalar_at(original_idx as usize) + .execute_scalar( + original_idx as usize, + &mut LEGACY_SESSION.create_execution_ctx() + ) .vortex_expect("scalar_at should succeed in conformance test"), result - .scalar_at(result_idx) + .execute_scalar(result_idx, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); } @@ -139,23 +145,23 @@ fn test_take_first_and_last(array: &ArrayRef) { assert_eq!(result.len(), 2); assert_eq!( array - .scalar_at(0) + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), result - .scalar_at(0) + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); assert_eq!( array - .scalar_at(len - 1) + .execute_scalar(len - 1, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), result - .scalar_at(1) + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); } -#[allow(clippy::cast_possible_truncation)] +#[expect(clippy::cast_possible_truncation)] fn test_take_with_nullable_indices(array: &ArrayRef) { let len = array.len(); @@ -184,17 +190,17 @@ fn test_take_with_nullable_indices(array: &ArrayRef) { match idx_opt { Some(idx) => { let expected = array - .scalar_at(*idx as usize) + .execute_scalar(*idx as usize, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); let actual = result - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); assert_eq!(expected, actual); } None => { assert!( result - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") .is_null() ); @@ -216,12 +222,12 @@ fn test_take_repeated_indices(array: &ArrayRef) { assert_eq!(result.len(), 3); let first_elem = array - .scalar_at(0) + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"); for i in 0..3 { assert_eq!( result - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), first_elem ); @@ -252,10 +258,10 @@ fn test_take_reverse(array: &ArrayRef) { for i in 0..len { assert_eq!( array - .scalar_at(len - 1 - i) + .execute_scalar(len - 1 - i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), result - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); } @@ -273,15 +279,15 @@ fn test_take_single_middle(array: &ArrayRef) { assert_eq!(result.len(), 1); assert_eq!( array - .scalar_at(middle_idx) + .execute_scalar(middle_idx, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), result - .scalar_at(0) + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); } -#[allow(clippy::cast_possible_truncation)] +#[expect(clippy::cast_possible_truncation)] fn test_take_random_unsorted(array: &ArrayRef) { let len = array.len(); @@ -304,10 +310,10 @@ fn test_take_random_unsorted(array: &ArrayRef) { for (i, &idx) in indices.iter().enumerate() { assert_eq!( array - .scalar_at(idx as usize) + .execute_scalar(idx as usize, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), result - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); } @@ -330,16 +336,16 @@ fn test_take_contiguous_range(array: &ArrayRef) { for i in 0..(end - start) { assert_eq!( array - .scalar_at(start + i) + .execute_scalar(start + i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), result - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); } } -#[allow(clippy::cast_possible_truncation)] +#[expect(clippy::cast_possible_truncation)] fn test_take_mixed_repeated(array: &ArrayRef) { let len = array.len(); @@ -366,16 +372,16 @@ fn test_take_mixed_repeated(array: &ArrayRef) { for (i, &idx) in indices.iter().enumerate() { assert_eq!( array - .scalar_at(idx as usize) + .execute_scalar(idx as usize, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), result - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); } } -#[allow(clippy::cast_possible_truncation)] +#[expect(clippy::cast_possible_truncation)] fn test_take_large_indices(array: &ArrayRef) { // Test with a large number of indices to stress test performance let len = array.len(); @@ -398,10 +404,10 @@ fn test_take_large_indices(array: &ArrayRef) { let expected_idx = indices[i] as usize; assert_eq!( array - .scalar_at(expected_idx) + .execute_scalar(expected_idx, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test"), result - .scalar_at(i) + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at should succeed in conformance test") ); } diff --git a/vortex-array/src/display/mod.rs b/vortex-array/src/display/mod.rs index f299bb7aeeb..8907667bd6e 100644 --- a/vortex-array/src/display/mod.rs +++ b/vortex-array/src/display/mod.rs @@ -19,6 +19,8 @@ use itertools::Itertools as _; pub use tree_display::TreeDisplay; use crate::ArrayRef; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; /// Describe how to convert an array to a string. /// @@ -534,7 +536,7 @@ impl ArrayRef { let is_truncated = self.len() > limit; let fmt_scalar = |i| { - self.scalar_at(i) + self.execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) .map_or_else(|e| format!(""), |s| s.to_string()) }; write!( @@ -577,7 +579,8 @@ impl ArrayRef { } #[cfg(feature = "table-display")] DisplayOptions::TableDisplay => { - use crate::canonical::ToCanonical; + #[expect(deprecated)] + use crate::canonical::ToCanonical as _; use crate::dtype::DType; let mut builder = tabled::builder::Builder::default(); @@ -587,7 +590,7 @@ impl ArrayRef { // For non-struct arrays, simply display a single column table without header. for row_idx in 0..self.len() { let value = self - .scalar_at(row_idx) + .execute_scalar(row_idx, &mut LEGACY_SESSION.create_execution_ctx()) .map_or_else(|e| format!(""), |s| s.to_string()); builder.push_record([value]); } @@ -598,11 +601,15 @@ impl ArrayRef { return write!(f, "{table}"); }; + #[expect(deprecated)] let struct_ = self.to_struct(); builder.push_record(sf.names().iter().map(|name| name.to_string())); for row_idx in 0..self.len() { - if !self.is_valid(row_idx).unwrap_or(false) { + if !self + .is_valid(row_idx, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap_or(false) + { let null_row = vec!["null".to_string(); sf.names().len()]; builder.push_record(null_row); } else { @@ -611,7 +618,7 @@ impl ArrayRef { crate::arrays::struct_::StructArrayExt::iter_unmasked_fields(&struct_) { let value = field_array - .scalar_at(row_idx) + .execute_scalar(row_idx, &mut LEGACY_SESSION.create_execution_ctx()) .map_or_else(|e| format!(""), |s| s.to_string()); row.push(value); } @@ -628,7 +635,10 @@ impl ArrayRef { } for row_idx in 0..self.len() { - if !self.is_valid(row_idx).unwrap_or(false) { + if !self + .is_valid(row_idx, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap_or(false) + { table.modify( (1 + row_idx, 0), tabled::settings::Span::column(sf.names().len() as isize), diff --git a/vortex-array/src/dtype/bigint/mod.rs b/vortex-array/src/dtype/bigint/mod.rs index 6846d202529..3ebf01425d6 100644 --- a/vortex-array/src/dtype/bigint/mod.rs +++ b/vortex-array/src/dtype/bigint/mod.rs @@ -330,7 +330,7 @@ define_as_primitive!(i64); define_as_primitive!(i128); #[cfg(test)] -#[allow(clippy::many_single_char_names)] +#[expect(clippy::many_single_char_names)] mod tests { use num_traits::ToPrimitive; diff --git a/vortex-array/src/dtype/coercion.rs b/vortex-array/src/dtype/coercion.rs index 528d9b673b2..d57c0b9c1f4 100644 --- a/vortex-array/src/dtype/coercion.rs +++ b/vortex-array/src/dtype/coercion.rs @@ -3,6 +3,8 @@ //! Utilities for performing type coercion. +use std::sync::Arc; + use crate::dtype::DType; use crate::dtype::PType; use crate::dtype::decimal::DecimalDType; @@ -77,14 +79,29 @@ impl DType { /// The core primitive — what type can hold both `self` and `other`? /// Returns `None` if no common supertype exists. pub fn least_supertype(&self, other: &DType) -> Option { - // 1. Identity (ignoring nullability): return self with union nullability - if self.eq_ignore_nullability(other) { - return Some(self.with_nullability(self.nullability() | other.nullability())); + let union_null = self.nullability() | other.nullability(); + + if let ( + DType::FixedSizeList(lhs_elem, lhs_size, _), + DType::FixedSizeList(rhs_elem, rhs_size, _), + ) = (self, other) + && lhs_size == rhs_size + { + let elem = lhs_elem.least_supertype(rhs_elem)?; + return Some(DType::FixedSizeList(Arc::new(elem), *lhs_size, union_null)); } - let union_null = self.nullability() | other.nullability(); + if let (DType::List(lhs_elem, _), DType::List(rhs_elem, _)) = (self, other) { + let elem = lhs_elem.least_supertype(rhs_elem)?; + return Some(DType::List(Arc::new(elem), union_null)); + } - // 2. Null + X: return X as nullable + // Identity (ignoring nullability): return self with union nullability + if self.eq_ignore_nullability(other) { + return Some(self.with_nullability(union_null)); + } + + // Null + X: return X as nullable if matches!(self, DType::Null) { return Some(other.as_nullable()); } @@ -92,7 +109,7 @@ impl DType { return Some(self.as_nullable()); } - // 3. Bool + numeric: return the numeric type (with union nullability) + // Bool + numeric: return the numeric type (with union nullability) if self.is_boolean() && other.is_numeric() { return Some(other.with_nullability(union_null)); } @@ -100,19 +117,19 @@ impl DType { return Some(self.with_nullability(union_null)); } - // 4. Primitive + Primitive (different ptypes): delegate to PType::least_supertype + // Primitive + Primitive (different ptypes): delegate to PType::least_supertype if let (DType::Primitive(lhs_p, _), DType::Primitive(rhs_p, _)) = (self, other) { return lhs_p .least_supertype(*rhs_p) .map(|p| DType::Primitive(p, union_null)); } - // 5. Decimal + Decimal: compute wider decimal + // Decimal + Decimal: compute wider decimal if let (DType::Decimal(lhs_d, _), DType::Decimal(rhs_d, _)) = (self, other) { return decimal_least_supertype(*lhs_d, *rhs_d).map(|d| DType::Decimal(d, union_null)); } - // 6. Decimal + integer Primitive: convert integer to Decimal, then widen + // Decimal + integer Primitive: convert integer to Decimal, then widen if let (DType::Decimal(dec, _), DType::Primitive(p, _)) = (self, other) && p.is_int() { @@ -126,7 +143,7 @@ impl DType { return decimal_least_supertype(int_dec, *dec).map(|d| DType::Decimal(d, union_null)); } - // 7. Extension + anything: delegate to vtable + // Extension + anything: delegate to vtable if let DType::Extension(ext) = self { return ext .least_supertype(other) @@ -138,7 +155,6 @@ impl DType { .map(|dt| dt.with_nullability(union_null)); } - // 8. Everything else: no common supertype None } @@ -152,22 +168,37 @@ impl DType { /// Is there any implicit coercion path from `other` to `self`? pub fn can_coerce_from(&self, other: &DType) -> bool { - // 1. Same type (ignoring nullability): check nullability compatibility + if let ( + DType::FixedSizeList(target_elem, target_size, _), + DType::FixedSizeList(source_elem, source_size, _), + ) = (self, other) + { + return target_size == source_size + && (self.is_nullable() || !other.is_nullable()) + && target_elem.can_coerce_from(source_elem); + } + + if let (DType::List(target_elem, _), DType::List(source_elem, _)) = (self, other) { + return (self.is_nullable() || !other.is_nullable()) + && target_elem.can_coerce_from(source_elem); + } + + // Same type (ignoring nullability): check nullability compatibility if self.eq_ignore_nullability(other) { return self.is_nullable() || !other.is_nullable(); } - // 2. Null → nullable target + // Null → nullable target if matches!(other, DType::Null) { return self.is_nullable(); } - // 3. Bool → numeric + // Bool → numeric if other.is_boolean() && self.is_numeric() { return self.is_nullable() || !other.is_nullable(); } - // 4. Primitive widening: true if least_supertype(source, target) == target + // Primitive widening: true if least_supertype(source, target) == target if let (DType::Primitive(..), DType::Primitive(..)) = (self, other) { return other .least_supertype(self) @@ -175,7 +206,7 @@ impl DType { && (self.is_nullable() || !other.is_nullable()); } - // 5. Decimal widening + // Decimal widening if let (DType::Decimal(target, _), DType::Decimal(source, _)) = (self, other) { let target_integral = target.precision() as i16 - target.scale() as i16; let source_integral = source.precision() as i16 - source.scale() as i16; @@ -184,7 +215,7 @@ impl DType { && (self.is_nullable() || !other.is_nullable()); } - // 6. Integer → Decimal + // Integer → Decimal if let (DType::Decimal(dec, _), DType::Primitive(p, _)) = (self, other) && p.is_int() { @@ -194,12 +225,11 @@ impl DType { && (self.is_nullable() || !other.is_nullable()); } - // 7. Extension: delegate to vtable + // Extension: delegate to vtable if let DType::Extension(ext) = self { return ext.can_coerce_from(other); } - // 8. Everything else: false false } @@ -274,6 +304,8 @@ fn decimal_least_supertype(a: DecimalDType, b: DecimalDType) -> Option = &'a ScalarValue; fn id(&self) -> ExtId { - self.id.clone() + self.id } fn serialize_metadata(&self, metadata: &Self::Metadata) -> VortexResult> { diff --git a/vortex-array/src/dtype/extension/mod.rs b/vortex-array/src/dtype/extension/mod.rs index 0eae1587b40..96dfa7e22a3 100644 --- a/vortex-array/src/dtype/extension/mod.rs +++ b/vortex-array/src/dtype/extension/mod.rs @@ -30,9 +30,10 @@ pub use erased::*; mod matcher; pub use matcher::*; +use vortex_session::registry::Id; /// A unique identifier for an extension type -pub type ExtId = arcref::ArcRef; +pub type ExtId = Id; /// Private module to seal [`typed::DynExtDType`]. mod sealed { diff --git a/vortex-array/src/dtype/extension/typed.rs b/vortex-array/src/dtype/extension/typed.rs index 747b813afad..5b176bbb34d 100644 --- a/vortex-array/src/dtype/extension/typed.rs +++ b/vortex-array/src/dtype/extension/typed.rs @@ -48,7 +48,7 @@ impl ExtDType { } } -#[allow(clippy::same_name_method)] +#[expect(clippy::same_name_method)] impl ExtDType { /// Creates a new extension dtype with the given metadata and storage dtype. pub fn try_with_vtable( diff --git a/vortex-array/src/dtype/extension/vtable.rs b/vortex-array/src/dtype/extension/vtable.rs index 1b763f53635..d4a00fbdec4 100644 --- a/vortex-array/src/dtype/extension/vtable.rs +++ b/vortex-array/src/dtype/extension/vtable.rs @@ -14,8 +14,8 @@ use crate::scalar::ScalarValue; /// The public API for defining new extension types. /// -/// This is the non-object-safe trait that plugin authors implement to define a new extension -/// type. It specifies the type's identity, metadata, serialization, and validation. +/// This is the non-object-safe trait that plugin authors implement to define a new extension type. +/// It specifies the type's identity, metadata, serialization, and validation. pub trait ExtVTable: 'static + Sized + Send + Sync + Clone + Debug + Eq + Hash { /// Associated type containing the deserialized metadata for this extension type. type Metadata: 'static + Send + Sync + Clone + Debug + Display + Eq + Hash; @@ -39,11 +39,11 @@ pub trait ExtVTable: 'static + Sized + Send + Sync + Clone + Debug + Eq + Hash { /// Validate that the given storage type is compatible with this extension type. fn validate_dtype(ext_dtype: &ExtDType) -> VortexResult<()>; - /// Can a value of `other` be implicitly widened into this type? - /// e.g. GeographyType might accept Point, LineString, etc. + /// Can a value of `other` be implicitly widened into this type? (e.g. GeographyType might + /// accept Point, LineString, etc.) /// - /// Implementors only need to override one of `can_coerce_from` or `can_coerce_to` — both - /// exist so that either side of the coercion can provide the logic. + /// Implementors only need to override one of `can_coerce_from` or `can_coerce_to`. We have both + /// so that either side of the coercion can provide the logic. fn can_coerce_from(ext_dtype: &ExtDType, other: &DType) -> bool { let _ = (ext_dtype, other); false @@ -51,14 +51,15 @@ pub trait ExtVTable: 'static + Sized + Send + Sync + Clone + Debug + Eq + Hash { /// Can this type be implicitly widened into `other`? /// - /// Implementors only need to override one of `can_coerce_from` or `can_coerce_to` — both - /// exist so that either side of the coercion can provide the logic. + /// Implementors only need to override one of `can_coerce_from` or `can_coerce_to`. We have both + /// so that either side of the coercion can provide the logic. fn can_coerce_to(ext_dtype: &ExtDType, other: &DType) -> bool { let _ = (ext_dtype, other); false } /// Given two types in a Uniform context, what is their least supertype? + /// /// Return None if no supertype exists. fn least_supertype(ext_dtype: &ExtDType, other: &DType) -> Option { let _ = (ext_dtype, other); @@ -69,7 +70,8 @@ pub trait ExtVTable: 'static + Sized + Send + Sync + Clone + Debug + Eq + Hash { /// Validate the given storage value is compatible with the extension type. /// - /// By default, this calls [`unpack_native()`](ExtVTable::unpack_native) and discards the result. + /// By default, this calls [`unpack_native()`](ExtVTable::unpack_native) and discards the + /// result. /// /// # Errors /// diff --git a/vortex-array/src/dtype/field_names.rs b/vortex-array/src/dtype/field_names.rs index 737ad0cfd52..331d70aecd6 100644 --- a/vortex-array/src/dtype/field_names.rs +++ b/vortex-array/src/dtype/field_names.rs @@ -10,9 +10,16 @@ use itertools::Itertools; use vortex_utils::aliases::StringEscape; /// A name for a field in a struct. -#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] +#[derive(Clone, Debug, Eq, PartialOrd, Ord, Hash)] +#[allow(clippy::derived_hash_with_manual_eq)] // manual PartialEq adds Arc::ptr_eq fast path only pub struct FieldName(Arc); +impl PartialEq for FieldName { + fn eq(&self, other: &Self) -> bool { + Arc::ptr_eq(&self.0, &other.0) || *self.0 == *other.0 + } +} + impl FieldName { /// Returns a reference to the inner string pub fn inner(&self) -> &Arc { @@ -139,10 +146,17 @@ impl From for Arc { } /// An ordered list of field names in a struct. -#[derive(Clone, PartialEq, Eq, Debug, Default, Hash)] +#[derive(Clone, Eq, Debug, Default, Hash)] +#[allow(clippy::derived_hash_with_manual_eq)] // manual PartialEq adds Arc::ptr_eq fast path only #[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))] pub struct FieldNames(Arc<[FieldName]>); +impl PartialEq for FieldNames { + fn eq(&self, other: &Self) -> bool { + Arc::ptr_eq(&self.0, &other.0) || *self.0 == *other.0 + } +} + impl fmt::Display for FieldNames { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { write!( diff --git a/vortex-array/src/dtype/mod.rs b/vortex-array/src/dtype/mod.rs index 8041faae414..62f0f1f757c 100644 --- a/vortex-array/src/dtype/mod.rs +++ b/vortex-array/src/dtype/mod.rs @@ -50,7 +50,8 @@ use std::sync::Arc; /// /// [`I32`]: PType::I32 /// [`NonNullable`]: Nullability::NonNullable -#[derive(Debug, Clone, PartialEq, Eq, Hash)] +#[derive(Debug, Clone, Eq, Hash)] +#[allow(clippy::derived_hash_with_manual_eq)] // manual PartialEq adds Arc::ptr_eq fast path only pub enum DType { /// A logical null type. /// @@ -106,6 +107,42 @@ pub enum DType { Variant(Nullability), } +impl PartialEq for DType { + fn eq(&self, other: &Self) -> bool { + match (self, other) { + (Self::Null, Self::Null) => true, + (Self::Bool(a), Self::Bool(b)) => a == b, + (Self::Primitive(pa, na), Self::Primitive(pb, nb)) => pa == pb && na == nb, + (Self::Decimal(da, na), Self::Decimal(db, nb)) => da == db && na == nb, + (Self::Utf8(a), Self::Utf8(b)) => a == b, + (Self::Binary(a), Self::Binary(b)) => a == b, + (Self::List(da, na), Self::List(db, nb)) => { + na == nb && (Arc::ptr_eq(da, db) || da == db) + } + (Self::FixedSizeList(da, sa, na), Self::FixedSizeList(db, sb, nb)) => { + sa == sb && na == nb && (Arc::ptr_eq(da, db) || da == db) + } + // StructFields handles its own Arc::ptr_eq in its PartialEq impl. + (Self::Struct(a, na), Self::Struct(b, nb)) => na == nb && a == b, + (Self::Extension(a), Self::Extension(b)) => a == b, + (Self::Variant(a), Self::Variant(b)) => a == b, + // Every variant is listed in the first position so that adding a new + // variant produces a non-exhaustive match compile error. + (Self::Null, _) + | (Self::Bool(_), _) + | (Self::Primitive(..), _) + | (Self::Decimal(..), _) + | (Self::Utf8(_), _) + | (Self::Binary(_), _) + | (Self::List(..), _) + | (Self::FixedSizeList(..), _) + | (Self::Struct(..), _) + | (Self::Extension(_), _) + | (Self::Variant(_), _) => false, + } + } +} + pub use bigint::*; pub use decimal::*; pub use dtype_impl::NativeDType; diff --git a/vortex-array/src/dtype/ptype.rs b/vortex-array/src/dtype/ptype.rs index 842f2d585a1..ab443618d66 100644 --- a/vortex-array/src/dtype/ptype.rs +++ b/vortex-array/src/dtype/ptype.rs @@ -891,7 +891,6 @@ macro_rules! try_from_bytes { ($T:ty) => { impl ToBytes for $T { #[inline] - #[allow(clippy::size_of_in_element_count)] fn to_le_bytes(&self) -> &[u8] { // NOTE(ngates): this assumes the platform is little-endian. Currently enforced // with a flag cfg(target_endian = "little") diff --git a/vortex-array/src/dtype/serde/flatbuffers.rs b/vortex-array/src/dtype/serde/flatbuffers.rs index af7ed7b8af3..ea4470039f1 100644 --- a/vortex-array/src/dtype/serde/flatbuffers.rs +++ b/vortex-array/src/dtype/serde/flatbuffers.rs @@ -195,13 +195,10 @@ impl TryFrom for DType { let fb_ext = fb .type__as_extension() .ok_or_else(|| vortex_err!("failed to parse extension from flatbuffer"))?; - let id = ExtId::new_arc( - fb_ext - .id() - .ok_or_else(|| vortex_err!("failed to parse extension id from flatbuffer"))? - .to_string() - .into(), - ); + let id = + ExtId::new(fb_ext.id().ok_or_else(|| { + vortex_err!("failed to parse extension id from flatbuffer") + })?); let storage_dtype = fb_ext.storage_dtype().ok_or_else(|| { vortex_err!( Serde: "storage_dtype must be present on DType fbs message") diff --git a/vortex-array/src/dtype/serde/mod.rs b/vortex-array/src/dtype/serde/mod.rs index 1dbeeb6cf8b..d0f43f0c4ac 100644 --- a/vortex-array/src/dtype/serde/mod.rs +++ b/vortex-array/src/dtype/serde/mod.rs @@ -7,7 +7,7 @@ pub(crate) mod flatbuffers; mod proto; -#[allow(clippy::module_inception)] +#[expect(clippy::module_inception)] #[cfg(feature = "serde")] mod serde; diff --git a/vortex-array/src/dtype/serde/proto.rs b/vortex-array/src/dtype/serde/proto.rs index 4ca8642120b..1dfe60f522e 100644 --- a/vortex-array/src/dtype/serde/proto.rs +++ b/vortex-array/src/dtype/serde/proto.rs @@ -87,7 +87,7 @@ impl DType { )) } DtypeType::Extension(e) => { - let id = ExtId::new_arc(e.id.as_str().to_string().into()); + let id = ExtId::new(e.id.as_str()); let storage_dtype = DType::from_proto( e.storage_dtype .as_ref() diff --git a/vortex-array/src/dtype/serde/serde.rs b/vortex-array/src/dtype/serde/serde.rs index e89a1a98658..1f7ea6fc2ca 100644 --- a/vortex-array/src/dtype/serde/serde.rs +++ b/vortex-array/src/dtype/serde/serde.rs @@ -571,7 +571,7 @@ impl<'de> DeserializeSeed<'de> for DTypeSerde<'_, ExtDTypeRef> { } let id = id.ok_or_else(|| de::Error::missing_field("id"))?; - let id = ExtId::new_arc(id); + let id = ExtId::new(&id); let storage_dtype = storage_dtype.ok_or_else(|| de::Error::missing_field("storage_dtype"))?; let metadata = metadata.ok_or_else(|| de::Error::missing_field("metadata"))?; diff --git a/vortex-array/src/dtype/session.rs b/vortex-array/src/dtype/session.rs index 669aab68165..2314658869f 100644 --- a/vortex-array/src/dtype/session.rs +++ b/vortex-array/src/dtype/session.rs @@ -3,10 +3,12 @@ //! Module for managing extension dtypes in a Vortex session. +use std::any::Any; use std::sync::Arc; use vortex_session::Ref; use vortex_session::SessionExt; +use vortex_session::SessionVar; use vortex_session::registry::Registry; use crate::dtype::extension::ExtDTypePluginRef; @@ -39,6 +41,16 @@ impl Default for DTypeSession { } } +impl SessionVar for DTypeSession { + fn as_any(&self) -> &dyn Any { + self + } + + fn as_any_mut(&mut self) -> &mut dyn Any { + self + } +} + impl DTypeSession { /// Register an extension DType with the Vortex session. pub fn register(&self, vtable: V) { diff --git a/vortex-array/src/dtype/struct_.rs b/vortex-array/src/dtype/struct_.rs index b824caf22dd..637c261b479 100644 --- a/vortex-array/src/dtype/struct_.rs +++ b/vortex-array/src/dtype/struct_.rs @@ -138,9 +138,16 @@ impl FieldDTypeInner { /// // Accessing a field by name will yield the first /// assert_eq!(fields.field("int_col").unwrap(), DType::Primitive(PType::I32, Nullability::Nullable)); /// ``` -#[derive(Clone, PartialEq, Eq, Hash)] +#[derive(Clone, Eq, Hash)] +#[allow(clippy::derived_hash_with_manual_eq)] // manual PartialEq adds Arc::ptr_eq fast path only pub struct StructFields(Arc); +impl PartialEq for StructFields { + fn eq(&self, other: &Self) -> bool { + Arc::ptr_eq(&self.0, &other.0) || self.0 == other.0 + } +} + impl std::fmt::Debug for StructFields { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("StructFields") diff --git a/vortex-array/src/executor.rs b/vortex-array/src/executor.rs index be47f178467..694fd25c3c6 100644 --- a/vortex-array/src/executor.rs +++ b/vortex-array/src/executor.rs @@ -1,31 +1,26 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -//! The execution engine: iteratively transforms arrays toward canonical form. +//! Iterative array execution. //! -//! Execution proceeds through four layers tried in order on each iteration: +//! The single-step [`Executable`] implementation for [`ArrayRef`] tries `reduce`, +//! `reduce_parent`, `execute_parent`, then `execute` once. The matcher-driven +//! [`ArrayRef::execute_until`] loop interprets [`ExecutionStep::ExecuteSlot`], +//! [`ExecutionStep::AppendChild`], and [`ExecutionStep::Done`] using an explicit stack plus an +//! optional builder, so encodings can advance without recursive descent. //! -//! 1. **`reduce`** -- metadata-only self-rewrite (cheapest). -//! 2. **`reduce_parent`** -- metadata-only child-driven parent rewrite. -//! 3. **`execute_parent`** -- child-driven fused execution (may read buffers). -//! 4. **`execute`** -- the encoding's own decode step (most expensive). -//! -//! The main entry point is [`DynArray::execute_until`], which uses an explicit work stack -//! to drive execution iteratively without recursion. Between steps, the optimizer runs -//! reduce/reduce_parent rules to fixpoint. -//! -//! See for a full description -//! of the model. +//! See for the full execution +//! narrative, diagrams, and walkthroughs. use std::env::VarError; use std::fmt; use std::fmt::Display; use std::sync::LazyLock; -use std::sync::atomic::AtomicUsize; use vortex_error::VortexExpect; use vortex_error::VortexResult; use vortex_error::vortex_bail; +use vortex_error::vortex_ensure; use vortex_error::vortex_panic; use vortex_session::VortexSession; @@ -33,23 +28,32 @@ use crate::AnyCanonical; use crate::ArrayRef; use crate::Canonical; use crate::IntoArray; +use crate::array::ArrayId; +use crate::builders::ArrayBuilder; +use crate::builders::builder_with_capacity_in; +use crate::dtype::DType; use crate::matcher::Matcher; use crate::memory::HostAllocatorRef; use crate::memory::MemorySessionExt; use crate::optimizer::ArrayOptimizer; - -/// Maximum number of iterations to attempt when executing an array before giving up and returning -/// an error. -pub(crate) static MAX_ITERATIONS: LazyLock = - LazyLock::new(|| match std::env::var("VORTEX_MAX_ITERATIONS") { - Ok(val) => val - .parse::() - .unwrap_or_else(|e| vortex_panic!("VORTEX_MAX_ITERATIONS is not a valid usize: {e}")), - Err(VarError::NotPresent) => 128, - Err(VarError::NotUnicode(_)) => { - vortex_panic!("VORTEX_MAX_ITERATIONS is not a valid unicode string") - } - }); +use crate::stats::ArrayStats; +use crate::stats::StatsSet; + +/// Returns the maximum number of iterations to attempt when executing an array before giving up and returning +/// an error, can be by the `VORTEX_MAX_ITERATIONS` env variables, otherwise defaults to 2^22. +pub(crate) fn max_iterations() -> usize { + static MAX_ITERATIONS: LazyLock = + LazyLock::new(|| match std::env::var("VORTEX_MAX_ITERATIONS") { + Ok(val) => val.parse::().unwrap_or_else(|e| { + vortex_panic!("VORTEX_MAX_ITERATIONS is not a valid usize: {e}") + }), + Err(VarError::NotPresent) => 2 << 21, // 2 ^ 22 + Err(VarError::NotUnicode(_)) => { + vortex_panic!("VORTEX_MAX_ITERATIONS is not a valid unicode string") + } + }); + *MAX_ITERATIONS +} /// Marker trait for types that an [`ArrayRef`] can be executed into. /// @@ -62,7 +66,7 @@ pub trait Executable: Sized { fn execute(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult; } -#[allow(clippy::same_name_method)] +#[expect(clippy::same_name_method)] impl ArrayRef { /// Execute this array to produce an instance of `E`. /// @@ -81,128 +85,237 @@ impl ArrayRef { } /// Iteratively execute this array until the [`Matcher`] matches, using an explicit work - /// stack. - /// - /// The scheduler repeatedly: - /// 1. Checks if the current array matches `M` — if so, pops the stack or returns. - /// 2. Runs `execute_parent` on each child for child-driven optimizations. - /// 3. Calls `execute` which returns an [`ExecutionStep`]. + /// stack plus an optional builder for `AppendChild`. /// /// Note: the returned array may not match `M`. If execution converges to a canonical form /// that does not match `M`, the canonical array is returned since no further execution /// progress is possible. /// - /// For safety, we will error when the number of execution iterations reaches a configurable - /// maximum (default 128, override with `VORTEX_MAX_ITERATIONS`). + /// For safety, this errors once execution reaches a configurable maximum number of + /// iterations (default `2^22`, override with `VORTEX_MAX_ITERATIONS`). + /// + /// # Loop state + /// + /// - `current_array: ArrayRef` -- the array currently in focus. + /// - `current_builder: Option>` -- active only for builder-mode + /// execution. `AppendChild` appends detached children here. `Done` finishes the builder + /// and turns it back into the next `current_array`. + /// - `stack: Vec` -- suspended parents from `ExecuteSlot`, including the + /// detached slot index, its [`DonePredicate`], and the parent builder that was active + /// before focus moved into the child. + /// + /// Example after `ExecuteSlot(1, pred)` has focused slot 1 of a parent: + /// + /// ```text + /// stack[top].parent_array: + /// RunEnd <-- suspended parent + /// +-- slot 0: ends + /// +-- slot 1: _ (detached) + /// + /// current_array: + /// DictEncoding <-- focused child + /// +-- slot 0: codes + /// +-- slot 1: dictionary + /// + /// current_builder: + /// None + /// ``` + /// + /// Each loop iteration works like this: + /// + /// ```text + /// loop: + /// Step 1: done(current_array)? + /// - root activation -> return current_array + /// - ExecuteSlot frame -> pop, reattach child, resume parent + /// + /// Step 2: current_builder active? + /// - yes -> skip Step 2a / 2b + /// - no -> try parent kernels + /// + /// Step 2a: current_array.execute_parent(stack.top.parent_array) + /// child looks up at the suspended parent from ExecuteSlot + /// + /// Step 2b: for child in current_array.children(): + /// child.execute_parent(current_array) + /// each child looks up at current_array + /// + /// Step 3: match current_array.execute() + /// ExecuteSlot(i, pred) -> push parent on stack, focus child `i` + /// AppendChild(i) -> detach child `i`, append it into current_builder, + /// keep parent as current_array + /// Done -> finish current_builder if present, else use returned array + /// ``` + /// + /// Step 2a and Step 2b are skipped while `current_builder` is active. `AppendChild` + /// partially consumes `current_array`: some slots already live in the builder, so a + /// parent rewrite would observe inconsistent state and could discard accumulated builder + /// data. pub fn execute_until(self, ctx: &mut ExecutionCtx) -> VortexResult { - static MAX_ITERATIONS: LazyLock = - LazyLock::new(|| match std::env::var("VORTEX_MAX_ITERATIONS") { - Ok(val) => val.parse::().unwrap_or_else(|e| { - vortex_panic!("VORTEX_MAX_ITERATIONS is not a valid usize: {e}") - }), - Err(VarError::NotPresent) => 128, - Err(VarError::NotUnicode(_)) => { - vortex_panic!("VORTEX_MAX_ITERATIONS is not a valid unicode string") - } - }); + let mut current_array = self; + let mut current_builder: Option> = None; + let mut stack: Vec = Vec::new(); + let max_iterations = max_iterations(); - let mut current = self.optimize()?; - // Stack frames: (parent, slot_idx, done_predicate_for_slot) - let mut stack: Vec<(ArrayRef, usize, DonePredicate)> = Vec::new(); - - for _ in 0..*MAX_ITERATIONS { - // Check for termination: use the stack frame's done predicate, or the root matcher. + for _ in 0..max_iterations { let is_done = stack .last() - .map_or(M::matches as DonePredicate, |frame| frame.2); - if is_done(¤t) { + .map_or(M::matches as DonePredicate, |frame| frame.done); + + if is_done(¤t_array) || AnyCanonical::matches(¤t_array) { match stack.pop() { None => { - ctx.log(format_args!("-> {}", current)); - return Ok(current); + debug_assert!( + current_builder.is_none(), + "root activation should not retain a builder" + ); + ctx.log(format_args!("-> {}", current_array)); + return Ok(current_array); } - Some((parent, slot_idx, _)) => { - current = parent.with_slot(slot_idx, current)?; - current = current.optimize()?; + Some(frame) => { + (current_array, current_builder) = pop_frame(frame, current_array)?; continue; } } } - // If we've reached canonical form, we can't execute any further regardless - // of whether the matcher matched. - if AnyCanonical::matches(¤t) { - match stack.pop() { - None => { - ctx.log(format_args!("-> canonical (unmatched) {}", current)); - return Ok(current); - } - Some((parent, slot_idx, _)) => { - current = parent.with_slot(slot_idx, current)?; - current = current.optimize()?; - continue; - } - } + // Step 2a: execute_parent against the suspended parent from ExecuteSlot. + // + // When executing a child for ExecuteSlot, try execute_parent against + // the suspended parent on the stack. This lets kernels like RunEnd's + // FilterKernel fire before the child is forced to canonical. + // + // Skip when a builder is active: the current array has been partially + // consumed by AppendChild (some slots are already in the builder), so + // a parent rewrite would see inconsistent state and the builder data + // would be lost when we restore frame.parent_builder. + if current_builder.is_none() + && let Some(frame) = stack.last() + && let Some(result) = + current_array.execute_parent(&frame.parent_array, frame.slot_idx, ctx)? + { + ctx.log(format_args!( + "execute_parent (stack) rewrote {} -> {}", + current_array, result + )); + let frame = stack.pop().vortex_expect("just peeked"); + current_array = result.optimize_ctx(ctx.session())?; + current_builder = frame.parent_builder; + continue; } - // Try execute_parent (child-driven optimized execution) - if let Some(rewritten) = try_execute_parent(¤t, ctx)? { + // Step 2b: execute_parent against current_array's own children. + if current_builder.is_none() + && let Some(rewritten) = try_execute_parent(¤t_array, ctx)? + { ctx.log(format_args!( "execute_parent rewrote {} -> {}", - current, rewritten + current_array, rewritten )); - current = rewritten.optimize()?; + current_array = rewritten.optimize_ctx(ctx.session())?; continue; } - // Execute the array itself. - let result = execute_step(current, ctx)?; + // execute step + let expected_len = current_array.len(); + let expected_dtype = current_array.dtype().clone(); + let stats = current_array.statistics().to_array_stats(); + let encoding_id = current_array.encoding_id(); + let result = current_array.execute_encoding_unchecked(ctx)?; let (array, step) = result.into_parts(); match step { ExecutionStep::ExecuteSlot(i, done) => { - let child = array.slots()[i] - .clone() - .vortex_expect("ExecuteSlot index in bounds"); + let (parent, child) = unsafe { array.take_slot_unchecked(i) }?; ctx.log(format_args!( "ExecuteSlot({i}): pushing {}, focusing on {}", - array, child + parent, child + )); + stack.push(StackFrame { + parent_array: parent, + parent_builder: current_builder.take(), + slot_idx: i, + done, + original_dtype: child.dtype().clone(), + original_len: child.len(), + }); + current_array = child; + current_builder = None; + } + ExecutionStep::AppendChild(i) => { + if current_builder.is_none() { + current_builder = Some(builder_with_capacity_in( + ctx.allocator(), + array.dtype(), + array.len(), + )); + } + let (parent, child) = unsafe { array.take_slot_unchecked(i) }?; + ctx.log(format_args!( + "AppendChild({i}): appending {} into builder", + child )); - stack.push((array, i, done)); - current = child.optimize()?; + // TODO(joe)[7674]: replace with a builder kernel registry so we don't + // need to go through the VTable append_to_builder indirection. + child.append_to_builder( + current_builder + .as_deref_mut() + .vortex_expect("builder must exist"), + ctx, + )?; + current_array = parent; } ExecutionStep::Done => { ctx.log(format_args!("Done: {}", array)); - current = array; + (current_array, current_builder) = finalize_done( + array, + current_builder, + expected_len, + expected_dtype, + stats, + encoding_id, + )?; } } } vortex_bail!( "Exceeded maximum execution iterations ({}) while executing array", - *MAX_ITERATIONS, + max_iterations, ) } } +struct StackFrame { + parent_array: ArrayRef, + parent_builder: Option>, + slot_idx: usize, + done: DonePredicate, + original_dtype: DType, + original_len: usize, +} + /// Execution context for batch CPU compute. -/// -/// Accumulates a trace of execution steps. Individual steps are logged at TRACE level for -/// real-time following, and the full trace is dumped at DEBUG level when the context is dropped. #[derive(Debug, Clone)] pub struct ExecutionCtx { - id: usize, session: VortexSession, + #[cfg(debug_assertions)] + id: usize, + #[cfg(debug_assertions)] ops: Vec, } impl ExecutionCtx { /// Create a new execution context with the given session. pub fn new(session: VortexSession) -> Self { - static EXEC_CTX_ID: AtomicUsize = AtomicUsize::new(0); - let id = EXEC_CTX_ID.fetch_add(1, std::sync::atomic::Ordering::Relaxed); Self { - id, session, + #[cfg(debug_assertions)] + id: { + static EXEC_CTX_ID: std::sync::atomic::AtomicUsize = + std::sync::atomic::AtomicUsize::new(0); + EXEC_CTX_ID.fetch_add(1, std::sync::atomic::Ordering::Relaxed) + }, + #[cfg(debug_assertions)] ops: Vec::new(), } } @@ -224,20 +337,26 @@ impl ExecutionCtx { /// /// Use the [`format_args!`] macro to create the `msg` argument. pub fn log(&mut self, msg: fmt::Arguments<'_>) { + #[cfg(debug_assertions)] if tracing::enabled!(tracing::Level::DEBUG) { let formatted = format!(" - {msg}"); tracing::trace!("exec[{}]: {formatted}", self.id); self.ops.push(formatted); } + let _ = msg; } } impl Display for ExecutionCtx { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - write!(f, "exec[{}]", self.id) + #[cfg(debug_assertions)] + return write!(f, "exec[{}]", self.id); + #[cfg(not(debug_assertions))] + write!(f, "exec") } } +#[cfg(debug_assertions)] impl Drop for ExecutionCtx { fn drop(&mut self) { if !self.ops.is_empty() && tracing::enabled!(tracing::Level::DEBUG) { @@ -259,42 +378,33 @@ impl Drop for ExecutionCtx { } } -/// Executing an [`ArrayRef`] into an [`ArrayRef`] is the atomic execution loop within Vortex. +/// Single-step execution: takes one step toward canonical form. /// -/// It attempts to take the smallest possible step of execution such that the returned array -/// is incrementally more "executed" than the input array. In other words, it is closer to becoming -/// a canonical array. +/// Steps through reduce, reduce_parent, execute_parent, then execute. For `ExecuteSlot`, +/// only a single child execution step is performed — the child is executed once and put back, +/// making this a lightweight, bounded operation. /// -/// The execution steps are as follows: -/// 0. Check for canonical. -/// 1. Attempt to `reduce` the array with metadata-only optimizations. -/// 2. Attempt to call `reduce_parent` on each child. -/// 3. Attempt to call `execute_parent` on each child. -/// 4. Call `execute` on the array itself (which returns an [`ExecutionStep`]). -/// -/// Most users will not call this method directly, instead preferring to specify an executable -/// target such as [`crate::Columnar`], [`Canonical`], or any of the canonical array types (such as -/// [`crate::arrays::PrimitiveArray`]). +/// **However**, if `execute_step` returns [`ExecutionStep::AppendChild`], this implementation +/// drives the *entire* array to completion via [`execute_into_builder`] in a single call. +/// This can do substantially more work than a normal step because it creates a builder and +/// fully decodes the array into that builder before returning. Callers should be aware that a +/// single `.execute::(ctx)` call may perform O(n_children * decode_cost) work when +/// `AppendChild` is returned. impl Executable for ArrayRef { fn execute(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { - // 0. Check for canonical if let Some(canonical) = array.as_opt::() { ctx.log(format_args!("-> canonical {}", array)); return Ok(Canonical::from(canonical).into_array()); } - // 1. reduce (metadata-only rewrites) if let Some(reduced) = array.reduce()? { ctx.log(format_args!("reduce: rewrote {} -> {}", array, reduced)); reduced.statistics().inherit_from(array.statistics()); return Ok(reduced); } - // 2. reduce_parent (child-driven metadata-only rewrites) for (slot_idx, slot) in array.slots().iter().enumerate() { - let Some(child) = slot else { - continue; - }; + let Some(child) = slot else { continue }; if let Some(reduced_parent) = child.reduce_parent(&array, slot_idx)? { ctx.log(format_args!( "reduce_parent: slot[{}]({}) rewrote {} -> {}", @@ -308,11 +418,8 @@ impl Executable for ArrayRef { } } - // 3. execute_parent (child-driven optimized execution) for (slot_idx, slot) in array.slots().iter().enumerate() { - let Some(child) = slot else { - continue; - }; + let Some(child) = slot else { continue }; if let Some(executed_parent) = child.execute_parent(&array, slot_idx, ctx)? { ctx.log(format_args!( "execute_parent: slot[{}]({}) rewrote {} -> {}", @@ -328,9 +435,8 @@ impl Executable for ArrayRef { } } - // 4. execute (returns an ExecutionResult) ctx.log(format_args!("executing {}", array)); - let result = execute_step(array, ctx)?; + let result = array.execute_encoding(ctx)?; let (array, step) = result.into_parts(); match step { ExecutionStep::Done => { @@ -338,21 +444,87 @@ impl Executable for ArrayRef { Ok(array) } ExecutionStep::ExecuteSlot(i, _) => { - // For single-step execution, handle ExecuteSlot by executing the slot, - // replacing it, and returning the updated array. let child = array.slots()[i].clone().vortex_expect("valid slot index"); let executed_child = child.execute::(ctx)?; array.with_slot(i, executed_child) } + ExecutionStep::AppendChild(_) => { + // Single-step: build the entire parent via the builder path. + let builder = builder_with_capacity_in(ctx.allocator(), array.dtype(), array.len()); + let mut builder = execute_into_builder(array, builder, ctx)?; + Ok(builder.finish()) + } } } } -/// Execute a single step on an array, consuming it. +/// Execute `array` into the given `builder`. /// -/// Extracts the vtable before consuming the array to avoid borrow conflicts. -fn execute_step(array: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { - array.execute_encoding(ctx) +/// This uses the encoding's [`crate::array::VTable::append_to_builder`] implementation. Most +/// encodings use the default path of `execute::` followed by `builder.extend_from_array`, +/// while encodings like `Chunked` can override that to append child-by-child without materializing +/// the entire parent. +/// +/// The builder must have a [`DType`] that is a nullability-superset of `array.dtype()`. +pub fn execute_into_builder( + array: ArrayRef, + mut builder: Box, + ctx: &mut ExecutionCtx, +) -> VortexResult> { + array.append_to_builder(builder.as_mut(), ctx)?; + Ok(builder) +} + +/// Pop a stack frame, restoring the parent with the finished child in its slot. +fn pop_frame( + frame: StackFrame, + child: ArrayRef, +) -> VortexResult<(ArrayRef, Option>)> { + debug_assert_eq!( + child.dtype(), + &frame.original_dtype, + "child dtype changed during execution" + ); + debug_assert_eq!( + child.len(), + frame.original_len, + "child len changed during execution" + ); + let parent_array = unsafe { frame.parent_array.put_slot_unchecked(frame.slot_idx, child) }?; + Ok((parent_array, frame.parent_builder)) +} + +fn finalize_done( + result: ArrayRef, + mut builder: Option>, + expected_len: usize, + expected_dtype: DType, + stats: ArrayStats, + encoding_id: ArrayId, +) -> VortexResult<(ArrayRef, Option>)> { + let output = if let Some(mut builder) = builder.take() { + builder.finish() + } else { + result + }; + + if cfg!(debug_assertions) { + vortex_ensure!( + output.len() == expected_len, + "Result length mismatch for {:?}", + encoding_id + ); + vortex_ensure!( + output.dtype() == &expected_dtype, + "Executed canonical dtype mismatch for {:?}", + encoding_id + ); + } + + output + .statistics() + .set_iter(StatsSet::from(stats).into_iter()); + Ok((output, None)) } /// Try execute_parent on each occupied slot of the array. @@ -372,23 +544,59 @@ fn try_execute_parent(array: &ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult< /// A predicate that determines when an array has reached a desired form during execution. pub type DonePredicate = fn(&ArrayRef) -> bool; -/// Metadata-only step indicator returned alongside an array in [`ExecutionResult`]. +/// Scheduler step indicator returned alongside an array in [`ExecutionResult`]. /// /// Instead of recursively executing children, encodings return an `ExecutionStep` that tells the /// scheduler what to do next. This enables the scheduler to manage execution iteratively using -/// an explicit work stack, run cross-step optimizations, and cache shared sub-expressions. +/// an explicit work stack plus an optional builder. +/// +/// # Semantics +/// +/// Each variant describes a different execution strategy with distinct cost profiles: +/// +/// - [`Done`](ExecutionStep::Done): The current activation has finished its work. If no builder +/// is active, the returned array is the result. If a builder is active, the scheduler ignores +/// the placeholder array and finishes the builder instead. The scheduler may continue +/// executing if the target form (e.g. canonical) has not yet been reached. +/// +/// - [`ExecuteSlot`](ExecutionStep::ExecuteSlot): The encoding needs one of its children +/// decoded before it can make further progress. The scheduler detaches that child, pushes +/// the parent onto the explicit stack, executes the child until the [`DonePredicate`] +/// matches, puts it back, and re-enters the parent. This is a cooperative yield: the +/// encoding does a bounded amount of work per step while the loop tracks the parent-child +/// relationship explicitly. +/// +/// - [`AppendChild`](ExecutionStep::AppendChild): The encoding needs one child executed to +/// canonical form and then appended into a builder owned by the current activation. The +/// scheduler detaches that child, lazily creates `current_builder` if needed, appends the +/// child into it, and keeps the parent as `current_array` for the next iteration. While the +/// builder is active, parent-kernel rewrites are skipped because the parent is partially +/// consumed. **Important:** in the single-step executor ([`Executable`] for [`ArrayRef`]), +/// returning `AppendChild` still causes the executor to drive the *entire* array to +/// completion via [`execute_into_builder`] in one call — this can do significantly more +/// work than a single `ExecuteSlot` step. pub enum ExecutionStep { /// Request that the scheduler execute the slot at the given index, using the provided /// [`DonePredicate`] to determine when the slot is "done", then replace the slot in this /// array and re-enter execution. /// - /// Between steps, the scheduler runs reduce/reduce_parent rules to fixpoint, enabling - /// cross-step optimization (e.g., pushing scalar functions through newly-decoded children). - /// /// Use [`ExecutionResult::execute_slot`] instead of constructing this variant directly. ExecuteSlot(usize, DonePredicate), - /// Execution is complete. The array in the accompanying [`ExecutionResult`] is the result. + /// Detach the slot at the given index, append that child into the current activation's + /// canonical builder, and keep the returned parent as `current_array`. + /// + /// `Done` finalizes that builder and turns it into the result of the activation. + /// + /// **Note:** In the single-step executor ([`Executable`] for [`ArrayRef`]), this variant + /// drives the entire parent to completion in one call via [`execute_into_builder`], which + /// may perform substantially more work than a single `ExecuteSlot` step. + AppendChild(usize), + + /// Execution is complete. If no builder is active, the array in the accompanying + /// [`ExecutionResult`] is the result. Otherwise, the scheduler finalizes the active + /// builder and uses that finished array instead. + /// /// The scheduler will continue executing if it has not yet reached the target form. Done, } @@ -397,6 +605,7 @@ impl fmt::Debug for ExecutionStep { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { match self { ExecutionStep::ExecuteSlot(idx, _) => f.debug_tuple("ExecuteSlot").field(idx).finish(), + ExecutionStep::AppendChild(idx) => f.debug_tuple("AppendChild").field(idx).finish(), ExecutionStep::Done => write!(f, "Done"), } } @@ -430,6 +639,16 @@ impl ExecutionResult { } } + /// Request that the child slot at `slot_idx` be detached, appended into the current + /// activation's canonical builder, and leave the returned parent as the next + /// `current_array`. + pub fn append_child(array: impl IntoArray, slot_idx: usize) -> Self { + Self { + array: array.into_array(), + step: ExecutionStep::AppendChild(slot_idx), + } + } + /// Returns a reference to the array. pub fn array(&self) -> &ArrayRef { &self.array @@ -481,7 +700,7 @@ macro_rules! require_child { /// execution of child `$idx`. /// /// Unlike `require_child!`, this is a statement macro (no value produced) and does not clone -/// `$parent` — it is moved into the early-return path. +/// `$parent` - it is moved into the early-return path. /// /// ```ignore /// require_opt_child!(array, array.patches().map(|p| p.indices()), 1 => Primitive); diff --git a/vortex-array/src/expr/exprs.rs b/vortex-array/src/expr/exprs.rs index cbf34cdb3c2..13821669034 100644 --- a/vortex-array/src/expr/exprs.rs +++ b/vortex-array/src/expr/exprs.rs @@ -29,6 +29,7 @@ use crate::scalar_fn::fns::dynamic::DynamicComparisonExpr; use crate::scalar_fn::fns::dynamic::Rhs; use crate::scalar_fn::fns::fill_null::FillNull; use crate::scalar_fn::fns::get_item::GetItem; +use crate::scalar_fn::fns::is_not_null::IsNotNull; use crate::scalar_fn::fns::is_null::IsNull; use crate::scalar_fn::fns::like::Like; use crate::scalar_fn::fns::like::LikeOptions; @@ -396,17 +397,19 @@ where /// /// ``` /// # use vortex_array::IntoArray; -/// # use vortex_array::arrow::IntoArrowArray as _; +/// # use vortex_array::arrow::ArrowArrayExecutor; +/// # use vortex_array::{LEGACY_SESSION, VortexSessionExecute}; /// # use vortex_buffer::buffer; /// # use vortex_array::expr::{checked_add, lit, root}; /// let xs = buffer![1, 2, 3].into_array(); /// let result = xs.apply(&checked_add(root(), lit(5))).unwrap(); /// +/// let mut ctx = LEGACY_SESSION.create_execution_ctx(); /// assert_eq!( -/// &result.into_arrow_preferred().unwrap(), +/// &result.execute_arrow(None, &mut ctx).unwrap(), /// &buffer![6, 7, 8] /// .into_array() -/// .into_arrow_preferred() +/// .execute_arrow(None, &mut ctx) /// .unwrap() /// ); /// ``` @@ -555,6 +558,20 @@ pub fn is_null(child: Expression) -> Expression { IsNull.new_expr(EmptyOptions, vec![child]) } +// ---- IsNotNull ---- + +/// Creates an expression that checks for non-null values. +/// +/// Returns a boolean array indicating which positions contain non-null values. +/// +/// ```rust +/// # use vortex_array::expr::{is_not_null, root}; +/// let expr = is_not_null(root()); +/// ``` +pub fn is_not_null(child: Expression) -> Expression { + IsNotNull.new_expr(EmptyOptions, vec![child]) +} + // ---- Like ---- /// Creates a SQL LIKE expression. diff --git a/vortex-array/src/expr/proto.rs b/vortex-array/src/expr/proto.rs index 22c5aa8b467..27859235e88 100644 --- a/vortex-array/src/expr/proto.rs +++ b/vortex-array/src/expr/proto.rs @@ -1,8 +1,6 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -use std::sync::Arc; - use itertools::Itertools; use vortex_error::VortexResult; use vortex_error::vortex_err; @@ -41,7 +39,7 @@ impl ExprSerializeProtoExt for Expression { impl Expression { pub fn from_proto(expr: &pb::Expr, session: &VortexSession) -> VortexResult { - let expr_id = ScalarFnId::new_arc(Arc::from(expr.id.to_string())); + let expr_id = ScalarFnId::new(expr.id.as_str()); let children = expr .children .iter() diff --git a/vortex-array/src/expr/pruning/pruning_expr.rs b/vortex-array/src/expr/pruning/pruning_expr.rs index bf985cad861..00d29fbcf99 100644 --- a/vortex-array/src/expr/pruning/pruning_expr.rs +++ b/vortex-array/src/expr/pruning/pruning_expr.rs @@ -86,6 +86,11 @@ pub fn field_path_stat_field_name(field_path: &FieldPath, stat: Stat) -> FieldNa /// cannot hold, and false if it cannot be determined from stats alone whether the positions can /// be pruned. /// +/// Some rewrites, such as `is_not_null(...)`, emit +/// [`row_count`][crate::scalar_fn::internal::row_count] placeholders. The evaluation layer must +/// replace those placeholders with the row count for its current scope before +/// executing the returned expression. +/// /// If the falsification logic attempts to access an unknown stat, /// this function will return `None`. pub fn checked_pruning_expr( diff --git a/vortex-array/src/expr/stats/mod.rs b/vortex-array/src/expr/stats/mod.rs index 5f998041dea..53f50bbb9e0 100644 --- a/vortex-array/src/expr/stats/mod.rs +++ b/vortex-array/src/expr/stats/mod.rs @@ -213,6 +213,8 @@ impl Display for Stat { mod test { use enum_iterator::all; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::arrays::PrimitiveArray; use crate::expr::stats::Stat; @@ -220,7 +222,7 @@ mod test { fn min_of_nulls_is_not_panic() { let min = PrimitiveArray::from_option_iter::([None, None, None, None]) .statistics() - .compute_as::(Stat::Min); + .compute_as::(Stat::Min, &mut LEGACY_SESSION.create_execution_ctx()); assert_eq!(min, None); } diff --git a/vortex-array/src/extension/datetime/date.rs b/vortex-array/src/extension/datetime/date.rs index a0434b3ebc5..8f3ae954535 100644 --- a/vortex-array/src/extension/datetime/date.rs +++ b/vortex-array/src/extension/datetime/date.rs @@ -79,7 +79,7 @@ impl ExtVTable for Date { type NativeValue<'a> = DateValue; fn id(&self) -> ExtId { - ExtId::new_ref("vortex.date") + ExtId::new("vortex.date") } fn serialize_metadata(&self, metadata: &Self::Metadata) -> VortexResult> { diff --git a/vortex-array/src/extension/datetime/time.rs b/vortex-array/src/extension/datetime/time.rs index 08c43e9e20c..d3a62b5ef65 100644 --- a/vortex-array/src/extension/datetime/time.rs +++ b/vortex-array/src/extension/datetime/time.rs @@ -80,7 +80,7 @@ impl ExtVTable for Time { type NativeValue<'a> = TimeValue; fn id(&self) -> ExtId { - ExtId::new_ref("vortex.time") + ExtId::new("vortex.time") } fn serialize_metadata(&self, metadata: &Self::Metadata) -> VortexResult> { diff --git a/vortex-array/src/extension/datetime/timestamp.rs b/vortex-array/src/extension/datetime/timestamp.rs index b33571ed40e..ab65ce8b84c 100644 --- a/vortex-array/src/extension/datetime/timestamp.rs +++ b/vortex-array/src/extension/datetime/timestamp.rs @@ -114,7 +114,7 @@ impl ExtVTable for Timestamp { type NativeValue<'a> = TimestampValue<'a>; fn id(&self) -> ExtId { - ExtId::new_ref("vortex.timestamp") + ExtId::new("vortex.timestamp") } // NOTE(ngates): unfortunately we're stuck with this hand-rolled serialization format for diff --git a/vortex-array/src/extension/tests/divisible_int.rs b/vortex-array/src/extension/tests/divisible_int.rs index 8b9612e9438..4ab08f7b9f5 100644 --- a/vortex-array/src/extension/tests/divisible_int.rs +++ b/vortex-array/src/extension/tests/divisible_int.rs @@ -35,7 +35,7 @@ impl ExtVTable for DivisibleInt { type NativeValue<'a> = u64; fn id(&self) -> ExtId { - ExtId::new_ref("test.divisible_int") + ExtId::new("test.divisible_int") } fn serialize_metadata(&self, metadata: &Self::Metadata) -> VortexResult> { diff --git a/vortex-array/src/extension/uuid/metadata.rs b/vortex-array/src/extension/uuid/metadata.rs index 7e7dd8b16d8..a9f595bac70 100644 --- a/vortex-array/src/extension/uuid/metadata.rs +++ b/vortex-array/src/extension/uuid/metadata.rs @@ -21,6 +21,8 @@ pub(crate) fn u8_to_version(b: u8) -> VortexResult { 6 => Ok(Version::SortMac), 7 => Ok(Version::SortRand), 8 => Ok(Version::Custom), + // UUID crate changed from 0xff to 0x0f for maximum uuid version in 1.23.0 + 0x0f => Ok(Version::Max), 0xff => Ok(Version::Max), _ => vortex_bail!("unknown UUID version discriminant: {b}"), } diff --git a/vortex-array/src/extension/uuid/vtable.rs b/vortex-array/src/extension/uuid/vtable.rs index c76c7352942..cd4d4d971ae 100644 --- a/vortex-array/src/extension/uuid/vtable.rs +++ b/vortex-array/src/extension/uuid/vtable.rs @@ -27,7 +27,7 @@ impl ExtVTable for Uuid { type NativeValue<'a> = uuid::Uuid; fn id(&self) -> ExtId { - ExtId::new_ref("vortex.uuid") + ExtId::new("vortex.uuid") } fn serialize_metadata(&self, metadata: &Self::Metadata) -> VortexResult> { @@ -312,8 +312,7 @@ mod tests { version: Some(Version::Random), }, uuid_storage_dtype(Nullability::NonNullable), - ) - .unwrap(); + )?; let storage_value = uuid_storage_scalar(&v4_uuid); let result = Uuid::unpack_native(&ext_dtype, &storage_value)?; @@ -330,8 +329,7 @@ mod tests { let ext_dtype = ExtDType::try_new( UuidMetadata::default(), uuid_storage_dtype(Nullability::NonNullable), - ) - .unwrap(); + )?; let storage_value = uuid_storage_scalar(&v4_uuid); let result = Uuid::unpack_native(&ext_dtype, &storage_value)?; diff --git a/vortex-array/src/lib.rs b/vortex-array/src/lib.rs index 463cb75a162..0869eac3ff4 100644 --- a/vortex-array/src/lib.rs +++ b/vortex-array/src/lib.rs @@ -1,7 +1,5 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors - -#![cfg_attr(vortex_nightly, feature(portable_simd))] //! Vortex crate containing core logic for encoding and memory representation of [arrays](ArrayRef). //! //! At the heart of Vortex are [arrays](ArrayRef). diff --git a/vortex-array/src/mask.rs b/vortex-array/src/mask.rs index c23a6f79652..ab035670e24 100644 --- a/vortex-array/src/mask.rs +++ b/vortex-array/src/mask.rs @@ -34,7 +34,10 @@ impl Executable for Mask { } Columnar::Canonical(a) => { let bool = a.into_array().execute::(ctx)?; - let mask = bool.validity_mask()?; + let mask = bool + .as_ref() + .validity()? + .execute_mask(bool.as_ref().len(), ctx)?; let bits = bool.into_bit_buffer(); // To handle nullable boolean arrays, we treat nulls as false in the mask. // TODO(ngates): is this correct? Feels like we should just force the caller to diff --git a/vortex-array/src/memory.rs b/vortex-array/src/memory.rs index 47030b18131..d3669e49f93 100644 --- a/vortex-array/src/memory.rs +++ b/vortex-array/src/memory.rs @@ -3,6 +3,7 @@ //! Session-scoped memory allocation for host-side buffers. +use std::any::Any; use std::fmt::Debug; use std::mem::size_of; use std::sync::Arc; @@ -18,6 +19,7 @@ use vortex_error::vortex_err; use vortex_session::Ref; use vortex_session::RefMut; use vortex_session::SessionExt; +use vortex_session::SessionVar; /// Mutable host buffer contract used by [`WritableHostBuffer`]. pub trait HostBufferMut: Send + 'static { @@ -199,6 +201,16 @@ impl Default for MemorySession { } } +impl SessionVar for MemorySession { + fn as_any(&self) -> &dyn Any { + self + } + + fn as_any_mut(&mut self) -> &mut dyn Any { + self + } +} + /// Extension trait for accessing session-scoped memory configuration. pub trait MemorySessionExt: SessionExt { /// Returns the memory session for this execution/session context. diff --git a/vortex-array/src/normalize.rs b/vortex-array/src/normalize.rs index 3235a480809..2833bf467d6 100644 --- a/vortex-array/src/normalize.rs +++ b/vortex-array/src/normalize.rs @@ -105,6 +105,7 @@ mod tests { use crate::ArrayRef; use crate::ExecutionCtx; use crate::IntoArray; + use crate::array::VTable; use crate::arrays::Dict; use crate::arrays::DictArray; use crate::arrays::Primitive; @@ -203,27 +204,25 @@ mod tests { // Slice the dict array to get a SliceArray wrapping a DictArray. let sliced = SliceArray::new(dict, 1..4).into_array(); - assert_eq!(sliced.encoding_id(), Slice::ID); + assert_eq!(sliced.encoding_id(), Slice.id()); - let allowed = HashSet::from_iter([Dict::ID, Primitive::ID]); + let allowed = HashSet::from_iter([Dict.id(), Primitive.id()]); let mut ctx = ExecutionCtx::new(VortexSession::empty()); - println!("sliced {}", sliced.display_tree()); - let normalized = sliced.normalize(&mut NormalizeOptions { allowed: &allowed, operation: Operation::Execute(&mut ctx), })?; - println!("after {}", normalized.display_tree()); - // The normalized result should be a DictArray, not a SliceArray. - assert_eq!(normalized.encoding_id(), Dict::ID); + assert_eq!(normalized.encoding_id(), Dict.id()); assert_eq!(normalized.len(), 3); // Verify the data: codes [1,0,1] -> values [20, 10, 20] + #[expect(deprecated)] + let normalized_canonical = normalized.to_canonical()?; assert_arrays_eq!( - normalized.to_canonical()?, + normalized_canonical, PrimitiveArray::from_iter(vec![20i32, 10, 20]) ); diff --git a/vortex-array/src/optimizer/kernels.rs b/vortex-array/src/optimizer/kernels.rs new file mode 100644 index 00000000000..a31192f8016 --- /dev/null +++ b/vortex-array/src/optimizer/kernels.rs @@ -0,0 +1,125 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Session-scoped registry for optimizer kernels. +//! +//! [`ArrayKernels`] stores function pointers that participate in array optimization without +//! adding rules to an encoding vtable. The optimizer currently consults it for parent-reduce +//! rewrites before the child encoding's static `PARENT_RULES`. A registered function can +//! therefore add a rule for an extension encoding or take precedence over a built-in rule. +//! +//! Kernel entries are addressed by `(outer_id, child_id, kind)`. For parent-reduce kernels, +//! `outer_id` is the id returned by the parent array's `encoding_id()` and `child_id` is the +//! child array's `encoding_id()`. For [`ScalarFn`](crate::arrays::ScalarFn) parents, the parent +//! id is the scalar function id. +//! +//! Sessions created by the top-level `vortex` crate install an empty registry by default. Other +//! sessions can add it with [`VortexSession::with`](vortex_session::VortexSession::with) or rely +//! on [`ArrayKernelsExt::kernels`] to insert the default value. + +use std::any::Any; +use std::hash::BuildHasher; +use std::sync::Arc; +use std::sync::LazyLock; + +use arc_swap::ArcSwap; +use vortex_error::VortexResult; +use vortex_session::Ref; +use vortex_session::SessionExt; +use vortex_session::SessionVar; +use vortex_session::registry::Id; +use vortex_utils::aliases::DefaultHashBuilder; +use vortex_utils::aliases::hash_map::HashMap; + +use crate::ArrayRef; + +/// Shared hasher used to combine `(outer, child, FnKind)` tuples into [`FnRegistry`] keys. +static FN_HASHER: LazyLock = LazyLock::new(DefaultHashBuilder::default); + +/// Function pointer for a plugin-provided parent-reduce rewrite. +/// +/// The optimizer calls this with the matched `child`, its `parent`, and the slot index where the +/// child appears. Return `Ok(Some(new_parent))` to replace the parent, or `Ok(None)` when the +/// rewrite does not apply. +/// +/// Implementations must preserve the parent's logical length and dtype, matching the invariant +/// required of static parent-reduce rules. +pub type ReduceParentFn = + fn(child: &ArrayRef, parent: &ArrayRef, child_idx: usize) -> VortexResult>; + +/// Session-scoped registry of optimizer kernel functions. +#[derive(Debug, Default)] +pub struct ArrayKernels { + reduce_parent: ArcSwap>>, +} + +impl ArrayKernels { + /// Create an empty [`ArrayKernels`] with no kernels registered. + pub fn empty() -> Self { + Self::default() + } + + /// Register a [`ReduceParentFn`] for `(outer, child)`. + /// + /// The optimizer will invoke `f` when it sees a parent with encoding id `outer` holding a + /// child with encoding id `child` during a `reduce_parent` step, before trying the child + /// encoding's static `PARENT_RULES`. `outer` is usually the parent array's encoding id. For + /// `ScalarFnArray`, it is the scalar function id, for example `Cast.id()`. + /// + /// Replaces any function already registered for the same pair. + pub fn register_reduce_parent>( + &self, + parent: Id, + child: Id, + fns: I, + ) { + let registry = self.reduce_parent.load(); + let id = self.hash_fn_ids(parent, child); + let mut owned_registry = registry.as_ref().clone(); + if let Some(existing) = owned_registry.remove(&id) { + owned_registry.insert(id, existing.as_ref().iter().cloned().chain(fns).collect()); + } else { + owned_registry.insert(id, fns.into_iter().collect()); + } + self.reduce_parent.store(Arc::new(owned_registry)); + } + + /// Look up the [`ReduceParentFn`] registered for `(outer, child)`. + /// + /// Returns an owned [`Arc`] so the session-variable borrow can be dropped before invoking the + /// function. + pub fn find_reduce_parent(&self, parent: Id, child: Id) -> Option> { + let id = self.hash_fn_ids(parent, child); + let map = self.reduce_parent.load(); + let entry = map.get(&id)?; + Some(Arc::clone(entry)) + } + + /// Combine a typed kernel id tuple into the `u64` key expected by the underlying + /// [`FnRegistry`]. All typed helpers use this path so registration and lookup agree. + fn hash_fn_ids(&self, parent: Id, child: Id) -> u64 { + FN_HASHER.hash_one((parent, child)) + } +} + +impl SessionVar for ArrayKernels { + fn as_any(&self) -> &dyn Any { + self + } + + fn as_any_mut(&mut self) -> &mut dyn Any { + self + } +} + +/// Extension trait for accessing optimizer kernels from a +/// [`VortexSession`](vortex_session::VortexSession). +pub trait ArrayKernelsExt: SessionExt { + /// Returns the [`ArrayKernels`] session variable, inserting a default-constructed one if + /// none has been registered on the session yet. + fn kernels(&self) -> Ref<'_, ArrayKernels> { + self.get::() + } +} + +impl ArrayKernelsExt for S {} diff --git a/vortex-array/src/optimizer/mod.rs b/vortex-array/src/optimizer/mod.rs index 0401e69cb9a..f991dbb2c1d 100644 --- a/vortex-array/src/optimizer/mod.rs +++ b/vortex-array/src/optimizer/mod.rs @@ -6,36 +6,69 @@ //! //! Optimization runs between execution steps, which is what enables cross-step optimizations: //! after a child is decoded, new `reduce_parent` rules may match that were previously blocked. +//! +//! There are three public entry points on [`ArrayOptimizer`]: +//! +//! - [`ArrayOptimizer::optimize`] uses only static rules registered on encoding vtables. +//! - [`ArrayOptimizer::optimize_ctx`] also consults session-scoped [`ArrayKernels`] before +//! static parent-reduce rules, so this is the entry point used by execution. +//! - [`ArrayOptimizer::optimize_recursive`] applies the session-aware optimizer to the root and +//! every descendant. use vortex_error::VortexResult; use vortex_error::vortex_bail; +use vortex_session::SessionExt; +use vortex_session::VortexSession; use crate::ArrayRef; +use crate::optimizer::kernels::ArrayKernels; +pub mod kernels; pub mod rules; /// Extension trait for optimizing array trees using reduce/reduce_parent rules. pub trait ArrayOptimizer { - /// Optimize the root array node only by running reduce and reduce_parent rules to fixpoint. + /// Optimize the root array node by running reduce and reduce_parent rules to fixpoint. + /// + /// This uses only static rules registered on encoding vtables. Use [`Self::optimize_ctx`] + /// when session-registered [`ArrayKernels`] should participate. fn optimize(&self) -> VortexResult; + /// Optimize the root array node using static rules and any [`ArrayKernels`] on `session`. + /// + /// Session kernels are checked for each `(parent_encoding_id, child_encoding_id)` pair before + /// the child's static `PARENT_RULES`. If `session` does not contain [`ArrayKernels`], this + /// behaves like [`Self::optimize`]. + fn optimize_ctx(&self, session: &VortexSession) -> VortexResult; + /// Optimize the entire array tree recursively (root and all descendants). - fn optimize_recursive(&self) -> VortexResult; + /// + /// This uses the same session-aware rule ordering as [`Self::optimize_ctx`] for every node in + /// the tree. + fn optimize_recursive(&self, session: &VortexSession) -> VortexResult; } impl ArrayOptimizer for ArrayRef { fn optimize(&self) -> VortexResult { - Ok(try_optimize(self)?.unwrap_or_else(|| self.clone())) + Ok(try_optimize(self, None)?.unwrap_or_else(|| self.clone())) + } + + fn optimize_ctx(&self, session: &VortexSession) -> VortexResult { + Ok(try_optimize(self, Some(session))?.unwrap_or_else(|| self.clone())) } - fn optimize_recursive(&self) -> VortexResult { - Ok(try_optimize_recursive(self)?.unwrap_or_else(|| self.clone())) + fn optimize_recursive(&self, session: &VortexSession) -> VortexResult { + Ok(try_optimize_recursive(self, session)?.unwrap_or_else(|| self.clone())) } } -fn try_optimize(array: &ArrayRef) -> VortexResult> { +fn try_optimize( + array: &ArrayRef, + session: Option<&VortexSession>, +) -> VortexResult> { let mut current_array = array.clone(); let mut any_optimizations = false; + let array_ref = session.and_then(|s| s.get_opt::()); // Apply reduction rules to the current array until no more rules apply. let mut loop_counter = 0; @@ -55,6 +88,21 @@ fn try_optimize(array: &ArrayRef) -> VortexResult> { // Its important to take all slots here, as `current_array` can change inside the loop. for (slot_idx, slot) in current_array.slots().iter().enumerate() { let Some(child) = slot else { continue }; + + // Session kernels take precedence over the child encoding's static PARENT_RULES. + if let Some(array_ref) = &array_ref + && let Some(plugins) = + array_ref.find_reduce_parent(current_array.encoding_id(), child.encoding_id()) + { + for plugin in plugins.as_ref() { + if let Some(new_array) = plugin(child, ¤t_array, slot_idx)? { + current_array = new_array; + any_optimizations = true; + continue 'outer; + } + } + } + if let Some(new_array) = child.reduce_parent(¤t_array, slot_idx)? { // If the parent was replaced, then we attempt to reduce it again. current_array = new_array; @@ -76,11 +124,14 @@ fn try_optimize(array: &ArrayRef) -> VortexResult> { } } -fn try_optimize_recursive(array: &ArrayRef) -> VortexResult> { +fn try_optimize_recursive( + array: &ArrayRef, + session: &VortexSession, +) -> VortexResult> { let mut current_array = array.clone(); let mut any_optimizations = false; - if let Some(new_array) = try_optimize(¤t_array)? { + if let Some(new_array) = try_optimize(¤t_array, Some(session))? { current_array = new_array; any_optimizations = true; } @@ -90,7 +141,7 @@ fn try_optimize_recursive(array: &ArrayRef) -> VortexResult> { for slot in current_array.slots() { match slot { Some(child) => { - if let Some(new_child) = try_optimize_recursive(child)? { + if let Some(new_child) = try_optimize_recursive(child, session)? { new_slots.push(Some(new_child)); any_slot_optimized = true; } else { diff --git a/vortex-array/src/patches.rs b/vortex-array/src/patches.rs index 1a614ceacf8..fa532546b50 100644 --- a/vortex-array/src/patches.rs +++ b/vortex-array/src/patches.rs @@ -6,28 +6,26 @@ use std::fmt::Debug; use std::hash::Hash; use std::ops::Range; -use itertools::Itertools; use num_traits::NumCast; use vortex_buffer::BitBuffer; use vortex_buffer::BufferMut; use vortex_error::VortexError; -use vortex_error::VortexExpect; use vortex_error::VortexResult; use vortex_error::vortex_bail; use vortex_error::vortex_ensure; use vortex_error::vortex_err; use vortex_mask::AllOr; use vortex_mask::Mask; -use vortex_mask::MaskMut; use vortex_utils::aliases::hash_map::HashMap; use crate::ArrayRef; use crate::ExecutionCtx; use crate::IntoArray; -use crate::ToCanonical; -use crate::arrays::BoolArray; +use crate::LEGACY_SESSION; +#[expect(deprecated)] +use crate::ToCanonical as _; +use crate::VortexSessionExecute; use crate::arrays::PrimitiveArray; -use crate::arrays::bool::BoolArrayExt; use crate::builtins::ArrayBuiltins; use crate::dtype::DType; use crate::dtype::IntegerPType; @@ -177,8 +175,11 @@ impl Patches { // Perform validation of components when they are host-resident. // This is not possible to do eagerly when the data is on GPU memory. if indices.is_host() && values.is_host() { - let max = usize::try_from(&indices.scalar_at(indices.len() - 1)?) - .map_err(|_| vortex_err!("indices must be a number"))?; + let max = usize::try_from(&indices.execute_scalar( + indices.len() - 1, + &mut LEGACY_SESSION.create_execution_ctx(), + )?) + .map_err(|_| vortex_err!("indices must be a number"))?; vortex_ensure!( max - offset < array_len, "Patch indices {max:?}, offset {offset} are longer than the array length {array_len}" @@ -187,7 +188,7 @@ impl Patches { #[cfg(debug_assertions)] { use crate::VortexSessionExecute; - let mut ctx = crate::LEGACY_SESSION.create_execution_ctx(); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); assert!( crate::aggregate_fn::fns::is_sorted::is_sorted(&indices, &mut ctx) .unwrap_or(false), @@ -297,7 +298,7 @@ impl Patches { }; chunk_offsets - .scalar_at(idx)? + .execute_scalar(idx, &mut LEGACY_SESSION.create_execution_ctx())? .as_primitive() .as_::() .ok_or_else(|| vortex_err!("chunk offset does not fit in usize")) @@ -368,7 +369,10 @@ impl Patches { pub fn get_patched(&self, index: usize) -> VortexResult> { self.search_index(index)? .to_found() - .map(|patch_idx| self.values().scalar_at(patch_idx)) + .map(|patch_idx| { + self.values() + .execute_scalar(patch_idx, &mut LEGACY_SESSION.create_execution_ctx()) + }) .transpose() } @@ -404,6 +408,7 @@ impl Patches { /// with the insertion point if not found. fn search_index_binary_search(indices: &ArrayRef, needle: usize) -> VortexResult { if indices.is_canonical() { + #[expect(deprecated)] let primitive = indices.to_primitive(); match_each_integer_ptype!(primitive.ptype(), |T| { let Ok(needle) = T::try_from(needle) else { @@ -550,7 +555,7 @@ impl Patches { pub fn min_index(&self) -> VortexResult { let first = self .indices - .scalar_at(0)? + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx())? .as_primitive() .as_::() .ok_or_else(|| vortex_err!("index does not fit in usize"))?; @@ -561,7 +566,10 @@ impl Patches { pub fn max_index(&self) -> VortexResult { let last = self .indices - .scalar_at(self.indices.len() - 1)? + .execute_scalar( + self.indices.len() - 1, + &mut LEGACY_SESSION.create_execution_ctx(), + )? .as_primitive() .as_::() .ok_or_else(|| vortex_err!("index does not fit in usize"))?; @@ -670,7 +678,7 @@ impl Patches { .as_ref() .map(|new_chunk_offsets| -> VortexResult { let new_chunk_base = new_chunk_offsets - .scalar_at(0)? + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx())? .as_primitive() .as_::() .ok_or_else(|| vortex_err!("chunk offset does not fit in usize"))?; @@ -768,7 +776,10 @@ impl Patches { take_indices_with_search_fn( patch_indices_slice, take_slice, - take_indices.validity_mask()?, + take_indices + .as_ref() + .validity()? + .execute_mask(take_indices.as_ref().len(), ctx)?, include_nulls, |take_idx| { self.search_index_chunked_batch( @@ -783,7 +794,10 @@ impl Patches { take_indices_with_search_fn( patch_indices_slice, take_slice, - take_indices.validity_mask()?, + take_indices + .as_ref() + .validity()? + .execute_mask(take_indices.as_ref().len(), ctx)?, include_nulls, |take_idx| { let Some(offset) = ::from(self.offset) else { @@ -868,59 +882,6 @@ impl Patches { })) } - /// Apply patches to a mutable buffer and validity mask. - /// - /// This method applies the patch values to the given buffer at the positions specified by the - /// patch indices. For non-null patch values, it updates the buffer and marks the position as - /// valid. For null patch values, it marks the position as invalid. - /// - /// # Safety - /// - /// - All patch indices after offset adjustment must be valid indices into the buffer. - /// - The buffer and validity mask must have the same length. - pub unsafe fn apply_to_buffer( - &self, - buffer: &mut [P], - validity: &mut MaskMut, - ctx: &mut ExecutionCtx, - ) { - let patch_indices = self - .indices - .clone() - .execute::(ctx) - .vortex_expect("patch indices must be convertible to PrimitiveArray"); - let patch_values = self - .values - .clone() - .execute::(ctx) - .vortex_expect("patch values must be convertible to PrimitiveArray"); - let patches_validity = patch_values - .validity() - .vortex_expect("patch values validity should be derivable"); - - let patch_values_slice = patch_values.as_slice::

(); - match_each_unsigned_integer_ptype!(patch_indices.ptype(), |I| { - let patch_indices_slice = patch_indices.as_slice::(); - - // SAFETY: - // - `Patches` invariant guarantees indices are sorted and within array bounds. - // - `patch_indices` and `patch_values` have equal length (from `Patches` invariant). - // - `buffer` and `validity` have equal length (precondition). - // - All patch indices are valid after offset adjustment (precondition). - unsafe { - apply_patches_to_buffer_inner( - buffer, - validity, - patch_indices_slice, - self.offset, - patch_values_slice, - &patches_validity, - ctx, - ); - } - }); - } - pub fn map_values(self, f: F) -> VortexResult where F: FnOnce(ArrayRef) -> VortexResult, @@ -945,83 +906,7 @@ impl Patches { } } -/// Helper function to apply patches to a buffer. -/// -/// # Safety -/// -/// - All indices in `patch_indices` after subtracting `patch_offset` must be valid indices -/// into both `buffer` and `validity`. -/// - `patch_indices` must be sorted in ascending order. -/// - `patch_indices` and `patch_values` must have the same length. -/// - `buffer` and `validity` must have the same length. -unsafe fn apply_patches_to_buffer_inner( - buffer: &mut [P], - validity: &mut MaskMut, - patch_indices: &[I], - patch_offset: usize, - patch_values: &[P], - patches_validity: &Validity, - ctx: &mut ExecutionCtx, -) where - P: NativePType, - I: UnsignedPType, -{ - debug_assert!(!patch_indices.is_empty()); - debug_assert_eq!(patch_indices.len(), patch_values.len()); - debug_assert_eq!(buffer.len(), validity.len()); - - match patches_validity { - Validity::NonNullable | Validity::AllValid => { - // All patch values are valid, apply them all. - for (&i, &value) in patch_indices.iter().zip_eq(patch_values) { - let index = i.as_() - patch_offset; - - // SAFETY: `index` is valid because caller guarantees all patch indices are within - // bounds after offset adjustment. - unsafe { - validity.set_unchecked(index); - } - buffer[index] = value; - } - } - Validity::AllInvalid => { - // All patch values are null, just mark positions as invalid. - for &i in patch_indices { - let index = i.as_() - patch_offset; - - // SAFETY: `index` is valid because caller guarantees all patch indices are within - // bounds after offset adjustment. - unsafe { - validity.unset_unchecked(index); - } - } - } - Validity::Array(array) => { - // Some patch values may be null, check each one. - let bool_array = array - .clone() - .execute::(ctx) - .vortex_expect("validity array must be convertible to BoolArray"); - let mask = bool_array.to_bit_buffer(); - for (patch_idx, (&i, &value)) in patch_indices.iter().zip_eq(patch_values).enumerate() { - let index = i.as_() - patch_offset; - - // SAFETY: `index` and `patch_idx` are valid because caller guarantees all patch - // indices are within bounds after offset adjustment. - unsafe { - if mask.value_unchecked(patch_idx) { - buffer[index] = value; - validity.set_unchecked(index); - } else { - validity.unset_unchecked(index); - } - } - } - } - } -} - -#[allow(clippy::too_many_arguments)] // private function, can clean up one day +#[expect(clippy::too_many_arguments)] // private function, can clean up one day fn take_map, T: NativePType>( indices: &[I], take_indices: &[T], @@ -1232,7 +1117,8 @@ mod test { use crate::IntoArray; use crate::LEGACY_SESSION; - use crate::ToCanonical; + #[expect(deprecated)] + use crate::ToCanonical as _; use crate::VortexSessionExecute; use crate::assert_arrays_eq; use crate::patches::Patches; @@ -1282,7 +1168,9 @@ mod test { ) .unwrap() .unwrap(); + #[expect(deprecated)] let primitive_values = taken.values().to_primitive(); + #[expect(deprecated)] let primitive_indices = taken.indices().to_primitive(); assert_eq!(taken.array_len(), 2); assert_arrays_eq!( @@ -1291,7 +1179,15 @@ mod test { ); assert_arrays_eq!(primitive_indices, PrimitiveArray::from_iter([0u64])); assert_eq!( - primitive_values.validity_mask().unwrap(), + primitive_values + .as_ref() + .validity() + .unwrap() + .execute_mask( + primitive_values.as_ref().len(), + &mut LEGACY_SESSION.create_execution_ctx() + ) + .unwrap(), Mask::from_iter(vec![true]) ); } @@ -1316,6 +1212,7 @@ mod test { .unwrap() .unwrap(); + #[expect(deprecated)] let primitive_values = taken.values().to_primitive(); assert_eq!(taken.array_len(), 2); assert_arrays_eq!( @@ -1325,7 +1222,15 @@ mod test { assert_arrays_eq!(taken.indices(), PrimitiveArray::from_iter([0u64, 1])); assert_eq!( - primitive_values.validity_mask().unwrap(), + primitive_values + .as_ref() + .validity() + .unwrap() + .execute_mask( + primitive_values.as_ref().len(), + &mut LEGACY_SESSION.create_execution_ctx() + ) + .unwrap(), Mask::from_iter([true, false]) ); } @@ -1499,9 +1404,24 @@ mod test { masked.values(), PrimitiveArray::from_iter([100i32, 200, 300]) ); - assert!(masked.values().is_valid(0).unwrap()); - assert!(masked.values().is_valid(1).unwrap()); - assert!(masked.values().is_valid(2).unwrap()); + assert!( + masked + .values() + .is_valid(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); + assert!( + masked + .values() + .is_valid(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); + assert!( + masked + .values() + .is_valid(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); // Indices should remain unchanged assert_arrays_eq!(masked.indices(), PrimitiveArray::from_iter([2u64, 5, 8])); @@ -1585,12 +1505,26 @@ mod test { assert_arrays_eq!(masked.indices(), PrimitiveArray::from_iter([5u64, 8])); // Values should be the null and 300 + #[expect(deprecated)] let masked_values = masked.values().to_primitive(); assert_eq!(masked_values.len(), 2); - assert!(!masked_values.is_valid(0).unwrap()); // the null value at index 5 - assert!(masked_values.is_valid(1).unwrap()); // the 300 value at index 8 + assert!( + !masked_values + .is_valid(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); // the null value at index 5 + assert!( + masked_values + .is_valid(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); // the 300 value at index 8 assert_eq!( - i32::try_from(&masked_values.scalar_at(1).unwrap()).unwrap(), + i32::try_from( + &masked_values + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ) + .unwrap(), 300i32 ); } @@ -1747,17 +1681,33 @@ mod test { ) .unwrap(); + #[expect(deprecated)] let values = patches.values().to_primitive(); assert_eq!( - i32::try_from(&values.scalar_at(0).unwrap()).unwrap(), + i32::try_from( + &values + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ) + .unwrap(), 100i32 ); assert_eq!( - i32::try_from(&values.scalar_at(1).unwrap()).unwrap(), + i32::try_from( + &values + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ) + .unwrap(), 200i32 ); assert_eq!( - i32::try_from(&values.scalar_at(2).unwrap()).unwrap(), + i32::try_from( + &values + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ) + .unwrap(), 300i32 ); } diff --git a/vortex-array/src/scalar/arbitrary.rs b/vortex-array/src/scalar/arbitrary.rs index 6abf02ffbc3..4b74e726749 100644 --- a/vortex-array/src/scalar/arbitrary.rs +++ b/vortex-array/src/scalar/arbitrary.rs @@ -66,7 +66,7 @@ pub fn random_scalar(u: &mut Unstructured, dtype: &DType) -> Result { .vortex_expect("unable to construct random `Scalar`_"), DType::Struct(sdt, _) => Scalar::try_new( dtype.clone(), - Some(ScalarValue::List( + Some(ScalarValue::Tuple( sdt.fields() .map(|d| random_scalar(u, &d).map(|s| s.into_value())) .collect::>>()?, @@ -75,7 +75,7 @@ pub fn random_scalar(u: &mut Unstructured, dtype: &DType) -> Result { .vortex_expect("unable to construct random `Scalar`_"), DType::List(edt, _) => Scalar::try_new( dtype.clone(), - Some(ScalarValue::List( + Some(ScalarValue::Tuple( iter::from_fn(|| { // Generate elements with 1/4 probability. u.arbitrary() @@ -88,7 +88,7 @@ pub fn random_scalar(u: &mut Unstructured, dtype: &DType) -> Result { .vortex_expect("unable to construct random `Scalar`_"), DType::FixedSizeList(edt, size, _) => Scalar::try_new( dtype.clone(), - Some(ScalarValue::List( + Some(ScalarValue::Tuple( (0..*size) .map(|_| random_scalar(u, edt).map(|s| s.into_value())) .collect::>>()?, diff --git a/vortex-array/src/scalar/arrow.rs b/vortex-array/src/scalar/arrow.rs index 73142281b0d..811417144f3 100644 --- a/vortex-array/src/scalar/arrow.rs +++ b/vortex-array/src/scalar/arrow.rs @@ -455,7 +455,7 @@ mod tests { type NativeValue<'a> = &'a str; fn id(&self) -> ExtId { - ExtId::new_ref("some_ext") + ExtId::new("some_ext") } fn serialize_metadata(&self, _options: &Self::Metadata) -> VortexResult> { diff --git a/vortex-array/src/scalar/constructor.rs b/vortex-array/src/scalar/constructor.rs index faaddc1dcef..2baf53c9528 100644 --- a/vortex-array/src/scalar/constructor.rs +++ b/vortex-array/src/scalar/constructor.rs @@ -166,7 +166,7 @@ impl Scalar { ListKind::FixedSize => DType::FixedSizeList(element_dtype, size, nullability), }; - Self::try_new(dtype, Some(ScalarValue::List(children))) + Self::try_new(dtype, Some(ScalarValue::Tuple(children))) .vortex_expect("unable to construct a list `Scalar`") } diff --git a/vortex-array/src/scalar/convert/into_scalar.rs b/vortex-array/src/scalar/convert/into_scalar.rs index ac804c06dde..f37173d5be7 100644 --- a/vortex-array/src/scalar/convert/into_scalar.rs +++ b/vortex-array/src/scalar/convert/into_scalar.rs @@ -82,7 +82,7 @@ where Scalar: From, { fn from(vec: Vec) -> Self { - ScalarValue::List( + ScalarValue::Tuple( vec.into_iter() .map(|elem| Scalar::from(elem).into_value()) .collect(), diff --git a/vortex-array/src/scalar/downcast.rs b/vortex-array/src/scalar/downcast.rs index ce9996514e3..1276141fc35 100644 --- a/vortex-array/src/scalar/downcast.rs +++ b/vortex-array/src/scalar/downcast.rs @@ -115,12 +115,13 @@ impl Scalar { /// Returns a view of the scalar as a list scalar. /// - /// Note that we use [`ListScalar`] to represent **both** [`List`](crate::dtype::DType::List) and - /// [`FixedSizeList`](crate::dtype::DType::FixedSizeList). + /// Note that we use [`ListScalar`] to represent **both** [`List`](crate::dtype::DType::List) + /// and [`FixedSizeList`](crate::dtype::DType::FixedSizeList). /// /// # Panics /// - /// Panics if the scalar does not have a [`List`](crate::dtype::DType::List) or [`FixedSizeList`](crate::dtype::DType::FixedSizeList) type. + /// Panics if the scalar does not have a [`List`](crate::dtype::DType::List) or + /// [`FixedSizeList`](crate::dtype::DType::FixedSizeList) type. pub fn as_list(&self) -> ListScalar<'_> { self.as_list_opt() .vortex_expect("Failed to convert scalar to list") @@ -128,8 +129,8 @@ impl Scalar { /// Returns a view of the scalar as a list scalar if it has a list type. /// - /// Note that we use [`ListScalar`] to represent **both** [`List`](crate::dtype::DType::List) and - /// [`FixedSizeList`](crate::dtype::DType::FixedSizeList). + /// Note that we use [`ListScalar`] to represent **both** [`List`](crate::dtype::DType::List) + /// and [`FixedSizeList`](crate::dtype::DType::FixedSizeList). pub fn as_list_opt(&self) -> Option> { ListScalar::try_new(self.dtype(), self.value()).ok() } @@ -172,7 +173,7 @@ impl Scalar { } impl ScalarValue { - /// Returns the boolean value, panicking if the value is not a [`Bool`][ScalarValue::Bool]. + /// Returns the boolean value, panicking if the value is not a [`Bool`](ScalarValue::Bool). pub fn as_bool(&self) -> bool { match self { ScalarValue::Bool(b) => *b, @@ -181,7 +182,7 @@ impl ScalarValue { } /// Returns the primitive value, panicking if the value is not a - /// [`Primitive`][ScalarValue::Primitive]. + /// [`Primitive`](ScalarValue::Primitive). pub fn as_primitive(&self) -> &PValue { match self { ScalarValue::Primitive(p) => p, @@ -190,7 +191,7 @@ impl ScalarValue { } /// Returns the decimal value, panicking if the value is not a - /// [`Decimal`][ScalarValue::Decimal]. + /// [`Decimal`](ScalarValue::Decimal). pub fn as_decimal(&self) -> &DecimalValue { match self { ScalarValue::Decimal(d) => d, @@ -198,7 +199,7 @@ impl ScalarValue { } } - /// Returns the UTF-8 string value, panicking if the value is not a [`Utf8`][ScalarValue::Utf8]. + /// Returns the UTF-8 string value, panicking if the value is not a [`Utf8`](ScalarValue::Utf8). pub fn as_utf8(&self) -> &BufferString { match self { ScalarValue::Utf8(s) => s, @@ -206,7 +207,7 @@ impl ScalarValue { } } - /// Returns the binary value, panicking if the value is not a [`Binary`][ScalarValue::Binary]. + /// Returns the binary value, panicking if the value is not a [`Binary`](ScalarValue::Binary). pub fn as_binary(&self) -> &ByteBuffer { match self { ScalarValue::Binary(b) => b, @@ -214,15 +215,15 @@ impl ScalarValue { } } - /// Returns the list elements, panicking if the value is not a [`List`][ScalarValue::List]. + /// Returns the tuple elements, panicking if the value is not a [`Tuple`](ScalarValue::Tuple). pub fn as_list(&self) -> &[Option] { match self { - ScalarValue::List(elements) => elements, - _ => vortex_panic!("ScalarValue is not a List"), + ScalarValue::Tuple(elements) => elements, + _ => vortex_panic!("ScalarValue is not a Tuple"), } } - /// Returns the boolean value, panicking if the value is not a [`Bool`][ScalarValue::Bool]. + /// Returns the boolean value, panicking if the value is not a [`Bool`](ScalarValue::Bool). pub fn into_bool(self) -> bool { match self { ScalarValue::Bool(b) => b, @@ -231,7 +232,7 @@ impl ScalarValue { } /// Returns the primitive value, panicking if the value is not a - /// [`Primitive`][ScalarValue::Primitive]. + /// [`Primitive`](ScalarValue::Primitive). pub fn into_primitive(self) -> PValue { match self { ScalarValue::Primitive(p) => p, @@ -240,7 +241,7 @@ impl ScalarValue { } /// Returns the decimal value, panicking if the value is not a - /// [`Decimal`][ScalarValue::Decimal]. + /// [`Decimal`](ScalarValue::Decimal). pub fn into_decimal(self) -> DecimalValue { match self { ScalarValue::Decimal(d) => d, @@ -248,7 +249,7 @@ impl ScalarValue { } } - /// Returns the UTF-8 string value, panicking if the value is not a [`Utf8`][ScalarValue::Utf8]. + /// Returns the UTF-8 string value, panicking if the value is not a [`Utf8`](ScalarValue::Utf8). pub fn into_utf8(self) -> BufferString { match self { ScalarValue::Utf8(s) => s, @@ -256,7 +257,7 @@ impl ScalarValue { } } - /// Returns the binary value, panicking if the value is not a [`Binary`][ScalarValue::Binary]. + /// Returns the binary value, panicking if the value is not a [`Binary`](ScalarValue::Binary). pub fn into_binary(self) -> ByteBuffer { match self { ScalarValue::Binary(b) => b, @@ -264,11 +265,11 @@ impl ScalarValue { } } - /// Returns the list elements, panicking if the value is not a [`List`][ScalarValue::List]. + /// Returns the tuple elements, panicking if the value is not a [`Tuple`](ScalarValue::Tuple). pub fn into_list(self) -> Vec> { match self { - ScalarValue::List(elements) => elements, - _ => vortex_panic!("ScalarValue is not a List"), + ScalarValue::Tuple(elements) => elements, + _ => vortex_panic!("ScalarValue is not a Tuple"), } } diff --git a/vortex-array/src/scalar/proto.rs b/vortex-array/src/scalar/proto.rs index 2c45a450bfa..938eeb2f778 100644 --- a/vortex-array/src/scalar/proto.rs +++ b/vortex-array/src/scalar/proto.rs @@ -101,7 +101,7 @@ impl From<&ScalarValue> for pb::ScalarValue { ScalarValue::Binary(v) => pb::ScalarValue { kind: Some(Kind::BytesValue(v.to_vec())), }, - ScalarValue::List(v) => { + ScalarValue::Tuple(v) => { let mut values = Vec::with_capacity(v.len()); for elem in v.iter() { values.push(ScalarValue::to_proto(elem.as_ref())); @@ -435,7 +435,7 @@ fn bytes_from_proto(bytes: &[u8], dtype: &DType) -> VortexResult { } } -/// Deserialize a [`ScalarValue::List`] from a protobuf `ListValue`. +/// Deserialize a [`ScalarValue::Tuple`] from a protobuf `ListValue`. fn list_from_proto( v: &ListValue, dtype: &DType, @@ -454,7 +454,7 @@ fn list_from_proto( )?); } - Ok(ScalarValue::List(values)) + Ok(ScalarValue::Tuple(values)) } #[cfg(test)] @@ -531,7 +531,7 @@ mod tests { Arc::new(DType::Primitive(PType::I32, Nullability::Nullable)), Nullability::Nullable, ), - Some(ScalarValue::List(vec![ + Some(ScalarValue::Tuple(vec![ Some(ScalarValue::Primitive(42i32.into())), Some(ScalarValue::Primitive(43i32.into())), ])), diff --git a/vortex-array/src/scalar/scalar_impl.rs b/vortex-array/src/scalar/scalar_impl.rs index edb102fe384..e6d0f721b6b 100644 --- a/vortex-array/src/scalar/scalar_impl.rs +++ b/vortex-array/src/scalar/scalar_impl.rs @@ -14,6 +14,7 @@ use vortex_error::vortex_panic; use crate::dtype::DType; use crate::dtype::NativeDType; use crate::dtype::PType; +use crate::dtype::StructFields; use crate::scalar::Scalar; use crate::scalar::ScalarValue; @@ -263,6 +264,16 @@ impl Scalar { } } +/// We implement `Hash` manually to be consistent with `PartialEq`. Since we ignore nullability in +/// equality comparisons, we must also ignore it when hashing to maintain the invariant that equal +/// values have equal hashes. +impl Hash for Scalar { + fn hash(&self, state: &mut H) { + self.dtype.as_nonnullable().hash(state); + self.value.hash(state); + } +} + /// We implement `PartialEq` manually because we want to ignore nullability when comparing scalars. /// Two scalars with the same value but different nullability should be considered equal. /// @@ -288,7 +299,14 @@ impl PartialOrd for Scalar { /// - Non-null values are compared according to their natural ordering /// /// # Examples - /// ```ignore + /// + /// ``` + /// use std::cmp::Ordering; + /// use vortex_array::dtype::DType; + /// use vortex_array::dtype::Nullability; + /// use vortex_array::dtype::PType; + /// use vortex_array::scalar::Scalar; + /// /// // Same types compare successfully /// let a = Scalar::primitive(10i32, Nullability::NonNullable); /// let b = Scalar::primitive(20i32, Nullability::NonNullable); @@ -308,16 +326,101 @@ impl PartialOrd for Scalar { if !self.dtype().eq_ignore_nullability(other.dtype()) { return None; } - self.value().partial_cmp(&other.value()) + + partial_cmp_scalar_values(self.dtype(), self.value(), other.value()) } } -/// We implement `Hash` manually to be consistent with `PartialEq`. Since we ignore nullability -/// in equality comparisons, we must also ignore it when hashing to maintain the invariant that -/// equal values have equal hashes. -impl Hash for Scalar { - fn hash(&self, state: &mut H) { - self.dtype.as_nonnullable().hash(state); - self.value.hash(state); +/// Compare two optional scalar values using `dtype` for nested tuple interpretation. +fn partial_cmp_scalar_values( + dtype: &DType, + lhs: Option<&ScalarValue>, + rhs: Option<&ScalarValue>, +) -> Option { + match (lhs, rhs) { + (None, None) => Some(Ordering::Equal), + (None, Some(_)) => Some(Ordering::Less), + (Some(_), None) => Some(Ordering::Greater), + (Some(lhs), Some(rhs)) => partial_cmp_non_null_scalar_values(dtype, lhs, rhs), + } +} + +/// Compare two non-null scalar values, consulting `dtype` only for tuple-backed values. +fn partial_cmp_non_null_scalar_values( + dtype: &DType, + lhs: &ScalarValue, + rhs: &ScalarValue, +) -> Option { + // `Scalar::validate` guarantees that a scalar's value matches its dtype. Most of the scalar + // value variants have only 1 method of comparison, regardless of the dtype. + match (lhs, rhs) { + (ScalarValue::Bool(lhs), ScalarValue::Bool(rhs)) => lhs.partial_cmp(rhs), + (ScalarValue::Primitive(lhs), ScalarValue::Primitive(rhs)) => lhs.partial_cmp(rhs), + (ScalarValue::Decimal(lhs), ScalarValue::Decimal(rhs)) => lhs.partial_cmp(rhs), + (ScalarValue::Utf8(lhs), ScalarValue::Utf8(rhs)) => lhs.partial_cmp(rhs), + (ScalarValue::Binary(lhs), ScalarValue::Binary(rhs)) => lhs.partial_cmp(rhs), + // `Tuple` is the exception here. Since it backs lists, fixed-size lists, and structs, we + // need the dtype to know whether children share one element dtype or use per-field dtypes. + (ScalarValue::Tuple(lhs), ScalarValue::Tuple(rhs)) => { + partial_cmp_tuple_values(dtype, lhs, rhs) + } + // Variant values can have a different dtype in each row, so it doesn't make sense to + // compare them. + (ScalarValue::Variant(_), ScalarValue::Variant(_)) => None, + _ => None, + } +} + +/// Compare tuple values according to the list, fixed-size list, or struct dtype layout. +fn partial_cmp_tuple_values( + dtype: &DType, + lhs: &[Option], + rhs: &[Option], +) -> Option { + match dtype { + DType::List(element_dtype, _) | DType::FixedSizeList(element_dtype, ..) => { + partial_cmp_list_values(element_dtype, lhs, rhs) + } + DType::Struct(fields, _) => partial_cmp_struct_values(fields, lhs, rhs), + DType::Extension(ext_dtype) => { + partial_cmp_tuple_values(ext_dtype.storage_dtype(), lhs, rhs) + } + _ => None, + } +} + +/// Compare list tuple values using the shared element dtype for each element. +fn partial_cmp_list_values( + element_dtype: &DType, + lhs: &[Option], + rhs: &[Option], +) -> Option { + for (lhs, rhs) in lhs.iter().zip(rhs.iter()) { + match partial_cmp_scalar_values(element_dtype, lhs.as_ref(), rhs.as_ref())? { + Ordering::Equal => continue, + ordering => return Some(ordering), + } + } + + Some(lhs.len().cmp(&rhs.len())) +} + +/// Compare struct tuple values using each field's dtype in field order. +fn partial_cmp_struct_values( + fields: &StructFields, + lhs: &[Option], + rhs: &[Option], +) -> Option { + if lhs.len() != fields.nfields() || rhs.len() != fields.nfields() { + return None; } + + for ((field_dtype, lhs), rhs) in fields.fields().zip(lhs.iter()).zip(rhs.iter()) { + match partial_cmp_scalar_values(&field_dtype, lhs.as_ref(), rhs.as_ref())? { + Ordering::Equal => continue, + ordering => return Some(ordering), + } + } + + Some(Ordering::Equal) } diff --git a/vortex-array/src/scalar/scalar_value.rs b/vortex-array/src/scalar/scalar_value.rs index da32bafa947..6c25fe70a73 100644 --- a/vortex-array/src/scalar/scalar_value.rs +++ b/vortex-array/src/scalar/scalar_value.rs @@ -3,7 +3,6 @@ //! Core [`ScalarValue`] type definition. -use std::cmp::Ordering; use std::fmt::Display; use std::fmt::Formatter; @@ -33,8 +32,10 @@ pub enum ScalarValue { Utf8(BufferString), /// A binary (byte array) value. Binary(ByteBuffer), - /// A list of potentially null scalar values. - List(Vec>), + /// A tuple of potentially null scalar values. + /// + /// Used as the underlying representation for list, fixed-size list, and struct scalars. + Tuple(Vec>), /// A row-specific scalar wrapped by `DType::Variant`. Variant(Box), } @@ -49,17 +50,17 @@ impl ScalarValue { DType::Decimal(dt, ..) => Self::Decimal(DecimalValue::zero(dt)), DType::Utf8(_) => Self::Utf8(BufferString::empty()), DType::Binary(_) => Self::Binary(ByteBuffer::empty()), - DType::List(..) => Self::List(vec![]), + DType::List(..) => Self::Tuple(vec![]), DType::FixedSizeList(edt, size, _) => { let elements = (0..*size).map(|_| Some(Self::zero_value(edt))).collect(); - Self::List(elements) + Self::Tuple(elements) } DType::Struct(fields, _) => { let field_values = fields .fields() .map(|f| Some(Self::zero_value(&f))) .collect(); - Self::List(field_values) + Self::Tuple(field_values) } DType::Extension(ext_dtype) => { // Since we have no way to define a "zero" extension value (since we have no idea @@ -89,14 +90,14 @@ impl ScalarValue { DType::Decimal(dt, ..) => Self::Decimal(DecimalValue::zero(dt)), DType::Utf8(_) => Self::Utf8(BufferString::empty()), DType::Binary(_) => Self::Binary(ByteBuffer::empty()), - DType::List(..) => Self::List(vec![]), + DType::List(..) => Self::Tuple(vec![]), DType::FixedSizeList(edt, size, _) => { let elements = (0..*size).map(|_| Self::default_value(edt)).collect(); - Self::List(elements) + Self::Tuple(elements) } DType::Struct(fields, _) => { let field_values = fields.fields().map(|f| Self::default_value(&f)).collect(); - Self::List(field_values) + Self::Tuple(field_values) } DType::Extension(ext_dtype) => { // Since we have no way to define a "default" extension value (since we have no idea @@ -109,22 +110,6 @@ impl ScalarValue { } } -impl PartialOrd for ScalarValue { - fn partial_cmp(&self, other: &Self) -> Option { - match (self, other) { - (ScalarValue::Bool(a), ScalarValue::Bool(b)) => a.partial_cmp(b), - (ScalarValue::Primitive(a), ScalarValue::Primitive(b)) => a.partial_cmp(b), - (ScalarValue::Decimal(a), ScalarValue::Decimal(b)) => a.partial_cmp(b), - (ScalarValue::Utf8(a), ScalarValue::Utf8(b)) => a.partial_cmp(b), - (ScalarValue::Binary(a), ScalarValue::Binary(b)) => a.partial_cmp(b), - (ScalarValue::List(a), ScalarValue::List(b)) => a.partial_cmp(b), - (ScalarValue::Variant(a), ScalarValue::Variant(b)) => a.partial_cmp(b), - // (ScalarValue::Extension(a), ScalarValue::Extension(b)) => a.partial_cmp(b), - _ => None, - } - } -} - impl Display for ScalarValue { fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { match self { @@ -156,7 +141,7 @@ impl Display for ScalarValue { write!(f, "{}", to_hex(b)) } } - ScalarValue::List(elements) => { + ScalarValue::Tuple(elements) => { write!(f, "[")?; for (i, element) in elements.iter().enumerate() { if i > 0 { diff --git a/vortex-array/src/scalar/tests/casting.rs b/vortex-array/src/scalar/tests/casting.rs index 4e6ecdd60b9..ae63232330b 100644 --- a/vortex-array/src/scalar/tests/casting.rs +++ b/vortex-array/src/scalar/tests/casting.rs @@ -32,7 +32,7 @@ mod tests { type NativeValue<'a> = &'a str; fn id(&self) -> ExtId { - ExtId::new_ref("apples") + ExtId::new("apples") } fn serialize_metadata(&self, _metadata: &Self::Metadata) -> VortexResult> { @@ -152,7 +152,7 @@ mod tests { Some(ScalarValue::Primitive(PValue::F32(f32_value))), ]; - let scalar = Scalar::new(struct_dtype, Some(ScalarValue::List(field_values))); + let scalar = Scalar::new(struct_dtype, Some(ScalarValue::Tuple(field_values))); let struct_scalar = scalar.as_struct(); let fields: Vec<_> = (0..3) @@ -209,7 +209,7 @@ mod tests { ))), ]; - let scalar = Scalar::new(list_dtype, Some(ScalarValue::List(elements))); + let scalar = Scalar::new(list_dtype, Some(ScalarValue::Tuple(elements))); let list_scalar = scalar.as_list(); let elements = list_scalar.elements().unwrap(); @@ -247,7 +247,7 @@ mod tests { type NativeValue<'a> = &'a str; fn id(&self) -> ExtId { - ExtId::new_ref("f16_ext") + ExtId::new("f16_ext") } fn serialize_metadata(&self, _metadata: &Self::Metadata) -> VortexResult> { @@ -304,7 +304,7 @@ mod tests { type NativeValue<'a> = &'a str; fn id(&self) -> ExtId { - ExtId::new_ref("struct_ext") + ExtId::new("struct_ext") } fn serialize_metadata(&self, _metadata: &Self::Metadata) -> VortexResult> { @@ -353,7 +353,7 @@ mod tests { let scalar = Scalar::new( DType::Extension(ext_dtype.erased()), - Some(ScalarValue::List(field_values)), + Some(ScalarValue::Tuple(field_values)), ); // Verify the struct field was coerced diff --git a/vortex-array/src/scalar/tests/nested.rs b/vortex-array/src/scalar/tests/nested.rs index 0aadd4ce2c9..d02bf43c631 100644 --- a/vortex-array/src/scalar/tests/nested.rs +++ b/vortex-array/src/scalar/tests/nested.rs @@ -311,7 +311,7 @@ mod tests { Arc::from(DType::Primitive(PType::U16, Nullability::Nullable)), Nullability::Nullable, ), - Some(ScalarValue::List(vec![ + Some(ScalarValue::Tuple(vec![ Some(ScalarValue::Primitive(PValue::U16(6))), Some(ScalarValue::Primitive(PValue::U16(100))), ])), @@ -350,7 +350,7 @@ mod tests { Arc::from(DType::Primitive(PType::U16, Nullability::Nullable)), Nullability::Nullable, ), - Some(ScalarValue::List(vec![ + Some(ScalarValue::Tuple(vec![ Some(ScalarValue::Primitive(PValue::U16(100))), Some(ScalarValue::Primitive(PValue::U16(256))), // Too large for U8 Some(ScalarValue::Primitive(PValue::U16(1000))), // Too large for U8 diff --git a/vortex-array/src/scalar/tests/nullability.rs b/vortex-array/src/scalar/tests/nullability.rs index 4d2bff418a5..d862d08d837 100644 --- a/vortex-array/src/scalar/tests/nullability.rs +++ b/vortex-array/src/scalar/tests/nullability.rs @@ -54,7 +54,7 @@ mod tests { Arc::from(DType::Primitive(PType::U16, Nullability::Nullable)), Nullability::Nullable, ), - Some(ScalarValue::List(vec![Some(ScalarValue::Primitive( + Some(ScalarValue::Tuple(vec![Some(ScalarValue::Primitive( PValue::U16(6), ))])), ); @@ -97,7 +97,7 @@ mod tests { Arc::from(DType::Primitive(PType::U16, Nullability::Nullable)), Nullability::Nullable, ), - Some(ScalarValue::List(vec![ + Some(ScalarValue::Tuple(vec![ Some(ScalarValue::Primitive(PValue::U16(6))), None, Some(ScalarValue::Primitive(PValue::U16(10))), @@ -200,7 +200,7 @@ mod tests { Arc::from(DType::Primitive(PType::U16, Nullability::Nullable)), Nullability::Nullable, ), - Some(ScalarValue::List(vec![ + Some(ScalarValue::Tuple(vec![ Some(ScalarValue::Primitive(PValue::U16(6))), None, ])), diff --git a/vortex-array/src/scalar/truncation.rs b/vortex-array/src/scalar/truncation.rs index cc1d7a231a7..3a2b562d7aa 100644 --- a/vortex-array/src/scalar/truncation.rs +++ b/vortex-array/src/scalar/truncation.rs @@ -14,7 +14,7 @@ use crate::scalar::Scalar; use crate::scalar::StringLike; /// A trait for truncating [`Scalar`]s to a given length in bytes. -#[allow(clippy::len_without_is_empty)] +#[expect(clippy::len_without_is_empty)] pub trait ScalarTruncation: Send + Sized { /// Unwrap a Scalar into a ScalarTruncation object /// diff --git a/vortex-array/src/scalar/typed_view/binary.rs b/vortex-array/src/scalar/typed_view/binary.rs index 9235d4e17ef..cb107e3613c 100644 --- a/vortex-array/src/scalar/typed_view/binary.rs +++ b/vortex-array/src/scalar/typed_view/binary.rs @@ -20,7 +20,7 @@ use crate::scalar::ScalarValue; /// /// This type provides a view into a binary scalar value, which can be either /// a valid byte buffer or null. -#[derive(Debug, Clone, Hash)] +#[derive(Debug, Clone, Copy, Hash)] pub struct BinaryScalar<'a> { /// The data type of this scalar. dtype: &'a DType, diff --git a/vortex-array/src/scalar/typed_view/bool.rs b/vortex-array/src/scalar/typed_view/bool.rs index 10f8cd47d9f..c971d1169c6 100644 --- a/vortex-array/src/scalar/typed_view/bool.rs +++ b/vortex-array/src/scalar/typed_view/bool.rs @@ -19,7 +19,7 @@ use crate::scalar::ScalarValue; /// /// This type provides a view into a boolean scalar value, which can be either /// true, false, or null. -#[derive(Debug, Clone, Hash, Eq)] +#[derive(Debug, Clone, Copy, Hash, Eq)] pub struct BoolScalar<'a> { /// The data type of this scalar. dtype: &'a DType, diff --git a/vortex-array/src/scalar/typed_view/decimal/tests.rs b/vortex-array/src/scalar/typed_view/decimal/tests.rs index fa6ccfad96b..ca5b06778d9 100644 --- a/vortex-array/src/scalar/typed_view/decimal/tests.rs +++ b/vortex-array/src/scalar/typed_view/decimal/tests.rs @@ -3,8 +3,6 @@ //! Tests for decimal scalar casting functionality. -#![allow(clippy::disallowed_types, clippy::panic)] - use rstest::rstest; use vortex_utils::aliases::hash_set::HashSet; diff --git a/vortex-array/src/scalar/typed_view/extension/mod.rs b/vortex-array/src/scalar/typed_view/extension/mod.rs index dfbc25f936d..3508883fc92 100644 --- a/vortex-array/src/scalar/typed_view/extension/mod.rs +++ b/vortex-array/src/scalar/typed_view/extension/mod.rs @@ -19,7 +19,7 @@ use crate::scalar::ScalarValue; /// A scalar value representing an extension type. /// /// Extension types allow wrapping a storage type with custom semantics. -#[derive(Debug, Clone)] +#[derive(Debug, Clone, Copy)] pub struct ExtScalar<'a> { /// A reference to the `DType` of the extension type. This **must** be the [`DType::Extension`] /// variant. diff --git a/vortex-array/src/scalar/typed_view/extension/tests.rs b/vortex-array/src/scalar/typed_view/extension/tests.rs index 5faa9550402..909f8a4b11a 100644 --- a/vortex-array/src/scalar/typed_view/extension/tests.rs +++ b/vortex-array/src/scalar/typed_view/extension/tests.rs @@ -21,7 +21,7 @@ impl ExtVTable for TestI32Ext { type NativeValue<'a> = &'a str; fn id(&self) -> ExtId { - ExtId::new_ref("test_ext") + ExtId::new("test_ext") } fn serialize_metadata(&self, _metadata: &Self::Metadata) -> VortexResult> { @@ -104,7 +104,7 @@ fn test_ext_scalar_partial_ord_different_types() { type NativeValue<'a> = &'a str; fn id(&self) -> ExtId { - ExtId::new_ref("test_ext_2") + ExtId::new("test_ext_2") } fn serialize_metadata(&self, _metadata: &Self::Metadata) -> VortexResult> { @@ -286,7 +286,7 @@ fn test_ext_scalar_with_metadata() { type NativeValue<'a> = &'a str; fn id(&self) -> ExtId { - ExtId::new_ref("test_ext_metadata") + ExtId::new("test_ext_metadata") } fn serialize_metadata(&self, _metadata: &Self::Metadata) -> VortexResult> { diff --git a/vortex-array/src/scalar/typed_view/list.rs b/vortex-array/src/scalar/typed_view/list.rs index 3a158dd59ca..15de731089a 100644 --- a/vortex-array/src/scalar/typed_view/list.rs +++ b/vortex-array/src/scalar/typed_view/list.rs @@ -29,7 +29,7 @@ use crate::scalar::ScalarValue; /// number of `elements` is equal to the `size` field of the [`FixedSizeList`]. /// /// [`FixedSizeList`]: DType::FixedSizeList -#[derive(Debug, Clone)] +#[derive(Debug, Clone, Copy)] pub struct ListScalar<'a> { /// The data type of this scalar. dtype: &'a DType, @@ -207,7 +207,7 @@ impl<'a> ListScalar<'a> { Scalar::try_new( dtype.clone(), - Some(ScalarValue::List( + Some(ScalarValue::Tuple( self.elements .ok_or_else(|| vortex_err!("nullness should be handled in Scalar::cast"))? .iter() diff --git a/vortex-array/src/scalar/typed_view/struct_.rs b/vortex-array/src/scalar/typed_view/struct_.rs index 93b796f055f..d8069bab77c 100644 --- a/vortex-array/src/scalar/typed_view/struct_.rs +++ b/vortex-array/src/scalar/typed_view/struct_.rs @@ -27,7 +27,7 @@ use crate::scalar::ScalarValue; /// /// This type provides a view into a struct scalar value, which can contain /// named fields with different types, or be null. -#[derive(Debug, Clone)] +#[derive(Debug, Clone, Copy)] pub struct StructScalar<'a> { /// The data type of this scalar. dtype: &'a DType, @@ -236,7 +236,7 @@ impl<'a> StructScalar<'a> { .map(|s| s.into_value()) }) .collect::>>()?; - Scalar::try_new(dtype.clone(), Some(ScalarValue::List(fields))) + Scalar::try_new(dtype.clone(), Some(ScalarValue::Tuple(fields))) } else { Ok(Scalar::null(dtype.clone())) } @@ -261,7 +261,7 @@ impl<'a> StructScalar<'a> { return Ok(Scalar::null(projected_dtype)); }; - let new_fields = ScalarValue::List( + let new_fields = ScalarValue::Tuple( projection .iter() .map(|name| { @@ -306,7 +306,7 @@ impl Scalar { } let value_children: Vec<_> = children.into_iter().map(|x| x.into_value()).collect(); - Self::try_new(dtype, Some(ScalarValue::List(value_children))) + Self::try_new(dtype, Some(ScalarValue::Tuple(value_children))) .vortex_expect("unable to construct a struct `Scalar`") } @@ -323,7 +323,7 @@ impl Scalar { children: impl IntoIterator, ) -> Self { let value_children: Vec<_> = children.into_iter().map(|s| s.into_value()).collect(); - unsafe { Self::new_unchecked(dtype, Some(ScalarValue::List(value_children))) } + unsafe { Self::new_unchecked(dtype, Some(ScalarValue::Tuple(value_children))) } } } diff --git a/vortex-array/src/scalar/typed_view/utf8.rs b/vortex-array/src/scalar/typed_view/utf8.rs index 7e6dcba8635..5a9ad84064b 100644 --- a/vortex-array/src/scalar/typed_view/utf8.rs +++ b/vortex-array/src/scalar/typed_view/utf8.rs @@ -22,7 +22,7 @@ use crate::scalar::ScalarValue; /// /// This type provides a view into a UTF-8 string scalar value, which can be either /// a valid UTF-8 string or null. -#[derive(Debug, Clone, Hash, Eq)] +#[derive(Debug, Clone, Copy, Hash, Eq)] pub struct Utf8Scalar<'a> { /// The data type of this scalar. dtype: &'a DType, @@ -159,7 +159,7 @@ mod private { impl Sealed for BufferString {} impl StringLike for BufferString { - #[allow(clippy::unwrap_in_result, clippy::expect_used)] + #[expect(clippy::expect_used)] fn increment(self) -> Result { if self.is_empty() { return Err(self); diff --git a/vortex-array/src/scalar/validate.rs b/vortex-array/src/scalar/validate.rs index 2696dc1f804..4df38187a9a 100644 --- a/vortex-array/src/scalar/validate.rs +++ b/vortex-array/src/scalar/validate.rs @@ -74,8 +74,8 @@ impl Scalar { ); } DType::List(elem_dtype, _) => { - let ScalarValue::List(elements) = value else { - vortex_bail!("list dtype expected List value, got {value}"); + let ScalarValue::Tuple(elements) = value else { + vortex_bail!("list dtype expected Tuple value, got {value}"); }; for (i, element) in elements.iter().enumerate() { @@ -84,8 +84,8 @@ impl Scalar { } } DType::FixedSizeList(elem_dtype, size, _) => { - let ScalarValue::List(elements) = value else { - vortex_bail!("fixed-size list dtype expected List value, got {value}",); + let ScalarValue::Tuple(elements) = value else { + vortex_bail!("fixed-size list dtype expected Tuple value, got {value}",); }; let len = elements.len(); @@ -102,8 +102,8 @@ impl Scalar { } } DType::Struct(fields, _) => { - let ScalarValue::List(values) = value else { - vortex_bail!("struct dtype expected List value, got {value}"); + let ScalarValue::Tuple(values) = value else { + vortex_bail!("struct dtype expected Tuple value, got {value}"); }; let nfields = fields.nfields(); diff --git a/vortex-array/src/scalar_fn/erased.rs b/vortex-array/src/scalar_fn/erased.rs index 154ad13bace..10e82d25455 100644 --- a/vortex-array/src/scalar_fn/erased.rs +++ b/vortex-array/src/scalar_fn/erased.rs @@ -31,20 +31,19 @@ use crate::scalar_fn::ScalarFnId; use crate::scalar_fn::ScalarFnVTable; use crate::scalar_fn::ScalarFnVTableExt; use crate::scalar_fn::SimplifyCtx; -use crate::scalar_fn::fns::is_null::IsNull; -use crate::scalar_fn::fns::not::Not; +use crate::scalar_fn::fns::is_not_null::IsNotNull; use crate::scalar_fn::options::ScalarFnOptions; use crate::scalar_fn::signature::ScalarFnSignature; use crate::scalar_fn::typed::DynScalarFn; -use crate::scalar_fn::typed::ScalarFn; +use crate::scalar_fn::typed::TypedScalarFnInstance; /// A type-erased scalar function, pairing a vtable with bound options behind a trait object. /// /// This stores a [`ScalarFnVTable`] and its options behind an `Arc`, allowing /// heterogeneous storage inside [`Expression`] and [`crate::arrays::ScalarFnArray`]. /// -/// Use [`super::ScalarFn::new()`] to construct, and [`super::ScalarFn::erased()`] to obtain a -/// [`ScalarFnRef`]. +/// Use [`super::TypedScalarFnInstance::new()`] to construct, and [`super::TypedScalarFnInstance::erased()`] to +/// obtain a [`ScalarFnRef`]. #[derive(Clone)] pub struct ScalarFnRef(pub(super) Arc); @@ -56,14 +55,14 @@ impl ScalarFnRef { /// Returns whether the scalar function is of the given vtable type. pub fn is(&self) -> bool { - self.0.as_any().is::>() + self.0.as_any().is::>() } /// Returns the typed options for this scalar function if it matches the given vtable type. pub fn as_opt(&self) -> Option<&V::Options> { self.0 .as_any() - .downcast_ref::>() + .downcast_ref::>() .map(|sf| sf.options()) } @@ -77,24 +76,26 @@ impl ScalarFnRef { .vortex_expect("Expression options type mismatch") } - /// Downcast to the concrete [`ScalarFn`]. + /// Downcast to the concrete [`TypedScalarFnInstance`]. /// /// Returns `Err(self)` if the downcast fails. - pub fn try_downcast(self) -> Result>, ScalarFnRef> { - if self.0.as_any().is::>() { - let ptr = Arc::into_raw(self.0) as *const ScalarFn; + pub fn try_downcast( + self, + ) -> Result>, ScalarFnRef> { + if self.0.as_any().is::>() { + let ptr = Arc::into_raw(self.0) as *const TypedScalarFnInstance; Ok(unsafe { Arc::from_raw(ptr) }) } else { Err(self) } } - /// Downcast to the concrete [`ScalarFn`]. + /// Downcast to the concrete [`TypedScalarFnInstance`]. /// /// # Panics /// /// Panics if the downcast fails. - pub fn downcast(self) -> Arc> { + pub fn downcast(self) -> Arc> { self.try_downcast::() .map_err(|this| { vortex_err!( @@ -106,9 +107,9 @@ impl ScalarFnRef { .vortex_expect("Failed to downcast ScalarFnRef") } - /// Try to downcast into a typed [`ScalarFn`]. - pub fn downcast_ref(&self) -> Option<&ScalarFn> { - self.0.as_any().downcast_ref::>() + /// Try to downcast into a typed [`TypedScalarFnInstance`]. + pub fn downcast_ref(&self) -> Option<&TypedScalarFnInstance> { + self.0.as_any().downcast_ref::>() } /// The type-erased options for this scalar function. @@ -135,11 +136,7 @@ impl ScalarFnRef { pub fn validity(&self, expr: &Expression) -> VortexResult { Ok(self.0.validity(expr)?.unwrap_or_else(|| { // TODO(ngates): make validity a mandatory method on VTable to avoid this fallback. - // TODO(ngates): add an IsNotNull expression. - Not.new_expr( - EmptyOptions, - [IsNull.new_expr(EmptyOptions, [expr.clone()])], - ) + IsNotNull.new_expr(EmptyOptions, [expr.clone()]) })) } diff --git a/vortex-array/src/scalar_fn/fns/between/kernel.rs b/vortex-array/src/scalar_fn/fns/between/kernel.rs index 3bf543940c4..ee4fb688982 100644 --- a/vortex-array/src/scalar_fn/fns/between/kernel.rs +++ b/vortex-array/src/scalar_fn/fns/between/kernel.rs @@ -11,7 +11,7 @@ use crate::ArrayRef; use crate::ExecutionCtx; use crate::array::ArrayView; use crate::array::VTable; -use crate::arrays::ScalarFnVTable; +use crate::arrays::ScalarFn; use crate::arrays::scalar_fn::ExactScalarFn; use crate::arrays::scalar_fn::ScalarFnArrayExt; use crate::arrays::scalar_fn::ScalarFnArrayView; @@ -64,7 +64,7 @@ where return Ok(None); } let scalar_fn_array = parent - .as_opt::() + .as_opt::() .vortex_expect("ExactScalarFn matcher confirmed ScalarFnArray"); let children = scalar_fn_array.children(); let lower = &children[1]; @@ -99,7 +99,7 @@ where return Ok(None); } let scalar_fn_array = parent - .as_opt::() + .as_opt::() .vortex_expect("ExactScalarFn matcher confirmed ScalarFnArray"); let children = scalar_fn_array.children(); let lower = &children[1]; diff --git a/vortex-array/src/scalar_fn/fns/between/mod.rs b/vortex-array/src/scalar_fn/fns/between/mod.rs index aaec8108808..bb44d71c846 100644 --- a/vortex-array/src/scalar_fn/fns/between/mod.rs +++ b/vortex-array/src/scalar_fn/fns/between/mod.rs @@ -8,6 +8,7 @@ use std::fmt::Formatter; pub use kernel::*; use prost::Message; +use vortex_array::expr::and; use vortex_error::VortexResult; use vortex_error::vortex_bail; use vortex_proto::expr as pb; @@ -106,16 +107,6 @@ pub(super) fn precondition( return Ok(Some(Canonical::empty(&return_dtype).into_array())); } - // A quick check to see if either bound is a null constant array. - if (lower.is_invalid(0)? || upper.is_invalid(0)?) - && let (Some(c_lower), Some(c_upper)) = (lower.as_constant(), upper.as_constant()) - && (c_lower.is_null() || c_upper.is_null()) - { - return Ok(Some( - ConstantArray::new(Scalar::null(return_dtype), arr.len()).into_array(), - )); - } - if lower.as_constant().is_some_and(|v| v.is_null()) || upper.as_constant().is_some_and(|v| v.is_null()) { @@ -164,7 +155,7 @@ fn between_canonical( upper.clone(), Operator::from(options.upper_strict.to_compare_operator()), )?; - execute_boolean(&lower_cmp, &upper_cmp, Operator::And) + execute_boolean(&lower_cmp, &upper_cmp, Operator::And, ctx) } /// An optimized scalar expression to compute whether values fall between two bounds. @@ -185,7 +176,7 @@ impl ScalarFnVTable for Between { type Options = BetweenOptions; fn id(&self) -> ScalarFnId { - ScalarFnId::from("vortex.between") + ScalarFnId::new("vortex.between") } fn serialize(&self, instance: &Self::Options) -> VortexResult>> { @@ -318,9 +309,18 @@ impl ScalarFnVTable for Between { let lhs = Binary.new_expr(options.lower_strict.to_operator(), [lower, arr.clone()]); let rhs = Binary.new_expr(options.upper_strict.to_operator(), [arr, upper]); - Binary - .new_expr(Operator::And, [lhs, rhs]) - .stat_falsification(catalog) + and(lhs, rhs).stat_falsification(catalog) + } + + fn validity( + &self, + _options: &Self::Options, + expression: &Expression, + ) -> VortexResult> { + let arr = expression.child(0).validity()?; + let lower = expression.child(1).validity()?; + let upper = expression.child(2).validity()?; + Ok(Some(and(and(arr, lower), upper))) } fn is_null_sensitive(&self, _instance: &Self::Options) -> bool { @@ -334,13 +334,13 @@ impl ScalarFnVTable for Between { #[cfg(test)] mod tests { + use std::sync::LazyLock; + use rstest::rstest; use vortex_buffer::buffer; use super::*; use crate::IntoArray; - use crate::LEGACY_SESSION; - use crate::ToCanonical; use crate::VortexSessionExecute; use crate::arrays::BoolArray; use crate::arrays::DecimalArray; @@ -355,9 +355,13 @@ mod tests { use crate::expr::root; use crate::scalar::DecimalValue; use crate::scalar::Scalar; + use crate::session::ArraySession; use crate::test_harness::to_int_indices; use crate::validity::Validity; + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); + #[test] fn test_display() { let expr = between( @@ -405,10 +409,11 @@ mod tests { lower_strict, upper_strict, }, - &mut LEGACY_SESSION.create_execution_ctx(), + &mut SESSION.create_execution_ctx(), ) .unwrap() - .to_bool(); + .execute::(&mut SESSION.create_execution_ctx()) + .unwrap(); let indices = to_int_indices(matches).unwrap(); assert_eq!(indices, expected); @@ -434,10 +439,11 @@ mod tests { lower_strict: StrictComparison::NonStrict, upper_strict: StrictComparison::NonStrict, }, - &mut LEGACY_SESSION.create_execution_ctx(), + &mut SESSION.create_execution_ctx(), ) .unwrap() - .to_bool(); + .execute::(&mut SESSION.create_execution_ctx()) + .unwrap(); let indices = to_int_indices(matches).unwrap(); assert!(indices.is_empty()); @@ -452,10 +458,11 @@ mod tests { lower_strict: StrictComparison::NonStrict, upper_strict: StrictComparison::NonStrict, }, - &mut LEGACY_SESSION.create_execution_ctx(), + &mut SESSION.create_execution_ctx(), ) .unwrap() - .to_bool(); + .execute::(&mut SESSION.create_execution_ctx()) + .unwrap(); let indices = to_int_indices(matches).unwrap(); assert_eq!(indices, vec![0, 1, 3]); @@ -470,10 +477,11 @@ mod tests { lower_strict: StrictComparison::NonStrict, upper_strict: StrictComparison::NonStrict, }, - &mut LEGACY_SESSION.create_execution_ctx(), + &mut SESSION.create_execution_ctx(), ) .unwrap() - .to_bool(); + .execute::(&mut SESSION.create_execution_ctx()) + .unwrap(); let indices = to_int_indices(matches).unwrap(); assert_eq!(indices, vec![0, 1, 2, 3, 4]); } @@ -512,7 +520,7 @@ mod tests { lower_strict: StrictComparison::Strict, upper_strict: StrictComparison::NonStrict, }, - &mut LEGACY_SESSION.create_execution_ctx(), + &mut SESSION.create_execution_ctx(), ) .unwrap(); assert_arrays_eq!( @@ -529,7 +537,7 @@ mod tests { lower_strict: StrictComparison::NonStrict, upper_strict: StrictComparison::Strict, }, - &mut LEGACY_SESSION.create_execution_ctx(), + &mut SESSION.create_execution_ctx(), ) .unwrap(); assert_arrays_eq!( diff --git a/vortex-array/src/scalar_fn/fns/binary/boolean.rs b/vortex-array/src/scalar_fn/fns/binary/boolean.rs index 57face819e8..68ef0d61156 100644 --- a/vortex-array/src/scalar_fn/fns/binary/boolean.rs +++ b/vortex-array/src/scalar_fn/fns/binary/boolean.rs @@ -10,10 +10,11 @@ use crate::ArrayRef; use crate::IntoArray; use crate::arrays::Constant; use crate::arrays::ConstantArray; +use crate::arrow::ArrowArrayExecutor; use crate::arrow::FromArrowArray; -use crate::arrow::IntoArrowArray; use crate::builtins::ArrayBuiltins; use crate::dtype::DType; +use crate::executor::ExecutionCtx; use crate::scalar::Scalar; use crate::scalar_fn::fns::operators::Operator; @@ -37,19 +38,31 @@ pub(crate) fn execute_boolean( lhs: &ArrayRef, rhs: &ArrayRef, op: Operator, + ctx: &mut ExecutionCtx, ) -> VortexResult { if let Some(result) = constant_boolean(lhs, rhs, op)? { return Ok(result); } - arrow_execute_boolean(lhs.clone(), rhs.clone(), op) + arrow_execute_boolean(lhs.clone(), rhs.clone(), op, ctx) } /// Arrow implementation for Kleene boolean operations using [`Operator`]. -fn arrow_execute_boolean(lhs: ArrayRef, rhs: ArrayRef, op: Operator) -> VortexResult { +fn arrow_execute_boolean( + lhs: ArrayRef, + rhs: ArrayRef, + op: Operator, + ctx: &mut ExecutionCtx, +) -> VortexResult { let nullable = lhs.dtype().is_nullable() || rhs.dtype().is_nullable(); - let lhs = lhs.into_arrow(&DataType::Boolean)?.as_boolean().clone(); - let rhs = rhs.into_arrow(&DataType::Boolean)?.as_boolean().clone(); + let lhs = lhs + .execute_arrow(Some(&DataType::Boolean), ctx)? + .as_boolean() + .clone(); + let rhs = rhs + .execute_arrow(Some(&DataType::Boolean), ctx)? + .as_boolean() + .clone(); let array = match op { Operator::And => arrow_arith::boolean::and_kleene(&lhs, &rhs)?, @@ -106,9 +119,12 @@ mod tests { use crate::ArrayRef; use crate::IntoArray; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::arrays::BoolArray; use crate::builtins::ArrayBuiltins; - use crate::canonical::ToCanonical; + #[expect(deprecated)] + use crate::canonical::ToCanonical as _; use crate::scalar_fn::fns::operators::Operator; #[rstest] @@ -122,12 +138,29 @@ mod tests { )] fn test_or(#[case] lhs: ArrayRef, #[case] rhs: ArrayRef) { let r = lhs.binary(rhs, Operator::Or).unwrap(); + #[expect(deprecated)] let r = r.to_bool().into_array(); - let v0 = r.scalar_at(0).unwrap().as_bool().value(); - let v1 = r.scalar_at(1).unwrap().as_bool().value(); - let v2 = r.scalar_at(2).unwrap().as_bool().value(); - let v3 = r.scalar_at(3).unwrap().as_bool().value(); + let v0 = r + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .as_bool() + .value(); + let v1 = r + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .as_bool() + .value(); + let v2 = r + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .as_bool() + .value(); + let v3 = r + .execute_scalar(3, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .as_bool() + .value(); assert!(v0.unwrap()); assert!(v1.unwrap()); @@ -145,16 +178,33 @@ mod tests { BoolArray::from_iter([Some(true), Some(true), Some(false), Some(false)]).into_array(), )] fn test_and(#[case] lhs: ArrayRef, #[case] rhs: ArrayRef) { + #[expect(deprecated)] let r = lhs .binary(rhs, Operator::And) .unwrap() .to_bool() .into_array(); - let v0 = r.scalar_at(0).unwrap().as_bool().value(); - let v1 = r.scalar_at(1).unwrap().as_bool().value(); - let v2 = r.scalar_at(2).unwrap().as_bool().value(); - let v3 = r.scalar_at(3).unwrap().as_bool().value(); + let v0 = r + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .as_bool() + .value(); + let v1 = r + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .as_bool() + .value(); + let v2 = r + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .as_bool() + .value(); + let v3 = r + .execute_scalar(3, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .as_bool() + .value(); assert!(v0.unwrap()); assert!(!v1.unwrap()); diff --git a/vortex-array/src/scalar_fn/fns/binary/compare.rs b/vortex-array/src/scalar_fn/fns/binary/compare.rs index 0108aac6509..546501fbb5e 100644 --- a/vortex-array/src/scalar_fn/fns/binary/compare.rs +++ b/vortex-array/src/scalar_fn/fns/binary/compare.rs @@ -19,12 +19,12 @@ use crate::array::ArrayView; use crate::array::VTable; use crate::arrays::Constant; use crate::arrays::ConstantArray; -use crate::arrays::ScalarFnVTable; +use crate::arrays::ScalarFn; use crate::arrays::scalar_fn::ExactScalarFn; use crate::arrays::scalar_fn::ScalarFnArrayExt; use crate::arrays::scalar_fn::ScalarFnArrayView; +use crate::arrow::ArrowArrayExecutor; use crate::arrow::Datum; -use crate::arrow::IntoArrowArray; use crate::arrow::from_arrow_array_with_len; use crate::dtype::DType; use crate::dtype::Nullability; @@ -74,7 +74,7 @@ where }; // Get the ScalarFnArray to access children - let Some(scalar_fn_array) = parent.as_opt::() else { + let Some(scalar_fn_array) = parent.as_opt::() else { return Ok(None); }; // Normalize so `array` is always LHS, swapping the operator if needed @@ -114,6 +114,7 @@ pub(crate) fn execute_compare( lhs: &ArrayRef, rhs: &ArrayRef, op: CompareOperator, + ctx: &mut ExecutionCtx, ) -> VortexResult { let nullable = lhs.dtype().is_nullable() || rhs.dtype().is_nullable(); @@ -136,7 +137,7 @@ pub(crate) fn execute_compare( return Ok(ConstantArray::new(result, lhs.len()).into_array()); } - arrow_compare_arrays(lhs, rhs, op) + arrow_compare_arrays(lhs, rhs, op, ctx) } /// Fall back to Arrow for comparison. @@ -144,6 +145,7 @@ fn arrow_compare_arrays( left: &ArrayRef, right: &ArrayRef, operator: CompareOperator, + ctx: &mut ExecutionCtx, ) -> VortexResult { assert_eq!(left.len(), right.len()); @@ -152,8 +154,8 @@ fn arrow_compare_arrays( // Arrow's vectorized comparison kernels don't support nested types. // For nested types, fall back to `make_comparator` which does element-wise comparison. let arrow_array: BooleanArray = if left.dtype().is_nested() || right.dtype().is_nested() { - let rhs = right.clone().into_arrow_preferred()?; - let lhs = left.clone().into_arrow(rhs.data_type())?; + let rhs = right.clone().execute_arrow(None, ctx)?; + let lhs = left.clone().execute_arrow(Some(rhs.data_type()), ctx)?; assert!( lhs.data_type().equals_datatype(rhs.data_type()), @@ -165,8 +167,8 @@ fn arrow_compare_arrays( compare_nested_arrow_arrays(lhs.as_ref(), rhs.as_ref(), operator)? } else { // Fast path: use vectorized kernels for primitive types. - let lhs = Datum::try_new(left)?; - let rhs = Datum::try_new_with_target_datatype(right, lhs.data_type())?; + let lhs = Datum::try_new(left, ctx)?; + let rhs = Datum::try_new_with_target_datatype(right, lhs.data_type(), ctx)?; match operator { CompareOperator::Eq => cmp::eq(&lhs, &rhs)?, @@ -249,7 +251,10 @@ mod tests { use crate::ArrayRef; use crate::IntoArray; - use crate::ToCanonical; + use crate::LEGACY_SESSION; + #[expect(deprecated)] + use crate::ToCanonical as _; + use crate::VortexSessionExecute; use crate::arrays::BoolArray; use crate::arrays::ListArray; use crate::arrays::ListViewArray; @@ -284,6 +289,7 @@ mod tests { Validity::from_iter([false, true, true, true, true]), ); + #[expect(deprecated)] let matches = arr .clone() .into_array() @@ -292,6 +298,7 @@ mod tests { .to_bool(); assert_eq!(to_int_indices(matches).unwrap(), [1u64, 2, 3, 4]); + #[expect(deprecated)] let matches = arr .clone() .into_array() @@ -306,6 +313,7 @@ mod tests { Validity::from_iter([false, true, true, true, true]), ); + #[expect(deprecated)] let matches = arr .clone() .into_array() @@ -314,6 +322,7 @@ mod tests { .to_bool(); assert_eq!(to_int_indices(matches).unwrap(), [2u64, 3, 4]); + #[expect(deprecated)] let matches = arr .clone() .into_array() @@ -322,6 +331,7 @@ mod tests { .to_bool(); assert_eq!(to_int_indices(matches).unwrap(), [4u64]); + #[expect(deprecated)] let matches = other .clone() .into_array() @@ -330,6 +340,7 @@ mod tests { .to_bool(); assert_eq!(to_int_indices(matches).unwrap(), [2u64, 3, 4]); + #[expect(deprecated)] let matches = other .into_array() .binary(arr.into_array(), Operator::Gt) @@ -348,7 +359,9 @@ mod tests { .binary(right.into_array(), Operator::Gt) .unwrap(); assert_eq!(result.len(), 10); - let scalar = result.scalar_at(0).unwrap(); + let scalar = result + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); assert_eq!(scalar.as_bool().value(), Some(false)); } @@ -540,8 +553,23 @@ mod tests { .into_array() .binary(list.into_array(), Operator::Eq) .unwrap(); - assert!(result.scalar_at(0).unwrap().is_valid()); - assert!(result.scalar_at(1).unwrap().is_valid()); - assert!(result.scalar_at(2).unwrap().is_valid()); + assert!( + result + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_valid() + ); + assert!( + result + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_valid() + ); + assert!( + result + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .is_valid() + ); } } diff --git a/vortex-array/src/scalar_fn/fns/binary/mod.rs b/vortex-array/src/scalar_fn/fns/binary/mod.rs index 83385268397..374d34d1ec9 100644 --- a/vortex-array/src/scalar_fn/fns/binary/mod.rs +++ b/vortex-array/src/scalar_fn/fns/binary/mod.rs @@ -52,7 +52,7 @@ impl ScalarFnVTable for Binary { type Options = Operator; fn id(&self) -> ScalarFnId { - ScalarFnId::from("vortex.binary") + ScalarFnId::new("vortex.binary") } fn serialize(&self, instance: &Self::Options) -> VortexResult>> { @@ -142,24 +142,24 @@ impl ScalarFnVTable for Binary { &self, op: &Operator, args: &dyn ExecutionArgs, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { let lhs = args.get(0)?; let rhs = args.get(1)?; match op { - Operator::Eq => execute_compare(&lhs, &rhs, CompareOperator::Eq), - Operator::NotEq => execute_compare(&lhs, &rhs, CompareOperator::NotEq), - Operator::Lt => execute_compare(&lhs, &rhs, CompareOperator::Lt), - Operator::Lte => execute_compare(&lhs, &rhs, CompareOperator::Lte), - Operator::Gt => execute_compare(&lhs, &rhs, CompareOperator::Gt), - Operator::Gte => execute_compare(&lhs, &rhs, CompareOperator::Gte), - Operator::And => execute_boolean(&lhs, &rhs, Operator::And), - Operator::Or => execute_boolean(&lhs, &rhs, Operator::Or), - Operator::Add => execute_numeric(&lhs, &rhs, NumericOperator::Add), - Operator::Sub => execute_numeric(&lhs, &rhs, NumericOperator::Sub), - Operator::Mul => execute_numeric(&lhs, &rhs, NumericOperator::Mul), - Operator::Div => execute_numeric(&lhs, &rhs, NumericOperator::Div), + Operator::Eq => execute_compare(&lhs, &rhs, CompareOperator::Eq, ctx), + Operator::NotEq => execute_compare(&lhs, &rhs, CompareOperator::NotEq, ctx), + Operator::Lt => execute_compare(&lhs, &rhs, CompareOperator::Lt, ctx), + Operator::Lte => execute_compare(&lhs, &rhs, CompareOperator::Lte, ctx), + Operator::Gt => execute_compare(&lhs, &rhs, CompareOperator::Gt, ctx), + Operator::Gte => execute_compare(&lhs, &rhs, CompareOperator::Gte, ctx), + Operator::And => execute_boolean(&lhs, &rhs, Operator::And, ctx), + Operator::Or => execute_boolean(&lhs, &rhs, Operator::Or, ctx), + Operator::Add => execute_numeric(&lhs, &rhs, NumericOperator::Add, ctx), + Operator::Sub => execute_numeric(&lhs, &rhs, NumericOperator::Sub, ctx), + Operator::Mul => execute_numeric(&lhs, &rhs, NumericOperator::Mul, ctx), + Operator::Div => execute_numeric(&lhs, &rhs, NumericOperator::Div, ctx), } } @@ -314,6 +314,8 @@ mod tests { use vortex_error::VortexExpect; use super::*; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::assert_arrays_eq; use crate::builtins::ArrayBuiltins; use crate::dtype::DType; @@ -494,7 +496,9 @@ mod tests { // Test using binary method directly let result_equal = lhs_struct.binary(rhs_struct_equal, Operator::Eq).unwrap(); assert_eq!( - result_equal.scalar_at(0).vortex_expect("value"), + result_equal + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .vortex_expect("value"), Scalar::bool(true, Nullability::NonNullable), "Equal structs should be equal" ); @@ -503,7 +507,9 @@ mod tests { .binary(rhs_struct_different, Operator::Eq) .unwrap(); assert_eq!( - result_different.scalar_at(0).vortex_expect("value"), + result_different + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .vortex_expect("value"), Scalar::bool(false, Nullability::NonNullable), "Different structs should not be equal" ); diff --git a/vortex-array/src/scalar_fn/fns/binary/numeric.rs b/vortex-array/src/scalar_fn/fns/binary/numeric.rs index 360c65ef6a3..9429a7310d1 100644 --- a/vortex-array/src/scalar_fn/fns/binary/numeric.rs +++ b/vortex-array/src/scalar_fn/fns/binary/numeric.rs @@ -9,6 +9,7 @@ use crate::arrays::Constant; use crate::arrays::ConstantArray; use crate::arrow::Datum; use crate::arrow::from_arrow_array_with_len; +use crate::executor::ExecutionCtx; use crate::scalar::NumericOperator; /// Execute a numeric operation between two arrays. @@ -19,11 +20,12 @@ pub(crate) fn execute_numeric( lhs: &ArrayRef, rhs: &ArrayRef, op: NumericOperator, + ctx: &mut ExecutionCtx, ) -> VortexResult { if let Some(result) = constant_numeric(lhs, rhs, op)? { return Ok(result); } - arrow_numeric(lhs, rhs, op) + arrow_numeric(lhs, rhs, op, ctx) } /// Implementation of numeric operations using the Arrow crate. @@ -31,12 +33,13 @@ pub(crate) fn arrow_numeric( lhs: &ArrayRef, rhs: &ArrayRef, operator: NumericOperator, + ctx: &mut ExecutionCtx, ) -> VortexResult { let nullable = lhs.dtype().is_nullable() || rhs.dtype().is_nullable(); let len = lhs.len(); - let left = Datum::try_new(lhs)?; - let right = Datum::try_new_with_target_datatype(rhs, left.data_type())?; + let left = Datum::try_new(lhs, ctx)?; + let right = Datum::try_new_with_target_datatype(rhs, left.data_type(), ctx)?; let array = match operator { NumericOperator::Add => arrow_arith::numeric::add(&left, &right)?, @@ -70,7 +73,6 @@ fn constant_numeric( } #[cfg(test)] -#[allow(deprecated)] mod test { use vortex_buffer::buffer; use vortex_error::VortexResult; diff --git a/vortex-array/src/scalar_fn/fns/case_when.rs b/vortex-array/src/scalar_fn/fns/case_when.rs index f30a17ee580..577e3eb3d2f 100644 --- a/vortex-array/src/scalar_fn/fns/case_when.rs +++ b/vortex-array/src/scalar_fn/fns/case_when.rs @@ -79,7 +79,7 @@ impl ScalarFnVTable for CaseWhen { type Options = CaseWhenOptions; fn id(&self) -> ScalarFnId { - ScalarFnId::from("vortex.case_when") + ScalarFnId::new("vortex.case_when") } fn serialize(&self, _options: &Self::Options) -> VortexResult>> { @@ -223,7 +223,7 @@ impl ScalarFnVTable for CaseWhen { let condition = args.get(i * 2)?; let cond_bool = condition.execute::(ctx)?; - let cond_mask = cond_bool.to_mask_fill_null_false(); + let cond_mask = cond_bool.to_mask_fill_null_false(ctx); let effective_mask = &remaining & &cond_mask; if effective_mask.all_false() { @@ -246,7 +246,7 @@ impl ScalarFnVTable for CaseWhen { return Ok(else_value); } - merge_case_branches(branches, else_value) + merge_case_branches(branches, else_value, ctx) } fn is_null_sensitive(&self, _options: &Self::Options) -> bool { @@ -268,6 +268,7 @@ const SLICE_CROSSOVER_RUN_LEN: usize = 4; fn merge_case_branches( branches: Vec<(Mask, ArrayRef)>, else_value: ArrayRef, + ctx: &mut ExecutionCtx, ) -> VortexResult { if branches.len() == 1 { let (mask, then_value) = &branches[0]; @@ -304,7 +305,14 @@ fn merge_case_branches( let fragmented = spans.len() > else_value.len() / SLICE_CROSSOVER_RUN_LEN; if fragmented { - merge_row_by_row(&branch_arrays, &else_value, &spans, &output_dtype, builder) + merge_row_by_row( + &branch_arrays, + &else_value, + &spans, + &output_dtype, + builder, + ctx, + ) } else { merge_run_by_run(&branch_arrays, &else_value, &spans, &output_dtype, builder) } @@ -318,21 +326,22 @@ fn merge_row_by_row( spans: &[(usize, usize, usize)], output_dtype: &DType, mut builder: Box, + ctx: &mut ExecutionCtx, ) -> VortexResult { let mut pos = 0; for &(start, end, branch_idx) in spans { for row in pos..start { - let scalar = else_value.scalar_at(row)?; + let scalar = else_value.execute_scalar(row, ctx)?; builder.append_scalar(&scalar.cast(output_dtype)?)?; } for row in start..end { - let scalar = branch_arrays[branch_idx].scalar_at(row)?; + let scalar = branch_arrays[branch_idx].execute_scalar(row, ctx)?; builder.append_scalar(&scalar.cast(output_dtype)?)?; } pos = end; } for row in pos..else_value.len() { - let scalar = else_value.scalar_at(row)?; + let scalar = else_value.execute_scalar(row, ctx)?; builder.append_scalar(&scalar.cast(output_dtype)?)?; } @@ -379,7 +388,8 @@ mod tests { use super::*; use crate::Canonical; use crate::IntoArray; - use crate::VortexSessionExecute as _; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::arrays::BoolArray; use crate::arrays::PrimitiveArray; use crate::arrays::StructArray; @@ -848,9 +858,8 @@ mod tests { // WHEN value = 1 THEN nullable(20) -- Nullable // ELSE 0 -- NonNullable // → result must be Nullable(i32) - let test_array = StructArray::from_fields(&[("value", buffer![0i32, 1, 2].into_array())]) - .unwrap() - .into_array(); + let test_array = + StructArray::from_fields(&[("value", buffer![0i32, 1, 2].into_array())])?.into_array(); let nullable_20 = Scalar::from(20i32).cast(&DType::Primitive(PType::I32, Nullability::Nullable))?; @@ -1107,8 +1116,7 @@ mod tests { fn test_evaluate_nary_string_output() -> VortexResult<()> { // Exercises merge_case_branches with a non-primitive (Utf8) builder. let test_array = - StructArray::from_fields(&[("value", buffer![1i32, 2, 3, 4].into_array())]) - .unwrap() + StructArray::from_fields(&[("value", buffer![1i32, 2, 3, 4].into_array())])? .into_array(); // CASE WHEN value > 2 THEN 'high' WHEN value > 0 THEN 'low' ELSE 'none' END @@ -1124,19 +1132,19 @@ mod tests { let result = evaluate_expr(&expr, &test_array); assert_eq!( - result.scalar_at(0)?, + result.execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx())?, Scalar::utf8("low", Nullability::NonNullable) ); assert_eq!( - result.scalar_at(1)?, + result.execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx())?, Scalar::utf8("low", Nullability::NonNullable) ); assert_eq!( - result.scalar_at(2)?, + result.execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx())?, Scalar::utf8("high", Nullability::NonNullable) ); assert_eq!( - result.scalar_at(3)?, + result.execute_scalar(3, &mut LEGACY_SESSION.create_execution_ctx())?, Scalar::utf8("high", Nullability::NonNullable) ); Ok(()) @@ -1194,6 +1202,7 @@ mod tests { ), ], PrimitiveArray::from_option_iter(vec![Some(99i32); n]).into_array(), + &mut SESSION.create_execution_ctx(), )?; // Even rows → 0, odd rows → 1. diff --git a/vortex-array/src/scalar_fn/fns/cast/mod.rs b/vortex-array/src/scalar_fn/fns/cast/mod.rs index 66a257d340b..cbab9615e97 100644 --- a/vortex-array/src/scalar_fn/fns/cast/mod.rs +++ b/vortex-array/src/scalar_fn/fns/cast/mod.rs @@ -53,7 +53,7 @@ impl ScalarFnVTable for Cast { type Options = DType; fn id(&self) -> ScalarFnId { - ScalarFnId::from("vortex.cast") + ScalarFnId::new("vortex.cast") } fn serialize(&self, dtype: &DType) -> VortexResult>> { @@ -198,7 +198,13 @@ impl ScalarFnVTable for Cast { } /// Cast a canonical array to the target dtype by dispatching to the appropriate -/// [`CastReduce`] or [`CastKernel`] for each canonical encoding. +/// [`CastKernel`] for each canonical encoding. +/// +/// Canonical encodings that can manipulate validity directly all implement [`CastKernel`] — +/// the kernel is the execution-time complement of their [`CastReduce`] rule and can compute +/// statistics (e.g. min of the validity array) when the reduce rule had to give up. +/// Encodings that delegate to scalars or storage (e.g. [`Null`], [`Constant`], [`Extension`]) +/// only implement [`CastReduce`] because they never need execution-level information. fn cast_canonical( canonical: CanonicalView<'_>, dtype: &DType, @@ -206,12 +212,12 @@ fn cast_canonical( ) -> VortexResult> { match canonical { CanonicalView::Null(a) => ::cast(a, dtype), - CanonicalView::Bool(a) => ::cast(a, dtype), + CanonicalView::Bool(a) => ::cast(a, dtype, ctx), CanonicalView::Primitive(a) => ::cast(a, dtype, ctx), CanonicalView::Decimal(a) => ::cast(a, dtype, ctx), - CanonicalView::VarBinView(a) => ::cast(a, dtype), - CanonicalView::List(a) => ::cast(a, dtype), - CanonicalView::FixedSizeList(a) => ::cast(a, dtype), + CanonicalView::VarBinView(a) => ::cast(a, dtype, ctx), + CanonicalView::List(a) => ::cast(a, dtype, ctx), + CanonicalView::FixedSizeList(a) => ::cast(a, dtype, ctx), CanonicalView::Struct(a) => ::cast(a, dtype, ctx), CanonicalView::Extension(a) => ::cast(a, dtype), CanonicalView::Variant(_) => { diff --git a/vortex-array/src/scalar_fn/fns/fill_null/kernel.rs b/vortex-array/src/scalar_fn/fns/fill_null/kernel.rs index dd2a6c9b16b..eea3dd6ef7b 100644 --- a/vortex-array/src/scalar_fn/fns/fill_null/kernel.rs +++ b/vortex-array/src/scalar_fn/fns/fill_null/kernel.rs @@ -12,7 +12,7 @@ use crate::array::ArrayView; use crate::array::VTable; use crate::arrays::Constant; use crate::arrays::ConstantArray; -use crate::arrays::ScalarFnVTable; +use crate::arrays::ScalarFn; use crate::arrays::scalar_fn::ExactScalarFn; use crate::arrays::scalar_fn::ScalarFnArrayExt; use crate::arrays::scalar_fn::ScalarFnArrayView; @@ -21,6 +21,7 @@ use crate::kernel::ExecuteParentKernel; use crate::optimizer::rules::ArrayParentReduceRule; use crate::scalar::Scalar; use crate::scalar_fn::fns::fill_null::FillNull as FillNullExpr; +use crate::validity::Validity; /// Fill nulls in an array with a scalar value without reading buffers. /// @@ -68,12 +69,17 @@ pub(super) fn precondition( ); // If the array has no nulls, fill_null is a no-op (just cast for nullability). - if !array.dtype().is_nullable() || array.all_valid()? { + if !array.dtype().is_nullable() + || matches!( + array.validity()?, + Validity::NonNullable | Validity::AllValid + ) + { return array.clone().cast(fill_value.dtype().clone()).map(Some); } // If all values are null, replace the entire array with the fill value. - if array.all_invalid()? { + if matches!(array.validity()?, Validity::AllInvalid) { return Ok(Some( ConstantArray::new(fill_value.clone(), array.len()).into_array(), )); @@ -117,7 +123,7 @@ where return Ok(None); } let scalar_fn_array = parent - .as_opt::() + .as_opt::() .vortex_expect("ExactScalarFn matcher confirmed ScalarFnArray"); let fill_value = scalar_fn_array .get_child(1) @@ -153,7 +159,7 @@ where return Ok(None); } let scalar_fn_array = parent - .as_opt::() + .as_opt::() .vortex_expect("ExactScalarFn matcher confirmed ScalarFnArray"); let fill_value = scalar_fn_array .get_child(1) diff --git a/vortex-array/src/scalar_fn/fns/fill_null/mod.rs b/vortex-array/src/scalar_fn/fns/fill_null/mod.rs index 656029bf82d..16557cd77db 100644 --- a/vortex-array/src/scalar_fn/fns/fill_null/mod.rs +++ b/vortex-array/src/scalar_fn/fns/fill_null/mod.rs @@ -39,7 +39,7 @@ impl ScalarFnVTable for FillNull { type Options = EmptyOptions; fn id(&self) -> ScalarFnId { - ScalarFnId::from("vortex.fill_null") + ScalarFnId::new("vortex.fill_null") } fn serialize(&self, _options: &Self::Options) -> VortexResult>> { diff --git a/vortex-array/src/scalar_fn/fns/get_item.rs b/vortex-array/src/scalar_fn/fns/get_item.rs index 353ebc0925b..0b562420945 100644 --- a/vortex-array/src/scalar_fn/fns/get_item.rs +++ b/vortex-array/src/scalar_fn/fns/get_item.rs @@ -44,7 +44,7 @@ impl ScalarFnVTable for GetItem { type Options = FieldName; fn id(&self) -> ScalarFnId { - ScalarFnId::from("vortex.get_item") + ScalarFnId::new("vortex.get_item") } fn serialize(&self, instance: &Self::Options) -> VortexResult>> { diff --git a/vortex-array/src/scalar_fn/fns/is_not_null.rs b/vortex-array/src/scalar_fn/fns/is_not_null.rs new file mode 100644 index 00000000000..26e474cc447 --- /dev/null +++ b/vortex-array/src/scalar_fn/fns/is_not_null.rs @@ -0,0 +1,284 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +use std::fmt::Formatter; + +use vortex_array::scalar_fn::internal::row_count::RowCount; +use vortex_error::VortexResult; +use vortex_session::VortexSession; + +use crate::ArrayRef; +use crate::ExecutionCtx; +use crate::IntoArray; +use crate::arrays::ConstantArray; +use crate::dtype::DType; +use crate::dtype::Nullability; +use crate::expr::Expression; +use crate::expr::StatsCatalog; +use crate::expr::eq; +use crate::expr::stats::Stat; +use crate::scalar_fn::Arity; +use crate::scalar_fn::ChildName; +use crate::scalar_fn::EmptyOptions; +use crate::scalar_fn::ExecutionArgs; +use crate::scalar_fn::ScalarFnId; +use crate::scalar_fn::ScalarFnVTable; +use crate::scalar_fn::ScalarFnVTableExt; +use crate::validity::Validity; + +/// Expression that checks for non-null values. +#[derive(Clone)] +pub struct IsNotNull; + +impl ScalarFnVTable for IsNotNull { + type Options = EmptyOptions; + + fn id(&self) -> ScalarFnId { + ScalarFnId::new("vortex.is_not_null") + } + + fn serialize(&self, _instance: &Self::Options) -> VortexResult>> { + Ok(Some(vec![])) + } + + fn deserialize( + &self, + _metadata: &[u8], + _session: &VortexSession, + ) -> VortexResult { + Ok(EmptyOptions) + } + + fn arity(&self, _options: &Self::Options) -> Arity { + Arity::Exact(1) + } + + fn child_name(&self, _instance: &Self::Options, child_idx: usize) -> ChildName { + match child_idx { + 0 => ChildName::from("input"), + _ => unreachable!("Invalid child index {} for IsNotNull expression", child_idx), + } + } + + fn fmt_sql( + &self, + _options: &Self::Options, + expr: &Expression, + f: &mut Formatter<'_>, + ) -> std::fmt::Result { + write!(f, "is_not_null(")?; + expr.child(0).fmt_sql(f)?; + write!(f, ")") + } + + fn return_dtype(&self, _options: &Self::Options, _arg_dtypes: &[DType]) -> VortexResult { + Ok(DType::Bool(Nullability::NonNullable)) + } + + fn execute( + &self, + _data: &Self::Options, + args: &dyn ExecutionArgs, + _ctx: &mut ExecutionCtx, + ) -> VortexResult { + let child = args.get(0)?; + match child.validity()? { + Validity::NonNullable | Validity::AllValid => { + Ok(ConstantArray::new(true, args.row_count()).into_array()) + } + Validity::AllInvalid => Ok(ConstantArray::new(false, args.row_count()).into_array()), + Validity::Array(a) => Ok(a), + } + } + + fn is_null_sensitive(&self, _instance: &Self::Options) -> bool { + true + } + + fn is_fallible(&self, _instance: &Self::Options) -> bool { + false + } + + fn stat_falsification( + &self, + _options: &Self::Options, + expr: &Expression, + catalog: &dyn StatsCatalog, + ) -> Option { + // is_not_null is falsified when ALL values are null, i.e. null_count == row_count. + let child = expr.child(0); + let null_count_expr = child.stat_expression(Stat::NullCount, catalog)?; + Some(eq(null_count_expr, RowCount.new_expr(EmptyOptions, []))) + } +} + +#[cfg(test)] +mod tests { + use vortex_buffer::buffer; + use vortex_error::VortexExpect as _; + use vortex_utils::aliases::hash_map::HashMap; + use vortex_utils::aliases::hash_set::HashSet; + + use crate::IntoArray; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; + use crate::arrays::PrimitiveArray; + use crate::arrays::StructArray; + use crate::dtype::DType; + use crate::dtype::Field; + use crate::dtype::FieldPath; + use crate::dtype::FieldPathSet; + use crate::dtype::Nullability; + use crate::expr::col; + use crate::expr::eq; + use crate::expr::get_item; + use crate::expr::is_not_null; + use crate::expr::pruning::checked_pruning_expr; + use crate::expr::root; + use crate::expr::stats::Stat; + use crate::expr::test_harness; + use crate::scalar::Scalar; + use crate::scalar_fn::EmptyOptions; + use crate::scalar_fn::internal::row_count::RowCount; + use crate::scalar_fn::vtable::ScalarFnVTableExt; + + #[test] + fn dtype() { + let dtype = test_harness::struct_dtype(); + assert_eq!( + is_not_null(root()).return_dtype(&dtype).unwrap(), + DType::Bool(Nullability::NonNullable) + ); + } + + #[test] + fn replace_children() { + let expr = is_not_null(root()); + expr.with_children([root()]) + .vortex_expect("operation should succeed in test"); + } + + #[test] + fn evaluate_mask() { + let test_array = + PrimitiveArray::from_option_iter(vec![Some(1), None, Some(2), None, Some(3)]) + .into_array(); + let expected = [true, false, true, false, true]; + + let result = test_array.clone().apply(&is_not_null(root())).unwrap(); + + assert_eq!(result.len(), test_array.len()); + assert_eq!(result.dtype(), &DType::Bool(Nullability::NonNullable)); + + for (i, expected_value) in expected.iter().enumerate() { + assert_eq!( + result + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + Scalar::bool(*expected_value, Nullability::NonNullable) + ); + } + } + + #[test] + fn evaluate_all_true() { + let test_array = buffer![1, 2, 3, 4, 5].into_array(); + + let result = test_array.clone().apply(&is_not_null(root())).unwrap(); + + assert_eq!(result.len(), test_array.len()); + for i in 0..result.len() { + assert_eq!( + result + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + Scalar::bool(true, Nullability::NonNullable) + ); + } + } + + #[test] + fn evaluate_all_false() { + let test_array = + PrimitiveArray::from_option_iter(vec![None::, None, None, None, None]) + .into_array(); + + let result = test_array.clone().apply(&is_not_null(root())).unwrap(); + + assert_eq!(result.len(), test_array.len()); + for i in 0..result.len() { + assert_eq!( + result + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + Scalar::bool(false, Nullability::NonNullable) + ); + } + } + + #[test] + fn evaluate_struct() { + let test_array = StructArray::from_fields(&[( + "a", + PrimitiveArray::from_option_iter(vec![Some(1), None, Some(2), None, Some(3)]) + .into_array(), + )]) + .unwrap() + .into_array(); + let expected = [true, false, true, false, true]; + + let result = test_array + .clone() + .apply(&is_not_null(get_item("a", root()))) + .unwrap(); + + assert_eq!(result.len(), test_array.len()); + assert_eq!(result.dtype(), &DType::Bool(Nullability::NonNullable)); + + for (i, expected_value) in expected.iter().enumerate() { + assert_eq!( + result + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), + Scalar::bool(*expected_value, Nullability::NonNullable) + ); + } + } + + #[test] + fn test_display() { + let expr = is_not_null(get_item("name", root())); + assert_eq!(expr.to_string(), "is_not_null($.name)"); + + let expr2 = is_not_null(root()); + assert_eq!(expr2.to_string(), "is_not_null($)"); + } + + #[test] + fn test_is_not_null_sensitive() { + assert!(is_not_null(col("a")).signature().is_null_sensitive()); + } + + #[test] + fn test_is_not_null_falsification() { + let expr = is_not_null(col("a")); + + let (pruning_expr, st) = checked_pruning_expr( + &expr, + &FieldPathSet::from_iter([FieldPath::from_iter([ + Field::Name("a".into()), + Field::Name("null_count".into()), + ])]), + ) + .unwrap(); + + assert_eq!( + &pruning_expr, + &eq(col("a_null_count"), RowCount.new_expr(EmptyOptions, [])) + ); + assert_eq!( + st.map(), + &HashMap::from_iter([(FieldPath::from_name("a"), HashSet::from([Stat::NullCount]))]) + ); + } +} diff --git a/vortex-array/src/scalar_fn/fns/is_null.rs b/vortex-array/src/scalar_fn/fns/is_null.rs index dd3ef3366c6..2d801cd6a0d 100644 --- a/vortex-array/src/scalar_fn/fns/is_null.rs +++ b/vortex-array/src/scalar_fn/fns/is_null.rs @@ -122,6 +122,8 @@ mod tests { use vortex_utils::aliases::hash_set::HashSet; use crate::IntoArray; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::arrays::PrimitiveArray; use crate::arrays::StructArray; use crate::dtype::DType; @@ -170,7 +172,9 @@ mod tests { for (i, expected_value) in expected.iter().enumerate() { assert_eq!( - result.scalar_at(i).unwrap(), + result + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::bool(*expected_value, Nullability::NonNullable) ); } @@ -186,7 +190,9 @@ mod tests { // All values should be false (non-nullable input) for i in 0..result.len() { assert_eq!( - result.scalar_at(i).unwrap(), + result + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::bool(false, Nullability::NonNullable) ); } @@ -204,7 +210,9 @@ mod tests { // All values should be true (all nulls) for i in 0..result.len() { assert_eq!( - result.scalar_at(i).unwrap(), + result + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::bool(true, Nullability::NonNullable) ); } @@ -231,7 +239,9 @@ mod tests { for (i, expected_value) in expected.iter().enumerate() { assert_eq!( - result.scalar_at(i).unwrap(), + result + .execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::bool(*expected_value, Nullability::NonNullable) ); } diff --git a/vortex-array/src/scalar_fn/fns/like/kernel.rs b/vortex-array/src/scalar_fn/fns/like/kernel.rs index 0489504cc93..b3b683212ff 100644 --- a/vortex-array/src/scalar_fn/fns/like/kernel.rs +++ b/vortex-array/src/scalar_fn/fns/like/kernel.rs @@ -8,7 +8,7 @@ use crate::ArrayRef; use crate::ExecutionCtx; use crate::array::ArrayView; use crate::array::VTable; -use crate::arrays::ScalarFnVTable; +use crate::arrays::ScalarFn; use crate::arrays::scalar_fn::ExactScalarFn; use crate::arrays::scalar_fn::ScalarFnArrayExt; use crate::arrays::scalar_fn::ScalarFnArrayView; @@ -69,7 +69,7 @@ where return Ok(None); } let scalar_fn_array = parent - .as_opt::() + .as_opt::() .vortex_expect("ExactScalarFn matcher confirmed ScalarFnArray"); let pattern = scalar_fn_array.get_child(1); let options = *parent.options; @@ -98,7 +98,7 @@ where return Ok(None); } let scalar_fn_array = parent - .as_opt::() + .as_opt::() .vortex_expect("ExactScalarFn matcher confirmed ScalarFnArray"); let pattern = scalar_fn_array.get_child(1); let options = *parent.options; diff --git a/vortex-array/src/scalar_fn/fns/like/mod.rs b/vortex-array/src/scalar_fn/fns/like/mod.rs index da52f730061..2e908ca17bd 100644 --- a/vortex-array/src/scalar_fn/fns/like/mod.rs +++ b/vortex-array/src/scalar_fn/fns/like/mod.rs @@ -62,7 +62,7 @@ impl ScalarFnVTable for Like { type Options = LikeOptions; fn id(&self) -> ScalarFnId { - ScalarFnId::from("vortex.like") + ScalarFnId::new("vortex.like") } fn serialize(&self, instance: &Self::Options) -> VortexResult>> { @@ -140,12 +140,12 @@ impl ScalarFnVTable for Like { &self, options: &Self::Options, args: &dyn ExecutionArgs, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { let child = args.get(0)?; let pattern = args.get(1)?; - arrow_like(&child, &pattern, *options) + arrow_like(&child, &pattern, *options, ctx) } fn validity( @@ -206,6 +206,7 @@ pub(crate) fn arrow_like( array: &ArrayRef, pattern: &ArrayRef, options: LikeOptions, + ctx: &mut ExecutionCtx, ) -> VortexResult { let nullable = array.dtype().is_nullable() | pattern.dtype().is_nullable(); let len = array.len(); @@ -217,8 +218,8 @@ pub(crate) fn arrow_like( ); // convert the pattern to the preferred array datatype - let lhs = Datum::try_new(array)?; - let rhs = Datum::try_new_with_target_datatype(pattern, lhs.data_type())?; + let lhs = Datum::try_new(array, ctx)?; + let rhs = Datum::try_new_with_target_datatype(pattern, lhs.data_type(), ctx)?; let result = match (options.negated, options.case_insensitive) { (false, false) => arrow_string::like::like(&lhs, &rhs)?, diff --git a/vortex-array/src/scalar_fn/fns/list_contains/kernel.rs b/vortex-array/src/scalar_fn/fns/list_contains/kernel.rs index 7bc643606a7..563600bfeee 100644 --- a/vortex-array/src/scalar_fn/fns/list_contains/kernel.rs +++ b/vortex-array/src/scalar_fn/fns/list_contains/kernel.rs @@ -8,7 +8,7 @@ use crate::ArrayRef; use crate::ExecutionCtx; use crate::array::ArrayView; use crate::array::VTable; -use crate::arrays::ScalarFnVTable; +use crate::arrays::ScalarFn; use crate::arrays::scalar_fn::ExactScalarFn; use crate::arrays::scalar_fn::ScalarFnArrayExt; use crate::arrays::scalar_fn::ScalarFnArrayView; @@ -67,7 +67,7 @@ where return Ok(None); } let scalar_fn_array = parent - .as_opt::() + .as_opt::() .vortex_expect("ExactScalarFn matcher confirmed ScalarFnArray"); let list = scalar_fn_array.get_child(0); ::list_contains(list, array) @@ -96,7 +96,7 @@ where return Ok(None); } let scalar_fn_array = parent - .as_opt::() + .as_opt::() .vortex_expect("ExactScalarFn matcher confirmed ScalarFnArray"); let list = scalar_fn_array.get_child(0); ::list_contains(list, array, ctx) diff --git a/vortex-array/src/scalar_fn/fns/list_contains/mod.rs b/vortex-array/src/scalar_fn/fns/list_contains/mod.rs index b4d6c3e6bc4..c1b8687d0b7 100644 --- a/vortex-array/src/scalar_fn/fns/list_contains/mod.rs +++ b/vortex-array/src/scalar_fn/fns/list_contains/mod.rs @@ -60,7 +60,7 @@ impl ScalarFnVTable for ListContains { type Options = EmptyOptions; fn id(&self) -> ScalarFnId { - ScalarFnId::from("vortex.list.contains") + ScalarFnId::new("vortex.list.contains") } fn serialize(&self, _instance: &Self::Options) -> VortexResult>> { @@ -219,7 +219,7 @@ fn compute_list_contains( ); } - if value.all_invalid()? || array.all_invalid()? { + if value.all_invalid(ctx)? || array.all_invalid(ctx)? { return Ok(ConstantArray::new( Scalar::null(DType::Bool(Nullability::Nullable)), array.len(), @@ -280,7 +280,7 @@ fn list_contains_scalar( // If the list array is constant, we perform a single comparison. if array.len() > 1 && array.is::() { let contains = list_contains_scalar(&array.slice(0..1)?, value, nullability, ctx)?; - return Ok(ConstantArray::new(contains.scalar_at(0)?, array.len()).into_array()); + return Ok(ConstantArray::new(contains.execute_scalar(0, ctx)?, array.len()).into_array()); } let list_array = array.clone().execute::(ctx)?; @@ -456,10 +456,13 @@ mod tests { use crate::ArrayRef; use crate::IntoArray; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::arrays::ListArray; use crate::arrays::VarBinArray; use crate::assert_arrays_eq; - use crate::canonical::ToCanonical; + #[expect(deprecated)] + use crate::canonical::ToCanonical as _; use crate::dtype::DType; use crate::dtype::Field; use crate::dtype::FieldPath; @@ -480,7 +483,6 @@ mod tests { use crate::expr::stats::Stat; use crate::scalar::Scalar; use crate::scalar_fn::fns::list_contains::BoolArray; - use crate::scalar_fn::fns::list_contains::Constant; use crate::scalar_fn::fns::list_contains::ConstantArray; use crate::scalar_fn::fns::list_contains::ListViewArray; use crate::scalar_fn::fns::list_contains::PrimitiveArray; @@ -504,11 +506,13 @@ mod tests { let item = arr.apply(&expr).unwrap(); assert_eq!( - item.scalar_at(0).unwrap(), + item.execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::bool(true, Nullability::Nullable) ); assert_eq!( - item.scalar_at(1).unwrap(), + item.execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::bool(false, Nullability::Nullable) ); } @@ -521,11 +525,13 @@ mod tests { let item = arr.apply(&expr).unwrap(); assert_eq!( - item.scalar_at(0).unwrap(), + item.execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::bool(true, Nullability::Nullable) ); assert_eq!( - item.scalar_at(1).unwrap(), + item.execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::bool(true, Nullability::Nullable) ); } @@ -538,11 +544,13 @@ mod tests { let item = arr.apply(&expr).unwrap(); assert_eq!( - item.scalar_at(0).unwrap(), + item.execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::bool(false, Nullability::Nullable) ); assert_eq!( - item.scalar_at(1).unwrap(), + item.execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::bool(false, Nullability::Nullable) ); } @@ -561,11 +569,13 @@ mod tests { let item = arr.apply(&expr).unwrap(); assert_eq!( - item.scalar_at(0).unwrap(), + item.execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::bool(true, Nullability::Nullable) ); assert_eq!( - item.scalar_at(1).unwrap(), + item.execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::bool(false, Nullability::Nullable) ); } @@ -584,10 +594,15 @@ mod tests { let item = arr.apply(&expr).unwrap(); assert_eq!( - item.scalar_at(0).unwrap(), + item.execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::bool(true, Nullability::Nullable) ); - assert!(!item.is_valid(1).unwrap()); + assert!( + !item + .is_valid(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + ); } #[test] @@ -676,7 +691,9 @@ mod tests { let expr = list_contains(lit(list_scalar.clone()), lit(2i32)); let result = arr.clone().apply(&expr).unwrap(); assert_eq!( - result.scalar_at(0).unwrap(), + result + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::bool(true, Nullability::NonNullable) ); @@ -684,7 +701,9 @@ mod tests { let expr = list_contains(lit(list_scalar), lit(42i32)); let result = arr.apply(&expr).unwrap(); assert_eq!( - result.scalar_at(0).unwrap(), + result + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::bool(false, Nullability::NonNullable) ); } @@ -692,10 +711,15 @@ mod tests { // -- Tests migrated from compute/list_contains.rs -- fn nonnull_strings(values: Vec>) -> ArrayRef { - ListArray::from_iter_slow::(values, Arc::new(DType::Utf8(Nullability::NonNullable))) - .unwrap() - .to_listview() - .into_array() + #[expect(deprecated)] + let result = ListArray::from_iter_slow::( + values, + Arc::new(DType::Utf8(Nullability::NonNullable)), + ) + .unwrap() + .to_listview() + .into_array(); + result } fn null_strings(values: Vec>>) -> ArrayRef { @@ -714,11 +738,13 @@ mod tests { let elements = VarBinArray::from_iter(elements, DType::Utf8(Nullability::Nullable)).into_array(); - ListArray::try_new(elements, offsets, Validity::NonNullable) + #[expect(deprecated)] + let result = ListArray::try_new(elements, offsets, Validity::NonNullable) .unwrap() .as_array() .to_listview() - .into_array() + .into_array(); + result } fn bool_array(values: Vec, validity: Validity) -> BoolArray { @@ -800,7 +826,6 @@ mod tests { let expr = list_contains(root(), lit(2i32)); let contains = list_array.apply(&expr).unwrap(); - assert!(contains.is::(), "Expected constant result"); let expected = BoolArray::from_iter([true, true]); assert_arrays_eq!(contains, expected); } @@ -818,7 +843,6 @@ mod tests { let expr = list_contains(root(), lit(2i32)); let contains = list_array.apply(&expr).unwrap(); - assert!(contains.is::(), "Expected constant result"); let expected = BoolArray::new( [false, false, false, false, false].into_iter().collect(), diff --git a/vortex-array/src/scalar_fn/fns/mask/mod.rs b/vortex-array/src/scalar_fn/fns/mask/mod.rs index eaf83307b35..b4841576cf6 100644 --- a/vortex-array/src/scalar_fn/fns/mask/mod.rs +++ b/vortex-array/src/scalar_fn/fns/mask/mod.rs @@ -14,12 +14,11 @@ use crate::ArrayRef; use crate::Canonical; use crate::ExecutionCtx; use crate::IntoArray; -use crate::arrays::BoolArray; use crate::arrays::Constant; use crate::arrays::ConstantArray; -use crate::arrays::bool::BoolArrayExt; use crate::arrays::masked::mask_validity_canonical; use crate::builtins::ArrayBuiltins; +use crate::child_to_validity; use crate::dtype::DType; use crate::dtype::Nullability; use crate::expr::Expression; @@ -46,7 +45,7 @@ impl ScalarFnVTable for Mask { type Options = EmptyOptions; fn id(&self) -> ScalarFnId { - ScalarFnId::from("vortex.mask") + ScalarFnId::new("vortex.mask") } fn serialize(&self, _options: &Self::Options) -> VortexResult>> { @@ -182,11 +181,9 @@ fn execute_canonical( mask_array: ArrayRef, ctx: &mut ExecutionCtx, ) -> VortexResult { - let mask_bool = mask_array.execute::(ctx)?; - let validity_mask = vortex_mask::Mask::from(mask_bool.to_bit_buffer()); - + let validity = child_to_validity(Some(&mask_array), Nullability::Nullable); let canonical = input.execute::(ctx)?; - Ok(mask_validity_canonical(canonical, &validity_mask, ctx)?.into_array()) + Ok(mask_validity_canonical(canonical, validity, ctx)?.into_array()) } #[cfg(test)] diff --git a/vortex-array/src/scalar_fn/fns/merge.rs b/vortex-array/src/scalar_fn/fns/merge.rs index 01445eb6c54..0c79ef86731 100644 --- a/vortex-array/src/scalar_fn/fns/merge.rs +++ b/vortex-array/src/scalar_fn/fns/merge.rs @@ -291,7 +291,8 @@ mod tests { use crate::ArrayRef; use crate::IntoArray; - use crate::ToCanonical; + #[expect(deprecated)] + use crate::ToCanonical as _; use crate::arrays::PrimitiveArray; use crate::arrays::struct_::StructArrayExt; use crate::assert_arrays_eq; @@ -317,11 +318,16 @@ mod tests { vortex_bail!("empty field path"); }; + #[expect(deprecated)] let mut array = array.to_struct().unmasked_field_by_name(field)?.clone(); for field in field_path { - array = array.to_struct().unmasked_field_by_name(field)?.clone(); + #[expect(deprecated)] + let next = array.to_struct().unmasked_field_by_name(field)?.clone(); + array = next; } - Ok(array.to_primitive()) + #[expect(deprecated)] + let result = array.to_primitive(); + Ok(result) } #[test] @@ -491,13 +497,16 @@ mod tests { ]) .unwrap() .into_array(); + #[expect(deprecated)] let actual_array = test_array.apply(&expr).unwrap().to_struct(); + #[expect(deprecated)] + let inner_struct = actual_array + .unmasked_field_by_name("a") + .unwrap() + .to_struct(); assert_eq!( - actual_array - .unmasked_field_by_name("a") - .unwrap() - .to_struct() + inner_struct .names() .iter() .map(|name| name.as_ref()) @@ -532,6 +541,7 @@ mod tests { ]) .unwrap() .into_array(); + #[expect(deprecated)] let actual_array = test_array.apply(&expr).unwrap().to_struct(); assert_eq!(actual_array.names(), ["a", "c", "b", "d"]); diff --git a/vortex-array/src/scalar_fn/fns/mod.rs b/vortex-array/src/scalar_fn/fns/mod.rs index 94fc8fb0384..8fa1b66532d 100644 --- a/vortex-array/src/scalar_fn/fns/mod.rs +++ b/vortex-array/src/scalar_fn/fns/mod.rs @@ -8,6 +8,7 @@ pub mod cast; pub mod dynamic; pub mod fill_null; pub mod get_item; +pub mod is_not_null; pub mod is_null; pub mod like; pub mod list_contains; diff --git a/vortex-array/src/scalar_fn/fns/not/mod.rs b/vortex-array/src/scalar_fn/fns/not/mod.rs index c70c8794fe6..66953709640 100644 --- a/vortex-array/src/scalar_fn/fns/not/mod.rs +++ b/vortex-array/src/scalar_fn/fns/not/mod.rs @@ -36,7 +36,7 @@ impl ScalarFnVTable for Not { type Options = EmptyOptions; fn id(&self) -> ScalarFnId { - ScalarFnId::from("vortex.not") + ScalarFnId::new("vortex.not") } fn serialize(&self, _options: &Self::Options) -> VortexResult>> { @@ -122,7 +122,8 @@ impl ScalarFnVTable for Not { #[cfg(test)] mod tests { use crate::IntoArray; - use crate::ToCanonical; + #[expect(deprecated)] + use crate::ToCanonical as _; use crate::arrays::bool::BoolArrayExt; use crate::dtype::DType; use crate::dtype::Nullability; @@ -137,15 +138,10 @@ mod tests { fn invert_booleans() { let not_expr = not(root()); let bools = BoolArray::from_iter([false, true, false, false, true, true]); + #[expect(deprecated)] + let result = bools.into_array().apply(¬_expr).unwrap().to_bool(); assert_eq!( - bools - .into_array() - .apply(¬_expr) - .unwrap() - .to_bool() - .to_bit_buffer() - .iter() - .collect::>(), + result.to_bit_buffer().iter().collect::>(), vec![true, false, true, true, false, false] ); } diff --git a/vortex-array/src/scalar_fn/fns/pack.rs b/vortex-array/src/scalar_fn/fns/pack.rs index 13626bb107b..0a24f721064 100644 --- a/vortex-array/src/scalar_fn/fns/pack.rs +++ b/vortex-array/src/scalar_fn/fns/pack.rs @@ -168,7 +168,8 @@ mod tests { use super::PackOptions; use crate::ArrayRef; use crate::IntoArray; - use crate::ToCanonical; + #[expect(deprecated)] + use crate::ToCanonical as _; use crate::arrays::PrimitiveArray; use crate::arrays::struct_::StructArrayExt; use crate::assert_arrays_eq; @@ -195,11 +196,16 @@ mod tests { vortex_bail!("empty field path"); }; + #[expect(deprecated)] let mut array = array.to_struct().unmasked_field_by_name(field)?.clone(); for field in field_path { - array = array.to_struct().unmasked_field_by_name(field)?.clone(); + #[expect(deprecated)] + let next = array.to_struct().unmasked_field_by_name(field)?.clone(); + array = next; } - Ok(array.to_primitive()) + #[expect(deprecated)] + let result = array.to_primitive(); + Ok(result) } #[test] @@ -215,7 +221,9 @@ mod tests { let test_array = test_array(); let actual_array = test_array.clone().apply(&expr).unwrap(); assert_eq!(actual_array.len(), test_array.len()); - assert_eq!(actual_array.to_struct().struct_fields().nfields(), 0); + #[expect(deprecated)] + let nfields = actual_array.to_struct().struct_fields().nfields(); + assert_eq!(nfields, 0); } #[test] @@ -228,6 +236,7 @@ mod tests { [col("a"), col("b"), col("a")], ); + #[expect(deprecated)] let actual_array = test_array().apply(&expr).unwrap().to_struct(); assert_eq!(actual_array.names(), ["one", "two", "three"]); @@ -267,6 +276,7 @@ mod tests { ], ); + #[expect(deprecated)] let actual_array = test_array().apply(&expr).unwrap().to_struct(); assert_eq!(actual_array.names(), ["one", "two", "three"]); @@ -299,6 +309,7 @@ mod tests { [col("a"), col("b"), col("a")], ); + #[expect(deprecated)] let actual_array = test_array().apply(&expr).unwrap().to_struct(); assert_eq!(actual_array.names(), ["one", "two", "three"]); diff --git a/vortex-array/src/scalar_fn/fns/root.rs b/vortex-array/src/scalar_fn/fns/root.rs index 7e04746db19..99bb6fecf6e 100644 --- a/vortex-array/src/scalar_fn/fns/root.rs +++ b/vortex-array/src/scalar_fn/fns/root.rs @@ -30,7 +30,7 @@ impl ScalarFnVTable for Root { type Options = EmptyOptions; fn id(&self) -> ScalarFnId { - ScalarFnId::from("vortex.root") + ScalarFnId::new("vortex.root") } fn serialize(&self, _instance: &Self::Options) -> VortexResult>> { diff --git a/vortex-array/src/scalar_fn/fns/select.rs b/vortex-array/src/scalar_fn/fns/select.rs index 2f93df5e789..2ca070c2bc4 100644 --- a/vortex-array/src/scalar_fn/fns/select.rs +++ b/vortex-array/src/scalar_fn/fns/select.rs @@ -94,7 +94,7 @@ impl ScalarFnVTable for Select { fn child_name(&self, _instance: &FieldSelection, child_idx: usize) -> ChildName { match child_idx { - 0 => ChildName::new_ref("child"), + 0 => ChildName::from("child"), _ => unreachable!(), } } @@ -307,7 +307,8 @@ mod tests { use vortex_buffer::buffer; use crate::IntoArray; - use crate::ToCanonical; + #[expect(deprecated)] + use crate::ToCanonical as _; use crate::arrays::struct_::StructArrayExt; use crate::dtype::DType; use crate::dtype::FieldName; @@ -335,6 +336,7 @@ mod tests { pub fn include_columns() { let st = test_array(); let select = select(vec![FieldName::from("a")], root()); + #[expect(deprecated)] let selected = st.into_array().apply(&select).unwrap().to_struct(); let selected_names = selected.names().clone(); assert_eq!(selected_names.as_ref(), &["a"]); @@ -344,6 +346,7 @@ mod tests { pub fn exclude_columns() { let st = test_array(); let select = select_exclude(vec![FieldName::from("a")], root()); + #[expect(deprecated)] let selected = st.into_array().apply(&select).unwrap().to_struct(); let selected_names = selected.names().clone(); assert_eq!(selected_names.as_ref(), &["b"]); diff --git a/vortex-array/src/scalar_fn/fns/zip/kernel.rs b/vortex-array/src/scalar_fn/fns/zip/kernel.rs index 5cfefee70d7..575c2f4c55a 100644 --- a/vortex-array/src/scalar_fn/fns/zip/kernel.rs +++ b/vortex-array/src/scalar_fn/fns/zip/kernel.rs @@ -8,7 +8,7 @@ use crate::ArrayRef; use crate::ExecutionCtx; use crate::array::ArrayView; use crate::array::VTable; -use crate::arrays::ScalarFnVTable; +use crate::arrays::ScalarFn; use crate::arrays::scalar_fn::ExactScalarFn; use crate::arrays::scalar_fn::ScalarFnArrayExt; use crate::arrays::scalar_fn::ScalarFnArrayView; @@ -68,7 +68,7 @@ where return Ok(None); } let scalar_fn_array = parent - .as_opt::() + .as_opt::() .vortex_expect("ExactScalarFn matcher confirmed ScalarFnArray"); let if_false = scalar_fn_array.get_child(1); let mask_array = scalar_fn_array.get_child(2); @@ -97,7 +97,7 @@ where return Ok(None); } let scalar_fn_array = parent - .as_opt::() + .as_opt::() .vortex_expect("ExactScalarFn matcher confirmed ScalarFnArray"); let if_false = scalar_fn_array.get_child(1); let mask_array = scalar_fn_array.get_child(2); diff --git a/vortex-array/src/scalar_fn/fns/zip/mod.rs b/vortex-array/src/scalar_fn/fns/zip/mod.rs index 5de3524be9a..ddb662459a7 100644 --- a/vortex-array/src/scalar_fn/fns/zip/mod.rs +++ b/vortex-array/src/scalar_fn/fns/zip/mod.rs @@ -46,7 +46,7 @@ impl ScalarFnVTable for Zip { type Options = EmptyOptions; fn id(&self) -> ScalarFnId { - ScalarFnId::from("vortex.zip") + ScalarFnId::new("vortex.zip") } fn serialize(&self, _options: &Self::Options) -> VortexResult>> { @@ -118,7 +118,7 @@ impl ScalarFnVTable for Zip { let mask = mask_array .execute::(ctx)? - .to_mask_fill_null_false(); + .to_mask_fill_null_false(ctx); let return_dtype = if_true .dtype() @@ -238,7 +238,7 @@ mod tests { use crate::arrays::Struct; use crate::arrays::StructArray; use crate::arrays::VarBinView; - use crate::arrow::IntoArrowArray; + use crate::arrow::ArrowArrayExecutor; use crate::assert_arrays_eq; use crate::builders::ArrayBuilder; use crate::builders::BufferGrowthStrategy; @@ -444,17 +444,22 @@ mod tests { let zipped = zipped.as_opt::().unwrap(); assert_eq!(zipped.data_buffers().len(), 2); + let mut arrow_ctx = LEGACY_SESSION.create_execution_ctx(); let expected = arrow_zip( mask.into_array() - .into_arrow_preferred() + .execute_arrow(None, &mut arrow_ctx) .unwrap() .as_boolean(), - &if_true.into_arrow_preferred().unwrap(), - &if_false.into_arrow_preferred().unwrap(), + &if_true.execute_arrow(None, &mut arrow_ctx).unwrap(), + &if_false.execute_arrow(None, &mut arrow_ctx).unwrap(), ) .unwrap(); - let actual = zipped.array().clone().into_arrow_preferred().unwrap(); + let actual = zipped + .array() + .clone() + .execute_arrow(None, &mut arrow_ctx) + .unwrap(); assert_eq!(actual.as_ref(), expected.as_ref()); } } diff --git a/vortex-array/src/scalar_fn/foreign.rs b/vortex-array/src/scalar_fn/foreign.rs index e7d3ee641cc..d2abd3b9a95 100644 --- a/vortex-array/src/scalar_fn/foreign.rs +++ b/vortex-array/src/scalar_fn/foreign.rs @@ -17,10 +17,10 @@ use crate::expr::Expression; use crate::scalar_fn::Arity; use crate::scalar_fn::ChildName; use crate::scalar_fn::ExecutionArgs; -use crate::scalar_fn::ScalarFn; use crate::scalar_fn::ScalarFnId; use crate::scalar_fn::ScalarFnRef; use crate::scalar_fn::ScalarFnVTable; +use crate::scalar_fn::TypedScalarFnInstance; /// Options payload for a foreign scalar function. #[derive(Clone, Debug, PartialEq, Eq, Hash)] @@ -58,7 +58,8 @@ impl ForeignScalarFnVTable { } pub fn make_scalar_fn(id: ScalarFnId, metadata: Vec, arity: usize) -> ScalarFnRef { - ScalarFn::new(Self::new(id), ForeignScalarFnOptions::new(metadata, arity)).erased() + TypedScalarFnInstance::new(Self::new(id), ForeignScalarFnOptions::new(metadata, arity)) + .erased() } } @@ -66,7 +67,7 @@ impl ScalarFnVTable for ForeignScalarFnVTable { type Options = ForeignScalarFnOptions; fn id(&self) -> ScalarFnId { - self.id.clone() + self.id } fn serialize(&self, options: &Self::Options) -> VortexResult>> { diff --git a/vortex-array/src/arrays/extension/vtable/canonical.rs b/vortex-array/src/scalar_fn/internal/mod.rs similarity index 83% rename from vortex-array/src/arrays/extension/vtable/canonical.rs rename to vortex-array/src/scalar_fn/internal/mod.rs index 0d735177e5d..c91a7541c8a 100644 --- a/vortex-array/src/arrays/extension/vtable/canonical.rs +++ b/vortex-array/src/scalar_fn/internal/mod.rs @@ -1,2 +1,4 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors + +pub mod row_count; diff --git a/vortex-array/src/scalar_fn/internal/row_count.rs b/vortex-array/src/scalar_fn/internal/row_count.rs new file mode 100644 index 00000000000..f411626321d --- /dev/null +++ b/vortex-array/src/scalar_fn/internal/row_count.rs @@ -0,0 +1,166 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +use std::fmt::Formatter; + +use vortex_array::ArrayRef; +use vortex_array::ExecutionCtx; +use vortex_array::arrays::ScalarFn; +use vortex_array::arrays::scalar_fn::ExactScalarFn; +use vortex_array::arrays::scalar_fn::ScalarFnArrayExt; +use vortex_array::dtype::DType; +use vortex_array::dtype::Nullability; +use vortex_array::dtype::PType; +use vortex_array::expr::Expression; +use vortex_array::scalar_fn::Arity; +use vortex_array::scalar_fn::ChildName; +use vortex_array::scalar_fn::EmptyOptions; +use vortex_array::scalar_fn::ExecutionArgs; +use vortex_array::scalar_fn::ScalarFnId; +use vortex_array::scalar_fn::ScalarFnVTable; +use vortex_error::VortexResult; +use vortex_error::vortex_bail; +use vortex_error::vortex_ensure; + +/// Zero-argument placeholder for the row count of the current evaluation scope. +/// +/// This expression *MUST* be replaced with a concrete array before evaluation. +/// Currently, the rewrite only happens in the context of stats pruning. +/// +/// `RowCount` is emitted while building pruning predicates that need a +/// scope-level value which is not stored as a regular stats column, such as the +/// row count of the current file or zone. The layer that owns that scope must +/// replace each placeholder with a concrete array via [`substitute_row_count`] +/// before evaluation. +/// +/// Calling [`ScalarFnVTable::execute`] directly returns an error because this +/// node is only a marker in a lazy expression tree. +#[derive(Clone)] +pub struct RowCount; + +impl ScalarFnVTable for RowCount { + type Options = EmptyOptions; + + fn id(&self) -> ScalarFnId { + ScalarFnId::from("vortex.row_count") + } + + fn arity(&self, _options: &Self::Options) -> Arity { + Arity::Exact(0) + } + + fn child_name(&self, _options: &Self::Options, _child_idx: usize) -> ChildName { + unreachable!("RowCount has arity 0") + } + + fn fmt_sql( + &self, + _options: &Self::Options, + _expr: &Expression, + f: &mut Formatter<'_>, + ) -> std::fmt::Result { + write!(f, "row_count()") + } + + fn return_dtype(&self, _options: &Self::Options, _args: &[DType]) -> VortexResult { + Ok(DType::Primitive(PType::U64, Nullability::NonNullable)) + } + + fn execute( + &self, + _options: &Self::Options, + _args: &dyn ExecutionArgs, + _ctx: &mut ExecutionCtx, + ) -> VortexResult { + vortex_bail!("RowCount must be substituted before evaluation") + } + + fn is_null_sensitive(&self, _options: &Self::Options) -> bool { + false + } + + fn is_fallible(&self, _options: &Self::Options) -> bool { + false + } +} + +/// Returns whether `array` contains a [`RowCount`] placeholder. +/// +/// Traversal is limited to lazy [`ScalarFnArray`] nodes produced by +/// [`ArrayRef::apply`][crate::ArrayRef::apply]. Other arrays are evaluation +/// leaves and cannot contain unevaluated placeholders. +/// +/// [`ScalarFnArray`]: vortex_array::arrays::ScalarFnArray +pub fn contains_row_count(array: &ArrayRef) -> bool { + if array.is::>() { + return true; + } + match array.as_opt::() { + Some(view) => view.iter_children().any(contains_row_count), + None => false, + } +} + +/// Replaces every [`RowCount`] placeholder with `replacement`. +/// +/// The replacement must have the same dtype and length as each placeholder. +/// Lazy [`ScalarFnArray`] ancestors are rewritten through slot take/put so +/// unaffected children are preserved, while non-[`ScalarFn`] arrays are returned +/// unchanged. +/// +/// [`ScalarFnArray`]: vortex_array::arrays::ScalarFnArray +pub fn substitute_row_count(array: ArrayRef, replacement: &ArrayRef) -> VortexResult { + if array.is::>() { + vortex_ensure!( + replacement.len() == array.len(), + "RowCount replacement length {} does not match scope length {}", + replacement.len(), + array.len(), + ); + vortex_ensure!( + replacement.dtype() == array.dtype(), + "RowCount replacement dtype {} does not match scope dtype {}", + replacement.dtype(), + array.dtype(), + ); + return Ok(replacement.clone()); + } + + if !array.is::() { + return Ok(array); + } + + let nchildren = array.nchildren(); + let mut array = array; + for slot_idx in 0..nchildren { + // SAFETY: `substitute_row_count` always returns an array with the same dtype and + // length as its input — `RowCount` placeholders are replaced with a checked + // replacement (same dtype and length), and `ScalarFn` recursion preserves both by + // operating on each slot in place. + let (taken, child) = unsafe { array.take_slot_unchecked(slot_idx)? }; + let new_child = substitute_row_count(child, replacement)?; + array = unsafe { taken.put_slot_unchecked(slot_idx, new_child)? }; + } + Ok(array) +} + +#[cfg(test)] +mod tests { + use vortex_array::dtype::DType; + use vortex_array::dtype::Nullability; + use vortex_array::dtype::PType; + + use crate::scalar_fn::EmptyOptions; + use crate::scalar_fn::internal::row_count::RowCount; + use crate::scalar_fn::vtable::ScalarFnVTableExt; + + #[test] + fn row_count_helper_dtype() { + let expr = RowCount.new_expr(EmptyOptions, []); + assert_eq!( + expr.return_dtype(&DType::Primitive(PType::I32, Nullability::Nullable)) + .unwrap(), + DType::Primitive(PType::U64, Nullability::NonNullable), + ); + } +} diff --git a/vortex-array/src/scalar_fn/mod.rs b/vortex-array/src/scalar_fn/mod.rs index 39e45f00e08..590ccb44224 100644 --- a/vortex-array/src/scalar_fn/mod.rs +++ b/vortex-array/src/scalar_fn/mod.rs @@ -7,7 +7,7 @@ //! implementations. Expressions ([`crate::expr::Expression`]) reference scalar functions //! at each node. -use arcref::ArcRef; +use vortex_session::registry::Id; mod vtable; pub use vtable::*; @@ -31,19 +31,20 @@ mod signature; pub use signature::*; pub mod fns; +pub mod internal; pub mod session; /// A unique identifier for a scalar function. -pub type ScalarFnId = ArcRef; +pub type ScalarFnId = Id; /// Private module to seal [`typed::DynScalarFn`]. mod sealed { use crate::scalar_fn::ScalarFnVTable; - use crate::scalar_fn::typed::ScalarFn; + use crate::scalar_fn::typed::TypedScalarFnInstance; /// Marker trait to prevent external implementations of [`super::typed::DynScalarFn`]. pub(crate) trait Sealed {} /// This can be the **only** implementor for [`super::typed::DynScalarFn`]. - impl Sealed for ScalarFn {} + impl Sealed for TypedScalarFnInstance {} } diff --git a/vortex-array/src/scalar_fn/plugin.rs b/vortex-array/src/scalar_fn/plugin.rs index bafaf2b6137..ef06bc3be0e 100644 --- a/vortex-array/src/scalar_fn/plugin.rs +++ b/vortex-array/src/scalar_fn/plugin.rs @@ -7,10 +7,10 @@ use std::sync::Arc; use vortex_error::VortexResult; use vortex_session::VortexSession; -use crate::scalar_fn::ScalarFn; use crate::scalar_fn::ScalarFnId; use crate::scalar_fn::ScalarFnRef; use crate::scalar_fn::ScalarFnVTable; +use crate::scalar_fn::TypedScalarFnInstance; /// Reference-counted pointer to a scalar function plugin. pub type ScalarFnPluginRef = Arc; @@ -43,6 +43,6 @@ impl ScalarFnPlugin for V { fn deserialize(&self, metadata: &[u8], session: &VortexSession) -> VortexResult { let options = ScalarFnVTable::deserialize(self, metadata, session)?; - Ok(ScalarFn::new(self.clone(), options).erased()) + Ok(TypedScalarFnInstance::new(self.clone(), options).erased()) } } diff --git a/vortex-array/src/scalar_fn/session.rs b/vortex-array/src/scalar_fn/session.rs index eef759bf8e3..32783e9df89 100644 --- a/vortex-array/src/scalar_fn/session.rs +++ b/vortex-array/src/scalar_fn/session.rs @@ -1,10 +1,12 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +use std::any::Any; use std::sync::Arc; use vortex_session::Ref; use vortex_session::SessionExt; +use vortex_session::SessionVar; use vortex_session::registry::Registry; use crate::scalar_fn::ScalarFnPluginRef; @@ -14,6 +16,7 @@ use crate::scalar_fn::fns::binary::Binary; use crate::scalar_fn::fns::cast::Cast; use crate::scalar_fn::fns::fill_null::FillNull; use crate::scalar_fn::fns::get_item::GetItem; +use crate::scalar_fn::fns::is_not_null::IsNotNull; use crate::scalar_fn::fns::is_null::IsNull; use crate::scalar_fn::fns::like::Like; use crate::scalar_fn::fns::list_contains::ListContains; @@ -58,6 +61,7 @@ impl Default for ScalarFnSession { this.register(Cast); this.register(FillNull); this.register(GetItem); + this.register(IsNotNull); this.register(IsNull); this.register(Like); this.register(ListContains); @@ -72,6 +76,16 @@ impl Default for ScalarFnSession { } } +impl SessionVar for ScalarFnSession { + fn as_any(&self) -> &dyn Any { + self + } + + fn as_any_mut(&mut self) -> &mut dyn Any { + self + } +} + /// Extension trait for accessing scalar function session data. pub trait ScalarFnSessionExt: SessionExt { /// Returns the scalar function vtable registry. diff --git a/vortex-array/src/scalar_fn/typed.rs b/vortex-array/src/scalar_fn/typed.rs index 88072ff4002..a2ef9549bff 100644 --- a/vortex-array/src/scalar_fn/typed.rs +++ b/vortex-array/src/scalar_fn/typed.rs @@ -40,14 +40,14 @@ use crate::scalar_fn::SimplifyCtx; /// You can construct one via [`new()`], and erase the type with [`erased()`] to obtain a /// [`ScalarFnRef`]. /// -/// [`new()`]: ScalarFn::new -/// [`erased()`]: ScalarFn::erased -pub struct ScalarFn { +/// [`new()`]: TypedScalarFnInstance::new +/// [`erased()`]: TypedScalarFnInstance::erased +pub struct TypedScalarFnInstance { vtable: V, options: V::Options, } -impl ScalarFn { +impl TypedScalarFnInstance { /// Create a new typed scalar function instance. pub fn new(vtable: V, options: V::Options) -> Self { Self { vtable, options } @@ -121,7 +121,7 @@ pub(super) trait DynScalarFn: 'static + Send + Sync + super::sealed::Sealed { fn options_debug(&self, f: &mut Formatter<'_>) -> fmt::Result; } -impl DynScalarFn for ScalarFn { +impl DynScalarFn for TypedScalarFnInstance { #[inline(always)] fn as_any(&self) -> &dyn Any { self diff --git a/vortex-array/src/scalar_fn/vtable.rs b/vortex-array/src/scalar_fn/vtable.rs index c9588e8aa13..7c4f65cedff 100644 --- a/vortex-array/src/scalar_fn/vtable.rs +++ b/vortex-array/src/scalar_fn/vtable.rs @@ -22,9 +22,9 @@ use crate::dtype::DType; use crate::expr::Expression; use crate::expr::StatsCatalog; use crate::expr::stats::Stat; -use crate::scalar_fn::ScalarFn; use crate::scalar_fn::ScalarFnId; use crate::scalar_fn::ScalarFnRef; +use crate::scalar_fn::TypedScalarFnInstance; /// This trait defines the interface for scalar function vtables, including methods for /// serialization, deserialization, validation, child naming, return type computation, @@ -388,7 +388,7 @@ impl Display for EmptyOptions { pub trait ScalarFnVTableExt: ScalarFnVTable { /// Bind this vtable with the given options into a [`ScalarFnRef`]. fn bind(&self, options: Self::Options) -> ScalarFnRef { - ScalarFn::new(self.clone(), options).erased() + TypedScalarFnInstance::new(self.clone(), options).erased() } /// Create a new expression with this vtable and the given options and children. diff --git a/vortex-array/src/search_sorted.rs b/vortex-array/src/search_sorted.rs index cf096bbb8e2..14741652330 100644 --- a/vortex-array/src/search_sorted.rs +++ b/vortex-array/src/search_sorted.rs @@ -13,6 +13,8 @@ use std::hint; use vortex_error::VortexResult; use crate::ArrayRef; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::scalar::Scalar; #[derive(Debug, Copy, Clone, Eq, PartialEq)] @@ -264,7 +266,7 @@ fn search_sorted_side_idx VortexResult>( impl IndexOrd for ArrayRef { fn index_cmp(&self, idx: usize, elem: &Scalar) -> VortexResult> { - let scalar_a = self.scalar_at(idx)?; + let scalar_a = self.execute_scalar(idx, &mut LEGACY_SESSION.create_execution_ctx())?; Ok(scalar_a.partial_cmp(elem)) } diff --git a/vortex-array/src/serde.rs b/vortex-array/src/serde.rs index 68a00569caf..26eb20faa66 100644 --- a/vortex-array/src/serde.rs +++ b/vortex-array/src/serde.rs @@ -29,6 +29,7 @@ use vortex_utils::aliases::hash_map::HashMap; use crate::ArrayContext; use crate::ArrayRef; +use crate::array::ArrayId; use crate::array::new_foreign_array; use crate::buffer::BufferHandle; use crate::dtype::DType; @@ -170,16 +171,6 @@ impl<'a> ArrayNodeFlatBuffer<'a> { session: &'a VortexSession, array: &'a ArrayRef, ) -> VortexResult { - // Depth-first traversal of the array to ensure it supports serialization. - // FIXME(ngates): this serializes the metadata and throws it away! - for child in array.depth_first_traversal() { - if child.metadata(session)?.is_none() { - vortex_bail!( - "Array {} does not support serialization", - child.encoding_id() - ); - } - } let n_buffers_recursive = array.nbuffers_recursive(); if n_buffers_recursive > u16::MAX as usize { vortex_bail!( @@ -210,13 +201,13 @@ impl<'a> ArrayNodeFlatBuffer<'a> { ) })?; - let metadata = self.array.metadata(self.session)?.ok_or_else(|| { + let metadata_bytes = self.session.array_serialize(self.array)?.ok_or_else(|| { vortex_err!( "Array {} does not support serialization", self.array.encoding_id() ) })?; - let metadata = Some(fbb.create_vector(metadata.as_slice())); + let metadata = Some(fbb.create_vector(metadata_bytes.as_slice())); // Assign buffer indices for all child arrays. let nbuffers = u16::try_from(self.array.nbuffers()) @@ -367,9 +358,9 @@ impl SerializedArray { decoded.dtype(), dtype, ); - assert_eq!( - decoded.encoding_id(), - encoding_id, + + assert!( + plugin.is_supported_encoding(&decoded.encoding_id()), "Array decoded from {} has incorrect encoding {}", encoding_id, decoded.encoding_id(), @@ -387,7 +378,7 @@ impl SerializedArray { fn decode_foreign( &self, - encoding_id: crate::array::ArrayId, + encoding_id: ArrayId, dtype: &DType, len: usize, ctx: &ReadContext, @@ -701,101 +692,3 @@ impl TryFrom for SerializedArray { Self::try_from(value.try_to_host_sync()?) } } - -#[cfg(test)] -mod tests { - use std::sync::LazyLock; - - use flatbuffers::FlatBufferBuilder; - use vortex_session::VortexSession; - use vortex_session::registry::ReadContext; - - use super::SerializeOptions; - use super::SerializedArray; - use crate::ArrayContext; - use crate::array::ArrayId; - use crate::dtype::DType; - use crate::dtype::Nullability; - use crate::flatbuffers as fba; - use crate::session::ArraySession; - - static SESSION: LazyLock = LazyLock::new(VortexSession::empty); - - #[test] - fn unknown_array_encoding_allow_unknown() { - let mut fbb = FlatBufferBuilder::new(); - - let child_metadata = fbb.create_vector(&[9u8]); - let child = fba::ArrayNode::create( - &mut fbb, - &fba::ArrayNodeArgs { - encoding: 1, - metadata: Some(child_metadata), - children: None, - buffers: None, - stats: None, - }, - ); - - let children = fbb.create_vector(&[child]); - let metadata = fbb.create_vector(&[1u8, 2, 3]); - let root = fba::ArrayNode::create( - &mut fbb, - &fba::ArrayNodeArgs { - encoding: 0, - metadata: Some(metadata), - children: Some(children), - buffers: None, - stats: None, - }, - ); - let array = fba::Array::create( - &mut fbb, - &fba::ArrayArgs { - root: Some(root), - buffers: None, - }, - ); - fbb.finish_minimal(array); - let (buf, start) = fbb.collapse(); - let tree = vortex_buffer::ByteBuffer::from(buf).slice(start..); - - let ser = SerializedArray::from_array_tree(tree).unwrap(); - let ctx = ReadContext::new([ - ArrayId::new_ref("vortex.test.foreign_array"), - ArrayId::new_ref("vortex.test.foreign_child"), - ]); - let session = VortexSession::empty() - .with::() - .allow_unknown(); - - let decoded = ser - .decode(&DType::Variant(Nullability::Nullable), 5, &ctx, &session) - .unwrap(); - assert_eq!(decoded.encoding_id().as_ref(), "vortex.test.foreign_array"); - assert_eq!(decoded.nchildren(), 1); - assert_eq!( - decoded.nth_child(0).unwrap().encoding_id().as_ref(), - "vortex.test.foreign_child" - ); - assert_eq!(decoded.metadata(&SESSION).unwrap().unwrap(), vec![1, 2, 3]); - assert_eq!( - decoded - .nth_child(0) - .unwrap() - .metadata(&SESSION) - .unwrap() - .unwrap(), - vec![9] - ); - - let serialized = decoded - .serialize( - &ArrayContext::default(), - &SESSION, - &SerializeOptions::default(), - ) - .unwrap(); - assert!(!serialized.is_empty()); - } -} diff --git a/vortex-array/src/session/mod.rs b/vortex-array/src/session/mod.rs index 62ed46cb34c..98a7417722b 100644 --- a/vortex-array/src/session/mod.rs +++ b/vortex-array/src/session/mod.rs @@ -1,11 +1,14 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +use std::any::Any; use std::sync::Arc; use vortex_error::VortexResult; +use vortex_error::vortex_bail; use vortex_session::Ref; use vortex_session::SessionExt; +use vortex_session::SessionVar; use vortex_session::registry::Registry; use crate::ArrayRef; @@ -15,17 +18,18 @@ use crate::arrays::Bool; use crate::arrays::Chunked; use crate::arrays::Constant; use crate::arrays::Decimal; +use crate::arrays::Dict; use crate::arrays::Extension; use crate::arrays::FixedSizeList; use crate::arrays::List; use crate::arrays::ListView; use crate::arrays::Masked; use crate::arrays::Null; -use crate::arrays::Patched; use crate::arrays::Primitive; use crate::arrays::Struct; use crate::arrays::VarBin; use crate::arrays::VarBinView; +use crate::arrays::Variant; pub type ArrayRegistry = Registry; @@ -68,20 +72,31 @@ impl Default for ArraySession { this.register(ListView); this.register(FixedSizeList); this.register(Struct); + this.register(Variant); this.register(Extension); // Register the utility encodings. this.register(Chunked); this.register(Constant); + this.register(Dict); this.register(List); this.register(Masked); - this.register(Patched); this.register(VarBin); this } } +impl SessionVar for ArraySession { + fn as_any(&self) -> &dyn Any { + self + } + + fn as_any_mut(&mut self) -> &mut dyn Any { + self + } +} + /// Session data for Vortex arrays. pub trait ArraySessionExt: SessionExt { /// Returns the array encoding registry. @@ -92,8 +107,12 @@ pub trait ArraySessionExt: SessionExt { /// Serialize an array using a plugin from the registry. fn array_serialize(&self, array: &ArrayRef) -> VortexResult>> { let Some(plugin) = self.arrays().registry.find(&array.encoding_id()) else { - return Ok(None); + vortex_bail!( + "Array {} is not registered for serializations", + array.encoding_id() + ); }; + plugin.serialize(array, &self.session()) } } diff --git a/vortex-array/src/stats/array.rs b/vortex-array/src/stats/array.rs index bd2e3cf7dea..248ca3587f2 100644 --- a/vortex-array/src/stats/array.rs +++ b/vortex-array/src/stats/array.rs @@ -6,6 +6,7 @@ use std::sync::Arc; use parking_lot::RwLock; +use vortex_array::ExecutionCtx; use vortex_error::VortexError; use vortex_error::VortexResult; use vortex_error::vortex_panic; @@ -15,8 +16,6 @@ use super::StatsSet; use super::StatsSetIntoIter; use super::TypedStatsSetRef; use crate::ArrayRef; -use crate::LEGACY_SESSION; -use crate::VortexSessionExecute; use crate::aggregate_fn::fns::is_constant::is_constant; use crate::aggregate_fn::fns::is_sorted::is_sorted; use crate::aggregate_fn::fns::is_sorted::is_strict_sorted; @@ -154,41 +153,35 @@ impl StatsSetRef<'_> { f(&mut lock.iter()) } - pub fn compute_stat(&self, stat: Stat) -> VortexResult> { - let mut ctx = LEGACY_SESSION.create_execution_ctx(); - + pub fn compute_stat(&self, stat: Stat, ctx: &mut ExecutionCtx) -> VortexResult> { // If it's already computed and exact, we can return it. if let Some(Precision::Exact(s)) = self.get(stat) { return Ok(Some(s)); } Ok(match stat { - Stat::Min => { - min_max(self.dyn_array_ref, &mut ctx)?.map(|MinMaxResult { min, max: _ }| min) - } - Stat::Max => { - min_max(self.dyn_array_ref, &mut ctx)?.map(|MinMaxResult { min: _, max }| max) - } + Stat::Min => min_max(self.dyn_array_ref, ctx)?.map(|MinMaxResult { min, max: _ }| min), + Stat::Max => min_max(self.dyn_array_ref, ctx)?.map(|MinMaxResult { min: _, max }| max), Stat::Sum => { Stat::Sum .dtype(self.dyn_array_ref.dtype()) .is_some() .then(|| { // Sum is supported for this dtype. - sum(self.dyn_array_ref, &mut ctx) + sum(self.dyn_array_ref, ctx) }) .transpose()? } - Stat::NullCount => self.dyn_array_ref.invalid_count().ok().map(Into::into), + Stat::NullCount => self.dyn_array_ref.invalid_count(ctx).ok().map(Into::into), Stat::IsConstant => { if self.dyn_array_ref.is_empty() { None } else { - Some(is_constant(self.dyn_array_ref, &mut ctx)?.into()) + Some(is_constant(self.dyn_array_ref, ctx)?.into()) } } - Stat::IsSorted => Some(is_sorted(self.dyn_array_ref, &mut ctx)?.into()), - Stat::IsStrictSorted => Some(is_strict_sorted(self.dyn_array_ref, &mut ctx)?.into()), + Stat::IsSorted => Some(is_sorted(self.dyn_array_ref, ctx)?.into()), + Stat::IsStrictSorted => Some(is_strict_sorted(self.dyn_array_ref, ctx)?.into()), Stat::UncompressedSizeInBytes => { let mut builder = builder_with_capacity(self.dyn_array_ref.dtype(), self.dyn_array_ref.len()); @@ -205,7 +198,7 @@ impl StatsSetRef<'_> { .is_some() .then(|| { // NaNCount is supported for this dtype. - nan_count(self.dyn_array_ref, &mut ctx) + nan_count(self.dyn_array_ref, ctx) }) .transpose()? .map(|s| s.into()) @@ -213,10 +206,10 @@ impl StatsSetRef<'_> { }) } - pub fn compute_all(&self, stats: &[Stat]) -> VortexResult { + pub fn compute_all(&self, stats: &[Stat], ctx: &mut ExecutionCtx) -> VortexResult { let mut stats_set = StatsSet::default(); for &stat in stats { - if let Some(s) = self.compute_stat(stat)? + if let Some(s) = self.compute_stat(stat, ctx)? && let Some(value) = s.into_value() { stats_set.set(stat, Precision::exact(value)); @@ -230,8 +223,9 @@ impl StatsSetRef<'_> { pub fn compute_as TryFrom<&'a Scalar, Error = VortexError>>( &self, stat: Stat, + ctx: &mut ExecutionCtx, ) -> Option { - self.compute_stat(stat) + self.compute_stat(stat, ctx) .inspect_err(|e| tracing::warn!("Failed to compute stat {stat}: {e}")) .ok() .flatten() @@ -255,32 +249,38 @@ impl StatsSetRef<'_> { self.array_stats.clear(stat); } - pub fn compute_min TryFrom<&'a Scalar, Error = VortexError>>(&self) -> Option { - self.compute_as(Stat::Min) + pub fn compute_min TryFrom<&'a Scalar, Error = VortexError>>( + &self, + ctx: &mut ExecutionCtx, + ) -> Option { + self.compute_as(Stat::Min, ctx) } - pub fn compute_max TryFrom<&'a Scalar, Error = VortexError>>(&self) -> Option { - self.compute_as(Stat::Max) + pub fn compute_max TryFrom<&'a Scalar, Error = VortexError>>( + &self, + ctx: &mut ExecutionCtx, + ) -> Option { + self.compute_as(Stat::Max, ctx) } - pub fn compute_is_sorted(&self) -> Option { - self.compute_as(Stat::IsSorted) + pub fn compute_is_sorted(&self, ctx: &mut ExecutionCtx) -> Option { + self.compute_as(Stat::IsSorted, ctx) } - pub fn compute_is_strict_sorted(&self) -> Option { - self.compute_as(Stat::IsStrictSorted) + pub fn compute_is_strict_sorted(&self, ctx: &mut ExecutionCtx) -> Option { + self.compute_as(Stat::IsStrictSorted, ctx) } - pub fn compute_is_constant(&self) -> Option { - self.compute_as(Stat::IsConstant) + pub fn compute_is_constant(&self, ctx: &mut ExecutionCtx) -> Option { + self.compute_as(Stat::IsConstant, ctx) } - pub fn compute_null_count(&self) -> Option { - self.compute_as(Stat::NullCount) + pub fn compute_null_count(&self, ctx: &mut ExecutionCtx) -> Option { + self.compute_as(Stat::NullCount, ctx) } - pub fn compute_uncompressed_size_in_bytes(&self) -> Option { - self.compute_as(Stat::UncompressedSizeInBytes) + pub fn compute_uncompressed_size_in_bytes(&self, ctx: &mut ExecutionCtx) -> Option { + self.compute_as(Stat::UncompressedSizeInBytes, ctx) } } diff --git a/vortex-array/src/stats/stats_set.rs b/vortex-array/src/stats/stats_set.rs index 5ddd0e908b1..a0ca186fed6 100644 --- a/vortex-array/src/stats/stats_set.rs +++ b/vortex-array/src/stats/stats_set.rs @@ -459,11 +459,28 @@ impl MutTypedStatsSetRef<'_, '_> { ) { (Some(m1), Some(m2)) => { // If the combine sum is exact, then we can sum them. - if let Some(scalar_value) = m1.zip(m2).as_exact().and_then(|(s1, s2)| { - s1.as_primitive() - .checked_add(&s2.as_primitive()) - .and_then(|pscalar| pscalar.pvalue().map(ScalarValue::Primitive)) - }) { + if let Some(scalar_value) = + m1.zip(m2).as_exact().and_then(|(s1, s2)| match s1.dtype() { + DType::Primitive(..) => s1 + .as_primitive() + .checked_add(&s2.as_primitive()) + .and_then(|pscalar| pscalar.pvalue().map(ScalarValue::Primitive)), + DType::Decimal(..) => s1 + .as_decimal() + .checked_binary_numeric( + &s2.as_decimal(), + crate::scalar::NumericOperator::Add, + ) + .map(|scalar| { + ScalarValue::Decimal( + scalar + .decimal_value() + .vortex_expect("no decimal value in scalar"), + ) + }), + _ => None, + }) + { self.set(Stat::Sum, Precision::Exact(scalar_value)); } } @@ -558,6 +575,8 @@ mod test { use enum_iterator::all; use itertools::Itertools; + use crate::LEGACY_SESSION; + use crate::VortexSessionExecute; use crate::arrays::PrimitiveArray; use crate::dtype::DType; use crate::dtype::Nullability; @@ -881,7 +900,10 @@ mod test { .filter(|s| !matches!(s, Stat::Sum)) .filter(|s| !matches!(s, Stat::NaNCount)) .collect_vec(); - array.statistics().compute_all(&all_stats).unwrap(); + array + .statistics() + .compute_all(&all_stats, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(); let stats = array.statistics().to_owned(); for stat in &all_stats { diff --git a/vortex-array/src/test_harness.rs b/vortex-array/src/test_harness.rs index 099b51bd289..dabeb8bdde4 100644 --- a/vortex-array/src/test_harness.rs +++ b/vortex-array/src/test_harness.rs @@ -8,13 +8,15 @@ use goldenfile::differs::binary_diff; use itertools::Itertools; use vortex_error::VortexResult; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::arrays::BoolArray; use crate::arrays::bool::BoolArrayExt; /// Check that a named metadata matches its previous versioning. /// /// Goldenfile takes care of checking for equality against a checked-in file. -#[allow(clippy::unwrap_used)] +#[expect(clippy::unwrap_used)] pub fn check_metadata(name: &str, metadata: &[u8]) { let mut mint = Mint::new("goldenfiles/"); let mut f = mint @@ -26,7 +28,10 @@ pub fn check_metadata(name: &str, metadata: &[u8]) { /// Outputs the indices of the true values in a BoolArray pub fn to_int_indices(indices_bits: BoolArray) -> VortexResult> { let buffer = indices_bits.to_bit_buffer(); - let mask = indices_bits.validity_mask()?; + let mask = indices_bits.as_ref().validity()?.execute_mask( + indices_bits.as_ref().len(), + &mut LEGACY_SESSION.create_execution_ctx(), + )?; Ok(buffer .iter() .enumerate() diff --git a/vortex-array/src/validity.rs b/vortex-array/src/validity.rs index a5f6903e712..b081783b809 100644 --- a/vortex-array/src/validity.rs +++ b/vortex-array/src/validity.rs @@ -6,13 +6,12 @@ use std::fmt::Debug; use std::ops::Range; +use itertools::Itertools as _; use vortex_buffer::BitBuffer; use vortex_error::VortexExpect as _; use vortex_error::VortexResult; use vortex_error::vortex_bail; use vortex_error::vortex_err; -use vortex_error::vortex_panic; -use vortex_mask::AllOr; use vortex_mask::Mask; use vortex_mask::MaskValues; @@ -20,10 +19,11 @@ use crate::ArrayRef; use crate::Canonical; use crate::ExecutionCtx; use crate::IntoArray; -use crate::ToCanonical; +use crate::LEGACY_SESSION; +use crate::VortexSessionExecute; use crate::arrays::BoolArray; +use crate::arrays::ChunkedArray; use crate::arrays::ConstantArray; -use crate::arrays::bool::BoolArrayExt; use crate::arrays::scalar_fn::ScalarFnFactoryExt; use crate::builtins::ArrayBuiltins; use crate::dtype::DType; @@ -112,6 +112,13 @@ impl Validity { } } + /// Returns `true` if this validity guarantees no null values, i.e. it is either + /// [`Validity::NonNullable`] or [`Validity::AllValid`]. + #[inline] + pub fn no_nulls(&self) -> bool { + matches!(self, Self::NonNullable | Self::AllValid) + } + /// The union nullability and validity. #[inline] pub fn union_nullability(self, nullability: Nullability) -> Self { @@ -128,8 +135,8 @@ impl Validity { Self::NonNullable | Self::AllValid => true, Self::AllInvalid => false, Self::Array(a) => a - .scalar_at(index) - .vortex_expect("Validity array must support scalar_at") + .execute_scalar(index, &mut LEGACY_SESSION.create_execution_ctx()) + .vortex_expect("Validity array must support execute_scalar") .as_bool() .value() .vortex_expect("Validity must be non-nullable"), @@ -151,22 +158,11 @@ impl Validity { pub fn take(&self, indices: &ArrayRef) -> VortexResult { match self { - Self::NonNullable => match indices.validity_mask()?.bit_buffer() { - AllOr::All => { - if indices.dtype().is_nullable() { - Ok(Self::AllValid) - } else { - Ok(Self::NonNullable) - } - } - AllOr::None => Ok(Self::AllInvalid), - AllOr::Some(buf) => Ok(Validity::from(buf.clone())), - }, - Self::AllValid => match indices.validity_mask()?.bit_buffer() { - AllOr::All => Ok(Self::AllValid), - AllOr::None => Ok(Self::AllInvalid), - AllOr::Some(buf) => Ok(Validity::from(buf.clone())), - }, + Self::NonNullable => indices.validity(), + Self::AllValid => Ok(match indices.validity()? { + Self::NonNullable => Self::AllValid, + v => v, + }), Self::AllInvalid => Ok(Self::AllInvalid), Self::Array(is_valid) => { let maybe_is_valid = is_valid.take(indices.clone())?; @@ -208,11 +204,12 @@ impl Validity { /// Converts this validity into a [`Mask`] of the given length. /// /// Valid elements are `true` and invalid elements are `false`. - pub fn to_mask(&self, length: usize) -> Mask { + #[deprecated(note = "Use execute_mask")] + pub fn to_mask(&self, length: usize, ctx: &mut ExecutionCtx) -> VortexResult { match self { - Self::NonNullable | Self::AllValid => Mask::new_true(length), - Self::AllInvalid => Mask::new_false(length), - Self::Array(a) => a.to_bool().to_mask(), + Self::NonNullable | Self::AllValid => Ok(Mask::new_true(length)), + Self::AllInvalid => Ok(Mask::new_false(length)), + Self::Array(arr) => arr.clone().execute::(ctx), } } @@ -297,12 +294,11 @@ impl Validity { _ => {} }; - let own_nullability = if matches!(self, Validity::NonNullable) { - Nullability::NonNullable - } else { - Nullability::Nullable - }; + if matches!(self, Validity::NonNullable) { + return Ok(Self::NonNullable); + } + // From here on we know that the validity is nullable let source = match self { Validity::NonNullable => BoolArray::from(BitBuffer::new_set(len)), Validity::AllValid => BoolArray::from(BitBuffer::new_set(len)), @@ -326,13 +322,10 @@ impl Validity { None, )?; - Ok(Self::from_array( - source.patch(&patches, ctx)?.into_array(), - own_nullability, - )) + Ok(Self::Array(source.patch(&patches, ctx)?.into_array())) } - /// Convert into a nullable variant + /// Convert into a nullable variant. #[inline] pub fn into_nullable(self) -> Validity { match self { @@ -341,9 +334,13 @@ impl Validity { } } - /// Convert into a non-nullable variant + /// Convert into a non-nullable variant, computing statistics if necessary. + /// + /// Returns `None` when the array contains invalid values (so the cast cannot be performed), + /// either because it is [`Validity::AllInvalid`] or because the validity array's minimum is + /// `false`. #[inline] - pub fn into_non_nullable(self, len: usize) -> Option { + pub fn into_non_nullable(self, len: usize, ctx: &mut ExecutionCtx) -> Option { match self { _ if len == 0 => Some(Validity::NonNullable), Self::NonNullable => Some(Self::NonNullable), @@ -352,7 +349,7 @@ impl Validity { Self::Array(is_valid) => { is_valid .statistics() - .compute_min::() + .compute_min::(ctx) .vortex_expect("validity array must support min") .then(|| { // min true => all true @@ -362,37 +359,92 @@ impl Validity { } } - /// Convert into a variant compatible with the given nullability, if possible. + /// Convert into a non-nullable variant without running execution. + /// + /// This is the cheap counterpart to [`Self::into_non_nullable`]: it inspects already-computed + /// statistics rather than triggering execution. + /// + /// Return values: + /// - `Ok(Some(NonNullable))` — the cast is provably safe. + /// - `Ok(None)` — We need to perform compute to determine whether cast is valid. Callers should fall back to [`Self::into_non_nullable`], typically by + /// returning `Ok(None)` from a `CastReduce` rule so the corresponding `CastKernel` runs. + /// - `Err(_)` — we know the cast must fail (e.g. [`Validity::AllInvalid`]). + #[inline] + pub fn trivial_into_non_nullable(self, len: usize) -> VortexResult> { + match self { + _ if len == 0 => Ok(Some(Validity::NonNullable)), + Self::NonNullable => Ok(Some(Self::NonNullable)), + Self::AllValid => Ok(Some(Self::NonNullable)), + Self::AllInvalid => { + Err(vortex_err!(InvalidArgument: "Cannot cast AllInvalid to NonNullable")) + } + Self::Array(_) => Ok(None), + } + } + + /// Convert into a variant compatible with the given nullability. + /// + /// This is the execution-time half of the nullability-cast pair. It is paired with + /// [`Self::trivial_cast_nullability`], which is used by `CastReduce` rules. The pattern is: + /// + /// - **`CastReduce` rules** (metadata-only rewrites in the optimizer) call + /// [`Self::trivial_cast_nullability`]. If it returns `Ok(None)`, the rule returns `Ok(None)` + /// and the cast is deferred to execution. + /// - **`CastKernel` impls** (executed via [`ExecuteParentKernel`]) call this method, which + /// may run the underlying validity array to compute statistics. + /// + /// Returns `Err` when nullability cannot be cast (for example, casting to non-nullable while + /// invalid values are present). + /// + /// [`ExecuteParentKernel`]: crate::kernel::ExecuteParentKernel #[inline] - pub fn cast_nullability(self, nullability: Nullability, len: usize) -> VortexResult { + pub fn cast_nullability( + self, + nullability: Nullability, + len: usize, + ctx: &mut ExecutionCtx, + ) -> VortexResult { match nullability { - Nullability::NonNullable => self.into_non_nullable(len).ok_or_else(|| { + Nullability::NonNullable => self.into_non_nullable(len, ctx).ok_or_else(|| { vortex_err!(InvalidArgument: "Cannot cast array with invalid values to non-nullable type.") }), Nullability::Nullable => Ok(self.into_nullable()), } } - /// Create Validity by copying the given array's validity. - #[inline] - pub fn copy_from_array(array: &ArrayRef) -> VortexResult { - Ok(Validity::from_mask( - array.validity_mask()?, - array.dtype().nullability(), - )) - } - - /// Create Validity from boolean array with given nullability of the array. + /// Best-effort, non-executing variant of [`Self::cast_nullability`]. /// - /// Note: You want to pass the nullability of parent array and not the nullability of the validity array itself - /// as that is always nonnullable - fn from_array(value: ArrayRef, nullability: Nullability) -> Self { - if !matches!(value.dtype(), DType::Bool(Nullability::NonNullable)) { - vortex_panic!("Expected a non-nullable boolean array") - } + /// Use this from `CastReduce` rules — they run inside the optimizer where execution is not + /// available. The pairing with [`Self::cast_nullability`] is symmetric: every encoding that + /// implements `CastReduce` and inspects validity should also implement `CastKernel` so that + /// the harder cases (where statistics are not yet cached) can still be handled at execution + /// time. + /// + /// Return values: + /// - `Ok(Some(_))` — the cast is provably safe and the new [`Validity`] is returned. + /// - `Ok(None)` — the cast cannot be reduced cheaply (the `CastKernel` should be tried via + /// [`Self::cast_nullability`]). + /// - `Err(_)` — the cast is provably impossible. + /// + /// Typical usage inside a `CastReduce`: + /// + /// ```ignore + /// let Some(new_validity) = array + /// .validity()? + /// .trivial_cast_nullability(dtype.nullability(), array.len())? + /// else { + /// return Ok(None); + /// }; + /// ``` + #[inline] + pub fn trivial_cast_nullability( + self, + nullability: Nullability, + len: usize, + ) -> VortexResult> { match nullability { - Nullability::NonNullable => Self::NonNullable, - Nullability::Nullable => Self::Array(value), + Nullability::NonNullable => self.trivial_into_non_nullable(len), + Nullability::Nullable => Ok(Some(self.into_nullable())), } } @@ -404,15 +456,6 @@ impl Validity { Self::Array(a) => Some(a.len()), } } - - #[inline] - pub fn uncompressed_size(&self) -> usize { - if let Validity::Array(a) = self { - a.len().div_ceil(8) - } else { - 0 - } - } } impl From for Validity { @@ -460,6 +503,45 @@ impl From<&Nullability> for Validity { } } +impl Validity { + /// Concatenate one or more validities together. + /// + /// Returns None if the vector is empty. + pub fn concat(validities: Vec<(Validity, usize)>) -> Option { + let mut validity_kinds = validities + .iter() + .map(|(v, _)| std::mem::discriminant(v)) + .unique(); + let validity_kind = validity_kinds.next()?; + if validity_kinds.next().is_none() { + // If there is only one kind of validity and its not Validity::Array, avoid constructing + // a Validity::Array. + if validity_kind == std::mem::discriminant(&Validity::AllValid) { + return Some(Validity::AllValid); + } + if validity_kind == std::mem::discriminant(&Validity::AllInvalid) { + return Some(Validity::AllInvalid); + } + if validity_kind == std::mem::discriminant(&Validity::NonNullable) { + return Some(Validity::NonNullable); + } + } + + Some(Validity::Array( + unsafe { + ChunkedArray::new_unchecked( + validities + .into_iter() + .map(|(v, len)| v.to_array(len)) + .collect(), + DType::Bool(Nullability::NonNullable), + ) + } + .into_array(), + )) + } +} + impl Validity { pub fn from_bit_buffer(buffer: BitBuffer, nullability: Nullability) -> Self { if buffer.true_count() == buffer.len() { @@ -588,13 +670,7 @@ mod tests { assert!( validity - .patch( - len, - 0, - &indices, - &patches, - &mut LEGACY_SESSION.create_execution_ctx(), - ) + .patch(len, 0, &indices, &patches, &mut ctx,) .unwrap() .mask_eq(&expected, &mut ctx) .unwrap() @@ -604,13 +680,14 @@ mod tests { #[test] #[should_panic] fn out_of_bounds_patch() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); Validity::NonNullable .patch( 2, 0, &buffer![4].into_array(), &Validity::AllInvalid, - &mut LEGACY_SESSION.create_execution_ctx(), + &mut ctx, ) .unwrap(); } diff --git a/vortex-array/src/variants.rs b/vortex-array/src/variants.rs index f60eea316a9..ca64d1ccffb 100644 --- a/vortex-array/src/variants.rs +++ b/vortex-array/src/variants.rs @@ -100,7 +100,9 @@ impl ArrayRef { .clone() .fill_null(Scalar::bool(false, self.dtype().nullability()))?; - Ok(array.execute::(ctx)?.to_mask_fill_null_false()) + Ok(array + .execute::(ctx)? + .to_mask_fill_null_false(ctx)) } } @@ -133,7 +135,7 @@ impl PrimitiveTyped<'_> { /// Return the primitive value at the given index. pub fn value(&self, idx: usize) -> VortexResult> { self.0 - .is_valid(idx)? + .is_valid(idx, &mut LEGACY_SESSION.create_execution_ctx())? .then(|| self.value_unchecked(idx)) .transpose() } @@ -142,7 +144,7 @@ impl PrimitiveTyped<'_> { pub fn value_unchecked(&self, idx: usize) -> VortexResult { Ok(self .0 - .scalar_at(idx)? + .execute_scalar(idx, &mut LEGACY_SESSION.create_execution_ctx())? .as_primitive() .pvalue() .unwrap_or_else(|| PValue::zero(&self.ptype()))) @@ -163,7 +165,10 @@ impl IndexOrd> for PrimitiveTyped<'_> { // TODO(ngates): add generics to the `value` function and implement this over T. impl IndexOrd for PrimitiveTyped<'_> { fn index_cmp(&self, idx: usize, elem: &PValue) -> VortexResult> { - assert!(self.0.all_valid()?); + assert!( + self.0 + .all_valid(&mut LEGACY_SESSION.create_execution_ctx())? + ); let value = self.value_unchecked(idx)?; Ok(value.partial_cmp(elem)) } diff --git a/vortex-bench/Cargo.toml b/vortex-bench/Cargo.toml index 62d302f12e1..2e1e1a9a423 100644 --- a/vortex-bench/Cargo.toml +++ b/vortex-bench/Cargo.toml @@ -17,6 +17,14 @@ version = { workspace = true } workspace = true [dependencies] +vortex = { workspace = true, features = [ + "object_store", + "files", + "tokio", + "zstd", +] } +vortex-tensor = { workspace = true } # TODO(connor): In the future, this might be inside vortex. + anyhow = { workspace = true } arrow-array = { workspace = true } arrow-schema = { workspace = true } @@ -53,13 +61,11 @@ tracing = { workspace = true } tracing-perfetto = { workspace = true } tracing-subscriber = { workspace = true, features = [ "env-filter", + "json", "tracing-log", ] } url = { workspace = true } uuid = { workspace = true, features = ["v4"] } -vortex = { workspace = true, features = [ - "object_store", - "files", - "tokio", - "zstd", -] } + +[features] +unstable_encodings = ["vortex/unstable_encodings"] diff --git a/vortex-bench/src/bin/data-gen.rs b/vortex-bench/src/bin/data-gen.rs index d38c977a6c6..35c77d70c48 100644 --- a/vortex-bench/src/bin/data-gen.rs +++ b/vortex-bench/src/bin/data-gen.rs @@ -17,12 +17,13 @@ use vortex_bench::Benchmark; use vortex_bench::BenchmarkArg; use vortex_bench::CompactionStrategy; use vortex_bench::Format; +use vortex_bench::LogFormat; use vortex_bench::Opt; use vortex_bench::Opts; use vortex_bench::conversions::convert_parquet_directory_to_vortex; use vortex_bench::create_benchmark; use vortex_bench::generate_duckdb_registration_sql; -use vortex_bench::setup_logging_and_tracing; +use vortex_bench::setup_logging_and_tracing_with_format; #[derive(Parser)] #[command(name = "bench-data-gen")] @@ -37,6 +38,11 @@ struct Args { #[arg(long)] tracing: bool, + /// Format for the primary stderr log sink. `text` is the default human-readable format; + /// `json` emits one JSON object per event, suitable for piping into `jq`. + #[arg(long, value_enum, default_value_t = LogFormat::Text)] + log_format: LogFormat, + #[arg(long, value_delimiter = ',', value_parser = value_parser!(Format))] formats: Vec, @@ -49,7 +55,7 @@ async fn main() -> anyhow::Result<()> { let args = Args::parse(); let opts = Opts::from(args.options); - setup_logging_and_tracing(args.verbose, args.tracing)?; + setup_logging_and_tracing_with_format(args.verbose, args.tracing, args.log_format)?; let benchmark = create_benchmark(args.benchmark, &opts)?; diff --git a/vortex-bench/src/clickbench/benchmark.rs b/vortex-bench/src/clickbench/benchmark.rs index 5e14cbcf40e..a68ce0b54e8 100644 --- a/vortex-bench/src/clickbench/benchmark.rs +++ b/vortex-bench/src/clickbench/benchmark.rs @@ -6,15 +6,14 @@ use std::fs; use std::path::Path; use anyhow::Result; -use reqwest::Client; use url::Url; -use vortex::error::VortexExpect; use crate::Benchmark; use crate::BenchmarkDataset; use crate::IdempotentPath; use crate::TableSpec; use crate::clickbench::*; +use crate::utils::file::resolve_data_url; /// ClickBench benchmark implementation pub struct ClickBenchBenchmark { @@ -29,7 +28,7 @@ impl ClickBenchBenchmark { queries_file: Option, use_remote_data_dir: Option, ) -> Result { - let url = Self::create_data_url(&use_remote_data_dir, flavor)?; + let url = Self::create_data_url(use_remote_data_dir.as_deref(), flavor)?; Ok(Self { flavor, queries_file, @@ -37,32 +36,8 @@ impl ClickBenchBenchmark { }) } - fn create_data_url(remote_data_dir: &Option, flavor: Flavor) -> Result { - match remote_data_dir { - None => { - let basepath = format!("clickbench_{flavor}").to_data_path(); - Ok(Url::parse(&format!( - "file:{}/", - basepath.to_str().vortex_expect("path should be utf8") - ))?) - } - Some(remote_data_dir) => { - if !remote_data_dir.ends_with("/") { - tracing::warn!( - "Supply a --use-remote-data-dir argument which ends in a slash e.g. s3://vortex-bench-dev-eu/parquet/" - ); - } - tracing::info!( - concat!( - "Assuming data already exists at this remote (e.g. S3, GCS) URL: {}.\\n", - "If it does not, you should kill this command, locally generate the files (by running without\\n", - "--use-remote-data-dir) and upload data/clickbench/ to some remote location.", - ), - remote_data_dir, - ); - Ok(Url::parse(remote_data_dir)?) - } - } + fn create_data_url(remote_data_dir: Option<&str>, flavor: Flavor) -> Result { + resolve_data_url(remote_data_dir, &format!("clickbench_{flavor}")) } } @@ -89,7 +64,7 @@ impl Benchmark for ClickBenchBenchmark { } let basepath = clickbench_flavor(self.flavor).to_data_path(); - self.flavor.download(Client::default(), basepath).await?; + self.flavor.download(basepath).await?; Ok(()) } diff --git a/vortex-bench/src/clickbench/data.rs b/vortex-bench/src/clickbench/data.rs index c0ac621a6b4..ba2a2104192 100644 --- a/vortex-bench/src/clickbench/data.rs +++ b/vortex-bench/src/clickbench/data.rs @@ -6,31 +6,22 @@ use std::fmt::Display; use std::path::Path; use std::str::FromStr; use std::sync::LazyLock; -use std::time::Duration; use arrow_schema::DataType; use arrow_schema::Field; use arrow_schema::Schema; use arrow_schema::TimeUnit; -use bytes::Bytes; use clap::ValueEnum; -use futures::StreamExt; -use indicatif::ProgressBar; -use indicatif::ProgressStyle; -use reqwest::IntoUrl; use serde::Deserialize; use serde::Serialize; -use tokio::fs::File; -use tokio::io::AsyncWriteExt; -use tokio::task::JoinSet; use tracing::info; -use tracing::warn; use vortex::error::VortexExpect; use crate::Format; // Re-export for use by clickbench_benchmark pub use crate::conversions::convert_parquet_directory_to_vortex; -use crate::idempotent_async; +use crate::datasets::data_downloads::download_data; +use crate::datasets::data_downloads::download_many; pub static HITS_SCHEMA: LazyLock = LazyLock::new(|| { use DataType::*; @@ -190,110 +181,28 @@ impl Display for Flavor { impl Flavor { // TODO(joe): move these elsewhere. - pub async fn download( - &self, - client: reqwest::Client, - basepath: impl AsRef, - ) -> anyhow::Result<()> { + pub async fn download(&self, basepath: impl AsRef) -> anyhow::Result<()> { let basepath = basepath.as_ref(); match self { Flavor::Single => { let output_path = basepath.join(Format::Parquet.name()).join("hits.parquet"); - idempotent_async(output_path.as_path(), |output_path| async move { - info!("Downloading single clickbench file"); - let url = "https://pub-3ba949c0f0354ac18db1f0f14f0a2c52.r2.dev/clickbench/parquet_single/hits.parquet"; - download_large_file(&client, url, &output_path).await?; - anyhow::Ok(()) - }) - .await?; + info!("Downloading single clickbench file"); + let url = "https://pub-3ba949c0f0354ac18db1f0f14f0a2c52.r2.dev/clickbench/parquet_single/hits.parquet"; + download_data(output_path, url).await?; } Flavor::Partitioned => { // The clickbench-provided file is missing some higher-level type info, so we reprocess it // to add that info, see https://github.com/ClickHouse/ClickBench/issues/7. - - let mut tasks = (0_u32..100).map(|idx| { - let output_path = basepath.join(Format::Parquet.name()).join(format!("hits_{idx}.parquet")); - let client = client.clone(); - - idempotent_async(output_path, move|output_path| async move { - info!("Downloading file {idx}"); - let url = format!("https://pub-3ba949c0f0354ac18db1f0f14f0a2c52.r2.dev/clickbench/parquet_many/hits_{idx}.parquet"); - let body = retry_get(&client, url).await?; - let mut file = File::create(output_path).await?; - file.write_all(&body).await?; - - anyhow::Ok(()) - }) - }).collect::>(); - - while let Some(task) = tasks.join_next().await { - task??; - } + info!("Downloading 100 ClickBench parquet shards"); + let parquet_dir = basepath.join(Format::Parquet.name()); + let downloads = (0_u32..100).map(|idx| { + let output_path = parquet_dir.join(format!("hits_{idx}.parquet")); + let url = format!("https://pub-3ba949c0f0354ac18db1f0f14f0a2c52.r2.dev/clickbench/parquet_many/hits_{idx}.parquet"); + (output_path, url) + }); + download_many(downloads).await?; } } Ok(()) } } - -/// Downloads a large file with streaming and progress indication. -async fn download_large_file( - client: &reqwest::Client, - url: &str, - output_path: &Path, -) -> anyhow::Result<()> { - let response = client.get(url).send().await?.error_for_status()?; - - let total_size = response.content_length().unwrap_or(0); - - let progress = ProgressBar::new(total_size); - progress.set_style( - ProgressStyle::with_template( - "[{elapsed_precise}] {bar:40.cyan/blue} {bytes}/{total_bytes} ({bytes_per_sec})", - ) - .expect("valid template"), - ); - - let mut file = File::create(output_path).await?; - let mut stream = response.bytes_stream(); - - while let Some(chunk) = stream.next().await { - let chunk = chunk?; - file.write_all(&chunk).await?; - progress.inc(chunk.len() as u64); - } - - progress.finish(); - Ok(()) -} - -async fn retry_get(client: &reqwest::Client, url: impl IntoUrl) -> anyhow::Result { - let url = url.as_str(); - let make_req = || async { client.get(url).send().await }; - - let mut body = None; - - for attempt in 0..3 { - match make_req().await.and_then(|r| r.error_for_status()) { - Ok(r) => match r.bytes().await { - Ok(b) => { - body = Some(b); - break; - } - Err(e) => { - warn!("Request errored with {e}, retying for the {attempt} time"); - } - }, - Err(e) => { - warn!("Request errored with {e}, retying for the {attempt} time"); - } - } - - // Very basic backoff mechanism - tokio::time::sleep(Duration::from_secs(attempt + 1)).await; - } - - match body { - Some(v) => Ok(v), - None => anyhow::bail!("Exahusted retry attempts for {url}"), - } -} diff --git a/vortex-bench/src/datasets/data_downloads.rs b/vortex-bench/src/datasets/data_downloads.rs index e846edbeaec..ae1281f8366 100644 --- a/vortex-bench/src/datasets/data_downloads.rs +++ b/vortex-bench/src/datasets/data_downloads.rs @@ -1,84 +1,763 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +use std::cmp::Ordering; use std::fs::File; -use std::io::Read; -use std::io::Write; +use std::io; +use std::path::Path; use std::path::PathBuf; +use std::sync::Arc; +use std::sync::LazyLock; +use std::sync::atomic::AtomicBool; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::AtomicUsize; +use std::sync::atomic::Ordering as AtomicOrdering; use std::time::Duration; +use std::time::Instant; use anyhow::Context; +use anyhow::Error; use anyhow::Result; use bzip2::read::BzDecoder; +use futures::StreamExt; +use futures::stream; +use indicatif::MultiProgress; +use indicatif::ProgressBar; +use indicatif::ProgressStyle; +use parking_lot::Mutex; use reqwest::Client; use tokio::fs::File as TokioFile; use tokio::io::AsyncWriteExt; +use tokio::sync::OwnedSemaphorePermit; +use tokio::sync::Semaphore; use tracing::info; +use tracing::warn; use crate::utils::file::idempotent; use crate::utils::file::idempotent_async; -pub async fn download_data(fname: PathBuf, data_url: &str) -> Result { - idempotent_async(&fname, async |path| { - info!( - "Downloading {} from {}", - fname.to_str().context("Failed to convert path to string")?, - data_url - ); - let mut file = TokioFile::create(path) - .await - .context("Failed to create file")?; - let mut response = Client::builder() - .read_timeout(Duration::from_secs(60)) - .timeout(Duration::from_secs(60 * 15)) - .build() - .context("Failed to build HTTP client")? - .get(data_url) - .send() - .await - .context("Failed to send HTTP request")? - .error_for_status() - .context("HTTP request returned error status")?; +//////////////////////////////////////////////////////////////////////////////////////////////////// +// Public API +//////////////////////////////////////////////////////////////////////////////////////////////////// - while let Some(chunk) = response - .chunk() - .await - .context("Failed to read response chunk")? - { - AsyncWriteExt::write_all(&mut file, &chunk) - .await - .context("Failed to write to file")?; - } - Ok(()) - }) - .await +/// Anything that can be described as a `(target_path, url)` pair accepted by +/// [`download_many`]. +pub trait IntoDownload { + fn into_download(self) -> (PathBuf, String); +} + +impl IntoDownload for (P, S) +where + P: Into, + S: Into, +{ + fn into_download(self) -> (PathBuf, String) { + (self.0.into(), self.1.into()) + } +} + +/// Idempotently download a single URL to `fname`. +/// +/// Uses the shared HTTP client, a 3-attempt exponential backoff retry loop with jitter, +/// and an [`indicatif::ProgressBar`]. If `fname` already exists, the download is +/// skipped. +#[tracing::instrument(skip_all, fields(url = %data_url.as_ref(), path = %fname.display()))] +pub async fn download_data(fname: PathBuf, data_url: impl AsRef) -> Result { + download_one(fname, data_url.as_ref(), None).await +} + +/// Idempotently download many `(path, url)` pairs with adaptive parallelism. +/// +/// This is the preferred way to fetch multi-shard datasets (ClickBench partitioned, +/// vector dataset train shards, Public BI tables, etc.) because it: +/// +/// - starts at `INITIAL_IN_FLIGHT` concurrent downloads and ramps up to +/// `MAX_IN_FLIGHT` as clean completions come in (TCP-style slow-start), then +/// halves on retries to back off from upstream rate limits, +/// - reuses the shared HTTP client across every shard, +/// - renders a top-of-block `N/total` bar plus a fixed number of reusable slot bars via +/// a shared [`MultiProgress`]: the terminal block size stays constant for the entire +/// run, so nothing "jumps" as shards cycle, +/// - short-circuits on the first error (the remaining in-flight downloads are dropped +/// when the returned future is dropped), +/// - returns the resolved on-disk paths in completion order (not submission order). +#[tracing::instrument(skip_all, fields(count = tracing::field::Empty))] +pub async fn download_many(downloads: I) -> Result> +where + I: IntoIterator, + I::Item: IntoDownload, +{ + let downloads: Vec<(PathBuf, String)> = downloads + .into_iter() + .map(IntoDownload::into_download) + .collect(); + tracing::Span::current().record("count", downloads.len()); + + if downloads.is_empty() { + return Ok(Vec::new()); + } + + let num_slots = downloads.len().min(MAX_IN_FLIGHT); + let initial_in_flight = INITIAL_IN_FLIGHT.min(num_slots); + let batch = BatchProgress::new(downloads.len() as u64, num_slots, initial_in_flight); + + let results: Vec> = stream::iter(downloads) + .map(|(path, url)| { + let batch = batch.clone(); + async move { + let result = download_one(path, &url, Some(&batch)).await; + if result.is_ok() { + batch.advance(); + } + result + } + }) + .buffer_unordered(num_slots) + .collect() + .await; + + batch.finish(); + + results.into_iter().collect() } +/// Idempotently decompress a bzip2 file into `output_path`, streaming the decompressed bytes +/// straight to disk so memory stays bounded. +/// +/// This is used for the public BI dataset. +#[tracing::instrument(skip_all, fields(input = %input_path.display(), output = %output_path.display()))] pub fn decompress_bz2(input_path: PathBuf, output_path: PathBuf) -> Result { idempotent(&output_path, |path| { info!( "Decompressing bzip from {} to {}", - input_path - .to_str() - .context("Failed to convert input path to string")?, - output_path - .to_str() - .context("Failed to convert output path to string")? + input_path.display(), + output_path.display() ); let input_file = File::open(&input_path) .with_context(|| format!("Failed to open input file: {:?}", input_path))?; let mut decoder = BzDecoder::new(input_file); - let mut buffer = Vec::new(); - decoder - .read_to_end(&mut buffer) - .context("Failed to decompress bzip2 data")?; - let mut output_file = File::create(path) .with_context(|| format!("Failed to create output file: {:?}", path))?; - output_file - .write_all(&buffer) - .context("Failed to write decompressed data")?; + io::copy(&mut decoder, &mut output_file).context("Failed to decompress bzip2 stream")?; Ok(output_path.clone()) }) } + +//////////////////////////////////////////////////////////////////////////////////////////////////// +// Shared HTTP client +//////////////////////////////////////////////////////////////////////////////////////////////////// + +/// Shared HTTP client used by every dataset download. +/// +/// Reusing a single client gives us connection pooling, DNS caching, and consistent +/// timeouts across all callers. Each benchmark used to build its own +/// [`reqwest::Client`] on every download, which both wasted TLS handshakes and made it +/// hard to reason about total in-flight concurrency. +static HTTP_CLIENT: LazyLock = LazyLock::new(|| { + Client::builder() + .read_timeout(Duration::from_secs(60)) + .timeout(Duration::from_secs(60 * 15)) + .build() + .expect("failed to build shared benchmark HTTP client") +}); + +//////////////////////////////////////////////////////////////////////////////////////////////////// +// Progress-bar templates +//////////////////////////////////////////////////////////////////////////////////////////////////// + +/// Template for a slot that currently holds no download. `{msg}` with an empty message +/// renders as a blank line while still reserving the terminal row. +const IDLE_TEMPLATE: &str = "{msg}"; + +/// Template for a slot that has acquired a download but has not yet received response +/// headers. Shows the filename plus an animated spinner so the user can see we are +/// still making forward progress during TLS + request + first-byte latency. +const CONNECTING_TEMPLATE: &str = "{prefix:>28!} {spinner} connecting..."; + +/// Template for an active download when the response advertised a `Content-Length`. +const KNOWN_SIZE_TEMPLATE: &str = + "{prefix:>28!} [{bar:30.cyan/blue}] {bytes:>9}/{total_bytes:>9} ({bytes_per_sec})"; + +/// Template for an active download when the response size is unknown. +const UNKNOWN_SIZE_TEMPLATE: &str = "{prefix:>28!} {spinner} {bytes} ({bytes_per_sec})"; + +/// Template for the top-of-block summary bar rendered by [`download_many`]. `{pos}/{len}` +/// tracks completed-of-total; `{msg}` is updated on every slot acquire / release to show +/// how many downloads are currently in flight. +const SHARDS_TEMPLATE: &str = + "[{elapsed_precise}] shards [{bar:30.green/white}] {pos}/{len} {msg}"; + +/// How often slot spinners redraw. Fast enough to feel alive; slow enough that stderr +/// writes sneaking past `MultiProgress` do not constantly fight for cursor position. +const SLOT_TICK: Duration = Duration::from_millis(80); + +//////////////////////////////////////////////////////////////////////////////////////////////////// +// Dynamic concurrency controller +//////////////////////////////////////////////////////////////////////////////////////////////////// + +/// Number of in-flight downloads to start at. Matches TCP-style slow-start: start small so +/// we don't hammer the upstream on the very first connection, and double from there. +const INITIAL_IN_FLIGHT: usize = 4; + +/// Upper bound on the number of concurrent downloads the controller can ramp up to, and +/// the number of slot rows pre-allocated in the [`MultiProgress`] block. Chosen large +/// enough that the retry-based controller is the effective ceiling, not this constant. +/// The trade-off is that on large batches the MP block will exceed a typical local +/// terminal height — indicatif handles this by drawing the most recent rows plus the +/// top shards bar — but on CI there is no TTY so the visual overflow does not apply. +const MAX_IN_FLIGHT: usize = 256; + +/// Never let the controller drive concurrency below this floor on a flaky network. +/// A value of `1` means the fallback is serial downloads. +const MIN_IN_FLIGHT: usize = 1; + +/// Minimum time between successive halves, in milliseconds. Coalesces simultaneous +/// retries from one upstream hiccup into a single reaction, preventing over-halving. +const HALVE_COOLDOWN_MS: u64 = 1000; + +/// Decide the next in-flight limit after a clean (no-retry) download completes. +/// +/// Returns `Some(new_limit)` if the limit should change, or `None` if it is already at +/// the cap or the computed move would be a no-op. +fn decide_on_success(current: usize, in_slow_start: bool) -> Option { + if current >= MAX_IN_FLIGHT { + return None; + } + let new = if in_slow_start { + current.saturating_mul(2) + } else { + current.saturating_add(1) + } + .min(MAX_IN_FLIGHT); + (new != current).then_some(new) +} + +/// Decide the next in-flight limit after a failed download attempt. +/// +/// Returns `Some(new_limit)` if the limit should be halved now. Returns `None` if the +/// halve is debounced (another halve fired within [`HALVE_COOLDOWN_MS`]) or we are +/// already at the [`MIN_IN_FLIGHT`] floor. +fn decide_on_retry(current: usize, now_ms: u64, last_halve_ms: u64) -> Option { + if now_ms.saturating_sub(last_halve_ms) < HALVE_COOLDOWN_MS { + return None; + } + if current <= MIN_IN_FLIGHT { + return None; + } + let new = (current / 2).max(MIN_IN_FLIGHT); + (new != current).then_some(new) +} + +//////////////////////////////////////////////////////////////////////////////////////////////////// +// Batch download internals +//////////////////////////////////////////////////////////////////////////////////////////////////// + +/// Shared rendering state for a batched download. +/// +/// Layout is built once at construction: the shards bar is registered first (top row), +/// then `num_slots` slot bars are registered below it. None of these are ever added to +/// or removed from the [`MultiProgress`] again. The block size on the terminal is +/// exactly `num_slots + 1` rows for the entire run of [`download_many`]. +/// +/// Per-download lifecycle reuses a single slot bar by swapping its style between an +/// idle placeholder and the active progress-bar / spinner variants. A +/// [`tokio::sync::Semaphore`] gates how many slots can be in use at any instant, which +/// lets a future controller adjust concurrency at runtime via +/// [`BatchProgress::set_max_in_flight`] without ever touching the MP layout. +#[derive(Clone)] +struct BatchProgress { + inner: Arc, +} + +struct BatchInner { + shards_bar: ProgressBar, + free: Mutex>, + in_flight: Arc, + /// Current concurrency limit — the source of truth read by the controller and + /// written via [`BatchProgress::set_max_in_flight`]. + current_in_flight: AtomicUsize, + num_slots: usize, + /// Controller state: are we still in slow-start (double on success) or have we + /// dropped into additive-increase (`+=1` on success) after the first retry? + in_slow_start: AtomicBool, + /// Millis since [`BatchInner::created_at`] of the most recent halve event, used to + /// debounce bursts of retries from a single upstream hiccup. + last_halve_at_ms: AtomicU64, + created_at: Instant, + // The MP is kept alive alongside the Arc so bars stay registered and rendered. + // Once the last BatchProgress clone drops, the MP drops and clears the block. + _mp: MultiProgress, +} + +impl BatchInner { + fn elapsed_ms(&self) -> u64 { + u64::try_from(self.created_at.elapsed().as_millis()).unwrap_or(u64::MAX) + } + + /// Refresh the shards bar message to reflect the current in-flight count. Called on + /// every slot acquire / release and from `set_max_in_flight`. + fn refresh_shards_message(&self) { + let active = self.num_slots - self.free.lock().len(); + let limit = self.current_in_flight.load(AtomicOrdering::Relaxed); + self.shards_bar + .set_message(format!("({active} active, limit {limit})")); + } +} + +impl BatchProgress { + fn new(total: u64, num_slots: usize, initial_in_flight: usize) -> Self { + let initial_in_flight = initial_in_flight.min(num_slots); + let mp = MultiProgress::new(); + + let shards_bar = mp.add(ProgressBar::new(total)); + shards_bar + .set_style(ProgressStyle::with_template(SHARDS_TEMPLATE).expect("valid template")); + + let idle_style = ProgressStyle::with_template(IDLE_TEMPLATE).expect("valid template"); + let mut slots = Vec::with_capacity(num_slots); + for _ in 0..num_slots { + let bar = mp.add(ProgressBar::new(0)); + bar.set_style(idle_style.clone()); + bar.set_message(""); + bar.enable_steady_tick(SLOT_TICK); + slots.push(bar); + } + + let inner = BatchInner { + shards_bar, + free: Mutex::new(slots), + in_flight: Arc::new(Semaphore::new(initial_in_flight)), + current_in_flight: AtomicUsize::new(initial_in_flight), + num_slots, + in_slow_start: AtomicBool::new(true), + last_halve_at_ms: AtomicU64::new(0), + created_at: Instant::now(), + _mp: mp, + }; + inner.refresh_shards_message(); + Self { + inner: Arc::new(inner), + } + } + + /// Wait for an in-flight permit, then claim an idle slot and switch it to the + /// [`CONNECTING_TEMPLATE`] style. The returned guard releases both the permit and + /// the slot on drop. + async fn acquire(&self, prefix: &str) -> SlotGuard { + let permit = Arc::clone(&self.inner.in_flight) + .acquire_owned() + .await + .expect("batch semaphore is never closed while a download is in flight"); + let bar = self + .inner + .free + .lock() + .pop() + .expect("slot free list invariant broken: permits outnumber pre-allocated slots"); + bar.set_style(ProgressStyle::with_template(CONNECTING_TEMPLATE).expect("valid template")); + bar.set_prefix(prefix.to_owned()); + bar.set_message(""); + bar.set_length(0); + bar.reset(); + self.inner.refresh_shards_message(); + SlotGuard { + bar, + owner: Arc::clone(&self.inner), + _permit: permit, + } + } + + fn advance(&self) { + self.inner.shards_bar.inc(1); + } + + fn finish(&self) { + self.inner.shards_bar.finish_and_clear(); + } + + /// Called when a download completed on its first attempt (no retries). Drives the + /// slow-start / additive-increase side of AIMD. + fn report_clean_success(&self) { + let current = self.inner.current_in_flight.load(AtomicOrdering::Relaxed); + let in_slow_start = self.inner.in_slow_start.load(AtomicOrdering::Relaxed); + if let Some(new) = decide_on_success(current, in_slow_start) { + self.set_max_in_flight(new); + } + } + + /// Called when a download attempt failed. Drives the halving side of AIMD, with an + /// internal cooldown so a burst of simultaneous retries from one upstream hiccup + /// halves the limit at most once. + fn report_retry(&self) { + let now_ms = self.inner.elapsed_ms(); + let current = self.inner.current_in_flight.load(AtomicOrdering::Relaxed); + let last_halve_ms = self.inner.last_halve_at_ms.load(AtomicOrdering::Relaxed); + if let Some(new) = decide_on_retry(current, now_ms, last_halve_ms) { + self.inner + .in_slow_start + .store(false, AtomicOrdering::Relaxed); + self.inner + .last_halve_at_ms + .store(now_ms, AtomicOrdering::Relaxed); + self.set_max_in_flight(new); + } + } + + /// Adjust how many downloads may run concurrently. Clamped to the pre-allocated + /// slot count. Raising the limit returns immediately; lowering spawns a background + /// task that acquires and forgets the delta so the limit takes effect as active + /// downloads complete naturally, never interrupting an in-flight transfer. + fn set_max_in_flight(&self, target: usize) { + let target = target.min(self.inner.num_slots); + let prev = self + .inner + .current_in_flight + .swap(target, AtomicOrdering::Relaxed); + match target.cmp(&prev) { + Ordering::Greater => { + self.inner.in_flight.add_permits(target - prev); + } + Ordering::Less => { + let delta = u32::try_from(prev - target).expect("delta fits in u32"); + let sem = Arc::clone(&self.inner.in_flight); + tokio::spawn(async move { + if let Ok(permit) = sem.acquire_many_owned(delta).await { + permit.forget(); + } + }); + } + Ordering::Equal => {} + } + self.inner.refresh_shards_message(); + } +} + +/// RAII handle for a borrowed slot bar. Drives the bar through its active lifecycle and +/// resets it back to the idle placeholder on drop. +struct SlotGuard { + bar: ProgressBar, + owner: Arc, + _permit: OwnedSemaphorePermit, +} + +impl SlotGuard { + fn activate_known(&self, total: u64) { + self.bar + .set_style(ProgressStyle::with_template(KNOWN_SIZE_TEMPLATE).expect("valid template")); + self.bar.set_length(total); + self.bar.reset(); + } + + fn activate_unknown(&self) { + self.bar.set_style( + ProgressStyle::with_template(UNKNOWN_SIZE_TEMPLATE).expect("valid template"), + ); + self.bar.set_length(0); + self.bar.reset(); + } + + fn inc(&self, n: u64) { + self.bar.inc(n); + } + + fn reset_for_retry(&self) { + self.bar + .set_style(ProgressStyle::with_template(CONNECTING_TEMPLATE).expect("valid template")); + self.bar.set_length(0); + self.bar.reset(); + } +} + +impl Drop for SlotGuard { + fn drop(&mut self) { + let bar = self.bar.clone(); + bar.set_style(ProgressStyle::with_template(IDLE_TEMPLATE).expect("valid template")); + bar.set_prefix(""); + bar.set_message(""); + bar.set_length(0); + bar.reset(); + self.owner.free.lock().push(bar); + self.owner.refresh_shards_message(); + } +} + +//////////////////////////////////////////////////////////////////////////////////////////////////// +// Core download implementation +//////////////////////////////////////////////////////////////////////////////////////////////////// + +/// Core download implementation shared by [`download_data`] and [`download_many`]. +/// +/// When `batch` is `Some`, the download reuses one of the pre-allocated slot bars from +/// the batch's [`MultiProgress`]. When `batch` is `None` the download renders its own +/// standalone bar. +async fn download_one(fname: PathBuf, url: &str, batch: Option<&BatchProgress>) -> Result { + let display_name = fname + .file_name() + .and_then(|n| n.to_str()) + .unwrap_or("") + .to_owned(); + idempotent_async(&fname, async move |tmp_path| { + retry_get(&HTTP_CLIENT, url, &tmp_path, &display_name, batch).await + }) + .await +} + +/// Perform an HTTP GET into `tmp_path`, retrying up to three times with exponential +/// backoff and a small jitter to avoid lockstep retries across concurrent shards. A +/// partial temp file from an exhausted retry loop is removed before returning the final +/// error. +/// +/// When `batch` is `Some`, a pre-allocated slot bar is reused across retries. When +/// `batch` is `None`, a standalone [`ProgressBar`] is created and cleared at the end. +async fn retry_get( + client: &Client, + url: &str, + tmp_path: &Path, + display_name: &str, + batch: Option<&BatchProgress>, +) -> Result<()> { + const MAX_ATTEMPTS: u32 = 3; + let progress = DownloadProgress::new(batch, display_name).await; + let mut last_err: Option = None; + + for attempt in 0..MAX_ATTEMPTS { + if attempt > 0 { + progress.reset_for_retry(); + } + match single_attempt(client, url, tmp_path, &progress).await { + Ok(()) => { + if attempt == 0 + && let Some(b) = batch + { + b.report_clean_success(); + } + progress.finalize(); + return Ok(()); + } + Err(e) => { + if let Some(b) = batch { + b.report_retry(); + } + last_err = Some(e); + } + } + if attempt + 1 < MAX_ATTEMPTS { + sleep_with_jitter(attempt).await; + } + } + + progress.finalize(); + cleanup_partial_temp(tmp_path); + Err(last_err.unwrap_or_else(|| anyhow::anyhow!("retry_get exhausted with no recorded error"))) +} + +/// Perform one download attempt end to end: create the temp file, issue the GET, stream +/// bytes to disk while advancing the progress bar. Returns on the first error so the +/// retry loop in [`retry_get`] can decide whether to try again. +async fn single_attempt( + client: &Client, + url: &str, + tmp_path: &Path, + progress: &DownloadProgress, +) -> Result<()> { + let mut file = TokioFile::create(tmp_path) + .await + .context("Failed to create file")?; + let response = client + .get(url) + .send() + .await + .context("Failed to send HTTP request")? + .error_for_status() + .context("HTTP request returned error status")?; + + progress.activate(response.content_length()); + + let mut stream = response.bytes_stream(); + while let Some(chunk) = stream.next().await { + let chunk = chunk?; + AsyncWriteExt::write_all(&mut file, &chunk) + .await + .context("Failed to write to file")?; + progress.inc(chunk.len() as u64); + } + + AsyncWriteExt::flush(&mut file).await?; + Ok(()) +} + +/// Sleep `2^attempt` seconds plus 0-500 ms of jitter before the next retry. +async fn sleep_with_jitter(attempt: u32) { + let jitter = Duration::from_millis(rand::random::() % 500); + let backoff = Duration::from_secs(1u64 << attempt) + jitter; + warn!( + "download attempt {} failed; retrying in {:?}", + attempt + 1, + backoff + ); + tokio::time::sleep(backoff).await; +} + +/// Best-effort removal of a partial temp file left behind when every retry attempt +/// failed. The UUID-named temp lives under `target/`; leaking it would be mostly +/// harmless but adds up over many CI runs. +fn cleanup_partial_temp(tmp_path: &Path) { + if let Err(err) = std::fs::remove_file(tmp_path) { + warn!( + "failed to remove leftover temp download {}: {}", + tmp_path.display(), + err + ); + } +} + +/// Unified progress handle for a single download. Hides the split between pooled slot +/// bars (batched path) and one-off standalone bars (single-download path) so +/// [`retry_get`] does not have to branch on every update. +enum DownloadProgress { + Slot(SlotGuard), + Standalone(ProgressBar), +} + +impl DownloadProgress { + async fn new(batch: Option<&BatchProgress>, display_name: &str) -> Self { + match batch { + Some(b) => Self::Slot(b.acquire(display_name).await), + None => Self::Standalone(new_standalone_bar(display_name)), + } + } + + fn reset_for_retry(&self) { + match self { + Self::Slot(s) => s.reset_for_retry(), + Self::Standalone(bar) => { + bar.set_style( + ProgressStyle::with_template(CONNECTING_TEMPLATE).expect("valid template"), + ); + bar.set_length(0); + bar.reset(); + } + } + } + + fn activate(&self, content_length: Option) { + match (self, content_length) { + (Self::Slot(s), Some(total)) => s.activate_known(total), + (Self::Slot(s), None) => s.activate_unknown(), + (Self::Standalone(bar), Some(total)) => { + bar.set_style( + ProgressStyle::with_template(KNOWN_SIZE_TEMPLATE).expect("valid template"), + ); + bar.set_length(total); + bar.reset(); + } + (Self::Standalone(bar), None) => { + bar.set_style( + ProgressStyle::with_template(UNKNOWN_SIZE_TEMPLATE).expect("valid template"), + ); + bar.set_length(0); + bar.reset(); + } + } + } + + fn inc(&self, n: u64) { + match self { + Self::Slot(s) => s.inc(n), + Self::Standalone(bar) => bar.inc(n), + } + } + + /// Tear down any visible state. Standalone bars are explicitly cleared here; + /// slot bars clean themselves up when their [`SlotGuard`] drops. + fn finalize(&self) { + if let Self::Standalone(bar) = self { + bar.finish_and_clear(); + } + } +} + +fn new_standalone_bar(display_name: &str) -> ProgressBar { + let bar = ProgressBar::new(0); + bar.set_style(ProgressStyle::with_template(CONNECTING_TEMPLATE).expect("valid template")); + bar.set_prefix(display_name.to_owned()); + bar.enable_steady_tick(SLOT_TICK); + bar +} + +//////////////////////////////////////////////////////////////////////////////////////////////////// +// Tests +//////////////////////////////////////////////////////////////////////////////////////////////////// + +#[cfg(test)] +mod tests { + use super::*; + + const COOLDOWN_MS: u64 = HALVE_COOLDOWN_MS; + + #[test] + fn ramp_up_doubles_in_slow_start() { + // Start at INITIAL (4). Each clean success in slow-start doubles until MAX. + let mut cur = INITIAL_IN_FLIGHT; + let expected = [8, 16, 32, 64, 128, 256]; + for want in expected { + let next = decide_on_success(cur, true).expect("should ramp"); + assert_eq!(next, want); + cur = next; + } + // At MAX, further successes are no-ops. + assert_eq!(cur, MAX_IN_FLIGHT); + assert_eq!(decide_on_success(cur, true), None); + assert_eq!(decide_on_success(cur, false), None); + } + + #[test] + fn additive_increase_after_slow_start_exits() { + // Once out of slow-start, successes add 1 instead of doubling. + assert_eq!(decide_on_success(16, false), Some(17)); + assert_eq!(decide_on_success(17, false), Some(18)); + } + + #[test] + fn retry_halves() { + // At 64, a single retry (past the cooldown) halves to 32. + assert_eq!(decide_on_retry(64, COOLDOWN_MS + 1, 0), Some(32)); + assert_eq!(decide_on_retry(32, COOLDOWN_MS + 1, 0), Some(16)); + assert_eq!(decide_on_retry(2, COOLDOWN_MS + 1, 0), Some(1)); + } + + #[test] + fn halve_is_debounced() { + // Three retries at t=100, t=200, t=500 (all within the 1 s cooldown after the + // first halve at t=100) only produce one halve. + let last_halve = 100; + assert_eq!(decide_on_retry(64, 200, last_halve), None); + assert_eq!(decide_on_retry(64, 500, last_halve), None); + // A retry past the cooldown halves again. + assert_eq!( + decide_on_retry(64, last_halve + COOLDOWN_MS + 1, last_halve), + Some(32) + ); + } + + #[test] + fn halve_respects_min_floor() { + // At MIN (1), retries are no-ops — we never go below 1. + assert_eq!(decide_on_retry(MIN_IN_FLIGHT, COOLDOWN_MS + 1, 0), None); + // At 2, halving to 1 is the last step. + assert_eq!(decide_on_retry(2, COOLDOWN_MS + 1, 0), Some(1)); + } + + #[test] + fn ramp_up_respects_max_cap() { + // Even from a large `current`, we never exceed MAX. + assert_eq!( + decide_on_success(MAX_IN_FLIGHT - 1, true), + Some(MAX_IN_FLIGHT) + ); + assert_eq!(decide_on_success(MAX_IN_FLIGHT, true), None); + // Additive at the cap is also a no-op. + assert_eq!(decide_on_success(MAX_IN_FLIGHT, false), None); + } +} diff --git a/vortex-bench/src/datasets/mod.rs b/vortex-bench/src/datasets/mod.rs index e7f53156500..7136d5451e1 100644 --- a/vortex-bench/src/datasets/mod.rs +++ b/vortex-bench/src/datasets/mod.rs @@ -8,6 +8,7 @@ use async_trait::async_trait; use serde::Deserialize; use serde::Serialize; use vortex::array::ArrayRef; +use vortex::array::ExecutionCtx; use crate::clickbench::Flavor; @@ -21,11 +22,22 @@ pub mod tpch_l_comment; use std::path::PathBuf; +pub(crate) const DEFAULT_BENCHMARK_RUNNER_ID: &str = "unknown"; + +pub(crate) fn normalize_benchmark_runner_id(benchmark_runner: &str) -> String { + let benchmark_runner = benchmark_runner.trim().replace('/', "_"); + if benchmark_runner.is_empty() { + DEFAULT_BENCHMARK_RUNNER_ID.to_string() + } else { + benchmark_runner + } +} + #[async_trait] pub trait Dataset { fn name(&self) -> &str; - async fn to_vortex_array(&self) -> Result; + async fn to_vortex_array(&self, ctx: &mut ExecutionCtx) -> Result; /// Get the path to the parquet file for this dataset. /// diff --git a/vortex-bench/src/datasets/struct_list_of_ints.rs b/vortex-bench/src/datasets/struct_list_of_ints.rs index 4999c05adac..6bed5151c5e 100644 --- a/vortex-bench/src/datasets/struct_list_of_ints.rs +++ b/vortex-bench/src/datasets/struct_list_of_ints.rs @@ -5,20 +5,23 @@ use std::fs::File; use std::path::PathBuf; use anyhow::Result; -use arrow_array::RecordBatch; use async_trait::async_trait; use parquet::arrow::ArrowWriter; use rand::RngExt; use rand::SeedableRng; use rand::rngs::StdRng; use vortex::array::ArrayRef; +use vortex::array::ExecutionCtx; use vortex::array::IntoArray; +use vortex::array::LEGACY_SESSION; +use vortex::array::VortexSessionExecute; use vortex::array::arrays::ChunkedArray; use vortex::array::arrays::ListArray; use vortex::array::arrays::PrimitiveArray; use vortex::array::arrays::StructArray; use vortex::array::arrays::chunked::ChunkedArrayExt; use vortex::array::arrays::listview::recursive_list_from_list_view; +use vortex::array::arrow::ArrowArrayExecutor; use vortex::array::validity::Validity; use vortex::dtype::FieldNames; @@ -61,7 +64,7 @@ impl Dataset for StructListOfInts { &self.name } - async fn to_vortex_array(&self) -> Result { + async fn to_vortex_array(&self, _ctx: &mut ExecutionCtx) -> Result { let names: FieldNames = (0..self.num_columns) .map(|col_idx| col_idx.to_string()) .collect(); @@ -113,7 +116,8 @@ impl Dataset for StructListOfInts { idempotent_async(&parquet_path, |temp_path| async move { // Generate the data - let array = self.to_vortex_array().await?; + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let array = self.to_vortex_array(&mut ctx).await?; // Convert to Arrow RecordBatches and write to parquet let chunked = array.as_::(); @@ -123,7 +127,9 @@ impl Dataset for StructListOfInts { for chunk in chunked.iter_chunks() { let converted = recursive_list_from_list_view(chunk.clone())?; - let batch = RecordBatch::try_from(&converted)?; + let schema = converted.dtype().to_arrow_schema()?; + let batch = converted + .execute_record_batch(&schema, &mut LEGACY_SESSION.create_execution_ctx())?; if writer.is_none() { writer = Some(ArrowWriter::try_new( diff --git a/vortex-bench/src/datasets/taxi_data.rs b/vortex-bench/src/datasets/taxi_data.rs index cc3c6f61b81..919a0fa4dff 100644 --- a/vortex-bench/src/datasets/taxi_data.rs +++ b/vortex-bench/src/datasets/taxi_data.rs @@ -8,6 +8,7 @@ use async_trait::async_trait; use tokio::fs::File as TokioFile; use tokio::io::AsyncWriteExt; use vortex::array::ArrayRef; +use vortex::array::ExecutionCtx; use vortex::array::IntoArray; use vortex::array::stream::ArrayStreamExt; use vortex::file::OpenOptionsSessionExt; @@ -37,7 +38,7 @@ impl Dataset for TaxiData { "taxi" } - async fn to_vortex_array(&self) -> Result { + async fn to_vortex_array(&self, _ctx: &mut ExecutionCtx) -> Result { fetch_taxi_data().await } diff --git a/vortex-bench/src/datasets/tpch_l_comment.rs b/vortex-bench/src/datasets/tpch_l_comment.rs index 00a141f03ba..ebe6ff2e96a 100644 --- a/vortex-bench/src/datasets/tpch_l_comment.rs +++ b/vortex-bench/src/datasets/tpch_l_comment.rs @@ -8,9 +8,11 @@ use async_trait::async_trait; use futures::TryStreamExt; use glob::glob; use vortex::array::ArrayRef; +use vortex::array::Canonical; +use vortex::array::ExecutionCtx; use vortex::array::IntoArray; -use vortex::array::ToCanonical; use vortex::array::arrays::ChunkedArray; +use vortex::array::arrays::StructArray; use vortex::dtype::Nullability::NonNullable; use vortex::expr::col; use vortex::expr::pack; @@ -51,7 +53,7 @@ impl Dataset for TPCHLCommentChunked { "TPC-H l_comment chunked" } - async fn to_vortex_array(&self) -> Result { + async fn to_vortex_array(&self, ctx: &mut ExecutionCtx) -> Result { let base_path = "tpch".to_data_path(); let scale_factor_dir = base_path.join("1.0"); let data_dir = scale_factor_dir.join(Format::OnDiskVortex.name()); @@ -76,7 +78,14 @@ impl Dataset for TPCHLCommentChunked { let file_chunks: Vec<_> = file .scan()? .with_projection(pack(vec![("l_comment", col("l_comment"))], NonNullable)) - .map(|a| Ok(a.to_canonical()?.into_array())) + .map({ + let ctx = ctx.clone(); + move |a| { + let mut ctx = ctx.clone(); + let canonical = a.execute::(&mut ctx)?; + Ok(canonical.into_array()) + } + }) .into_array_stream()? .try_collect() .await?; @@ -99,12 +108,9 @@ impl Dataset for TPCHLCommentCanonical { "TPC-H l_comment canonical" } - async fn to_vortex_array(&self) -> Result { - let comments_canonical = TPCHLCommentChunked - .to_vortex_array() - .await? - .to_struct() - .into_array(); + async fn to_vortex_array(&self, ctx: &mut ExecutionCtx) -> Result { + let comments_chunked = TPCHLCommentChunked.to_vortex_array(ctx).await?; + let comments_canonical = comments_chunked.execute::(ctx)?.into_array(); Ok(ChunkedArray::from_iter([comments_canonical]).into_array()) } diff --git a/vortex-bench/src/downloadable_dataset.rs b/vortex-bench/src/downloadable_dataset.rs index 78ab162d688..cd73ba5af84 100644 --- a/vortex-bench/src/downloadable_dataset.rs +++ b/vortex-bench/src/downloadable_dataset.rs @@ -4,6 +4,7 @@ use async_trait::async_trait; use tokio::fs::File; use vortex::array::ArrayRef; +use vortex::array::ExecutionCtx; use vortex::array::IntoArray; use vortex::array::stream::ArrayStreamExt; use vortex::file::OpenOptionsSessionExt; @@ -57,7 +58,7 @@ impl Dataset for DownloadableDataset { } } - async fn to_vortex_array(&self) -> anyhow::Result { + async fn to_vortex_array(&self, _ctx: &mut ExecutionCtx) -> anyhow::Result { let parquet = self.to_parquet_path().await?; let dir = format!("{}/", self.name()).to_data_path(); let vortex = dir.join(format!("{}.vortex", self.name())); diff --git a/vortex-bench/src/fineweb/mod.rs b/vortex-bench/src/fineweb/mod.rs index 743e7737313..93151cc6fca 100644 --- a/vortex-bench/src/fineweb/mod.rs +++ b/vortex-bench/src/fineweb/mod.rs @@ -3,15 +3,14 @@ use std::path::PathBuf; -use tokio::io::AsyncWriteExt; use tracing::info; use url::Url; use crate::Benchmark; use crate::BenchmarkDataset; -use crate::IdempotentPath; use crate::TableSpec; -use crate::idempotent_async; +use crate::datasets::data_downloads::download_data; +use crate::utils::file::resolve_data_url; /// URL to the sample file const SAMPLE_URL: &str = "https://huggingface.co/datasets/HuggingFaceFW/fineweb/resolve/v1.4.0/sample/10BT/001_00000.parquet"; @@ -51,35 +50,12 @@ impl FinewebBenchmark { } pub fn with_remote_data_dir(use_remote_data_dir: Option) -> anyhow::Result { - let data_url = Self::create_data_url(&use_remote_data_dir)?; + let data_url = Self::create_data_url(use_remote_data_dir.as_deref())?; Ok(Self { data_url }) } - fn create_data_url(remote_data_dir: &Option) -> anyhow::Result { - match remote_data_dir { - None => { - let data_dir = "fineweb".to_data_path(); - Url::from_directory_path(&data_dir).map_err(|_| { - anyhow::anyhow!("Failed to create URL from directory path: {:?}", &data_dir) - }) - } - Some(remote_data_dir) => { - if !remote_data_dir.ends_with("/") { - tracing::warn!( - "Supply a --use-remote-data-dir argument which ends in a slash e.g. s3://vortex-bench-dev-eu/develop/12345/fineweb/" - ); - } - tracing::info!( - concat!( - "Assuming data already exists at this remote (e.g. S3, GCS) URL: {}.\n", - "If it does not, you should kill this command, locally generate the files (by running without\n", - "--use-remote-data-dir) and upload data/fineweb/ to some remote location.", - ), - remote_data_dir, - ); - Ok(Url::parse(remote_data_dir)?) - } - } + fn create_data_url(remote_data_dir: Option<&str>) -> anyhow::Result { + resolve_data_url(remote_data_dir, "fineweb") } } @@ -104,27 +80,7 @@ impl Benchmark for FinewebBenchmark { return Ok(()); } - let parquet = idempotent_async(&self.parquet_path()?, |parquet_path| async move { - info!("Downloading FineWeb Parquet source from HuggingFace"); - - let response = reqwest::get(SAMPLE_URL) - .await? - .error_for_status() - .map_err(|err| { - anyhow::anyhow!("error fetching fineweb sample from HuggingFace: {err}") - })?; - - let bytes = response.bytes().await?; - let mut w = tokio::fs::File::create(parquet_path).await?; - - w.write_all(&bytes).await?; - - w.flush().await?; - - Ok(()) - }) - .await?; - + let parquet = download_data(self.parquet_path()?, SAMPLE_URL).await?; info!("fineweb base data generated in {}", parquet.display()); Ok(()) diff --git a/vortex-bench/src/lib.rs b/vortex-bench/src/lib.rs index a1b7c46bb16..db9bd69c6ce 100644 --- a/vortex-bench/src/lib.rs +++ b/vortex-bench/src/lib.rs @@ -1,8 +1,8 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] -#![allow(clippy::expect_used)] +#![expect(clippy::unwrap_used)] +#![expect(clippy::expect_used)] use std::clone::Clone; use std::fmt::Display; @@ -53,6 +53,7 @@ pub mod statpopgen; pub mod tpcds; pub mod tpch; pub mod utils; +pub mod vector_dataset; pub use benchmark::Benchmark; pub use benchmark::TableSpec; diff --git a/vortex-bench/src/measurements.rs b/vortex-bench/src/measurements.rs index f49349cd95e..4f6abe65563 100644 --- a/vortex-bench/src/measurements.rs +++ b/vortex-bench/src/measurements.rs @@ -243,7 +243,8 @@ pub struct QueryMeasurement { pub query_idx: usize, pub target: Target, pub benchmark_dataset: BenchmarkDataset, - /// The storage backend against which this test was run. One of: s3, gcs, nvme. + pub benchmark_runner: String, + /// The storage backend against which this test was run. One of: s3, nvme. pub storage: String, pub runs: Vec, } @@ -279,6 +280,8 @@ pub struct QueryMeasurementJson { pub name: String, pub storage: String, pub dataset: BenchmarkDataset, + /// The cloud runner used to run this + pub runner: String, pub unit: String, pub value: u128, pub all_runtimes: Vec, @@ -310,6 +313,7 @@ impl ToJson for QueryMeasurement { name, storage: self.storage.clone(), dataset: self.benchmark_dataset.clone(), + runner: self.benchmark_runner.clone(), unit: "ns".to_string(), value: self.median_run().as_nanos(), all_runtimes: self.runs.iter().map(|r| r.as_nanos()).collect_vec(), @@ -430,6 +434,7 @@ pub struct MemoryMeasurement { pub query_idx: usize, pub target: Target, pub benchmark_dataset: BenchmarkDataset, + pub benchmark_runner: String, pub storage: String, pub physical_memory_delta: i64, pub virtual_memory_delta: i64, @@ -442,6 +447,7 @@ impl MemoryMeasurement { query_idx: usize, target: Target, benchmark_dataset: BenchmarkDataset, + benchmark_runner: String, storage: String, memory_result: MemoryMeasurementResult, ) -> Self { @@ -449,6 +455,7 @@ impl MemoryMeasurement { query_idx, target, benchmark_dataset, + benchmark_runner, storage, physical_memory_delta: memory_result.physical_memory_delta, virtual_memory_delta: memory_result.virtual_memory_delta, @@ -474,6 +481,7 @@ impl ToJson for MemoryMeasurement { name, storage: self.storage.clone(), dataset: self.benchmark_dataset.clone(), + runner: self.benchmark_runner.clone(), physical_memory_delta: self.physical_memory_delta, virtual_memory_delta: self.virtual_memory_delta, peak_physical_memory: self.peak_physical_memory, @@ -507,6 +515,7 @@ pub struct MemoryMeasurementJson { pub name: String, pub storage: String, pub dataset: BenchmarkDataset, + pub runner: String, pub physical_memory_delta: i64, pub virtual_memory_delta: i64, pub peak_physical_memory: u64, diff --git a/vortex-bench/src/polarsignals/benchmark.rs b/vortex-bench/src/polarsignals/benchmark.rs index 16ba6a9886b..0ef7b4246e6 100644 --- a/vortex-bench/src/polarsignals/benchmark.rs +++ b/vortex-bench/src/polarsignals/benchmark.rs @@ -119,7 +119,7 @@ impl Benchmark for PolarSignalsBenchmark { )] } - #[expect(clippy::expect_used, clippy::unwrap_in_result)] + #[expect(clippy::expect_used)] fn pattern(&self, _table_name: &str, format: Format) -> Option { Some( format!("{FILE_NAME}.{}", format.ext()) diff --git a/vortex-bench/src/polarsignals/data.rs b/vortex-bench/src/polarsignals/data.rs index eb2db956ee4..d1a67785b2a 100644 --- a/vortex-bench/src/polarsignals/data.rs +++ b/vortex-bench/src/polarsignals/data.rs @@ -25,6 +25,7 @@ use parquet::file::properties::WriterProperties; use rand::RngExt; use rand::SeedableRng; use rand::rngs::StdRng; +use vortex::utils::parallelism::get_available_parallelism; use super::schema::Int64DictBuilder; use super::schema::LABELS; @@ -85,7 +86,7 @@ fn generate_sorted_label_sets() -> LabelSets { .iter() .map(|&idx| { let (_, fill, distinct) = LABELS[idx]; - #[allow(clippy::cast_possible_truncation, clippy::cast_sign_loss)] + #[expect(clippy::cast_possible_truncation, clippy::cast_sign_loss)] let null_count = ((1.0 - fill) * NUM_LABEL_SETS as f64).round() as usize; if s < null_count || distinct == 0 { None @@ -146,9 +147,7 @@ pub fn generate_polarsignals_parquet(n_rows: usize, output_path: &Path) -> Resul let mut writer = ArrowWriter::try_new(file, Arc::clone(&schema), Some(props))?; let batch_size = 10_000; - let num_threads = std::thread::available_parallelism() - .map(|n| n.get()) - .unwrap_or(1); + let num_threads = get_available_parallelism().unwrap_or(1); let batch_ranges: Vec<(usize, usize)> = (0..n_rows) .step_by(batch_size) @@ -191,7 +190,7 @@ pub fn generate_polarsignals_parquet(n_rows: usize, output_path: &Path) -> Resul Ok(()) } -#[allow(clippy::too_many_arguments)] +#[expect(clippy::too_many_arguments)] fn build_batch( schema: &Arc, n: usize, @@ -414,7 +413,7 @@ fn build_locations( } #[cfg(test)] -#[allow(clippy::disallowed_types)] +#[expect(clippy::disallowed_types)] mod tests { use std::collections::HashSet; diff --git a/vortex-bench/src/public_bi.rs b/vortex-bench/src/public_bi.rs index 3602cd2cd33..7406347d370 100644 --- a/vortex-bench/src/public_bi.rs +++ b/vortex-bench/src/public_bi.rs @@ -26,6 +26,7 @@ use tracing::info; use tracing::trace; use url::Url; use vortex::array::ArrayRef; +use vortex::array::ExecutionCtx; use vortex::array::IntoArray; use vortex::array::stream::ArrayStreamExt; use vortex::error::VortexResult; @@ -43,7 +44,7 @@ use crate::TableSpec; use crate::conversions::parquet_to_vortex_chunks; use crate::datasets::Dataset; use crate::datasets::data_downloads::decompress_bz2; -use crate::datasets::data_downloads::download_data; +use crate::datasets::data_downloads::download_many; use crate::idempotent_async; use crate::workspace_root; @@ -289,16 +290,13 @@ pub struct PBIData { impl PBIData { async fn download_bzips(&self) -> anyhow::Result<()> { - let download_futures = self.tables.iter().map(|table| { - download_data( + let downloads = self.tables.iter().map(|table| { + ( self.get_file_path(&table.name, FileType::CsvBzip2), - table.data_url.as_str(), + table.data_url.as_str().to_owned(), ) }); - let results = join_all(download_futures).await; - for result in results { - result?; - } + download_many(downloads).await?; Ok(()) } @@ -455,7 +453,7 @@ impl Dataset for PBIBenchmark { &self.name } - async fn to_vortex_array(&self) -> anyhow::Result { + async fn to_vortex_array(&self, _ctx: &mut ExecutionCtx) -> anyhow::Result { let dataset = self.dataset()?; dataset.write_as_vortex().await?; // reading only the first table, each table in a PBI benchmark diff --git a/vortex-bench/src/realnest/gharchive.rs b/vortex-bench/src/realnest/gharchive.rs index 154cf945185..3ec6a67f340 100644 --- a/vortex-bench/src/realnest/gharchive.rs +++ b/vortex-bench/src/realnest/gharchive.rs @@ -14,10 +14,10 @@ use url::Url; use crate::Benchmark; use crate::BenchmarkDataset; -use crate::IdempotentPath; use crate::TableSpec; use crate::idempotent; use crate::idempotent_async; +use crate::utils::file::resolve_data_url; /// Template URL for raw JSON dataset fn raw_json_url(hour: usize) -> String { @@ -43,35 +43,12 @@ impl GithubArchiveBenchmark { } pub fn with_remote_data_dir(use_remote_data_dir: Option) -> anyhow::Result { - let data_url = Self::create_data_url(&use_remote_data_dir)?; + let data_url = Self::create_data_url(use_remote_data_dir.as_deref())?; Ok(Self { data_url }) } - fn create_data_url(remote_data_dir: &Option) -> anyhow::Result { - match remote_data_dir { - None => { - let data_dir = "gharchive".to_data_path(); - Url::from_directory_path(&data_dir).map_err(|_| { - anyhow::anyhow!("Failed to create URL from directory path: {:?}", &data_dir) - }) - } - Some(remote_data_dir) => { - if !remote_data_dir.ends_with("/") { - tracing::warn!( - "Supply a --use-remote-data-dir argument which ends in a slash e.g. s3://vortex-bench-dev-eu/develop/12345/gharchive/" - ); - } - tracing::info!( - concat!( - "Assuming data already exists at this remote (e.g. S3, GCS) URL: {}.\n", - "If it does not, you should kill this command, locally generate the files (by running without\n", - "--use-remote-data-dir) and upload data/gharchive/ to some remote location.", - ), - remote_data_dir, - ); - Ok(Url::parse(remote_data_dir)?) - } - } + fn create_data_url(remote_data_dir: Option<&str>) -> anyhow::Result { + resolve_data_url(remote_data_dir, "gharchive") } } diff --git a/vortex-bench/src/runner.rs b/vortex-bench/src/runner.rs index 6c9064b6a29..3885bace2af 100644 --- a/vortex-bench/src/runner.rs +++ b/vortex-bench/src/runner.rs @@ -13,12 +13,15 @@ use std::time::Instant; use indicatif::ProgressBar; use vortex::error::vortex_panic; +use vortex::utils::aliases::hash_set::HashSet; use crate::Benchmark; use crate::BenchmarkDataset; use crate::Engine; use crate::Format; use crate::Target; +use crate::datasets::DEFAULT_BENCHMARK_RUNNER_ID; +use crate::datasets::normalize_benchmark_runner_id; /// Controls whether queries are benchmarked or explained. pub enum BenchmarkMode { @@ -65,8 +68,10 @@ pub struct BenchmarkResults { pub struct SqlBenchmarkRunner { engine: Engine, benchmark_dataset: BenchmarkDataset, + benchmark_runner: String, storage: String, expected_row_counts: Option>, + /// Deduplicated, preserving insertion order. formats: Vec, memory_tracker: Option, hide_progress_bar: bool, @@ -79,17 +84,23 @@ impl SqlBenchmarkRunner { pub fn new( benchmark: &B, engine: Engine, - formats: Vec, + benchmark_runner: String, + formats: impl IntoIterator, track_memory: bool, hide_progress_bar: bool, ) -> anyhow::Result { + let mut seen = HashSet::new(); + let formats: Vec = formats.into_iter().filter(|f| seen.insert(*f)).collect(); let storage = url_scheme_to_storage(benchmark.data_url())?; + let benchmark_runner = normalize_benchmark_runner_id(&benchmark_runner); + validate_benchmark_runner_id(&benchmark_runner, is_ci())?; let memory_tracker = track_memory.then(BenchmarkMemoryTracker::new); Ok(Self { engine, benchmark_dataset: benchmark.dataset(), + benchmark_runner, storage, expected_row_counts: benchmark.expected_row_counts().map(|s| s.to_vec()), formats, @@ -164,6 +175,7 @@ impl SqlBenchmarkRunner { query_idx, target, benchmark_dataset: self.benchmark_dataset.clone(), + benchmark_runner: self.benchmark_runner.clone(), storage: self.storage.clone(), runs, }); @@ -188,6 +200,7 @@ impl SqlBenchmarkRunner { query_idx, target, self.benchmark_dataset.clone(), + self.benchmark_runner.clone(), self.storage.clone(), memory_result, )); @@ -407,6 +420,18 @@ impl SqlBenchmarkRunner { } } +fn is_ci() -> bool { + matches!(std::env::var("CI").as_deref(), Ok("true")) +} + +fn validate_benchmark_runner_id(benchmark_runner: &str, is_ci: bool) -> anyhow::Result<()> { + anyhow::ensure!( + !is_ci || benchmark_runner != DEFAULT_BENCHMARK_RUNNER_ID, + "benchmark runner must not be unknown in CI; pass --runner" + ); + Ok(()) +} + pub fn export_results( queries: Vec, memory: Vec, @@ -456,3 +481,23 @@ pub fn filter_queries( }) .collect() } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn ci_rejects_unknown_benchmark_runner() { + assert!(validate_benchmark_runner_id("unknown", true).is_err()); + } + + #[test] + fn ci_accepts_explicit_benchmark_runner() { + assert!(validate_benchmark_runner_id("ec2_c6id.8xlarge", true).is_ok()); + } + + #[test] + fn local_accepts_unknown_benchmark_runner() { + assert!(validate_benchmark_runner_id("unknown", false).is_ok()); + } +} diff --git a/vortex-bench/src/statpopgen/builder.rs b/vortex-bench/src/statpopgen/builder.rs index 7c46d1ff2b8..4e11b4e43e4 100644 --- a/vortex-bench/src/statpopgen/builder.rs +++ b/vortex-bench/src/statpopgen/builder.rs @@ -28,8 +28,7 @@ use vortex::utils::aliases::hash_set::HashSet; use super::vcf_conversion::*; -#[allow(dead_code)] -#[allow(non_snake_case)] +#[expect(non_snake_case)] pub struct GnomADBuilder<'a> { /// The schema of the to-be-generated Parquet file. schema: SchemaRef, @@ -240,7 +239,6 @@ impl InfoArrayBuilder { } impl<'a> GnomADBuilder<'a> { - #[allow(non_snake_case)] pub fn new(header: &'a Header, schema: SchemaRef) -> Self { let info_builder: HashMap<&'a str, InfoArrayBuilder> = header .infos() diff --git a/vortex-bench/src/statpopgen/statpopgen_benchmark.rs b/vortex-bench/src/statpopgen/statpopgen_benchmark.rs index dc5fa6448ad..58c4b6aed51 100644 --- a/vortex-bench/src/statpopgen/statpopgen_benchmark.rs +++ b/vortex-bench/src/statpopgen/statpopgen_benchmark.rs @@ -129,7 +129,7 @@ impl Benchmark for StatPopGenBenchmark { Ok(()) } - #[allow(clippy::cast_possible_truncation)] + #[expect(clippy::cast_possible_truncation)] fn expected_row_counts(&self) -> Option> { let n_rows = self.n_rows as usize; match self.scale_factor { @@ -168,7 +168,7 @@ impl Benchmark for StatPopGenBenchmark { vec![TableSpec::new("statpopgen", None)] } - #[expect(clippy::expect_used, clippy::unwrap_in_result)] + #[expect(clippy::expect_used)] fn pattern(&self, _table_name: &str, format: Format) -> Option { Some( format!("{}.{}", Self::FILE_NAME, format.ext()) diff --git a/vortex-bench/src/tpcds/tpcds_benchmark.rs b/vortex-bench/src/tpcds/tpcds_benchmark.rs index 3ad60876521..7f8f2d5281e 100644 --- a/vortex-bench/src/tpcds/tpcds_benchmark.rs +++ b/vortex-bench/src/tpcds/tpcds_benchmark.rs @@ -27,11 +27,11 @@ impl TpcDsBenchmark { pub fn new(scale_factor: String, use_remote_data_dir: Option) -> Result { Ok(Self { scale_factor: scale_factor.clone(), - data_url: Self::create_data_url(&use_remote_data_dir, &scale_factor)?, + data_url: Self::create_data_url(use_remote_data_dir.as_deref(), &scale_factor)?, }) } - fn create_data_url(remote_data_dir: &Option, scale_factor: &str) -> Result { + fn create_data_url(remote_data_dir: Option<&str>, scale_factor: &str) -> Result { match remote_data_dir { None => { let data_dir = "tpcds".to_data_path(); @@ -124,7 +124,7 @@ impl Benchmark for TpcDsBenchmark { ] } - #[expect(clippy::expect_used, clippy::unwrap_in_result)] + #[expect(clippy::expect_used)] fn pattern(&self, table_name: &str, format: Format) -> Option { Some( format!("{}.{}", table_name, format.ext()) diff --git a/vortex-bench/src/tpch/benchmark.rs b/vortex-bench/src/tpch/benchmark.rs index 71290f860e0..6851e5f5f06 100644 --- a/vortex-bench/src/tpch/benchmark.rs +++ b/vortex-bench/src/tpch/benchmark.rs @@ -37,14 +37,11 @@ impl TpcHBenchmark { pub fn new(scale_factor: String, use_remote_data_dir: Option) -> anyhow::Result { Ok(Self { scale_factor: scale_factor.clone(), - data_url: Self::create_data_url(&use_remote_data_dir, &scale_factor)?, + data_url: Self::create_data_url(use_remote_data_dir.as_deref(), &scale_factor)?, }) } - fn create_data_url( - remote_data_dir: &Option, - scale_factor: &str, - ) -> anyhow::Result { + fn create_data_url(remote_data_dir: Option<&str>, scale_factor: &str) -> anyhow::Result { match remote_data_dir { None => { let data_dir = "tpch".to_data_path(); @@ -139,7 +136,7 @@ impl Benchmark for TpcHBenchmark { ] } - #[expect(clippy::expect_used, clippy::unwrap_in_result)] + #[expect(clippy::expect_used)] fn pattern(&self, table_name: &str, format: Format) -> Option { Some( format!("{}_*.{}", table_name, format.ext()) diff --git a/vortex-bench/src/tpch/tpchgen.rs b/vortex-bench/src/tpch/tpchgen.rs index 1d7273f3ccc..5dbeb380a7e 100644 --- a/vortex-bench/src/tpch/tpchgen.rs +++ b/vortex-bench/src/tpch/tpchgen.rs @@ -269,7 +269,7 @@ fn generate_table_files( } /// Calculate the number of partitions without creating expensive iterators -#[allow(clippy::cast_possible_truncation)] +#[expect(clippy::cast_possible_truncation)] fn calculate_num_parts(generator: TableGenerator, options: &TpchGenOptions) -> Result { let scale_factor = options.scale_factor.parse::()?; @@ -277,7 +277,7 @@ fn calculate_num_parts(generator: TableGenerator, options: &TpchGenOptions) -> R .uncompressed_data_size() .zip(options.max_file_size_mb) { - #[allow(clippy::cast_precision_loss)] + #[expect(clippy::cast_precision_loss)] let file_size = (data_size as f64 * scale_factor).ceil() as u64; file_size.div_ceil(max_file_size) } else { @@ -288,7 +288,7 @@ fn calculate_num_parts(generator: TableGenerator, options: &TpchGenOptions) -> R } /// Create a single batch iterator for a specific partition -#[allow(clippy::cast_possible_truncation)] +#[expect(clippy::cast_possible_truncation)] fn create_single_batch_iterator( generator: TableGenerator, options: &TpchGenOptions, diff --git a/vortex-bench/src/utils/constants.rs b/vortex-bench/src/utils/constants.rs index 959ad63b7f4..c9f5ca4c442 100644 --- a/vortex-bench/src/utils/constants.rs +++ b/vortex-bench/src/utils/constants.rs @@ -4,4 +4,3 @@ /// Storage type constants pub const STORAGE_NVME: &str = "nvme"; pub const STORAGE_S3: &str = "s3"; -pub const STORAGE_GCS: &str = "gcs"; diff --git a/vortex-bench/src/utils/file.rs b/vortex-bench/src/utils/file.rs index f13dbdf93c8..fb7ecc668fb 100644 --- a/vortex-bench/src/utils/file.rs +++ b/vortex-bench/src/utils/file.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -use std::fs::create_dir_all; +use std::fs; use std::future::Future; use std::path::Path; use std::path::PathBuf; @@ -25,10 +25,10 @@ pub fn idempotent( if !data_path.exists() { // Ensure parent directory exists if let Some(parent) = data_path.parent() { - create_dir_all(parent).context("Failed to create parent directories")?; + fs::create_dir_all(parent).context("Failed to create parent directories")?; } f(temp_path.as_path())?; - std::fs::rename(temp_path, &data_path).context("Failed to rename temp file")?; + fs::rename(temp_path, &data_path).context("Failed to rename temp file")?; } Ok(data_path) } @@ -44,10 +44,10 @@ where if !data_path.exists() { // Ensure parent directory exists if let Some(parent) = data_path.parent() { - create_dir_all(parent).context("Failed to create parent directories")?; + fs::create_dir_all(parent).context("Failed to create parent directories")?; } f(temp_path.clone()).await?; - std::fs::rename(temp_path, &data_path).context("Failed to rename temp file")?; + fs::rename(temp_path, &data_path).context("Failed to rename temp file")?; } Ok(data_path) } @@ -110,22 +110,62 @@ impl IdempotentPath for &Path { } } +/// Resolve the `--use-remote-data-dir` CLI option to a `Url` for a named dataset. +/// +/// When `remote_data_dir` is `None`, returns a `file://` URL pointing at the dataset's local cache +/// directory (`//`). +/// +/// When `remote_data_dir` is `Some(...)`, parses it as a remote URL (typically `s3://` or `gs://`). +/// The user must have pre-uploaded the expected data layout; a warning is logged if the URL does +/// not end in `/`, and an informational message describes the expected layout. +/// +/// This helper replaces the boilerplate `create_data_url()` that used to be duplicated across every +/// benchmark that supports remote data directories (ClickBench, Fineweb, GhArchive, ...). +pub fn resolve_data_url(remote_data_dir: Option<&str>, local_subdir: &str) -> Result { + match remote_data_dir { + None => { + let data_dir = data_dir().join(local_subdir); + Url::from_directory_path(&data_dir).map_err(|_| { + anyhow::anyhow!("Failed to create URL from directory path: {:?}", &data_dir) + }) + } + Some(remote_data_dir) => { + if !remote_data_dir.ends_with('/') { + tracing::warn!( + "Supply a --use-remote-data-dir argument which ends in a slash \ + e.g. s3://vortex-bench-dev-eu/develop/12345/{}/", + local_subdir, + ); + } + tracing::info!( + concat!( + "Assuming data already exists at this remote (e.g. S3, GCS) URL: {}.\n", + "If it does not, you should kill this command, locally generate the files ", + "(by running without\n", + "--use-remote-data-dir) and upload data/{}/ to some remote location.", + ), + remote_data_dir, + local_subdir, + ); + Ok(Url::parse(remote_data_dir)?) + } + } +} + /// Convert a URL scheme to a storage type string /// -/// Maps URL schemes (s3, gcs, file) to storage type identifiers +/// Maps URL schemes (s3, file) to storage type identifiers /// for benchmark reporting. /// /// # Returns -/// - A storage type string ("s3", "gcs", "nvme") +/// - A storage type string ("s3", "nvme") /// - Or an error if the scheme is unknown pub fn url_scheme_to_storage(url: &Url) -> Result { - use super::constants::STORAGE_GCS; use super::constants::STORAGE_NVME; use super::constants::STORAGE_S3; match url.scheme() { STORAGE_S3 => Ok(STORAGE_S3.to_owned()), - STORAGE_GCS => Ok(STORAGE_GCS.to_owned()), "file" => Ok(STORAGE_NVME.to_owned()), otherwise => { bail!("unknown URL scheme: {}", otherwise) diff --git a/vortex-bench/src/utils/logging.rs b/vortex-bench/src/utils/logging.rs index cdc99b8fb59..b414184f930 100644 --- a/vortex-bench/src/utils/logging.rs +++ b/vortex-bench/src/utils/logging.rs @@ -4,34 +4,77 @@ use std::fs::File; use std::io::IsTerminal; +use clap::ValueEnum; use tracing::level_filters::LevelFilter; use tracing_perfetto::PerfettoLayer; use tracing_subscriber::EnvFilter; +use tracing_subscriber::Layer; use tracing_subscriber::prelude::*; -/// Initialize logging/tracing for a benchmark -pub fn setup_logging_and_tracing(verbose: bool, tracing: bool) -> anyhow::Result<()> { +/// Format for the primary stderr log sink. +/// +/// `Text` is the default human-readable formatter matching the historical behavior of this crate. +/// `Json` emits one newline-delimited JSON object per event, suitable for piping into `jq` or a log +/// aggregator. +#[derive(Copy, Clone, Debug, Default, PartialEq, Eq, ValueEnum)] +pub enum LogFormat { + #[default] + Text, + Json, +} + +/// Initialize logging/tracing for a benchmark, hardcoding [`LogFormat::Text`]. +/// +/// See [`setup_logging_and_tracing_with_format`] if you want to select JSON +/// output from a CLI flag. +pub fn setup_logging_and_tracing(verbose: bool, perfetto: bool) -> anyhow::Result<()> { + setup_logging_and_tracing_with_format(verbose, perfetto, LogFormat::Text) +} + +/// Initialize logging/tracing for a benchmark with an explicit stderr format. +/// +/// - `verbose`: when `RUST_LOG` is unset, raises the default filter from `INFO` to `TRACE`. Has no +/// effect when `RUST_LOG` is set (the env var wins). +/// - `perfetto`: when `true`, additionally attaches a [`tracing_perfetto::PerfettoLayer`] that +/// writes span begin/end events to `trace.json` in the current directory. Intended to be loaded +/// into the Perfetto UI for flamegraph visualization. +/// - `format`: controls the primary stderr sink's formatting. See [`LogFormat`]. +pub fn setup_logging_and_tracing_with_format( + verbose: bool, + perfetto: bool, + format: LogFormat, +) -> anyhow::Result<()> { let filter = default_env_filter(verbose); - let fmt_layer = tracing_subscriber::fmt::layer() - .with_writer(std::io::stderr) - .with_level(true) - .with_file(true) - .with_line_number(true) - .with_ansi(std::io::stderr().is_terminal()); + let perfetto_layer = perfetto + .then(|| { + Ok::<_, anyhow::Error>( + PerfettoLayer::new(File::create("trace.json")?).with_debug_annotations(true), + ) + }) + .transpose()?; + + // `fmt::layer()` and `fmt::layer().json()` produce different concrete types, + // so erase each to a `dyn Layer` via `.boxed()` and keep the registry uniform. + let fmt_layer: Box + Send + Sync> = match format { + LogFormat::Text => tracing_subscriber::fmt::layer() + .with_writer(std::io::stderr) + .with_level(true) + .with_file(true) + .with_line_number(true) + .with_ansi(std::io::stderr().is_terminal()) + .boxed(), + LogFormat::Json => tracing_subscriber::fmt::layer() + .json() + .with_writer(std::io::stderr) + .with_current_span(true) + .with_span_list(true) + .boxed(), + }; tracing_subscriber::registry() .with(filter) - .with( - tracing - .then(|| { - Ok::<_, anyhow::Error>( - PerfettoLayer::new(File::create("trace.json")?) - .with_debug_annotations(true), - ) - }) - .transpose()?, - ) + .with(perfetto_layer) .with(fmt_layer) .init(); diff --git a/vortex-bench/src/vector_dataset/catalog.rs b/vortex-bench/src/vector_dataset/catalog.rs new file mode 100644 index 00000000000..f34eaf812ec --- /dev/null +++ b/vortex-bench/src/vector_dataset/catalog.rs @@ -0,0 +1,416 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! The static catalog of hosted vector benchmark corpora. +//! +//! Every entry in [`ALL_VECTOR_DATASETS`] declares its dimensionality, row count, element type, +//! distance metric, the set of train-split layouts hosted on the upstream bucket, and whether +//! ground-truth `neighbors.parquet` and `scalar_labels` columns are available. +//! Most entries mirror VectorDBBench's dataset model; a few are extra prefixes that are present +//! on the same public bucket and use the same file layout. +//! +//! Higher-level code (downloaders, ingest pipelines, recall measurement) should be parameterized +//! over these descriptors rather than hardcoding per-dataset URLs. + +use std::num::NonZeroU32; + +use anyhow::Result; +use anyhow::bail; +use clap::ValueEnum; +use vortex::dtype::PType; + +use super::layout::LayoutSpec; +use super::layout::TrainLayout; +use super::layout::VectorMetric; + +const TEN_SHARDS: NonZeroU32 = NonZeroU32::new(10).unwrap(); +const FIFTY_SHARDS: NonZeroU32 = NonZeroU32::new(50).unwrap(); +const ONE_HUNDRED_SHARDS: NonZeroU32 = NonZeroU32::new(100).unwrap(); + +/// Every [`VectorDataset`] variant in catalog order. Useful for CLI help and metadata-consistency +/// tests. +pub const ALL_VECTOR_DATASETS: &[VectorDataset] = &[ + VectorDataset::CohereSmall100k, + VectorDataset::CohereMedium1m, + VectorDataset::CohereLarge10m, + VectorDataset::OpenaiSmall50k, + VectorDataset::OpenaiMedium500k, + VectorDataset::OpenaiLarge5m, + VectorDataset::BioasqMedium1m, + VectorDataset::BioasqLarge10m, + VectorDataset::GloveSmall100k, + VectorDataset::GloveMedium1m, + VectorDataset::GistSmall100k, + VectorDataset::GistMedium1m, + VectorDataset::SiftSmall500k, + VectorDataset::SiftMedium5m, + VectorDataset::SiftLarge50m, + VectorDataset::LaionLarge100m, +]; + +// NB: We can't do `#[clap(rename_all = "kebab-case")]` here because it won't put a dash in front of +// any numbers. +/// The publicly hosted vector benchmark datasets. +/// +/// Variants are named ``, kebab-cased on the CLI (e.g. `cohere-large-10m`). +/// +/// The static metadata for each variant (dimensionality, row count, hosted layouts, etc.) is +/// exposed via the inherent methods below; the full table is reachable via [`ALL_VECTOR_DATASETS`]. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, ValueEnum)] +pub enum VectorDataset { + /// Cohere wiki-22-12, 100K × 768 f32, cosine. Single + SingleShuffled. + #[clap(name = "cohere-small-100k")] + CohereSmall100k, + /// Cohere wiki-22-12, 1M × 768 f32, cosine. Single + SingleShuffled. + #[clap(name = "cohere-medium-1m")] + CohereMedium1m, + /// Cohere wiki-22-12, 10M × 768 f32, cosine. Partitioned + PartitionedShuffled (10 shards). + #[clap(name = "cohere-large-10m")] + CohereLarge10m, + + /// OpenAI embeddings on C4, 50K × 1536 f64, cosine. Single + SingleShuffled. + #[clap(name = "openai-small-50k")] + OpenaiSmall50k, + /// OpenAI embeddings on C4, 500K × 1536 f64, cosine. Single + SingleShuffled. + #[clap(name = "openai-medium-500k")] + OpenaiMedium500k, + /// OpenAI embeddings on C4, 5M × 1536 f64, cosine. Partitioned + PartitionedShuffled (10 + /// shards). + #[clap(name = "openai-large-5m")] + OpenaiLarge5m, + + /// Bioasq biomedical, 1M × 1024 f32, cosine. SingleShuffled only. + #[clap(name = "bioasq-medium-1m")] + BioasqMedium1m, + /// Bioasq biomedical, 10M × 1024 f32, cosine. PartitionedShuffled only (10 shards). + #[clap(name = "bioasq-large-10m")] + BioasqLarge10m, + + /// GloVe word vectors, 100K × 200 f32, cosine. Single only. No neighbors / labels. + #[clap(name = "glove-small-100k")] + GloveSmall100k, + /// GloVe word vectors, 1M × 200 f32, cosine. Single only. No neighbors / labels. + #[clap(name = "glove-medium-1m")] + GloveMedium1m, + + /// GIST image features, 100K × 960 f32, L2. Single only. No neighbors / labels. + #[clap(name = "gist-small-100k")] + GistSmall100k, + /// GIST image features, 1M × 960 f32, L2. Single only. No neighbors / labels. + #[clap(name = "gist-medium-1m")] + GistMedium1m, + + /// SIFT image features, 500K × 128 f32, L2. Single only. No neighbors / labels. + #[clap(name = "sift-small-500k")] + SiftSmall500k, + /// SIFT image features, 5M × 128 f32, L2. Single only. No neighbors / labels. + #[clap(name = "sift-medium-5m")] + SiftMedium5m, + /// SIFT image features, 50M × 128 f32, L2. Partitioned only (50 shards). No labels. + #[clap(name = "sift-large-50m")] + SiftLarge50m, + + /// LAION image embeddings, 100M × 768 f32, L2. Partitioned only (100 shards). + /// Has `neighbors.parquet` and `scalar_labels.parquet`. + #[clap(name = "laion-large-100m")] + LaionLarge100m, +} + +impl VectorDataset { + /// Stable kebab-cased label used in CLI args, file paths, and metric names. + pub fn name(&self) -> &'static str { + match self { + VectorDataset::CohereSmall100k => "cohere-small-100k", + VectorDataset::CohereMedium1m => "cohere-medium-1m", + VectorDataset::CohereLarge10m => "cohere-large-10m", + VectorDataset::OpenaiSmall50k => "openai-small-50k", + VectorDataset::OpenaiMedium500k => "openai-medium-500k", + VectorDataset::OpenaiLarge5m => "openai-large-5m", + VectorDataset::BioasqMedium1m => "bioasq-medium-1m", + VectorDataset::BioasqLarge10m => "bioasq-large-10m", + VectorDataset::GloveSmall100k => "glove-small-100k", + VectorDataset::GloveMedium1m => "glove-medium-1m", + VectorDataset::GistSmall100k => "gist-small-100k", + VectorDataset::GistMedium1m => "gist-medium-1m", + VectorDataset::SiftSmall500k => "sift-small-500k", + VectorDataset::SiftMedium5m => "sift-medium-5m", + VectorDataset::SiftLarge50m => "sift-large-50m", + VectorDataset::LaionLarge100m => "laion-large-100m", + } + } + + /// The directory name on `assets.zilliz.com/benchmark//`. Snake-cased to + /// match the upstream bucket's existing naming convention. + pub fn s3_prefix(&self) -> &'static str { + match self { + VectorDataset::CohereSmall100k => "cohere_small_100k", + VectorDataset::CohereMedium1m => "cohere_medium_1m", + VectorDataset::CohereLarge10m => "cohere_large_10m", + VectorDataset::OpenaiSmall50k => "openai_small_50k", + VectorDataset::OpenaiMedium500k => "openai_medium_500k", + VectorDataset::OpenaiLarge5m => "openai_large_5m", + VectorDataset::BioasqMedium1m => "bioasq_medium_1m", + VectorDataset::BioasqLarge10m => "bioasq_large_10m", + VectorDataset::GloveSmall100k => "glove_small_100k", + VectorDataset::GloveMedium1m => "glove_medium_1m", + VectorDataset::GistSmall100k => "gist_small_100k", + VectorDataset::GistMedium1m => "gist_medium_1m", + VectorDataset::SiftSmall500k => "sift_small_500k", + VectorDataset::SiftMedium5m => "sift_medium_5m", + VectorDataset::SiftLarge50m => "sift_large_50m", + VectorDataset::LaionLarge100m => "laion_large_100m", + } + } + + /// Vector dimensionality. + pub fn dim(&self) -> u32 { + match self { + VectorDataset::CohereSmall100k + | VectorDataset::CohereMedium1m + | VectorDataset::CohereLarge10m + | VectorDataset::LaionLarge100m => 768, + VectorDataset::OpenaiSmall50k + | VectorDataset::OpenaiMedium500k + | VectorDataset::OpenaiLarge5m => 1536, + VectorDataset::BioasqMedium1m | VectorDataset::BioasqLarge10m => 1024, + VectorDataset::GloveSmall100k | VectorDataset::GloveMedium1m => 200, + VectorDataset::GistSmall100k | VectorDataset::GistMedium1m => 960, + VectorDataset::SiftSmall500k + | VectorDataset::SiftMedium5m + | VectorDataset::SiftLarge50m => 128, + } + } + + /// Number of rows in the train split (sum across shards if partitioned). + pub fn num_rows(&self) -> u64 { + match self { + VectorDataset::CohereSmall100k => 100_000, + VectorDataset::CohereMedium1m => 1_000_000, + VectorDataset::CohereLarge10m => 10_000_000, + VectorDataset::OpenaiSmall50k => 50_000, + VectorDataset::OpenaiMedium500k => 500_000, + VectorDataset::OpenaiLarge5m => 5_000_000, + VectorDataset::BioasqMedium1m => 1_000_000, + VectorDataset::BioasqLarge10m => 10_000_000, + VectorDataset::GloveSmall100k => 100_000, + VectorDataset::GloveMedium1m => 1_000_000, + VectorDataset::GistSmall100k => 100_000, + VectorDataset::GistMedium1m => 1_000_000, + VectorDataset::SiftSmall500k => 500_000, + VectorDataset::SiftMedium5m => 5_000_000, + VectorDataset::SiftLarge50m => 50_000_000, + VectorDataset::LaionLarge100m => 100_000_000, + } + } + + /// Element scalar type as stored in the upstream parquet. The benchmark always casts to + /// `f32` after ingest (TurboQuant + the handrolled baseline operate in f32), so this is + /// only consulted by the parquet readers. + pub fn element_ptype(&self) -> PType { + match self { + VectorDataset::OpenaiSmall50k + | VectorDataset::OpenaiMedium500k + | VectorDataset::OpenaiLarge5m => PType::F64, + _ => PType::F32, + } + } + + /// Distance metric the upstream dataset was curated for. + pub fn metric(&self) -> VectorMetric { + match self { + VectorDataset::GistSmall100k + | VectorDataset::GistMedium1m + | VectorDataset::SiftSmall500k + | VectorDataset::SiftMedium5m + | VectorDataset::SiftLarge50m + | VectorDataset::LaionLarge100m => VectorMetric::L2, + _ => VectorMetric::Cosine, + } + } + + /// The set of train-split layouts hosted on the upstream bucket for this dataset. + /// + /// Always non-empty since the catalog never declares a dataset with zero layouts. + pub fn layouts(&self) -> Vec { + match self { + VectorDataset::CohereSmall100k + | VectorDataset::CohereMedium1m + | VectorDataset::OpenaiSmall50k + | VectorDataset::OpenaiMedium500k => { + vec![LayoutSpec::single(), LayoutSpec::single_shuffled()] + } + VectorDataset::CohereLarge10m | VectorDataset::OpenaiLarge5m => vec![ + LayoutSpec::partitioned(TEN_SHARDS), + LayoutSpec::partitioned_shuffled(TEN_SHARDS), + ], + VectorDataset::BioasqMedium1m => vec![LayoutSpec::single_shuffled()], + VectorDataset::BioasqLarge10m => { + vec![LayoutSpec::partitioned_shuffled(TEN_SHARDS)] + } + VectorDataset::GloveSmall100k + | VectorDataset::GloveMedium1m + | VectorDataset::GistSmall100k + | VectorDataset::GistMedium1m + | VectorDataset::SiftSmall500k + | VectorDataset::SiftMedium5m => vec![LayoutSpec::single()], + VectorDataset::SiftLarge50m => vec![LayoutSpec::partitioned(FIFTY_SHARDS)], + VectorDataset::LaionLarge100m => vec![LayoutSpec::partitioned(ONE_HUNDRED_SHARDS)], + } + } + + /// Whether `neighbors.parquet` (top-K ground truth for recall) is hosted. + pub fn has_neighbors(&self) -> bool { + match self { + VectorDataset::CohereSmall100k + | VectorDataset::CohereMedium1m + | VectorDataset::CohereLarge10m + | VectorDataset::OpenaiSmall50k + | VectorDataset::OpenaiMedium500k + | VectorDataset::OpenaiLarge5m + | VectorDataset::BioasqMedium1m + | VectorDataset::BioasqLarge10m + | VectorDataset::LaionLarge100m => true, + VectorDataset::GloveSmall100k + | VectorDataset::GloveMedium1m + | VectorDataset::GistSmall100k + | VectorDataset::GistMedium1m + | VectorDataset::SiftSmall500k + | VectorDataset::SiftMedium5m + | VectorDataset::SiftLarge50m => false, + } + } + + /// Whether the train split carries a `scalar_labels` column (for filtered-search + /// benchmarks). The benchmark does not exercise filtered search yet, but the ingest + /// pipeline copies the column through when present so future filtered-recall work does + /// not require re-ingest. + pub fn has_scalar_labels(&self) -> bool { + matches!( + self, + VectorDataset::CohereSmall100k + | VectorDataset::CohereMedium1m + | VectorDataset::CohereLarge10m + | VectorDataset::OpenaiSmall50k + | VectorDataset::OpenaiMedium500k + | VectorDataset::OpenaiLarge5m + | VectorDataset::BioasqMedium1m + | VectorDataset::BioasqLarge10m + | VectorDataset::LaionLarge100m + ) + } + + /// Validate that `layout` is one of the layouts hosted for this dataset and return its + /// [`LayoutSpec`]. + /// + /// Bails with an error message that lists the allowed values. + pub fn validate_layout(&self, layout: TrainLayout) -> Result { + let layouts = self.layouts(); + match layouts.iter().find(|spec| spec.layout() == layout) { + Some(spec) => Ok(*spec), + None => { + let allowed = layouts + .iter() + .map(|s| s.layout().label()) + .collect::>() + .join(", "); + bail!( + "dataset {} does not have layout '{}'; allowed layouts: [{}]", + self.name(), + layout, + allowed, + ); + } + } + } +} + +#[cfg(test)] +mod tests { + use vortex::utils::aliases::hash_set::HashSet; + + use super::*; + + #[test] + fn all_datasets_have_consistent_metadata() { + let mut seen: HashSet<&'static str> = HashSet::default(); + for &ds in ALL_VECTOR_DATASETS { + assert!(seen.insert(ds.name()), "duplicate name {}", ds.name()); + assert!(ds.dim() > 0); + assert!(ds.num_rows() > 0); + assert!(!ds.layouts().is_empty(), "{} has no layouts", ds.name()); + assert_eq!( + ds.s3_prefix().chars().filter(|c| *c == '_').count() + 1, + ds.name().split('-').count(), + "{}: s3_prefix '{}' shape disagrees with name '{}'", + ds.name(), + ds.s3_prefix(), + ds.name(), + ); + } + } + + #[test] + fn validate_layout_accepts_hosted_layout() { + let ds = VectorDataset::CohereSmall100k; + let spec = ds.validate_layout(TrainLayout::Single).unwrap(); + assert_eq!(spec.layout(), TrainLayout::Single); + assert_eq!(spec.num_files(), 1); + } + + #[test] + fn validate_layout_rejects_unhosted_layout() { + let ds = VectorDataset::SiftSmall500k; + let err = ds + .validate_layout(TrainLayout::SingleShuffled) + .unwrap_err() + .to_string(); + assert!( + err.contains("does not have layout 'single-shuffled'"), + "{err}" + ); + assert!(err.contains("[single]"), "{err}"); + } + + #[test] + fn partitioned_datasets_declare_shard_count() { + let layouts = VectorDataset::CohereLarge10m.layouts(); + assert_eq!(layouts.len(), 2); + for spec in layouts { + assert!(spec.layout().is_partitioned()); + assert_eq!(spec.num_files(), 10); + } + } + + #[test] + fn l2_datasets_match_upstream_metric() { + for ds in [ + VectorDataset::GistSmall100k, + VectorDataset::GistMedium1m, + VectorDataset::SiftSmall500k, + VectorDataset::SiftMedium5m, + VectorDataset::SiftLarge50m, + VectorDataset::LaionLarge100m, + ] { + assert_eq!(ds.metric(), VectorMetric::L2, "{} should use L2", ds.name()); + } + } + + #[test] + fn datasets_without_neighbors_skip_recall() { + for ds in [ + VectorDataset::GloveSmall100k, + VectorDataset::GloveMedium1m, + VectorDataset::GistSmall100k, + VectorDataset::GistMedium1m, + VectorDataset::SiftSmall500k, + VectorDataset::SiftMedium5m, + VectorDataset::SiftLarge50m, + ] { + assert!( + !ds.has_neighbors(), + "{} unexpectedly has neighbors", + ds.name() + ); + } + } +} diff --git a/vortex-bench/src/vector_dataset/convert.rs b/vortex-bench/src/vector_dataset/convert.rs new file mode 100644 index 00000000000..f0c13ff33aa --- /dev/null +++ b/vortex-bench/src/vector_dataset/convert.rs @@ -0,0 +1,503 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +// TODO(connor): Should we re-export this through `conversions.rs`? + +use vortex::array::ArrayRef; +use vortex::array::IntoArray; +use vortex::array::arrays::Chunked; +use vortex::array::arrays::ChunkedArray; +use vortex::array::arrays::ExtensionArray; +use vortex::array::arrays::FixedSizeListArray; +use vortex::array::arrays::List; +use vortex::array::arrays::ListView; +use vortex::array::arrays::Primitive; +use vortex::array::arrays::PrimitiveArray; +use vortex::array::arrays::chunked::ChunkedArrayExt; +use vortex::array::arrays::list::ListArrayExt; +use vortex::array::arrays::listview::recursive_list_from_list_view; +use vortex::array::validity::Validity; +use vortex::dtype::DType; +use vortex::dtype::extension::ExtDType; +use vortex::error::VortexExpect; +use vortex::error::VortexResult; +use vortex::error::vortex_bail; +use vortex::error::vortex_err; +use vortex::extension::EmptyMetadata; +use vortex_tensor::vector::Vector; + +/// Rewrap a list-of-float column as a [`vortex_tensor::vector::Vector`] extension array. +/// +/// Parquet has no fixed-size list logical type, so an embedding column ingested via +/// `parquet_to_vortex_chunks` arrives as `List` (or `List`) even when every row has the +/// same length. +/// +/// This helper validates that every list in `input` has the same length `D` and reconstructs the +/// column as an `Extension(FixedSizeList)` array, which is the type expected by the +/// vector search scalar functions in `vortex-tensor`. +/// +/// The input may be either a single [`ListView`] array or a [`Chunked`] array of lists (the common +/// case after `parquet_to_vortex_chunks`). Chunked inputs are converted chunk-by-chunk and +/// reassembled as a [`ChunkedArray`] of `Extension`. We also convert [`ListView`] to +/// [`List`] so that we know all elements are contiguous (this might be slow). +/// +/// # Errors +/// +/// Returns an error if: +/// - `input` is not a `ListView`, `List`, or `Chunked` array. +/// - The element type is not a float primitive (`f16`, `f32`, or `f64`). +/// - A nullable element dtype (`List`) is accepted as long as the runtime validity is +/// `NonNullable` or `AllValid` since parquet has no non-nullable-element list logical type, so +/// arrow-rs always marks list-of-float element fields as nullable on read regardless of whether +/// any element is actually missing. In that case the elements are rewrapped as non-nullable +/// before being embedded in the FSL. +/// - The element dtype is nullable *and* any element is actually null (i.e., `Validity::AllInvalid` +/// or any `Validity::Array` mask). Vector extension elements must be non-null, and that is +/// verified on construction. +/// - Any row has a different length than the first row. +/// - The list validity is nullable (vector elements cannot be null at the row level). +/// - The input has zero rows (the dimension cannot be inferred from empty input). +pub fn list_to_vector_ext(input: ArrayRef) -> VortexResult { + if let Some(chunked) = input.as_opt::() { + let converted: Vec = chunked + .iter_chunks() + .map(|chunk| list_to_vector_ext(chunk.clone())) + .collect::>()?; + + let Some(first) = converted.first() else { + vortex_bail!("list_to_vector_ext: chunked input has no chunks"); + }; + + let dtype = first.dtype().clone(); + return Ok(ChunkedArray::try_new(converted, dtype)?.into_array()); + } + + // `parquet_to_vortex_chunks` produces `ListView` arrays for list columns by default; + // materialize them into a flat `List` representation before we validate offsets. + if input.as_opt::().is_some() { + let flat = recursive_list_from_list_view(input)?; + return list_to_vector_ext(flat); + } + + let Some(list) = input.as_opt::() else { + vortex_bail!( + "list_to_vector_ext: expected a List array, got dtype {}", + input.dtype() + ); + }; + + if !matches!( + list.list_validity(), + Validity::NonNullable | Validity::AllValid + ) { + vortex_bail!( + "list_to_vector_ext: list rows must be non-nullable for Vector extension wrapping" + ); + } + + let element_dtype = list.element_dtype().clone(); + let DType::Primitive(ptype, elem_nullability) = &element_dtype else { + vortex_bail!( + "list_to_vector_ext: element dtype must be a primitive float, got {}", + element_dtype + ); + }; + if !ptype.is_float() { + vortex_bail!( + "list_to_vector_ext: element type must be float (f16/f32/f64), got {}", + ptype + ); + } + + // Extract the flat elements buffer up front: the nullable-handling branch below + // needs to inspect runtime validity before we can decide whether to rewrap it. + let raw_elements = list.sliced_elements()?; + + let num_rows = input.len(); + if num_rows == 0 { + vortex_bail!("list_to_vector_ext: cannot infer vector dimension from empty input"); + } + + // Walk the offsets array once, reusing the previous iteration's `end` as the + // next iteration's `start`. Each `offset_at` call goes through + // `ListArrayExt::offset_at`, which has a fast path when the offsets child is a + // `Primitive` array (direct slice index). That's the common case after + // `parquet_to_vortex_chunks`, so for a 100K-row column we do ~100K primitive + // slice indexes rather than 200K. The loop body is O(1) either way. + let mut prev_end = list.offset_at(0)?; + let first_end = list.offset_at(1)?; + + let dim = first_end.checked_sub(prev_end).ok_or_else(|| { + vortex_err!("list_to_vector_ext: offsets are not monotonically increasing") + })?; + if dim == 0 { + vortex_bail!("list_to_vector_ext: first row has zero elements"); + } + + prev_end = first_end; + + for i in 1..num_rows { + let end = list.offset_at(i + 1)?; + + let row_len = end + .checked_sub(prev_end) + .vortex_expect("list offsets must be monotonically increasing"); + if row_len != dim { + vortex_bail!( + "list_to_vector_ext: row {} has length {} but expected {}", + i, + row_len, + dim + ); + } + + prev_end = end; + } + + let expected_elements = num_rows + .checked_mul(dim) + .ok_or_else(|| vortex_err!("list_to_vector_ext: num_rows * dim overflows usize"))?; + if raw_elements.len() != expected_elements { + vortex_bail!( + "list_to_vector_ext: elements buffer has length {} but expected {}", + raw_elements.len(), + expected_elements + ); + } + + // Parquet has no non-nullable-element list logical type, so arrow-rs marks every + // `List`'s element field as nullable on read regardless of what the writer intended. + // That propagates through `DType::from_arrow`, so every real embedding parquet file arrives + // shaped as `List` even when every value is present. A nullable element dtype is + // losslessly convertible to a non-nullable FSL as long as the runtime validity is + // `NonNullable`/`AllValid`; we must only reject when a real null is present. + let elements = if elem_nullability.is_nullable() { + let primitive = raw_elements.as_opt::().ok_or_else(|| { + vortex_err!( + "list_to_vector_ext: expected nullable-float elements to downcast to \ + Primitive, got dtype {}", + raw_elements.dtype() + ) + })?; + match primitive.validity()? { + Validity::NonNullable | Validity::AllValid => { + // `to_host_sync` is a no-op for host-resident buffers, so this is a + // metadata change (rebuilding the array with a non-nullable dtype), + // not a data copy. + let byte_buffer = primitive.buffer_handle().to_host_sync(); + PrimitiveArray::from_byte_buffer(byte_buffer, *ptype, Validity::NonNullable) + .into_array() + } + Validity::AllInvalid => { + vortex_bail!( + "list_to_vector_ext: list has nullable element dtype with all-invalid \ + elements; Vector extension elements must be non-null" + ); + } + Validity::Array(_) => { + vortex_bail!( + "list_to_vector_ext: list has nullable element dtype with one or more \ + actual null elements; Vector extension elements must be non-null" + ); + } + } + } else { + raw_elements + }; + + let dim_u32 = u32::try_from(dim) + .map_err(|_| vortex_err!("list_to_vector_ext: dimension {dim} does not fit in u32"))?; + + // Finally, construct the `FixedSizeListArray` and wrap it in a Vector array. + let fsl = FixedSizeListArray::try_new(elements, dim_u32, Validity::NonNullable, num_rows)?; + let ext_dtype = ExtDType::::try_new(EmptyMetadata, fsl.dtype().clone())?.erased(); + Ok(ExtensionArray::new(ext_dtype, fsl.into_array()).into_array()) +} + +#[cfg(test)] +mod tests { + use vortex::array::Array; + use vortex::array::ArrayRef; + use vortex::array::IntoArray; + use vortex::array::arrays::BoolArray; + use vortex::array::arrays::ChunkedArray; + use vortex::array::arrays::Extension; + use vortex::array::arrays::List; + use vortex::array::arrays::ListViewArray; + use vortex::array::arrays::PrimitiveArray; + use vortex::array::arrays::extension::ExtensionArrayExt; + use vortex::array::validity::Validity; + use vortex::buffer::BufferMut; + use vortex::dtype::DType; + + use super::list_to_vector_ext; + + /// Build a `List` whose elements carry the given [`Validity`]. Passing + /// `Validity::NonNullable` produces a `List`; any other variant produces + /// a `List`, matching the shape `parquet_to_vortex_chunks` produces for + /// embedding columns after arrow-rs' canonicalization. + fn list_f32_with_element_validity( + values: &[f32], + dim: usize, + element_validity: Validity, + ) -> ArrayRef { + assert_eq!( + values.len() % dim, + 0, + "values.len() must be a multiple of dim" + ); + let num_rows = values.len() / dim; + let elements = PrimitiveArray::new::( + BufferMut::::from_iter(values.iter().copied()).freeze(), + element_validity, + ) + .into_array(); + let mut offsets_buf = BufferMut::::with_capacity(num_rows + 1); + for i in 0..=num_rows { + offsets_buf.push(i32::try_from(i * dim).unwrap()); + } + let offsets = + PrimitiveArray::new::(offsets_buf.freeze(), Validity::NonNullable).into_array(); + Array::::new(elements, offsets, Validity::NonNullable).into_array() + } + + fn list_f32(rows: &[&[f32]]) -> ArrayRef { + let mut elements = BufferMut::::with_capacity(rows.iter().map(|r| r.len()).sum()); + let mut offsets = BufferMut::::with_capacity(rows.len() + 1); + offsets.push(0); + for row in rows { + for &v in row.iter() { + elements.push(v); + } + offsets.push(i32::try_from(elements.len()).unwrap()); + } + + let elements_array = + PrimitiveArray::new::(elements.freeze(), Validity::NonNullable).into_array(); + let offsets_array = + PrimitiveArray::new::(offsets.freeze(), Validity::NonNullable).into_array(); + Array::::new(elements_array, offsets_array, Validity::NonNullable).into_array() + } + + #[test] + fn uniform_list_becomes_vector_extension() { + let list = list_f32(&[&[1.0, 2.0, 3.0], &[4.0, 5.0, 6.0], &[7.0, 8.0, 9.0]]); + let wrapped = list_to_vector_ext(list).unwrap(); + assert_eq!(wrapped.len(), 3); + let ext = wrapped.as_opt::().expect("returns Extension"); + assert!(matches!( + ext.storage_array().dtype(), + DType::FixedSizeList(_, 3, _) + )); + } + + #[test] + fn mismatched_row_length_is_rejected() { + let list = list_f32(&[&[1.0, 2.0, 3.0], &[4.0, 5.0]]); + let err = list_to_vector_ext(list).unwrap_err().to_string(); + assert!( + err.contains("row 1 has length 2 but expected 3"), + "unexpected error: {err}", + ); + } + + #[test] + fn non_list_input_is_rejected() { + let primitive = PrimitiveArray::new::( + BufferMut::::from_iter([1.0f32, 2.0, 3.0]).freeze(), + Validity::NonNullable, + ) + .into_array(); + let err = list_to_vector_ext(primitive).unwrap_err().to_string(); + assert!( + err.contains("expected a List array"), + "unexpected error: {err}" + ); + } + + #[test] + fn empty_input_is_rejected() { + let list = list_f32(&[]); + let err = list_to_vector_ext(list).unwrap_err().to_string(); + assert!( + err.contains("cannot infer vector dimension from empty input"), + "unexpected error: {err}", + ); + } + + /// Build a `ListView` whose every row is a length-`dim` slice of the flattened + /// `values` buffer. This shape matches what `parquet_to_vortex_chunks` produces for + /// embedding columns after arrow-rs' canonicalization, and exercises the + /// `list_to_vector_ext` fast-path that collapses `ListView` → `List` before + /// validating offsets. + fn list_view_f32(dim: usize, rows: &[&[f32]]) -> ArrayRef { + let mut values = BufferMut::::with_capacity(rows.len() * dim); + for row in rows { + assert_eq!(row.len(), dim); + for &v in row.iter() { + values.push(v); + } + } + let elements = + PrimitiveArray::new::(values.freeze(), Validity::NonNullable).into_array(); + + let dim_i32 = i32::try_from(dim).unwrap(); + let num_rows = rows.len(); + + let mut offsets_buf = BufferMut::::with_capacity(num_rows); + for i in 0..num_rows { + offsets_buf.push(i32::try_from(i).unwrap() * dim_i32); + } + let offsets = + PrimitiveArray::new::(offsets_buf.freeze(), Validity::NonNullable).into_array(); + + let mut sizes_buf = BufferMut::::with_capacity(num_rows); + for _ in 0..num_rows { + sizes_buf.push(dim_i32); + } + let sizes = + PrimitiveArray::new::(sizes_buf.freeze(), Validity::NonNullable).into_array(); + + ListViewArray::try_new(elements, offsets, sizes, Validity::NonNullable) + .unwrap() + .into_array() + } + + #[test] + fn list_view_input_is_rewrapped_as_vector_extension() { + // Simulates the post-parquet-ingest shape: the `emb` column arrives as a + // ListView, not a List. `list_to_vector_ext` must materialize it via + // `recursive_list_from_list_view` and then validate offsets on the flattened + // `List` form. + let list_view = list_view_f32(3, &[&[1.0, 2.0, 3.0], &[4.0, 5.0, 6.0]]); + let wrapped = list_to_vector_ext(list_view).unwrap(); + assert_eq!(wrapped.len(), 2); + let ext = wrapped.as_opt::().expect("returns Extension"); + assert!(matches!( + ext.storage_array().dtype(), + DType::FixedSizeList(_, 3, _) + )); + } + + #[test] + fn all_invalid_list_validity_is_rejected() { + // A list with `Validity::AllInvalid` means every row is null. The Vector + // extension type requires non-nullable elements at the FSL level, so we + // must reject this input rather than silently dropping the validity mask. + let elements = PrimitiveArray::new::( + BufferMut::::from_iter([1.0f32, 2.0, 3.0, 4.0, 5.0, 6.0]).freeze(), + Validity::NonNullable, + ) + .into_array(); + let offsets = PrimitiveArray::new::( + BufferMut::::from_iter([0i32, 3, 6]).freeze(), + Validity::NonNullable, + ) + .into_array(); + let list = Array::::new(elements, offsets, Validity::AllInvalid).into_array(); + + let err = list_to_vector_ext(list).unwrap_err().to_string(); + assert!( + err.contains("list rows must be non-nullable"), + "unexpected error: {err}" + ); + } + + #[test] + fn non_float_element_type_is_rejected() { + // Build a List. + let elements = PrimitiveArray::new::( + BufferMut::::from_iter([1i32, 2, 3, 4]).freeze(), + Validity::NonNullable, + ) + .into_array(); + let offsets = PrimitiveArray::new::( + BufferMut::::from_iter([0i32, 2, 4]).freeze(), + Validity::NonNullable, + ) + .into_array(); + let list = Array::::new(elements, offsets, Validity::NonNullable).into_array(); + + let err = list_to_vector_ext(list).unwrap_err().to_string(); + assert!( + err.contains("element type must be float"), + "unexpected error: {err}", + ); + } + + #[test] + fn nullable_elements_with_real_nulls_are_rejected() { + // A `List` whose elements carry a real `Validity::Array` mask with + // at least one `false` bit has one or more actually-missing values. The + // rejection here is about runtime nulls, not dtype metadata: a nullable + // element dtype with all-valid runtime validity is accepted (see + // `nullable_element_dtype_with_all_valid_elements_is_accepted`), because + // parquet-ingested embeddings always arrive shaped that way even when + // every value is present. A real null, on the other hand, cannot be + // represented in the Vector extension FSL and must be rejected rather + // than silently dropped. + let element_validity = Validity::Array( + BoolArray::from_iter([true, true, false, true, true, true]).into_array(), + ); + let list = + list_f32_with_element_validity(&[1.0, 2.0, 3.0, 4.0, 5.0, 6.0], 3, element_validity); + + let err = list_to_vector_ext(list).unwrap_err().to_string(); + assert!( + err.contains("one or more actual null elements"), + "unexpected error: {err}" + ); + } + + #[test] + fn nullable_element_dtype_with_all_valid_elements_is_accepted() { + // This is the regression test for the Cohere parquet case: every real + // VectorDBBench parquet file arrives as `List` with + // `Validity::AllValid` elements because parquet has no non-nullable + // list-element logical type and arrow-rs propagates the nullable bit + // through `DType::from_arrow`. `list_to_vector_ext` must accept this + // shape by rewrapping the elements as non-nullable before building the + // FSL, rather than rejecting outright on the dtype metadata. + let list = + list_f32_with_element_validity(&[1.0, 2.0, 3.0, 4.0, 5.0, 6.0], 3, Validity::AllValid); + + let wrapped = list_to_vector_ext(list).unwrap(); + assert_eq!(wrapped.len(), 2); + let ext = wrapped.as_opt::().expect("returns Extension"); + assert!(matches!( + ext.storage_array().dtype(), + DType::FixedSizeList(_, 3, _) + )); + } + + #[test] + fn nullable_element_dtype_with_all_invalid_elements_is_rejected() { + // A `List` whose elements are `Validity::AllInvalid` means every + // value is missing. Rewrapping as non-nullable would silently drop the + // validity and produce bogus vectors, so this must be rejected. + let list = list_f32_with_element_validity( + &[1.0, 2.0, 3.0, 4.0, 5.0, 6.0], + 3, + Validity::AllInvalid, + ); + + let err = list_to_vector_ext(list).unwrap_err().to_string(); + assert!( + err.contains("all-invalid elements"), + "unexpected error: {err}" + ); + } + + #[test] + fn chunked_input_with_mixed_dimensions_returns_error() { + let dim_three = list_f32(&[&[1.0, 2.0, 3.0]]); + let dim_two = list_f32(&[&[4.0, 5.0]]); + let chunked = + ChunkedArray::try_new(vec![dim_three.clone(), dim_two], dim_three.dtype().clone()) + .unwrap() + .into_array(); + + let err = list_to_vector_ext(chunked).unwrap_err().to_string(); + assert!(err.contains("Mismatched types"), "unexpected error: {err}"); + } +} diff --git a/vortex-bench/src/vector_dataset/download.rs b/vortex-bench/src/vector_dataset/download.rs new file mode 100644 index 00000000000..a7b7d8c6e89 --- /dev/null +++ b/vortex-bench/src/vector_dataset/download.rs @@ -0,0 +1,133 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! URL builders and idempotent download driver for vector benchmark datasets. +//! +//! The upstream bucket is `https://assets.zilliz.com/benchmark//`. Within each +//! prefix the train split is named according to a four-way convention: +//! +//! - `Single`: `train.parquet` +//! - `SingleShuffled`: `shuffle_train.parquet` +//! - `Partitioned`: `train-NN-of-MM.parquet` +//! - `PartitionedShuffled`: `shuffle_train-NN-of-MM.parquet` +//! +//! `test.parquet` and (when present) `neighbors.parquet` live alongside the train files. + +use std::path::PathBuf; + +use anyhow::Context; +use anyhow::Result; + +use crate::datasets::data_downloads::download_data; +use crate::datasets::data_downloads::download_many; +use crate::vector_dataset::catalog::VectorDataset; +use crate::vector_dataset::layout::LayoutSpec; +use crate::vector_dataset::layout::TrainLayout; +use crate::vector_dataset::paths; + +/// Bucket root for all VectorDBBench datasets we mirror against. +const BENCHMARK_ROOT: &str = "https://assets.zilliz.com/benchmark"; + +/// All train-shard URLs for a `(dataset, layout)` pair. Length matches `layout.num_files()`. +pub fn train_urls(ds: VectorDataset, spec: LayoutSpec) -> Vec { + let prefix = format!("{BENCHMARK_ROOT}/{}", ds.s3_prefix()); + let layout = spec.layout(); + if layout.is_partitioned() { + let n = spec.num_files(); + (0..n) + .map(|i| format!("{prefix}/{}", partitioned_file_name(layout, i, n),)) + .collect() + } else { + let name = match layout { + TrainLayout::Single => "train.parquet", + TrainLayout::SingleShuffled => "shuffle_train.parquet", + _ => unreachable!("non-partitioned guard above"), + }; + vec![format!("{prefix}/{name}")] + } +} + +/// URL for `test.parquet`. +pub fn test_url(ds: VectorDataset) -> String { + format!("{BENCHMARK_ROOT}/{}/test.parquet", ds.s3_prefix()) +} + +/// URL for `neighbors.parquet`, or `None` when the dataset doesn't host one. +pub fn neighbors_url(ds: VectorDataset) -> Option { + ds.has_neighbors() + .then(|| format!("{BENCHMARK_ROOT}/{}/neighbors.parquet", ds.s3_prefix())) +} + +fn partitioned_file_name(layout: TrainLayout, shard_idx: u32, num_files: u32) -> String { + let prefix = match layout { + TrainLayout::Partitioned => "train", + TrainLayout::PartitionedShuffled => "shuffle_train", + _ => unreachable!("partitioned guard"), + }; + format!( + "{prefix}-{shard_idx:0width$}-of-{num_files:0width$}.parquet", + width = num_files_width(num_files), + ) +} + +fn num_files_width(num_files: u32) -> usize { + let digits = num_files.checked_ilog10().unwrap_or(0) as usize + 1; + digits.max(2) +} + +/// Local on-disk paths to the cached parquet files for a `(dataset, layout)` pair after +/// [`download`] returns successfully. +#[derive(Debug, Clone)] +pub struct DatasetPaths { + /// Per-shard train parquet paths in shard order. + pub train_files: Vec, + /// `test.parquet`. + pub test: PathBuf, + /// `neighbors.parquet` if the dataset hosts top-K ground truth. + pub neighbors: Option, +} + +/// Download every parquet file required to run a `(dataset, layout)` benchmark, returning local +/// on-disk paths. +/// +/// This has idempotent semantics, so files already present on disk are skipped, and re-runs only +/// pay for new files. +/// +/// Train shards download via [`download_many`] with adaptive parallelism; the small +/// `test.parquet` and `neighbors.parquet` files use the simple [`download_data`] helper. +/// All HTTP requests share a single pooled client. +pub async fn download(ds: VectorDataset, layout: TrainLayout) -> Result { + let spec = ds.validate_layout(layout)?; + let urls = train_urls(ds, spec); + let train_targets = paths::train_files(ds, layout, spec.num_files()); + debug_assert_eq!(urls.len(), train_targets.len()); + + let train_downloads: Vec<(PathBuf, String)> = train_targets + .iter() + .cloned() + .zip(urls.into_iter()) + .collect(); + download_many(train_downloads) + .await + .with_context(|| format!("download train shards for {}", ds.name()))?; + + let test = download_data(paths::test_path(ds, layout), &test_url(ds)) + .await + .with_context(|| format!("download test.parquet for {}", ds.name()))?; + + let neighbors = if let Some(url) = neighbors_url(ds) { + Some( + download_data(paths::neighbors_path(ds, layout), &url) + .await + .with_context(|| format!("download neighbors.parquet for {}", ds.name()))?, + ) + } else { + None + }; + + Ok(DatasetPaths { + train_files: train_targets, + test, + neighbors, + }) +} diff --git a/vortex-bench/src/vector_dataset/layout.rs b/vortex-bench/src/vector_dataset/layout.rs new file mode 100644 index 00000000000..3356776d0c3 --- /dev/null +++ b/vortex-bench/src/vector_dataset/layout.rs @@ -0,0 +1,185 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Train-split layout variants for vector benchmark datasets. +//! +//! VectorDBBench corpora are published under `assets.zilliz.com/benchmark//` in up to four +//! different shapes: +//! - a single train file +//! - a single shuffled-rows train file +//! - a partitioned train file split into N shards +//! - the same partitioned shape in shuffled-rows order. +//! +//! Not every dataset hosts every layout. See [`VectorDataset::layouts`] for the per-dataset list. +//! +//! [`VectorDataset::layouts`]: super::VectorDataset::layouts + +use std::fmt; +use std::num::NonZeroU32; + +use clap::ValueEnum; +use serde::Deserialize; +use serde::Serialize; + +/// Distance metric a dataset was curated for. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum VectorMetric { + /// `dot(a, b) / (||a|| * ||b||)`. + Cosine, + /// `sum((a - b)^2)`. + L2, + // TODO(connor): Do we even need this? + /// `dot(a, b)`. + InnerProduct, +} + +/// A specific train layout published for a dataset, plus the shard count when partitioned +/// (always `1` for `Single` / `SingleShuffled`). +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct LayoutSpec { + layout: TrainLayout, + num_files: NonZeroU32, +} + +impl LayoutSpec { + /// Build a single-file layout spec. + pub const fn single() -> Self { + Self { + layout: TrainLayout::Single, + num_files: NonZeroU32::MIN, + } + } + + /// Build a shuffled single-file layout spec. + pub const fn single_shuffled() -> Self { + Self { + layout: TrainLayout::SingleShuffled, + num_files: NonZeroU32::MIN, + } + } + + /// Build a partitioned layout spec with the given shard count. + pub const fn partitioned(num_files: NonZeroU32) -> Self { + Self { + layout: TrainLayout::Partitioned, + num_files, + } + } + + /// Build a shuffled partitioned layout spec with the given shard count. + pub const fn partitioned_shuffled(num_files: NonZeroU32) -> Self { + Self { + layout: TrainLayout::PartitionedShuffled, + num_files, + } + } + + /// Which of the four published shapes this entry describes. + pub const fn layout(&self) -> TrainLayout { + self.layout + } + + /// Number of parquet shards on the bucket. `1` for the single-file layouts. + pub const fn num_files(&self) -> u32 { + self.num_files.get() + } +} + +/// One of the four published train-split shapes for a VectorDBBench corpus. +/// +/// `Single` and `SingleShuffled` are one-file layouts; `Partitioned` and `PartitionedShuffled` are +/// sharded into N files. The shuffled variants randomize the row order, which is useful when you +/// want the on-disk arrangement to be representative of a query workload rather than of the +/// upstream ingest order. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, ValueEnum, Serialize, Deserialize)] +#[serde(rename_all = "kebab-case")] +pub enum TrainLayout { + /// One `train.parquet` file. Row order matches the upstream curation. + #[clap(name = "single")] + Single, + /// One `shuffle_train.parquet` file. Row order is randomized. + #[clap(name = "single-shuffled")] + SingleShuffled, + /// Multiple `train-NN-of-N.parquet` shards. Row order matches the upstream curation. + #[clap(name = "partitioned")] + Partitioned, + /// Multiple `shuffle_train-NN-of-N.parquet` shards. Row order is randomized. + #[clap(name = "partitioned-shuffled")] + PartitionedShuffled, +} + +impl TrainLayout { + /// Stable kebab-cased label used in CLI args, file paths, and metric names. + pub fn label(&self) -> &'static str { + match self { + TrainLayout::Single => "single", + TrainLayout::SingleShuffled => "single-shuffled", + TrainLayout::Partitioned => "partitioned", + TrainLayout::PartitionedShuffled => "partitioned-shuffled", + } + } + + /// Whether this layout is split across multiple parquet files. + pub fn is_partitioned(&self) -> bool { + matches!( + self, + TrainLayout::Partitioned | TrainLayout::PartitionedShuffled + ) + } +} + +impl fmt::Display for TrainLayout { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str(self.label()) + } +} + +#[cfg(test)] +mod tests { + use std::num::NonZeroU32; + + use super::*; + + #[test] + fn label_round_trips_through_value_enum() { + for layout in [ + TrainLayout::Single, + TrainLayout::SingleShuffled, + TrainLayout::Partitioned, + TrainLayout::PartitionedShuffled, + ] { + let parsed = TrainLayout::from_str(layout.label(), true).unwrap(); + assert_eq!(parsed, layout); + } + } + + #[test] + fn is_partitioned_matches_variant() { + assert!(!TrainLayout::Single.is_partitioned()); + assert!(!TrainLayout::SingleShuffled.is_partitioned()); + assert!(TrainLayout::Partitioned.is_partitioned()); + assert!(TrainLayout::PartitionedShuffled.is_partitioned()); + } + + #[test] + fn layout_specs_encode_valid_shape_and_count() { + assert_eq!(LayoutSpec::single().layout(), TrainLayout::Single); + assert_eq!(LayoutSpec::single().num_files(), 1); + assert_eq!( + LayoutSpec::single_shuffled().layout(), + TrainLayout::SingleShuffled + ); + assert_eq!( + LayoutSpec::partitioned(NonZeroU32::new(10).unwrap()).layout(), + TrainLayout::Partitioned + ); + assert_eq!( + LayoutSpec::partitioned_shuffled(NonZeroU32::new(10).unwrap()).layout(), + TrainLayout::PartitionedShuffled + ); + assert_eq!( + LayoutSpec::partitioned_shuffled(NonZeroU32::new(10).unwrap()).num_files(), + 10 + ); + } +} diff --git a/vortex-bench/src/vector_dataset/mod.rs b/vortex-bench/src/vector_dataset/mod.rs new file mode 100644 index 00000000000..d826aa9fc5d --- /dev/null +++ b/vortex-bench/src/vector_dataset/mod.rs @@ -0,0 +1,40 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Public catalog of VectorDBBench corpora used by the vector-search benchmark. +//! +//! The catalog is intentionally separate from the [`crate::datasets::Dataset`] trait used by the +//! row-table benchmarks: the train split of a vector dataset is sometimes partitioned across many +//! parquet files, sometimes single-file, sometimes shuffled, sometimes not, and its `emb` column +//! has to be rewrapped into an `Extension` before it's useful to a cosine-similarity scan. +//! None of that fits the row-table `Dataset` contract. +//! +//! The four sub-modules split the catalog into roughly orthogonal concerns: +//! +//! - `catalog`: the static [`VectorDataset`] enum + per-dataset metadata. +//! - `layout`: [`TrainLayout`] / [`LayoutSpec`] (the four hosted train shapes) and +//! [`VectorMetric`]. +//! - `download`: URL builders and the idempotent download driver. +//! - `paths`: local filesystem layout (`/vector-search/...`). +//! +//! Higher-level callers (the bench crate's ingest + scan pipeline) compose these: +//! [`download::download`] returns a [`download::DatasetPaths`] handle that the ingest pass turns +//! into per-flavor `.vortex` files, after which the scan driver re-opens those files per iteration. + +mod catalog; +mod convert; +mod download; +mod layout; +mod paths; + +pub use catalog::ALL_VECTOR_DATASETS; +pub use catalog::VectorDataset; +pub use convert::list_to_vector_ext; +pub use download::DatasetPaths; +pub use download::download; +pub use download::neighbors_url; +pub use download::test_url; +pub use download::train_urls; +pub use layout::LayoutSpec; +pub use layout::TrainLayout; +pub use layout::VectorMetric; diff --git a/vortex-bench/src/vector_dataset/paths.rs b/vortex-bench/src/vector_dataset/paths.rs new file mode 100644 index 00000000000..3fbf9452ff7 --- /dev/null +++ b/vortex-bench/src/vector_dataset/paths.rs @@ -0,0 +1,116 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Local-filesystem layout for cached vector benchmark datasets. +//! +//! ```text +//! /vector-search/// +//! train<> single-file: train.parquet +//! partitioned: 00-of-N.parquet, 01-of-N.parquet, ... +//! test.parquet +//! neighbors.parquet only when ds.has_neighbors() +//! +//! + some more things +//! ``` +//! +//! This module exists purely to centralize the path-construction logic used by both the downloader +//! and the ingest pipeline. + +use std::path::PathBuf; + +use crate::utils::file::data_dir; +use crate::vector_dataset::VectorDataset; +use crate::vector_dataset::layout::TrainLayout; + +/// Top-level cache root: `/vector-search/`. +pub fn root() -> PathBuf { + data_dir().join("vector-search") +} + +/// Per-dataset directory: `///`. +pub fn dataset_dir(ds: VectorDataset, layout: TrainLayout) -> PathBuf { + root().join(ds.name()).join(layout.label()) +} + +/// Train-shard directory: `/train/`. +pub fn train_dir(ds: VectorDataset, layout: TrainLayout) -> PathBuf { + dataset_dir(ds, layout).join("train") +} + +/// File name for one train shard within [`train_dir`]. +/// +/// Single-file layouts produce `train.parquet`; partitioned layouts produce `NN-of-MM.parquet` so a +/// directory listing sorts shards in sequence order. +pub fn train_file_name(layout: TrainLayout, shard_idx: u32, num_files: u32) -> String { + if layout.is_partitioned() { + format!( + "{shard_idx:0width$}-of-{num_files:0width$}.parquet", + width = num_files_width(num_files), + ) + } else { + "train.parquet".to_owned() + } +} + +/// All train-shard paths for a dataset/layout pair, in shard order. +pub fn train_files(ds: VectorDataset, layout: TrainLayout, num_files: u32) -> Vec { + let dir = train_dir(ds, layout); + (0..num_files) + .map(|i| dir.join(train_file_name(layout, i, num_files))) + .collect() +} + +/// Path to the cached `test.parquet` for a dataset/layout pair. +pub fn test_path(ds: VectorDataset, layout: TrainLayout) -> PathBuf { + dataset_dir(ds, layout).join("test.parquet") +} + +/// Path to the cached `neighbors.parquet` for a dataset/layout pair. +pub fn neighbors_path(ds: VectorDataset, layout: TrainLayout) -> PathBuf { + dataset_dir(ds, layout).join("neighbors.parquet") +} + +/// Width used to zero-pad shard indices in partitioned filenames. `10` shards is 2 digits, `100` +/// shards is 3 digits. +fn num_files_width(num_files: u32) -> usize { + let digits = num_files.checked_ilog10().unwrap_or(0) as usize + 1; + digits.max(2) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn single_layout_uses_train_parquet() { + assert_eq!(train_file_name(TrainLayout::Single, 0, 1), "train.parquet"); + } + + #[test] + fn partitioned_filename_zero_pads_to_match_total() { + assert_eq!( + train_file_name(TrainLayout::Partitioned, 0, 10), + "00-of-10.parquet" + ); + assert_eq!( + train_file_name(TrainLayout::Partitioned, 9, 10), + "09-of-10.parquet" + ); + assert_eq!( + train_file_name(TrainLayout::Partitioned, 99, 100), + "099-of-100.parquet" + ); + } + + #[test] + fn train_files_lists_all_shards_in_order() { + let files = train_files(VectorDataset::CohereLarge10m, TrainLayout::Partitioned, 10); + assert_eq!(files.len(), 10); + for (i, path) in files.iter().enumerate() { + assert!( + path.to_string_lossy() + .ends_with(&format!("{i:02}-of-10.parquet")) + ); + } + } +} diff --git a/vortex-btrblocks/Cargo.toml b/vortex-btrblocks/Cargo.toml index 35383a5f5ed..40b0ae52aae 100644 --- a/vortex-btrblocks/Cargo.toml +++ b/vortex-btrblocks/Cargo.toml @@ -14,7 +14,6 @@ rust-version = { workspace = true } version = { workspace = true } [dependencies] -getrandom_v03 = { workspace = true } itertools = { workspace = true } num-traits = { workspace = true } pco = { workspace = true, optional = true } @@ -45,6 +44,7 @@ divan = { workspace = true } rstest = { workspace = true } test-with = { workspace = true } vortex-array = { workspace = true, features = ["_test-harness"] } +vortex-session = { workspace = true } [features] # This feature enabled unstable encodings for which we don't guarantee stability. @@ -64,6 +64,3 @@ test = false name = "compress_listview" harness = false test = false - -[package.metadata.cargo-machete] -ignored = ["getrandom_v03"] diff --git a/vortex-btrblocks/benches/compress.rs b/vortex-btrblocks/benches/compress.rs index 18a48881678..8bc47d16404 100644 --- a/vortex-btrblocks/benches/compress.rs +++ b/vortex-btrblocks/benches/compress.rs @@ -1,11 +1,12 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] -#![allow(unexpected_cfgs)] +#![expect(clippy::unwrap_used)] #[cfg(not(codspeed))] mod benchmarks { + use std::sync::LazyLock; + use divan::Bencher; use divan::counter::BytesCount; use divan::counter::ItemsCount; @@ -14,11 +15,17 @@ mod benchmarks { use rand::prelude::StdRng; use vortex_array::ArrayRef; use vortex_array::IntoArray; - use vortex_array::ToCanonical; + use vortex_array::VortexSessionExecute; + use vortex_array::arrays::PrimitiveArray; + use vortex_array::session::ArraySession; use vortex_btrblocks::BtrBlocksCompressor; use vortex_buffer::buffer_mut; + use vortex_session::VortexSession; use vortex_utils::aliases::hash_set::HashSet; + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); + fn make_clickbench_window_name() -> ArrayRef { // A test that's meant to mirror the WindowName column from ClickBench. let mut values = buffer_mut![-1i32; 65_536]; @@ -40,13 +47,20 @@ mod benchmarks { #[divan::bench] fn btrblocks(bencher: Bencher) { - let array = make_clickbench_window_name().to_primitive(); + let mut ctx = SESSION.create_execution_ctx(); + let array = make_clickbench_window_name() + .execute::(&mut ctx) + .unwrap(); let compressor = BtrBlocksCompressor::default(); bencher - .with_inputs(|| &array) - .input_counter(|array| ItemsCount::new(array.len())) - .input_counter(|array| BytesCount::of_many::(array.len())) - .bench_refs(|array| compressor.compress(&array.clone().into_array()).unwrap()); + .with_inputs(|| (&array, SESSION.create_execution_ctx())) + .input_counter(|(array, _)| ItemsCount::new(array.len())) + .input_counter(|(array, _)| BytesCount::of_many::(array.len())) + .bench_refs(|(array, ctx)| { + compressor + .compress(&array.clone().into_array(), ctx) + .unwrap() + }); } } diff --git a/vortex-btrblocks/benches/compress_listview.rs b/vortex-btrblocks/benches/compress_listview.rs index 11f5f92437e..e1b0595ac27 100644 --- a/vortex-btrblocks/benches/compress_listview.rs +++ b/vortex-btrblocks/benches/compress_listview.rs @@ -1,12 +1,13 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] -#![allow(clippy::cast_possible_truncation)] -#![allow(unexpected_cfgs)] +#![expect(clippy::unwrap_used)] +#![expect(clippy::cast_possible_truncation)] #[cfg(not(codspeed))] mod benchmarks { + use std::sync::LazyLock; + use divan::Bencher; use divan::counter::BytesCount; use divan::counter::ItemsCount; @@ -15,17 +16,23 @@ mod benchmarks { use rand::prelude::StdRng; use vortex_array::ArrayRef; use vortex_array::IntoArray; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::ListViewArray; use vortex_array::arrays::StructArray; use vortex_array::arrays::VarBinViewArray; use vortex_array::dtype::FieldNames; + use vortex_array::session::ArraySession; use vortex_array::validity::Validity; use vortex_btrblocks::BtrBlocksCompressor; use vortex_buffer::buffer_mut; + use vortex_session::VortexSession; const NUM_ROWS: usize = 8192; const SEED: u64 = 42; + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); + const SHORT_STRINGS: &[&str] = &[ "alpha_one", "bravo_two", @@ -178,10 +185,10 @@ mod benchmarks { let nbytes = array.nbytes(); let compressor = BtrBlocksCompressor::default(); bencher - .with_inputs(|| &array) + .with_inputs(|| (&array, SESSION.create_execution_ctx())) .input_counter(|_| ItemsCount::new(NUM_ROWS)) .input_counter(move |_| BytesCount::new(nbytes as usize)) - .bench_refs(|array| compressor.compress(array).unwrap()); + .bench_refs(|(array, ctx)| compressor.compress(array, ctx).unwrap()); } } diff --git a/vortex-btrblocks/public-api.lock b/vortex-btrblocks/public-api.lock index 06928421496..797365ae043 100644 --- a/vortex-btrblocks/public-api.lock +++ b/vortex-btrblocks/public-api.lock @@ -44,11 +44,11 @@ impl core::cmp::Eq for vortex_btrblocks::schemes::decimal::DecimalScheme impl core::cmp::PartialEq for vortex_btrblocks::schemes::decimal::DecimalScheme -pub fn vortex_btrblocks::schemes::decimal::DecimalScheme::eq(&self, other: &vortex_btrblocks::schemes::decimal::DecimalScheme) -> bool +pub fn vortex_btrblocks::schemes::decimal::DecimalScheme::eq(&self, &vortex_btrblocks::schemes::decimal::DecimalScheme) -> bool impl core::fmt::Debug for vortex_btrblocks::schemes::decimal::DecimalScheme -pub fn vortex_btrblocks::schemes::decimal::DecimalScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_btrblocks::schemes::decimal::DecimalScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_btrblocks::schemes::decimal::DecimalScheme @@ -56,11 +56,11 @@ impl core::marker::StructuralPartialEq for vortex_btrblocks::schemes::decimal::D impl vortex_compressor::scheme::Scheme for vortex_btrblocks::schemes::decimal::DecimalScheme -pub fn vortex_btrblocks::schemes::decimal::DecimalScheme::compress(&self, compressor: &vortex_compressor::compressor::CascadingCompressor, data: &mut vortex_compressor::stats::cache::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_btrblocks::schemes::decimal::DecimalScheme::compress(&self, &vortex_compressor::compressor::CascadingCompressor, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_btrblocks::schemes::decimal::DecimalScheme::expected_compression_ratio(&self, _data: &mut vortex_compressor::stats::cache::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_btrblocks::schemes::decimal::DecimalScheme::expected_compression_ratio(&self, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_btrblocks::schemes::decimal::DecimalScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_btrblocks::schemes::decimal::DecimalScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_btrblocks::schemes::decimal::DecimalScheme::num_children(&self) -> usize @@ -86,11 +86,11 @@ impl core::cmp::Eq for vortex_btrblocks::schemes::float::ALPRDScheme impl core::cmp::PartialEq for vortex_btrblocks::schemes::float::ALPRDScheme -pub fn vortex_btrblocks::schemes::float::ALPRDScheme::eq(&self, other: &vortex_btrblocks::schemes::float::ALPRDScheme) -> bool +pub fn vortex_btrblocks::schemes::float::ALPRDScheme::eq(&self, &vortex_btrblocks::schemes::float::ALPRDScheme) -> bool impl core::fmt::Debug for vortex_btrblocks::schemes::float::ALPRDScheme -pub fn vortex_btrblocks::schemes::float::ALPRDScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_btrblocks::schemes::float::ALPRDScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_btrblocks::schemes::float::ALPRDScheme @@ -98,11 +98,11 @@ impl core::marker::StructuralPartialEq for vortex_btrblocks::schemes::float::ALP impl vortex_compressor::scheme::Scheme for vortex_btrblocks::schemes::float::ALPRDScheme -pub fn vortex_btrblocks::schemes::float::ALPRDScheme::compress(&self, _compressor: &vortex_compressor::compressor::CascadingCompressor, data: &mut vortex_compressor::stats::cache::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_btrblocks::schemes::float::ALPRDScheme::compress(&self, &vortex_compressor::compressor::CascadingCompressor, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_btrblocks::schemes::float::ALPRDScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::cache::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_btrblocks::schemes::float::ALPRDScheme::expected_compression_ratio(&self, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_btrblocks::schemes::float::ALPRDScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_btrblocks::schemes::float::ALPRDScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_btrblocks::schemes::float::ALPRDScheme::scheme_name(&self) -> &'static str @@ -116,11 +116,11 @@ impl core::cmp::Eq for vortex_btrblocks::schemes::float::ALPScheme impl core::cmp::PartialEq for vortex_btrblocks::schemes::float::ALPScheme -pub fn vortex_btrblocks::schemes::float::ALPScheme::eq(&self, other: &vortex_btrblocks::schemes::float::ALPScheme) -> bool +pub fn vortex_btrblocks::schemes::float::ALPScheme::eq(&self, &vortex_btrblocks::schemes::float::ALPScheme) -> bool impl core::fmt::Debug for vortex_btrblocks::schemes::float::ALPScheme -pub fn vortex_btrblocks::schemes::float::ALPScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_btrblocks::schemes::float::ALPScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_btrblocks::schemes::float::ALPScheme @@ -128,11 +128,11 @@ impl core::marker::StructuralPartialEq for vortex_btrblocks::schemes::float::ALP impl vortex_compressor::scheme::Scheme for vortex_btrblocks::schemes::float::ALPScheme -pub fn vortex_btrblocks::schemes::float::ALPScheme::compress(&self, compressor: &vortex_compressor::compressor::CascadingCompressor, data: &mut vortex_compressor::stats::cache::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_btrblocks::schemes::float::ALPScheme::compress(&self, &vortex_compressor::compressor::CascadingCompressor, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_btrblocks::schemes::float::ALPScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::cache::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_btrblocks::schemes::float::ALPScheme::expected_compression_ratio(&self, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_btrblocks::schemes::float::ALPScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_btrblocks::schemes::float::ALPScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_btrblocks::schemes::float::ALPScheme::num_children(&self) -> usize @@ -148,11 +148,11 @@ impl core::cmp::Eq for vortex_btrblocks::schemes::float::FloatRLEScheme impl core::cmp::PartialEq for vortex_btrblocks::schemes::float::FloatRLEScheme -pub fn vortex_btrblocks::schemes::float::FloatRLEScheme::eq(&self, other: &vortex_btrblocks::schemes::float::FloatRLEScheme) -> bool +pub fn vortex_btrblocks::schemes::float::FloatRLEScheme::eq(&self, &vortex_btrblocks::schemes::float::FloatRLEScheme) -> bool impl core::fmt::Debug for vortex_btrblocks::schemes::float::FloatRLEScheme -pub fn vortex_btrblocks::schemes::float::FloatRLEScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_btrblocks::schemes::float::FloatRLEScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_btrblocks::schemes::float::FloatRLEScheme @@ -162,13 +162,13 @@ impl vortex_compressor::scheme::Scheme for vortex_btrblocks::schemes::float::Flo pub fn vortex_btrblocks::schemes::float::FloatRLEScheme::ancestor_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_btrblocks::schemes::float::FloatRLEScheme::compress(&self, compressor: &vortex_compressor::compressor::CascadingCompressor, data: &mut vortex_compressor::stats::cache::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_btrblocks::schemes::float::FloatRLEScheme::compress(&self, &vortex_compressor::compressor::CascadingCompressor, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_btrblocks::schemes::float::FloatRLEScheme::descendant_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_btrblocks::schemes::float::FloatRLEScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::cache::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_btrblocks::schemes::float::FloatRLEScheme::expected_compression_ratio(&self, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_btrblocks::schemes::float::FloatRLEScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_btrblocks::schemes::float::FloatRLEScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_btrblocks::schemes::float::FloatRLEScheme::num_children(&self) -> usize @@ -184,11 +184,11 @@ impl core::cmp::Eq for vortex_btrblocks::schemes::float::NullDominatedSparseSche impl core::cmp::PartialEq for vortex_btrblocks::schemes::float::NullDominatedSparseScheme -pub fn vortex_btrblocks::schemes::float::NullDominatedSparseScheme::eq(&self, other: &vortex_btrblocks::schemes::float::NullDominatedSparseScheme) -> bool +pub fn vortex_btrblocks::schemes::float::NullDominatedSparseScheme::eq(&self, &vortex_btrblocks::schemes::float::NullDominatedSparseScheme) -> bool impl core::fmt::Debug for vortex_btrblocks::schemes::float::NullDominatedSparseScheme -pub fn vortex_btrblocks::schemes::float::NullDominatedSparseScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_btrblocks::schemes::float::NullDominatedSparseScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_btrblocks::schemes::float::NullDominatedSparseScheme @@ -196,13 +196,13 @@ impl core::marker::StructuralPartialEq for vortex_btrblocks::schemes::float::Nul impl vortex_compressor::scheme::Scheme for vortex_btrblocks::schemes::float::NullDominatedSparseScheme -pub fn vortex_btrblocks::schemes::float::NullDominatedSparseScheme::compress(&self, compressor: &vortex_compressor::compressor::CascadingCompressor, data: &mut vortex_compressor::stats::cache::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_btrblocks::schemes::float::NullDominatedSparseScheme::compress(&self, &vortex_compressor::compressor::CascadingCompressor, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_btrblocks::schemes::float::NullDominatedSparseScheme::descendant_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_btrblocks::schemes::float::NullDominatedSparseScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::cache::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_btrblocks::schemes::float::NullDominatedSparseScheme::expected_compression_ratio(&self, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_btrblocks::schemes::float::NullDominatedSparseScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_btrblocks::schemes::float::NullDominatedSparseScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_btrblocks::schemes::float::NullDominatedSparseScheme::num_children(&self) -> usize @@ -218,11 +218,11 @@ impl core::cmp::Eq for vortex_btrblocks::schemes::float::PcoScheme impl core::cmp::PartialEq for vortex_btrblocks::schemes::float::PcoScheme -pub fn vortex_btrblocks::schemes::float::PcoScheme::eq(&self, other: &vortex_btrblocks::schemes::float::PcoScheme) -> bool +pub fn vortex_btrblocks::schemes::float::PcoScheme::eq(&self, &vortex_btrblocks::schemes::float::PcoScheme) -> bool impl core::fmt::Debug for vortex_btrblocks::schemes::float::PcoScheme -pub fn vortex_btrblocks::schemes::float::PcoScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_btrblocks::schemes::float::PcoScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_btrblocks::schemes::float::PcoScheme @@ -230,11 +230,11 @@ impl core::marker::StructuralPartialEq for vortex_btrblocks::schemes::float::Pco impl vortex_compressor::scheme::Scheme for vortex_btrblocks::schemes::float::PcoScheme -pub fn vortex_btrblocks::schemes::float::PcoScheme::compress(&self, _compressor: &vortex_compressor::compressor::CascadingCompressor, data: &mut vortex_compressor::stats::cache::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_btrblocks::schemes::float::PcoScheme::compress(&self, &vortex_compressor::compressor::CascadingCompressor, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_btrblocks::schemes::float::PcoScheme::expected_compression_ratio(&self, _data: &mut vortex_compressor::stats::cache::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_btrblocks::schemes::float::PcoScheme::expected_compression_ratio(&self, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_btrblocks::schemes::float::PcoScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_btrblocks::schemes::float::PcoScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_btrblocks::schemes::float::PcoScheme::scheme_name(&self) -> &'static str @@ -258,11 +258,11 @@ impl core::cmp::Eq for vortex_btrblocks::schemes::integer::BitPackingScheme impl core::cmp::PartialEq for vortex_btrblocks::schemes::integer::BitPackingScheme -pub fn vortex_btrblocks::schemes::integer::BitPackingScheme::eq(&self, other: &vortex_btrblocks::schemes::integer::BitPackingScheme) -> bool +pub fn vortex_btrblocks::schemes::integer::BitPackingScheme::eq(&self, &vortex_btrblocks::schemes::integer::BitPackingScheme) -> bool impl core::fmt::Debug for vortex_btrblocks::schemes::integer::BitPackingScheme -pub fn vortex_btrblocks::schemes::integer::BitPackingScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_btrblocks::schemes::integer::BitPackingScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_btrblocks::schemes::integer::BitPackingScheme @@ -270,11 +270,11 @@ impl core::marker::StructuralPartialEq for vortex_btrblocks::schemes::integer::B impl vortex_compressor::scheme::Scheme for vortex_btrblocks::schemes::integer::BitPackingScheme -pub fn vortex_btrblocks::schemes::integer::BitPackingScheme::compress(&self, _compressor: &vortex_compressor::compressor::CascadingCompressor, data: &mut vortex_compressor::stats::cache::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_btrblocks::schemes::integer::BitPackingScheme::compress(&self, &vortex_compressor::compressor::CascadingCompressor, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_btrblocks::schemes::integer::BitPackingScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::cache::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_btrblocks::schemes::integer::BitPackingScheme::expected_compression_ratio(&self, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_btrblocks::schemes::integer::BitPackingScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_btrblocks::schemes::integer::BitPackingScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_btrblocks::schemes::integer::BitPackingScheme::scheme_name(&self) -> &'static str @@ -288,11 +288,11 @@ impl core::cmp::Eq for vortex_btrblocks::schemes::integer::FoRScheme impl core::cmp::PartialEq for vortex_btrblocks::schemes::integer::FoRScheme -pub fn vortex_btrblocks::schemes::integer::FoRScheme::eq(&self, other: &vortex_btrblocks::schemes::integer::FoRScheme) -> bool +pub fn vortex_btrblocks::schemes::integer::FoRScheme::eq(&self, &vortex_btrblocks::schemes::integer::FoRScheme) -> bool impl core::fmt::Debug for vortex_btrblocks::schemes::integer::FoRScheme -pub fn vortex_btrblocks::schemes::integer::FoRScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_btrblocks::schemes::integer::FoRScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_btrblocks::schemes::integer::FoRScheme @@ -302,11 +302,11 @@ impl vortex_compressor::scheme::Scheme for vortex_btrblocks::schemes::integer::F pub fn vortex_btrblocks::schemes::integer::FoRScheme::ancestor_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_btrblocks::schemes::integer::FoRScheme::compress(&self, compressor: &vortex_compressor::compressor::CascadingCompressor, data: &mut vortex_compressor::stats::cache::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_btrblocks::schemes::integer::FoRScheme::compress(&self, &vortex_compressor::compressor::CascadingCompressor, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_btrblocks::schemes::integer::FoRScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::cache::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_btrblocks::schemes::integer::FoRScheme::expected_compression_ratio(&self, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_btrblocks::schemes::integer::FoRScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_btrblocks::schemes::integer::FoRScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_btrblocks::schemes::integer::FoRScheme::scheme_name(&self) -> &'static str @@ -320,11 +320,11 @@ impl core::cmp::Eq for vortex_btrblocks::schemes::integer::IntRLEScheme impl core::cmp::PartialEq for vortex_btrblocks::schemes::integer::IntRLEScheme -pub fn vortex_btrblocks::schemes::integer::IntRLEScheme::eq(&self, other: &vortex_btrblocks::schemes::integer::IntRLEScheme) -> bool +pub fn vortex_btrblocks::schemes::integer::IntRLEScheme::eq(&self, &vortex_btrblocks::schemes::integer::IntRLEScheme) -> bool impl core::fmt::Debug for vortex_btrblocks::schemes::integer::IntRLEScheme -pub fn vortex_btrblocks::schemes::integer::IntRLEScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_btrblocks::schemes::integer::IntRLEScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_btrblocks::schemes::integer::IntRLEScheme @@ -334,13 +334,13 @@ impl vortex_compressor::scheme::Scheme for vortex_btrblocks::schemes::integer::I pub fn vortex_btrblocks::schemes::integer::IntRLEScheme::ancestor_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_btrblocks::schemes::integer::IntRLEScheme::compress(&self, compressor: &vortex_compressor::compressor::CascadingCompressor, data: &mut vortex_compressor::stats::cache::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_btrblocks::schemes::integer::IntRLEScheme::compress(&self, &vortex_compressor::compressor::CascadingCompressor, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_btrblocks::schemes::integer::IntRLEScheme::descendant_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_btrblocks::schemes::integer::IntRLEScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::cache::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_btrblocks::schemes::integer::IntRLEScheme::expected_compression_ratio(&self, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_btrblocks::schemes::integer::IntRLEScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_btrblocks::schemes::integer::IntRLEScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_btrblocks::schemes::integer::IntRLEScheme::num_children(&self) -> usize @@ -356,11 +356,11 @@ impl core::cmp::Eq for vortex_btrblocks::schemes::integer::PcoScheme impl core::cmp::PartialEq for vortex_btrblocks::schemes::integer::PcoScheme -pub fn vortex_btrblocks::schemes::integer::PcoScheme::eq(&self, other: &vortex_btrblocks::schemes::integer::PcoScheme) -> bool +pub fn vortex_btrblocks::schemes::integer::PcoScheme::eq(&self, &vortex_btrblocks::schemes::integer::PcoScheme) -> bool impl core::fmt::Debug for vortex_btrblocks::schemes::integer::PcoScheme -pub fn vortex_btrblocks::schemes::integer::PcoScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_btrblocks::schemes::integer::PcoScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_btrblocks::schemes::integer::PcoScheme @@ -368,11 +368,11 @@ impl core::marker::StructuralPartialEq for vortex_btrblocks::schemes::integer::P impl vortex_compressor::scheme::Scheme for vortex_btrblocks::schemes::integer::PcoScheme -pub fn vortex_btrblocks::schemes::integer::PcoScheme::compress(&self, _compressor: &vortex_compressor::compressor::CascadingCompressor, data: &mut vortex_compressor::stats::cache::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_btrblocks::schemes::integer::PcoScheme::compress(&self, &vortex_compressor::compressor::CascadingCompressor, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_btrblocks::schemes::integer::PcoScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::cache::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_btrblocks::schemes::integer::PcoScheme::expected_compression_ratio(&self, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_btrblocks::schemes::integer::PcoScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_btrblocks::schemes::integer::PcoScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_btrblocks::schemes::integer::PcoScheme::scheme_name(&self) -> &'static str @@ -386,11 +386,11 @@ impl core::cmp::Eq for vortex_btrblocks::schemes::integer::RunEndScheme impl core::cmp::PartialEq for vortex_btrblocks::schemes::integer::RunEndScheme -pub fn vortex_btrblocks::schemes::integer::RunEndScheme::eq(&self, other: &vortex_btrblocks::schemes::integer::RunEndScheme) -> bool +pub fn vortex_btrblocks::schemes::integer::RunEndScheme::eq(&self, &vortex_btrblocks::schemes::integer::RunEndScheme) -> bool impl core::fmt::Debug for vortex_btrblocks::schemes::integer::RunEndScheme -pub fn vortex_btrblocks::schemes::integer::RunEndScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_btrblocks::schemes::integer::RunEndScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_btrblocks::schemes::integer::RunEndScheme @@ -400,13 +400,13 @@ impl vortex_compressor::scheme::Scheme for vortex_btrblocks::schemes::integer::R pub fn vortex_btrblocks::schemes::integer::RunEndScheme::ancestor_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_btrblocks::schemes::integer::RunEndScheme::compress(&self, compressor: &vortex_compressor::compressor::CascadingCompressor, data: &mut vortex_compressor::stats::cache::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_btrblocks::schemes::integer::RunEndScheme::compress(&self, &vortex_compressor::compressor::CascadingCompressor, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_btrblocks::schemes::integer::RunEndScheme::descendant_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_btrblocks::schemes::integer::RunEndScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::cache::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_btrblocks::schemes::integer::RunEndScheme::expected_compression_ratio(&self, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_btrblocks::schemes::integer::RunEndScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_btrblocks::schemes::integer::RunEndScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_btrblocks::schemes::integer::RunEndScheme::num_children(&self) -> usize @@ -422,11 +422,11 @@ impl core::cmp::Eq for vortex_btrblocks::schemes::integer::SequenceScheme impl core::cmp::PartialEq for vortex_btrblocks::schemes::integer::SequenceScheme -pub fn vortex_btrblocks::schemes::integer::SequenceScheme::eq(&self, other: &vortex_btrblocks::schemes::integer::SequenceScheme) -> bool +pub fn vortex_btrblocks::schemes::integer::SequenceScheme::eq(&self, &vortex_btrblocks::schemes::integer::SequenceScheme) -> bool impl core::fmt::Debug for vortex_btrblocks::schemes::integer::SequenceScheme -pub fn vortex_btrblocks::schemes::integer::SequenceScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_btrblocks::schemes::integer::SequenceScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_btrblocks::schemes::integer::SequenceScheme @@ -436,11 +436,11 @@ impl vortex_compressor::scheme::Scheme for vortex_btrblocks::schemes::integer::S pub fn vortex_btrblocks::schemes::integer::SequenceScheme::ancestor_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_btrblocks::schemes::integer::SequenceScheme::compress(&self, _compressor: &vortex_compressor::compressor::CascadingCompressor, data: &mut vortex_compressor::stats::cache::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_btrblocks::schemes::integer::SequenceScheme::compress(&self, &vortex_compressor::compressor::CascadingCompressor, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_btrblocks::schemes::integer::SequenceScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::cache::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_btrblocks::schemes::integer::SequenceScheme::expected_compression_ratio(&self, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_btrblocks::schemes::integer::SequenceScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_btrblocks::schemes::integer::SequenceScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_btrblocks::schemes::integer::SequenceScheme::scheme_name(&self) -> &'static str @@ -454,11 +454,11 @@ impl core::cmp::Eq for vortex_btrblocks::schemes::integer::SparseScheme impl core::cmp::PartialEq for vortex_btrblocks::schemes::integer::SparseScheme -pub fn vortex_btrblocks::schemes::integer::SparseScheme::eq(&self, other: &vortex_btrblocks::schemes::integer::SparseScheme) -> bool +pub fn vortex_btrblocks::schemes::integer::SparseScheme::eq(&self, &vortex_btrblocks::schemes::integer::SparseScheme) -> bool impl core::fmt::Debug for vortex_btrblocks::schemes::integer::SparseScheme -pub fn vortex_btrblocks::schemes::integer::SparseScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_btrblocks::schemes::integer::SparseScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_btrblocks::schemes::integer::SparseScheme @@ -466,13 +466,13 @@ impl core::marker::StructuralPartialEq for vortex_btrblocks::schemes::integer::S impl vortex_compressor::scheme::Scheme for vortex_btrblocks::schemes::integer::SparseScheme -pub fn vortex_btrblocks::schemes::integer::SparseScheme::compress(&self, compressor: &vortex_compressor::compressor::CascadingCompressor, data: &mut vortex_compressor::stats::cache::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_btrblocks::schemes::integer::SparseScheme::compress(&self, &vortex_compressor::compressor::CascadingCompressor, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_btrblocks::schemes::integer::SparseScheme::descendant_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_btrblocks::schemes::integer::SparseScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::cache::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_btrblocks::schemes::integer::SparseScheme::expected_compression_ratio(&self, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_btrblocks::schemes::integer::SparseScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_btrblocks::schemes::integer::SparseScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_btrblocks::schemes::integer::SparseScheme::num_children(&self) -> usize @@ -490,11 +490,11 @@ impl core::cmp::Eq for vortex_btrblocks::schemes::integer::ZigZagScheme impl core::cmp::PartialEq for vortex_btrblocks::schemes::integer::ZigZagScheme -pub fn vortex_btrblocks::schemes::integer::ZigZagScheme::eq(&self, other: &vortex_btrblocks::schemes::integer::ZigZagScheme) -> bool +pub fn vortex_btrblocks::schemes::integer::ZigZagScheme::eq(&self, &vortex_btrblocks::schemes::integer::ZigZagScheme) -> bool impl core::fmt::Debug for vortex_btrblocks::schemes::integer::ZigZagScheme -pub fn vortex_btrblocks::schemes::integer::ZigZagScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_btrblocks::schemes::integer::ZigZagScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_btrblocks::schemes::integer::ZigZagScheme @@ -504,13 +504,13 @@ impl vortex_compressor::scheme::Scheme for vortex_btrblocks::schemes::integer::Z pub fn vortex_btrblocks::schemes::integer::ZigZagScheme::ancestor_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_btrblocks::schemes::integer::ZigZagScheme::compress(&self, compressor: &vortex_compressor::compressor::CascadingCompressor, data: &mut vortex_compressor::stats::cache::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_btrblocks::schemes::integer::ZigZagScheme::compress(&self, &vortex_compressor::compressor::CascadingCompressor, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_btrblocks::schemes::integer::ZigZagScheme::descendant_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_btrblocks::schemes::integer::ZigZagScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::cache::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_btrblocks::schemes::integer::ZigZagScheme::expected_compression_ratio(&self, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_btrblocks::schemes::integer::ZigZagScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_btrblocks::schemes::integer::ZigZagScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_btrblocks::schemes::integer::ZigZagScheme::num_children(&self) -> usize @@ -536,11 +536,11 @@ impl core::cmp::Eq for vortex_btrblocks::schemes::string::FSSTScheme impl core::cmp::PartialEq for vortex_btrblocks::schemes::string::FSSTScheme -pub fn vortex_btrblocks::schemes::string::FSSTScheme::eq(&self, other: &vortex_btrblocks::schemes::string::FSSTScheme) -> bool +pub fn vortex_btrblocks::schemes::string::FSSTScheme::eq(&self, &vortex_btrblocks::schemes::string::FSSTScheme) -> bool impl core::fmt::Debug for vortex_btrblocks::schemes::string::FSSTScheme -pub fn vortex_btrblocks::schemes::string::FSSTScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_btrblocks::schemes::string::FSSTScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_btrblocks::schemes::string::FSSTScheme @@ -548,11 +548,11 @@ impl core::marker::StructuralPartialEq for vortex_btrblocks::schemes::string::FS impl vortex_compressor::scheme::Scheme for vortex_btrblocks::schemes::string::FSSTScheme -pub fn vortex_btrblocks::schemes::string::FSSTScheme::compress(&self, compressor: &vortex_compressor::compressor::CascadingCompressor, data: &mut vortex_compressor::stats::cache::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_btrblocks::schemes::string::FSSTScheme::compress(&self, &vortex_compressor::compressor::CascadingCompressor, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_btrblocks::schemes::string::FSSTScheme::expected_compression_ratio(&self, _data: &mut vortex_compressor::stats::cache::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_btrblocks::schemes::string::FSSTScheme::expected_compression_ratio(&self, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_btrblocks::schemes::string::FSSTScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_btrblocks::schemes::string::FSSTScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_btrblocks::schemes::string::FSSTScheme::num_children(&self) -> usize @@ -568,11 +568,11 @@ impl core::cmp::Eq for vortex_btrblocks::schemes::string::NullDominatedSparseSch impl core::cmp::PartialEq for vortex_btrblocks::schemes::string::NullDominatedSparseScheme -pub fn vortex_btrblocks::schemes::string::NullDominatedSparseScheme::eq(&self, other: &vortex_btrblocks::schemes::string::NullDominatedSparseScheme) -> bool +pub fn vortex_btrblocks::schemes::string::NullDominatedSparseScheme::eq(&self, &vortex_btrblocks::schemes::string::NullDominatedSparseScheme) -> bool impl core::fmt::Debug for vortex_btrblocks::schemes::string::NullDominatedSparseScheme -pub fn vortex_btrblocks::schemes::string::NullDominatedSparseScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_btrblocks::schemes::string::NullDominatedSparseScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_btrblocks::schemes::string::NullDominatedSparseScheme @@ -580,13 +580,13 @@ impl core::marker::StructuralPartialEq for vortex_btrblocks::schemes::string::Nu impl vortex_compressor::scheme::Scheme for vortex_btrblocks::schemes::string::NullDominatedSparseScheme -pub fn vortex_btrblocks::schemes::string::NullDominatedSparseScheme::compress(&self, compressor: &vortex_compressor::compressor::CascadingCompressor, data: &mut vortex_compressor::stats::cache::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_btrblocks::schemes::string::NullDominatedSparseScheme::compress(&self, &vortex_compressor::compressor::CascadingCompressor, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_btrblocks::schemes::string::NullDominatedSparseScheme::descendant_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_btrblocks::schemes::string::NullDominatedSparseScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::cache::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_btrblocks::schemes::string::NullDominatedSparseScheme::expected_compression_ratio(&self, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_btrblocks::schemes::string::NullDominatedSparseScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_btrblocks::schemes::string::NullDominatedSparseScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_btrblocks::schemes::string::NullDominatedSparseScheme::num_children(&self) -> usize @@ -602,11 +602,11 @@ impl core::cmp::Eq for vortex_btrblocks::schemes::string::ZstdScheme impl core::cmp::PartialEq for vortex_btrblocks::schemes::string::ZstdScheme -pub fn vortex_btrblocks::schemes::string::ZstdScheme::eq(&self, other: &vortex_btrblocks::schemes::string::ZstdScheme) -> bool +pub fn vortex_btrblocks::schemes::string::ZstdScheme::eq(&self, &vortex_btrblocks::schemes::string::ZstdScheme) -> bool impl core::fmt::Debug for vortex_btrblocks::schemes::string::ZstdScheme -pub fn vortex_btrblocks::schemes::string::ZstdScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_btrblocks::schemes::string::ZstdScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_btrblocks::schemes::string::ZstdScheme @@ -614,11 +614,11 @@ impl core::marker::StructuralPartialEq for vortex_btrblocks::schemes::string::Zs impl vortex_compressor::scheme::Scheme for vortex_btrblocks::schemes::string::ZstdScheme -pub fn vortex_btrblocks::schemes::string::ZstdScheme::compress(&self, _compressor: &vortex_compressor::compressor::CascadingCompressor, data: &mut vortex_compressor::stats::cache::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_btrblocks::schemes::string::ZstdScheme::compress(&self, &vortex_compressor::compressor::CascadingCompressor, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_btrblocks::schemes::string::ZstdScheme::expected_compression_ratio(&self, _data: &mut vortex_compressor::stats::cache::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_btrblocks::schemes::string::ZstdScheme::expected_compression_ratio(&self, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_btrblocks::schemes::string::ZstdScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_btrblocks::schemes::string::ZstdScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_btrblocks::schemes::string::ZstdScheme::scheme_name(&self) -> &'static str @@ -634,11 +634,11 @@ impl core::cmp::Eq for vortex_btrblocks::schemes::temporal::TemporalScheme impl core::cmp::PartialEq for vortex_btrblocks::schemes::temporal::TemporalScheme -pub fn vortex_btrblocks::schemes::temporal::TemporalScheme::eq(&self, other: &vortex_btrblocks::schemes::temporal::TemporalScheme) -> bool +pub fn vortex_btrblocks::schemes::temporal::TemporalScheme::eq(&self, &vortex_btrblocks::schemes::temporal::TemporalScheme) -> bool impl core::fmt::Debug for vortex_btrblocks::schemes::temporal::TemporalScheme -pub fn vortex_btrblocks::schemes::temporal::TemporalScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_btrblocks::schemes::temporal::TemporalScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_btrblocks::schemes::temporal::TemporalScheme @@ -646,11 +646,11 @@ impl core::marker::StructuralPartialEq for vortex_btrblocks::schemes::temporal:: impl vortex_compressor::scheme::Scheme for vortex_btrblocks::schemes::temporal::TemporalScheme -pub fn vortex_btrblocks::schemes::temporal::TemporalScheme::compress(&self, compressor: &vortex_compressor::compressor::CascadingCompressor, data: &mut vortex_compressor::stats::cache::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_btrblocks::schemes::temporal::TemporalScheme::compress(&self, &vortex_compressor::compressor::CascadingCompressor, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_btrblocks::schemes::temporal::TemporalScheme::expected_compression_ratio(&self, _data: &mut vortex_compressor::stats::cache::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_btrblocks::schemes::temporal::TemporalScheme::expected_compression_ratio(&self, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_btrblocks::schemes::temporal::TemporalScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_btrblocks::schemes::temporal::TemporalScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_btrblocks::schemes::temporal::TemporalScheme::num_children(&self) -> usize @@ -660,7 +660,7 @@ pub struct vortex_btrblocks::BtrBlocksCompressor(pub vortex_compressor::compress impl vortex_btrblocks::BtrBlocksCompressor -pub fn vortex_btrblocks::BtrBlocksCompressor::compress(&self, array: &vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_btrblocks::BtrBlocksCompressor::compress(&self, &vortex_array::array::erased::ArrayRef, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult impl core::clone::Clone for vortex_btrblocks::BtrBlocksCompressor @@ -672,7 +672,7 @@ pub fn vortex_btrblocks::BtrBlocksCompressor::default() -> Self impl core::fmt::Debug for vortex_btrblocks::BtrBlocksCompressor -pub fn vortex_btrblocks::BtrBlocksCompressor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_btrblocks::BtrBlocksCompressor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::ops::deref::Deref for vortex_btrblocks::BtrBlocksCompressor @@ -686,13 +686,15 @@ impl vortex_btrblocks::BtrBlocksCompressorBuilder pub fn vortex_btrblocks::BtrBlocksCompressorBuilder::build(self) -> vortex_btrblocks::BtrBlocksCompressor -pub fn vortex_btrblocks::BtrBlocksCompressorBuilder::exclude_schemes(self, ids: impl core::iter::traits::collect::IntoIterator) -> Self +pub fn vortex_btrblocks::BtrBlocksCompressorBuilder::empty() -> Self + +pub fn vortex_btrblocks::BtrBlocksCompressorBuilder::exclude_schemes(self, impl core::iter::traits::collect::IntoIterator) -> Self pub fn vortex_btrblocks::BtrBlocksCompressorBuilder::only_cuda_compatible(self) -> Self pub fn vortex_btrblocks::BtrBlocksCompressorBuilder::with_compact(self) -> Self -pub fn vortex_btrblocks::BtrBlocksCompressorBuilder::with_new_scheme(self, scheme: &'static dyn vortex_compressor::scheme::Scheme) -> Self +pub fn vortex_btrblocks::BtrBlocksCompressorBuilder::with_new_scheme(self, &'static dyn vortex_compressor::scheme::Scheme) -> Self impl core::clone::Clone for vortex_btrblocks::BtrBlocksCompressorBuilder @@ -704,8 +706,8 @@ pub fn vortex_btrblocks::BtrBlocksCompressorBuilder::default() -> Self impl core::fmt::Debug for vortex_btrblocks::BtrBlocksCompressorBuilder -pub fn vortex_btrblocks::BtrBlocksCompressorBuilder::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_btrblocks::BtrBlocksCompressorBuilder::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub const vortex_btrblocks::ALL_SCHEMES: &[&dyn vortex_compressor::scheme::Scheme] -pub fn vortex_btrblocks::compress_patches(patches: vortex_array::patches::Patches) -> vortex_error::VortexResult +pub fn vortex_btrblocks::compress_patches(vortex_array::patches::Patches, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult diff --git a/vortex-btrblocks/src/builder.rs b/vortex-btrblocks/src/builder.rs index da405b153e8..ab77f625764 100644 --- a/vortex-btrblocks/src/builder.rs +++ b/vortex-btrblocks/src/builder.rs @@ -97,6 +97,15 @@ impl Default for BtrBlocksCompressorBuilder { } impl BtrBlocksCompressorBuilder { + /// Creates a builder with no schemes registered. + /// + /// Useful when the caller wants explicit, scheme-by-scheme control over the compressor. + pub fn empty() -> Self { + Self { + schemes: Vec::new(), + } + } + /// Adds an external compression scheme not in [`ALL_SCHEMES`]. /// /// This allows encoding crates outside of `vortex-btrblocks` to register their own schemes @@ -188,3 +197,20 @@ impl BtrBlocksCompressorBuilder { BtrBlocksCompressor(CascadingCompressor::new(self.schemes)) } } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn empty_starts_with_no_schemes() { + let builder = BtrBlocksCompressorBuilder::empty(); + assert!(builder.schemes.is_empty()); + } + + #[test] + fn default_includes_all_schemes() { + let builder = BtrBlocksCompressorBuilder::default(); + assert_eq!(builder.schemes.len(), ALL_SCHEMES.len()); + } +} diff --git a/vortex-btrblocks/src/canonical_compressor.rs b/vortex-btrblocks/src/canonical_compressor.rs index 1739a41214f..b0a8ea67423 100644 --- a/vortex-btrblocks/src/canonical_compressor.rs +++ b/vortex-btrblocks/src/canonical_compressor.rs @@ -6,6 +6,7 @@ use std::ops::Deref; use vortex_array::ArrayRef; +use vortex_array::ExecutionCtx; use vortex_error::VortexResult; use crate::BtrBlocksCompressorBuilder; @@ -38,8 +39,8 @@ pub struct BtrBlocksCompressor( impl BtrBlocksCompressor { /// Compresses an array using BtrBlocks-inspired compression. - pub fn compress(&self, array: &ArrayRef) -> VortexResult { - self.0.compress(array) + pub fn compress(&self, array: &ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + self.0.compress(array, ctx) } } @@ -67,21 +68,29 @@ impl Default for BtrBlocksCompressor { #[cfg(test)] mod tests { + use std::sync::LazyLock; + use rstest::rstest; use vortex_array::IntoArray; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; use vortex_array::arrays::Constant; use vortex_array::arrays::List; use vortex_array::arrays::ListView; use vortex_array::arrays::ListViewArray; use vortex_array::assert_arrays_eq; + use vortex_array::session::ArraySession; use vortex_array::validity::Validity; use vortex_buffer::BitBuffer; use vortex_buffer::buffer; use vortex_error::VortexResult; + use vortex_session::VortexSession; use crate::BtrBlocksCompressor; + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); + #[rstest] #[case::zctl( unsafe { @@ -108,7 +117,8 @@ mod tests { #[case] expect_list: bool, ) -> VortexResult<()> { let array_ref = input.clone().into_array(); - let result = BtrBlocksCompressor::default().compress(&array_ref)?; + let result = BtrBlocksCompressor::default() + .compress(&array_ref, &mut SESSION.create_execution_ctx())?; if expect_list { assert!(result.as_opt::().is_some()); } else { @@ -122,7 +132,10 @@ mod tests { fn test_constant_all_true() -> VortexResult<()> { let array = BoolArray::new(BitBuffer::from(vec![true; 100]), Validity::NonNullable); let btr = BtrBlocksCompressor::default(); - let compressed = btr.compress(&array.clone().into_array())?; + let compressed = btr.compress( + &array.clone().into_array(), + &mut SESSION.create_execution_ctx(), + )?; assert!(compressed.is::()); assert_arrays_eq!(compressed, array); Ok(()) @@ -132,7 +145,10 @@ mod tests { fn test_constant_all_false() -> VortexResult<()> { let array = BoolArray::new(BitBuffer::from(vec![false; 100]), Validity::NonNullable); let btr = BtrBlocksCompressor::default(); - let compressed = btr.compress(&array.clone().into_array())?; + let compressed = btr.compress( + &array.clone().into_array(), + &mut SESSION.create_execution_ctx(), + )?; assert!(compressed.is::()); assert_arrays_eq!(compressed, array); Ok(()) @@ -145,7 +161,10 @@ mod tests { Validity::from(BitBuffer::from(vec![true; 100])), ); let btr = BtrBlocksCompressor::default(); - let compressed = btr.compress(&array.clone().into_array())?; + let compressed = btr.compress( + &array.clone().into_array(), + &mut SESSION.create_execution_ctx(), + )?; assert!(compressed.is::()); assert_arrays_eq!(compressed, array); Ok(()) @@ -156,7 +175,10 @@ mod tests { let validity = Validity::from(BitBuffer::from_iter((0..100).map(|i| i % 3 != 0))); let array = BoolArray::new(BitBuffer::from(vec![true; 100]), validity); let btr = BtrBlocksCompressor::default(); - let compressed = btr.compress(&array.clone().into_array())?; + let compressed = btr.compress( + &array.clone().into_array(), + &mut SESSION.create_execution_ctx(), + )?; assert!(!compressed.is::()); assert_arrays_eq!(compressed, array); Ok(()) @@ -169,7 +191,10 @@ mod tests { Validity::NonNullable, ); let btr = BtrBlocksCompressor::default(); - let compressed = btr.compress(&array.clone().into_array())?; + let compressed = btr.compress( + &array.clone().into_array(), + &mut SESSION.create_execution_ctx(), + )?; assert!(!compressed.is::()); assert_arrays_eq!(compressed, array); Ok(()) diff --git a/vortex-btrblocks/src/lib.rs b/vortex-btrblocks/src/lib.rs index 1f6d5a2d816..39db05246a6 100644 --- a/vortex-btrblocks/src/lib.rs +++ b/vortex-btrblocks/src/lib.rs @@ -22,8 +22,8 @@ //! //! # How It Works //! -//! [`BtrBlocksCompressor::compress()`] takes an `&ArrayRef` and returns an `ArrayRef` that may -//! use a different encoding. It first canonicalizes the input, then dispatches by type. +//! [`BtrBlocksCompressor::compress()`] takes an `&ArrayRef` plus a mutable execution context and +//! returns an `ArrayRef` that may use a different encoding. It first canonicalizes the input, then dispatches by type. //! Primitives and strings go through `choose_and_compress`, which evaluates every enabled //! [`Scheme`] and picks the one with the best compression ratio. Compound types like structs //! and lists recurse into their fields and elements. diff --git a/vortex-btrblocks/src/schemes/decimal.rs b/vortex-btrblocks/src/schemes/decimal.rs index ce5262ba5ce..d60e5c6b175 100644 --- a/vortex-btrblocks/src/schemes/decimal.rs +++ b/vortex-btrblocks/src/schemes/decimal.rs @@ -5,12 +5,14 @@ use vortex_array::ArrayRef; use vortex_array::Canonical; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; -use vortex_array::ToCanonical; +use vortex_array::arrays::DecimalArray; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::decimal::narrowed_decimal; use vortex_array::dtype::DecimalType; use vortex_compressor::estimate::CompressionEstimate; +use vortex_compressor::estimate::EstimateVerdict; use vortex_decimal_byte_parts::DecimalByteParts; use vortex_error::VortexResult; @@ -43,22 +45,24 @@ impl Scheme for DecimalScheme { fn expected_compression_ratio( &self, - _data: &mut ArrayAndStats, - _ctx: CompressorContext, + _data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { // Decimal compression is almost always beneficial (narrowing + primitive compression). - CompressionEstimate::AlwaysUse + CompressionEstimate::Verdict(EstimateVerdict::AlwaysUse) } fn compress( &self, compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { // TODO(joe): add support splitting i128/256 buffers into chunks of primitive values // for compression. 2 for i128 and 4 for i256. - let decimal = data.array().clone().to_decimal(); + let decimal = data.array().clone().execute::(exec_ctx)?; let decimal = narrowed_decimal(decimal); let validity = decimal.validity()?; let prim = match decimal.values_type() { @@ -69,7 +73,8 @@ impl Scheme for DecimalScheme { _ => return Ok(decimal.into_array()), }; - let compressed = compressor.compress_child(&prim.into_array(), &ctx, self.id(), 0)?; + let compressed = + compressor.compress_child(&prim.into_array(), &compress_ctx, self.id(), 0, exec_ctx)?; DecimalByteParts::try_new(compressed, decimal.decimal_dtype()).map(|d| d.into_array()) } diff --git a/vortex-btrblocks/src/schemes/float.rs b/vortex-btrblocks/src/schemes/float.rs index e74b130bbd0..8c55b38d345 100644 --- a/vortex-btrblocks/src/schemes/float.rs +++ b/vortex-btrblocks/src/schemes/float.rs @@ -12,11 +12,16 @@ use vortex_alp::RDEncoder; use vortex_alp::alp_encode; use vortex_array::ArrayRef; use vortex_array::Canonical; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; -use vortex_array::ToCanonical; +use vortex_array::arrays::Patched; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::patched::use_experimental_patches; use vortex_array::arrays::primitive::PrimitiveArrayExt; use vortex_array::dtype::PType; use vortex_compressor::estimate::CompressionEstimate; +use vortex_compressor::estimate::DeferredEstimate; +use vortex_compressor::estimate::EstimateVerdict; use vortex_compressor::scheme::ChildSelection; use vortex_compressor::scheme::DescendantExclusion; use vortex_error::VortexResult; @@ -78,40 +83,65 @@ impl Scheme for ALPScheme { fn expected_compression_ratio( &self, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { // ALP encodes floats as integers. Without integer compression afterward, the encoded ints // are the same size. - if ctx.finished_cascading() { - return CompressionEstimate::Skip; + if compress_ctx.finished_cascading() { + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } // We don't support ALP for f16. if data.array_as_primitive().ptype() == PType::F16 { - return CompressionEstimate::Skip; + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } - CompressionEstimate::Sample + CompressionEstimate::Deferred(DeferredEstimate::Sample) } fn compress( &self, compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { - let alp_encoded = alp_encode(&data.array_as_primitive(), None)?; + let alp_encoded = alp_encode(data.array_as_primitive(), None, exec_ctx)?; // Compress the ALP ints. - let compressed_alp_ints = - compressor.compress_child(alp_encoded.encoded(), &ctx, self.id(), 0)?; - - // Patches are not compressed. They should be infrequent, and if they are not then we want - // to keep them linear for easy indexing. - let patches = alp_encoded.patches().map(compress_patches).transpose()?; + let compressed_alp_ints = compressor.compress_child( + alp_encoded.encoded(), + &compress_ctx, + self.id(), + 0, + exec_ctx, + )?; + + let alp_stats = alp_encoded.as_array().statistics().to_owned(); + let exponents = alp_encoded.exponents(); + + if use_experimental_patches() { + let patches = alp_encoded.patches(); + + // Create ALP array without interior patches. + let alp_array = ALP::new(compressed_alp_ints, exponents, None).into_array(); + + match patches { + None => Ok(alp_array), + Some(p) => Ok(Patched::from_array_and_patches(alp_array, &p, exec_ctx)? + .with_stats_set(alp_stats) + .into_array()), + } + } else { + let patches = alp_encoded + .patches() + .map(|p| compress_patches(p, exec_ctx)) + .transpose()?; - Ok(ALP::new(compressed_alp_ints, alp_encoded.exponents(), patches).into_array()) + Ok(ALP::new(compressed_alp_ints, exponents, patches).into_array()) + } } } @@ -126,22 +156,24 @@ impl Scheme for ALPRDScheme { fn expected_compression_ratio( &self, - data: &mut ArrayAndStats, - _ctx: CompressorContext, + data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { // We don't support ALPRD for f16. if data.array_as_primitive().ptype() == PType::F16 { - return CompressionEstimate::Skip; + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } - CompressionEstimate::Sample + CompressionEstimate::Deferred(DeferredEstimate::Sample) } fn compress( &self, _compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - _ctx: CompressorContext, + data: &ArrayAndStats, + _compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { let primitive_array = data.array_as_primitive(); @@ -151,11 +183,14 @@ impl Scheme for ALPRDScheme { ptype => vortex_panic!("cannot ALPRD compress ptype {ptype}"), }; - let alp_rd = encoder.encode(&primitive_array); + let alp_rd = encoder.encode(primitive_array, exec_ctx); let dtype = alp_rd.dtype().clone(); let right_bit_width = alp_rd.right_bit_width(); let mut parts = ALPRDArrayOwnedExt::into_data_parts(alp_rd); - parts.left_parts_patches = parts.left_parts_patches.map(compress_patches).transpose()?; + parts.left_parts_patches = parts + .left_parts_patches + .map(|p| compress_patches(p, exec_ctx)) + .transpose()?; Ok(vortex_alp::ALPRD::try_new( dtype, @@ -164,6 +199,7 @@ impl Scheme for ALPRDScheme { parts.right_parts, right_bit_width, parts.left_parts_patches, + exec_ctx, )? .into_array()) } @@ -193,40 +229,52 @@ impl Scheme for NullDominatedSparseScheme { fn expected_compression_ratio( &self, - data: &mut ArrayAndStats, - _ctx: CompressorContext, + data: &ArrayAndStats, + _compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { let len = data.array_len() as f64; - let stats = data.float_stats(); + let stats = data.float_stats(exec_ctx); let value_count = stats.value_count(); // All-null arrays should be compressed as constant instead anyways. if value_count == 0 { - return CompressionEstimate::Skip; + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } // If the majority (90%) of values is null, this will compress well. if stats.null_count() as f64 / len > 0.9 { - return CompressionEstimate::Ratio(len / value_count as f64); + return CompressionEstimate::Verdict(EstimateVerdict::Ratio(len / value_count as f64)); } // Otherwise we don't go this route. - CompressionEstimate::Skip + CompressionEstimate::Verdict(EstimateVerdict::Skip) } fn compress( &self, compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { // We pass None as we only run this pathway for NULL-dominated float arrays. - let sparse_encoded = Sparse::encode(data.array(), None)?; + let sparse_encoded = Sparse::encode(data.array(), None, exec_ctx)?; if let Some(sparse) = sparse_encoded.as_opt::() { - let indices = sparse.patches().indices().to_primitive().narrow()?; - let compressed_indices = - compressor.compress_child(&indices.into_array(), &ctx, self.id(), 0)?; + let indices = sparse + .patches() + .indices() + .clone() + .execute::(exec_ctx)? + .narrow()?; + let compressed_indices = compressor.compress_child( + &indices.into_array(), + &compress_ctx, + self.id(), + 0, + exec_ctx, + )?; Sparse::try_new( compressed_indices, @@ -253,22 +301,25 @@ impl Scheme for PcoScheme { fn expected_compression_ratio( &self, - _data: &mut ArrayAndStats, - _ctx: CompressorContext, + _data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { - CompressionEstimate::Sample + CompressionEstimate::Deferred(DeferredEstimate::Sample) } fn compress( &self, _compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - _ctx: CompressorContext, + data: &ArrayAndStats, + _compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { Ok(vortex_pco::Pco::from_primitive( - &data.array_as_primitive(), + data.array_as_primitive(), pco::DEFAULT_COMPRESSION_LEVEL, 8192, + exec_ctx, )? .into_array()) } @@ -298,57 +349,66 @@ impl Scheme for FloatRLEScheme { fn expected_compression_ratio( &self, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { // RLE is only useful when we cascade it with another encoding. - if ctx.finished_cascading() { - return CompressionEstimate::Skip; + if compress_ctx.finished_cascading() { + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } - if data.float_stats().average_run_length() < super::integer::RUN_LENGTH_THRESHOLD { - return CompressionEstimate::Skip; + if data.float_stats(exec_ctx).average_run_length() < super::integer::RUN_LENGTH_THRESHOLD { + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } - CompressionEstimate::Sample + CompressionEstimate::Deferred(DeferredEstimate::Sample) } fn compress( &self, compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { - super::integer::rle_compress(self, compressor, data, ctx) + super::integer::rle_compress(self, compressor, data, compress_ctx, exec_ctx) } } #[cfg(test)] mod tests { use std::iter; + use std::sync::LazyLock; use vortex_array::IntoArray; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; use vortex_array::builders::ArrayBuilder; use vortex_array::builders::PrimitiveBuilder; use vortex_array::display::DisplayOptions; use vortex_array::dtype::Nullability; + use vortex_array::session::ArraySession; use vortex_array::validity::Validity; use vortex_buffer::Buffer; use vortex_buffer::buffer_mut; use vortex_compressor::CascadingCompressor; use vortex_error::VortexResult; use vortex_fastlanes::RLE; + use vortex_session::VortexSession; use crate::BtrBlocksCompressor; use crate::schemes::float::FloatRLEScheme; + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); + #[test] fn test_empty() -> VortexResult<()> { let btr = BtrBlocksCompressor::default(); let array = PrimitiveArray::new(Buffer::::empty(), Validity::NonNullable).into_array(); - let result = btr.compress(&array)?; + let result = btr.compress(&array, &mut SESSION.create_execution_ctx())?; assert!(result.is_empty()); Ok(()) @@ -363,7 +423,7 @@ mod tests { let array = values.into_array(); let btr = BtrBlocksCompressor::default(); - let compressed = btr.compress(&array)?; + let compressed = btr.compress(&array, &mut SESSION.create_execution_ctx())?; assert_eq!(compressed.len(), 1024); let display = compressed @@ -385,7 +445,8 @@ mod tests { let array = PrimitiveArray::new(Buffer::copy_from(&values), Validity::NonNullable); let compressor = CascadingCompressor::new(vec![&FloatRLEScheme]); - let compressed = compressor.compress(&array.into_array())?; + let compressed = + compressor.compress(&array.into_array(), &mut SESSION.create_execution_ctx())?; assert!(compressed.is::()); let expected = Buffer::copy_from(&values).into_array(); @@ -406,7 +467,7 @@ mod tests { let array = array.finish_into_primitive().into_array(); let btr = BtrBlocksCompressor::default(); - let compressed = btr.compress(&array)?; + let compressed = btr.compress(&array, &mut SESSION.create_execution_ctx())?; assert_eq!(compressed.len(), 96); let display = compressed @@ -422,26 +483,34 @@ mod tests { /// Tests to verify that each float compression scheme produces the expected encoding. #[cfg(test)] mod scheme_selection_tests { + use std::sync::LazyLock; + use vortex_alp::ALP; use vortex_array::IntoArray; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::Constant; use vortex_array::arrays::Dict; use vortex_array::arrays::PrimitiveArray; use vortex_array::builders::ArrayBuilder; use vortex_array::builders::PrimitiveBuilder; use vortex_array::dtype::Nullability; + use vortex_array::session::ArraySession; use vortex_array::validity::Validity; use vortex_buffer::Buffer; use vortex_error::VortexResult; + use vortex_session::VortexSession; use crate::BtrBlocksCompressor; + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); + #[test] fn test_constant_compressed() -> VortexResult<()> { let values: Vec = vec![42.5; 100]; let array = PrimitiveArray::new(Buffer::copy_from(&values), Validity::NonNullable); let btr = BtrBlocksCompressor::default(); - let compressed = btr.compress(&array.into_array())?; + let compressed = btr.compress(&array.into_array(), &mut SESSION.create_execution_ctx())?; assert!(compressed.is::()); Ok(()) } @@ -451,7 +520,7 @@ mod scheme_selection_tests { let values: Vec = (0..1000).map(|i| (i as f64) * 0.01).collect(); let array = PrimitiveArray::new(Buffer::copy_from(&values), Validity::NonNullable); let btr = BtrBlocksCompressor::default(); - let compressed = btr.compress(&array.into_array())?; + let compressed = btr.compress(&array.into_array(), &mut SESSION.create_execution_ctx())?; assert!(compressed.is::()); Ok(()) } @@ -464,7 +533,7 @@ mod scheme_selection_tests { .collect(); let array = PrimitiveArray::new(Buffer::copy_from(&values), Validity::NonNullable); let btr = BtrBlocksCompressor::default(); - let compressed = btr.compress(&array.into_array())?; + let compressed = btr.compress(&array.into_array(), &mut SESSION.create_execution_ctx())?; assert!(compressed.is::()); assert!(compressed.children()[0].is::()); Ok(()) @@ -479,7 +548,7 @@ mod scheme_selection_tests { builder.append_nulls(95); let array = builder.finish_into_primitive(); let btr = BtrBlocksCompressor::default(); - let compressed = btr.compress(&array.into_array())?; + let compressed = btr.compress(&array.into_array(), &mut SESSION.create_execution_ctx())?; // Verify the compressed array preserves values. assert_eq!(compressed.len(), 100); Ok(()) diff --git a/vortex-btrblocks/src/schemes/integer.rs b/vortex-btrblocks/src/schemes/integer.rs index f56b8d478f0..dfd61deb80b 100644 --- a/vortex-btrblocks/src/schemes/integer.rs +++ b/vortex-btrblocks/src/schemes/integer.rs @@ -5,17 +5,20 @@ use vortex_array::ArrayRef; use vortex_array::Canonical; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; -use vortex_array::LEGACY_SESSION; -use vortex_array::ToCanonical; -use vortex_array::VortexSessionExecute; use vortex_array::arrays::ConstantArray; use vortex_array::arrays::Patched; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::patched::use_experimental_patches; use vortex_array::arrays::primitive::PrimitiveArrayExt; use vortex_array::scalar::Scalar; use vortex_compressor::builtins::FloatDictScheme; use vortex_compressor::builtins::StringDictScheme; use vortex_compressor::estimate::CompressionEstimate; +use vortex_compressor::estimate::DeferredEstimate; +use vortex_compressor::estimate::EstimateScore; +use vortex_compressor::estimate::EstimateVerdict; use vortex_compressor::scheme::AncestorExclusion; use vortex_compressor::scheme::ChildSelection; use vortex_compressor::scheme::DescendantExclusion; @@ -32,7 +35,6 @@ use vortex_fastlanes::FoR; use vortex_fastlanes::FoRArrayExt; use vortex_fastlanes::RLE; use vortex_fastlanes::RLEArrayExt; -use vortex_fastlanes::USE_EXPERIMENTAL_PATCHES; use vortex_fastlanes::bitpack_compress::bit_width_histogram; use vortex_fastlanes::bitpack_compress::bitpack_encode; use vortex_fastlanes::bitpack_compress::find_best_bit_width; @@ -128,27 +130,27 @@ impl Scheme for FoRScheme { fn expected_compression_ratio( &self, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { // FoR only subtracts the min. Without further compression (e.g. BitPacking), the output is // the same size. - if ctx.finished_cascading() { - return CompressionEstimate::Skip; + if compress_ctx.finished_cascading() { + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } - - let stats = data.integer_stats(); + let stats = data.integer_stats(exec_ctx); // Only apply when the min is not already zero. if stats.erased().min_is_zero() { - return CompressionEstimate::Skip; + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } // Difference between max and min. let for_bitwidth = match stats.erased().max_minus_min().checked_ilog2() { Some(l) => l + 1, // If max-min == 0, the we should be compressing this as a constant array. - None => return CompressionEstimate::Skip, + None => return CompressionEstimate::Verdict(EstimateVerdict::Skip), }; // If BitPacking can be applied (only non-negative values) and FoR doesn't reduce bit width @@ -162,7 +164,7 @@ impl Scheme for FoRScheme { { let bitpack_bitwidth = max_log + 1; if for_bitwidth >= bitpack_bitwidth { - return CompressionEstimate::Skip; + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } } @@ -173,26 +175,33 @@ impl Scheme for FoRScheme { .try_into() .vortex_expect("bit width must fit in u32"); - CompressionEstimate::Ratio(full_width as f64 / for_bitwidth as f64) + CompressionEstimate::Verdict(EstimateVerdict::Ratio( + full_width as f64 / for_bitwidth as f64, + )) } fn compress( &self, compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { - let primitive = data.array().to_primitive(); + let primitive = data.array().clone().execute::(exec_ctx)?; let for_array = FoR::encode(primitive)?; - let biased = for_array.encoded().to_primitive(); + let biased = for_array + .encoded() + .clone() + .execute::(exec_ctx)?; // Immediately bitpack. If any other scheme was preferable, it would be chosen instead // of bitpacking. // NOTE: we could delegate in the future if we had another downstream codec that performs // as well. - let leaf_ctx = ctx.clone().as_leaf(); - let mut biased_data = ArrayAndStats::new(biased.into_array(), ctx.merged_stats_options()); - let compressed = BitPackingScheme.compress(compressor, &mut biased_data, leaf_ctx)?; + let leaf_ctx = compress_ctx.clone().as_leaf(); + let biased_data = + ArrayAndStats::new(biased.into_array(), compress_ctx.merged_stats_options()); + let compressed = BitPackingScheme.compress(compressor, &biased_data, leaf_ctx, exec_ctx)?; // TODO(connor): This should really be `new_unchecked`. let for_compressed = FoR::try_new(compressed, for_array.reference_scalar().clone())?; @@ -259,38 +268,43 @@ impl Scheme for ZigZagScheme { fn expected_compression_ratio( &self, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { // ZigZag only transforms negative values to positive. Without further compression, // the output is the same size. - if ctx.finished_cascading() { - return CompressionEstimate::Skip; + if compress_ctx.finished_cascading() { + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } - - let stats = data.integer_stats(); + let stats = data.integer_stats(exec_ctx); // ZigZag is only useful when there are negative values. if !stats.erased().min_is_negative() { - return CompressionEstimate::Skip; + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } - CompressionEstimate::Sample + CompressionEstimate::Deferred(DeferredEstimate::Sample) } fn compress( &self, compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { // Zigzag encode the values, then recursively compress the inner values. let zag = zigzag_encode(data.array_as_primitive())?; - let encoded = zag.encoded().to_primitive(); - - let compressed = compressor.compress_child(&encoded.into_array(), &ctx, self.id(), 0)?; - - tracing::debug!("zigzag output: {}", compressed.encoding_id()); + let encoded = zag.encoded().clone().execute::(exec_ctx)?; + + let compressed = compressor.compress_child( + &encoded.into_array(), + &compress_ctx, + self.id(), + 0, + exec_ctx, + )?; Ok(ZigZag::try_new(compressed)?.into_array()) } @@ -307,43 +321,46 @@ impl Scheme for BitPackingScheme { fn expected_compression_ratio( &self, - data: &mut ArrayAndStats, - _ctx: CompressorContext, + data: &ArrayAndStats, + _compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { - let stats = data.integer_stats(); + let stats = data.integer_stats(exec_ctx); // BitPacking only works for non-negative values. if stats.erased().min_is_negative() { - return CompressionEstimate::Skip; + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } - CompressionEstimate::Sample + CompressionEstimate::Deferred(DeferredEstimate::Sample) } fn compress( &self, _compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - _ctx: CompressorContext, + data: &ArrayAndStats, + _compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { let primitive_array = data.array_as_primitive(); - let histogram = bit_width_histogram(&primitive_array)?; + let histogram = bit_width_histogram(primitive_array, exec_ctx)?; let bw = find_best_bit_width(primitive_array.ptype(), &histogram)?; // If best bw is determined to be the current bit-width, return the original array. if bw as usize == primitive_array.ptype().bit_width() { - return Ok(primitive_array.into_array()); + return Ok(primitive_array.array().clone()); } // Otherwise we can bitpack the array. - let packed = bitpack_encode(&primitive_array, bw, Some(&histogram))?; + let primitive_array = primitive_array.into_owned(); + let packed = bitpack_encode(&primitive_array, bw, Some(&histogram), exec_ctx)?; let packed_stats = packed.statistics().to_owned(); let ptype = packed.dtype().as_ptype(); let mut parts = BitPacked::into_parts(packed); - let array = if *USE_EXPERIMENTAL_PATCHES { + let array = if use_experimental_patches() { let patches = parts.patches.take(); // Transpose patches into G-ALP style PatchedArray, wrapping an inner BitPackedArray. let array = BitPacked::try_new( @@ -359,17 +376,17 @@ impl Scheme for BitPackingScheme { match patches { None => array, - Some(p) => Patched::from_array_and_patches( - array, - &p, - &mut LEGACY_SESSION.create_execution_ctx(), - )? - .with_stats_set(packed_stats) - .into_array(), + Some(p) => Patched::from_array_and_patches(array, &p, exec_ctx)? + .with_stats_set(packed_stats) + .into_array(), } } else { // Compress patches and place back into BitPackedArray. - let patches = parts.patches.take().map(compress_patches).transpose()?; + let patches = parts + .patches + .take() + .map(|p| compress_patches(p, exec_ctx)) + .transpose()?; parts.patches = patches; BitPacked::try_new( parts.packed, @@ -433,21 +450,22 @@ impl Scheme for SparseScheme { fn expected_compression_ratio( &self, - data: &mut ArrayAndStats, - _ctx: CompressorContext, + data: &ArrayAndStats, + _compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { let len = data.array_len() as f64; - let stats = data.integer_stats(); + let stats = data.integer_stats(exec_ctx); let value_count = stats.value_count(); // All-null arrays should be compressed as constant instead anyways. if value_count == 0 { - return CompressionEstimate::Skip; + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } // If the majority (90%) of values is null, this will compress well. if stats.null_count() as f64 / len > 0.9 { - return CompressionEstimate::Ratio(len / value_count as f64); + return CompressionEstimate::Verdict(EstimateVerdict::Ratio(len / value_count as f64)); } let (_, most_frequent_count) = stats @@ -459,29 +477,31 @@ impl Scheme for SparseScheme { // If the most frequent value is the only value, we should compress as constant instead. if most_frequent_count == value_count { - return CompressionEstimate::Skip; + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } debug_assert!(value_count > most_frequent_count); // See if the most frequent value accounts for >= 90% of the set values. let freq = most_frequent_count as f64 / value_count as f64; if freq < 0.9 { - return CompressionEstimate::Skip; + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } // We only store the positions of the non-top values. - CompressionEstimate::Ratio(value_count as f64 / (value_count - most_frequent_count) as f64) + CompressionEstimate::Verdict(EstimateVerdict::Ratio( + value_count as f64 / (value_count - most_frequent_count) as f64, + )) } fn compress( &self, compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { let len = data.array_len(); - // TODO(connor): Fight the borrow checker (needs interior mutability)! - let stats = data.integer_stats().clone(); + let stats = data.integer_stats(exec_ctx); let array = data.array(); let (most_frequent_value, most_frequent_count) = stats @@ -511,20 +531,37 @@ impl Scheme for SparseScheme { most_frequent_value.ptype(), array.dtype().nullability(), )), + exec_ctx, )?; if let Some(sparse) = sparse_encoded.as_opt::() { + let sparse_values_primitive = sparse + .patches() + .values() + .clone() + .execute::(exec_ctx)?; let compressed_values = compressor.compress_child( - &sparse.patches().values().to_primitive().into_array(), - &ctx, + &sparse_values_primitive.into_array(), + &compress_ctx, self.id(), 0, + exec_ctx, )?; - let indices = sparse.patches().indices().to_primitive().narrow()?; + let indices = sparse + .patches() + .indices() + .clone() + .execute::(exec_ctx)? + .narrow()?; - let compressed_indices = - compressor.compress_child(&indices.into_array(), &ctx, self.id(), 1)?; + let compressed_indices = compressor.compress_child( + &indices.into_array(), + &compress_ctx, + self.id(), + 1, + exec_ctx, + )?; Sparse::try_new( compressed_indices, @@ -597,30 +634,39 @@ impl Scheme for RunEndScheme { fn expected_compression_ratio( &self, - data: &mut ArrayAndStats, - _ctx: CompressorContext, + data: &ArrayAndStats, + _compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { // If the run length is below the threshold, drop it. - if data.integer_stats().average_run_length() < RUN_END_THRESHOLD { - return CompressionEstimate::Skip; + if data.integer_stats(exec_ctx).average_run_length() < RUN_END_THRESHOLD { + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } - CompressionEstimate::Sample + CompressionEstimate::Deferred(DeferredEstimate::Sample) } fn compress( &self, compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { // Run-end encode the ends. - let (ends, values) = runend_encode(data.array_as_primitive().as_view()); - - let compressed_values = - compressor.compress_child(&values.to_primitive().into_array(), &ctx, self.id(), 0)?; + let (ends, values) = runend_encode(data.array_as_primitive(), exec_ctx); + + let values_primitive = values.execute::(exec_ctx)?; + let compressed_values = compressor.compress_child( + &values_primitive.into_array(), + &compress_ctx, + self.id(), + 0, + exec_ctx, + )?; - let compressed_ends = compressor.compress_child(&ends.into_array(), &ctx, self.id(), 1)?; + let compressed_ends = + compressor.compress_child(&ends.into_array(), &compress_ctx, self.id(), 1, exec_ctx)?; // SAFETY: compression doesn't affect invariants. Ok(unsafe { @@ -661,20 +707,20 @@ impl Scheme for SequenceScheme { fn expected_compression_ratio( &self, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { // It is pointless checking if a sample is a sequence since it will not correspond to the // entire array. - if ctx.is_sample() { - return CompressionEstimate::Skip; + if compress_ctx.is_sample() { + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } - - let stats = data.integer_stats(); + let stats = data.integer_stats(exec_ctx); // `SequenceArray` does not support nulls. if stats.null_count() > 0 { - return CompressionEstimate::Skip; + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } // If the distinct_values_count was computed, and not all values are unique, then this @@ -683,37 +729,49 @@ impl Scheme for SequenceScheme { .distinct_count() .is_some_and(|count| count as usize != data.array_len()) { - return CompressionEstimate::Skip; + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } - // TODO(connor): Why do we sequence encode the whole thing and then throw it away? And then - // why do we divide the ratio by 2??? - - CompressionEstimate::Estimate(Box::new(|_compressor, data, _ctx| { - let Some(encoded) = sequence_encode(&data.array_as_primitive())? else { - // If we are unable to sequence encode this array, make sure we skip. - return Ok(CompressionEstimate::Skip); - }; - - // TODO(connor): This doesn't really make sense? - // Since two values are required to store base and multiplier the compression ratio is - // divided by 2. - Ok(CompressionEstimate::Ratio(encoded.len() as f64 / 2.0)) - })) + // TODO(connor): `sequence_encode` allocates the encoded array just to confirm feasibility. + // A cheaper `is_sequence` probe would let us skip the allocation entirely. + CompressionEstimate::Deferred(DeferredEstimate::Callback(Box::new( + |_compressor, data, best_so_far, _ctx, exec_ctx| { + // `SequenceArray` stores exactly two scalars (base and multiplier), so the best + // achievable compression ratio is `array_len / 2`. + let compressed_size = 2usize; + let max_ratio = data.array_len() as f64 / compressed_size as f64; + + // If we cannot beat the best so far, then we do not want to even try sequence + // encoding the data. + let threshold = best_so_far.and_then(EstimateScore::finite_ratio); + if threshold.is_some_and(|t| max_ratio <= t) { + return Ok(EstimateVerdict::Skip); + } + + // TODO(connor): We should pass this array back to the compressor in the case that + // we do want to sequence encode this so that we do not need to recompress. + if sequence_encode(data.array_as_primitive(), exec_ctx)?.is_none() { + return Ok(EstimateVerdict::Skip); + } + // TODO(connor): Should we get the actual ratio here? + Ok(EstimateVerdict::Ratio(max_ratio)) + }, + ))) } fn compress( &self, _compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - _ctx: CompressorContext, + data: &ArrayAndStats, + _compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { - let stats = data.integer_stats(); + let stats = data.integer_stats(exec_ctx); if stats.null_count() > 0 { vortex_bail!("sequence encoding does not support nulls"); } - sequence_encode(&data.array_as_primitive())? + sequence_encode(data.array_as_primitive(), exec_ctx)? .ok_or_else(|| vortex_err!("cannot sequence encode array")) } } @@ -730,29 +788,32 @@ impl Scheme for PcoScheme { fn expected_compression_ratio( &self, - data: &mut ArrayAndStats, - _ctx: CompressorContext, + data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { use vortex_array::dtype::PType; // Pco does not support I8 or U8. if matches!(data.array_as_primitive().ptype(), PType::I8 | PType::U8) { - return CompressionEstimate::Skip; + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } - CompressionEstimate::Sample + CompressionEstimate::Deferred(DeferredEstimate::Sample) } fn compress( &self, _compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - _ctx: CompressorContext, + data: &ArrayAndStats, + _compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { Ok(vortex_pco::Pco::from_primitive( - &data.array_as_primitive(), + data.array_as_primitive(), pco::DEFAULT_COMPRESSION_LEVEL, 8192, + exec_ctx, )? .into_array()) } @@ -762,46 +823,69 @@ impl Scheme for PcoScheme { pub(crate) fn rle_compress( scheme: &dyn Scheme, compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { - let array = data.array_as_primitive(); - let rle_array = RLE::encode(&array)?; + let rle_array = RLE::encode(data.array_as_primitive(), exec_ctx)?; + let rle_values_primitive = rle_array + .values() + .clone() + .execute::(exec_ctx)?; let compressed_values = compressor.compress_child( - &rle_array.values().to_primitive().into_array(), - &ctx, + &rle_values_primitive.into_array(), + &compress_ctx, scheme.id(), 0, + exec_ctx, )?; // Delta is an unstable encoding, once we deem it stable we can switch over to this always. #[cfg(feature = "unstable_encodings")] - let compressed_indices = try_compress_delta( - compressor, - &rle_array.indices().to_primitive().narrow()?.into_array(), - &ctx, - scheme.id(), - 1, - )?; + let compressed_indices = { + let rle_indices_primitive = rle_array + .indices() + .clone() + .execute::(exec_ctx)? + .narrow()?; + try_compress_delta( + compressor, + &rle_indices_primitive.into_array(), + &compress_ctx, + scheme.id(), + 1, + exec_ctx, + )? + }; #[cfg(not(feature = "unstable_encodings"))] - let compressed_indices = compressor.compress_child( - &rle_array.indices().to_primitive().narrow()?.into_array(), - &ctx, - scheme.id(), - 1, - )?; + let compressed_indices = { + let rle_indices_primitive = rle_array + .indices() + .clone() + .execute::(exec_ctx)? + .narrow()?; + compressor.compress_child( + &rle_indices_primitive.into_array(), + &compress_ctx, + scheme.id(), + 1, + exec_ctx, + )? + }; + let rle_offsets_primitive = rle_array + .values_idx_offsets() + .clone() + .execute::(exec_ctx)? + .narrow()?; let compressed_offsets = compressor.compress_child( - &rle_array - .values_idx_offsets() - .to_primitive() - .narrow()? - .into_array(), - &ctx, + &rle_offsets_primitive.into_array(), + &compress_ctx, scheme.id(), 2, + exec_ctx, )?; // SAFETY: Recursive compression doesn't affect the invariants. @@ -824,14 +908,25 @@ fn try_compress_delta( parent_ctx: &CompressorContext, parent_id: SchemeId, child_index: usize, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { - let (bases, deltas) = - vortex_fastlanes::delta_compress(&child.to_primitive(), &mut compressor.execution_ctx())?; - - let compressed_bases = - compressor.compress_child(&bases.into_array(), parent_ctx, parent_id, child_index)?; - let compressed_deltas = - compressor.compress_child(&deltas.into_array(), parent_ctx, parent_id, child_index)?; + let child_primitive = child.clone().execute::(exec_ctx)?; + let (bases, deltas) = vortex_fastlanes::delta_compress(&child_primitive, exec_ctx)?; + + let compressed_bases = compressor.compress_child( + &bases.into_array(), + parent_ctx, + parent_id, + child_index, + exec_ctx, + )?; + let compressed_deltas = compressor.compress_child( + &deltas.into_array(), + parent_ctx, + parent_id, + child_index, + exec_ctx, + )?; Delta::try_new(compressed_bases, compressed_deltas, 0, child.len()).map(IntoArray::into_array) } @@ -860,45 +955,49 @@ impl Scheme for IntRLEScheme { fn expected_compression_ratio( &self, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { // RLE is only useful when we cascade it with another encoding. - if ctx.finished_cascading() { - return CompressionEstimate::Skip; + if compress_ctx.finished_cascading() { + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } - - if data.integer_stats().average_run_length() < RUN_LENGTH_THRESHOLD { - return CompressionEstimate::Skip; + if data.integer_stats(exec_ctx).average_run_length() < RUN_LENGTH_THRESHOLD { + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } - CompressionEstimate::Sample + CompressionEstimate::Deferred(DeferredEstimate::Sample) } fn compress( &self, compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { - rle_compress(self, compressor, data, ctx) + rle_compress(self, compressor, data, compress_ctx, exec_ctx) } } #[cfg(test)] mod tests { use std::iter; + use std::sync::LazyLock; use itertools::Itertools; use rand::Rng; use rand::SeedableRng; use rand::rngs::StdRng; use vortex_array::IntoArray; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::Constant; use vortex_array::arrays::Dict; use vortex_array::arrays::Masked; use vortex_array::arrays::PrimitiveArray; use vortex_array::assert_arrays_eq; + use vortex_array::session::ArraySession; use vortex_array::validity::Validity; use vortex_buffer::Buffer; use vortex_buffer::BufferMut; @@ -907,16 +1006,20 @@ mod tests { use vortex_error::VortexResult; use vortex_fastlanes::RLE; use vortex_sequence::Sequence; + use vortex_session::VortexSession; use crate::BtrBlocksCompressor; use crate::schemes::integer::IntRLEScheme; + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); + #[test] fn test_empty() -> VortexResult<()> { // Make sure empty array compression does not fail. let btr = BtrBlocksCompressor::default(); let array = PrimitiveArray::new(Buffer::::empty(), Validity::NonNullable); - let result = btr.compress(&array.into_array())?; + let result = btr.compress(&array.into_array(), &mut SESSION.create_execution_ctx())?; assert!(result.is_empty()); Ok(()) @@ -943,7 +1046,10 @@ mod tests { } let btr = BtrBlocksCompressor::default(); - let compressed = btr.compress(&codes.freeze().into_array())?; + let compressed = btr.compress( + &codes.freeze().into_array(), + &mut SESSION.create_execution_ctx(), + )?; assert!(compressed.is::()); Ok(()) } @@ -959,7 +1065,7 @@ mod tests { let validity = array.validity()?; let btr = BtrBlocksCompressor::default(); - let compressed = btr.compress(&array.into_array())?; + let compressed = btr.compress(&array.into_array(), &mut SESSION.create_execution_ctx())?; assert!(compressed.is::()); assert!(compressed.children()[0].is::()); @@ -977,7 +1083,7 @@ mod tests { let array = PrimitiveArray::from_option_iter(values.clone().into_iter().map(Some)); let btr = BtrBlocksCompressor::default(); - let compressed = btr.compress(&array.into_array())?; + let compressed = btr.compress(&array.into_array(), &mut SESSION.create_execution_ctx())?; assert!(compressed.is::()); let decoded = compressed; @@ -995,7 +1101,8 @@ mod tests { let array = PrimitiveArray::new(Buffer::copy_from(&values), Validity::NonNullable); let compressor = CascadingCompressor::new(vec![&IntRLEScheme]); - let compressed = compressor.compress(&array.into_array())?; + let compressed = + compressor.compress(&array.into_array(), &mut SESSION.create_execution_ctx())?; assert!(compressed.is::()); let expected = Buffer::copy_from(&values).into_array(); @@ -1017,7 +1124,7 @@ mod tests { .into_array(); let btr = BtrBlocksCompressor::default(); - btr.compress(&prim)?; + btr.compress(&prim, &mut SESSION.create_execution_ctx())?; Ok(()) } @@ -1027,17 +1134,20 @@ mod tests { #[cfg(test)] mod scheme_selection_tests { use std::iter; + use std::sync::LazyLock; use rand::Rng; use rand::SeedableRng; use rand::rngs::StdRng; use vortex_array::IntoArray; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::Constant; use vortex_array::arrays::Dict; use vortex_array::arrays::PrimitiveArray; use vortex_array::expr::stats::Precision; use vortex_array::expr::stats::Stat; use vortex_array::expr::stats::StatsProviderExt; + use vortex_array::session::ArraySession; use vortex_array::validity::Validity; use vortex_buffer::Buffer; use vortex_error::VortexResult; @@ -1045,16 +1155,20 @@ mod scheme_selection_tests { use vortex_fastlanes::FoR; use vortex_runend::RunEnd; use vortex_sequence::Sequence; + use vortex_session::VortexSession; use vortex_sparse::Sparse; use crate::BtrBlocksCompressor; + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); + #[test] fn test_constant_compressed() -> VortexResult<()> { let values: Vec = iter::repeat_n(42, 100).collect(); let array = PrimitiveArray::new(Buffer::copy_from(&values), Validity::NonNullable); let btr = BtrBlocksCompressor::default(); - let compressed = btr.compress(&array.into_array())?; + let compressed = btr.compress(&array.into_array(), &mut SESSION.create_execution_ctx())?; assert!(compressed.is::()); Ok(()) } @@ -1064,7 +1178,7 @@ mod scheme_selection_tests { let values: Vec = (0..1000).map(|i| 1_000_000 + ((i * 37) % 100)).collect(); let array = PrimitiveArray::new(Buffer::copy_from(&values), Validity::NonNullable); let btr = BtrBlocksCompressor::default(); - let compressed = btr.compress(&array.into_array())?; + let compressed = btr.compress(&array.into_array(), &mut SESSION.create_execution_ctx())?; assert!(compressed.is::()); Ok(()) } @@ -1074,7 +1188,7 @@ mod scheme_selection_tests { let values: Vec = (0..1000).map(|i| i % 16).collect(); let array = PrimitiveArray::new(Buffer::copy_from(&values), Validity::NonNullable); let btr = BtrBlocksCompressor::default(); - let compressed = btr.compress(&array.into_array())?; + let compressed = btr.compress(&array.into_array(), &mut SESSION.create_execution_ctx())?; assert!(compressed.is::()); assert_eq!( compressed.statistics().get_as::(Stat::NullCount), @@ -1103,7 +1217,7 @@ mod scheme_selection_tests { } let array = PrimitiveArray::new(Buffer::copy_from(&values), Validity::NonNullable); let btr = BtrBlocksCompressor::default(); - let compressed = btr.compress(&array.into_array())?; + let compressed = btr.compress(&array.into_array(), &mut SESSION.create_execution_ctx())?; assert!(compressed.is::()); Ok(()) } @@ -1127,7 +1241,7 @@ mod scheme_selection_tests { let array = PrimitiveArray::new(Buffer::copy_from(&codes), Validity::NonNullable); let btr = BtrBlocksCompressor::default(); - let compressed = btr.compress(&array.into_array())?; + let compressed = btr.compress(&array.into_array(), &mut SESSION.create_execution_ctx())?; assert!(compressed.is::()); Ok(()) } @@ -1140,7 +1254,7 @@ mod scheme_selection_tests { } let array = PrimitiveArray::new(Buffer::copy_from(&values), Validity::NonNullable); let btr = BtrBlocksCompressor::default(); - let compressed = btr.compress(&array.into_array())?; + let compressed = btr.compress(&array.into_array(), &mut SESSION.create_execution_ctx())?; assert!(compressed.is::()); Ok(()) } @@ -1150,7 +1264,7 @@ mod scheme_selection_tests { let values: Vec = (0..1000).map(|i| i * 7).collect(); let array = PrimitiveArray::new(Buffer::copy_from(&values), Validity::NonNullable); let btr = BtrBlocksCompressor::default(); - let compressed = btr.compress(&array.into_array())?; + let compressed = btr.compress(&array.into_array(), &mut SESSION.create_execution_ctx())?; assert!(compressed.is::()); Ok(()) } @@ -1163,7 +1277,7 @@ mod scheme_selection_tests { } let array = PrimitiveArray::new(Buffer::copy_from(&values), Validity::NonNullable); let btr = BtrBlocksCompressor::default(); - let compressed = btr.compress(&array.into_array())?; + let compressed = btr.compress(&array.into_array(), &mut SESSION.create_execution_ctx())?; eprintln!("{}", compressed.display_tree()); assert!(compressed.is::()); Ok(()) diff --git a/vortex-btrblocks/src/schemes/patches.rs b/vortex-btrblocks/src/schemes/patches.rs index 8a82058ddb7..0d9854c3b23 100644 --- a/vortex-btrblocks/src/schemes/patches.rs +++ b/vortex-btrblocks/src/schemes/patches.rs @@ -2,34 +2,47 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors use vortex_array::ArrayRef; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; -use vortex_array::ToCanonical; use vortex_array::arrays::ConstantArray; +use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::primitive::PrimitiveArrayExt; use vortex_array::patches::Patches; use vortex_error::VortexError; use vortex_error::VortexResult; /// Compresses the given patches by downscaling integers and checking for constant values. -pub fn compress_patches(patches: Patches) -> VortexResult { +pub fn compress_patches(patches: Patches, ctx: &mut ExecutionCtx) -> VortexResult { // Downscale the patch indices. - let indices = patches.indices().to_primitive().narrow()?.into_array(); + let indices = patches + .indices() + .clone() + .execute::(ctx)? + .narrow()? + .into_array(); // Check if the values are constant. let values = patches.values(); let values = if values .statistics() - .compute_is_constant() + .compute_is_constant(ctx) .unwrap_or_default() { - ConstantArray::new(values.scalar_at(0)?, values.len()).into_array() + ConstantArray::new(values.execute_scalar(0, ctx)?, values.len()).into_array() } else { values.clone() }; let chunk_offsets = patches .chunk_offsets() .as_ref() - .map(|offsets| Ok::(offsets.to_primitive().narrow()?.into_array())) + .map(|offsets| { + let offsets_primitive = offsets + .clone() + .execute::(ctx)? + .narrow()? + .into_array(); + Ok::(offsets_primitive) + }) .transpose()?; Patches::new( diff --git a/vortex-btrblocks/src/schemes/string.rs b/vortex-btrblocks/src/schemes/string.rs index 7aec016534e..0df5a268157 100644 --- a/vortex-btrblocks/src/schemes/string.rs +++ b/vortex-btrblocks/src/schemes/string.rs @@ -5,12 +5,15 @@ use vortex_array::ArrayRef; use vortex_array::Canonical; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; -use vortex_array::ToCanonical; +use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::VarBinArray; use vortex_array::arrays::primitive::PrimitiveArrayExt; use vortex_array::arrays::varbin::VarBinArrayExt; use vortex_compressor::estimate::CompressionEstimate; +use vortex_compressor::estimate::DeferredEstimate; +use vortex_compressor::estimate::EstimateVerdict; use vortex_compressor::scheme::ChildSelection; use vortex_compressor::scheme::DescendantExclusion; use vortex_error::VortexResult; @@ -70,38 +73,49 @@ impl Scheme for FSSTScheme { fn expected_compression_ratio( &self, - _data: &mut ArrayAndStats, - _ctx: CompressorContext, + _data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { - CompressionEstimate::Sample + CompressionEstimate::Deferred(DeferredEstimate::Sample) } fn compress( &self, compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { - let utf8 = data.array_as_utf8(); + let utf8 = data.array_as_utf8().into_owned(); let compressor_fsst = fsst_train_compressor(&utf8); - let fsst = fsst_compress(&utf8, utf8.len(), utf8.dtype(), &compressor_fsst); + let fsst = fsst_compress(&utf8, utf8.len(), utf8.dtype(), &compressor_fsst, exec_ctx); + let uncompressed_lengths_primitive = fsst + .uncompressed_lengths() + .clone() + .execute::(exec_ctx)? + .narrow()?; let compressed_original_lengths = compressor.compress_child( - &fsst - .uncompressed_lengths() - .to_primitive() - .narrow()? - .into_array(), - &ctx, + &uncompressed_lengths_primitive.into_array(), + &compress_ctx, self.id(), 0, + exec_ctx, )?; + let codes_offsets_primitive = fsst + .codes() + .offsets() + .clone() + .execute::(exec_ctx)? + .narrow()?; let compressed_codes_offsets = compressor.compress_child( - &fsst.codes().offsets().to_primitive().narrow()?.into_array(), - &ctx, + &codes_offsets_primitive.into_array(), + &compress_ctx, self.id(), 1, + exec_ctx, )?; let compressed_codes = VarBinArray::try_new( compressed_codes_offsets, @@ -116,6 +130,7 @@ impl Scheme for FSSTScheme { fsst.symbol_lengths().clone(), compressed_codes, compressed_original_lengths, + exec_ctx, )?; Ok(fsst.into_array()) @@ -152,41 +167,53 @@ impl Scheme for NullDominatedSparseScheme { fn expected_compression_ratio( &self, - data: &mut ArrayAndStats, - _ctx: CompressorContext, + data: &ArrayAndStats, + _compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { let len = data.array_len() as f64; - let stats = data.string_stats(); + let stats = data.string_stats(exec_ctx); let value_count = stats.value_count(); // All-null arrays should be compressed as constant instead anyways. if value_count == 0 { - return CompressionEstimate::Skip; + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } // If the majority (90%) of values is null, this will compress well. if stats.null_count() as f64 / len > 0.9 { - return CompressionEstimate::Ratio(len / value_count as f64); + return CompressionEstimate::Verdict(EstimateVerdict::Ratio(len / value_count as f64)); } // Otherwise we don't go this route. - CompressionEstimate::Skip + CompressionEstimate::Verdict(EstimateVerdict::Skip) } fn compress( &self, compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { // We pass None as we only run this pathway for NULL-dominated string arrays. - let sparse_encoded = Sparse::encode(data.array(), None)?; + let sparse_encoded = Sparse::encode(data.array(), None, exec_ctx)?; if let Some(sparse) = sparse_encoded.as_opt::() { // Compress the indices only (not the values for strings). - let indices = sparse.patches().indices().to_primitive().narrow()?; - let compressed_indices = - compressor.compress_child(&indices.into_array(), &ctx, self.id(), 0)?; + let indices = sparse + .patches() + .indices() + .clone() + .execute::(exec_ctx)? + .narrow()?; + let compressed_indices = compressor.compress_child( + &indices.into_array(), + &compress_ctx, + self.id(), + 0, + exec_ctx, + )?; Sparse::try_new( compressed_indices, @@ -213,20 +240,25 @@ impl Scheme for ZstdScheme { fn expected_compression_ratio( &self, - _data: &mut ArrayAndStats, - _ctx: CompressorContext, + _data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { - CompressionEstimate::Sample + CompressionEstimate::Deferred(DeferredEstimate::Sample) } fn compress( &self, _compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - _ctx: CompressorContext, + data: &ArrayAndStats, + _compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { - let compacted = data.array_as_utf8().compact_buffers()?; - Ok(vortex_zstd::Zstd::from_var_bin_view_without_dict(&compacted, 3, 8192)?.into_array()) + let compacted = data.array_as_utf8().into_owned().compact_buffers()?; + Ok( + vortex_zstd::Zstd::from_var_bin_view_without_dict(&compacted, 3, 8192, exec_ctx)? + .into_array(), + ) } } @@ -242,38 +274,45 @@ impl Scheme for ZstdBuffersScheme { fn expected_compression_ratio( &self, - _data: &mut ArrayAndStats, - _ctx: CompressorContext, + _data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { - CompressionEstimate::Sample + CompressionEstimate::Deferred(DeferredEstimate::Sample) } fn compress( &self, _compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - _ctx: CompressorContext, + data: &ArrayAndStats, + _compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { - Ok( - vortex_zstd::ZstdBuffers::compress(data.array(), 3, &vortex_array::LEGACY_SESSION)? - .into_array(), - ) + Ok(vortex_zstd::ZstdBuffers::compress(data.array(), 3, exec_ctx.session())?.into_array()) } } #[cfg(test)] mod tests { + use std::sync::LazyLock; + use vortex_array::IntoArray; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::VarBinViewArray; use vortex_array::builders::ArrayBuilder; use vortex_array::builders::VarBinViewBuilder; use vortex_array::display::DisplayOptions; use vortex_array::dtype::DType; use vortex_array::dtype::Nullability; + use vortex_array::session::ArraySession; use vortex_error::VortexResult; + use vortex_session::VortexSession; use crate::BtrBlocksCompressor; + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); + #[test] fn test_strings() -> VortexResult<()> { let mut strings = Vec::new(); @@ -287,7 +326,7 @@ mod tests { let array_ref = strings.into_array(); let btr = BtrBlocksCompressor::default(); - let compressed = btr.compress(&array_ref)?; + let compressed = btr.compress(&array_ref, &mut SESSION.create_execution_ctx())?; assert_eq!(compressed.len(), 2048); let display = compressed @@ -310,7 +349,7 @@ mod tests { let array_ref = strings.into_array(); let btr = BtrBlocksCompressor::default(); - let compressed = btr.compress(&array_ref)?; + let compressed = btr.compress(&array_ref, &mut SESSION.create_execution_ctx())?; assert_eq!(compressed.len(), 100); let display = compressed @@ -326,23 +365,32 @@ mod tests { /// Tests to verify that each string compression scheme produces the expected encoding. #[cfg(test)] mod scheme_selection_tests { + use std::sync::LazyLock; + use vortex_array::IntoArray; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::Constant; use vortex_array::arrays::Dict; use vortex_array::arrays::VarBinViewArray; use vortex_array::dtype::DType; use vortex_array::dtype::Nullability; + use vortex_array::session::ArraySession; use vortex_error::VortexResult; use vortex_fsst::FSST; + use vortex_session::VortexSession; use crate::BtrBlocksCompressor; + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); + #[test] fn test_constant_compressed() -> VortexResult<()> { let strings: Vec> = vec![Some("constant_value"); 100]; let array = VarBinViewArray::from_iter(strings, DType::Utf8(Nullability::NonNullable)); let array_ref = array.into_array(); - let compressed = BtrBlocksCompressor::default().compress(&array_ref)?; + let compressed = BtrBlocksCompressor::default() + .compress(&array_ref, &mut SESSION.create_execution_ctx())?; assert!(compressed.is::()); Ok(()) } @@ -356,7 +404,8 @@ mod scheme_selection_tests { } let array = VarBinViewArray::from_iter(strings, DType::Utf8(Nullability::NonNullable)); let array_ref = array.into_array(); - let compressed = BtrBlocksCompressor::default().compress(&array_ref)?; + let compressed = BtrBlocksCompressor::default() + .compress(&array_ref, &mut SESSION.create_execution_ctx())?; assert!(compressed.is::()); Ok(()) } @@ -371,7 +420,8 @@ mod scheme_selection_tests { } let array = VarBinViewArray::from_iter(strings, DType::Utf8(Nullability::NonNullable)); let array_ref = array.into_array(); - let compressed = BtrBlocksCompressor::default().compress(&array_ref)?; + let compressed = BtrBlocksCompressor::default() + .compress(&array_ref, &mut SESSION.create_execution_ctx())?; assert!(compressed.is::()); Ok(()) } diff --git a/vortex-btrblocks/src/schemes/temporal.rs b/vortex-btrblocks/src/schemes/temporal.rs index 306ccaf982d..73aa9eedfb4 100644 --- a/vortex-btrblocks/src/schemes/temporal.rs +++ b/vortex-btrblocks/src/schemes/temporal.rs @@ -5,16 +5,20 @@ use vortex_array::ArrayRef; use vortex_array::Canonical; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; -use vortex_array::ToCanonical; use vortex_array::aggregate_fn::fns::is_constant::is_constant; use vortex_array::arrays::ConstantArray; +use vortex_array::arrays::ExtensionArray; +use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::TemporalArray; +use vortex_array::arrays::extension::ExtensionArrayExt; use vortex_array::arrays::primitive::PrimitiveArrayExt; use vortex_array::dtype::extension::Matcher; use vortex_array::extension::datetime::AnyTemporal; use vortex_array::extension::datetime::TemporalMetadata; use vortex_compressor::estimate::CompressionEstimate; +use vortex_compressor::estimate::EstimateVerdict; use vortex_datetime_parts::DateTimeParts; use vortex_datetime_parts::TemporalParts; use vortex_datetime_parts::split_temporal; @@ -58,31 +62,33 @@ impl Scheme for TemporalScheme { fn expected_compression_ratio( &self, - _data: &mut ArrayAndStats, - _ctx: CompressorContext, + _data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { // Temporal compression (splitting into parts) is almost always beneficial. - CompressionEstimate::AlwaysUse + CompressionEstimate::Verdict(EstimateVerdict::AlwaysUse) } fn compress( &self, compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { let array = data.array().clone(); - let ext_array = array.to_extension(); + let ext_array = array.execute::(exec_ctx)?; let temporal_array = TemporalArray::try_from(ext_array.clone().into_array())?; // Check for constant array and return early if so. - let is_constant = is_constant( - &ext_array.clone().into_array(), - &mut compressor.execution_ctx(), - )?; + let is_constant = is_constant(&ext_array.clone().into_array(), exec_ctx)?; if is_constant { - return Ok(ConstantArray::new(ext_array.scalar_at(0)?, ext_array.len()).into_array()); + return Ok( + ConstantArray::new(ext_array.execute_scalar(0, exec_ctx)?, ext_array.len()) + .into_array(), + ); } let dtype = temporal_array.dtype().clone(); @@ -90,25 +96,31 @@ impl Scheme for TemporalScheme { days, seconds, subseconds, - } = split_temporal(temporal_array)?; + } = split_temporal(temporal_array, exec_ctx)?; + let days_primitive = days.execute::(exec_ctx)?.narrow()?; let days = compressor.compress_child( - &days.to_primitive().narrow()?.into_array(), - &ctx, + &days_primitive.into_array(), + &compress_ctx, self.id(), 0, + exec_ctx, )?; + let seconds_primitive = seconds.execute::(exec_ctx)?.narrow()?; let seconds = compressor.compress_child( - &seconds.to_primitive().narrow()?.into_array(), - &ctx, + &seconds_primitive.into_array(), + &compress_ctx, self.id(), 1, + exec_ctx, )?; + let subseconds_primitive = subseconds.execute::(exec_ctx)?.narrow()?; let subseconds = compressor.compress_child( - &subseconds.to_primitive().narrow()?.into_array(), - &ctx, + &subseconds_primitive.into_array(), + &compress_ctx, self.id(), 2, + exec_ctx, )?; Ok(DateTimeParts::try_new(dtype, days, seconds, subseconds)?.into_array()) diff --git a/vortex-buffer/benches/vortex_bitbuffer.rs b/vortex-buffer/benches/vortex_bitbuffer.rs index 807f890b706..67ce88da889 100644 --- a/vortex-buffer/benches/vortex_bitbuffer.rs +++ b/vortex-buffer/benches/vortex_bitbuffer.rs @@ -1,8 +1,6 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] - use std::iter::Iterator; use arrow_buffer::BooleanBuffer; diff --git a/vortex-buffer/benches/vortex_buffer.rs b/vortex-buffer/benches/vortex_buffer.rs index 1fc28023c75..5c97cbb3915 100644 --- a/vortex-buffer/benches/vortex_buffer.rs +++ b/vortex-buffer/benches/vortex_buffer.rs @@ -1,8 +1,6 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] - use std::iter::Iterator; use arrow_buffer::ArrowNativeType; diff --git a/vortex-buffer/public-api.lock b/vortex-buffer/public-api.lock index 29b941c77d4..f6c52f9df89 100644 --- a/vortex-buffer/public-api.lock +++ b/vortex-buffer/public-api.lock @@ -160,11 +160,11 @@ impl vortex_buffer::Alignment pub fn vortex_buffer::Alignment::exponent(&self) -> u8 -pub const fn vortex_buffer::Alignment::from_exponent(exponent: u8) -> Self +pub const fn vortex_buffer::Alignment::from_exponent(u8) -> Self -pub fn vortex_buffer::Alignment::is_aligned_to(&self, other: vortex_buffer::Alignment) -> bool +pub fn vortex_buffer::Alignment::is_aligned_to(&self, vortex_buffer::Alignment) -> bool -pub const fn vortex_buffer::Alignment::new(align: usize) -> Self +pub const fn vortex_buffer::Alignment::new(usize) -> Self pub const fn vortex_buffer::Alignment::none() -> Self @@ -178,53 +178,53 @@ impl core::cmp::Eq for vortex_buffer::Alignment impl core::cmp::Ord for vortex_buffer::Alignment -pub fn vortex_buffer::Alignment::cmp(&self, other: &vortex_buffer::Alignment) -> core::cmp::Ordering +pub fn vortex_buffer::Alignment::cmp(&self, &vortex_buffer::Alignment) -> core::cmp::Ordering impl core::cmp::PartialEq for vortex_buffer::Alignment -pub fn vortex_buffer::Alignment::eq(&self, other: &vortex_buffer::Alignment) -> bool +pub fn vortex_buffer::Alignment::eq(&self, &vortex_buffer::Alignment) -> bool impl core::cmp::PartialOrd for vortex_buffer::Alignment -pub fn vortex_buffer::Alignment::partial_cmp(&self, other: &vortex_buffer::Alignment) -> core::option::Option +pub fn vortex_buffer::Alignment::partial_cmp(&self, &vortex_buffer::Alignment) -> core::option::Option impl core::convert::From for vortex_buffer::Alignment -pub fn vortex_buffer::Alignment::from(value: u16) -> Self +pub fn vortex_buffer::Alignment::from(u16) -> Self impl core::convert::From for vortex_buffer::Alignment -pub fn vortex_buffer::Alignment::from(value: usize) -> Self +pub fn vortex_buffer::Alignment::from(usize) -> Self impl core::convert::From for u16 -pub fn u16::from(value: vortex_buffer::Alignment) -> Self +pub fn u16::from(vortex_buffer::Alignment) -> Self impl core::convert::From for u32 -pub fn u32::from(value: vortex_buffer::Alignment) -> Self +pub fn u32::from(vortex_buffer::Alignment) -> Self impl core::convert::From for usize -pub fn usize::from(value: vortex_buffer::Alignment) -> Self +pub fn usize::from(vortex_buffer::Alignment) -> Self impl core::convert::TryFrom for vortex_buffer::Alignment pub type vortex_buffer::Alignment::Error = vortex_error::VortexError -pub fn vortex_buffer::Alignment::try_from(value: u32) -> core::result::Result +pub fn vortex_buffer::Alignment::try_from(u32) -> core::result::Result impl core::fmt::Debug for vortex_buffer::Alignment -pub fn vortex_buffer::Alignment::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_buffer::Alignment::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_buffer::Alignment -pub fn vortex_buffer::Alignment::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_buffer::Alignment::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_buffer::Alignment -pub fn vortex_buffer::Alignment::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_buffer::Alignment::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_buffer::Alignment @@ -240,9 +240,9 @@ pub struct vortex_buffer::BitBuffer impl vortex_buffer::BitBuffer -pub fn vortex_buffer::BitBuffer::bitand_not(&self, rhs: &vortex_buffer::BitBuffer) -> vortex_buffer::BitBuffer +pub fn vortex_buffer::BitBuffer::bitand_not(&self, &vortex_buffer::BitBuffer) -> vortex_buffer::BitBuffer -pub fn vortex_buffer::BitBuffer::iter_bits(&self, f: F) where F: core::ops::function::FnMut(usize, bool) +pub fn vortex_buffer::BitBuffer::iter_bits(&self, F) where F: core::ops::function::FnMut(usize, bool) impl vortex_buffer::BitBuffer @@ -250,15 +250,15 @@ pub fn vortex_buffer::BitBuffer::chunks(&self) -> arrow_buffer::util::bit_chunk_ pub fn vortex_buffer::BitBuffer::clear(&mut self) -pub fn vortex_buffer::BitBuffer::collect_bool bool>(len: usize, f: F) -> Self +pub fn vortex_buffer::BitBuffer::collect_bool bool>(usize, F) -> Self pub fn vortex_buffer::BitBuffer::empty() -> Self pub fn vortex_buffer::BitBuffer::false_count(&self) -> usize -pub fn vortex_buffer::BitBuffer::from_indices(len: usize, indices: &[usize]) -> vortex_buffer::BitBuffer +pub fn vortex_buffer::BitBuffer::from_indices(usize, &[usize]) -> vortex_buffer::BitBuffer -pub fn vortex_buffer::BitBuffer::full(value: bool, len: usize) -> Self +pub fn vortex_buffer::BitBuffer::full(bool, usize) -> Self pub fn vortex_buffer::BitBuffer::inner(&self) -> &vortex_buffer::ByteBuffer @@ -268,15 +268,15 @@ pub fn vortex_buffer::BitBuffer::iter(&self) -> arrow_buffer::util::bit_iterator pub fn vortex_buffer::BitBuffer::len(&self) -> usize -pub fn vortex_buffer::BitBuffer::map_cmp(&self, f: F) -> Self where F: core::ops::function::FnMut(usize, bool) -> bool +pub fn vortex_buffer::BitBuffer::map_cmp(&self, F) -> Self where F: core::ops::function::FnMut(usize, bool) -> bool -pub fn vortex_buffer::BitBuffer::new(buffer: vortex_buffer::ByteBuffer, len: usize) -> Self +pub fn vortex_buffer::BitBuffer::new(vortex_buffer::ByteBuffer, usize) -> Self -pub fn vortex_buffer::BitBuffer::new_set(len: usize) -> Self +pub fn vortex_buffer::BitBuffer::new_set(usize) -> Self -pub fn vortex_buffer::BitBuffer::new_unset(len: usize) -> Self +pub fn vortex_buffer::BitBuffer::new_unset(usize) -> Self -pub fn vortex_buffer::BitBuffer::new_with_offset(buffer: vortex_buffer::ByteBuffer, len: usize, offset: usize) -> Self +pub fn vortex_buffer::BitBuffer::new_with_offset(vortex_buffer::ByteBuffer, usize, usize) -> Self pub fn vortex_buffer::BitBuffer::offset(&self) -> usize @@ -286,7 +286,7 @@ pub fn vortex_buffer::BitBuffer::set_slices(&self) -> arrow_buffer::util::bit_it pub fn vortex_buffer::BitBuffer::shrink_offset(self) -> Self -pub fn vortex_buffer::BitBuffer::slice(&self, range: impl core::ops::range::RangeBounds) -> Self +pub fn vortex_buffer::BitBuffer::slice(&self, impl core::ops::range::RangeBounds) -> Self pub fn vortex_buffer::BitBuffer::sliced(&self) -> Self @@ -294,9 +294,9 @@ pub fn vortex_buffer::BitBuffer::true_count(&self) -> usize pub fn vortex_buffer::BitBuffer::unaligned_chunks(&self) -> arrow_buffer::util::bit_chunk_iterator::UnalignedBitChunk<'_> -pub fn vortex_buffer::BitBuffer::value(&self, index: usize) -> bool +pub fn vortex_buffer::BitBuffer::value(&self, usize) -> bool -pub unsafe fn vortex_buffer::BitBuffer::value_unchecked(&self, index: usize) -> bool +pub unsafe fn vortex_buffer::BitBuffer::value_unchecked(&self, usize) -> bool impl vortex_buffer::BitBuffer @@ -312,89 +312,89 @@ impl core::cmp::Eq for vortex_buffer::BitBuffer impl core::cmp::PartialEq for vortex_buffer::BitBuffer -pub fn vortex_buffer::BitBuffer::eq(&self, other: &Self) -> bool +pub fn vortex_buffer::BitBuffer::eq(&self, &Self) -> bool impl core::convert::From<&[bool]> for vortex_buffer::BitBuffer -pub fn vortex_buffer::BitBuffer::from(value: &[bool]) -> Self +pub fn vortex_buffer::BitBuffer::from(&[bool]) -> Self impl core::convert::From> for vortex_buffer::BitBuffer -pub fn vortex_buffer::BitBuffer::from(value: alloc::vec::Vec) -> Self +pub fn vortex_buffer::BitBuffer::from(alloc::vec::Vec) -> Self impl core::convert::From for vortex_buffer::BitBuffer -pub fn vortex_buffer::BitBuffer::from(value: arrow_buffer::buffer::boolean::BooleanBuffer) -> Self +pub fn vortex_buffer::BitBuffer::from(arrow_buffer::buffer::boolean::BooleanBuffer) -> Self impl core::convert::From for arrow_buffer::buffer::boolean::BooleanBuffer -pub fn arrow_buffer::buffer::boolean::BooleanBuffer::from(value: vortex_buffer::BitBuffer) -> Self +pub fn arrow_buffer::buffer::boolean::BooleanBuffer::from(vortex_buffer::BitBuffer) -> Self impl core::fmt::Debug for vortex_buffer::BitBuffer -pub fn vortex_buffer::BitBuffer::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_buffer::BitBuffer::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_buffer::BitBuffer -pub fn vortex_buffer::BitBuffer::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_buffer::BitBuffer::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::iter::traits::collect::FromIterator for vortex_buffer::BitBuffer -pub fn vortex_buffer::BitBuffer::from_iter>(iter: T) -> Self +pub fn vortex_buffer::BitBuffer::from_iter>(T) -> Self impl core::ops::bit::BitAnd for &vortex_buffer::BitBuffer pub type &vortex_buffer::BitBuffer::Output = vortex_buffer::BitBuffer -pub fn &vortex_buffer::BitBuffer::bitand(self, rhs: Self) -> Self::Output +pub fn &vortex_buffer::BitBuffer::bitand(self, Self) -> Self::Output impl core::ops::bit::BitAnd for vortex_buffer::BitBuffer pub type vortex_buffer::BitBuffer::Output = vortex_buffer::BitBuffer -pub fn vortex_buffer::BitBuffer::bitand(self, rhs: vortex_buffer::BitBuffer) -> Self::Output +pub fn vortex_buffer::BitBuffer::bitand(self, vortex_buffer::BitBuffer) -> Self::Output impl core::ops::bit::BitAnd<&vortex_buffer::BitBuffer> for vortex_buffer::BitBuffer pub type vortex_buffer::BitBuffer::Output = vortex_buffer::BitBuffer -pub fn vortex_buffer::BitBuffer::bitand(self, rhs: &vortex_buffer::BitBuffer) -> Self::Output +pub fn vortex_buffer::BitBuffer::bitand(self, &vortex_buffer::BitBuffer) -> Self::Output impl core::ops::bit::BitAnd for &vortex_buffer::BitBuffer pub type &vortex_buffer::BitBuffer::Output = vortex_buffer::BitBuffer -pub fn &vortex_buffer::BitBuffer::bitand(self, rhs: vortex_buffer::BitBuffer) -> Self::Output +pub fn &vortex_buffer::BitBuffer::bitand(self, vortex_buffer::BitBuffer) -> Self::Output impl core::ops::bit::BitOr for &vortex_buffer::BitBuffer pub type &vortex_buffer::BitBuffer::Output = vortex_buffer::BitBuffer -pub fn &vortex_buffer::BitBuffer::bitor(self, rhs: Self) -> Self::Output +pub fn &vortex_buffer::BitBuffer::bitor(self, Self) -> Self::Output impl core::ops::bit::BitOr for vortex_buffer::BitBuffer pub type vortex_buffer::BitBuffer::Output = vortex_buffer::BitBuffer -pub fn vortex_buffer::BitBuffer::bitor(self, rhs: Self) -> Self::Output +pub fn vortex_buffer::BitBuffer::bitor(self, Self) -> Self::Output impl core::ops::bit::BitOr<&vortex_buffer::BitBuffer> for vortex_buffer::BitBuffer pub type vortex_buffer::BitBuffer::Output = vortex_buffer::BitBuffer -pub fn vortex_buffer::BitBuffer::bitor(self, rhs: &vortex_buffer::BitBuffer) -> Self::Output +pub fn vortex_buffer::BitBuffer::bitor(self, &vortex_buffer::BitBuffer) -> Self::Output impl core::ops::bit::BitXor for &vortex_buffer::BitBuffer pub type &vortex_buffer::BitBuffer::Output = vortex_buffer::BitBuffer -pub fn &vortex_buffer::BitBuffer::bitxor(self, rhs: Self) -> Self::Output +pub fn &vortex_buffer::BitBuffer::bitxor(self, Self) -> Self::Output impl core::ops::bit::BitXor<&vortex_buffer::BitBuffer> for vortex_buffer::BitBuffer pub type vortex_buffer::BitBuffer::Output = vortex_buffer::BitBuffer -pub fn vortex_buffer::BitBuffer::bitxor(self, rhs: &vortex_buffer::BitBuffer) -> Self::Output +pub fn vortex_buffer::BitBuffer::bitxor(self, &vortex_buffer::BitBuffer) -> Self::Output impl core::ops::bit::Not for &vortex_buffer::BitBuffer @@ -420,47 +420,41 @@ pub struct vortex_buffer::BitBufferMut impl vortex_buffer::BitBufferMut -pub fn vortex_buffer::BitBufferMut::append(&mut self, value: bool) +pub fn vortex_buffer::BitBufferMut::append(&mut self, bool) -pub fn vortex_buffer::BitBufferMut::append_buffer(&mut self, buffer: &vortex_buffer::BitBuffer) +pub fn vortex_buffer::BitBufferMut::append_buffer(&mut self, &vortex_buffer::BitBuffer) pub fn vortex_buffer::BitBufferMut::append_false(&mut self) -pub fn vortex_buffer::BitBufferMut::append_n(&mut self, value: bool, n: usize) +pub fn vortex_buffer::BitBufferMut::append_n(&mut self, bool, usize) pub fn vortex_buffer::BitBufferMut::append_true(&mut self) -pub fn vortex_buffer::BitBufferMut::as_mut_ptr(&mut self) -> *mut u8 - pub fn vortex_buffer::BitBufferMut::as_mut_slice(&mut self) -> &mut [u8] pub fn vortex_buffer::BitBufferMut::as_slice(&self) -> &[u8] pub fn vortex_buffer::BitBufferMut::capacity(&self) -> usize -pub fn vortex_buffer::BitBufferMut::chunks(&self) -> arrow_buffer::util::bit_chunk_iterator::BitChunks<'_> - pub fn vortex_buffer::BitBufferMut::clear(&mut self) -pub fn vortex_buffer::BitBufferMut::collect_bool bool>(len: usize, f: F) -> Self +pub fn vortex_buffer::BitBufferMut::collect_bool bool>(usize, F) -> Self -pub fn vortex_buffer::BitBufferMut::copy_from(bit_buffer: &vortex_buffer::BitBuffer) -> Self +pub fn vortex_buffer::BitBufferMut::copy_from(&vortex_buffer::BitBuffer) -> Self pub fn vortex_buffer::BitBufferMut::empty() -> Self -pub fn vortex_buffer::BitBufferMut::false_count(&self) -> usize +pub fn vortex_buffer::BitBufferMut::fill_range(&mut self, usize, usize, bool) -pub fn vortex_buffer::BitBufferMut::fill_range(&mut self, start: usize, end: usize, value: bool) - -pub unsafe fn vortex_buffer::BitBufferMut::fill_range_unchecked(&mut self, start: usize, end: usize, value: bool) +pub unsafe fn vortex_buffer::BitBufferMut::fill_range_unchecked(&mut self, usize, usize, bool) pub fn vortex_buffer::BitBufferMut::freeze(self) -> vortex_buffer::BitBuffer -pub fn vortex_buffer::BitBufferMut::from_buffer(buffer: vortex_buffer::ByteBufferMut, offset: usize, len: usize) -> Self +pub fn vortex_buffer::BitBufferMut::from_buffer(vortex_buffer::ByteBufferMut, usize, usize) -> Self -pub fn vortex_buffer::BitBufferMut::from_indices(len: usize, indices: &[usize]) -> vortex_buffer::BitBufferMut +pub fn vortex_buffer::BitBufferMut::from_indices(usize, &[usize]) -> vortex_buffer::BitBufferMut -pub fn vortex_buffer::BitBufferMut::full(value: bool, len: usize) -> Self +pub fn vortex_buffer::BitBufferMut::full(bool, usize) -> Self pub fn vortex_buffer::BitBufferMut::inner(&self) -> &vortex_buffer::ByteBufferMut @@ -470,59 +464,53 @@ pub fn vortex_buffer::BitBufferMut::is_empty(&self) -> bool pub fn vortex_buffer::BitBufferMut::len(&self) -> usize -pub fn vortex_buffer::BitBufferMut::new_set(len: usize) -> Self +pub fn vortex_buffer::BitBufferMut::new_set(usize) -> Self -pub fn vortex_buffer::BitBufferMut::new_unset(len: usize) -> Self +pub fn vortex_buffer::BitBufferMut::new_unset(usize) -> Self pub fn vortex_buffer::BitBufferMut::offset(&self) -> usize -pub fn vortex_buffer::BitBufferMut::reserve(&mut self, additional: usize) - -pub fn vortex_buffer::BitBufferMut::set(&mut self, index: usize) +pub fn vortex_buffer::BitBufferMut::reserve(&mut self, usize) -pub unsafe fn vortex_buffer::BitBufferMut::set_len(&mut self, new_len: usize) +pub fn vortex_buffer::BitBufferMut::set(&mut self, usize) -pub fn vortex_buffer::BitBufferMut::set_to(&mut self, index: usize, value: bool) +pub unsafe fn vortex_buffer::BitBufferMut::set_len(&mut self, usize) -pub unsafe fn vortex_buffer::BitBufferMut::set_to_unchecked(&mut self, index: usize, value: bool) +pub fn vortex_buffer::BitBufferMut::set_to(&mut self, usize, bool) -pub fn vortex_buffer::BitBufferMut::split_off(&mut self, at: usize) -> Self +pub unsafe fn vortex_buffer::BitBufferMut::set_to_unchecked(&mut self, usize, bool) -pub fn vortex_buffer::BitBufferMut::true_count(&self) -> usize +pub unsafe fn vortex_buffer::BitBufferMut::set_unchecked(&mut self, usize) -pub fn vortex_buffer::BitBufferMut::truncate(&mut self, len: usize) +pub fn vortex_buffer::BitBufferMut::truncate(&mut self, usize) -pub fn vortex_buffer::BitBufferMut::unaligned_chunks(&self) -> arrow_buffer::util::bit_chunk_iterator::UnalignedBitChunk<'_> +pub fn vortex_buffer::BitBufferMut::unset(&mut self, usize) -pub fn vortex_buffer::BitBufferMut::unset(&mut self, index: usize) +pub unsafe fn vortex_buffer::BitBufferMut::unset_unchecked(&mut self, usize) -pub unsafe fn vortex_buffer::BitBufferMut::unset_unchecked(&mut self, index: usize) +pub fn vortex_buffer::BitBufferMut::unsplit(&mut self, Self) -pub fn vortex_buffer::BitBufferMut::unsplit(&mut self, other: Self) +pub fn vortex_buffer::BitBufferMut::value(&self, usize) -> bool -pub fn vortex_buffer::BitBufferMut::value(&self, index: usize) -> bool +pub unsafe fn vortex_buffer::BitBufferMut::value_unchecked(&self, usize) -> bool -pub unsafe fn vortex_buffer::BitBufferMut::value_unchecked(&self, index: usize) -> bool - -pub fn vortex_buffer::BitBufferMut::with_capacity(capacity: usize) -> Self +pub fn vortex_buffer::BitBufferMut::with_capacity(usize) -> Self impl core::clone::Clone for vortex_buffer::BitBufferMut pub fn vortex_buffer::BitBufferMut::clone(&self) -> vortex_buffer::BitBufferMut -impl core::cmp::Eq for vortex_buffer::BitBufferMut - -impl core::cmp::PartialEq for vortex_buffer::BitBufferMut +impl core::convert::From<&[bool]> for vortex_buffer::BitBufferMut -pub fn vortex_buffer::BitBufferMut::eq(&self, other: &Self) -> bool +pub fn vortex_buffer::BitBufferMut::from(&[bool]) -> Self -impl core::convert::From<&[bool]> for vortex_buffer::BitBufferMut +impl core::convert::From<&[u8]> for vortex_buffer::BitBufferMut -pub fn vortex_buffer::BitBufferMut::from(value: &[bool]) -> Self +pub fn vortex_buffer::BitBufferMut::from(&[u8]) -> Self impl core::convert::From> for vortex_buffer::BitBufferMut -pub fn vortex_buffer::BitBufferMut::from(value: alloc::vec::Vec) -> Self +pub fn vortex_buffer::BitBufferMut::from(alloc::vec::Vec) -> Self impl core::default::Default for vortex_buffer::BitBufferMut @@ -530,11 +518,11 @@ pub fn vortex_buffer::BitBufferMut::default() -> Self impl core::fmt::Debug for vortex_buffer::BitBufferMut -pub fn vortex_buffer::BitBufferMut::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_buffer::BitBufferMut::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::iter::traits::collect::FromIterator for vortex_buffer::BitBufferMut -pub fn vortex_buffer::BitBufferMut::from_iter>(iter: T) -> Self +pub fn vortex_buffer::BitBufferMut::from_iter>(T) -> Self impl core::ops::bit::Not for vortex_buffer::BitBufferMut @@ -546,13 +534,13 @@ pub struct vortex_buffer::Buffer impl vortex_buffer::Buffer -pub fn vortex_buffer::Buffer::from_arrow_buffer(arrow: arrow_buffer::buffer::immutable::Buffer, alignment: vortex_buffer::Alignment) -> Self +pub fn vortex_buffer::Buffer::from_arrow_buffer(arrow_buffer::buffer::immutable::Buffer, vortex_buffer::Alignment) -> Self pub fn vortex_buffer::Buffer::into_arrow_buffer(self) -> arrow_buffer::buffer::immutable::Buffer impl vortex_buffer::Buffer -pub fn vortex_buffer::Buffer::from_arrow_scalar_buffer(arrow: arrow_buffer::buffer::scalar::ScalarBuffer) -> Self +pub fn vortex_buffer::Buffer::from_arrow_scalar_buffer(arrow_buffer::buffer::scalar::ScalarBuffer) -> Self pub fn vortex_buffer::Buffer::into_arrow_offset_buffer(self) -> arrow_buffer::buffer::offset::OffsetBuffer @@ -560,7 +548,7 @@ pub fn vortex_buffer::Buffer::into_arrow_scalar_buffer(self) -> arrow_buffer: impl vortex_buffer::Buffer -pub fn vortex_buffer::Buffer::aligned(self, alignment: vortex_buffer::Alignment) -> Self +pub fn vortex_buffer::Buffer::aligned(self, vortex_buffer::Alignment) -> Self pub fn vortex_buffer::Buffer::alignment(&self) -> vortex_buffer::Alignment @@ -570,25 +558,25 @@ pub fn vortex_buffer::Buffer::as_slice(&self) -> &[T] pub fn vortex_buffer::Buffer::clear(&mut self) -pub fn vortex_buffer::Buffer::copy_from(values: impl core::convert::AsRef<[T]>) -> Self +pub fn vortex_buffer::Buffer::copy_from(impl core::convert::AsRef<[T]>) -> Self -pub fn vortex_buffer::Buffer::copy_from_aligned(values: impl core::convert::AsRef<[T]>, alignment: vortex_buffer::Alignment) -> Self +pub fn vortex_buffer::Buffer::copy_from_aligned(impl core::convert::AsRef<[T]>, vortex_buffer::Alignment) -> Self pub fn vortex_buffer::Buffer::empty() -> Self -pub fn vortex_buffer::Buffer::empty_aligned(alignment: vortex_buffer::Alignment) -> Self +pub fn vortex_buffer::Buffer::empty_aligned(vortex_buffer::Alignment) -> Self -pub fn vortex_buffer::Buffer::ensure_aligned(self, alignment: vortex_buffer::Alignment) -> Self +pub fn vortex_buffer::Buffer::ensure_aligned(self, vortex_buffer::Alignment) -> Self -pub fn vortex_buffer::Buffer::from_byte_buffer(buffer: vortex_buffer::ByteBuffer) -> Self +pub fn vortex_buffer::Buffer::from_byte_buffer(vortex_buffer::ByteBuffer) -> Self -pub fn vortex_buffer::Buffer::from_byte_buffer_aligned(buffer: vortex_buffer::ByteBuffer, alignment: vortex_buffer::Alignment) -> Self +pub fn vortex_buffer::Buffer::from_byte_buffer_aligned(vortex_buffer::ByteBuffer, vortex_buffer::Alignment) -> Self -pub fn vortex_buffer::Buffer::from_bytes_aligned(bytes: bytes::bytes::Bytes, alignment: vortex_buffer::Alignment) -> Self +pub fn vortex_buffer::Buffer::from_bytes_aligned(bytes::bytes::Bytes, vortex_buffer::Alignment) -> Self -pub fn vortex_buffer::Buffer::from_trusted_len_iter>(iter: I) -> Self +pub fn vortex_buffer::Buffer::from_trusted_len_iter>(I) -> Self -pub fn vortex_buffer::Buffer::full(item: T, len: usize) -> Self where T: core::marker::Copy +pub fn vortex_buffer::Buffer::full(T, usize) -> Self where T: core::marker::Copy pub fn vortex_buffer::Buffer::inner(&self) -> &bytes::bytes::Bytes @@ -598,7 +586,7 @@ pub fn vortex_buffer::Buffer::into_inner(self) -> bytes::bytes::Bytes pub fn vortex_buffer::Buffer::into_mut(self) -> vortex_buffer::BufferMut -pub fn vortex_buffer::Buffer::is_aligned(&self, alignment: vortex_buffer::Alignment) -> bool +pub fn vortex_buffer::Buffer::is_aligned(&self, vortex_buffer::Alignment) -> bool pub fn vortex_buffer::Buffer::is_empty(&self) -> bool @@ -606,23 +594,23 @@ pub fn vortex_buffer::Buffer::iter(&self) -> vortex_buffer::Iter<'_, T> pub fn vortex_buffer::Buffer::len(&self) -> usize -pub fn vortex_buffer::Buffer::map_each_in_place(self, f: F) -> vortex_buffer::BufferMut where T: core::marker::Copy, F: core::ops::function::FnMut(T) -> R +pub fn vortex_buffer::Buffer::map_each_in_place(self, F) -> vortex_buffer::BufferMut where T: core::marker::Copy, F: core::ops::function::FnMut(T) -> R -pub fn vortex_buffer::Buffer::slice(&self, range: impl core::ops::range::RangeBounds) -> Self +pub fn vortex_buffer::Buffer::slice(&self, impl core::ops::range::RangeBounds) -> Self -pub fn vortex_buffer::Buffer::slice_ref(&self, subset: &[T]) -> Self +pub fn vortex_buffer::Buffer::slice_ref(&self, &[T]) -> Self -pub fn vortex_buffer::Buffer::slice_ref_with_alignment(&self, subset: &[T], alignment: vortex_buffer::Alignment) -> Self +pub fn vortex_buffer::Buffer::slice_ref_with_alignment(&self, &[T], vortex_buffer::Alignment) -> Self -pub fn vortex_buffer::Buffer::slice_unaligned(&self, range: impl core::ops::range::RangeBounds) -> Self +pub fn vortex_buffer::Buffer::slice_unaligned(&self, impl core::ops::range::RangeBounds) -> Self -pub fn vortex_buffer::Buffer::slice_with_alignment(&self, range: impl core::ops::range::RangeBounds, alignment: vortex_buffer::Alignment) -> Self +pub fn vortex_buffer::Buffer::slice_with_alignment(&self, impl core::ops::range::RangeBounds, vortex_buffer::Alignment) -> Self pub fn vortex_buffer::Buffer::try_into_mut(self) -> core::result::Result, Self> -pub fn vortex_buffer::Buffer::zeroed(len: usize) -> Self +pub fn vortex_buffer::Buffer::zeroed(usize) -> Self -pub fn vortex_buffer::Buffer::zeroed_aligned(len: usize, alignment: vortex_buffer::Alignment) -> Self +pub fn vortex_buffer::Buffer::zeroed_aligned(usize, vortex_buffer::Alignment) -> Self impl vortex_buffer::Buffer @@ -632,7 +620,7 @@ impl core::convert::TryFrom> for vortex_buffer::Buffer pub type vortex_buffer::BufferString::Error = vortex_error::VortexError -pub fn vortex_buffer::BufferString::try_from(value: vortex_buffer::ByteBuffer) -> core::result::Result +pub fn vortex_buffer::BufferString::try_from(vortex_buffer::ByteBuffer) -> core::result::Result impl core::convert::AsRef> for vortex_buffer::ConstBuffer @@ -642,19 +630,19 @@ impl core::convert::TryFrom> for vor pub type vortex_buffer::ConstBuffer::Error = vortex_error::VortexError -pub fn vortex_buffer::ConstBuffer::try_from(value: vortex_buffer::Buffer) -> core::result::Result +pub fn vortex_buffer::ConstBuffer::try_from(vortex_buffer::Buffer) -> core::result::Result impl core::cmp::PartialEq> for vortex_buffer::Buffer -pub fn vortex_buffer::Buffer::eq(&self, other: &alloc::vec::Vec) -> bool +pub fn vortex_buffer::Buffer::eq(&self, &alloc::vec::Vec) -> bool impl core::cmp::PartialEq> for alloc::vec::Vec -pub fn alloc::vec::Vec::eq(&self, other: &vortex_buffer::Buffer) -> bool +pub fn alloc::vec::Vec::eq(&self, &vortex_buffer::Buffer) -> bool impl core::fmt::Debug for vortex_buffer::Buffer -pub fn vortex_buffer::Buffer::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_buffer::Buffer::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::iter::traits::collect::IntoIterator for vortex_buffer::Buffer @@ -672,15 +660,15 @@ impl core::cmp::Eq for vortex_buffer::Buffer impl core::cmp::Ord for vortex_buffer::Buffer -pub fn vortex_buffer::Buffer::cmp(&self, other: &Self) -> core::cmp::Ordering +pub fn vortex_buffer::Buffer::cmp(&self, &Self) -> core::cmp::Ordering impl core::cmp::PartialEq for vortex_buffer::Buffer -pub fn vortex_buffer::Buffer::eq(&self, other: &Self) -> bool +pub fn vortex_buffer::Buffer::eq(&self, &Self) -> bool impl core::cmp::PartialOrd for vortex_buffer::Buffer -pub fn vortex_buffer::Buffer::partial_cmp(&self, other: &Self) -> core::option::Option +pub fn vortex_buffer::Buffer::partial_cmp(&self, &Self) -> core::option::Option impl core::convert::AsRef<[T]> for vortex_buffer::Buffer @@ -688,11 +676,11 @@ pub fn vortex_buffer::Buffer::as_ref(&self) -> &[T] impl core::convert::From> for vortex_buffer::Buffer where T: core::marker::Send + 'static -pub fn vortex_buffer::Buffer::from(value: alloc::vec::Vec) -> Self +pub fn vortex_buffer::Buffer::from(alloc::vec::Vec) -> Self impl core::convert::From> for vortex_buffer::Buffer -pub fn vortex_buffer::Buffer::from(value: vortex_buffer::BufferMut) -> Self +pub fn vortex_buffer::Buffer::from(vortex_buffer::BufferMut) -> Self impl core::default::Default for vortex_buffer::Buffer @@ -700,11 +688,11 @@ pub fn vortex_buffer::Buffer::default() -> Self impl core::hash::Hash for vortex_buffer::Buffer -pub fn vortex_buffer::Buffer::hash(&self, state: &mut H) +pub fn vortex_buffer::Buffer::hash(&self, &mut H) impl core::iter::traits::collect::FromIterator for vortex_buffer::Buffer -pub fn vortex_buffer::Buffer::from_iter>(iter: I) -> Self +pub fn vortex_buffer::Buffer::from_iter>(I) -> Self impl core::ops::deref::Deref for vortex_buffer::Buffer @@ -730,7 +718,7 @@ pub struct vortex_buffer::BufferMut impl vortex_buffer::BufferMut -pub fn vortex_buffer::BufferMut::aligned(self, alignment: vortex_buffer::Alignment) -> Self +pub fn vortex_buffer::BufferMut::aligned(self, vortex_buffer::Alignment) -> Self pub fn vortex_buffer::BufferMut::alignment(&self) -> vortex_buffer::Alignment @@ -742,19 +730,19 @@ pub fn vortex_buffer::BufferMut::capacity(&self) -> usize pub fn vortex_buffer::BufferMut::clear(&mut self) -pub fn vortex_buffer::BufferMut::copy_from(other: impl core::convert::AsRef<[T]>) -> Self +pub fn vortex_buffer::BufferMut::copy_from(impl core::convert::AsRef<[T]>) -> Self -pub fn vortex_buffer::BufferMut::copy_from_aligned(other: impl core::convert::AsRef<[T]>, alignment: vortex_buffer::Alignment) -> Self +pub fn vortex_buffer::BufferMut::copy_from_aligned(impl core::convert::AsRef<[T]>, vortex_buffer::Alignment) -> Self pub fn vortex_buffer::BufferMut::empty() -> Self -pub fn vortex_buffer::BufferMut::empty_aligned(alignment: vortex_buffer::Alignment) -> Self +pub fn vortex_buffer::BufferMut::empty_aligned(vortex_buffer::Alignment) -> Self -pub fn vortex_buffer::BufferMut::extend_from_slice(&mut self, slice: &[T]) +pub fn vortex_buffer::BufferMut::extend_from_slice(&mut self, &[T]) pub fn vortex_buffer::BufferMut::freeze(self) -> vortex_buffer::Buffer -pub fn vortex_buffer::BufferMut::full(item: T, len: usize) -> Self where T: core::marker::Copy +pub fn vortex_buffer::BufferMut::full(T, usize) -> Self where T: core::marker::Copy pub fn vortex_buffer::BufferMut::into_byte_buffer(self) -> vortex_buffer::ByteBufferMut @@ -762,57 +750,57 @@ pub fn vortex_buffer::BufferMut::is_empty(&self) -> bool pub fn vortex_buffer::BufferMut::len(&self) -> usize -pub fn vortex_buffer::BufferMut::map_each_in_place(self, f: F) -> vortex_buffer::BufferMut where T: core::marker::Copy, F: core::ops::function::FnMut(T) -> R +pub fn vortex_buffer::BufferMut::map_each_in_place(self, F) -> vortex_buffer::BufferMut where T: core::marker::Copy, F: core::ops::function::FnMut(T) -> R -pub fn vortex_buffer::BufferMut::push(&mut self, value: T) +pub fn vortex_buffer::BufferMut::push(&mut self, T) -pub fn vortex_buffer::BufferMut::push_n(&mut self, item: T, n: usize) where T: core::marker::Copy +pub fn vortex_buffer::BufferMut::push_n(&mut self, T, usize) where T: core::marker::Copy -pub unsafe fn vortex_buffer::BufferMut::push_n_unchecked(&mut self, item: T, n: usize) where T: core::marker::Copy +pub unsafe fn vortex_buffer::BufferMut::push_n_unchecked(&mut self, T, usize) where T: core::marker::Copy -pub unsafe fn vortex_buffer::BufferMut::push_unchecked(&mut self, item: T) +pub unsafe fn vortex_buffer::BufferMut::push_unchecked(&mut self, T) -pub fn vortex_buffer::BufferMut::reserve(&mut self, additional: usize) +pub fn vortex_buffer::BufferMut::reserve(&mut self, usize) -pub unsafe fn vortex_buffer::BufferMut::set_len(&mut self, len: usize) +pub unsafe fn vortex_buffer::BufferMut::set_len(&mut self, usize) pub fn vortex_buffer::BufferMut::spare_capacity_mut(&mut self) -> &mut [core::mem::maybe_uninit::MaybeUninit] -pub fn vortex_buffer::BufferMut::split_off(&mut self, at: usize) -> Self +pub fn vortex_buffer::BufferMut::split_off(&mut self, usize) -> Self pub unsafe fn vortex_buffer::BufferMut::transmute(self) -> vortex_buffer::BufferMut -pub fn vortex_buffer::BufferMut::truncate(&mut self, len: usize) +pub fn vortex_buffer::BufferMut::truncate(&mut self, usize) -pub fn vortex_buffer::BufferMut::unsplit(&mut self, other: Self) +pub fn vortex_buffer::BufferMut::unsplit(&mut self, Self) -pub fn vortex_buffer::BufferMut::with_capacity(capacity: usize) -> Self +pub fn vortex_buffer::BufferMut::with_capacity(usize) -> Self -pub fn vortex_buffer::BufferMut::with_capacity_aligned(capacity: usize, alignment: vortex_buffer::Alignment) -> Self +pub fn vortex_buffer::BufferMut::with_capacity_aligned(usize, vortex_buffer::Alignment) -> Self -pub fn vortex_buffer::BufferMut::zeroed(len: usize) -> Self +pub fn vortex_buffer::BufferMut::zeroed(usize) -> Self -pub fn vortex_buffer::BufferMut::zeroed_aligned(len: usize, alignment: vortex_buffer::Alignment) -> Self +pub fn vortex_buffer::BufferMut::zeroed_aligned(usize, vortex_buffer::Alignment) -> Self impl vortex_buffer::BufferMut -pub fn vortex_buffer::BufferMut::extend_trusted>(&mut self, iter: I) +pub fn vortex_buffer::BufferMut::extend_trusted>(&mut self, I) -pub fn vortex_buffer::BufferMut::from_trusted_len_iter(iter: I) -> Self where I: vortex_buffer::trusted_len::TrustedLen +pub fn vortex_buffer::BufferMut::from_trusted_len_iter(I) -> Self where I: vortex_buffer::trusted_len::TrustedLen impl<'a, T> core::iter::traits::collect::Extend<&'a T> for vortex_buffer::BufferMut where T: core::marker::Copy + 'a -pub fn vortex_buffer::BufferMut::extend>(&mut self, iter: I) +pub fn vortex_buffer::BufferMut::extend>(&mut self, I) impl core::cmp::Eq for vortex_buffer::BufferMut impl core::cmp::PartialEq for vortex_buffer::BufferMut -pub fn vortex_buffer::BufferMut::eq(&self, other: &vortex_buffer::BufferMut) -> bool +pub fn vortex_buffer::BufferMut::eq(&self, &vortex_buffer::BufferMut) -> bool impl core::fmt::Debug for vortex_buffer::BufferMut -pub fn vortex_buffer::BufferMut::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_buffer::BufferMut::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::clone::Clone for vortex_buffer::BufferMut @@ -828,7 +816,7 @@ pub fn vortex_buffer::BufferMut::as_ref(&self) -> &[T] impl core::convert::From> for vortex_buffer::Buffer -pub fn vortex_buffer::Buffer::from(value: vortex_buffer::BufferMut) -> Self +pub fn vortex_buffer::Buffer::from(vortex_buffer::BufferMut) -> Self impl core::default::Default for vortex_buffer::BufferMut @@ -836,11 +824,11 @@ pub fn vortex_buffer::BufferMut::default() -> Self impl core::iter::traits::collect::Extend for vortex_buffer::BufferMut -pub fn vortex_buffer::BufferMut::extend>(&mut self, iter: I) +pub fn vortex_buffer::BufferMut::extend>(&mut self, I) impl core::iter::traits::collect::FromIterator for vortex_buffer::BufferMut -pub fn vortex_buffer::BufferMut::from_iter>(iter: I) -> Self +pub fn vortex_buffer::BufferMut::from_iter>(I) -> Self impl core::marker::StructuralPartialEq for vortex_buffer::BufferMut @@ -866,7 +854,7 @@ pub fn vortex_buffer::BufferString::inner(&self) -> &vortex_buffer::ByteBuffer pub fn vortex_buffer::BufferString::into_inner(self) -> vortex_buffer::ByteBuffer -pub unsafe const fn vortex_buffer::BufferString::new_unchecked(buffer: vortex_buffer::ByteBuffer) -> Self +pub unsafe const fn vortex_buffer::BufferString::new_unchecked(vortex_buffer::ByteBuffer) -> Self impl core::clone::Clone for vortex_buffer::BufferString @@ -876,15 +864,15 @@ impl core::cmp::Eq for vortex_buffer::BufferString impl core::cmp::Ord for vortex_buffer::BufferString -pub fn vortex_buffer::BufferString::cmp(&self, other: &vortex_buffer::BufferString) -> core::cmp::Ordering +pub fn vortex_buffer::BufferString::cmp(&self, &vortex_buffer::BufferString) -> core::cmp::Ordering impl core::cmp::PartialEq for vortex_buffer::BufferString -pub fn vortex_buffer::BufferString::eq(&self, other: &vortex_buffer::BufferString) -> bool +pub fn vortex_buffer::BufferString::eq(&self, &vortex_buffer::BufferString) -> bool impl core::cmp::PartialOrd for vortex_buffer::BufferString -pub fn vortex_buffer::BufferString::partial_cmp(&self, other: &vortex_buffer::BufferString) -> core::option::Option +pub fn vortex_buffer::BufferString::partial_cmp(&self, &vortex_buffer::BufferString) -> core::option::Option impl core::convert::AsRef<[u8]> for vortex_buffer::BufferString @@ -896,35 +884,35 @@ pub fn vortex_buffer::BufferString::as_ref(&self) -> &str impl core::convert::From<&str> for vortex_buffer::BufferString -pub fn vortex_buffer::BufferString::from(value: &str) -> Self +pub fn vortex_buffer::BufferString::from(&str) -> Self impl core::convert::From for vortex_buffer::BufferString -pub fn vortex_buffer::BufferString::from(value: alloc::string::String) -> Self +pub fn vortex_buffer::BufferString::from(alloc::string::String) -> Self impl core::convert::From for vortex_buffer::ByteBuffer -pub fn vortex_buffer::ByteBuffer::from(value: vortex_buffer::BufferString) -> Self +pub fn vortex_buffer::ByteBuffer::from(vortex_buffer::BufferString) -> Self impl core::convert::TryFrom<&[u8]> for vortex_buffer::BufferString pub type vortex_buffer::BufferString::Error = vortex_error::VortexError -pub fn vortex_buffer::BufferString::try_from(value: &[u8]) -> core::result::Result +pub fn vortex_buffer::BufferString::try_from(&[u8]) -> core::result::Result impl core::convert::TryFrom> for vortex_buffer::BufferString pub type vortex_buffer::BufferString::Error = vortex_error::VortexError -pub fn vortex_buffer::BufferString::try_from(value: vortex_buffer::ByteBuffer) -> core::result::Result +pub fn vortex_buffer::BufferString::try_from(vortex_buffer::ByteBuffer) -> core::result::Result impl core::fmt::Debug for vortex_buffer::BufferString -pub fn vortex_buffer::BufferString::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_buffer::BufferString::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_buffer::BufferString -pub fn vortex_buffer::BufferString::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_buffer::BufferString::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_buffer::BufferString @@ -938,13 +926,13 @@ pub struct vortex_buffer::ConstBuffer(_) impl vortex_buffer::ConstBuffer -pub fn vortex_buffer::ConstBuffer::align_from>>(buf: B) -> Self +pub fn vortex_buffer::ConstBuffer::align_from>>(B) -> Self pub const fn vortex_buffer::ConstBuffer::alignment() -> vortex_buffer::Alignment pub fn vortex_buffer::ConstBuffer::as_slice(&self) -> &[T] -pub fn vortex_buffer::ConstBuffer::copy_from>(buf: B) -> Self +pub fn vortex_buffer::ConstBuffer::copy_from>(B) -> Self pub fn vortex_buffer::ConstBuffer::inner(&self) -> &vortex_buffer::Buffer @@ -958,7 +946,7 @@ impl core::convert::TryFrom> for vor pub type vortex_buffer::ConstBuffer::Error = vortex_error::VortexError -pub fn vortex_buffer::ConstBuffer::try_from(value: vortex_buffer::Buffer) -> core::result::Result +pub fn vortex_buffer::ConstBuffer::try_from(vortex_buffer::Buffer) -> core::result::Result impl core::marker::StructuralPartialEq for vortex_buffer::ConstBuffer @@ -976,19 +964,19 @@ impl core::cmp::Eq for vortex_buffer::ConstBuf impl core::cmp::PartialEq for vortex_buffer::ConstBuffer -pub fn vortex_buffer::ConstBuffer::eq(&self, other: &vortex_buffer::ConstBuffer) -> bool +pub fn vortex_buffer::ConstBuffer::eq(&self, &vortex_buffer::ConstBuffer) -> bool impl core::cmp::PartialOrd for vortex_buffer::ConstBuffer -pub fn vortex_buffer::ConstBuffer::partial_cmp(&self, other: &vortex_buffer::ConstBuffer) -> core::option::Option +pub fn vortex_buffer::ConstBuffer::partial_cmp(&self, &vortex_buffer::ConstBuffer) -> core::option::Option impl core::fmt::Debug for vortex_buffer::ConstBuffer -pub fn vortex_buffer::ConstBuffer::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_buffer::ConstBuffer::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_buffer::ConstBuffer -pub fn vortex_buffer::ConstBuffer::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_buffer::ConstBuffer::hash<__H: core::hash::Hasher>(&self, &mut __H) pub struct vortex_buffer::Iter<'a, T> @@ -1002,7 +990,7 @@ pub fn vortex_buffer::Iter<'a, T>::last(self) -> core::option::Option::next(&mut self) -> core::option::Option -pub fn vortex_buffer::Iter<'a, T>::nth(&mut self, n: usize) -> core::option::Option +pub fn vortex_buffer::Iter<'a, T>::nth(&mut self, usize) -> core::option::Option pub fn vortex_buffer::Iter<'a, T>::size_hint(&self) -> (usize, core::option::Option) @@ -1016,23 +1004,23 @@ pub const vortex_buffer::ALIGNMENT_TO_HOST_COPY: vortex_buffer::Alignment pub trait vortex_buffer::AlignedBuf: bytes::buf::buf_impl::Buf -pub fn vortex_buffer::AlignedBuf::copy_to_aligned(&mut self, len: usize, alignment: vortex_buffer::Alignment) -> vortex_buffer::ByteBuffer +pub fn vortex_buffer::AlignedBuf::copy_to_aligned(&mut self, usize, vortex_buffer::Alignment) -> vortex_buffer::ByteBuffer -pub fn vortex_buffer::AlignedBuf::copy_to_const_aligned(&mut self, len: usize) -> vortex_buffer::ConstByteBuffer +pub fn vortex_buffer::AlignedBuf::copy_to_const_aligned(&mut self, usize) -> vortex_buffer::ConstByteBuffer impl vortex_buffer::AlignedBuf for B -pub fn B::copy_to_aligned(&mut self, len: usize, alignment: vortex_buffer::Alignment) -> vortex_buffer::ByteBuffer +pub fn B::copy_to_aligned(&mut self, usize, vortex_buffer::Alignment) -> vortex_buffer::ByteBuffer -pub fn B::copy_to_const_aligned(&mut self, len: usize) -> vortex_buffer::ConstByteBuffer +pub fn B::copy_to_const_aligned(&mut self, usize) -> vortex_buffer::ConstByteBuffer -pub fn vortex_buffer::get_bit(buf: &[u8], index: usize) -> bool +pub fn vortex_buffer::get_bit(&[u8], usize) -> bool -pub unsafe fn vortex_buffer::get_bit_unchecked(buf: *const u8, index: usize) -> bool +pub unsafe fn vortex_buffer::get_bit_unchecked(*const u8, usize) -> bool -pub unsafe fn vortex_buffer::set_bit_unchecked(buf: *mut u8, index: usize) +pub unsafe fn vortex_buffer::set_bit_unchecked(*mut u8, usize) -pub unsafe fn vortex_buffer::unset_bit_unchecked(buf: *mut u8, index: usize) +pub unsafe fn vortex_buffer::unset_bit_unchecked(*mut u8, usize) pub type vortex_buffer::ByteBuffer = vortex_buffer::Buffer diff --git a/vortex-buffer/src/bit/buf_mut.rs b/vortex-buffer/src/bit/buf_mut.rs index e0e2f85877d..bf42e92d571 100644 --- a/vortex-buffer/src/bit/buf_mut.rs +++ b/vortex-buffer/src/bit/buf_mut.rs @@ -3,8 +3,6 @@ use std::ops::Not; -use arrow_buffer::bit_chunk_iterator::BitChunks; -use arrow_buffer::bit_chunk_iterator::UnalignedBitChunk; use bitvec::view::BitView; use crate::BitBuffer; @@ -90,7 +88,7 @@ fn fill_bits(slice: &mut [u8], start_bit: usize, end_bit: usize, value: bool) { /// ``` /// /// See also: [`BitBuffer`]. -#[derive(Debug, Clone, Eq)] +#[derive(Debug, Clone)] pub struct BitBufferMut { buffer: ByteBufferMut, /// Represents the offset of the bit buffer into the first byte. @@ -100,19 +98,6 @@ pub struct BitBufferMut { len: usize, } -impl PartialEq for BitBufferMut { - fn eq(&self, other: &Self) -> bool { - if self.len != other.len { - return false; - } - - self.chunks() - .iter_padded() - .zip(other.chunks().iter_padded()) - .all(|(a, b)| a == b) - } -} - impl BitBufferMut { /// Create new bit buffer from given byte buffer and logical bit length pub fn from_buffer(buffer: ByteBufferMut, offset: usize, len: usize) -> Self { @@ -271,13 +256,6 @@ impl BitBufferMut { unsafe { get_bit_unchecked(self.buffer.as_ptr(), self.offset + index) } } - /// Access chunks of the underlying buffer as 8 byte chunks with a final trailer - /// - /// If you're performing operations on a single buffer, prefer [BitBuffer::unaligned_chunks] - pub fn chunks(&self) -> BitChunks<'_> { - BitChunks::new(self.buffer.as_slice(), self.offset, self.len) - } - /// Get the bit capacity of the buffer. #[inline(always)] pub fn capacity(&self) -> usize { @@ -354,7 +332,8 @@ impl BitBufferMut { /// # Safety /// /// The caller must ensure that `index` does not exceed the largest bit index in the backing buffer. - unsafe fn set_unchecked(&mut self, index: usize) { + #[inline] + pub unsafe fn set_unchecked(&mut self, index: usize) { // SAFETY: checked by caller unsafe { set_bit_unchecked(self.buffer.as_mut_ptr(), self.offset + index) } } @@ -536,52 +515,6 @@ impl BitBufferMut { self.len += bit_len; } - /// Splits the bit buffer into two at the given index. - /// - /// Afterward, self contains elements `[0, at)`, and the returned buffer contains elements - /// `[at, capacity)`. - /// - /// Unlike bytes, if the split position is not on a byte-boundary this operation will copy - /// data into the result type, and mutate self. - #[must_use = "consider BitBufferMut::truncate if you don't need the other half"] - pub fn split_off(&mut self, at: usize) -> Self { - assert!( - at <= self.capacity(), - "index {at} exceeds capacity {}", - self.capacity() - ); - - // The length of the tail is any bits after `at` - let tail_len = self.len.saturating_sub(at); - let byte_pos = (self.offset + at).div_ceil(8); - - // If we are splitting on a byte boundary, we can just slice the buffer - // Or if `at > self.len`, then the tail is empty anyway and we can just return as much - // of the existing capacity as possible. - if at > self.len() || (self.offset + at).is_multiple_of(8) { - let tail_buffer = self.buffer.split_off(byte_pos); - self.len = self.len.min(at); - - // Return the tail buffer - return Self { - buffer: tail_buffer, - offset: 0, - len: tail_len, - }; - } - - // Otherwise, we truncate ourselves, and copy any bits into a new tail buffer. - // Note that in this case we do not preserve the capacity. - let u64_cap = tail_len.div_ceil(8); - let mut tail_buffer_u64 = BufferMut::::with_capacity(u64_cap); - tail_buffer_u64.extend( - BitChunks::new(self.buffer.as_slice(), self.offset + at, tail_len).iter_padded(), - ); - - self.truncate(at); - BitBufferMut::from_buffer(tail_buffer_u64.into_byte_buffer(), 0, tail_len) - } - /// Absorbs a mutable buffer that was previously split off. /// /// If the two buffers were previously contiguous and not mutated in a way that causes @@ -615,26 +548,6 @@ impl BitBufferMut { pub fn as_mut_slice(&mut self) -> &mut [u8] { self.buffer.as_mut_slice() } - - /// Returns a raw mutable pointer to the internal buffer. - pub fn as_mut_ptr(&mut self) -> *mut u8 { - self.buffer.as_mut_ptr() - } - - /// Access chunks of the buffer aligned to 8 byte boundary as [prefix, \, suffix] - pub fn unaligned_chunks(&self) -> UnalignedBitChunk<'_> { - UnalignedBitChunk::new(self.buffer.as_slice(), self.offset, self.len) - } - - /// Get the number of set bits in the buffer. - pub fn true_count(&self) -> usize { - self.unaligned_chunks().count_ones() - } - - /// Get the number of unset bits in the buffer. - pub fn false_count(&self) -> usize { - self.len - self.true_count() - } } impl Default for BitBufferMut { @@ -660,6 +573,13 @@ impl From<&[bool]> for BitBufferMut { } } +// allow building a buffer from a set of truthy byte values. +impl From<&[u8]> for BitBufferMut { + fn from(value: &[u8]) -> Self { + BitBufferMut::collect_bool(value.len(), |i| value[i] > 0) + } +} + impl From> for BitBufferMut { fn from(value: Vec) -> Self { value.as_slice().into() @@ -1176,40 +1096,4 @@ mod tests { assert_eq!(bit_buf.value(i), i % 2 == 0); } } - - #[test] - fn test_split_off() { - // Test splitting at various positions and across a byte boundary - for i in 0..10 { - let buf = bitbuffer![0 1 0 1 0 1 0 1 0 1]; - - let mut buf_mut = BitBufferMut::copy_from(&buf); - assert_eq!(buf_mut.len(), 10); - - let tail = buf_mut.split_off(i); - assert_eq!(buf_mut.len(), i); - assert_eq!(buf_mut.freeze(), buf.slice(0..i)); - - assert_eq!(tail.len(), 10 - i); - assert_eq!(tail.freeze(), buf.slice(i..10)); - } - } - - #[test] - fn test_split_off_with_offset() { - // Test splitting at various positions and across a byte boundary - for i in 0..10 { - let buf = bitbuffer![0 1 0 1 0 1 0 1 0 1 0 1].slice(2..); - - let mut buf_mut = BitBufferMut::copy_from(&buf); - assert_eq!(buf_mut.len(), 10); - - let tail = buf_mut.split_off(i); - assert_eq!(buf_mut.len(), i); - assert_eq!(buf_mut.freeze(), buf.slice(0..i)); - - assert_eq!(tail.len(), 10 - i); - assert_eq!(tail.freeze(), buf.slice(i..10)); - } - } } diff --git a/vortex-buffer/src/lib.rs b/vortex-buffer/src/lib.rs index 8f6d0cf24c3..8319fffa387 100644 --- a/vortex-buffer/src/lib.rs +++ b/vortex-buffer/src/lib.rs @@ -2,8 +2,6 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors #![deny(missing_docs)] -// cudarc HostSlice has len and is_empty methods that duplicate BufferMut methods. -#![allow(clippy::same_name_method)] //! A library for working with custom aligned buffers of sized values. //! diff --git a/vortex-compressor/Cargo.toml b/vortex-compressor/Cargo.toml index da9bd07889c..2d28d86a5e6 100644 --- a/vortex-compressor/Cargo.toml +++ b/vortex-compressor/Cargo.toml @@ -19,7 +19,7 @@ num-traits = { workspace = true } parking_lot = { workspace = true } rand = { workspace = true } rustc-hash = { workspace = true } -tracing = { workspace = true } +tracing = { workspace = true, features = ["std", "attributes"] } vortex-array = { workspace = true } vortex-buffer = { workspace = true } vortex-error = { workspace = true } @@ -29,7 +29,9 @@ vortex-utils = { workspace = true } [dev-dependencies] divan = { workspace = true } rstest = { workspace = true } +tracing-subscriber = { workspace = true, features = ["env-filter"] } vortex-array = { workspace = true, features = ["_test-harness"] } +vortex-session = { workspace = true } [lints] workspace = true diff --git a/vortex-compressor/benches/dict_encode.rs b/vortex-compressor/benches/dict_encode.rs index 1c47e951fb8..02eaa906cb1 100644 --- a/vortex-compressor/benches/dict_encode.rs +++ b/vortex-compressor/benches/dict_encode.rs @@ -1,10 +1,12 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] +#![expect(clippy::unwrap_used)] use divan::Bencher; use vortex_array::IntoArray; +use vortex_array::LEGACY_SESSION; +use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; use vortex_array::arrays::PrimitiveArray; use vortex_array::builders::dict::dict_encode; @@ -40,10 +42,10 @@ fn encode_generic(bencher: Bencher) { #[divan::bench] fn encode_specialized(bencher: Bencher) { let array = make_array(); - let stats = IntegerStats::generate(&array); + let stats = IntegerStats::generate(&array, &mut LEGACY_SESSION.create_execution_ctx()); bencher .with_inputs(|| &stats) - .bench_refs(|stats| integer_dictionary_encode(array.clone(), stats)); + .bench_refs(|stats| integer_dictionary_encode(array.as_view(), stats)); } fn main() { diff --git a/vortex-compressor/benches/stats_calc.rs b/vortex-compressor/benches/stats_calc.rs index 5675c8de434..c93a8373801 100644 --- a/vortex-compressor/benches/stats_calc.rs +++ b/vortex-compressor/benches/stats_calc.rs @@ -1,13 +1,12 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::cast_possible_truncation, clippy::use_debug)] -#![allow(unexpected_cfgs)] - #[cfg(not(codspeed))] #[divan::bench_group(items_count = 64_000u32, bytes_count = 256_000u32)] mod benchmarks { use divan::Bencher; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::validity::Validity; use vortex_buffer::Buffer; @@ -56,9 +55,11 @@ mod benchmarks { Distribution::LongRuns => generate_runs(64), }; - bencher.with_inputs(|| &values).bench_refs(|values| { - IntegerStats::generate_opts(values, GenerateStatsOptions::default()); - }); + bencher + .with_inputs(|| (&values, LEGACY_SESSION.create_execution_ctx())) + .bench_refs(|(values, ctx)| { + IntegerStats::generate_opts(values, GenerateStatsOptions::default(), ctx); + }); } #[divan::bench(args = [Distribution::LowCardinality, Distribution::ShortRuns, Distribution::LongRuns])] @@ -69,14 +70,17 @@ mod benchmarks { Distribution::LongRuns => generate_runs(64), }; - bencher.with_inputs(|| &values).bench_refs(|values| { - IntegerStats::generate_opts( - values, - GenerateStatsOptions { - count_distinct_values: false, - }, - ); - }); + bencher + .with_inputs(|| (&values, LEGACY_SESSION.create_execution_ctx())) + .bench_refs(|(values, ctx)| { + IntegerStats::generate_opts( + values, + GenerateStatsOptions { + count_distinct_values: false, + }, + ctx, + ); + }); } } diff --git a/vortex-compressor/public-api.lock b/vortex-compressor/public-api.lock index 0f16ccc61ab..cbdc65377f0 100644 --- a/vortex-compressor/public-api.lock +++ b/vortex-compressor/public-api.lock @@ -12,11 +12,11 @@ impl core::cmp::Eq for vortex_compressor::builtins::BoolConstantScheme impl core::cmp::PartialEq for vortex_compressor::builtins::BoolConstantScheme -pub fn vortex_compressor::builtins::BoolConstantScheme::eq(&self, other: &vortex_compressor::builtins::BoolConstantScheme) -> bool +pub fn vortex_compressor::builtins::BoolConstantScheme::eq(&self, &vortex_compressor::builtins::BoolConstantScheme) -> bool impl core::fmt::Debug for vortex_compressor::builtins::BoolConstantScheme -pub fn vortex_compressor::builtins::BoolConstantScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_compressor::builtins::BoolConstantScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_compressor::builtins::BoolConstantScheme @@ -26,13 +26,13 @@ impl vortex_compressor::scheme::Scheme for vortex_compressor::builtins::BoolCons pub fn vortex_compressor::builtins::BoolConstantScheme::ancestor_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::BoolConstantScheme::compress(&self, _compressor: &vortex_compressor::CascadingCompressor, data: &mut vortex_compressor::stats::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_compressor::builtins::BoolConstantScheme::compress(&self, &vortex_compressor::CascadingCompressor, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_compressor::builtins::BoolConstantScheme::descendant_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::BoolConstantScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_compressor::builtins::BoolConstantScheme::expected_compression_ratio(&self, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_compressor::builtins::BoolConstantScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_compressor::builtins::BoolConstantScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_compressor::builtins::BoolConstantScheme::num_children(&self) -> usize @@ -50,11 +50,11 @@ impl core::cmp::Eq for vortex_compressor::builtins::FloatConstantScheme impl core::cmp::PartialEq for vortex_compressor::builtins::FloatConstantScheme -pub fn vortex_compressor::builtins::FloatConstantScheme::eq(&self, other: &vortex_compressor::builtins::FloatConstantScheme) -> bool +pub fn vortex_compressor::builtins::FloatConstantScheme::eq(&self, &vortex_compressor::builtins::FloatConstantScheme) -> bool impl core::fmt::Debug for vortex_compressor::builtins::FloatConstantScheme -pub fn vortex_compressor::builtins::FloatConstantScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_compressor::builtins::FloatConstantScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_compressor::builtins::FloatConstantScheme @@ -64,13 +64,13 @@ impl vortex_compressor::scheme::Scheme for vortex_compressor::builtins::FloatCon pub fn vortex_compressor::builtins::FloatConstantScheme::ancestor_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::FloatConstantScheme::compress(&self, _compressor: &vortex_compressor::CascadingCompressor, data: &mut vortex_compressor::stats::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_compressor::builtins::FloatConstantScheme::compress(&self, &vortex_compressor::CascadingCompressor, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_compressor::builtins::FloatConstantScheme::descendant_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::FloatConstantScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_compressor::builtins::FloatConstantScheme::expected_compression_ratio(&self, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_compressor::builtins::FloatConstantScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_compressor::builtins::FloatConstantScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_compressor::builtins::FloatConstantScheme::num_children(&self) -> usize @@ -88,11 +88,11 @@ impl core::cmp::Eq for vortex_compressor::builtins::FloatDictScheme impl core::cmp::PartialEq for vortex_compressor::builtins::FloatDictScheme -pub fn vortex_compressor::builtins::FloatDictScheme::eq(&self, other: &vortex_compressor::builtins::FloatDictScheme) -> bool +pub fn vortex_compressor::builtins::FloatDictScheme::eq(&self, &vortex_compressor::builtins::FloatDictScheme) -> bool impl core::fmt::Debug for vortex_compressor::builtins::FloatDictScheme -pub fn vortex_compressor::builtins::FloatDictScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_compressor::builtins::FloatDictScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_compressor::builtins::FloatDictScheme @@ -102,13 +102,13 @@ impl vortex_compressor::scheme::Scheme for vortex_compressor::builtins::FloatDic pub fn vortex_compressor::builtins::FloatDictScheme::ancestor_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::FloatDictScheme::compress(&self, compressor: &vortex_compressor::CascadingCompressor, data: &mut vortex_compressor::stats::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_compressor::builtins::FloatDictScheme::compress(&self, &vortex_compressor::CascadingCompressor, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_compressor::builtins::FloatDictScheme::descendant_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::FloatDictScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_compressor::builtins::FloatDictScheme::expected_compression_ratio(&self, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_compressor::builtins::FloatDictScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_compressor::builtins::FloatDictScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_compressor::builtins::FloatDictScheme::num_children(&self) -> usize @@ -126,11 +126,11 @@ impl core::cmp::Eq for vortex_compressor::builtins::IntConstantScheme impl core::cmp::PartialEq for vortex_compressor::builtins::IntConstantScheme -pub fn vortex_compressor::builtins::IntConstantScheme::eq(&self, other: &vortex_compressor::builtins::IntConstantScheme) -> bool +pub fn vortex_compressor::builtins::IntConstantScheme::eq(&self, &vortex_compressor::builtins::IntConstantScheme) -> bool impl core::fmt::Debug for vortex_compressor::builtins::IntConstantScheme -pub fn vortex_compressor::builtins::IntConstantScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_compressor::builtins::IntConstantScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_compressor::builtins::IntConstantScheme @@ -140,13 +140,13 @@ impl vortex_compressor::scheme::Scheme for vortex_compressor::builtins::IntConst pub fn vortex_compressor::builtins::IntConstantScheme::ancestor_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::IntConstantScheme::compress(&self, _compressor: &vortex_compressor::CascadingCompressor, data: &mut vortex_compressor::stats::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_compressor::builtins::IntConstantScheme::compress(&self, &vortex_compressor::CascadingCompressor, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_compressor::builtins::IntConstantScheme::descendant_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::IntConstantScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_compressor::builtins::IntConstantScheme::expected_compression_ratio(&self, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_compressor::builtins::IntConstantScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_compressor::builtins::IntConstantScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_compressor::builtins::IntConstantScheme::num_children(&self) -> usize @@ -164,11 +164,11 @@ impl core::cmp::Eq for vortex_compressor::builtins::IntDictScheme impl core::cmp::PartialEq for vortex_compressor::builtins::IntDictScheme -pub fn vortex_compressor::builtins::IntDictScheme::eq(&self, other: &vortex_compressor::builtins::IntDictScheme) -> bool +pub fn vortex_compressor::builtins::IntDictScheme::eq(&self, &vortex_compressor::builtins::IntDictScheme) -> bool impl core::fmt::Debug for vortex_compressor::builtins::IntDictScheme -pub fn vortex_compressor::builtins::IntDictScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_compressor::builtins::IntDictScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_compressor::builtins::IntDictScheme @@ -178,13 +178,13 @@ impl vortex_compressor::scheme::Scheme for vortex_compressor::builtins::IntDictS pub fn vortex_compressor::builtins::IntDictScheme::ancestor_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::IntDictScheme::compress(&self, compressor: &vortex_compressor::CascadingCompressor, data: &mut vortex_compressor::stats::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_compressor::builtins::IntDictScheme::compress(&self, &vortex_compressor::CascadingCompressor, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_compressor::builtins::IntDictScheme::descendant_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::IntDictScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_compressor::builtins::IntDictScheme::expected_compression_ratio(&self, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_compressor::builtins::IntDictScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_compressor::builtins::IntDictScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_compressor::builtins::IntDictScheme::num_children(&self) -> usize @@ -202,11 +202,11 @@ impl core::cmp::Eq for vortex_compressor::builtins::StringConstantScheme impl core::cmp::PartialEq for vortex_compressor::builtins::StringConstantScheme -pub fn vortex_compressor::builtins::StringConstantScheme::eq(&self, other: &vortex_compressor::builtins::StringConstantScheme) -> bool +pub fn vortex_compressor::builtins::StringConstantScheme::eq(&self, &vortex_compressor::builtins::StringConstantScheme) -> bool impl core::fmt::Debug for vortex_compressor::builtins::StringConstantScheme -pub fn vortex_compressor::builtins::StringConstantScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_compressor::builtins::StringConstantScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_compressor::builtins::StringConstantScheme @@ -216,13 +216,13 @@ impl vortex_compressor::scheme::Scheme for vortex_compressor::builtins::StringCo pub fn vortex_compressor::builtins::StringConstantScheme::ancestor_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::StringConstantScheme::compress(&self, _compressor: &vortex_compressor::CascadingCompressor, data: &mut vortex_compressor::stats::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_compressor::builtins::StringConstantScheme::compress(&self, &vortex_compressor::CascadingCompressor, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_compressor::builtins::StringConstantScheme::descendant_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::StringConstantScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_compressor::builtins::StringConstantScheme::expected_compression_ratio(&self, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_compressor::builtins::StringConstantScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_compressor::builtins::StringConstantScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_compressor::builtins::StringConstantScheme::num_children(&self) -> usize @@ -240,11 +240,11 @@ impl core::cmp::Eq for vortex_compressor::builtins::StringDictScheme impl core::cmp::PartialEq for vortex_compressor::builtins::StringDictScheme -pub fn vortex_compressor::builtins::StringDictScheme::eq(&self, other: &vortex_compressor::builtins::StringDictScheme) -> bool +pub fn vortex_compressor::builtins::StringDictScheme::eq(&self, &vortex_compressor::builtins::StringDictScheme) -> bool impl core::fmt::Debug for vortex_compressor::builtins::StringDictScheme -pub fn vortex_compressor::builtins::StringDictScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_compressor::builtins::StringDictScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_compressor::builtins::StringDictScheme @@ -254,13 +254,13 @@ impl vortex_compressor::scheme::Scheme for vortex_compressor::builtins::StringDi pub fn vortex_compressor::builtins::StringDictScheme::ancestor_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::StringDictScheme::compress(&self, compressor: &vortex_compressor::CascadingCompressor, data: &mut vortex_compressor::stats::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_compressor::builtins::StringDictScheme::compress(&self, &vortex_compressor::CascadingCompressor, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_compressor::builtins::StringDictScheme::descendant_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::StringDictScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_compressor::builtins::StringDictScheme::expected_compression_ratio(&self, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_compressor::builtins::StringDictScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_compressor::builtins::StringDictScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_compressor::builtins::StringDictScheme::num_children(&self) -> usize @@ -268,15 +268,15 @@ pub fn vortex_compressor::builtins::StringDictScheme::scheme_name(&self) -> &'st pub fn vortex_compressor::builtins::StringDictScheme::stats_options(&self) -> vortex_compressor::stats::GenerateStatsOptions -pub fn vortex_compressor::builtins::float_dictionary_encode(array: vortex_array::arrays::primitive::vtable::PrimitiveArray, stats: &vortex_compressor::stats::FloatStats) -> vortex_error::VortexResult +pub fn vortex_compressor::builtins::float_dictionary_encode(vortex_array::array::view::ArrayView<'_, vortex_array::arrays::primitive::vtable::Primitive>, &vortex_compressor::stats::FloatStats) -> vortex_error::VortexResult -pub fn vortex_compressor::builtins::integer_dictionary_encode(array: vortex_array::arrays::primitive::vtable::PrimitiveArray, stats: &vortex_compressor::stats::IntegerStats) -> vortex_error::VortexResult +pub fn vortex_compressor::builtins::integer_dictionary_encode(vortex_array::array::view::ArrayView<'_, vortex_array::arrays::primitive::vtable::Primitive>, &vortex_compressor::stats::IntegerStats) -> vortex_error::VortexResult -pub fn vortex_compressor::builtins::is_float_primitive(canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_compressor::builtins::is_float_primitive(&vortex_array::canonical::Canonical) -> bool -pub fn vortex_compressor::builtins::is_integer_primitive(canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_compressor::builtins::is_integer_primitive(&vortex_array::canonical::Canonical) -> bool -pub fn vortex_compressor::builtins::is_utf8_string(canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_compressor::builtins::is_utf8_string(&vortex_array::canonical::Canonical) -> bool pub mod vortex_compressor::ctx @@ -300,7 +300,7 @@ pub fn vortex_compressor::ctx::CompressorContext::clone(&self) -> vortex_compres impl core::fmt::Debug for vortex_compressor::ctx::CompressorContext -pub fn vortex_compressor::ctx::CompressorContext::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_compressor::ctx::CompressorContext::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub const vortex_compressor::ctx::MAX_CASCADE: usize @@ -308,21 +308,63 @@ pub mod vortex_compressor::estimate pub enum vortex_compressor::estimate::CompressionEstimate -pub vortex_compressor::estimate::CompressionEstimate::AlwaysUse +pub vortex_compressor::estimate::CompressionEstimate::Deferred(vortex_compressor::estimate::DeferredEstimate) -pub vortex_compressor::estimate::CompressionEstimate::Estimate(alloc::boxed::Box) +pub vortex_compressor::estimate::CompressionEstimate::Verdict(vortex_compressor::estimate::EstimateVerdict) -pub vortex_compressor::estimate::CompressionEstimate::Ratio(f64) +impl core::fmt::Debug for vortex_compressor::estimate::CompressionEstimate -pub vortex_compressor::estimate::CompressionEstimate::Sample +pub fn vortex_compressor::estimate::CompressionEstimate::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result -pub vortex_compressor::estimate::CompressionEstimate::Skip +pub enum vortex_compressor::estimate::DeferredEstimate -impl core::fmt::Debug for vortex_compressor::estimate::CompressionEstimate +pub vortex_compressor::estimate::DeferredEstimate::Callback(alloc::boxed::Box) + +pub vortex_compressor::estimate::DeferredEstimate::Sample + +impl core::fmt::Debug for vortex_compressor::estimate::DeferredEstimate + +pub fn vortex_compressor::estimate::DeferredEstimate::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +pub enum vortex_compressor::estimate::EstimateScore + +pub vortex_compressor::estimate::EstimateScore::FiniteCompression(f64) + +pub vortex_compressor::estimate::EstimateScore::ZeroBytes + +impl vortex_compressor::estimate::EstimateScore + +pub fn vortex_compressor::estimate::EstimateScore::finite_ratio(self) -> core::option::Option + +impl core::clone::Clone for vortex_compressor::estimate::EstimateScore + +pub fn vortex_compressor::estimate::EstimateScore::clone(&self) -> vortex_compressor::estimate::EstimateScore -pub fn vortex_compressor::estimate::CompressionEstimate::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +impl core::cmp::PartialEq for vortex_compressor::estimate::EstimateScore -pub type vortex_compressor::estimate::EstimateFn = (dyn core::ops::function::FnOnce(&vortex_compressor::CascadingCompressor, &mut vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult + core::marker::Send + core::marker::Sync) +pub fn vortex_compressor::estimate::EstimateScore::eq(&self, &vortex_compressor::estimate::EstimateScore) -> bool + +impl core::fmt::Debug for vortex_compressor::estimate::EstimateScore + +pub fn vortex_compressor::estimate::EstimateScore::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +impl core::marker::Copy for vortex_compressor::estimate::EstimateScore + +impl core::marker::StructuralPartialEq for vortex_compressor::estimate::EstimateScore + +pub enum vortex_compressor::estimate::EstimateVerdict + +pub vortex_compressor::estimate::EstimateVerdict::AlwaysUse + +pub vortex_compressor::estimate::EstimateVerdict::Ratio(f64) + +pub vortex_compressor::estimate::EstimateVerdict::Skip + +impl core::fmt::Debug for vortex_compressor::estimate::EstimateVerdict + +pub fn vortex_compressor::estimate::EstimateVerdict::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +pub type vortex_compressor::estimate::EstimateFn = (dyn core::ops::function::FnOnce(&vortex_compressor::CascadingCompressor, &vortex_compressor::stats::ArrayAndStats, core::option::Option, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult + core::marker::Send + core::marker::Sync) pub mod vortex_compressor::scheme @@ -336,7 +378,7 @@ pub vortex_compressor::scheme::ChildSelection::One(usize) impl vortex_compressor::scheme::ChildSelection -pub fn vortex_compressor::scheme::ChildSelection::contains(&self, child_index: usize) -> bool +pub fn vortex_compressor::scheme::ChildSelection::contains(&self, usize) -> bool impl core::clone::Clone for vortex_compressor::scheme::ChildSelection @@ -344,7 +386,7 @@ pub fn vortex_compressor::scheme::ChildSelection::clone(&self) -> vortex_compres impl core::fmt::Debug for vortex_compressor::scheme::ChildSelection -pub fn vortex_compressor::scheme::ChildSelection::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_compressor::scheme::ChildSelection::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_compressor::scheme::ChildSelection @@ -360,7 +402,7 @@ pub fn vortex_compressor::scheme::AncestorExclusion::clone(&self) -> vortex_comp impl core::fmt::Debug for vortex_compressor::scheme::AncestorExclusion -pub fn vortex_compressor::scheme::AncestorExclusion::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_compressor::scheme::AncestorExclusion::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_compressor::scheme::AncestorExclusion @@ -376,7 +418,7 @@ pub fn vortex_compressor::scheme::DescendantExclusion::clone(&self) -> vortex_co impl core::fmt::Debug for vortex_compressor::scheme::DescendantExclusion -pub fn vortex_compressor::scheme::DescendantExclusion::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_compressor::scheme::DescendantExclusion::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_compressor::scheme::DescendantExclusion @@ -390,19 +432,19 @@ impl core::cmp::Eq for vortex_compressor::scheme::SchemeId impl core::cmp::PartialEq for vortex_compressor::scheme::SchemeId -pub fn vortex_compressor::scheme::SchemeId::eq(&self, other: &vortex_compressor::scheme::SchemeId) -> bool +pub fn vortex_compressor::scheme::SchemeId::eq(&self, &vortex_compressor::scheme::SchemeId) -> bool impl core::fmt::Debug for vortex_compressor::scheme::SchemeId -pub fn vortex_compressor::scheme::SchemeId::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_compressor::scheme::SchemeId::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_compressor::scheme::SchemeId -pub fn vortex_compressor::scheme::SchemeId::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_compressor::scheme::SchemeId::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_compressor::scheme::SchemeId -pub fn vortex_compressor::scheme::SchemeId::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_compressor::scheme::SchemeId::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_compressor::scheme::SchemeId @@ -412,13 +454,13 @@ pub trait vortex_compressor::scheme::Scheme: core::fmt::Debug + core::marker::Se pub fn vortex_compressor::scheme::Scheme::ancestor_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::scheme::Scheme::compress(&self, compressor: &vortex_compressor::CascadingCompressor, data: &mut vortex_compressor::stats::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_compressor::scheme::Scheme::compress(&self, &vortex_compressor::CascadingCompressor, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_compressor::scheme::Scheme::descendant_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::scheme::Scheme::expected_compression_ratio(&self, _data: &mut vortex_compressor::stats::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_compressor::scheme::Scheme::expected_compression_ratio(&self, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_compressor::scheme::Scheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_compressor::scheme::Scheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_compressor::scheme::Scheme::num_children(&self) -> usize @@ -430,13 +472,13 @@ impl vortex_compressor::scheme::Scheme for vortex_compressor::builtins::BoolCons pub fn vortex_compressor::builtins::BoolConstantScheme::ancestor_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::BoolConstantScheme::compress(&self, _compressor: &vortex_compressor::CascadingCompressor, data: &mut vortex_compressor::stats::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_compressor::builtins::BoolConstantScheme::compress(&self, &vortex_compressor::CascadingCompressor, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_compressor::builtins::BoolConstantScheme::descendant_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::BoolConstantScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_compressor::builtins::BoolConstantScheme::expected_compression_ratio(&self, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_compressor::builtins::BoolConstantScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_compressor::builtins::BoolConstantScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_compressor::builtins::BoolConstantScheme::num_children(&self) -> usize @@ -448,13 +490,13 @@ impl vortex_compressor::scheme::Scheme for vortex_compressor::builtins::FloatCon pub fn vortex_compressor::builtins::FloatConstantScheme::ancestor_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::FloatConstantScheme::compress(&self, _compressor: &vortex_compressor::CascadingCompressor, data: &mut vortex_compressor::stats::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_compressor::builtins::FloatConstantScheme::compress(&self, &vortex_compressor::CascadingCompressor, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_compressor::builtins::FloatConstantScheme::descendant_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::FloatConstantScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_compressor::builtins::FloatConstantScheme::expected_compression_ratio(&self, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_compressor::builtins::FloatConstantScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_compressor::builtins::FloatConstantScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_compressor::builtins::FloatConstantScheme::num_children(&self) -> usize @@ -466,13 +508,13 @@ impl vortex_compressor::scheme::Scheme for vortex_compressor::builtins::FloatDic pub fn vortex_compressor::builtins::FloatDictScheme::ancestor_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::FloatDictScheme::compress(&self, compressor: &vortex_compressor::CascadingCompressor, data: &mut vortex_compressor::stats::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_compressor::builtins::FloatDictScheme::compress(&self, &vortex_compressor::CascadingCompressor, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_compressor::builtins::FloatDictScheme::descendant_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::FloatDictScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_compressor::builtins::FloatDictScheme::expected_compression_ratio(&self, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_compressor::builtins::FloatDictScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_compressor::builtins::FloatDictScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_compressor::builtins::FloatDictScheme::num_children(&self) -> usize @@ -484,13 +526,13 @@ impl vortex_compressor::scheme::Scheme for vortex_compressor::builtins::IntConst pub fn vortex_compressor::builtins::IntConstantScheme::ancestor_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::IntConstantScheme::compress(&self, _compressor: &vortex_compressor::CascadingCompressor, data: &mut vortex_compressor::stats::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_compressor::builtins::IntConstantScheme::compress(&self, &vortex_compressor::CascadingCompressor, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_compressor::builtins::IntConstantScheme::descendant_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::IntConstantScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_compressor::builtins::IntConstantScheme::expected_compression_ratio(&self, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_compressor::builtins::IntConstantScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_compressor::builtins::IntConstantScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_compressor::builtins::IntConstantScheme::num_children(&self) -> usize @@ -502,13 +544,13 @@ impl vortex_compressor::scheme::Scheme for vortex_compressor::builtins::IntDictS pub fn vortex_compressor::builtins::IntDictScheme::ancestor_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::IntDictScheme::compress(&self, compressor: &vortex_compressor::CascadingCompressor, data: &mut vortex_compressor::stats::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_compressor::builtins::IntDictScheme::compress(&self, &vortex_compressor::CascadingCompressor, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_compressor::builtins::IntDictScheme::descendant_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::IntDictScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_compressor::builtins::IntDictScheme::expected_compression_ratio(&self, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_compressor::builtins::IntDictScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_compressor::builtins::IntDictScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_compressor::builtins::IntDictScheme::num_children(&self) -> usize @@ -520,13 +562,13 @@ impl vortex_compressor::scheme::Scheme for vortex_compressor::builtins::StringCo pub fn vortex_compressor::builtins::StringConstantScheme::ancestor_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::StringConstantScheme::compress(&self, _compressor: &vortex_compressor::CascadingCompressor, data: &mut vortex_compressor::stats::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_compressor::builtins::StringConstantScheme::compress(&self, &vortex_compressor::CascadingCompressor, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_compressor::builtins::StringConstantScheme::descendant_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::StringConstantScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_compressor::builtins::StringConstantScheme::expected_compression_ratio(&self, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_compressor::builtins::StringConstantScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_compressor::builtins::StringConstantScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_compressor::builtins::StringConstantScheme::num_children(&self) -> usize @@ -538,13 +580,13 @@ impl vortex_compressor::scheme::Scheme for vortex_compressor::builtins::StringDi pub fn vortex_compressor::builtins::StringDictScheme::ancestor_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::StringDictScheme::compress(&self, compressor: &vortex_compressor::CascadingCompressor, data: &mut vortex_compressor::stats::ArrayAndStats, ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_compressor::builtins::StringDictScheme::compress(&self, &vortex_compressor::CascadingCompressor, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_compressor::builtins::StringDictScheme::descendant_exclusions(&self) -> alloc::vec::Vec -pub fn vortex_compressor::builtins::StringDictScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_compressor::builtins::StringDictScheme::expected_compression_ratio(&self, &vortex_compressor::stats::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_compressor::builtins::StringDictScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_compressor::builtins::StringDictScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_compressor::builtins::StringDictScheme::num_children(&self) -> usize @@ -576,19 +618,19 @@ pub fn vortex_compressor::stats::FloatErasedStats::clone(&self) -> vortex_compre impl core::convert::From> for vortex_compressor::stats::FloatErasedStats -pub fn vortex_compressor::stats::FloatErasedStats::from(typed: vortex_compressor::stats::FloatTypedStats) -> Self +pub fn vortex_compressor::stats::FloatErasedStats::from(vortex_compressor::stats::FloatTypedStats) -> Self impl core::convert::From> for vortex_compressor::stats::FloatErasedStats -pub fn vortex_compressor::stats::FloatErasedStats::from(typed: vortex_compressor::stats::FloatTypedStats) -> Self +pub fn vortex_compressor::stats::FloatErasedStats::from(vortex_compressor::stats::FloatTypedStats) -> Self impl core::convert::From> for vortex_compressor::stats::FloatErasedStats -pub fn vortex_compressor::stats::FloatErasedStats::from(typed: vortex_compressor::stats::FloatTypedStats) -> Self +pub fn vortex_compressor::stats::FloatErasedStats::from(vortex_compressor::stats::FloatTypedStats) -> Self impl core::fmt::Debug for vortex_compressor::stats::FloatErasedStats -pub fn vortex_compressor::stats::FloatErasedStats::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_compressor::stats::FloatErasedStats::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub enum vortex_compressor::stats::IntegerErasedStats @@ -628,39 +670,39 @@ pub fn vortex_compressor::stats::IntegerErasedStats::clone(&self) -> vortex_comp impl core::convert::From> for vortex_compressor::stats::IntegerErasedStats -pub fn vortex_compressor::stats::IntegerErasedStats::from(typed: vortex_compressor::stats::IntegerTypedStats) -> Self +pub fn vortex_compressor::stats::IntegerErasedStats::from(vortex_compressor::stats::IntegerTypedStats) -> Self impl core::convert::From> for vortex_compressor::stats::IntegerErasedStats -pub fn vortex_compressor::stats::IntegerErasedStats::from(typed: vortex_compressor::stats::IntegerTypedStats) -> Self +pub fn vortex_compressor::stats::IntegerErasedStats::from(vortex_compressor::stats::IntegerTypedStats) -> Self impl core::convert::From> for vortex_compressor::stats::IntegerErasedStats -pub fn vortex_compressor::stats::IntegerErasedStats::from(typed: vortex_compressor::stats::IntegerTypedStats) -> Self +pub fn vortex_compressor::stats::IntegerErasedStats::from(vortex_compressor::stats::IntegerTypedStats) -> Self impl core::convert::From> for vortex_compressor::stats::IntegerErasedStats -pub fn vortex_compressor::stats::IntegerErasedStats::from(typed: vortex_compressor::stats::IntegerTypedStats) -> Self +pub fn vortex_compressor::stats::IntegerErasedStats::from(vortex_compressor::stats::IntegerTypedStats) -> Self impl core::convert::From> for vortex_compressor::stats::IntegerErasedStats -pub fn vortex_compressor::stats::IntegerErasedStats::from(typed: vortex_compressor::stats::IntegerTypedStats) -> Self +pub fn vortex_compressor::stats::IntegerErasedStats::from(vortex_compressor::stats::IntegerTypedStats) -> Self impl core::convert::From> for vortex_compressor::stats::IntegerErasedStats -pub fn vortex_compressor::stats::IntegerErasedStats::from(typed: vortex_compressor::stats::IntegerTypedStats) -> Self +pub fn vortex_compressor::stats::IntegerErasedStats::from(vortex_compressor::stats::IntegerTypedStats) -> Self impl core::convert::From> for vortex_compressor::stats::IntegerErasedStats -pub fn vortex_compressor::stats::IntegerErasedStats::from(typed: vortex_compressor::stats::IntegerTypedStats) -> Self +pub fn vortex_compressor::stats::IntegerErasedStats::from(vortex_compressor::stats::IntegerTypedStats) -> Self impl core::convert::From> for vortex_compressor::stats::IntegerErasedStats -pub fn vortex_compressor::stats::IntegerErasedStats::from(typed: vortex_compressor::stats::IntegerTypedStats) -> Self +pub fn vortex_compressor::stats::IntegerErasedStats::from(vortex_compressor::stats::IntegerTypedStats) -> Self impl core::fmt::Debug for vortex_compressor::stats::IntegerErasedStats -pub fn vortex_compressor::stats::IntegerErasedStats::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_compressor::stats::IntegerErasedStats::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_compressor::stats::ArrayAndStats @@ -668,31 +710,31 @@ impl vortex_compressor::stats::ArrayAndStats pub fn vortex_compressor::stats::ArrayAndStats::array(&self) -> &vortex_array::array::erased::ArrayRef -pub fn vortex_compressor::stats::ArrayAndStats::array_as_primitive(&self) -> vortex_array::arrays::primitive::vtable::PrimitiveArray +pub fn vortex_compressor::stats::ArrayAndStats::array_as_primitive(&self) -> vortex_array::array::view::ArrayView<'_, vortex_array::arrays::primitive::vtable::Primitive> -pub fn vortex_compressor::stats::ArrayAndStats::array_as_utf8(&self) -> vortex_array::arrays::varbinview::vtable::VarBinViewArray +pub fn vortex_compressor::stats::ArrayAndStats::array_as_utf8(&self) -> vortex_array::array::view::ArrayView<'_, vortex_array::arrays::varbinview::vtable::VarBinView> pub fn vortex_compressor::stats::ArrayAndStats::array_len(&self) -> usize -pub fn vortex_compressor::stats::ArrayAndStats::bool_stats(&mut self) -> &vortex_compressor::stats::BoolStats +pub fn vortex_compressor::stats::ArrayAndStats::bool_stats(&self, &mut vortex_array::executor::ExecutionCtx) -> alloc::sync::Arc -pub fn vortex_compressor::stats::ArrayAndStats::float_stats(&mut self) -> &vortex_compressor::stats::FloatStats +pub fn vortex_compressor::stats::ArrayAndStats::float_stats(&self, &mut vortex_array::executor::ExecutionCtx) -> alloc::sync::Arc -pub fn vortex_compressor::stats::ArrayAndStats::get_or_insert_with(&mut self, f: impl core::ops::function::FnOnce() -> T) -> &T +pub fn vortex_compressor::stats::ArrayAndStats::get_or_insert_with(&self, impl core::ops::function::FnOnce() -> T) -> alloc::sync::Arc -pub fn vortex_compressor::stats::ArrayAndStats::integer_stats(&mut self) -> &vortex_compressor::stats::IntegerStats +pub fn vortex_compressor::stats::ArrayAndStats::integer_stats(&self, &mut vortex_array::executor::ExecutionCtx) -> alloc::sync::Arc pub fn vortex_compressor::stats::ArrayAndStats::into_array(self) -> vortex_array::array::erased::ArrayRef -pub fn vortex_compressor::stats::ArrayAndStats::new(array: vortex_array::array::erased::ArrayRef, opts: vortex_compressor::stats::GenerateStatsOptions) -> Self +pub fn vortex_compressor::stats::ArrayAndStats::new(vortex_array::array::erased::ArrayRef, vortex_compressor::stats::GenerateStatsOptions) -> Self -pub fn vortex_compressor::stats::ArrayAndStats::string_stats(&mut self) -> &vortex_compressor::stats::StringStats +pub fn vortex_compressor::stats::ArrayAndStats::string_stats(&self, &mut vortex_array::executor::ExecutionCtx) -> alloc::sync::Arc pub struct vortex_compressor::stats::BoolStats impl vortex_compressor::stats::BoolStats -pub fn vortex_compressor::stats::BoolStats::generate(input: &vortex_array::arrays::bool::vtable::BoolArray) -> vortex_error::VortexResult +pub fn vortex_compressor::stats::BoolStats::generate(&vortex_array::arrays::bool::vtable::BoolArray, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_compressor::stats::BoolStats::is_constant(&self) -> bool @@ -708,7 +750,7 @@ pub fn vortex_compressor::stats::BoolStats::clone(&self) -> vortex_compressor::s impl core::fmt::Debug for vortex_compressor::stats::BoolStats -pub fn vortex_compressor::stats::BoolStats::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_compressor::stats::BoolStats::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_compressor::stats::FloatDistinctInfo @@ -722,7 +764,7 @@ pub fn vortex_compressor::stats::FloatDistinctInfo::clone(&self) -> vortex_co impl core::fmt::Debug for vortex_compressor::stats::FloatDistinctInfo -pub fn vortex_compressor::stats::FloatDistinctInfo::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_compressor::stats::FloatDistinctInfo::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_compressor::stats::FloatStats @@ -732,9 +774,9 @@ pub fn vortex_compressor::stats::FloatStats::average_run_length(&self) -> u32 pub fn vortex_compressor::stats::FloatStats::erased(&self) -> &vortex_compressor::stats::FloatErasedStats -pub fn vortex_compressor::stats::FloatStats::generate(input: &vortex_array::arrays::primitive::vtable::PrimitiveArray) -> Self +pub fn vortex_compressor::stats::FloatStats::generate(&vortex_array::arrays::primitive::vtable::PrimitiveArray, &mut vortex_array::executor::ExecutionCtx) -> Self -pub fn vortex_compressor::stats::FloatStats::generate_opts(input: &vortex_array::arrays::primitive::vtable::PrimitiveArray, opts: vortex_compressor::stats::GenerateStatsOptions) -> Self +pub fn vortex_compressor::stats::FloatStats::generate_opts(&vortex_array::arrays::primitive::vtable::PrimitiveArray, vortex_compressor::stats::GenerateStatsOptions, &mut vortex_array::executor::ExecutionCtx) -> Self pub fn vortex_compressor::stats::FloatStats::null_count(&self) -> u32 @@ -750,7 +792,7 @@ pub fn vortex_compressor::stats::FloatStats::clone(&self) -> vortex_compressor:: impl core::fmt::Debug for vortex_compressor::stats::FloatStats -pub fn vortex_compressor::stats::FloatStats::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_compressor::stats::FloatStats::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_compressor::stats::FloatTypedStats @@ -760,15 +802,15 @@ pub fn vortex_compressor::stats::FloatTypedStats::distinct(&self) -> core::op impl core::convert::From> for vortex_compressor::stats::FloatErasedStats -pub fn vortex_compressor::stats::FloatErasedStats::from(typed: vortex_compressor::stats::FloatTypedStats) -> Self +pub fn vortex_compressor::stats::FloatErasedStats::from(vortex_compressor::stats::FloatTypedStats) -> Self impl core::convert::From> for vortex_compressor::stats::FloatErasedStats -pub fn vortex_compressor::stats::FloatErasedStats::from(typed: vortex_compressor::stats::FloatTypedStats) -> Self +pub fn vortex_compressor::stats::FloatErasedStats::from(vortex_compressor::stats::FloatTypedStats) -> Self impl core::convert::From> for vortex_compressor::stats::FloatErasedStats -pub fn vortex_compressor::stats::FloatErasedStats::from(typed: vortex_compressor::stats::FloatTypedStats) -> Self +pub fn vortex_compressor::stats::FloatErasedStats::from(vortex_compressor::stats::FloatTypedStats) -> Self impl core::clone::Clone for vortex_compressor::stats::FloatTypedStats @@ -776,7 +818,7 @@ pub fn vortex_compressor::stats::FloatTypedStats::clone(&self) -> vortex_comp impl core::fmt::Debug for vortex_compressor::stats::FloatTypedStats -pub fn vortex_compressor::stats::FloatTypedStats::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_compressor::stats::FloatTypedStats::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_compressor::stats::GenerateStatsOptions @@ -784,7 +826,7 @@ pub vortex_compressor::stats::GenerateStatsOptions::count_distinct_values: bool impl vortex_compressor::stats::GenerateStatsOptions -pub fn vortex_compressor::stats::GenerateStatsOptions::merge(self, other: Self) -> Self +pub fn vortex_compressor::stats::GenerateStatsOptions::merge(self, Self) -> Self impl core::clone::Clone for vortex_compressor::stats::GenerateStatsOptions @@ -796,7 +838,7 @@ pub fn vortex_compressor::stats::GenerateStatsOptions::default() -> vortex_compr impl core::fmt::Debug for vortex_compressor::stats::GenerateStatsOptions -pub fn vortex_compressor::stats::GenerateStatsOptions::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_compressor::stats::GenerateStatsOptions::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_compressor::stats::GenerateStatsOptions @@ -812,7 +854,7 @@ pub fn vortex_compressor::stats::IntegerDistinctInfo::clone(&self) -> vortex_ impl core::fmt::Debug for vortex_compressor::stats::IntegerDistinctInfo -pub fn vortex_compressor::stats::IntegerDistinctInfo::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_compressor::stats::IntegerDistinctInfo::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_compressor::stats::IntegerStats @@ -822,9 +864,9 @@ pub fn vortex_compressor::stats::IntegerStats::average_run_length(&self) -> u32 pub fn vortex_compressor::stats::IntegerStats::erased(&self) -> &vortex_compressor::stats::IntegerErasedStats -pub fn vortex_compressor::stats::IntegerStats::generate(input: &vortex_array::arrays::primitive::vtable::PrimitiveArray) -> Self +pub fn vortex_compressor::stats::IntegerStats::generate(&vortex_array::arrays::primitive::vtable::PrimitiveArray, &mut vortex_array::executor::ExecutionCtx) -> Self -pub fn vortex_compressor::stats::IntegerStats::generate_opts(input: &vortex_array::arrays::primitive::vtable::PrimitiveArray, opts: vortex_compressor::stats::GenerateStatsOptions) -> Self +pub fn vortex_compressor::stats::IntegerStats::generate_opts(&vortex_array::arrays::primitive::vtable::PrimitiveArray, vortex_compressor::stats::GenerateStatsOptions, &mut vortex_array::executor::ExecutionCtx) -> Self pub fn vortex_compressor::stats::IntegerStats::null_count(&self) -> u32 @@ -842,7 +884,7 @@ pub fn vortex_compressor::stats::IntegerStats::clone(&self) -> vortex_compressor impl core::fmt::Debug for vortex_compressor::stats::IntegerStats -pub fn vortex_compressor::stats::IntegerStats::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_compressor::stats::IntegerStats::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_compressor::stats::IntegerTypedStats @@ -852,35 +894,35 @@ pub fn vortex_compressor::stats::IntegerTypedStats::distinct(&self) -> core:: impl core::convert::From> for vortex_compressor::stats::IntegerErasedStats -pub fn vortex_compressor::stats::IntegerErasedStats::from(typed: vortex_compressor::stats::IntegerTypedStats) -> Self +pub fn vortex_compressor::stats::IntegerErasedStats::from(vortex_compressor::stats::IntegerTypedStats) -> Self impl core::convert::From> for vortex_compressor::stats::IntegerErasedStats -pub fn vortex_compressor::stats::IntegerErasedStats::from(typed: vortex_compressor::stats::IntegerTypedStats) -> Self +pub fn vortex_compressor::stats::IntegerErasedStats::from(vortex_compressor::stats::IntegerTypedStats) -> Self impl core::convert::From> for vortex_compressor::stats::IntegerErasedStats -pub fn vortex_compressor::stats::IntegerErasedStats::from(typed: vortex_compressor::stats::IntegerTypedStats) -> Self +pub fn vortex_compressor::stats::IntegerErasedStats::from(vortex_compressor::stats::IntegerTypedStats) -> Self impl core::convert::From> for vortex_compressor::stats::IntegerErasedStats -pub fn vortex_compressor::stats::IntegerErasedStats::from(typed: vortex_compressor::stats::IntegerTypedStats) -> Self +pub fn vortex_compressor::stats::IntegerErasedStats::from(vortex_compressor::stats::IntegerTypedStats) -> Self impl core::convert::From> for vortex_compressor::stats::IntegerErasedStats -pub fn vortex_compressor::stats::IntegerErasedStats::from(typed: vortex_compressor::stats::IntegerTypedStats) -> Self +pub fn vortex_compressor::stats::IntegerErasedStats::from(vortex_compressor::stats::IntegerTypedStats) -> Self impl core::convert::From> for vortex_compressor::stats::IntegerErasedStats -pub fn vortex_compressor::stats::IntegerErasedStats::from(typed: vortex_compressor::stats::IntegerTypedStats) -> Self +pub fn vortex_compressor::stats::IntegerErasedStats::from(vortex_compressor::stats::IntegerTypedStats) -> Self impl core::convert::From> for vortex_compressor::stats::IntegerErasedStats -pub fn vortex_compressor::stats::IntegerErasedStats::from(typed: vortex_compressor::stats::IntegerTypedStats) -> Self +pub fn vortex_compressor::stats::IntegerErasedStats::from(vortex_compressor::stats::IntegerTypedStats) -> Self impl core::convert::From> for vortex_compressor::stats::IntegerErasedStats -pub fn vortex_compressor::stats::IntegerErasedStats::from(typed: vortex_compressor::stats::IntegerTypedStats) -> Self +pub fn vortex_compressor::stats::IntegerErasedStats::from(vortex_compressor::stats::IntegerTypedStats) -> Self impl core::clone::Clone for vortex_compressor::stats::IntegerTypedStats @@ -888,7 +930,7 @@ pub fn vortex_compressor::stats::IntegerTypedStats::clone(&self) -> vortex_co impl core::fmt::Debug for vortex_compressor::stats::IntegerTypedStats -pub fn vortex_compressor::stats::IntegerTypedStats::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_compressor::stats::IntegerTypedStats::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_compressor::stats::StringStats @@ -896,9 +938,9 @@ impl vortex_compressor::stats::StringStats pub fn vortex_compressor::stats::StringStats::estimated_distinct_count(&self) -> core::option::Option -pub fn vortex_compressor::stats::StringStats::generate(input: &vortex_array::arrays::varbinview::vtable::VarBinViewArray) -> Self +pub fn vortex_compressor::stats::StringStats::generate(&vortex_array::arrays::varbinview::vtable::VarBinViewArray, &mut vortex_array::executor::ExecutionCtx) -> Self -pub fn vortex_compressor::stats::StringStats::generate_opts(input: &vortex_array::arrays::varbinview::vtable::VarBinViewArray, opts: vortex_compressor::stats::GenerateStatsOptions) -> Self +pub fn vortex_compressor::stats::StringStats::generate_opts(&vortex_array::arrays::varbinview::vtable::VarBinViewArray, vortex_compressor::stats::GenerateStatsOptions, &mut vortex_array::executor::ExecutionCtx) -> Self pub fn vortex_compressor::stats::StringStats::null_count(&self) -> u32 @@ -910,19 +952,17 @@ pub fn vortex_compressor::stats::StringStats::clone(&self) -> vortex_compressor: impl core::fmt::Debug for vortex_compressor::stats::StringStats -pub fn vortex_compressor::stats::StringStats::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_compressor::stats::StringStats::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_compressor::CascadingCompressor impl vortex_compressor::CascadingCompressor -pub fn vortex_compressor::CascadingCompressor::compress(&self, array: &vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult - -pub fn vortex_compressor::CascadingCompressor::compress_child(&self, child: &vortex_array::array::erased::ArrayRef, parent_ctx: &vortex_compressor::ctx::CompressorContext, parent_id: vortex_compressor::scheme::SchemeId, child_index: usize) -> vortex_error::VortexResult +pub fn vortex_compressor::CascadingCompressor::compress(&self, &vortex_array::array::erased::ArrayRef, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_compressor::CascadingCompressor::execution_ctx(&self) -> parking_lot::mutex::MutexGuard<'_, vortex_array::executor::ExecutionCtx> +pub fn vortex_compressor::CascadingCompressor::compress_child(&self, &vortex_array::array::erased::ArrayRef, &vortex_compressor::ctx::CompressorContext, vortex_compressor::scheme::SchemeId, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_compressor::CascadingCompressor::new(schemes: alloc::vec::Vec<&'static dyn vortex_compressor::scheme::Scheme>) -> Self +pub fn vortex_compressor::CascadingCompressor::new(alloc::vec::Vec<&'static dyn vortex_compressor::scheme::Scheme>) -> Self impl core::clone::Clone for vortex_compressor::CascadingCompressor @@ -930,4 +970,4 @@ pub fn vortex_compressor::CascadingCompressor::clone(&self) -> vortex_compressor impl core::fmt::Debug for vortex_compressor::CascadingCompressor -pub fn vortex_compressor::CascadingCompressor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_compressor::CascadingCompressor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result diff --git a/vortex-compressor/src/builtins/constant/bool.rs b/vortex-compressor/src/builtins/constant/bool.rs index 62e156379e9..3716e85e9ea 100644 --- a/vortex-compressor/src/builtins/constant/bool.rs +++ b/vortex-compressor/src/builtins/constant/bool.rs @@ -5,6 +5,7 @@ use vortex_array::ArrayRef; use vortex_array::Canonical; +use vortex_array::ExecutionCtx; use vortex_error::VortexResult; use crate::CascadingCompressor; @@ -12,6 +13,7 @@ use crate::builtins::BoolConstantScheme; use crate::builtins::constant::compress_constant_array_with_validity; use crate::ctx::CompressorContext; use crate::estimate::CompressionEstimate; +use crate::estimate::EstimateVerdict; use crate::scheme::Scheme; use crate::stats::ArrayAndStats; @@ -26,37 +28,39 @@ impl Scheme for BoolConstantScheme { fn expected_compression_ratio( &self, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { // Constant detection on a sample is a false positive, since the sample being constant does // not mean the full array is constant. - if ctx.is_sample() { - return CompressionEstimate::Skip; + if compress_ctx.is_sample() { + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } let array_len = data.array().len(); - let stats = data.bool_stats(); + let stats = data.bool_stats(exec_ctx); // We want to use `Constant` if there are only nulls in the array. if stats.value_count() == 0 { debug_assert_eq!(stats.null_count() as usize, array_len); - return CompressionEstimate::AlwaysUse; + return CompressionEstimate::Verdict(EstimateVerdict::AlwaysUse); } if stats.is_constant() { - return CompressionEstimate::AlwaysUse; + return CompressionEstimate::Verdict(EstimateVerdict::AlwaysUse); } - CompressionEstimate::Skip + CompressionEstimate::Verdict(EstimateVerdict::Skip) } fn compress( &self, _compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - _ctx: CompressorContext, + data: &ArrayAndStats, + _compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { - compress_constant_array_with_validity(data.array()) + compress_constant_array_with_validity(data.array(), exec_ctx) } } diff --git a/vortex-compressor/src/builtins/constant/float.rs b/vortex-compressor/src/builtins/constant/float.rs index df8ab7464b6..501e3cd9623 100644 --- a/vortex-compressor/src/builtins/constant/float.rs +++ b/vortex-compressor/src/builtins/constant/float.rs @@ -5,6 +5,7 @@ use vortex_array::ArrayRef; use vortex_array::Canonical; +use vortex_array::ExecutionCtx; use vortex_array::aggregate_fn::fns::is_constant::is_constant; use vortex_error::VortexResult; @@ -14,6 +15,8 @@ use crate::builtins::FloatConstantScheme; use crate::builtins::constant::compress_constant_array_with_validity; use crate::ctx::CompressorContext; use crate::estimate::CompressionEstimate; +use crate::estimate::DeferredEstimate; +use crate::estimate::EstimateVerdict; use crate::scheme::Scheme; use crate::stats::ArrayAndStats; @@ -28,32 +31,33 @@ impl Scheme for FloatConstantScheme { fn expected_compression_ratio( &self, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { // Constant detection on a sample is a false positive, since the sample being constant does // not mean the full array is constant. - if ctx.is_sample() { - return CompressionEstimate::Skip; + if compress_ctx.is_sample() { + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } let array_len = data.array().len(); - let stats = data.float_stats(); + let stats = data.float_stats(exec_ctx); // Note that we only compute distinct counts if other schemes have requested it. if let Some(distinct_count) = stats.distinct_count() { if distinct_count > 1 { - return CompressionEstimate::Skip; + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } else { debug_assert_eq!(distinct_count, 1); - return CompressionEstimate::AlwaysUse; + return CompressionEstimate::Verdict(EstimateVerdict::AlwaysUse); } } // We want to use `Constant` if there are only nulls in the array. if stats.value_count() == 0 { debug_assert_eq!(stats.null_count() as usize, array_len); - return CompressionEstimate::AlwaysUse; + return CompressionEstimate::Verdict(EstimateVerdict::AlwaysUse); } // TODO(connor): Can we be smart here with the max and min like with integers? @@ -61,21 +65,24 @@ impl Scheme for FloatConstantScheme { // Otherwise our best bet is to actually check if the array is constant. // This is an expensive check, but in practice the distinct count is known because we often // include dictionary encoding in our set of schemes, so we rarely call this. - CompressionEstimate::Estimate(Box::new(|compressor, data, _ctx| { - if is_constant(data.array(), &mut compressor.execution_ctx())? { - Ok(CompressionEstimate::AlwaysUse) - } else { - Ok(CompressionEstimate::Skip) - } - })) + CompressionEstimate::Deferred(DeferredEstimate::Callback(Box::new( + |_compressor, data, _best_so_far, _ctx, exec_ctx| { + if is_constant(data.array(), exec_ctx)? { + Ok(EstimateVerdict::AlwaysUse) + } else { + Ok(EstimateVerdict::Skip) + } + }, + ))) } fn compress( &self, _compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - _ctx: CompressorContext, + data: &ArrayAndStats, + _compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { - compress_constant_array_with_validity(data.array()) + compress_constant_array_with_validity(data.array(), exec_ctx) } } diff --git a/vortex-compressor/src/builtins/constant/integer.rs b/vortex-compressor/src/builtins/constant/integer.rs index 0264893e5c8..a10481ad100 100644 --- a/vortex-compressor/src/builtins/constant/integer.rs +++ b/vortex-compressor/src/builtins/constant/integer.rs @@ -5,6 +5,7 @@ use vortex_array::ArrayRef; use vortex_array::Canonical; +use vortex_array::ExecutionCtx; use vortex_error::VortexResult; use super::is_integer_primitive; @@ -13,6 +14,7 @@ use crate::builtins::IntConstantScheme; use crate::builtins::constant::compress_constant_array_with_validity; use crate::ctx::CompressorContext; use crate::estimate::CompressionEstimate; +use crate::estimate::EstimateVerdict; use crate::scheme::Scheme; use crate::stats::ArrayAndStats; @@ -27,48 +29,50 @@ impl Scheme for IntConstantScheme { fn expected_compression_ratio( &self, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { // Constant detection on a sample is a false positive, since the sample being constant does // not mean the full array is constant. - if ctx.is_sample() { - return CompressionEstimate::Skip; + if compress_ctx.is_sample() { + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } let array_len = data.array().len(); - let stats = data.integer_stats(); + let stats = data.integer_stats(exec_ctx); // Note that we only compute distinct counts if other schemes have requested it. if let Some(distinct_count) = stats.distinct_count() { if distinct_count > 1 { - return CompressionEstimate::Skip; + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } else { debug_assert_eq!(distinct_count, 1); - return CompressionEstimate::AlwaysUse; + return CompressionEstimate::Verdict(EstimateVerdict::AlwaysUse); } } // We want to use `Constant` if there are only nulls in the array. if stats.value_count() == 0 { debug_assert_eq!(stats.null_count() as usize, array_len); - return CompressionEstimate::AlwaysUse; + return CompressionEstimate::Verdict(EstimateVerdict::AlwaysUse); } // Otherwise, use the max and min to determine if there is a single value. match stats.erased().max_minus_min().checked_ilog2() { - Some(_) => CompressionEstimate::Skip, + Some(_) => CompressionEstimate::Verdict(EstimateVerdict::Skip), // If max-min == 0, then we know that there is only 1 value. - None => CompressionEstimate::AlwaysUse, + None => CompressionEstimate::Verdict(EstimateVerdict::AlwaysUse), } } fn compress( &self, _compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - _ctx: CompressorContext, + data: &ArrayAndStats, + _compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { - compress_constant_array_with_validity(data.array()) + compress_constant_array_with_validity(data.array(), exec_ctx) } } diff --git a/vortex-compressor/src/builtins/constant/mod.rs b/vortex-compressor/src/builtins/constant/mod.rs index 1b177fc530b..139cb8d2790 100644 --- a/vortex-compressor/src/builtins/constant/mod.rs +++ b/vortex-compressor/src/builtins/constant/mod.rs @@ -4,6 +4,7 @@ //! Constant encoding schemes for bool, float, integer, and string arrays. use vortex_array::ArrayRef; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; use vortex_array::arrays::ConstantArray; use vortex_array::arrays::MaskedArray; @@ -42,21 +43,24 @@ mod string; /// Assumes that the source array has constant valid scalars. /// /// If the array has any nulls, returns a [`MaskedArray`] with a [`ConstantArray`] child.` -fn compress_constant_array_with_validity(source: &ArrayRef) -> VortexResult { - if source.all_invalid()? { +fn compress_constant_array_with_validity( + source: &ArrayRef, + ctx: &mut ExecutionCtx, +) -> VortexResult { + if source.all_invalid(ctx)? { return Ok( ConstantArray::new(Scalar::null(source.dtype().clone()), source.len()).into_array(), ); } let scalar_idx = (0..source.len()) - .position(|idx| source.is_valid(idx).unwrap_or(false)) + .position(|idx| source.is_valid(idx, ctx).unwrap_or(false)) .vortex_expect("We checked that there exists a scalar that is not invalid"); - let scalar = source.scalar_at(scalar_idx)?; + let scalar = source.execute_scalar(scalar_idx, ctx)?; let const_arr = ConstantArray::new(scalar, source.len()).into_array(); - if !source.all_valid()? { + if !source.all_valid(ctx)? { Ok(MaskedArray::try_new(const_arr, source.validity()?)?.into_array()) } else { Ok(const_arr) diff --git a/vortex-compressor/src/builtins/constant/string.rs b/vortex-compressor/src/builtins/constant/string.rs index 96e4e7ba674..fcd6138bca2 100644 --- a/vortex-compressor/src/builtins/constant/string.rs +++ b/vortex-compressor/src/builtins/constant/string.rs @@ -5,6 +5,7 @@ use vortex_array::ArrayRef; use vortex_array::Canonical; +use vortex_array::ExecutionCtx; use vortex_array::aggregate_fn::fns::is_constant::is_constant; use vortex_error::VortexResult; @@ -14,6 +15,8 @@ use crate::builtins::StringConstantScheme; use crate::builtins::constant::compress_constant_array_with_validity; use crate::ctx::CompressorContext; use crate::estimate::CompressionEstimate; +use crate::estimate::DeferredEstimate; +use crate::estimate::EstimateVerdict; use crate::scheme::Scheme; use crate::stats::ArrayAndStats; @@ -28,48 +31,52 @@ impl Scheme for StringConstantScheme { fn expected_compression_ratio( &self, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { // Constant detection on a sample is a false positive, since the sample being constant does // not mean the full array is constant. - if ctx.is_sample() { - return CompressionEstimate::Skip; + if compress_ctx.is_sample() { + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } let array_len = data.array().len(); - let stats = data.string_stats(); + let stats = data.string_stats(exec_ctx); // We want to use `Constant` if there are only nulls in the array. if stats.value_count() == 0 { debug_assert_eq!(stats.null_count() as usize, array_len); - return CompressionEstimate::AlwaysUse; + return CompressionEstimate::Verdict(EstimateVerdict::AlwaysUse); } // Since the estimated distinct count is always going to be less than or equal to the actual // distinct count, if this is not equal to 1 the actual is definitely not equal to 1. if stats.estimated_distinct_count().is_some_and(|c| c > 1) { - return CompressionEstimate::Skip; + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } // Otherwise our best bet is to actually check if the array is constant. // This is an expensive check, but the alternative of not compressing a constant array is // far less preferable. - CompressionEstimate::Estimate(Box::new(|compressor, data, _ctx| { - if is_constant(data.array(), &mut compressor.execution_ctx())? { - Ok(CompressionEstimate::AlwaysUse) - } else { - Ok(CompressionEstimate::Skip) - } - })) + CompressionEstimate::Deferred(DeferredEstimate::Callback(Box::new( + |_compressor, data, _best_so_far, _ctx, exec_ctx| { + if is_constant(data.array(), exec_ctx)? { + Ok(EstimateVerdict::AlwaysUse) + } else { + Ok(EstimateVerdict::Skip) + } + }, + ))) } fn compress( &self, _compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - _ctx: CompressorContext, + data: &ArrayAndStats, + _compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { - compress_constant_array_with_validity(data.array()) + compress_constant_array_with_validity(data.array(), exec_ctx) } } diff --git a/vortex-compressor/src/builtins/dict/float.rs b/vortex-compressor/src/builtins/dict/float.rs index c2cdbeafaf2..51d553b591f 100644 --- a/vortex-compressor/src/builtins/dict/float.rs +++ b/vortex-compressor/src/builtins/dict/float.rs @@ -7,9 +7,12 @@ //! external compatibility. use vortex_array::ArrayRef; +use vortex_array::ArrayView; use vortex_array::Canonical; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; use vortex_array::arrays::DictArray; +use vortex_array::arrays::Primitive; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::dict::DictArrayExt; use vortex_array::arrays::dict::DictArraySlotsExt; @@ -26,6 +29,8 @@ use crate::builtins::IntDictScheme; use crate::builtins::is_float_primitive; use crate::ctx::CompressorContext; use crate::estimate::CompressionEstimate; +use crate::estimate::DeferredEstimate; +use crate::estimate::EstimateVerdict; use crate::scheme::ChildSelection; use crate::scheme::DescendantExclusion; use crate::scheme::Scheme; @@ -78,13 +83,14 @@ impl Scheme for FloatDictScheme { fn expected_compression_ratio( &self, - data: &mut ArrayAndStats, - _ctx: CompressorContext, + data: &ArrayAndStats, + _compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { - let stats = data.float_stats(); + let stats = data.float_stats(exec_ctx); if stats.value_count() == 0 { - return CompressionEstimate::Skip; + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } let distinct_values_count = stats.distinct_count().vortex_expect( @@ -93,36 +99,38 @@ impl Scheme for FloatDictScheme { // If > 50% of the values are distinct, skip dictionary scheme. if distinct_values_count > stats.value_count() / 2 { - return CompressionEstimate::Skip; + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } // Let sampling determine the expected ratio. - CompressionEstimate::Sample + CompressionEstimate::Deferred(DeferredEstimate::Sample) } fn compress( &self, compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { - // TODO(connor): Fight the borrow checker (needs interior mutability)! - let stats = data.float_stats().clone(); + let stats = data.float_stats(exec_ctx); let dict = dictionary_encode(data.array_as_primitive(), &stats)?; let has_all_values_referenced = dict.has_all_values_referenced(); // Values = child 0. - let compressed_values = compressor.compress_child(dict.values(), &ctx, self.id(), 0)?; + let compressed_values = + compressor.compress_child(dict.values(), &compress_ctx, self.id(), 0, exec_ctx)?; // Codes = child 1. let narrowed_codes = dict .codes() .clone() - .execute::(&mut compressor.execution_ctx())? + .execute::(exec_ctx)? .narrow()? .into_array(); - let compressed_codes = compressor.compress_child(&narrowed_codes, &ctx, self.id(), 1)?; + let compressed_codes = + compressor.compress_child(&narrowed_codes, &compress_ctx, self.id(), 1, exec_ctx)?; // SAFETY: compressing codes or values does not alter the invariants. unsafe { @@ -182,7 +190,10 @@ macro_rules! typed_encode { /// # Errors /// /// Returns an error if unable to compute validity. -pub fn dictionary_encode(array: PrimitiveArray, stats: &FloatStats) -> VortexResult { +pub fn dictionary_encode( + array: ArrayView<'_, Primitive>, + stats: &FloatStats, +) -> VortexResult { match stats.erased() { FloatErasedStats::F16(typed) => typed_encode!(array, stats, typed, f16), FloatErasedStats::F32(typed) => typed_encode!(array, stats, typed, f32), @@ -205,7 +216,7 @@ macro_rules! impl_encode { ($typ:ty, $utyp:ty, $($ityp:ty),+) => { $( impl Encode<$typ, $ityp> for DictEncoder { - #[allow(clippy::cast_possible_truncation)] + #[expect(clippy::cast_possible_truncation)] fn encode(distinct: &[$typ], values: &[$typ]) -> Buffer<$ityp> { let mut codes = vortex_utils::aliases::hash_map::HashMap::<$utyp, $ityp>::with_capacity( @@ -235,20 +246,26 @@ impl_encode!(f64, u64); #[cfg(test)] mod tests { use vortex_array::IntoArray; - use vortex_array::ToCanonical; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::dict::DictArraySlotsExt; use vortex_array::assert_arrays_eq; + use vortex_array::session::ArraySession; use vortex_array::validity::Validity; use vortex_buffer::buffer; + use vortex_error::VortexResult; + use vortex_session::VortexSession; use super::dictionary_encode; use crate::stats::FloatStats; use crate::stats::GenerateStatsOptions; #[test] - fn test_float_dict_encode() { + fn test_float_dict_encode() -> VortexResult<()> { + let mut ctx = VortexSession::empty() + .with::() + .create_execution_ctx(); let values = buffer![1f32, 2f32, 2f32, 0f32, 1f32]; let validity = Validity::Array(BoolArray::from_iter([true, true, true, false, true]).into_array()); @@ -259,8 +276,9 @@ mod tests { GenerateStatsOptions { count_distinct_values: true, }, + &mut ctx, ); - let dict_array = dictionary_encode(array, &stats).unwrap(); + let dict_array = dictionary_encode(array.as_view(), &stats)?; assert_eq!(dict_array.values().len(), 2); assert_eq!(dict_array.codes().len(), 5); @@ -269,7 +287,12 @@ mod tests { Validity::Array(BoolArray::from_iter([true, true, true, false, true]).into_array()), ) .into_array(); - let undict = dict_array.as_array().to_primitive().into_array(); + let undict = dict_array + .as_array() + .clone() + .execute::(&mut ctx)? + .into_array(); assert_arrays_eq!(undict, expected); + Ok(()) } } diff --git a/vortex-compressor/src/builtins/dict/integer.rs b/vortex-compressor/src/builtins/dict/integer.rs index 8671f9d79e6..140afdcebf1 100644 --- a/vortex-compressor/src/builtins/dict/integer.rs +++ b/vortex-compressor/src/builtins/dict/integer.rs @@ -7,9 +7,12 @@ //! for external compatibility. use vortex_array::ArrayRef; +use vortex_array::ArrayView; use vortex_array::Canonical; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; use vortex_array::arrays::DictArray; +use vortex_array::arrays::Primitive; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::dict::DictArrayExt; use vortex_array::arrays::dict::DictArraySlotsExt; @@ -24,6 +27,7 @@ use crate::builtins::IntDictScheme; use crate::builtins::is_integer_primitive; use crate::ctx::CompressorContext; use crate::estimate::CompressionEstimate; +use crate::estimate::EstimateVerdict; use crate::scheme::Scheme; use crate::scheme::SchemeExt; use crate::stats::ArrayAndStats; @@ -53,14 +57,15 @@ impl Scheme for IntDictScheme { fn expected_compression_ratio( &self, - data: &mut ArrayAndStats, - _ctx: CompressorContext, + data: &ArrayAndStats, + _compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { let bit_width = data.array_as_primitive().ptype().bit_width(); - let stats = data.integer_stats(); + let stats = data.integer_stats(exec_ctx); if stats.value_count() == 0 { - return CompressionEstimate::Skip; + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } let distinct_values_count = stats.distinct_count().vortex_expect( @@ -69,7 +74,7 @@ impl Scheme for IntDictScheme { // If > 50% of the values are distinct, skip dictionary scheme. if distinct_values_count > stats.value_count() / 2 { - return CompressionEstimate::Skip; + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } // Ignore nulls encoding for the estimate. We only focus on values. @@ -90,30 +95,34 @@ impl Scheme for IntDictScheme { let before = stats.value_count() as usize * bit_width; - CompressionEstimate::Ratio(before as f64 / (values_size + codes_size) as f64) + CompressionEstimate::Verdict(EstimateVerdict::Ratio( + before as f64 / (values_size + codes_size) as f64, + )) } fn compress( &self, compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { - // TODO(connor): Fight the borrow checker (needs interior mutability)! - let stats = data.integer_stats().clone(); + let stats = data.integer_stats(exec_ctx); let dict = dictionary_encode(data.array_as_primitive(), &stats)?; // Values = child 0. - let compressed_values = compressor.compress_child(dict.values(), &ctx, self.id(), 0)?; + let compressed_values = + compressor.compress_child(dict.values(), &compress_ctx, self.id(), 0, exec_ctx)?; // Codes = child 1. let narrowed_codes = dict .codes() .clone() - .execute::(&mut compressor.execution_ctx())? + .execute::(exec_ctx)? .narrow()? .into_array(); - let compressed_codes = compressor.compress_child(&narrowed_codes, &ctx, self.id(), 1)?; + let compressed_codes = + compressor.compress_child(&narrowed_codes, &compress_ctx, self.id(), 1, exec_ctx)?; // SAFETY: compressing codes does not change their values. unsafe { @@ -177,7 +186,10 @@ macro_rules! typed_encode { clippy::cognitive_complexity, reason = "complexity from match on all integer types" )] -pub fn dictionary_encode(array: PrimitiveArray, stats: &IntegerStats) -> VortexResult { +pub fn dictionary_encode( + array: ArrayView<'_, Primitive>, + stats: &IntegerStats, +) -> VortexResult { match stats.erased() { IntegerErasedStats::U8(typed) => typed_encode!(array, stats, typed, u8), IntegerErasedStats::U16(typed) => typed_encode!(array, stats, typed, u16), @@ -205,7 +217,7 @@ macro_rules! impl_encode { ($typ:ty, $($ityp:ty),+) => { $( impl Encode<$typ, $ityp> for DictEncoder { - #[allow(clippy::cast_possible_truncation)] + #[expect(clippy::cast_possible_truncation)] fn encode(distinct: &[$typ], values: &[$typ]) -> Buffer<$ityp> { let mut codes = vortex_utils::aliases::hash_map::HashMap::<$typ, $ityp>::with_capacity( @@ -241,19 +253,25 @@ impl_encode!(i64); #[cfg(test)] mod tests { use vortex_array::IntoArray; - use vortex_array::ToCanonical; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::dict::DictArraySlotsExt; use vortex_array::assert_arrays_eq; + use vortex_array::session::ArraySession; use vortex_array::validity::Validity; use vortex_buffer::buffer; + use vortex_error::VortexResult; + use vortex_session::VortexSession; use super::dictionary_encode; use crate::stats::IntegerStats; #[test] - fn test_dict_encode_integer_stats() { + fn test_dict_encode_integer_stats() -> VortexResult<()> { + let mut ctx = VortexSession::empty() + .with::() + .create_execution_ctx(); let data = buffer![100i32, 200, 100, 0, 100]; let validity = Validity::Array(BoolArray::from_iter([true, true, true, false, true]).into_array()); @@ -264,8 +282,9 @@ mod tests { crate::stats::GenerateStatsOptions { count_distinct_values: true, }, + &mut ctx, ); - let dict_array = dictionary_encode(array, &stats).unwrap(); + let dict_array = dictionary_encode(array.as_view(), &stats)?; assert_eq!(dict_array.values().len(), 2); assert_eq!(dict_array.codes().len(), 5); @@ -274,7 +293,12 @@ mod tests { Validity::Array(BoolArray::from_iter([true, true, true, false, true]).into_array()), ) .into_array(); - let undict = dict_array.as_array().to_primitive().into_array(); + let undict = dict_array + .as_array() + .clone() + .execute::(&mut ctx)? + .into_array(); assert_arrays_eq!(undict, expected); + Ok(()) } } diff --git a/vortex-compressor/src/builtins/dict/string.rs b/vortex-compressor/src/builtins/dict/string.rs index 399910a00ff..ac6affdf854 100644 --- a/vortex-compressor/src/builtins/dict/string.rs +++ b/vortex-compressor/src/builtins/dict/string.rs @@ -8,6 +8,7 @@ use vortex_array::ArrayRef; use vortex_array::Canonical; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; use vortex_array::arrays::DictArray; use vortex_array::arrays::PrimitiveArray; @@ -24,6 +25,8 @@ use crate::builtins::StringDictScheme; use crate::builtins::is_utf8_string; use crate::ctx::CompressorContext; use crate::estimate::CompressionEstimate; +use crate::estimate::DeferredEstimate; +use crate::estimate::EstimateVerdict; use crate::scheme::ChildSelection; use crate::scheme::DescendantExclusion; use crate::scheme::Scheme; @@ -65,13 +68,14 @@ impl Scheme for StringDictScheme { fn expected_compression_ratio( &self, - data: &mut ArrayAndStats, - _ctx: CompressorContext, + data: &ArrayAndStats, + _compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate { - let stats = data.string_stats(); + let stats = data.string_stats(exec_ctx); if stats.value_count() == 0 { - return CompressionEstimate::Skip; + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } let estimated_distinct_values_count = stats.estimated_distinct_count().vortex_expect( @@ -80,32 +84,35 @@ impl Scheme for StringDictScheme { // If > 50% of the values are distinct, skip dictionary scheme. if estimated_distinct_values_count > stats.value_count() / 2 { - return CompressionEstimate::Skip; + return CompressionEstimate::Verdict(EstimateVerdict::Skip); } // Let sampling determine the expected ratio. - CompressionEstimate::Sample + CompressionEstimate::Deferred(DeferredEstimate::Sample) } fn compress( &self, compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { let dict = dict_encode(data.array())?; // Values = child 0. - let compressed_values = compressor.compress_child(dict.values(), &ctx, self.id(), 0)?; + let compressed_values = + compressor.compress_child(dict.values(), &compress_ctx, self.id(), 0, exec_ctx)?; // Codes = child 1. let narrowed_codes = dict .codes() .clone() - .execute::(&mut compressor.execution_ctx())? + .execute::(exec_ctx)? .narrow()? .into_array(); - let compressed_codes = compressor.compress_child(&narrowed_codes, &ctx, self.id(), 1)?; + let compressed_codes = + compressor.compress_child(&narrowed_codes, &compress_ctx, self.id(), 1, exec_ctx)?; // SAFETY: compressing codes or values does not alter the invariants. unsafe { diff --git a/vortex-compressor/src/compressor.rs b/vortex-compressor/src/compressor.rs index 099cc858f92..d3ad0d9d5c3 100644 --- a/vortex-compressor/src/compressor.rs +++ b/vortex-compressor/src/compressor.rs @@ -3,23 +3,17 @@ //! Cascading array compression implementation. -use std::sync::Arc; - -use parking_lot::Mutex; -use parking_lot::MutexGuard; use vortex_array::ArrayRef; use vortex_array::Canonical; use vortex_array::CanonicalValidity; use vortex_array::ExecutionCtx; use vortex_array::IntoArray; -use vortex_array::LEGACY_SESSION; -use vortex_array::ToCanonical; -use vortex_array::VortexSessionExecute; use vortex_array::arrays::ConstantArray; use vortex_array::arrays::ExtensionArray; use vortex_array::arrays::FixedSizeListArray; use vortex_array::arrays::ListArray; use vortex_array::arrays::ListViewArray; +use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::StructArray; use vortex_array::arrays::extension::ExtensionArrayExt; use vortex_array::arrays::fixed_size_list::FixedSizeListArrayExt; @@ -27,19 +21,23 @@ use vortex_array::arrays::list::ListArrayExt; use vortex_array::arrays::listview::ListViewArrayExt; use vortex_array::arrays::listview::list_from_list_view; use vortex_array::arrays::primitive::PrimitiveArrayExt; +use vortex_array::arrays::scalar_fn::AnyScalarFn; use vortex_array::arrays::struct_::StructArrayExt; use vortex_array::dtype::DType; use vortex_array::dtype::Nullability; use vortex_array::scalar::Scalar; use vortex_error::VortexResult; use vortex_error::vortex_bail; -use vortex_error::vortex_panic; use crate::builtins::IntDictScheme; use crate::ctx::CompressorContext; use crate::estimate::CompressionEstimate; +use crate::estimate::DeferredEstimate; +use crate::estimate::EstimateScore; +use crate::estimate::EstimateVerdict; +use crate::estimate::WinnerEstimate; use crate::estimate::estimate_compression_ratio_with_sampling; -use crate::estimate::is_better_ratio; +use crate::estimate::is_better_score; use crate::scheme::ChildSelection; use crate::scheme::DescendantExclusion; use crate::scheme::Scheme; @@ -47,11 +45,10 @@ use crate::scheme::SchemeExt; use crate::scheme::SchemeId; use crate::stats::ArrayAndStats; use crate::stats::GenerateStatsOptions; +use crate::trace; -/// The implicit root scheme ID for the compressor's own cascading (e.g. list offset compression). -/// -/// This is the **only** [`SchemeId`] that is not auto-provided via [`SchemeExt`]. -const ROOT_SCHEME_ID: SchemeId = SchemeId { +/// Synthetic scheme ID used for the compressor's own root-level cascading. +pub(crate) const ROOT_SCHEME_ID: SchemeId = SchemeId { name: "vortex.compressor.root", }; @@ -72,7 +69,7 @@ mod root_list_children { /// The compressor works by: /// 1. Canonicalizing input arrays to a standard representation. /// 2. Pre-filtering schemes by [`Scheme::matches`] and exclusion rules. -/// 3. Evaluating each matching scheme's compression ratio on a sample. +/// 3. Evaluating each matching scheme's compression estimate and resolving deferred work. /// 4. Compressing with the best scheme and verifying the result is smaller. /// /// No scheme may appear twice in a cascade chain. The compressor enforces this automatically @@ -85,11 +82,6 @@ pub struct CascadingCompressor { /// Descendant exclusion rules for the compressor's own cascading (e.g. excluding Dict from /// list offsets). root_exclusions: Vec, - - /// Shared execution context for array operations during compression. - /// - /// This should have low contention as we only execute arrays one at a time during compression. - ctx: Arc>, } impl CascadingCompressor { @@ -107,16 +99,9 @@ impl CascadingCompressor { Self { schemes, root_exclusions, - // TODO(connor): The caller should probably pass this in. - ctx: Arc::new(Mutex::new(LEGACY_SESSION.create_execution_ctx())), } } - /// Returns a mutable borrow of the execution context. - pub fn execution_ctx(&self) -> MutexGuard<'_, ExecutionCtx> { - self.ctx.lock() - } - /// Compresses an array using cascading adaptive compression. /// /// First canonicalizes and compacts the array, then applies optimal compression schemes. @@ -124,23 +109,29 @@ impl CascadingCompressor { /// # Errors /// /// Returns an error if canonicalization or compression fails. - pub fn compress(&self, array: &ArrayRef) -> VortexResult { - let canonical = array - .clone() - .execute::(&mut self.execution_ctx())? - .0; + pub fn compress( + &self, + array: &ArrayRef, + exec_ctx: &mut ExecutionCtx, + ) -> VortexResult { + let before_nbytes = array.nbytes(); + let span = trace::compress_span(array.len(), array.dtype(), before_nbytes); + let _enter = span.enter(); - // Compact it, removing any wasted space before we attempt to compress it. + let canonical = array.clone().execute::(exec_ctx)?.0; let compact = canonical.compact()?; + let compressed = self.compress_canonical(compact, CompressorContext::new(), exec_ctx)?; + + trace::record_compress_outcome(&span, before_nbytes, compressed.nbytes()); - self.compress_canonical(compact, CompressorContext::new()) + Ok(compressed) } /// Compresses a child array produced by a cascading scheme. /// - /// If the cascade budget is exhausted, the canonical array is returned as-is. Otherwise, - /// the child context is created by descending and recording the parent scheme + child - /// index, and compression proceeds normally. + /// If the cascade budget is exhausted, the canonical array is returned as-is. Otherwise, the + /// child context is created by descending and recording the parent scheme + child index, and + /// compression proceeds normally. /// /// # Errors /// @@ -151,21 +142,20 @@ impl CascadingCompressor { parent_ctx: &CompressorContext, parent_id: SchemeId, child_index: usize, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { if parent_ctx.finished_cascading() { + trace::cascade_exhausted(parent_id, child_index); return Ok(child.clone()); } - let canonical = child - .clone() - .execute::(&mut self.execution_ctx())? - .0; + let canonical = child.clone().execute::(exec_ctx)?.0; let compact = canonical.compact()?; let child_ctx = parent_ctx .clone() .descend_with_scheme(parent_id, child_index); - self.compress_canonical(compact, child_ctx) + self.compress_canonical(compact, child_ctx, exec_ctx) } /// Compresses a canonical array by dispatching to type-specific logic. @@ -176,23 +166,24 @@ impl CascadingCompressor { fn compress_canonical( &self, array: Canonical, - ctx: CompressorContext, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { match array { Canonical::Null(null_array) => Ok(null_array.into_array()), Canonical::Bool(bool_array) => { - self.choose_and_compress(Canonical::Bool(bool_array), ctx) + self.choose_and_compress(Canonical::Bool(bool_array), compress_ctx, exec_ctx) } Canonical::Primitive(primitive) => { - self.choose_and_compress(Canonical::Primitive(primitive), ctx) + self.choose_and_compress(Canonical::Primitive(primitive), compress_ctx, exec_ctx) } Canonical::Decimal(decimal) => { - self.choose_and_compress(Canonical::Decimal(decimal), ctx) + self.choose_and_compress(Canonical::Decimal(decimal), compress_ctx, exec_ctx) } Canonical::Struct(struct_array) => { let fields = struct_array .iter_unmasked_fields() - .map(|field| self.compress(field)) + .map(|field| self.compress(field, exec_ctx)) .collect::, _>>()?; Ok(StructArray::try_new( @@ -206,13 +197,13 @@ impl CascadingCompressor { Canonical::List(list_view_array) => { if list_view_array.is_zero_copy_to_list() || list_view_array.elements().is_empty() { let list_array = list_from_list_view(list_view_array)?; - self.compress_list_array(list_array, ctx) + self.compress_list_array(list_array, compress_ctx, exec_ctx) } else { - self.compress_list_view_array(list_view_array, ctx) + self.compress_list_view_array(list_view_array, compress_ctx, exec_ctx) } } Canonical::FixedSizeList(fsl_array) => { - let compressed_elems = self.compress(fsl_array.elements())?; + let compressed_elems = self.compress(fsl_array.elements(), exec_ctx)?; Ok(FixedSizeListArray::try_new( compressed_elems, @@ -227,7 +218,7 @@ impl CascadingCompressor { .dtype() .eq_ignore_nullability(&DType::Utf8(Nullability::NonNullable)) { - self.choose_and_compress(Canonical::VarBinView(strings), ctx) + self.choose_and_compress(Canonical::VarBinView(strings), compress_ctx, exec_ctx) } else { // We do not compress binary arrays. Ok(strings.into_array()) @@ -237,14 +228,22 @@ impl CascadingCompressor { let before_nbytes = ext_array.as_ref().nbytes(); // Try scheme-based compression first. - let result = - self.choose_and_compress(Canonical::Extension(ext_array.clone()), ctx)?; + let result = self.choose_and_compress( + Canonical::Extension(ext_array.clone()), + compress_ctx, + exec_ctx, + )?; if result.nbytes() < before_nbytes { return Ok(result); } + // TODO(connor): HACK TO SUPPORT L2 DENORMALIZATION!!! + if result.is::() { + return Ok(result); + } + // Otherwise, fall back to compressing the underlying storage array. - let compressed_storage = self.compress(ext_array.storage_array())?; + let compressed_storage = self.compress(ext_array.storage_array(), exec_ctx)?; Ok( ExtensionArray::new(ext_array.ext_dtype().clone(), compressed_storage) @@ -260,41 +259,36 @@ impl CascadingCompressor { /// The main scheme-selection entry point for a single leaf array. /// /// Filters allowed schemes by [`matches`] and exclusion rules, merges their [`stats_options`] - /// into a single [`GenerateStatsOptions`], then delegates to [`choose_scheme`] to pick the - /// winner by estimated compression ratio. + /// into a single [`GenerateStatsOptions`], and picks the winner by estimated compression + /// ratio. /// - /// If a winner is found and its compressed output is actually smaller, that output is returned. - /// Otherwise, the original array is returned unchanged. + /// If a winner is found and its compressed output is actually smaller, that output is + /// returned. Otherwise, the original array is returned unchanged. /// /// Empty and all-null arrays are short-circuited before any scheme evaluation. /// /// [`matches`]: Scheme::matches /// [`stats_options`]: Scheme::stats_options - /// [`choose_scheme`]: Self::choose_scheme fn choose_and_compress( &self, canonical: Canonical, - ctx: CompressorContext, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { let eligible_schemes: Vec<&'static dyn Scheme> = self .schemes .iter() .copied() - .filter(|s| s.matches(&canonical) && !self.is_excluded(*s, &ctx)) + .filter(|s| s.matches(&canonical) && !self.is_excluded(*s, &compress_ctx)) .collect(); let array: ArrayRef = canonical.into(); - // If there are no schemes that we can compress into, then just return it uncompressed. - if eligible_schemes.is_empty() { + if eligible_schemes.is_empty() || array.is_empty() { return Ok(array); } - // Nothing to compress if empty or all-null. - if array.is_empty() { - return Ok(array); - } - if array.all_invalid()? { + if array.all_invalid(exec_ctx)? { return Ok( ConstantArray::new(Scalar::null(array.dtype().clone()), array.len()).into_array(), ); @@ -307,91 +301,135 @@ impl CascadingCompressor { .fold(GenerateStatsOptions::default(), |acc, s| { acc.merge(s.stats_options()) }); - let ctx = ctx.with_merged_stats_options(merged_opts); - - let mut data = ArrayAndStats::new(array, merged_opts); - - if let Some(winner) = self.choose_best_scheme(&eligible_schemes, &mut data, ctx.clone())? { - // TODO(connor): Add a tracing warning here if compression with the chosen scheme - // failed, since there was likely more we could have done while choosing schemes. - - // Sampling and estimation chose a scheme, so let's compress the whole array with it. - let compressed = winner.compress(self, &mut data, ctx)?; - - // Only choose the compressed array if it is smaller than the canonical one. - if compressed.nbytes() < before_nbytes { - // TODO(connor): Add a tracing warning here too. - return Ok(compressed); - } + let compress_ctx = compress_ctx.with_merged_stats_options(merged_opts); + + let data = ArrayAndStats::new(array, merged_opts); + + let Some((winner, winner_estimate)) = + self.choose_best_scheme(&eligible_schemes, &data, compress_ctx.clone(), exec_ctx)? + else { + return Ok(data.into_array()); + }; + + // Run the winning scheme's `compress`. On failure, emit an ERROR event carrying the + // scheme name and cascade history before propagating. + let error_ctx = trace::enabled_error_context(&compress_ctx); + let _winner_span = trace::winner_compress_span(winner.id(), before_nbytes).entered(); + let compressed = winner + .compress(self, &data, compress_ctx, exec_ctx) + .inspect_err(|err| { + // NB: this is the only way we can tell which scheme panicked / bailed on their + // data, especially for third-party schemes where the error site may not carry any + // compressor context. + trace::scheme_compress_failed(winner.id(), before_nbytes, error_ctx.as_ref(), err); + })?; + + let after_nbytes = compressed.nbytes(); + let actual_ratio = (after_nbytes != 0).then(|| before_nbytes as f64 / after_nbytes as f64); + + // TODO(connor): HACK TO SUPPORT L2 DENORMALIZATION!!! + let accepted = after_nbytes < before_nbytes || compressed.is::(); + + trace::record_winner_compress_result( + after_nbytes, + winner_estimate.trace_ratio(), + actual_ratio, + accepted, + ); + + if accepted { + Ok(compressed) + } else { + Ok(data.into_array()) } - - // No scheme improved on the original. - Ok(data.into_array()) } - /// Calls [`expected_compression_ratio`] on each candidate and returns the scheme with the - /// highest ratio, or `None` if no scheme exceeds 1.0. Ties are broken by registration order - /// (earlier in the list wins). + /// Calls [`expected_compression_ratio`] on each candidate and returns the winning scheme along + /// with its resolved winner estimate, or `None` if no scheme beats the canonical encoding. + /// + /// Selection runs in two passes. Pass 1 evaluates every immediate + /// [`CompressionEstimate::Verdict`] and tracks the running best. [`Scheme`]s returning + /// [`CompressionEstimate::Deferred`] are stashed for pass 2 so that we do not make any + /// expensive computations if we don't have to. + /// + /// Pass 2 evaluates the deferred work and, for each [`DeferredEstimate::Callback`], passes the + /// current best [`EstimateScore`] as an early-exit hint so the callback can return + /// [`EstimateVerdict::Skip`] without doing expensive work when it cannot beat the threshold. + /// + /// Ties are broken by registration order within each pass. /// /// [`expected_compression_ratio`]: Scheme::expected_compression_ratio fn choose_best_scheme( &self, schemes: &[&'static dyn Scheme], - data: &mut ArrayAndStats, - ctx: CompressorContext, - ) -> VortexResult> { - let mut best: Option<(&'static dyn Scheme, f64)> = None; - - // TODO(connor): Might want to use an `im` data structure inside of `ctx` if the clones here - // are expensive. - for &scheme in schemes { - let estimate = scheme.expected_compression_ratio(data, ctx.clone()); - - match estimate { - CompressionEstimate::Skip => {} - CompressionEstimate::AlwaysUse => return Ok(Some(scheme)), - CompressionEstimate::Ratio(ratio) => { - if is_better_ratio(ratio, &best) { - best = Some((scheme, ratio)); + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, + ) -> VortexResult> { + let mut best: Option<(&'static dyn Scheme, EstimateScore)> = None; + let mut deferred: Vec<(&'static dyn Scheme, DeferredEstimate)> = Vec::new(); + + // Pass 1: evaluate every immediate verdict. Stash deferred work for pass 2. + { + let _verdict_pass = trace::verdict_pass_span().entered(); + for &scheme in schemes { + match scheme.expected_compression_ratio(data, compress_ctx.clone(), exec_ctx) { + CompressionEstimate::Verdict(EstimateVerdict::Skip) => {} + CompressionEstimate::Verdict(EstimateVerdict::AlwaysUse) => { + return Ok(Some((scheme, WinnerEstimate::AlwaysUse))); + } + CompressionEstimate::Verdict(EstimateVerdict::Ratio(ratio)) => { + let score = EstimateScore::FiniteCompression(ratio); + + if is_better_score(score, best.as_ref()) { + best = Some((scheme, score)); + } + } + CompressionEstimate::Deferred(deferred_estimate) => { + deferred.push((scheme, deferred_estimate)); } } - CompressionEstimate::Sample => { - let sample_ratio = estimate_compression_ratio_with_sampling( - scheme, + } + } + + // Pass 2: run deferred work. Callbacks receive the current best as a threshold so they can + // short-circuit with `Skip` when they cannot beat it. + for (scheme, deferred_estimate) in deferred { + let _span = trace::scheme_eval_span(scheme.id()).entered(); + let threshold: Option = best.map(|(_, score)| score); + match deferred_estimate { + DeferredEstimate::Sample => { + let score = estimate_compression_ratio_with_sampling( self, + scheme, data.array(), - ctx.clone(), + compress_ctx.clone(), + exec_ctx, )?; - if is_better_ratio(sample_ratio, &best) { - best = Some((scheme, sample_ratio)); + if is_better_score(score, best.as_ref()) { + best = Some((scheme, score)); } } - // TODO(connor): Is there a way to deduplicate some of this code? - CompressionEstimate::Estimate(estimate_callback) => { - let estimate = estimate_callback(self, data, ctx.clone())?; - - match estimate { - CompressionEstimate::Skip => {} - CompressionEstimate::AlwaysUse => return Ok(Some(scheme)), - CompressionEstimate::Ratio(ratio) => { - if is_better_ratio(ratio, &best) { - best = Some((scheme, ratio)); - } + DeferredEstimate::Callback(callback) => { + match callback(self, data, threshold, compress_ctx.clone(), exec_ctx)? { + EstimateVerdict::Skip => {} + EstimateVerdict::AlwaysUse => { + return Ok(Some((scheme, WinnerEstimate::AlwaysUse))); } - e @ (CompressionEstimate::Sample | CompressionEstimate::Estimate(_)) => { - vortex_panic!( - "an estimation function returned an invalid variant {e:?}" - ) + EstimateVerdict::Ratio(ratio) => { + let score = EstimateScore::FiniteCompression(ratio); + + if is_better_score(score, best.as_ref()) { + best = Some((scheme, score)); + } } } } } - - // tracing::debug!(scheme = %scheme.id(), estimate, "evaluated compression ratio"); } - Ok(best.map(|(s, _)| s)) + Ok(best.map(|(scheme, score)| (scheme, WinnerEstimate::Score(score)))) } // TODO(connor): Lots of room for optimization here. @@ -448,17 +486,25 @@ impl CascadingCompressor { fn compress_list_array( &self, list_array: ListArray, - ctx: CompressorContext, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { let list_array = list_array.reset_offsets(true)?; - let compressed_elems = self.compress(list_array.elements())?; + let compressed_elems = self.compress(list_array.elements(), exec_ctx)?; // Record the root scheme with the offsets child index so root exclusion rules apply. - let offset_ctx = ctx.descend_with_scheme(ROOT_SCHEME_ID, root_list_children::OFFSETS); + let offset_ctx = + compress_ctx.descend_with_scheme(ROOT_SCHEME_ID, root_list_children::OFFSETS); + let list_offsets_primitive = list_array + .offsets() + .clone() + .execute::(exec_ctx)? + .narrow()?; let compressed_offsets = self.compress_canonical( - Canonical::Primitive(list_array.offsets().to_primitive().narrow()?), + Canonical::Primitive(list_offsets_primitive), offset_ctx, + exec_ctx, )?; Ok( @@ -472,22 +518,35 @@ impl CascadingCompressor { fn compress_list_view_array( &self, list_view: ListViewArray, - ctx: CompressorContext, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult { - let compressed_elems = self.compress(list_view.elements())?; + let compressed_elems = self.compress(list_view.elements(), exec_ctx)?; - let offset_ctx = ctx + let offset_ctx = compress_ctx .clone() .descend_with_scheme(ROOT_SCHEME_ID, root_list_children::OFFSETS); + let list_view_offsets_primitive = list_view + .offsets() + .clone() + .execute::(exec_ctx)? + .narrow()?; let compressed_offsets = self.compress_canonical( - Canonical::Primitive(list_view.offsets().to_primitive().narrow()?), + Canonical::Primitive(list_view_offsets_primitive), offset_ctx, + exec_ctx, )?; - let sizes_ctx = ctx.descend_with_scheme(ROOT_SCHEME_ID, root_list_children::SIZES); + let sizes_ctx = compress_ctx.descend_with_scheme(ROOT_SCHEME_ID, root_list_children::SIZES); + let list_view_sizes_primitive = list_view + .sizes() + .clone() + .execute::(exec_ctx)? + .narrow()?; let compressed_sizes = self.compress_canonical( - Canonical::Primitive(list_view.sizes().to_primitive().narrow()?), + Canonical::Primitive(list_view_sizes_primitive), sizes_ctx, + exec_ctx, )?; Ok(ListViewArray::try_new( @@ -502,23 +561,279 @@ impl CascadingCompressor { #[cfg(test)] mod tests { + use std::sync::LazyLock; + + use parking_lot::Mutex; + use vortex_array::ArrayRef; + use vortex_array::Canonical; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; use vortex_array::arrays::Constant; + use vortex_array::arrays::NullArray; use vortex_array::arrays::PrimitiveArray; + use vortex_array::session::ArraySession; use vortex_array::validity::Validity; use vortex_buffer::buffer; + use vortex_session::VortexSession; use super::*; use crate::builtins::FloatDictScheme; use crate::builtins::IntDictScheme; use crate::builtins::StringDictScheme; use crate::ctx::CompressorContext; + use crate::estimate::CompressionEstimate; + use crate::estimate::DeferredEstimate; + use crate::estimate::EstimateScore; + use crate::estimate::EstimateVerdict; + use crate::estimate::WinnerEstimate; use crate::scheme::SchemeExt; + static SESSION: LazyLock = + LazyLock::new(|| VortexSession::empty().with::()); + fn compressor() -> CascadingCompressor { CascadingCompressor::new(vec![&IntDictScheme, &FloatDictScheme, &StringDictScheme]) } + fn estimate_test_data() -> ArrayAndStats { + let array = PrimitiveArray::new(buffer![1i32, 2, 3, 4], Validity::NonNullable).into_array(); + ArrayAndStats::new(array, GenerateStatsOptions::default()) + } + + fn matches_integer_primitive(canonical: &Canonical) -> bool { + matches!(canonical, Canonical::Primitive(primitive) if primitive.ptype().is_int()) + } + + #[derive(Debug)] + struct DirectRatioScheme; + + impl Scheme for DirectRatioScheme { + fn scheme_name(&self) -> &'static str { + "test.direct_ratio" + } + + fn matches(&self, canonical: &Canonical) -> bool { + matches_integer_primitive(canonical) + } + + fn expected_compression_ratio( + &self, + _data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, + ) -> CompressionEstimate { + CompressionEstimate::Verdict(EstimateVerdict::Ratio(2.0)) + } + + fn compress( + &self, + _compressor: &CascadingCompressor, + _data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, + ) -> VortexResult { + unreachable!("test helper should never be selected for compression") + } + } + + #[derive(Debug)] + struct ImmediateAlwaysUseScheme; + + impl Scheme for ImmediateAlwaysUseScheme { + fn scheme_name(&self) -> &'static str { + "test.immediate_always_use" + } + + fn matches(&self, canonical: &Canonical) -> bool { + matches_integer_primitive(canonical) + } + + fn expected_compression_ratio( + &self, + _data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, + ) -> CompressionEstimate { + CompressionEstimate::Verdict(EstimateVerdict::AlwaysUse) + } + + fn compress( + &self, + _compressor: &CascadingCompressor, + _data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, + ) -> VortexResult { + unreachable!("test helper should never be selected for compression") + } + } + + #[derive(Debug)] + struct CallbackAlwaysUseScheme; + + impl Scheme for CallbackAlwaysUseScheme { + fn scheme_name(&self) -> &'static str { + "test.callback_always_use" + } + + fn matches(&self, canonical: &Canonical) -> bool { + matches_integer_primitive(canonical) + } + + fn expected_compression_ratio( + &self, + _data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, + ) -> CompressionEstimate { + CompressionEstimate::Deferred(DeferredEstimate::Callback(Box::new( + |_compressor, _data, _ctx, _exec_ctx, _best_so_far| Ok(EstimateVerdict::AlwaysUse), + ))) + } + + fn compress( + &self, + _compressor: &CascadingCompressor, + _data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, + ) -> VortexResult { + unreachable!("test helper should never be selected for compression") + } + } + + #[derive(Debug)] + struct CallbackSkipScheme; + + impl Scheme for CallbackSkipScheme { + fn scheme_name(&self) -> &'static str { + "test.callback_skip" + } + + fn matches(&self, canonical: &Canonical) -> bool { + matches_integer_primitive(canonical) + } + + fn expected_compression_ratio( + &self, + _data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, + ) -> CompressionEstimate { + CompressionEstimate::Deferred(DeferredEstimate::Callback(Box::new( + |_compressor, _data, _ctx, _exec_ctx, _best_so_far| Ok(EstimateVerdict::Skip), + ))) + } + + fn compress( + &self, + _compressor: &CascadingCompressor, + _data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, + ) -> VortexResult { + unreachable!("test helper should never be selected for compression") + } + } + + #[derive(Debug)] + struct CallbackRatioScheme; + + impl Scheme for CallbackRatioScheme { + fn scheme_name(&self) -> &'static str { + "test.callback_ratio" + } + + fn matches(&self, canonical: &Canonical) -> bool { + matches_integer_primitive(canonical) + } + + fn expected_compression_ratio( + &self, + _data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, + ) -> CompressionEstimate { + CompressionEstimate::Deferred(DeferredEstimate::Callback(Box::new( + |_compressor, _data, _ctx, _exec_ctx, _best_so_far| Ok(EstimateVerdict::Ratio(3.0)), + ))) + } + + fn compress( + &self, + _compressor: &CascadingCompressor, + _data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, + ) -> VortexResult { + unreachable!("test helper should never be selected for compression") + } + } + + #[derive(Debug)] + struct HugeRatioScheme; + + impl Scheme for HugeRatioScheme { + fn scheme_name(&self) -> &'static str { + "test.huge_ratio" + } + + fn matches(&self, canonical: &Canonical) -> bool { + matches_integer_primitive(canonical) + } + + fn expected_compression_ratio( + &self, + _data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, + ) -> CompressionEstimate { + CompressionEstimate::Verdict(EstimateVerdict::Ratio(100.0)) + } + + fn compress( + &self, + _compressor: &CascadingCompressor, + _data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, + ) -> VortexResult { + unreachable!("test helper should never be selected for compression") + } + } + + #[derive(Debug)] + struct ZeroBytesSamplingScheme; + + impl Scheme for ZeroBytesSamplingScheme { + fn scheme_name(&self) -> &'static str { + "test.zero_bytes_sampling" + } + + fn matches(&self, canonical: &Canonical) -> bool { + matches_integer_primitive(canonical) + } + + fn expected_compression_ratio( + &self, + _data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, + ) -> CompressionEstimate { + CompressionEstimate::Deferred(DeferredEstimate::Sample) + } + + fn compress( + &self, + _compressor: &CascadingCompressor, + data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, + ) -> VortexResult { + Ok(NullArray::new(data.array().len()).into_array()) + } + } + #[test] fn test_self_exclusion() { let c = compressor(); @@ -568,6 +883,368 @@ mod tests { assert!(!c.is_excluded(&IntDictScheme, &ctx)); } + #[test] + fn immediate_always_use_wins_immediately() -> VortexResult<()> { + let compressor = + CascadingCompressor::new(vec![&DirectRatioScheme, &ImmediateAlwaysUseScheme]); + let schemes: [&'static dyn Scheme; 2] = [&DirectRatioScheme, &ImmediateAlwaysUseScheme]; + let data = estimate_test_data(); + let mut exec_ctx = SESSION.create_execution_ctx(); + + let winner = compressor.choose_best_scheme( + &schemes, + &data, + CompressorContext::new(), + &mut exec_ctx, + )?; + + assert!(matches!( + winner, + Some((scheme, WinnerEstimate::AlwaysUse)) + if scheme.id() == ImmediateAlwaysUseScheme.id() + )); + Ok(()) + } + + #[test] + fn callback_always_use_wins_immediately() -> VortexResult<()> { + let compressor = + CascadingCompressor::new(vec![&DirectRatioScheme, &CallbackAlwaysUseScheme]); + let schemes: [&'static dyn Scheme; 2] = [&DirectRatioScheme, &CallbackAlwaysUseScheme]; + let data = estimate_test_data(); + let mut exec_ctx = SESSION.create_execution_ctx(); + + let winner = compressor.choose_best_scheme( + &schemes, + &data, + CompressorContext::new(), + &mut exec_ctx, + )?; + + assert!(matches!( + winner, + Some((scheme, WinnerEstimate::AlwaysUse)) + if scheme.id() == CallbackAlwaysUseScheme.id() + )); + Ok(()) + } + + #[test] + fn callback_skip_is_ignored() -> VortexResult<()> { + let compressor = CascadingCompressor::new(vec![&CallbackSkipScheme, &DirectRatioScheme]); + let schemes: [&'static dyn Scheme; 2] = [&CallbackSkipScheme, &DirectRatioScheme]; + let data = estimate_test_data(); + let mut exec_ctx = SESSION.create_execution_ctx(); + + let winner = compressor.choose_best_scheme( + &schemes, + &data, + CompressorContext::new(), + &mut exec_ctx, + )?; + + assert!(matches!( + winner, + Some((scheme, WinnerEstimate::Score(EstimateScore::FiniteCompression(2.0)))) + if scheme.id() == DirectRatioScheme.id() + )); + Ok(()) + } + + #[test] + fn callback_ratio_competes_numerically() -> VortexResult<()> { + let compressor = CascadingCompressor::new(vec![&DirectRatioScheme, &CallbackRatioScheme]); + let schemes: [&'static dyn Scheme; 2] = [&DirectRatioScheme, &CallbackRatioScheme]; + let data = estimate_test_data(); + let mut exec_ctx = SESSION.create_execution_ctx(); + + let winner = compressor.choose_best_scheme( + &schemes, + &data, + CompressorContext::new(), + &mut exec_ctx, + )?; + + assert!(matches!( + winner, + Some((scheme, WinnerEstimate::Score(EstimateScore::FiniteCompression(3.0)))) + if scheme.id() == CallbackRatioScheme.id() + )); + Ok(()) + } + + #[test] + fn zero_byte_sample_loses_to_finite_ratio() -> VortexResult<()> { + let compressor = CascadingCompressor::new(vec![&HugeRatioScheme, &ZeroBytesSamplingScheme]); + let schemes: [&'static dyn Scheme; 2] = [&HugeRatioScheme, &ZeroBytesSamplingScheme]; + let data = estimate_test_data(); + let mut exec_ctx = SESSION.create_execution_ctx(); + + let winner = compressor.choose_best_scheme( + &schemes, + &data, + CompressorContext::new(), + &mut exec_ctx, + )?; + + assert!(matches!( + winner, + Some((scheme, WinnerEstimate::Score(EstimateScore::FiniteCompression(100.0)))) + if scheme.id() == HugeRatioScheme.id() + )); + Ok(()) + } + + #[test] + fn finite_ratio_displaces_zero_byte_sample() -> VortexResult<()> { + let compressor = CascadingCompressor::new(vec![&ZeroBytesSamplingScheme, &HugeRatioScheme]); + let schemes: [&'static dyn Scheme; 2] = [&ZeroBytesSamplingScheme, &HugeRatioScheme]; + let data = estimate_test_data(); + let mut exec_ctx = SESSION.create_execution_ctx(); + + let winner = compressor.choose_best_scheme( + &schemes, + &data, + CompressorContext::new(), + &mut exec_ctx, + )?; + + assert!(matches!( + winner, + Some((scheme, WinnerEstimate::Score(EstimateScore::FiniteCompression(100.0)))) + if scheme.id() == HugeRatioScheme.id() + )); + Ok(()) + } + + #[test] + fn zero_byte_sample_alone_selects_no_scheme() -> VortexResult<()> { + let compressor = CascadingCompressor::new(vec![&ZeroBytesSamplingScheme]); + let schemes: [&'static dyn Scheme; 1] = [&ZeroBytesSamplingScheme]; + let data = estimate_test_data(); + let mut exec_ctx = SESSION.create_execution_ctx(); + + let winner = compressor.choose_best_scheme( + &schemes, + &data, + CompressorContext::new(), + &mut exec_ctx, + )?; + + assert!(winner.is_none()); + Ok(()) + } + + // Observer helper used by threshold-related tests. Captures the `best_so_far` value the + // compressor passes to its deferred callback. `OBSERVER_LOCK` serializes tests that share + // `OBSERVED_THRESHOLD` so they do not race. + static OBSERVER_LOCK: Mutex<()> = Mutex::new(()); + static OBSERVED_THRESHOLD: Mutex>> = Mutex::new(None); + + #[derive(Debug)] + struct ThresholdObservingScheme; + + impl Scheme for ThresholdObservingScheme { + fn scheme_name(&self) -> &'static str { + "test.threshold_observing" + } + + fn matches(&self, canonical: &Canonical) -> bool { + matches_integer_primitive(canonical) + } + + fn expected_compression_ratio( + &self, + _data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, + ) -> CompressionEstimate { + CompressionEstimate::Deferred(DeferredEstimate::Callback(Box::new( + |_compressor, _data, best_so_far, _ctx, _exec_ctx| { + *OBSERVED_THRESHOLD.lock() = Some(best_so_far); + Ok(EstimateVerdict::Skip) + }, + ))) + } + + fn compress( + &self, + _compressor: &CascadingCompressor, + _data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, + ) -> VortexResult { + unreachable!("test helper should never be selected for compression") + } + } + + #[derive(Debug)] + struct CallbackMatchingRatioScheme; + + impl Scheme for CallbackMatchingRatioScheme { + fn scheme_name(&self) -> &'static str { + "test.callback_matching_ratio" + } + + fn matches(&self, canonical: &Canonical) -> bool { + matches_integer_primitive(canonical) + } + + fn expected_compression_ratio( + &self, + _data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, + ) -> CompressionEstimate { + CompressionEstimate::Deferred(DeferredEstimate::Callback(Box::new( + |_compressor, _data, _ctx, _exec_ctx, _best_so_far| Ok(EstimateVerdict::Ratio(2.0)), + ))) + } + + fn compress( + &self, + _compressor: &CascadingCompressor, + _data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, + ) -> VortexResult { + unreachable!("test helper should never be selected for compression") + } + } + + #[test] + fn callback_always_use_overrides_pass_one_best() -> VortexResult<()> { + // `HugeRatioScheme` returns an immediate `Ratio(100.0)` in pass 1; + // `CallbackAlwaysUseScheme` returns `AlwaysUse` from its deferred callback in pass 2. + // The deferred `AlwaysUse` must still win. + let compressor = CascadingCompressor::new(vec![&HugeRatioScheme, &CallbackAlwaysUseScheme]); + let schemes: [&'static dyn Scheme; 2] = [&HugeRatioScheme, &CallbackAlwaysUseScheme]; + let data = estimate_test_data(); + let mut exec_ctx = SESSION.create_execution_ctx(); + + let winner = compressor.choose_best_scheme( + &schemes, + &data, + CompressorContext::new(), + &mut exec_ctx, + )?; + + assert!(matches!( + winner, + Some((scheme, WinnerEstimate::AlwaysUse)) + if scheme.id() == CallbackAlwaysUseScheme.id() + )); + Ok(()) + } + + #[test] + fn threshold_reflects_pass_one_best() -> VortexResult<()> { + let _guard = OBSERVER_LOCK.lock(); + *OBSERVED_THRESHOLD.lock() = None; + + let compressor = + CascadingCompressor::new(vec![&DirectRatioScheme, &ThresholdObservingScheme]); + let schemes: [&'static dyn Scheme; 2] = [&DirectRatioScheme, &ThresholdObservingScheme]; + let data = estimate_test_data(); + let mut exec_ctx = SESSION.create_execution_ctx(); + + compressor.choose_best_scheme(&schemes, &data, CompressorContext::new(), &mut exec_ctx)?; + + let observed = *OBSERVED_THRESHOLD.lock(); + assert!(matches!( + observed, + Some(Some(EstimateScore::FiniteCompression(r))) if r == 2.0 + )); + Ok(()) + } + + #[test] + fn threshold_is_none_when_only_prior_is_zero_bytes() -> VortexResult<()> { + let _guard = OBSERVER_LOCK.lock(); + *OBSERVED_THRESHOLD.lock() = None; + + let compressor = + CascadingCompressor::new(vec![&ZeroBytesSamplingScheme, &ThresholdObservingScheme]); + let schemes: [&'static dyn Scheme; 2] = + [&ZeroBytesSamplingScheme, &ThresholdObservingScheme]; + let data = estimate_test_data(); + let mut exec_ctx = SESSION.create_execution_ctx(); + + compressor.choose_best_scheme(&schemes, &data, CompressorContext::new(), &mut exec_ctx)?; + + // The observing callback was invoked (outer `Some`) and `best_so_far` was `None` (inner + // `None`) because the zero-byte sample is never stored as the best. + let observed = *OBSERVED_THRESHOLD.lock(); + assert_eq!(observed, Some(None)); + Ok(()) + } + + #[test] + fn threshold_is_none_when_no_prior_scheme() -> VortexResult<()> { + let _guard = OBSERVER_LOCK.lock(); + *OBSERVED_THRESHOLD.lock() = None; + + let compressor = CascadingCompressor::new(vec![&ThresholdObservingScheme]); + let schemes: [&'static dyn Scheme; 1] = [&ThresholdObservingScheme]; + let data = estimate_test_data(); + let mut exec_ctx = SESSION.create_execution_ctx(); + + compressor.choose_best_scheme(&schemes, &data, CompressorContext::new(), &mut exec_ctx)?; + + let observed = *OBSERVED_THRESHOLD.lock(); + assert_eq!(observed, Some(None)); + Ok(()) + } + + #[test] + fn threshold_updates_from_earlier_deferred_callback() -> VortexResult<()> { + let _guard = OBSERVER_LOCK.lock(); + *OBSERVED_THRESHOLD.lock() = None; + + // Both schemes are deferred. The first callback registers `Ratio(3.0)`; the second + // callback must observe it as its threshold. + let compressor = + CascadingCompressor::new(vec![&CallbackRatioScheme, &ThresholdObservingScheme]); + let schemes: [&'static dyn Scheme; 2] = [&CallbackRatioScheme, &ThresholdObservingScheme]; + let data = estimate_test_data(); + let mut exec_ctx = SESSION.create_execution_ctx(); + + compressor.choose_best_scheme(&schemes, &data, CompressorContext::new(), &mut exec_ctx)?; + + let observed = *OBSERVED_THRESHOLD.lock(); + assert!(matches!( + observed, + Some(Some(EstimateScore::FiniteCompression(r))) if r == 3.0 + )); + Ok(()) + } + + #[test] + fn ratio_tie_between_immediate_and_deferred_favors_immediate() -> VortexResult<()> { + // Both schemes produce the same `Ratio(2.0)`, one from pass 1 (immediate) and one from + // pass 2 (deferred callback). Pass 1 locks in first, and strict `>` tie-breaking means + // the deferred callback's equal ratio cannot displace it. + let compressor = + CascadingCompressor::new(vec![&CallbackMatchingRatioScheme, &DirectRatioScheme]); + let schemes: [&'static dyn Scheme; 2] = [&CallbackMatchingRatioScheme, &DirectRatioScheme]; + let data = estimate_test_data(); + let mut exec_ctx = SESSION.create_execution_ctx(); + + let winner = compressor.choose_best_scheme( + &schemes, + &data, + CompressorContext::new(), + &mut exec_ctx, + )?; + + assert!(matches!( + winner, + Some((scheme, WinnerEstimate::Score(EstimateScore::FiniteCompression(r)))) + if scheme.id() == DirectRatioScheme.id() && r == 2.0 + )); + Ok(()) + } + #[test] fn all_null_array_compresses_to_constant() -> VortexResult<()> { let array = PrimitiveArray::new( @@ -579,7 +1256,8 @@ mod tests { // The compressor should produce a `ConstantArray` for an all-null array regardless of // which schemes are registered. let compressor = CascadingCompressor::new(vec![&IntDictScheme]); - let compressed = compressor.compress(&array)?; + let mut exec_ctx = SESSION.create_execution_ctx(); + let compressed = compressor.compress(&array, &mut exec_ctx)?; assert!(compressed.is::()); Ok(()) } @@ -608,9 +1286,15 @@ mod tests { // Before the fix this panicked with: // "this must be present since `DictScheme` declared that we need distinct values" - let ratio = - estimate_compression_ratio_with_sampling(&FloatDictScheme, &compressor, &array, ctx)?; - assert!(ratio.is_finite()); + let mut exec_ctx = SESSION.create_execution_ctx(); + let score = estimate_compression_ratio_with_sampling( + &compressor, + &FloatDictScheme, + &array, + ctx, + &mut exec_ctx, + )?; + assert!(matches!(score, EstimateScore::FiniteCompression(ratio) if ratio.is_finite())); Ok(()) } } diff --git a/vortex-compressor/src/ctx.rs b/vortex-compressor/src/ctx.rs index a488bef17bf..4e0619ff8ee 100644 --- a/vortex-compressor/src/ctx.rs +++ b/vortex-compressor/src/ctx.rs @@ -3,8 +3,11 @@ //! Compression context for recursive compression. +use std::fmt; + use vortex_error::VortexExpect; +use crate::compressor::ROOT_SCHEME_ID; use crate::scheme::SchemeId; use crate::stats::GenerateStatsOptions; @@ -27,6 +30,7 @@ pub struct CompressorContext { /// Merged stats options from all eligible schemes at this compression site. merged_stats_options: GenerateStatsOptions, + // TODO(connor): Replace this with an `im::Vector` /// The cascade chain: `(scheme_id, child_index)` pairs from root to current depth. /// Used for self-exclusion, push rules ([`descendant_exclusions`]), and pull rules /// ([`ancestor_exclusions`]). @@ -73,6 +77,16 @@ impl CompressorContext { &self.cascade_history } + /// Returns a display wrapper for the current cascade ancestry. + pub(crate) fn cascade_path(&self) -> impl fmt::Display + '_ { + CascadePath(&self.cascade_history) + } + + /// Returns the current cascade ancestry depth. + pub(crate) fn cascade_depth(&self) -> usize { + self.cascade_history.len() + } + /// Whether cascading is exhausted (no further cascade levels allowed). /// /// This should only be used in the implementation of a [`Scheme`](crate::scheme::Scheme) if the @@ -113,3 +127,28 @@ impl CompressorContext { self } } + +/// Display wrapper for a cascade ancestry path. +struct CascadePath<'a>(&'a [(SchemeId, usize)]); + +impl fmt::Display for CascadePath<'_> { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + if self.0.is_empty() { + return f.write_str("root"); + } + + for (index, (scheme_id, child_index)) in self.0.iter().enumerate() { + if index > 0 { + f.write_str(" > ")?; + } + + if *scheme_id == ROOT_SCHEME_ID { + write!(f, "root[{child_index}]")?; + } else { + write!(f, "{scheme_id}[{child_index}]")?; + } + } + + Ok(()) + } +} diff --git a/vortex-compressor/src/estimate.rs b/vortex-compressor/src/estimate.rs index b1e3ae3c659..70dd75d13b0 100644 --- a/vortex-compressor/src/estimate.rs +++ b/vortex-compressor/src/estimate.rs @@ -7,6 +7,7 @@ use std::fmt; use vortex_array::ArrayRef; use vortex_array::Canonical; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; use vortex_error::VortexResult; @@ -16,37 +17,57 @@ use crate::sample::SAMPLE_SIZE; use crate::sample::sample; use crate::sample::sample_count_approx_one_percent; use crate::scheme::Scheme; +use crate::scheme::SchemeExt; use crate::stats::ArrayAndStats; +use crate::trace; -/// Closure type for [`CompressionEstimate::Estimate`]. The compressor calls this with the same -/// arguments it would pass to sampling. +/// Closure type for [`DeferredEstimate::Callback`]. +/// +/// The compressor calls this with the same arguments it would pass to sampling, plus the best +/// [`EstimateScore`] observed so far (if any). The closure must resolve directly to a terminal +/// [`EstimateVerdict`]. +/// +/// The `best_so_far` threshold is an early-exit hint. If your scheme's maximum achievable +/// compression ratio is not strictly greater than +/// `best_so_far.and_then(EstimateScore::finite_ratio)`, you should return +/// [`EstimateVerdict::Skip`]. Returning a ratio equal to the threshold is permitted but will +/// lose to the prior best due to strict `>` tie-breaking in the selector. Use the threshold +/// only as an early-exit hint, never to perform additional work. #[rustfmt::skip] pub type EstimateFn = dyn FnOnce( &CascadingCompressor, - &mut ArrayAndStats, + &ArrayAndStats, + Option, CompressorContext, - ) -> VortexResult + &mut ExecutionCtx, + ) -> VortexResult + Send + Sync; -// TODO(connor): We should make use of the fact that some checks are cheap and some checks are -// expensive (sample or estimate variants). /// The result of a [`Scheme`]'s compression ratio estimation. /// /// This type is returned by [`Scheme::expected_compression_ratio`] to tell the compressor how /// promising this scheme is for a given array without performing any expensive work. /// -/// All expensive or fallible operations (sampling, trial encoding) are deferred to the compressor -/// via the [`Sample`](CompressionEstimate::Sample) and [`Estimate`](CompressionEstimate::Estimate) -/// variants. -/// -/// [`Sample`]: CompressionEstimate::Sample -/// [`Estimate`]: CompressionEstimate::Estimate +/// [`CompressionEstimate::Verdict`] means the scheme already knows the terminal answer. +/// [`CompressionEstimate::Deferred`] means the compressor must do extra work before the scheme can +/// produce a terminal answer. +#[derive(Debug)] pub enum CompressionEstimate { + /// The scheme already knows the terminal estimation verdict. + Verdict(EstimateVerdict), + + /// The compressor must perform deferred work to resolve the terminal estimation verdict. + Deferred(DeferredEstimate), +} + +/// The terminal answer to a compression estimate request. +#[derive(Debug)] +pub enum EstimateVerdict { /// Do not use this scheme for this array. Skip, - /// Always use this scheme, as we know it is definitively the best choice. + /// Always use this scheme, as it is definitively the best choice. /// /// Some examples include constant detection, decimal byte parts, and temporal decomposition. /// @@ -60,27 +81,110 @@ pub enum CompressionEstimate { /// The estimated compression ratio. This must be greater than `1.0` to be considered by the /// compressor, otherwise it is worse than the canonical encoding. Ratio(f64), +} +/// Deferred work that can resolve to a terminal [`EstimateVerdict`]. +pub enum DeferredEstimate { /// The scheme cannot cheaply estimate its ratio, so the compressor should compress a small /// sample to determine effectiveness. Sample, - /// A fallible estimation requiring a custom expensive computation. The compressor will call the - /// closure and handle the result. + /// A fallible estimation requiring a custom expensive computation. /// /// Use this only when the scheme needs to perform trial encoding or other costly checks to - /// determine its compression ratio. + /// determine its compression ratio. The callback returns an [`EstimateVerdict`] directly, so + /// it cannot request more sampling or another deferred callback. + /// + /// The compressor evaluates all immediate [`CompressionEstimate::Verdict`] results before + /// invoking any deferred callback, and passes the best [`EstimateScore`] observed so far to + /// the callback. This lets the callback return [`EstimateVerdict::Skip`] without performing + /// expensive work when its maximum achievable ratio cannot beat the current best. See + /// [`EstimateFn`] for the full contract. + Callback(Box), +} + +/// Ranked estimate used for comparing non-terminal compression candidates. +#[derive(Debug, Clone, Copy, PartialEq)] +pub enum EstimateScore { + /// A finite compression ratio. Higher means a smaller amount of data, so it is better. + FiniteCompression(f64), + /// Trial compression produced a 0-byte output. + /// + /// This has no finite ratio and is not eligible for scheme selection. /// - /// The estimation function must **not** return a [`Sample`](CompressionEstimate::Sample) or - /// [`Estimate`](CompressionEstimate::Estimate) variant to ensure the estimation process is - /// bounded. - Estimate(Box), + /// TODO(connor): A zero-byte sample usually means the sampler happened to hit an all-null + /// sample. Improve this logic so we can distinguish real zero-byte wins from sampling artifacts. + ZeroBytes, +} + +impl EstimateScore { + /// Converts measured sample sizes into a ranked estimate. + pub(super) fn from_sample_sizes(before_nbytes: u64, after_nbytes: u64) -> Self { + if after_nbytes == 0 { + Self::ZeroBytes + } else { + Self::FiniteCompression(before_nbytes as f64 / after_nbytes as f64) + } + } + + /// Returns the finite compression ratio, or [`None`] for the zero-byte special case. + /// + /// Callers comparing a scheme's maximum achievable ratio against a "best so far" threshold + /// should use this to extract a numeric value from an [`EstimateScore`]. + pub fn finite_ratio(self) -> Option { + match self { + Self::FiniteCompression(ratio) => Some(ratio), + Self::ZeroBytes => None, + } + } + + /// Returns whether this estimate is eligible to compete. + fn is_valid(self) -> bool { + match self { + Self::FiniteCompression(ratio) => { + ratio.is_finite() && !ratio.is_subnormal() && ratio > 1.0 + } + Self::ZeroBytes => false, + } + } + + /// Returns whether this estimate beats another valid estimate. + fn beats(self, other: Self) -> bool { + match (self, other) { + (Self::ZeroBytes, _) => false, + (Self::FiniteCompression(_), Self::ZeroBytes) => true, + (Self::FiniteCompression(ratio), Self::FiniteCompression(best_ratio)) => { + ratio > best_ratio + } + } + } +} + +/// Winner estimate carried from scheme selection into result tracing. +#[derive(Debug, Clone, Copy, PartialEq)] +pub(super) enum WinnerEstimate { + /// The scheme must be used immediately. + AlwaysUse, + /// The scheme won by a ranked estimate. + Score(EstimateScore), +} + +impl WinnerEstimate { + /// Returns the traceable numeric ratio for the winning estimate. + pub(super) fn trace_ratio(self) -> Option { + match self { + Self::AlwaysUse => None, + Self::Score(score) => score.finite_ratio(), + } + } } -/// Returns `true` if `ratio` is a valid compression ratio (> 1.0, finite, not subnormal) that -/// beats the current best. -pub(super) fn is_better_ratio(ratio: f64, best: &Option<(&'static dyn Scheme, f64)>) -> bool { - ratio.is_finite() && !ratio.is_subnormal() && ratio > 1.0 && best.is_none_or(|(_, r)| ratio > r) +/// Returns `true` if `score` beats the current best estimate. +pub(super) fn is_better_score( + score: EstimateScore, + best: Option<&(&'static dyn Scheme, EstimateScore)>, +) -> bool { + score.is_valid() && best.is_none_or(|(_, best_score)| score.beats(*best_score)) } /// Estimates compression ratio by compressing a ~1% sample of the data. @@ -92,60 +196,50 @@ pub(super) fn is_better_ratio(ratio: f64, best: &Option<(&'static dyn Scheme, f6 /// /// Returns an error if sample compression fails. pub(super) fn estimate_compression_ratio_with_sampling( - scheme: &S, compressor: &CascadingCompressor, + scheme: &S, array: &ArrayRef, - ctx: CompressorContext, -) -> VortexResult { - let sample_array = if ctx.is_sample() { + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, +) -> VortexResult { + let sample_array = if compress_ctx.is_sample() { array.clone() } else { - let source_len = array.len(); - let sample_count = sample_count_approx_one_percent(source_len); - - tracing::trace!( - "Sampling {} values out of {}", - SAMPLE_SIZE as u64 * sample_count as u64, - source_len - ); - + let sample_count = sample_count_approx_one_percent(array.len()); // `ArrayAndStats` expects a canonical array (so that it can easily compute lazy stats). - let canonical: Canonical = - sample(array, SAMPLE_SIZE, sample_count).execute(&mut compressor.execution_ctx())?; + let canonical: Canonical = sample(array, SAMPLE_SIZE, sample_count).execute(exec_ctx)?; canonical.into_array() }; - let mut sample_data = ArrayAndStats::new(sample_array, scheme.stats_options()); - let sample_ctx = ctx.with_sampling(); + let sample_data = ArrayAndStats::new(sample_array, scheme.stats_options()); + let error_ctx = trace::enabled_error_context(&compress_ctx); + let sample_ctx = compress_ctx.with_sampling(); - let after = scheme - .compress(compressor, &mut sample_data, sample_ctx)? - .nbytes(); - let before = sample_data.array().nbytes(); + let compressed = match scheme.compress(compressor, &sample_data, sample_ctx, exec_ctx) { + Ok(compressed) => compressed, + Err(err) => { + trace::sample_compress_failed(scheme.id(), error_ctx.as_ref(), &err); + return Err(err); + } + }; - // TODO(connor): Issue https://github.com/vortex-data/vortex/issues/7268. - // if after == 0 { - // tracing::warn!( - // scheme = %scheme.id(), - // "sample compressed to 0 bytes, which should only happen for constant arrays", - // ); - // } + let after = compressed.nbytes(); + let before = sample_data.array().nbytes(); - let ratio = before as f64 / after as f64; + let score = EstimateScore::from_sample_sizes(before, after); - tracing::debug!("estimate_compression_ratio_with_sampling(compressor={scheme:#?}) = {ratio}",); + if matches!(score, EstimateScore::ZeroBytes) { + trace::zero_byte_sample_result(scheme.id(), before); + } - Ok(ratio) + Ok(score) } -impl fmt::Debug for CompressionEstimate { +impl fmt::Debug for DeferredEstimate { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { match self { - CompressionEstimate::Skip => write!(f, "Skip"), - CompressionEstimate::AlwaysUse => write!(f, "AlwaysUse"), - CompressionEstimate::Ratio(r) => f.debug_tuple("Ratio").field(r).finish(), - CompressionEstimate::Sample => write!(f, "Sample"), - CompressionEstimate::Estimate(_) => write!(f, "Estimate(..)"), + DeferredEstimate::Sample => write!(f, "Sample"), + DeferredEstimate::Callback(_) => write!(f, "Callback(..)"), } } } diff --git a/vortex-compressor/src/lib.rs b/vortex-compressor/src/lib.rs index 65fd3f09c56..1ecc9e4d5b4 100644 --- a/vortex-compressor/src/lib.rs +++ b/vortex-compressor/src/lib.rs @@ -15,6 +15,26 @@ //! //! This crate contains no encoding dependencies. Batteries-included compressors are provided by //! downstream crates like `vortex-btrblocks`, which register different encodings to the compressor. +//! +//! # Observability +//! +//! The compressor emits a small set of `tracing` spans and events on a single target so you can +//! see what it's doing without attaching a profiler. +//! +//! For example, set `RUST_LOG=vortex_compressor::encode=debug` to see compression decision spans +//! and exceptional events. The `vortex_compressor::encode` target carries the top-level `compress` +//! span, per-scheme evaluation and winning-compression spans, the `cascade_exhausted` event, +//! `sample.result` events for zero-byte sample outputs, and both `*.compress_failed` events. +//! +//! The winning-compression span carries `scheme_chosen`, `input_nbytes`, `compressed_nbytes`, +//! `estimated_ratio` (absent when the scheme returned `AlwaysUse` or sampled to 0 bytes), +//! `achieved_ratio` (absent when the compressed output is 0 bytes), and `accepted`. +//! +//! Failure events additionally carry `cascade_path` and `cascade_depth`, so nested compression +//! errors can be tied back to the ancestor branch that triggered them. +//! +//! From those fields you can derive per-scheme savings, rejection counts, and estimator accuracy +//! with a short `jq` query. pub mod builtins; pub mod ctx; @@ -25,4 +45,5 @@ pub mod stats; mod sample; mod compressor; +mod trace; pub use compressor::CascadingCompressor; diff --git a/vortex-compressor/src/scheme.rs b/vortex-compressor/src/scheme.rs index 66b35051e95..4c5e7f0e940 100644 --- a/vortex-compressor/src/scheme.rs +++ b/vortex-compressor/src/scheme.rs @@ -10,6 +10,7 @@ use std::hash::Hasher; use vortex_array::ArrayRef; use vortex_array::Canonical; +use vortex_array::ExecutionCtx; use vortex_error::VortexResult; use crate::CascadingCompressor; @@ -20,10 +21,10 @@ use crate::stats::GenerateStatsOptions; /// Unique identifier for a compression scheme. /// -/// The only way to obtain a [`SchemeId`] is through [`SchemeExt::id()`], which is -/// auto-implemented for all [`Scheme`] types. There is no public constructor. +/// The only way to obtain a [`SchemeId`] is through [`SchemeExt::id()`], which is auto-implemented +/// for all [`Scheme`] types. There is no public constructor. /// -/// The only exception to this is for `ROOT_SCHEME_ID` in `compressor.rs`. +/// The only exception to this is for the compressor's synthetic `ROOT_SCHEME_ID`. #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] pub struct SchemeId { /// Only constructable within `vortex-compressor`. @@ -129,10 +130,10 @@ pub struct AncestorExclusion { /// /// # Implementing a scheme /// -/// [`expected_compression_ratio`] should return [`CompressionEstimate::Sample`] when a cheap -/// heuristic is not available, asking the compressor to estimate via sampling. Implementors should -/// return a more specific variant when possible (e.g. [`CompressionEstimate::AlwaysUse`] for -/// constant detection or [`CompressionEstimate::Skip`] for early rejection based on stats). +/// [`expected_compression_ratio`] should return +/// `CompressionEstimate::Deferred(DeferredEstimate::Sample)` when a cheap heuristic is not +/// available, asking the compressor to estimate via sampling. Implementors should return an +/// immediate [`CompressionEstimate::Verdict`] when possible. /// /// Schemes that need statistics that may be expensive to compute should override [`stats_options`] /// to declare what they require. The compressor merges all eligible schemes' options before @@ -184,14 +185,21 @@ pub trait Scheme: Debug + Send + Sync { /// Cheaply estimate the compression ratio for this scheme on the given array. /// - /// This method should be fast and infallible. Any expensive or fallible work should be deferred - /// to the compressor by returning [`CompressionEstimate::Sample`] or - /// [`CompressionEstimate::Estimate`]. + /// This method should be fast and infallible. Any expensive or fallible work should be + /// deferred to the compressor by returning + /// `CompressionEstimate::Deferred(DeferredEstimate::Sample)` or + /// `CompressionEstimate::Deferred(DeferredEstimate::Callback(...))`. /// /// The compressor will ask all schemes what their expected compression ratio is given the array /// and statistics. The scheme with the highest estimated ratio will then be applied to the /// entire array. /// + /// [`CompressionEstimate::Verdict`] means the scheme already knows the terminal + /// [`crate::estimate::EstimateVerdict`]. `CompressionEstimate::Deferred(DeferredEstimate::Sample)` + /// asks the compressor to sample. `CompressionEstimate::Deferred(DeferredEstimate::Callback(...))` + /// asks the compressor to run custom deferred work. Deferred callbacks must return a + /// [`crate::estimate::EstimateVerdict`] directly, never another deferred request. + /// /// Note that the compressor will also use this method when compressing samples, so some /// statistics that might hold for the samples may not hold for the entire array (e.g., /// `Constant`). Implementations should check `ctx.is_sample` to make sure that they are @@ -203,8 +211,9 @@ pub trait Scheme: Debug + Send + Sync { /// that are not all-null. fn expected_compression_ratio( &self, - _data: &mut ArrayAndStats, - _ctx: CompressorContext, + _data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, ) -> CompressionEstimate; /// Compress the array using this scheme. @@ -215,8 +224,9 @@ pub trait Scheme: Debug + Send + Sync { fn compress( &self, compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - ctx: CompressorContext, + data: &ArrayAndStats, + compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, ) -> VortexResult; } diff --git a/vortex-compressor/src/stats/bool.rs b/vortex-compressor/src/stats/bool.rs index 4f357f048aa..8825ec8a7f6 100644 --- a/vortex-compressor/src/stats/bool.rs +++ b/vortex-compressor/src/stats/bool.rs @@ -3,6 +3,7 @@ //! Bool compression statistics. +use vortex_array::ExecutionCtx; use vortex_array::arrays::BoolArray; use vortex_array::arrays::bool::BoolArrayExt; use vortex_error::VortexResult; @@ -25,7 +26,7 @@ impl BoolStats { /// # Errors /// /// Returns an error if getting validity mask fails or values exceed `u32` bounds. - pub fn generate(input: &BoolArray) -> VortexResult { + pub fn generate(input: &BoolArray, ctx: &mut ExecutionCtx) -> VortexResult { if input.is_empty() { return Ok(Self { null_count: 0, @@ -34,7 +35,7 @@ impl BoolStats { }); } - if input.all_invalid()? { + if input.all_invalid(ctx)? { return Ok(Self { null_count: u32::try_from(input.len())?, value_count: 0, @@ -42,7 +43,10 @@ impl BoolStats { }); } - let validity = input.validity_mask()?; + let validity = input + .as_ref() + .validity()? + .execute_mask(input.as_ref().len(), ctx)?; let null_count = validity.false_count(); let value_count = validity.true_count(); @@ -88,6 +92,8 @@ impl BoolStats { #[cfg(test)] mod tests { + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; use vortex_array::validity::Validity; use vortex_buffer::BitBuffer; @@ -97,11 +103,12 @@ mod tests { #[test] fn test_all_true() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = BoolArray::new( BitBuffer::from(vec![true, true, true]), Validity::NonNullable, ); - let stats = BoolStats::generate(&array)?; + let stats = BoolStats::generate(&array, &mut ctx)?; assert_eq!(stats.value_count, 3); assert_eq!(stats.null_count, 0); assert_eq!(stats.true_count, 3); @@ -111,11 +118,12 @@ mod tests { #[test] fn test_all_false() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = BoolArray::new( BitBuffer::from(vec![false, false, false]), Validity::NonNullable, ); - let stats = BoolStats::generate(&array)?; + let stats = BoolStats::generate(&array, &mut ctx)?; assert_eq!(stats.value_count, 3); assert_eq!(stats.null_count, 0); assert_eq!(stats.true_count, 0); @@ -125,11 +133,12 @@ mod tests { #[test] fn test_mixed() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = BoolArray::new( BitBuffer::from(vec![true, false, true]), Validity::NonNullable, ); - let stats = BoolStats::generate(&array)?; + let stats = BoolStats::generate(&array, &mut ctx)?; assert_eq!(stats.value_count, 3); assert_eq!(stats.null_count, 0); assert_eq!(stats.true_count, 2); @@ -139,11 +148,12 @@ mod tests { #[test] fn test_with_nulls() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = BoolArray::new( BitBuffer::from(vec![true, false, true]), Validity::from_iter([true, false, true]), ); - let stats = BoolStats::generate(&array)?; + let stats = BoolStats::generate(&array, &mut ctx)?; assert_eq!(stats.value_count, 2); assert_eq!(stats.null_count, 1); assert_eq!(stats.true_count, 2); diff --git a/vortex-compressor/src/stats/cache.rs b/vortex-compressor/src/stats/cache.rs index 4582f9e3b09..6f7020191a1 100644 --- a/vortex-compressor/src/stats/cache.rs +++ b/vortex-compressor/src/stats/cache.rs @@ -5,13 +5,15 @@ use std::any::Any; use std::any::TypeId; +use std::sync::Arc; +use parking_lot::Mutex; use vortex_array::ArrayRef; -use vortex_array::ToCanonical; +use vortex_array::ArrayView; +use vortex_array::ExecutionCtx; +use vortex_array::arrays::Bool; use vortex_array::arrays::Primitive; -use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::VarBinView; -use vortex_array::arrays::VarBinViewArray; use vortex_error::VortexExpect; use super::BoolStats; @@ -19,43 +21,50 @@ use super::FloatStats; use super::GenerateStatsOptions; use super::IntegerStats; use super::StringStats; +use crate::trace; + +/// A single cache entry: a concrete [`TypeId`] paired with a type-erased value. +type StatsEntry = (TypeId, Arc); /// Cache for compression statistics, keyed by concrete type. +/// +/// The cache is interior-mutable: entries can be inserted through a shared [`&StatsCache`] +/// borrow. Values are stored as [`Arc`] so that cached entries can be +/// cloned out of the lock cheaply and handed back to callers as [`Arc`]. struct StatsCache { // TODO(connor): We could further optimize this with a `SmallVec` here. /// The cache entries, keyed by [`TypeId`]. /// /// The total number of statistics types in this stats should be relatively small, so we use a /// vector instead of a hash map. - entries: Vec<(TypeId, Box)>, + entries: Arc>>, } impl StatsCache { /// Creates a new empty cache. fn new() -> Self { Self { - entries: Vec::new(), + entries: Arc::new(Mutex::new(Vec::new())), } } /// Returns a cached value, computing it on first access. - fn get_or_insert_with(&mut self, f: impl FnOnce() -> T) -> &T { + fn get_or_insert_with(&self, f: impl FnOnce() -> T) -> Arc { let type_id = TypeId::of::(); - let pos = self.entries.iter().position(|(id, _)| *id == type_id); + let mut guard = self.entries.lock(); - if let Some(pos) = pos { - self.entries[pos] - .1 - .downcast_ref::() + if let Some(pos) = guard.iter().position(|(id, _)| *id == type_id) { + Arc::clone(&guard[pos].1) + .downcast::() + .ok() .vortex_expect("we just checked the TypeID") } else { - self.entries.push((type_id, Box::new(f()))); - self.entries - .last() - .vortex_expect("just pushed") - .1 - .downcast_ref::() - .vortex_expect("we just checked the TypeID") + let new_arc: Arc = { + let _span = trace::generate_stats_span(std::any::type_name::()).entered(); + Arc::new(f()) + }; + guard.push((type_id, Arc::clone(&new_arc) as Arc)); + new_arc } } } @@ -66,10 +75,16 @@ impl StatsCache { /// FoR bias subtraction), it must create a new [`ArrayAndStats`] so that stale stats from the /// original array are not reused. /// -/// Built-in stats are accessed via typed methods (`integer_stats`, `float_stats`, `string_stats`) -/// which generate stats lazily on first access using the stored [`GenerateStatsOptions`]. +/// Built-in stats are accessed via typed methods ([`integer_stats`], [`float_stats`], +/// [`string_stats`]) which generate stats lazily on first access using the stored +/// [`GenerateStatsOptions`]. /// -/// Extension schemes can use `get_or_insert_with` for custom stats types. +/// Extension schemes can use [`get_or_insert_with`] for custom stats types. +/// +/// [`integer_stats`]: ArrayAndStats::integer_stats +/// [`float_stats`]: ArrayAndStats::float_stats +/// [`string_stats`]: ArrayAndStats::string_stats +/// [`get_or_insert_with`]: ArrayAndStats::get_or_insert_with pub struct ArrayAndStats { /// The array. This is always in canonical form. array: ArrayRef, @@ -105,30 +120,26 @@ impl ArrayAndStats { &self.array } - // TODO(connor): This should return an `ArrayView` once more vtable changes land. - /// Returns the array as a [`PrimitiveArray`]. + /// Returns the array as an [`ArrayView`]. /// /// # Panics /// /// Panics if the array is not a primitive array. - pub fn array_as_primitive(&self) -> PrimitiveArray { + pub fn array_as_primitive(&self) -> ArrayView<'_, Primitive> { self.array .as_opt::() .vortex_expect("the array is guaranteed to already be canonical by construction") - .into_owned() } - // TODO(connor): This should return an `ArrayView` once more vtable changes land. - /// Returns the array as a [`VarBinViewArray`]. + /// Returns the array as an [`ArrayView`]. /// /// # Panics /// /// Panics if the array is not a UTF-8 string array. - pub fn array_as_utf8(&self) -> VarBinViewArray { + pub fn array_as_utf8(&self) -> ArrayView<'_, VarBinView> { self.array .as_opt::() .vortex_expect("the array is guaranteed to already be canonical by construction") - .into_owned() } /// Consumes the bundle and returns the array. @@ -142,48 +153,58 @@ impl ArrayAndStats { } /// Returns bool stats, generating them lazily on first access. - pub fn bool_stats(&mut self) -> &BoolStats { + pub fn bool_stats(&self, ctx: &mut ExecutionCtx) -> Arc { let array = self.array.clone(); - self.cache.get_or_insert_with::(|| { - BoolStats::generate(&array.to_bool()).vortex_expect("BoolStats shouldn't fail") + let bool_array = array + .as_opt::() + .vortex_expect("the array is guaranteed to already be canonical by construction") + .into_owned(); + BoolStats::generate(&bool_array, ctx).vortex_expect("BoolStats shouldn't fail") }) } - // TODO(connor): These should all have interior mutability instead!!! - /// Returns integer stats, generating them lazily on first access. - pub fn integer_stats(&mut self) -> &IntegerStats { + pub fn integer_stats(&self, ctx: &mut ExecutionCtx) -> Arc { let array = self.array.clone(); let opts = self.opts; - self.cache.get_or_insert_with::(|| { - IntegerStats::generate_opts(&array.to_primitive(), opts) + let primitive = array + .as_opt::() + .vortex_expect("the array is guaranteed to already be canonical by construction") + .into_owned(); + IntegerStats::generate_opts(&primitive, opts, ctx) }) } /// Returns float stats, generating them lazily on first access. - pub fn float_stats(&mut self) -> &FloatStats { + pub fn float_stats(&self, ctx: &mut ExecutionCtx) -> Arc { let array = self.array.clone(); let opts = self.opts; - self.cache.get_or_insert_with::(|| { - FloatStats::generate_opts(&array.to_primitive(), opts) + let primitive = array + .as_opt::() + .vortex_expect("the array is guaranteed to already be canonical by construction") + .into_owned(); + FloatStats::generate_opts(&primitive, opts, ctx) }) } /// Returns string stats, generating them lazily on first access. - pub fn string_stats(&mut self) -> &StringStats { + pub fn string_stats(&self, ctx: &mut ExecutionCtx) -> Arc { let array = self.array.clone(); let opts = self.opts; - self.cache.get_or_insert_with::(|| { - StringStats::generate_opts(&array.to_varbinview(), opts) + let varbinview = array + .as_opt::() + .vortex_expect("the array is guaranteed to already be canonical by construction") + .into_owned(); + StringStats::generate_opts(&varbinview, opts, ctx) }) } /// For extension schemes with custom stats types. - pub fn get_or_insert_with(&mut self, f: impl FnOnce() -> T) -> &T { + pub fn get_or_insert_with(&self, f: impl FnOnce() -> T) -> Arc { self.cache.get_or_insert_with::(f) } } diff --git a/vortex-compressor/src/stats/float.rs b/vortex-compressor/src/stats/float.rs index c89de9c9893..d968e8d368f 100644 --- a/vortex-compressor/src/stats/float.rs +++ b/vortex-compressor/src/stats/float.rs @@ -8,6 +8,7 @@ use std::hash::Hash; use itertools::Itertools; use num_traits::Float; use rustc_hash::FxBuildHasher; +use vortex_array::ExecutionCtx; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::primitive::NativeValue; use vortex_array::dtype::NativePType; @@ -107,11 +108,12 @@ impl FloatStats { fn generate_opts_fallible( input: &PrimitiveArray, opts: GenerateStatsOptions, + ctx: &mut ExecutionCtx, ) -> VortexResult { match input.ptype() { - PType::F16 => typed_float_stats::(input, opts.count_distinct_values), - PType::F32 => typed_float_stats::(input, opts.count_distinct_values), - PType::F64 => typed_float_stats::(input, opts.count_distinct_values), + PType::F16 => typed_float_stats::(input, opts.count_distinct_values, ctx), + PType::F32 => typed_float_stats::(input, opts.count_distinct_values, ctx), + PType::F64 => typed_float_stats::(input, opts.count_distinct_values, ctx), _ => vortex_panic!("cannot generate FloatStats from ptype {}", input.ptype()), } } @@ -124,13 +126,17 @@ impl FloatStats { impl FloatStats { /// Generates stats with default options. - pub fn generate(input: &PrimitiveArray) -> Self { - Self::generate_opts(input, GenerateStatsOptions::default()) + pub fn generate(input: &PrimitiveArray, ctx: &mut ExecutionCtx) -> Self { + Self::generate_opts(input, GenerateStatsOptions::default(), ctx) } /// Generates stats with provided options. - pub fn generate_opts(input: &PrimitiveArray, opts: GenerateStatsOptions) -> Self { - Self::generate_opts_fallible(input, opts) + pub fn generate_opts( + input: &PrimitiveArray, + opts: GenerateStatsOptions, + ctx: &mut ExecutionCtx, + ) -> Self { + Self::generate_opts_fallible(input, opts, ctx) .vortex_expect("FloatStats::generate_opts should not fail") } @@ -159,6 +165,7 @@ impl FloatStats { fn typed_float_stats( array: &PrimitiveArray, count_distinct_values: bool, + ctx: &mut ExecutionCtx, ) -> VortexResult where NativeValue: Hash + Eq, @@ -174,18 +181,24 @@ where }); } - if array.all_invalid()? { + if array.all_invalid(ctx)? { return Ok(FloatStats { null_count: u32::try_from(array.len())?, value_count: 0, average_run_length: 0, - erased: TypedStats { distinct: None }.into(), + erased: TypedStats { + distinct: Some(DistinctInfo { + distinct_values: HashSet::with_capacity_and_hasher(0, FxBuildHasher), + distinct_count: 0, + }), + } + .into(), }); } let null_count = array .statistics() - .compute_null_count() + .compute_null_count(ctx) .ok_or_else(|| vortex_err!("Failed to compute null_count"))?; let value_count = array.len() - null_count; @@ -197,7 +210,10 @@ where HashSet::with_hasher(FxBuildHasher) }; - let validity = array.validity_mask()?; + let validity = array + .as_ref() + .validity()? + .execute_mask(array.as_ref().len(), ctx)?; let mut runs = 1; let head_idx = validity @@ -260,33 +276,39 @@ where #[cfg(test)] mod tests { use vortex_array::IntoArray; - use vortex_array::ToCanonical; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::validity::Validity; use vortex_buffer::buffer; + use vortex_error::VortexResult; use super::FloatStats; #[test] - fn test_float_stats() { + fn test_float_stats() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let floats = buffer![0.0f32, 1.0f32, 2.0f32].into_array(); - let floats = floats.to_primitive(); + let floats = floats.execute::(&mut ctx)?; let stats = FloatStats::generate_opts( &floats, crate::stats::GenerateStatsOptions { count_distinct_values: true, }, + &mut ctx, ); assert_eq!(stats.value_count, 3); assert_eq!(stats.null_count, 0); assert_eq!(stats.average_run_length, 1); assert_eq!(stats.distinct_count().unwrap(), 3); + Ok(()) } #[test] fn test_float_stats_leading_nulls() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let floats = PrimitiveArray::new( buffer![0.0f32, 1.0f32, 2.0f32], Validity::from_iter([false, true, true]), @@ -297,6 +319,7 @@ mod tests { crate::stats::GenerateStatsOptions { count_distinct_values: true, }, + &mut ctx, ); assert_eq!(stats.value_count, 2); diff --git a/vortex-compressor/src/stats/integer.rs b/vortex-compressor/src/stats/integer.rs index b1ff843b667..64345ac5f06 100644 --- a/vortex-compressor/src/stats/integer.rs +++ b/vortex-compressor/src/stats/integer.rs @@ -7,6 +7,7 @@ use std::hash::Hash; use num_traits::PrimInt; use rustc_hash::FxBuildHasher; +use vortex_array::ExecutionCtx; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::primitive::NativeValue; use vortex_array::dtype::IntegerPType; @@ -255,9 +256,10 @@ impl IntegerStats { fn generate_opts_fallible( input: &PrimitiveArray, opts: GenerateStatsOptions, + ctx: &mut ExecutionCtx, ) -> VortexResult { match_each_integer_ptype!(input.ptype(), |T| { - typed_int_stats::(input, opts.count_distinct_values) + typed_int_stats::(input, opts.count_distinct_values, ctx) }) } @@ -274,13 +276,17 @@ impl IntegerStats { impl IntegerStats { /// Generates stats with default options. - pub fn generate(input: &PrimitiveArray) -> Self { - Self::generate_opts(input, GenerateStatsOptions::default()) + pub fn generate(input: &PrimitiveArray, ctx: &mut ExecutionCtx) -> Self { + Self::generate_opts(input, GenerateStatsOptions::default(), ctx) } /// Generates stats with provided options. - pub fn generate_opts(input: &PrimitiveArray, opts: GenerateStatsOptions) -> Self { - Self::generate_opts_fallible(input, opts) + pub fn generate_opts( + input: &PrimitiveArray, + opts: GenerateStatsOptions, + ctx: &mut ExecutionCtx, + ) -> Self { + Self::generate_opts_fallible(input, opts, ctx) .vortex_expect("IntegerStats::generate_opts should not fail") } @@ -309,6 +315,7 @@ impl IntegerStats { fn typed_int_stats( array: &PrimitiveArray, count_distinct_values: bool, + ctx: &mut ExecutionCtx, ) -> VortexResult where T: IntegerPType + PrimInt + for<'a> TryFrom<&'a Scalar, Error = VortexError>, @@ -330,7 +337,7 @@ where }); } - if array.all_invalid()? { + if array.all_invalid(ctx)? { return Ok(IntegerStats { null_count: u32::try_from(array.len())?, value_count: 0, @@ -338,13 +345,21 @@ where erased: TypedStats { min: T::max_value(), max: T::min_value(), - distinct: None, + distinct: Some(DistinctInfo { + distinct_values: HashMap::with_capacity_and_hasher(0, FxBuildHasher), + distinct_count: 0, + most_frequent_value: T::zero(), + top_frequency: 0, + }), } .into(), }); } - let validity = array.validity_mask()?; + let validity = array + .as_ref() + .validity()? + .execute_mask(array.as_ref().len(), ctx)?; let null_count = validity.false_count(); let value_count = validity.true_count(); @@ -426,12 +441,12 @@ where let array_ref = array.as_ref(); let min = array_ref .statistics() - .compute_as::(Stat::Min) + .compute_as::(Stat::Min, ctx) .vortex_expect("min should be computed"); let max = array_ref .statistics() - .compute_as::(Stat::Max) + .compute_as::(Stat::Max, ctx) .vortex_expect("max should be computed"); let distinct = count_distinct_values.then(|| { @@ -546,6 +561,8 @@ fn inner_loop_naive( mod tests { use std::iter; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::validity::Validity; use vortex_buffer::BitBuffer; @@ -558,46 +575,51 @@ mod tests { #[test] fn test_naive_count_distinct_values() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = PrimitiveArray::new(buffer![217u8, 0], Validity::NonNullable); - let stats = typed_int_stats::(&array, true)?; + let stats = typed_int_stats::(&array, true, &mut ctx)?; assert_eq!(stats.distinct_count().unwrap(), 2); Ok(()) } #[test] fn test_naive_count_distinct_values_nullable() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = PrimitiveArray::new( buffer![217u8, 0], Validity::from(BitBuffer::from(vec![true, false])), ); - let stats = typed_int_stats::(&array, true)?; + let stats = typed_int_stats::(&array, true, &mut ctx)?; assert_eq!(stats.distinct_count().unwrap(), 1); Ok(()) } #[test] fn test_count_distinct_values() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = PrimitiveArray::new((0..128u8).collect::>(), Validity::NonNullable); - let stats = typed_int_stats::(&array, true)?; + let stats = typed_int_stats::(&array, true, &mut ctx)?; assert_eq!(stats.distinct_count().unwrap(), 128); Ok(()) } #[test] fn test_count_distinct_values_nullable() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let array = PrimitiveArray::new( (0..128u8).collect::>(), Validity::from(BitBuffer::from_iter( iter::repeat_n(vec![true, false], 64).flatten(), )), ); - let stats = typed_int_stats::(&array, true)?; + let stats = typed_int_stats::(&array, true, &mut ctx)?; assert_eq!(stats.distinct_count().unwrap(), 64); Ok(()) } #[test] fn test_integer_stats_leading_nulls() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let ints = PrimitiveArray::new(buffer![0, 1, 2], Validity::from_iter([false, true, true])); let stats = IntegerStats::generate_opts( @@ -605,6 +627,7 @@ mod tests { crate::stats::GenerateStatsOptions { count_distinct_values: true, }, + &mut ctx, ); assert_eq!(stats.value_count, 2); diff --git a/vortex-compressor/src/stats/string.rs b/vortex-compressor/src/stats/string.rs index d35d8381611..8613aa5cc37 100644 --- a/vortex-compressor/src/stats/string.rs +++ b/vortex-compressor/src/stats/string.rs @@ -3,6 +3,7 @@ //! String compression statistics. +use vortex_array::ExecutionCtx; use vortex_array::arrays::VarBinViewArray; use vortex_error::VortexExpect; use vortex_error::VortexResult; @@ -47,10 +48,11 @@ impl StringStats { fn generate_opts_fallible( input: &VarBinViewArray, opts: GenerateStatsOptions, + ctx: &mut ExecutionCtx, ) -> VortexResult { let null_count = input .statistics() - .compute_null_count() + .compute_null_count(ctx) .ok_or_else(|| vortex_err!("Failed to compute null_count"))?; let value_count = input.len() - null_count; let estimated_distinct_count = opts @@ -68,13 +70,17 @@ impl StringStats { impl StringStats { /// Generates stats with default options. - pub fn generate(input: &VarBinViewArray) -> Self { - Self::generate_opts(input, GenerateStatsOptions::default()) + pub fn generate(input: &VarBinViewArray, ctx: &mut ExecutionCtx) -> Self { + Self::generate_opts(input, GenerateStatsOptions::default(), ctx) } /// Generates stats with provided options. - pub fn generate_opts(input: &VarBinViewArray, opts: GenerateStatsOptions) -> Self { - Self::generate_opts_fallible(input, opts) + pub fn generate_opts( + input: &VarBinViewArray, + opts: GenerateStatsOptions, + ctx: &mut ExecutionCtx, + ) -> Self { + Self::generate_opts_fallible(input, opts, ctx) .vortex_expect("StringStats::generate_opts should not fail") } diff --git a/vortex-compressor/src/trace.rs b/vortex-compressor/src/trace.rs new file mode 100644 index 00000000000..84027272b31 --- /dev/null +++ b/vortex-compressor/src/trace.rs @@ -0,0 +1,196 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Internal tracing helpers for compressor observability. + +use std::fmt; + +use crate::ctx::CompressorContext; +use crate::scheme::SchemeId; + +/// Shared tracing target for compressor decisions and coarse cascade structure. +pub(super) const TARGET_TRACE: &str = "vortex_compressor::encode"; + +/// Builds the top-level compression span. +/// +/// `input_nbytes` is known up front; `compressed_nbytes` / `compression_ratio` are filled in by +/// [`record_compress_outcome`] once the cascade returns. +#[inline] +pub(super) fn compress_span( + len: usize, + dtype: &impl fmt::Display, + before_nbytes: u64, +) -> tracing::Span { + tracing::debug_span!( + target: TARGET_TRACE, + "compress", + array_len = len, + dtype = %dtype, + input_nbytes = before_nbytes, + compressed_nbytes = tracing::field::Empty, + compression_ratio = tracing::field::Empty, + ) +} + +/// Builds a span covering on-demand materialization of a cached stats type. +/// +/// Child of whatever span is active when a stats accessor first fires. Typically that's +/// [`verdict_pass_span`]; entering this span disambiguates stats cost from the rest of Pass 1. +/// `kind` is usually `std::any::type_name::()` so the args identify which group was generated +/// (e.g. `IntegerStats`, `FloatStats`). +#[inline] +pub(super) fn generate_stats_span(kind: &'static str) -> tracing::Span { + tracing::debug_span!( + target: TARGET_TRACE, + "generate_stats", + stats_kind = kind, + ) +} + +/// Builds a span covering Pass 1 of scheme selection (the cheap-verdict pass). +/// +/// Stats batches merged across eligible schemes are materialized lazily by the first +/// `expected_compression_ratio` call that touches them. Grouping those calls under one span makes +/// the stats cost (and unexpectedly slow verdicts) visible independently of per-candidate sampling. +#[inline] +pub(super) fn verdict_pass_span() -> tracing::Span { + tracing::debug_span!( + target: TARGET_TRACE, + "verdict_pass", + ) +} + +/// Builds a span covering one deferred per-scheme evaluation (sample or callback). +/// +/// `scheme_candidate` is the scheme being evaluated, not necessarily chosen. +#[inline] +pub(super) fn scheme_eval_span(scheme: SchemeId) -> tracing::Span { + tracing::debug_span!( + target: TARGET_TRACE, + "scheme_eval", + scheme_candidate = %scheme, + ) +} + +/// Emits the sampling result event for zero-byte sample outputs. +#[inline] +pub(super) fn zero_byte_sample_result(scheme: SchemeId, sampled_before: u64) { + tracing::debug!( + target: TARGET_TRACE, + scheme = %scheme, + sampled_before, + sampled_after = 0_u64, + "sample.result", + ); +} + +/// Builds a span covering the winning scheme's full-array compression. +/// +/// `scheme_chosen` and `input_nbytes` are known up front. `compressed_nbytes`, +/// `estimated_ratio`, `achieved_ratio`, and `accepted` are filled in by +/// [`record_winner_compress_result`] once the encode completes. +#[inline] +pub(super) fn winner_compress_span(scheme: SchemeId, before_nbytes: u64) -> tracing::Span { + tracing::debug_span!( + target: TARGET_TRACE, + "winner_compress", + scheme_chosen = %scheme, + input_nbytes = before_nbytes, + compressed_nbytes = tracing::field::Empty, + estimated_ratio = tracing::field::Empty, + achieved_ratio = tracing::field::Empty, + accepted = tracing::field::Empty, + ) +} + +/// Records the outcome of a winning-scheme compression on the current `winner_compress` span. +#[inline] +pub(super) fn record_winner_compress_result( + compressed_nbytes: u64, + estimated_ratio: Option, + achieved_ratio: Option, + accepted: bool, +) { + let span = tracing::Span::current(); + span.record("compressed_nbytes", compressed_nbytes); + if let Some(r) = estimated_ratio { + span.record("estimated_ratio", r); + } + if let Some(r) = achieved_ratio { + span.record("achieved_ratio", r); + } + span.record("accepted", accepted); +} + +/// Records the final output size and, when finite, the top-level compression ratio. +#[inline] +pub(super) fn record_compress_outcome( + span: &tracing::Span, + input_nbytes: u64, + compressed_nbytes: u64, +) { + span.record("compressed_nbytes", compressed_nbytes); + if compressed_nbytes != 0 { + span.record( + "compression_ratio", + input_nbytes as f64 / compressed_nbytes as f64, + ); + } +} + +/// Emits the MAX_CASCADE short-circuit event. +#[inline] +pub(super) fn cascade_exhausted(parent: SchemeId, child_index: usize) { + tracing::debug!( + target: TARGET_TRACE, + parent = %parent, + child_index, + "cascade_exhausted", + ); +} + +/// Captures the context needed for error tracing only when ERROR logs are enabled. +#[inline] +pub(super) fn enabled_error_context(ctx: &CompressorContext) -> Option { + tracing::enabled!(target: TARGET_TRACE, tracing::Level::ERROR).then(|| ctx.clone()) +} + +/// Emits a compression-failure event for a winning scheme. +#[inline] +pub(super) fn scheme_compress_failed( + scheme: SchemeId, + before_nbytes: u64, + ctx: Option<&CompressorContext>, + err: &impl fmt::Display, +) { + if let Some(ctx) = ctx { + tracing::error!( + target: TARGET_TRACE, + scheme = %scheme, + before_nbytes, + cascade_path = %ctx.cascade_path(), + cascade_depth = ctx.cascade_depth(), + error = %err, + "scheme.compress_failed", + ); + } +} + +/// Emits a sampling-failure event. +#[inline] +pub(super) fn sample_compress_failed( + scheme: SchemeId, + ctx: Option<&CompressorContext>, + err: &impl fmt::Display, +) { + if let Some(ctx) = ctx { + tracing::error!( + target: TARGET_TRACE, + scheme = %scheme, + cascade_path = %ctx.cascade_path(), + cascade_depth = ctx.cascade_depth(), + error = %err, + "sample.compress_failed", + ); + } +} diff --git a/vortex-cuda/Cargo.toml b/vortex-cuda/Cargo.toml index 9d21976f92b..a58c1c379ce 100644 --- a/vortex-cuda/Cargo.toml +++ b/vortex-cuda/Cargo.toml @@ -30,6 +30,7 @@ cudarc = { workspace = true, features = ["f16"] } futures = { workspace = true, features = ["executor"] } itertools = { workspace = true } kanal = { workspace = true } +num-traits = { workspace = true } object_store = { workspace = true, features = ["fs"] } parking_lot = { workspace = true } prost = { workspace = true } @@ -83,13 +84,13 @@ name = "bitpacked_cuda" harness = false [[bench]] -name = "dynamic_dispatch_cuda" +name = "alp_cuda" harness = false [[bench]] -name = "throughput_cuda" +name = "dynamic_dispatch_cuda" harness = false [[bench]] -name = "transpose_patches" +name = "throughput_cuda" harness = false diff --git a/vortex-cuda/benches/alp_cuda.rs b/vortex-cuda/benches/alp_cuda.rs new file mode 100644 index 00000000000..af9006cc3a8 --- /dev/null +++ b/vortex-cuda/benches/alp_cuda.rs @@ -0,0 +1,147 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! CUDA benchmarks for ALP decompression. + +#![expect(clippy::unwrap_used)] +#![expect(clippy::cast_possible_truncation)] + +mod bench_config; +mod timed_launch_strategy; + +use std::mem::size_of; +use std::sync::Arc; +use std::sync::atomic::Ordering; +use std::time::Duration; + +use criterion::BenchmarkId; +use criterion::Criterion; +use criterion::Throughput; +use cudarc::driver::DeviceRepr; +use futures::executor::block_on; +use vortex::array::IntoArray; +use vortex::array::LEGACY_SESSION; +use vortex::array::VortexSessionExecute; +use vortex::array::arrays::PrimitiveArray; +use vortex::array::validity::Validity::NonNullable; +use vortex::buffer::Buffer; +use vortex::dtype::NativePType; +use vortex::encodings::alp::ALPArray; +use vortex::encodings::alp::ALPArrayExt; +use vortex::encodings::alp::ALPFloat; +use vortex::encodings::alp::alp_encode; +use vortex::error::VortexExpect; +use vortex::session::VortexSession; +use vortex_cuda::CudaDispatchMode; +use vortex_cuda::CudaSession; +use vortex_cuda::executor::CudaArrayExt; +use vortex_cuda_macros::cuda_available; +use vortex_cuda_macros::cuda_not_available; + +use crate::bench_config::BENCH_SIZES; +use crate::timed_launch_strategy::TimedLaunchStrategy; + +/// Patch frequencies to benchmark (as fractions). +const PATCH_FREQUENCIES: &[(f64, &str)] = &[(0.0, "0%"), (0.01, "1%"), (0.10, "10%")]; + +/// Create an ALP-encoded array of `len` floats with the requested patch frequency. +/// +/// Base values are integer-valued floats which encode cleanly. When `patch_frequency > 0`, +/// PI is placed at regular intervals; PI cannot round-trip through ALP at the chosen +/// exponents, so each PI becomes a patch. +fn make_alp_array(len: usize, patch_frequency: f64) -> ALPArray +where + T: ALPFloat + NativePType, +{ + let patch_interval = if patch_frequency > 0.0 { + (1.0 / patch_frequency) as usize + } else { + usize::MAX + }; + let outlier = T::from(std::f64::consts::PI).unwrap(); + + let values: Buffer = (0..len) + .map(|i| { + if patch_interval != usize::MAX && i % patch_interval == 0 { + outlier + } else { + T::from((i % 256) as u32).unwrap() + } + }) + .collect(); + + let primitive_array = PrimitiveArray::new(values, NonNullable); + let encoded = alp_encode( + primitive_array.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .vortex_expect("failed to ALP-encode array"); + + if patch_frequency > 0.0 { + assert!( + encoded.patches().is_some(), + "expected patches for patch_frequency={patch_frequency}", + ); + } + + encoded +} + +fn benchmark_alp_decode_typed(c: &mut Criterion, type_name: &str) +where + T: ALPFloat + NativePType + DeviceRepr, +{ + let mut group = c.benchmark_group(format!("cuda/alp_{}", type_name)); + + for &(len, len_str) in BENCH_SIZES { + group.throughput(Throughput::Bytes((len * size_of::()) as u64)); + + for &(patch_freq, patch_label) in PATCH_FREQUENCIES { + let array = make_alp_array::(len, patch_freq); + + group.bench_with_input( + BenchmarkId::new(patch_label, len_str), + &array, + |b, array| { + b.iter_custom(|iters| { + let timed = TimedLaunchStrategy::default(); + let timer = timed.timer(); + + let mut cuda_ctx = + CudaSession::create_execution_ctx(&VortexSession::empty()) + .vortex_expect("failed to create execution context") + .with_dispatch_mode(CudaDispatchMode::StandaloneOnly) + .with_launch_strategy(Arc::new(timed)); + + for _ in 0..iters { + block_on(array.clone().into_array().execute_cuda(&mut cuda_ctx)) + .unwrap(); + } + + Duration::from_nanos(timer.load(Ordering::Relaxed)) + }); + }, + ); + } + } + + group.finish(); +} + +fn benchmark_alp_decode(c: &mut Criterion) { + benchmark_alp_decode_typed::(c, "f32"); + benchmark_alp_decode_typed::(c, "f64"); +} + +criterion::criterion_group! { + name = benches; + config = bench_config::cuda_bench_config(); + targets = benchmark_alp_decode +} + +#[cuda_available] +criterion::criterion_main!(benches); + +#[cuda_not_available] +fn main() {} diff --git a/vortex-cuda/benches/bench_config/mod.rs b/vortex-cuda/benches/bench_config/mod.rs new file mode 100644 index 00000000000..c8afa537fab --- /dev/null +++ b/vortex-cuda/benches/bench_config/mod.rs @@ -0,0 +1,48 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +use std::time::Duration; + +use criterion::Criterion; + +/// Benchmark input sizes. +/// +/// On codspeed, only the 100M variant runs — kernels under ~200 µs +/// (i.e. the 10M cases) swing 15-45% across ephemeral GPU instances, +/// drowning real regressions in noise. Locally both sizes run. +#[cfg(not(codspeed))] +pub const BENCH_SIZES: &[(usize, &str)] = &[(10_000_000, "10M"), (100_000_000, "100M")]; +#[cfg(codspeed)] +pub const BENCH_SIZES: &[(usize, &str)] = &[(100_000_000, "100M")]; + +/// Returns a [`Criterion`] configuration tuned for CUDA benchmarks. +/// +/// All benchmarks use `iter_custom` with precise CUDA event timing. +/// criterion's iteration planner estimates `iters` from **wall time** during +/// warmup, which includes GPU context setup and memory copies — not just +/// the kernel. Setting `measurement_time = 1ns` forces `iters = 1` so +/// each sample is exactly one `iter_custom` call returning GPU-timed duration. +/// Stability comes from a high `sample_size` (many independent launches) +/// rather than many iterations per sample. +/// +/// `warm_up_time` is set to 500 ms — long enough to JIT-compile PTX, +/// boost GPU clocks, and warm caches, while keeping total runtime under +/// 2 minutes even for the largest benchmark binary (~18 benchmarks). +/// +/// `sample_size` is 10: with 100M inputs the kernels are long enough +/// (>500 µs) that within-run variance is low. Cross-run stability +/// comes from the large input size, not from averaging many samples. +pub(super) fn cuda_bench_config() -> Criterion { + let sample_size = 10; + + Criterion::default() + .without_plots() + .sample_size(sample_size) + // Enough for PTX JIT, GPU clock boost, and cache warming. + .warm_up_time(Duration::from_millis(500)) + // Forces `iters = 1`: criterion's planner estimates iteration cost + // from wall time (which includes GPU context setup), not the + // GPU-timed duration returned by `iter_custom`. A real + // measurement_time would cause wildly inflated iteration counts. + .measurement_time(Duration::from_nanos(1)) +} diff --git a/vortex-cuda/benches/bitpacked_cuda.rs b/vortex-cuda/benches/bitpacked_cuda.rs index a0cc4985ea5..8ecf7718f15 100644 --- a/vortex-cuda/benches/bitpacked_cuda.rs +++ b/vortex-cuda/benches/bitpacked_cuda.rs @@ -3,10 +3,11 @@ //! CUDA benchmarks for bit unpacking. -#![allow(clippy::unwrap_used)] -#![allow(clippy::cast_possible_truncation)] +#![expect(clippy::unwrap_used)] +#![expect(clippy::cast_possible_truncation)] -mod common; +mod bench_config; +mod timed_launch_strategy; use std::mem::size_of; use std::ops::Add; @@ -20,6 +21,8 @@ use criterion::Throughput; use cudarc::driver::DeviceRepr; use futures::executor::block_on; use vortex::array::IntoArray; +use vortex::array::LEGACY_SESSION; +use vortex::array::VortexSessionExecute; use vortex::array::arrays::PrimitiveArray; use vortex::array::validity::Validity::NonNullable; use vortex::buffer::Buffer; @@ -29,18 +32,16 @@ use vortex::encodings::fastlanes::BitPackedData; use vortex::encodings::fastlanes::unpack_iter::BitPacked; use vortex::error::VortexExpect; use vortex::session::VortexSession; +use vortex_cuda::CudaDispatchMode; use vortex_cuda::CudaSession; use vortex_cuda::executor::CudaArrayExt; use vortex_cuda_macros::cuda_available; use vortex_cuda_macros::cuda_not_available; -use crate::common::TimedLaunchStrategy; - -const N_ROWS: usize = 100_000_000; +use crate::timed_launch_strategy::TimedLaunchStrategy; /// Patch frequencies to benchmark (as fractions) -const PATCH_FREQUENCIES: &[(f64, &str)] = - &[(0.001, "0.1%"), (0.01, "1%"), (0.05, "5%"), (0.10, "10%")]; +const PATCH_FREQUENCIES: &[(f64, &str)] = &[(0.01, "1%"), (0.10, "10%")]; /// Create a bit-packed array with the given bit width fn make_bitpacked_array(bit_width: u8, len: usize) -> BitPackedArray @@ -57,7 +58,8 @@ where .collect(); let primitive_array = PrimitiveArray::new(Buffer::from(values), NonNullable); - BitPackedData::encode(&primitive_array.into_array(), bit_width) + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + BitPackedData::encode(&primitive_array.into_array(), bit_width, &mut ctx) .vortex_expect("failed to create BitPacked array") } @@ -97,7 +99,8 @@ where .collect(); let primitive_array = PrimitiveArray::new(Buffer::from(values), NonNullable).into_array(); - BitPackedData::encode(&primitive_array, bit_width) + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + BitPackedData::encode(&primitive_array, bit_width, &mut ctx) .vortex_expect("failed to create BitPacked array with patches") } @@ -107,34 +110,36 @@ where T: BitPacked + NativePType + DeviceRepr + Add + From, T::Physical: DeviceRepr, { - let mut group = c.benchmark_group(format!("bitunpack_cuda_{}", type_name)); - group.sample_size(10); + let mut group = c.benchmark_group(format!("cuda/bitpacked_{}", type_name)); - let array = make_bitpacked_array::(bit_width, N_ROWS); - let nbytes = N_ROWS * size_of::(); + for &(n_rows, size_str) in bench_config::BENCH_SIZES { + let array = make_bitpacked_array::(bit_width, n_rows); + let nbytes = n_rows * size_of::(); - group.throughput(Throughput::Bytes(nbytes as u64)); + group.throughput(Throughput::Bytes(nbytes as u64)); - group.bench_with_input( - BenchmarkId::new("bitunpack", format!("{}bw", bit_width)), - &array, - |b, array| { - b.iter_custom(|iters| { - let timed = TimedLaunchStrategy::default(); - let timer = Arc::clone(&timed.total_time_ns); + group.bench_with_input( + BenchmarkId::new(format!("unpack/{}bw", bit_width), size_str), + &array, + |b, array| { + b.iter_custom(|iters| { + let timed = TimedLaunchStrategy::default(); + let timer = timed.timer(); - let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) - .vortex_expect("failed to create execution context") - .with_launch_strategy(Arc::new(timed)); + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) + .vortex_expect("failed to create execution context") + .with_dispatch_mode(CudaDispatchMode::StandaloneOnly) + .with_launch_strategy(Arc::new(timed)); - for _ in 0..iters { - block_on(array.clone().into_array().execute_cuda(&mut cuda_ctx)).unwrap(); - } + for _ in 0..iters { + block_on(array.clone().into_array().execute_cuda(&mut cuda_ctx)).unwrap(); + } - Duration::from_nanos(timer.load(Ordering::Relaxed)) - }); - }, - ); + Duration::from_nanos(timer.load(Ordering::Relaxed)) + }); + }, + ); + } group.finish(); } @@ -152,35 +157,39 @@ where T: BitPacked + NativePType + DeviceRepr + Add + From, T::Physical: DeviceRepr, { - let mut group = c.benchmark_group(format!("bitunpack_cuda_patched_{}", type_name)); - group.sample_size(10); - - let nbytes = N_ROWS * size_of::(); - group.throughput(Throughput::Bytes(nbytes as u64)); - - for &(patch_freq, patch_label) in PATCH_FREQUENCIES { - let array = make_bitpacked_array_with_patches::(N_ROWS, patch_freq); - - group.bench_with_input( - BenchmarkId::new("bitunpack_patched", patch_label), - &array, - |b, array| { - b.iter_custom(|iters| { - let timed = TimedLaunchStrategy::default(); - let timer = Arc::clone(&timed.total_time_ns); - - let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) - .vortex_expect("failed to create execution context") - .with_launch_strategy(Arc::new(timed)); - - for _ in 0..iters { - block_on(array.clone().into_array().execute_cuda(&mut cuda_ctx)).unwrap(); - } - - Duration::from_nanos(timer.load(Ordering::Relaxed)) - }); - }, - ); + let mut group = c.benchmark_group(format!("cuda/bitpacked_patched_{}", type_name)); + + for &(n_rows, size_str) in bench_config::BENCH_SIZES { + let nbytes = n_rows * size_of::(); + group.throughput(Throughput::Bytes(nbytes as u64)); + + for &(patch_freq, patch_label) in PATCH_FREQUENCIES { + let array = make_bitpacked_array_with_patches::(n_rows, patch_freq); + + group.bench_with_input( + BenchmarkId::new(format!("unpack/{}", patch_label), size_str), + &array, + |b, array| { + b.iter_custom(|iters| { + let timed = TimedLaunchStrategy::default(); + let timer = timed.timer(); + + let mut cuda_ctx = + CudaSession::create_execution_ctx(&VortexSession::empty()) + .vortex_expect("failed to create execution context") + .with_dispatch_mode(CudaDispatchMode::StandaloneOnly) + .with_launch_strategy(Arc::new(timed)); + + for _ in 0..iters { + block_on(array.clone().into_array().execute_cuda(&mut cuda_ctx)) + .unwrap(); + } + + Duration::from_nanos(timer.load(Ordering::Relaxed)) + }); + }, + ); + } } group.finish(); @@ -193,11 +202,11 @@ fn benchmark_bitunpack_with_patches(c: &mut Criterion) { benchmark_bitunpack_with_patches_typed::(c, "u64"); } -criterion::criterion_group!( - benches, - benchmark_bitunpack, - benchmark_bitunpack_with_patches -); +criterion::criterion_group! { + name = benches; + config = bench_config::cuda_bench_config(); + targets = benchmark_bitunpack, benchmark_bitunpack_with_patches +} #[cuda_available] criterion::criterion_main!(benches); diff --git a/vortex-cuda/benches/date_time_parts_cuda.rs b/vortex-cuda/benches/date_time_parts_cuda.rs index 53f303d0b2c..2a5fe5ede3f 100644 --- a/vortex-cuda/benches/date_time_parts_cuda.rs +++ b/vortex-cuda/benches/date_time_parts_cuda.rs @@ -3,10 +3,11 @@ //! CUDA benchmarks for DateTimeParts decoding. -#![allow(clippy::unwrap_used)] -#![allow(clippy::cast_possible_truncation)] +#![expect(clippy::unwrap_used)] +#![expect(clippy::cast_possible_truncation)] -mod common; +mod bench_config; +mod timed_launch_strategy; use std::mem::size_of; use std::sync::Arc; @@ -30,12 +31,13 @@ use vortex::error::VortexExpect; use vortex::extension::datetime::TimeUnit; use vortex::extension::datetime::Timestamp; use vortex::session::VortexSession; +use vortex_cuda::CudaDispatchMode; use vortex_cuda::CudaSession; use vortex_cuda::executor::CudaArrayExt; use vortex_cuda_macros::cuda_available; use vortex_cuda_macros::cuda_not_available; -use crate::common::TimedLaunchStrategy; +use crate::timed_launch_strategy::TimedLaunchStrategy; fn make_datetimeparts_array(len: usize, time_unit: TimeUnit) -> DateTimePartsArray { let days: Vec = (0..len).map(|i| (i / 1000) as i16).collect(); @@ -50,29 +52,25 @@ fn make_datetimeparts_array(len: usize, time_unit: TimeUnit) -> DateTimePartsArr } fn benchmark_datetimeparts(c: &mut Criterion) { - let mut group = c.benchmark_group("datetimeparts_cuda"); - group.sample_size(10); - - for (len, len_str) in [ - (1_000_000usize, "1M"), - (10_000_000usize, "10M"), - (100_000_000usize, "100M"), - ] { + let mut group = c.benchmark_group("cuda/datetimeparts"); + + for &(len, len_str) in bench_config::BENCH_SIZES { group.throughput(Throughput::Bytes((len * size_of::()) as u64)); let (time_unit, unit_str) = (TimeUnit::Milliseconds, "ms"); let dtp_array = make_datetimeparts_array(len, time_unit); group.bench_with_input( - BenchmarkId::new("datetimeparts", format!("{len_str}_{unit_str}")), + BenchmarkId::new(unit_str, len_str), &dtp_array, |b, dtp_array| { b.iter_custom(|iters| { let timed = TimedLaunchStrategy::default(); - let timer = Arc::clone(&timed.total_time_ns); + let timer = timed.timer(); let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) .vortex_expect("failed to create execution context") + .with_dispatch_mode(CudaDispatchMode::StandaloneOnly) .with_launch_strategy(Arc::new(timed)); for _ in 0..iters { @@ -90,7 +88,11 @@ fn benchmark_datetimeparts(c: &mut Criterion) { group.finish(); } -criterion::criterion_group!(benches, benchmark_datetimeparts); +criterion::criterion_group! { + name = benches; + config = bench_config::cuda_bench_config(); + targets = benchmark_datetimeparts +} #[cuda_available] criterion::criterion_main!(benches); diff --git a/vortex-cuda/benches/dict_cuda.rs b/vortex-cuda/benches/dict_cuda.rs index db650eacadd..48bc6ab687c 100644 --- a/vortex-cuda/benches/dict_cuda.rs +++ b/vortex-cuda/benches/dict_cuda.rs @@ -3,11 +3,13 @@ //! CUDA benchmarks for dictionary decoding. -#![allow(clippy::unwrap_used)] -#![allow(clippy::cast_possible_truncation)] +#![expect(clippy::unwrap_used)] +#![expect(clippy::cast_possible_truncation)] -mod common; +mod bench_config; +mod timed_launch_strategy; +use std::fmt::Debug; use std::mem::size_of; use std::sync::Arc; use std::sync::atomic::Ordering; @@ -26,14 +28,14 @@ use vortex::buffer::Buffer; use vortex::dtype::NativePType; use vortex::error::VortexExpect; use vortex::session::VortexSession; +use vortex_cuda::CudaDispatchMode; use vortex_cuda::CudaSession; use vortex_cuda::executor::CudaArrayExt; use vortex_cuda_macros::cuda_available; use vortex_cuda_macros::cuda_not_available; -use crate::common::TimedLaunchStrategy; - -const BENCH_ARGS: &[(usize, &str)] = &[(10_000_000, "10M")]; +use crate::bench_config::BENCH_SIZES; +use crate::timed_launch_strategy::TimedLaunchStrategy; /// Configuration for a dictionary benchmark specifying value and code types along with dictionary size. struct DictBenchConfig { @@ -47,7 +49,7 @@ fn make_dict_array_typed(len: usize, dict_size: usize) -> DictArray where V: NativePType + From, C: NativePType + TryFrom, - >::Error: std::fmt::Debug, + >::Error: Debug, { // Dictionary values let values: Vec = (0..dict_size) @@ -70,12 +72,11 @@ fn benchmark_dict_typed(c: &mut Criterion, config: &DictBenchConfig) where V: NativePType + DeviceRepr + From, C: NativePType + DeviceRepr + TryFrom, - >::Error: std::fmt::Debug, + >::Error: Debug, { - let mut group = c.benchmark_group("dict_cuda"); - group.sample_size(10); + let mut group = c.benchmark_group("cuda/dict"); - for (len, len_str) in BENCH_ARGS { + for (len, len_str) in BENCH_SIZES { // Throughput is based on output size (values read from dictionary) group.throughput(Throughput::Bytes((len * size_of::()) as u64)); @@ -83,20 +84,21 @@ where group.bench_with_input( BenchmarkId::new( - "dict", format!( - "{len_str}_{}_values_{}_codes", + "{}_values_{}_codes", config.value_type_name, config.code_type_name ), + len_str, ), &dict_array, |b, dict_array| { b.iter_custom(|iters| { let timed = TimedLaunchStrategy::default(); - let timer = Arc::clone(&timed.total_time_ns); + let timer = timed.timer(); let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) .vortex_expect("failed to create execution context") + .with_dispatch_mode(CudaDispatchMode::StandaloneOnly) .with_launch_strategy(Arc::new(timed)); for _ in 0..iters { @@ -156,7 +158,11 @@ fn benchmark_dict(c: &mut Criterion) { ); } -criterion::criterion_group!(benches, benchmark_dict); +criterion::criterion_group! { + name = benches; + config = bench_config::cuda_bench_config(); + targets = benchmark_dict +} #[cuda_available] criterion::criterion_main!(benches); diff --git a/vortex-cuda/benches/dynamic_dispatch_cuda.rs b/vortex-cuda/benches/dynamic_dispatch_cuda.rs index 9cff91f9856..0034d55dfc1 100644 --- a/vortex-cuda/benches/dynamic_dispatch_cuda.rs +++ b/vortex-cuda/benches/dynamic_dispatch_cuda.rs @@ -1,10 +1,13 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] -#![allow(clippy::cast_possible_truncation)] -#![allow(clippy::expect_used)] +#![expect(clippy::unwrap_used)] +#![expect(clippy::cast_possible_truncation)] +#![expect(clippy::expect_used)] +mod bench_config; + +use std::marker::PhantomData; use std::mem::size_of; use std::sync::Arc; use std::time::Duration; @@ -14,17 +17,22 @@ use criterion::Criterion; use criterion::Throughput; use cudarc::driver::CudaSlice; use cudarc::driver::DevicePtr; +use cudarc::driver::DeviceRepr; use cudarc::driver::LaunchConfig; use cudarc::driver::PushKernelArg; use cudarc::driver::sys::CUevent_flags; +use futures::executor::block_on; +use vortex::array::ArrayRef; use vortex::array::IntoArray; -use vortex::array::ToCanonical; +use vortex::array::LEGACY_SESSION; +use vortex::array::VortexSessionExecute; use vortex::array::arrays::DictArray; use vortex::array::arrays::PrimitiveArray; +use vortex::array::buffer; use vortex::array::scalar::Scalar; use vortex::array::validity::Validity::NonNullable; use vortex::buffer::Buffer; -use vortex::dtype::PType; +use vortex::dtype::NativePType; use vortex::encodings::alp::ALP; use vortex::encodings::alp::ALPArrayExt; use vortex::encodings::alp::ALPArraySlotsExt; @@ -41,6 +49,7 @@ use vortex::error::VortexResult; use vortex::error::vortex_err; use vortex::session::VortexSession; use vortex_cuda::CudaDeviceBuffer; +use vortex_cuda::CudaDispatchMode; use vortex_cuda::CudaExecutionCtx; use vortex_cuda::CudaSession; use vortex_cuda::dynamic_dispatch::CudaDispatchPlan; @@ -49,27 +58,23 @@ use vortex_cuda::dynamic_dispatch::MaterializedPlan; use vortex_cuda_macros::cuda_available; use vortex_cuda_macros::cuda_not_available; -const BENCH_ARGS: &[(usize, &str)] = &[ - (1_000_000, "1M"), - (10_000_000, "10M"), - (100_000_000, "100M"), -]; +use crate::bench_config::BENCH_SIZES; /// Launch the dynamic_dispatch kernel and return GPU-timed duration. /// /// This deliberately does not use `CudaDispatchPlan::execute` because the /// benchmark pre-allocates the output buffer and device plan once, then reuses /// them across iterations. -fn run_timed( +fn run_timed( cuda_ctx: &mut CudaExecutionCtx, array_len: usize, output_buf: &CudaDeviceBuffer, device_plan: &Arc>, shared_mem_bytes: u32, ) -> VortexResult { - let cuda_function = cuda_ctx.load_function("dynamic_dispatch", &[PType::U32])?; + let cuda_function = cuda_ctx.load_function("dynamic_dispatch", &[T::PTYPE])?; let array_len_u64 = array_len as u64; - let output_view = output_buf.as_view::(); + let output_view = output_buf.as_view::(); let (output_ptr, record_output) = output_view.device_ptr(cuda_ctx.stream()); let (plan_ptr, record_plan) = device_plan.device_ptr(cuda_ctx.stream()); @@ -116,18 +121,24 @@ fn run_timed( } /// Benchmark runner: builds a dynamic plan and launches the kernel. -struct BenchRunner { +/// +/// `T` is the unsigned integer type matching the output element width +/// (e.g. `u32` for f32/i32/u32, `u64` for f64/i64/u64). +struct BenchRunner { _plan: CudaDispatchPlan, smem_bytes: u32, len: usize, device_plan: Arc>, output_buf: CudaDeviceBuffer, - _plan_buffers: Vec, + _plan_buffers: Vec, + _phantom: PhantomData, } -impl BenchRunner { - fn new(array: &vortex::array::ArrayRef, len: usize, cuda_ctx: &CudaExecutionCtx) -> Self { - let plan = match DispatchPlan::new(array).vortex_expect("build_dyn_dispatch_plan") { +impl BenchRunner { + fn new(array: &ArrayRef, len: usize, cuda_ctx: &mut CudaExecutionCtx) -> Self { + let plan = match DispatchPlan::new(array, CudaDispatchMode::DynDispatchOnly) + .vortex_expect("build_dyn_dispatch_plan") + { DispatchPlan::Fused(plan) => plan, _ => unreachable!("encoding not fusable"), }; @@ -135,7 +146,8 @@ impl BenchRunner { dispatch_plan, device_buffers, shared_mem_bytes, - } = plan.materialize(cuda_ctx).vortex_expect("materialize plan"); + .. + } = block_on(plan.materialize(cuda_ctx)).vortex_expect("materialize plan"); let device_plan = Arc::new( cuda_ctx @@ -151,16 +163,17 @@ impl BenchRunner { device_plan, output_buf: CudaDeviceBuffer::new( cuda_ctx - .device_alloc::(len.next_multiple_of(1024)) + .device_alloc::(len.next_multiple_of(1024)) .expect("alloc output"), ), _plan_buffers: device_buffers, + _phantom: PhantomData, } } fn run(&self, cuda_ctx: &mut CudaExecutionCtx) -> Duration { cuda_ctx.stream().synchronize().unwrap(); - run_timed( + run_timed::( cuda_ctx, self.len, &self.output_buf, @@ -175,13 +188,12 @@ impl BenchRunner { // Benchmark: FoR(BitPacked) // --------------------------------------------------------------------------- fn bench_for_bitpacked(c: &mut Criterion) { - let mut group = c.benchmark_group("for_bitpacked_6bw"); - group.sample_size(10); + let mut group = c.benchmark_group("cuda/for_bitpacked_6bw"); let bit_width: u8 = 6; let reference = 100_000u32; - for (len, len_str) in BENCH_ARGS { + for (len, len_str) in BENCH_SIZES { group.throughput(Throughput::Bytes((len * size_of::()) as u64)); // FoR(BitPacked): residuals 0..max_val, reference adds 100_000 @@ -190,29 +202,27 @@ fn bench_for_bitpacked(c: &mut Criterion) { .map(|i| (i as u64 % (max_val + 1)) as u32) .collect(); let prim = PrimitiveArray::new(Buffer::from(residuals), NonNullable); - let bp = BitPackedData::encode(&prim.into_array(), bit_width).vortex_expect("bitpack"); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let bp = + BitPackedData::encode(&prim.into_array(), bit_width, &mut ctx).vortex_expect("bitpack"); let array = FoR::try_new(bp.into_array(), Scalar::from(reference)) .vortex_expect("for") .into_array(); - group.bench_with_input( - BenchmarkId::new("dynamic_dispatch_u32", len_str), - len, - |b, &n| { - let mut cuda_ctx = - CudaSession::create_execution_ctx(&VortexSession::empty()).vortex_expect("ctx"); - - let bench_runner = BenchRunner::new(&array, n, &cuda_ctx); - - b.iter_custom(|iters| { - let mut total_time = Duration::ZERO; - for _ in 0..iters { - total_time += bench_runner.run(&mut cuda_ctx); - } - total_time - }); - }, - ); + group.bench_with_input(BenchmarkId::new("dispatch_u32", len_str), len, |b, &n| { + let mut cuda_ctx = + CudaSession::create_execution_ctx(&VortexSession::empty()).vortex_expect("ctx"); + + let bench_runner = BenchRunner::::new(&array, n, &mut cuda_ctx); + + b.iter_custom(|iters| { + let mut total_time = Duration::ZERO; + for _ in 0..iters { + total_time += bench_runner.run(&mut cuda_ctx); + } + total_time + }); + }); } group.finish(); @@ -222,42 +232,38 @@ fn bench_for_bitpacked(c: &mut Criterion) { // Benchmark: Dict(codes=BitPacked, values=Primitive) // --------------------------------------------------------------------------- fn bench_dict_bp_codes(c: &mut Criterion) { - let mut group = c.benchmark_group("dict_256vals_bp8bw_codes"); - group.sample_size(10); + let mut group = c.benchmark_group("cuda/dict_256vals_bp8bw_codes"); let dict_size: usize = 256; let dict_bit_width: u8 = 8; let dict_values: Vec = (0..dict_size as u32).map(|i| i * 1000 + 42).collect(); - for (len, len_str) in BENCH_ARGS { + for (len, len_str) in BENCH_SIZES { group.throughput(Throughput::Bytes((len * size_of::()) as u64)); let codes: Vec = (0..*len).map(|i| (i % dict_size) as u32).collect(); let codes_prim = PrimitiveArray::new(Buffer::from(codes), NonNullable); - let codes_bp = BitPackedData::encode(&codes_prim.into_array(), dict_bit_width) + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let codes_bp = BitPackedData::encode(&codes_prim.into_array(), dict_bit_width, &mut ctx) .vortex_expect("bitpack codes"); let values_prim = PrimitiveArray::new(Buffer::from(dict_values.clone()), NonNullable); let dict = DictArray::new(codes_bp.into_array(), values_prim.into_array()); let array = dict.into_array(); - group.bench_with_input( - BenchmarkId::new("dynamic_dispatch_u32", len_str), - len, - |b, &n| { - let mut cuda_ctx = - CudaSession::create_execution_ctx(&VortexSession::empty()).vortex_expect("ctx"); - - let bench_runner = BenchRunner::new(&array, n, &cuda_ctx); - - b.iter_custom(|iters| { - let mut total_time = Duration::ZERO; - for _ in 0..iters { - total_time += bench_runner.run(&mut cuda_ctx); - } - total_time - }); - }, - ); + group.bench_with_input(BenchmarkId::new("dispatch_u32", len_str), len, |b, &n| { + let mut cuda_ctx = + CudaSession::create_execution_ctx(&VortexSession::empty()).vortex_expect("ctx"); + + let bench_runner = BenchRunner::::new(&array, n, &mut cuda_ctx); + + b.iter_custom(|iters| { + let mut total_time = Duration::ZERO; + for _ in 0..iters { + total_time += bench_runner.run(&mut cuda_ctx); + } + total_time + }); + }); } group.finish(); @@ -267,41 +273,98 @@ fn bench_dict_bp_codes(c: &mut Criterion) { // Benchmark: RunEnd(ends=Prim, values=Prim) // --------------------------------------------------------------------------- fn bench_runend(c: &mut Criterion) { - let mut group = c.benchmark_group("runend_100runs"); - group.sample_size(10); + let mut group = c.benchmark_group("cuda/runend_100runs"); let num_runs: usize = 100; - for (len, len_str) in BENCH_ARGS { + for (len, len_str) in BENCH_SIZES { group.throughput(Throughput::Bytes((len * size_of::()) as u64)); let run_len = *len / num_runs; let ends: Vec = (1..=num_runs).map(|i| (i * run_len) as u32).collect(); let values: Vec = (0..num_runs).map(|i| (i * 7 + 42) as u32).collect(); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let ends_arr = PrimitiveArray::new(Buffer::from(ends), NonNullable).into_array(); let values_arr = PrimitiveArray::new(Buffer::from(values), NonNullable).into_array(); - let re = RunEnd::new(ends_arr, values_arr); + let re = RunEnd::new(ends_arr, values_arr, &mut ctx); let array = re.into_array(); - group.bench_with_input( - BenchmarkId::new("dynamic_dispatch_u32", len_str), - len, - |b, &n| { - let mut cuda_ctx = - CudaSession::create_execution_ctx(&VortexSession::empty()).vortex_expect("ctx"); - - let bench_runner = BenchRunner::new(&array, n, &cuda_ctx); - - b.iter_custom(|iters| { - let mut total_time = Duration::ZERO; - for _ in 0..iters { - total_time += bench_runner.run(&mut cuda_ctx); - } - total_time - }); - }, + group.bench_with_input(BenchmarkId::new("dispatch_u32", len_str), len, |b, &n| { + let mut cuda_ctx = + CudaSession::create_execution_ctx(&VortexSession::empty()).vortex_expect("ctx"); + + let bench_runner = BenchRunner::::new(&array, n, &mut cuda_ctx); + + b.iter_custom(|iters| { + let mut total_time = Duration::ZERO; + for _ in 0..iters { + total_time += bench_runner.run(&mut cuda_ctx); + } + total_time + }); + }); + } + + group.finish(); +} + +// --------------------------------------------------------------------------- +// Benchmark: ALP(FoR(BitPacked)) — f64 +// --------------------------------------------------------------------------- +fn bench_alp_for_bitpacked_f64(c: &mut Criterion) { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let mut group = c.benchmark_group("cuda/alp_for_bp_6bw_f64"); + + let exponents = Exponents { e: 2, f: 0 }; + let bit_width: u8 = 6; + + for (len, len_str) in BENCH_SIZES { + group.throughput(Throughput::Bytes((len * size_of::()) as u64)); + + // Generate f64 values that ALP-encode without patches. + let floats: Vec = (0..*len) + .map(|i| ::decode_single(10 + (i as i64 % 64), exponents)) + .collect(); + let float_prim = PrimitiveArray::new(Buffer::from(floats), NonNullable); + + // Encode: ALP → FoR → BitPacked + let alp = + alp_encode(float_prim.as_view(), Some(exponents), &mut ctx).vortex_expect("alp_encode"); + assert!(alp.patches().is_none()); + let for_arr = FoRData::encode( + alp.encoded() + .clone() + .execute::(&mut ctx) + .vortex_expect("to primitive"), + ) + .vortex_expect("for encode"); + let bp = BitPackedData::encode(for_arr.encoded(), bit_width, &mut ctx) + .vortex_expect("bitpack encode"); + + let tree = ALP::new( + FoR::try_new(bp.into_array(), for_arr.reference_scalar().clone()) + .vortex_expect("for_new") + .into_array(), + exponents, + None, ); + let array = tree.into_array(); + + group.bench_with_input(BenchmarkId::new("dispatch_f64", len_str), len, |b, &n| { + let mut cuda_ctx = + CudaSession::create_execution_ctx(&VortexSession::empty()).vortex_expect("ctx"); + + let bench_runner = BenchRunner::::new(&array, n, &mut cuda_ctx); + + b.iter_custom(|iters| { + let mut total_time = Duration::ZERO; + for _ in 0..iters { + total_time += bench_runner.run(&mut cuda_ctx); + } + total_time + }); + }); } group.finish(); @@ -311,8 +374,7 @@ fn bench_runend(c: &mut Criterion) { // Benchmark: Dict(codes=BitPacked, values=FoR(BitPacked)) // --------------------------------------------------------------------------- fn bench_dict_bp_codes_bp_for_values(c: &mut Criterion) { - let mut group = c.benchmark_group("dict_64vals_bp6bw_codes_for_bp6bw_values"); - group.sample_size(10); + let mut group = c.benchmark_group("cuda/dict_64vals_bp6bw_codes_for_bp6bw_values"); let dict_size: usize = 64; let dict_bit_width: u8 = 6; @@ -322,40 +384,37 @@ fn bench_dict_bp_codes_bp_for_values(c: &mut Criterion) { // Dict values: residuals 0..63 bitpacked, FoR adds 1_000_000 let dict_residuals: Vec = (0..dict_size as u32).collect(); let dict_prim = PrimitiveArray::new(Buffer::from(dict_residuals), NonNullable); - let dict_bp = BitPackedData::encode(&dict_prim.into_array(), dict_bit_width) + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let dict_bp = BitPackedData::encode(&dict_prim.into_array(), dict_bit_width, &mut ctx) .vortex_expect("bitpack dict"); let dict_for = FoR::try_new(dict_bp.into_array(), Scalar::from(dict_reference)).vortex_expect("for dict"); - for (len, len_str) in BENCH_ARGS { + for (len, len_str) in BENCH_SIZES { group.throughput(Throughput::Bytes((len * size_of::()) as u64)); let codes: Vec = (0..*len).map(|i| (i % dict_size) as u32).collect(); let codes_prim = PrimitiveArray::new(Buffer::from(codes), NonNullable); - let codes_bp = BitPackedData::encode(&codes_prim.into_array(), codes_bit_width) + let codes_bp = BitPackedData::encode(&codes_prim.into_array(), codes_bit_width, &mut ctx) .vortex_expect("bitpack codes"); let dict = DictArray::new(codes_bp.into_array(), dict_for.clone().into_array()); let array = dict.into_array(); - group.bench_with_input( - BenchmarkId::new("dynamic_dispatch_u32", len_str), - len, - |b, &n| { - let mut cuda_ctx = - CudaSession::create_execution_ctx(&VortexSession::empty()).vortex_expect("ctx"); - - let bench_runner = BenchRunner::new(&array, n, &cuda_ctx); - - b.iter_custom(|iters| { - let mut total_time = Duration::ZERO; - for _ in 0..iters { - total_time += bench_runner.run(&mut cuda_ctx); - } - total_time - }); - }, - ); + group.bench_with_input(BenchmarkId::new("dispatch_u32", len_str), len, |b, &n| { + let mut cuda_ctx = + CudaSession::create_execution_ctx(&VortexSession::empty()).vortex_expect("ctx"); + + let bench_runner = BenchRunner::::new(&array, n, &mut cuda_ctx); + + b.iter_custom(|iters| { + let mut total_time = Duration::ZERO; + for _ in 0..iters { + total_time += bench_runner.run(&mut cuda_ctx); + } + total_time + }); + }); } group.finish(); @@ -365,13 +424,13 @@ fn bench_dict_bp_codes_bp_for_values(c: &mut Criterion) { // Benchmark: ALP(FoR(BitPacked)) for f32 // --------------------------------------------------------------------------- fn bench_alp_for_bitpacked(c: &mut Criterion) { - let mut group = c.benchmark_group("alp_for_bp_6bw_f32"); - group.sample_size(10); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let mut group = c.benchmark_group("cuda/alp_for_bp_6bw_f32"); let exponents = Exponents { e: 2, f: 0 }; let bit_width: u8 = 6; - for (len, len_str) in BENCH_ARGS { + for (len, len_str) in BENCH_SIZES { group.throughput(Throughput::Bytes((len * size_of::()) as u64)); // Generate f32 values that ALP-encode without patches. @@ -381,11 +440,18 @@ fn bench_alp_for_bitpacked(c: &mut Criterion) { let float_prim = PrimitiveArray::new(Buffer::from(floats), NonNullable); // Encode: ALP → FoR → BitPacked - let alp = alp_encode(&float_prim, Some(exponents)).vortex_expect("alp_encode"); + let alp = + alp_encode(float_prim.as_view(), Some(exponents), &mut ctx).vortex_expect("alp_encode"); assert!(alp.patches().is_none()); - let for_arr = FoRData::encode(alp.encoded().to_primitive()).vortex_expect("for encode"); - let bp = - BitPackedData::encode(for_arr.encoded(), bit_width).vortex_expect("bitpack encode"); + let for_arr = FoRData::encode( + alp.encoded() + .clone() + .execute::(&mut ctx) + .vortex_expect("to primitive"), + ) + .vortex_expect("for encode"); + let bp = BitPackedData::encode(for_arr.encoded(), bit_width, &mut ctx) + .vortex_expect("bitpack encode"); let tree = ALP::new( FoR::try_new(bp.into_array(), for_arr.reference_scalar().clone()) @@ -396,24 +462,141 @@ fn bench_alp_for_bitpacked(c: &mut Criterion) { ); let array = tree.into_array(); - group.bench_with_input( - BenchmarkId::new("dynamic_dispatch_f32", len_str), - len, - |b, &n| { - let mut cuda_ctx = - CudaSession::create_execution_ctx(&VortexSession::empty()).vortex_expect("ctx"); - - let bench_runner = BenchRunner::new(&array, n, &cuda_ctx); - - b.iter_custom(|iters| { - let mut total_time = Duration::ZERO; - for _ in 0..iters { - total_time += bench_runner.run(&mut cuda_ctx); - } - total_time - }); - }, - ); + group.bench_with_input(BenchmarkId::new("dispatch_f32", len_str), len, |b, &n| { + let mut cuda_ctx = + CudaSession::create_execution_ctx(&VortexSession::empty()).vortex_expect("ctx"); + + let bench_runner = BenchRunner::::new(&array, n, &mut cuda_ctx); + + b.iter_custom(|iters| { + let mut total_time = Duration::ZERO; + for _ in 0..iters { + total_time += bench_runner.run(&mut cuda_ctx); + } + total_time + }); + }); + } + + group.finish(); +} + +// --------------------------------------------------------------------------- +// Benchmark: Dict with narrower BitPacked codes (exercises widen_inplace) +// --------------------------------------------------------------------------- + +/// Dict(codes=BitPacked, values=Prim) — widens u8 → u32 in smem. +fn bench_dict_bp_u8_codes_u32_values(c: &mut Criterion) { + let mut group = c.benchmark_group("cuda/dict_widen_u8_to_u32"); + + let dict_size: usize = 4; // 2-bit codes + let bit_width: u8 = 2; + let dict_values: Vec = (0..dict_size as u32).map(|i| i * 1000 + 42).collect(); + + for (len, len_str) in BENCH_SIZES { + group.throughput(Throughput::Bytes((len * size_of::()) as u64)); + + let codes: Vec = (0..*len).map(|i| (i % dict_size) as u8).collect(); + let codes_prim = PrimitiveArray::new(Buffer::from(codes), NonNullable); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let codes_bp = BitPackedData::encode(&codes_prim.into_array(), bit_width, &mut ctx) + .vortex_expect("bitpack u8 codes"); + let values_prim = PrimitiveArray::new(Buffer::from(dict_values.clone()), NonNullable); + let dict = DictArray::new(codes_bp.into_array(), values_prim.into_array()); + let array = dict.into_array(); + + group.bench_with_input(BenchmarkId::new("dispatch_u32", len_str), len, |b, &n| { + let mut cuda_ctx = + CudaSession::create_execution_ctx(&VortexSession::empty()).vortex_expect("ctx"); + + let bench_runner = BenchRunner::::new(&array, n, &mut cuda_ctx); + + b.iter_custom(|iters| { + let mut total_time = Duration::ZERO; + for _ in 0..iters { + total_time += bench_runner.run(&mut cuda_ctx); + } + total_time + }); + }); + } + + group.finish(); +} + +/// Dict(codes=BitPacked, values=Prim) — widens u16 → u32 in smem. +fn bench_dict_bp_u16_codes_u32_values(c: &mut Criterion) { + let mut group = c.benchmark_group("cuda/dict_widen_u16_to_u32"); + + let dict_size: usize = 8; // 3-bit codes + let bit_width: u8 = 3; + let dict_values: Vec = (0..dict_size as u32).map(|i| i * 5000 + 100).collect(); + + for (len, len_str) in BENCH_SIZES { + group.throughput(Throughput::Bytes((len * size_of::()) as u64)); + + let codes: Vec = (0..*len).map(|i| (i % dict_size) as u16).collect(); + let codes_prim = PrimitiveArray::new(Buffer::from(codes), NonNullable); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let codes_bp = BitPackedData::encode(&codes_prim.into_array(), bit_width, &mut ctx) + .vortex_expect("bitpack u16 codes"); + let values_prim = PrimitiveArray::new(Buffer::from(dict_values.clone()), NonNullable); + let dict = DictArray::new(codes_bp.into_array(), values_prim.into_array()); + let array = dict.into_array(); + + group.bench_with_input(BenchmarkId::new("dispatch_u32", len_str), len, |b, &n| { + let mut cuda_ctx = + CudaSession::create_execution_ctx(&VortexSession::empty()).vortex_expect("ctx"); + + let bench_runner = BenchRunner::::new(&array, n, &mut cuda_ctx); + + b.iter_custom(|iters| { + let mut total_time = Duration::ZERO; + for _ in 0..iters { + total_time += bench_runner.run(&mut cuda_ctx); + } + total_time + }); + }); + } + + group.finish(); +} + +/// Dict(codes=BitPacked, values=Prim) — same-width baseline, no widen. +fn bench_dict_bp_u32_codes_u32_values(c: &mut Criterion) { + let mut group = c.benchmark_group("cuda/dict_nowiden_u32_to_u32"); + + let dict_size: usize = 8; // 3-bit codes + let bit_width: u8 = 3; + let dict_values: Vec = (0..dict_size as u32).map(|i| i * 5000 + 100).collect(); + + for (len, len_str) in BENCH_SIZES { + group.throughput(Throughput::Bytes((len * size_of::()) as u64)); + + let codes: Vec = (0..*len).map(|i| (i % dict_size) as u32).collect(); + let codes_prim = PrimitiveArray::new(Buffer::from(codes), NonNullable); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let codes_bp = BitPackedData::encode(&codes_prim.into_array(), bit_width, &mut ctx) + .vortex_expect("bitpack u32 codes"); + let values_prim = PrimitiveArray::new(Buffer::from(dict_values.clone()), NonNullable); + let dict = DictArray::new(codes_bp.into_array(), values_prim.into_array()); + let array = dict.into_array(); + + group.bench_with_input(BenchmarkId::new("dispatch_u32", len_str), len, |b, &n| { + let mut cuda_ctx = + CudaSession::create_execution_ctx(&VortexSession::empty()).vortex_expect("ctx"); + + let bench_runner = BenchRunner::::new(&array, n, &mut cuda_ctx); + + b.iter_custom(|iters| { + let mut total_time = Duration::ZERO; + for _ in 0..iters { + total_time += bench_runner.run(&mut cuda_ctx); + } + total_time + }); + }); } group.finish(); @@ -425,9 +608,17 @@ fn benchmark_dynamic_dispatch(c: &mut Criterion) { bench_runend(c); bench_dict_bp_codes_bp_for_values(c); bench_alp_for_bitpacked(c); + bench_alp_for_bitpacked_f64(c); + bench_dict_bp_u8_codes_u32_values(c); + bench_dict_bp_u16_codes_u32_values(c); + bench_dict_bp_u32_codes_u32_values(c); } -criterion::criterion_group!(benches, benchmark_dynamic_dispatch); +criterion::criterion_group! { + name = benches; + config = bench_config::cuda_bench_config(); + targets = benchmark_dynamic_dispatch +} #[cuda_available] criterion::criterion_main!(benches); diff --git a/vortex-cuda/benches/filter_cuda.rs b/vortex-cuda/benches/filter_cuda.rs index 3eb623ac357..5a7869d6bff 100644 --- a/vortex-cuda/benches/filter_cuda.rs +++ b/vortex-cuda/benches/filter_cuda.rs @@ -3,8 +3,10 @@ //! CUDA benchmarks for GPU filtering using CUB DeviceSelect::Flagged. -#![allow(clippy::unwrap_used)] -#![allow(clippy::cast_possible_truncation)] +#![expect(clippy::unwrap_used)] +#![expect(clippy::cast_possible_truncation)] + +mod bench_config; use std::ffi::c_void; use std::fmt::Debug; @@ -18,6 +20,7 @@ use cudarc::driver::CudaSlice; use cudarc::driver::CudaView; use cudarc::driver::DevicePtr; use cudarc::driver::DevicePtrMut; +use cudarc::driver::DeviceRepr; use cudarc::driver::sys::CUevent_flags; use futures::executor::block_on; use vortex::error::VortexExpect; @@ -32,7 +35,7 @@ use vortex_cuda::CudaSession; use vortex_cuda_macros::cuda_available; use vortex_cuda_macros::cuda_not_available; -const BENCH_SIZES: &[(usize, &str)] = &[(1_000_000, "1M"), (10_000_000, "10M")]; +use crate::bench_config::BENCH_SIZES; const SELECTIVITIES: &[(f64, &str)] = &[(0.1, "10%"), (0.5, "50%"), (0.9, "90%")]; /// Creates input data of the given length. @@ -64,7 +67,7 @@ fn make_bitmask(len: usize, selectivity: f64) -> (Vec, usize) { /// Runs the CUB filter kernel and returns elapsed GPU time. #[expect(clippy::too_many_arguments)] -async fn run_filter_timed( +async fn run_filter_timed( d_input: CudaView<'_, T>, d_bitmask: CudaView<'_, u8>, d_output: &mut CudaSlice, @@ -132,17 +135,9 @@ async fn run_filter_timed( /// Benchmark filter for a specific type. fn benchmark_filter_type(c: &mut Criterion, type_name: &str) where - T: CubFilterable - + cudarc::driver::DeviceRepr - + From - + Debug - + Clone - + Send - + Sync - + 'static, + T: CubFilterable + DeviceRepr + From + Debug + Clone + Send + Sync + 'static, { - let mut group = c.benchmark_group(format!("filter_cuda_{type_name}")); - group.sample_size(10); + let mut group = c.benchmark_group(format!("cuda/filter_{type_name}")); for (len, len_label) in BENCH_SIZES { for (selectivity, sel_label) in SELECTIVITIES { @@ -153,7 +148,7 @@ where group.throughput(Throughput::Bytes((len * size_of::()) as u64)); group.bench_with_input( - BenchmarkId::new(format!("{len_label}_{sel_label}"), true_count), + BenchmarkId::new(format!("select/{sel_label}"), len_label), &(input_data, bitmask, true_count), |b, (input_data, bitmask, true_count)| { b.iter_custom(|iters| { @@ -229,10 +224,13 @@ where fn benchmark_filter(c: &mut Criterion) { benchmark_filter_type::(c, "i32"); benchmark_filter_type::(c, "i64"); - benchmark_filter_type::(c, "f64"); } -criterion::criterion_group!(benches, benchmark_filter); +criterion::criterion_group! { + name = benches; + config = bench_config::cuda_bench_config(); + targets = benchmark_filter +} #[cuda_available] criterion::criterion_main!(benches); diff --git a/vortex-cuda/benches/for_cuda.rs b/vortex-cuda/benches/for_cuda.rs index cc5fab5e22c..91b30c4b021 100644 --- a/vortex-cuda/benches/for_cuda.rs +++ b/vortex-cuda/benches/for_cuda.rs @@ -3,10 +3,11 @@ //! CUDA benchmarks for FoR decompression. -#![allow(clippy::unwrap_used)] -#![allow(clippy::cast_possible_truncation)] +#![expect(clippy::unwrap_used)] +#![expect(clippy::cast_possible_truncation)] -mod common; +mod bench_config; +mod timed_launch_strategy; use std::mem::size_of; use std::ops::Add; @@ -20,6 +21,8 @@ use criterion::Throughput; use cudarc::driver::DeviceRepr; use futures::executor::block_on; use vortex::array::IntoArray; +use vortex::array::LEGACY_SESSION; +use vortex::array::VortexSessionExecute; use vortex::array::arrays::PrimitiveArray; use vortex::array::validity::Validity; use vortex::buffer::Buffer; @@ -31,14 +34,14 @@ use vortex::encodings::fastlanes::FoRArray; use vortex::error::VortexExpect; use vortex::scalar::Scalar; use vortex::session::VortexSession; +use vortex_cuda::CudaDispatchMode; use vortex_cuda::CudaSession; use vortex_cuda::executor::CudaArrayExt; use vortex_cuda_macros::cuda_available; use vortex_cuda_macros::cuda_not_available; -use crate::common::TimedLaunchStrategy; - -const BENCH_ARGS: &[(usize, &str)] = &[(10_000_000, "10M")]; +use crate::bench_config::BENCH_SIZES; +use crate::timed_launch_strategy::TimedLaunchStrategy; const REFERENCE_VALUE: u8 = 10; /// Creates a FoR array with the specified type and length. @@ -56,7 +59,9 @@ where PrimitiveArray::new(Buffer::from(data), Validity::NonNullable).into_array(); if bp && T::PTYPE != PType::U8 { - let child = BitPackedData::encode(&primitive_array, 8).vortex_expect("failed to bitpack"); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let child = + BitPackedData::encode(&primitive_array, 8, &mut ctx).vortex_expect("failed to bitpack"); FoR::try_new(child.into_array(), reference.into()) .vortex_expect("failed to create FoR array") } else { @@ -70,24 +75,24 @@ where T: NativePType + DeviceRepr + From + Add, Scalar: From, { - let mut group = c.benchmark_group("for_cuda"); - group.sample_size(10); + let mut group = c.benchmark_group("cuda/for"); - for &(len, len_str) in BENCH_ARGS { + for &(len, len_str) in BENCH_SIZES { group.throughput(Throughput::Bytes((len * size_of::()) as u64)); let for_array = make_for_array_typed::(len, false); group.bench_with_input( - BenchmarkId::new("for", format!("{len_str}_{type_name}")), + BenchmarkId::new(type_name, len_str), &for_array, |b, for_array| { b.iter_custom(|iters| { let timed = TimedLaunchStrategy::default(); - let timer = Arc::clone(&timed.total_time_ns); + let timer = timed.timer(); let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) .vortex_expect("failed to create execution context") + .with_dispatch_mode(CudaDispatchMode::StandaloneOnly) .with_launch_strategy(Arc::new(timed)); for _ in 0..iters { @@ -109,24 +114,24 @@ where T: NativePType + DeviceRepr + From + Add, Scalar: From, { - let mut group = c.benchmark_group("ffor_cuda"); - group.sample_size(10); + let mut group = c.benchmark_group("cuda/ffor"); - for &(len, len_str) in BENCH_ARGS { + for &(len, len_str) in BENCH_SIZES { group.throughput(Throughput::Bytes((len * size_of::()) as u64)); let for_array = make_for_array_typed::(len, true); group.bench_with_input( - BenchmarkId::new("for", format!("{len_str}_{type_name}")), + BenchmarkId::new(type_name, len_str), &for_array, |b, for_array| { b.iter_custom(|iters| { let timed = TimedLaunchStrategy::default(); - let timer = Arc::clone(&timed.total_time_ns); + let timer = timed.timer(); let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) .vortex_expect("failed to create execution context") + .with_dispatch_mode(CudaDispatchMode::StandaloneOnly) .with_launch_strategy(Arc::new(timed)); for _ in 0..iters { @@ -159,7 +164,11 @@ fn benchmark_ffor(c: &mut Criterion) { benchmark_ffor_typed::(c, "u64"); } -criterion::criterion_group!(benches, benchmark_for, benchmark_ffor); +criterion::criterion_group! { + name = benches; + config = bench_config::cuda_bench_config(); + targets = benchmark_for, benchmark_ffor +} #[cuda_available] criterion::criterion_main!(benches); diff --git a/vortex-cuda/benches/runend_cuda.rs b/vortex-cuda/benches/runend_cuda.rs index c073f60634c..f02e0458080 100644 --- a/vortex-cuda/benches/runend_cuda.rs +++ b/vortex-cuda/benches/runend_cuda.rs @@ -3,10 +3,11 @@ //! CUDA benchmarks for run-end decoding. -#![allow(clippy::unwrap_used)] -#![allow(clippy::cast_possible_truncation)] +#![expect(clippy::unwrap_used)] +#![expect(clippy::cast_possible_truncation)] -mod common; +mod bench_config; +mod timed_launch_strategy; use std::mem::size_of; use std::sync::Arc; @@ -18,6 +19,7 @@ use criterion::Criterion; use criterion::Throughput; use cudarc::driver::DeviceRepr; use futures::executor::block_on; +use vortex::array::ExecutionCtx; use vortex::array::IntoArray; use vortex::array::arrays::PrimitiveArray; use vortex::array::validity::Validity; @@ -31,10 +33,14 @@ use vortex_cuda::executor::CudaArrayExt; use vortex_cuda_macros::cuda_available; use vortex_cuda_macros::cuda_not_available; -use crate::common::TimedLaunchStrategy; +use crate::timed_launch_strategy::TimedLaunchStrategy; /// Creates a run-end encoded array with the specified output length and average run length. -fn make_runend_array_typed(output_len: usize, avg_run_len: usize) -> RunEndArray +fn make_runend_array_typed( + output_len: usize, + avg_run_len: usize, + ctx: &mut ExecutionCtx, +) -> RunEndArray where T: NativePType + From, { @@ -55,7 +61,7 @@ where let ends_array = PrimitiveArray::new(Buffer::from(ends), Validity::NonNullable).into_array(); let values_array = PrimitiveArray::new(Buffer::from(values), Validity::NonNullable).into_array(); - RunEnd::new(ends_array, values_array) + RunEnd::new(ends_array, values_array, ctx) } /// Benchmark run-end decoding for a specific type with varying run lengths @@ -63,26 +69,22 @@ fn benchmark_runend_typed(c: &mut Criterion, type_name: &str) where T: NativePType + DeviceRepr + From, { - let mut group = c.benchmark_group("runend_cuda"); - group.sample_size(10); - - for (len, len_str) in [ - (1_000_000usize, "1M"), - (10_000_000usize, "10M"), - (100_000_000usize, "100M"), - ] { + let mut group = c.benchmark_group("cuda/runend"); + + for &(len, len_str) in bench_config::BENCH_SIZES { group.throughput(Throughput::Bytes((len * size_of::()) as u64)); - for run_len in [10, 100, 1000, 10000, 100000] { - let runend_array = make_runend_array_typed::(len, run_len); + for run_len in [10, 1000, 100000] { + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()).unwrap(); + let runend_array = make_runend_array_typed::(len, run_len, cuda_ctx.execution_ctx()); group.bench_with_input( - BenchmarkId::new("runend", format!("{len_str}_{type_name}_runlen_{run_len}")), + BenchmarkId::new(format!("{type_name}_runlen_{run_len}"), len_str), &runend_array, |b, runend_array| { b.iter_custom(|iters| { let timed = TimedLaunchStrategy::default(); - let timer = Arc::clone(&timed.total_time_ns); + let timer = timed.timer(); let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) @@ -114,7 +116,11 @@ fn benchmark_runend(c: &mut Criterion) { benchmark_runend_typed::(c, "i32"); } -criterion::criterion_group!(benches, benchmark_runend); +criterion::criterion_group! { + name = benches; + config = bench_config::cuda_bench_config(); + targets = benchmark_runend +} #[cuda_available] criterion::criterion_main!(benches); diff --git a/vortex-cuda/benches/throughput_cuda.rs b/vortex-cuda/benches/throughput_cuda.rs index b911263ec26..1ae85c1ff45 100644 --- a/vortex-cuda/benches/throughput_cuda.rs +++ b/vortex-cuda/benches/throughput_cuda.rs @@ -6,8 +6,11 @@ //! //! Cases: 100% input only, 100% output only, and 50/50 mixed. -#![allow(clippy::unwrap_used)] -#![allow(clippy::cast_possible_truncation)] +#![expect(clippy::unwrap_used)] + +mod bench_config; +// Unused here but suppresses dead_code warning for the shared module. +const _: &[(usize, &str)] = bench_config::BENCH_SIZES; use std::time::Duration; @@ -29,9 +32,9 @@ const TOTAL_BYTES: usize = 100 * 1024 * 1024; /// Benchmark configurations: (input_bytes, output_bytes, label). const MIXES: &[(usize, usize, &str)] = &[ - (TOTAL_BYTES, 0, "100%_in/0%_out"), - (TOTAL_BYTES / 2, TOTAL_BYTES / 2, "50%_in/50%_out"), - (0, TOTAL_BYTES, "0%_in/100%_out"), + (TOTAL_BYTES, 0, "100%_in_0%_out"), + (TOTAL_BYTES / 2, TOTAL_BYTES / 2, "50%_in_50%_out"), + (0, TOTAL_BYTES, "0%_in_100%_out"), ]; fn transfer_mix_timed( @@ -94,15 +97,13 @@ fn transfer_mix_timed( } fn benchmark_transfer_throughput(c: &mut Criterion) { - let mut group = c.benchmark_group("transfer_throughput_cuda"); - group.sample_size(10); - + let mut group = c.benchmark_group("cuda/throughput"); for &(input_bytes, output_bytes, label) in MIXES { let total_mem_bytes = input_bytes * 2 + output_bytes; group.throughput(Throughput::Bytes(total_mem_bytes as u64)); group.bench_with_input( - BenchmarkId::new("mix", label), + BenchmarkId::new(format!("transfer/{label}"), "100MiB"), &(input_bytes, output_bytes), |b, &(in_bytes, out_bytes)| { b.iter_custom(|iters| { @@ -123,7 +124,11 @@ fn benchmark_transfer_throughput(c: &mut Criterion) { group.finish(); } -criterion::criterion_group!(benches, benchmark_transfer_throughput); +criterion::criterion_group! { + name = benches; + config = bench_config::cuda_bench_config(); + targets = benchmark_transfer_throughput +} #[cuda_available] criterion::criterion_main!(benches); diff --git a/vortex-cuda/benches/common/mod.rs b/vortex-cuda/benches/timed_launch_strategy/mod.rs similarity index 79% rename from vortex-cuda/benches/common/mod.rs rename to vortex-cuda/benches/timed_launch_strategy/mod.rs index 71e21c85c34..9535f6e1c1c 100644 --- a/vortex-cuda/benches/common/mod.rs +++ b/vortex-cuda/benches/timed_launch_strategy/mod.rs @@ -16,6 +16,13 @@ pub struct TimedLaunchStrategy { pub total_time_ns: Arc, } +impl TimedLaunchStrategy { + /// Returns a shared handle to the accumulated kernel time, for reading after launches complete. + pub fn timer(&self) -> Arc { + Arc::clone(&self.total_time_ns) + } +} + impl LaunchStrategy for TimedLaunchStrategy { fn event_flags(&self) -> CUevent_flags { // using blocking_sync to make sure all events flush before we complete. @@ -24,6 +31,7 @@ impl LaunchStrategy for TimedLaunchStrategy { fn on_complete(&self, events: &CudaKernelEvents, _len: usize) -> VortexResult<()> { // NOTE: as long as the duration < 584 years this cast is safe. + #[allow(clippy::cast_possible_truncation)] let elapsed_nanos = events.duration()?.as_nanos() as u64; self.total_time_ns .fetch_add(elapsed_nanos, Ordering::Relaxed); diff --git a/vortex-cuda/benches/transpose_patches.rs b/vortex-cuda/benches/transpose_patches.rs deleted file mode 100644 index a4a4dfaa099..00000000000 --- a/vortex-cuda/benches/transpose_patches.rs +++ /dev/null @@ -1,76 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -#![allow(clippy::unwrap_used)] -#![allow(clippy::cast_possible_truncation)] - -use std::time::Duration; - -use criterion::BenchmarkId; -use criterion::Criterion; -use criterion::Throughput; -use futures::executor::block_on; -use vortex::array::IntoArray; -use vortex::array::arrays::PrimitiveArray; -use vortex::array::dtype::PType; -use vortex::array::patches::Patches; -use vortex::buffer::Buffer; -use vortex::buffer::buffer; -use vortex::session::VortexSession; -use vortex_array::validity::Validity; -use vortex_cuda::CudaSession; -use vortex_cuda::transpose_patches; -use vortex_cuda_macros::cuda_available; -use vortex_cuda_macros::cuda_not_available; -use vortex_error::VortexExpect; - -fn benchmark_transpose(c: &mut Criterion) { - let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) - .vortex_expect("failed to create execution context"); - - let patches = block_on(async { - // Assume that we have 64k values, and we have 1024 patches evenly disbursed across - // the range. - let indices = (0..1024).map(|x| x * 64).collect::>(); - - let values = buffer![-1.0f32; 1024]; - - let device_indices = cuda_ctx.copy_to_device(indices)?.await?; - let device_values = cuda_ctx.copy_to_device(values)?.await?; - - Patches::new( - 64 * 1024, - 0, - PrimitiveArray::from_buffer_handle(device_indices, PType::U32, Validity::NonNullable) - .into_array(), - PrimitiveArray::from_buffer_handle(device_values, PType::F32, Validity::NonNullable) - .into_array(), - None, - ) - }) - .unwrap(); - - let mut group = c.benchmark_group("transpose"); - group.sample_size(100); - group.measurement_time(Duration::from_secs(10)); - - group.throughput(Throughput::Bytes( - patches.indices().nbytes() + patches.values().nbytes(), - )); - - group.bench_with_input( - BenchmarkId::new("transpose_patches", 0), - &patches, - |b, patches| { - b.iter(|| block_on(async { transpose_patches(patches, &mut cuda_ctx).await.unwrap() })) - }, - ); -} - -criterion::criterion_group!(benches, benchmark_transpose); - -#[cuda_available] -criterion::criterion_main!(benches); - -#[cuda_not_available] -fn main() {} diff --git a/vortex-cuda/benches/zstd_cuda.rs b/vortex-cuda/benches/zstd_cuda.rs index a5d285d2d2a..6176422563a 100644 --- a/vortex-cuda/benches/zstd_cuda.rs +++ b/vortex-cuda/benches/zstd_cuda.rs @@ -1,8 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] -#![allow(clippy::cast_possible_truncation)] +mod bench_config; use std::time::Duration; @@ -28,11 +27,7 @@ use vortex_cuda::zstd_kernel_prepare; use vortex_cuda_macros::cuda_available; use vortex_cuda_macros::cuda_not_available; -const BENCH_ARGS: &[(usize, &str)] = &[ - (1_000_000, "1M"), - (10_000_000, "10M"), - (100_000_000, "100M"), -]; +use crate::bench_config::BENCH_SIZES; /// Generate compressible string data by repeating patterns. fn generate_string_data(count: usize) -> Vec<&'static str> { @@ -53,14 +48,22 @@ fn generate_string_data(count: usize) -> Vec<&'static str> { } /// Create a ZSTD-compressed array -fn make_zstd_array(num_strings: usize) -> VortexResult<(ZstdArray, usize)> { +fn make_zstd_array( + num_strings: usize, + cuda_ctx: &mut vortex_cuda::CudaExecutionCtx, +) -> VortexResult<(ZstdArray, usize)> { let strings = generate_string_data(num_strings); let var_bin_view = VarBinViewArray::from_iter_str(strings.iter().copied()); let uncompressed_size: usize = strings.iter().map(|s| s.len()).sum(); let zstd_compression_level = -10; // Less compression but faster. let zstd_array = // Disable dictionary as nvCOMP doesn't support ZSTD dictionaries. - Zstd::from_var_bin_view_without_dict(&var_bin_view, zstd_compression_level, 2048)?; + Zstd::from_var_bin_view_without_dict( + &var_bin_view, + zstd_compression_level, + 2048, + cuda_ctx.execution_ctx(), + )?; Ok((zstd_array, uncompressed_size)) } @@ -120,16 +123,17 @@ async fn execute_zstd_kernel( /// Benchmark ZSTD CUDA decompression kernel performance fn benchmark_zstd_cuda_decompress(c: &mut Criterion) { - let mut group = c.benchmark_group("ZSTD_cuda"); - group.sample_size(10); + let mut group = c.benchmark_group("cuda/zstd"); - for (num_strings, label) in BENCH_ARGS { - let (zstd_array, uncompressed_size) = - make_zstd_array(*num_strings).vortex_expect("failed to create ZSTD array"); + for (num_strings, label) in BENCH_SIZES { + let mut setup_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) + .vortex_expect("failed to create execution context"); + let (zstd_array, uncompressed_size) = make_zstd_array(*num_strings, &mut setup_ctx) + .vortex_expect("failed to create ZSTD array"); group.throughput(Throughput::Bytes(uncompressed_size as u64)); group.bench_with_input( - BenchmarkId::new("decompress_kernel", label), + BenchmarkId::new("decompress", label), &zstd_array, |b, zstd_array| { b.iter_custom(|iters| { @@ -143,7 +147,7 @@ fn benchmark_zstd_cuda_decompress(c: &mut Criterion) { frames, metadata, .. } = { let validity = child_to_validity( - &zstd_array.as_ref().slots()[0], + zstd_array.as_ref().slots()[0].as_ref(), zstd_array.dtype().nullability(), ); zstd_array.clone().into_data().into_parts(validity) @@ -165,7 +169,11 @@ fn benchmark_zstd_cuda_decompress(c: &mut Criterion) { group.finish(); } -criterion::criterion_group!(benches, benchmark_zstd_cuda_decompress); +criterion::criterion_group! { + name = benches; + config = bench_config::cuda_bench_config(); + targets = benchmark_zstd_cuda_decompress +} #[cuda_available] criterion::criterion_main!(benches); diff --git a/vortex-cuda/build.rs b/vortex-cuda/build.rs index fd035398b14..b8673cdda83 100644 --- a/vortex-cuda/build.rs +++ b/vortex-cuda/build.rs @@ -1,9 +1,9 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] -#![allow(clippy::expect_used)] -#![allow(clippy::use_debug)] +#![expect(clippy::unwrap_used)] +#![expect(clippy::expect_used)] +#![expect(clippy::use_debug)] use std::env; use std::fs::File; @@ -14,7 +14,8 @@ use std::process::Command; use fastlanes::FastLanes; -use crate::bit_unpack_gen::generate_cuda_unpack; +use crate::bit_unpack_gen::generate_cuda_unpack_kernels; +use crate::bit_unpack_gen::generate_cuda_unpack_lanes; #[path = "src/bit_unpack_gen.rs"] pub mod bit_unpack_gen; @@ -72,7 +73,11 @@ fn main() { match path.extension().and_then(|e| e.to_str()) { Some("cuh") | Some("h") => { - println!("cargo:rerun-if-changed={}", path.display()) + // Only watch hand-written .cuh/.h files, not generated ones + // (generated files are rebuilt when cuda_kernel_generator changes) + if !is_generated { + println!("cargo:rerun-if-changed={}", path.display()); + } } Some("cu") => { // Only watch hand-written .cu files, not generated ones @@ -94,10 +99,19 @@ fn main() { } fn generate_unpack(output_dir: &Path, thread_count: usize) -> io::Result { - let path = output_dir.join(format!("bit_unpack_{}.cu", T::T)); - let mut cu_file = File::create(&path)?; - generate_cuda_unpack::(&mut cu_file, thread_count)?; - Ok(path) + // Generate the lanes header (.cuh) — device functions only, no __global__ kernels. + // This is what dynamic_dispatch.cu includes (via bit_unpack.cuh). + let cuh_path = output_dir.join(format!("bit_unpack_{}_lanes.cuh", T::T)); + let mut cuh_file = File::create(&cuh_path)?; + generate_cuda_unpack_lanes::(&mut cuh_file)?; + + // Generate the standalone kernels (.cu) — includes the lanes header, + // adds _device template + __global__ wrappers. Compiled to its own PTX. + let cu_path = output_dir.join(format!("bit_unpack_{}.cu", T::T)); + let mut cu_file = File::create(&cu_path)?; + generate_cuda_unpack_kernels::(&mut cu_file, thread_count)?; + + Ok(cu_path) } fn nvcc_compile_ptx( diff --git a/vortex-cuda/cub/src/filter.rs b/vortex-cuda/cub/src/filter.rs index 5fdf5e474ed..fff38857706 100644 --- a/vortex-cuda/cub/src/filter.rs +++ b/vortex-cuda/cub/src/filter.rs @@ -29,7 +29,7 @@ pub trait CubFilterable: Copy + 'static { /// - `d_flags` must have at least `num_items` bytes (one per element, 0 or 1) /// - `d_out` must have enough space for selected elements /// - `d_num_selected` must point to a valid i64 on the device - #[allow(clippy::too_many_arguments)] + #[expect(clippy::too_many_arguments)] unsafe fn filter_bytemask( d_temp: *mut c_void, temp_bytes: usize, @@ -55,7 +55,7 @@ pub trait CubFilterable: Copy + 'static { /// - `d_bitmask` must contain enough bytes to hold `bit_offset + num_items` bits /// - `d_out` must have enough space for selected elements /// - `d_num_selected` must point to a valid i64 on the device - #[allow(clippy::too_many_arguments)] + #[expect(clippy::too_many_arguments)] unsafe fn filter_bitmask( d_temp: *mut c_void, temp_bytes: usize, @@ -135,7 +135,7 @@ macro_rules! impl_filter { /// - `d_flags` must have at least `num_items` bytes (one per element, 0 or 1) /// - `d_out` must have enough space for selected elements /// - `d_num_selected` must point to a valid i64 on the device - #[allow(clippy::too_many_arguments)] + #[expect(clippy::too_many_arguments)] pub unsafe fn []( d_temp: *mut c_void, temp_bytes: usize, @@ -175,7 +175,7 @@ macro_rules! impl_filter { /// - `d_bitmask` must contain enough bytes to hold `bit_offset + num_items` bits /// - `d_out` must have enough space for selected elements /// - `d_num_selected` must point to a valid i64 on the device - #[allow(clippy::too_many_arguments)] + #[expect(clippy::too_many_arguments)] pub unsafe fn []( d_temp: *mut c_void, temp_bytes: usize, diff --git a/vortex-cuda/cub/src/lib.rs b/vortex-cuda/cub/src/lib.rs index 706373255aa..c0532576604 100644 --- a/vortex-cuda/cub/src/lib.rs +++ b/vortex-cuda/cub/src/lib.rs @@ -18,13 +18,7 @@ use std::path::PathBuf; use std::sync::OnceLock; /// Raw FFI type definitions and dynamically-loaded function pointers from bindgen. -#[allow( - non_upper_case_globals, - non_camel_case_types, - non_snake_case, - dead_code, - clippy::all -)] +#[allow(non_camel_case_types, dead_code, clippy::all)] pub mod sys; mod error; diff --git a/vortex-cuda/gpu-scan-cli/src/main.rs b/vortex-cuda/gpu-scan-cli/src/main.rs index 4c343742000..9078e30c691 100644 --- a/vortex-cuda/gpu-scan-cli/src/main.rs +++ b/vortex-cuda/gpu-scan-cli/src/main.rs @@ -18,8 +18,9 @@ use tracing_subscriber::fmt::format::FmtSpan; use tracing_subscriber::layer::SubscriberExt; use tracing_subscriber::util::SubscriberInitExt; use vortex::VortexSessionDefault; -use vortex::array::ToCanonical; +use vortex::array::VortexSessionExecute; use vortex::array::arrays::Dict; +use vortex::array::arrays::StructArray; use vortex::array::arrays::struct_::StructArrayExt; use vortex::buffer::ByteBufferMut; use vortex::compressor::BtrBlocksCompressorBuilder; @@ -178,7 +179,7 @@ async fn cmd_scan(path: PathBuf, gpu_file: bool, json_output: bool) -> VortexRes let mut chunk = 0; while let Some(next) = batches.next().await.transpose()? { - let record = next.to_struct(); + let record = next.execute::(cuda_ctx.execution_ctx())?; for (field, field_name) in record .iter_unmasked_fields() diff --git a/vortex-cuda/kernels/src/alp.cu b/vortex-cuda/kernels/src/alp.cu index e34285d9d9b..95502131257 100644 --- a/vortex-cuda/kernels/src/alp.cu +++ b/vortex-cuda/kernels/src/alp.cu @@ -1,36 +1,92 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#include "scalar_kernel.cuh" - -// ALP (Adaptive Lossless floating-Point) decode operation. -// Converts integers to floats by multiplying by precomputed exponent factors. -// Formula: decoded = (float)encoded * f * e -// Where f = F10[exponents.f] and e = IF10[exponents.e] are passed directly. -template -struct AlpOp { - FloatT f; // F10[exponents.f] - power of 10 - FloatT e; // IF10[exponents.e] - inverse power of 10 - - __device__ inline FloatT operator()(EncodedT value) const { - return static_cast(value) * f * e; +#include "patches.cuh" + +// ALP (Adaptive Lossless floating-Point) decode: out[i] = (FloatT)in[i] * f * e. +// +// Each block processes one 1024-element chunk cooperatively and applies patches +// into shared memory before writing to global memory, mirroring the strategy +// used by bit_unpack. f = F10[exponents.f], e = IF10[exponents.e]. +// +// The cast from EncT to FloatT must preserve ALP's lossless contract: f32 is +// only encoded as i32, and f64 is only encoded as i64. The i64 → double cast +// is lossless for all values ALP can produce. +template +__device__ void alp_device(const EncT *__restrict in, + FloatT *__restrict out, + FloatT f, + FloatT e, + uint64_t array_len, + int thread_idx, + GPUPatches &patches) { + constexpr int ThreadCount = 32; + // ThreadCount == 32 (one warp) is baked into this kernel: + // - __syncwarp() below is only sufficient because all threads live in one warp. + // - per_thread must evenly divide 1024 so the unrolled loops cover the chunk. + static_assert(ThreadCount == 32, "alp kernel requires exactly one warp per block"); + static_assert(1024 % ThreadCount == 0, "ThreadCount must evenly divide 1024"); + __shared__ FloatT shared_out[1024]; + + constexpr int per_thread = 1024 / ThreadCount; + uint64_t chunk_base = static_cast(blockIdx.x) * 1024; + + // Step 1: decode the chunk into shared memory. The tail block is bounds-checked; + // all interior blocks take the fast path with no per-element branch. + if (chunk_base + 1024 <= array_len) { +#pragma unroll + for (int i = 0; i < per_thread; i++) { + int idx = i * ThreadCount + thread_idx; + shared_out[idx] = static_cast(in[idx]) * f * e; + } + } else { +#pragma unroll + for (int i = 0; i < per_thread; i++) { + int idx = i * ThreadCount + thread_idx; + uint64_t global_idx = chunk_base + static_cast(idx); + if (global_idx < array_len) { + shared_out[idx] = static_cast(in[idx]) * f * e; + } else { + shared_out[idx] = FloatT {}; + } + } } -}; - -// Macro to generate ALP kernel for each type combination. -// Input is integer (encoded), output is float (decoded). -#define GENERATE_ALP_KERNEL(enc_suffix, float_suffix, EncType, FloatType) \ - extern "C" __global__ void alp_##enc_suffix##_##float_suffix(const EncType *__restrict encoded, \ - FloatType *__restrict decoded, \ - FloatType f, \ - FloatType e, \ - uint64_t array_len) { \ - scalar_kernel(encoded, decoded, array_len, AlpOp {f, e}); \ + __syncwarp(); + + // Step 2: apply patches in parallel across the warp. + PatchesCursor cursor(patches, blockIdx.x, thread_idx, static_cast(ThreadCount)); + auto patch = cursor.next(); + while (patch.index != 1024) { + shared_out[patch.index] = patch.value; + patch = cursor.next(); } + __syncwarp(); -// f32 variants (ALP for f32 encodes as i32 or i64) -GENERATE_ALP_KERNEL(i32, f32, int32_t, float) -GENERATE_ALP_KERNEL(i64, f32, int64_t, float) +// Step 3: coalesced write-out of the full 1024-element chunk. The caller +// allocates `full_out` rounded up to a multiple of 1024, so every block +// writes entirely within bounds. Positions in `[array_len, rounded_len)` +// of the tail chunk hold don't-care values; the caller slices them off. +#pragma unroll + for (int i = 0; i < per_thread; i++) { + int idx = i * ThreadCount + thread_idx; + out[idx] = shared_out[idx]; + } +} + +#define GENERATE_ALP_KERNEL(enc_suffix, float_suffix, EncT, FloatT) \ + extern "C" __global__ void alp_##enc_suffix##_##float_suffix##_32t(const EncT *__restrict full_in, \ + FloatT *__restrict full_out, \ + FloatT f, \ + FloatT e, \ + uint64_t array_len, \ + GPUPatches patches) { \ + int thread_idx = threadIdx.x; \ + auto in = full_in + (blockIdx.x * 1024); \ + auto out = full_out + (blockIdx.x * 1024); \ + alp_device(in, out, f, e, array_len, thread_idx, patches); \ + } -// f64 variants (ALP for f64 encodes as i64) +// The only ALPInt bindings produced by the encoder are (f32, i32) and (f64, i64). +// i64 → double is lossless; i32 → float is lossless for all values ALP emits. +GENERATE_ALP_KERNEL(i32, f32, int32_t, float) GENERATE_ALP_KERNEL(i64, f64, int64_t, double) diff --git a/vortex-cuda/kernels/src/bit_unpack.cuh b/vortex-cuda/kernels/src/bit_unpack.cuh index fb814e3f4fa..8e6bc9fec93 100644 --- a/vortex-cuda/kernels/src/bit_unpack.cuh +++ b/vortex-cuda/kernels/src/bit_unpack.cuh @@ -7,10 +7,10 @@ #include #include -#include "bit_unpack_8.cu" -#include "bit_unpack_16.cu" -#include "bit_unpack_32.cu" -#include "bit_unpack_64.cu" +#include "bit_unpack_8_lanes.cuh" +#include "bit_unpack_16_lanes.cuh" +#include "bit_unpack_32_lanes.cuh" +#include "bit_unpack_64_lanes.cuh" #include "patches.h" /// Decodes a single lane of packed data. @@ -26,22 +26,22 @@ /// * `lane` - Lane index within the block (used to determine which packed words to process) /// * `bit_width` - Number of bits with which each value is encoded template -__device__ inline void bit_unpack_lane(const T *__restrict packed_chunk, - T *__restrict output_buffer, - T reference, - unsigned int lane, - uint32_t bit_width); +__device__ __noinline__ void bit_unpack_lane(const T *__restrict packed_chunk, + T *__restrict output_buffer, + T reference, + unsigned int lane, + uint32_t bit_width); /// Template specializations for `bitunpack_lane_to_smem` for different integer types. /// /// Generates template specializations for each supported integer size (8, 16, 32, 64 bits). #define BIT_UNPACK_LANE(bits) \ template <> \ - __device__ inline void bit_unpack_lane(const uint##bits##_t *in, \ - uint##bits##_t *out, \ - uint##bits##_t reference, \ - unsigned int lane, \ - uint32_t bw) { \ + __device__ __noinline__ void bit_unpack_lane(const uint##bits##_t *in, \ + uint##bits##_t *out, \ + uint##bits##_t reference, \ + unsigned int lane, \ + uint32_t bw) { \ bit_unpack_##bits##_lane(in, out, reference, lane, bw); \ } diff --git a/vortex-cuda/kernels/src/bit_unpack_16.cu b/vortex-cuda/kernels/src/bit_unpack_16.cu index c329e85157a..a784df201d3 100644 --- a/vortex-cuda/kernels/src/bit_unpack_16.cu +++ b/vortex-cuda/kernels/src/bit_unpack_16.cu @@ -1,1006 +1,151 @@ // AUTO-GENERATED. Do not edit by hand! -#include -#include -#include -#include "fastlanes_common.cuh" +#include "bit_unpack_16_lanes.cuh" #include "patches.cuh" template -__device__ void _bit_unpack_16_lane(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane); - -template <> -__device__ void _bit_unpack_16_lane<0>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { - #pragma unroll - for (int row = 0; row < 16; row++) { - out[INDEX(row, lane)] = reference; - } -} - -template <> -__device__ void _bit_unpack_16_lane<1>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 64; - uint16_t src; - uint16_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint16_t, 1); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint16_t, 1); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint16_t, 1); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint16_t, 1); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint16_t, 1); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint16_t, 1); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint16_t, 1); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint16_t, 1); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 1); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint16_t, 1); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint16_t, 1); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint16_t, 1); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint16_t, 1); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint16_t, 1); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint16_t, 1); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint16_t, 1); - out[INDEX(15, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_16_lane<2>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 64; - uint16_t src; - uint16_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint16_t, 2); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint16_t, 2); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint16_t, 2); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint16_t, 2); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 2); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint16_t, 2); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint16_t, 2); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint16_t, 2); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint16_t, 0)) << 2; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint16_t, 2); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint16_t, 2); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint16_t, 2); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint16_t, 2); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 2); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint16_t, 2); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint16_t, 2); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint16_t, 2); - out[INDEX(15, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_16_lane<3>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 64; - uint16_t src; - uint16_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint16_t, 3); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint16_t, 3); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint16_t, 3); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint16_t, 3); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint16_t, 3); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint16_t, 1); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint16_t, 2)) << 1; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint16_t, 3); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint16_t, 3); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 3); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint16_t, 3); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint16_t, 2); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint16_t, 1)) << 2; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint16_t, 3); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint16_t, 3); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint16_t, 3); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint16_t, 3); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint16_t, 3); - out[INDEX(15, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_16_lane<4>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 64; - uint16_t src; - uint16_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint16_t, 4); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint16_t, 4); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 4); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint16_t, 4); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint16_t, 0)) << 4; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint16_t, 4); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint16_t, 4); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 4); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint16_t, 4); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint16_t, 0)) << 4; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint16_t, 4); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint16_t, 4); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 4); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint16_t, 4); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint16_t, 0)) << 4; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint16_t, 4); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint16_t, 4); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 4); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint16_t, 4); - out[INDEX(15, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_16_lane<5>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 64; - uint16_t src; - uint16_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint16_t, 5); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint16_t, 5); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint16_t, 5); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint16_t, 1); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint16_t, 4)) << 1; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint16_t, 5); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint16_t, 5); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint16_t, 2); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint16_t, 3)) << 2; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint16_t, 5); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 5); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint16_t, 3); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint16_t, 2)) << 3; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint16_t, 5); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint16_t, 5); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint16_t, 4); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint16_t, 1)) << 4; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint16_t, 5); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint16_t, 5); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint16_t, 5); - out[INDEX(15, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_16_lane<6>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 64; - uint16_t src; - uint16_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint16_t, 6); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint16_t, 6); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint16_t, 4); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint16_t, 2)) << 4; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint16_t, 6); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 6); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint16_t, 2); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint16_t, 4)) << 2; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint16_t, 6); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint16_t, 6); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint16_t, 0)) << 6; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint16_t, 6); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint16_t, 6); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint16_t, 4); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint16_t, 2)) << 4; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint16_t, 6); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 6); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint16_t, 2); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint16_t, 4)) << 2; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint16_t, 6); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint16_t, 6); - out[INDEX(15, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_16_lane<7>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 64; - uint16_t src; - uint16_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint16_t, 7); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint16_t, 7); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint16_t, 2); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint16_t, 5)) << 2; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint16_t, 7); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint16_t, 4); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint16_t, 3)) << 4; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint16_t, 7); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint16_t, 6); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint16_t, 1)) << 6; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint16_t, 7); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 7); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint16_t, 1); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint16_t, 6)) << 1; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint16_t, 7); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint16_t, 3); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint16_t, 4)) << 3; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint16_t, 7); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint16_t, 5); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint16_t, 2)) << 5; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint16_t, 7); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint16_t, 7); - out[INDEX(15, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_16_lane<8>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 64; - uint16_t src; - uint16_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint16_t, 8); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 8); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint16_t, 0)) << 8; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint16_t, 8); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 8); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint16_t, 0)) << 8; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint16_t, 8); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 8); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint16_t, 0)) << 8; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint16_t, 8); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 8); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint16_t, 0)) << 8; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint16_t, 8); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 8); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint16_t, 0)) << 8; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint16_t, 8); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 8); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint16_t, 0)) << 8; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint16_t, 8); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 8); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint16_t, 0)) << 8; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint16_t, 8); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 8); - out[INDEX(15, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_16_lane<9>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 64; - uint16_t src; - uint16_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint16_t, 9); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint16_t, 7); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint16_t, 2)) << 7; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint16_t, 9); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint16_t, 5); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint16_t, 4)) << 5; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint16_t, 9); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint16_t, 3); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint16_t, 6)) << 3; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint16_t, 9); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint16_t, 1); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint16_t, 8)) << 1; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 8); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint16_t, 1)) << 8; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint16_t, 9); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint16_t, 6); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint16_t, 3)) << 6; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint16_t, 9); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint16_t, 4); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint16_t, 5)) << 4; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint16_t, 9); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint16_t, 2); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint16_t, 7)) << 2; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint16_t, 9); - out[INDEX(15, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_16_lane<10>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 64; - uint16_t src; - uint16_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint16_t, 10); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint16_t, 6); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint16_t, 4)) << 6; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint16_t, 10); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint16_t, 2); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint16_t, 8)) << 2; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 8); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint16_t, 2)) << 8; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint16_t, 10); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint16_t, 4); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint16_t, 6)) << 4; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint16_t, 10); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint16_t, 0)) << 10; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint16_t, 10); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint16_t, 6); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint16_t, 4)) << 6; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint16_t, 10); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint16_t, 2); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint16_t, 8)) << 2; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 8); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint16_t, 2)) << 8; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint16_t, 10); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint16_t, 4); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint16_t, 6)) << 4; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint16_t, 10); - out[INDEX(15, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_16_lane<11>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 64; - uint16_t src; - uint16_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint16_t, 11); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint16_t, 5); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint16_t, 6)) << 5; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint16_t, 10); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint16_t, 1)) << 10; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint16_t, 11); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint16_t, 4); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint16_t, 7)) << 4; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint16_t, 9); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint16_t, 2)) << 9; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint16_t, 11); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint16_t, 3); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint16_t, 8)) << 3; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 8); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint16_t, 3)) << 8; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint16_t, 11); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint16_t, 2); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint16_t, 9)) << 2; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint16_t, 7); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint16_t, 4)) << 7; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint16_t, 11); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint16_t, 1); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint16_t, 10)) << 1; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint16_t, 6); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint16_t, 5)) << 6; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint16_t, 11); - out[INDEX(15, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_16_lane<12>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 64; - uint16_t src; - uint16_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint16_t, 12); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint16_t, 4); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint16_t, 8)) << 4; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 8); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint16_t, 4)) << 8; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint16_t, 12); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint16_t, 0)) << 12; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint16_t, 12); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint16_t, 4); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint16_t, 8)) << 4; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 8); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint16_t, 4)) << 8; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint16_t, 12); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint16_t, 0)) << 12; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint16_t, 12); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint16_t, 4); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint16_t, 8)) << 4; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 8); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint16_t, 4)) << 8; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint16_t, 12); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint16_t, 0)) << 12; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint16_t, 12); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint16_t, 4); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint16_t, 8)) << 4; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 8); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint16_t, 4)) << 8; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint16_t, 12); - out[INDEX(15, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_16_lane<13>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 64; - uint16_t src; - uint16_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint16_t, 13); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint16_t, 3); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint16_t, 10)) << 3; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint16_t, 6); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint16_t, 7)) << 6; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint16_t, 9); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint16_t, 4)) << 9; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint16_t, 12); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint16_t, 1)) << 12; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint16_t, 13); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint16_t, 2); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint16_t, 11)) << 2; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint16_t, 5); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint16_t, 8)) << 5; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 8); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint16_t, 5)) << 8; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint16_t, 11); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint16_t, 2)) << 11; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint16_t, 13); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint16_t, 1); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint16_t, 12)) << 1; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint16_t, 4); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint16_t, 9)) << 4; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint16_t, 7); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint16_t, 6)) << 7; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint16_t, 10); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint16_t, 3)) << 10; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint16_t, 13); - out[INDEX(15, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_16_lane<14>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 64; - uint16_t src; - uint16_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint16_t, 14); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint16_t, 2); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint16_t, 12)) << 2; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint16_t, 4); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint16_t, 10)) << 4; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint16_t, 6); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint16_t, 8)) << 6; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 8); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint16_t, 6)) << 8; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint16_t, 10); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint16_t, 4)) << 10; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint16_t, 12); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint16_t, 2)) << 12; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint16_t, 14); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint16_t, 0)) << 14; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint16_t, 14); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint16_t, 2); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint16_t, 12)) << 2; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint16_t, 4); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint16_t, 10)) << 4; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint16_t, 6); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint16_t, 8)) << 6; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 8); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint16_t, 6)) << 8; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint16_t, 10); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint16_t, 4)) << 10; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint16_t, 12); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint16_t, 2)) << 12; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint16_t, 14); - out[INDEX(15, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_16_lane<15>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 64; - uint16_t src; - uint16_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint16_t, 15); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint16_t, 1); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint16_t, 14)) << 1; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint16_t, 2); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint16_t, 13)) << 2; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint16_t, 3); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint16_t, 12)) << 3; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint16_t, 4); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint16_t, 11)) << 4; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint16_t, 5); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint16_t, 10)) << 5; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint16_t, 6); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint16_t, 9)) << 6; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint16_t, 7); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint16_t, 8)) << 7; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint16_t, 8); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint16_t, 7)) << 8; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint16_t, 9); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint16_t, 6)) << 9; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint16_t, 10); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint16_t, 5)) << 10; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint16_t, 11); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint16_t, 4)) << 11; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint16_t, 12); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint16_t, 3)) << 12; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint16_t, 13); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint16_t, 2)) << 13; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint16_t, 14); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint16_t, 1)) << 14; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint16_t, 15); - out[INDEX(15, lane)] = tmp + reference; -} +__device__ void _bit_unpack_16_device(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, int thread_idx, GPUPatches& patches) { + __shared__ uint16_t shared_out[FL_CHUNK]; -template <> -__device__ void _bit_unpack_16_lane<16>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 64; + // Step 1: Unpack into shared memory #pragma unroll - for (int row = 0; row < 16; row++) { - out[INDEX(row, lane)] = in[LANE_COUNT * row + lane] + reference; + for (int i = 0; i < FL_LANES / 32; i++) { + _bit_unpack_16_lane(in, shared_out, reference, thread_idx * (FL_LANES / 32) + i); } -} + __syncwarp(); -/// Runtime dispatch to the optimized lane decoder for the given bit width. -__device__ inline void bit_unpack_16_lane( - const uint16_t *__restrict in, - uint16_t *__restrict out, - uint16_t reference, - unsigned int lane, - uint32_t bit_width -) { - switch (bit_width) { - case 0: _bit_unpack_16_lane<0>(in, out, reference, lane); break; - case 1: _bit_unpack_16_lane<1>(in, out, reference, lane); break; - case 2: _bit_unpack_16_lane<2>(in, out, reference, lane); break; - case 3: _bit_unpack_16_lane<3>(in, out, reference, lane); break; - case 4: _bit_unpack_16_lane<4>(in, out, reference, lane); break; - case 5: _bit_unpack_16_lane<5>(in, out, reference, lane); break; - case 6: _bit_unpack_16_lane<6>(in, out, reference, lane); break; - case 7: _bit_unpack_16_lane<7>(in, out, reference, lane); break; - case 8: _bit_unpack_16_lane<8>(in, out, reference, lane); break; - case 9: _bit_unpack_16_lane<9>(in, out, reference, lane); break; - case 10: _bit_unpack_16_lane<10>(in, out, reference, lane); break; - case 11: _bit_unpack_16_lane<11>(in, out, reference, lane); break; - case 12: _bit_unpack_16_lane<12>(in, out, reference, lane); break; - case 13: _bit_unpack_16_lane<13>(in, out, reference, lane); break; - case 14: _bit_unpack_16_lane<14>(in, out, reference, lane); break; - case 15: _bit_unpack_16_lane<15>(in, out, reference, lane); break; - case 16: _bit_unpack_16_lane<16>(in, out, reference, lane); break; + // Step 2: Apply patches to shared memory in parallel + PatchesCursor cursor(patches, blockIdx.x, thread_idx, 32); + auto patch = cursor.next(); + while (patch.index != FL_CHUNK) { + shared_out[patch.index] = patch.value; + patch = cursor.next(); } -} + __syncwarp(); -template -__device__ void _bit_unpack_16_device(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, int thread_idx, GPUPatches& patches) { - __shared__ uint16_t shared_out[1024]; + // Step 3: Copy to global memory #pragma unroll - for (int i = 0; i < 2; i++) { - _bit_unpack_16_lane(in, shared_out, reference, thread_idx * 2 + i); - } - __syncwarp(); - PatchesCursor cursor(patches, blockIdx.x, thread_idx, 32); - auto patch = cursor.next(); - for (int i = 0; i < 32; i++) { + for (int i = 0; i < FL_CHUNK / 32; i++) { auto idx = i * 32 + thread_idx; - if (idx == patch.index) { - out[idx] = patch.value; - patch = cursor.next(); - } else { - out[idx] = shared_out[idx]; - } + out[idx] = shared_out[idx]; } } extern "C" __global__ void bit_unpack_16_0bw_32t(const uint16_t *__restrict full_in, uint16_t *__restrict full_out, uint16_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 0 / sizeof(uint16_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 0)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_16_device<0>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_16_1bw_32t(const uint16_t *__restrict full_in, uint16_t *__restrict full_out, uint16_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 1 / sizeof(uint16_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 1)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_16_device<1>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_16_2bw_32t(const uint16_t *__restrict full_in, uint16_t *__restrict full_out, uint16_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 2 / sizeof(uint16_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 2)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_16_device<2>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_16_3bw_32t(const uint16_t *__restrict full_in, uint16_t *__restrict full_out, uint16_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 3 / sizeof(uint16_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 3)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_16_device<3>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_16_4bw_32t(const uint16_t *__restrict full_in, uint16_t *__restrict full_out, uint16_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 4 / sizeof(uint16_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 4)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_16_device<4>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_16_5bw_32t(const uint16_t *__restrict full_in, uint16_t *__restrict full_out, uint16_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 5 / sizeof(uint16_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 5)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_16_device<5>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_16_6bw_32t(const uint16_t *__restrict full_in, uint16_t *__restrict full_out, uint16_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 6 / sizeof(uint16_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 6)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_16_device<6>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_16_7bw_32t(const uint16_t *__restrict full_in, uint16_t *__restrict full_out, uint16_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 7 / sizeof(uint16_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 7)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_16_device<7>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_16_8bw_32t(const uint16_t *__restrict full_in, uint16_t *__restrict full_out, uint16_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 8 / sizeof(uint16_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 8)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_16_device<8>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_16_9bw_32t(const uint16_t *__restrict full_in, uint16_t *__restrict full_out, uint16_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 9 / sizeof(uint16_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 9)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_16_device<9>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_16_10bw_32t(const uint16_t *__restrict full_in, uint16_t *__restrict full_out, uint16_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 10 / sizeof(uint16_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 10)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_16_device<10>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_16_11bw_32t(const uint16_t *__restrict full_in, uint16_t *__restrict full_out, uint16_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 11 / sizeof(uint16_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 11)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_16_device<11>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_16_12bw_32t(const uint16_t *__restrict full_in, uint16_t *__restrict full_out, uint16_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 12 / sizeof(uint16_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 12)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_16_device<12>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_16_13bw_32t(const uint16_t *__restrict full_in, uint16_t *__restrict full_out, uint16_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 13 / sizeof(uint16_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 13)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_16_device<13>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_16_14bw_32t(const uint16_t *__restrict full_in, uint16_t *__restrict full_out, uint16_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 14 / sizeof(uint16_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 14)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_16_device<14>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_16_15bw_32t(const uint16_t *__restrict full_in, uint16_t *__restrict full_out, uint16_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 15 / sizeof(uint16_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 15)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_16_device<15>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_16_16bw_32t(const uint16_t *__restrict full_in, uint16_t *__restrict full_out, uint16_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 16 / sizeof(uint16_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 16)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_16_device<16>(in, out, reference, thread_idx, patches); } diff --git a/vortex-cuda/kernels/src/bit_unpack_16_lanes.cuh b/vortex-cuda/kernels/src/bit_unpack_16_lanes.cuh new file mode 100644 index 00000000000..ff992fa4ffb --- /dev/null +++ b/vortex-cuda/kernels/src/bit_unpack_16_lanes.cuh @@ -0,0 +1,867 @@ +// AUTO-GENERATED. Do not edit by hand! +#pragma once + +#include +#include +#include +#include "fastlanes_common.cuh" + +template +__device__ void _bit_unpack_16_lane(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane); + +template <> +__device__ void _bit_unpack_16_lane<0>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { + #pragma unroll + for (int row = 0; row < 16; row++) { + out[INDEX(row, lane)] = reference; + } +} + +template <> +__device__ void _bit_unpack_16_lane<1>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint16_t src; + uint16_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint16_t, 1); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint16_t, 1); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint16_t, 1); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint16_t, 1); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint16_t, 1); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint16_t, 1); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint16_t, 1); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint16_t, 1); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 1); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint16_t, 1); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint16_t, 1); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint16_t, 1); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint16_t, 1); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint16_t, 1); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint16_t, 1); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint16_t, 1); + out[INDEX(15, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_16_lane<2>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint16_t src; + uint16_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint16_t, 2); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint16_t, 2); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint16_t, 2); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint16_t, 2); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 2); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint16_t, 2); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint16_t, 2); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint16_t, 2); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint16_t, 0)) << 2; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint16_t, 2); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint16_t, 2); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint16_t, 2); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint16_t, 2); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 2); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint16_t, 2); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint16_t, 2); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint16_t, 2); + out[INDEX(15, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_16_lane<3>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint16_t src; + uint16_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint16_t, 3); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint16_t, 3); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint16_t, 3); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint16_t, 3); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint16_t, 3); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint16_t, 1); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint16_t, 2)) << 1; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint16_t, 3); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint16_t, 3); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 3); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint16_t, 3); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint16_t, 2); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint16_t, 1)) << 2; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint16_t, 3); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint16_t, 3); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint16_t, 3); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint16_t, 3); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint16_t, 3); + out[INDEX(15, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_16_lane<4>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint16_t src; + uint16_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint16_t, 4); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint16_t, 4); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 4); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint16_t, 4); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint16_t, 0)) << 4; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint16_t, 4); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint16_t, 4); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 4); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint16_t, 4); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint16_t, 0)) << 4; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint16_t, 4); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint16_t, 4); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 4); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint16_t, 4); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint16_t, 0)) << 4; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint16_t, 4); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint16_t, 4); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 4); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint16_t, 4); + out[INDEX(15, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_16_lane<5>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint16_t src; + uint16_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint16_t, 5); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint16_t, 5); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint16_t, 5); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint16_t, 1); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint16_t, 4)) << 1; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint16_t, 5); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint16_t, 5); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint16_t, 2); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint16_t, 3)) << 2; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint16_t, 5); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 5); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint16_t, 3); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint16_t, 2)) << 3; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint16_t, 5); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint16_t, 5); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint16_t, 4); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint16_t, 1)) << 4; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint16_t, 5); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint16_t, 5); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint16_t, 5); + out[INDEX(15, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_16_lane<6>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint16_t src; + uint16_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint16_t, 6); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint16_t, 6); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint16_t, 4); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint16_t, 2)) << 4; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint16_t, 6); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 6); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint16_t, 2); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint16_t, 4)) << 2; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint16_t, 6); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint16_t, 6); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint16_t, 0)) << 6; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint16_t, 6); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint16_t, 6); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint16_t, 4); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint16_t, 2)) << 4; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint16_t, 6); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 6); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint16_t, 2); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint16_t, 4)) << 2; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint16_t, 6); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint16_t, 6); + out[INDEX(15, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_16_lane<7>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint16_t src; + uint16_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint16_t, 7); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint16_t, 7); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint16_t, 2); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint16_t, 5)) << 2; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint16_t, 7); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint16_t, 4); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint16_t, 3)) << 4; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint16_t, 7); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint16_t, 6); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint16_t, 1)) << 6; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint16_t, 7); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 7); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint16_t, 1); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint16_t, 6)) << 1; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint16_t, 7); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint16_t, 3); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint16_t, 4)) << 3; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint16_t, 7); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint16_t, 5); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint16_t, 2)) << 5; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint16_t, 7); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint16_t, 7); + out[INDEX(15, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_16_lane<8>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint16_t src; + uint16_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint16_t, 8); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 8); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint16_t, 0)) << 8; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint16_t, 8); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 8); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint16_t, 0)) << 8; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint16_t, 8); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 8); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint16_t, 0)) << 8; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint16_t, 8); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 8); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint16_t, 0)) << 8; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint16_t, 8); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 8); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint16_t, 0)) << 8; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint16_t, 8); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 8); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint16_t, 0)) << 8; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint16_t, 8); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 8); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint16_t, 0)) << 8; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint16_t, 8); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 8); + out[INDEX(15, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_16_lane<9>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint16_t src; + uint16_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint16_t, 9); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint16_t, 7); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint16_t, 2)) << 7; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint16_t, 9); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint16_t, 5); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint16_t, 4)) << 5; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint16_t, 9); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint16_t, 3); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint16_t, 6)) << 3; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint16_t, 9); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint16_t, 1); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint16_t, 8)) << 1; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 8); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint16_t, 1)) << 8; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint16_t, 9); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint16_t, 6); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint16_t, 3)) << 6; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint16_t, 9); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint16_t, 4); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint16_t, 5)) << 4; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint16_t, 9); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint16_t, 2); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint16_t, 7)) << 2; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint16_t, 9); + out[INDEX(15, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_16_lane<10>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint16_t src; + uint16_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint16_t, 10); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint16_t, 6); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint16_t, 4)) << 6; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint16_t, 10); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint16_t, 2); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint16_t, 8)) << 2; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 8); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint16_t, 2)) << 8; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint16_t, 10); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint16_t, 4); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint16_t, 6)) << 4; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint16_t, 10); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint16_t, 0)) << 10; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint16_t, 10); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint16_t, 6); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint16_t, 4)) << 6; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint16_t, 10); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint16_t, 2); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint16_t, 8)) << 2; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 8); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint16_t, 2)) << 8; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint16_t, 10); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint16_t, 4); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint16_t, 6)) << 4; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint16_t, 10); + out[INDEX(15, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_16_lane<11>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint16_t src; + uint16_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint16_t, 11); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint16_t, 5); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint16_t, 6)) << 5; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint16_t, 10); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint16_t, 1)) << 10; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint16_t, 11); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint16_t, 4); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint16_t, 7)) << 4; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint16_t, 9); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint16_t, 2)) << 9; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint16_t, 11); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint16_t, 3); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint16_t, 8)) << 3; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 8); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint16_t, 3)) << 8; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint16_t, 11); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint16_t, 2); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint16_t, 9)) << 2; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint16_t, 7); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint16_t, 4)) << 7; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint16_t, 11); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint16_t, 1); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint16_t, 10)) << 1; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint16_t, 6); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint16_t, 5)) << 6; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint16_t, 11); + out[INDEX(15, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_16_lane<12>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint16_t src; + uint16_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint16_t, 12); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint16_t, 4); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint16_t, 8)) << 4; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 8); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint16_t, 4)) << 8; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint16_t, 12); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint16_t, 0)) << 12; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint16_t, 12); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint16_t, 4); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint16_t, 8)) << 4; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 8); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint16_t, 4)) << 8; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint16_t, 12); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint16_t, 0)) << 12; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint16_t, 12); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint16_t, 4); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint16_t, 8)) << 4; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 8); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint16_t, 4)) << 8; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint16_t, 12); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint16_t, 0)) << 12; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint16_t, 12); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint16_t, 4); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint16_t, 8)) << 4; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 8); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint16_t, 4)) << 8; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint16_t, 12); + out[INDEX(15, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_16_lane<13>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint16_t src; + uint16_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint16_t, 13); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint16_t, 3); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint16_t, 10)) << 3; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint16_t, 6); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint16_t, 7)) << 6; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint16_t, 9); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint16_t, 4)) << 9; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint16_t, 12); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint16_t, 1)) << 12; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint16_t, 13); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint16_t, 2); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint16_t, 11)) << 2; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint16_t, 5); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint16_t, 8)) << 5; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 8); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint16_t, 5)) << 8; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint16_t, 11); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint16_t, 2)) << 11; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint16_t, 13); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint16_t, 1); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint16_t, 12)) << 1; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint16_t, 4); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint16_t, 9)) << 4; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint16_t, 7); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint16_t, 6)) << 7; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint16_t, 10); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint16_t, 3)) << 10; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint16_t, 13); + out[INDEX(15, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_16_lane<14>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint16_t src; + uint16_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint16_t, 14); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint16_t, 2); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint16_t, 12)) << 2; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint16_t, 4); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint16_t, 10)) << 4; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint16_t, 6); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint16_t, 8)) << 6; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 8); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint16_t, 6)) << 8; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint16_t, 10); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint16_t, 4)) << 10; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint16_t, 12); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint16_t, 2)) << 12; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint16_t, 14); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint16_t, 0)) << 14; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint16_t, 14); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint16_t, 2); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint16_t, 12)) << 2; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint16_t, 4); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint16_t, 10)) << 4; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint16_t, 6); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint16_t, 8)) << 6; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 8); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint16_t, 6)) << 8; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint16_t, 10); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint16_t, 4)) << 10; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint16_t, 12); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint16_t, 2)) << 12; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint16_t, 14); + out[INDEX(15, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_16_lane<15>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint16_t src; + uint16_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint16_t, 15); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint16_t, 1); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint16_t, 14)) << 1; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint16_t, 2); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint16_t, 13)) << 2; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint16_t, 3); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint16_t, 12)) << 3; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint16_t, 4); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint16_t, 11)) << 4; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint16_t, 5); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint16_t, 10)) << 5; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint16_t, 6); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint16_t, 9)) << 6; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint16_t, 7); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint16_t, 8)) << 7; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint16_t, 8); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint16_t, 7)) << 8; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint16_t, 9); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint16_t, 6)) << 9; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint16_t, 10); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint16_t, 5)) << 10; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint16_t, 11); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint16_t, 4)) << 11; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint16_t, 12); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint16_t, 3)) << 12; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint16_t, 13); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint16_t, 2)) << 13; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint16_t, 14); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint16_t, 1)) << 14; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint16_t, 15); + out[INDEX(15, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_16_lane<16>(const uint16_t *__restrict in, uint16_t *__restrict out, uint16_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + #pragma unroll + for (int row = 0; row < 16; row++) { + out[INDEX(row, lane)] = in[LANE_COUNT * row + lane] + reference; + } +} + +/// Runtime dispatch to the optimized lane decoder for the given bit width. +__device__ __noinline__ void bit_unpack_16_lane( + const uint16_t *__restrict in, + uint16_t *__restrict out, + uint16_t reference, + unsigned int lane, + uint32_t bit_width +) { + switch (bit_width) { + case 0: _bit_unpack_16_lane<0>(in, out, reference, lane); break; + case 1: _bit_unpack_16_lane<1>(in, out, reference, lane); break; + case 2: _bit_unpack_16_lane<2>(in, out, reference, lane); break; + case 3: _bit_unpack_16_lane<3>(in, out, reference, lane); break; + case 4: _bit_unpack_16_lane<4>(in, out, reference, lane); break; + case 5: _bit_unpack_16_lane<5>(in, out, reference, lane); break; + case 6: _bit_unpack_16_lane<6>(in, out, reference, lane); break; + case 7: _bit_unpack_16_lane<7>(in, out, reference, lane); break; + case 8: _bit_unpack_16_lane<8>(in, out, reference, lane); break; + case 9: _bit_unpack_16_lane<9>(in, out, reference, lane); break; + case 10: _bit_unpack_16_lane<10>(in, out, reference, lane); break; + case 11: _bit_unpack_16_lane<11>(in, out, reference, lane); break; + case 12: _bit_unpack_16_lane<12>(in, out, reference, lane); break; + case 13: _bit_unpack_16_lane<13>(in, out, reference, lane); break; + case 14: _bit_unpack_16_lane<14>(in, out, reference, lane); break; + case 15: _bit_unpack_16_lane<15>(in, out, reference, lane); break; + case 16: _bit_unpack_16_lane<16>(in, out, reference, lane); break; + } +} + diff --git a/vortex-cuda/kernels/src/bit_unpack_32.cu b/vortex-cuda/kernels/src/bit_unpack_32.cu index 7eb2d49cc49..3f8fcb5c227 100644 --- a/vortex-cuda/kernels/src/bit_unpack_32.cu +++ b/vortex-cuda/kernels/src/bit_unpack_32.cu @@ -1,3486 +1,263 @@ // AUTO-GENERATED. Do not edit by hand! -#include -#include -#include -#include "fastlanes_common.cuh" +#include "bit_unpack_32_lanes.cuh" #include "patches.cuh" template -__device__ void _bit_unpack_32_lane(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane); - -template <> -__device__ void _bit_unpack_32_lane<0>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - #pragma unroll - for (int row = 0; row < 32; row++) { - out[INDEX(row, lane)] = reference; - } -} - -template <> -__device__ void _bit_unpack_32_lane<1>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 1); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint32_t, 1); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 1); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint32_t, 1); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 1); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint32_t, 1); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 1); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint32_t, 1); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 1); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint32_t, 1); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 1); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint32_t, 1); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 1); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint32_t, 1); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 1); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint32_t, 1); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 1); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint32_t, 1); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 1); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint32_t, 1); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 1); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint32_t, 1); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 1); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint32_t, 1); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 1); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint32_t, 1); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 1); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint32_t, 1); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 1); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint32_t, 1); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 1); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint32_t, 1); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<2>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 2); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 2); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 2); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 2); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 2); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 2); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 2); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 2); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 2); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 2); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 2); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 2); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 2); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 2); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 2); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 0)) << 2; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 2); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 2); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 2); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 2); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 2); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 2); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 2); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 2); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 2); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 2); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 2); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 2); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 2); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 2); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 2); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<3>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 3); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint32_t, 3); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 3); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint32_t, 3); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 3); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint32_t, 3); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 3); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint32_t, 3); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 3); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint32_t, 3); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 1)) << 2; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint32_t, 3); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 3); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint32_t, 3); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 3); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint32_t, 3); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 3); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint32_t, 3); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 3); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint32_t, 3); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 3); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint32_t, 1); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 2)) << 1; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 3); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint32_t, 3); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 3); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint32_t, 3); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 3); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint32_t, 3); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 3); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint32_t, 3); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 3); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint32_t, 3); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<4>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 4); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 4); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 4); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 4); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 4); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 4); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 4); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 0)) << 4; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 4); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 4); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 4); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 4); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 4); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 4); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 4); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 0)) << 4; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 4); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 4); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 4); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 4); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 4); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 4); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 4); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 0)) << 4; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 4); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 4); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 4); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 4); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 4); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 4); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 4); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<5>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 5); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint32_t, 5); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 5); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint32_t, 5); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 5); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint32_t, 5); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 3)) << 2; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint32_t, 5); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 5); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint32_t, 5); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 5); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint32_t, 5); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 1)) << 4; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint32_t, 5); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 5); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint32_t, 5); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 5); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint32_t, 5); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 5); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint32_t, 1); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 4)) << 1; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 5); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint32_t, 5); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 5); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint32_t, 5); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 5); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint32_t, 3); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 2)) << 3; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 5); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint32_t, 5); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 5); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint32_t, 5); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 5); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint32_t, 5); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<6>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 6); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 6); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 6); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 6); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 6); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 4)) << 2; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 6); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 6); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 6); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 6); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 2)) << 4; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 6); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 6); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 6); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 6); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 0)) << 6; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 6); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 6); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 6); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 6); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 6); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 4)) << 2; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 6); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 6); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 6); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 6); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 2)) << 4; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 6); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 6); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 6); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 6); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<7>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 7); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint32_t, 7); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 7); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint32_t, 7); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 3)) << 4; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint32_t, 7); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 7); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint32_t, 7); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 7); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint32_t, 1); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 6)) << 1; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 7); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint32_t, 7); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 7); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint32_t, 5); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 2)) << 5; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 7); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint32_t, 7); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 7); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint32_t, 7); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 5)) << 2; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint32_t, 7); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 7); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint32_t, 7); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 1)) << 6; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint32_t, 7); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 7); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint32_t, 7); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 7); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint32_t, 3); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint32_t, 4)) << 3; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 7); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint32_t, 7); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 7); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint32_t, 7); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<8>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 8); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 8); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 8); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 0)) << 8; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 8); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 8); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 8); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 0)) << 8; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 8); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 8); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 8); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 0)) << 8; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 8); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 8); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 8); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 0)) << 8; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 8); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 8); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 8); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 0)) << 8; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 8); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 8); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 8); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint32_t, 0)) << 8; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 8); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 8); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 8); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint32_t, 0)) << 8; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 8); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 8); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 8); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<9>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 9); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint32_t, 9); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 9); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint32_t, 5); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 4)) << 5; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 9); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint32_t, 9); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 9); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint32_t, 1); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 8)) << 1; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 9); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint32_t, 9); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 3)) << 6; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint32_t, 9); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 9); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint32_t, 9); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 7)) << 2; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint32_t, 9); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 9); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint32_t, 7); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 2)) << 7; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 9); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint32_t, 9); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 9); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint32_t, 3); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint32_t, 6)) << 3; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 9); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint32_t, 9); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint32_t, 1)) << 8; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint32_t, 9); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 9); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint32_t, 9); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint32_t, 5)) << 4; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint32_t, 9); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 9); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint32_t, 9); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<10>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 10); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 10); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 10); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 8)) << 2; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 10); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 10); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 6)) << 4; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 10); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 10); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 4)) << 6; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 10); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 10); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 2)) << 8; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 10); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 10); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 10); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 0)) << 10; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 10); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 10); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 10); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint32_t, 8)) << 2; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 10); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 10); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint32_t, 6)) << 4; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 10); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 10); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint32_t, 4)) << 6; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 10); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 10); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint32_t, 2)) << 8; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 10); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 10); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 10); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<11>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 11); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint32_t, 11); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 10); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 1)) << 10; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint32_t, 11); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 11); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint32_t, 9); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 2)) << 9; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 11); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint32_t, 11); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 3)) << 8; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint32_t, 11); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 11); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint32_t, 7); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 4)) << 7; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 11); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint32_t, 11); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 5)) << 6; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint32_t, 11); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 11); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint32_t, 5); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint32_t, 6)) << 5; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 11); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint32_t, 11); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint32_t, 7)) << 4; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint32_t, 11); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 11); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint32_t, 3); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint32_t, 8)) << 3; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 11); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint32_t, 11); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint32_t, 9)) << 2; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint32_t, 11); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 11); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint32_t, 1); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint32_t, 10)) << 1; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 11); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint32_t, 11); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<12>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 12); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 12); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 4)) << 8; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 12); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 12); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 8)) << 4; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 12); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 0)) << 12; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 12); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 12); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 4)) << 8; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 12); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 12); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 8)) << 4; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 12); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint32_t, 0)) << 12; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 12); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 12); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint32_t, 4)) << 8; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 12); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 12); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint32_t, 8)) << 4; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 12); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint32_t, 0)) << 12; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 12); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 12); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint32_t, 4)) << 8; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 12); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 12); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint32_t, 8)) << 4; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 12); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<13>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 13); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint32_t, 13); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 7)) << 6; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint32_t, 13); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 1)) << 12; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint32_t, 13); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 13); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint32_t, 5); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 8)) << 5; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 13); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint32_t, 11); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 2)) << 11; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 13); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint32_t, 13); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 9)) << 4; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint32_t, 13); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 10); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint32_t, 3)) << 10; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint32_t, 13); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 13); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint32_t, 3); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint32_t, 10)) << 3; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 13); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint32_t, 9); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint32_t, 4)) << 9; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 13); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint32_t, 13); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint32_t, 11)) << 2; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint32_t, 13); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint32_t, 5)) << 8; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint32_t, 13); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 13); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint32_t, 1); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint32_t, 12)) << 1; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 13); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint32_t, 7); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint32_t, 6)) << 7; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 13); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint32_t, 13); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<14>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 14); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 14); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 10)) << 4; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 14); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 6)) << 8; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 14); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 2)) << 12; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 14); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 14); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 12)) << 2; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 14); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 8)) << 6; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 14); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 10); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint32_t, 4)) << 10; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 14); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 14); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint32_t, 0)) << 14; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 14); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 14); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint32_t, 10)) << 4; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 14); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint32_t, 6)) << 8; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 14); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint32_t, 2)) << 12; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 14); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 14); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint32_t, 12)) << 2; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 14); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint32_t, 8)) << 6; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 14); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 10); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint32_t, 4)) << 10; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 14); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 14); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<15>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 15); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint32_t, 15); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 13)) << 2; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint32_t, 15); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 11)) << 4; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint32_t, 15); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 9)) << 6; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint32_t, 15); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 7)) << 8; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint32_t, 15); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 10); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 5)) << 10; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint32_t, 15); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint32_t, 3)) << 12; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint32_t, 15); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 14); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint32_t, 1)) << 14; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint32_t, 15); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 15); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint32_t, 1); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint32_t, 14)) << 1; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 15); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint32_t, 3); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint32_t, 12)) << 3; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 15); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint32_t, 5); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint32_t, 10)) << 5; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 15); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint32_t, 7); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint32_t, 8)) << 7; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 15); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint32_t, 9); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint32_t, 6)) << 9; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 15); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint32_t, 11); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint32_t, 4)) << 11; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 15); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint32_t, 13); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint32_t, 2)) << 13; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 15); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint32_t, 15); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<16>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 16); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 0)) << 16; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 16); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 0)) << 16; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 16); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 0)) << 16; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 16); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 0)) << 16; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 16); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 0)) << 16; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 16); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint32_t, 0)) << 16; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 16); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint32_t, 0)) << 16; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 16); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint32_t, 0)) << 16; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 16); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint32_t, 0)) << 16; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 16); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint32_t, 0)) << 16; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 16); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint32_t, 0)) << 16; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 16); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint32_t, 0)) << 16; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 16); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint32_t, 0)) << 16; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 16); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint32_t, 0)) << 16; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 16); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint32_t, 0)) << 16; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 16); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<17>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 17); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint32_t, 15); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 2)) << 15; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 17); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint32_t, 13); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 4)) << 13; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 17); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint32_t, 11); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 6)) << 11; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 17); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint32_t, 9); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 8)) << 9; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 17); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint32_t, 7); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 10)) << 7; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 17); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint32_t, 5); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint32_t, 12)) << 5; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 17); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint32_t, 3); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint32_t, 14)) << 3; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 17); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint32_t, 1); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint32_t, 16)) << 1; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint32_t, 1)) << 16; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint32_t, 17); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 14); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint32_t, 3)) << 14; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint32_t, 17); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint32_t, 5)) << 12; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint32_t, 17); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 10); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint32_t, 7)) << 10; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint32_t, 17); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint32_t, 9)) << 8; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint32_t, 17); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint32_t, 11)) << 6; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint32_t, 17); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint32_t, 13)) << 4; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint32_t, 17); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint32_t, 15)) << 2; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint32_t, 17); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<18>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 18); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 14); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 4)) << 14; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 18); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 10); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 8)) << 10; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 18); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 12)) << 6; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 18); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 16)) << 2; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 2)) << 16; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 18); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint32_t, 6)) << 12; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 18); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint32_t, 10)) << 8; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 18); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint32_t, 14)) << 4; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 18); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint32_t, 0)) << 18; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 18); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 14); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint32_t, 4)) << 14; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 18); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 10); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint32_t, 8)) << 10; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 18); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint32_t, 12)) << 6; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 18); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint32_t, 16)) << 2; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint32_t, 2)) << 16; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 18); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint32_t, 6)) << 12; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 18); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint32_t, 10)) << 8; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 18); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint32_t, 14)) << 4; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 18); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<19>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 19); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint32_t, 13); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 6)) << 13; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 19); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint32_t, 7); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 12)) << 7; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 19); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint32_t, 1); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 18)) << 1; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 14); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 5)) << 14; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint32_t, 19); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 11)) << 8; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint32_t, 19); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint32_t, 17)) << 2; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint32_t, 15); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint32_t, 4)) << 15; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 19); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint32_t, 9); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint32_t, 10)) << 9; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 19); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint32_t, 3); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint32_t, 16)) << 3; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint32_t, 3)) << 16; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint32_t, 19); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 10); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint32_t, 9)) << 10; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint32_t, 19); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint32_t, 15)) << 4; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint32_t, 17); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint32_t, 2)) << 17; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 19); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint32_t, 11); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint32_t, 8)) << 11; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 19); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint32_t, 5); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint32_t, 14)) << 5; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 18); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint32_t, 1)) << 18; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint32_t, 19); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint32_t, 7)) << 12; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint32_t, 19); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint32_t, 13)) << 6; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint32_t, 19); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<20>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 20); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 8)) << 12; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 20); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 16)) << 4; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 4)) << 16; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 20); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 12)) << 8; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 20); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 0)) << 20; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 20); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint32_t, 8)) << 12; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 20); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint32_t, 16)) << 4; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint32_t, 4)) << 16; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 20); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint32_t, 12)) << 8; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 20); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint32_t, 0)) << 20; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 20); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint32_t, 8)) << 12; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 20); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint32_t, 16)) << 4; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint32_t, 4)) << 16; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 20); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint32_t, 12)) << 8; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 20); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint32_t, 0)) << 20; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 20); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint32_t, 8)) << 12; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 20); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint32_t, 16)) << 4; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint32_t, 4)) << 16; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 20); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint32_t, 12)) << 8; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 20); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<21>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 21); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint32_t, 11); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 10)) << 11; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 21); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint32_t, 1); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 20)) << 1; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 9)) << 12; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint32_t, 21); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 19)) << 2; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint32_t, 13); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 8)) << 13; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 21); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint32_t, 3); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint32_t, 18)) << 3; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 14); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint32_t, 7)) << 14; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint32_t, 21); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint32_t, 17)) << 4; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint32_t, 15); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint32_t, 6)) << 15; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 21); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint32_t, 5); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint32_t, 16)) << 5; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint32_t, 5)) << 16; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint32_t, 21); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint32_t, 15)) << 6; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint32_t, 17); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint32_t, 4)) << 17; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 21); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint32_t, 7); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint32_t, 14)) << 7; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 18); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint32_t, 3)) << 18; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint32_t, 21); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint32_t, 13)) << 8; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint32_t, 19); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint32_t, 2)) << 19; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 21); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint32_t, 9); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint32_t, 12)) << 9; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 20); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint32_t, 1)) << 20; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint32_t, 21); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 10); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint32_t, 11)) << 10; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint32_t, 21); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<22>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 22); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 10); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 12)) << 10; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 20); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 2)) << 20; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 22); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 14)) << 8; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 18); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 4)) << 18; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 22); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 16)) << 6; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint32_t, 6)) << 16; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 22); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint32_t, 18)) << 4; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 14); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint32_t, 8)) << 14; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 22); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint32_t, 20)) << 2; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint32_t, 10)) << 12; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 22); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint32_t, 0)) << 22; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 22); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 10); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint32_t, 12)) << 10; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 20); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint32_t, 2)) << 20; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 22); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint32_t, 14)) << 8; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 18); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint32_t, 4)) << 18; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 22); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint32_t, 16)) << 6; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint32_t, 6)) << 16; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 22); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint32_t, 18)) << 4; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 14); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint32_t, 8)) << 14; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 22); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint32_t, 20)) << 2; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint32_t, 10)) << 12; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 22); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<23>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 23); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint32_t, 9); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 14)) << 9; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 18); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 5)) << 18; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint32_t, 23); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 19)) << 4; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint32_t, 13); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 10)) << 13; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 22); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 1)) << 22; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint32_t, 23); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint32_t, 15)) << 8; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint32_t, 17); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint32_t, 6)) << 17; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 23); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint32_t, 3); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint32_t, 20)) << 3; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint32_t, 11)) << 12; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint32_t, 21); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint32_t, 2)) << 21; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 23); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint32_t, 7); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint32_t, 16)) << 7; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint32_t, 7)) << 16; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint32_t, 23); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint32_t, 21)) << 2; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint32_t, 11); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint32_t, 12)) << 11; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 20); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint32_t, 3)) << 20; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint32_t, 23); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint32_t, 17)) << 6; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint32_t, 15); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint32_t, 8)) << 15; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 23); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint32_t, 1); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint32_t, 22)) << 1; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 10); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint32_t, 13)) << 10; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint32_t, 19); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint32_t, 4)) << 19; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 23); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint32_t, 5); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint32_t, 18)) << 5; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 14); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint32_t, 9)) << 14; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint32_t, 23); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<24>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 24); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 16)) << 8; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 8)) << 16; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 24); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 0)) << 24; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 24); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 16)) << 8; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 8)) << 16; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 24); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint32_t, 0)) << 24; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 24); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint32_t, 16)) << 8; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint32_t, 8)) << 16; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 24); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint32_t, 0)) << 24; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 24); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint32_t, 16)) << 8; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint32_t, 8)) << 16; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 24); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint32_t, 0)) << 24; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 24); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint32_t, 16)) << 8; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint32_t, 8)) << 16; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 24); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint32_t, 0)) << 24; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 24); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint32_t, 16)) << 8; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint32_t, 8)) << 16; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 24); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint32_t, 0)) << 24; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 24); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint32_t, 16)) << 8; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint32_t, 8)) << 16; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 24); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint32_t, 0)) << 24; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 24); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint32_t, 16)) << 8; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint32_t, 8)) << 16; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 24); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<25>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 25); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint32_t, 7); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 18)) << 7; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 14); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 11)) << 14; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint32_t, 21); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 4)) << 21; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 25); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint32_t, 3); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 22)) << 3; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 10); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 15)) << 10; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint32_t, 17); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint32_t, 8)) << 17; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 24); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint32_t, 1)) << 24; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint32_t, 25); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint32_t, 19)) << 6; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint32_t, 13); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint32_t, 12)) << 13; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 20); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint32_t, 5)) << 20; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint32_t, 25); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint32_t, 23)) << 2; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint32_t, 9); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint32_t, 16)) << 9; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint32_t, 9)) << 16; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint32_t, 23); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint32_t, 2)) << 23; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 25); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint32_t, 5); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint32_t, 20)) << 5; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint32_t, 13)) << 12; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint32_t, 19); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint32_t, 6)) << 19; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 25); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint32_t, 1); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint32_t, 24)) << 1; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint32_t, 17)) << 8; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint32_t, 15); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint32_t, 10)) << 15; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 22); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint32_t, 3)) << 22; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint32_t, 25); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint32_t, 21)) << 4; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint32_t, 11); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint32_t, 14)) << 11; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 18); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint32_t, 7)) << 18; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint32_t, 25); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<26>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 26); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 20)) << 6; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 14)) << 12; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 18); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 8)) << 18; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 24); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 2)) << 24; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 26); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 22)) << 4; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 10); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint32_t, 16)) << 10; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint32_t, 10)) << 16; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 22); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint32_t, 4)) << 22; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 26); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint32_t, 24)) << 2; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint32_t, 18)) << 8; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 14); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint32_t, 12)) << 14; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 20); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint32_t, 6)) << 20; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 26); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint32_t, 0)) << 26; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 26); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint32_t, 20)) << 6; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint32_t, 14)) << 12; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 18); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint32_t, 8)) << 18; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 24); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint32_t, 2)) << 24; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 26); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint32_t, 22)) << 4; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 10); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint32_t, 16)) << 10; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint32_t, 10)) << 16; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 22); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint32_t, 4)) << 22; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 26); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint32_t, 24)) << 2; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint32_t, 18)) << 8; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 14); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint32_t, 12)) << 14; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 20); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint32_t, 6)) << 20; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 26); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<27>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 27); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint32_t, 5); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 22)) << 5; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 10); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 17)) << 10; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint32_t, 15); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 12)) << 15; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 20); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 7)) << 20; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint32_t, 25); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 2)) << 25; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 27); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint32_t, 3); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint32_t, 24)) << 3; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint32_t, 19)) << 8; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint32_t, 13); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint32_t, 14)) << 13; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 18); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint32_t, 9)) << 18; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint32_t, 23); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint32_t, 4)) << 23; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 27); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint32_t, 1); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint32_t, 26)) << 1; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint32_t, 21)) << 6; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint32_t, 11); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint32_t, 16)) << 11; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint32_t, 11)) << 16; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint32_t, 21); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint32_t, 6)) << 21; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 26); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint32_t, 1)) << 26; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint32_t, 27); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint32_t, 23)) << 4; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint32_t, 9); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint32_t, 18)) << 9; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 14); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint32_t, 13)) << 14; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint32_t, 19); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint32_t, 8)) << 19; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 24); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint32_t, 3)) << 24; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint32_t, 27); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint32_t, 25)) << 2; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint32_t, 7); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint32_t, 20)) << 7; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint32_t, 15)) << 12; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint32_t, 17); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint32_t, 10)) << 17; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 22); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint32_t, 5)) << 22; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint32_t, 27); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<28>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 28); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 24)) << 4; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 20)) << 8; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 16)) << 12; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 12)) << 16; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 20); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 8)) << 20; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 24); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint32_t, 4)) << 24; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 28); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint32_t, 0)) << 28; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 28); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint32_t, 24)) << 4; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint32_t, 20)) << 8; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint32_t, 16)) << 12; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint32_t, 12)) << 16; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 20); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint32_t, 8)) << 20; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 24); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint32_t, 4)) << 24; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 28); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint32_t, 0)) << 28; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 28); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint32_t, 24)) << 4; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint32_t, 20)) << 8; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint32_t, 16)) << 12; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint32_t, 12)) << 16; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 20); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint32_t, 8)) << 20; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 24); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint32_t, 4)) << 24; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 28); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint32_t, 0)) << 28; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 28); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint32_t, 24)) << 4; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint32_t, 20)) << 8; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint32_t, 16)) << 12; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint32_t, 12)) << 16; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 20); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint32_t, 8)) << 20; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 24); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint32_t, 4)) << 24; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 28); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<29>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 29); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint32_t, 3); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 26)) << 3; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 23)) << 6; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint32_t, 9); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 20)) << 9; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 17)) << 12; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint32_t, 15); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 14)) << 15; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 18); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint32_t, 11)) << 18; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint32_t, 21); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint32_t, 8)) << 21; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 24); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint32_t, 5)) << 24; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint32_t, 27); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint32_t, 2)) << 27; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 29); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint32_t, 1); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint32_t, 28)) << 1; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint32_t, 25)) << 4; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint32_t, 7); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint32_t, 22)) << 7; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 10); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint32_t, 19)) << 10; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint32_t, 13); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint32_t, 16)) << 13; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint32_t, 13)) << 16; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint32_t, 19); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint32_t, 10)) << 19; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 22); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint32_t, 7)) << 22; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint32_t, 25); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint32_t, 4)) << 25; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 28); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint32_t, 1)) << 28; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint32_t, 29); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint32_t, 27)) << 2; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint32_t, 5); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint32_t, 24)) << 5; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint32_t, 21)) << 8; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint32_t, 11); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint32_t, 18)) << 11; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 14); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint32_t, 15)) << 14; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint32_t, 17); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint32_t, 12)) << 17; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 20); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint32_t, 9)) << 20; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint32_t, 23); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint32_t, 6)) << 23; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 26); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint32_t, 3)) << 26; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint32_t, 29); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<30>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 30); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 28)) << 2; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 26)) << 4; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 24)) << 6; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 22)) << 8; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 10); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 20)) << 10; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint32_t, 18)) << 12; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 14); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint32_t, 16)) << 14; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint32_t, 14)) << 16; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 18); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint32_t, 12)) << 18; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 20); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint32_t, 10)) << 20; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 22); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint32_t, 8)) << 22; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 24); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint32_t, 6)) << 24; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 26); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint32_t, 4)) << 26; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 28); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint32_t, 2)) << 28; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 30); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint32_t, 0)) << 30; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint32_t, 30); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint32_t, 28)) << 2; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint32_t, 26)) << 4; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint32_t, 24)) << 6; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint32_t, 22)) << 8; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 10); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint32_t, 20)) << 10; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint32_t, 18)) << 12; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 14); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint32_t, 16)) << 14; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint32_t, 14)) << 16; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 18); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint32_t, 12)) << 18; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 20); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint32_t, 10)) << 20; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 22); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint32_t, 8)) << 22; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 24); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint32_t, 6)) << 24; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 26); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint32_t, 4)) << 26; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 28); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint32_t, 2)) << 28; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 30); - out[INDEX(31, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_32_lane<31>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; - uint32_t src; - uint32_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint32_t, 31); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint32_t, 1); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint32_t, 30)) << 1; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint32_t, 2); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint32_t, 29)) << 2; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint32_t, 3); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint32_t, 28)) << 3; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint32_t, 4); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint32_t, 27)) << 4; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint32_t, 5); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint32_t, 26)) << 5; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint32_t, 6); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint32_t, 25)) << 6; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint32_t, 7); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint32_t, 24)) << 7; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint32_t, 8); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint32_t, 23)) << 8; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint32_t, 9); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint32_t, 22)) << 9; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint32_t, 10); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint32_t, 21)) << 10; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint32_t, 11); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint32_t, 20)) << 11; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint32_t, 12); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint32_t, 19)) << 12; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint32_t, 13); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint32_t, 18)) << 13; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint32_t, 14); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint32_t, 17)) << 14; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint32_t, 15); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint32_t, 16)) << 15; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint32_t, 16); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint32_t, 15)) << 16; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint32_t, 17); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint32_t, 14)) << 17; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint32_t, 18); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint32_t, 13)) << 18; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint32_t, 19); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint32_t, 12)) << 19; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint32_t, 20); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint32_t, 11)) << 20; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint32_t, 21); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint32_t, 10)) << 21; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint32_t, 22); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint32_t, 9)) << 22; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint32_t, 23); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint32_t, 8)) << 23; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint32_t, 24); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint32_t, 7)) << 24; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint32_t, 25); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint32_t, 6)) << 25; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint32_t, 26); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint32_t, 5)) << 26; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint32_t, 27); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint32_t, 4)) << 27; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint32_t, 28); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint32_t, 3)) << 28; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint32_t, 29); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint32_t, 2)) << 29; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint32_t, 30); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint32_t, 1)) << 30; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint32_t, 31); - out[INDEX(31, lane)] = tmp + reference; -} +__device__ void _bit_unpack_32_device(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, int thread_idx, GPUPatches& patches) { + __shared__ uint32_t shared_out[FL_CHUNK]; -template <> -__device__ void _bit_unpack_32_lane<32>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 32; + // Step 1: Unpack into shared memory #pragma unroll - for (int row = 0; row < 32; row++) { - out[INDEX(row, lane)] = in[LANE_COUNT * row + lane] + reference; + for (int i = 0; i < FL_LANES / 32; i++) { + _bit_unpack_32_lane(in, shared_out, reference, thread_idx * (FL_LANES / 32) + i); } -} + __syncwarp(); -/// Runtime dispatch to the optimized lane decoder for the given bit width. -__device__ inline void bit_unpack_32_lane( - const uint32_t *__restrict in, - uint32_t *__restrict out, - uint32_t reference, - unsigned int lane, - uint32_t bit_width -) { - switch (bit_width) { - case 0: _bit_unpack_32_lane<0>(in, out, reference, lane); break; - case 1: _bit_unpack_32_lane<1>(in, out, reference, lane); break; - case 2: _bit_unpack_32_lane<2>(in, out, reference, lane); break; - case 3: _bit_unpack_32_lane<3>(in, out, reference, lane); break; - case 4: _bit_unpack_32_lane<4>(in, out, reference, lane); break; - case 5: _bit_unpack_32_lane<5>(in, out, reference, lane); break; - case 6: _bit_unpack_32_lane<6>(in, out, reference, lane); break; - case 7: _bit_unpack_32_lane<7>(in, out, reference, lane); break; - case 8: _bit_unpack_32_lane<8>(in, out, reference, lane); break; - case 9: _bit_unpack_32_lane<9>(in, out, reference, lane); break; - case 10: _bit_unpack_32_lane<10>(in, out, reference, lane); break; - case 11: _bit_unpack_32_lane<11>(in, out, reference, lane); break; - case 12: _bit_unpack_32_lane<12>(in, out, reference, lane); break; - case 13: _bit_unpack_32_lane<13>(in, out, reference, lane); break; - case 14: _bit_unpack_32_lane<14>(in, out, reference, lane); break; - case 15: _bit_unpack_32_lane<15>(in, out, reference, lane); break; - case 16: _bit_unpack_32_lane<16>(in, out, reference, lane); break; - case 17: _bit_unpack_32_lane<17>(in, out, reference, lane); break; - case 18: _bit_unpack_32_lane<18>(in, out, reference, lane); break; - case 19: _bit_unpack_32_lane<19>(in, out, reference, lane); break; - case 20: _bit_unpack_32_lane<20>(in, out, reference, lane); break; - case 21: _bit_unpack_32_lane<21>(in, out, reference, lane); break; - case 22: _bit_unpack_32_lane<22>(in, out, reference, lane); break; - case 23: _bit_unpack_32_lane<23>(in, out, reference, lane); break; - case 24: _bit_unpack_32_lane<24>(in, out, reference, lane); break; - case 25: _bit_unpack_32_lane<25>(in, out, reference, lane); break; - case 26: _bit_unpack_32_lane<26>(in, out, reference, lane); break; - case 27: _bit_unpack_32_lane<27>(in, out, reference, lane); break; - case 28: _bit_unpack_32_lane<28>(in, out, reference, lane); break; - case 29: _bit_unpack_32_lane<29>(in, out, reference, lane); break; - case 30: _bit_unpack_32_lane<30>(in, out, reference, lane); break; - case 31: _bit_unpack_32_lane<31>(in, out, reference, lane); break; - case 32: _bit_unpack_32_lane<32>(in, out, reference, lane); break; + // Step 2: Apply patches to shared memory in parallel + PatchesCursor cursor(patches, blockIdx.x, thread_idx, 32); + auto patch = cursor.next(); + while (patch.index != FL_CHUNK) { + shared_out[patch.index] = patch.value; + patch = cursor.next(); } -} + __syncwarp(); -template -__device__ void _bit_unpack_32_device(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, int thread_idx, GPUPatches& patches) { - __shared__ uint32_t shared_out[1024]; + // Step 3: Copy to global memory #pragma unroll - for (int i = 0; i < 1; i++) { - _bit_unpack_32_lane(in, shared_out, reference, thread_idx * 1 + i); - } - __syncwarp(); - PatchesCursor cursor(patches, blockIdx.x, thread_idx, 32); - auto patch = cursor.next(); - for (int i = 0; i < 32; i++) { + for (int i = 0; i < FL_CHUNK / 32; i++) { auto idx = i * 32 + thread_idx; - if (idx == patch.index) { - out[idx] = patch.value; - patch = cursor.next(); - } else { - out[idx] = shared_out[idx]; - } + out[idx] = shared_out[idx]; } } extern "C" __global__ void bit_unpack_32_0bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 0 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 0)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<0>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_1bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 1 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 1)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<1>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_2bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 2 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 2)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<2>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_3bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 3 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 3)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<3>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_4bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 4 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 4)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<4>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_5bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 5 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 5)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<5>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_6bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 6 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 6)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<6>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_7bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 7 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 7)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<7>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_8bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 8 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 8)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<8>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_9bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 9 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 9)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<9>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_10bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 10 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 10)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<10>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_11bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 11 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 11)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<11>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_12bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 12 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 12)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<12>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_13bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 13 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 13)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<13>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_14bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 14 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 14)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<14>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_15bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 15 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 15)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<15>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_16bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 16 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 16)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<16>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_17bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 17 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 17)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<17>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_18bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 18 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 18)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<18>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_19bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 19 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 19)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<19>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_20bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 20 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 20)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<20>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_21bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 21 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 21)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<21>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_22bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 22 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 22)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<22>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_23bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 23 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 23)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<23>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_24bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 24 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 24)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<24>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_25bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 25 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 25)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<25>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_26bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 26 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 26)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<26>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_27bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 27 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 27)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<27>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_28bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 28 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 28)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<28>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_29bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 29 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 29)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<29>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_30bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 30 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 30)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<30>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_31bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 31 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 31)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<31>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_32_32bw_32t(const uint32_t *__restrict full_in, uint32_t *__restrict full_out, uint32_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 32 / sizeof(uint32_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 32)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_32_device<32>(in, out, reference, thread_idx, patches); } diff --git a/vortex-cuda/kernels/src/bit_unpack_32_lanes.cuh b/vortex-cuda/kernels/src/bit_unpack_32_lanes.cuh new file mode 100644 index 00000000000..58d4195e3aa --- /dev/null +++ b/vortex-cuda/kernels/src/bit_unpack_32_lanes.cuh @@ -0,0 +1,3235 @@ +// AUTO-GENERATED. Do not edit by hand! +#pragma once + +#include +#include +#include +#include "fastlanes_common.cuh" + +template +__device__ void _bit_unpack_32_lane(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane); + +template <> +__device__ void _bit_unpack_32_lane<0>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + #pragma unroll + for (int row = 0; row < 32; row++) { + out[INDEX(row, lane)] = reference; + } +} + +template <> +__device__ void _bit_unpack_32_lane<1>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 1); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint32_t, 1); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 1); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint32_t, 1); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 1); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint32_t, 1); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 1); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint32_t, 1); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 1); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint32_t, 1); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 1); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint32_t, 1); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 1); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint32_t, 1); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 1); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint32_t, 1); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 1); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint32_t, 1); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 1); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint32_t, 1); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 1); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint32_t, 1); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 1); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint32_t, 1); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 1); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint32_t, 1); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 1); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint32_t, 1); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 1); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint32_t, 1); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 1); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint32_t, 1); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<2>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 2); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 2); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 2); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 2); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 2); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 2); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 2); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 2); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 2); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 2); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 2); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 2); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 2); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 2); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 2); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 0)) << 2; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 2); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 2); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 2); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 2); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 2); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 2); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 2); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 2); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 2); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 2); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 2); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 2); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 2); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 2); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 2); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<3>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 3); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint32_t, 3); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 3); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint32_t, 3); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 3); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint32_t, 3); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 3); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint32_t, 3); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 3); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint32_t, 3); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 1)) << 2; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint32_t, 3); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 3); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint32_t, 3); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 3); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint32_t, 3); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 3); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint32_t, 3); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 3); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint32_t, 3); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 3); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint32_t, 1); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 2)) << 1; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 3); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint32_t, 3); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 3); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint32_t, 3); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 3); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint32_t, 3); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 3); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint32_t, 3); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 3); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint32_t, 3); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<4>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 4); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 4); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 4); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 4); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 4); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 4); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 4); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 0)) << 4; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 4); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 4); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 4); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 4); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 4); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 4); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 4); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 0)) << 4; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 4); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 4); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 4); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 4); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 4); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 4); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 4); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 0)) << 4; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 4); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 4); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 4); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 4); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 4); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 4); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 4); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<5>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 5); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint32_t, 5); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 5); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint32_t, 5); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 5); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint32_t, 5); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 3)) << 2; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint32_t, 5); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 5); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint32_t, 5); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 5); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint32_t, 5); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 1)) << 4; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint32_t, 5); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 5); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint32_t, 5); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 5); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint32_t, 5); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 5); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint32_t, 1); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 4)) << 1; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 5); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint32_t, 5); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 5); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint32_t, 5); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 5); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint32_t, 3); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 2)) << 3; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 5); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint32_t, 5); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 5); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint32_t, 5); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 5); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint32_t, 5); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<6>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 6); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 6); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 6); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 6); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 6); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 4)) << 2; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 6); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 6); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 6); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 6); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 2)) << 4; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 6); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 6); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 6); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 6); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 0)) << 6; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 6); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 6); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 6); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 6); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 6); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 4)) << 2; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 6); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 6); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 6); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 6); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 2)) << 4; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 6); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 6); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 6); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 6); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<7>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 7); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint32_t, 7); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 7); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint32_t, 7); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 3)) << 4; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint32_t, 7); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 7); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint32_t, 7); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 7); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint32_t, 1); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 6)) << 1; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 7); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint32_t, 7); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 7); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint32_t, 5); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 2)) << 5; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 7); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint32_t, 7); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 7); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint32_t, 7); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 5)) << 2; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint32_t, 7); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 7); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint32_t, 7); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 1)) << 6; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint32_t, 7); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 7); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint32_t, 7); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 7); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint32_t, 3); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint32_t, 4)) << 3; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 7); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint32_t, 7); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 7); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint32_t, 7); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<8>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 8); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 8); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 8); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 0)) << 8; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 8); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 8); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 8); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 0)) << 8; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 8); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 8); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 8); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 0)) << 8; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 8); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 8); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 8); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 0)) << 8; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 8); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 8); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 8); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 0)) << 8; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 8); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 8); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 8); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint32_t, 0)) << 8; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 8); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 8); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 8); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint32_t, 0)) << 8; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 8); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 8); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 8); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<9>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 9); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint32_t, 9); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 9); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint32_t, 5); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 4)) << 5; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 9); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint32_t, 9); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 9); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint32_t, 1); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 8)) << 1; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 9); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint32_t, 9); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 3)) << 6; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint32_t, 9); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 9); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint32_t, 9); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 7)) << 2; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint32_t, 9); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 9); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint32_t, 7); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 2)) << 7; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 9); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint32_t, 9); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 9); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint32_t, 3); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint32_t, 6)) << 3; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 9); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint32_t, 9); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint32_t, 1)) << 8; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint32_t, 9); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 9); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint32_t, 9); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint32_t, 5)) << 4; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint32_t, 9); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 9); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint32_t, 9); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<10>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 10); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 10); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 10); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 8)) << 2; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 10); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 10); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 6)) << 4; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 10); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 10); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 4)) << 6; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 10); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 10); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 2)) << 8; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 10); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 10); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 10); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 0)) << 10; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 10); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 10); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 10); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint32_t, 8)) << 2; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 10); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 10); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint32_t, 6)) << 4; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 10); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 10); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint32_t, 4)) << 6; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 10); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 10); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint32_t, 2)) << 8; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 10); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 10); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 10); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<11>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 11); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint32_t, 11); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 10); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 1)) << 10; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint32_t, 11); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 11); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint32_t, 9); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 2)) << 9; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 11); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint32_t, 11); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 3)) << 8; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint32_t, 11); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 11); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint32_t, 7); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 4)) << 7; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 11); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint32_t, 11); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 5)) << 6; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint32_t, 11); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 11); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint32_t, 5); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint32_t, 6)) << 5; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 11); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint32_t, 11); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint32_t, 7)) << 4; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint32_t, 11); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 11); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint32_t, 3); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint32_t, 8)) << 3; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 11); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint32_t, 11); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint32_t, 9)) << 2; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint32_t, 11); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 11); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint32_t, 1); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint32_t, 10)) << 1; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 11); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint32_t, 11); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<12>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 12); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 12); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 4)) << 8; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 12); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 12); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 8)) << 4; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 12); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 0)) << 12; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 12); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 12); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 4)) << 8; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 12); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 12); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 8)) << 4; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 12); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint32_t, 0)) << 12; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 12); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 12); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint32_t, 4)) << 8; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 12); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 12); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint32_t, 8)) << 4; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 12); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint32_t, 0)) << 12; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 12); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 12); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint32_t, 4)) << 8; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 12); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 12); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint32_t, 8)) << 4; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 12); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<13>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 13); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint32_t, 13); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 7)) << 6; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint32_t, 13); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 1)) << 12; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint32_t, 13); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 13); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint32_t, 5); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 8)) << 5; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 13); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint32_t, 11); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 2)) << 11; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 13); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint32_t, 13); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 9)) << 4; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint32_t, 13); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 10); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint32_t, 3)) << 10; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint32_t, 13); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 13); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint32_t, 3); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint32_t, 10)) << 3; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 13); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint32_t, 9); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint32_t, 4)) << 9; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 13); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint32_t, 13); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint32_t, 11)) << 2; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint32_t, 13); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint32_t, 5)) << 8; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint32_t, 13); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 13); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint32_t, 1); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint32_t, 12)) << 1; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 13); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint32_t, 7); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint32_t, 6)) << 7; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 13); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint32_t, 13); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<14>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 14); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 14); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 10)) << 4; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 14); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 6)) << 8; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 14); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 2)) << 12; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 14); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 14); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 12)) << 2; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 14); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 8)) << 6; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 14); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 10); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint32_t, 4)) << 10; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 14); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 14); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint32_t, 0)) << 14; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 14); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 14); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint32_t, 10)) << 4; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 14); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint32_t, 6)) << 8; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 14); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint32_t, 2)) << 12; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 14); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 14); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint32_t, 12)) << 2; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 14); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint32_t, 8)) << 6; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 14); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 10); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint32_t, 4)) << 10; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 14); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 14); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<15>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 15); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint32_t, 15); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 13)) << 2; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint32_t, 15); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 11)) << 4; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint32_t, 15); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 9)) << 6; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint32_t, 15); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 7)) << 8; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint32_t, 15); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 10); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 5)) << 10; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint32_t, 15); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint32_t, 3)) << 12; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint32_t, 15); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 14); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint32_t, 1)) << 14; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint32_t, 15); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 15); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint32_t, 1); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint32_t, 14)) << 1; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 15); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint32_t, 3); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint32_t, 12)) << 3; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 15); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint32_t, 5); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint32_t, 10)) << 5; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 15); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint32_t, 7); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint32_t, 8)) << 7; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 15); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint32_t, 9); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint32_t, 6)) << 9; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 15); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint32_t, 11); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint32_t, 4)) << 11; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 15); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint32_t, 13); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint32_t, 2)) << 13; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 15); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint32_t, 15); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<16>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 16); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 0)) << 16; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 16); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 0)) << 16; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 16); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 0)) << 16; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 16); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 0)) << 16; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 16); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 0)) << 16; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 16); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint32_t, 0)) << 16; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 16); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint32_t, 0)) << 16; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 16); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint32_t, 0)) << 16; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 16); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint32_t, 0)) << 16; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 16); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint32_t, 0)) << 16; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 16); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint32_t, 0)) << 16; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 16); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint32_t, 0)) << 16; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 16); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint32_t, 0)) << 16; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 16); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint32_t, 0)) << 16; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 16); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint32_t, 0)) << 16; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 16); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<17>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 17); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint32_t, 15); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 2)) << 15; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 17); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint32_t, 13); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 4)) << 13; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 17); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint32_t, 11); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 6)) << 11; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 17); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint32_t, 9); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 8)) << 9; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 17); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint32_t, 7); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 10)) << 7; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 17); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint32_t, 5); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint32_t, 12)) << 5; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 17); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint32_t, 3); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint32_t, 14)) << 3; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 17); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint32_t, 1); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint32_t, 16)) << 1; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint32_t, 1)) << 16; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint32_t, 17); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 14); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint32_t, 3)) << 14; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint32_t, 17); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint32_t, 5)) << 12; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint32_t, 17); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 10); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint32_t, 7)) << 10; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint32_t, 17); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint32_t, 9)) << 8; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint32_t, 17); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint32_t, 11)) << 6; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint32_t, 17); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint32_t, 13)) << 4; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint32_t, 17); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint32_t, 15)) << 2; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint32_t, 17); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<18>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 18); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 14); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 4)) << 14; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 18); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 10); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 8)) << 10; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 18); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 12)) << 6; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 18); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 16)) << 2; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 2)) << 16; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 18); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint32_t, 6)) << 12; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 18); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint32_t, 10)) << 8; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 18); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint32_t, 14)) << 4; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 18); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint32_t, 0)) << 18; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 18); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 14); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint32_t, 4)) << 14; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 18); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 10); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint32_t, 8)) << 10; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 18); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint32_t, 12)) << 6; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 18); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint32_t, 16)) << 2; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint32_t, 2)) << 16; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 18); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint32_t, 6)) << 12; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 18); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint32_t, 10)) << 8; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 18); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint32_t, 14)) << 4; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 18); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<19>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 19); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint32_t, 13); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 6)) << 13; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 19); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint32_t, 7); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 12)) << 7; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 19); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint32_t, 1); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 18)) << 1; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 14); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 5)) << 14; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint32_t, 19); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 11)) << 8; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint32_t, 19); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint32_t, 17)) << 2; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint32_t, 15); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint32_t, 4)) << 15; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 19); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint32_t, 9); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint32_t, 10)) << 9; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 19); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint32_t, 3); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint32_t, 16)) << 3; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint32_t, 3)) << 16; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint32_t, 19); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 10); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint32_t, 9)) << 10; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint32_t, 19); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint32_t, 15)) << 4; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint32_t, 17); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint32_t, 2)) << 17; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 19); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint32_t, 11); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint32_t, 8)) << 11; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 19); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint32_t, 5); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint32_t, 14)) << 5; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 18); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint32_t, 1)) << 18; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint32_t, 19); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint32_t, 7)) << 12; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint32_t, 19); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint32_t, 13)) << 6; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint32_t, 19); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<20>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 20); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 8)) << 12; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 20); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 16)) << 4; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 4)) << 16; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 20); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 12)) << 8; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 20); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 0)) << 20; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 20); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint32_t, 8)) << 12; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 20); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint32_t, 16)) << 4; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint32_t, 4)) << 16; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 20); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint32_t, 12)) << 8; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 20); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint32_t, 0)) << 20; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 20); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint32_t, 8)) << 12; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 20); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint32_t, 16)) << 4; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint32_t, 4)) << 16; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 20); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint32_t, 12)) << 8; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 20); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint32_t, 0)) << 20; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 20); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint32_t, 8)) << 12; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 20); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint32_t, 16)) << 4; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint32_t, 4)) << 16; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 20); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint32_t, 12)) << 8; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 20); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<21>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 21); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint32_t, 11); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 10)) << 11; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 21); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint32_t, 1); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 20)) << 1; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 9)) << 12; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint32_t, 21); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 19)) << 2; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint32_t, 13); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 8)) << 13; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 21); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint32_t, 3); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint32_t, 18)) << 3; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 14); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint32_t, 7)) << 14; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint32_t, 21); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint32_t, 17)) << 4; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint32_t, 15); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint32_t, 6)) << 15; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 21); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint32_t, 5); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint32_t, 16)) << 5; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint32_t, 5)) << 16; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint32_t, 21); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint32_t, 15)) << 6; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint32_t, 17); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint32_t, 4)) << 17; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 21); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint32_t, 7); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint32_t, 14)) << 7; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 18); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint32_t, 3)) << 18; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint32_t, 21); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint32_t, 13)) << 8; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint32_t, 19); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint32_t, 2)) << 19; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 21); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint32_t, 9); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint32_t, 12)) << 9; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 20); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint32_t, 1)) << 20; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint32_t, 21); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 10); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint32_t, 11)) << 10; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint32_t, 21); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<22>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 22); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 10); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 12)) << 10; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 20); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 2)) << 20; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 22); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 14)) << 8; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 18); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 4)) << 18; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 22); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 16)) << 6; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint32_t, 6)) << 16; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 22); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint32_t, 18)) << 4; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 14); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint32_t, 8)) << 14; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 22); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint32_t, 20)) << 2; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint32_t, 10)) << 12; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 22); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint32_t, 0)) << 22; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 22); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 10); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint32_t, 12)) << 10; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 20); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint32_t, 2)) << 20; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 22); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint32_t, 14)) << 8; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 18); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint32_t, 4)) << 18; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 22); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint32_t, 16)) << 6; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint32_t, 6)) << 16; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 22); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint32_t, 18)) << 4; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 14); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint32_t, 8)) << 14; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 22); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint32_t, 20)) << 2; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint32_t, 10)) << 12; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 22); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<23>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 23); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint32_t, 9); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 14)) << 9; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 18); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 5)) << 18; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint32_t, 23); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 19)) << 4; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint32_t, 13); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 10)) << 13; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 22); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 1)) << 22; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint32_t, 23); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint32_t, 15)) << 8; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint32_t, 17); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint32_t, 6)) << 17; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 23); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint32_t, 3); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint32_t, 20)) << 3; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint32_t, 11)) << 12; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint32_t, 21); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint32_t, 2)) << 21; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 23); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint32_t, 7); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint32_t, 16)) << 7; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint32_t, 7)) << 16; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint32_t, 23); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint32_t, 21)) << 2; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint32_t, 11); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint32_t, 12)) << 11; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 20); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint32_t, 3)) << 20; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint32_t, 23); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint32_t, 17)) << 6; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint32_t, 15); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint32_t, 8)) << 15; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 23); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint32_t, 1); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint32_t, 22)) << 1; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 10); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint32_t, 13)) << 10; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint32_t, 19); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint32_t, 4)) << 19; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 23); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint32_t, 5); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint32_t, 18)) << 5; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 14); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint32_t, 9)) << 14; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint32_t, 23); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<24>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 24); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 16)) << 8; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 8)) << 16; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 24); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 0)) << 24; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 24); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 16)) << 8; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 8)) << 16; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 24); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint32_t, 0)) << 24; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 24); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint32_t, 16)) << 8; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint32_t, 8)) << 16; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 24); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint32_t, 0)) << 24; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 24); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint32_t, 16)) << 8; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint32_t, 8)) << 16; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 24); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint32_t, 0)) << 24; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 24); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint32_t, 16)) << 8; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint32_t, 8)) << 16; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 24); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint32_t, 0)) << 24; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 24); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint32_t, 16)) << 8; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint32_t, 8)) << 16; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 24); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint32_t, 0)) << 24; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 24); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint32_t, 16)) << 8; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint32_t, 8)) << 16; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 24); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint32_t, 0)) << 24; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 24); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint32_t, 16)) << 8; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint32_t, 8)) << 16; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 24); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<25>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 25); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint32_t, 7); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 18)) << 7; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 14); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 11)) << 14; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint32_t, 21); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 4)) << 21; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 25); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint32_t, 3); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 22)) << 3; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 10); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 15)) << 10; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint32_t, 17); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint32_t, 8)) << 17; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 24); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint32_t, 1)) << 24; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint32_t, 25); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint32_t, 19)) << 6; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint32_t, 13); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint32_t, 12)) << 13; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 20); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint32_t, 5)) << 20; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint32_t, 25); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint32_t, 23)) << 2; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint32_t, 9); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint32_t, 16)) << 9; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint32_t, 9)) << 16; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint32_t, 23); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint32_t, 2)) << 23; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 25); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint32_t, 5); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint32_t, 20)) << 5; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint32_t, 13)) << 12; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint32_t, 19); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint32_t, 6)) << 19; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 25); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint32_t, 1); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint32_t, 24)) << 1; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint32_t, 17)) << 8; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint32_t, 15); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint32_t, 10)) << 15; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 22); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint32_t, 3)) << 22; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint32_t, 25); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint32_t, 21)) << 4; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint32_t, 11); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint32_t, 14)) << 11; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 18); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint32_t, 7)) << 18; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint32_t, 25); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<26>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 26); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 20)) << 6; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 14)) << 12; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 18); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 8)) << 18; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 24); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 2)) << 24; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 26); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 22)) << 4; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 10); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint32_t, 16)) << 10; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint32_t, 10)) << 16; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 22); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint32_t, 4)) << 22; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 26); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint32_t, 24)) << 2; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint32_t, 18)) << 8; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 14); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint32_t, 12)) << 14; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 20); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint32_t, 6)) << 20; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 26); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint32_t, 0)) << 26; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 26); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint32_t, 20)) << 6; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint32_t, 14)) << 12; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 18); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint32_t, 8)) << 18; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 24); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint32_t, 2)) << 24; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 26); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint32_t, 22)) << 4; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 10); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint32_t, 16)) << 10; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint32_t, 10)) << 16; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 22); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint32_t, 4)) << 22; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 26); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint32_t, 24)) << 2; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint32_t, 18)) << 8; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 14); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint32_t, 12)) << 14; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 20); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint32_t, 6)) << 20; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 26); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<27>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 27); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint32_t, 5); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 22)) << 5; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 10); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 17)) << 10; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint32_t, 15); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 12)) << 15; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 20); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 7)) << 20; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint32_t, 25); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 2)) << 25; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 27); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint32_t, 3); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint32_t, 24)) << 3; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint32_t, 19)) << 8; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint32_t, 13); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint32_t, 14)) << 13; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 18); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint32_t, 9)) << 18; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint32_t, 23); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint32_t, 4)) << 23; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 27); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint32_t, 1); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint32_t, 26)) << 1; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint32_t, 21)) << 6; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint32_t, 11); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint32_t, 16)) << 11; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint32_t, 11)) << 16; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint32_t, 21); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint32_t, 6)) << 21; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 26); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint32_t, 1)) << 26; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint32_t, 27); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint32_t, 23)) << 4; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint32_t, 9); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint32_t, 18)) << 9; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 14); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint32_t, 13)) << 14; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint32_t, 19); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint32_t, 8)) << 19; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 24); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint32_t, 3)) << 24; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint32_t, 27); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint32_t, 25)) << 2; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint32_t, 7); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint32_t, 20)) << 7; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint32_t, 15)) << 12; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint32_t, 17); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint32_t, 10)) << 17; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 22); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint32_t, 5)) << 22; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint32_t, 27); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<28>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 28); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 24)) << 4; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 20)) << 8; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 16)) << 12; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 12)) << 16; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 20); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 8)) << 20; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 24); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint32_t, 4)) << 24; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 28); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint32_t, 0)) << 28; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 28); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint32_t, 24)) << 4; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint32_t, 20)) << 8; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint32_t, 16)) << 12; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint32_t, 12)) << 16; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 20); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint32_t, 8)) << 20; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 24); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint32_t, 4)) << 24; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 28); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint32_t, 0)) << 28; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 28); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint32_t, 24)) << 4; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint32_t, 20)) << 8; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint32_t, 16)) << 12; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint32_t, 12)) << 16; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 20); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint32_t, 8)) << 20; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 24); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint32_t, 4)) << 24; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 28); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint32_t, 0)) << 28; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 28); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint32_t, 24)) << 4; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint32_t, 20)) << 8; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint32_t, 16)) << 12; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint32_t, 12)) << 16; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 20); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint32_t, 8)) << 20; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 24); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint32_t, 4)) << 24; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 28); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<29>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 29); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint32_t, 3); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 26)) << 3; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 23)) << 6; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint32_t, 9); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 20)) << 9; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 17)) << 12; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint32_t, 15); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 14)) << 15; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 18); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint32_t, 11)) << 18; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint32_t, 21); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint32_t, 8)) << 21; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 24); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint32_t, 5)) << 24; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint32_t, 27); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint32_t, 2)) << 27; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 29); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint32_t, 1); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint32_t, 28)) << 1; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint32_t, 25)) << 4; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint32_t, 7); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint32_t, 22)) << 7; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 10); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint32_t, 19)) << 10; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint32_t, 13); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint32_t, 16)) << 13; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint32_t, 13)) << 16; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint32_t, 19); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint32_t, 10)) << 19; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 22); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint32_t, 7)) << 22; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint32_t, 25); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint32_t, 4)) << 25; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 28); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint32_t, 1)) << 28; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint32_t, 29); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint32_t, 27)) << 2; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint32_t, 5); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint32_t, 24)) << 5; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint32_t, 21)) << 8; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint32_t, 11); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint32_t, 18)) << 11; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 14); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint32_t, 15)) << 14; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint32_t, 17); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint32_t, 12)) << 17; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 20); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint32_t, 9)) << 20; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint32_t, 23); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint32_t, 6)) << 23; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 26); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint32_t, 3)) << 26; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint32_t, 29); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<30>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 30); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 28)) << 2; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 26)) << 4; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 24)) << 6; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 22)) << 8; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 10); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 20)) << 10; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint32_t, 18)) << 12; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 14); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint32_t, 16)) << 14; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint32_t, 14)) << 16; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 18); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint32_t, 12)) << 18; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 20); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint32_t, 10)) << 20; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 22); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint32_t, 8)) << 22; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 24); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint32_t, 6)) << 24; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 26); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint32_t, 4)) << 26; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 28); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint32_t, 2)) << 28; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 30); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint32_t, 0)) << 30; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint32_t, 30); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint32_t, 28)) << 2; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint32_t, 26)) << 4; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint32_t, 24)) << 6; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint32_t, 22)) << 8; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 10); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint32_t, 20)) << 10; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint32_t, 18)) << 12; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 14); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint32_t, 16)) << 14; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint32_t, 14)) << 16; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 18); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint32_t, 12)) << 18; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 20); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint32_t, 10)) << 20; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 22); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint32_t, 8)) << 22; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 24); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint32_t, 6)) << 24; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 26); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint32_t, 4)) << 26; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 28); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint32_t, 2)) << 28; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 30); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<31>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint32_t src; + uint32_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint32_t, 31); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint32_t, 1); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint32_t, 30)) << 1; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint32_t, 2); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint32_t, 29)) << 2; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint32_t, 3); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint32_t, 28)) << 3; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint32_t, 4); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint32_t, 27)) << 4; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint32_t, 5); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint32_t, 26)) << 5; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint32_t, 6); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint32_t, 25)) << 6; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint32_t, 7); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint32_t, 24)) << 7; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint32_t, 8); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint32_t, 23)) << 8; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint32_t, 9); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint32_t, 22)) << 9; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint32_t, 10); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint32_t, 21)) << 10; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint32_t, 11); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint32_t, 20)) << 11; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint32_t, 12); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint32_t, 19)) << 12; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint32_t, 13); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint32_t, 18)) << 13; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint32_t, 14); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint32_t, 17)) << 14; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint32_t, 15); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint32_t, 16)) << 15; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint32_t, 16); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint32_t, 15)) << 16; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint32_t, 17); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint32_t, 14)) << 17; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint32_t, 18); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint32_t, 13)) << 18; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint32_t, 19); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint32_t, 12)) << 19; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint32_t, 20); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint32_t, 11)) << 20; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint32_t, 21); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint32_t, 10)) << 21; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint32_t, 22); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint32_t, 9)) << 22; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint32_t, 23); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint32_t, 8)) << 23; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint32_t, 24); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint32_t, 7)) << 24; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint32_t, 25); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint32_t, 6)) << 25; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint32_t, 26); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint32_t, 5)) << 26; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint32_t, 27); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint32_t, 4)) << 27; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint32_t, 28); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint32_t, 3)) << 28; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint32_t, 29); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint32_t, 2)) << 29; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint32_t, 30); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint32_t, 1)) << 30; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint32_t, 31); + out[INDEX(31, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_32_lane<32>(const uint32_t *__restrict in, uint32_t *__restrict out, uint32_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + #pragma unroll + for (int row = 0; row < 32; row++) { + out[INDEX(row, lane)] = in[LANE_COUNT * row + lane] + reference; + } +} + +/// Runtime dispatch to the optimized lane decoder for the given bit width. +__device__ __noinline__ void bit_unpack_32_lane( + const uint32_t *__restrict in, + uint32_t *__restrict out, + uint32_t reference, + unsigned int lane, + uint32_t bit_width +) { + switch (bit_width) { + case 0: _bit_unpack_32_lane<0>(in, out, reference, lane); break; + case 1: _bit_unpack_32_lane<1>(in, out, reference, lane); break; + case 2: _bit_unpack_32_lane<2>(in, out, reference, lane); break; + case 3: _bit_unpack_32_lane<3>(in, out, reference, lane); break; + case 4: _bit_unpack_32_lane<4>(in, out, reference, lane); break; + case 5: _bit_unpack_32_lane<5>(in, out, reference, lane); break; + case 6: _bit_unpack_32_lane<6>(in, out, reference, lane); break; + case 7: _bit_unpack_32_lane<7>(in, out, reference, lane); break; + case 8: _bit_unpack_32_lane<8>(in, out, reference, lane); break; + case 9: _bit_unpack_32_lane<9>(in, out, reference, lane); break; + case 10: _bit_unpack_32_lane<10>(in, out, reference, lane); break; + case 11: _bit_unpack_32_lane<11>(in, out, reference, lane); break; + case 12: _bit_unpack_32_lane<12>(in, out, reference, lane); break; + case 13: _bit_unpack_32_lane<13>(in, out, reference, lane); break; + case 14: _bit_unpack_32_lane<14>(in, out, reference, lane); break; + case 15: _bit_unpack_32_lane<15>(in, out, reference, lane); break; + case 16: _bit_unpack_32_lane<16>(in, out, reference, lane); break; + case 17: _bit_unpack_32_lane<17>(in, out, reference, lane); break; + case 18: _bit_unpack_32_lane<18>(in, out, reference, lane); break; + case 19: _bit_unpack_32_lane<19>(in, out, reference, lane); break; + case 20: _bit_unpack_32_lane<20>(in, out, reference, lane); break; + case 21: _bit_unpack_32_lane<21>(in, out, reference, lane); break; + case 22: _bit_unpack_32_lane<22>(in, out, reference, lane); break; + case 23: _bit_unpack_32_lane<23>(in, out, reference, lane); break; + case 24: _bit_unpack_32_lane<24>(in, out, reference, lane); break; + case 25: _bit_unpack_32_lane<25>(in, out, reference, lane); break; + case 26: _bit_unpack_32_lane<26>(in, out, reference, lane); break; + case 27: _bit_unpack_32_lane<27>(in, out, reference, lane); break; + case 28: _bit_unpack_32_lane<28>(in, out, reference, lane); break; + case 29: _bit_unpack_32_lane<29>(in, out, reference, lane); break; + case 30: _bit_unpack_32_lane<30>(in, out, reference, lane); break; + case 31: _bit_unpack_32_lane<31>(in, out, reference, lane); break; + case 32: _bit_unpack_32_lane<32>(in, out, reference, lane); break; + } +} + diff --git a/vortex-cuda/kernels/src/bit_unpack_64.cu b/vortex-cuda/kernels/src/bit_unpack_64.cu index 581ee159376..ebe0b125369 100644 --- a/vortex-cuda/kernels/src/bit_unpack_64.cu +++ b/vortex-cuda/kernels/src/bit_unpack_64.cu @@ -1,13054 +1,487 @@ // AUTO-GENERATED. Do not edit by hand! -#include -#include -#include -#include "fastlanes_common.cuh" +#include "bit_unpack_64_lanes.cuh" #include "patches.cuh" template -__device__ void _bit_unpack_64_lane(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane); - -template <> -__device__ void _bit_unpack_64_lane<0>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - #pragma unroll - for (int row = 0; row < 64; row++) { - out[INDEX(row, lane)] = reference; - } -} - -template <> -__device__ void _bit_unpack_64_lane<1>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 1); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 1); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 1); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 1); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 1); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 1); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 1); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 1); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 1); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 1); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 1); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 1); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 1); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 1); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 1); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 1); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 1); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 1); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 1); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 1); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 1); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 1); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 1); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 1); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 1); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 1); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 1); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 1); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 1); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 1); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 1); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 1); - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 1); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 1); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 1); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 1); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 1); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 1); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 1); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 1); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 1); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 1); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 1); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 1); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 1); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 1); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 1); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 1); - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 1); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 1); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 1); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 1); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 1); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 1); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 1); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 1); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 1); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 1); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 1); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 1); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 1); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 1); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 1); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<2>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 2); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 2); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 2); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 2); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 2); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 2); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 2); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 2); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 2); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 2); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 2); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 2); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 2); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 2); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 2); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 2); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 2); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 2); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 2); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 2); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 2); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 2); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 2); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 2); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 2); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 2); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 2); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 2); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 2); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 2); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 2); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 0)) << 2; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 2); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 2); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 2); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 2); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 2); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 2); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 2); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 2); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 2); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 2); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 2); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 2); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 2); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 2); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 2); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 2); - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 2); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 2); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 2); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 2); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 2); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 2); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 2); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 2); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 2); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 2); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 2); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 2); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 2); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 2); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 2); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<3>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 3); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 3); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 3); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 3); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 3); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 3); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 3); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 3); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 3); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 3); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 3); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 3); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 3); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 3); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 3); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 3); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 3); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 3); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 3); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 3); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 3); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 2)) << 1; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 3); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 3); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 3); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 3); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 3); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 3); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 3); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 3); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 3); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 3); - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 3); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 3); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 3); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 3); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 3); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 3); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 3); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 3); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 3); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 3); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 1)) << 2; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 3); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 3); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 3); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 3); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 3); - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 3); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 3); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 3); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 3); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 3); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 3); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 3); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 3); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 3); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 3); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 3); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 3); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 3); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 3); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 3); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<4>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 4); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 4); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 4); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 4); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 4); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 4); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 4); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 4); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 4); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 4); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 4); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 4); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 4); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 4); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 4); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 0)) << 4; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 4); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 4); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 4); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 4); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 4); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 4); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 4); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 4); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 4); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 4); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 4); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 4); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 4); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 4); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 4); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 0)) << 4; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 4); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 4); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 4); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 4); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 4); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 4); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 4); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 4); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 4); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 4); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 4); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 4); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 4); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 4); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 4); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 0)) << 4; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 4); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 4); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 4); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 4); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 4); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 4); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 4); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 4); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 4); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 4); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 4); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 4); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 4); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 4); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 4); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<5>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 5); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 5); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 5); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 5); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 5); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 5); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 5); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 5); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 5); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 5); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 5); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 5); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 1)) << 4; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 5); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 5); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 5); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 5); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 5); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 5); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 5); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 5); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 5); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 5); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 5); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 5); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 2)) << 3; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 5); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 5); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 5); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 5); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 5); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 5); - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 5); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 5); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 5); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 5); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 5); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 5); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 3)) << 2; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 5); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 5); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 5); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 5); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 5); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 5); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 5); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 5); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 5); - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 5); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 5); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 5); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 4)) << 1; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 5); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 5); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 5); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 5); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 5); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 5); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 5); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 5); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 5); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 5); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 5); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<6>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 6); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 6); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 6); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 6); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 6); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 6); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 6); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 6); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 6); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 6); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 2)) << 4; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 6); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 6); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 6); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 6); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 6); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 6); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 6); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 6); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 6); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 6); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 4)) << 2; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 6); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 6); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 6); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 6); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 6); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 6); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 6); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 6); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 6); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 0)) << 6; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 6); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 6); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 6); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 6); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 6); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 6); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 6); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 6); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 6); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 6); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 2)) << 4; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 6); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 6); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 6); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 6); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 6); - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 6); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 6); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 6); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 6); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 6); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 4)) << 2; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 6); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 6); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 6); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 6); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 6); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 6); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 6); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 6); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 6); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<7>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 7); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 7); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 7); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 7); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 7); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 7); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 7); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 7); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 7); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 6)) << 1; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 7); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 7); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 7); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 7); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 7); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 7); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 7); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 7); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 5)) << 2; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 7); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 7); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 7); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 7); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 7); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 7); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 7); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 7); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 4)) << 3; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 7); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 7); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 7); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 7); - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 7); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 7); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 7); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 7); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 3)) << 4; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 7); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 7); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 7); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 7); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 7); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 7); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 7); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 7); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 2)) << 5; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 7); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 7); - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 7); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 7); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 7); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 7); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 7); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 7); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 1)) << 6; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 7); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 7); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 7); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 7); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 7); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 7); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 7); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 7); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<8>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 8); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 8); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 8); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 8); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 8); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 8); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 8); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 0)) << 8; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 8); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 8); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 8); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 8); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 8); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 8); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 8); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 0)) << 8; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 8); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 8); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 8); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 8); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 8); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 8); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 8); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 0)) << 8; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 8); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 8); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 8); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 8); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 8); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 8); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 8); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 0)) << 8; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 8); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 8); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 8); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 8); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 8); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 8); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 8); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 0)) << 8; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 8); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 8); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 8); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 8); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 8); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 8); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 8); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 0)) << 8; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 8); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 8); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 8); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 8); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 8); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 8); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 8); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 0)) << 8; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 8); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 8); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 8); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 8); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 8); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 8); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 8); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<9>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 9); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 9); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 9); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 9); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 9); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 9); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 9); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 8)) << 1; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 9); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 9); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 9); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 9); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 9); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 9); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 7)) << 2; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 9); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 9); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 9); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 9); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 9); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 9); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 6)) << 3; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 9); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 9); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 9); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 9); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 9); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 9); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 5)) << 4; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 9); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 9); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 9); - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 9); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 9); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 9); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 4)) << 5; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 9); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 9); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 9); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 9); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 9); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 9); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 3)) << 6; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 9); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 9); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 9); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 9); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 9); - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 9); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 2)) << 7; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 9); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 9); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 9); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 9); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 9); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 9); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 1)) << 8; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 9); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 9); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 9); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 9); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 9); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 9); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<10>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 10); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 10); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 10); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 10); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 10); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 10); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 6)) << 4; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 10); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 10); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 10); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 10); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 10); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 2)) << 8; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 10); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 10); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 10); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 10); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 10); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 10); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 8)) << 2; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 10); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 10); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 10); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 10); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 10); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 4)) << 6; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 10); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 10); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 10); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 10); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 10); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 0)) << 10; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 10); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 10); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 10); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 10); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 10); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 10); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 6)) << 4; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 10); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 10); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 10); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 10); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 10); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 2)) << 8; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 10); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 10); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 10); - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 10); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 10); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 10); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 8)) << 2; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 10); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 10); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 10); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 10); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 10); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 4)) << 6; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 10); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 10); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 10); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 10); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 10); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<11>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 11); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 11); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 11); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 11); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 11); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 2)) << 9; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 11); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 11); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 11); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 11); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 11); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 4)) << 7; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 11); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 11); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 11); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 11); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 11); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 6)) << 5; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 11); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 11); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 11); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 11); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 11); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 8)) << 3; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 11); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 11); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 11); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 11); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 11); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 10)) << 1; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 11); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 11); - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 11); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 11); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 1)) << 10; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 11); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 11); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 11); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 11); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 11); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 3)) << 8; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 11); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 11); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 11); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 11); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 11); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 5)) << 6; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 11); - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 11); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 11); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 11); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 11); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 7)) << 4; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 11); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 11); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 11); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 11); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 11); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 9)) << 2; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 11); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 11); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 11); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 11); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<12>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 12); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 12); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 12); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 12); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 12); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 8)) << 4; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 12); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 12); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 12); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 12); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 4)) << 8; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 12); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 12); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 12); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 12); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 0)) << 12; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 12); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 12); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 12); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 12); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 12); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 8)) << 4; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 12); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 12); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 12); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 12); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 4)) << 8; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 12); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 12); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 12); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 12); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 0)) << 12; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 12); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 12); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 12); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 12); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 12); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 8)) << 4; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 12); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 12); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 12); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 12); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 4)) << 8; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 12); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 12); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 12); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 12); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 0)) << 12; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 12); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 12); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 12); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 12); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 12); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 8)) << 4; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 12); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 12); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 12); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 12); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 4)) << 8; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 12); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 12); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 12); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 12); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<13>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 13); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 13); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 13); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 13); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 1)) << 12; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 13); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 13); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 13); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 13); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 2)) << 11; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 13); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 13); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 13); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 13); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 3)) << 10; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 13); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 13); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 13); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 13); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 4)) << 9; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 13); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 13); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 13); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 13); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 5)) << 8; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 13); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 13); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 13); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 13); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 6)) << 7; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 13); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 13); - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 13); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 13); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 7)) << 6; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 13); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 13); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 13); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 13); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 8)) << 5; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 13); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 13); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 13); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 13); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 9)) << 4; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 13); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 13); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 13); - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 13); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 10)) << 3; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 13); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 13); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 13); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 13); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 11)) << 2; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 13); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 13); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 13); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 13); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 12)) << 1; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 13); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 13); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 13); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<14>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 14); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 14); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 14); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 14); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 6)) << 8; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 14); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 14); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 14); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 14); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 12)) << 2; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 14); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 14); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 14); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 4)) << 10; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 14); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 14); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 14); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 14); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 10)) << 4; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 14); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 14); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 14); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 2)) << 12; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 14); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 14); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 14); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 14); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 8)) << 6; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 14); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 14); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 14); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 0)) << 14; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 14); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 14); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 14); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 14); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 6)) << 8; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 14); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 14); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 14); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 14); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 12)) << 2; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 14); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 14); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 14); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 4)) << 10; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 14); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 14); - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 14); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 14); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 10)) << 4; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 14); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 14); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 14); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 2)) << 12; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 14); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 14); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 14); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 14); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 8)) << 6; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 14); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 14); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 14); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<15>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 15); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 15); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 15); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 15); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 11)) << 4; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 15); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 15); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 15); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 7)) << 8; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 15); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 15); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 15); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 3)) << 12; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 15); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 15); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 15); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 15); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 14)) << 1; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 15); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 15); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 15); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 10)) << 5; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 15); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 15); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 15); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 6)) << 9; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 15); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 15); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 15); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 2)) << 13; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 15); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 15); - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 15); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 15); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 13)) << 2; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 15); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 15); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 15); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 9)) << 6; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 15); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 15); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 15); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 5)) << 10; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 15); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 15); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 15); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 1)) << 14; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 15); - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 15); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 15); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 15); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 12)) << 3; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 15); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 15); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 15); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 8)) << 7; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 15); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 15); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 15); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 4)) << 11; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 15); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 15); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 15); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 15); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<16>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 16); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 16); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 16); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 0)) << 16; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 16); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 16); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 16); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 0)) << 16; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 16); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 16); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 16); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 0)) << 16; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 16); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 16); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 16); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 0)) << 16; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 16); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 16); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 16); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 0)) << 16; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 16); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 16); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 16); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 0)) << 16; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 16); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 16); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 16); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 0)) << 16; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 16); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 16); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 16); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 0)) << 16; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 16); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 16); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 16); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 0)) << 16; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 16); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 16); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 16); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 0)) << 16; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 16); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 16); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 16); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 0)) << 16; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 16); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 16); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 16); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 0)) << 16; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 16); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 16); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 16); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 0)) << 16; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 16); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 16); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 16); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 0)) << 16; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 16); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 16); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 16); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 0)) << 16; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 16); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 16); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 16); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<17>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 17); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 17); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 17); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 4)) << 13; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 17); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 17); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 17); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 8)) << 9; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 17); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 17); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 17); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 12)) << 5; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 17); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 17); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 17); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 16)) << 1; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 17); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 17); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 3)) << 14; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 17); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 17); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 17); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 7)) << 10; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 17); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 17); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 17); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 11)) << 6; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 17); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 17); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 17); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 15)) << 2; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 17); - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 17); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 15); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 2)) << 15; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 17); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 17); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 17); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 6)) << 11; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 17); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 17); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 17); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 10)) << 7; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 17); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 17); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 17); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 14)) << 3; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 17); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 17); - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 1)) << 16; - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 17); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 17); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 17); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 5)) << 12; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 17); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 17); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 17); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 9)) << 8; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 17); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 17); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 17); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 13)) << 4; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 17); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 17); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 17); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<18>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 18); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 18); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 18); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 8)) << 10; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 18); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 18); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 18); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 16)) << 2; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 18); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 18); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 6)) << 12; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 18); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 18); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 18); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 14)) << 4; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 18); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 18); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 4)) << 14; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 18); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 18); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 18); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 12)) << 6; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 18); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 18); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 2)) << 16; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 18); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 18); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 18); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 10)) << 8; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 18); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 18); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 0)) << 18; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 18); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 18); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 18); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 8)) << 10; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 18); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 18); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 18); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 16)) << 2; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 18); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 18); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 6)) << 12; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 18); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 18); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 18); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 14)) << 4; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 18); - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 18); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 4)) << 14; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 18); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 18); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 18); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 12)) << 6; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 18); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 18); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 2)) << 16; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 18); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 18); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 18); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 10)) << 8; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 18); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 18); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<19>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 19); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 19); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 19); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 12)) << 7; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 19); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 19); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 5)) << 14; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 19); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 19); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 19); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 17)) << 2; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 19); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 19); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 10)) << 9; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 19); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 19); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 3)) << 16; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 19); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 19); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 19); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 15)) << 4; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 19); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 19); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 8)) << 11; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 19); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 19); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 1)) << 18; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 19); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 19); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 19); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 13)) << 6; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 19); - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 19); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 6)) << 13; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 19); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 19); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 19); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 18)) << 1; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 19); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 19); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 11)) << 8; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 19); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 19); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 15); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 4)) << 15; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 19); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 19); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 19); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 16)) << 3; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 19); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 19); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 9)) << 10; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 19); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 19); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 17); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 2)) << 17; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 19); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 19); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 19); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 14)) << 5; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 19); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 19); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 7)) << 12; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 19); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 19); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 19); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<20>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 20); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 20); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 20); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 16)) << 4; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 20); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 20); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 12)) << 8; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 20); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 20); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 8)) << 12; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 20); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 20); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 4)) << 16; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 20); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 20); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 0)) << 20; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 20); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 20); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 20); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 16)) << 4; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 20); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 20); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 12)) << 8; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 20); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 20); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 8)) << 12; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 20); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 20); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 4)) << 16; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 20); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 20); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 0)) << 20; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 20); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 20); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 20); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 16)) << 4; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 20); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 20); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 12)) << 8; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 20); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 20); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 8)) << 12; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 20); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 20); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 4)) << 16; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 20); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 20); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 0)) << 20; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 20); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 20); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 20); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 16)) << 4; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 20); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 20); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 12)) << 8; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 20); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 20); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 8)) << 12; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 20); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 20); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 4)) << 16; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 20); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 20); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<21>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 21); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 21); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 21); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 20)) << 1; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 21); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 21); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 19)) << 2; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 21); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 21); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 18)) << 3; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 21); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 21); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 17)) << 4; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 21); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 21); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 16)) << 5; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 21); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 21); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 15)) << 6; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 21); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 21); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 14)) << 7; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 21); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 21); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 13)) << 8; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 21); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 21); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 12)) << 9; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 21); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 21); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 11)) << 10; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 21); - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 21); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 10)) << 11; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 21); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 21); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 9)) << 12; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 21); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 21); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 8)) << 13; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 21); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 21); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 7)) << 14; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 21); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 21); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 15); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 6)) << 15; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 21); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 21); - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 5)) << 16; - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 21); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 21); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 17); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 4)) << 17; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 21); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 21); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 3)) << 18; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 21); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 21); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 19); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 2)) << 19; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 21); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 21); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 1)) << 20; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 21); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 21); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 21); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<22>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 22); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 22); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 2)) << 20; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 22); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 22); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 4)) << 18; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 22); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 22); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 6)) << 16; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 22); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 22); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 8)) << 14; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 22); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 22); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 10)) << 12; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 22); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 22); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 12)) << 10; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 22); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 22); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 14)) << 8; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 22); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 22); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 16)) << 6; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 22); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 22); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 18)) << 4; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 22); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 22); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 20)) << 2; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 22); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 0)) << 22; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 22); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 22); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 2)) << 20; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 22); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 22); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 4)) << 18; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 22); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 22); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 6)) << 16; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 22); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 22); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 8)) << 14; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 22); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 22); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 10)) << 12; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 22); - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 22); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 12)) << 10; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 22); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 22); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 14)) << 8; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 22); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 22); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 16)) << 6; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 22); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 22); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 18)) << 4; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 22); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 22); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 20)) << 2; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 22); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<23>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 23); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 23); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 5)) << 18; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 23); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 23); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 10)) << 13; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 23); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 23); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 15)) << 8; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 23); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 23); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 20)) << 3; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 23); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 21); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 2)) << 21; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 23); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 23); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 7)) << 16; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 23); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 23); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 12)) << 11; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 23); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 23); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 17)) << 6; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 23); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 23); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 22)) << 1; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 23); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 19); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 4)) << 19; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 23); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 23); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 9)) << 14; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 23); - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 23); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 14)) << 9; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 23); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 23); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 19)) << 4; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 23); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 1)) << 22; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 23); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 23); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 17); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 6)) << 17; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 23); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 23); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 11)) << 12; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 23); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 23); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 16)) << 7; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 23); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 23); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 21)) << 2; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 23); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 3)) << 20; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 23); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 23); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 15); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 8)) << 15; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 23); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 23); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 13)) << 10; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 23); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 23); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 18)) << 5; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 23); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 23); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<24>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 24); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 24); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 8)) << 16; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 24); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 24); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 16)) << 8; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 24); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 0)) << 24; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 24); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 24); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 8)) << 16; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 24); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 24); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 16)) << 8; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 24); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 0)) << 24; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 24); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 24); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 8)) << 16; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 24); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 24); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 16)) << 8; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 24); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 0)) << 24; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 24); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 24); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 8)) << 16; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 24); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 24); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 16)) << 8; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 24); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 0)) << 24; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 24); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 24); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 8)) << 16; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 24); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 24); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 16)) << 8; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 24); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 0)) << 24; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 24); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 24); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 8)) << 16; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 24); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 24); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 16)) << 8; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 24); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 0)) << 24; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 24); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 24); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 8)) << 16; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 24); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 24); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 16)) << 8; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 24); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 0)) << 24; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 24); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 24); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 8)) << 16; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 24); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 24); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 16)) << 8; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 24); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<25>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 25); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 25); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 11)) << 14; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 25); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 25); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 22)) << 3; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 25); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 17); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 8)) << 17; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 25); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 25); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 19)) << 6; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 25); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 5)) << 20; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 25); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 25); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 16)) << 9; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 25); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 23); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 2)) << 23; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 25); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 25); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 13)) << 12; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 25); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 25); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 24)) << 1; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 25); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 15); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 10)) << 15; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 25); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 25); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 21)) << 4; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 25); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 7)) << 18; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 25); - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 25); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 18)) << 7; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 25); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 21); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 4)) << 21; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 25); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 25); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 15)) << 10; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 25); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 1)) << 24; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 25); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 25); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 12)) << 13; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 25); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 25); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 23)) << 2; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 25); - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 9)) << 16; - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 25); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 25); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 20)) << 5; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 25); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 19); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 6)) << 19; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 25); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 25); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 17)) << 8; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 25); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 3)) << 22; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 25); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 25); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 14)) << 11; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 25); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 25); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<26>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 26); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 26); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 14)) << 12; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 26); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 2)) << 24; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 26); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 26); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 16)) << 10; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 26); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 4)) << 22; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 26); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 26); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 18)) << 8; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 26); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 6)) << 20; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 26); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 26); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 20)) << 6; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 26); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 8)) << 18; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 26); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 26); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 22)) << 4; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 26); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 10)) << 16; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 26); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 26); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 24)) << 2; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 26); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 12)) << 14; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 26); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 0)) << 26; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 26); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 26); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 14)) << 12; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 26); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 2)) << 24; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 26); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 26); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 16)) << 10; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 26); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 4)) << 22; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 26); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 26); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 18)) << 8; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 26); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 6)) << 20; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 26); - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 26); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 20)) << 6; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 26); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 8)) << 18; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 26); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 26); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 22)) << 4; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 26); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 10)) << 16; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 26); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 26); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 24)) << 2; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 26); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 12)) << 14; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 26); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<27>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 27); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 27); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 17)) << 10; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 27); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 7)) << 20; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 27); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 27); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 24)) << 3; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 27); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 14)) << 13; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 27); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 23); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 4)) << 23; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 27); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 27); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 21)) << 6; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 27); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 11)) << 16; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 27); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 1)) << 26; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 27); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 27); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 18)) << 9; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 27); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 19); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 8)) << 19; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 27); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 27); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 25)) << 2; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 27); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 15)) << 12; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 27); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 5)) << 22; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 27); - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 27); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 22)) << 5; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 27); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 15); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 12)) << 15; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 27); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 25); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 2)) << 25; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 27); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 27); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 19)) << 8; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 27); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 9)) << 18; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 27); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 27); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 26)) << 1; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 27); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 16)) << 11; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 27); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 21); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 6)) << 21; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 27); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 27); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 23)) << 4; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 27); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 13)) << 14; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 27); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 3)) << 24; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 27); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 27); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 20)) << 7; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 27); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 17); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 10)) << 17; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 27); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 27); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<28>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 28); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 28); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 20)) << 8; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 28); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 12)) << 16; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 28); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 4)) << 24; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 28); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 28); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 24)) << 4; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 28); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 16)) << 12; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 28); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 8)) << 20; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 28); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 0)) << 28; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 28); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 28); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 20)) << 8; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 28); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 12)) << 16; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 28); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 4)) << 24; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 28); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 28); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 24)) << 4; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 28); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 16)) << 12; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 28); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 8)) << 20; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 28); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 0)) << 28; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 28); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 28); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 20)) << 8; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 28); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 12)) << 16; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 28); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 4)) << 24; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 28); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 28); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 24)) << 4; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 28); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 16)) << 12; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 28); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 8)) << 20; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 28); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 0)) << 28; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 28); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 28); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 20)) << 8; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 28); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 12)) << 16; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 28); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 4)) << 24; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 28); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 28); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 24)) << 4; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 28); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 16)) << 12; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 28); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 8)) << 20; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 28); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<29>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 29); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 29); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 23)) << 6; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 29); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 17)) << 12; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 29); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 11)) << 18; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 29); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 5)) << 24; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 29); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 29); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 28)) << 1; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 29); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 22)) << 7; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 29); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 16)) << 13; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 29); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 19); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 10)) << 19; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 29); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 25); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 4)) << 25; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 29); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 29); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 27)) << 2; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 29); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 21)) << 8; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 29); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 15)) << 14; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 29); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 9)) << 20; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 29); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 3)) << 26; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 29); - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 29); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 26)) << 3; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 29); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 20)) << 9; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 29); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 15); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 14)) << 15; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 29); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 21); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 8)) << 21; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 29); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 27); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 2)) << 27; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 29); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 29); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 25)) << 4; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 29); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 19)) << 10; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 29); - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 13)) << 16; - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 29); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 7)) << 22; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 29); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 1)) << 28; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 29); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 29); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 24)) << 5; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 29); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 18)) << 11; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 29); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 17); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 12)) << 17; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 29); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 23); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 6)) << 23; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 29); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 29); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<30>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 30); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 30); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 26)) << 4; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 30); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 22)) << 8; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 30); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 18)) << 12; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 30); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 14)) << 16; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 30); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 10)) << 20; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 30); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 6)) << 24; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 30); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 2)) << 28; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 30); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 30); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 28)) << 2; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 30); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 24)) << 6; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 30); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 20)) << 10; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 30); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 16)) << 14; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 30); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 12)) << 18; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 30); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 8)) << 22; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 30); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 4)) << 26; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 30); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 0)) << 30; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 30); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 30); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 26)) << 4; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 30); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 22)) << 8; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 30); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 18)) << 12; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 30); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 14)) << 16; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 30); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 10)) << 20; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 30); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 6)) << 24; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 30); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 2)) << 28; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 30); - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 30); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 28)) << 2; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 30); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 24)) << 6; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 30); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 20)) << 10; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 30); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 16)) << 14; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 30); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 12)) << 18; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 30); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 8)) << 22; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 30); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 4)) << 26; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 30); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<31>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 31); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 31); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 29)) << 2; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 31); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 27)) << 4; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 31); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 25)) << 6; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 31); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 23)) << 8; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 31); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 21)) << 10; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 31); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 19)) << 12; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 31); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 17)) << 14; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 31); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 15)) << 16; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 31); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 13)) << 18; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 31); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 11)) << 20; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 31); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 9)) << 22; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 31); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 7)) << 24; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 31); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 5)) << 26; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 31); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 3)) << 28; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 31); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 1)) << 30; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 31); - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 31); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 30)) << 1; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 31); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 28)) << 3; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 31); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 26)) << 5; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 31); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 24)) << 7; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 31); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 22)) << 9; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 31); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 20)) << 11; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 31); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 18)) << 13; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 31); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 15); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 16)) << 15; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 31); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 17); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 14)) << 17; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 31); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 19); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 12)) << 19; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 31); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 21); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 10)) << 21; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 31); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 23); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 8)) << 23; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 31); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 25); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 6)) << 25; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 31); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 27); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 4)) << 27; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 31); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 29); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 2)) << 29; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 31); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 31); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<32>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 0)) << 32; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 32); - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<33>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 33); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 31); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 2)) << 31; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 33); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 29); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 4)) << 29; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 33); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 27); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 6)) << 27; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 33); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 25); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 8)) << 25; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 33); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 23); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 10)) << 23; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 33); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 21); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 12)) << 21; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 33); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 19); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 14)) << 19; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 33); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 17); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 16)) << 17; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 33); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 15); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 18)) << 15; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 33); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 20)) << 13; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 33); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 22)) << 11; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 33); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 24)) << 9; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 33); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 26)) << 7; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 33); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 28)) << 5; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 33); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 30)) << 3; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 33); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 32)) << 1; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 1)) << 32; - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 33); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 3)) << 30; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 33); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 5)) << 28; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 33); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 7)) << 26; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 33); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 9)) << 24; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 33); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 11)) << 22; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 33); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 13)) << 20; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 33); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 15)) << 18; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 33); - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 17)) << 16; - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 33); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 19)) << 14; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 33); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 21)) << 12; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 33); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 23)) << 10; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 33); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 25)) << 8; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 33); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 27)) << 6; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 33); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 29)) << 4; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 33); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 31)) << 2; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 33); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<34>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 34); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 4)) << 30; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 34); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 8)) << 26; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 34); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 12)) << 22; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 34); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 16)) << 18; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 34); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 20)) << 14; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 34); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 24)) << 10; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 34); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 28)) << 6; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 34); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 32)) << 2; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 2)) << 32; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 34); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 6)) << 28; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 34); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 10)) << 24; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 34); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 14)) << 20; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 34); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 18)) << 16; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 34); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 22)) << 12; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 34); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 26)) << 8; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 34); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 30)) << 4; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 0)) << 34; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 34); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 4)) << 30; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 34); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 8)) << 26; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 34); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 12)) << 22; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 34); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 16)) << 18; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 34); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 20)) << 14; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 34); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 24)) << 10; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 34); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 28)) << 6; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 34); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 32)) << 2; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 2)) << 32; - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 34); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 6)) << 28; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 34); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 10)) << 24; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 34); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 14)) << 20; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 34); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 18)) << 16; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 34); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 22)) << 12; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 34); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 26)) << 8; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 34); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 30)) << 4; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<35>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 35); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 29); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 6)) << 29; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 35); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 23); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 12)) << 23; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 35); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 17); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 18)) << 17; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 35); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 24)) << 11; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 35); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 30)) << 5; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 1)) << 34; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 35); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 7)) << 28; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 35); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 13)) << 22; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 35); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 19)) << 16; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 35); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 25)) << 10; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 35); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 31)) << 4; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 33); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 2)) << 33; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 35); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 27); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 8)) << 27; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 35); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 21); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 14)) << 21; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 35); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 15); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 20)) << 15; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 35); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 26)) << 9; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 35); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 32)) << 3; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 3)) << 32; - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 35); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 9)) << 26; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 35); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 15)) << 20; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 35); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 21)) << 14; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 35); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 27)) << 8; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 35); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 33)) << 2; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 31); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 4)) << 31; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 35); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 25); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 10)) << 25; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 35); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 19); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 16)) << 19; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 35); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 22)) << 13; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 35); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 28)) << 7; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 35); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 34)) << 1; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 5)) << 30; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 35); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 11)) << 24; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 35); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 17)) << 18; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 35); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 23)) << 12; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 35); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 29)) << 6; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 35); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<36>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 36); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 8)) << 28; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 36); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 16)) << 20; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 36); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 24)) << 12; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 36); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 32)) << 4; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 4)) << 32; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 36); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 12)) << 24; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 36); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 20)) << 16; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 36); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 28)) << 8; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 0)) << 36; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 36); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 8)) << 28; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 36); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 16)) << 20; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 36); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 24)) << 12; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 36); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 32)) << 4; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 4)) << 32; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 36); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 12)) << 24; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 36); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 20)) << 16; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 36); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 28)) << 8; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 0)) << 36; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 36); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 8)) << 28; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 36); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 16)) << 20; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 36); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 24)) << 12; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 36); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 32)) << 4; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 4)) << 32; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 36); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 12)) << 24; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 36); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 20)) << 16; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 36); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 28)) << 8; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 0)) << 36; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 36); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 8)) << 28; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 36); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 16)) << 20; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 36); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 24)) << 12; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 36); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 32)) << 4; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 4)) << 32; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 36); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 12)) << 24; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 36); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 20)) << 16; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 36); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 28)) << 8; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<37>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 37); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 27); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 10)) << 27; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 37); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 17); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 20)) << 17; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 37); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 30)) << 7; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 3)) << 34; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 37); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 13)) << 24; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 37); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 23)) << 14; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 37); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 33)) << 4; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 31); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 6)) << 31; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 37); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 21); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 16)) << 21; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 37); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 26)) << 11; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 37); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 36)) << 1; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 9)) << 28; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 37); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 19)) << 18; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 37); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 29)) << 8; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 35); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 2)) << 35; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 37); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 25); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 12)) << 25; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 37); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 15); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 22)) << 15; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 37); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 32)) << 5; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 5)) << 32; - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 37); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 15)) << 22; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 37); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 25)) << 12; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 37); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 35)) << 2; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 29); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 8)) << 29; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 37); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 19); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 18)) << 19; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 37); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 28)) << 9; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 1)) << 36; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 37); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 11)) << 26; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 37); - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 21)) << 16; - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 37); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 31)) << 6; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 33); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 4)) << 33; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 37); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 23); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 14)) << 23; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 37); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 24)) << 13; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 37); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 34)) << 3; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 7)) << 30; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 37); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 17)) << 20; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 37); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 27)) << 10; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 37); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<38>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 38); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 12)) << 26; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 38); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 24)) << 14; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 38); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 36)) << 2; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 10)) << 28; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 38); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 22)) << 16; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 38); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 34)) << 4; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 8)) << 30; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 38); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 20)) << 18; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 38); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 32)) << 6; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 6)) << 32; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 38); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 18)) << 20; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 38); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 30)) << 8; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 4)) << 34; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 38); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 16)) << 22; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 38); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 28)) << 10; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 2)) << 36; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 38); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 14)) << 24; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 38); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 26)) << 12; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 0)) << 38; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 38); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 12)) << 26; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 38); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 24)) << 14; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 38); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 36)) << 2; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 10)) << 28; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 38); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 22)) << 16; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 38); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 34)) << 4; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 8)) << 30; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 38); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 20)) << 18; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 38); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 32)) << 6; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 6)) << 32; - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 38); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 18)) << 20; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 38); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 30)) << 8; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 4)) << 34; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 38); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 16)) << 22; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 38); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 28)) << 10; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 2)) << 36; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 38); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 14)) << 24; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 38); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 26)) << 12; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<39>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 39); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 25); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 14)) << 25; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 39); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 28)) << 11; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 3)) << 36; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 39); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 17)) << 22; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 39); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 31)) << 8; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 33); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 6)) << 33; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 39); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 19); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 20)) << 19; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 39); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 34)) << 5; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 9)) << 30; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 39); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 23)) << 16; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 39); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 37)) << 2; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 27); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 12)) << 27; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 39); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 26)) << 13; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 1)) << 38; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 39); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 15)) << 24; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 39); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 29)) << 10; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 35); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 4)) << 35; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 39); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 21); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 18)) << 21; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 39); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 32)) << 7; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 7)) << 32; - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 39); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 21)) << 18; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 39); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 35)) << 4; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 29); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 10)) << 29; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 39); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 15); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 24)) << 15; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 39); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 38)) << 1; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 13)) << 26; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 39); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 27)) << 12; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 37); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 2)) << 37; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 39); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 23); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 16)) << 23; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 39); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 30)) << 9; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 5)) << 34; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 39); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 19)) << 20; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 39); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 33)) << 6; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 31); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 8)) << 31; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 39); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 17); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 22)) << 17; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 39); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 36)) << 3; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 11)) << 28; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 39); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 38]; - tmp |= (src & MASK(uint64_t, 25)) << 14; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 39); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<40>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 40); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 16)) << 24; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 40); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 32)) << 8; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 8)) << 32; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 40); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 24)) << 16; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 0)) << 40; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 40); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 16)) << 24; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 40); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 32)) << 8; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 8)) << 32; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 40); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 24)) << 16; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 0)) << 40; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 40); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 16)) << 24; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 40); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 32)) << 8; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 8)) << 32; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 40); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 24)) << 16; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 0)) << 40; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 40); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 16)) << 24; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 40); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 32)) << 8; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 8)) << 32; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 40); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 24)) << 16; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 0)) << 40; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 40); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 16)) << 24; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 40); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 32)) << 8; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 8)) << 32; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 40); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 24)) << 16; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 0)) << 40; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 40); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 16)) << 24; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 40); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 32)) << 8; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 8)) << 32; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 40); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 24)) << 16; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 0)) << 40; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 40); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 16)) << 24; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 40); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 32)) << 8; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 8)) << 32; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 40); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 24)) << 16; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 0)) << 40; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 40); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 16)) << 24; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 40); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 32)) << 8; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 38]; - tmp |= (src & MASK(uint64_t, 8)) << 32; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 40); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 39]; - tmp |= (src & MASK(uint64_t, 24)) << 16; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<41>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 41); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 23); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 18)) << 23; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 41); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 36)) << 5; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 13)) << 28; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 41); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 31)) << 10; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 33); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 8)) << 33; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 41); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 15); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 26)) << 15; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 3)) << 38; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 41); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 21)) << 20; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 41); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 39)) << 2; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 25); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 16)) << 25; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 41); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 34)) << 7; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 11)) << 30; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 41); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 29)) << 12; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 35); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 6)) << 35; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 41); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 17); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 24)) << 17; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 1)) << 40; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 41); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 19)) << 22; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 41); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 37)) << 4; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 27); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 14)) << 27; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 41); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 32)) << 9; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 9)) << 32; - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 41); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 27)) << 14; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 37); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 4)) << 37; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 41); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 19); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 22)) << 19; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 41); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 40)) << 1; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 17)) << 24; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 41); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 35)) << 6; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 29); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 12)) << 29; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 41); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 30)) << 11; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 7)) << 34; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 41); - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 25)) << 16; - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 39); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 2)) << 39; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 41); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 21); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 20)) << 21; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 41); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 38)) << 3; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 15)) << 26; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 41); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 33)) << 8; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 31); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 10)) << 31; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 41); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - src = in[lane + LANE_COUNT * 38]; - tmp |= (src & MASK(uint64_t, 28)) << 13; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 39]; - tmp |= (src & MASK(uint64_t, 5)) << 36; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 41); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 40]; - tmp |= (src & MASK(uint64_t, 23)) << 18; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 41); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<42>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 42); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 20)) << 22; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 42); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 40)) << 2; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 18)) << 24; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 42); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 38)) << 4; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 16)) << 26; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 42); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 36)) << 6; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 14)) << 28; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 42); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 34)) << 8; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 12)) << 30; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 42); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 32)) << 10; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 10)) << 32; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 42); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 30)) << 12; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 8)) << 34; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 42); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 28)) << 14; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 6)) << 36; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 42); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 26)) << 16; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 4)) << 38; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 42); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 24)) << 18; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 2)) << 40; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 42); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 22)) << 20; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 42); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 0)) << 42; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 42); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 20)) << 22; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 42); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 40)) << 2; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 18)) << 24; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 42); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 38)) << 4; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 16)) << 26; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 42); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 36)) << 6; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 14)) << 28; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 42); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 34)) << 8; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 12)) << 30; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 42); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 32)) << 10; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 10)) << 32; - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 42); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 30)) << 12; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 8)) << 34; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 42); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 28)) << 14; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 6)) << 36; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 42); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 26)) << 16; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 38]; - tmp |= (src & MASK(uint64_t, 4)) << 38; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 42); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 39]; - tmp |= (src & MASK(uint64_t, 24)) << 18; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 40]; - tmp |= (src & MASK(uint64_t, 2)) << 40; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 42); - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 41]; - tmp |= (src & MASK(uint64_t, 22)) << 20; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 42); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<43>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 43); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 21); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 22)) << 21; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 42); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 1)) << 42; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 43); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 23)) << 20; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 41); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 2)) << 41; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 43); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 19); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 24)) << 19; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 3)) << 40; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 43); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 25)) << 18; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 39); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 4)) << 39; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 43); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 17); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 26)) << 17; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 5)) << 38; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 43); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 27)) << 16; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 37); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 6)) << 37; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 43); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 15); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 28)) << 15; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 7)) << 36; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 43); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 29)) << 14; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 35); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 8)) << 35; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 43); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 30)) << 13; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 9)) << 34; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 43); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 31)) << 12; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 33); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 10)) << 33; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 43); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 32)) << 11; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 11)) << 32; - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 43); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 33)) << 10; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 31); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 12)) << 31; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 43); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 34)) << 9; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 13)) << 30; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 43); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 35)) << 8; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 29); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 14)) << 29; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 43); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 36)) << 7; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 15)) << 28; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 43); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 37)) << 6; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 27); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 16)) << 27; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 43); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 38)) << 5; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 17)) << 26; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 43); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 39)) << 4; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 25); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 18)) << 25; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 43); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 40)) << 3; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 38]; - tmp |= (src & MASK(uint64_t, 19)) << 24; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 43); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 39]; - tmp |= (src & MASK(uint64_t, 41)) << 2; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 23); - src = in[lane + LANE_COUNT * 40]; - tmp |= (src & MASK(uint64_t, 20)) << 23; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 43); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 41]; - tmp |= (src & MASK(uint64_t, 42)) << 1; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 42]; - tmp |= (src & MASK(uint64_t, 21)) << 22; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 43); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<44>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 44); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 24)) << 20; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 4)) << 40; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 44); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 28)) << 16; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 8)) << 36; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 44); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 32)) << 12; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 12)) << 32; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 44); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 36)) << 8; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 16)) << 28; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 44); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 40)) << 4; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 20)) << 24; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 0)) << 44; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 44); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 24)) << 20; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 4)) << 40; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 44); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 28)) << 16; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 8)) << 36; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 44); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 32)) << 12; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 12)) << 32; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 44); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 36)) << 8; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 16)) << 28; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 44); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 40)) << 4; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 20)) << 24; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 0)) << 44; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 44); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 24)) << 20; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 4)) << 40; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 44); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 28)) << 16; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 8)) << 36; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 44); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 32)) << 12; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 12)) << 32; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 44); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 36)) << 8; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 16)) << 28; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 44); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 40)) << 4; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 20)) << 24; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 0)) << 44; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 44); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 24)) << 20; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 4)) << 40; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 44); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 28)) << 16; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 8)) << 36; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 44); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 38]; - tmp |= (src & MASK(uint64_t, 32)) << 12; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 39]; - tmp |= (src & MASK(uint64_t, 12)) << 32; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 44); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 40]; - tmp |= (src & MASK(uint64_t, 36)) << 8; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 41]; - tmp |= (src & MASK(uint64_t, 16)) << 28; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 44); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 42]; - tmp |= (src & MASK(uint64_t, 40)) << 4; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 43]; - tmp |= (src & MASK(uint64_t, 20)) << 24; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<45>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 45); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 19); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 26)) << 19; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 7)) << 38; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 45); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 33)) << 12; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 31); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 14)) << 31; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 45); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 40)) << 5; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 21)) << 24; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 43); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 2)) << 43; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 45); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 17); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 28)) << 17; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 9)) << 36; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 45); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 35)) << 10; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 29); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 16)) << 29; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 45); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 42)) << 3; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 23)) << 22; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 41); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 4)) << 41; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 45); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 15); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 30)) << 15; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 11)) << 34; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 45); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 37)) << 8; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 27); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 18)) << 27; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 45); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 44)) << 1; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 25)) << 20; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 39); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 6)) << 39; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 45); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 32)) << 13; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 13)) << 32; - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 45); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 39)) << 6; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 25); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 20)) << 25; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 1)) << 44; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 45); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 27)) << 18; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 37); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 8)) << 37; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 45); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 34)) << 11; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 15)) << 30; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 45); - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 41)) << 4; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 23); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 22)) << 23; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 42); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 3)) << 42; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 45); - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 29)) << 16; - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 35); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 10)) << 35; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 45); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 36)) << 9; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 17)) << 28; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 45); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 38]; - tmp |= (src & MASK(uint64_t, 43)) << 2; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 21); - src = in[lane + LANE_COUNT * 39]; - tmp |= (src & MASK(uint64_t, 24)) << 21; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 40]; - tmp |= (src & MASK(uint64_t, 5)) << 40; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 45); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 41]; - tmp |= (src & MASK(uint64_t, 31)) << 14; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 33); - src = in[lane + LANE_COUNT * 42]; - tmp |= (src & MASK(uint64_t, 12)) << 33; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 45); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 43]; - tmp |= (src & MASK(uint64_t, 38)) << 7; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 44]; - tmp |= (src & MASK(uint64_t, 19)) << 26; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 45); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<46>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 46); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 28)) << 18; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 10)) << 36; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 46); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 38)) << 8; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 20)) << 26; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 2)) << 44; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 46); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 30)) << 16; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 12)) << 34; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 46); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 40)) << 6; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 22)) << 24; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 42); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 4)) << 42; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 46); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 32)) << 14; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 14)) << 32; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 46); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 42)) << 4; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 24)) << 22; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 6)) << 40; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 46); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 34)) << 12; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 16)) << 30; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 46); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 44)) << 2; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 26)) << 20; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 8)) << 38; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 46); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 36)) << 10; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 18)) << 28; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 46); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 0)) << 46; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 46); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 28)) << 18; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 10)) << 36; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 46); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 38)) << 8; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 20)) << 26; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 2)) << 44; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 46); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 30)) << 16; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 12)) << 34; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 46); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 40)) << 6; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 22)) << 24; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 42); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 4)) << 42; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 46); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 32)) << 14; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 14)) << 32; - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 46); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 42)) << 4; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 24)) << 22; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 38]; - tmp |= (src & MASK(uint64_t, 6)) << 40; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 46); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 39]; - tmp |= (src & MASK(uint64_t, 34)) << 12; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 40]; - tmp |= (src & MASK(uint64_t, 16)) << 30; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 46); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 41]; - tmp |= (src & MASK(uint64_t, 44)) << 2; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 42]; - tmp |= (src & MASK(uint64_t, 26)) << 20; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 43]; - tmp |= (src & MASK(uint64_t, 8)) << 38; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 46); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 44]; - tmp |= (src & MASK(uint64_t, 36)) << 10; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 45]; - tmp |= (src & MASK(uint64_t, 18)) << 28; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 46); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<47>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 47); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 17); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 30)) << 17; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 13)) << 34; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 47); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 43)) << 4; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 21); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 26)) << 21; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 9)) << 38; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 47); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 39)) << 8; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 25); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 22)) << 25; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 42); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 5)) << 42; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 47); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 35)) << 12; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 29); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 18)) << 29; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 46); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 1)) << 46; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 47); - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 31)) << 16; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 33); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 14)) << 33; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 47); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 44)) << 3; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 27)) << 20; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 37); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 10)) << 37; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 47); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 40)) << 7; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 23)) << 24; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 41); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 6)) << 41; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 47); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 36)) << 11; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 19)) << 28; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 45); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 2)) << 45; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 47); - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 15); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 32)) << 15; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 15)) << 32; - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 47); - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 45)) << 2; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 19); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 28)) << 19; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 11)) << 36; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 47); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 41)) << 6; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 23); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 24)) << 23; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 7)) << 40; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 47); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 37)) << 10; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 27); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 20)) << 27; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 3)) << 44; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 47); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 33)) << 14; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 31); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 16)) << 31; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 47); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 46)) << 1; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 29)) << 18; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 35); - src = in[lane + LANE_COUNT * 38]; - tmp |= (src & MASK(uint64_t, 12)) << 35; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 47); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 39]; - tmp |= (src & MASK(uint64_t, 42)) << 5; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 40]; - tmp |= (src & MASK(uint64_t, 25)) << 22; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 39); - src = in[lane + LANE_COUNT * 41]; - tmp |= (src & MASK(uint64_t, 8)) << 39; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 47); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 42]; - tmp |= (src & MASK(uint64_t, 38)) << 9; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 43]; - tmp |= (src & MASK(uint64_t, 21)) << 26; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 43); - src = in[lane + LANE_COUNT * 44]; - tmp |= (src & MASK(uint64_t, 4)) << 43; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 47); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - src = in[lane + LANE_COUNT * 45]; - tmp |= (src & MASK(uint64_t, 34)) << 13; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 46]; - tmp |= (src & MASK(uint64_t, 17)) << 30; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 47); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<48>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 48); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 32)) << 16; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 16)) << 32; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 0)) << 48; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 48); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 32)) << 16; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 16)) << 32; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 0)) << 48; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 48); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 32)) << 16; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 16)) << 32; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 0)) << 48; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 48); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 32)) << 16; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 16)) << 32; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 0)) << 48; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 48); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 32)) << 16; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 16)) << 32; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 0)) << 48; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 48); - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 32)) << 16; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 16)) << 32; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 0)) << 48; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 48); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 32)) << 16; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 16)) << 32; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 0)) << 48; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 48); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 32)) << 16; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 16)) << 32; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 0)) << 48; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 48); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 32)) << 16; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 16)) << 32; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 0)) << 48; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 48); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 32)) << 16; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 16)) << 32; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 0)) << 48; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 48); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 32)) << 16; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 16)) << 32; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 0)) << 48; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 48); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 32)) << 16; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 16)) << 32; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 0)) << 48; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 48); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 32)) << 16; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 38]; - tmp |= (src & MASK(uint64_t, 16)) << 32; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 39]; - tmp |= (src & MASK(uint64_t, 0)) << 48; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 48); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 40]; - tmp |= (src & MASK(uint64_t, 32)) << 16; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 41]; - tmp |= (src & MASK(uint64_t, 16)) << 32; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 42]; - tmp |= (src & MASK(uint64_t, 0)) << 48; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 48); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 43]; - tmp |= (src & MASK(uint64_t, 32)) << 16; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 44]; - tmp |= (src & MASK(uint64_t, 16)) << 32; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 45]; - tmp |= (src & MASK(uint64_t, 0)) << 48; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 48); - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 46]; - tmp |= (src & MASK(uint64_t, 32)) << 16; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 47]; - tmp |= (src & MASK(uint64_t, 16)) << 32; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<49>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 49); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 15); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 34)) << 15; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 19)) << 30; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 45); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 4)) << 45; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 49); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 38)) << 11; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 23)) << 26; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 41); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 8)) << 41; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 49); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 42)) << 7; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 27)) << 22; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 37); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 12)) << 37; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 49); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 46)) << 3; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 31)) << 18; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 33); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 16)) << 33; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 1)) << 48; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 49); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 35)) << 14; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 29); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 20)) << 29; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 5)) << 44; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 49); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 39)) << 10; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 25); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 24)) << 25; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 9)) << 40; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 49); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 43)) << 6; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 21); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 28)) << 21; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 13)) << 36; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 49); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 47)) << 2; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 17); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 32)) << 17; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 17)) << 32; - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 47); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 2)) << 47; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 49); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 36)) << 13; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 21)) << 28; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 43); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 6)) << 43; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 49); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 40)) << 9; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 25)) << 24; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 39); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 10)) << 39; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 49); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 44)) << 5; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 29)) << 20; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 35); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 14)) << 35; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 49); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 48)) << 1; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 33)) << 16; - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 31); - src = in[lane + LANE_COUNT * 38]; - tmp |= (src & MASK(uint64_t, 18)) << 31; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 46); - src = in[lane + LANE_COUNT * 39]; - tmp |= (src & MASK(uint64_t, 3)) << 46; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 49); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 40]; - tmp |= (src & MASK(uint64_t, 37)) << 12; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 27); - src = in[lane + LANE_COUNT * 41]; - tmp |= (src & MASK(uint64_t, 22)) << 27; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 42); - src = in[lane + LANE_COUNT * 42]; - tmp |= (src & MASK(uint64_t, 7)) << 42; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 49); - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 43]; - tmp |= (src & MASK(uint64_t, 41)) << 8; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 23); - src = in[lane + LANE_COUNT * 44]; - tmp |= (src & MASK(uint64_t, 26)) << 23; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 45]; - tmp |= (src & MASK(uint64_t, 11)) << 38; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 49); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 46]; - tmp |= (src & MASK(uint64_t, 45)) << 4; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 19); - src = in[lane + LANE_COUNT * 47]; - tmp |= (src & MASK(uint64_t, 30)) << 19; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 48]; - tmp |= (src & MASK(uint64_t, 15)) << 34; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 49); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<50>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 50); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 36)) << 14; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 22)) << 28; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 42); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 8)) << 42; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 50); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 44)) << 6; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 30)) << 20; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 16)) << 34; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 2)) << 48; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 50); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 38)) << 12; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 24)) << 26; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 10)) << 40; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 50); - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 46)) << 4; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 32)) << 18; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 18)) << 32; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 46); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 4)) << 46; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 50); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 40)) << 10; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 26)) << 24; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 12)) << 38; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 50); - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 48)) << 2; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 34)) << 16; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 20)) << 30; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 6)) << 44; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 50); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 42)) << 8; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 28)) << 22; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 14)) << 36; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 50); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 0)) << 50; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 50); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 36)) << 14; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 22)) << 28; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 42); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 8)) << 42; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 50); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 44)) << 6; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 30)) << 20; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 16)) << 34; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 2)) << 48; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 50); - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 38)) << 12; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 24)) << 26; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 10)) << 40; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 50); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 46)) << 4; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 32)) << 18; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 38]; - tmp |= (src & MASK(uint64_t, 18)) << 32; - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 46); - src = in[lane + LANE_COUNT * 39]; - tmp |= (src & MASK(uint64_t, 4)) << 46; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 50); - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 40]; - tmp |= (src & MASK(uint64_t, 40)) << 10; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 41]; - tmp |= (src & MASK(uint64_t, 26)) << 24; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 42]; - tmp |= (src & MASK(uint64_t, 12)) << 38; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 50); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 43]; - tmp |= (src & MASK(uint64_t, 48)) << 2; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 44]; - tmp |= (src & MASK(uint64_t, 34)) << 16; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 45]; - tmp |= (src & MASK(uint64_t, 20)) << 30; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 46]; - tmp |= (src & MASK(uint64_t, 6)) << 44; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 50); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 47]; - tmp |= (src & MASK(uint64_t, 42)) << 8; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 48]; - tmp |= (src & MASK(uint64_t, 28)) << 22; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 49]; - tmp |= (src & MASK(uint64_t, 14)) << 36; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 50); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<51>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 51); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 38)) << 13; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 25)) << 26; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 39); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 12)) << 39; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 51); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 50)) << 1; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 37)) << 14; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 27); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 24)) << 27; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 11)) << 40; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 51); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 49)) << 2; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 15); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 36)) << 15; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 23)) << 28; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 41); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 10)) << 41; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 51); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 48)) << 3; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 35)) << 16; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 29); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 22)) << 29; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 42); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 9)) << 42; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 51); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 47)) << 4; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 17); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 34)) << 17; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 21)) << 30; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 43); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 8)) << 43; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 51); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 46)) << 5; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 33)) << 18; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 31); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 20)) << 31; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 7)) << 44; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 51); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 45)) << 6; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 19); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 32)) << 19; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 19)) << 32; - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 45); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 6)) << 45; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 51); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 44)) << 7; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 31)) << 20; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 33); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 18)) << 33; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 46); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 5)) << 46; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 51); - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 43)) << 8; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 21); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 30)) << 21; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 17)) << 34; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 47); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 4)) << 47; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 51); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 42)) << 9; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 29)) << 22; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 35); - src = in[lane + LANE_COUNT * 38]; - tmp |= (src & MASK(uint64_t, 16)) << 35; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 39]; - tmp |= (src & MASK(uint64_t, 3)) << 48; - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 51); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 40]; - tmp |= (src & MASK(uint64_t, 41)) << 10; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 23); - src = in[lane + LANE_COUNT * 41]; - tmp |= (src & MASK(uint64_t, 28)) << 23; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 42]; - tmp |= (src & MASK(uint64_t, 15)) << 36; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 49); - src = in[lane + LANE_COUNT * 43]; - tmp |= (src & MASK(uint64_t, 2)) << 49; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 51); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 44]; - tmp |= (src & MASK(uint64_t, 40)) << 11; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 45]; - tmp |= (src & MASK(uint64_t, 27)) << 24; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 37); - src = in[lane + LANE_COUNT * 46]; - tmp |= (src & MASK(uint64_t, 14)) << 37; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 50); - src = in[lane + LANE_COUNT * 47]; - tmp |= (src & MASK(uint64_t, 1)) << 50; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 51); - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 48]; - tmp |= (src & MASK(uint64_t, 39)) << 12; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 25); - src = in[lane + LANE_COUNT * 49]; - tmp |= (src & MASK(uint64_t, 26)) << 25; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 50]; - tmp |= (src & MASK(uint64_t, 13)) << 38; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 51); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<52>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 52); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 40)) << 12; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 28)) << 24; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 16)) << 36; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 4)) << 48; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 52); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 44)) << 8; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 32)) << 20; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 20)) << 32; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 8)) << 44; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 52); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 48)) << 4; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 36)) << 16; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 24)) << 28; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 12)) << 40; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 52); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 0)) << 52; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 52); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 40)) << 12; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 28)) << 24; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 16)) << 36; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 4)) << 48; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 52); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 44)) << 8; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 32)) << 20; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 20)) << 32; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 8)) << 44; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 52); - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 48)) << 4; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 36)) << 16; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 24)) << 28; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 12)) << 40; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 52); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 0)) << 52; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 52); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 40)) << 12; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 28)) << 24; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 16)) << 36; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 4)) << 48; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 52); - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 44)) << 8; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 32)) << 20; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 20)) << 32; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 8)) << 44; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 52); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 48)) << 4; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 36)) << 16; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 24)) << 28; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 38]; - tmp |= (src & MASK(uint64_t, 12)) << 40; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 52); - src = in[lane + LANE_COUNT * 39]; - tmp |= (src & MASK(uint64_t, 0)) << 52; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 52); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 40]; - tmp |= (src & MASK(uint64_t, 40)) << 12; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 41]; - tmp |= (src & MASK(uint64_t, 28)) << 24; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 42]; - tmp |= (src & MASK(uint64_t, 16)) << 36; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 43]; - tmp |= (src & MASK(uint64_t, 4)) << 48; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 52); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 44]; - tmp |= (src & MASK(uint64_t, 44)) << 8; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 45]; - tmp |= (src & MASK(uint64_t, 32)) << 20; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 46]; - tmp |= (src & MASK(uint64_t, 20)) << 32; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 47]; - tmp |= (src & MASK(uint64_t, 8)) << 44; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 52); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 48]; - tmp |= (src & MASK(uint64_t, 48)) << 4; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 49]; - tmp |= (src & MASK(uint64_t, 36)) << 16; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 50]; - tmp |= (src & MASK(uint64_t, 24)) << 28; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 51]; - tmp |= (src & MASK(uint64_t, 12)) << 40; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 52); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<53>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 53); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 42)) << 11; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 31)) << 22; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 33); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 20)) << 33; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 9)) << 44; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 53); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 51)) << 2; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 40)) << 13; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 29)) << 24; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 35); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 18)) << 35; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 46); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 7)) << 46; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 53); - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 49)) << 4; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 15); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 38)) << 15; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 27)) << 26; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 37); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 16)) << 37; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 5)) << 48; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 53); - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 47)) << 6; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 17); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 36)) << 17; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 25)) << 28; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 39); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 14)) << 39; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 50); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 3)) << 50; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 53); - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 45)) << 8; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 19); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 34)) << 19; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 23)) << 30; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 41); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 12)) << 41; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 52); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 1)) << 52; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 53); - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 43)) << 10; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 21); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 32)) << 21; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 21)) << 32; - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 43); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 10)) << 43; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 53); - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 52)) << 1; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 41)) << 12; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 23); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 30)) << 23; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 19)) << 34; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 45); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 8)) << 45; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 53); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 50)) << 3; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 39)) << 14; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 25); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 28)) << 25; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 17)) << 36; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 47); - src = in[lane + LANE_COUNT * 38]; - tmp |= (src & MASK(uint64_t, 6)) << 47; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 53); - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 39]; - tmp |= (src & MASK(uint64_t, 48)) << 5; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 40]; - tmp |= (src & MASK(uint64_t, 37)) << 16; - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 27); - src = in[lane + LANE_COUNT * 41]; - tmp |= (src & MASK(uint64_t, 26)) << 27; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 42]; - tmp |= (src & MASK(uint64_t, 15)) << 38; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 49); - src = in[lane + LANE_COUNT * 43]; - tmp |= (src & MASK(uint64_t, 4)) << 49; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 53); - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 44]; - tmp |= (src & MASK(uint64_t, 46)) << 7; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 45]; - tmp |= (src & MASK(uint64_t, 35)) << 18; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 29); - src = in[lane + LANE_COUNT * 46]; - tmp |= (src & MASK(uint64_t, 24)) << 29; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 47]; - tmp |= (src & MASK(uint64_t, 13)) << 40; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 51); - src = in[lane + LANE_COUNT * 48]; - tmp |= (src & MASK(uint64_t, 2)) << 51; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 53); - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 49]; - tmp |= (src & MASK(uint64_t, 44)) << 9; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 50]; - tmp |= (src & MASK(uint64_t, 33)) << 20; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 31); - src = in[lane + LANE_COUNT * 51]; - tmp |= (src & MASK(uint64_t, 22)) << 31; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 42); - src = in[lane + LANE_COUNT * 52]; - tmp |= (src & MASK(uint64_t, 11)) << 42; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 53); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<54>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 54); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 44)) << 10; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 34)) << 20; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 24)) << 30; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 14)) << 40; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 50); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 4)) << 50; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 54); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 48)) << 6; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 38)) << 16; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 28)) << 26; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 18)) << 36; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 46); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 8)) << 46; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 54); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 52)) << 2; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 42)) << 12; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 32)) << 22; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 22)) << 32; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 42); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 12)) << 42; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 52); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 2)) << 52; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 54); - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 46)) << 8; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 36)) << 18; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 26)) << 28; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 16)) << 38; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 6)) << 48; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 54); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 50)) << 4; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 40)) << 14; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 30)) << 24; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 20)) << 34; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 10)) << 44; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 54); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 0)) << 54; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 54); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 44)) << 10; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 34)) << 20; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 24)) << 30; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 14)) << 40; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 50); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 4)) << 50; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 54); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 48)) << 6; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 38)) << 16; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 28)) << 26; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 18)) << 36; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 46); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 8)) << 46; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 54); - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 38]; - tmp |= (src & MASK(uint64_t, 52)) << 2; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 39]; - tmp |= (src & MASK(uint64_t, 42)) << 12; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 40]; - tmp |= (src & MASK(uint64_t, 32)) << 22; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 41]; - tmp |= (src & MASK(uint64_t, 22)) << 32; - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 42); - src = in[lane + LANE_COUNT * 42]; - tmp |= (src & MASK(uint64_t, 12)) << 42; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 52); - src = in[lane + LANE_COUNT * 43]; - tmp |= (src & MASK(uint64_t, 2)) << 52; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 54); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 44]; - tmp |= (src & MASK(uint64_t, 46)) << 8; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 45]; - tmp |= (src & MASK(uint64_t, 36)) << 18; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 46]; - tmp |= (src & MASK(uint64_t, 26)) << 28; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 47]; - tmp |= (src & MASK(uint64_t, 16)) << 38; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 48]; - tmp |= (src & MASK(uint64_t, 6)) << 48; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 54); - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 49]; - tmp |= (src & MASK(uint64_t, 50)) << 4; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 50]; - tmp |= (src & MASK(uint64_t, 40)) << 14; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 51]; - tmp |= (src & MASK(uint64_t, 30)) << 24; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 52]; - tmp |= (src & MASK(uint64_t, 20)) << 34; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 53]; - tmp |= (src & MASK(uint64_t, 10)) << 44; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 54); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<55>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 55); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 46)) << 9; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 37)) << 18; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 27); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 28)) << 27; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 19)) << 36; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 45); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 10)) << 45; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 54); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 1)) << 54; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 55); - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 47)) << 8; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 17); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 38)) << 17; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 29)) << 26; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 35); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 20)) << 35; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 11)) << 44; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 53); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 2)) << 53; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 55); - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 48)) << 7; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 39)) << 16; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 25); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 30)) << 25; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 21)) << 34; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 43); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 12)) << 43; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 52); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 3)) << 52; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 55); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 49)) << 6; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 15); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 40)) << 15; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 31)) << 24; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 33); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 22)) << 33; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 42); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 13)) << 42; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 51); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 4)) << 51; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 55); - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 50)) << 5; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 41)) << 14; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 23); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 32)) << 23; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 23)) << 32; - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 41); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 14)) << 41; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 50); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 5)) << 50; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 55); - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 51)) << 4; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 42)) << 13; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 33)) << 22; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 31); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 24)) << 31; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 15)) << 40; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 49); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 6)) << 49; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 55); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 52)) << 3; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 38]; - tmp |= (src & MASK(uint64_t, 43)) << 12; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 21); - src = in[lane + LANE_COUNT * 39]; - tmp |= (src & MASK(uint64_t, 34)) << 21; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 40]; - tmp |= (src & MASK(uint64_t, 25)) << 30; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 39); - src = in[lane + LANE_COUNT * 41]; - tmp |= (src & MASK(uint64_t, 16)) << 39; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 42]; - tmp |= (src & MASK(uint64_t, 7)) << 48; - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 55); - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 43]; - tmp |= (src & MASK(uint64_t, 53)) << 2; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 44]; - tmp |= (src & MASK(uint64_t, 44)) << 11; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 45]; - tmp |= (src & MASK(uint64_t, 35)) << 20; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 29); - src = in[lane + LANE_COUNT * 46]; - tmp |= (src & MASK(uint64_t, 26)) << 29; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 47]; - tmp |= (src & MASK(uint64_t, 17)) << 38; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 47); - src = in[lane + LANE_COUNT * 48]; - tmp |= (src & MASK(uint64_t, 8)) << 47; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 55); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 49]; - tmp |= (src & MASK(uint64_t, 54)) << 1; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 50]; - tmp |= (src & MASK(uint64_t, 45)) << 10; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 19); - src = in[lane + LANE_COUNT * 51]; - tmp |= (src & MASK(uint64_t, 36)) << 19; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 52]; - tmp |= (src & MASK(uint64_t, 27)) << 28; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 37); - src = in[lane + LANE_COUNT * 53]; - tmp |= (src & MASK(uint64_t, 18)) << 37; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 46); - src = in[lane + LANE_COUNT * 54]; - tmp |= (src & MASK(uint64_t, 9)) << 46; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 55); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<56>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 56); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 48)) << 8; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 40)) << 16; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 32)) << 24; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 24)) << 32; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 16)) << 40; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 8)) << 48; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 56); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 0)) << 56; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 56); - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 48)) << 8; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 40)) << 16; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 32)) << 24; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 24)) << 32; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 16)) << 40; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 8)) << 48; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 56); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 0)) << 56; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 56); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 48)) << 8; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 40)) << 16; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 32)) << 24; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 24)) << 32; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 16)) << 40; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 8)) << 48; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 56); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 0)) << 56; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 56); - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 48)) << 8; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 40)) << 16; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 32)) << 24; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 24)) << 32; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 16)) << 40; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 8)) << 48; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 56); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 0)) << 56; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 56); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 48)) << 8; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 40)) << 16; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 32)) << 24; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 24)) << 32; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 16)) << 40; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 8)) << 48; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 56); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 0)) << 56; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 56); - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 48)) << 8; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 40)) << 16; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 38]; - tmp |= (src & MASK(uint64_t, 32)) << 24; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 39]; - tmp |= (src & MASK(uint64_t, 24)) << 32; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 40]; - tmp |= (src & MASK(uint64_t, 16)) << 40; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 41]; - tmp |= (src & MASK(uint64_t, 8)) << 48; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 56); - src = in[lane + LANE_COUNT * 42]; - tmp |= (src & MASK(uint64_t, 0)) << 56; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 56); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 43]; - tmp |= (src & MASK(uint64_t, 48)) << 8; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 44]; - tmp |= (src & MASK(uint64_t, 40)) << 16; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 45]; - tmp |= (src & MASK(uint64_t, 32)) << 24; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 46]; - tmp |= (src & MASK(uint64_t, 24)) << 32; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 47]; - tmp |= (src & MASK(uint64_t, 16)) << 40; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 48]; - tmp |= (src & MASK(uint64_t, 8)) << 48; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 56); - src = in[lane + LANE_COUNT * 49]; - tmp |= (src & MASK(uint64_t, 0)) << 56; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 56); - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 50]; - tmp |= (src & MASK(uint64_t, 48)) << 8; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 51]; - tmp |= (src & MASK(uint64_t, 40)) << 16; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 52]; - tmp |= (src & MASK(uint64_t, 32)) << 24; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 53]; - tmp |= (src & MASK(uint64_t, 24)) << 32; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 54]; - tmp |= (src & MASK(uint64_t, 16)) << 40; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 55]; - tmp |= (src & MASK(uint64_t, 8)) << 48; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 56); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<57>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 57); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 50)) << 7; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 43)) << 14; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 21); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 36)) << 21; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 29)) << 28; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 35); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 22)) << 35; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 42); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 15)) << 42; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 49); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 8)) << 49; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 56); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 1)) << 56; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 57); - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 51)) << 6; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 44)) << 13; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 37)) << 20; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 27); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 30)) << 27; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 23)) << 34; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 41); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 16)) << 41; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 9)) << 48; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 55); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 2)) << 55; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 57); - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 52)) << 5; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 45)) << 12; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 19); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 38)) << 19; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 31)) << 26; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 33); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 24)) << 33; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 17)) << 40; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 47); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 10)) << 47; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 54); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 3)) << 54; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 57); - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 53)) << 4; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 46)) << 11; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 39)) << 18; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 25); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 32)) << 25; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 25)) << 32; - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 39); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 18)) << 39; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 46); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 11)) << 46; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 53); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 4)) << 53; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 57); - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 54)) << 3; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 47)) << 10; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 17); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 40)) << 17; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 33)) << 24; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 31); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 26)) << 31; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 38]; - tmp |= (src & MASK(uint64_t, 19)) << 38; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 45); - src = in[lane + LANE_COUNT * 39]; - tmp |= (src & MASK(uint64_t, 12)) << 45; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 52); - src = in[lane + LANE_COUNT * 40]; - tmp |= (src & MASK(uint64_t, 5)) << 52; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 57); - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 41]; - tmp |= (src & MASK(uint64_t, 55)) << 2; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 42]; - tmp |= (src & MASK(uint64_t, 48)) << 9; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 43]; - tmp |= (src & MASK(uint64_t, 41)) << 16; - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 23); - src = in[lane + LANE_COUNT * 44]; - tmp |= (src & MASK(uint64_t, 34)) << 23; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 45]; - tmp |= (src & MASK(uint64_t, 27)) << 30; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 37); - src = in[lane + LANE_COUNT * 46]; - tmp |= (src & MASK(uint64_t, 20)) << 37; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 47]; - tmp |= (src & MASK(uint64_t, 13)) << 44; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 51); - src = in[lane + LANE_COUNT * 48]; - tmp |= (src & MASK(uint64_t, 6)) << 51; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 57); - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 49]; - tmp |= (src & MASK(uint64_t, 56)) << 1; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 50]; - tmp |= (src & MASK(uint64_t, 49)) << 8; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 15); - src = in[lane + LANE_COUNT * 51]; - tmp |= (src & MASK(uint64_t, 42)) << 15; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 52]; - tmp |= (src & MASK(uint64_t, 35)) << 22; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 29); - src = in[lane + LANE_COUNT * 53]; - tmp |= (src & MASK(uint64_t, 28)) << 29; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 54]; - tmp |= (src & MASK(uint64_t, 21)) << 36; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 43); - src = in[lane + LANE_COUNT * 55]; - tmp |= (src & MASK(uint64_t, 14)) << 43; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 50); - src = in[lane + LANE_COUNT * 56]; - tmp |= (src & MASK(uint64_t, 7)) << 50; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 57); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<58>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 58); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 52)) << 6; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 46)) << 12; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 40)) << 18; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 34)) << 24; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 28)) << 30; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 22)) << 36; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 42); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 16)) << 42; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 10)) << 48; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 54); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 4)) << 54; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 58); - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 56)) << 2; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 50)) << 8; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 44)) << 14; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 38)) << 20; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 32)) << 26; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 26)) << 32; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 20)) << 38; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 14)) << 44; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 50); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 8)) << 50; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 56); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 2)) << 56; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 58); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 54)) << 4; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 48)) << 10; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 42)) << 16; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 36)) << 22; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 30)) << 28; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 24)) << 34; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 18)) << 40; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 46); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 12)) << 46; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 52); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 6)) << 52; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 58); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 0)) << 58; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 58); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 52)) << 6; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 46)) << 12; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 40)) << 18; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 34)) << 24; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 28)) << 30; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 22)) << 36; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 42); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 16)) << 42; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 10)) << 48; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 54); - src = in[lane + LANE_COUNT * 38]; - tmp |= (src & MASK(uint64_t, 4)) << 54; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 58); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 39]; - tmp |= (src & MASK(uint64_t, 56)) << 2; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 40]; - tmp |= (src & MASK(uint64_t, 50)) << 8; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 41]; - tmp |= (src & MASK(uint64_t, 44)) << 14; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 42]; - tmp |= (src & MASK(uint64_t, 38)) << 20; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 43]; - tmp |= (src & MASK(uint64_t, 32)) << 26; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 44]; - tmp |= (src & MASK(uint64_t, 26)) << 32; - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 45]; - tmp |= (src & MASK(uint64_t, 20)) << 38; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 46]; - tmp |= (src & MASK(uint64_t, 14)) << 44; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 50); - src = in[lane + LANE_COUNT * 47]; - tmp |= (src & MASK(uint64_t, 8)) << 50; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 56); - src = in[lane + LANE_COUNT * 48]; - tmp |= (src & MASK(uint64_t, 2)) << 56; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 58); - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 49]; - tmp |= (src & MASK(uint64_t, 54)) << 4; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 50]; - tmp |= (src & MASK(uint64_t, 48)) << 10; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 51]; - tmp |= (src & MASK(uint64_t, 42)) << 16; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 52]; - tmp |= (src & MASK(uint64_t, 36)) << 22; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 53]; - tmp |= (src & MASK(uint64_t, 30)) << 28; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 54]; - tmp |= (src & MASK(uint64_t, 24)) << 34; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 55]; - tmp |= (src & MASK(uint64_t, 18)) << 40; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 46); - src = in[lane + LANE_COUNT * 56]; - tmp |= (src & MASK(uint64_t, 12)) << 46; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 52); - src = in[lane + LANE_COUNT * 57]; - tmp |= (src & MASK(uint64_t, 6)) << 52; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 58); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<59>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 59); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 54)) << 5; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 49)) << 10; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 15); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 44)) << 15; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 39)) << 20; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 25); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 34)) << 25; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 29)) << 30; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 35); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 24)) << 35; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 19)) << 40; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 45); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 14)) << 45; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 50); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 9)) << 50; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 55); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 4)) << 55; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 59); - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 58)) << 1; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 53)) << 6; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 48)) << 11; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 43)) << 16; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 21); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 38)) << 21; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 33)) << 26; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 31); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 28)) << 31; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 23)) << 36; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 41); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 18)) << 41; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 46); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 13)) << 46; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 51); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 8)) << 51; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 56); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 3)) << 56; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 59); - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 57)) << 2; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 52)) << 7; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 47)) << 12; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 17); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 42)) << 17; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 37)) << 22; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 27); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 32)) << 27; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 27)) << 32; - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 37); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 22)) << 37; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 42); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 17)) << 42; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 47); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 12)) << 47; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 52); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 7)) << 52; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 57); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 2)) << 57; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 59); - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 56)) << 3; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 51)) << 8; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - src = in[lane + LANE_COUNT * 38]; - tmp |= (src & MASK(uint64_t, 46)) << 13; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 39]; - tmp |= (src & MASK(uint64_t, 41)) << 18; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 23); - src = in[lane + LANE_COUNT * 40]; - tmp |= (src & MASK(uint64_t, 36)) << 23; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 41]; - tmp |= (src & MASK(uint64_t, 31)) << 28; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 33); - src = in[lane + LANE_COUNT * 42]; - tmp |= (src & MASK(uint64_t, 26)) << 33; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 43]; - tmp |= (src & MASK(uint64_t, 21)) << 38; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 43); - src = in[lane + LANE_COUNT * 44]; - tmp |= (src & MASK(uint64_t, 16)) << 43; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 45]; - tmp |= (src & MASK(uint64_t, 11)) << 48; - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 53); - src = in[lane + LANE_COUNT * 46]; - tmp |= (src & MASK(uint64_t, 6)) << 53; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 58); - src = in[lane + LANE_COUNT * 47]; - tmp |= (src & MASK(uint64_t, 1)) << 58; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 59); - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 48]; - tmp |= (src & MASK(uint64_t, 55)) << 4; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 49]; - tmp |= (src & MASK(uint64_t, 50)) << 9; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 50]; - tmp |= (src & MASK(uint64_t, 45)) << 14; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 19); - src = in[lane + LANE_COUNT * 51]; - tmp |= (src & MASK(uint64_t, 40)) << 19; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 52]; - tmp |= (src & MASK(uint64_t, 35)) << 24; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 29); - src = in[lane + LANE_COUNT * 53]; - tmp |= (src & MASK(uint64_t, 30)) << 29; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 54]; - tmp |= (src & MASK(uint64_t, 25)) << 34; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 39); - src = in[lane + LANE_COUNT * 55]; - tmp |= (src & MASK(uint64_t, 20)) << 39; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 56]; - tmp |= (src & MASK(uint64_t, 15)) << 44; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 49); - src = in[lane + LANE_COUNT * 57]; - tmp |= (src & MASK(uint64_t, 10)) << 49; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 54); - src = in[lane + LANE_COUNT * 58]; - tmp |= (src & MASK(uint64_t, 5)) << 54; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 59); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<60>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 60); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 56)) << 4; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 52)) << 8; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 48)) << 12; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 44)) << 16; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 40)) << 20; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 36)) << 24; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 32)) << 28; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 28)) << 32; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 24)) << 36; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 20)) << 40; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 16)) << 44; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 12)) << 48; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 52); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 8)) << 52; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 56); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 4)) << 56; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 60); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 0)) << 60; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 60); - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 56)) << 4; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 52)) << 8; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 48)) << 12; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 44)) << 16; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 40)) << 20; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 36)) << 24; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 32)) << 28; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 28)) << 32; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 24)) << 36; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 20)) << 40; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 16)) << 44; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 12)) << 48; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 52); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 8)) << 52; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 56); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 4)) << 56; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 60); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 0)) << 60; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 60); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 56)) << 4; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 52)) << 8; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 48)) << 12; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 44)) << 16; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 40)) << 20; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 36)) << 24; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 32)) << 28; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 38]; - tmp |= (src & MASK(uint64_t, 28)) << 32; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 39]; - tmp |= (src & MASK(uint64_t, 24)) << 36; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 40]; - tmp |= (src & MASK(uint64_t, 20)) << 40; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 41]; - tmp |= (src & MASK(uint64_t, 16)) << 44; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 42]; - tmp |= (src & MASK(uint64_t, 12)) << 48; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 52); - src = in[lane + LANE_COUNT * 43]; - tmp |= (src & MASK(uint64_t, 8)) << 52; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 56); - src = in[lane + LANE_COUNT * 44]; - tmp |= (src & MASK(uint64_t, 4)) << 56; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 60); - src = in[lane + LANE_COUNT * 45]; - tmp |= (src & MASK(uint64_t, 0)) << 60; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 60); - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 46]; - tmp |= (src & MASK(uint64_t, 56)) << 4; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 47]; - tmp |= (src & MASK(uint64_t, 52)) << 8; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 48]; - tmp |= (src & MASK(uint64_t, 48)) << 12; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 49]; - tmp |= (src & MASK(uint64_t, 44)) << 16; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 50]; - tmp |= (src & MASK(uint64_t, 40)) << 20; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 51]; - tmp |= (src & MASK(uint64_t, 36)) << 24; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 52]; - tmp |= (src & MASK(uint64_t, 32)) << 28; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 53]; - tmp |= (src & MASK(uint64_t, 28)) << 32; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 54]; - tmp |= (src & MASK(uint64_t, 24)) << 36; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 55]; - tmp |= (src & MASK(uint64_t, 20)) << 40; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 56]; - tmp |= (src & MASK(uint64_t, 16)) << 44; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 57]; - tmp |= (src & MASK(uint64_t, 12)) << 48; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 52); - src = in[lane + LANE_COUNT * 58]; - tmp |= (src & MASK(uint64_t, 8)) << 52; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 56); - src = in[lane + LANE_COUNT * 59]; - tmp |= (src & MASK(uint64_t, 4)) << 56; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 60); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<61>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 61); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 58)) << 3; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 55)) << 6; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 52)) << 9; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 49)) << 12; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 15); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 46)) << 15; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 43)) << 18; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 21); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 40)) << 21; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 37)) << 24; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 27); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 34)) << 27; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 31)) << 30; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 33); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 28)) << 33; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 25)) << 36; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 39); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 22)) << 39; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 42); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 19)) << 42; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 45); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 16)) << 45; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 13)) << 48; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 51); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 10)) << 51; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 54); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 7)) << 54; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 57); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 4)) << 57; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 60); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 1)) << 60; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 61); - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 59)) << 2; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 56)) << 5; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 53)) << 8; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 50)) << 11; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 47)) << 14; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 17); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 44)) << 17; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 41)) << 20; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 23); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 38)) << 23; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 35)) << 26; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 29); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 32)) << 29; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 29)) << 32; - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 35); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 26)) << 35; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 23)) << 38; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 41); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 20)) << 41; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 17)) << 44; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 47); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 14)) << 47; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 50); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 11)) << 50; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 53); - src = in[lane + LANE_COUNT * 38]; - tmp |= (src & MASK(uint64_t, 8)) << 53; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 56); - src = in[lane + LANE_COUNT * 39]; - tmp |= (src & MASK(uint64_t, 5)) << 56; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 59); - src = in[lane + LANE_COUNT * 40]; - tmp |= (src & MASK(uint64_t, 2)) << 59; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 61); - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 41]; - tmp |= (src & MASK(uint64_t, 60)) << 1; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 42]; - tmp |= (src & MASK(uint64_t, 57)) << 4; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 43]; - tmp |= (src & MASK(uint64_t, 54)) << 7; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 44]; - tmp |= (src & MASK(uint64_t, 51)) << 10; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - src = in[lane + LANE_COUNT * 45]; - tmp |= (src & MASK(uint64_t, 48)) << 13; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 46]; - tmp |= (src & MASK(uint64_t, 45)) << 16; - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 19); - src = in[lane + LANE_COUNT * 47]; - tmp |= (src & MASK(uint64_t, 42)) << 19; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 48]; - tmp |= (src & MASK(uint64_t, 39)) << 22; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 25); - src = in[lane + LANE_COUNT * 49]; - tmp |= (src & MASK(uint64_t, 36)) << 25; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 50]; - tmp |= (src & MASK(uint64_t, 33)) << 28; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 31); - src = in[lane + LANE_COUNT * 51]; - tmp |= (src & MASK(uint64_t, 30)) << 31; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 52]; - tmp |= (src & MASK(uint64_t, 27)) << 34; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 37); - src = in[lane + LANE_COUNT * 53]; - tmp |= (src & MASK(uint64_t, 24)) << 37; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 54]; - tmp |= (src & MASK(uint64_t, 21)) << 40; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 43); - src = in[lane + LANE_COUNT * 55]; - tmp |= (src & MASK(uint64_t, 18)) << 43; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 46); - src = in[lane + LANE_COUNT * 56]; - tmp |= (src & MASK(uint64_t, 15)) << 46; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 49); - src = in[lane + LANE_COUNT * 57]; - tmp |= (src & MASK(uint64_t, 12)) << 49; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 52); - src = in[lane + LANE_COUNT * 58]; - tmp |= (src & MASK(uint64_t, 9)) << 52; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 55); - src = in[lane + LANE_COUNT * 59]; - tmp |= (src & MASK(uint64_t, 6)) << 55; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 58); - src = in[lane + LANE_COUNT * 60]; - tmp |= (src & MASK(uint64_t, 3)) << 58; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 61); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<62>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 62); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 60)) << 2; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 58)) << 4; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 56)) << 6; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 54)) << 8; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 52)) << 10; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 50)) << 12; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 48)) << 14; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 46)) << 16; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 44)) << 18; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 42)) << 20; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 40)) << 22; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 38)) << 24; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 36)) << 26; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 34)) << 28; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 32)) << 30; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 30)) << 32; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 28)) << 34; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 26)) << 36; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 24)) << 38; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 22)) << 40; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 42); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 20)) << 42; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 18)) << 44; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 46); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 16)) << 46; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 14)) << 48; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 50); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 12)) << 50; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 52); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 10)) << 52; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 54); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 8)) << 54; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 56); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 6)) << 56; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 58); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 4)) << 58; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 60); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 2)) << 60; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 62); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 0)) << 62; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint64_t, 62); - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 60)) << 2; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 58)) << 4; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 56)) << 6; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 54)) << 8; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 52)) << 10; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 50)) << 12; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 38]; - tmp |= (src & MASK(uint64_t, 48)) << 14; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 39]; - tmp |= (src & MASK(uint64_t, 46)) << 16; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 40]; - tmp |= (src & MASK(uint64_t, 44)) << 18; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 41]; - tmp |= (src & MASK(uint64_t, 42)) << 20; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 42]; - tmp |= (src & MASK(uint64_t, 40)) << 22; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 43]; - tmp |= (src & MASK(uint64_t, 38)) << 24; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 44]; - tmp |= (src & MASK(uint64_t, 36)) << 26; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 45]; - tmp |= (src & MASK(uint64_t, 34)) << 28; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 46]; - tmp |= (src & MASK(uint64_t, 32)) << 30; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 47]; - tmp |= (src & MASK(uint64_t, 30)) << 32; - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 48]; - tmp |= (src & MASK(uint64_t, 28)) << 34; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 49]; - tmp |= (src & MASK(uint64_t, 26)) << 36; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 50]; - tmp |= (src & MASK(uint64_t, 24)) << 38; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 51]; - tmp |= (src & MASK(uint64_t, 22)) << 40; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 42); - src = in[lane + LANE_COUNT * 52]; - tmp |= (src & MASK(uint64_t, 20)) << 42; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 53]; - tmp |= (src & MASK(uint64_t, 18)) << 44; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 46); - src = in[lane + LANE_COUNT * 54]; - tmp |= (src & MASK(uint64_t, 16)) << 46; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 55]; - tmp |= (src & MASK(uint64_t, 14)) << 48; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 50); - src = in[lane + LANE_COUNT * 56]; - tmp |= (src & MASK(uint64_t, 12)) << 50; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 52); - src = in[lane + LANE_COUNT * 57]; - tmp |= (src & MASK(uint64_t, 10)) << 52; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 54); - src = in[lane + LANE_COUNT * 58]; - tmp |= (src & MASK(uint64_t, 8)) << 54; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 56); - src = in[lane + LANE_COUNT * 59]; - tmp |= (src & MASK(uint64_t, 6)) << 56; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 58); - src = in[lane + LANE_COUNT * 60]; - tmp |= (src & MASK(uint64_t, 4)) << 58; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 60); - src = in[lane + LANE_COUNT * 61]; - tmp |= (src & MASK(uint64_t, 2)) << 60; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 62); - out[INDEX(63, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_64_lane<63>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; - uint64_t src; - uint64_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint64_t, 63); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 63) & MASK(uint64_t, 1); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint64_t, 62)) << 1; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 62) & MASK(uint64_t, 2); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint64_t, 61)) << 2; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 61) & MASK(uint64_t, 3); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint64_t, 60)) << 3; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 60) & MASK(uint64_t, 4); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint64_t, 59)) << 4; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 59) & MASK(uint64_t, 5); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint64_t, 58)) << 5; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 58) & MASK(uint64_t, 6); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint64_t, 57)) << 6; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 57) & MASK(uint64_t, 7); - src = in[lane + LANE_COUNT * 7]; - tmp |= (src & MASK(uint64_t, 56)) << 7; - out[INDEX(7, lane)] = tmp + reference; - tmp = (src >> 56) & MASK(uint64_t, 8); - src = in[lane + LANE_COUNT * 8]; - tmp |= (src & MASK(uint64_t, 55)) << 8; - out[INDEX(8, lane)] = tmp + reference; - tmp = (src >> 55) & MASK(uint64_t, 9); - src = in[lane + LANE_COUNT * 9]; - tmp |= (src & MASK(uint64_t, 54)) << 9; - out[INDEX(9, lane)] = tmp + reference; - tmp = (src >> 54) & MASK(uint64_t, 10); - src = in[lane + LANE_COUNT * 10]; - tmp |= (src & MASK(uint64_t, 53)) << 10; - out[INDEX(10, lane)] = tmp + reference; - tmp = (src >> 53) & MASK(uint64_t, 11); - src = in[lane + LANE_COUNT * 11]; - tmp |= (src & MASK(uint64_t, 52)) << 11; - out[INDEX(11, lane)] = tmp + reference; - tmp = (src >> 52) & MASK(uint64_t, 12); - src = in[lane + LANE_COUNT * 12]; - tmp |= (src & MASK(uint64_t, 51)) << 12; - out[INDEX(12, lane)] = tmp + reference; - tmp = (src >> 51) & MASK(uint64_t, 13); - src = in[lane + LANE_COUNT * 13]; - tmp |= (src & MASK(uint64_t, 50)) << 13; - out[INDEX(13, lane)] = tmp + reference; - tmp = (src >> 50) & MASK(uint64_t, 14); - src = in[lane + LANE_COUNT * 14]; - tmp |= (src & MASK(uint64_t, 49)) << 14; - out[INDEX(14, lane)] = tmp + reference; - tmp = (src >> 49) & MASK(uint64_t, 15); - src = in[lane + LANE_COUNT * 15]; - tmp |= (src & MASK(uint64_t, 48)) << 15; - out[INDEX(15, lane)] = tmp + reference; - tmp = (src >> 48) & MASK(uint64_t, 16); - src = in[lane + LANE_COUNT * 16]; - tmp |= (src & MASK(uint64_t, 47)) << 16; - out[INDEX(16, lane)] = tmp + reference; - tmp = (src >> 47) & MASK(uint64_t, 17); - src = in[lane + LANE_COUNT * 17]; - tmp |= (src & MASK(uint64_t, 46)) << 17; - out[INDEX(17, lane)] = tmp + reference; - tmp = (src >> 46) & MASK(uint64_t, 18); - src = in[lane + LANE_COUNT * 18]; - tmp |= (src & MASK(uint64_t, 45)) << 18; - out[INDEX(18, lane)] = tmp + reference; - tmp = (src >> 45) & MASK(uint64_t, 19); - src = in[lane + LANE_COUNT * 19]; - tmp |= (src & MASK(uint64_t, 44)) << 19; - out[INDEX(19, lane)] = tmp + reference; - tmp = (src >> 44) & MASK(uint64_t, 20); - src = in[lane + LANE_COUNT * 20]; - tmp |= (src & MASK(uint64_t, 43)) << 20; - out[INDEX(20, lane)] = tmp + reference; - tmp = (src >> 43) & MASK(uint64_t, 21); - src = in[lane + LANE_COUNT * 21]; - tmp |= (src & MASK(uint64_t, 42)) << 21; - out[INDEX(21, lane)] = tmp + reference; - tmp = (src >> 42) & MASK(uint64_t, 22); - src = in[lane + LANE_COUNT * 22]; - tmp |= (src & MASK(uint64_t, 41)) << 22; - out[INDEX(22, lane)] = tmp + reference; - tmp = (src >> 41) & MASK(uint64_t, 23); - src = in[lane + LANE_COUNT * 23]; - tmp |= (src & MASK(uint64_t, 40)) << 23; - out[INDEX(23, lane)] = tmp + reference; - tmp = (src >> 40) & MASK(uint64_t, 24); - src = in[lane + LANE_COUNT * 24]; - tmp |= (src & MASK(uint64_t, 39)) << 24; - out[INDEX(24, lane)] = tmp + reference; - tmp = (src >> 39) & MASK(uint64_t, 25); - src = in[lane + LANE_COUNT * 25]; - tmp |= (src & MASK(uint64_t, 38)) << 25; - out[INDEX(25, lane)] = tmp + reference; - tmp = (src >> 38) & MASK(uint64_t, 26); - src = in[lane + LANE_COUNT * 26]; - tmp |= (src & MASK(uint64_t, 37)) << 26; - out[INDEX(26, lane)] = tmp + reference; - tmp = (src >> 37) & MASK(uint64_t, 27); - src = in[lane + LANE_COUNT * 27]; - tmp |= (src & MASK(uint64_t, 36)) << 27; - out[INDEX(27, lane)] = tmp + reference; - tmp = (src >> 36) & MASK(uint64_t, 28); - src = in[lane + LANE_COUNT * 28]; - tmp |= (src & MASK(uint64_t, 35)) << 28; - out[INDEX(28, lane)] = tmp + reference; - tmp = (src >> 35) & MASK(uint64_t, 29); - src = in[lane + LANE_COUNT * 29]; - tmp |= (src & MASK(uint64_t, 34)) << 29; - out[INDEX(29, lane)] = tmp + reference; - tmp = (src >> 34) & MASK(uint64_t, 30); - src = in[lane + LANE_COUNT * 30]; - tmp |= (src & MASK(uint64_t, 33)) << 30; - out[INDEX(30, lane)] = tmp + reference; - tmp = (src >> 33) & MASK(uint64_t, 31); - src = in[lane + LANE_COUNT * 31]; - tmp |= (src & MASK(uint64_t, 32)) << 31; - out[INDEX(31, lane)] = tmp + reference; - tmp = (src >> 32) & MASK(uint64_t, 32); - src = in[lane + LANE_COUNT * 32]; - tmp |= (src & MASK(uint64_t, 31)) << 32; - out[INDEX(32, lane)] = tmp + reference; - tmp = (src >> 31) & MASK(uint64_t, 33); - src = in[lane + LANE_COUNT * 33]; - tmp |= (src & MASK(uint64_t, 30)) << 33; - out[INDEX(33, lane)] = tmp + reference; - tmp = (src >> 30) & MASK(uint64_t, 34); - src = in[lane + LANE_COUNT * 34]; - tmp |= (src & MASK(uint64_t, 29)) << 34; - out[INDEX(34, lane)] = tmp + reference; - tmp = (src >> 29) & MASK(uint64_t, 35); - src = in[lane + LANE_COUNT * 35]; - tmp |= (src & MASK(uint64_t, 28)) << 35; - out[INDEX(35, lane)] = tmp + reference; - tmp = (src >> 28) & MASK(uint64_t, 36); - src = in[lane + LANE_COUNT * 36]; - tmp |= (src & MASK(uint64_t, 27)) << 36; - out[INDEX(36, lane)] = tmp + reference; - tmp = (src >> 27) & MASK(uint64_t, 37); - src = in[lane + LANE_COUNT * 37]; - tmp |= (src & MASK(uint64_t, 26)) << 37; - out[INDEX(37, lane)] = tmp + reference; - tmp = (src >> 26) & MASK(uint64_t, 38); - src = in[lane + LANE_COUNT * 38]; - tmp |= (src & MASK(uint64_t, 25)) << 38; - out[INDEX(38, lane)] = tmp + reference; - tmp = (src >> 25) & MASK(uint64_t, 39); - src = in[lane + LANE_COUNT * 39]; - tmp |= (src & MASK(uint64_t, 24)) << 39; - out[INDEX(39, lane)] = tmp + reference; - tmp = (src >> 24) & MASK(uint64_t, 40); - src = in[lane + LANE_COUNT * 40]; - tmp |= (src & MASK(uint64_t, 23)) << 40; - out[INDEX(40, lane)] = tmp + reference; - tmp = (src >> 23) & MASK(uint64_t, 41); - src = in[lane + LANE_COUNT * 41]; - tmp |= (src & MASK(uint64_t, 22)) << 41; - out[INDEX(41, lane)] = tmp + reference; - tmp = (src >> 22) & MASK(uint64_t, 42); - src = in[lane + LANE_COUNT * 42]; - tmp |= (src & MASK(uint64_t, 21)) << 42; - out[INDEX(42, lane)] = tmp + reference; - tmp = (src >> 21) & MASK(uint64_t, 43); - src = in[lane + LANE_COUNT * 43]; - tmp |= (src & MASK(uint64_t, 20)) << 43; - out[INDEX(43, lane)] = tmp + reference; - tmp = (src >> 20) & MASK(uint64_t, 44); - src = in[lane + LANE_COUNT * 44]; - tmp |= (src & MASK(uint64_t, 19)) << 44; - out[INDEX(44, lane)] = tmp + reference; - tmp = (src >> 19) & MASK(uint64_t, 45); - src = in[lane + LANE_COUNT * 45]; - tmp |= (src & MASK(uint64_t, 18)) << 45; - out[INDEX(45, lane)] = tmp + reference; - tmp = (src >> 18) & MASK(uint64_t, 46); - src = in[lane + LANE_COUNT * 46]; - tmp |= (src & MASK(uint64_t, 17)) << 46; - out[INDEX(46, lane)] = tmp + reference; - tmp = (src >> 17) & MASK(uint64_t, 47); - src = in[lane + LANE_COUNT * 47]; - tmp |= (src & MASK(uint64_t, 16)) << 47; - out[INDEX(47, lane)] = tmp + reference; - tmp = (src >> 16) & MASK(uint64_t, 48); - src = in[lane + LANE_COUNT * 48]; - tmp |= (src & MASK(uint64_t, 15)) << 48; - out[INDEX(48, lane)] = tmp + reference; - tmp = (src >> 15) & MASK(uint64_t, 49); - src = in[lane + LANE_COUNT * 49]; - tmp |= (src & MASK(uint64_t, 14)) << 49; - out[INDEX(49, lane)] = tmp + reference; - tmp = (src >> 14) & MASK(uint64_t, 50); - src = in[lane + LANE_COUNT * 50]; - tmp |= (src & MASK(uint64_t, 13)) << 50; - out[INDEX(50, lane)] = tmp + reference; - tmp = (src >> 13) & MASK(uint64_t, 51); - src = in[lane + LANE_COUNT * 51]; - tmp |= (src & MASK(uint64_t, 12)) << 51; - out[INDEX(51, lane)] = tmp + reference; - tmp = (src >> 12) & MASK(uint64_t, 52); - src = in[lane + LANE_COUNT * 52]; - tmp |= (src & MASK(uint64_t, 11)) << 52; - out[INDEX(52, lane)] = tmp + reference; - tmp = (src >> 11) & MASK(uint64_t, 53); - src = in[lane + LANE_COUNT * 53]; - tmp |= (src & MASK(uint64_t, 10)) << 53; - out[INDEX(53, lane)] = tmp + reference; - tmp = (src >> 10) & MASK(uint64_t, 54); - src = in[lane + LANE_COUNT * 54]; - tmp |= (src & MASK(uint64_t, 9)) << 54; - out[INDEX(54, lane)] = tmp + reference; - tmp = (src >> 9) & MASK(uint64_t, 55); - src = in[lane + LANE_COUNT * 55]; - tmp |= (src & MASK(uint64_t, 8)) << 55; - out[INDEX(55, lane)] = tmp + reference; - tmp = (src >> 8) & MASK(uint64_t, 56); - src = in[lane + LANE_COUNT * 56]; - tmp |= (src & MASK(uint64_t, 7)) << 56; - out[INDEX(56, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint64_t, 57); - src = in[lane + LANE_COUNT * 57]; - tmp |= (src & MASK(uint64_t, 6)) << 57; - out[INDEX(57, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint64_t, 58); - src = in[lane + LANE_COUNT * 58]; - tmp |= (src & MASK(uint64_t, 5)) << 58; - out[INDEX(58, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint64_t, 59); - src = in[lane + LANE_COUNT * 59]; - tmp |= (src & MASK(uint64_t, 4)) << 59; - out[INDEX(59, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint64_t, 60); - src = in[lane + LANE_COUNT * 60]; - tmp |= (src & MASK(uint64_t, 3)) << 60; - out[INDEX(60, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint64_t, 61); - src = in[lane + LANE_COUNT * 61]; - tmp |= (src & MASK(uint64_t, 2)) << 61; - out[INDEX(61, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint64_t, 62); - src = in[lane + LANE_COUNT * 62]; - tmp |= (src & MASK(uint64_t, 1)) << 62; - out[INDEX(62, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint64_t, 63); - out[INDEX(63, lane)] = tmp + reference; -} +__device__ void _bit_unpack_64_device(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, int thread_idx, GPUPatches& patches) { + __shared__ uint64_t shared_out[FL_CHUNK]; -template <> -__device__ void _bit_unpack_64_lane<64>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 16; + // Step 1: Unpack into shared memory #pragma unroll - for (int row = 0; row < 64; row++) { - out[INDEX(row, lane)] = in[LANE_COUNT * row + lane] + reference; + for (int i = 0; i < FL_LANES / 16; i++) { + _bit_unpack_64_lane(in, shared_out, reference, thread_idx * (FL_LANES / 16) + i); } -} + __syncwarp(); -/// Runtime dispatch to the optimized lane decoder for the given bit width. -__device__ inline void bit_unpack_64_lane( - const uint64_t *__restrict in, - uint64_t *__restrict out, - uint64_t reference, - unsigned int lane, - uint32_t bit_width -) { - switch (bit_width) { - case 0: _bit_unpack_64_lane<0>(in, out, reference, lane); break; - case 1: _bit_unpack_64_lane<1>(in, out, reference, lane); break; - case 2: _bit_unpack_64_lane<2>(in, out, reference, lane); break; - case 3: _bit_unpack_64_lane<3>(in, out, reference, lane); break; - case 4: _bit_unpack_64_lane<4>(in, out, reference, lane); break; - case 5: _bit_unpack_64_lane<5>(in, out, reference, lane); break; - case 6: _bit_unpack_64_lane<6>(in, out, reference, lane); break; - case 7: _bit_unpack_64_lane<7>(in, out, reference, lane); break; - case 8: _bit_unpack_64_lane<8>(in, out, reference, lane); break; - case 9: _bit_unpack_64_lane<9>(in, out, reference, lane); break; - case 10: _bit_unpack_64_lane<10>(in, out, reference, lane); break; - case 11: _bit_unpack_64_lane<11>(in, out, reference, lane); break; - case 12: _bit_unpack_64_lane<12>(in, out, reference, lane); break; - case 13: _bit_unpack_64_lane<13>(in, out, reference, lane); break; - case 14: _bit_unpack_64_lane<14>(in, out, reference, lane); break; - case 15: _bit_unpack_64_lane<15>(in, out, reference, lane); break; - case 16: _bit_unpack_64_lane<16>(in, out, reference, lane); break; - case 17: _bit_unpack_64_lane<17>(in, out, reference, lane); break; - case 18: _bit_unpack_64_lane<18>(in, out, reference, lane); break; - case 19: _bit_unpack_64_lane<19>(in, out, reference, lane); break; - case 20: _bit_unpack_64_lane<20>(in, out, reference, lane); break; - case 21: _bit_unpack_64_lane<21>(in, out, reference, lane); break; - case 22: _bit_unpack_64_lane<22>(in, out, reference, lane); break; - case 23: _bit_unpack_64_lane<23>(in, out, reference, lane); break; - case 24: _bit_unpack_64_lane<24>(in, out, reference, lane); break; - case 25: _bit_unpack_64_lane<25>(in, out, reference, lane); break; - case 26: _bit_unpack_64_lane<26>(in, out, reference, lane); break; - case 27: _bit_unpack_64_lane<27>(in, out, reference, lane); break; - case 28: _bit_unpack_64_lane<28>(in, out, reference, lane); break; - case 29: _bit_unpack_64_lane<29>(in, out, reference, lane); break; - case 30: _bit_unpack_64_lane<30>(in, out, reference, lane); break; - case 31: _bit_unpack_64_lane<31>(in, out, reference, lane); break; - case 32: _bit_unpack_64_lane<32>(in, out, reference, lane); break; - case 33: _bit_unpack_64_lane<33>(in, out, reference, lane); break; - case 34: _bit_unpack_64_lane<34>(in, out, reference, lane); break; - case 35: _bit_unpack_64_lane<35>(in, out, reference, lane); break; - case 36: _bit_unpack_64_lane<36>(in, out, reference, lane); break; - case 37: _bit_unpack_64_lane<37>(in, out, reference, lane); break; - case 38: _bit_unpack_64_lane<38>(in, out, reference, lane); break; - case 39: _bit_unpack_64_lane<39>(in, out, reference, lane); break; - case 40: _bit_unpack_64_lane<40>(in, out, reference, lane); break; - case 41: _bit_unpack_64_lane<41>(in, out, reference, lane); break; - case 42: _bit_unpack_64_lane<42>(in, out, reference, lane); break; - case 43: _bit_unpack_64_lane<43>(in, out, reference, lane); break; - case 44: _bit_unpack_64_lane<44>(in, out, reference, lane); break; - case 45: _bit_unpack_64_lane<45>(in, out, reference, lane); break; - case 46: _bit_unpack_64_lane<46>(in, out, reference, lane); break; - case 47: _bit_unpack_64_lane<47>(in, out, reference, lane); break; - case 48: _bit_unpack_64_lane<48>(in, out, reference, lane); break; - case 49: _bit_unpack_64_lane<49>(in, out, reference, lane); break; - case 50: _bit_unpack_64_lane<50>(in, out, reference, lane); break; - case 51: _bit_unpack_64_lane<51>(in, out, reference, lane); break; - case 52: _bit_unpack_64_lane<52>(in, out, reference, lane); break; - case 53: _bit_unpack_64_lane<53>(in, out, reference, lane); break; - case 54: _bit_unpack_64_lane<54>(in, out, reference, lane); break; - case 55: _bit_unpack_64_lane<55>(in, out, reference, lane); break; - case 56: _bit_unpack_64_lane<56>(in, out, reference, lane); break; - case 57: _bit_unpack_64_lane<57>(in, out, reference, lane); break; - case 58: _bit_unpack_64_lane<58>(in, out, reference, lane); break; - case 59: _bit_unpack_64_lane<59>(in, out, reference, lane); break; - case 60: _bit_unpack_64_lane<60>(in, out, reference, lane); break; - case 61: _bit_unpack_64_lane<61>(in, out, reference, lane); break; - case 62: _bit_unpack_64_lane<62>(in, out, reference, lane); break; - case 63: _bit_unpack_64_lane<63>(in, out, reference, lane); break; - case 64: _bit_unpack_64_lane<64>(in, out, reference, lane); break; + // Step 2: Apply patches to shared memory in parallel + PatchesCursor cursor(patches, blockIdx.x, thread_idx, 16); + auto patch = cursor.next(); + while (patch.index != FL_CHUNK) { + shared_out[patch.index] = patch.value; + patch = cursor.next(); } -} + __syncwarp(); -template -__device__ void _bit_unpack_64_device(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, int thread_idx, GPUPatches& patches) { - __shared__ uint64_t shared_out[1024]; + // Step 3: Copy to global memory #pragma unroll - for (int i = 0; i < 1; i++) { - _bit_unpack_64_lane(in, shared_out, reference, thread_idx * 1 + i); - } - __syncwarp(); - PatchesCursor cursor(patches, blockIdx.x, thread_idx, 16); - auto patch = cursor.next(); - for (int i = 0; i < 64; i++) { + for (int i = 0; i < FL_CHUNK / 16; i++) { auto idx = i * 16 + thread_idx; - if (idx == patch.index) { - out[idx] = patch.value; - patch = cursor.next(); - } else { - out[idx] = shared_out[idx]; - } + out[idx] = shared_out[idx]; } } extern "C" __global__ void bit_unpack_64_0bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 0 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 0)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<0>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_1bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 1 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 1)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<1>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_2bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 2 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 2)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<2>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_3bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 3 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 3)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<3>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_4bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 4 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 4)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<4>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_5bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 5 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 5)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<5>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_6bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 6 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 6)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<6>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_7bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 7 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 7)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<7>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_8bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 8 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 8)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<8>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_9bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 9 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 9)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<9>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_10bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 10 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 10)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<10>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_11bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 11 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 11)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<11>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_12bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 12 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 12)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<12>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_13bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 13 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 13)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<13>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_14bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 14 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 14)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<14>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_15bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 15 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 15)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<15>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_16bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 16 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 16)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<16>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_17bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 17 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 17)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<17>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_18bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 18 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 18)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<18>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_19bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 19 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 19)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<19>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_20bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 20 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 20)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<20>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_21bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 21 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 21)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<21>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_22bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 22 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 22)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<22>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_23bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 23 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 23)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<23>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_24bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 24 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 24)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<24>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_25bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 25 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 25)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<25>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_26bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 26 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 26)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<26>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_27bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 27 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 27)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<27>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_28bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 28 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 28)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<28>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_29bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 29 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 29)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<29>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_30bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 30 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 30)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<30>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_31bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 31 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 31)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<31>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_32bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 32 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 32)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<32>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_33bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 33 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 33)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<33>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_34bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 34 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 34)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<34>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_35bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 35 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 35)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<35>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_36bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 36 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 36)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<36>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_37bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 37 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 37)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<37>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_38bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 38 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 38)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<38>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_39bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 39 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 39)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<39>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_40bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 40 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 40)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<40>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_41bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 41 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 41)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<41>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_42bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 42 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 42)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<42>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_43bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 43 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 43)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<43>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_44bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 44 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 44)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<44>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_45bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 45 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 45)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<45>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_46bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 46 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 46)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<46>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_47bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 47 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 47)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<47>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_48bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 48 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 48)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<48>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_49bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 49 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 49)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<49>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_50bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 50 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 50)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<50>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_51bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 51 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 51)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<51>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_52bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 52 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 52)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<52>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_53bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 53 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 53)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<53>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_54bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 54 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 54)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<54>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_55bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 55 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 55)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<55>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_56bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 56 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 56)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<56>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_57bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 57 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 57)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<57>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_58bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 58 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 58)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<58>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_59bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 59 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 59)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<59>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_60bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 60 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 60)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<60>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_61bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 61 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 61)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<61>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_62bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 62 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 62)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<62>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_63bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 63 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 63)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<63>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_64_64bw_16t(const uint64_t *__restrict full_in, uint64_t *__restrict full_out, uint64_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 64 / sizeof(uint64_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 64)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_64_device<64>(in, out, reference, thread_idx, patches); } diff --git a/vortex-cuda/kernels/src/bit_unpack_64_lanes.cuh b/vortex-cuda/kernels/src/bit_unpack_64_lanes.cuh new file mode 100644 index 00000000000..afc17737b40 --- /dev/null +++ b/vortex-cuda/kernels/src/bit_unpack_64_lanes.cuh @@ -0,0 +1,12579 @@ +// AUTO-GENERATED. Do not edit by hand! +#pragma once + +#include +#include +#include +#include "fastlanes_common.cuh" + +template +__device__ void _bit_unpack_64_lane(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane); + +template <> +__device__ void _bit_unpack_64_lane<0>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + #pragma unroll + for (int row = 0; row < 64; row++) { + out[INDEX(row, lane)] = reference; + } +} + +template <> +__device__ void _bit_unpack_64_lane<1>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 1); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 1); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 1); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 1); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 1); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 1); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 1); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 1); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 1); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 1); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 1); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 1); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 1); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 1); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 1); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 1); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 1); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 1); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 1); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 1); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 1); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 1); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 1); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 1); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 1); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 1); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 1); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 1); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 1); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 1); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 1); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 1); + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 1); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 1); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 1); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 1); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 1); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 1); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 1); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 1); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 1); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 1); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 1); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 1); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 1); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 1); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 1); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 1); + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 1); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 1); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 1); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 1); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 1); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 1); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 1); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 1); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 1); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 1); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 1); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 1); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 1); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 1); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 1); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<2>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 2); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 2); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 2); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 2); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 2); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 2); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 2); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 2); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 2); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 2); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 2); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 2); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 2); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 2); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 2); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 2); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 2); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 2); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 2); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 2); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 2); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 2); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 2); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 2); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 2); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 2); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 2); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 2); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 2); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 2); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 2); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 0)) << 2; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 2); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 2); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 2); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 2); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 2); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 2); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 2); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 2); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 2); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 2); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 2); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 2); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 2); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 2); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 2); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 2); + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 2); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 2); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 2); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 2); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 2); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 2); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 2); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 2); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 2); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 2); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 2); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 2); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 2); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 2); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 2); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<3>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 3); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 3); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 3); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 3); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 3); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 3); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 3); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 3); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 3); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 3); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 3); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 3); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 3); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 3); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 3); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 3); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 3); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 3); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 3); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 3); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 3); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 2)) << 1; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 3); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 3); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 3); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 3); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 3); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 3); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 3); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 3); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 3); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 3); + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 3); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 3); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 3); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 3); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 3); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 3); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 3); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 3); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 3); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 3); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 1)) << 2; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 3); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 3); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 3); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 3); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 3); + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 3); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 3); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 3); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 3); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 3); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 3); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 3); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 3); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 3); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 3); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 3); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 3); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 3); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 3); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 3); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<4>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 4); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 4); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 4); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 4); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 4); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 4); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 4); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 4); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 4); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 4); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 4); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 4); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 4); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 4); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 4); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 0)) << 4; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 4); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 4); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 4); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 4); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 4); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 4); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 4); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 4); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 4); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 4); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 4); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 4); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 4); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 4); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 4); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 0)) << 4; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 4); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 4); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 4); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 4); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 4); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 4); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 4); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 4); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 4); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 4); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 4); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 4); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 4); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 4); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 4); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 0)) << 4; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 4); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 4); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 4); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 4); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 4); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 4); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 4); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 4); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 4); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 4); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 4); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 4); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 4); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 4); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 4); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<5>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 5); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 5); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 5); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 5); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 5); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 5); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 5); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 5); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 5); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 5); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 5); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 5); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 1)) << 4; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 5); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 5); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 5); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 5); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 5); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 5); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 5); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 5); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 5); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 5); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 5); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 5); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 2)) << 3; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 5); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 5); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 5); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 5); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 5); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 5); + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 5); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 5); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 5); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 5); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 5); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 5); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 3)) << 2; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 5); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 5); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 5); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 5); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 5); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 5); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 5); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 5); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 5); + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 5); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 5); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 5); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 4)) << 1; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 5); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 5); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 5); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 5); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 5); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 5); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 5); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 5); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 5); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 5); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 5); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<6>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 6); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 6); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 6); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 6); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 6); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 6); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 6); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 6); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 6); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 6); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 2)) << 4; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 6); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 6); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 6); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 6); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 6); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 6); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 6); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 6); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 6); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 6); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 4)) << 2; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 6); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 6); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 6); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 6); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 6); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 6); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 6); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 6); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 6); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 0)) << 6; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 6); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 6); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 6); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 6); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 6); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 6); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 6); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 6); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 6); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 6); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 2)) << 4; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 6); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 6); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 6); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 6); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 6); + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 6); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 6); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 6); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 6); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 6); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 4)) << 2; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 6); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 6); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 6); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 6); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 6); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 6); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 6); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 6); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 6); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<7>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 7); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 7); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 7); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 7); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 7); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 7); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 7); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 7); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 7); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 6)) << 1; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 7); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 7); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 7); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 7); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 7); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 7); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 7); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 7); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 5)) << 2; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 7); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 7); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 7); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 7); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 7); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 7); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 7); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 7); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 4)) << 3; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 7); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 7); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 7); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 7); + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 7); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 7); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 7); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 7); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 3)) << 4; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 7); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 7); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 7); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 7); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 7); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 7); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 7); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 7); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 2)) << 5; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 7); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 7); + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 7); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 7); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 7); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 7); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 7); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 7); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 1)) << 6; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 7); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 7); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 7); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 7); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 7); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 7); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 7); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 7); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<8>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 8); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 8); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 8); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 8); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 8); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 8); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 8); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 0)) << 8; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 8); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 8); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 8); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 8); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 8); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 8); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 8); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 0)) << 8; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 8); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 8); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 8); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 8); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 8); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 8); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 8); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 0)) << 8; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 8); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 8); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 8); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 8); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 8); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 8); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 8); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 0)) << 8; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 8); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 8); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 8); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 8); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 8); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 8); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 8); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 0)) << 8; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 8); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 8); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 8); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 8); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 8); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 8); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 8); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 0)) << 8; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 8); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 8); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 8); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 8); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 8); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 8); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 8); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 0)) << 8; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 8); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 8); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 8); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 8); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 8); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 8); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 8); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<9>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 9); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 9); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 9); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 9); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 9); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 9); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 9); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 8)) << 1; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 9); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 9); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 9); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 9); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 9); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 9); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 7)) << 2; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 9); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 9); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 9); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 9); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 9); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 9); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 6)) << 3; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 9); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 9); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 9); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 9); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 9); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 9); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 5)) << 4; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 9); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 9); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 9); + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 9); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 9); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 9); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 4)) << 5; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 9); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 9); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 9); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 9); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 9); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 9); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 3)) << 6; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 9); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 9); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 9); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 9); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 9); + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 9); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 2)) << 7; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 9); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 9); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 9); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 9); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 9); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 9); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 1)) << 8; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 9); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 9); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 9); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 9); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 9); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 9); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<10>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 10); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 10); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 10); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 10); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 10); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 10); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 6)) << 4; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 10); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 10); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 10); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 10); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 10); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 2)) << 8; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 10); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 10); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 10); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 10); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 10); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 10); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 8)) << 2; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 10); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 10); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 10); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 10); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 10); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 4)) << 6; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 10); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 10); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 10); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 10); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 10); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 0)) << 10; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 10); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 10); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 10); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 10); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 10); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 10); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 6)) << 4; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 10); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 10); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 10); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 10); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 10); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 2)) << 8; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 10); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 10); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 10); + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 10); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 10); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 10); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 8)) << 2; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 10); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 10); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 10); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 10); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 10); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 4)) << 6; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 10); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 10); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 10); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 10); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 10); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<11>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 11); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 11); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 11); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 11); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 11); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 2)) << 9; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 11); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 11); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 11); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 11); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 11); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 4)) << 7; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 11); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 11); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 11); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 11); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 11); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 6)) << 5; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 11); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 11); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 11); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 11); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 11); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 8)) << 3; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 11); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 11); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 11); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 11); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 11); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 10)) << 1; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 11); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 11); + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 11); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 11); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 1)) << 10; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 11); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 11); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 11); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 11); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 11); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 3)) << 8; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 11); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 11); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 11); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 11); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 11); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 5)) << 6; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 11); + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 11); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 11); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 11); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 11); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 7)) << 4; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 11); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 11); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 11); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 11); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 11); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 9)) << 2; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 11); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 11); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 11); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 11); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<12>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 12); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 12); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 12); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 12); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 12); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 8)) << 4; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 12); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 12); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 12); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 12); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 4)) << 8; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 12); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 12); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 12); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 12); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 0)) << 12; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 12); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 12); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 12); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 12); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 12); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 8)) << 4; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 12); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 12); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 12); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 12); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 4)) << 8; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 12); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 12); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 12); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 12); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 0)) << 12; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 12); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 12); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 12); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 12); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 12); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 8)) << 4; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 12); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 12); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 12); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 12); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 4)) << 8; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 12); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 12); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 12); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 12); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 0)) << 12; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 12); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 12); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 12); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 12); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 12); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 8)) << 4; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 12); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 12); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 12); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 12); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 4)) << 8; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 12); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 12); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 12); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 12); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<13>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 13); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 13); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 13); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 13); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 1)) << 12; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 13); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 13); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 13); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 13); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 2)) << 11; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 13); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 13); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 13); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 13); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 3)) << 10; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 13); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 13); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 13); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 13); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 4)) << 9; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 13); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 13); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 13); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 13); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 5)) << 8; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 13); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 13); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 13); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 13); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 6)) << 7; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 13); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 13); + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 13); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 13); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 7)) << 6; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 13); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 13); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 13); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 13); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 8)) << 5; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 13); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 13); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 13); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 13); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 9)) << 4; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 13); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 13); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 13); + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 13); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 10)) << 3; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 13); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 13); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 13); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 13); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 11)) << 2; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 13); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 13); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 13); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 13); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 12)) << 1; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 13); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 13); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 13); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<14>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 14); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 14); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 14); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 14); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 6)) << 8; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 14); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 14); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 14); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 14); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 12)) << 2; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 14); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 14); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 14); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 4)) << 10; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 14); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 14); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 14); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 14); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 10)) << 4; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 14); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 14); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 14); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 2)) << 12; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 14); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 14); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 14); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 14); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 8)) << 6; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 14); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 14); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 14); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 0)) << 14; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 14); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 14); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 14); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 14); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 6)) << 8; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 14); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 14); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 14); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 14); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 12)) << 2; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 14); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 14); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 14); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 4)) << 10; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 14); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 14); + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 14); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 14); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 10)) << 4; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 14); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 14); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 14); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 2)) << 12; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 14); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 14); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 14); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 14); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 8)) << 6; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 14); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 14); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 14); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<15>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 15); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 15); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 15); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 15); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 11)) << 4; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 15); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 15); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 15); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 7)) << 8; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 15); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 15); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 15); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 3)) << 12; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 15); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 15); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 15); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 15); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 14)) << 1; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 15); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 15); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 15); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 10)) << 5; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 15); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 15); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 15); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 6)) << 9; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 15); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 15); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 15); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 2)) << 13; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 15); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 15); + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 15); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 15); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 13)) << 2; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 15); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 15); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 15); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 9)) << 6; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 15); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 15); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 15); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 5)) << 10; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 15); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 15); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 15); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 1)) << 14; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 15); + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 15); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 15); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 15); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 12)) << 3; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 15); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 15); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 15); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 8)) << 7; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 15); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 15); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 15); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 4)) << 11; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 15); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 15); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 15); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 15); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<16>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 16); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 16); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 16); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 0)) << 16; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 16); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 16); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 16); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 0)) << 16; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 16); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 16); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 16); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 0)) << 16; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 16); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 16); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 16); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 0)) << 16; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 16); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 16); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 16); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 0)) << 16; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 16); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 16); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 16); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 0)) << 16; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 16); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 16); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 16); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 0)) << 16; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 16); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 16); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 16); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 0)) << 16; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 16); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 16); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 16); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 0)) << 16; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 16); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 16); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 16); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 0)) << 16; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 16); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 16); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 16); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 0)) << 16; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 16); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 16); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 16); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 0)) << 16; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 16); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 16); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 16); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 0)) << 16; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 16); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 16); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 16); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 0)) << 16; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 16); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 16); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 16); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 0)) << 16; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 16); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 16); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 16); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<17>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 17); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 17); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 17); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 4)) << 13; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 17); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 17); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 17); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 8)) << 9; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 17); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 17); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 17); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 12)) << 5; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 17); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 17); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 17); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 16)) << 1; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 17); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 17); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 3)) << 14; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 17); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 17); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 17); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 7)) << 10; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 17); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 17); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 17); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 11)) << 6; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 17); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 17); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 17); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 15)) << 2; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 17); + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 17); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 15); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 2)) << 15; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 17); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 17); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 17); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 6)) << 11; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 17); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 17); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 17); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 10)) << 7; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 17); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 17); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 17); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 14)) << 3; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 17); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 17); + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 1)) << 16; + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 17); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 17); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 17); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 5)) << 12; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 17); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 17); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 17); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 9)) << 8; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 17); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 17); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 17); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 13)) << 4; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 17); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 17); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 17); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<18>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 18); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 18); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 18); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 8)) << 10; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 18); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 18); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 18); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 16)) << 2; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 18); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 18); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 6)) << 12; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 18); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 18); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 18); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 14)) << 4; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 18); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 18); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 4)) << 14; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 18); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 18); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 18); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 12)) << 6; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 18); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 18); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 2)) << 16; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 18); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 18); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 18); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 10)) << 8; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 18); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 18); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 0)) << 18; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 18); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 18); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 18); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 8)) << 10; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 18); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 18); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 18); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 16)) << 2; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 18); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 18); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 6)) << 12; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 18); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 18); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 18); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 14)) << 4; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 18); + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 18); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 4)) << 14; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 18); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 18); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 18); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 12)) << 6; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 18); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 18); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 2)) << 16; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 18); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 18); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 18); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 10)) << 8; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 18); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 18); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<19>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 19); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 19); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 19); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 12)) << 7; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 19); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 19); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 5)) << 14; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 19); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 19); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 19); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 17)) << 2; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 19); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 19); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 10)) << 9; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 19); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 19); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 3)) << 16; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 19); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 19); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 19); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 15)) << 4; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 19); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 19); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 8)) << 11; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 19); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 19); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 1)) << 18; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 19); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 19); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 19); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 13)) << 6; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 19); + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 19); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 6)) << 13; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 19); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 19); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 19); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 18)) << 1; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 19); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 19); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 11)) << 8; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 19); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 19); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 15); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 4)) << 15; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 19); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 19); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 19); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 16)) << 3; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 19); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 19); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 9)) << 10; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 19); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 19); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 17); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 2)) << 17; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 19); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 19); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 19); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 14)) << 5; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 19); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 19); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 7)) << 12; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 19); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 19); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 19); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<20>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 20); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 20); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 20); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 16)) << 4; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 20); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 20); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 12)) << 8; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 20); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 20); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 8)) << 12; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 20); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 20); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 4)) << 16; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 20); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 20); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 0)) << 20; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 20); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 20); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 20); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 16)) << 4; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 20); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 20); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 12)) << 8; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 20); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 20); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 8)) << 12; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 20); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 20); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 4)) << 16; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 20); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 20); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 0)) << 20; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 20); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 20); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 20); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 16)) << 4; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 20); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 20); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 12)) << 8; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 20); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 20); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 8)) << 12; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 20); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 20); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 4)) << 16; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 20); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 20); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 0)) << 20; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 20); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 20); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 20); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 16)) << 4; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 20); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 20); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 12)) << 8; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 20); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 20); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 8)) << 12; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 20); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 20); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 4)) << 16; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 20); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 20); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<21>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 21); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 21); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 21); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 20)) << 1; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 21); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 21); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 19)) << 2; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 21); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 21); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 18)) << 3; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 21); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 21); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 17)) << 4; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 21); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 21); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 16)) << 5; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 21); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 21); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 15)) << 6; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 21); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 21); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 14)) << 7; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 21); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 21); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 13)) << 8; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 21); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 21); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 12)) << 9; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 21); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 21); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 11)) << 10; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 21); + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 21); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 10)) << 11; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 21); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 21); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 9)) << 12; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 21); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 21); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 8)) << 13; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 21); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 21); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 7)) << 14; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 21); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 21); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 15); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 6)) << 15; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 21); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 21); + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 5)) << 16; + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 21); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 21); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 17); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 4)) << 17; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 21); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 21); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 3)) << 18; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 21); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 21); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 19); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 2)) << 19; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 21); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 21); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 1)) << 20; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 21); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 21); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 21); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<22>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 22); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 22); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 2)) << 20; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 22); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 22); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 4)) << 18; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 22); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 22); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 6)) << 16; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 22); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 22); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 8)) << 14; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 22); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 22); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 10)) << 12; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 22); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 22); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 12)) << 10; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 22); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 22); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 14)) << 8; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 22); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 22); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 16)) << 6; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 22); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 22); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 18)) << 4; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 22); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 22); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 20)) << 2; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 22); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 0)) << 22; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 22); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 22); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 2)) << 20; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 22); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 22); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 4)) << 18; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 22); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 22); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 6)) << 16; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 22); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 22); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 8)) << 14; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 22); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 22); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 10)) << 12; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 22); + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 22); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 12)) << 10; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 22); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 22); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 14)) << 8; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 22); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 22); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 16)) << 6; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 22); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 22); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 18)) << 4; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 22); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 22); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 20)) << 2; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 22); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<23>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 23); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 23); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 5)) << 18; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 23); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 23); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 10)) << 13; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 23); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 23); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 15)) << 8; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 23); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 23); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 20)) << 3; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 23); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 21); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 2)) << 21; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 23); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 23); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 7)) << 16; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 23); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 23); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 12)) << 11; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 23); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 23); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 17)) << 6; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 23); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 23); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 22)) << 1; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 23); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 19); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 4)) << 19; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 23); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 23); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 9)) << 14; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 23); + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 23); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 14)) << 9; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 23); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 23); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 19)) << 4; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 23); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 1)) << 22; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 23); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 23); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 17); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 6)) << 17; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 23); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 23); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 11)) << 12; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 23); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 23); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 16)) << 7; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 23); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 23); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 21)) << 2; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 23); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 3)) << 20; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 23); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 23); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 15); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 8)) << 15; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 23); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 23); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 13)) << 10; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 23); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 23); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 18)) << 5; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 23); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 23); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<24>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 24); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 24); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 8)) << 16; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 24); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 24); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 16)) << 8; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 24); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 0)) << 24; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 24); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 24); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 8)) << 16; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 24); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 24); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 16)) << 8; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 24); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 0)) << 24; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 24); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 24); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 8)) << 16; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 24); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 24); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 16)) << 8; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 24); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 0)) << 24; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 24); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 24); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 8)) << 16; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 24); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 24); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 16)) << 8; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 24); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 0)) << 24; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 24); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 24); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 8)) << 16; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 24); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 24); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 16)) << 8; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 24); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 0)) << 24; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 24); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 24); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 8)) << 16; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 24); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 24); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 16)) << 8; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 24); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 0)) << 24; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 24); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 24); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 8)) << 16; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 24); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 24); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 16)) << 8; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 24); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 0)) << 24; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 24); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 24); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 8)) << 16; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 24); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 24); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 16)) << 8; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 24); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<25>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 25); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 25); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 11)) << 14; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 25); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 25); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 22)) << 3; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 25); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 17); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 8)) << 17; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 25); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 25); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 19)) << 6; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 25); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 5)) << 20; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 25); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 25); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 16)) << 9; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 25); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 23); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 2)) << 23; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 25); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 25); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 13)) << 12; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 25); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 25); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 24)) << 1; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 25); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 15); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 10)) << 15; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 25); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 25); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 21)) << 4; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 25); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 7)) << 18; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 25); + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 25); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 18)) << 7; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 25); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 21); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 4)) << 21; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 25); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 25); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 15)) << 10; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 25); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 1)) << 24; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 25); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 25); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 12)) << 13; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 25); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 25); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 23)) << 2; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 25); + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 9)) << 16; + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 25); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 25); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 20)) << 5; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 25); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 19); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 6)) << 19; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 25); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 25); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 17)) << 8; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 25); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 3)) << 22; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 25); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 25); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 14)) << 11; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 25); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 25); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<26>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 26); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 26); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 14)) << 12; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 26); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 2)) << 24; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 26); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 26); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 16)) << 10; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 26); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 4)) << 22; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 26); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 26); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 18)) << 8; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 26); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 6)) << 20; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 26); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 26); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 20)) << 6; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 26); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 8)) << 18; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 26); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 26); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 22)) << 4; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 26); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 10)) << 16; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 26); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 26); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 24)) << 2; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 26); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 12)) << 14; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 26); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 0)) << 26; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 26); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 26); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 14)) << 12; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 26); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 2)) << 24; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 26); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 26); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 16)) << 10; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 26); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 4)) << 22; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 26); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 26); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 18)) << 8; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 26); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 6)) << 20; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 26); + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 26); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 20)) << 6; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 26); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 8)) << 18; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 26); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 26); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 22)) << 4; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 26); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 10)) << 16; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 26); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 26); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 24)) << 2; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 26); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 12)) << 14; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 26); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<27>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 27); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 27); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 17)) << 10; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 27); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 7)) << 20; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 27); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 27); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 24)) << 3; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 27); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 14)) << 13; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 27); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 23); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 4)) << 23; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 27); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 27); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 21)) << 6; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 27); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 11)) << 16; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 27); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 1)) << 26; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 27); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 27); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 18)) << 9; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 27); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 19); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 8)) << 19; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 27); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 27); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 25)) << 2; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 27); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 15)) << 12; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 27); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 5)) << 22; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 27); + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 27); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 22)) << 5; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 27); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 15); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 12)) << 15; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 27); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 25); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 2)) << 25; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 27); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 27); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 19)) << 8; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 27); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 9)) << 18; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 27); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 27); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 26)) << 1; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 27); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 16)) << 11; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 27); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 21); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 6)) << 21; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 27); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 27); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 23)) << 4; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 27); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 13)) << 14; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 27); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 3)) << 24; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 27); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 27); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 20)) << 7; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 27); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 17); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 10)) << 17; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 27); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 27); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<28>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 28); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 28); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 20)) << 8; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 28); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 12)) << 16; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 28); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 4)) << 24; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 28); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 28); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 24)) << 4; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 28); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 16)) << 12; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 28); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 8)) << 20; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 28); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 0)) << 28; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 28); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 28); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 20)) << 8; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 28); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 12)) << 16; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 28); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 4)) << 24; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 28); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 28); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 24)) << 4; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 28); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 16)) << 12; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 28); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 8)) << 20; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 28); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 0)) << 28; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 28); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 28); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 20)) << 8; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 28); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 12)) << 16; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 28); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 4)) << 24; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 28); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 28); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 24)) << 4; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 28); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 16)) << 12; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 28); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 8)) << 20; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 28); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 0)) << 28; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 28); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 28); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 20)) << 8; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 28); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 12)) << 16; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 28); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 4)) << 24; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 28); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 28); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 24)) << 4; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 28); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 16)) << 12; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 28); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 8)) << 20; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 28); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<29>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 29); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 29); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 23)) << 6; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 29); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 17)) << 12; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 29); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 11)) << 18; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 29); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 5)) << 24; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 29); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 29); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 28)) << 1; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 29); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 22)) << 7; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 29); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 16)) << 13; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 29); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 19); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 10)) << 19; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 29); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 25); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 4)) << 25; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 29); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 29); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 27)) << 2; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 29); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 21)) << 8; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 29); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 15)) << 14; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 29); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 9)) << 20; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 29); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 3)) << 26; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 29); + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 29); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 26)) << 3; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 29); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 20)) << 9; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 29); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 15); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 14)) << 15; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 29); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 21); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 8)) << 21; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 29); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 27); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 2)) << 27; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 29); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 29); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 25)) << 4; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 29); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 19)) << 10; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 29); + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 13)) << 16; + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 29); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 7)) << 22; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 29); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 1)) << 28; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 29); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 29); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 24)) << 5; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 29); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 18)) << 11; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 29); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 17); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 12)) << 17; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 29); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 23); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 6)) << 23; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 29); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 29); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<30>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 30); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 30); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 26)) << 4; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 30); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 22)) << 8; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 30); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 18)) << 12; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 30); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 14)) << 16; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 30); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 10)) << 20; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 30); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 6)) << 24; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 30); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 2)) << 28; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 30); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 30); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 28)) << 2; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 30); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 24)) << 6; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 30); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 20)) << 10; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 30); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 16)) << 14; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 30); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 12)) << 18; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 30); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 8)) << 22; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 30); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 4)) << 26; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 30); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 0)) << 30; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 30); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 30); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 26)) << 4; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 30); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 22)) << 8; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 30); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 18)) << 12; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 30); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 14)) << 16; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 30); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 10)) << 20; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 30); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 6)) << 24; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 30); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 2)) << 28; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 30); + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 30); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 28)) << 2; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 30); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 24)) << 6; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 30); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 20)) << 10; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 30); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 16)) << 14; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 30); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 12)) << 18; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 30); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 8)) << 22; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 30); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 4)) << 26; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 30); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<31>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 31); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 31); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 29)) << 2; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 31); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 27)) << 4; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 31); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 25)) << 6; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 31); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 23)) << 8; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 31); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 21)) << 10; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 31); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 19)) << 12; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 31); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 17)) << 14; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 31); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 15)) << 16; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 31); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 13)) << 18; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 31); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 11)) << 20; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 31); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 9)) << 22; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 31); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 7)) << 24; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 31); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 5)) << 26; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 31); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 3)) << 28; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 31); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 1)) << 30; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 31); + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 31); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 30)) << 1; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 31); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 28)) << 3; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 31); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 26)) << 5; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 31); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 24)) << 7; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 31); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 22)) << 9; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 31); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 20)) << 11; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 31); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 18)) << 13; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 31); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 15); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 16)) << 15; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 31); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 17); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 14)) << 17; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 31); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 19); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 12)) << 19; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 31); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 21); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 10)) << 21; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 31); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 23); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 8)) << 23; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 31); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 25); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 6)) << 25; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 31); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 27); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 4)) << 27; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 31); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 29); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 2)) << 29; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 31); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 31); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<32>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 0)) << 32; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 32); + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<33>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 33); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 31); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 2)) << 31; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 33); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 29); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 4)) << 29; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 33); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 27); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 6)) << 27; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 33); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 25); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 8)) << 25; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 33); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 23); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 10)) << 23; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 33); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 21); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 12)) << 21; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 33); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 19); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 14)) << 19; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 33); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 17); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 16)) << 17; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 33); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 15); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 18)) << 15; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 33); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 20)) << 13; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 33); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 22)) << 11; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 33); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 24)) << 9; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 33); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 26)) << 7; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 33); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 28)) << 5; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 33); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 30)) << 3; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 33); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 32)) << 1; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 1)) << 32; + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 33); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 3)) << 30; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 33); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 5)) << 28; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 33); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 7)) << 26; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 33); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 9)) << 24; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 33); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 11)) << 22; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 33); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 13)) << 20; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 33); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 15)) << 18; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 33); + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 17)) << 16; + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 33); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 19)) << 14; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 33); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 21)) << 12; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 33); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 23)) << 10; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 33); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 25)) << 8; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 33); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 27)) << 6; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 33); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 29)) << 4; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 33); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 31)) << 2; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 33); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<34>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 34); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 4)) << 30; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 34); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 8)) << 26; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 34); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 12)) << 22; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 34); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 16)) << 18; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 34); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 20)) << 14; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 34); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 24)) << 10; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 34); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 28)) << 6; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 34); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 32)) << 2; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 2)) << 32; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 34); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 6)) << 28; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 34); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 10)) << 24; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 34); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 14)) << 20; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 34); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 18)) << 16; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 34); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 22)) << 12; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 34); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 26)) << 8; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 34); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 30)) << 4; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 0)) << 34; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 34); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 4)) << 30; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 34); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 8)) << 26; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 34); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 12)) << 22; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 34); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 16)) << 18; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 34); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 20)) << 14; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 34); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 24)) << 10; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 34); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 28)) << 6; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 34); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 32)) << 2; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 2)) << 32; + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 34); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 6)) << 28; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 34); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 10)) << 24; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 34); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 14)) << 20; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 34); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 18)) << 16; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 34); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 22)) << 12; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 34); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 26)) << 8; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 34); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 30)) << 4; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<35>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 35); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 29); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 6)) << 29; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 35); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 23); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 12)) << 23; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 35); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 17); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 18)) << 17; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 35); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 24)) << 11; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 35); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 30)) << 5; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 1)) << 34; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 35); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 7)) << 28; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 35); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 13)) << 22; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 35); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 19)) << 16; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 35); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 25)) << 10; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 35); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 31)) << 4; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 33); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 2)) << 33; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 35); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 27); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 8)) << 27; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 35); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 21); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 14)) << 21; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 35); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 15); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 20)) << 15; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 35); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 26)) << 9; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 35); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 32)) << 3; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 3)) << 32; + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 35); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 9)) << 26; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 35); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 15)) << 20; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 35); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 21)) << 14; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 35); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 27)) << 8; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 35); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 33)) << 2; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 31); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 4)) << 31; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 35); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 25); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 10)) << 25; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 35); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 19); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 16)) << 19; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 35); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 22)) << 13; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 35); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 28)) << 7; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 35); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 34)) << 1; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 5)) << 30; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 35); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 11)) << 24; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 35); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 17)) << 18; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 35); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 23)) << 12; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 35); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 29)) << 6; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 35); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<36>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 36); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 8)) << 28; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 36); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 16)) << 20; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 36); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 24)) << 12; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 36); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 32)) << 4; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 4)) << 32; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 36); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 12)) << 24; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 36); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 20)) << 16; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 36); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 28)) << 8; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 0)) << 36; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 36); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 8)) << 28; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 36); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 16)) << 20; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 36); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 24)) << 12; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 36); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 32)) << 4; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 4)) << 32; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 36); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 12)) << 24; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 36); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 20)) << 16; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 36); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 28)) << 8; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 0)) << 36; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 36); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 8)) << 28; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 36); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 16)) << 20; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 36); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 24)) << 12; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 36); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 32)) << 4; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 4)) << 32; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 36); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 12)) << 24; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 36); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 20)) << 16; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 36); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 28)) << 8; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 0)) << 36; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 36); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 8)) << 28; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 36); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 16)) << 20; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 36); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 24)) << 12; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 36); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 32)) << 4; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 4)) << 32; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 36); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 12)) << 24; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 36); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 20)) << 16; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 36); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 28)) << 8; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<37>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 37); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 27); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 10)) << 27; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 37); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 17); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 20)) << 17; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 37); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 30)) << 7; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 3)) << 34; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 37); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 13)) << 24; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 37); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 23)) << 14; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 37); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 33)) << 4; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 31); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 6)) << 31; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 37); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 21); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 16)) << 21; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 37); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 26)) << 11; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 37); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 36)) << 1; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 9)) << 28; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 37); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 19)) << 18; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 37); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 29)) << 8; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 35); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 2)) << 35; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 37); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 25); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 12)) << 25; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 37); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 15); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 22)) << 15; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 37); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 32)) << 5; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 5)) << 32; + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 37); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 15)) << 22; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 37); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 25)) << 12; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 37); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 35)) << 2; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 29); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 8)) << 29; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 37); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 19); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 18)) << 19; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 37); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 28)) << 9; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 1)) << 36; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 37); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 11)) << 26; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 37); + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 21)) << 16; + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 37); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 31)) << 6; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 33); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 4)) << 33; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 37); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 23); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 14)) << 23; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 37); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 24)) << 13; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 37); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 34)) << 3; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 7)) << 30; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 37); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 17)) << 20; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 37); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 27)) << 10; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 37); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<38>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 38); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 12)) << 26; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 38); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 24)) << 14; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 38); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 36)) << 2; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 10)) << 28; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 38); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 22)) << 16; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 38); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 34)) << 4; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 8)) << 30; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 38); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 20)) << 18; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 38); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 32)) << 6; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 6)) << 32; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 38); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 18)) << 20; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 38); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 30)) << 8; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 4)) << 34; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 38); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 16)) << 22; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 38); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 28)) << 10; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 2)) << 36; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 38); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 14)) << 24; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 38); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 26)) << 12; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 0)) << 38; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 38); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 12)) << 26; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 38); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 24)) << 14; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 38); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 36)) << 2; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 10)) << 28; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 38); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 22)) << 16; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 38); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 34)) << 4; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 8)) << 30; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 38); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 20)) << 18; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 38); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 32)) << 6; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 6)) << 32; + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 38); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 18)) << 20; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 38); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 30)) << 8; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 4)) << 34; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 38); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 16)) << 22; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 38); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 28)) << 10; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 2)) << 36; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 38); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 14)) << 24; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 38); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 26)) << 12; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<39>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 39); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 25); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 14)) << 25; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 39); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 28)) << 11; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 3)) << 36; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 39); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 17)) << 22; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 39); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 31)) << 8; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 33); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 6)) << 33; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 39); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 19); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 20)) << 19; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 39); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 34)) << 5; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 9)) << 30; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 39); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 23)) << 16; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 39); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 37)) << 2; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 27); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 12)) << 27; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 39); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 26)) << 13; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 1)) << 38; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 39); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 15)) << 24; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 39); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 29)) << 10; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 35); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 4)) << 35; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 39); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 21); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 18)) << 21; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 39); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 32)) << 7; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 7)) << 32; + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 39); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 21)) << 18; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 39); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 35)) << 4; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 29); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 10)) << 29; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 39); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 15); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 24)) << 15; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 39); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 38)) << 1; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 13)) << 26; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 39); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 27)) << 12; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 37); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 2)) << 37; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 39); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 23); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 16)) << 23; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 39); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 30)) << 9; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 5)) << 34; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 39); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 19)) << 20; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 39); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 33)) << 6; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 31); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 8)) << 31; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 39); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 17); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 22)) << 17; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 39); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 36)) << 3; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 11)) << 28; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 39); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 38]; + tmp |= (src & MASK(uint64_t, 25)) << 14; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 39); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<40>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 40); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 16)) << 24; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 40); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 32)) << 8; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 8)) << 32; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 40); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 24)) << 16; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 0)) << 40; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 40); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 16)) << 24; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 40); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 32)) << 8; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 8)) << 32; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 40); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 24)) << 16; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 0)) << 40; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 40); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 16)) << 24; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 40); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 32)) << 8; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 8)) << 32; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 40); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 24)) << 16; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 0)) << 40; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 40); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 16)) << 24; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 40); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 32)) << 8; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 8)) << 32; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 40); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 24)) << 16; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 0)) << 40; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 40); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 16)) << 24; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 40); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 32)) << 8; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 8)) << 32; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 40); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 24)) << 16; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 0)) << 40; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 40); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 16)) << 24; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 40); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 32)) << 8; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 8)) << 32; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 40); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 24)) << 16; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 0)) << 40; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 40); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 16)) << 24; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 40); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 32)) << 8; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 8)) << 32; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 40); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 24)) << 16; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 0)) << 40; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 40); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 16)) << 24; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 40); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 32)) << 8; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 38]; + tmp |= (src & MASK(uint64_t, 8)) << 32; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 40); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 39]; + tmp |= (src & MASK(uint64_t, 24)) << 16; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<41>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 41); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 23); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 18)) << 23; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 41); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 36)) << 5; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 13)) << 28; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 41); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 31)) << 10; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 33); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 8)) << 33; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 41); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 15); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 26)) << 15; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 3)) << 38; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 41); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 21)) << 20; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 41); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 39)) << 2; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 25); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 16)) << 25; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 41); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 34)) << 7; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 11)) << 30; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 41); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 29)) << 12; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 35); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 6)) << 35; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 41); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 17); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 24)) << 17; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 1)) << 40; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 41); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 19)) << 22; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 41); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 37)) << 4; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 27); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 14)) << 27; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 41); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 32)) << 9; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 9)) << 32; + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 41); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 27)) << 14; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 37); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 4)) << 37; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 41); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 19); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 22)) << 19; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 41); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 40)) << 1; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 17)) << 24; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 41); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 35)) << 6; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 29); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 12)) << 29; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 41); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 30)) << 11; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 7)) << 34; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 41); + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 25)) << 16; + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 39); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 2)) << 39; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 41); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 21); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 20)) << 21; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 41); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 38)) << 3; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 15)) << 26; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 41); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 33)) << 8; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 31); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 10)) << 31; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 41); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + src = in[lane + LANE_COUNT * 38]; + tmp |= (src & MASK(uint64_t, 28)) << 13; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 39]; + tmp |= (src & MASK(uint64_t, 5)) << 36; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 41); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 40]; + tmp |= (src & MASK(uint64_t, 23)) << 18; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 41); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<42>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 42); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 20)) << 22; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 42); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 40)) << 2; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 18)) << 24; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 42); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 38)) << 4; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 16)) << 26; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 42); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 36)) << 6; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 14)) << 28; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 42); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 34)) << 8; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 12)) << 30; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 42); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 32)) << 10; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 10)) << 32; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 42); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 30)) << 12; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 8)) << 34; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 42); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 28)) << 14; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 6)) << 36; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 42); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 26)) << 16; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 4)) << 38; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 42); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 24)) << 18; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 2)) << 40; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 42); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 22)) << 20; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 42); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 0)) << 42; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 42); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 20)) << 22; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 42); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 40)) << 2; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 18)) << 24; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 42); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 38)) << 4; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 16)) << 26; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 42); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 36)) << 6; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 14)) << 28; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 42); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 34)) << 8; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 12)) << 30; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 42); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 32)) << 10; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 10)) << 32; + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 42); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 30)) << 12; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 8)) << 34; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 42); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 28)) << 14; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 6)) << 36; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 42); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 26)) << 16; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 38]; + tmp |= (src & MASK(uint64_t, 4)) << 38; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 42); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 39]; + tmp |= (src & MASK(uint64_t, 24)) << 18; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 40]; + tmp |= (src & MASK(uint64_t, 2)) << 40; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 42); + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 41]; + tmp |= (src & MASK(uint64_t, 22)) << 20; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 42); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<43>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 43); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 21); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 22)) << 21; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 42); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 1)) << 42; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 43); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 23)) << 20; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 41); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 2)) << 41; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 43); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 19); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 24)) << 19; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 3)) << 40; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 43); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 25)) << 18; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 39); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 4)) << 39; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 43); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 17); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 26)) << 17; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 5)) << 38; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 43); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 27)) << 16; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 37); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 6)) << 37; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 43); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 15); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 28)) << 15; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 7)) << 36; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 43); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 29)) << 14; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 35); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 8)) << 35; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 43); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 30)) << 13; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 9)) << 34; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 43); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 31)) << 12; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 33); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 10)) << 33; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 43); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 32)) << 11; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 11)) << 32; + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 43); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 33)) << 10; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 31); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 12)) << 31; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 43); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 34)) << 9; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 13)) << 30; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 43); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 35)) << 8; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 29); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 14)) << 29; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 43); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 36)) << 7; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 15)) << 28; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 43); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 37)) << 6; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 27); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 16)) << 27; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 43); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 38)) << 5; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 17)) << 26; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 43); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 39)) << 4; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 25); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 18)) << 25; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 43); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 40)) << 3; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 38]; + tmp |= (src & MASK(uint64_t, 19)) << 24; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 43); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 39]; + tmp |= (src & MASK(uint64_t, 41)) << 2; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 23); + src = in[lane + LANE_COUNT * 40]; + tmp |= (src & MASK(uint64_t, 20)) << 23; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 43); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 41]; + tmp |= (src & MASK(uint64_t, 42)) << 1; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 42]; + tmp |= (src & MASK(uint64_t, 21)) << 22; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 43); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<44>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 44); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 24)) << 20; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 4)) << 40; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 44); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 28)) << 16; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 8)) << 36; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 44); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 32)) << 12; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 12)) << 32; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 44); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 36)) << 8; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 16)) << 28; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 44); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 40)) << 4; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 20)) << 24; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 0)) << 44; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 44); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 24)) << 20; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 4)) << 40; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 44); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 28)) << 16; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 8)) << 36; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 44); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 32)) << 12; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 12)) << 32; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 44); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 36)) << 8; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 16)) << 28; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 44); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 40)) << 4; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 20)) << 24; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 0)) << 44; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 44); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 24)) << 20; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 4)) << 40; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 44); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 28)) << 16; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 8)) << 36; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 44); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 32)) << 12; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 12)) << 32; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 44); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 36)) << 8; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 16)) << 28; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 44); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 40)) << 4; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 20)) << 24; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 0)) << 44; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 44); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 24)) << 20; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 4)) << 40; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 44); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 28)) << 16; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 8)) << 36; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 44); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 38]; + tmp |= (src & MASK(uint64_t, 32)) << 12; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 39]; + tmp |= (src & MASK(uint64_t, 12)) << 32; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 44); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 40]; + tmp |= (src & MASK(uint64_t, 36)) << 8; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 41]; + tmp |= (src & MASK(uint64_t, 16)) << 28; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 44); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 42]; + tmp |= (src & MASK(uint64_t, 40)) << 4; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 43]; + tmp |= (src & MASK(uint64_t, 20)) << 24; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<45>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 45); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 19); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 26)) << 19; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 7)) << 38; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 45); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 33)) << 12; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 31); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 14)) << 31; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 45); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 40)) << 5; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 21)) << 24; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 43); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 2)) << 43; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 45); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 17); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 28)) << 17; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 9)) << 36; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 45); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 35)) << 10; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 29); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 16)) << 29; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 45); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 42)) << 3; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 23)) << 22; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 41); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 4)) << 41; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 45); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 15); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 30)) << 15; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 11)) << 34; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 45); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 37)) << 8; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 27); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 18)) << 27; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 45); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 44)) << 1; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 25)) << 20; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 39); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 6)) << 39; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 45); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 32)) << 13; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 13)) << 32; + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 45); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 39)) << 6; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 25); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 20)) << 25; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 1)) << 44; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 45); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 27)) << 18; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 37); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 8)) << 37; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 45); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 34)) << 11; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 15)) << 30; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 45); + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 41)) << 4; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 23); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 22)) << 23; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 42); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 3)) << 42; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 45); + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 29)) << 16; + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 35); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 10)) << 35; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 45); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 36)) << 9; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 17)) << 28; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 45); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 38]; + tmp |= (src & MASK(uint64_t, 43)) << 2; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 21); + src = in[lane + LANE_COUNT * 39]; + tmp |= (src & MASK(uint64_t, 24)) << 21; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 40]; + tmp |= (src & MASK(uint64_t, 5)) << 40; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 45); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 41]; + tmp |= (src & MASK(uint64_t, 31)) << 14; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 33); + src = in[lane + LANE_COUNT * 42]; + tmp |= (src & MASK(uint64_t, 12)) << 33; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 45); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 43]; + tmp |= (src & MASK(uint64_t, 38)) << 7; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 44]; + tmp |= (src & MASK(uint64_t, 19)) << 26; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 45); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<46>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 46); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 28)) << 18; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 10)) << 36; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 46); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 38)) << 8; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 20)) << 26; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 2)) << 44; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 46); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 30)) << 16; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 12)) << 34; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 46); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 40)) << 6; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 22)) << 24; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 42); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 4)) << 42; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 46); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 32)) << 14; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 14)) << 32; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 46); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 42)) << 4; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 24)) << 22; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 6)) << 40; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 46); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 34)) << 12; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 16)) << 30; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 46); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 44)) << 2; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 26)) << 20; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 8)) << 38; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 46); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 36)) << 10; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 18)) << 28; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 46); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 0)) << 46; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 46); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 28)) << 18; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 10)) << 36; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 46); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 38)) << 8; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 20)) << 26; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 2)) << 44; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 46); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 30)) << 16; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 12)) << 34; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 46); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 40)) << 6; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 22)) << 24; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 42); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 4)) << 42; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 46); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 32)) << 14; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 14)) << 32; + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 46); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 42)) << 4; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 24)) << 22; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 38]; + tmp |= (src & MASK(uint64_t, 6)) << 40; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 46); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 39]; + tmp |= (src & MASK(uint64_t, 34)) << 12; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 40]; + tmp |= (src & MASK(uint64_t, 16)) << 30; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 46); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 41]; + tmp |= (src & MASK(uint64_t, 44)) << 2; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 42]; + tmp |= (src & MASK(uint64_t, 26)) << 20; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 43]; + tmp |= (src & MASK(uint64_t, 8)) << 38; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 46); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 44]; + tmp |= (src & MASK(uint64_t, 36)) << 10; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 45]; + tmp |= (src & MASK(uint64_t, 18)) << 28; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 46); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<47>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 47); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 17); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 30)) << 17; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 13)) << 34; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 47); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 43)) << 4; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 21); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 26)) << 21; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 9)) << 38; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 47); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 39)) << 8; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 25); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 22)) << 25; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 42); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 5)) << 42; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 47); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 35)) << 12; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 29); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 18)) << 29; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 46); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 1)) << 46; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 47); + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 31)) << 16; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 33); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 14)) << 33; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 47); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 44)) << 3; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 27)) << 20; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 37); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 10)) << 37; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 47); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 40)) << 7; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 23)) << 24; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 41); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 6)) << 41; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 47); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 36)) << 11; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 19)) << 28; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 45); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 2)) << 45; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 47); + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 15); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 32)) << 15; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 15)) << 32; + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 47); + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 45)) << 2; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 19); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 28)) << 19; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 11)) << 36; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 47); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 41)) << 6; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 23); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 24)) << 23; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 7)) << 40; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 47); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 37)) << 10; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 27); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 20)) << 27; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 3)) << 44; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 47); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 33)) << 14; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 31); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 16)) << 31; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 47); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 46)) << 1; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 29)) << 18; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 35); + src = in[lane + LANE_COUNT * 38]; + tmp |= (src & MASK(uint64_t, 12)) << 35; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 47); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 39]; + tmp |= (src & MASK(uint64_t, 42)) << 5; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 40]; + tmp |= (src & MASK(uint64_t, 25)) << 22; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 39); + src = in[lane + LANE_COUNT * 41]; + tmp |= (src & MASK(uint64_t, 8)) << 39; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 47); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 42]; + tmp |= (src & MASK(uint64_t, 38)) << 9; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 43]; + tmp |= (src & MASK(uint64_t, 21)) << 26; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 43); + src = in[lane + LANE_COUNT * 44]; + tmp |= (src & MASK(uint64_t, 4)) << 43; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 47); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + src = in[lane + LANE_COUNT * 45]; + tmp |= (src & MASK(uint64_t, 34)) << 13; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 46]; + tmp |= (src & MASK(uint64_t, 17)) << 30; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 47); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<48>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 48); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 32)) << 16; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 16)) << 32; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 0)) << 48; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 48); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 32)) << 16; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 16)) << 32; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 0)) << 48; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 48); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 32)) << 16; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 16)) << 32; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 0)) << 48; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 48); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 32)) << 16; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 16)) << 32; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 0)) << 48; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 48); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 32)) << 16; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 16)) << 32; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 0)) << 48; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 48); + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 32)) << 16; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 16)) << 32; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 0)) << 48; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 48); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 32)) << 16; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 16)) << 32; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 0)) << 48; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 48); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 32)) << 16; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 16)) << 32; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 0)) << 48; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 48); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 32)) << 16; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 16)) << 32; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 0)) << 48; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 48); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 32)) << 16; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 16)) << 32; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 0)) << 48; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 48); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 32)) << 16; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 16)) << 32; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 0)) << 48; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 48); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 32)) << 16; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 16)) << 32; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 0)) << 48; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 48); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 32)) << 16; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 38]; + tmp |= (src & MASK(uint64_t, 16)) << 32; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 39]; + tmp |= (src & MASK(uint64_t, 0)) << 48; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 48); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 40]; + tmp |= (src & MASK(uint64_t, 32)) << 16; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 41]; + tmp |= (src & MASK(uint64_t, 16)) << 32; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 42]; + tmp |= (src & MASK(uint64_t, 0)) << 48; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 48); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 43]; + tmp |= (src & MASK(uint64_t, 32)) << 16; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 44]; + tmp |= (src & MASK(uint64_t, 16)) << 32; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 45]; + tmp |= (src & MASK(uint64_t, 0)) << 48; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 48); + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 46]; + tmp |= (src & MASK(uint64_t, 32)) << 16; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 47]; + tmp |= (src & MASK(uint64_t, 16)) << 32; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<49>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 49); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 15); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 34)) << 15; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 19)) << 30; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 45); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 4)) << 45; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 49); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 38)) << 11; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 23)) << 26; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 41); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 8)) << 41; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 49); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 42)) << 7; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 27)) << 22; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 37); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 12)) << 37; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 49); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 46)) << 3; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 31)) << 18; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 33); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 16)) << 33; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 1)) << 48; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 49); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 35)) << 14; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 29); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 20)) << 29; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 5)) << 44; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 49); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 39)) << 10; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 25); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 24)) << 25; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 9)) << 40; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 49); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 43)) << 6; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 21); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 28)) << 21; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 13)) << 36; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 49); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 47)) << 2; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 17); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 32)) << 17; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 17)) << 32; + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 47); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 2)) << 47; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 49); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 36)) << 13; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 21)) << 28; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 43); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 6)) << 43; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 49); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 40)) << 9; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 25)) << 24; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 39); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 10)) << 39; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 49); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 44)) << 5; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 29)) << 20; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 35); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 14)) << 35; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 49); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 48)) << 1; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 33)) << 16; + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 31); + src = in[lane + LANE_COUNT * 38]; + tmp |= (src & MASK(uint64_t, 18)) << 31; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 46); + src = in[lane + LANE_COUNT * 39]; + tmp |= (src & MASK(uint64_t, 3)) << 46; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 49); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 40]; + tmp |= (src & MASK(uint64_t, 37)) << 12; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 27); + src = in[lane + LANE_COUNT * 41]; + tmp |= (src & MASK(uint64_t, 22)) << 27; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 42); + src = in[lane + LANE_COUNT * 42]; + tmp |= (src & MASK(uint64_t, 7)) << 42; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 49); + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 43]; + tmp |= (src & MASK(uint64_t, 41)) << 8; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 23); + src = in[lane + LANE_COUNT * 44]; + tmp |= (src & MASK(uint64_t, 26)) << 23; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 45]; + tmp |= (src & MASK(uint64_t, 11)) << 38; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 49); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 46]; + tmp |= (src & MASK(uint64_t, 45)) << 4; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 19); + src = in[lane + LANE_COUNT * 47]; + tmp |= (src & MASK(uint64_t, 30)) << 19; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 48]; + tmp |= (src & MASK(uint64_t, 15)) << 34; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 49); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<50>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 50); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 36)) << 14; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 22)) << 28; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 42); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 8)) << 42; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 50); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 44)) << 6; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 30)) << 20; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 16)) << 34; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 2)) << 48; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 50); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 38)) << 12; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 24)) << 26; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 10)) << 40; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 50); + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 46)) << 4; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 32)) << 18; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 18)) << 32; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 46); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 4)) << 46; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 50); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 40)) << 10; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 26)) << 24; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 12)) << 38; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 50); + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 48)) << 2; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 34)) << 16; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 20)) << 30; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 6)) << 44; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 50); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 42)) << 8; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 28)) << 22; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 14)) << 36; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 50); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 0)) << 50; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 50); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 36)) << 14; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 22)) << 28; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 42); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 8)) << 42; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 50); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 44)) << 6; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 30)) << 20; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 16)) << 34; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 2)) << 48; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 50); + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 38)) << 12; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 24)) << 26; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 10)) << 40; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 50); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 46)) << 4; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 32)) << 18; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 38]; + tmp |= (src & MASK(uint64_t, 18)) << 32; + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 46); + src = in[lane + LANE_COUNT * 39]; + tmp |= (src & MASK(uint64_t, 4)) << 46; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 50); + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 40]; + tmp |= (src & MASK(uint64_t, 40)) << 10; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 41]; + tmp |= (src & MASK(uint64_t, 26)) << 24; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 42]; + tmp |= (src & MASK(uint64_t, 12)) << 38; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 50); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 43]; + tmp |= (src & MASK(uint64_t, 48)) << 2; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 44]; + tmp |= (src & MASK(uint64_t, 34)) << 16; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 45]; + tmp |= (src & MASK(uint64_t, 20)) << 30; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 46]; + tmp |= (src & MASK(uint64_t, 6)) << 44; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 50); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 47]; + tmp |= (src & MASK(uint64_t, 42)) << 8; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 48]; + tmp |= (src & MASK(uint64_t, 28)) << 22; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 49]; + tmp |= (src & MASK(uint64_t, 14)) << 36; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 50); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<51>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 51); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 38)) << 13; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 25)) << 26; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 39); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 12)) << 39; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 51); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 50)) << 1; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 37)) << 14; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 27); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 24)) << 27; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 11)) << 40; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 51); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 49)) << 2; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 15); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 36)) << 15; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 23)) << 28; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 41); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 10)) << 41; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 51); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 48)) << 3; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 35)) << 16; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 29); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 22)) << 29; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 42); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 9)) << 42; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 51); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 47)) << 4; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 17); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 34)) << 17; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 21)) << 30; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 43); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 8)) << 43; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 51); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 46)) << 5; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 33)) << 18; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 31); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 20)) << 31; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 7)) << 44; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 51); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 45)) << 6; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 19); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 32)) << 19; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 19)) << 32; + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 45); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 6)) << 45; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 51); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 44)) << 7; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 31)) << 20; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 33); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 18)) << 33; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 46); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 5)) << 46; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 51); + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 43)) << 8; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 21); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 30)) << 21; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 17)) << 34; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 47); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 4)) << 47; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 51); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 42)) << 9; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 29)) << 22; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 35); + src = in[lane + LANE_COUNT * 38]; + tmp |= (src & MASK(uint64_t, 16)) << 35; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 39]; + tmp |= (src & MASK(uint64_t, 3)) << 48; + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 51); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 40]; + tmp |= (src & MASK(uint64_t, 41)) << 10; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 23); + src = in[lane + LANE_COUNT * 41]; + tmp |= (src & MASK(uint64_t, 28)) << 23; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 42]; + tmp |= (src & MASK(uint64_t, 15)) << 36; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 49); + src = in[lane + LANE_COUNT * 43]; + tmp |= (src & MASK(uint64_t, 2)) << 49; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 51); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 44]; + tmp |= (src & MASK(uint64_t, 40)) << 11; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 45]; + tmp |= (src & MASK(uint64_t, 27)) << 24; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 37); + src = in[lane + LANE_COUNT * 46]; + tmp |= (src & MASK(uint64_t, 14)) << 37; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 50); + src = in[lane + LANE_COUNT * 47]; + tmp |= (src & MASK(uint64_t, 1)) << 50; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 51); + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 48]; + tmp |= (src & MASK(uint64_t, 39)) << 12; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 25); + src = in[lane + LANE_COUNT * 49]; + tmp |= (src & MASK(uint64_t, 26)) << 25; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 50]; + tmp |= (src & MASK(uint64_t, 13)) << 38; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 51); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<52>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 52); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 40)) << 12; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 28)) << 24; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 16)) << 36; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 4)) << 48; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 52); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 44)) << 8; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 32)) << 20; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 20)) << 32; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 8)) << 44; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 52); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 48)) << 4; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 36)) << 16; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 24)) << 28; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 12)) << 40; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 52); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 0)) << 52; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 52); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 40)) << 12; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 28)) << 24; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 16)) << 36; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 4)) << 48; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 52); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 44)) << 8; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 32)) << 20; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 20)) << 32; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 8)) << 44; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 52); + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 48)) << 4; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 36)) << 16; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 24)) << 28; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 12)) << 40; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 52); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 0)) << 52; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 52); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 40)) << 12; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 28)) << 24; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 16)) << 36; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 4)) << 48; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 52); + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 44)) << 8; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 32)) << 20; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 20)) << 32; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 8)) << 44; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 52); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 48)) << 4; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 36)) << 16; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 24)) << 28; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 38]; + tmp |= (src & MASK(uint64_t, 12)) << 40; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 52); + src = in[lane + LANE_COUNT * 39]; + tmp |= (src & MASK(uint64_t, 0)) << 52; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 52); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 40]; + tmp |= (src & MASK(uint64_t, 40)) << 12; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 41]; + tmp |= (src & MASK(uint64_t, 28)) << 24; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 42]; + tmp |= (src & MASK(uint64_t, 16)) << 36; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 43]; + tmp |= (src & MASK(uint64_t, 4)) << 48; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 52); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 44]; + tmp |= (src & MASK(uint64_t, 44)) << 8; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 45]; + tmp |= (src & MASK(uint64_t, 32)) << 20; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 46]; + tmp |= (src & MASK(uint64_t, 20)) << 32; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 47]; + tmp |= (src & MASK(uint64_t, 8)) << 44; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 52); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 48]; + tmp |= (src & MASK(uint64_t, 48)) << 4; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 49]; + tmp |= (src & MASK(uint64_t, 36)) << 16; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 50]; + tmp |= (src & MASK(uint64_t, 24)) << 28; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 51]; + tmp |= (src & MASK(uint64_t, 12)) << 40; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 52); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<53>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 53); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 42)) << 11; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 31)) << 22; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 33); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 20)) << 33; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 9)) << 44; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 53); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 51)) << 2; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 40)) << 13; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 29)) << 24; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 35); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 18)) << 35; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 46); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 7)) << 46; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 53); + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 49)) << 4; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 15); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 38)) << 15; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 27)) << 26; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 37); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 16)) << 37; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 5)) << 48; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 53); + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 47)) << 6; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 17); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 36)) << 17; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 25)) << 28; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 39); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 14)) << 39; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 50); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 3)) << 50; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 53); + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 45)) << 8; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 19); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 34)) << 19; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 23)) << 30; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 41); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 12)) << 41; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 52); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 1)) << 52; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 53); + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 43)) << 10; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 21); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 32)) << 21; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 21)) << 32; + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 43); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 10)) << 43; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 53); + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 52)) << 1; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 41)) << 12; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 23); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 30)) << 23; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 19)) << 34; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 45); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 8)) << 45; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 53); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 50)) << 3; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 39)) << 14; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 25); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 28)) << 25; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 17)) << 36; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 47); + src = in[lane + LANE_COUNT * 38]; + tmp |= (src & MASK(uint64_t, 6)) << 47; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 53); + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 39]; + tmp |= (src & MASK(uint64_t, 48)) << 5; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 40]; + tmp |= (src & MASK(uint64_t, 37)) << 16; + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 27); + src = in[lane + LANE_COUNT * 41]; + tmp |= (src & MASK(uint64_t, 26)) << 27; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 42]; + tmp |= (src & MASK(uint64_t, 15)) << 38; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 49); + src = in[lane + LANE_COUNT * 43]; + tmp |= (src & MASK(uint64_t, 4)) << 49; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 53); + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 44]; + tmp |= (src & MASK(uint64_t, 46)) << 7; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 45]; + tmp |= (src & MASK(uint64_t, 35)) << 18; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 29); + src = in[lane + LANE_COUNT * 46]; + tmp |= (src & MASK(uint64_t, 24)) << 29; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 47]; + tmp |= (src & MASK(uint64_t, 13)) << 40; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 51); + src = in[lane + LANE_COUNT * 48]; + tmp |= (src & MASK(uint64_t, 2)) << 51; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 53); + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 49]; + tmp |= (src & MASK(uint64_t, 44)) << 9; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 50]; + tmp |= (src & MASK(uint64_t, 33)) << 20; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 31); + src = in[lane + LANE_COUNT * 51]; + tmp |= (src & MASK(uint64_t, 22)) << 31; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 42); + src = in[lane + LANE_COUNT * 52]; + tmp |= (src & MASK(uint64_t, 11)) << 42; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 53); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<54>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 54); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 44)) << 10; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 34)) << 20; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 24)) << 30; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 14)) << 40; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 50); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 4)) << 50; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 54); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 48)) << 6; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 38)) << 16; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 28)) << 26; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 18)) << 36; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 46); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 8)) << 46; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 54); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 52)) << 2; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 42)) << 12; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 32)) << 22; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 22)) << 32; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 42); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 12)) << 42; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 52); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 2)) << 52; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 54); + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 46)) << 8; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 36)) << 18; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 26)) << 28; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 16)) << 38; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 6)) << 48; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 54); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 50)) << 4; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 40)) << 14; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 30)) << 24; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 20)) << 34; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 10)) << 44; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 54); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 0)) << 54; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 54); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 44)) << 10; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 34)) << 20; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 24)) << 30; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 14)) << 40; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 50); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 4)) << 50; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 54); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 48)) << 6; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 38)) << 16; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 28)) << 26; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 18)) << 36; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 46); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 8)) << 46; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 54); + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 38]; + tmp |= (src & MASK(uint64_t, 52)) << 2; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 39]; + tmp |= (src & MASK(uint64_t, 42)) << 12; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 40]; + tmp |= (src & MASK(uint64_t, 32)) << 22; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 41]; + tmp |= (src & MASK(uint64_t, 22)) << 32; + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 42); + src = in[lane + LANE_COUNT * 42]; + tmp |= (src & MASK(uint64_t, 12)) << 42; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 52); + src = in[lane + LANE_COUNT * 43]; + tmp |= (src & MASK(uint64_t, 2)) << 52; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 54); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 44]; + tmp |= (src & MASK(uint64_t, 46)) << 8; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 45]; + tmp |= (src & MASK(uint64_t, 36)) << 18; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 46]; + tmp |= (src & MASK(uint64_t, 26)) << 28; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 47]; + tmp |= (src & MASK(uint64_t, 16)) << 38; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 48]; + tmp |= (src & MASK(uint64_t, 6)) << 48; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 54); + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 49]; + tmp |= (src & MASK(uint64_t, 50)) << 4; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 50]; + tmp |= (src & MASK(uint64_t, 40)) << 14; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 51]; + tmp |= (src & MASK(uint64_t, 30)) << 24; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 52]; + tmp |= (src & MASK(uint64_t, 20)) << 34; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 53]; + tmp |= (src & MASK(uint64_t, 10)) << 44; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 54); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<55>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 55); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 46)) << 9; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 37)) << 18; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 27); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 28)) << 27; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 19)) << 36; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 45); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 10)) << 45; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 54); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 1)) << 54; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 55); + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 47)) << 8; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 17); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 38)) << 17; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 29)) << 26; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 35); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 20)) << 35; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 11)) << 44; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 53); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 2)) << 53; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 55); + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 48)) << 7; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 39)) << 16; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 25); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 30)) << 25; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 21)) << 34; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 43); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 12)) << 43; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 52); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 3)) << 52; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 55); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 49)) << 6; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 15); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 40)) << 15; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 31)) << 24; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 33); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 22)) << 33; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 42); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 13)) << 42; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 51); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 4)) << 51; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 55); + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 50)) << 5; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 41)) << 14; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 23); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 32)) << 23; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 23)) << 32; + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 41); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 14)) << 41; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 50); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 5)) << 50; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 55); + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 51)) << 4; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 42)) << 13; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 33)) << 22; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 31); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 24)) << 31; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 15)) << 40; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 49); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 6)) << 49; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 55); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 52)) << 3; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 38]; + tmp |= (src & MASK(uint64_t, 43)) << 12; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 21); + src = in[lane + LANE_COUNT * 39]; + tmp |= (src & MASK(uint64_t, 34)) << 21; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 40]; + tmp |= (src & MASK(uint64_t, 25)) << 30; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 39); + src = in[lane + LANE_COUNT * 41]; + tmp |= (src & MASK(uint64_t, 16)) << 39; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 42]; + tmp |= (src & MASK(uint64_t, 7)) << 48; + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 55); + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 43]; + tmp |= (src & MASK(uint64_t, 53)) << 2; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 44]; + tmp |= (src & MASK(uint64_t, 44)) << 11; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 45]; + tmp |= (src & MASK(uint64_t, 35)) << 20; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 29); + src = in[lane + LANE_COUNT * 46]; + tmp |= (src & MASK(uint64_t, 26)) << 29; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 47]; + tmp |= (src & MASK(uint64_t, 17)) << 38; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 47); + src = in[lane + LANE_COUNT * 48]; + tmp |= (src & MASK(uint64_t, 8)) << 47; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 55); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 49]; + tmp |= (src & MASK(uint64_t, 54)) << 1; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 50]; + tmp |= (src & MASK(uint64_t, 45)) << 10; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 19); + src = in[lane + LANE_COUNT * 51]; + tmp |= (src & MASK(uint64_t, 36)) << 19; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 52]; + tmp |= (src & MASK(uint64_t, 27)) << 28; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 37); + src = in[lane + LANE_COUNT * 53]; + tmp |= (src & MASK(uint64_t, 18)) << 37; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 46); + src = in[lane + LANE_COUNT * 54]; + tmp |= (src & MASK(uint64_t, 9)) << 46; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 55); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<56>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 56); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 48)) << 8; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 40)) << 16; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 32)) << 24; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 24)) << 32; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 16)) << 40; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 8)) << 48; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 56); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 0)) << 56; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 56); + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 48)) << 8; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 40)) << 16; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 32)) << 24; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 24)) << 32; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 16)) << 40; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 8)) << 48; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 56); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 0)) << 56; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 56); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 48)) << 8; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 40)) << 16; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 32)) << 24; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 24)) << 32; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 16)) << 40; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 8)) << 48; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 56); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 0)) << 56; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 56); + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 48)) << 8; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 40)) << 16; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 32)) << 24; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 24)) << 32; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 16)) << 40; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 8)) << 48; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 56); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 0)) << 56; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 56); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 48)) << 8; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 40)) << 16; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 32)) << 24; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 24)) << 32; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 16)) << 40; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 8)) << 48; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 56); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 0)) << 56; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 56); + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 48)) << 8; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 40)) << 16; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 38]; + tmp |= (src & MASK(uint64_t, 32)) << 24; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 39]; + tmp |= (src & MASK(uint64_t, 24)) << 32; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 40]; + tmp |= (src & MASK(uint64_t, 16)) << 40; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 41]; + tmp |= (src & MASK(uint64_t, 8)) << 48; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 56); + src = in[lane + LANE_COUNT * 42]; + tmp |= (src & MASK(uint64_t, 0)) << 56; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 56); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 43]; + tmp |= (src & MASK(uint64_t, 48)) << 8; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 44]; + tmp |= (src & MASK(uint64_t, 40)) << 16; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 45]; + tmp |= (src & MASK(uint64_t, 32)) << 24; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 46]; + tmp |= (src & MASK(uint64_t, 24)) << 32; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 47]; + tmp |= (src & MASK(uint64_t, 16)) << 40; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 48]; + tmp |= (src & MASK(uint64_t, 8)) << 48; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 56); + src = in[lane + LANE_COUNT * 49]; + tmp |= (src & MASK(uint64_t, 0)) << 56; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 56); + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 50]; + tmp |= (src & MASK(uint64_t, 48)) << 8; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 51]; + tmp |= (src & MASK(uint64_t, 40)) << 16; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 52]; + tmp |= (src & MASK(uint64_t, 32)) << 24; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 53]; + tmp |= (src & MASK(uint64_t, 24)) << 32; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 54]; + tmp |= (src & MASK(uint64_t, 16)) << 40; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 55]; + tmp |= (src & MASK(uint64_t, 8)) << 48; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 56); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<57>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 57); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 50)) << 7; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 43)) << 14; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 21); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 36)) << 21; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 29)) << 28; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 35); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 22)) << 35; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 42); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 15)) << 42; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 49); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 8)) << 49; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 56); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 1)) << 56; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 57); + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 51)) << 6; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 44)) << 13; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 37)) << 20; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 27); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 30)) << 27; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 23)) << 34; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 41); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 16)) << 41; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 9)) << 48; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 55); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 2)) << 55; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 57); + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 52)) << 5; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 45)) << 12; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 19); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 38)) << 19; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 31)) << 26; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 33); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 24)) << 33; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 17)) << 40; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 47); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 10)) << 47; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 54); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 3)) << 54; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 57); + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 53)) << 4; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 46)) << 11; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 39)) << 18; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 25); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 32)) << 25; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 25)) << 32; + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 39); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 18)) << 39; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 46); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 11)) << 46; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 53); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 4)) << 53; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 57); + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 54)) << 3; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 47)) << 10; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 17); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 40)) << 17; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 33)) << 24; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 31); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 26)) << 31; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 38]; + tmp |= (src & MASK(uint64_t, 19)) << 38; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 45); + src = in[lane + LANE_COUNT * 39]; + tmp |= (src & MASK(uint64_t, 12)) << 45; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 52); + src = in[lane + LANE_COUNT * 40]; + tmp |= (src & MASK(uint64_t, 5)) << 52; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 57); + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 41]; + tmp |= (src & MASK(uint64_t, 55)) << 2; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 42]; + tmp |= (src & MASK(uint64_t, 48)) << 9; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 43]; + tmp |= (src & MASK(uint64_t, 41)) << 16; + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 23); + src = in[lane + LANE_COUNT * 44]; + tmp |= (src & MASK(uint64_t, 34)) << 23; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 45]; + tmp |= (src & MASK(uint64_t, 27)) << 30; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 37); + src = in[lane + LANE_COUNT * 46]; + tmp |= (src & MASK(uint64_t, 20)) << 37; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 47]; + tmp |= (src & MASK(uint64_t, 13)) << 44; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 51); + src = in[lane + LANE_COUNT * 48]; + tmp |= (src & MASK(uint64_t, 6)) << 51; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 57); + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 49]; + tmp |= (src & MASK(uint64_t, 56)) << 1; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 50]; + tmp |= (src & MASK(uint64_t, 49)) << 8; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 15); + src = in[lane + LANE_COUNT * 51]; + tmp |= (src & MASK(uint64_t, 42)) << 15; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 52]; + tmp |= (src & MASK(uint64_t, 35)) << 22; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 29); + src = in[lane + LANE_COUNT * 53]; + tmp |= (src & MASK(uint64_t, 28)) << 29; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 54]; + tmp |= (src & MASK(uint64_t, 21)) << 36; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 43); + src = in[lane + LANE_COUNT * 55]; + tmp |= (src & MASK(uint64_t, 14)) << 43; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 50); + src = in[lane + LANE_COUNT * 56]; + tmp |= (src & MASK(uint64_t, 7)) << 50; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 57); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<58>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 58); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 52)) << 6; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 46)) << 12; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 40)) << 18; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 34)) << 24; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 28)) << 30; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 22)) << 36; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 42); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 16)) << 42; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 10)) << 48; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 54); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 4)) << 54; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 58); + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 56)) << 2; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 50)) << 8; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 44)) << 14; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 38)) << 20; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 32)) << 26; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 26)) << 32; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 20)) << 38; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 14)) << 44; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 50); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 8)) << 50; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 56); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 2)) << 56; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 58); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 54)) << 4; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 48)) << 10; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 42)) << 16; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 36)) << 22; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 30)) << 28; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 24)) << 34; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 18)) << 40; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 46); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 12)) << 46; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 52); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 6)) << 52; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 58); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 0)) << 58; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 58); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 52)) << 6; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 46)) << 12; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 40)) << 18; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 34)) << 24; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 28)) << 30; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 22)) << 36; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 42); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 16)) << 42; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 10)) << 48; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 54); + src = in[lane + LANE_COUNT * 38]; + tmp |= (src & MASK(uint64_t, 4)) << 54; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 58); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 39]; + tmp |= (src & MASK(uint64_t, 56)) << 2; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 40]; + tmp |= (src & MASK(uint64_t, 50)) << 8; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 41]; + tmp |= (src & MASK(uint64_t, 44)) << 14; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 42]; + tmp |= (src & MASK(uint64_t, 38)) << 20; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 43]; + tmp |= (src & MASK(uint64_t, 32)) << 26; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 44]; + tmp |= (src & MASK(uint64_t, 26)) << 32; + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 45]; + tmp |= (src & MASK(uint64_t, 20)) << 38; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 46]; + tmp |= (src & MASK(uint64_t, 14)) << 44; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 50); + src = in[lane + LANE_COUNT * 47]; + tmp |= (src & MASK(uint64_t, 8)) << 50; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 56); + src = in[lane + LANE_COUNT * 48]; + tmp |= (src & MASK(uint64_t, 2)) << 56; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 58); + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 49]; + tmp |= (src & MASK(uint64_t, 54)) << 4; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 50]; + tmp |= (src & MASK(uint64_t, 48)) << 10; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 51]; + tmp |= (src & MASK(uint64_t, 42)) << 16; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 52]; + tmp |= (src & MASK(uint64_t, 36)) << 22; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 53]; + tmp |= (src & MASK(uint64_t, 30)) << 28; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 54]; + tmp |= (src & MASK(uint64_t, 24)) << 34; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 55]; + tmp |= (src & MASK(uint64_t, 18)) << 40; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 46); + src = in[lane + LANE_COUNT * 56]; + tmp |= (src & MASK(uint64_t, 12)) << 46; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 52); + src = in[lane + LANE_COUNT * 57]; + tmp |= (src & MASK(uint64_t, 6)) << 52; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 58); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<59>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 59); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 54)) << 5; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 49)) << 10; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 15); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 44)) << 15; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 39)) << 20; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 25); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 34)) << 25; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 29)) << 30; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 35); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 24)) << 35; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 19)) << 40; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 45); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 14)) << 45; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 50); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 9)) << 50; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 55); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 4)) << 55; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 59); + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 58)) << 1; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 53)) << 6; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 48)) << 11; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 43)) << 16; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 21); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 38)) << 21; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 33)) << 26; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 31); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 28)) << 31; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 23)) << 36; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 41); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 18)) << 41; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 46); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 13)) << 46; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 51); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 8)) << 51; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 56); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 3)) << 56; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 59); + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 57)) << 2; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 52)) << 7; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 47)) << 12; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 17); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 42)) << 17; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 37)) << 22; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 27); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 32)) << 27; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 27)) << 32; + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 37); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 22)) << 37; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 42); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 17)) << 42; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 47); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 12)) << 47; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 52); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 7)) << 52; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 57); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 2)) << 57; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 59); + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 56)) << 3; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 51)) << 8; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + src = in[lane + LANE_COUNT * 38]; + tmp |= (src & MASK(uint64_t, 46)) << 13; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 39]; + tmp |= (src & MASK(uint64_t, 41)) << 18; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 23); + src = in[lane + LANE_COUNT * 40]; + tmp |= (src & MASK(uint64_t, 36)) << 23; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 41]; + tmp |= (src & MASK(uint64_t, 31)) << 28; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 33); + src = in[lane + LANE_COUNT * 42]; + tmp |= (src & MASK(uint64_t, 26)) << 33; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 43]; + tmp |= (src & MASK(uint64_t, 21)) << 38; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 43); + src = in[lane + LANE_COUNT * 44]; + tmp |= (src & MASK(uint64_t, 16)) << 43; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 45]; + tmp |= (src & MASK(uint64_t, 11)) << 48; + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 53); + src = in[lane + LANE_COUNT * 46]; + tmp |= (src & MASK(uint64_t, 6)) << 53; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 58); + src = in[lane + LANE_COUNT * 47]; + tmp |= (src & MASK(uint64_t, 1)) << 58; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 59); + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 48]; + tmp |= (src & MASK(uint64_t, 55)) << 4; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 49]; + tmp |= (src & MASK(uint64_t, 50)) << 9; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 50]; + tmp |= (src & MASK(uint64_t, 45)) << 14; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 19); + src = in[lane + LANE_COUNT * 51]; + tmp |= (src & MASK(uint64_t, 40)) << 19; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 52]; + tmp |= (src & MASK(uint64_t, 35)) << 24; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 29); + src = in[lane + LANE_COUNT * 53]; + tmp |= (src & MASK(uint64_t, 30)) << 29; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 54]; + tmp |= (src & MASK(uint64_t, 25)) << 34; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 39); + src = in[lane + LANE_COUNT * 55]; + tmp |= (src & MASK(uint64_t, 20)) << 39; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 56]; + tmp |= (src & MASK(uint64_t, 15)) << 44; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 49); + src = in[lane + LANE_COUNT * 57]; + tmp |= (src & MASK(uint64_t, 10)) << 49; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 54); + src = in[lane + LANE_COUNT * 58]; + tmp |= (src & MASK(uint64_t, 5)) << 54; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 59); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<60>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 60); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 56)) << 4; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 52)) << 8; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 48)) << 12; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 44)) << 16; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 40)) << 20; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 36)) << 24; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 32)) << 28; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 28)) << 32; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 24)) << 36; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 20)) << 40; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 16)) << 44; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 12)) << 48; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 52); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 8)) << 52; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 56); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 4)) << 56; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 60); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 0)) << 60; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 60); + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 56)) << 4; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 52)) << 8; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 48)) << 12; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 44)) << 16; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 40)) << 20; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 36)) << 24; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 32)) << 28; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 28)) << 32; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 24)) << 36; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 20)) << 40; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 16)) << 44; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 12)) << 48; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 52); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 8)) << 52; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 56); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 4)) << 56; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 60); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 0)) << 60; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 60); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 56)) << 4; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 52)) << 8; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 48)) << 12; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 44)) << 16; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 40)) << 20; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 36)) << 24; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 32)) << 28; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 38]; + tmp |= (src & MASK(uint64_t, 28)) << 32; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 39]; + tmp |= (src & MASK(uint64_t, 24)) << 36; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 40]; + tmp |= (src & MASK(uint64_t, 20)) << 40; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 41]; + tmp |= (src & MASK(uint64_t, 16)) << 44; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 42]; + tmp |= (src & MASK(uint64_t, 12)) << 48; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 52); + src = in[lane + LANE_COUNT * 43]; + tmp |= (src & MASK(uint64_t, 8)) << 52; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 56); + src = in[lane + LANE_COUNT * 44]; + tmp |= (src & MASK(uint64_t, 4)) << 56; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 60); + src = in[lane + LANE_COUNT * 45]; + tmp |= (src & MASK(uint64_t, 0)) << 60; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 60); + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 46]; + tmp |= (src & MASK(uint64_t, 56)) << 4; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 47]; + tmp |= (src & MASK(uint64_t, 52)) << 8; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 48]; + tmp |= (src & MASK(uint64_t, 48)) << 12; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 49]; + tmp |= (src & MASK(uint64_t, 44)) << 16; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 50]; + tmp |= (src & MASK(uint64_t, 40)) << 20; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 51]; + tmp |= (src & MASK(uint64_t, 36)) << 24; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 52]; + tmp |= (src & MASK(uint64_t, 32)) << 28; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 53]; + tmp |= (src & MASK(uint64_t, 28)) << 32; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 54]; + tmp |= (src & MASK(uint64_t, 24)) << 36; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 55]; + tmp |= (src & MASK(uint64_t, 20)) << 40; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 56]; + tmp |= (src & MASK(uint64_t, 16)) << 44; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 57]; + tmp |= (src & MASK(uint64_t, 12)) << 48; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 52); + src = in[lane + LANE_COUNT * 58]; + tmp |= (src & MASK(uint64_t, 8)) << 52; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 56); + src = in[lane + LANE_COUNT * 59]; + tmp |= (src & MASK(uint64_t, 4)) << 56; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 60); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<61>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 61); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 58)) << 3; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 55)) << 6; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 52)) << 9; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 49)) << 12; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 15); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 46)) << 15; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 43)) << 18; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 21); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 40)) << 21; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 37)) << 24; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 27); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 34)) << 27; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 31)) << 30; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 33); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 28)) << 33; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 25)) << 36; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 39); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 22)) << 39; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 42); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 19)) << 42; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 45); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 16)) << 45; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 13)) << 48; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 51); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 10)) << 51; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 54); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 7)) << 54; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 57); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 4)) << 57; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 60); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 1)) << 60; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 61); + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 59)) << 2; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 56)) << 5; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 53)) << 8; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 50)) << 11; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 47)) << 14; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 17); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 44)) << 17; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 41)) << 20; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 23); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 38)) << 23; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 35)) << 26; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 29); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 32)) << 29; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 29)) << 32; + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 35); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 26)) << 35; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 23)) << 38; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 41); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 20)) << 41; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 17)) << 44; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 47); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 14)) << 47; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 50); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 11)) << 50; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 53); + src = in[lane + LANE_COUNT * 38]; + tmp |= (src & MASK(uint64_t, 8)) << 53; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 56); + src = in[lane + LANE_COUNT * 39]; + tmp |= (src & MASK(uint64_t, 5)) << 56; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 59); + src = in[lane + LANE_COUNT * 40]; + tmp |= (src & MASK(uint64_t, 2)) << 59; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 61); + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 41]; + tmp |= (src & MASK(uint64_t, 60)) << 1; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 42]; + tmp |= (src & MASK(uint64_t, 57)) << 4; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 43]; + tmp |= (src & MASK(uint64_t, 54)) << 7; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 44]; + tmp |= (src & MASK(uint64_t, 51)) << 10; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + src = in[lane + LANE_COUNT * 45]; + tmp |= (src & MASK(uint64_t, 48)) << 13; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 46]; + tmp |= (src & MASK(uint64_t, 45)) << 16; + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 19); + src = in[lane + LANE_COUNT * 47]; + tmp |= (src & MASK(uint64_t, 42)) << 19; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 48]; + tmp |= (src & MASK(uint64_t, 39)) << 22; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 25); + src = in[lane + LANE_COUNT * 49]; + tmp |= (src & MASK(uint64_t, 36)) << 25; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 50]; + tmp |= (src & MASK(uint64_t, 33)) << 28; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 31); + src = in[lane + LANE_COUNT * 51]; + tmp |= (src & MASK(uint64_t, 30)) << 31; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 52]; + tmp |= (src & MASK(uint64_t, 27)) << 34; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 37); + src = in[lane + LANE_COUNT * 53]; + tmp |= (src & MASK(uint64_t, 24)) << 37; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 54]; + tmp |= (src & MASK(uint64_t, 21)) << 40; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 43); + src = in[lane + LANE_COUNT * 55]; + tmp |= (src & MASK(uint64_t, 18)) << 43; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 46); + src = in[lane + LANE_COUNT * 56]; + tmp |= (src & MASK(uint64_t, 15)) << 46; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 49); + src = in[lane + LANE_COUNT * 57]; + tmp |= (src & MASK(uint64_t, 12)) << 49; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 52); + src = in[lane + LANE_COUNT * 58]; + tmp |= (src & MASK(uint64_t, 9)) << 52; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 55); + src = in[lane + LANE_COUNT * 59]; + tmp |= (src & MASK(uint64_t, 6)) << 55; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 58); + src = in[lane + LANE_COUNT * 60]; + tmp |= (src & MASK(uint64_t, 3)) << 58; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 61); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<62>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 62); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 60)) << 2; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 58)) << 4; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 56)) << 6; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 54)) << 8; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 52)) << 10; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 50)) << 12; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 48)) << 14; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 46)) << 16; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 44)) << 18; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 42)) << 20; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 40)) << 22; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 38)) << 24; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 36)) << 26; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 34)) << 28; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 32)) << 30; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 30)) << 32; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 28)) << 34; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 26)) << 36; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 24)) << 38; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 22)) << 40; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 42); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 20)) << 42; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 18)) << 44; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 46); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 16)) << 46; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 14)) << 48; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 50); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 12)) << 50; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 52); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 10)) << 52; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 54); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 8)) << 54; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 56); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 6)) << 56; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 58); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 4)) << 58; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 60); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 2)) << 60; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 62); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 0)) << 62; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint64_t, 62); + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 60)) << 2; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 58)) << 4; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 56)) << 6; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 54)) << 8; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 52)) << 10; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 50)) << 12; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 38]; + tmp |= (src & MASK(uint64_t, 48)) << 14; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 39]; + tmp |= (src & MASK(uint64_t, 46)) << 16; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 40]; + tmp |= (src & MASK(uint64_t, 44)) << 18; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 41]; + tmp |= (src & MASK(uint64_t, 42)) << 20; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 42]; + tmp |= (src & MASK(uint64_t, 40)) << 22; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 43]; + tmp |= (src & MASK(uint64_t, 38)) << 24; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 44]; + tmp |= (src & MASK(uint64_t, 36)) << 26; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 45]; + tmp |= (src & MASK(uint64_t, 34)) << 28; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 46]; + tmp |= (src & MASK(uint64_t, 32)) << 30; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 47]; + tmp |= (src & MASK(uint64_t, 30)) << 32; + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 48]; + tmp |= (src & MASK(uint64_t, 28)) << 34; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 49]; + tmp |= (src & MASK(uint64_t, 26)) << 36; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 50]; + tmp |= (src & MASK(uint64_t, 24)) << 38; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 51]; + tmp |= (src & MASK(uint64_t, 22)) << 40; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 42); + src = in[lane + LANE_COUNT * 52]; + tmp |= (src & MASK(uint64_t, 20)) << 42; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 53]; + tmp |= (src & MASK(uint64_t, 18)) << 44; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 46); + src = in[lane + LANE_COUNT * 54]; + tmp |= (src & MASK(uint64_t, 16)) << 46; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 55]; + tmp |= (src & MASK(uint64_t, 14)) << 48; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 50); + src = in[lane + LANE_COUNT * 56]; + tmp |= (src & MASK(uint64_t, 12)) << 50; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 52); + src = in[lane + LANE_COUNT * 57]; + tmp |= (src & MASK(uint64_t, 10)) << 52; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 54); + src = in[lane + LANE_COUNT * 58]; + tmp |= (src & MASK(uint64_t, 8)) << 54; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 56); + src = in[lane + LANE_COUNT * 59]; + tmp |= (src & MASK(uint64_t, 6)) << 56; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 58); + src = in[lane + LANE_COUNT * 60]; + tmp |= (src & MASK(uint64_t, 4)) << 58; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 60); + src = in[lane + LANE_COUNT * 61]; + tmp |= (src & MASK(uint64_t, 2)) << 60; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 62); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<63>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint64_t src; + uint64_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint64_t, 63); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 63) & MASK(uint64_t, 1); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint64_t, 62)) << 1; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 62) & MASK(uint64_t, 2); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint64_t, 61)) << 2; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 61) & MASK(uint64_t, 3); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint64_t, 60)) << 3; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 60) & MASK(uint64_t, 4); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint64_t, 59)) << 4; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 59) & MASK(uint64_t, 5); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint64_t, 58)) << 5; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 58) & MASK(uint64_t, 6); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint64_t, 57)) << 6; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 57) & MASK(uint64_t, 7); + src = in[lane + LANE_COUNT * 7]; + tmp |= (src & MASK(uint64_t, 56)) << 7; + out[INDEX(7, lane)] = tmp + reference; + tmp = (src >> 56) & MASK(uint64_t, 8); + src = in[lane + LANE_COUNT * 8]; + tmp |= (src & MASK(uint64_t, 55)) << 8; + out[INDEX(8, lane)] = tmp + reference; + tmp = (src >> 55) & MASK(uint64_t, 9); + src = in[lane + LANE_COUNT * 9]; + tmp |= (src & MASK(uint64_t, 54)) << 9; + out[INDEX(9, lane)] = tmp + reference; + tmp = (src >> 54) & MASK(uint64_t, 10); + src = in[lane + LANE_COUNT * 10]; + tmp |= (src & MASK(uint64_t, 53)) << 10; + out[INDEX(10, lane)] = tmp + reference; + tmp = (src >> 53) & MASK(uint64_t, 11); + src = in[lane + LANE_COUNT * 11]; + tmp |= (src & MASK(uint64_t, 52)) << 11; + out[INDEX(11, lane)] = tmp + reference; + tmp = (src >> 52) & MASK(uint64_t, 12); + src = in[lane + LANE_COUNT * 12]; + tmp |= (src & MASK(uint64_t, 51)) << 12; + out[INDEX(12, lane)] = tmp + reference; + tmp = (src >> 51) & MASK(uint64_t, 13); + src = in[lane + LANE_COUNT * 13]; + tmp |= (src & MASK(uint64_t, 50)) << 13; + out[INDEX(13, lane)] = tmp + reference; + tmp = (src >> 50) & MASK(uint64_t, 14); + src = in[lane + LANE_COUNT * 14]; + tmp |= (src & MASK(uint64_t, 49)) << 14; + out[INDEX(14, lane)] = tmp + reference; + tmp = (src >> 49) & MASK(uint64_t, 15); + src = in[lane + LANE_COUNT * 15]; + tmp |= (src & MASK(uint64_t, 48)) << 15; + out[INDEX(15, lane)] = tmp + reference; + tmp = (src >> 48) & MASK(uint64_t, 16); + src = in[lane + LANE_COUNT * 16]; + tmp |= (src & MASK(uint64_t, 47)) << 16; + out[INDEX(16, lane)] = tmp + reference; + tmp = (src >> 47) & MASK(uint64_t, 17); + src = in[lane + LANE_COUNT * 17]; + tmp |= (src & MASK(uint64_t, 46)) << 17; + out[INDEX(17, lane)] = tmp + reference; + tmp = (src >> 46) & MASK(uint64_t, 18); + src = in[lane + LANE_COUNT * 18]; + tmp |= (src & MASK(uint64_t, 45)) << 18; + out[INDEX(18, lane)] = tmp + reference; + tmp = (src >> 45) & MASK(uint64_t, 19); + src = in[lane + LANE_COUNT * 19]; + tmp |= (src & MASK(uint64_t, 44)) << 19; + out[INDEX(19, lane)] = tmp + reference; + tmp = (src >> 44) & MASK(uint64_t, 20); + src = in[lane + LANE_COUNT * 20]; + tmp |= (src & MASK(uint64_t, 43)) << 20; + out[INDEX(20, lane)] = tmp + reference; + tmp = (src >> 43) & MASK(uint64_t, 21); + src = in[lane + LANE_COUNT * 21]; + tmp |= (src & MASK(uint64_t, 42)) << 21; + out[INDEX(21, lane)] = tmp + reference; + tmp = (src >> 42) & MASK(uint64_t, 22); + src = in[lane + LANE_COUNT * 22]; + tmp |= (src & MASK(uint64_t, 41)) << 22; + out[INDEX(22, lane)] = tmp + reference; + tmp = (src >> 41) & MASK(uint64_t, 23); + src = in[lane + LANE_COUNT * 23]; + tmp |= (src & MASK(uint64_t, 40)) << 23; + out[INDEX(23, lane)] = tmp + reference; + tmp = (src >> 40) & MASK(uint64_t, 24); + src = in[lane + LANE_COUNT * 24]; + tmp |= (src & MASK(uint64_t, 39)) << 24; + out[INDEX(24, lane)] = tmp + reference; + tmp = (src >> 39) & MASK(uint64_t, 25); + src = in[lane + LANE_COUNT * 25]; + tmp |= (src & MASK(uint64_t, 38)) << 25; + out[INDEX(25, lane)] = tmp + reference; + tmp = (src >> 38) & MASK(uint64_t, 26); + src = in[lane + LANE_COUNT * 26]; + tmp |= (src & MASK(uint64_t, 37)) << 26; + out[INDEX(26, lane)] = tmp + reference; + tmp = (src >> 37) & MASK(uint64_t, 27); + src = in[lane + LANE_COUNT * 27]; + tmp |= (src & MASK(uint64_t, 36)) << 27; + out[INDEX(27, lane)] = tmp + reference; + tmp = (src >> 36) & MASK(uint64_t, 28); + src = in[lane + LANE_COUNT * 28]; + tmp |= (src & MASK(uint64_t, 35)) << 28; + out[INDEX(28, lane)] = tmp + reference; + tmp = (src >> 35) & MASK(uint64_t, 29); + src = in[lane + LANE_COUNT * 29]; + tmp |= (src & MASK(uint64_t, 34)) << 29; + out[INDEX(29, lane)] = tmp + reference; + tmp = (src >> 34) & MASK(uint64_t, 30); + src = in[lane + LANE_COUNT * 30]; + tmp |= (src & MASK(uint64_t, 33)) << 30; + out[INDEX(30, lane)] = tmp + reference; + tmp = (src >> 33) & MASK(uint64_t, 31); + src = in[lane + LANE_COUNT * 31]; + tmp |= (src & MASK(uint64_t, 32)) << 31; + out[INDEX(31, lane)] = tmp + reference; + tmp = (src >> 32) & MASK(uint64_t, 32); + src = in[lane + LANE_COUNT * 32]; + tmp |= (src & MASK(uint64_t, 31)) << 32; + out[INDEX(32, lane)] = tmp + reference; + tmp = (src >> 31) & MASK(uint64_t, 33); + src = in[lane + LANE_COUNT * 33]; + tmp |= (src & MASK(uint64_t, 30)) << 33; + out[INDEX(33, lane)] = tmp + reference; + tmp = (src >> 30) & MASK(uint64_t, 34); + src = in[lane + LANE_COUNT * 34]; + tmp |= (src & MASK(uint64_t, 29)) << 34; + out[INDEX(34, lane)] = tmp + reference; + tmp = (src >> 29) & MASK(uint64_t, 35); + src = in[lane + LANE_COUNT * 35]; + tmp |= (src & MASK(uint64_t, 28)) << 35; + out[INDEX(35, lane)] = tmp + reference; + tmp = (src >> 28) & MASK(uint64_t, 36); + src = in[lane + LANE_COUNT * 36]; + tmp |= (src & MASK(uint64_t, 27)) << 36; + out[INDEX(36, lane)] = tmp + reference; + tmp = (src >> 27) & MASK(uint64_t, 37); + src = in[lane + LANE_COUNT * 37]; + tmp |= (src & MASK(uint64_t, 26)) << 37; + out[INDEX(37, lane)] = tmp + reference; + tmp = (src >> 26) & MASK(uint64_t, 38); + src = in[lane + LANE_COUNT * 38]; + tmp |= (src & MASK(uint64_t, 25)) << 38; + out[INDEX(38, lane)] = tmp + reference; + tmp = (src >> 25) & MASK(uint64_t, 39); + src = in[lane + LANE_COUNT * 39]; + tmp |= (src & MASK(uint64_t, 24)) << 39; + out[INDEX(39, lane)] = tmp + reference; + tmp = (src >> 24) & MASK(uint64_t, 40); + src = in[lane + LANE_COUNT * 40]; + tmp |= (src & MASK(uint64_t, 23)) << 40; + out[INDEX(40, lane)] = tmp + reference; + tmp = (src >> 23) & MASK(uint64_t, 41); + src = in[lane + LANE_COUNT * 41]; + tmp |= (src & MASK(uint64_t, 22)) << 41; + out[INDEX(41, lane)] = tmp + reference; + tmp = (src >> 22) & MASK(uint64_t, 42); + src = in[lane + LANE_COUNT * 42]; + tmp |= (src & MASK(uint64_t, 21)) << 42; + out[INDEX(42, lane)] = tmp + reference; + tmp = (src >> 21) & MASK(uint64_t, 43); + src = in[lane + LANE_COUNT * 43]; + tmp |= (src & MASK(uint64_t, 20)) << 43; + out[INDEX(43, lane)] = tmp + reference; + tmp = (src >> 20) & MASK(uint64_t, 44); + src = in[lane + LANE_COUNT * 44]; + tmp |= (src & MASK(uint64_t, 19)) << 44; + out[INDEX(44, lane)] = tmp + reference; + tmp = (src >> 19) & MASK(uint64_t, 45); + src = in[lane + LANE_COUNT * 45]; + tmp |= (src & MASK(uint64_t, 18)) << 45; + out[INDEX(45, lane)] = tmp + reference; + tmp = (src >> 18) & MASK(uint64_t, 46); + src = in[lane + LANE_COUNT * 46]; + tmp |= (src & MASK(uint64_t, 17)) << 46; + out[INDEX(46, lane)] = tmp + reference; + tmp = (src >> 17) & MASK(uint64_t, 47); + src = in[lane + LANE_COUNT * 47]; + tmp |= (src & MASK(uint64_t, 16)) << 47; + out[INDEX(47, lane)] = tmp + reference; + tmp = (src >> 16) & MASK(uint64_t, 48); + src = in[lane + LANE_COUNT * 48]; + tmp |= (src & MASK(uint64_t, 15)) << 48; + out[INDEX(48, lane)] = tmp + reference; + tmp = (src >> 15) & MASK(uint64_t, 49); + src = in[lane + LANE_COUNT * 49]; + tmp |= (src & MASK(uint64_t, 14)) << 49; + out[INDEX(49, lane)] = tmp + reference; + tmp = (src >> 14) & MASK(uint64_t, 50); + src = in[lane + LANE_COUNT * 50]; + tmp |= (src & MASK(uint64_t, 13)) << 50; + out[INDEX(50, lane)] = tmp + reference; + tmp = (src >> 13) & MASK(uint64_t, 51); + src = in[lane + LANE_COUNT * 51]; + tmp |= (src & MASK(uint64_t, 12)) << 51; + out[INDEX(51, lane)] = tmp + reference; + tmp = (src >> 12) & MASK(uint64_t, 52); + src = in[lane + LANE_COUNT * 52]; + tmp |= (src & MASK(uint64_t, 11)) << 52; + out[INDEX(52, lane)] = tmp + reference; + tmp = (src >> 11) & MASK(uint64_t, 53); + src = in[lane + LANE_COUNT * 53]; + tmp |= (src & MASK(uint64_t, 10)) << 53; + out[INDEX(53, lane)] = tmp + reference; + tmp = (src >> 10) & MASK(uint64_t, 54); + src = in[lane + LANE_COUNT * 54]; + tmp |= (src & MASK(uint64_t, 9)) << 54; + out[INDEX(54, lane)] = tmp + reference; + tmp = (src >> 9) & MASK(uint64_t, 55); + src = in[lane + LANE_COUNT * 55]; + tmp |= (src & MASK(uint64_t, 8)) << 55; + out[INDEX(55, lane)] = tmp + reference; + tmp = (src >> 8) & MASK(uint64_t, 56); + src = in[lane + LANE_COUNT * 56]; + tmp |= (src & MASK(uint64_t, 7)) << 56; + out[INDEX(56, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint64_t, 57); + src = in[lane + LANE_COUNT * 57]; + tmp |= (src & MASK(uint64_t, 6)) << 57; + out[INDEX(57, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint64_t, 58); + src = in[lane + LANE_COUNT * 58]; + tmp |= (src & MASK(uint64_t, 5)) << 58; + out[INDEX(58, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint64_t, 59); + src = in[lane + LANE_COUNT * 59]; + tmp |= (src & MASK(uint64_t, 4)) << 59; + out[INDEX(59, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint64_t, 60); + src = in[lane + LANE_COUNT * 60]; + tmp |= (src & MASK(uint64_t, 3)) << 60; + out[INDEX(60, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint64_t, 61); + src = in[lane + LANE_COUNT * 61]; + tmp |= (src & MASK(uint64_t, 2)) << 61; + out[INDEX(61, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint64_t, 62); + src = in[lane + LANE_COUNT * 62]; + tmp |= (src & MASK(uint64_t, 1)) << 62; + out[INDEX(62, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint64_t, 63); + out[INDEX(63, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_64_lane<64>(const uint64_t *__restrict in, uint64_t *__restrict out, uint64_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + #pragma unroll + for (int row = 0; row < 64; row++) { + out[INDEX(row, lane)] = in[LANE_COUNT * row + lane] + reference; + } +} + +/// Runtime dispatch to the optimized lane decoder for the given bit width. +__device__ __noinline__ void bit_unpack_64_lane( + const uint64_t *__restrict in, + uint64_t *__restrict out, + uint64_t reference, + unsigned int lane, + uint32_t bit_width +) { + switch (bit_width) { + case 0: _bit_unpack_64_lane<0>(in, out, reference, lane); break; + case 1: _bit_unpack_64_lane<1>(in, out, reference, lane); break; + case 2: _bit_unpack_64_lane<2>(in, out, reference, lane); break; + case 3: _bit_unpack_64_lane<3>(in, out, reference, lane); break; + case 4: _bit_unpack_64_lane<4>(in, out, reference, lane); break; + case 5: _bit_unpack_64_lane<5>(in, out, reference, lane); break; + case 6: _bit_unpack_64_lane<6>(in, out, reference, lane); break; + case 7: _bit_unpack_64_lane<7>(in, out, reference, lane); break; + case 8: _bit_unpack_64_lane<8>(in, out, reference, lane); break; + case 9: _bit_unpack_64_lane<9>(in, out, reference, lane); break; + case 10: _bit_unpack_64_lane<10>(in, out, reference, lane); break; + case 11: _bit_unpack_64_lane<11>(in, out, reference, lane); break; + case 12: _bit_unpack_64_lane<12>(in, out, reference, lane); break; + case 13: _bit_unpack_64_lane<13>(in, out, reference, lane); break; + case 14: _bit_unpack_64_lane<14>(in, out, reference, lane); break; + case 15: _bit_unpack_64_lane<15>(in, out, reference, lane); break; + case 16: _bit_unpack_64_lane<16>(in, out, reference, lane); break; + case 17: _bit_unpack_64_lane<17>(in, out, reference, lane); break; + case 18: _bit_unpack_64_lane<18>(in, out, reference, lane); break; + case 19: _bit_unpack_64_lane<19>(in, out, reference, lane); break; + case 20: _bit_unpack_64_lane<20>(in, out, reference, lane); break; + case 21: _bit_unpack_64_lane<21>(in, out, reference, lane); break; + case 22: _bit_unpack_64_lane<22>(in, out, reference, lane); break; + case 23: _bit_unpack_64_lane<23>(in, out, reference, lane); break; + case 24: _bit_unpack_64_lane<24>(in, out, reference, lane); break; + case 25: _bit_unpack_64_lane<25>(in, out, reference, lane); break; + case 26: _bit_unpack_64_lane<26>(in, out, reference, lane); break; + case 27: _bit_unpack_64_lane<27>(in, out, reference, lane); break; + case 28: _bit_unpack_64_lane<28>(in, out, reference, lane); break; + case 29: _bit_unpack_64_lane<29>(in, out, reference, lane); break; + case 30: _bit_unpack_64_lane<30>(in, out, reference, lane); break; + case 31: _bit_unpack_64_lane<31>(in, out, reference, lane); break; + case 32: _bit_unpack_64_lane<32>(in, out, reference, lane); break; + case 33: _bit_unpack_64_lane<33>(in, out, reference, lane); break; + case 34: _bit_unpack_64_lane<34>(in, out, reference, lane); break; + case 35: _bit_unpack_64_lane<35>(in, out, reference, lane); break; + case 36: _bit_unpack_64_lane<36>(in, out, reference, lane); break; + case 37: _bit_unpack_64_lane<37>(in, out, reference, lane); break; + case 38: _bit_unpack_64_lane<38>(in, out, reference, lane); break; + case 39: _bit_unpack_64_lane<39>(in, out, reference, lane); break; + case 40: _bit_unpack_64_lane<40>(in, out, reference, lane); break; + case 41: _bit_unpack_64_lane<41>(in, out, reference, lane); break; + case 42: _bit_unpack_64_lane<42>(in, out, reference, lane); break; + case 43: _bit_unpack_64_lane<43>(in, out, reference, lane); break; + case 44: _bit_unpack_64_lane<44>(in, out, reference, lane); break; + case 45: _bit_unpack_64_lane<45>(in, out, reference, lane); break; + case 46: _bit_unpack_64_lane<46>(in, out, reference, lane); break; + case 47: _bit_unpack_64_lane<47>(in, out, reference, lane); break; + case 48: _bit_unpack_64_lane<48>(in, out, reference, lane); break; + case 49: _bit_unpack_64_lane<49>(in, out, reference, lane); break; + case 50: _bit_unpack_64_lane<50>(in, out, reference, lane); break; + case 51: _bit_unpack_64_lane<51>(in, out, reference, lane); break; + case 52: _bit_unpack_64_lane<52>(in, out, reference, lane); break; + case 53: _bit_unpack_64_lane<53>(in, out, reference, lane); break; + case 54: _bit_unpack_64_lane<54>(in, out, reference, lane); break; + case 55: _bit_unpack_64_lane<55>(in, out, reference, lane); break; + case 56: _bit_unpack_64_lane<56>(in, out, reference, lane); break; + case 57: _bit_unpack_64_lane<57>(in, out, reference, lane); break; + case 58: _bit_unpack_64_lane<58>(in, out, reference, lane); break; + case 59: _bit_unpack_64_lane<59>(in, out, reference, lane); break; + case 60: _bit_unpack_64_lane<60>(in, out, reference, lane); break; + case 61: _bit_unpack_64_lane<61>(in, out, reference, lane); break; + case 62: _bit_unpack_64_lane<62>(in, out, reference, lane); break; + case 63: _bit_unpack_64_lane<63>(in, out, reference, lane); break; + case 64: _bit_unpack_64_lane<64>(in, out, reference, lane); break; + } +} + diff --git a/vortex-cuda/kernels/src/bit_unpack_8.cu b/vortex-cuda/kernels/src/bit_unpack_8.cu index 13ef84a2b2f..b2fcfd26f04 100644 --- a/vortex-cuda/kernels/src/bit_unpack_8.cu +++ b/vortex-cuda/kernels/src/bit_unpack_8.cu @@ -1,342 +1,95 @@ // AUTO-GENERATED. Do not edit by hand! -#include -#include -#include -#include "fastlanes_common.cuh" +#include "bit_unpack_8_lanes.cuh" #include "patches.cuh" template -__device__ void _bit_unpack_8_lane(const uint8_t *__restrict in, uint8_t *__restrict out, uint8_t reference, unsigned int lane); - -template <> -__device__ void _bit_unpack_8_lane<0>(const uint8_t *__restrict in, uint8_t *__restrict out, uint8_t reference, unsigned int lane) { - #pragma unroll - for (int row = 0; row < 8; row++) { - out[INDEX(row, lane)] = reference; - } -} - -template <> -__device__ void _bit_unpack_8_lane<1>(const uint8_t *__restrict in, uint8_t *__restrict out, uint8_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 128; - uint8_t src; - uint8_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint8_t, 1); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint8_t, 1); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint8_t, 1); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint8_t, 1); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint8_t, 1); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint8_t, 1); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint8_t, 1); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint8_t, 1); - out[INDEX(7, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_8_lane<2>(const uint8_t *__restrict in, uint8_t *__restrict out, uint8_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 128; - uint8_t src; - uint8_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint8_t, 2); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint8_t, 2); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint8_t, 2); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint8_t, 2); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint8_t, 0)) << 2; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint8_t, 2); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint8_t, 2); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint8_t, 2); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint8_t, 2); - out[INDEX(7, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_8_lane<3>(const uint8_t *__restrict in, uint8_t *__restrict out, uint8_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 128; - uint8_t src; - uint8_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint8_t, 3); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint8_t, 3); - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint8_t, 2); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint8_t, 1)) << 2; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint8_t, 3); - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint8_t, 3); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint8_t, 1); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint8_t, 2)) << 1; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint8_t, 3); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint8_t, 3); - out[INDEX(7, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_8_lane<4>(const uint8_t *__restrict in, uint8_t *__restrict out, uint8_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 128; - uint8_t src; - uint8_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint8_t, 4); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint8_t, 4); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint8_t, 0)) << 4; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint8_t, 4); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint8_t, 4); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint8_t, 0)) << 4; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint8_t, 4); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint8_t, 4); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint8_t, 0)) << 4; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint8_t, 4); - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint8_t, 4); - out[INDEX(7, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_8_lane<5>(const uint8_t *__restrict in, uint8_t *__restrict out, uint8_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 128; - uint8_t src; - uint8_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint8_t, 5); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint8_t, 3); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint8_t, 2)) << 3; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint8_t, 5); - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint8_t, 1); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint8_t, 4)) << 1; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint8_t, 4); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint8_t, 1)) << 4; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint8_t, 5); - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint8_t, 2); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint8_t, 3)) << 2; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint8_t, 5); - out[INDEX(7, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_8_lane<6>(const uint8_t *__restrict in, uint8_t *__restrict out, uint8_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 128; - uint8_t src; - uint8_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint8_t, 6); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint8_t, 2); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint8_t, 4)) << 2; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint8_t, 4); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint8_t, 2)) << 4; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint8_t, 6); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint8_t, 0)) << 6; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 0) & MASK(uint8_t, 6); - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint8_t, 2); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint8_t, 4)) << 2; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint8_t, 4); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint8_t, 2)) << 4; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint8_t, 6); - out[INDEX(7, lane)] = tmp + reference; -} - -template <> -__device__ void _bit_unpack_8_lane<7>(const uint8_t *__restrict in, uint8_t *__restrict out, uint8_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 128; - uint8_t src; - uint8_t tmp; - src = in[lane]; - tmp = (src >> 0) & MASK(uint8_t, 7); - out[INDEX(0, lane)] = tmp + reference; - tmp = (src >> 7) & MASK(uint8_t, 1); - src = in[lane + LANE_COUNT * 1]; - tmp |= (src & MASK(uint8_t, 6)) << 1; - out[INDEX(1, lane)] = tmp + reference; - tmp = (src >> 6) & MASK(uint8_t, 2); - src = in[lane + LANE_COUNT * 2]; - tmp |= (src & MASK(uint8_t, 5)) << 2; - out[INDEX(2, lane)] = tmp + reference; - tmp = (src >> 5) & MASK(uint8_t, 3); - src = in[lane + LANE_COUNT * 3]; - tmp |= (src & MASK(uint8_t, 4)) << 3; - out[INDEX(3, lane)] = tmp + reference; - tmp = (src >> 4) & MASK(uint8_t, 4); - src = in[lane + LANE_COUNT * 4]; - tmp |= (src & MASK(uint8_t, 3)) << 4; - out[INDEX(4, lane)] = tmp + reference; - tmp = (src >> 3) & MASK(uint8_t, 5); - src = in[lane + LANE_COUNT * 5]; - tmp |= (src & MASK(uint8_t, 2)) << 5; - out[INDEX(5, lane)] = tmp + reference; - tmp = (src >> 2) & MASK(uint8_t, 6); - src = in[lane + LANE_COUNT * 6]; - tmp |= (src & MASK(uint8_t, 1)) << 6; - out[INDEX(6, lane)] = tmp + reference; - tmp = (src >> 1) & MASK(uint8_t, 7); - out[INDEX(7, lane)] = tmp + reference; -} +__device__ void _bit_unpack_8_device(const uint8_t *__restrict in, uint8_t *__restrict out, uint8_t reference, int thread_idx, GPUPatches& patches) { + __shared__ uint8_t shared_out[FL_CHUNK]; -template <> -__device__ void _bit_unpack_8_lane<8>(const uint8_t *__restrict in, uint8_t *__restrict out, uint8_t reference, unsigned int lane) { - unsigned int LANE_COUNT = 128; + // Step 1: Unpack into shared memory #pragma unroll - for (int row = 0; row < 8; row++) { - out[INDEX(row, lane)] = in[LANE_COUNT * row + lane] + reference; + for (int i = 0; i < FL_LANES / 32; i++) { + _bit_unpack_8_lane(in, shared_out, reference, thread_idx * (FL_LANES / 32) + i); } -} + __syncwarp(); -/// Runtime dispatch to the optimized lane decoder for the given bit width. -__device__ inline void bit_unpack_8_lane( - const uint8_t *__restrict in, - uint8_t *__restrict out, - uint8_t reference, - unsigned int lane, - uint32_t bit_width -) { - switch (bit_width) { - case 0: _bit_unpack_8_lane<0>(in, out, reference, lane); break; - case 1: _bit_unpack_8_lane<1>(in, out, reference, lane); break; - case 2: _bit_unpack_8_lane<2>(in, out, reference, lane); break; - case 3: _bit_unpack_8_lane<3>(in, out, reference, lane); break; - case 4: _bit_unpack_8_lane<4>(in, out, reference, lane); break; - case 5: _bit_unpack_8_lane<5>(in, out, reference, lane); break; - case 6: _bit_unpack_8_lane<6>(in, out, reference, lane); break; - case 7: _bit_unpack_8_lane<7>(in, out, reference, lane); break; - case 8: _bit_unpack_8_lane<8>(in, out, reference, lane); break; + // Step 2: Apply patches to shared memory in parallel + PatchesCursor cursor(patches, blockIdx.x, thread_idx, 32); + auto patch = cursor.next(); + while (patch.index != FL_CHUNK) { + shared_out[patch.index] = patch.value; + patch = cursor.next(); } -} + __syncwarp(); -template -__device__ void _bit_unpack_8_device(const uint8_t *__restrict in, uint8_t *__restrict out, uint8_t reference, int thread_idx, GPUPatches& patches) { - __shared__ uint8_t shared_out[1024]; + // Step 3: Copy to global memory #pragma unroll - for (int i = 0; i < 4; i++) { - _bit_unpack_8_lane(in, shared_out, reference, thread_idx * 4 + i); - } - __syncwarp(); - PatchesCursor cursor(patches, blockIdx.x, thread_idx, 32); - auto patch = cursor.next(); - for (int i = 0; i < 32; i++) { + for (int i = 0; i < FL_CHUNK / 32; i++) { auto idx = i * 32 + thread_idx; - if (idx == patch.index) { - out[idx] = patch.value; - patch = cursor.next(); - } else { - out[idx] = shared_out[idx]; - } + out[idx] = shared_out[idx]; } } extern "C" __global__ void bit_unpack_8_0bw_32t(const uint8_t *__restrict full_in, uint8_t *__restrict full_out, uint8_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 0 / sizeof(uint8_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 0)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_8_device<0>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_8_1bw_32t(const uint8_t *__restrict full_in, uint8_t *__restrict full_out, uint8_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 1 / sizeof(uint8_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 1)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_8_device<1>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_8_2bw_32t(const uint8_t *__restrict full_in, uint8_t *__restrict full_out, uint8_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 2 / sizeof(uint8_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 2)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_8_device<2>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_8_3bw_32t(const uint8_t *__restrict full_in, uint8_t *__restrict full_out, uint8_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 3 / sizeof(uint8_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 3)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_8_device<3>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_8_4bw_32t(const uint8_t *__restrict full_in, uint8_t *__restrict full_out, uint8_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 4 / sizeof(uint8_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 4)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_8_device<4>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_8_5bw_32t(const uint8_t *__restrict full_in, uint8_t *__restrict full_out, uint8_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 5 / sizeof(uint8_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 5)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_8_device<5>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_8_6bw_32t(const uint8_t *__restrict full_in, uint8_t *__restrict full_out, uint8_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 6 / sizeof(uint8_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 6)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_8_device<6>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_8_7bw_32t(const uint8_t *__restrict full_in, uint8_t *__restrict full_out, uint8_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 7 / sizeof(uint8_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 7)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_8_device<7>(in, out, reference, thread_idx, patches); } extern "C" __global__ void bit_unpack_8_8bw_32t(const uint8_t *__restrict full_in, uint8_t *__restrict full_out, uint8_t reference, GPUPatches patches) { int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * 8 / sizeof(uint8_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * 8)); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_8_device<8>(in, out, reference, thread_idx, patches); } diff --git a/vortex-cuda/kernels/src/bit_unpack_8_lanes.cuh b/vortex-cuda/kernels/src/bit_unpack_8_lanes.cuh new file mode 100644 index 00000000000..d9c3550cb18 --- /dev/null +++ b/vortex-cuda/kernels/src/bit_unpack_8_lanes.cuh @@ -0,0 +1,259 @@ +// AUTO-GENERATED. Do not edit by hand! +#pragma once + +#include +#include +#include +#include "fastlanes_common.cuh" + +template +__device__ void _bit_unpack_8_lane(const uint8_t *__restrict in, uint8_t *__restrict out, uint8_t reference, unsigned int lane); + +template <> +__device__ void _bit_unpack_8_lane<0>(const uint8_t *__restrict in, uint8_t *__restrict out, uint8_t reference, unsigned int lane) { + #pragma unroll + for (int row = 0; row < 8; row++) { + out[INDEX(row, lane)] = reference; + } +} + +template <> +__device__ void _bit_unpack_8_lane<1>(const uint8_t *__restrict in, uint8_t *__restrict out, uint8_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint8_t src; + uint8_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint8_t, 1); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint8_t, 1); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint8_t, 1); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint8_t, 1); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint8_t, 1); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint8_t, 1); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint8_t, 1); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint8_t, 1); + out[INDEX(7, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_8_lane<2>(const uint8_t *__restrict in, uint8_t *__restrict out, uint8_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint8_t src; + uint8_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint8_t, 2); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint8_t, 2); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint8_t, 2); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint8_t, 2); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint8_t, 0)) << 2; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint8_t, 2); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint8_t, 2); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint8_t, 2); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint8_t, 2); + out[INDEX(7, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_8_lane<3>(const uint8_t *__restrict in, uint8_t *__restrict out, uint8_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint8_t src; + uint8_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint8_t, 3); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint8_t, 3); + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint8_t, 2); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint8_t, 1)) << 2; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint8_t, 3); + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint8_t, 3); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint8_t, 1); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint8_t, 2)) << 1; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint8_t, 3); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint8_t, 3); + out[INDEX(7, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_8_lane<4>(const uint8_t *__restrict in, uint8_t *__restrict out, uint8_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint8_t src; + uint8_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint8_t, 4); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint8_t, 4); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint8_t, 0)) << 4; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint8_t, 4); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint8_t, 4); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint8_t, 0)) << 4; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint8_t, 4); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint8_t, 4); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint8_t, 0)) << 4; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint8_t, 4); + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint8_t, 4); + out[INDEX(7, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_8_lane<5>(const uint8_t *__restrict in, uint8_t *__restrict out, uint8_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint8_t src; + uint8_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint8_t, 5); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint8_t, 3); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint8_t, 2)) << 3; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint8_t, 5); + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint8_t, 1); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint8_t, 4)) << 1; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint8_t, 4); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint8_t, 1)) << 4; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint8_t, 5); + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint8_t, 2); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint8_t, 3)) << 2; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint8_t, 5); + out[INDEX(7, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_8_lane<6>(const uint8_t *__restrict in, uint8_t *__restrict out, uint8_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint8_t src; + uint8_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint8_t, 6); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint8_t, 2); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint8_t, 4)) << 2; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint8_t, 4); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint8_t, 2)) << 4; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint8_t, 6); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint8_t, 0)) << 6; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 0) & MASK(uint8_t, 6); + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint8_t, 2); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint8_t, 4)) << 2; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint8_t, 4); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint8_t, 2)) << 4; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint8_t, 6); + out[INDEX(7, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_8_lane<7>(const uint8_t *__restrict in, uint8_t *__restrict out, uint8_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + uint8_t src; + uint8_t tmp; + src = in[lane]; + tmp = (src >> 0) & MASK(uint8_t, 7); + out[INDEX(0, lane)] = tmp + reference; + tmp = (src >> 7) & MASK(uint8_t, 1); + src = in[lane + LANE_COUNT * 1]; + tmp |= (src & MASK(uint8_t, 6)) << 1; + out[INDEX(1, lane)] = tmp + reference; + tmp = (src >> 6) & MASK(uint8_t, 2); + src = in[lane + LANE_COUNT * 2]; + tmp |= (src & MASK(uint8_t, 5)) << 2; + out[INDEX(2, lane)] = tmp + reference; + tmp = (src >> 5) & MASK(uint8_t, 3); + src = in[lane + LANE_COUNT * 3]; + tmp |= (src & MASK(uint8_t, 4)) << 3; + out[INDEX(3, lane)] = tmp + reference; + tmp = (src >> 4) & MASK(uint8_t, 4); + src = in[lane + LANE_COUNT * 4]; + tmp |= (src & MASK(uint8_t, 3)) << 4; + out[INDEX(4, lane)] = tmp + reference; + tmp = (src >> 3) & MASK(uint8_t, 5); + src = in[lane + LANE_COUNT * 5]; + tmp |= (src & MASK(uint8_t, 2)) << 5; + out[INDEX(5, lane)] = tmp + reference; + tmp = (src >> 2) & MASK(uint8_t, 6); + src = in[lane + LANE_COUNT * 6]; + tmp |= (src & MASK(uint8_t, 1)) << 6; + out[INDEX(6, lane)] = tmp + reference; + tmp = (src >> 1) & MASK(uint8_t, 7); + out[INDEX(7, lane)] = tmp + reference; +} + +template <> +__device__ void _bit_unpack_8_lane<8>(const uint8_t *__restrict in, uint8_t *__restrict out, uint8_t reference, unsigned int lane) { + constexpr unsigned int LANE_COUNT = FL_LANES; + #pragma unroll + for (int row = 0; row < 8; row++) { + out[INDEX(row, lane)] = in[LANE_COUNT * row + lane] + reference; + } +} + +/// Runtime dispatch to the optimized lane decoder for the given bit width. +__device__ __noinline__ void bit_unpack_8_lane( + const uint8_t *__restrict in, + uint8_t *__restrict out, + uint8_t reference, + unsigned int lane, + uint32_t bit_width +) { + switch (bit_width) { + case 0: _bit_unpack_8_lane<0>(in, out, reference, lane); break; + case 1: _bit_unpack_8_lane<1>(in, out, reference, lane); break; + case 2: _bit_unpack_8_lane<2>(in, out, reference, lane); break; + case 3: _bit_unpack_8_lane<3>(in, out, reference, lane); break; + case 4: _bit_unpack_8_lane<4>(in, out, reference, lane); break; + case 5: _bit_unpack_8_lane<5>(in, out, reference, lane); break; + case 6: _bit_unpack_8_lane<6>(in, out, reference, lane); break; + case 7: _bit_unpack_8_lane<7>(in, out, reference, lane); break; + case 8: _bit_unpack_8_lane<8>(in, out, reference, lane); break; + } +} + diff --git a/vortex-cuda/kernels/src/dynamic_dispatch.cu b/vortex-cuda/kernels/src/dynamic_dispatch.cu index 29bf26efefb..0f2c4c157b9 100644 --- a/vortex-cuda/kernels/src/dynamic_dispatch.cu +++ b/vortex-cuda/kernels/src/dynamic_dispatch.cu @@ -39,9 +39,16 @@ // // ## Mixed-width support // -// LOAD sources from pending subtrees may have a narrower type than the -// output (e.g. u8 dict codes in a u32 plan). load_element() widens -// to T via static_cast — no separate widen kernel or smem intermediate. +// Dict codes, RunEnd ends, and other child arrays may have a narrower +// element type than the output T. Two mechanisms handle this: +// +// LOAD load_element() dispatches on the per-stage PTypeTag to +// read at the source's native width and static_cast to T. +// BITUNPACK bitunpack_typed() unpacks at the source's native width, +// then widens to T in-place via a backward scan +// (widen_inplace). The smem region is pre-allocated at +// max(source_width, T) bytes per element by the Rust plan +// builder, so the widen never overflows. #include #include @@ -52,6 +59,7 @@ #include "bit_unpack.cuh" #include "dynamic_dispatch.h" +#include "patches.cuh" #include "types.cuh" // ═══════════════════════════════════════════════════════════════════════════ @@ -109,8 +117,12 @@ __shared__ uint64_t runend_cursors[BLOCK_SIZE]; // ═══════════════════════════════════════════════════════════════════════════ /// Apply one scalar operation to N values in registers. +/// +/// `abs_pos` is the absolute output position of the first value to process. +/// It is used by scalar operations that apply patches, e.g. ALP. template -__device__ inline void scalar_op(T *values, const struct ScalarOp &op, char *__restrict smem) { +__device__ inline void +scalar_op(T *values, const struct ScalarOp &op, char *__restrict smem, uint64_t abs_pos = 0) { switch (op.op_code) { case ScalarOp::FOR: { const T ref = static_cast(op.params.frame_of_ref.reference); @@ -128,11 +140,57 @@ __device__ inline void scalar_op(T *values, const struct ScalarOp &op, char *__r break; } case ScalarOp::ALP: { - const float f = op.params.alp.f, e = op.params.alp.e; + if constexpr (sizeof(T) == 4) { + // The plan builder stores f32 F10/IF10 table entries as f64 + // in AlpParams. The round-trip f32→f64→f32 is exact per the + // C++ standard: [conv.fpprom] guarantees the widening is + // value-preserving, and [conv.double] guarantees the narrowing + // recovers the original value when it is exactly representable + // in the destination type (which it is, having originated as f32). + const float f = static_cast(op.params.alp.f); + const float e = static_cast(op.params.alp.e); #pragma unroll - for (uint32_t i = 0; i < N; ++i) { - float r = static_cast(static_cast(values[i])) * f * e; - values[i] = static_cast(__float_as_uint(r)); + for (uint32_t i = 0; i < N; ++i) { + float r = static_cast(static_cast(values[i])) * f * e; + values[i] = static_cast(__float_as_uint(r)); + } + } else if constexpr (sizeof(T) == 8) { + const double f = op.params.alp.f, e = op.params.alp.e; +#pragma unroll + for (uint32_t i = 0; i < N; ++i) { + double r = static_cast(static_cast(values[i])) * f * e; + // __double_as_longlong reinterprets f64 bits as int64, and + // static_cast to T (uint64_t) preserves the bit pattern + // under C++20's two's complement guarantee. + values[i] = static_cast(__double_as_longlong(r)); + } + } + // Apply ALP patches: override positions whose float value couldn't + // be reconstructed through the ALP encode/decode cycle. + // Per-value cursor — with a slice offset, a tile's N values can + // straddle two FL chunks, so each value needs its own lookup. + if (op.params.alp.patches_ptr != 0) { + const auto &patches = *reinterpret_cast(op.params.alp.patches_ptr); + // The sliced chunk_offsets array starts at original chunk + // (offset / FL_CHUNK). PatchesCursor indexes from 0, so + // subtract that base to get the index into chunk_offsets. + const uint32_t chunk_start = patches.offset / FL_CHUNK; +#pragma unroll + for (uint32_t i = 0; i < N; ++i) { + uint64_t my_pos = (N > 1) ? abs_pos + i * blockDim.x + threadIdx.x : abs_pos; + uint64_t orig = my_pos + patches.offset; + uint32_t chunk = static_cast(orig / FL_CHUNK) - chunk_start; + uint32_t within = static_cast(orig % FL_CHUNK); + PatchesCursor cursor(patches, chunk, 0, 1); + auto patch = cursor.next(); + while (patch.index != FL_CHUNK) { + if (patch.index == within) { + values[i] = patch.value; + break; + } + patch = cursor.next(); + } + } } break; } @@ -149,10 +207,50 @@ __device__ inline void scalar_op(T *values, const struct ScalarOp &op, char *__r } } +// ═══════════════════════════════════════════════════════════════════════════ +// Patches +// ═══════════════════════════════════════════════════════════════════════════ + +/// Scatter patches for a single chunk into `out` using PatchesCursor. +/// All threads in the block cooperate. Caller must issue __syncthreads() +/// afterward if other threads read from `out`. +template +__device__ __forceinline__ void +scatter_patches_chunk(const GPUPatches &patches, T *__restrict out, uint32_t chunk) { + PatchesCursor cursor(patches, chunk, threadIdx.x, blockDim.x); + auto patch = cursor.next(); + while (patch.index != FL_CHUNK) { + out[patch.index] = patch.value; + patch = cursor.next(); + } +} + // ═══════════════════════════════════════════════════════════════════════════ // Source ops // ═══════════════════════════════════════════════════════════════════════════ +/// Widen SOURCE-sized elements in shared memory to DESTINATION-sized in-place. +/// +/// A single warp performs the backward scan so that lockstep execution +/// guarantees every load at index i retires before the store at i, and +/// higher indices are already consumed. Using multiple warps would introduce +/// a cross-warp race: a fast warp writing dst[low] can clobber source +/// bytes that a slow warp has not yet read. +template +__device__ inline void widen_inplace(DESTINATION *dst, uint32_t len) { + if constexpr (sizeof(DESTINATION) <= sizeof(SOURCE)) { + return; + } + const SOURCE *src = reinterpret_cast(dst); + if (threadIdx.x < warpSize) { + for (int32_t i = static_cast(len) - 1 - static_cast(threadIdx.x); i >= 0; + i -= warpSize) { + dst[i] = static_cast(src[i]); + } + } + __syncthreads(); +} + /// FastLanes cooperative unpack — all threads in the block scatter-write /// decoded elements into `dst`. Caller must issue __syncthreads() before /// any thread reads from `dst`. @@ -162,11 +260,8 @@ __device__ inline void bitunpack(const T *__restrict packed, uint64_t chunk_start, uint32_t chunk_len, const struct SourceOp &src) { - constexpr uint32_t T_BITS = sizeof(T) * 8; - constexpr uint32_t FL_CHUNK = 1024; - constexpr uint32_t LANES = FL_CHUNK / T_BITS; const uint32_t bw = src.params.bitunpack.bit_width; - const uint32_t words_per_block = LANES * bw; + const uint32_t words_per_block = FL_LANES * bw; const uint32_t elem_off = src.params.bitunpack.element_offset; const uint32_t dst_off = (chunk_start + elem_off) % FL_CHUNK; const uint64_t first_block = (chunk_start + elem_off) / FL_CHUNK; @@ -177,9 +272,77 @@ __device__ inline void bitunpack(const T *__restrict packed, for (uint32_t c = 0; c < n_chunks; ++c) { const T *src_chunk = packed + (first_block + c) * words_per_block; T *chunk_dst = dst + c * FL_CHUNK; - for (uint32_t lane = threadIdx.x; lane < LANES; lane += blockDim.x) { + for (uint32_t lane = threadIdx.x; lane < FL_LANES; lane += blockDim.x) { bit_unpack_lane(src_chunk, chunk_dst, 0, lane, bw); } + // Apply BitPacked patches inline, matching the standalone kernel pattern. + if (src.params.bitunpack.patches_ptr != 0) { + __syncthreads(); + const auto &patches = *reinterpret_cast(src.params.bitunpack.patches_ptr); + scatter_patches_chunk(patches, chunk_dst, first_block + c); + } + } +} + +/// Dispatch bitunpack at the source's native element width, then widen +/// to T in-place so all downstream scalar ops and smem consumers see +/// T-sized elements. Falls back to the direct `bitunpack` path when +/// the source ptype already matches T. Issues __syncthreads() before +/// returning on all paths. +/// +/// Accepts explicit chunk_start / chunk_len so it works for both input +/// stages (full decode with chunk_start=0, chunk_len=stage.len) and +/// the output stage (tiled with varying chunk_start / chunk_len). +template +__device__ inline void bitunpack_typed(T *__restrict dst, + const void *__restrict packed, + uint64_t chunk_start, + uint32_t chunk_len, + const struct SourceOp &src, + PTypeTag source_ptype) { + // Fast path: source width matches T — no widening needed. + if (ptype_byte_width(source_ptype) == sizeof(T)) { + bitunpack(reinterpret_cast(packed), dst, chunk_start, chunk_len, src); + __syncthreads(); + return; + } + + // Compute total elements written by bitunpack (including alignment + // padding) so widen_inplace covers the full scratch region. + const uint32_t elem_off = src.params.bitunpack.element_offset; + const uint32_t dst_off = (chunk_start + elem_off) % FL_CHUNK; + const uint32_t n_chunks = (chunk_len + dst_off + FL_CHUNK - 1) / FL_CHUNK; + const uint32_t total_elems = n_chunks * FL_CHUNK; + + // Narrow source: unpack at native width, then widen to T. + switch (source_ptype) { + case PTYPE_U8: + case PTYPE_I8: { + auto *narrow = reinterpret_cast(dst); + bitunpack(reinterpret_cast(packed), narrow, chunk_start, chunk_len, src); + __syncthreads(); + widen_inplace(dst, total_elems); + break; + } + case PTYPE_U16: + case PTYPE_I16: { + auto *narrow = reinterpret_cast(dst); + bitunpack(reinterpret_cast(packed), narrow, chunk_start, chunk_len, src); + __syncthreads(); + widen_inplace(dst, total_elems); + break; + } + case PTYPE_U32: + case PTYPE_I32: + case PTYPE_F32: { + auto *narrow = reinterpret_cast(dst); + bitunpack(reinterpret_cast(packed), narrow, chunk_start, chunk_len, src); + __syncthreads(); + widen_inplace(dst, total_elems); + break; + } + default: + __builtin_unreachable(); } } @@ -279,17 +442,15 @@ __device__ void execute_output_stage(T *__restrict output, char *__restrict smem, uint64_t block_start, uint32_t block_len) { - constexpr uint32_t VALUES_PER_TILE = 32 / sizeof(T); + // Cap at 4 values per thread per tile to minimise register pressure. + constexpr uint32_t VALUES_PER_TILE = (32 / sizeof(T)) < 4 ? (32 / sizeof(T)) : 4; const uint32_t tile_size = blockDim.x * VALUES_PER_TILE; + const auto &src = stage.source; const void *raw_input = reinterpret_cast(stage.input_ptr); const PTypeTag ptype = stage.source_ptype; if (src.op_code == SourceOp::RUNEND) { - // Seed each thread's cursor with the run containing its first - // strided position. The RUNEND arm in source_op advances the - // cursor monotonically, so this avoids a full binary search on - // every element. const T *ends = reinterpret_cast(smem + src.params.runend.ends_smem_byte_offset); runend_cursors[threadIdx.x] = upper_bound(ends, src.params.runend.num_runs, @@ -300,23 +461,17 @@ __device__ void execute_output_stage(T *__restrict output, uint32_t chunk_len; const T *smem_src = nullptr; - // BITUNPACK uses smem scratch, so the outer loop advances one - // chunk at a time. LOAD, SEQUENCE, and RUNEND need no smem - // scratch, so chunk_len = block_len (single outer iteration); - // tiling happens in the inner tile_idx loop. if (src.op_code == SourceOp::BITUNPACK) { chunk_len = bitunpack_tile_len(stage, block_len, elem_idx); T *scratch = reinterpret_cast(smem + stage.smem_byte_offset); - bitunpack(reinterpret_cast(stage.input_ptr), - scratch, - block_start + elem_idx, - chunk_len, - src); - constexpr uint32_t FL_CHUNK = 1024; // FastLanes chunk size + bitunpack_typed(scratch, + reinterpret_cast(stage.input_ptr), + block_start + elem_idx, + chunk_len, + src, + ptype); const uint32_t align = (block_start + elem_idx + src.params.bitunpack.element_offset) % FL_CHUNK; smem_src = scratch + align; - // Write barrier: all threads finished bitunpack, safe to read from scratch. - __syncthreads(); } else { chunk_len = block_len; } @@ -336,7 +491,7 @@ __device__ void execute_output_stage(T *__restrict output, smem); for (uint8_t op = 0; op < stage.num_scalar_ops; ++op) { - scalar_op(values, stage.scalar_ops[op], smem); + scalar_op(values, stage.scalar_ops[op], smem, tile_start); } #pragma unroll @@ -358,7 +513,7 @@ __device__ void execute_output_stage(T *__restrict output, source_op(&val, src, raw_input, ptype, smem_src, i, gpos, smem); for (uint8_t op = 0; op < stage.num_scalar_ops; ++op) { - scalar_op(&val, stage.scalar_ops[op], smem); + scalar_op(&val, stage.scalar_ops[op], smem, gpos); } __stcs(&output[gpos], val); } @@ -391,24 +546,28 @@ __device__ void execute_input_stage(const Stage &stage, char *__restrict smem) { const auto &src = stage.source; if (src.op_code == SourceOp::BITUNPACK) { - bitunpack(reinterpret_cast(stage.input_ptr), smem_out, 0, stage.len, src); + bitunpack_typed(smem_out, + reinterpret_cast(stage.input_ptr), + 0, + stage.len, + src, + stage.source_ptype); + smem_out += src.params.bitunpack.element_offset % SMEM_TILE_SIZE; - // Write barrier: cooperative bitunpack finished, safe to read - // decoded elements in the scalar-op loop below. - __syncthreads(); if (stage.num_scalar_ops > 0) { - for (uint32_t i = threadIdx.x; i < stage.len; i += blockDim.x) { - T val = smem_out[i]; + for (uint32_t elem_idx = threadIdx.x; elem_idx < stage.len; elem_idx += blockDim.x) { + T val = smem_out[elem_idx]; for (uint8_t op = 0; op < stage.num_scalar_ops; ++op) { - scalar_op(&val, stage.scalar_ops[op], smem); + scalar_op(&val, stage.scalar_ops[op], smem, elem_idx); } - smem_out[i] = val; + smem_out[elem_idx] = val; } // Write barrier: scalar ops applied in-place, smem region is // now fully populated for subsequent stages to read. __syncthreads(); } + } else { if (src.op_code == SourceOp::RUNEND) { // Seed each thread's cursor with the run containing its first @@ -420,13 +579,13 @@ __device__ void execute_input_stage(const Stage &stage, char *__restrict smem) { upper_bound(ends, src.params.runend.num_runs, threadIdx.x + src.params.runend.offset); } const void *raw_input = reinterpret_cast(stage.input_ptr); - for (uint32_t i = threadIdx.x; i < stage.len; i += blockDim.x) { + for (uint32_t elem_idx = threadIdx.x; elem_idx < stage.len; elem_idx += blockDim.x) { T val; - source_op(&val, src, raw_input, stage.source_ptype, nullptr, 0, i, smem); + source_op(&val, src, raw_input, stage.source_ptype, nullptr, 0, elem_idx, smem); for (uint8_t op = 0; op < stage.num_scalar_ops; ++op) { - scalar_op(&val, stage.scalar_ops[op], smem); + scalar_op(&val, stage.scalar_ops[op], smem, elem_idx); } - smem_out[i] = val; + smem_out[elem_idx] = val; } // Write barrier: smem region is fully populated for subsequent // stages to read. @@ -472,9 +631,10 @@ dynamic_dispatch(T *__restrict output, uint64_t array_len, const uint8_t *__rest // matters is load_element(), which dispatches on the per-op PTypeTag to // sign-extend or zero-extend when widening a narrow source to T. #define GENERATE_KERNEL(suffix, Type) \ - extern "C" __global__ void dynamic_dispatch_##suffix(Type *__restrict output, \ - uint64_t array_len, \ - const uint8_t *__restrict packed_plan) { \ + extern "C" __global__ void __launch_bounds__(BLOCK_SIZE, 32) \ + dynamic_dispatch_##suffix(Type *__restrict output, \ + uint64_t array_len, \ + const uint8_t *__restrict packed_plan) { \ dynamic_dispatch(output, array_len, packed_plan); \ } diff --git a/vortex-cuda/kernels/src/dynamic_dispatch.h b/vortex-cuda/kernels/src/dynamic_dispatch.h index 95540c51581..c4dbe55c9f3 100644 --- a/vortex-cuda/kernels/src/dynamic_dispatch.h +++ b/vortex-cuda/kernels/src/dynamic_dispatch.h @@ -20,7 +20,7 @@ /// /// Each source op and scalar op may produce a different PType than its input. /// For example, DICT transforms codes (e.g. u8) into values (e.g. f32), and -/// ALP transforms encoded integers (i32) into floats (f32). +/// ALP transforms encoded integers into floats (e.g. i32 → f32, i64 → f64). /// /// `PTypeTag` is a compact enum that identifies the primitive type at each /// point in the pipeline. The kernel uses it to dispatch typed memory @@ -78,6 +78,27 @@ PTYPE_HOST_DEVICE constexpr PTypeTag ptype_to_unsigned(PTypeTag tag) { return tag; } } + +PTYPE_HOST_DEVICE constexpr uint8_t ptype_byte_width(PTypeTag tag) { + switch (tag) { + case PTYPE_U8: + case PTYPE_I8: + return 1; + case PTYPE_U16: + case PTYPE_I16: + return 2; + case PTYPE_U32: + case PTYPE_I32: + case PTYPE_F32: + return 4; + case PTYPE_U64: + case PTYPE_I64: + case PTYPE_F64: + return 8; + default: + return 0; + } +} #endif /// Number of threads per CUDA block. @@ -103,11 +124,18 @@ extern "C" { #endif /// Parameters for source ops, which decode data into a stage's shared memory region. +/// +/// patches_ptr lives on the union variant that owns it (BitunpackParams, +/// AlpParams) — not per-stage — so the pointer is tied to its op. +/// Adding a u64 can grow the union and every ScalarOp/SourceOp read in the +/// tile loop; SourceParams was already 24 B (no change), ScalarParams grew +/// 8→16 B (no measurable impact — tile loop is compute-bound). union SourceParams { /// Unpack FastLanes bit-packed data. struct BitunpackParams { uint8_t bit_width; uint32_t element_offset; // Sub-byte offset + uint64_t patches_ptr; // device pointer to GPUPatches struct (0 = none) } bitunpack; /// Copy from global to shared memory. @@ -143,7 +171,7 @@ struct SourceOp { /// Each scalar op declares its `output_ptype` — the PType of the values it /// produces. Most ops preserve the input type (FOR, ZIGZAG), but some /// change it: -/// - ALP: encoded int → float (e.g. i32 → f32) +/// - ALP: encoded int → float (e.g. i32 → f32, i64 → f64) /// - DICT: codes type → values type (e.g. u8 → u32) /// /// The plan builder uses `output_ptype` to determine the element width @@ -155,8 +183,9 @@ union ScalarParams { } frame_of_ref; struct AlpParams { - float f; - float e; + double f; + double e; + uint64_t patches_ptr; // device pointer to GPUPatches struct (0 = none) } alp; /// Dictionary gather: use current value as index into decoded values in smem. @@ -193,6 +222,7 @@ struct PackedStage { uint32_t len; // number of elements this stage produces struct SourceOp source; + uint8_t num_scalar_ops; enum PTypeTag source_ptype; // PType produced by the source op }; @@ -220,11 +250,12 @@ struct __attribute__((aligned(8))) PlanHeader { /// change the type; the final output PType is given by the last scalar op's /// `output_ptype` (or `source_ptype` if there are no scalar ops). struct Stage { - uint64_t input_ptr; // encoded input in global memory - uint32_t smem_byte_offset; // byte offset within dynamic shared memory - uint32_t len; // elements produced - enum PTypeTag source_ptype; // PType produced by the source op - struct SourceOp source; // source decode op + uint64_t input_ptr; // encoded input in global memory + uint32_t smem_byte_offset; // byte offset within dynamic shared memory + uint32_t len; // elements produced + enum PTypeTag source_ptype; // PType produced by the source op + struct SourceOp source; // source decode op + uint8_t num_scalar_ops; // number of scalar ops const struct ScalarOp *scalar_ops; // scalar decode ops }; @@ -247,6 +278,7 @@ __device__ inline Stage parse_stage(const uint8_t *&cursor) { .len = packed_stage->len, .source_ptype = packed_stage->source_ptype, .source = packed_stage->source, + .num_scalar_ops = packed_stage->num_scalar_ops, .scalar_ops = ops, }; diff --git a/vortex-cuda/kernels/src/fastlanes_common.cuh b/vortex-cuda/kernels/src/fastlanes_common.cuh index 660a1c554f4..8536de26789 100644 --- a/vortex-cuda/kernels/src/fastlanes_common.cuh +++ b/vortex-cuda/kernels/src/fastlanes_common.cuh @@ -8,8 +8,25 @@ // FastLanes ordering array __constant__ int FL_ORDER[] = {0, 4, 2, 6, 1, 5, 3, 7}; +// FastLanes organises every 1024-element vector into a transposed layout +// of FL_LANES columns × (1024 / FL_LANES) rows. Each column is a "lane" +// that can be processed independently of every other lane, which is what +// makes all FastLanes encodings (FFOR, DELTA, RLE, ALP, …) fully +// data-parallel. One CUDA thread or one CPU SIMD lane handles one +// FastLanes lane. +// +// Paper: https://ir.cwi.nl/pub/35881/35881.pdf +// Repo: https://github.com/cwida/FastLanes + +/// FastLanes chunk size in elements. +constexpr uint32_t FL_CHUNK = 1024; + +/// Number of FastLanes lanes for element type T (1024 / bit-width). +template +constexpr uint32_t FL_LANES = FL_CHUNK / (sizeof(T) * 8); + // Compute the index in the FastLanes layout #define INDEX(row, lane) (FL_ORDER[row / 8] * 16 + (row % 8) * 128 + lane) // Create a mask with 'width' bits set -#define MASK(T, width) (((T)1 << width) - 1) \ No newline at end of file +#define MASK(T, width) (((T)1 << width) - 1) diff --git a/vortex-cuda/kernels/src/patches.cuh b/vortex-cuda/kernels/src/patches.cuh index 24b1705164f..fa9ff18def9 100644 --- a/vortex-cuda/kernels/src/patches.cuh +++ b/vortex-cuda/kernels/src/patches.cuh @@ -3,57 +3,105 @@ #pragma once +#include "fastlanes_common.cuh" #include "patches.h" +/// Load a chunk offset value, dispatching on the runtime type. +__device__ inline uint32_t load_chunk_offset(const GPUPatches &patches, uint32_t idx) { + switch (patches.chunk_offset_type) { + case CO_U8: + return reinterpret_cast(patches.chunk_offsets)[idx]; + case CO_U16: + return reinterpret_cast(patches.chunk_offsets)[idx]; + case CO_U32: + return reinterpret_cast(patches.chunk_offsets)[idx]; + case CO_U64: + return static_cast(reinterpret_cast(patches.chunk_offsets)[idx]); + } + return 0; +} + /// A single patch: a within-chunk index and its replacement value. -/// A sentinel patch has index == 1024, which can never match a valid -/// within-chunk position (0–1023). +/// A sentinel patch has index == FL_CHUNK, which can never match a valid +/// within-chunk position (0–FL_CHUNK-1). template struct Patch { uint16_t index; T value; }; -/// Cursor for iterating over a single lane's patches within a chunk. +/// Cursor for iterating over a thread's portion of patches within a chunk. /// -/// Usage in the generated merge-loop: +/// Patches are divided evenly among threads. Each thread applies its patches +/// to shared memory, then all threads sync and copy to global memory. +/// +/// Usage in the generated kernel: /// /// PatchesCursor cursor(patches, blockIdx.x, thread_idx, 32); /// auto patch = cursor.next(); -/// for (int i = 0; i < 32; i++) { -/// auto idx = i * 32 + thread_idx; -/// if (idx == patch.index) { -/// out[idx] = patch.value; -/// patch = cursor.next(); -/// } else { -/// out[idx] = shared_out[idx]; -/// } +/// while (patch.index != FL_CHUNK) { +/// shared_out[patch.index] = patch.value; +/// patch = cursor.next(); /// } template class PatchesCursor { public: - /// Construct a cursor positioned at the patches for the given (chunk, lane). - /// n_lanes is a compile-time constant emitted by the code generator (16 or 32). - __device__ PatchesCursor(const GPUPatches &patches, uint32_t chunk, uint32_t lane, uint32_t n_lanes) { - if (patches.lane_offsets == nullptr) { + /// Construct a cursor for this thread's portion of patches in the chunk. + __device__ + PatchesCursor(const GPUPatches &patches, uint32_t chunk, uint32_t thread_idx, uint32_t n_threads) { + if (patches.chunk_offsets == nullptr) { indices = nullptr; values = nullptr; remaining = 0; return; } - auto slot = chunk * n_lanes + lane; - auto start = patches.lane_offsets[slot]; - remaining = patches.lane_offsets[slot + 1] - start; + + // mirrors the logic from vortex-array/src/arrays/primitive/array/patch.rs + + // Compute base_offset from the first chunk offset. + uint32_t base_offset = load_chunk_offset(patches, 0); + + uint32_t patches_start_idx = load_chunk_offset(patches, chunk) - base_offset; + patches_start_idx -= min(patches_start_idx, patches.offset_within_chunk); + + // calculate the ending index. + uint32_t patches_end_idx; + if ((chunk + 1) < patches.n_chunks) { + patches_end_idx = load_chunk_offset(patches, chunk + 1) - base_offset; + // if this is the end of times, we should drop it out here... + patches_end_idx -= min(patches_end_idx, patches.offset_within_chunk); + } else { + patches_end_idx = patches.num_patches; + } + + // calculate how many patches are in the chunk + uint32_t num_patches = patches_end_idx - patches_start_idx; + + // Divide patches among threads (ceil division) + uint32_t patches_per_thread = (num_patches + n_threads - 1) / n_threads; + uint32_t my_start = min(thread_idx * patches_per_thread, num_patches); + uint32_t my_end = min((thread_idx + 1) * patches_per_thread, num_patches); + + uint32_t start = patches_start_idx + my_start; + remaining = my_end - my_start; indices = patches.indices + start; values = reinterpret_cast(patches.values) + start; + + // The iterator returns indices relative to the start of the chunk. + // `chunk_base` is the index of the first element within a chunk, accounting + // for the slice offset. + chunk_base = chunk * FL_CHUNK + patches.offset; + chunk_base -= min(chunk_base, patches.offset % FL_CHUNK); } - /// Return the current patch and advance, or a sentinel {1024, 0} if exhausted. + /// Return the current patch (with within-chunk index) and advance, + /// or a sentinel {1024, 0} if exhausted. __device__ Patch next() { if (remaining == 0) { - return {1024, T {}}; + return {FL_CHUNK, T {}}; } - Patch patch = {*indices, *values}; + uint16_t within_chunk = static_cast(*indices - chunk_base); + Patch patch = {within_chunk, *values}; indices++; values++; remaining--; @@ -61,7 +109,8 @@ public: } private: - const uint16_t *indices; + const uint32_t *indices; const T *values; - uint8_t remaining; -}; \ No newline at end of file + uint32_t remaining; + uint32_t chunk_base; +}; diff --git a/vortex-cuda/kernels/src/patches.h b/vortex-cuda/kernels/src/patches.h index acc628ae595..ed7240821e6 100644 --- a/vortex-cuda/kernels/src/patches.h +++ b/vortex-cuda/kernels/src/patches.h @@ -9,18 +9,26 @@ extern "C" { #endif +/// Type tag for chunk_offsets pointer. +typedef enum { CO_U8 = 0, CO_U16 = 1, CO_U32 = 2, CO_U64 = 3 } ChunkOffsetType; + /// GPU-resident patches for fused exception patching during bit-unpacking. /// -/// Patches are stored in a lane-wise transposed layout: for each (chunk, lane) pair, -/// the corresponding patch indices and values are stored contiguously. The lane_offsets -/// array is a CSR-style offset array of size (n_chunks * n_lanes + 1) that maps each -/// (chunk, lane) slot to its range in the indices and values arrays. +/// Patches are stored in sorted order within each chunk. The chunk_offsets +/// array maps each chunk to the start of its range in the indices/values arrays. +/// The array has n_chunks elements (not n_chunks+1); the final offset is implicit +/// and equals num_patches. /// -/// A NULL lane_offsets pointer indicates no patches are present. +/// A NULL chunk_offsets pointer indicates no patches are present. typedef struct { - uint32_t *lane_offsets; - uint16_t *indices; + void *chunk_offsets; + ChunkOffsetType chunk_offset_type; + uint32_t *indices; void *values; + uint32_t offset; + uint32_t offset_within_chunk; + uint32_t num_patches; + uint32_t n_chunks; } GPUPatches; #ifdef __cplusplus diff --git a/vortex-cuda/nvcomp/src/lib.rs b/vortex-cuda/nvcomp/src/lib.rs index 2f9fd00deb9..3ab502baa63 100644 --- a/vortex-cuda/nvcomp/src/lib.rs +++ b/vortex-cuda/nvcomp/src/lib.rs @@ -22,11 +22,10 @@ use std::path::PathBuf; use std::sync::OnceLock; /// Raw FFI type definitions and dynamically-loaded function pointers from bindgen. -#[allow( +#[expect( non_upper_case_globals, non_camel_case_types, non_snake_case, - dead_code, clippy::all )] pub mod sys; diff --git a/vortex-cuda/nvcomp/src/zstd.rs b/vortex-cuda/nvcomp/src/zstd.rs index 7a326d3fb25..44111901071 100644 --- a/vortex-cuda/nvcomp/src/zstd.rs +++ b/vortex-cuda/nvcomp/src/zstd.rs @@ -134,7 +134,7 @@ pub fn get_decompress_temp_size_with_opts( /// - Each output buffer must have at least the corresponding `device_uncompressed_bytes` size /// - `device_temp_ptr` must have at least `temp_bytes` allocated /// - The stream must be valid -#[allow(clippy::too_many_arguments)] +#[expect(clippy::too_many_arguments)] pub unsafe fn decompress_async( device_compressed_ptrs: *const *const c_void, device_compressed_bytes: *const usize, @@ -170,7 +170,7 @@ pub unsafe fn decompress_async( /// # Safety /// /// Same requirements as [`decompress_async`]. -#[allow(clippy::too_many_arguments)] +#[expect(clippy::too_many_arguments)] pub unsafe fn decompress_async_with_opts( device_compressed_ptrs: *const *const c_void, device_compressed_bytes: *const usize, diff --git a/vortex-cuda/src/arrow/canonical.rs b/vortex-cuda/src/arrow/canonical.rs index 660c2872259..0ccace213c9 100644 --- a/vortex-cuda/src/arrow/canonical.rs +++ b/vortex-cuda/src/arrow/canonical.rs @@ -1,11 +1,14 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +use std::mem; +use std::ptr; + use async_trait::async_trait; use futures::future::BoxFuture; use vortex::array::ArrayRef; use vortex::array::Canonical; -use vortex::array::ToCanonical; +use vortex::array::arrays::PrimitiveArray; use vortex::array::arrays::StructArray; use vortex::array::arrays::bool::BoolDataParts; use vortex::array::arrays::decimal::DecimalDataParts; @@ -116,7 +119,10 @@ fn export_canonical( vortex_bail!("only support temporal extension types currently"); } - let values = extension.storage_array().to_primitive(); + let values = extension + .storage_array() + .clone() + .execute::(ctx.execution_ctx())?; let len = extension.len(); let PrimitiveDataParts { @@ -161,9 +167,9 @@ fn export_canonical( n_buffers: 2, buffers: private_data.buffer_ptrs.as_mut_ptr(), n_children: 0, - children: std::ptr::null_mut(), + children: ptr::null_mut(), release: Some(release_array), - dictionary: std::ptr::null_mut(), + dictionary: ptr::null_mut(), private_data: Box::into_raw(private_data).cast(), }; @@ -241,9 +247,9 @@ fn export_fixed_size( n_buffers: 2, buffers: private_data.buffer_ptrs.as_mut_ptr(), n_children: 0, - children: std::ptr::null_mut(), + children: ptr::null_mut(), release: Some(release_array), - dictionary: std::ptr::null_mut(), + dictionary: ptr::null_mut(), private_data: Box::into_raw(private_data).cast(), }; @@ -255,12 +261,11 @@ unsafe extern "C" fn release_array(array: *mut ArrowArray) { // code. This is necessary to ensure that the fields inside the CudaPrivateData // get dropped to free native/GPU memory. unsafe { - let private_data_ptr = - std::ptr::replace(&raw mut (*array).private_data, std::ptr::null_mut()); + let private_data_ptr = ptr::replace(&raw mut (*array).private_data, ptr::null_mut()); if !private_data_ptr.is_null() { let mut private_data = Box::from_raw(private_data_ptr.cast::()); - let children = std::mem::take(&mut private_data.children); + let children = mem::take(&mut private_data.children); for child in children { release_array(child); } @@ -274,6 +279,7 @@ unsafe extern "C" fn release_array(array: *mut ArrowArray) { #[cfg(test)] mod tests { use rstest::rstest; + use vortex::array::ArrayRef; use vortex::array::IntoArray; use vortex::array::arrays::DecimalArray; use vortex::array::arrays::NullArray; @@ -305,7 +311,7 @@ mod tests { #[case::f64(PrimitiveArray::from_iter([1.0f64, 2.0, 3.0]).into_array(), 3)] #[crate::test] async fn test_export_primitive( - #[case] array: vortex::array::ArrayRef, + #[case] array: ArrayRef, #[case] expected_len: i64, ) -> VortexResult<()> { let mut ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) diff --git a/vortex-cuda/src/arrow/mod.rs b/vortex-cuda/src/arrow/mod.rs index 27533a986dd..89580290a6a 100644 --- a/vortex-cuda/src/arrow/mod.rs +++ b/vortex-cuda/src/arrow/mod.rs @@ -95,7 +95,6 @@ pub(crate) struct ArrowArray { } impl ArrowArray { - #[allow(unused)] pub fn empty() -> Self { Self { length: 0, diff --git a/vortex-cuda/src/bit_unpack_gen.rs b/vortex-cuda/src/bit_unpack_gen.rs index e4efa9bf2eb..2482c0996b8 100644 --- a/vortex-cuda/src/bit_unpack_gen.rs +++ b/vortex-cuda/src/bit_unpack_gen.rs @@ -48,12 +48,7 @@ fn write_row(output: &mut impl Write, bits: usize, bit_width: usize, row: usize) /// loop. For all other bit widths, emits pre-computed per-row bit extraction /// with register-cached `src` words — identical to the original hand-unrolled /// codegen, preserving minimal memory loads and zero extra work. -fn generate_lane_decoder( - output: &mut impl Write, - bits: usize, - lanes: usize, - bit_width: usize, -) -> io::Result<()> { +fn generate_lane_decoder(output: &mut impl Write, bits: usize, bit_width: usize) -> io::Result<()> { if bit_width == 0 { write!( output, @@ -71,7 +66,7 @@ __device__ void _bit_unpack_{bits}_lane<0>(const uint{bits}_t *__restrict in, ui output, r#"template <> __device__ void _bit_unpack_{bits}_lane<{bit_width}>(const uint{bits}_t *__restrict in, uint{bits}_t *__restrict out, uint{bits}_t reference, unsigned int lane) {{ - unsigned int LANE_COUNT = {lanes}; + constexpr unsigned int LANE_COUNT = FL_LANES; #pragma unroll for (int row = 0; row < {bits}; row++) {{ out[INDEX(row, lane)] = in[LANE_COUNT * row + lane] + reference; @@ -84,7 +79,7 @@ __device__ void _bit_unpack_{bits}_lane<{bit_width}>(const uint{bits}_t *__restr output, r#"template <> __device__ void _bit_unpack_{bits}_lane<{bit_width}>(const uint{bits}_t *__restrict in, uint{bits}_t *__restrict out, uint{bits}_t reference, unsigned int lane) {{ - unsigned int LANE_COUNT = {lanes}; + constexpr unsigned int LANE_COUNT = FL_LANES; uint{bits}_t src; uint{bits}_t tmp; src = in[lane]; @@ -108,7 +103,7 @@ fn generate_lane_dispatch(output: &mut impl Write, bits: usize) -> io::Result<() write!( output, r#"/// Runtime dispatch to the optimized lane decoder for the given bit width. -__device__ inline void bit_unpack_{bits}_lane( +__device__ __noinline__ void bit_unpack_{bits}_lane( const uint{bits}_t *__restrict in, uint{bits}_t *__restrict out, uint{bits}_t reference, @@ -142,32 +137,35 @@ __device__ inline void bit_unpack_{bits}_lane( fn generate_device_kernel_template( output: &mut impl Write, bits: usize, - lanes: usize, thread_count: usize, ) -> io::Result<()> { - let per_thread_loop_count = lanes / thread_count; - let shared_copy_ncount = 1024 / thread_count; - write!( output, r#"template __device__ void _bit_unpack_{bits}_device(const uint{bits}_t *__restrict in, uint{bits}_t *__restrict out, uint{bits}_t reference, int thread_idx, GPUPatches& patches) {{ - __shared__ uint{bits}_t shared_out[1024]; + __shared__ uint{bits}_t shared_out[FL_CHUNK]; + + // Step 1: Unpack into shared memory #pragma unroll - for (int i = 0; i < {per_thread_loop_count}; i++) {{ - _bit_unpack_{bits}_lane(in, shared_out, reference, thread_idx * {per_thread_loop_count} + i); + for (int i = 0; i < FL_LANES / {thread_count}; i++) {{ + _bit_unpack_{bits}_lane(in, shared_out, reference, thread_idx * (FL_LANES / {thread_count}) + i); }} __syncwarp(); + + // Step 2: Apply patches to shared memory in parallel PatchesCursor cursor(patches, blockIdx.x, thread_idx, {thread_count}); auto patch = cursor.next(); - for (int i = 0; i < {shared_copy_ncount}; i++) {{ + while (patch.index != FL_CHUNK) {{ + shared_out[patch.index] = patch.value; + patch = cursor.next(); + }} + __syncwarp(); + + // Step 3: Copy to global memory + #pragma unroll + for (int i = 0; i < FL_CHUNK / {thread_count}; i++) {{ auto idx = i * {thread_count} + thread_idx; - if (idx == patch.index) {{ - out[idx] = patch.value; - patch = cursor.next(); - }} else {{ - out[idx] = shared_out[idx]; - }} + out[idx] = shared_out[idx]; }} }} "# @@ -187,31 +185,32 @@ fn generate_global_kernel( output, r#"extern "C" __global__ void {func_name}(const uint{bits}_t *__restrict full_in, uint{bits}_t *__restrict full_out, uint{bits}_t reference, GPUPatches patches) {{ int thread_idx = threadIdx.x; - auto in = full_in + (blockIdx.x * (128 * {bit_width} / sizeof(uint{bits}_t))); - auto out = full_out + (blockIdx.x * 1024); + auto in = full_in + (blockIdx.x * (FL_LANES * {bit_width})); + auto out = full_out + (blockIdx.x * FL_CHUNK); _bit_unpack_{bits}_device<{bit_width}>(in, out, reference, thread_idx, patches); }} "# ) } -/// Generate CUDA lane decoders, dispatch function, and kernel wrappers for all bit widths. -pub fn generate_cuda_unpack( - output: &mut impl Write, - thread_count: usize, -) -> io::Result<()> { +/// Generate the lane-decoder header: template specializations + runtime dispatch. +/// +/// This produces a `.cuh` file that is included by `bit_unpack.cuh` (and +/// transitively by `dynamic_dispatch.cu`). It contains only `__device__` +/// functions — no `__global__` kernels — so that `dynamic_dispatch.cu` does +/// not pull in the 129 standalone bit-unpack kernel entry points. +pub fn generate_cuda_unpack_lanes(output: &mut impl Write) -> io::Result<()> { let bits = T::T; - let lanes = T::LANES; - // File header + forward declaration for the lane-decoder template. write!( output, r#"// AUTO-GENERATED. Do not edit by hand! +#pragma once + #include #include #include #include "fastlanes_common.cuh" -#include "patches.cuh" template __device__ void _bit_unpack_{bits}_lane(const uint{bits}_t *__restrict in, uint{bits}_t *__restrict out, uint{bits}_t reference, unsigned int lane); @@ -221,7 +220,7 @@ __device__ void _bit_unpack_{bits}_lane(const uint{bits}_t *__restrict in, uint{ // Lane-decoder template specializations (one per bit width). for bit_width in 0..=bits { - generate_lane_decoder(output, bits, lanes, bit_width)?; + generate_lane_decoder(output, bits, bit_width)?; writeln!(output)?; } @@ -229,8 +228,30 @@ __device__ void _bit_unpack_{bits}_lane(const uint{bits}_t *__restrict in, uint{ generate_lane_dispatch(output, bits)?; writeln!(output)?; + Ok(()) +} + +/// Generate the standalone kernel file: `_device` template + `__global__` wrappers. +/// +/// This produces a `.cu` file that is compiled to PTX on its own. It includes +/// the corresponding `_lanes.cuh` header for the lane decoders. +pub fn generate_cuda_unpack_kernels( + output: &mut impl Write, + thread_count: usize, +) -> io::Result<()> { + let bits = T::T; + + write!( + output, + r#"// AUTO-GENERATED. Do not edit by hand! +#include "bit_unpack_{bits}_lanes.cuh" +#include "patches.cuh" + +"# + )?; + // Device kernel template (written once, instantiated per bit width). - generate_device_kernel_template(output, bits, lanes, thread_count)?; + generate_device_kernel_template(output, bits, thread_count)?; writeln!(output)?; // Thin extern "C" global-kernel wrappers (one per bit width). diff --git a/vortex-cuda/src/canonical.rs b/vortex-cuda/src/canonical.rs index cde3841e196..591f1be404f 100644 --- a/vortex-cuda/src/canonical.rs +++ b/vortex-cuda/src/canonical.rs @@ -7,6 +7,8 @@ use async_trait::async_trait; use futures::future::try_join_all; use vortex::array::Canonical; use vortex::array::IntoArray; +use vortex::array::LEGACY_SESSION; +use vortex::array::VortexSessionExecute; use vortex::array::arrays::BoolArray; use vortex::array::arrays::DecimalArray; use vortex::array::arrays::ExtensionArray; @@ -50,7 +52,14 @@ impl CanonicalCudaExt for Canonical { let mut host_fields = vec![]; for field in fields.iter() { - host_fields.push(field.to_canonical()?.into_host().await?.into_array()); + host_fields.push( + field + .clone() + .execute::(&mut LEGACY_SESSION.create_execution_ctx())? + .into_host() + .await? + .into_array(), + ); } Ok(Canonical::Struct(StructArray::new( @@ -132,7 +141,8 @@ impl CanonicalCudaExt for Canonical { // Copy the storage array to host and rewrap in ExtensionArray. let host_storage = ext .storage_array() - .to_canonical()? + .clone() + .execute::(&mut LEGACY_SESSION.create_execution_ctx())? .into_host() .await? .into_array(); diff --git a/vortex-cuda/src/device_buffer.rs b/vortex-cuda/src/device_buffer.rs index 3257318cde4..e0f6477a9e3 100644 --- a/vortex-cuda/src/device_buffer.rs +++ b/vortex-cuda/src/device_buffer.rs @@ -1,8 +1,12 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +use std::any::Any; use std::cmp::min; use std::fmt::Debug; +use std::fmt::Formatter; +use std::hash::Hash; +use std::hash::Hasher; use std::ops::Range; use std::sync::Arc; @@ -11,6 +15,7 @@ use cudarc::driver::CudaView; use cudarc::driver::DevicePtr; use cudarc::driver::DeviceRepr; use cudarc::driver::sys; +use futures::executor::block_on; use futures::future::BoxFuture; use vortex::array::buffer::BufferHandle; use vortex::array::buffer::DeviceBuffer; @@ -174,7 +179,7 @@ impl CudaBufferExt for BufferHandle { } impl Debug for CudaDeviceBuffer { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { f.debug_struct("CudaDeviceBuffer") .field("allocation", &self.allocation) .field("device_ptr", &self.device_ptr) @@ -184,8 +189,8 @@ impl Debug for CudaDeviceBuffer { } } -impl std::hash::Hash for CudaDeviceBuffer { - fn hash(&self, state: &mut H) { +impl Hash for CudaDeviceBuffer { + fn hash(&self, state: &mut H) { self.device_ptr.hash(state); self.len.hash(state); self.offset.hash(state); @@ -224,7 +229,7 @@ impl DeviceBuffer for CudaDeviceBuffer { /// /// Returns an error if the CUDA memory copy operation fails. fn copy_to_host_sync(&self, alignment: Alignment) -> VortexResult { - futures::executor::block_on(self.copy_to_host(alignment)?) + block_on(self.copy_to_host(alignment)?) } /// Copies a device buffer to host memory asynchronously. @@ -322,7 +327,7 @@ impl DeviceBuffer for CudaDeviceBuffer { }) } - fn as_any(&self) -> &dyn std::any::Any { + fn as_any(&self) -> &dyn Any { self } diff --git a/vortex-cuda/src/dynamic_dispatch/mod.rs b/vortex-cuda/src/dynamic_dispatch/mod.rs index 192438d0329..c582c5d8945 100644 --- a/vortex-cuda/src/dynamic_dispatch/mod.rs +++ b/vortex-cuda/src/dynamic_dispatch/mod.rs @@ -24,14 +24,18 @@ use cudarc::driver::DevicePtr; use cudarc::driver::LaunchConfig; use cudarc::driver::PushKernelArg; use vortex::array::Canonical; +use vortex::array::IntoArray; +use vortex::array::arrays::ConstantArray; use vortex::array::arrays::PrimitiveArray; use vortex::array::buffer::BufferHandle; use vortex::array::buffer::DeviceBufferExt; use vortex::array::match_each_unsigned_integer_ptype; +use vortex::array::scalar::Scalar; use vortex::array::validity::Validity; use vortex::buffer::Alignment; use vortex::buffer::ByteBuffer; use vortex::buffer::ByteBufferMut; +use vortex::dtype::DType; use vortex::dtype::Nullability; use vortex::dtype::PType; use vortex::error::VortexResult; @@ -82,22 +86,26 @@ pub fn tag_to_ptype(tag: PTypeTag) -> PType { } } -/// Serialize a `#[repr(C)]` struct to a byte vector for the packed plan. +/// Reinterpret a reference to a packed type as a raw bytes slice. /// -/// Copies field data into a pre-zeroed buffer so padding holes are -/// deterministically zero, avoiding UB from reading uninitialised bytes. -fn as_bytes(val: &T) -> Vec { - let n = size_of::(); - let mut buf = vec![0u8; n]; - // SAFETY: T is a bindgen-generated #[repr(C)] struct with only - // integer/float/enum fields. We overwrite the zeroed buffer with - // the struct's bytes; padding holes keep their zero value. - unsafe { - std::ptr::copy_nonoverlapping(std::ptr::addr_of!(*val).cast::(), buf.as_mut_ptr(), n); - } - buf +/// # Safety +/// +/// The caller must ensure `T` is a `#[repr(C)]` type whose layout is +/// compatible with the C ABI. All the types we serialise (`PlanHeader`, +/// `PackedStage`, `ScalarOp`) are bindgen-generated `#[repr(C)]` structs. +/// Padding bytes may be uninitialised on the Rust side, but the C reader +/// never inspects them, so the values are irrelevant. +unsafe trait AsPackedBytes: Sized { + /// Reinterpret a `&T` as a byte slice for serialization into the packed plan. + fn as_packed_bytes(&self) -> &[u8] { + unsafe { std::slice::from_raw_parts(std::ptr::addr_of!(*self).cast(), size_of::()) } + } } +unsafe impl AsPackedBytes for PlanHeader {} +unsafe impl AsPackedBytes for PackedStage {} +unsafe impl AsPackedBytes for ScalarOp {} + /// A stage used to build a [`CudaDispatchPlan`] on the host side. /// /// This is NOT a C ABI struct — it exists purely on the Rust side and is @@ -206,7 +214,7 @@ impl CudaDispatchPlan { output_ptype, plan_size_bytes: total_size as u16, }; - buffer.extend_from_slice(&as_bytes(&header)); + buffer.extend_from_slice(header.as_packed_bytes()); for stage in &stages { let packed_stage = PackedStage { @@ -217,9 +225,9 @@ impl CudaDispatchPlan { num_scalar_ops: stage.scalar_ops.len() as u8, source_ptype: stage.source_ptype, }; - buffer.extend_from_slice(&as_bytes(&packed_stage)); + buffer.extend_from_slice(packed_stage.as_packed_bytes()); for op in &stage.scalar_ops { - buffer.extend_from_slice(&as_bytes(op)); + buffer.extend_from_slice(op.as_packed_bytes()); } } @@ -305,6 +313,7 @@ impl SourceOp { bitunpack: SourceParams_BitunpackParams { bit_width, element_offset: u32::from(element_offset), + patches_ptr: 0, }, }, } @@ -379,13 +388,17 @@ impl ScalarOp { } } - /// ALP floating-point decode. - pub fn alp(f: f32, e: f32) -> Self { + /// ALP floating-point decode (f32 or f64). + pub fn alp(f: f64, e: f64, output_ptype: PTypeTag) -> Self { Self { op_code: ScalarOp_ScalarOpCode_ALP, - output_ptype: PTypeTag_PTYPE_F32, + output_ptype, params: ScalarParams { - alp: ScalarParams_AlpParams { f, e }, + alp: ScalarParams_AlpParams { + f, + e, + patches_ptr: 0, + }, }, } } @@ -408,13 +421,22 @@ impl ScalarOp { impl MaterializedPlan { pub fn execute(self, len: usize, ctx: &mut CudaExecutionCtx) -> VortexResult { let output_ptype = self.dispatch_plan.output_ptype(); + + // All values are null — no need to touch the GPU. + if matches!(self.validity, Validity::AllInvalid) { + let dtype = DType::Primitive(output_ptype, Nullability::Nullable); + return ConstantArray::new(Scalar::null(dtype), len) + .into_array() + .execute::(ctx.execution_ctx()); + } + // The CUDA kernels are instantiated for unsigned integer types only; // map signed/float ptypes to their same-width unsigned counterpart. let unsigned_ptype = match output_ptype { PType::U8 | PType::I8 => PType::U8, PType::U16 | PType::I16 => PType::U16, PType::U32 | PType::I32 | PType::F32 => PType::U32, - PType::U64 | PType::I64 => PType::U64, + PType::U64 | PType::I64 | PType::F64 => PType::U64, other => vortex_bail!("dynamic dispatch does not support PType {:?}", other), }; match_each_unsigned_integer_ptype!(unsigned_ptype, |T| { @@ -431,9 +453,11 @@ impl MaterializedPlan { where T: cudarc::driver::DeviceRepr + vortex::dtype::NativePType, { + let nullability = self.validity.nullability(); + if len == 0 { return Ok(Canonical::Primitive(PrimitiveArray::empty::( - Nullability::NonNullable, + nullability, ))); } @@ -467,13 +491,14 @@ impl MaterializedPlan { Ok(Canonical::Primitive(PrimitiveArray::from_buffer_handle( BufferHandle::new_device(output_buf.slice_typed::(0..len)), output_ptype, - Validity::NonNullable, + self.validity, ))) } } #[cfg(test)] mod tests { + use std::ops::Range; use std::sync::Arc; use cudarc::driver::DevicePtr; @@ -481,10 +506,10 @@ mod tests { use cudarc::driver::PushKernelArg; use rstest::rstest; use vortex::array::IntoArray; - use vortex::array::ToCanonical; use vortex::array::arrays::DictArray; use vortex::array::arrays::PrimitiveArray; use vortex::array::scalar::Scalar; + use vortex::array::validity::Validity; use vortex::array::validity::Validity::NonNullable; use vortex::buffer::Buffer; use vortex::dtype::PType; @@ -496,6 +521,7 @@ mod tests { use vortex::encodings::alp::alp_encode; use vortex::encodings::fastlanes::BitPacked; use vortex::encodings::fastlanes::BitPackedArray; + use vortex::encodings::fastlanes::BitPackedArrayExt; use vortex::encodings::fastlanes::FoR; use vortex::encodings::fastlanes::FoRArrayExt; use vortex::encodings::runend::RunEnd; @@ -503,12 +529,16 @@ mod tests { use vortex::error::VortexExpect; use vortex::error::VortexResult; use vortex::session::VortexSession; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use super::*; use crate::CanonicalCudaExt; use crate::CudaBufferExt; use crate::CudaDeviceBuffer; use crate::CudaExecutionCtx; + use crate::executor::CudaArrayExt; + use crate::executor::CudaDispatchMode; use crate::hybrid_dispatch::try_gpu_dispatch; use crate::session::CudaSession; @@ -518,16 +548,20 @@ mod tests { .map(|i| ((i as u64) % (max_val + 1)) as u32) .collect(); let primitive = PrimitiveArray::new(Buffer::from(values), NonNullable); - BitPacked::encode(&primitive.into_array(), bit_width) - .vortex_expect("failed to create BitPacked array") + BitPacked::encode( + &primitive.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .vortex_expect("failed to create BitPacked array") } - fn dispatch_plan( + async fn dispatch_plan( array: &vortex::array::ArrayRef, - ctx: &CudaExecutionCtx, + ctx: &mut CudaExecutionCtx, ) -> VortexResult { - match DispatchPlan::new(array)? { - DispatchPlan::Fused(plan) => plan.materialize(ctx), + match DispatchPlan::new(array, CudaDispatchMode::DynDispatchOnly)? { + DispatchPlan::Fused(plan) => plan.materialize(ctx).await, _ => vortex_bail!("array encoding not fusable"), } } @@ -631,7 +665,12 @@ mod tests { cuda_ctx: &CudaExecutionCtx, data: &[u32], ) -> VortexResult<(u64, Arc>)> { - let device_buf = Arc::new(cuda_ctx.stream().clone_htod(data).expect("htod")); + let device_buf = Arc::new( + cuda_ctx + .stream() + .clone_htod(data) + .map_err(|e| vortex_err!("htod: {e}"))?, + ); let (ptr, _) = device_buf.device_ptr(cuda_ctx.stream()); Ok((ptr, device_buf)) } @@ -669,7 +708,7 @@ mod tests { &[ ScalarOp::frame_of_ref(reference as u64, PTypeTag_PTYPE_U32), ScalarOp::zigzag(PTypeTag_PTYPE_U32), - ScalarOp::alp(alp_f, alp_e), + ScalarOp::alp(alp_f as f64, alp_e as f64, PTypeTag_PTYPE_F32), ], )], PTypeTag_PTYPE_U32, @@ -699,12 +738,15 @@ mod tests { cuda_ctx .stream() .clone_htod(plan.as_bytes()) - .expect("copy plan to device"), + .map_err(|e| vortex_err!("copy plan to device: {e}"))?, ); let (plan_ptr, record_plan) = device_plan.device_ptr(cuda_ctx.stream()); let array_len_u64 = output_len as u64; - cuda_ctx.stream().synchronize().expect("sync"); + cuda_ctx + .stream() + .synchronize() + .map_err(|e| vortex_err!("sync: {e}"))?; let cuda_function = cuda_ctx .load_function("dynamic_dispatch", &[PType::U32]) @@ -721,14 +763,16 @@ mod tests { shared_mem_bytes, }; unsafe { - launch_builder.launch(config).expect("kernel launch"); + launch_builder + .launch(config) + .map_err(|e| vortex_err!("kernel launch: {e}"))?; } drop((record_output, record_plan)); - Ok(cuda_ctx + cuda_ctx .stream() .clone_dtoh(&output_buf.as_view::()) - .expect("copy back")) + .map_err(|e| vortex_err!("copy back: {e}")) } fn run_dispatch_plan_f32( @@ -743,7 +787,7 @@ mod tests { } #[crate::test] - fn test_bitpacked() -> VortexResult<()> { + async fn test_bitpacked() -> VortexResult<()> { let bit_width: u8 = 10; let len = 3000; let max_val = (1u64 << bit_width).saturating_sub(1); @@ -752,8 +796,8 @@ mod tests { .collect(); let bp = bitpacked_array_u32(bit_width, len); - let cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; - let plan = dispatch_plan(&bp.into_array(), &cuda_ctx)?; + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let plan = dispatch_plan(&bp.into_array(), &mut cuda_ctx).await?; let actual = run_dynamic_dispatch_plan(&cuda_ctx, len, &plan.dispatch_plan, plan.shared_mem_bytes)?; @@ -763,7 +807,7 @@ mod tests { } #[crate::test] - fn test_for_bitpacked() -> VortexResult<()> { + async fn test_for_bitpacked() -> VortexResult<()> { let bit_width: u8 = 6; let len = 3000; let reference = 42u32; @@ -777,8 +821,8 @@ mod tests { let bp = bitpacked_array_u32(bit_width, len); let for_arr = FoR::try_new(bp.into_array(), Scalar::from(reference))?; - let cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; - let plan = dispatch_plan(&for_arr.into_array(), &cuda_ctx)?; + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let plan = dispatch_plan(&for_arr.into_array(), &mut cuda_ctx).await?; let actual = run_dynamic_dispatch_plan(&cuda_ctx, len, &plan.dispatch_plan, plan.shared_mem_bytes)?; @@ -788,7 +832,7 @@ mod tests { } #[crate::test] - fn test_runend() -> VortexResult<()> { + async fn test_runend() -> VortexResult<()> { let ends: Vec = vec![1000, 2000, 3000]; let values: Vec = vec![10, 20, 30]; let len = 3000; @@ -799,12 +843,12 @@ mod tests { expected.push(values[run]); } + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; let ends_arr = PrimitiveArray::new(Buffer::from(ends), NonNullable).into_array(); let values_arr = PrimitiveArray::new(Buffer::from(values), NonNullable).into_array(); - let re = RunEnd::new(ends_arr, values_arr); + let re = RunEnd::new(ends_arr, values_arr, cuda_ctx.execution_ctx()); - let cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; - let plan = dispatch_plan(&re.into_array(), &cuda_ctx)?; + let plan = dispatch_plan(&re.into_array(), &mut cuda_ctx).await?; let actual = run_dynamic_dispatch_plan(&cuda_ctx, len, &plan.dispatch_plan, plan.shared_mem_bytes)?; @@ -814,7 +858,7 @@ mod tests { } #[crate::test] - fn test_dict_for_bp_values_bp_codes() -> VortexResult<()> { + async fn test_dict_for_bp_values_bp_codes() -> VortexResult<()> { // Dict where both codes and values are BitPacked+FoR. let dict_reference = 1_000_000u32; let dict_residuals: Vec = (0..64).collect(); @@ -827,17 +871,25 @@ mod tests { // BitPack+FoR the dict values let dict_prim = PrimitiveArray::new(Buffer::from(dict_residuals), NonNullable); - let dict_bp = BitPacked::encode(&dict_prim.into_array(), 6)?; + let dict_bp = BitPacked::encode( + &dict_prim.into_array(), + 6, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; let dict_for = FoR::try_new(dict_bp.into_array(), Scalar::from(dict_reference))?; // BitPack the codes let codes_prim = PrimitiveArray::new(Buffer::from(codes), NonNullable); - let codes_bp = BitPacked::encode(&codes_prim.into_array(), 6)?; + let codes_bp = BitPacked::encode( + &codes_prim.into_array(), + 6, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; let dict = DictArray::try_new(codes_bp.into_array(), dict_for.into_array())?; - let cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; - let plan = dispatch_plan(&dict.into_array(), &cuda_ctx)?; + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let plan = dispatch_plan(&dict.into_array(), &mut cuda_ctx).await?; let actual = run_dynamic_dispatch_plan(&cuda_ctx, len, &plan.dispatch_plan, plan.shared_mem_bytes)?; @@ -847,7 +899,8 @@ mod tests { } #[crate::test] - fn test_alp_for_bitpacked() -> VortexResult<()> { + async fn test_alp_for_bitpacked() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // ALP(FoR(BitPacked)): encode each layer, then reassemble the tree // bottom-up because encode() methods produce flat outputs. let len = 3000; @@ -857,10 +910,14 @@ mod tests { .collect(); let float_prim = PrimitiveArray::new(Buffer::from(floats.clone()), NonNullable); - let alp = alp_encode(&float_prim, Some(exponents))?; + let alp = alp_encode(float_prim.as_view(), Some(exponents), &mut ctx)?; assert!(alp.patches().is_none()); - let for_arr = FoR::encode(alp.encoded().to_primitive())?; - let bp = BitPacked::encode(for_arr.encoded(), 6)?; + let for_arr = FoR::encode(alp.encoded().clone().execute::(&mut ctx)?)?; + let bp = BitPacked::encode( + for_arr.encoded(), + 6, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; let tree = ALP::new( FoR::try_new(bp.into_array(), for_arr.reference_scalar().clone())?.into_array(), @@ -868,8 +925,8 @@ mod tests { None, ); - let cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; - let plan = dispatch_plan(&tree.into_array(), &cuda_ctx)?; + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let plan = dispatch_plan(&tree.into_array(), &mut cuda_ctx).await?; let actual = run_dispatch_plan_f32(&cuda_ctx, len, &plan.dispatch_plan, plan.shared_mem_bytes)?; @@ -879,7 +936,7 @@ mod tests { } #[crate::test] - fn test_zigzag_bitpacked() -> VortexResult<()> { + async fn test_zigzag_bitpacked() -> VortexResult<()> { // ZigZag(BitPacked): unpack then zigzag-decode. let bit_width: u8 = 4; let len = 3000; @@ -894,11 +951,15 @@ mod tests { .collect(); let prim = PrimitiveArray::new(Buffer::from(raw), NonNullable); - let bp = BitPacked::encode(&prim.into_array(), bit_width)?; + let bp = BitPacked::encode( + &prim.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; let zz = ZigZag::try_new(bp.into_array())?; - let cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; - let plan = dispatch_plan(&zz.into_array(), &cuda_ctx)?; + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let plan = dispatch_plan(&zz.into_array(), &mut cuda_ctx).await?; let actual = run_dynamic_dispatch_plan(&cuda_ctx, len, &plan.dispatch_plan, plan.shared_mem_bytes)?; @@ -908,7 +969,7 @@ mod tests { } #[crate::test] - fn test_for_runend() -> VortexResult<()> { + async fn test_for_runend() -> VortexResult<()> { // FoR(RunEnd): expand runs then add constant. let ends: Vec = vec![500, 1000, 1500, 2000, 2500, 3000]; let values: Vec = vec![1, 2, 3, 4, 5, 6]; @@ -921,13 +982,13 @@ mod tests { expected.push(values[run] + reference); } + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; let ends_arr = PrimitiveArray::new(Buffer::from(ends), NonNullable).into_array(); let values_arr = PrimitiveArray::new(Buffer::from(values), NonNullable).into_array(); - let re = RunEnd::new(ends_arr, values_arr); + let re = RunEnd::new(ends_arr, values_arr, cuda_ctx.execution_ctx()); let for_arr = FoR::try_new(re.into_array(), Scalar::from(reference))?; - let cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; - let plan = dispatch_plan(&for_arr.into_array(), &cuda_ctx)?; + let plan = dispatch_plan(&for_arr.into_array(), &mut cuda_ctx).await?; let actual = run_dynamic_dispatch_plan(&cuda_ctx, len, &plan.dispatch_plan, plan.shared_mem_bytes)?; @@ -937,7 +998,7 @@ mod tests { } #[crate::test] - fn test_for_dict() -> VortexResult<()> { + async fn test_for_dict() -> VortexResult<()> { // FoR(Dict(codes=Primitive, values=Primitive)): gather then add constant. let dict_values: Vec = vec![100, 200, 300, 400]; let dict_size = dict_values.len(); @@ -955,8 +1016,8 @@ mod tests { let dict = DictArray::try_new(codes_prim.into_array(), values_prim.into_array())?; let for_arr = FoR::try_new(dict.into_array(), Scalar::from(reference))?; - let cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; - let plan = dispatch_plan(&for_arr.into_array(), &cuda_ctx)?; + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let plan = dispatch_plan(&for_arr.into_array(), &mut cuda_ctx).await?; let actual = run_dynamic_dispatch_plan(&cuda_ctx, len, &plan.dispatch_plan, plan.shared_mem_bytes)?; @@ -966,7 +1027,7 @@ mod tests { } #[crate::test] - fn test_dict_for_bp_codes() -> VortexResult<()> { + async fn test_dict_for_bp_codes() -> VortexResult<()> { // Dict(codes=FoR(BitPacked), values=primitive) let dict_values: Vec = (0..8).map(|i| i * 1000 + 7).collect(); let dict_size = dict_values.len(); @@ -977,14 +1038,18 @@ mod tests { // BitPack codes, then wrap in FoR (reference=0 so values unchanged) let bit_width: u8 = 3; let codes_prim = PrimitiveArray::new(Buffer::from(codes), NonNullable); - let codes_bp = BitPacked::encode(&codes_prim.into_array(), bit_width)?; + let codes_bp = BitPacked::encode( + &codes_prim.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; let codes_for = FoR::try_new(codes_bp.into_array(), Scalar::from(0u32))?; let values_prim = PrimitiveArray::new(Buffer::from(dict_values), NonNullable); let dict = DictArray::try_new(codes_for.into_array(), values_prim.into_array())?; - let cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; - let plan = dispatch_plan(&dict.into_array(), &cuda_ctx)?; + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let plan = dispatch_plan(&dict.into_array(), &mut cuda_ctx).await?; let actual = run_dynamic_dispatch_plan(&cuda_ctx, len, &plan.dispatch_plan, plan.shared_mem_bytes)?; @@ -994,7 +1059,7 @@ mod tests { } #[crate::test] - fn test_dict_primitive_values_bp_codes() -> VortexResult<()> { + async fn test_dict_primitive_values_bp_codes() -> VortexResult<()> { let dict_values: Vec = vec![100, 200, 300, 400]; let dict_size = dict_values.len(); let len = 3000; @@ -1003,13 +1068,17 @@ mod tests { let bit_width: u8 = 2; let codes_prim = PrimitiveArray::new(Buffer::from(codes), NonNullable); - let codes_bp = BitPacked::encode(&codes_prim.into_array(), bit_width)?; + let codes_bp = BitPacked::encode( + &codes_prim.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; let values_prim = PrimitiveArray::new(Buffer::from(dict_values), NonNullable); let dict = DictArray::try_new(codes_bp.into_array(), values_prim.into_array())?; - let cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; - let plan = dispatch_plan(&dict.into_array(), &cuda_ctx)?; + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let plan = dispatch_plan(&dict.into_array(), &mut cuda_ctx).await?; let actual = run_dynamic_dispatch_plan(&cuda_ctx, len, &plan.dispatch_plan, plan.shared_mem_bytes)?; @@ -1032,7 +1101,7 @@ mod tests { // Mixed-width Dict (u8 codes, u32 values): both are Primitive, so // walk_mixed_width_child grabs the codes buffer directly as a LOAD // source. No pending subtrees → Fused. - let plan = DispatchPlan::new(&array)?; + let plan = DispatchPlan::new(&array, CudaDispatchMode::Auto)?; assert!( matches!(plan, DispatchPlan::Fused(..)), "expected Fused for mixed-width Dict with primitive codes" @@ -1064,7 +1133,7 @@ mod tests { // Mixed-width Dict (u16 codes, u32 values): both are Primitive, so // walk_mixed_width_child grabs the codes buffer directly as a LOAD // source. No pending subtrees → Fused. - let plan = DispatchPlan::new(&array)?; + let plan = DispatchPlan::new(&array, CudaDispatchMode::Auto)?; assert!( matches!(plan, DispatchPlan::Fused(..)), "expected Fused for mixed-width Dict with primitive codes" @@ -1088,21 +1157,21 @@ mod tests { let values: Vec = vec![10, 20, 30]; let len = 3000; + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; let ends_arr = PrimitiveArray::new(Buffer::from(ends), NonNullable).into_array(); let values_arr = PrimitiveArray::new(Buffer::from(values), NonNullable).into_array(); - let re = RunEnd::new(ends_arr, values_arr); + let re = RunEnd::new(ends_arr, values_arr, cuda_ctx.execution_ctx()); let array = re.into_array(); // Ends (u64) are wider than values (u32), so the kernel would truncate // ends via load_element. The plan builder rejects this as Unfused. - let plan = DispatchPlan::new(&array)?; + let plan = DispatchPlan::new(&array, CudaDispatchMode::Auto)?; assert!( matches!(plan, DispatchPlan::Unfused), "expected Unfused for RunEnd with wider ends" ); // Execute through the non-fused dispatch path. - let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; let canonical = try_gpu_dispatch(&array, &mut cuda_ctx).await?; let result = CanonicalCudaExt::into_host(canonical).await?.into_array(); @@ -1139,7 +1208,7 @@ mod tests { #[case(2500, 4500)] #[case(3333, 4444)] #[crate::test] - fn test_sliced_primitive( + async fn test_sliced_primitive( #[case] slice_start: usize, #[case] slice_end: usize, ) -> VortexResult<()> { @@ -1152,8 +1221,8 @@ mod tests { let expected: Vec = data[slice_start..slice_end].to_vec(); - let cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; - let plan = dispatch_plan(&sliced, &cuda_ctx)?; + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let plan = dispatch_plan(&sliced, &mut cuda_ctx).await?; let actual = run_dynamic_dispatch_plan( &cuda_ctx, @@ -1182,7 +1251,7 @@ mod tests { #[case(2500, 4500)] #[case(3333, 4444)] #[crate::test] - fn test_sliced_zigzag_bitpacked( + async fn test_sliced_zigzag_bitpacked( #[case] slice_start: usize, #[case] slice_end: usize, ) -> VortexResult<()> { @@ -1197,14 +1266,18 @@ mod tests { .collect(); let prim = PrimitiveArray::new(Buffer::from(raw), NonNullable); - let bp = BitPacked::encode(&prim.into_array(), bit_width)?; + let bp = BitPacked::encode( + &prim.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; let zz = ZigZag::try_new(bp.into_array())?; let sliced = zz.into_array().slice(slice_start..slice_end)?; let expected: Vec = all_decoded[slice_start..slice_end].to_vec(); - let cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; - let plan = dispatch_plan(&sliced, &cuda_ctx)?; + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let plan = dispatch_plan(&sliced, &mut cuda_ctx).await?; let actual = run_dynamic_dispatch_plan( &cuda_ctx, @@ -1233,7 +1306,7 @@ mod tests { #[case(2500, 4500)] #[case(3333, 4444)] #[crate::test] - fn test_sliced_dict_with_primitive_codes( + async fn test_sliced_dict_with_primitive_codes( #[case] slice_start: usize, #[case] slice_end: usize, ) -> VortexResult<()> { @@ -1253,8 +1326,8 @@ mod tests { .map(|&c| dict_values[c as usize]) .collect(); - let cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; - let plan = dispatch_plan(&sliced, &cuda_ctx)?; + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let plan = dispatch_plan(&sliced, &mut cuda_ctx).await?; let actual = run_dynamic_dispatch_plan( &cuda_ctx, @@ -1283,7 +1356,7 @@ mod tests { #[case(2500, 4500)] #[case(3333, 4444)] #[crate::test] - fn test_sliced_bitpacked( + async fn test_sliced_bitpacked( #[case] slice_start: usize, #[case] slice_end: usize, ) -> VortexResult<()> { @@ -1293,13 +1366,17 @@ mod tests { let data: Vec = (0..len).map(|i| (i as u32) % max_val).collect(); let prim = PrimitiveArray::new(Buffer::from(data.clone()), NonNullable); - let bp = BitPacked::encode(&prim.into_array(), bit_width)?; + let bp = BitPacked::encode( + &prim.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; let sliced = bp.into_array().slice(slice_start..slice_end)?; let expected: Vec = data[slice_start..slice_end].to_vec(); - let cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; - let plan = dispatch_plan(&sliced, &cuda_ctx)?; + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let plan = dispatch_plan(&sliced, &mut cuda_ctx).await?; let actual = run_dynamic_dispatch_plan( &cuda_ctx, @@ -1328,7 +1405,7 @@ mod tests { #[case(2500, 4500)] #[case(3333, 4444)] #[crate::test] - fn test_sliced_for_bitpacked( + async fn test_sliced_for_bitpacked( #[case] slice_start: usize, #[case] slice_end: usize, ) -> VortexResult<()> { @@ -1339,7 +1416,11 @@ mod tests { let encoded_data: Vec = (0..len).map(|i| (i as u32) % max_val).collect(); let prim = PrimitiveArray::new(Buffer::from(encoded_data.clone()), NonNullable); - let bp = BitPacked::encode(&prim.into_array(), bit_width)?; + let bp = BitPacked::encode( + &prim.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; let for_arr = FoR::try_new(bp.into_array(), Scalar::from(reference))?; let all_decoded: Vec = encoded_data.iter().map(|&v| v + reference).collect(); @@ -1347,8 +1428,8 @@ mod tests { let sliced = for_arr.into_array().slice(slice_start..slice_end)?; let expected: Vec = all_decoded[slice_start..slice_end].to_vec(); - let cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; - let plan = dispatch_plan(&sliced, &cuda_ctx)?; + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let plan = dispatch_plan(&sliced, &mut cuda_ctx).await?; let actual = run_dynamic_dispatch_plan( &cuda_ctx, @@ -1377,7 +1458,7 @@ mod tests { #[case(2500, 4500)] #[case(3333, 4444)] #[crate::test] - fn test_sliced_dict_for_bp_values_bp_codes( + async fn test_sliced_dict_for_bp_values_bp_codes( #[case] slice_start: usize, #[case] slice_end: usize, ) -> VortexResult<()> { @@ -1392,20 +1473,28 @@ mod tests { // BitPack+FoR the dict values let dict_prim = PrimitiveArray::new(Buffer::from(dict_residuals), NonNullable); - let dict_bp = BitPacked::encode(&dict_prim.into_array(), 6)?; + let dict_bp = BitPacked::encode( + &dict_prim.into_array(), + 6, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; let dict_for = FoR::try_new(dict_bp.into_array(), Scalar::from(dict_reference))?; // BitPack the codes let codes_prim = PrimitiveArray::new(Buffer::from(codes), NonNullable); - let codes_bp = BitPacked::encode(&codes_prim.into_array(), 6)?; + let codes_bp = BitPacked::encode( + &codes_prim.into_array(), + 6, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; let dict = DictArray::try_new(codes_bp.into_array(), dict_for.into_array())?; let sliced = dict.into_array().slice(slice_start..slice_end)?; let expected: Vec = all_decoded[slice_start..slice_end].to_vec(); - let cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; - let plan = dispatch_plan(&sliced, &cuda_ctx)?; + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let plan = dispatch_plan(&sliced, &mut cuda_ctx).await?; let actual = run_dynamic_dispatch_plan( &cuda_ctx, @@ -1424,7 +1513,7 @@ mod tests { #[case(0u32, 1u32, 4096)] #[case(100u32, 7u32, 5000)] #[crate::test] - fn test_sequence_unsigned( + async fn test_sequence_unsigned( #[case] base: u32, #[case] multiplier: u32, #[case] len: usize, @@ -1436,8 +1525,8 @@ mod tests { let seq = Sequence::try_new_typed(base, multiplier, Nullability::NonNullable, len)?; - let cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; - let plan = dispatch_plan(&seq.into_array(), &cuda_ctx)?; + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let plan = dispatch_plan(&seq.into_array(), &mut cuda_ctx).await?; let actual = run_dynamic_dispatch_plan( &cuda_ctx, @@ -1457,7 +1546,7 @@ mod tests { #[case(-500i32, -7i32, 50)] #[case(0i32, 1i32, 5000)] #[crate::test] - fn test_sequence_signed( + async fn test_sequence_signed( #[case] base: i32, #[case] multiplier: i32, #[case] len: usize, @@ -1469,8 +1558,8 @@ mod tests { let seq = Sequence::try_new_typed(base, multiplier, Nullability::NonNullable, len)?; - let cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; - let plan = dispatch_plan(&seq.into_array(), &cuda_ctx)?; + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let plan = dispatch_plan(&seq.into_array(), &mut cuda_ctx).await?; let actual_u32 = run_dynamic_dispatch_plan( &cuda_ctx, @@ -1497,7 +1586,12 @@ mod tests { .collect(); let primitive = PrimitiveArray::new(Buffer::from(residuals), NonNullable); - let bp = BitPacked::encode(&primitive.into_array(), bit_width).vortex_expect("bitpack u8"); + let bp = BitPacked::encode( + &primitive.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .vortex_expect("bitpack u8"); let for_arr = FoR::try_new( bp.into_array(), Scalar::primitive(reference, Nullability::NonNullable), @@ -1528,7 +1622,12 @@ mod tests { .collect(); let primitive = PrimitiveArray::new(Buffer::from(residuals), NonNullable); - let bp = BitPacked::encode(&primitive.into_array(), bit_width).vortex_expect("bitpack u16"); + let bp = BitPacked::encode( + &primitive.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .vortex_expect("bitpack u16"); let for_arr = FoR::try_new( bp.into_array(), Scalar::primitive(reference, Nullability::NonNullable), @@ -1557,7 +1656,12 @@ mod tests { .collect(); let primitive = PrimitiveArray::new(Buffer::from(residuals), NonNullable); - let bp = BitPacked::encode(&primitive.into_array(), bit_width).vortex_expect("bitpack u64"); + let bp = BitPacked::encode( + &primitive.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .vortex_expect("bitpack u64"); let for_arr = FoR::try_new( bp.into_array(), Scalar::primitive(reference, Nullability::NonNullable), @@ -1588,7 +1692,12 @@ mod tests { async fn test_single_element() -> VortexResult<()> { let values: Vec = vec![42]; let primitive = PrimitiveArray::new(Buffer::from(values.clone()), NonNullable); - let bp = BitPacked::encode(&primitive.into_array(), 6).vortex_expect("bitpack"); + let bp = BitPacked::encode( + &primitive.into_array(), + 6, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .vortex_expect("bitpack"); let for_arr = FoR::try_new( bp.into_array(), Scalar::primitive(0u32, Nullability::NonNullable), @@ -1617,7 +1726,12 @@ mod tests { let expected: Vec = residuals.iter().map(|&r| r + reference).collect(); let primitive = PrimitiveArray::new(Buffer::from(residuals), NonNullable); - let bp = BitPacked::encode(&primitive.into_array(), bit_width).vortex_expect("bitpack"); + let bp = BitPacked::encode( + &primitive.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .vortex_expect("bitpack"); let for_arr = FoR::try_new( bp.into_array(), Scalar::primitive(reference, Nullability::NonNullable), @@ -1633,17 +1747,228 @@ mod tests { Ok(()) } + // --------------------------------------------------------------- + // has_standalone_kernel classification tests + // --------------------------------------------------------------- + + #[crate::test] + fn test_has_standalone_kernel_true_cases() -> VortexResult<()> { + // BitPacked — leaf encoding, no children. + let bp = bitpacked_array_u32(6, 2048); + let bp_arr = bp.into_array(); + assert!(plan_builder::has_standalone_kernel(&bp_arr)); + + // Sequence — leaf encoding, no children. + use vortex::dtype::Nullability; + use vortex::encodings::sequence::Sequence; + let seq = Sequence::try_new_typed(0u32, 1u32, Nullability::NonNullable, 2048)?; + assert!(plan_builder::has_standalone_kernel(&seq.into_array())); + + // FoR(BitPacked) — FFOR fusion, single launch. + let for_bp = FoR::try_new(bp_arr.clone(), 100u32.into())?; + assert!(plan_builder::has_standalone_kernel(&for_bp.into_array())); + + // FoR(Slice(BitPacked)) — FFOR + slice fusion, single launch. + let sliced_bp = bp_arr.slice(100..1500)?; + let for_sliced_bp = FoR::try_new(sliced_bp, 100u32.into())?; + assert!(plan_builder::has_standalone_kernel( + &for_sliced_bp.into_array() + )); + + Ok(()) + } + + #[crate::test] + fn test_has_standalone_kernel_false_cases() -> VortexResult<()> { + // ALP(Primitive) — ALP always calls execute_cuda on its encoded child. + let encoded = + PrimitiveArray::new(Buffer::from((0i32..2048).collect::>()), NonNullable); + let alp = ALP::try_new(encoded.into_array(), Exponents { e: 2, f: 0 }, None)?; + assert!(!plan_builder::has_standalone_kernel(&alp.into_array())); + + // FoR(Primitive) — FoR standalone would recurse for non-BP child. + let prim = PrimitiveArray::new(Buffer::from(vec![1u32, 2, 3]), NonNullable); + let for_prim = FoR::try_new(prim.into_array(), 100u32.into())?; + assert!(!plan_builder::has_standalone_kernel(&for_prim.into_array())); + + // Dict and RunEnd always recurse into children. + let codes = PrimitiveArray::new(Buffer::from(vec![0u32, 1, 0, 1]), NonNullable); + let values = PrimitiveArray::new(Buffer::from(vec![100u32, 200]), NonNullable); + let dict = DictArray::try_new(codes.into_array(), values.into_array())?; + assert!(!plan_builder::has_standalone_kernel(&dict.into_array())); + + Ok(()) + } + + /// Every encoding `has_standalone_kernel` returns `true` for must have + /// a kernel registered in the CUDA session. #[crate::test] - fn test_f64_rejected() { - // F64 arrays should be rejected by the plan builder, not silently accepted. + fn test_has_standalone_kernel_implies_registered_kernel() -> VortexResult<()> { + use vortex::dtype::Nullability; + use vortex::encodings::sequence::Sequence; + + let session = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let cuda_session = session.cuda_session(); + + // Leaf encodings. + let bp = bitpacked_array_u32(6, 2048); + let bp_arr = bp.into_array(); + let seq = Sequence::try_new_typed(0u32, 1u32, Nullability::NonNullable, 2048)?; + let seq_arr = seq.into_array(); + + // FoR fusions. + let for_bp = FoR::try_new(bp_arr.clone(), 100u32.into())?; + let sliced_bp = bp_arr.slice(100..1500)?; + let for_sliced_bp = FoR::try_new(sliced_bp, 100u32.into())?; + + // With patches: some values exceed 2^4-1, creating overflow exceptions. + let values: Vec = (0..2048) + .map(|i| if i % 100 == 0 { 1000 } else { (i as u32) % 16 }) + .collect(); + let patched_bp = BitPacked::encode( + &PrimitiveArray::new(Buffer::from(values), NonNullable).into_array(), + 4, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; + assert!(patched_bp.patches().is_some(), "expected patches"); + let patched_bp_arr = patched_bp.into_array(); + let for_patched_bp = FoR::try_new(patched_bp_arr.clone(), 100u32.into())?; + + for array in [ + &bp_arr, + &seq_arr, + &for_bp.into_array(), + &for_sliced_bp.into_array(), + &patched_bp_arr, + &for_patched_bp.into_array(), + ] { + assert!( + plan_builder::has_standalone_kernel(array), + "expected has_standalone_kernel=true for {:?}", + array.encoding_id() + ); + assert!( + cuda_session.kernel(&array.encoding_id()).is_some(), + "has_standalone_kernel=true but no kernel registered for {:?}", + array.encoding_id() + ); + } + + Ok(()) + } + + #[crate::test] + fn test_f64_primitive_fuses() { + // Raw F64 primitives are now accepted — the kernel operates on uint64_t + // (same bit width), so LOAD correctly preserves the f64 bit pattern. let values: Vec = vec![1.0, 2.0, 3.0]; let primitive = PrimitiveArray::new(Buffer::from(values), NonNullable); - let plan = DispatchPlan::new(&primitive.into_array()) - .expect("DispatchPlan::new should not fail for f64"); + let plan = DispatchPlan::new(&primitive.into_array(), CudaDispatchMode::Auto) + .expect("DispatchPlan::new should not fail for f64 primitive"); assert!( - matches!(plan, DispatchPlan::Unfused), - "expected F64 to be classified as Unfused" + matches!(plan, DispatchPlan::Fused(_)), + "expected F64 primitive to be classified as Fused" + ); + } + + #[crate::test] + async fn test_alp_f64_for_bitpacked() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + // ALP(FoR(BitPacked)) with f64: same structure as the f32 test. + let len = 3000; + let exponents = Exponents { e: 2, f: 0 }; + let floats: Vec = (0..len) + .map(|i| ::decode_single(10 + (i as i64 % 64), exponents)) + .collect(); + let float_prim = PrimitiveArray::new(Buffer::from(floats), NonNullable); + + let alp = alp_encode(float_prim.as_view(), Some(exponents), &mut ctx)?; + assert!(alp.patches().is_none()); + let for_arr = FoR::encode(alp.encoded().clone().execute::(&mut ctx)?)?; + let bp = BitPacked::encode( + for_arr.encoded(), + 6, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; + + let tree = ALP::new( + FoR::try_new(bp.into_array(), for_arr.reference_scalar().clone())?.into_array(), + exponents, + None, ); + let array = tree.into_array(); + + // CPU decode as ground truth. + let cpu = array + .clone() + .execute::(&mut LEGACY_SESSION.create_execution_ctx())? + .into_array(); + + // GPU decode. + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let canonical = try_gpu_dispatch(&array, &mut cuda_ctx).await?; + let gpu = CanonicalCudaExt::into_host(canonical).await?.into_array(); + + vortex::array::assert_arrays_eq!(cpu, gpu); + + Ok(()) + } + + #[rstest] + #[case(4096, None)] + #[case(4096, Some(100..3000))] + #[crate::test] + async fn test_alp_f64_with_patches( + #[case] len: usize, + #[case] slice_range: Option>, + ) -> VortexResult<()> { + let mut values: Vec = (0..len).map(|i| (i as f64) * 1.1).collect(); + // Insert exception values that ALP can't encode. + values[0] = 99.9; + values[500] = std::f64::consts::PI; + values[1024] = std::f64::consts::E; + if len > 2048 { + values[2048] = std::f64::consts::LN_2; + } + if len > 3333 { + values[3333] = std::f64::consts::SQRT_2; + } + + let float_prim = PrimitiveArray::new(Buffer::from(values), NonNullable); + let encoded = alp_encode( + float_prim.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + )? + .into_array(); + + let (array, base_offset) = if let Some(range) = &slice_range { + (encoded.slice(range.clone())?, range.start) + } else { + (encoded, 0) + }; + + // Decode on CPU as ground truth (accounts for ALP precision loss + patches). + let cpu_decoded = array + .clone() + .execute::(&mut LEGACY_SESSION.create_execution_ctx())?; + let expected: Vec = cpu_decoded.as_slice::().to_vec(); + + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let canonical = try_gpu_dispatch(&array, &mut cuda_ctx).await?; + let result = CanonicalCudaExt::into_host(canonical).await?; + let result_prim = result.as_primitive(); + let actual: Vec = result_prim.as_slice::().to_vec(); + for (i, (&a, &e)) in actual.iter().zip(expected.iter()).enumerate() { + assert!( + a.to_bits() == e.to_bits(), + "mismatch at index {i} (original index {}): gpu={a} cpu={e} (bits: {:#018x} vs {:#018x})", + i + base_offset, + a.to_bits(), + e.to_bits(), + ); + } + Ok(()) } #[crate::test] @@ -1654,20 +1979,20 @@ mod tests { let values: Vec = vec![100, 200, 300, 400]; let len = 2000; + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; let ends_arr = PrimitiveArray::new(Buffer::from(ends), NonNullable).into_array(); let values_arr = PrimitiveArray::new(Buffer::from(values), NonNullable).into_array(); - let re = RunEnd::new(ends_arr, values_arr); + let re = RunEnd::new(ends_arr, values_arr, cuda_ctx.execution_ctx()); let array = re.into_array(); // Ends (u32) are wider than values (u16), so the kernel would truncate // ends via load_element. The plan builder rejects this as Unfused. - let plan = DispatchPlan::new(&array)?; + let plan = DispatchPlan::new(&array, CudaDispatchMode::Auto)?; assert!( matches!(plan, DispatchPlan::Unfused), "expected Unfused for RunEnd with wider ends" ); - let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; let canonical = try_gpu_dispatch(&array, &mut cuda_ctx).await?; let result = CanonicalCudaExt::into_host(canonical).await?.into_array(); @@ -1693,23 +2018,28 @@ mod tests { #[crate::test] async fn test_dict_bitpacked_u8_codes_u32_values() -> VortexResult<()> { // Dict with BitPacked u8 codes (narrower than u32 output) and u32 values. - // Codes become a pending subtree, values fuse. + // The kernel's bitunpack_typed decodes at the source's native width and + // widens to T, so this fuses into a single kernel launch. let dict_values: Vec = vec![100, 200, 300, 400]; let len = 2048; let codes: Vec = (0..len).map(|i| (i % dict_values.len()) as u8).collect(); let codes_prim = PrimitiveArray::new(Buffer::from(codes.clone()), NonNullable); // BitPack the u8 codes at 2 bits (4 values need 2 bits) - let codes_bp = - BitPacked::encode(&codes_prim.into_array(), 2).vortex_expect("bitpack codes"); + let codes_bp = BitPacked::encode( + &codes_prim.into_array(), + 2, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .vortex_expect("bitpack codes"); let values_prim = PrimitiveArray::new(Buffer::from(dict_values.clone()), NonNullable); let dict = DictArray::try_new(codes_bp.into_array(), values_prim.into_array())?; let array = dict.into_array(); - let plan = DispatchPlan::new(&array)?; + let plan = DispatchPlan::new(&array, CudaDispatchMode::Auto)?; assert!( - matches!(plan, DispatchPlan::PartiallyFused { .. }), - "expected PartiallyFused for mixed-width Dict with BitPacked codes" + matches!(plan, DispatchPlan::Fused(..)), + "expected Fused for mixed-width Dict with BitPacked codes" ); let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; @@ -1723,6 +2053,202 @@ mod tests { Ok(()) } + /// Dict with non-Primitive narrower codes (BitPacked) at various + /// code/value widths. These fuse into a single kernel because + /// bitunpack_typed decodes at the source's native width and widens to T. + #[rstest] + #[case::bp_u8_codes_u16_values(2u8, 3000usize, vec![100u16, 200, 300, 400])] + #[case::bp_u8_codes_u32_values(2u8, 3000usize, vec![100_000u32, 200_000, 300_000, 400_000])] + #[case::bp_u16_codes_u32_values(4u8, 2048usize, vec![1_000_000u32, 2_000_000, 3_000_000, 4_000_000, 5_000_000, 6_000_000, 7_000_000, 8_000_000])] + #[crate::test] + async fn test_dict_mixed_width_bitpacked_codes( + #[case] bit_width: u8, + #[case] len: usize, + #[case] dict_values: Vec, + ) -> VortexResult<()> { + let dict_size = dict_values.len(); + let codes: Vec = (0..len).map(|i| (i % dict_size) as u8).collect(); + + let codes_prim = PrimitiveArray::new(Buffer::from(codes.clone()), NonNullable); + let codes_bp = BitPacked::encode( + &codes_prim.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .vortex_expect("bitpack codes"); + let values_prim = PrimitiveArray::new(Buffer::from(dict_values.clone()), NonNullable); + let dict = DictArray::try_new(codes_bp.into_array(), values_prim.into_array())?; + let array = dict.into_array(); + + let plan = DispatchPlan::new(&array, CudaDispatchMode::Auto)?; + assert!( + matches!(plan, DispatchPlan::Fused(..)), + "expected Fused for mixed-width Dict with BitPacked codes" + ); + + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let canonical = try_gpu_dispatch(&array, &mut cuda_ctx).await?; + let result = CanonicalCudaExt::into_host(canonical).await?.into_array(); + + let expected: Vec = codes.iter().map(|&c| dict_values[c as usize]).collect(); + let expected_arr = PrimitiveArray::new(Buffer::from(expected), NonNullable).into_array(); + vortex::array::assert_arrays_eq!(expected_arr, result); + + Ok(()) + } + + /// Dict with FoR(BitPacked) codes narrower than the value type. + /// The entire codes decode chain runs at the narrower width; the kernel + /// decodes at native width and widens to T, fusing everything. + #[rstest] + #[case::for_bp_u8_codes_u32_values(3u8, 3000usize, vec![100u32, 200, 300, 400, 500, 600, 700, 800])] + #[case::for_bp_u16_codes_u32_values(4u8, 2048usize, vec![10_000u32, 20_000, 30_000, 40_000, 50_000, 60_000, 70_000, 80_000])] + #[crate::test] + async fn test_dict_mixed_width_for_bp_codes( + #[case] bit_width: u8, + #[case] len: usize, + #[case] dict_values: Vec, + ) -> VortexResult<()> { + let dict_size = dict_values.len(); + let codes: Vec = (0..len).map(|i| (i % dict_size) as u8).collect(); + + let codes_prim = PrimitiveArray::new(Buffer::from(codes.clone()), NonNullable); + let codes_bp = BitPacked::encode( + &codes_prim.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .vortex_expect("bitpack codes"); + let codes_for = FoR::try_new(codes_bp.into_array(), Scalar::from(0u8))?; + let values_prim = PrimitiveArray::new(Buffer::from(dict_values.clone()), NonNullable); + let dict = DictArray::try_new(codes_for.into_array(), values_prim.into_array())?; + let array = dict.into_array(); + + let plan = DispatchPlan::new(&array, CudaDispatchMode::Auto)?; + assert!( + matches!(plan, DispatchPlan::Fused(..)), + "expected Fused for mixed-width Dict with FoR(BitPacked) codes" + ); + + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let canonical = try_gpu_dispatch(&array, &mut cuda_ctx).await?; + let result = CanonicalCudaExt::into_host(canonical).await?.into_array(); + + let expected: Vec = codes.iter().map(|&c| dict_values[c as usize]).collect(); + let expected_arr = PrimitiveArray::new(Buffer::from(expected), NonNullable).into_array(); + vortex::array::assert_arrays_eq!(expected_arr, result); + + Ok(()) + } + + /// RunEnd with non-Primitive narrower ends (BitPacked) and wider output + /// values. The kernel decodes at the source's native width and widens + /// to T in shared memory, fusing everything into one launch. + #[rstest] + #[case::bp_u8_ends_u16_values( + vec![25u8, 50, 75, 100], + vec![10u16, 20, 30, 40], + )] + #[case::bp_u8_ends_u32_values( + vec![40u8, 80, 120], + vec![1000u32, 2000, 3000], + )] + #[case::bp_u16_ends_u32_values( + vec![500u16, 1000, 1500, 2000], + vec![100_000u32, 200_000, 300_000, 400_000], + )] + #[crate::test] + async fn test_runend_mixed_width_bitpacked_ends< + E: vortex::dtype::NativePType + Into, + V: vortex::dtype::NativePType, + >( + #[case] ends: Vec, + #[case] values: Vec, + ) -> VortexResult<()> { + let ends_u64: Vec = ends.iter().map(|e| (*e).into()).collect(); + let len = *ends_u64.last().unwrap() as usize; + let bit_width = 64 - ends_u64.iter().max().unwrap().leading_zeros() as u8; + let ends_prim = PrimitiveArray::new(Buffer::from(ends.clone()), NonNullable); + let ends_bp = BitPacked::encode( + &ends_prim.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .vortex_expect("bitpack ends"); + + let values_prim = PrimitiveArray::new(Buffer::from(values.clone()), NonNullable); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let re = RunEnd::new(ends_bp.into_array(), values_prim.into_array(), &mut ctx); + let array = re.into_array(); + + let plan = DispatchPlan::new(&array, CudaDispatchMode::Auto)?; + assert!( + matches!(plan, DispatchPlan::Fused(..)), + "expected Fused for mixed-width RunEnd with BitPacked ends" + ); + + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let canonical = try_gpu_dispatch(&array, &mut cuda_ctx).await?; + let result = CanonicalCudaExt::into_host(canonical).await?.into_array(); + + let mut expected: Vec = Vec::with_capacity(len); + let mut prev: u64 = 0; + for (end, val) in ends_u64.iter().zip(values.iter()) { + let run_len = (*end - prev) as usize; + expected.extend(std::iter::repeat_n(*val, run_len)); + prev = *end; + } + let expected_arr = PrimitiveArray::new(Buffer::from(expected), NonNullable).into_array(); + vortex::array::assert_arrays_eq!(expected_arr, result); + + Ok(()) + } + + /// RunEnd with FoR(BitPacked) narrower ends: the full decode chain for + /// ends runs at a narrower width; the kernel's bitunpack_typed decodes + /// at native width and widens to T, fusing everything. + #[crate::test] + async fn test_runend_mixed_width_for_bp_u16_ends_u32_values() -> VortexResult<()> { + let ends: Vec = vec![500, 1000, 1500, 2000]; + let values: Vec = vec![100, 200, 300, 400]; + let len = 2000usize; + + let ends_prim = PrimitiveArray::new(Buffer::from(ends.clone()), NonNullable); + let ends_bp = BitPacked::encode( + &ends_prim.into_array(), + 11, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .vortex_expect("bitpack ends"); + let ends_for = FoR::try_new(ends_bp.into_array(), Scalar::from(0u16))?; + + let values_prim = PrimitiveArray::new(Buffer::from(values.clone()), NonNullable); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let re = RunEnd::new(ends_for.into_array(), values_prim.into_array(), &mut ctx); + let array = re.into_array(); + + let plan = DispatchPlan::new(&array, CudaDispatchMode::Auto)?; + assert!( + matches!(plan, DispatchPlan::Fused(..)), + "expected Fused for mixed-width RunEnd with FoR(BitPacked) ends" + ); + + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let canonical = try_gpu_dispatch(&array, &mut cuda_ctx).await?; + let result = CanonicalCudaExt::into_host(canonical).await?.into_array(); + + let mut expected: Vec = Vec::with_capacity(len); + let mut prev = 0u16; + for (&end, &val) in ends.iter().zip(values.iter()) { + expected.extend(std::iter::repeat_n(val, (end - prev) as usize)); + prev = end; + } + let expected_arr = PrimitiveArray::new(Buffer::from(expected), NonNullable).into_array(); + vortex::array::assert_arrays_eq!(expected_arr, result); + + Ok(()) + } + #[crate::test] async fn test_sliced_dict_mixed_width() -> VortexResult<()> { // Sliced Dict with u8 codes and u32 values — combines PartiallyFused + slice handling. @@ -1762,7 +2288,12 @@ mod tests { let i8_values: Vec = vec![-1, -2, -3, 127, -128, 0, 1, 42]; let len = i8_values.len(); - let device_buf = Arc::new(cuda_ctx.stream().clone_htod(&i8_values).expect("htod")); + let device_buf = Arc::new( + cuda_ctx + .stream() + .clone_htod(&i8_values) + .map_err(|e| vortex_err!("htod: {e}"))?, + ); let (input_ptr, _) = device_buf.device_ptr(cuda_ctx.stream()); // Build a single-stage LOAD plan: source ptype = I8, output ptype = U32. @@ -1795,7 +2326,12 @@ mod tests { let i16_values: Vec = vec![-1, -256, -32768, 32767, 0, 1, -100, 12345]; let len = i16_values.len(); - let device_buf = Arc::new(cuda_ctx.stream().clone_htod(&i16_values).expect("htod")); + let device_buf = Arc::new( + cuda_ctx + .stream() + .clone_htod(&i16_values) + .map_err(|e| vortex_err!("htod: {e}"))?, + ); let (input_ptr, _) = device_buf.device_ptr(cuda_ctx.stream()); let plan = CudaDispatchPlan::new( @@ -1817,4 +2353,648 @@ mod tests { Ok(()) } + + // ═══════════════════════════════════════════════════════════════════ + // Validity propagation tests + // ═══════════════════════════════════════════════════════════════════ + + /// Nullable Primitive array — LOAD source with validity propagated. + #[crate::test] + async fn test_nullable_primitive() -> VortexResult<()> { + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + + let array = PrimitiveArray::from_option_iter( + (0..2048u32).map(|i| if i % 3 == 0 { None } else { Some(i) }), + ); + let cpu = crate::canonicalize_cpu(array.clone())?.into_array(); + + let gpu = try_gpu_dispatch(&array.into_array(), &mut cuda_ctx) + .await? + .into_host() + .await? + .into_array(); + + vortex::array::assert_arrays_eq!(cpu, gpu); + Ok(()) + } + + /// Nullable FoR(BitPacked) — validity from the root propagated through + /// the fused plan. The standard encoding flow is: subtract FoR reference + /// to get residuals, then bitpack. BitPacked::encode preserves input + /// validity, so this produces a real nullable FoR(BitPacked) tree. + #[crate::test] + async fn test_nullable_for_bitpacked() -> VortexResult<()> { + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + + let len = 2048; + let reference = 1000u32; + + // Original values in [reference, reference+63], every 5th null. + let values: Vec> = (0..len) + .map(|i| { + if i % 5 == 0 { + None + } else { + Some((i as u32 % 64) + reference) + } + }) + .collect(); + let prim = PrimitiveArray::from_option_iter(values.iter().copied()); + let cpu = crate::canonicalize_cpu(prim.clone())?.into_array(); + + // FoR encoding: subtract reference to get residuals [0..63]. + // Null positions get 0 (from from_option_iter), which is fine — + // after subtracting reference it wraps, but validity masks it. + let residuals = + PrimitiveArray::from_option_iter(values.iter().map(|v| v.map(|x| x - reference))); + + // BitPacked::encode preserves nullable validity from the input. + let bp = BitPacked::encode( + &residuals.into_array(), + 6, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; + let for_arr = FoR::try_new(bp.into_array(), reference.into())?; + + // Verify the plan actually fuses (not just a LOAD). + assert!( + matches!( + DispatchPlan::new(&for_arr.clone().into_array(), CudaDispatchMode::Auto)?, + DispatchPlan::Standalone | DispatchPlan::Fused(_) + ), + "FoR(BitPacked) with nullable validity should be fusable" + ); + + let gpu = try_gpu_dispatch(&for_arr.into_array(), &mut cuda_ctx) + .await? + .into_host() + .await? + .into_array(); + + vortex::array::assert_arrays_eq!(cpu, gpu); + Ok(()) + } + + /// AllInvalid array — kernel should be skipped entirely. + #[crate::test] + async fn test_all_invalid_skips_kernel() -> VortexResult<()> { + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + + let array = PrimitiveArray::new(Buffer::from(vec![0u32; 2048]), Validity::AllInvalid); + + let result = try_gpu_dispatch(&array.into_array(), &mut cuda_ctx) + .await? + .into_host() + .await?; + + let prim = result.into_primitive(); + assert_eq!(prim.len(), 2048); + assert!(matches!(prim.validity()?, Validity::AllInvalid)); + Ok(()) + } + + /// AllValid nullable array — should fuse and produce AllValid output. + #[crate::test] + async fn test_all_valid_nullable() -> VortexResult<()> { + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + + let values: Vec = (0..2048).collect(); + let array = PrimitiveArray::new(Buffer::from(values.clone()), Validity::AllValid); + + let cpu = crate::canonicalize_cpu(array.clone())?.into_array(); + let gpu = try_gpu_dispatch(&array.into_array(), &mut cuda_ctx) + .await? + .into_host() + .await? + .into_array(); + + vortex::array::assert_arrays_eq!(cpu, gpu); + Ok(()) + } + + /// Dict with nullable codes must fall back to Unfused (not fused). + #[crate::test] + fn test_dict_nullable_codes_rejected() -> VortexResult<()> { + use vortex::buffer::buffer; + + let codes = PrimitiveArray::from_option_iter([Some(0u32), None, Some(1), None, Some(2)]); + let values = PrimitiveArray::new(buffer![10u32, 20, 30], NonNullable); + let dict = DictArray::try_new(codes.into_array(), values.into_array())?; + + let plan = DispatchPlan::new(&dict.into_array(), CudaDispatchMode::Auto)?; + assert!( + matches!(plan, DispatchPlan::Unfused), + "Dict with nullable codes should fall back to Unfused" + ); + Ok(()) + } + + /// Dict with non-nullable codes but nullable values should still fuse. + #[crate::test] + async fn test_dict_nullable_values_fuses() -> VortexResult<()> { + use vortex::buffer::buffer; + + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + + let codes = PrimitiveArray::new(buffer![0u32, 1, 2, 2, 1, 0], NonNullable); + let values = PrimitiveArray::from_option_iter([Some(10u32), None, Some(30)]); + let dict = DictArray::try_new(codes.into_array(), values.into_array())?; + + let cpu = crate::canonicalize_cpu(dict.clone())?.into_array(); + let gpu = dict + .into_array() + .execute_cuda(&mut cuda_ctx) + .await? + .into_host() + .await? + .into_array(); + + vortex::array::assert_arrays_eq!(cpu, gpu); + Ok(()) + } + + /// Nullable FoR(BitPacked) through CUB filter — the original bug scenario. + /// Validity must survive through fused dispatch and into the filter. + #[crate::test] + async fn test_nullable_fused_then_filter() -> VortexResult<()> { + use vortex::array::arrays::FilterArray; + use vortex::mask::Mask; + + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + + let len = 2048usize; + let values: Vec> = (0..len) + .map(|i| { + if i % 7 == 0 { + None + } else { + Some((i % 64) as u32) + } + }) + .collect(); + let prim = PrimitiveArray::from_option_iter(values.iter().copied()); + + // Keep every other element. + let mask = Mask::from_iter((0..len).map(|i| i % 2 == 0)); + let filter_array = FilterArray::try_new(prim.into_array(), mask)?; + + let cpu = crate::canonicalize_cpu(filter_array.clone())?.into_array(); + let gpu = filter_array + .into_array() + .execute_cuda(&mut cuda_ctx) + .await? + .into_host() + .await? + .into_array(); + + vortex::array::assert_arrays_eq!(cpu, gpu); + Ok(()) + } + + /// Empty nullable array should preserve nullability. + #[crate::test] + async fn test_empty_nullable_array() -> VortexResult<()> { + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + + let array = PrimitiveArray::new(Buffer::::empty(), Validity::AllValid); + let result = try_gpu_dispatch(&array.into_array(), &mut cuda_ctx).await?; + let prim = result.into_primitive(); + assert_eq!(prim.len(), 0); + assert_eq!(prim.validity()?.nullability(), Nullability::Nullable); + Ok(()) + } + + // --------------------------------------------------------------- + // Patch tests — fused dynamic dispatch with exception values + // --------------------------------------------------------------- + + #[rstest] + #[case::unsliced(3000, None)] + #[case::mid_slice(5000, Some(500..3500))] + #[case::start_slice(5000, Some(0..1000))] + #[case::chunk_aligned(5000, Some(1024..3000))] + #[crate::test] + async fn test_bitpacked_with_patches( + #[case] len: usize, + #[case] slice_range: Option>, + ) -> VortexResult<()> { + let bit_width: u8 = 4; + let max_val = (1u32 << bit_width) - 1; + let values: Vec = (0..len) + .map(|i| { + if i % 100 == 0 { + 1000 + } else { + (i as u32) % (max_val + 1) + } + }) + .collect(); + + let prim = PrimitiveArray::new(Buffer::from(values.clone()), NonNullable); + let bp = BitPacked::encode( + &prim.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; + assert!(bp.patches().is_some(), "expected patches"); + + let (array, expected) = if let Some(range) = slice_range { + let sliced = bp.into_array().slice(range.clone())?; + (sliced, values[range].to_vec()) + } else { + (bp.into_array(), values) + }; + + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let plan = dispatch_plan(&array, &mut cuda_ctx).await?; + let actual = run_dynamic_dispatch_plan( + &cuda_ctx, + expected.len(), + &plan.dispatch_plan, + plan.shared_mem_bytes, + )?; + assert_eq!(actual, expected); + Ok(()) + } + + #[rstest] + #[case::unsliced(3000, None)] + #[case::mid_slice(5000, Some(500..3500))] + #[crate::test] + async fn test_for_bitpacked_with_patches( + #[case] len: usize, + #[case] slice_range: Option>, + ) -> VortexResult<()> { + let bit_width: u8 = 6; + let reference = 42u32; + let max_val = (1u32 << bit_width) - 1; + let residuals: Vec = (0..len) + .map(|i| { + if i % 200 == 0 { + 500 + } else { + (i as u32) % (max_val + 1) + } + }) + .collect(); + let all_values: Vec = residuals.iter().map(|&v| v + reference).collect(); + + let prim = PrimitiveArray::new(Buffer::from(residuals), NonNullable); + let bp = BitPacked::encode( + &prim.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; + assert!(bp.patches().is_some(), "expected patches"); + let for_arr = FoR::try_new(bp.into_array(), Scalar::from(reference))?; + + let (array, expected) = if let Some(range) = slice_range { + let sliced = for_arr.into_array().slice(range.clone())?; + (sliced, all_values[range].to_vec()) + } else { + (for_arr.into_array(), all_values) + }; + + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let plan = dispatch_plan(&array, &mut cuda_ctx).await?; + let actual = run_dynamic_dispatch_plan( + &cuda_ctx, + expected.len(), + &plan.dispatch_plan, + plan.shared_mem_bytes, + )?; + assert_eq!(actual, expected); + Ok(()) + } + + #[rstest] + #[case::unsliced(2000, None)] + #[case::mid_slice(5000, Some(100..4000))] + #[case::large_offset(5000, Some(1500..4500))] + #[crate::test] + async fn test_alp_with_patches( + #[case] len: usize, + #[case] slice_range: Option>, + ) -> VortexResult<()> { + let mut values: Vec = (0..len).map(|i| (i as f32) * 1.1).collect(); + // Insert exception values that ALP can't encode. + values[0] = 99.9; + values[500] = std::f32::consts::PI; + values[1024] = std::f32::consts::E; + if len > 2048 { + values[2048] = std::f32::consts::LN_2; + } + if len > 3333 { + values[3333] = std::f32::consts::SQRT_2; + } + + let float_prim = PrimitiveArray::new(Buffer::from(values), NonNullable); + let encoded = alp_encode( + float_prim.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + )? + .into_array(); + + let (array, base_offset) = if let Some(range) = &slice_range { + (encoded.slice(range.clone())?, range.start) + } else { + (encoded, 0) + }; + + // Decode on CPU as ground truth (accounts for ALP precision loss + patches). + let cpu_decoded = array + .clone() + .execute::(&mut LEGACY_SESSION.create_execution_ctx())?; + let expected: Vec = cpu_decoded.as_slice::().to_vec(); + + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let plan = dispatch_plan(&array, &mut cuda_ctx).await?; + let actual = run_dispatch_plan_f32( + &cuda_ctx, + expected.len(), + &plan.dispatch_plan, + plan.shared_mem_bytes, + )?; + for (i, (&a, &e)) in actual.iter().zip(expected.iter()).enumerate() { + assert!( + a.to_bits() == e.to_bits(), + "mismatch at index {i} (original index {}): gpu={a} cpu={e} (bits: {:#010x} vs {:#010x})", + i + base_offset, + a.to_bits(), + e.to_bits(), + ); + } + Ok(()) + } + + // --------------------------------------------------------------- + // Additional patch tests — typed widths, edge cases, composites + // --------------------------------------------------------------- + + /// u8 BitPacked with patches (bit_width=3, patch values > 7). + #[crate::test] + async fn test_bitpacked_with_patches_u8() -> VortexResult<()> { + let bit_width: u8 = 3; + let len = 3000usize; + let max_val = (1u8 << bit_width) - 1; + let values: Vec = (0..len) + .map(|i| { + if i % 100 == 0 { + 200u8 + } else { + (i as u8) % (max_val + 1) + } + }) + .collect(); + + let prim = PrimitiveArray::new(Buffer::from(values.clone()), NonNullable); + let bp = BitPacked::encode( + &prim.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; + assert!(bp.patches().is_some(), "expected patches"); + + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let canonical = try_gpu_dispatch(&bp.into_array(), &mut cuda_ctx).await?; + let result = CanonicalCudaExt::into_host(canonical).await?.into_array(); + + let expected_arr = PrimitiveArray::new(Buffer::from(values), NonNullable).into_array(); + vortex::array::assert_arrays_eq!(expected_arr, result); + Ok(()) + } + + /// u16 BitPacked with patches (bit_width=6, patch values > 63). + #[crate::test] + async fn test_bitpacked_with_patches_u16() -> VortexResult<()> { + let bit_width: u8 = 6; + let len = 3000usize; + let max_val = (1u16 << bit_width) - 1; + let values: Vec = (0..len) + .map(|i| { + if i % 150 == 0 { + 5000u16 + } else { + (i as u16) % (max_val + 1) + } + }) + .collect(); + + let prim = PrimitiveArray::new(Buffer::from(values.clone()), NonNullable); + let bp = BitPacked::encode( + &prim.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; + assert!(bp.patches().is_some(), "expected patches"); + + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let canonical = try_gpu_dispatch(&bp.into_array(), &mut cuda_ctx).await?; + let result = CanonicalCudaExt::into_host(canonical).await?.into_array(); + + let expected_arr = PrimitiveArray::new(Buffer::from(values), NonNullable).into_array(); + vortex::array::assert_arrays_eq!(expected_arr, result); + Ok(()) + } + + /// u64 BitPacked with patches (bit_width=4, patch values > 15). + #[crate::test] + async fn test_bitpacked_with_patches_u64() -> VortexResult<()> { + let bit_width: u8 = 4; + let len = 3000usize; + let max_val = (1u64 << bit_width) - 1; + let values: Vec = (0..len) + .map(|i| { + if i % 200 == 0 { + 1_000_000u64 + } else { + (i as u64) % (max_val + 1) + } + }) + .collect(); + + let prim = PrimitiveArray::new(Buffer::from(values.clone()), NonNullable); + let bp = BitPacked::encode( + &prim.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; + assert!(bp.patches().is_some(), "expected patches"); + + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let canonical = try_gpu_dispatch(&bp.into_array(), &mut cuda_ctx).await?; + let result = CanonicalCudaExt::into_host(canonical).await?.into_array(); + + let expected_arr = PrimitiveArray::new(Buffer::from(values), NonNullable).into_array(); + vortex::array::assert_arrays_eq!(expected_arr, result); + Ok(()) + } + + /// Dict where codes are BitPacked u32 with patches exceeding the bit width. + #[crate::test] + async fn test_dict_bitpacked_codes_with_patches() -> VortexResult<()> { + let dict_values: Vec = (0..256).map(|i| i * 1000 + 42).collect(); + let len = 3000; + let bit_width: u8 = 4; + let max_code = (1u32 << bit_width) - 1; + // Some codes exceed max_code (15), creating patches in the BitPacked codes. + let codes: Vec = (0..len) + .map(|i| { + if i % 100 == 0 { + 100u32 // exceeds max_code=15, becomes a patch + } else { + (i as u32) % (max_code + 1) + } + }) + .collect(); + let expected: Vec = codes.iter().map(|&c| dict_values[c as usize]).collect(); + + let codes_prim = PrimitiveArray::new(Buffer::from(codes), NonNullable); + let codes_bp = BitPacked::encode( + &codes_prim.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; + assert!(codes_bp.patches().is_some(), "expected patches on codes"); + + let values_prim = PrimitiveArray::new(Buffer::from(dict_values), NonNullable); + let dict = DictArray::try_new(codes_bp.into_array(), values_prim.into_array())?; + + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let plan = dispatch_plan(&dict.into_array(), &mut cuda_ctx).await?; + let actual = + run_dynamic_dispatch_plan(&cuda_ctx, len, &plan.dispatch_plan, plan.shared_mem_bytes)?; + assert_eq!(actual, expected); + Ok(()) + } + + /// Patches placed exactly at FastLanes chunk boundaries (1024-element chunks). + #[crate::test] + async fn test_bitpacked_patches_at_chunk_boundaries() -> VortexResult<()> { + let len = 4096usize; + let bit_width: u8 = 4; + let max_val = (1u32 << bit_width) - 1; + let mut values: Vec = (0..len).map(|i| (i as u32) % (max_val + 1)).collect(); + // Place patches at FL chunk boundaries (1024 elements per chunk). + values[1023] = 1000; // end of chunk 0 + values[1024] = 2000; // start of chunk 1 + values[2047] = 3000; // end of chunk 1 + values[2048] = 4000; // start of chunk 2 + + let prim = PrimitiveArray::new(Buffer::from(values.clone()), NonNullable); + let bp = BitPacked::encode( + &prim.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; + assert!(bp.patches().is_some(), "expected patches"); + + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let plan = dispatch_plan(&bp.into_array(), &mut cuda_ctx).await?; + let actual = + run_dynamic_dispatch_plan(&cuda_ctx, len, &plan.dispatch_plan, plan.shared_mem_bytes)?; + assert_eq!(actual, values); + Ok(()) + } + + /// Large array (100k elements) spanning many blocks with sparse patches. + #[crate::test] + async fn test_bitpacked_large_array_with_patches() -> VortexResult<()> { + let len = 100_000usize; + let bit_width: u8 = 6; + let max_val = (1u32 << bit_width) - 1; + let values: Vec = (0..len) + .map(|i| { + if i % 500 == 0 { + 1000 + } else { + (i as u32) % (max_val + 1) + } + }) + .collect(); + + let prim = PrimitiveArray::new(Buffer::from(values.clone()), NonNullable); + let bp = BitPacked::encode( + &prim.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; + assert!(bp.patches().is_some(), "expected patches"); + + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let plan = dispatch_plan(&bp.into_array(), &mut cuda_ctx).await?; + let actual = + run_dynamic_dispatch_plan(&cuda_ctx, len, &plan.dispatch_plan, plan.shared_mem_bytes)?; + assert_eq!(actual, values); + Ok(()) + } + + /// Nullable BitPacked with patches — validity must survive through fused + /// dispatch alongside patch application. + #[crate::test] + async fn test_nullable_bitpacked_with_patches() -> VortexResult<()> { + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + + let len = 3000usize; + let bit_width: u8 = 4; + let max_val = (1u32 << bit_width) - 1; + + // Every 7th element is null; every 100th (non-null) element is a patch. + let values: Vec> = (0..len) + .map(|i| { + if i % 7 == 0 { + None + } else if i % 100 == 0 { + Some(1000u32) // exceeds max_val=15, becomes a patch + } else { + Some((i as u32) % (max_val + 1)) + } + }) + .collect(); + + let prim = PrimitiveArray::from_option_iter(values.iter().copied()); + let cpu = crate::canonicalize_cpu(prim.clone())?.into_array(); + + let bp = BitPacked::encode( + &prim.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; + assert!(bp.patches().is_some(), "expected patches"); + + let gpu = try_gpu_dispatch(&bp.into_array(), &mut cuda_ctx) + .await? + .into_host() + .await? + .into_array(); + + vortex::array::assert_arrays_eq!(cpu, gpu); + Ok(()) + } + + /// Extreme case: ALL values are patches (bit_width=1, every value > 1). + #[crate::test] + async fn test_bitpacked_all_patches() -> VortexResult<()> { + let bit_width: u8 = 1; + let len = 2000usize; + // All values >= 2, so every single element exceeds max storable (1) and + // becomes a patch. + let values: Vec = (0..len).map(|i| (i as u32) + 2).collect(); + + let prim = PrimitiveArray::new(Buffer::from(values.clone()), NonNullable); + let bp = BitPacked::encode( + &prim.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; + assert!(bp.patches().is_some(), "expected patches"); + + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty())?; + let plan = dispatch_plan(&bp.into_array(), &mut cuda_ctx).await?; + let actual = + run_dynamic_dispatch_plan(&cuda_ctx, len, &plan.dispatch_plan, plan.shared_mem_bytes)?; + assert_eq!(actual, values); + Ok(()) + } } diff --git a/vortex-cuda/src/dynamic_dispatch/plan_builder.rs b/vortex-cuda/src/dynamic_dispatch/plan_builder.rs index c44b7224b7e..e887f1ba53f 100644 --- a/vortex-cuda/src/dynamic_dispatch/plan_builder.rs +++ b/vortex-cuda/src/dynamic_dispatch/plan_builder.rs @@ -9,17 +9,21 @@ use itertools::zip_eq; use tracing::trace; use vortex::array::ArrayRef; +use vortex::array::ArrayVTable; use vortex::array::arrays::Dict; use vortex::array::arrays::Primitive; use vortex::array::arrays::Slice; use vortex::array::arrays::dict::DictArraySlotsExt; use vortex::array::arrays::slice::SliceArrayExt; use vortex::array::buffer::BufferHandle; +use vortex::array::patches::Patches; +use vortex::array::validity::Validity; use vortex::dtype::PType; use vortex::encodings::alp::ALP; use vortex::encodings::alp::ALPArrayExt; use vortex::encodings::alp::ALPArraySlotsExt; use vortex::encodings::alp::ALPFloat; +use vortex::encodings::alp::Exponents; use vortex::encodings::fastlanes::BitPacked; use vortex::encodings::fastlanes::BitPackedArrayExt; use vortex::encodings::fastlanes::FoR; @@ -36,6 +40,8 @@ use vortex::error::vortex_err; use super::CudaDispatchPlan; use super::MaterializedStage; use super::PTypeTag; +use super::PTypeTag_PTYPE_F32; +use super::PTypeTag_PTYPE_F64; use super::SMEM_TILE_SIZE; use super::ScalarOp; use super::SourceOp; @@ -43,6 +49,8 @@ use super::ptype_to_tag; use super::tag_to_ptype; use crate::CudaBufferExt; use crate::CudaExecutionCtx; +use crate::executor::CudaDispatchMode; +use crate::kernel::load_patches_to_gpu; /// A plan whose source buffers have been copied to the device, ready for kernel launch. pub struct MaterializedPlan { @@ -52,26 +60,32 @@ pub struct MaterializedPlan { pub device_buffers: Vec, /// Dynamic shared memory bytes needed to launch this plan. pub shared_mem_bytes: u32, + /// Validity of the root array, propagated to the output. + pub validity: Validity, } /// Checks whether the encoding of an array can be fused into a dynamic-dispatch plan. fn is_dyn_dispatch_compatible(array: &ArrayRef) -> bool { - // The dynamic dispatch kernel only supports F32 floats (via ALP). - // F16 and F64 have no reinterpret path in the kernel. - if matches!(PType::try_from(array.dtype()), Ok(PType::F16 | PType::F64)) { + // F16 has no reinterpret path in the kernel. + if matches!(PType::try_from(array.dtype()), Ok(PType::F16)) { return false; } let id = array.encoding_id(); - if id == ALP::ID { + if id == ALP.id() { let arr = array.as_::(); - return arr.patches().is_none() && arr.dtype().as_ptype() == PType::F32; + return matches!(arr.dtype().as_ptype(), PType::F32 | PType::F64); } - if id == BitPacked::ID { - return array.as_::().patches().is_none(); + if id == BitPacked.id() { + return true; } - if id == Dict::ID { + if id == Dict.id() { let arr = array.as_::(); + // Nullable codes could hold garbage values at null positions, causing + // out-of-bounds shared memory reads in the DICT gather scalar op. + if arr.codes().dtype().is_nullable() { + return false; + } // Dict codes and values may have different byte widths. // The kernel handles mixed widths via widening input stages, // but only when codes are no wider than values (the output type). @@ -83,8 +97,14 @@ fn is_dyn_dispatch_compatible(array: &ArrayRef) -> bool { _ => false, }; } - if id == RunEnd::ID { + if id == RunEnd.id() { let arr = array.as_::(); + // Nullable ends could hold garbage values at null positions, causing + // unpredictable binary search / forward-scan behavior in the RUNEND + // source op. + if arr.ends().dtype().is_nullable() { + return false; + } // RunEnd ends and values may have different byte widths. // The kernel handles mixed widths via widening input stages, // but only when ends are no wider than values (the output type). @@ -96,17 +116,52 @@ fn is_dyn_dispatch_compatible(array: &ArrayRef) -> bool { _ => false, }; } - id == FoR::ID - || id == ZigZag::ID - || id == Primitive::ID - || id == Slice::ID - || id == Sequence::ID + id == FoR.id() + || id == ZigZag.id() + || id == Primitive.id() + || id == Slice.id() + || id == Sequence.id() +} + +/// Returns `true` if a registered standalone kernel can decode the entire +/// `array` tree in a single launch without recursing into `execute_cuda` +/// for child encodings. +pub fn has_standalone_kernel(array: &ArrayRef) -> bool { + let id = array.encoding_id(); + + // Leaf encodings: no children to recurse into. + if id == BitPacked.id() || id == Sequence.id() { + return true; + } + + // FoR fuses with BitPacked (FFOR) and Slice(BitPacked) in one launch. + if id == FoR.id() { + let for_arr = array.as_::(); + let child = for_arr.encoded(); + if child.encoding_id() == BitPacked.id() { + return true; + } + if let Some(slice) = child.as_opt::() { + return slice.child().encoding_id() == BitPacked.id(); + } + return false; + } + + false } /// An unmaterialized stage: a source op, scalar ops, and optional source buffer reference. +/// +/// Patches are tied to their owning ops, mirroring the CUDA side where +/// `patches_ptr` lives on `BitunpackParams` / `AlpParams`: +/// - `source_patches` for the source op (BitPacked exceptions) +/// - Each scalar op carries its own `Option` (ALP exceptions) struct Stage { source: SourceOp, - scalar_ops: Vec, + /// Patches from the source op (e.g. BitPacked overflow exceptions). + source_patches: Option, + /// Scalar ops with optional per-op patches (e.g. ALP float exceptions). + scalar_ops: Vec<(ScalarOp, Option)>, /// Index into `FusedPlan::source_buffers`, or `None` /// for sources that don't read from a device buffer. source_buffer_index: Option, @@ -118,6 +173,7 @@ impl Stage { fn new(source: SourceOp, source_buffer_index: Option, source_ptype: PTypeTag) -> Self { Self { source, + source_patches: None, scalar_ops: vec![], source_buffer_index, source_ptype, @@ -133,6 +189,9 @@ type OutputLen = u32; /// Constructed by [`DispatchPlan::new`], which inspects the encoding tree /// and determines whether it can be fully fused, partially fused, or not fused at all. pub enum DispatchPlan { + /// A registered standalone kernel can decode the entire tree in a single + /// launch without recursing into child encodings. + Standalone, /// Entire encoding tree is fusable into a single kernel launch. Fused(FusedPlan), /// Some subtrees need separate execution before the fused plan can run. @@ -151,55 +210,28 @@ pub enum DispatchPlan { /// /// Stages are stored in kernel execution order. There are two phases: /// -/// 1. All stages except the last run first and decode their output -/// into shared memory (e.g. all dict values, all run-end endpoints). -/// This data stays resident for the output stage to index into. -/// -/// 2. The last stage (the output stage) iterates over the input in tiles -/// of `SMEM_TILE_SIZE` (1024) elements, decoding each tile into a -/// scratch region of shared memory, applying scalar ops (which may -/// reference data from the earlier stages), and writing the result to -/// global memory. -/// -/// # Per-stage PType tracking +/// 1. All stages except the last decode into shared memory (dict values, +/// run-end endpoints). The kernel writes `T`-wide elements even when +/// a stage's source ptype is narrower, widening in-place as needed. /// -/// Each stage carries a `source_ptype` (`PTypeTag`) that identifies the -/// primitive type produced by its source op (LOAD, BITUNPACK, etc.). -/// Scalar ops may change the type (e.g. DICT transforms codes → values, -/// ALP transforms encoded ints → floats); each `ScalarOp` declares its -/// `output_ptype`. The kernel uses these tags to dispatch typed memory -/// operations and cross-stage references at the correct element width. +/// 2. The last stage (the output stage) tiles at `SMEM_TILE_SIZE` (1024) +/// elements, decoding each tile into a scratch region, applying scalar +/// ops (which may reference earlier stages), and streaming to global +/// memory. /// /// # Shared memory allocation /// -/// Total shared memory = `smem_byte_cursor` + `SMEM_TILE_SIZE` × `output_elem_bytes`. +/// Total = `smem_byte_cursor` + `SMEM_TILE_SIZE × output_elem_bytes`. /// -/// `smem_byte_cursor` is tracked in bytes and covers the preceding -/// fully-decoded stages (dict values, run-end endpoints). Each stage's -/// shared memory footprint is `len × final_ptype_byte_width`, where the -/// final ptype is determined by the last scalar op's `output_ptype` (or -/// `source_ptype` if there are no scalar ops). +/// Each input stage occupies `len × max(final_width, output_elem_bytes)` +/// bytes, where `final_width` is the byte width of the last scalar op's +/// `output_ptype` (or `source_ptype` if none). The `max` is necessary +/// because `execute_input_stage` writes `T`-wide elements even when +/// the stage's logical type is narrower. /// -/// All shared memory offsets are byte offsets — the C ABI uses byte -/// offsets and per-field `PTypeTag` values so that stages with different -/// element widths can coexist in the same shared memory pool. -/// -/// This is sufficient because: -/// -/// - Earlier stages only originate from dict (values) and run-end (ends, -/// values). `push_smem_stage` reserves the appropriate number of bytes -/// in `smem_byte_cursor`, so each stage's source op has room to decode -/// the complete input. -/// -/// - The output stage (last) tiles at `SMEM_TILE_SIZE` (1024 elements), -/// so its source op never writes more than 1024 elements into the -/// scratch region, even though each block is responsible for -/// `ELEMENTS_PER_BLOCK` (2048) output elements — it processes them in -/// two passes through the scratch. -/// -/// Note: `BITUNPACK` writes full FastLanes blocks (1024 elements), which can -/// exceed `stage.len` by up to 1023 elements. This overflow is absorbed by -/// the scratch region (`SMEM_TILE_SIZE` ≥ `FL_CHUNK_SIZE`). +/// `BITUNPACK` writes full FastLanes blocks (1024 elements) which may +/// exceed `stage.len` by up to 1023 elements; this overflow is absorbed +/// by the scratch region (`SMEM_TILE_SIZE` ≥ `FL_CHUNK_SIZE`). pub struct FusedPlan { /// Stages in kernel execution order; all but the last decode into /// shared memory, the last decodes into global memory. @@ -213,6 +245,8 @@ pub struct FusedPlan { output_elem_bytes: u32, /// PType of the root (output) array, as a C ABI tag. output_ptype: PTypeTag, + /// Validity of the root array, propagated to the output. + validity: Validity, } impl DispatchPlan { @@ -220,10 +254,21 @@ impl DispatchPlan { /// /// # Limitations /// - /// - Validity bitmaps are ignored; only `NonNullable`/`AllValid` is supported. - /// - `BitPackedArray` and `ALPArray` with patches are not supported. - /// - Only f32 ALP is supported (kernel stores multipliers as `float`). - pub fn new(array: &ArrayRef) -> VortexResult { + /// - **F16 primitives** are not supported (no reinterpret path in the kernel). + /// - **ALP** is supported for f32 and f64 only (including patches). + /// - **BitPacked** with patches is supported. + /// - **Dict** with nullable codes is rejected (garbage at null positions + /// could OOB the DICT gather). Dict with codes wider than values is + /// also rejected (load would truncate code indices). + /// - **RunEnd** with nullable ends is rejected (garbage values break the + /// binary search). RunEnd with ends wider than values is also rejected. + /// - Validity is propagated from the root array to the output. + /// - Unrecognized encodings fall back to `Unfused`. + pub fn new(array: &ArrayRef, mode: CudaDispatchMode) -> VortexResult { + if mode == CudaDispatchMode::Auto && has_standalone_kernel(array) { + return Ok(Self::Standalone); + } + if PType::try_from(array.dtype()).is_err() || !is_dyn_dispatch_compatible(array) { return Ok(Self::Unfused); } @@ -271,11 +316,9 @@ impl FusedPlan { array.dtype() ) })?; - if output_ptype_rust == PType::F64 { - vortex_bail!("dynamic dispatch does not support f64 output"); - } let output_elem_bytes = output_ptype_rust.byte_width() as u32; let output_ptype = ptype_to_tag(output_ptype_rust); + let validity = array.validity()?; let mut pending_subtrees: Vec = Vec::new(); let mut plan = Self { @@ -284,6 +327,7 @@ impl FusedPlan { source_buffers: Vec::new(), output_elem_bytes, output_ptype, + validity, }; let len = array.len() as u32; @@ -319,7 +363,7 @@ impl FusedPlan { } /// Copy source buffers to the device, producing a [`MaterializedPlan`]. - pub fn materialize(self, ctx: &CudaExecutionCtx) -> VortexResult { + pub async fn materialize(self, ctx: &mut CudaExecutionCtx) -> VortexResult { let shared_mem_bytes = self.dynamic_shared_mem_bytes(); let mut device_buffers = Vec::new(); @@ -346,25 +390,43 @@ impl FusedPlan { // Byte offsets are passed directly to the C ABI — the kernel now // indexes shared memory by byte offset and casts to the correct type // using source_ptype / output_ptype. - let stages: Vec = self - .stages - .iter() - .map(|(stage, smem_byte_offset, len)| { - MaterializedStage::new( - resolve_ptr(stage), - *smem_byte_offset, - *len, - stage.source_ptype, - stage.source, - &stage.scalar_ops, - ) - }) - .collect(); + let mut stages: Vec = Vec::new(); + for (stage, smem_byte_offset, len) in &self.stages { + let mut source = stage.source; + + // Upload source patches (e.g. BitPacked exceptions). + if let Some(patches) = &stage.source_patches { + let (ptr, bufs) = load_patches_to_gpu(patches, ctx).await?; + source.params.bitunpack.patches_ptr = ptr; + device_buffers.extend(bufs); + } + + // Upload patches for each scalar op that carries them. + let mut scalar_ops: Vec = Vec::with_capacity(stage.scalar_ops.len()); + for (mut op, patches) in stage.scalar_ops.clone() { + if let Some(patches) = &patches { + let (ptr, bufs) = load_patches_to_gpu(patches, ctx).await?; + op.params.alp.patches_ptr = ptr; + device_buffers.extend(bufs); + } + scalar_ops.push(op); + } + + stages.push(MaterializedStage::new( + resolve_ptr(stage), + *smem_byte_offset, + *len, + stage.source_ptype, + source, + &scalar_ops, + )); + } Ok(MaterializedPlan { dispatch_plan: CudaDispatchPlan::new(stages, self.output_ptype), device_buffers, shared_mem_bytes, + validity: self.validity, }) } @@ -373,10 +435,10 @@ impl FusedPlan { /// /// `subtree_buffers` must correspond 1:1 (in DFS order) to the /// `pending_subtrees` returned by `build`. - pub fn materialize_with_subtrees( + pub async fn materialize_with_subtrees( mut self, subtree_buffers: Vec, - ctx: &CudaExecutionCtx, + ctx: &mut CudaExecutionCtx, ) -> VortexResult { for (slot, buf) in zip_eq( self.source_buffers.iter_mut().filter(|s| s.is_none()), @@ -384,7 +446,7 @@ impl FusedPlan { ) { *slot = Some(buf); } - self.materialize(ctx) + self.materialize(ctx).await } /// Walk the encoding tree, producing a [`Stage`] for the root. @@ -394,43 +456,28 @@ impl FusedPlan { pending_subtrees: &mut Vec, ) -> VortexResult { if !is_dyn_dispatch_compatible(&array) { - // Subtree can't be fused — record it as a deferred LOAD source. - // Bail if dtype is non-primitive (can't become a LOAD stage). - let ptype = PType::try_from(array.dtype()).map_err(|_| { - vortex_err!( - "unfusable subtree has non-primitive dtype {:?}, cannot partially fuse", - array.dtype() - ) - })?; - let buf_idx = self.source_buffers.len(); - self.source_buffers.push(None); // placeholder, filled at materialize time - pending_subtrees.push(array); - return Ok(Stage::new( - SourceOp::load(), - Some(buf_idx), - ptype_to_tag(ptype), - )); + return self.push_subtree(array, pending_subtrees); } let id = array.encoding_id(); - if id == BitPacked::ID { + if id == BitPacked.id() { self.walk_bitpacked(array) - } else if id == FoR::ID { + } else if id == FoR.id() { self.walk_for(array, pending_subtrees) - } else if id == ZigZag::ID { + } else if id == ZigZag.id() { self.walk_zigzag(array, pending_subtrees) - } else if id == ALP::ID { + } else if id == ALP.id() { self.walk_alp(array, pending_subtrees) - } else if id == Dict::ID { + } else if id == Dict.id() { self.walk_dict(array, pending_subtrees) - } else if id == RunEnd::ID { + } else if id == RunEnd.id() { self.walk_runend(array, pending_subtrees) - } else if id == Primitive::ID { + } else if id == Primitive.id() { self.walk_primitive(array) - } else if id == Slice::ID { + } else if id == Slice.id() { self.walk_slice(array, pending_subtrees) - } else if id == Sequence::ID { + } else if id == Sequence.id() { self.walk_sequence(array) } else { vortex_bail!( @@ -443,7 +490,8 @@ impl FusedPlan { /// SliceArray → resolve the slice via reduce/execute rules. /// /// When the plan builder encounters a `SliceArray`, it resolves the slice - /// by invoking the child's `reduce_parent`, `execute_parent`. + /// by invoking the child's `reduce_parent`. If that fails (e.g. ALP + /// doesn't implement it), we manually slice the child's sub-arrays. fn walk_slice( &mut self, array: ArrayRef, @@ -456,6 +504,26 @@ impl FusedPlan { return self.walk(reduced, pending_subtrees); } + // ALP doesn't implement reduce_parent — slice encoded child and + // patches manually (Patches::slice adjusts offsets for the range). + if child.encoding_id() == ALP.id() { + let alp = child.as_::(); + let offset = slice_arr.data().slice_range().start; + let len = array.len(); + let sliced_encoded = alp.encoded().clone().slice(offset..offset + len)?; + let sliced_patches = alp + .patches() + .map(|p| p.slice(offset..offset + len)) + .transpose()? + .flatten(); + return self.walk_alp_inner( + sliced_encoded, + sliced_patches, + alp.exponents(), + pending_subtrees, + ); + } + vortex_bail!( "Cannot resolve SliceArray wrapping {:?} in dynamic dispatch plan builder", child.encoding_id() @@ -476,20 +544,18 @@ impl FusedPlan { fn walk_bitpacked(&mut self, array: ArrayRef) -> VortexResult { let bp = array.as_::(); - if bp.patches().is_some() { - vortex_bail!("Dynamic dispatch does not support BitPackedArray with patches"); - } - let source_ptype = ptype_to_tag(PType::try_from(bp.dtype()).map_err(|_| { vortex_err!("BitPacked must have primitive dtype, got {:?}", bp.dtype()) })?); let buf_index = self.source_buffers.len(); self.source_buffers.push(Some(bp.packed().clone())); - Ok(Stage::new( + let mut stage = Stage::new( SourceOp::bitunpack(bp.bit_width(), bp.offset()), Some(buf_index), source_ptype, - )) + ); + stage.source_patches = bp.patches(); + Ok(stage) } fn walk_for( @@ -515,7 +581,7 @@ impl FusedPlan { .cast::()?; pipeline .scalar_ops - .push(ScalarOp::frame_of_ref(ref_u64, output_ptype)); + .push((ScalarOp::frame_of_ref(ref_u64, output_ptype), None)); Ok(pipeline) } @@ -531,7 +597,9 @@ impl FusedPlan { })?); let mut pipeline = self.walk(encoded, pending_subtrees)?; - pipeline.scalar_ops.push(ScalarOp::zigzag(output_ptype)); + pipeline + .scalar_ops + .push((ScalarOp::zigzag(output_ptype), None)); Ok(pipeline) } @@ -541,47 +609,100 @@ impl FusedPlan { pending_subtrees: &mut Vec, ) -> VortexResult { let alp = array.as_::(); + self.walk_alp_inner( + alp.encoded().clone(), + alp.patches(), + alp.exponents(), + pending_subtrees, + ) + } - if alp.patches().is_some() { - vortex_bail!("Dynamic dispatch does not support ALPArray with patches"); - } - - let ptype = alp.dtype().as_ptype(); - if ptype != PType::F32 { - vortex_bail!( - "Dynamic dispatch only supports f32 ALP, got {:?}", - alp.dtype() - ); - } - - let exponents = alp.exponents(); - let alp_f = ::F10[exponents.f as usize]; - let alp_e = ::IF10[exponents.e as usize]; - let encoded = alp.encoded().clone(); + /// Shared ALP logic for both `walk_alp` and `walk_slice` (Slice(ALP)). + fn walk_alp_inner( + &mut self, + encoded: ArrayRef, + patches: Option, + exponents: Exponents, + pending_subtrees: &mut Vec, + ) -> VortexResult { + let encoded_ptype = PType::try_from(encoded.dtype()).map_err(|_| { + vortex_err!( + "ALP encoded child must have primitive dtype, got {:?}", + encoded.dtype() + ) + })?; + // ALP encodes f32 as i32 and f64 as i64. Select the correct + // exponent tables and output PType based on the encoded integer width. + let (alp_f, alp_e, output_ptype) = match encoded_ptype { + PType::I32 => ( + ::F10[exponents.f as usize] as f64, + ::IF10[exponents.e as usize] as f64, + PTypeTag_PTYPE_F32, + ), + PType::I64 => ( + ::F10[exponents.f as usize], + ::IF10[exponents.e as usize], + PTypeTag_PTYPE_F64, + ), + other => vortex_bail!( + "ALP encoded ptype must be I32 (f32) or I64 (f64), got {:?}", + other + ), + }; let mut pipeline = self.walk(encoded, pending_subtrees)?; - pipeline.scalar_ops.push(ScalarOp::alp(alp_f, alp_e)); + pipeline + .scalar_ops + .push((ScalarOp::alp(alp_f, alp_e, output_ptype), patches)); Ok(pipeline) } - /// Handle a child array whose element width differs from the output type. + /// Walk a child that may have a different element width than the output. /// - /// If the child is a `Primitive`, its buffer is grabbed directly as a LOAD - /// source — no separate kernel launch needed, since `load_element()` - /// handles the widening in-kernel. Otherwise, the child is recorded as a - /// pending subtree for separate execution. - fn walk_mixed_width_child( + /// Primitives are always handled directly (`load_element()` widens + /// in-kernel). Non-primitive children are recursively walked; the kernel's + /// `bitunpack_typed` decodes at the source's native width and widens to + /// `T` in shared memory, and `push_smem_stage` allocates accordingly. + fn walk_child( &mut self, - child: ArrayRef, + array: ArrayRef, pending_subtrees: &mut Vec, ) -> VortexResult { - let ptype = PType::try_from(child.dtype())?; - if child.encoding_id() == Primitive::ID { - return self.walk_primitive(child); + if array.encoding_id() == Primitive.id() { + return self.walk_primitive(array); } + self.walk(array, pending_subtrees) + } + + /// Reserve a placeholder buffer slot and record the array as a pending subtree. + /// + /// Called from [`walk`] when [`is_dyn_dispatch_compatible`] rejects a child. + /// Cases that require a separate kernel dispatch: + /// + /// - **F16 primitives** — no reinterpret path in the kernel. + /// - **Dict with nullable codes** — garbage at null positions could OOB + /// the DICT gather in shared memory. + /// - **Dict with codes wider than values** — `load_element()` would + /// truncate the code indices. + /// - **RunEnd with nullable ends** — garbage values break the binary + /// search / forward-scan. + /// - **RunEnd with ends wider than values** — same truncation issue. + /// - **Unrecognized encoding** — anything outside the kernel's allow-list + /// (e.g. FSST, Pco, Zstd). + fn push_subtree( + &mut self, + array: ArrayRef, + pending_subtrees: &mut Vec, + ) -> VortexResult { + let ptype = PType::try_from(array.dtype()).map_err(|_| { + vortex_err!( + "unfusable subtree has non-primitive dtype {:?}, cannot partially fuse", + array.dtype() + ) + })?; let buf_idx = self.source_buffers.len(); self.source_buffers.push(None); - pending_subtrees.push(child); + pending_subtrees.push(array); Ok(Stage::new( SourceOp::load(), Some(buf_idx), @@ -599,33 +720,17 @@ impl FusedPlan { let codes = dict.codes().clone(); let values_ptype = PType::try_from(values.dtype())?; - let values_elem_bytes = values_ptype.byte_width() as u32; - let codes_ptype = PType::try_from(codes.dtype())?; - let codes_elem_bytes = codes_ptype.byte_width() as u32; - - // If values have a different width than the output type, they - // can't be fused into the same kernel instantiation. Primitives - // are handled directly (just grab the buffer); other encodings - // become pending subtrees executed by a separate kernel. + let values_len = values.len() as u32; - let values_spec = if values_elem_bytes != self.output_elem_bytes { - self.walk_mixed_width_child(values, pending_subtrees)? - } else { - self.walk(values, pending_subtrees)? - }; + let values_spec = self.walk_child(values, pending_subtrees)?; let values_smem_byte_offset = self.push_smem_stage(values_spec, values_len); - // Same for codes. - let mut pipeline = if codes_elem_bytes != self.output_elem_bytes { - self.walk_mixed_width_child(codes, pending_subtrees)? - } else { - self.walk(codes, pending_subtrees)? - }; + let mut pipeline = self.walk_child(codes, pending_subtrees)?; // DICT scalar op: pass byte offset directly (C ABI uses byte offsets). // output_ptype is the values' ptype — DICT transforms codes → values. - pipeline.scalar_ops.push(ScalarOp::dict( - values_smem_byte_offset, - ptype_to_tag(values_ptype), + pipeline.scalar_ops.push(( + ScalarOp::dict(values_smem_byte_offset, ptype_to_tag(values_ptype)), + None, )); Ok(pipeline) } @@ -652,26 +757,10 @@ impl FusedPlan { let num_runs = ends.len() as u32; let num_values = values.len() as u32; - let ends_ptype = PType::try_from(ends.dtype())?; - let ends_elem_bytes = ends_ptype.byte_width() as u32; - let values_ptype = PType::try_from(values.dtype())?; - let values_elem_bytes = values_ptype.byte_width() as u32; - - // If ends or values have a different width than the output type, - // they can't be fused into the same kernel instantiation. - // Primitives are handled directly; others become pending subtrees. - let ends_spec = if ends_elem_bytes != self.output_elem_bytes { - self.walk_mixed_width_child(ends, pending_subtrees)? - } else { - self.walk(ends, pending_subtrees)? - }; + let ends_spec = self.walk_child(ends, pending_subtrees)?; let ends_smem_byte_offset = self.push_smem_stage(ends_spec, num_runs); - let values_spec = if values_elem_bytes != self.output_elem_bytes { - self.walk_mixed_width_child(values, pending_subtrees)? - } else { - self.walk(values, pending_subtrees)? - }; + let values_spec = self.walk_child(values, pending_subtrees)?; let values_smem_byte_offset = self.push_smem_stage(values_spec, num_values); // Pass byte offsets and PTypeTags directly — the C ABI now uses @@ -689,13 +778,10 @@ impl FusedPlan { } /// Add a stage that decodes fully into shared memory before the output - /// stage runs. Returns the shared memory byte offset where the data starts. - /// - /// The smem region is sized at the stage's output ptype width — i.e. - /// the ptype after all scalar ops have run. For stages that go through - /// type-changing scalar ops (e.g. dict values with FoR→ALP), the final - /// smem footprint is `len × final_ptype_byte_width`. If there are no - /// scalar ops, the source_ptype determines the width. + /// stage runs. Returns the shared memory byte offset where the data + /// starts. Allocates `len × max(final_width, output_elem_bytes)` bytes + /// so that narrower stages widened to `T` by `bitunpack_typed` never + /// overflow. fn push_smem_stage(&mut self, spec: Stage, len: u32) -> u32 { let smem_byte_offset = self.smem_byte_cursor; // The kernel's execute_input_stage always writes T-wide elements @@ -705,7 +791,7 @@ impl FusedPlan { let final_ptype = spec .scalar_ops .last() - .map(|op| op.output_ptype) + .map(|(op, _)| op.output_ptype) .unwrap_or(spec.source_ptype); let final_elem_bytes = tag_to_ptype(final_ptype).byte_width() as u32; let elem_bytes = final_elem_bytes.max(self.output_elem_bytes); diff --git a/vortex-cuda/src/executor.rs b/vortex-cuda/src/executor.rs index 1d6d1db76c3..d490d3f03fe 100644 --- a/vortex-cuda/src/executor.rs +++ b/vortex-cuda/src/executor.rs @@ -16,6 +16,7 @@ use futures::future::BoxFuture; use tracing::debug; use tracing::trace; use vortex::array::ArrayRef; +use vortex::array::ArrayVTable; use vortex::array::Canonical; use vortex::array::ExecutionCtx; use vortex::array::IntoArray; @@ -56,6 +57,25 @@ impl CudaKernelEvents { } } +/// Controls which GPU dispatch strategy is used when executing arrays. +/// +/// By default, `execute_cuda` tries fused dynamic dispatch first, +/// then falls back to standalone per-encoding kernels. Benchmarks and tests +/// can force a specific strategy to get stable, isolated measurements. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)] +pub enum CudaDispatchMode { + /// Automatically choose the best strategy (current default behavior). + /// Tries fused → partially-fused → standalone → CPU fallback. + #[default] + Auto, + /// Only use standalone per-encoding `CudaExecute` kernels. + /// Skips the fused/partially-fused dynamic dispatch planner entirely. + StandaloneOnly, + /// Only use fused or partially-fused dynamic dispatch. + /// Returns an error if the array is not dyn-dispatch-compatible. + DynDispatchOnly, +} + /// CUDA execution context. /// /// Provides access to the CUDA context and stream for kernel execution. @@ -65,6 +85,7 @@ pub struct CudaExecutionCtx { ctx: ExecutionCtx, cuda_session: CudaSession, strategy: Arc, + dispatch_mode: CudaDispatchMode, } impl CudaExecutionCtx { @@ -76,6 +97,7 @@ impl CudaExecutionCtx { ctx, cuda_session, strategy: Arc::new(DefaultLaunchStrategy), + dispatch_mode: CudaDispatchMode::Auto, } } @@ -93,6 +115,15 @@ impl CudaExecutionCtx { self } + /// Set the dispatch mode for the execution context. + /// + /// This controls whether `execute_cuda` uses fused dynamic dispatch, + /// standalone per-encoding kernels, or the automatic (default) strategy. + pub fn with_dispatch_mode(mut self, dispatch_mode: CudaDispatchMode) -> Self { + self.dispatch_mode = dispatch_mode; + self + } + /// Perform an external kernel launch, with events created and logged via the configured /// [`LaunchStrategy`]. /// @@ -275,6 +306,11 @@ impl CudaExecutionCtx { self.ctx.session() } + /// Returns the current dispatch mode. + pub fn dispatch_mode(&self) -> CudaDispatchMode { + self.dispatch_mode + } + /// Returns a reference to the CUDA session. pub(crate) fn cuda_session(&self) -> &CudaSession { &self.cuda_session @@ -350,9 +386,9 @@ pub trait CudaArrayExt { #[async_trait] impl CudaArrayExt for ArrayRef { - #[allow(clippy::unwrap_in_result, clippy::unwrap_used)] + #[expect(clippy::unwrap_used)] async fn execute_cuda(self, ctx: &mut CudaExecutionCtx) -> VortexResult { - if self.encoding_id() == Struct::ID { + if self.encoding_id() == Struct.id() { let len = self.len(); let StructDataParts { fields, diff --git a/vortex-cuda/src/hybrid_dispatch/mod.rs b/vortex-cuda/src/hybrid_dispatch/mod.rs index 0845a46fdc3..eb0727e992f 100644 --- a/vortex-cuda/src/hybrid_dispatch/mod.rs +++ b/vortex-cuda/src/hybrid_dispatch/mod.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -//! Hybrid dispatch: fuses dynamic-dispatch plans with single-kernel fallbacks. +//! Hybrid dispatch: routes arrays to standalone kernels or fused dynamic-dispatch plans. //! //! When an array is executed on the GPU, we fuse as much of its encoding //! tree as possible into a single kernel launch via [`DispatchPlan`]. @@ -18,10 +18,15 @@ //! //! Strategies tried in order: //! -//! 1. Fully fused — no unfusable nodes, entire tree compiles into one +//! 1. Standalone — a registered kernel covers the entire tree in a single +//! launch without recursing into child encodings. Preferred when applicable +//! because the kernel is hand-tuned for that exact scenario. Detected by +//! `has_standalone_kernel`. +//! +//! 2. Fully fused — no unfusable nodes, entire tree compiles into one //! [`DispatchPlan`] → `MaterializedPlan` → kernel launch. //! -//! 2. Partial fusion — `pending_subtrees` from the `PartiallyFused` +//! 3. Partial fusion — `pending_subtrees` from the `PartiallyFused` //! variant are executed first (sequentially, same stream), their device buffers //! become `LOAD` ops in a fused plan via `FusedPlan::materialize_with_subtrees`. //! Each subtree re-enters [`try_gpu_dispatch`] and may itself fuse. @@ -30,10 +35,10 @@ //! happens in-kernel via `load_element()` in the LOAD source op — no //! separate widen pass is needed. //! -//! 3. Fallback — root is not fusable. Delegate to its registered +//! 4. Fallback — root is not fusable. Delegate to its registered //! `CudaExecute` kernel; its children re-enter [`try_gpu_dispatch`]. //! -//! All three compose recursively to arbitrary depth. +//! All four compose recursively. //! //! Zone-map pruning is handled by ZonedReader before chunks reach the plan //! builder. Filtering within a chunk is done after decompression, not as push-down. @@ -55,12 +60,14 @@ use crate::dynamic_dispatch::plan_builder::DispatchPlan; use crate::executor::CudaArrayExt; use crate::executor::CudaExecutionCtx; -/// Try to execute `array` on the GPU, attempting three strategies in order: +/// Try to execute `array` on the GPU, attempting four strategies in order: /// -/// 1. Fully fused — [`DispatchPlan::new`] + `FusedPlan::materialize`. -/// 2. Partially fused — pending subtrees executed first, then +/// 1. Standalone — a registered kernel covers the entire tree without +/// recursing into child encodings (e.g. FFOR, ALP(Primitive)). +/// 2. Fully fused — [`DispatchPlan::new`] + `FusedPlan::materialize`. +/// 3. Partially fused — pending subtrees executed first, then /// `FusedPlan::materialize_with_subtrees`. -/// 3. Fallback — root encoding's `CudaExecute` kernel; children +/// 4. Fallback — root encoding's `CudaExecute` kernel; children /// re-enter this function recursively. /// /// Returns `Ok(Canonical)` on success. Returns `Err` when the array @@ -70,11 +77,34 @@ pub async fn try_gpu_dispatch( array: &ArrayRef, ctx: &mut CudaExecutionCtx, ) -> VortexResult { + use crate::executor::CudaDispatchMode; + trace!(encoding = %array.encoding_id(), dtype = ?array.dtype(), len = array.len(), "try GPU dispatch"); - match DispatchPlan::new(array)? { + // Short-circuit: standalone-only skips the plan builder entirely. + if ctx.dispatch_mode() == CudaDispatchMode::StandaloneOnly { + trace!(encoding = %array.encoding_id(), "standalone-only dispatch"); + return ctx + .cuda_session() + .kernel(&array.encoding_id()) + .ok_or_else(|| vortex_err!("No CUDA kernel for encoding {:?}", array.encoding_id()))? + .execute(array.clone(), ctx) + .await; + } + + match DispatchPlan::new(array, ctx.dispatch_mode())? { + DispatchPlan::Standalone => { + trace!(encoding = %array.encoding_id(), "standalone dispatch"); + ctx.cuda_session() + .kernel(&array.encoding_id()) + .ok_or_else(|| { + vortex_err!("No CUDA kernel for encoding {:?}", array.encoding_id()) + })? + .execute(array.clone(), ctx) + .await + } DispatchPlan::Fused(plan) => { - let materialized = plan.materialize(ctx)?; + let materialized = plan.materialize(ctx).await?; let num_stages = materialized.dispatch_plan.num_stages(); trace!(encoding = %array.encoding_id(), num_stages, "fully-fused dispatch"); materialized.execute(array.len(), ctx) @@ -93,12 +123,20 @@ pub async fn try_gpu_dispatch( } let num_subtrees = subtree_buffers.len(); - let materialized = plan.materialize_with_subtrees(subtree_buffers, ctx)?; + let materialized = plan.materialize_with_subtrees(subtree_buffers, ctx).await?; let num_stages = materialized.dispatch_plan.num_stages(); trace!(encoding = %array.encoding_id(), num_stages, num_subtrees, "partially-fused dispatch"); materialized.execute(array.len(), ctx) } DispatchPlan::Unfused => { + // In dyn-dispatch-only mode, don't fall back to standalone kernels. + if ctx.dispatch_mode() == CudaDispatchMode::DynDispatchOnly { + return Err(vortex_err!( + "Array with encoding {:?} is not dyn-dispatch-compatible", + array.encoding_id() + )); + } + // Unfused kernel dispatch fallback. ctx.cuda_session() .kernel(&array.encoding_id()) @@ -124,6 +162,8 @@ mod tests { use vortex::error::VortexResult; use vortex::mask::Mask; use vortex::session::VortexSession; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use crate::CanonicalCudaExt; use crate::executor::CudaArrayExt; @@ -138,11 +178,12 @@ mod tests { let bp = BitPacked::encode( &PrimitiveArray::new(Buffer::from(values), NonNullable).into_array(), 7, + &mut LEGACY_SESSION.create_execution_ctx(), ) .vortex_expect("bp"); let arr = FoR::try_new(bp.into_array(), 1000u32.into()).vortex_expect("for"); - let cpu = arr.to_canonical()?.into_array(); + let cpu = crate::canonicalize_cpu(arr.clone())?.into_array(); let gpu = arr .into_array() .execute_cuda(&mut ctx) @@ -167,6 +208,7 @@ mod tests { let bp = BitPacked::encode( &PrimitiveArray::new(Buffer::from(encoded), NonNullable).into_array(), 9, + &mut LEGACY_SESSION.create_execution_ctx(), ) .vortex_expect("bp"); let alp = ALP::try_new( @@ -177,7 +219,7 @@ mod tests { None, )?; - let cpu = alp.to_canonical()?.into_array(); + let cpu = crate::canonicalize_cpu(alp.clone())?.into_array(); let gpu = alp .into_array() .execute_cuda(&mut ctx) @@ -215,7 +257,7 @@ mod tests { .unwrap(); let arr = ALP::try_new(encoded, Exponents { e: 0, f: 2 }, Some(patches))?; - let cpu = arr.to_canonical()?.into_array(); + let cpu = crate::canonicalize_cpu(arr.clone())?.into_array(); let gpu = arr .into_array() .execute_cuda(&mut ctx) @@ -253,12 +295,13 @@ mod tests { ) .into_array(); let vals = FoR::try_new( - BitPacked::encode(&vals, 6).vortex_expect("bp").into_array(), + BitPacked::encode(&vals, 6, &mut LEGACY_SESSION.create_execution_ctx()) + .vortex_expect("bp") + .into_array(), 0u32.into(), ) .vortex_expect("for"); - let vals = ZstdBuffers::compress(&vals.into_array(), 3, &VortexSession::empty()) - .vortex_expect("zstd"); + let vals = ZstdBuffers::compress(&vals.into_array(), 3, &session).vortex_expect("zstd"); // codes = FoR(BitPacked) let codes = PrimitiveArray::new( @@ -267,7 +310,7 @@ mod tests { ) .into_array(); let codes = FoR::try_new( - BitPacked::encode(&codes, 6) + BitPacked::encode(&codes, 6, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("bp") .into_array(), 0u32.into(), @@ -303,6 +346,7 @@ mod tests { let bp = BitPacked::encode( &PrimitiveArray::new(Buffer::from(data.clone()), NonNullable).into_array(), 7, + &mut LEGACY_SESSION.create_execution_ctx(), ) .vortex_expect("bp"); let for_arr = FoR::try_new(bp.into_array(), 100u32.into()).vortex_expect("for"); diff --git a/vortex-cuda/src/kernel/arrays/constant.rs b/vortex-cuda/src/kernel/arrays/constant.rs index 2164378b9e9..8e5966da243 100644 --- a/vortex-cuda/src/kernel/arrays/constant.rs +++ b/vortex-cuda/src/kernel/arrays/constant.rs @@ -227,7 +227,7 @@ mod tests { let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) .vortex_expect("failed to create execution context"); - let cpu_result = constant_array.to_canonical()?; + let cpu_result = crate::canonicalize_cpu(constant_array.clone())?; let gpu_result = ConstantNumericExecutor .execute(constant_array.into_array(), &mut cuda_ctx) @@ -248,7 +248,7 @@ mod tests { .vortex_expect("failed to create execution context"); let constant_array = ConstantArray::new(42i32, 0); - let cpu_result = constant_array.to_canonical()?; + let cpu_result = crate::canonicalize_cpu(constant_array.clone())?; let gpu_result = ConstantNumericExecutor .execute(constant_array.into_array(), &mut cuda_ctx) @@ -270,7 +270,7 @@ mod tests { // Test with array smaller than one block (< 2048 elements) let constant_array = ConstantArray::new(99i32, 100); - let cpu_result = constant_array.to_canonical()?; + let cpu_result = crate::canonicalize_cpu(constant_array.clone())?; let gpu_result = ConstantNumericExecutor .execute(constant_array.into_array(), &mut cuda_ctx) diff --git a/vortex-cuda/src/kernel/arrays/dict.rs b/vortex-cuda/src/kernel/arrays/dict.rs index ce98c989d88..442b730a037 100644 --- a/vortex-cuda/src/kernel/arrays/dict.rs +++ b/vortex-cuda/src/kernel/arrays/dict.rs @@ -339,7 +339,7 @@ mod tests { .vortex_expect("failed to create Dict array"); // Get baseline from CPU canonicalization - let baseline = dict_array.to_canonical()?; + let baseline = crate::canonicalize_cpu(dict_array.clone())?; // Execute on CUDA let cuda_result = DictExecutor @@ -374,7 +374,7 @@ mod tests { .vortex_expect("failed to create Dict array"); // Get baseline from CPU canonicalization - let baseline = dict_array.to_canonical()?; + let baseline = crate::canonicalize_cpu(dict_array.clone())?; // Execute on CUDA let cuda_result = DictExecutor @@ -406,7 +406,7 @@ mod tests { .vortex_expect("failed to create Dict array"); // Get baseline from CPU canonicalization - let baseline = dict_array.to_canonical()?; + let baseline = crate::canonicalize_cpu(dict_array.clone())?; // Execute on CUDA let cuda_result = DictExecutor @@ -437,7 +437,7 @@ mod tests { .vortex_expect("failed to create Dict array"); // Get baseline from CPU canonicalization - let baseline = dict_array.to_canonical()?; + let baseline = crate::canonicalize_cpu(dict_array.clone())?; // Execute on CUDA let cuda_result = DictExecutor @@ -470,7 +470,7 @@ mod tests { .vortex_expect("failed to create Dict array"); // Get baseline from CPU canonicalization - let baseline = dict_array.to_canonical()?; + let baseline = crate::canonicalize_cpu(dict_array.clone())?; let cuda_result = DictExecutor .execute(dict_array.into_array(), &mut cuda_ctx) @@ -507,7 +507,7 @@ mod tests { .vortex_expect("failed to create Dict array"); // Get baseline from CPU canonicalization - let baseline = dict_array.to_canonical()?; + let baseline = crate::canonicalize_cpu(dict_array.clone())?; // Execute on CUDA let cuda_result = DictExecutor @@ -551,7 +551,7 @@ mod tests { .vortex_expect("failed to create Dict array"); // Get baseline from CPU canonicalization - let baseline = dict_array.to_canonical()?; + let baseline = crate::canonicalize_cpu(dict_array.clone())?; // Execute on CUDA let cuda_result = DictExecutor @@ -596,7 +596,7 @@ mod tests { .vortex_expect("failed to create Dict array"); // Get baseline from CPU canonicalization - let baseline = dict_array.to_canonical()?; + let baseline = crate::canonicalize_cpu(dict_array.clone())?; // Execute on CUDA let cuda_result = DictExecutor @@ -629,7 +629,7 @@ mod tests { .vortex_expect("failed to create Dict array"); // Get baseline from CPU canonicalization - let baseline = dict_array.to_canonical()?; + let baseline = crate::canonicalize_cpu(dict_array.clone())?; // Execute on CUDA let cuda_result = DictExecutor @@ -669,7 +669,7 @@ mod tests { let dict_array = DictArray::try_new(codes_array.into_array(), values.into_array()) .vortex_expect("failed to create Dict array"); - let baseline = dict_array.to_canonical()?; + let baseline = crate::canonicalize_cpu(dict_array.clone())?; let cuda_result = DictExecutor .execute(dict_array.into_array(), &mut cuda_ctx) @@ -697,7 +697,7 @@ mod tests { let dict_array = DictArray::try_new(codes_array.into_array(), values.into_array()) .vortex_expect("failed to create Dict array"); - let baseline = dict_array.to_canonical()?; + let baseline = crate::canonicalize_cpu(dict_array.clone())?; let cuda_result = DictExecutor .execute(dict_array.into_array(), &mut cuda_ctx) @@ -725,7 +725,7 @@ mod tests { let dict_array = DictArray::try_new(codes_array.into_array(), values.into_array()) .vortex_expect("failed to create Dict array"); - let baseline = dict_array.to_canonical()?; + let baseline = crate::canonicalize_cpu(dict_array.clone())?; let cuda_result = DictExecutor .execute(dict_array.into_array(), &mut cuda_ctx) @@ -756,7 +756,7 @@ mod tests { let dict_array = DictArray::try_new(codes_array.into_array(), values.into_array()) .vortex_expect("failed to create Dict array"); - let baseline = dict_array.to_canonical()?; + let baseline = crate::canonicalize_cpu(dict_array.clone())?; let cuda_result = DictExecutor .execute(dict_array.into_array(), &mut cuda_ctx) @@ -792,7 +792,7 @@ mod tests { let dict_array = DictArray::try_new(codes_array.into_array(), values.into_array()) .vortex_expect("failed to create Dict array"); - let baseline = dict_array.to_canonical()?; + let baseline = crate::canonicalize_cpu(dict_array.clone())?; let cuda_result = DictExecutor .execute(dict_array.into_array(), &mut cuda_ctx) @@ -825,7 +825,7 @@ mod tests { let dict_array = DictArray::try_new(codes_array.into_array(), values.into_array()) .vortex_expect("failed to create Dict array"); - let baseline = dict_array.to_canonical()?; + let baseline = crate::canonicalize_cpu(dict_array.clone())?; let cuda_result = DictExecutor .execute(dict_array.into_array(), &mut cuda_ctx) @@ -850,7 +850,7 @@ mod tests { let dict_array = DictArray::try_new(codes_array.into_array(), values.into_array()) .vortex_expect("failed to create Dict array"); - let baseline = dict_array.to_canonical()?; + let baseline = crate::canonicalize_cpu(dict_array.clone())?; let cuda_result = DictExecutor .execute(dict_array.into_array(), &mut cuda_ctx) @@ -877,7 +877,7 @@ mod tests { let dict_array = DictArray::try_new(codes_array.into_array(), values.into_array()) .vortex_expect("failed to create Dict array"); - let baseline = dict_array.to_canonical()?; + let baseline = crate::canonicalize_cpu(dict_array.clone())?; let cuda_result = DictExecutor .execute(dict_array.into_array(), &mut cuda_ctx) @@ -907,7 +907,7 @@ mod tests { let dict_array = DictArray::try_new(codes_array.into_array(), values.into_array()) .vortex_expect("failed to create Dict array"); - let baseline = dict_array.to_canonical()?; + let baseline = crate::canonicalize_cpu(dict_array.clone())?; let cuda_result = DictExecutor .execute(dict_array.into_array(), &mut cuda_ctx) @@ -932,7 +932,7 @@ mod tests { let dict_array = DictArray::try_new(codes_array.into_array(), values.into_array()) .vortex_expect("failed to create Dict array"); - let baseline = dict_array.to_canonical()?; + let baseline = crate::canonicalize_cpu(dict_array.clone())?; let cuda_result = DictExecutor .execute(dict_array.into_array(), &mut cuda_ctx) @@ -958,7 +958,7 @@ mod tests { let dict_array = DictArray::try_new(codes_array.into_array(), values.into_array()) .vortex_expect("failed to create Dict array"); - let baseline = dict_array.to_canonical()?; + let baseline = crate::canonicalize_cpu(dict_array.clone())?; let cuda_result = DictExecutor .execute(dict_array.into_array(), &mut cuda_ctx) @@ -991,7 +991,7 @@ mod tests { let dict_array = DictArray::try_new(codes_array.into_array(), values.into_array()) .vortex_expect("failed to create Dict array"); - let baseline = dict_array.to_canonical()?; + let baseline = crate::canonicalize_cpu(dict_array.clone())?; let cuda_result = DictExecutor .execute(dict_array.into_array(), &mut cuda_ctx) @@ -1027,7 +1027,7 @@ mod tests { let dict_array = DictArray::try_new(codes_array.into_array(), values.into_array()) .vortex_expect("failed to create Dict array"); - let baseline = dict_array.to_canonical()?; + let baseline = crate::canonicalize_cpu(dict_array.clone())?; let cuda_result = DictExecutor .execute(dict_array.into_array(), &mut cuda_ctx) diff --git a/vortex-cuda/src/kernel/arrays/shared.rs b/vortex-cuda/src/kernel/arrays/shared.rs index 085b30f9057..16e748d26ca 100644 --- a/vortex-cuda/src/kernel/arrays/shared.rs +++ b/vortex-cuda/src/kernel/arrays/shared.rs @@ -34,6 +34,6 @@ impl CudaExecute for SharedExecutor { shared .get_or_compute_async(|source| source.execute_cuda(ctx)) .await? - .to_canonical() + .execute::(ctx.execution_ctx()) } } diff --git a/vortex-cuda/src/kernel/encodings/alp.rs b/vortex-cuda/src/kernel/encodings/alp.rs index 6661e0fed5e..77dd4dc73b4 100644 --- a/vortex-cuda/src/kernel/encodings/alp.rs +++ b/vortex-cuda/src/kernel/encodings/alp.rs @@ -2,10 +2,10 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors use std::fmt::Debug; -use std::sync::Arc; use async_trait::async_trait; use cudarc::driver::DeviceRepr; +use cudarc::driver::LaunchConfig; use cudarc::driver::PushKernelArg; use tracing::instrument; use vortex::array::ArrayRef; @@ -13,7 +13,7 @@ use vortex::array::Canonical; use vortex::array::arrays::PrimitiveArray; use vortex::array::arrays::primitive::PrimitiveDataParts; use vortex::array::buffer::BufferHandle; -use vortex::array::match_each_unsigned_integer_ptype; +use vortex::array::buffer::DeviceBufferExt; use vortex::dtype::NativePType; use vortex::encodings::alp::ALP; use vortex::encodings::alp::ALPArray; @@ -30,7 +30,8 @@ use crate::CudaDeviceBuffer; use crate::executor::CudaArrayExt; use crate::executor::CudaExecute; use crate::executor::CudaExecutionCtx; -use crate::kernel::patches::execute_patches; +use crate::kernel::patches::build_gpu_patches; +use crate::kernel::patches::types::load_patches; /// CUDA decoder for ALP (Adaptive Lossless floating-Point) decompression. #[derive(Debug)] @@ -54,6 +55,11 @@ impl CudaExecute for ALPExecutor { } } +/// Threads per block. 32 threads × 32 elements = 1024 element chunks for both +/// (f32, i32) and (f64, i64). f64 uses 8 KB of shared memory per block. +const ALP_THREADS_PER_BLOCK: u32 = 32; + +#[instrument(skip_all)] async fn decode_alp(array: ALPArray, ctx: &mut CudaExecutionCtx) -> VortexResult where A: ALPFloat + NativePType + DeviceRepr + Send + Sync + 'static, @@ -67,7 +73,7 @@ where let f: A = A::F10[exponents.f as usize]; let e: A = A::IF10[exponents.e as usize]; - // Execute child and copy to device + // Execute child and copy to device. let canonical = array.encoded().clone().execute_cuda(ctx).await?; let primitive = canonical.into_primitive(); let PrimitiveDataParts { @@ -75,42 +81,53 @@ where } = primitive.into_data_parts(); let device_input = ctx.ensure_on_device(buffer).await?; - - // Get CUDA view of input let input_view = device_input.cuda_view::()?; - // Allocate output buffer - let output_slice = ctx.device_alloc::(array_len)?; + // Allocate output rounded up to a full chunk: the fused kernel writes a + // whole 1024-element chunk per block, and we slice off any padding below. + let output_slice = ctx.device_alloc::(array_len.next_multiple_of(1024))?; let output_buf = CudaDeviceBuffer::new(output_slice); let output_view = output_buf.as_view::(); - let array_len_u64 = array_len as u64; + // Patch validity does not need to be scattered: the ALP encoder strips null + // positions from the exception list, so patches only exist at valid + // positions. load_patches additionally rejects patches without + // chunk_offsets (required by the fused kernel's PatchesCursor). + let device_patches = if let Some(patches) = array.patches() { + Some(load_patches(&patches, ctx).await?) + } else { + None + }; + let patches_arg = build_gpu_patches(device_patches.as_ref())?; - // Load kernel function - let kernel_ptypes = [A::ALPInt::PTYPE, A::PTYPE]; - let cuda_function = ctx.load_function("alp", &kernel_ptypes)?; + // Load the kernel: alp_{enc}_{float}_32t + let enc_suffix = A::ALPInt::PTYPE.to_string(); + let float_suffix = A::PTYPE.to_string(); + let cuda_function = ctx + .load_function_with_suffixes("alp", &[enc_suffix.as_str(), float_suffix.as_str(), "32t"])?; - ctx.launch_kernel(&cuda_function, array_len, |args| { + let num_blocks = u32::try_from(array_len.div_ceil(1024))?; + let config = LaunchConfig { + grid_dim: (num_blocks, 1, 1), + block_dim: (ALP_THREADS_PER_BLOCK, 1, 1), + shared_mem_bytes: 0, + }; + + let array_len_u64 = array_len as u64; + ctx.launch_kernel_config(&cuda_function, config, array_len, |args| { args.arg(&input_view) .arg(&output_view) .arg(&f) .arg(&e) - .arg(&array_len_u64); + .arg(&array_len_u64) + .arg(&patches_arg); })?; - // Check if there are any patches to decode here - let output_buf = if let Some(patches) = array.patches() { - match_each_unsigned_integer_ptype!(patches.indices_ptype()?, |I| { - execute_patches::(patches.clone(), output_buf, ctx).await? - }) - } else { - output_buf - }; + // Synchronize so the device patches buffers remain alive for the kernel. + ctx.synchronize_stream()?; + drop(device_patches); - // TODO(aduffy): scatter patch values validity. There are several places we'll need to start - // handling validity. - - let output_handle = BufferHandle::new_device(Arc::new(output_buf)); + let output_handle = BufferHandle::new_device(output_buf.slice_typed::(0..array_len)); Ok(Canonical::Primitive(PrimitiveArray::from_buffer_handle( output_handle, A::PTYPE, @@ -121,6 +138,8 @@ where #[cfg(test)] mod tests { use vortex::array::IntoArray; + use vortex::array::LEGACY_SESSION; + use vortex::array::VortexSessionExecute; use vortex::array::arrays::PrimitiveArray; use vortex::array::assert_arrays_eq; use vortex::array::patches::Patches; @@ -129,34 +148,44 @@ mod tests { use vortex::buffer::buffer; use vortex::encodings::alp::ALP; use vortex::encodings::alp::Exponents; + use vortex::encodings::alp::alp_encode; use vortex::error::VortexExpect; use vortex::session::VortexSession; use super::*; use crate::CanonicalCudaExt; + use crate::canonicalize_cpu; + use crate::executor::CudaArrayExt; use crate::session::CudaSession; + /// Irrational values ALP cannot encode losslessly, guaranteed to land + /// in the exception list on round-trip through `alp_encode`. + const UNENCODABLE: f64 = std::f64::consts::PI; + const UNENCODABLE_F32: f32 = std::f32::consts::PI; + + /// Small manually-constructed ALP array with patches. Exercises the + /// custom-construction path (as opposed to going through `alp_encode`). + /// Patches must carry `chunk_offsets` — the fused kernel requires them. #[crate::test] async fn test_cuda_alp_decompression_f32() -> VortexResult<()> { let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) .vortex_expect("failed to create execution context"); - // Create encoded values (what ALP would produce) // For f32 with exponents (e=0, f=2): decoded = encoded * F10[2] * IF10[0] // = encoded * 100.0 * 1.0 - // So encoded value of 100 -> decoded 10000.0 + // Encoded value of 100 -> decoded 10000.0. let encoded_data: Vec = vec![100, 200, 300, 400, 500]; - let exponents = Exponents { e: 0, f: 2 }; // multiply by 100 + let exponents = Exponents { e: 0, f: 2 }; - // Patches + // One chunk holds all 5 elements. chunk_offsets[0] = 0: chunk 0's + // patches begin at patch index 0. let patches = Patches::new( 5, 0, PrimitiveArray::new(buffer![0u32, 4u32], Validity::NonNullable).into_array(), PrimitiveArray::new(buffer![0.0f32, 999f32], Validity::NonNullable).into_array(), - None, - ) - .unwrap(); + Some(PrimitiveArray::new(buffer![0u32], Validity::NonNullable).into_array()), + )?; let alp_array = ALP::try_new( PrimitiveArray::new(Buffer::from(encoded_data.clone()), Validity::NonNullable) @@ -165,7 +194,7 @@ mod tests { Some(patches), )?; - let cpu_result = alp_array.to_canonical()?.into_array(); + let cpu_result = canonicalize_cpu(alp_array.clone())?.into_array(); let gpu_result = ALPExecutor .execute(alp_array.into_array(), &mut cuda_ctx) @@ -179,4 +208,252 @@ mod tests { Ok(()) } + + /// ALP with nullable encoded data and patches — the encoder strips null + /// positions from the exception list, so patch validity doesn't need + /// scattering. This test verifies that the encoded child's validity is + /// preserved through the standalone ALP GPU executor. + #[crate::test] + async fn test_cuda_alp_nullable_with_patches() -> VortexResult<()> { + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) + .vortex_expect("failed to create execution context"); + + // Values that will produce ALP exceptions at non-null positions. + // Nulls at positions 1 and 3; the exception at position 4 can't be + // encoded losslessly by ALP. + let values: Vec> = vec![ + Some(1.0), + None, + Some(2.0), + None, + Some(UNENCODABLE_F32), + Some(3.0), + Some(4.0), + Some(5.0), + ]; + let prim = PrimitiveArray::from_option_iter(values); + let alp_array = alp_encode( + prim.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; + + let cpu_result = canonicalize_cpu(alp_array.clone())?.into_array(); + + let gpu_result = alp_array + .into_array() + .execute_cuda(&mut cuda_ctx) + .await? + .into_host() + .await? + .into_array(); + + assert_arrays_eq!(cpu_result, gpu_result); + Ok(()) + } + + /// ALP with all-valid nullable data — the dtype is nullable but no + /// elements are actually null. + #[crate::test] + async fn test_cuda_alp_all_valid_nullable() -> VortexResult<()> { + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) + .vortex_expect("failed to create execution context"); + + let values = PrimitiveArray::new( + Buffer::from(vec![1.0f32, 2.0, 3.0, 4.0, 5.0]), + Validity::AllValid, + ); + let alp_array = alp_encode( + values.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; + + let cpu_result = canonicalize_cpu(alp_array.clone())?.into_array(); + + let gpu_result = alp_array + .into_array() + .execute_cuda(&mut cuda_ctx) + .await? + .into_host() + .await? + .into_array(); + + assert_arrays_eq!(cpu_result, gpu_result); + Ok(()) + } + + /// Multi-chunk ALP (> 1024 elements) with patches in chunks 0 and 2 but + /// none in chunk 1. Exercises the `PatchesCursor` branch where a + /// non-trailing chunk has `chunk_offsets[c] == chunk_offsets[c+1]` + /// (zero patches) via the offset math rather than the NULL sentinel. + #[crate::test] + async fn test_cuda_alp_multi_chunk_sparse_patches() -> VortexResult<()> { + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) + .vortex_expect("failed to create execution context"); + + // 3072 values (3 chunks). Inject exceptions (values ALP can't encode + // losslessly) only in chunks 0 and 2; chunk 1 stays exception-free so + // its cursor slice is empty despite patches existing in the array. + let values: Buffer = (0u32..3072) + .map(|i| { + if matches!(i, 0 | 100 | 1023 | 3071) { + UNENCODABLE_F32 + } else { + i as f32 + } + }) + .collect(); + let prim = PrimitiveArray::new(values, Validity::NonNullable); + let alp_array = alp_encode( + prim.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; + assert!( + alp_array.patches().is_some(), + "expected patches from ALP exceptions" + ); + + let cpu_result = canonicalize_cpu(alp_array.clone())?.into_array(); + + let gpu_result = alp_array + .into_array() + .execute_cuda(&mut cuda_ctx) + .await? + .into_host() + .await? + .into_array(); + + assert_arrays_eq!(cpu_result, gpu_result); + Ok(()) + } + + /// Multi-chunk f64 decode with patches distributed across chunks. The f64 + /// path (i64 → double) is otherwise only covered by the partial-tail case, + /// so this guards the fast-path for the (i64, f64) kernel variant. + #[crate::test] + async fn test_cuda_alp_f64_multi_chunk_with_patches() -> VortexResult<()> { + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) + .vortex_expect("failed to create execution context"); + + // 3072 values (3 chunks). Sprinkle exceptions into each chunk. + let values: Buffer = (0u32..3072) + .map(|i| { + if matches!(i, 0 | 500 | 1024 | 1500 | 2048 | 3071) { + UNENCODABLE + } else { + i as f64 + } + }) + .collect(); + let prim = PrimitiveArray::new(values, Validity::NonNullable); + let alp_array = alp_encode( + prim.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; + assert!( + alp_array.patches().is_some(), + "expected patches from ALP exceptions" + ); + + let cpu_result = canonicalize_cpu(alp_array.clone())?.into_array(); + + let gpu_result = alp_array + .into_array() + .execute_cuda(&mut cuda_ctx) + .await? + .into_host() + .await? + .into_array(); + + assert_arrays_eq!(cpu_result, gpu_result); + Ok(()) + } + + /// Single chunk with more patches than threads per block (32). Forces + /// `PatchesCursor` to split patches across multiple threads, exercising + /// the per-thread ceil-division and clamping math that no other test hits + /// (existing tests have ≤ 6 patches per chunk). + #[crate::test] + async fn test_cuda_alp_dense_patches_single_chunk() -> VortexResult<()> { + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) + .vortex_expect("failed to create execution context"); + + // Build a 1024-element ALP array manually with exactly 40 patches + // all in the single chunk. 40 > 32 forces cursor division (each of + // the first 20 threads handles 2 patches; remaining threads idle). + const LEN: i32 = 1024; + const NUM_PATCHES: u32 = 40; + + let exponents = Exponents { e: 0, f: 2 }; + let encoded: Buffer = (0i32..LEN).map(|i| i * 100).collect(); + + let patch_indices: Buffer = (0u32..NUM_PATCHES).collect(); + let patch_values: Buffer = (0..NUM_PATCHES).map(|i| i as f32 * 0.125 + 0.5).collect(); + + let patches = Patches::new( + LEN as usize, + 0, + PrimitiveArray::new(patch_indices, Validity::NonNullable).into_array(), + PrimitiveArray::new(patch_values, Validity::NonNullable).into_array(), + Some(PrimitiveArray::new(buffer![0u32], Validity::NonNullable).into_array()), + )?; + + let alp_array = ALP::try_new( + PrimitiveArray::new(encoded, Validity::NonNullable).into_array(), + exponents, + Some(patches), + )?; + + let cpu_result = canonicalize_cpu(alp_array.clone())?.into_array(); + + let gpu_result = ALPExecutor + .execute(alp_array.into_array(), &mut cuda_ctx) + .await + .vortex_expect("GPU decompression failed") + .into_host() + .await? + .into_array(); + + assert_arrays_eq!(cpu_result, gpu_result); + Ok(()) + } + + /// Tail-chunk bounds check: an array whose length is not a multiple of + /// 1024 forces the kernel's tail-block path to bounds-check its decode + /// loop. Includes a patch in the tail. + #[crate::test] + async fn test_cuda_alp_partial_tail_chunk() -> VortexResult<()> { + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) + .vortex_expect("failed to create execution context"); + + let values: Buffer = (0u32..1500) + .map(|i| if i == 1400 { UNENCODABLE } else { i as f64 }) + .collect(); + let prim = PrimitiveArray::new(values, Validity::NonNullable); + let alp_array = alp_encode( + prim.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; + assert!( + alp_array.patches().is_some(), + "expected patches from ALP exceptions" + ); + + let cpu_result = canonicalize_cpu(alp_array.clone())?.into_array(); + + let gpu_result = alp_array + .into_array() + .execute_cuda(&mut cuda_ctx) + .await? + .into_host() + .await? + .into_array(); + + assert_arrays_eq!(cpu_result, gpu_result); + Ok(()) + } } diff --git a/vortex-cuda/src/kernel/encodings/bitpacked.rs b/vortex-cuda/src/kernel/encodings/bitpacked.rs index 357d444fae4..29b56436feb 100644 --- a/vortex-cuda/src/kernel/encodings/bitpacked.rs +++ b/vortex-cuda/src/kernel/encodings/bitpacked.rs @@ -28,8 +28,8 @@ use crate::CudaBufferExt; use crate::CudaDeviceBuffer; use crate::executor::CudaExecute; use crate::executor::CudaExecutionCtx; -use crate::kernel::patches::gpu::GPUPatches; -use crate::kernel::patches::types::transpose_patches; +use crate::kernel::patches::build_gpu_patches; +use crate::kernel::patches::types::load_patches; /// CUDA decoder for bit-packed arrays. #[derive(Debug)] @@ -84,8 +84,6 @@ pub fn bitpacked_cuda_launch_config(output_width: usize, len: usize) -> VortexRe }) } -unsafe impl DeviceRepr for GPUPatches {} - #[instrument(skip_all)] pub(crate) async fn decode_bitpacked( array: BitPackedArray, @@ -124,25 +122,12 @@ where // We hold this here to keep the device buffers alive. let device_patches = if let Some(patches) = patches { - Some(transpose_patches(&patches, ctx).await?) + Some(load_patches(&patches, ctx).await?) } else { None }; - let patches_arg = if let Some(p) = &device_patches { - GPUPatches { - lane_offsets: p.lane_offsets.cuda_device_ptr()? as _, - indices: p.indices.cuda_device_ptr()? as _, - values: p.values.cuda_device_ptr()? as _, - } - } else { - // NULL lane_offsets signals no patches to the kernel - GPUPatches { - lane_offsets: std::ptr::null_mut(), - indices: std::ptr::null_mut(), - values: std::ptr::null_mut(), - } - }; + let patches_arg = build_gpu_patches(device_patches.as_ref())?; ctx.launch_kernel_config(&cuda_function, config, len, |args| { args.arg(&input_view) @@ -175,9 +160,12 @@ mod tests { use vortex::array::dtype::NativePType; use vortex::array::validity::Validity::NonNullable; use vortex::buffer::Buffer; + use vortex::buffer::buffer; use vortex::encodings::fastlanes::BitPackedArrayExt; use vortex::error::VortexExpect; use vortex::session::VortexSession; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use super::*; use crate::CanonicalCudaExt; @@ -199,10 +187,14 @@ mod tests { let array = PrimitiveArray::new(iter.collect::>(), NonNullable); // Last two items should be patched - let bp_with_patches = BitPacked::encode(&array.into_array(), bw)?; + let bp_with_patches = BitPacked::encode( + &array.into_array(), + bw, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; assert!(bp_with_patches.patches().is_some()); - let cpu_result = bp_with_patches.to_canonical()?.into_array(); + let cpu_result = crate::canonicalize_cpu(bp_with_patches.clone())?.into_array(); let gpu_result = block_on(async { BitPackedExecutor @@ -230,10 +222,14 @@ mod tests { ); // Last two items should be patched - let bp_with_patches = BitPacked::encode(&array.into_array(), 9)?; + let bp_with_patches = BitPacked::encode( + &array.into_array(), + 9, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; assert!(bp_with_patches.patches().is_some()); - let cpu_result = bp_with_patches.to_canonical()?.into_array(); + let cpu_result = crate::canonicalize_cpu(bp_with_patches.clone())?.into_array(); let gpu_result = block_on(async { BitPackedExecutor @@ -272,9 +268,13 @@ mod tests { NonNullable, ); - let bitpacked_array = BitPacked::encode(&primitive_array.into_array(), bit_width) - .vortex_expect("operation should succeed in test"); - let cpu_result = bitpacked_array.to_canonical()?; + let bitpacked_array = BitPacked::encode( + &primitive_array.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .vortex_expect("operation should succeed in test"); + let cpu_result = crate::canonicalize_cpu(bitpacked_array.clone())?; let gpu_result = block_on(async { BitPackedExecutor @@ -321,9 +321,13 @@ mod tests { NonNullable, ); - let bitpacked_array = BitPacked::encode(&primitive_array.into_array(), bit_width) - .vortex_expect("operation should succeed in test"); - let cpu_result = bitpacked_array.to_canonical()?; + let bitpacked_array = BitPacked::encode( + &primitive_array.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .vortex_expect("operation should succeed in test"); + let cpu_result = crate::canonicalize_cpu(bitpacked_array.clone())?; let gpu_result = block_on(async { BitPackedExecutor @@ -386,9 +390,13 @@ mod tests { NonNullable, ); - let bitpacked_array = BitPacked::encode(&primitive_array.into_array(), bit_width) - .vortex_expect("operation should succeed in test"); - let cpu_result = bitpacked_array.to_canonical()?; + let bitpacked_array = BitPacked::encode( + &primitive_array.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .vortex_expect("operation should succeed in test"); + let cpu_result = crate::canonicalize_cpu(bitpacked_array.clone())?; let gpu_result = block_on(async { BitPackedExecutor @@ -483,9 +491,13 @@ mod tests { NonNullable, ); - let bitpacked_array = BitPacked::encode(&primitive_array.into_array(), bit_width) - .vortex_expect("operation should succeed in test"); - let cpu_result = bitpacked_array.to_canonical()?; + let bitpacked_array = BitPacked::encode( + &primitive_array.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .vortex_expect("operation should succeed in test"); + let cpu_result = crate::canonicalize_cpu(bitpacked_array.clone())?; let gpu_result = block_on(async { BitPackedExecutor .execute(bitpacked_array.into_array(), &mut cuda_ctx) @@ -516,11 +528,168 @@ mod tests { NonNullable, ); - let bitpacked_array = BitPacked::encode(&primitive_array.into_array(), bit_width) - .vortex_expect("operation should succeed in test"); + let bitpacked_array = BitPacked::encode( + &primitive_array.into_array(), + bit_width, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .vortex_expect("operation should succeed in test"); let sliced_array = bitpacked_array.into_array().slice(67..3969)?; assert!(sliced_array.is::()); - let cpu_result = sliced_array.to_canonical()?; + let cpu_result = crate::canonicalize_cpu(sliced_array.clone())?; + let gpu_result = block_on(async { + BitPackedExecutor + .execute(sliced_array, &mut cuda_ctx) + .await + .vortex_expect("GPU decompression failed") + .into_host() + .await + .map(|a| a.into_array()) + })?; + + assert_arrays_eq!(cpu_result.into_array(), gpu_result); + + Ok(()) + } + + /// Test slicing a bitpacked array with patches where the slice boundary + /// falls in the middle of a chunk's patch range, creating a non-zero + /// offset_within_chunk. + #[crate::test] + fn test_cuda_bitunpack_sliced_patches_offset_within_chunk() -> VortexResult<()> { + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) + .vortex_expect("failed to create execution context"); + + // Create an array with values that will generate patches. + // We use values 0-511 (fits in 9 bits) but include some larger values + // that will become patches. + let primitive_array = PrimitiveArray::new(buffer![100u8, 101, 102, 3, 4, 5], NonNullable); + + // Encode with bit width 4. First 3 elements patched, remainder will pack. + let bitpacked_array = BitPacked::encode( + &primitive_array.into_array(), + 4, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; + assert!( + bitpacked_array.patches().is_some(), + "Expected patches to be present" + ); + + let sliced_array = bitpacked_array.into_array().slice(2..6)?; + assert!(sliced_array.is::()); + + let cpu_result = sliced_array + .clone() + .execute::(cuda_ctx.execution_ctx())?; + let gpu_result = block_on(async { + BitPackedExecutor + .execute(sliced_array, &mut cuda_ctx) + .await + .vortex_expect("GPU decompression failed") + .into_host() + .await + .map(|a| a.into_array()) + })?; + + assert_arrays_eq!(cpu_result.into_array(), gpu_result); + + Ok(()) + } + + /// Test slicing a bitpacked array multiple times, accumulating offset_within_chunk. + #[crate::test] + fn test_cuda_bitunpack_double_sliced_patches() -> VortexResult<()> { + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) + .vortex_expect("failed to create execution context"); + + // Create an array with values that will generate patches. + let mut values: Vec = Vec::with_capacity(3072); + for i in 0u16..3072 { + if i == 50 || i == 100 || i == 200 || i == 300 || i == 400 || i == 1100 || i == 2100 { + values.push(600); + } else { + values.push(i % 512); + } + } + + let primitive_array = + PrimitiveArray::new(Buffer::from_iter(values.iter().copied()), NonNullable); + + let bitpacked_array = BitPacked::encode( + &primitive_array.into_array(), + 9, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; + assert!( + bitpacked_array.patches().is_some(), + "Expected patches to be present" + ); + + // First slice: drop the patch at index 50 from the front of chunk 0. + let first_slice = bitpacked_array.into_array().slice(75..3000)?; + // Second slice (relative to first): drop patch at original index 100. + // The second slice's range is kept wide enough that num_blocks still + // covers every chunk in the packed buffer. + let second_slice = first_slice.slice(50..2900)?; + assert!(second_slice.is::()); + + let cpu_result = second_slice + .clone() + .execute::(cuda_ctx.execution_ctx())?; + let gpu_result = block_on(async { + BitPackedExecutor + .execute(second_slice, &mut cuda_ctx) + .await + .vortex_expect("GPU decompression failed") + .into_host() + .await + .map(|a| a.into_array()) + })?; + + assert_arrays_eq!(cpu_result.into_array(), gpu_result); + + Ok(()) + } + + /// Test slicing to skip an entire chunk's worth of patches. + #[crate::test] + fn test_cuda_bitunpack_sliced_skip_first_chunk_patches() -> VortexResult<()> { + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) + .vortex_expect("failed to create execution context"); + + // Create patches in first chunk only, then slice past them all. + let mut values: Vec = Vec::with_capacity(3072); + for i in 0u16..3072 { + if i == 100 || i == 200 || i == 300 { + values.push(600); + } else if i == 1500 || i == 2500 { + values.push(700); + } else { + values.push(i % 512); + } + } + + let primitive_array = + PrimitiveArray::new(Buffer::from_iter(values.iter().copied()), NonNullable); + + let bitpacked_array = BitPacked::encode( + &primitive_array.into_array(), + 9, + &mut LEGACY_SESSION.create_execution_ctx(), + )?; + assert!( + bitpacked_array.patches().is_some(), + "Expected patches to be present" + ); + + // Slice to skip past all first chunk patches + let sliced_array = bitpacked_array.into_array().slice(1024..3072)?; + assert!(sliced_array.is::()); + + let cpu_result = sliced_array + .clone() + .execute::(cuda_ctx.execution_ctx())?; let gpu_result = block_on(async { BitPackedExecutor .execute(sliced_array, &mut cuda_ctx) diff --git a/vortex-cuda/src/kernel/encodings/date_time_parts.rs b/vortex-cuda/src/kernel/encodings/date_time_parts.rs index 421a759e374..d8c655c2558 100644 --- a/vortex-cuda/src/kernel/encodings/date_time_parts.rs +++ b/vortex-cuda/src/kernel/encodings/date_time_parts.rs @@ -22,7 +22,7 @@ use vortex::dtype::NativePType; use vortex::dtype::Nullability; use vortex::dtype::PType; use vortex::encodings::datetime_parts::DateTimeParts; -use vortex::encodings::datetime_parts::DateTimePartsArrayExt; +use vortex::encodings::datetime_parts::DateTimePartsArraySlotsExt; use vortex::error::VortexResult; use vortex::error::vortex_bail; use vortex::error::vortex_err; @@ -66,7 +66,7 @@ impl CudaExecute for DateTimePartsExecutor { let time_unit = options.unit; let time_zone = options.tz.clone(); - let validity = Validity::copy_from_array(&array.clone().into_array())?; + let validity = array.validity()?; if output_len == 0 { return Ok(Canonical::empty(array.dtype())); @@ -130,7 +130,7 @@ impl CudaExecute for DateTimePartsExecutor { } } -#[allow(clippy::too_many_arguments)] +#[expect(clippy::too_many_arguments)] async fn decode_datetimeparts_typed( days: PrimitiveArray, seconds: PrimitiveArray, @@ -284,7 +284,7 @@ mod tests { .vortex_expect("failed to create execution context"); let dtp_array = make_datetimeparts_array(days, seconds, subseconds, time_unit); - let cpu_result = dtp_array.to_canonical()?; + let cpu_result = crate::canonicalize_cpu(dtp_array.clone())?; let gpu_result = DateTimePartsExecutor .execute(dtp_array.into_array(), &mut cuda_ctx) @@ -310,7 +310,7 @@ mod tests { let subseconds: Vec = (0..len).map(|i| (i % 1000) as i64).collect(); let dtp_array = make_datetimeparts_array(days, seconds, subseconds, TimeUnit::Milliseconds); - let cpu_result = dtp_array.to_canonical()?; + let cpu_result = crate::canonicalize_cpu(dtp_array.clone())?; let gpu_result = DateTimePartsExecutor .execute(dtp_array.into_array(), &mut cuda_ctx) @@ -358,7 +358,7 @@ mod tests { ) .vortex_expect("Failed to create DateTimePartsArray"); - let cpu_result = dtp_array.to_canonical()?; + let cpu_result = crate::canonicalize_cpu(dtp_array.clone())?; let gpu_result = DateTimePartsExecutor .execute(dtp_array.into_array(), &mut cuda_ctx) diff --git a/vortex-cuda/src/kernel/encodings/decimal_byte_parts.rs b/vortex-cuda/src/kernel/encodings/decimal_byte_parts.rs index 3f4bdb92146..70d3c71a485 100644 --- a/vortex-cuda/src/kernel/encodings/decimal_byte_parts.rs +++ b/vortex-cuda/src/kernel/encodings/decimal_byte_parts.rs @@ -96,7 +96,8 @@ mod tests { ) .vortex_expect("create DecimalBytePartsArray"); - let cpu_result = dbp_array.to_canonical().vortex_expect("CPU canonicalize"); + let cpu_result = + crate::canonicalize_cpu(dbp_array.clone()).vortex_expect("CPU canonicalize"); let gpu_result = DecimalBytePartsExecutor .execute(dbp_array.into_array(), &mut cuda_ctx) diff --git a/vortex-cuda/src/kernel/encodings/for_.rs b/vortex-cuda/src/kernel/encodings/for_.rs index 490b797b9b8..cdca6eeab7d 100644 --- a/vortex-cuda/src/kernel/encodings/for_.rs +++ b/vortex-cuda/src/kernel/encodings/for_.rs @@ -74,7 +74,7 @@ impl CudaExecute for FoRExecutor { .into_primitive() .into_array() .slice(slice_range)? - .to_canonical(); + .execute::(ctx.execution_ctx()); } match_each_native_simd_ptype!(array.ptype(), |P| { decode_for::

(array, ctx).await }) @@ -139,6 +139,8 @@ mod tests { use vortex::error::VortexExpect; use vortex::scalar::Scalar; use vortex::session::VortexSession; + use vortex_array::LEGACY_SESSION; + use vortex_array::VortexSessionExecute; use super::*; use crate::CanonicalCudaExt; @@ -162,7 +164,7 @@ mod tests { let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) .vortex_expect("failed to create execution context"); - let cpu_result = for_array.to_canonical()?; + let cpu_result = crate::canonicalize_cpu(for_array.clone())?; let gpu_result = FoRExecutor .execute(for_array.into_array(), &mut cuda_ctx) @@ -187,10 +189,12 @@ mod tests { .take(1024) .collect::>() .into_array(); - let packed = BitPacked::encode(&values, 3).unwrap().into_array(); + let packed = BitPacked::encode(&values, 3, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .into_array(); let for_array = FoR::try_new(packed, (-8i8).into()).unwrap(); - let cpu_result = for_array.to_canonical().unwrap(); + let cpu_result = crate::canonicalize_cpu(for_array.clone()).unwrap(); let gpu_result = FoRExecutor .execute(for_array.into_array(), &mut cuda_ctx) diff --git a/vortex-cuda/src/kernel/encodings/runend.rs b/vortex-cuda/src/kernel/encodings/runend.rs index 3f3bce2579b..fca435478d8 100644 --- a/vortex-cuda/src/kernel/encodings/runend.rs +++ b/vortex-cuda/src/kernel/encodings/runend.rs @@ -78,7 +78,7 @@ impl CudaExecute for RunEndExecutor { if matches!(values.validity()?, Validity::AllInvalid) { return ConstantArray::new(Scalar::null(values.dtype().clone()), output_len) .into_array() - .to_canonical(); + .execute::(ctx.execution_ctx()); } let ends = ends.execute_cuda(ctx).await?.into_primitive(); @@ -148,7 +148,9 @@ async fn decode_runend_typed { - unreachable!("Array validity not yet supported for run-end decoding on GPU"); + vortex_bail!( + "RunEnd GPU decoding does not yet support per-element validity in values; falling back to CPU" + ); } }; @@ -163,6 +165,7 @@ async fn decode_runend_typed(ends: Vec, values: Vec) -> RunEndArray + fn make_runend_array( + ends: Vec, + values: Vec, + ctx: &mut vortex::array::ExecutionCtx, + ) -> RunEndArray where V: NativePType, E: NativePType, @@ -186,22 +194,25 @@ mod tests { PrimitiveArray::new(Buffer::from(ends), Validity::NonNullable).into_array(); let values_array = PrimitiveArray::new(Buffer::from(values), Validity::NonNullable).into_array(); - RunEnd::new(ends_array, values_array) + RunEnd::new(ends_array, values_array, ctx) } + type RunEndBuilder = fn(&mut vortex::array::ExecutionCtx) -> RunEndArray; + #[rstest] - #[case::u32_ends_u8_values(make_runend_array(vec![3u32, 6, 10], vec![10u8, 20, 30]))] - #[case::u32_ends_u32_values(make_runend_array(vec![2u32, 5, 10], vec![1u32, 2, 3]))] - #[case::u32_ends_f64_values(make_runend_array(vec![2u32, 5, 8], vec![1.5f64, 2.5, 3.5]))] - #[case::u8_ends_i32_values(make_runend_array(vec![2u8, 5, 10], vec![1i32, 2, 3]))] - #[case::u32_ends_i32_values(make_runend_array(vec![2u32, 5, 10], vec![1i32, 2, 3]))] - #[case::u64_ends_i32_values(make_runend_array(vec![2u64, 5, 10], vec![1i32, 2, 3]))] + #[case::u32_ends_u8_values(|ctx: &mut vortex::array::ExecutionCtx| make_runend_array(vec![3u32, 6, 10], vec![10u8, 20, 30], ctx))] + #[case::u32_ends_u32_values(|ctx: &mut vortex::array::ExecutionCtx| make_runend_array(vec![2u32, 5, 10], vec![1u32, 2, 3], ctx))] + #[case::u32_ends_f64_values(|ctx: &mut vortex::array::ExecutionCtx| make_runend_array(vec![2u32, 5, 8], vec![1.5f64, 2.5, 3.5], ctx))] + #[case::u8_ends_i32_values(|ctx: &mut vortex::array::ExecutionCtx| make_runend_array(vec![2u8, 5, 10], vec![1i32, 2, 3], ctx))] + #[case::u32_ends_i32_values(|ctx: &mut vortex::array::ExecutionCtx| make_runend_array(vec![2u32, 5, 10], vec![1i32, 2, 3], ctx))] + #[case::u64_ends_i32_values(|ctx: &mut vortex::array::ExecutionCtx| make_runend_array(vec![2u64, 5, 10], vec![1i32, 2, 3], ctx))] #[crate::test] - async fn test_cuda_runend_types(#[case] runend_array: RunEndArray) -> VortexResult<()> { + async fn test_cuda_runend_types(#[case] build: RunEndBuilder) -> VortexResult<()> { let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) .vortex_expect("failed to create execution context"); - let cpu_result = runend_array.to_canonical()?; + let runend_array = build(cuda_ctx.execution_ctx()); + let cpu_result = crate::canonicalize_cpu(runend_array.clone())?; let gpu_result = RunEndExecutor .execute(runend_array.into_array(), &mut cuda_ctx) @@ -228,10 +239,10 @@ mod tests { let ends: Vec = (1..=num_runs).map(|i| (i * run_length) as u64).collect(); let values: Vec = (0..num_runs).map(|i| i32::try_from(i).unwrap()).collect(); - let runend_array = make_runend_array(ends, values); + let runend_array = make_runend_array(ends, values, cuda_ctx.execution_ctx()); assert_eq!(runend_array.len(), total_len); - let cpu_result = runend_array.to_canonical()?; + let cpu_result = crate::canonicalize_cpu(runend_array.clone())?; let gpu_result = RunEndExecutor .execute(runend_array.into_array(), &mut cuda_ctx) @@ -251,9 +262,9 @@ mod tests { let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) .vortex_expect("failed to create execution context"); - let runend_array = make_runend_array(vec![100u32], vec![42i32]); + let runend_array = make_runend_array(vec![100u32], vec![42i32], cuda_ctx.execution_ctx()); - let cpu_result = runend_array.to_canonical()?; + let cpu_result = crate::canonicalize_cpu(runend_array.clone())?; let gpu_result = RunEndExecutor .execute(runend_array.into_array(), &mut cuda_ctx) @@ -278,9 +289,9 @@ mod tests { let ends: Vec = (1..=num_elements).collect(); let values: Vec = (0..num_elements as i32).collect(); - let runend_array = make_runend_array(ends, values); + let runend_array = make_runend_array(ends, values, cuda_ctx.execution_ctx()); - let cpu_result = runend_array.to_canonical()?; + let cpu_result = crate::canonicalize_cpu(runend_array.clone())?; let gpu_result = RunEndExecutor .execute(runend_array.into_array(), &mut cuda_ctx) @@ -294,4 +305,36 @@ mod tests { Ok(()) } + + #[crate::test] + async fn test_cuda_runend_nullable_values_falls_back_to_cpu() -> VortexResult<()> { + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) + .vortex_expect("failed to create execution context"); + + // Build a RunEnd array whose values have Validity::Array (some nulls). + let ends_array = + PrimitiveArray::new(Buffer::from(vec![3u32, 6, 10]), Validity::NonNullable) + .into_array(); + let validity = + Validity::Array(BoolArray::from_iter([true, false, true].into_iter()).into_array()); + let values_array = + PrimitiveArray::new(Buffer::from(vec![10i32, 0, 30]), validity).into_array(); + let runend_array = RunEnd::new(ends_array, values_array, cuda_ctx.execution_ctx()); + + let cpu_result = crate::canonicalize_cpu(runend_array.clone())?.into_array(); + + // execute_cuda should fall back to CPU and still produce the correct result. + let gpu_result = runend_array + .into_array() + .execute_cuda(&mut cuda_ctx) + .await + .vortex_expect("GPU/CPU fallback should succeed") + .into_host() + .await? + .into_array(); + + assert_arrays_eq!(cpu_result, gpu_result); + + Ok(()) + } } diff --git a/vortex-cuda/src/kernel/encodings/sequence.rs b/vortex-cuda/src/kernel/encodings/sequence.rs index bcc90ec77b7..c0721b8d920 100644 --- a/vortex-cuda/src/kernel/encodings/sequence.rs +++ b/vortex-cuda/src/kernel/encodings/sequence.rs @@ -129,7 +129,7 @@ mod tests { let array = Sequence::try_new_typed(base, multiplier, nullability, len).unwrap(); - let cpu_result = array.to_canonical().unwrap().into_array(); + let cpu_result = crate::canonicalize_cpu(array.clone()).unwrap().into_array(); let gpu_result = SequenceExecutor .execute(array.into_array(), &mut cuda_ctx) diff --git a/vortex-cuda/src/kernel/encodings/zigzag.rs b/vortex-cuda/src/kernel/encodings/zigzag.rs index f13a19bf0e1..98f9534be0a 100644 --- a/vortex-cuda/src/kernel/encodings/zigzag.rs +++ b/vortex-cuda/src/kernel/encodings/zigzag.rs @@ -124,7 +124,7 @@ mod tests { PrimitiveArray::new(Buffer::from(encoded_data), NonNullable).into_array(), )?; - let cpu_result = zigzag_array.to_canonical()?; + let cpu_result = crate::canonicalize_cpu(zigzag_array.clone())?; let gpu_result = ZigZagExecutor .execute(zigzag_array.into_array(), &mut cuda_ctx) diff --git a/vortex-cuda/src/kernel/encodings/zstd.rs b/vortex-cuda/src/kernel/encodings/zstd.rs index 77ade13a6b0..abadeb1698f 100644 --- a/vortex-cuda/src/kernel/encodings/zstd.rs +++ b/vortex-cuda/src/kernel/encodings/zstd.rs @@ -29,8 +29,10 @@ use vortex::encodings::zstd::Zstd; use vortex::encodings::zstd::ZstdArray; use vortex::encodings::zstd::ZstdDataParts; use vortex::encodings::zstd::ZstdMetadata; +use vortex::encodings::zstd::reconstruct_views; use vortex::error::VortexExpect; use vortex::error::VortexResult; +use vortex::error::vortex_bail; use vortex::error::vortex_err; use vortex::mask::AllOr; use vortex_nvcomp::sys::nvcompStatus_t; @@ -207,7 +209,8 @@ impl CudaExecute for ZstdExecutor { dtype = %_other, "Only Binary/Utf8 ZSTD arrays supported on GPU, falling back to CPU" ); - Zstd::decompress(&zstd, ctx.execution_ctx())?.to_canonical() + Zstd::decompress(&zstd, ctx.execution_ctx())? + .execute::(ctx.execution_ctx()) } } } @@ -215,7 +218,7 @@ impl CudaExecute for ZstdExecutor { async fn decode_zstd(array: ZstdArray, ctx: &mut CudaExecutionCtx) -> VortexResult { let dtype = array.dtype().clone(); - let validity = child_to_validity(&array.as_ref().slots()[0], dtype.nullability()); + let validity = child_to_validity(array.as_ref().slots()[0].as_ref(), dtype.nullability()); let ZstdDataParts { frames, metadata, @@ -328,8 +331,7 @@ async fn decode_zstd(array: ZstdArray, ctx: &mut CudaExecutionCtx) -> VortexResu .indices() { AllOr::All => { - let (buffers, all_views) = - vortex::encodings::zstd::reconstruct_views(&host_buffer, MAX_BUFFER_LEN); + let (buffers, all_views) = reconstruct_views(&host_buffer, MAX_BUFFER_LEN); let sliced_views = all_views.slice(slice_value_idx_start..slice_value_idx_stop); Ok(Canonical::VarBinView(unsafe { @@ -342,7 +344,7 @@ async fn decode_zstd(array: ZstdArray, ctx: &mut CudaExecutionCtx) -> VortexResu })) } _ => { - unimplemented!("CUDA ZSTD decompression does not yet support arrays with nulls") + vortex_bail!("CUDA ZSTD decompression does not yet support arrays with nulls") } } } @@ -357,6 +359,8 @@ mod tests { use vortex::session::VortexSession; use super::*; + use crate::CanonicalCudaExt; + use crate::executor::CudaArrayExt; use crate::session::CudaSession; #[crate::test] @@ -373,9 +377,10 @@ mod tests { "baz", ]); - let zstd_array = Zstd::from_var_bin_view(&strings, 3, 0)?; + let zstd_array = Zstd::from_var_bin_view(&strings, 3, 0, cuda_ctx.execution_ctx())?; - let cpu_result = Zstd::decompress(&zstd_array, cuda_ctx.execution_ctx())?.to_canonical()?; + let cpu_result = Zstd::decompress(&zstd_array, cuda_ctx.execution_ctx())? + .execute::(cuda_ctx.execution_ctx())?; let gpu_result = ZstdExecutor .execute(zstd_array.into_array(), &mut cuda_ctx) .await?; @@ -408,9 +413,10 @@ mod tests { // Compress with ZSTD using values_per_frame=3 to create multiple frames. // 14 strings and 3 values per frame = ceil(14/3) = 5 frames. - let zstd_array = Zstd::from_var_bin_view(&strings, 3, 3)?; + let zstd_array = Zstd::from_var_bin_view(&strings, 3, 3, cuda_ctx.execution_ctx())?; - let cpu_result = Zstd::decompress(&zstd_array, cuda_ctx.execution_ctx())?.to_canonical()?; + let cpu_result = Zstd::decompress(&zstd_array, cuda_ctx.execution_ctx())? + .execute::(cuda_ctx.execution_ctx())?; let gpu_result = ZstdExecutor .execute(zstd_array.into_array(), &mut cuda_ctx) .await?; @@ -437,12 +443,12 @@ mod tests { "final test string", ]); - let zstd_array = Zstd::from_var_bin_view(&strings, 3, 0)?; + let zstd_array = Zstd::from_var_bin_view(&strings, 3, 0, cuda_ctx.execution_ctx())?; // Slice the array to get a subset (indices 2..7) let sliced_zstd = zstd_array.slice(2..7)?; - let cpu_result = sliced_zstd.to_canonical()?; + let cpu_result = crate::canonicalize_cpu(sliced_zstd.clone())?; let gpu_result = ZstdExecutor .execute(sliced_zstd.clone(), &mut cuda_ctx) .await?; @@ -450,4 +456,38 @@ mod tests { assert_arrays_eq!(cpu_result.into_array(), gpu_result.into_array()); Ok(()) } + + /// Zstd with nullable data — the GPU kernel does not yet support nulls, + /// so `execute_cuda` should gracefully fall back to CPU and produce + /// correct results instead of panicking. + #[crate::test] + async fn test_cuda_zstd_nullable_falls_back_to_cpu() -> VortexResult<()> { + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()) + .vortex_expect("failed to create execution context"); + + let strings = VarBinViewArray::from_iter_nullable_str([ + Some("hello"), + None, + Some("world"), + None, + Some("testing nullable zstd"), + Some("another string"), + ]); + + let zstd_array = Zstd::from_var_bin_view(&strings, 3, 0, cuda_ctx.execution_ctx())?; + + let cpu_result = crate::canonicalize_cpu(zstd_array.clone())?.into_array(); + + // execute_cuda should fall back to CPU and still produce the correct result. + let gpu_result = zstd_array + .into_array() + .execute_cuda(&mut cuda_ctx) + .await? + .into_host() + .await? + .into_array(); + + assert_arrays_eq!(cpu_result, gpu_result); + Ok(()) + } } diff --git a/vortex-cuda/src/kernel/encodings/zstd_buffers.rs b/vortex-cuda/src/kernel/encodings/zstd_buffers.rs index c93cf8aaec9..8a67a338ca5 100644 --- a/vortex-cuda/src/kernel/encodings/zstd_buffers.rs +++ b/vortex-cuda/src/kernel/encodings/zstd_buffers.rs @@ -18,6 +18,7 @@ use vortex::buffer::Alignment; use vortex::buffer::Buffer; use vortex::encodings::zstd::ZstdBuffers; use vortex::encodings::zstd::ZstdBuffersArray; +use vortex::encodings::zstd::ZstdBuffersDecodePlan; use vortex::error::VortexResult; use vortex::error::vortex_err; use vortex_nvcomp::sys; @@ -175,7 +176,7 @@ async fn move_frames_to_device( // This performs D2H to retrieve the lengths and status arrays. async fn validate_decompress_results( - plan: &vortex::encodings::zstd::ZstdBuffersDecodePlan, + plan: &ZstdBuffersDecodePlan, device_actual_sizes: CudaSlice, device_statuses: CudaSlice, ) -> VortexResult<()> { @@ -239,7 +240,7 @@ mod tests { let input = PrimitiveArray::from_iter(0i64..1024).into_array(); let compressed = ZstdBuffers::compress(&input, 3, &VortexSession::empty())?; - let cpu_result = compressed.clone().into_array().to_canonical()?; + let cpu_result = crate::canonicalize_cpu(compressed.clone())?; let gpu_result = ZstdBuffersExecutor .execute(compressed.into_array(), &mut cuda_ctx) .await? @@ -266,7 +267,7 @@ mod tests { .into_array(); let compressed = ZstdBuffers::compress(&input, 3, &VortexSession::empty())?; - let cpu_result = compressed.clone().into_array().to_canonical()?; + let cpu_result = crate::canonicalize_cpu(compressed.clone())?; let gpu_result = ZstdBuffersExecutor .execute(compressed.into_array(), &mut cuda_ctx) .await? diff --git a/vortex-cuda/src/kernel/filter/decimal.rs b/vortex-cuda/src/kernel/filter/decimal.rs index cdfc4342fa6..566b1c3313d 100644 --- a/vortex-cuda/src/kernel/filter/decimal.rs +++ b/vortex-cuda/src/kernel/filter/decimal.rs @@ -94,7 +94,7 @@ mod tests { let filter_array = FilterArray::try_new(input.clone().into_array(), mask.clone())?; - let cpu_result = filter_array.to_canonical()?.into_array(); + let cpu_result = crate::canonicalize_cpu(filter_array.clone())?.into_array(); let gpu_result = FilterExecutor .execute(filter_array.into_array(), &mut cuda_ctx) @@ -123,7 +123,7 @@ mod tests { let filter_array = FilterArray::try_new(input.into_array(), mask)?; - let cpu_result = filter_array.to_canonical()?.into_array(); + let cpu_result = crate::canonicalize_cpu(filter_array.clone())?.into_array(); let gpu_result = FilterExecutor .execute(filter_array.into_array(), &mut cuda_ctx) diff --git a/vortex-cuda/src/kernel/filter/primitive.rs b/vortex-cuda/src/kernel/filter/primitive.rs index c74c56e5b06..6bff0b20c70 100644 --- a/vortex-cuda/src/kernel/filter/primitive.rs +++ b/vortex-cuda/src/kernel/filter/primitive.rs @@ -88,7 +88,7 @@ mod tests { let filter_array = FilterArray::try_new(input.clone().into_array(), mask.clone())?; - let cpu_result = filter_array.to_canonical()?.into_array(); + let cpu_result = crate::canonicalize_cpu(filter_array.clone())?.into_array(); let gpu_result = FilterExecutor .execute(filter_array.into_array(), &mut cuda_ctx) @@ -117,7 +117,7 @@ mod tests { let filter_array = FilterArray::try_new(input.into_array(), mask)?; - let cpu_result = filter_array.to_canonical()?.into_array(); + let cpu_result = crate::canonicalize_cpu(filter_array.clone())?.into_array(); let gpu_result = FilterExecutor .execute(filter_array.into_array(), &mut cuda_ctx) diff --git a/vortex-cuda/src/kernel/filter/varbinview.rs b/vortex-cuda/src/kernel/filter/varbinview.rs index 84632f5cca8..23a6c91b048 100644 --- a/vortex-cuda/src/kernel/filter/varbinview.rs +++ b/vortex-cuda/src/kernel/filter/varbinview.rs @@ -74,7 +74,7 @@ mod tests { let filter_array = FilterArray::try_new(input.into_array(), mask.clone())?; - let cpu_result = filter_array.to_canonical()?.into_array(); + let cpu_result = crate::canonicalize_cpu(filter_array.clone())?.into_array(); let gpu_result = FilterExecutor .execute(filter_array.into_array(), &mut cuda_ctx) diff --git a/vortex-cuda/src/kernel/mod.rs b/vortex-cuda/src/kernel/mod.rs index 93ffd768df5..5665bc6c3f3 100644 --- a/vortex-cuda/src/kernel/mod.rs +++ b/vortex-cuda/src/kernel/mod.rs @@ -34,7 +34,7 @@ pub use encodings::ZstdKernelPrep; pub use encodings::zstd_kernel_prepare; pub(crate) use encodings::*; pub(crate) use filter::FilterExecutor; -pub use patches::types::transpose_patches; +pub(crate) use patches::types::load_patches_to_gpu; pub(crate) use slice::SliceExecutor; use crate::CudaKernelEvents; diff --git a/vortex-cuda/src/kernel/patches/mod.rs b/vortex-cuda/src/kernel/patches/mod.rs index 206ac8bc1cd..90ec3fc9c99 100644 --- a/vortex-cuda/src/kernel/patches/mod.rs +++ b/vortex-cuda/src/kernel/patches/mod.rs @@ -4,7 +4,7 @@ pub mod types; #[rustfmt::skip] -#[allow(warnings, clippy::all, clippy::pedantic, clippy::nursery)] +#[expect(warnings, clippy::all, clippy::pedantic, clippy::nursery)] pub mod gpu { include!(concat!(env!("OUT_DIR"), "/patches.rs")); } @@ -16,15 +16,81 @@ use vortex::array::arrays::primitive::PrimitiveDataParts; use vortex::array::patches::Patches; use vortex::array::validity::Validity; use vortex::dtype::NativePType; +use vortex::dtype::PType; use vortex::error::VortexResult; +use vortex::error::vortex_bail; use vortex::error::vortex_ensure; use crate::CudaBufferExt; use crate::CudaDeviceBuffer; use crate::CudaExecutionCtx; use crate::executor::CudaArrayExt; +use crate::kernel::patches::gpu::ChunkOffsetType; +use crate::kernel::patches::gpu::ChunkOffsetType_CO_U8; +use crate::kernel::patches::gpu::ChunkOffsetType_CO_U16; +use crate::kernel::patches::gpu::ChunkOffsetType_CO_U32; +use crate::kernel::patches::gpu::ChunkOffsetType_CO_U64; +use crate::kernel::patches::gpu::GPUPatches; +use crate::kernel::patches::types::DevicePatches; + +// Safe because `GPUPatches` contains only raw pointers, POD integers, and an enum. +unsafe impl DeviceRepr for GPUPatches {} + +impl GPUPatches { + /// Sentinel value passed to kernels when no patches are present. A NULL + /// `chunk_offsets` pointer is the signal `PatchesCursor` checks for. + pub(crate) const NULL_PATCHES: Self = Self { + chunk_offsets: std::ptr::null_mut(), + chunk_offset_type: ChunkOffsetType_CO_U32, + indices: std::ptr::null_mut(), + values: std::ptr::null_mut(), + offset: 0, + offset_within_chunk: 0, + num_patches: 0, + n_chunks: 0, + }; +} + +/// Convert a [`PType`] to the corresponding [`ChunkOffsetType`] for GPU patches. +fn ptype_to_chunk_offset_type(ptype: PType) -> VortexResult { + match ptype { + PType::U8 => Ok(ChunkOffsetType_CO_U8), + PType::U16 => Ok(ChunkOffsetType_CO_U16), + PType::U32 => Ok(ChunkOffsetType_CO_U32), + PType::U64 => Ok(ChunkOffsetType_CO_U64), + _ => vortex_bail!("Invalid PType for chunk_offsets: {:?}", ptype), + } +} + +/// Build a [`GPUPatches`] kernel argument from optional device-resident patches. +/// +/// When `device_patches` is `None`, returns a sentinel value whose NULL +/// `chunk_offsets` signals "no patches" to the kernel. +pub(crate) fn build_gpu_patches( + device_patches: Option<&DevicePatches>, +) -> VortexResult { + #[expect(clippy::cast_possible_truncation)] + match device_patches { + Some(p) => Ok(GPUPatches { + chunk_offsets: p.chunk_offsets.cuda_device_ptr()? as _, + chunk_offset_type: ptype_to_chunk_offset_type(p.chunk_offset_ptype)?, + indices: p.indices.cuda_device_ptr()? as _, + values: p.values.cuda_device_ptr()? as _, + offset: p.offset as u32, + offset_within_chunk: p.offset_within_chunk as u32, + num_patches: p.num_patches as u32, + n_chunks: p.n_chunks as u32, + }), + None => Ok(GPUPatches::NULL_PATCHES), + } +} /// Apply a set of patches in-place onto a [`CudaDeviceBuffer`] holding `ValuesT`. +/// +/// Naive scatter kernel. Kept as a reusable fallback for encoders that cannot +/// use the chunk-based fused patching path (e.g., where `chunk_offsets` are +/// unavailable); no production caller uses it today. +#[allow(dead_code)] #[instrument(skip_all)] pub(crate) async fn execute_patches< ValuesT: NativePType + DeviceRepr, @@ -101,8 +167,9 @@ mod tests { use std::sync::Arc; use cudarc::driver::DeviceRepr; + use vortex::array::ExecutionCtx; use vortex::array::IntoArray; - use vortex::array::ToCanonical; + use vortex::array::LEGACY_SESSION; use vortex::array::VortexSessionExecute; use vortex::array::arrays::PrimitiveArray; use vortex::array::arrays::primitive::PrimitiveDataParts; @@ -144,35 +211,30 @@ mod tests { } async fn full_test_case() { - let mut ctx = CudaSession::create_execution_ctx(&VortexSession::empty()).unwrap(); + let mut cuda_ctx = CudaSession::create_execution_ctx(&VortexSession::empty()).unwrap(); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values = PrimitiveArray::from_iter(0..128); - let values = force_cast::(values); + let values = force_cast::(values, &mut ctx); let patch_idx = PrimitiveArray::new(buffer![0, 8, 16, 32], Validity::NonNullable); - let patch_idx = force_cast::(patch_idx); + let patch_idx = force_cast::(patch_idx, &mut ctx); let patch_val = PrimitiveArray::new(buffer![99, 99, 99, 99], Validity::NonNullable); - let patch_val = force_cast::(patch_val); + let patch_val = force_cast::(patch_val, &mut ctx); // Copy all to GPU let patches = Patches::new(128, 0, patch_idx.into_array(), patch_val.into_array(), None).unwrap(); - let cpu_result = values - .clone() - .patch( - &patches, - &mut vortex::array::LEGACY_SESSION.create_execution_ctx(), - ) - .unwrap(); + let cpu_result = values.clone().patch(&patches, &mut ctx).unwrap(); let PrimitiveDataParts { buffer: cuda_buffer, .. } = values.into_data_parts(); - let handle = ctx.ensure_on_device(cuda_buffer).await.unwrap(); + let handle = cuda_ctx.ensure_on_device(cuda_buffer).await.unwrap(); let device_buf = handle .as_device() .as_any() @@ -180,7 +242,7 @@ mod tests { .unwrap() .clone(); - let patched_buf = execute_patches::(patches, device_buf, &mut ctx) + let patched_buf = execute_patches::(patches, device_buf, &mut cuda_ctx) .await .unwrap(); @@ -189,7 +251,8 @@ mod tests { Values::PTYPE, Validity::NonNullable, ) - .to_canonical() + .into_array() + .execute::(&mut ctx) .unwrap() .into_host() .await @@ -199,11 +262,12 @@ mod tests { assert_arrays_eq!(cpu_result, gpu_result); } - fn force_cast(array: PrimitiveArray) -> PrimitiveArray { + fn force_cast(array: PrimitiveArray, ctx: &mut ExecutionCtx) -> PrimitiveArray { array .into_array() .cast(DType::Primitive(T::PTYPE, Nullability::NonNullable)) .unwrap() - .to_primitive() + .execute::(ctx) + .unwrap() } } diff --git a/vortex-cuda/src/kernel/patches/types.rs b/vortex-cuda/src/kernel/patches/types.rs index 78fa0b8429e..f16d2b06488 100644 --- a/vortex-cuda/src/kernel/patches/types.rs +++ b/vortex-cuda/src/kernel/patches/types.rs @@ -1,354 +1,217 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -//! An implementation of lane-wise patches instead of linear patches. This layout for exception -//! patching enables fully parallel GPU execution, as outlined by Hepkema et al. in -//! "G-ALP: Rethinking Light-weight Encodings for GPUs" +//! GPU patches loading for fused exception patching during bit-unpacking. -use vortex::array::Canonical; +use std::mem::size_of; + +use num_traits::ToPrimitive; use vortex::array::buffer::BufferHandle; -use vortex::array::dtype::IntegerPType; -use vortex::array::dtype::NativePType; +use vortex::buffer::Alignment; use vortex::buffer::Buffer; use vortex::buffer::BufferMut; -use vortex_array::match_each_native_ptype; +use vortex::buffer::ByteBufferMut; +use vortex::dtype::PType; use vortex_array::match_each_unsigned_integer_ptype; use vortex_array::patches::Patches; use vortex_error::VortexResult; +use vortex_error::vortex_bail; +use crate::CudaBufferExt; use crate::CudaExecutionCtx; - -/// A set of device-resident patches that live in the GPU. -/// -/// These are dynamically typed. -#[repr(C)] +use crate::executor::CudaArrayExt; +use crate::kernel::patches::gpu::ChunkOffsetType; +use crate::kernel::patches::gpu::ChunkOffsetType_CO_U8; +use crate::kernel::patches::gpu::ChunkOffsetType_CO_U16; +use crate::kernel::patches::gpu::ChunkOffsetType_CO_U32; +use crate::kernel::patches::gpu::ChunkOffsetType_CO_U64; +use crate::kernel::patches::gpu::GPUPatches; + +/// A set of device-resident patches. pub struct DevicePatches { - pub(crate) lane_offsets: BufferHandle, + pub(crate) chunk_offsets: BufferHandle, + pub(crate) chunk_offset_ptype: PType, pub(crate) indices: BufferHandle, pub(crate) values: BufferHandle, + pub(crate) offset: usize, + pub(crate) offset_within_chunk: usize, + pub(crate) num_patches: usize, + pub(crate) n_chunks: usize, } -/// Number of lanes used at patch time for a value of type `V`. +/// Load patches for GPU use (async). /// -/// This is *NOT* equal to the number of FastLanes lanes for the type `V`, rather this will -/// correspond with the number of CUDA threads dedicated to executing each 1024-element vector. -const fn patch_lanes() -> usize { - // For types 32-bits or smaller, we use a 32 lane configuration, and for 64-bit we use 16 lanes. - // This matches up with the number of lanes we use to execute copying results from bit-unpacking - // from shared to global memory. - if size_of::() < 8 { 32 } else { 16 } -} - -/// A set of patches of values `V` existing in host buffers. -#[allow(dead_code)] -pub struct HostPatches { - n_chunks: usize, - n_lanes: usize, - lane_offsets: Buffer, - indices: Buffer, - /// Values. This is a buffer handle which might live on the new buffer type here - values: Buffer, -} - -#[cfg(test)] -struct LanePatches<'a, V> { - indices: &'a [u16], - values: &'a [V], -} - -impl HostPatches { - /// Get number of patches for a specific lane. - #[cfg(test)] - fn patch_count(&self, chunk: usize, lane: usize) -> usize { - let start = chunk * self.n_lanes + lane; - let end = start + 1; - let count = self.lane_offsets[end] - self.lane_offsets[start]; - - count as usize - } - - /// Get an ordered list of patches for the given chunk/lane. - #[cfg(test)] - fn patches(&self, chunk: usize, lane: usize) -> LanePatches<'_, V> { - let start = chunk * self.n_lanes + lane; - let end = start + 1; - - let lane_start = self.lane_offsets[start] as usize; - let lane_stop = self.lane_offsets[end] as usize; - - LanePatches { - indices: &self.indices[lane_start..lane_stop], - values: &self.values[lane_start..lane_stop], - } - } - - /// Apply the patches on top of the other buffer. - #[cfg(test)] - fn apply(&self, output: &mut BufferMut) { - for chunk in 0..self.n_chunks { - for lane in 0..self.n_lanes { - let patches = self.patches(chunk, lane); - for (&index, &value) in std::iter::zip(patches.indices, patches.values) { - let full_index = chunk * 1024 + (index as usize); - output[full_index] = value; - } - } - } - } - - /// Export the patches for use on the device associated with the provided execution context. - pub async fn export_to_device( - mut self, - ctx: &mut CudaExecutionCtx, - ) -> VortexResult { - let lane_offsets = std::mem::take(&mut self.lane_offsets); - let indices = std::mem::take(&mut self.indices); - let values = std::mem::take(&mut self.values); - - // Convert each into a handle that can be passed around. - let lane_offsets_handle = BufferHandle::new_host(lane_offsets.into_byte_buffer()); - let indices_handle = BufferHandle::new_host(indices.into_byte_buffer()); - let values_handle = BufferHandle::new_host(values.into_byte_buffer()); - - let lane_offsets_handle = ctx.ensure_on_device(lane_offsets_handle).await?; - let indices_handle = ctx.ensure_on_device(indices_handle).await?; - let values_handle = ctx.ensure_on_device(values_handle).await?; - - Ok(DevicePatches { - lane_offsets: lane_offsets_handle, - indices: indices_handle, - values: values_handle, - }) - } -} - -/// Transpose a set of patches from the default sorted layout into the data parallel layout. -#[allow(clippy::cognitive_complexity)] -pub async fn transpose_patches( +/// # Errors +/// +/// If the patches do not have `chunk_offsets`. They have been written by +/// default in new Vortex files since 0.54.0. +pub(crate) async fn load_patches( patches: &Patches, ctx: &mut CudaExecutionCtx, ) -> VortexResult { - let array_len = patches.array_len(); let offset = patches.offset(); - + let offset_within_chunk = patches.offset_within_chunk().unwrap_or_default(); + // Get or compute chunk_offsets + let Some(co) = patches.chunk_offsets() else { + vortex_bail!("cannot execute_cuda for patched BitPacked array without chunk_offsets") + }; + + let (chunk_offsets, chunk_offset_ptype, n_chunks) = { + let co_canonical = co.clone().execute_cuda(ctx).await?.into_primitive(); + let ptype = co_canonical.ptype(); + let len = co_canonical.len(); + (co_canonical.buffer_handle().clone(), ptype, len) + }; + + // Load indices - must be converted to u32 for GPU use let indices = patches .indices() .clone() - .execute::(ctx.execution_ctx())? + .execute_cuda(ctx) + .await? .into_primitive(); + let indices_ptype = indices.ptype(); + #[expect(clippy::expect_used)] + let indices = if indices_ptype == PType::U32 { + indices.buffer_handle().clone() + } else { + // Convert indices to u32 + let indices_buf = indices.buffer_handle().to_host().await; + let indices_u32 = match_each_unsigned_integer_ptype!(indices_ptype, |I| { + let src: Buffer = Buffer::from_byte_buffer(indices_buf); + let mut dst: BufferMut = BufferMut::with_capacity(src.len()); + for &idx in src.as_slice() { + // Indices are limited to u32 range for GPU + dst.push(idx.to_u32().expect("index should fit in u32")); + } + dst.freeze() + }); + BufferHandle::new_host(indices_u32.into_byte_buffer()) + }; + // Load values let values = patches .values() .clone() - .execute::(ctx.execution_ctx())? + .execute_cuda(ctx) + .await? .into_primitive(); - let indices_ptype = indices.ptype(); - let values_ptype = values.ptype(); - - let indices = indices.buffer_handle().to_host().await; - let values = values.buffer_handle().to_host().await; - - match_each_unsigned_integer_ptype!(indices_ptype, |I| { - match_each_native_ptype!(values_ptype, |V| { - let indices: Buffer = Buffer::from_byte_buffer(indices); - let values: Buffer = Buffer::from_byte_buffer(values); - - let host_patches = transpose(indices.as_slice(), values.as_slice(), offset, array_len); - - host_patches.export_to_device(ctx).await - }) + // Ensure all on device + let chunk_offsets = ctx.ensure_on_device(chunk_offsets).await?; + let indices = ctx.ensure_on_device(indices).await?; + let values = ctx.ensure_on_device(values.buffer_handle().clone()).await?; + + let num_patches = patches.num_patches(); + + Ok(DevicePatches { + chunk_offsets, + chunk_offset_ptype, + indices, + values, + offset, + offset_within_chunk, + num_patches, + n_chunks, }) } -#[allow(clippy::cast_possible_truncation)] -fn transpose( - indices_in: &[I], - values_in: &[V], - offset: usize, - array_len: usize, -) -> HostPatches { - // Total number of slots is number of chunks times number of lanes. - let n_chunks = array_len.div_ceil(1024); - assert!( - n_chunks <= u32::MAX as usize, - "Cannot transpose patches for array with >= 4 trillion elements" - ); - - let n_lanes = patch_lanes::(); - - // We know upfront how many indices and values we'll have. - let mut indices_buffer = BufferMut::with_capacity(indices_in.len()); - let mut values_buffer = BufferMut::with_capacity(values_in.len()); - - // number of patches in each chunk. - let mut lane_offsets: BufferMut = BufferMut::zeroed(n_chunks * n_lanes + 1); - - // Scan the index/values once to get chunk/lane counts - for index in indices_in { - let index = index.as_() - offset; - let chunk = index / 1024; - let lane = index % n_lanes; - - lane_offsets[chunk * n_lanes + lane + 1] += 1; - } - - // Prefix-sum sizes -> offsets - for index in 1..lane_offsets.len() { - lane_offsets[index] += lane_offsets[index - 1]; - } - - // Loop over patches, writing them to final positions - let indices_out = indices_buffer.spare_capacity_mut(); - let values_out = values_buffer.spare_capacity_mut(); - for (index, &value) in std::iter::zip(indices_in, values_in) { - let index = index.as_() - offset; - let chunk = index / 1024; - let lane = index % n_lanes; - - let position = &mut lane_offsets[chunk * n_lanes + lane]; - indices_out[*position as usize].write((index % 1024) as u16); - values_out[*position as usize].write(value); - *position += 1; +/// Convert a PType to the corresponding `ChunkOffsetType` for GPU patches. +pub(crate) fn ptype_to_chunk_offset_type(ptype: PType) -> VortexResult { + match ptype { + PType::U8 => Ok(ChunkOffsetType_CO_U8), + PType::U16 => Ok(ChunkOffsetType_CO_U16), + PType::U32 => Ok(ChunkOffsetType_CO_U32), + PType::U64 => Ok(ChunkOffsetType_CO_U64), + _ => vortex_bail!("Invalid PType for chunk_offsets: {:?}", ptype), } +} - // SAFETY: we know there are exactly indices_in.len() indices/values, and we just - // set them to the appropriate values in the loop above. - unsafe { - indices_buffer.set_len(indices_in.len()); - values_buffer.set_len(values_in.len()); +/// Build a [`GPUPatches`] struct from [`DevicePatches`], serialize it to +/// bytes, and upload to the device. Returns the device pointer and a buffer +/// handle that must be kept alive for the kernel launch. +fn build_gpu_patches( + dp: &DevicePatches, + ctx: &CudaExecutionCtx, +) -> VortexResult<(BufferHandle, u64)> { + // Zero-initialize to avoid uninitialized padding bytes (e.g. between + // chunk_offset_type and indices) which would be UB when serialized. + let mut gpu_patches: GPUPatches = unsafe { std::mem::zeroed() }; + gpu_patches.chunk_offsets = dp.chunk_offsets.cuda_device_ptr()? as _; + gpu_patches.chunk_offset_type = ptype_to_chunk_offset_type(dp.chunk_offset_ptype)?; + gpu_patches.indices = dp.indices.cuda_device_ptr()? as _; + gpu_patches.values = dp.values.cuda_device_ptr()? as _; + #[expect(clippy::cast_possible_truncation)] + { + gpu_patches.offset = dp.offset as u32; + gpu_patches.offset_within_chunk = dp.offset_within_chunk as u32; + gpu_patches.num_patches = dp.num_patches as u32; + // n_chunks must match the chunk_offsets array length, not array_len / 1024. + // When patches are sliced, chunk_offsets is sliced to only include chunks + // overlapping the slice range — matching the CPU's patch_chunk which uses + // chunk_offsets_slice.len(). + gpu_patches.n_chunks = dp.n_chunks as u32; } - // Now, pass over all the indices and values again and subtract out the position increments. - for index in indices_in { - let index = index.as_() - offset; - let chunk = index / 1024; - let lane = index % n_lanes; - - lane_offsets[chunk * n_lanes + lane] -= 1; - } + let bytes = unsafe { + std::slice::from_raw_parts( + std::ptr::from_ref(&gpu_patches).cast::(), + size_of::(), + ) + }; + let mut buf = + ByteBufferMut::with_capacity_aligned(size_of::(), Alignment::of::()); + buf.extend_from_slice(bytes); + let gpu_buf = ctx.ensure_on_device_sync(BufferHandle::new_host(buf.freeze()))?; + let ptr = gpu_buf.cuda_device_ptr()?; + Ok((gpu_buf, ptr)) +} - HostPatches { - n_chunks, - n_lanes, - lane_offsets: lane_offsets.freeze(), - indices: indices_buffer.freeze(), - values: values_buffer.freeze(), - } +/// Transfers patches to the GPU and builds a [`GPUPatches`] descriptor. +/// +/// Returns the device pointer and all buffer handles that must be kept +/// alive for the kernel launch. +pub(crate) async fn load_patches_to_gpu( + patches: &Patches, + ctx: &mut CudaExecutionCtx, +) -> VortexResult<(u64, Vec)> { + let device_patches = load_patches(patches, ctx).await?; + let (gpu_buf, ptr) = build_gpu_patches(&device_patches, ctx)?; + let DevicePatches { + chunk_offsets, + indices, + values, + .. + } = device_patches; + Ok((ptr, vec![chunk_offsets, indices, values, gpu_buf])) } #[cfg(test)] mod tests { - use vortex::array::ExecutionCtx; - use vortex::buffer::BufferMut; - use vortex::buffer::buffer; - use vortex::buffer::buffer_mut; use vortex_array::IntoArray; - use vortex_array::LEGACY_SESSION; use vortex_array::arrays::PrimitiveArray; - use vortex_array::assert_arrays_eq; - use vortex_array::dtype::NativePType; use vortex_array::patches::Patches; use vortex_error::VortexResult; - use crate::kernel::patches::types::transpose; - - #[crate::test] - fn test_transpose_patches() { - let patch_values = buffer![0u32, 10, 20, 30, 40, 50, 60, 70, 80]; - - let mut patch_indices = BufferMut::empty(); - // CHUNK 0. patch_values have value type i32, which means there will be 32 lanes. - patch_indices.extend_from_slice(&[0, 31, 63, 64]); - - // CHUNK 1. - patch_indices.extend_from_slice(&[1024, 1056, 1058]); - - // CHUNK 2: empty - patch_indices.extend_from_slice(&[]); - - // CHUNK 3 - patch_indices.extend_from_slice(&[3073, 3076]); - - let patch_indices = patch_indices.freeze(); - - let transposed = transpose( - patch_indices.as_slice(), - patch_values.as_slice(), - 0, - 1024 * 5, - ); - - // Chunk 0 should have patches in lanes 0, 31 - assert_eq!(transposed.patches(0, 0).values, &[0, 30]); - assert_eq!(transposed.patches(0, 0).indices, &[0, 64]); - - assert_eq!(transposed.patches(0, 31).values, &[10, 20]); - assert_eq!(transposed.patches(0, 31).indices, &[31, 63]); - - // Chunk 1 should have patches in lanes 0, 2 - assert_eq!(transposed.patches(1, 0).values, &[40, 50]); - assert_eq!(transposed.patches(1, 0).indices, &[0, 32]); - assert_eq!(transposed.patches(1, 2).values, &[60]); - assert_eq!(transposed.patches(1, 2).indices, &[34]); - - // Chunk 2 should be empty - for lane in 0..transposed.n_lanes { - assert_eq!(transposed.patch_count(2, lane), 0); - } - - // Chunk 3 contains patches at lanes 1, 4 - assert_eq!(transposed.patches(3, 1).values, &[70]); - assert_eq!(transposed.patches(3, 1).indices, &[1]); - assert_eq!(transposed.patches(3, 4).values, &[80]); - assert_eq!(transposed.patches(3, 4).indices, &[4]); - } - #[test] - #[allow(clippy::cast_possible_truncation)] - fn test_transpose_complex() -> VortexResult<()> { - test_case(1024, 0, &[0], &[0f32])?; - test_case(512, 512, &[512, 513, 514], &[10i8, 20, 30])?; - test_case(10_000, 100, &[500, 1_000, 1_001, 1_002], &[1i16, 2, 3, 4])?; - - for len in (1..4096).step_by(10) { - let offset = len / 2; - - let indices: Vec = (offset..len).map(|x| x as u32).collect(); - - test_case(len, offset, &indices, &indices)?; - } - - Ok(()) - } - - fn test_case( - len: usize, - offset: usize, - patch_indices: &[u32], - patch_values: &[V], - ) -> VortexResult<()> { - let mut data = buffer_mut![V::default(); len]; - let array = PrimitiveArray::from_iter(data.iter().copied()); + fn test_patches_with_chunk_offsets() -> VortexResult<()> { + // Test creating patches with pre-computed chunk_offsets + let indices = PrimitiveArray::from_iter([0u32, 500, 1024, 2000]); + let values = PrimitiveArray::from_iter([10u32, 20, 30, 40]); + let chunk_offsets = PrimitiveArray::from_iter([0u32, 2, 3, 4]); let patches = Patches::new( - len, - offset, - PrimitiveArray::from_iter(patch_indices.iter().copied()).into_array(), - PrimitiveArray::from_iter(patch_values.iter().copied()).into_array(), - None, + 3072, + 0, + indices.into_array(), + values.into_array(), + Some(chunk_offsets.into_array()), )?; - // Verify that the outputs match between Patches and transpose_patches(). - let mut ctx = ExecutionCtx::new(LEGACY_SESSION.clone()); - let patched = array.patch(&patches, &mut ctx)?.into_array(); - - let transposed = transpose(patch_indices, patch_values, offset, len); - transposed.apply(&mut data); - - let patched_transposed = data.freeze().into_array(); - - assert_arrays_eq!(patched, patched_transposed); + assert!(patches.chunk_offsets().is_some()); + assert_eq!(patches.chunk_offset_at(0)?, 0); + assert_eq!(patches.chunk_offset_at(1)?, 2); + assert_eq!(patches.chunk_offset_at(2)?, 3); Ok(()) } diff --git a/vortex-cuda/src/kernel/slice/mod.rs b/vortex-cuda/src/kernel/slice/mod.rs index 647faa64fcd..30b1fdaf640 100644 --- a/vortex-cuda/src/kernel/slice/mod.rs +++ b/vortex-cuda/src/kernel/slice/mod.rs @@ -37,20 +37,30 @@ impl CudaExecute for SliceExecutor { let child = slice_array.child().clone().execute_cuda(ctx).await?; match child { - Canonical::Null(null_array) => null_array.into_array().slice(range)?.to_canonical(), - Canonical::Bool(bool_array) => bool_array.into_array().slice(range)?.to_canonical(), - Canonical::Primitive(prim_array) => { - prim_array.into_array().slice(range)?.to_canonical() - } - Canonical::Decimal(decimal_array) => { - decimal_array.into_array().slice(range)?.to_canonical() - } - Canonical::VarBinView(varbinview) => { - varbinview.into_array().slice(range)?.to_canonical() - } - Canonical::Extension(extension_array) => { - extension_array.into_array().slice(range)?.to_canonical() - } + Canonical::Null(null_array) => null_array + .into_array() + .slice(range)? + .execute::(ctx.execution_ctx()), + Canonical::Bool(bool_array) => bool_array + .into_array() + .slice(range)? + .execute::(ctx.execution_ctx()), + Canonical::Primitive(prim_array) => prim_array + .into_array() + .slice(range)? + .execute::(ctx.execution_ctx()), + Canonical::Decimal(decimal_array) => decimal_array + .into_array() + .slice(range)? + .execute::(ctx.execution_ctx()), + Canonical::VarBinView(varbinview) => varbinview + .into_array() + .slice(range)? + .execute::(ctx.execution_ctx()), + Canonical::Extension(extension_array) => extension_array + .into_array() + .slice(range)? + .execute::(ctx.execution_ctx()), c => todo!("Slice kernel not implemented for {}", c.dtype()), } } diff --git a/vortex-cuda/src/layout.rs b/vortex-cuda/src/layout.rs index 516968b8f82..3bec31ba163 100644 --- a/vortex-cuda/src/layout.rs +++ b/vortex-cuda/src/layout.rs @@ -3,6 +3,7 @@ //! A CUDA-optimized flat layout that inlines small constant array buffers into layout metadata. +use std::any::Any; use std::collections::BTreeSet; use std::ops::BitAnd; use std::ops::Range; @@ -16,6 +17,7 @@ use futures::future::BoxFuture; use vortex::array::ArrayContext; use vortex::array::ArrayId; use vortex::array::ArrayRef; +use vortex::array::ArrayVTable; use vortex::array::DeserializeMetadata; use vortex::array::MaskFuture; use vortex::array::ProstMetadata; @@ -127,7 +129,7 @@ impl VTable for CudaFlat { type Metadata = ProstMetadata; fn id(_encoding: &Self::Encoding) -> LayoutId { - LayoutId::new_ref("vortex.cuda_flat") + LayoutId::new("vortex.cuda_flat") } fn encoding(_layout: &Self::Layout) -> LayoutEncodingRef { @@ -380,6 +382,10 @@ impl LayoutReader for CudaFlatReader { } .boxed()) } + + fn as_any(&self) -> &dyn Any { + self + } } /// A [`LayoutStrategy`] that writes a [`CudaFlatLayout`] with constant array buffers inlined @@ -551,7 +557,7 @@ fn extract_constant_buffers(chunk: &ArrayRef) -> Vec { let mut buffer_idx = 0u32; for array in chunk.depth_first_traversal() { let n = array.nbuffers(); - if array.encoding_id() == Constant::ID { + if array.encoding_id() == Constant.id() { for buf in array.buffers() { result.push(InlinedBuffer { buffer_index: buffer_idx, diff --git a/vortex-cuda/src/lib.rs b/vortex-cuda/src/lib.rs index 274fe31b18f..bbebef9430d 100644 --- a/vortex-cuda/src/lib.rs +++ b/vortex-cuda/src/lib.rs @@ -27,6 +27,7 @@ pub use canonical::CanonicalCudaExt; pub use device_buffer::CudaBufferExt; pub use device_buffer::CudaDeviceBuffer; pub use device_read_at::CopyDeviceReadAt; +pub use executor::CudaDispatchMode; pub use executor::CudaExecutionCtx; pub use executor::CudaKernelEvents; use kernel::ALPExecutor; @@ -47,7 +48,6 @@ use kernel::ZigZagExecutor; use kernel::ZstdBuffersExecutor; use kernel::ZstdExecutor; pub use kernel::ZstdKernelPrep; -pub use kernel::transpose_patches; pub use kernel::zstd_kernel_prepare; pub use pinned::PinnedByteBufferPool; pub use pinned::PinnedPoolStats; @@ -59,6 +59,7 @@ pub use session::CudaSession; pub use session::CudaSessionExt; pub use stream::VortexCudaStream; pub use stream_pool::VortexCudaStreamPool; +use vortex::array::ArrayVTable; use vortex::array::arrays::Constant; use vortex::array::arrays::Dict; use vortex::array::arrays::Filter; @@ -82,6 +83,18 @@ pub use vortex_nvcomp as nvcomp; use crate::kernel::SequenceExecutor; use crate::kernel::SliceExecutor; +#[cfg(test)] +pub(crate) fn canonicalize_cpu( + array: impl vortex::array::IntoArray, +) -> vortex::error::VortexResult { + use vortex::array::LEGACY_SESSION; + use vortex::array::VortexSessionExecute; + + array + .into_array() + .execute::(&mut LEGACY_SESSION.create_execution_ctx()) +} + /// Checks if CUDA is available on the system by looking for nvcc. pub fn cuda_available() -> bool { Command::new("nvcc") @@ -93,22 +106,22 @@ pub fn cuda_available() -> bool { /// Registers CUDA kernels. pub fn initialize_cuda(session: &CudaSession) { info!("Registering CUDA kernels"); - session.register_kernel(ALP::ID, &ALPExecutor); - session.register_kernel(BitPacked::ID, &BitPackedExecutor); - session.register_kernel(Constant::ID, &ConstantNumericExecutor); - session.register_kernel(DateTimeParts::ID, &DateTimePartsExecutor); - session.register_kernel(DecimalByteParts::ID, &DecimalBytePartsExecutor); - session.register_kernel(Dict::ID, &DictExecutor); - session.register_kernel(Shared::ID, &SharedExecutor); - session.register_kernel(FoR::ID, &FoRExecutor); - session.register_kernel(RunEnd::ID, &RunEndExecutor); - session.register_kernel(Sequence::ID, &SequenceExecutor); - session.register_kernel(ZigZag::ID, &ZigZagExecutor); - session.register_kernel(Zstd::ID, &ZstdExecutor); + session.register_kernel(ALP.id(), &ALPExecutor); + session.register_kernel(BitPacked.id(), &BitPackedExecutor); + session.register_kernel(Constant.id(), &ConstantNumericExecutor); + session.register_kernel(DateTimeParts.id(), &DateTimePartsExecutor); + session.register_kernel(DecimalByteParts.id(), &DecimalBytePartsExecutor); + session.register_kernel(Dict.id(), &DictExecutor); + session.register_kernel(Shared.id(), &SharedExecutor); + session.register_kernel(FoR.id(), &FoRExecutor); + session.register_kernel(RunEnd.id(), &RunEndExecutor); + session.register_kernel(Sequence.id(), &SequenceExecutor); + session.register_kernel(ZigZag.id(), &ZigZagExecutor); + session.register_kernel(Zstd.id(), &ZstdExecutor); #[cfg(feature = "unstable_encodings")] - session.register_kernel(ZstdBuffers::ID, &ZstdBuffersExecutor); + session.register_kernel(ZstdBuffers.id(), &ZstdBuffersExecutor); // Operation kernels - session.register_kernel(Filter::ID, &FilterExecutor); - session.register_kernel(Slice::ID, &SliceExecutor); + session.register_kernel(Filter.id(), &FilterExecutor); + session.register_kernel(Slice.id(), &SliceExecutor); } diff --git a/vortex-cuda/src/pinned.rs b/vortex-cuda/src/pinned.rs index 7939bc588ec..cd97db63164 100644 --- a/vortex-cuda/src/pinned.rs +++ b/vortex-cuda/src/pinned.rs @@ -2,6 +2,8 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors use std::sync::Arc; +use std::sync::atomic::AtomicU64; +use std::sync::atomic::Ordering; use cudarc::driver::CudaContext; use cudarc::driver::CudaEvent; @@ -27,7 +29,7 @@ pub(crate) struct PinnedByteBuffer { logical_len: usize, } -#[allow(clippy::same_name_method)] +#[expect(clippy::same_name_method)] impl PinnedByteBuffer { /// Allocate a pinned host buffer with a given capacity and logical length. /// @@ -70,7 +72,6 @@ impl PinnedByteBuffer { } } -#[allow(clippy::same_name_method)] impl HostSlice for PinnedByteBuffer { fn len(&self) -> usize { self.len() @@ -106,10 +107,10 @@ pub struct PinnedByteBufferPool { max_keep_per_size: usize, buckets: Mutex>>, inflight: Mutex>, - hits: std::sync::atomic::AtomicU64, - misses: std::sync::atomic::AtomicU64, - allocs: std::sync::atomic::AtomicU64, - puts: std::sync::atomic::AtomicU64, + hits: AtomicU64, + misses: AtomicU64, + allocs: AtomicU64, + puts: AtomicU64, } struct InflightPinnedBuffer { @@ -130,10 +131,10 @@ impl PinnedByteBufferPool { max_keep_per_size: max_keep_per_size.max(1), buckets: Mutex::new(HashMap::new()), inflight: Mutex::new(Vec::new()), - hits: std::sync::atomic::AtomicU64::new(0), - misses: std::sync::atomic::AtomicU64::new(0), - allocs: std::sync::atomic::AtomicU64::new(0), - puts: std::sync::atomic::AtomicU64::new(0), + hits: AtomicU64::new(0), + misses: AtomicU64::new(0), + allocs: AtomicU64::new(0), + puts: AtomicU64::new(0), } } @@ -170,10 +171,10 @@ impl PinnedByteBufferPool { /// Snapshot pool reuse statistics. pub fn stats(&self) -> PinnedPoolStats { PinnedPoolStats { - hits: self.hits.load(std::sync::atomic::Ordering::Relaxed), - misses: self.misses.load(std::sync::atomic::Ordering::Relaxed), - allocs: self.allocs.load(std::sync::atomic::Ordering::Relaxed), - puts: self.puts.load(std::sync::atomic::Ordering::Relaxed), + hits: self.hits.load(Ordering::Relaxed), + misses: self.misses.load(Ordering::Relaxed), + allocs: self.allocs.load(Ordering::Relaxed), + puts: self.puts.load(Ordering::Relaxed), } } @@ -212,16 +213,14 @@ impl PinnedByteBufferPool { if let Some(bucket) = buckets.get_mut(&key_len) && let Some(buf) = bucket.pop() { - self.hits.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + self.hits.fetch_add(1, Ordering::Relaxed); let mut buf = buf; buf.set_logical_len(len); return Ok(buf); } } - self.misses - .fetch_add(1, std::sync::atomic::Ordering::Relaxed); - self.allocs - .fetch_add(1, std::sync::atomic::Ordering::Relaxed); + self.misses.fetch_add(1, Ordering::Relaxed); + self.allocs.fetch_add(1, Ordering::Relaxed); unsafe { PinnedByteBuffer::uninit_with_capacity(&self.ctx, key_len, len) } } @@ -239,7 +238,7 @@ impl PinnedByteBufferPool { }; // If the pool is full, the buffer (cuMemFreeHost) is dropped outside the lock. drop(overflow); - self.puts.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + self.puts.fetch_add(1, Ordering::Relaxed); } fn try_get_inner(&self, len: usize) -> VortexResult> { @@ -249,7 +248,7 @@ impl PinnedByteBufferPool { if let Some(bucket) = buckets.get_mut(&key_len) && let Some(mut buf) = bucket.pop() { - self.hits.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + self.hits.fetch_add(1, Ordering::Relaxed); buf.set_logical_len(len); Ok(Some(buf)) } else { @@ -277,7 +276,6 @@ pub struct PooledPinnedBuffer { pool: Arc, } -#[allow(clippy::same_name_method)] impl PooledPinnedBuffer { /// Create a new pooled buffer. pub(crate) fn new(inner: PinnedByteBuffer, pool: Arc) -> Self { diff --git a/vortex-cuda/src/session.rs b/vortex-cuda/src/session.rs index e6bf4710dda..4ef1736021c 100644 --- a/vortex-cuda/src/session.rs +++ b/vortex-cuda/src/session.rs @@ -1,6 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +use std::any::Any; use std::fmt::Debug; use std::sync::Arc; @@ -10,6 +11,7 @@ use vortex::array::VortexSessionExecute; use vortex::error::VortexResult; use vortex::session::Ref; use vortex::session::SessionExt; +use vortex::session::SessionVar; use vortex::utils::aliases::dash_map::DashMap; use crate::ExportDeviceArray; @@ -85,8 +87,12 @@ impl CudaSession { /// /// * `array_id` - The encoding ID to register support for /// * `executor` - A static reference to the CUDA support implementation - pub fn register_kernel(&self, array_id: ArrayId, executor: &'static dyn CudaExecute) { - self.kernels.insert(array_id, executor); + pub fn register_kernel( + &self, + array_id: impl Into, + executor: &'static dyn CudaExecute, + ) { + self.kernels.insert(array_id.into(), executor); } /// Retrieves the CUDA support implementation for an encoding, if registered. @@ -143,6 +149,16 @@ impl Default for CudaSession { } } +impl SessionVar for CudaSession { + fn as_any(&self) -> &dyn Any { + self + } + + fn as_any_mut(&mut self) -> &mut dyn Any { + self + } +} + /// Extension trait for accessing the CUDA session from a Vortex session. pub trait CudaSessionExt: SessionExt { /// Returns the CUDA session. diff --git a/vortex-cxx/src/lib.rs b/vortex-cxx/src/lib.rs index 1cc63071ebc..50d345be648 100644 --- a/vortex-cxx/src/lib.rs +++ b/vortex-cxx/src/lib.rs @@ -2,6 +2,7 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors #![allow(clippy::boxed_local)] + mod dtype; mod expr; mod read; diff --git a/vortex-cxx/src/read.rs b/vortex-cxx/src/read.rs index 1ef4aec2734..ac8ccf03466 100644 --- a/vortex-cxx/src/read.rs +++ b/vortex-cxx/src/read.rs @@ -15,7 +15,9 @@ use arrow_schema::Schema; use arrow_schema::SchemaRef; use futures::stream::TryStreamExt; use vortex::array::ArrayRef; -use vortex::array::arrow::IntoArrowArray; +use vortex::array::LEGACY_SESSION; +use vortex::array::VortexSessionExecute; +use vortex::array::arrow::ArrowArrayExecutor; use vortex::buffer::Buffer; use vortex::file::OpenOptionsSessionExt; use vortex::io::runtime::BlockingRuntime; @@ -163,7 +165,7 @@ pub(crate) fn scan_builder_into_threadsafe_cloneable_reader( let stream = builder .inner .map(move |b| { - b.into_arrow(&data_type) + b.execute_arrow(Some(&data_type), &mut LEGACY_SESSION.create_execution_ctx()) .map(|struct_array| RecordBatch::from(struct_array.as_struct())) }) .into_stream()? diff --git a/vortex-datafusion/public-api.lock b/vortex-datafusion/public-api.lock index c51d60ac700..8d55e677395 100644 --- a/vortex-datafusion/public-api.lock +++ b/vortex-datafusion/public-api.lock @@ -6,7 +6,7 @@ pub struct vortex_datafusion::metrics::VortexMetricsFinder(_) impl vortex_datafusion::metrics::VortexMetricsFinder -pub fn vortex_datafusion::metrics::VortexMetricsFinder::find_all(plan: &dyn datafusion_physical_plan::execution_plan::ExecutionPlan) -> alloc::vec::Vec +pub fn vortex_datafusion::metrics::VortexMetricsFinder::find_all(&dyn datafusion_physical_plan::execution_plan::ExecutionPlan) -> alloc::vec::Vec impl core::default::Default for vortex_datafusion::metrics::VortexMetricsFinder @@ -16,7 +16,7 @@ impl datafusion_physical_plan::visitor::ExecutionPlanVisitor for vortex_datafusi pub type vortex_datafusion::metrics::VortexMetricsFinder::Error = core::convert::Infallible -pub fn vortex_datafusion::metrics::VortexMetricsFinder::pre_visit(&mut self, plan: &dyn datafusion_physical_plan::execution_plan::ExecutionPlan) -> core::result::Result +pub fn vortex_datafusion::metrics::VortexMetricsFinder::pre_visit(&mut self, &dyn datafusion_physical_plan::execution_plan::ExecutionPlan) -> core::result::Result pub mod vortex_datafusion::reader @@ -24,23 +24,23 @@ pub struct vortex_datafusion::reader::DefaultVortexReaderFactory impl vortex_datafusion::reader::DefaultVortexReaderFactory -pub fn vortex_datafusion::reader::DefaultVortexReaderFactory::new(object_store: alloc::sync::Arc) -> Self +pub fn vortex_datafusion::reader::DefaultVortexReaderFactory::new(alloc::sync::Arc) -> Self impl core::fmt::Debug for vortex_datafusion::reader::DefaultVortexReaderFactory -pub fn vortex_datafusion::reader::DefaultVortexReaderFactory::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_datafusion::reader::DefaultVortexReaderFactory::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_datafusion::reader::VortexReaderFactory for vortex_datafusion::reader::DefaultVortexReaderFactory -pub fn vortex_datafusion::reader::DefaultVortexReaderFactory::create_reader(&self, file: &datafusion_datasource::PartitionedFile, session: &vortex_session::VortexSession) -> datafusion_common::error::Result> +pub fn vortex_datafusion::reader::DefaultVortexReaderFactory::create_reader(&self, &datafusion_datasource::PartitionedFile, &vortex_session::VortexSession) -> datafusion_common::error::Result> pub trait vortex_datafusion::reader::VortexReaderFactory: core::fmt::Debug + core::marker::Send + core::marker::Sync + 'static -pub fn vortex_datafusion::reader::VortexReaderFactory::create_reader(&self, file: &datafusion_datasource::PartitionedFile, session: &vortex_session::VortexSession) -> datafusion_common::error::Result> +pub fn vortex_datafusion::reader::VortexReaderFactory::create_reader(&self, &datafusion_datasource::PartitionedFile, &vortex_session::VortexSession) -> datafusion_common::error::Result> impl vortex_datafusion::reader::VortexReaderFactory for vortex_datafusion::reader::DefaultVortexReaderFactory -pub fn vortex_datafusion::reader::DefaultVortexReaderFactory::create_reader(&self, file: &datafusion_datasource::PartitionedFile, session: &vortex_session::VortexSession) -> datafusion_common::error::Result> +pub fn vortex_datafusion::reader::DefaultVortexReaderFactory::create_reader(&self, &datafusion_datasource::PartitionedFile, &vortex_session::VortexSession) -> datafusion_common::error::Result> pub mod vortex_datafusion::v2 @@ -48,7 +48,7 @@ pub struct vortex_datafusion::v2::VortexDataSource impl vortex_datafusion::v2::VortexDataSource -pub fn vortex_datafusion::v2::VortexDataSource::builder(data_source: vortex_scan::DataSourceRef, session: vortex_session::VortexSession) -> vortex_datafusion::v2::source::VortexDataSourceBuilder +pub fn vortex_datafusion::v2::VortexDataSource::builder(vortex_scan::DataSourceRef, vortex_session::VortexSession) -> vortex_datafusion::v2::source::VortexDataSourceBuilder impl core::clone::Clone for vortex_datafusion::v2::VortexDataSource @@ -56,7 +56,7 @@ pub fn vortex_datafusion::v2::VortexDataSource::clone(&self) -> vortex_datafusio impl core::fmt::Debug for vortex_datafusion::v2::VortexDataSource -pub fn vortex_datafusion::v2::VortexDataSource::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_datafusion::v2::VortexDataSource::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl datafusion_datasource::source::DataSource for vortex_datafusion::v2::VortexDataSource @@ -66,37 +66,37 @@ pub fn vortex_datafusion::v2::VortexDataSource::eq_properties(&self) -> datafusi pub fn vortex_datafusion::v2::VortexDataSource::fetch(&self) -> core::option::Option -pub fn vortex_datafusion::v2::VortexDataSource::fmt_as(&self, _t: datafusion_physical_plan::display::DisplayFormatType, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_datafusion::v2::VortexDataSource::fmt_as(&self, datafusion_physical_plan::display::DisplayFormatType, &mut core::fmt::Formatter<'_>) -> core::fmt::Result -pub fn vortex_datafusion::v2::VortexDataSource::open(&self, partition: usize, _context: alloc::sync::Arc) -> datafusion_common::error::Result +pub fn vortex_datafusion::v2::VortexDataSource::open(&self, usize, alloc::sync::Arc) -> datafusion_common::error::Result pub fn vortex_datafusion::v2::VortexDataSource::output_partitioning(&self) -> datafusion_physical_expr::partitioning::Partitioning -pub fn vortex_datafusion::v2::VortexDataSource::partition_statistics(&self, _partition: core::option::Option) -> datafusion_common::error::Result +pub fn vortex_datafusion::v2::VortexDataSource::partition_statistics(&self, core::option::Option) -> datafusion_common::error::Result -pub fn vortex_datafusion::v2::VortexDataSource::repartitioned(&self, target_partitions: usize, _repartition_file_min_size: usize, output_ordering: core::option::Option) -> datafusion_common::error::Result>> +pub fn vortex_datafusion::v2::VortexDataSource::repartitioned(&self, usize, usize, core::option::Option) -> datafusion_common::error::Result>> -pub fn vortex_datafusion::v2::VortexDataSource::try_pushdown_filters(&self, filters: alloc::vec::Vec>, _config: &datafusion_common::config::ConfigOptions) -> datafusion_common::error::Result>> +pub fn vortex_datafusion::v2::VortexDataSource::try_pushdown_filters(&self, alloc::vec::Vec>, &datafusion_common::config::ConfigOptions) -> datafusion_common::error::Result>> -pub fn vortex_datafusion::v2::VortexDataSource::try_swapping_with_projection(&self, projection: &datafusion_physical_expr::projection::ProjectionExprs) -> datafusion_common::error::Result>> +pub fn vortex_datafusion::v2::VortexDataSource::try_swapping_with_projection(&self, &datafusion_physical_expr::projection::ProjectionExprs) -> datafusion_common::error::Result>> -pub fn vortex_datafusion::v2::VortexDataSource::with_fetch(&self, limit: core::option::Option) -> core::option::Option> +pub fn vortex_datafusion::v2::VortexDataSource::with_fetch(&self, core::option::Option) -> core::option::Option> pub struct vortex_datafusion::v2::VortexTable impl vortex_datafusion::v2::VortexTable -pub fn vortex_datafusion::v2::VortexTable::new(data_source: vortex_scan::DataSourceRef, session: vortex_session::VortexSession, arrow_schema: arrow_schema::schema::SchemaRef) -> Self +pub fn vortex_datafusion::v2::VortexTable::new(vortex_scan::DataSourceRef, vortex_session::VortexSession, arrow_schema::schema::SchemaRef) -> Self impl core::fmt::Debug for vortex_datafusion::v2::VortexTable -pub fn vortex_datafusion::v2::VortexTable::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_datafusion::v2::VortexTable::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl datafusion_catalog::table::TableProvider for vortex_datafusion::v2::VortexTable pub fn vortex_datafusion::v2::VortexTable::as_any(&self) -> &dyn core::any::Any -pub fn vortex_datafusion::v2::VortexTable::scan<'life0, 'life1, 'life2, 'life3, 'async_trait>(&'life0 self, _state: &'life1 dyn datafusion_session::session::Session, projection: core::option::Option<&'life2 alloc::vec::Vec>, _filters: &'life3 [datafusion_expr::expr::Expr], _limit: core::option::Option) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait +pub fn vortex_datafusion::v2::VortexTable::scan<'life0, 'life1, 'life2, 'life3, 'async_trait>(&'life0 self, &'life1 dyn datafusion_session::session::Session, core::option::Option<&'life2 alloc::vec::Vec>, &'life3 [datafusion_expr::expr::Expr], core::option::Option) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait pub fn vortex_datafusion::v2::VortexTable::schema(&self) -> arrow_schema::schema::SchemaRef @@ -112,25 +112,25 @@ pub fn vortex_datafusion::DefaultExpressionConvertor::default() -> vortex_datafu impl vortex_datafusion::ExpressionConvertor for vortex_datafusion::DefaultExpressionConvertor -pub fn vortex_datafusion::DefaultExpressionConvertor::can_be_pushed_down(&self, expr: &alloc::sync::Arc, schema: &arrow_schema::schema::Schema) -> bool +pub fn vortex_datafusion::DefaultExpressionConvertor::can_be_pushed_down(&self, &alloc::sync::Arc, &arrow_schema::schema::Schema) -> bool -pub fn vortex_datafusion::DefaultExpressionConvertor::convert(&self, df: &dyn datafusion_physical_expr_common::physical_expr::PhysicalExpr) -> datafusion_common::error::Result +pub fn vortex_datafusion::DefaultExpressionConvertor::convert(&self, &dyn datafusion_physical_expr_common::physical_expr::PhysicalExpr) -> datafusion_common::error::Result -pub fn vortex_datafusion::DefaultExpressionConvertor::no_pushdown_projection(&self, source_projection: datafusion_physical_expr::projection::ProjectionExprs, input_schema: &arrow_schema::schema::Schema) -> datafusion_common::error::Result +pub fn vortex_datafusion::DefaultExpressionConvertor::no_pushdown_projection(&self, datafusion_physical_expr::projection::ProjectionExprs, &arrow_schema::schema::Schema) -> datafusion_common::error::Result -pub fn vortex_datafusion::DefaultExpressionConvertor::split_projection(&self, source_projection: datafusion_physical_expr::projection::ProjectionExprs, input_schema: &arrow_schema::schema::Schema, output_schema: &arrow_schema::schema::Schema) -> datafusion_common::error::Result +pub fn vortex_datafusion::DefaultExpressionConvertor::split_projection(&self, datafusion_physical_expr::projection::ProjectionExprs, &arrow_schema::schema::Schema, &arrow_schema::schema::Schema) -> datafusion_common::error::Result pub struct vortex_datafusion::VortexAccessPlan impl vortex_datafusion::VortexAccessPlan -pub fn vortex_datafusion::VortexAccessPlan::apply_to_builder(&self, scan_builder: vortex_layout::scan::scan_builder::ScanBuilder) -> vortex_layout::scan::scan_builder::ScanBuilder where A: 'static + core::marker::Send +pub fn vortex_datafusion::VortexAccessPlan::apply_to_builder(&self, vortex_layout::scan::scan_builder::ScanBuilder) -> vortex_layout::scan::scan_builder::ScanBuilder where A: 'static + core::marker::Send pub fn vortex_datafusion::VortexAccessPlan::selection(&self) -> core::option::Option<&vortex_scan::selection::Selection> impl vortex_datafusion::VortexAccessPlan -pub fn vortex_datafusion::VortexAccessPlan::with_selection(self, selection: vortex_scan::selection::Selection) -> Self +pub fn vortex_datafusion::VortexAccessPlan::with_selection(self, vortex_scan::selection::Selection) -> Self impl core::default::Default for vortex_datafusion::VortexAccessPlan @@ -140,15 +140,15 @@ pub struct vortex_datafusion::VortexFormat impl vortex_datafusion::VortexFormat -pub fn vortex_datafusion::VortexFormat::new(session: vortex_session::VortexSession) -> Self +pub fn vortex_datafusion::VortexFormat::new(vortex_session::VortexSession) -> Self -pub fn vortex_datafusion::VortexFormat::new_with_options(session: vortex_session::VortexSession, opts: vortex_datafusion::VortexTableOptions) -> Self +pub fn vortex_datafusion::VortexFormat::new_with_options(vortex_session::VortexSession, vortex_datafusion::VortexTableOptions) -> Self pub fn vortex_datafusion::VortexFormat::options(&self) -> &vortex_datafusion::VortexTableOptions impl core::fmt::Debug for vortex_datafusion::VortexFormat -pub fn vortex_datafusion::VortexFormat::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_datafusion::VortexFormat::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl datafusion_datasource::file_format::FileFormat for vortex_datafusion::VortexFormat @@ -156,19 +156,19 @@ pub fn vortex_datafusion::VortexFormat::as_any(&self) -> &dyn core::any::Any pub fn vortex_datafusion::VortexFormat::compression_type(&self) -> core::option::Option -pub fn vortex_datafusion::VortexFormat::create_physical_plan<'life0, 'life1, 'async_trait>(&'life0 self, state: &'life1 dyn datafusion_session::session::Session, file_scan_config: datafusion_datasource::file_scan_config::FileScanConfig) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait +pub fn vortex_datafusion::VortexFormat::create_physical_plan<'life0, 'life1, 'async_trait>(&'life0 self, &'life1 dyn datafusion_session::session::Session, datafusion_datasource::file_scan_config::FileScanConfig) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait -pub fn vortex_datafusion::VortexFormat::create_writer_physical_plan<'life0, 'life1, 'async_trait>(&'life0 self, input: alloc::sync::Arc, _state: &'life1 dyn datafusion_session::session::Session, conf: datafusion_datasource::file_sink_config::FileSinkConfig, order_requirements: core::option::Option) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait +pub fn vortex_datafusion::VortexFormat::create_writer_physical_plan<'life0, 'life1, 'async_trait>(&'life0 self, alloc::sync::Arc, &'life1 dyn datafusion_session::session::Session, datafusion_datasource::file_sink_config::FileSinkConfig, core::option::Option) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait -pub fn vortex_datafusion::VortexFormat::file_source(&self, table_schema: datafusion_datasource::table_schema::TableSchema) -> alloc::sync::Arc +pub fn vortex_datafusion::VortexFormat::file_source(&self, datafusion_datasource::table_schema::TableSchema) -> alloc::sync::Arc pub fn vortex_datafusion::VortexFormat::get_ext(&self) -> alloc::string::String -pub fn vortex_datafusion::VortexFormat::get_ext_with_compression(&self, file_compression_type: &datafusion_datasource::file_compression_type::FileCompressionType) -> datafusion_common::error::Result +pub fn vortex_datafusion::VortexFormat::get_ext_with_compression(&self, &datafusion_datasource::file_compression_type::FileCompressionType) -> datafusion_common::error::Result -pub fn vortex_datafusion::VortexFormat::infer_schema<'life0, 'life1, 'life2, 'life3, 'async_trait>(&'life0 self, state: &'life1 dyn datafusion_session::session::Session, store: &'life2 alloc::sync::Arc, objects: &'life3 [object_store::ObjectMeta]) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait +pub fn vortex_datafusion::VortexFormat::infer_schema<'life0, 'life1, 'life2, 'life3, 'async_trait>(&'life0 self, &'life1 dyn datafusion_session::session::Session, &'life2 alloc::sync::Arc, &'life3 [object_store::ObjectMeta]) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait -pub fn vortex_datafusion::VortexFormat::infer_stats<'life0, 'life1, 'life2, 'life3, 'async_trait>(&'life0 self, state: &'life1 dyn datafusion_session::session::Session, store: &'life2 alloc::sync::Arc, table_schema: arrow_schema::schema::SchemaRef, object: &'life3 object_store::ObjectMeta) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait +pub fn vortex_datafusion::VortexFormat::infer_stats<'life0, 'life1, 'life2, 'life3, 'async_trait>(&'life0 self, &'life1 dyn datafusion_session::session::Session, &'life2 alloc::sync::Arc, arrow_schema::schema::SchemaRef, &'life3 object_store::ObjectMeta) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait pub struct vortex_datafusion::VortexFormatFactory @@ -176,13 +176,13 @@ impl vortex_datafusion::VortexFormatFactory pub fn vortex_datafusion::VortexFormatFactory::new() -> Self -pub fn vortex_datafusion::VortexFormatFactory::new_with_options(session: vortex_session::VortexSession, options: vortex_datafusion::VortexTableOptions) -> Self +pub fn vortex_datafusion::VortexFormatFactory::new_with_options(vortex_session::VortexSession, vortex_datafusion::VortexTableOptions) -> Self -pub fn vortex_datafusion::VortexFormatFactory::with_options(self, options: vortex_datafusion::VortexTableOptions) -> Self +pub fn vortex_datafusion::VortexFormatFactory::with_options(self, vortex_datafusion::VortexTableOptions) -> Self impl core::fmt::Debug for vortex_datafusion::VortexFormatFactory -pub fn vortex_datafusion::VortexFormatFactory::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_datafusion::VortexFormatFactory::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl datafusion_common::file_options::file_type::GetExt for vortex_datafusion::VortexFormatFactory @@ -192,7 +192,7 @@ impl datafusion_datasource::file_format::FileFormatFactory for vortex_datafusion pub fn vortex_datafusion::VortexFormatFactory::as_any(&self) -> &dyn core::any::Any -pub fn vortex_datafusion::VortexFormatFactory::create(&self, _state: &dyn datafusion_session::session::Session, format_options: &std::collections::hash::map::HashMap) -> datafusion_common::error::Result> +pub fn vortex_datafusion::VortexFormatFactory::create(&self, &dyn datafusion_session::session::Session, &std::collections::hash::map::HashMap) -> datafusion_common::error::Result> pub fn vortex_datafusion::VortexFormatFactory::default(&self) -> alloc::sync::Arc @@ -202,21 +202,21 @@ impl vortex_datafusion::VortexSource pub fn vortex_datafusion::VortexSource::metrics_registry(&self) -> &alloc::sync::Arc -pub fn vortex_datafusion::VortexSource::new(table_schema: datafusion_datasource::table_schema::TableSchema, session: vortex_session::VortexSession) -> Self +pub fn vortex_datafusion::VortexSource::new(datafusion_datasource::table_schema::TableSchema, vortex_session::VortexSession) -> Self pub fn vortex_datafusion::VortexSource::options(&self) -> &vortex_datafusion::VortexTableOptions -pub fn vortex_datafusion::VortexSource::with_expression_convertor(self, expr_convertor: alloc::sync::Arc) -> Self +pub fn vortex_datafusion::VortexSource::with_expression_convertor(self, alloc::sync::Arc) -> Self -pub fn vortex_datafusion::VortexSource::with_file_metadata_cache(self, file_metadata_cache: alloc::sync::Arc) -> Self +pub fn vortex_datafusion::VortexSource::with_file_metadata_cache(self, alloc::sync::Arc) -> Self -pub fn vortex_datafusion::VortexSource::with_options(self, opts: vortex_datafusion::VortexTableOptions) -> Self +pub fn vortex_datafusion::VortexSource::with_options(self, vortex_datafusion::VortexTableOptions) -> Self -pub fn vortex_datafusion::VortexSource::with_projection_pushdown(self, enabled: bool) -> Self +pub fn vortex_datafusion::VortexSource::with_projection_pushdown(self, bool) -> Self -pub fn vortex_datafusion::VortexSource::with_scan_concurrency(self, scan_concurrency: usize) -> Self +pub fn vortex_datafusion::VortexSource::with_scan_concurrency(self, usize) -> Self -pub fn vortex_datafusion::VortexSource::with_vortex_reader_factory(self, vortex_reader_factory: alloc::sync::Arc) -> Self +pub fn vortex_datafusion::VortexSource::with_vortex_reader_factory(self, alloc::sync::Arc) -> Self impl core::clone::Clone for vortex_datafusion::VortexSource @@ -226,13 +226,13 @@ impl datafusion_datasource::file::FileSource for vortex_datafusion::VortexSource pub fn vortex_datafusion::VortexSource::as_any(&self) -> &dyn core::any::Any -pub fn vortex_datafusion::VortexSource::create_file_opener(&self, object_store: alloc::sync::Arc, base_config: &datafusion_datasource::file_scan_config::FileScanConfig, partition: usize) -> datafusion_common::error::Result> +pub fn vortex_datafusion::VortexSource::create_file_opener(&self, alloc::sync::Arc, &datafusion_datasource::file_scan_config::FileScanConfig, usize) -> datafusion_common::error::Result> pub fn vortex_datafusion::VortexSource::file_type(&self) -> &str pub fn vortex_datafusion::VortexSource::filter(&self) -> core::option::Option> -pub fn vortex_datafusion::VortexSource::fmt_extra(&self, t: datafusion_physical_plan::display::DisplayFormatType, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_datafusion::VortexSource::fmt_extra(&self, datafusion_physical_plan::display::DisplayFormatType, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_datafusion::VortexSource::metrics(&self) -> &datafusion_physical_expr_common::metrics::ExecutionPlanMetricsSet @@ -242,11 +242,11 @@ pub fn vortex_datafusion::VortexSource::supports_repartitioning(&self) -> bool pub fn vortex_datafusion::VortexSource::table_schema(&self) -> &datafusion_datasource::table_schema::TableSchema -pub fn vortex_datafusion::VortexSource::try_pushdown_filters(&self, filters: alloc::vec::Vec>, _config: &datafusion_common::config::ConfigOptions) -> datafusion_common::error::Result>> +pub fn vortex_datafusion::VortexSource::try_pushdown_filters(&self, alloc::vec::Vec>, &datafusion_common::config::ConfigOptions) -> datafusion_common::error::Result>> -pub fn vortex_datafusion::VortexSource::try_pushdown_projection(&self, projection: &datafusion_physical_expr::projection::ProjectionExprs) -> datafusion_common::error::Result>> +pub fn vortex_datafusion::VortexSource::try_pushdown_projection(&self, &datafusion_physical_expr::projection::ProjectionExprs) -> datafusion_common::error::Result>> -pub fn vortex_datafusion::VortexSource::with_batch_size(&self, batch_size: usize) -> alloc::sync::Arc +pub fn vortex_datafusion::VortexSource::with_batch_size(&self, usize) -> alloc::sync::Arc pub struct vortex_datafusion::VortexTableOptions @@ -266,7 +266,7 @@ impl core::cmp::Eq for vortex_datafusion::VortexTableOptions impl core::cmp::PartialEq for vortex_datafusion::VortexTableOptions -pub fn vortex_datafusion::VortexTableOptions::eq(&self, other: &vortex_datafusion::VortexTableOptions) -> bool +pub fn vortex_datafusion::VortexTableOptions::eq(&self, &vortex_datafusion::VortexTableOptions) -> bool impl core::default::Default for vortex_datafusion::VortexTableOptions @@ -274,34 +274,34 @@ pub fn vortex_datafusion::VortexTableOptions::default() -> Self impl core::fmt::Debug for vortex_datafusion::VortexTableOptions -pub fn vortex_datafusion::VortexTableOptions::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_datafusion::VortexTableOptions::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::StructuralPartialEq for vortex_datafusion::VortexTableOptions impl datafusion_common::config::ConfigField for vortex_datafusion::VortexTableOptions -pub fn vortex_datafusion::VortexTableOptions::reset(&mut self, key: &str) -> datafusion_common::error::Result<()> +pub fn vortex_datafusion::VortexTableOptions::reset(&mut self, &str) -> datafusion_common::error::Result<()> -pub fn vortex_datafusion::VortexTableOptions::set(&mut self, key: &str, value: &str) -> datafusion_common::error::Result<()> +pub fn vortex_datafusion::VortexTableOptions::set(&mut self, &str, &str) -> datafusion_common::error::Result<()> -pub fn vortex_datafusion::VortexTableOptions::visit(&self, v: &mut V, key_prefix: &str, _description: &'static str) +pub fn vortex_datafusion::VortexTableOptions::visit(&self, &mut V, &str, &'static str) pub trait vortex_datafusion::ExpressionConvertor: core::marker::Send + core::marker::Sync -pub fn vortex_datafusion::ExpressionConvertor::can_be_pushed_down(&self, expr: &alloc::sync::Arc, schema: &arrow_schema::schema::Schema) -> bool +pub fn vortex_datafusion::ExpressionConvertor::can_be_pushed_down(&self, &alloc::sync::Arc, &arrow_schema::schema::Schema) -> bool -pub fn vortex_datafusion::ExpressionConvertor::convert(&self, expr: &dyn datafusion_physical_expr_common::physical_expr::PhysicalExpr) -> datafusion_common::error::Result +pub fn vortex_datafusion::ExpressionConvertor::convert(&self, &dyn datafusion_physical_expr_common::physical_expr::PhysicalExpr) -> datafusion_common::error::Result -pub fn vortex_datafusion::ExpressionConvertor::no_pushdown_projection(&self, source_projection: datafusion_physical_expr::projection::ProjectionExprs, input_schema: &arrow_schema::schema::Schema) -> datafusion_common::error::Result +pub fn vortex_datafusion::ExpressionConvertor::no_pushdown_projection(&self, datafusion_physical_expr::projection::ProjectionExprs, &arrow_schema::schema::Schema) -> datafusion_common::error::Result -pub fn vortex_datafusion::ExpressionConvertor::split_projection(&self, source_projection: datafusion_physical_expr::projection::ProjectionExprs, input_schema: &arrow_schema::schema::Schema, output_schema: &arrow_schema::schema::Schema) -> datafusion_common::error::Result +pub fn vortex_datafusion::ExpressionConvertor::split_projection(&self, datafusion_physical_expr::projection::ProjectionExprs, &arrow_schema::schema::Schema, &arrow_schema::schema::Schema) -> datafusion_common::error::Result impl vortex_datafusion::ExpressionConvertor for vortex_datafusion::DefaultExpressionConvertor -pub fn vortex_datafusion::DefaultExpressionConvertor::can_be_pushed_down(&self, expr: &alloc::sync::Arc, schema: &arrow_schema::schema::Schema) -> bool +pub fn vortex_datafusion::DefaultExpressionConvertor::can_be_pushed_down(&self, &alloc::sync::Arc, &arrow_schema::schema::Schema) -> bool -pub fn vortex_datafusion::DefaultExpressionConvertor::convert(&self, df: &dyn datafusion_physical_expr_common::physical_expr::PhysicalExpr) -> datafusion_common::error::Result +pub fn vortex_datafusion::DefaultExpressionConvertor::convert(&self, &dyn datafusion_physical_expr_common::physical_expr::PhysicalExpr) -> datafusion_common::error::Result -pub fn vortex_datafusion::DefaultExpressionConvertor::no_pushdown_projection(&self, source_projection: datafusion_physical_expr::projection::ProjectionExprs, input_schema: &arrow_schema::schema::Schema) -> datafusion_common::error::Result +pub fn vortex_datafusion::DefaultExpressionConvertor::no_pushdown_projection(&self, datafusion_physical_expr::projection::ProjectionExprs, &arrow_schema::schema::Schema) -> datafusion_common::error::Result -pub fn vortex_datafusion::DefaultExpressionConvertor::split_projection(&self, source_projection: datafusion_physical_expr::projection::ProjectionExprs, input_schema: &arrow_schema::schema::Schema, output_schema: &arrow_schema::schema::Schema) -> datafusion_common::error::Result +pub fn vortex_datafusion::DefaultExpressionConvertor::split_projection(&self, datafusion_physical_expr::projection::ProjectionExprs, &arrow_schema::schema::Schema, &arrow_schema::schema::Schema) -> datafusion_common::error::Result diff --git a/vortex-datafusion/src/convert/exprs.rs b/vortex-datafusion/src/convert/exprs.rs index e5142b50ed1..ce6c9eb0aa8 100644 --- a/vortex-datafusion/src/convert/exprs.rs +++ b/vortex-datafusion/src/convert/exprs.rs @@ -25,6 +25,7 @@ use vortex::expr::Expression; use vortex::expr::and_collect; use vortex::expr::cast; use vortex::expr::get_item; +use vortex::expr::is_not_null; use vortex::expr::is_null; use vortex::expr::list_contains; use vortex::expr::lit; @@ -241,7 +242,7 @@ impl ExpressionConvertor for DefaultExpressionConvertor { if let Some(is_not_null_expr) = df.as_any().downcast_ref::() { let arg = self.convert(is_not_null_expr.arg().as_ref())?; - return Ok(not(is_null(arg))); + return Ok(is_not_null(arg)); } if let Some(in_list) = df.as_any().downcast_ref::() { diff --git a/vortex-datafusion/src/convert/stats.rs b/vortex-datafusion/src/convert/stats.rs index 2a6b1d0996f..3c7b31de95d 100644 --- a/vortex-datafusion/src/convert/stats.rs +++ b/vortex-datafusion/src/convert/stats.rs @@ -9,6 +9,7 @@ use vortex::dtype::Nullability; use vortex::dtype::PType; use vortex::error::VortexExpect; use vortex::error::VortexResult; +use vortex::expr::stats::Precision as VortexPrecision; use vortex::expr::stats::Stat; use vortex::scalar::Scalar; @@ -83,10 +84,40 @@ pub(crate) fn stats_set_to_df( min_value: min.to_df(), max_value: max.to_df(), sum_value: sum.to_df(), - distinct_count: stats_set - .get_as::(Stat::IsConstant, &DType::Bool(Nullability::NonNullable)) - .and_then(|is_constant| is_constant.as_exact().map(|_| Precision::Exact(1))) - .unwrap_or(Precision::Absent), + distinct_count: is_constant_to_distinct_count( + stats_set.get_as::(Stat::IsConstant, &DType::Bool(Nullability::NonNullable)), + ), byte_size: column_size.to_df(), }) } + +pub(crate) fn is_constant_to_distinct_count( + is_constant: Option>, +) -> Precision { + match is_constant.and_then(VortexPrecision::as_exact) { + Some(true) => Precision::Exact(1), + Some(false) | None => Precision::Absent, + } +} + +#[cfg(test)] +mod tests { + use vortex::expr::stats::Precision as VortexPrecision; + + use super::*; + + #[test] + fn is_constant_false_does_not_imply_one_distinct_value() -> VortexResult<()> { + let false_constant = StatsSet::of(Stat::IsConstant, VortexPrecision::exact(false)); + let false_stats = stats_set_to_df(&false_constant, &DType::Bool(Nullability::NonNullable))?; + + assert_eq!(false_stats.distinct_count, Precision::Absent); + + let true_constant = StatsSet::of(Stat::IsConstant, VortexPrecision::exact(true)); + let true_stats = stats_set_to_df(&true_constant, &DType::Bool(Nullability::NonNullable))?; + + assert_eq!(true_stats.distinct_count, Precision::Exact(1)); + + Ok(()) + } +} diff --git a/vortex-datafusion/src/lib.rs b/vortex-datafusion/src/lib.rs index ce266fbacd1..d08653318f9 100644 --- a/vortex-datafusion/src/lib.rs +++ b/vortex-datafusion/src/lib.rs @@ -1,7 +1,87 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -//! Connectors to enable [DataFusion](https://docs.rs/datafusion/latest/datafusion/) to read [`Vortex`](https://docs.rs/crate/vortex/latest) data. +//! Integrations between [`Vortex`] and [DataFusion]. +//! +//! The crate exposes two main entry points: +//! +//! - [`VortexFormatFactory`] for the file-based integration used by SQL, +//! `CREATE EXTERNAL TABLE`, and +//! [`ListingTable`]. +//! - [`v2`] for direct integration from an existing Vortex +//! [`DataSourceRef`]. +//! +//! # Registering The File Format +//! +//! Most applications register [`VortexFormatFactory`] with a DataFusion +//! [`SessionContext`] and then let DataFusion create [`VortexFormat`] and +//! [`VortexSource`] instances as queries are planned: +//! +//! ```no_run +//! use std::sync::Arc; +//! +//! use datafusion::datasource::provider::DefaultTableFactory; +//! use datafusion::execution::SessionStateBuilder; +//! use datafusion::prelude::SessionContext; +//! use datafusion_common::GetExt; +//! use vortex_datafusion::VortexFormatFactory; +//! +//! # #[tokio::main] +//! # async fn main() -> Result<(), Box> { +//! let factory = Arc::new(VortexFormatFactory::new()); +//! let mut state_builder = SessionStateBuilder::new() +//! .with_default_features() +//! .with_table_factory( +//! factory.get_ext().to_uppercase(), +//! Arc::new(DefaultTableFactory::new()), +//! ); +//! +//! if let Some(file_formats) = state_builder.file_formats() { +//! file_formats.push(factory.clone() as _); +//! } +//! +//! let ctx = SessionContext::new_with_state(state_builder.build()).enable_url_table(); +//! ctx.sql( +//! "CREATE EXTERNAL TABLE metrics (service VARCHAR, value BIGINT) \ +//! STORED AS vortex LOCATION 'file:///tmp/metrics/'", +//! ) +//! .await?; +//! # Ok(()) +//! # } +//! ``` +//! +//! # Registering An Existing Vortex Data Source +//! +//! If your application already has a Vortex [`DataSourceRef`], use +//! [`v2::VortexTable`] to register it directly with DataFusion: +//! +//! ```no_run +//! use std::sync::Arc; +//! +//! use arrow_schema::Schema; +//! use datafusion::prelude::SessionContext; +//! use vortex::VortexSessionDefault; +//! use vortex::scan::DataSourceRef; +//! use vortex::session::VortexSession; +//! use vortex_datafusion::v2::VortexTable; +//! +//! # let data_source: DataSourceRef = todo!(); +//! let table = Arc::new(VortexTable::new( +//! data_source, +//! VortexSession::default(), +//! Arc::new(Schema::empty()), +//! )); +//! +//! let ctx = SessionContext::new(); +//! ctx.register_table("vortex_data", table)?; +//! # Ok::<(), datafusion_common::DataFusionError>(()) +//! ``` +//! +//! [`Vortex`]: https://docs.rs/crate/vortex/latest +//! [DataFusion]: https://docs.rs/datafusion/latest/datafusion/ +//! [`ListingTable`]: https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html +//! [`DataSourceRef`]: vortex::scan::DataSourceRef +//! [`SessionContext`]: https://docs.rs/datafusion/latest/datafusion/prelude/struct.SessionContext.html #![deny(missing_docs)] use std::fmt::Debug; @@ -19,7 +99,11 @@ pub use convert::exprs::DefaultExpressionConvertor; pub use convert::exprs::ExpressionConvertor; pub use persistent::*; -/// Extension trait to convert our [`Precision`](vortex::stats::Precision) to Datafusion's [`Precision`](datafusion_common::stats::Precision) +/// Extension trait to convert our [`Precision`] to DataFusion's +/// [`DataFusionPrecision`]. +/// +/// [`Precision`]: vortex::expr::stats::Precision +/// [`DataFusionPrecision`]: datafusion_common::stats::Precision trait PrecisionExt where T: Debug + Clone + PartialEq + Eq + PartialOrd, diff --git a/vortex-datafusion/src/persistent/access_plan.rs b/vortex-datafusion/src/persistent/access_plan.rs index 78f81d01b05..99583b7c0e4 100644 --- a/vortex-datafusion/src/persistent/access_plan.rs +++ b/vortex-datafusion/src/persistent/access_plan.rs @@ -4,18 +4,46 @@ use vortex::layout::scan::scan_builder::ScanBuilder; use vortex::scan::selection::Selection; -/// Custom Vortex-specific information that can be provided by external indexes or other sources. +/// Additional Vortex-specific scan constraints attached to a +/// [`PartitionedFile`]. /// -/// This is intended as a low-level interface for users building their own data systems, see the [advance index] example from the DataFusion repo for a similar usage with Parquet. +/// `VortexAccessPlan` is the hook to use when an external index or planner +/// already knows that only part of a file needs to be scanned. The plan is +/// attached as `extensions` on `PartitionedFile`, and the internal +/// `VortexOpener` applies it before building the Vortex scan. /// -/// [advance index]: https://github.com/apache/datafusion/blob/47df535d2cd5aac5ad5a92bdc837f38e05ea0f0f/datafusion-examples/examples/data_io/parquet_advanced_index.rs +/// The current access plan surface is intentionally small: it lets callers +/// provide a [`Selection`] that narrows the rows considered by the scan. +/// +/// # Example +/// +/// ```no_run +/// # use std::sync::Arc; +/// # use datafusion_datasource::PartitionedFile; +/// # use vortex::scan::selection::Selection; +/// use vortex_datafusion::VortexAccessPlan; +/// +/// # let selection: Selection = todo!(); +/// let file = PartitionedFile::new("metrics.vortex", 1024).with_extensions(Arc::new( +/// VortexAccessPlan::default().with_selection(selection), +/// )); +/// # let _ = file; +/// ``` +/// +/// This is a low-level integration point for systems building their own access +/// paths on top of DataFusion. For a conceptually similar Parquet example, see +/// DataFusion's +/// [`parquet_advanced_index`]. +/// +/// [`PartitionedFile`]: datafusion_datasource::PartitionedFile +/// [`parquet_advanced_index`]: https://github.com/apache/datafusion/blob/47df535d2cd5aac5ad5a92bdc837f38e05ea0f0f/datafusion-examples/examples/data_io/parquet_advanced_index.rs #[derive(Default)] pub struct VortexAccessPlan { selection: Option, } impl VortexAccessPlan { - /// Sets a [`Selection`] for this plan. + /// Sets the row [`Selection`] to apply when the file is opened. pub fn with_selection(mut self, selection: Selection) -> Self { self.selection = Some(selection); self @@ -28,7 +56,10 @@ impl VortexAccessPlan { self.selection.as_ref() } - /// Apply the plan to the scan's builder. + /// Applies this access plan to a [`ScanBuilder`]. + /// + /// This is used internally by the file opener after it has translated a + /// `PartitionedFile` into a Vortex scan. pub fn apply_to_builder(&self, mut scan_builder: ScanBuilder) -> ScanBuilder where A: 'static + Send, diff --git a/vortex-datafusion/src/persistent/cache.rs b/vortex-datafusion/src/persistent/cache.rs index 037b4b1cd0a..de632a7597d 100644 --- a/vortex-datafusion/src/persistent/cache.rs +++ b/vortex-datafusion/src/persistent/cache.rs @@ -38,7 +38,7 @@ impl FileMetadata for CachedVortexMetadata { .unwrap_or(1024 * 64) } - #[allow(clippy::disallowed_types)] + #[expect(clippy::disallowed_types)] fn extra_info(&self) -> std::collections::HashMap { Default::default() } diff --git a/vortex-datafusion/src/persistent/format.rs b/vortex-datafusion/src/persistent/format.rs index 1c33063a4f8..b63475f7066 100644 --- a/vortex-datafusion/src/persistent/format.rs +++ b/vortex-datafusion/src/persistent/format.rs @@ -14,6 +14,7 @@ use datafusion_common::ColumnStatistics; use datafusion_common::DataFusionError; use datafusion_common::GetExt; use datafusion_common::Result as DFResult; +use datafusion_common::ScalarValue as DFScalarValue; use datafusion_common::Statistics; use datafusion_common::config::ConfigField; use datafusion_common::config_namespace; @@ -62,6 +63,7 @@ use vortex::file::VORTEX_FILE_EXTENSION; use vortex::io::object_store::ObjectStoreReadAt; use vortex::io::session::RuntimeSessionExt; use vortex::scalar::Scalar; +use vortex::scalar::ScalarValue as VortexScalarValue; use vortex::session::VortexSession; use super::cache::CachedVortexMetadata; @@ -69,11 +71,58 @@ use super::sink::VortexSink; use super::source::VortexSource; use crate::PrecisionExt as _; use crate::convert::TryToDataFusion; +use crate::convert::stats::is_constant_to_distinct_count; const DEFAULT_FOOTER_INITIAL_READ_SIZE_BYTES: usize = MAX_POSTSCRIPT_SIZE as usize + EOF_SIZE; const DEFAULT_TARGET_FILE_SIZE_MB: usize = 128; -/// Vortex implementation of a DataFusion [`FileFormat`]. +/// DataFusion [`FileFormat`] implementation for `.vortex` files. +/// +/// Most applications do not construct `VortexFormat` directly. Instead, they +/// register [`VortexFormatFactory`] with a [`SessionContext`] and let +/// DataFusion instantiate `VortexFormat` as tables are planned. +/// +/// Construct `VortexFormat` directly when you are wiring a [`ListingTable`] by +/// hand and need to pass a file format into [`ListingOptions`]. +/// +/// # Example +/// +/// ```no_run +/// use std::sync::Arc; +/// +/// use datafusion::datasource::listing::ListingOptions; +/// use datafusion::datasource::listing::ListingTable; +/// use datafusion::datasource::listing::ListingTableConfig; +/// use datafusion::datasource::listing::ListingTableUrl; +/// use datafusion::prelude::SessionContext; +/// use tempfile::tempdir; +/// use vortex::VortexSessionDefault; +/// use vortex::session::VortexSession; +/// use vortex_datafusion::VortexFormat; +/// +/// # #[tokio::main] +/// # async fn main() -> Result<(), Box> { +/// let ctx = SessionContext::new(); +/// let dir = tempdir()?; +/// +/// let format = Arc::new(VortexFormat::new(VortexSession::default())); +/// let table_url = ListingTableUrl::parse(dir.path().to_str().unwrap())?; +/// let config = ListingTableConfig::new(table_url) +/// .with_listing_options( +/// ListingOptions::new(format).with_session_config_options(ctx.state().config()), +/// ) +/// .infer_schema(&ctx.state()) +/// .await?; +/// +/// let table = ListingTable::try_new(config)?; +/// # let _ = table; +/// # Ok(()) +/// # } +/// ``` +/// +/// [`SessionContext`]: https://docs.rs/datafusion/latest/datafusion/prelude/struct.SessionContext.html +/// [`ListingTable`]: https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html +/// [`ListingOptions`]: https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingOptions.html pub struct VortexFormat { session: VortexSession, opts: VortexTableOptions, @@ -88,9 +137,24 @@ impl Debug for VortexFormat { } config_namespace! { - /// Options to configure the [`VortexFormat`]. + /// Options to configure [`VortexFormat`] and [`VortexSource`]. /// - /// Can be set through a DataFusion [`SessionConfig`]. + /// These options are usually set on a [`VortexFormatFactory`] and inherited + /// by the `VortexFormat` / `VortexSource` instances created for individual + /// tables. + /// + /// # Example + /// + /// ```rust + /// use vortex_datafusion::{VortexFormatFactory, VortexTableOptions}; + /// + /// let factory = VortexFormatFactory::new().with_options(VortexTableOptions { + /// projection_pushdown: true, + /// scan_concurrency: Some(8), + /// ..Default::default() + /// }); + /// # let _ = factory; + /// ``` /// /// [`SessionConfig`]: https://docs.rs/datafusion/latest/datafusion/prelude/struct.SessionConfig.html pub struct VortexTableOptions { @@ -122,7 +186,44 @@ config_namespace! { impl Eq for VortexTableOptions {} -/// Minimal factory to create [`VortexFormat`] instances. +/// Registration entry point for the file-backed Vortex integration. +/// +/// `VortexFormatFactory` is the type most applications use. Register it with a +/// DataFusion session, and DataFusion will create [`VortexFormat`] values for +/// `CREATE EXTERNAL TABLE`, [`ListingTable`], and URL-table scans. +/// +/// The factory stores a [`VortexSession`] and default [`VortexTableOptions`]. +/// Those defaults are copied into the formats and sources created for each +/// table. +/// +/// # Example +/// +/// ```no_run +/// use std::sync::Arc; +/// +/// use datafusion::datasource::provider::DefaultTableFactory; +/// use datafusion::execution::SessionStateBuilder; +/// use datafusion_common::GetExt; +/// use vortex_datafusion::{VortexFormatFactory, VortexTableOptions}; +/// +/// let factory = Arc::new(VortexFormatFactory::new().with_options(VortexTableOptions { +/// projection_pushdown: true, +/// ..Default::default() +/// })); +/// +/// let mut state_builder = SessionStateBuilder::new() +/// .with_default_features() +/// .with_table_factory( +/// factory.get_ext().to_uppercase(), +/// Arc::new(DefaultTableFactory::new()), +/// ); +/// +/// if let Some(file_formats) = state_builder.file_formats() { +/// file_formats.push(factory.clone() as _); +/// } +/// ``` +/// +/// [`ListingTable`]: https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html #[derive(Debug)] pub struct VortexFormatFactory { session: VortexSession, @@ -136,7 +237,7 @@ impl GetExt for VortexFormatFactory { } impl VortexFormatFactory { - /// Creates a new instance with a default [`VortexSession`] and default options. + /// Creates a factory with a default [`VortexSession`] and default options. #[expect( clippy::new_without_default, reason = "FormatFactory defines `default` method, so having `Default` implementation is confusing" @@ -148,9 +249,11 @@ impl VortexFormatFactory { } } - /// Creates a new instance with customized session and default options for all [`VortexFormat`] instances created from this factory. + /// Creates a factory with an explicit session and default options. /// - /// The options can be overridden by table-level configuration pass in [`FileFormatFactory::create`]. + /// The supplied options become the baseline for every [`VortexFormat`] + /// created by this factory. DataFusion may still override them with + /// table-level options passed into [`FileFormatFactory::create`]. pub fn new_with_options(session: VortexSession, options: VortexTableOptions) -> Self { Self { session, @@ -158,13 +261,21 @@ impl VortexFormatFactory { } } - /// Override the default options for this factory. + /// Overrides the default options for this factory. + /// + /// This is the usual way to turn on features such as projection pushdown for + /// every table created through the factory. + /// + /// # Example /// - /// For example: /// ```rust /// use vortex_datafusion::{VortexFormatFactory, VortexTableOptions}; /// - /// let factory = VortexFormatFactory::new().with_options(VortexTableOptions::default()); + /// let factory = VortexFormatFactory::new().with_options(VortexTableOptions { + /// projection_pushdown: true, + /// ..Default::default() + /// }); + /// # let _ = factory; /// ``` pub fn with_options(mut self, options: VortexTableOptions) -> Self { self.options = Some(options); @@ -204,17 +315,23 @@ impl FileFormatFactory for VortexFormatFactory { } impl VortexFormat { - /// Create a new instance with default options. + /// Creates a format with default [`VortexTableOptions`]. + /// + /// Prefer [`VortexFormatFactory`] when registering with a session. Construct + /// `VortexFormat` directly when building [`ListingOptions`] manually. + /// + /// [`ListingOptions`]: https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingOptions.html pub fn new(session: VortexSession) -> Self { Self::new_with_options(session, VortexTableOptions::default()) } - /// Creates a new instance with configured by a [`VortexTableOptions`]. + /// Creates a format with explicit [`VortexTableOptions`]. pub fn new_with_options(session: VortexSession, opts: VortexTableOptions) -> Self { Self { session, opts } } - /// Return the format specific configuration + /// Returns the format-specific configuration that will be copied into the + /// [`VortexSource`] created for a scan. pub fn options(&self) -> &VortexTableOptions { &self.opts } @@ -395,11 +512,13 @@ impl FileFormat for VortexFormat { .vortex_expect("Row count overflow"), ), total_byte_size: Precision::Absent, - column_statistics: vec![ColumnStatistics::default(); struct_dtype.nfields()], + column_statistics: vec![ + ColumnStatistics::default(); + table_schema.fields().len() + ], }); }; - let mut sum_of_column_byte_sizes = stats::Precision::exact(0_usize); let mut column_statistics = Vec::with_capacity(table_schema.fields().len()); for field in table_schema.fields().iter() { @@ -413,55 +532,22 @@ impl FileFormat for VortexFormat { let (stats_set, stats_dtype) = file_stats.get(col_idx); // Update the total size in bytes. - let column_size = stats_set - .get_as::(Stat::UncompressedSizeInBytes, &PType::U64.into()) - .unwrap_or_else(|| stats::Precision::inexact(0_usize)); - sum_of_column_byte_sizes = sum_of_column_byte_sizes - .zip(column_size) - .map(|(acc, size)| acc + size); - - // TODO(connor): There's a lot that can go wrong here, should probably handle this - // more gracefully... - // Find the min statistic. - let min = stats_set.get(Stat::Min).and_then(|pstat_val| { - pstat_val - .map(|stat_val| { - // Because of DataFusion's Schema evolution, it is possible that the - // type of the min/max stat has changed. Thus we construct the stat as - // the file datatype first and only then do we cast accordingly. - Scalar::try_new( - Stat::Min - .dtype(stats_dtype) - .vortex_expect("must have a valid dtype"), - Some(stat_val), - ) - .vortex_expect("`Stat::Min` somehow had an incompatible `DType`") - .cast(&DType::from_arrow(field.as_ref())) - .vortex_expect("Unable to cast to target type that DataFusion wants") - .try_to_df() - .ok() - }) - .transpose() - }); - - // Find the max statistic. - let max = stats_set.get(Stat::Max).and_then(|pstat_val| { - pstat_val - .map(|stat_val| { - Scalar::try_new( - Stat::Max - .dtype(stats_dtype) - .vortex_expect("must have a valid dtype"), - Some(stat_val), - ) - .vortex_expect("`Stat::Max` somehow had an incompatible `DType`") - .cast(&DType::from_arrow(field.as_ref())) - .vortex_expect("Unable to cast to target type that DataFusion wants") - .try_to_df() - .ok() - }) - .transpose() - }); + let column_size = + stats_set.get_as::(Stat::UncompressedSizeInBytes, &PType::U64.into()); + + let target_dtype = DType::from_arrow(field.as_ref()); + let min = scalar_stat_to_df( + Stat::Min, + stats_set.get(Stat::Min), + stats_dtype, + &target_dtype, + ); + let max = scalar_stat_to_df( + Stat::Max, + stats_set.get(Stat::Max), + stats_dtype, + &target_dtype, + ); let null_count = stats_set.get_as::(Stat::NullCount, &PType::U64.into()); @@ -470,16 +556,19 @@ impl FileFormat for VortexFormat { min_value: min.to_df(), max_value: max.to_df(), sum_value: Precision::Absent, - distinct_count: stats_set - .get_as::(Stat::IsConstant, &DType::Bool(Nullability::NonNullable)) - .and_then(|is_constant| is_constant.as_exact().map(|_| Precision::Exact(1))) - .unwrap_or(Precision::Absent), - // TODO(connor): Is this correct? + distinct_count: is_constant_to_distinct_count( + stats_set.get_as::( + Stat::IsConstant, + &DType::Bool(Nullability::NonNullable), + ), + ), byte_size: column_size.to_df(), }) } - let total_byte_size = sum_of_column_byte_sizes.to_df(); + let total_byte_size = column_statistics + .iter() + .fold(Precision::Exact(0), |acc, cs| acc.add(&cs.byte_size)); Ok(Statistics { num_rows: Precision::Exact( @@ -580,6 +669,23 @@ impl FileFormat for VortexFormat { } } +fn scalar_stat_to_df( + stat: Stat, + value: Option>, + stats_dtype: &DType, + target_dtype: &DType, +) -> Option> { + let stat_dtype = stat.dtype(stats_dtype)?; + + value? + .try_map(|stat_value| { + Scalar::try_new(stat_dtype, Some(stat_value))? + .cast(target_dtype)? + .try_to_df() + }) + .ok() +} + #[cfg(test)] mod tests { diff --git a/vortex-datafusion/src/persistent/metrics.rs b/vortex-datafusion/src/persistent/metrics.rs index 7e99430568f..d4c6456d6fa 100644 --- a/vortex-datafusion/src/persistent/metrics.rs +++ b/vortex-datafusion/src/persistent/metrics.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -//! Vortex table provider metrics. +//! Helpers for extracting Vortex scan metrics from DataFusion execution plans. use std::sync::Arc; use std::time::Duration; @@ -27,13 +27,40 @@ use crate::persistent::source::VortexSource; pub(crate) static PARTITION_LABEL: &str = "partition"; pub(crate) static PATH_LABEL: &str = "file_path"; -/// Extracts datafusion metrics from all VortexExec instances in -/// a given physical plan. +/// Walks a physical plan and returns one [`MetricsSet`] per +/// [`DataSourceExec`]. +/// +/// For Vortex-backed scans, the returned metrics include both the metrics +/// already attached to the `DataSourceExec` and the Vortex metrics accumulated +/// in [`VortexSource::metrics_registry`]. +/// +/// This helper exists because the Vortex read path records most scan metrics in +/// a Vortex [`MetricsRegistry`] rather than in DataFusion's native metrics set. +/// +/// # Example +/// +/// ```no_run +/// # use std::sync::Arc; +/// use datafusion_physical_plan::ExecutionPlan; +/// use vortex_datafusion::metrics::VortexMetricsFinder; +/// +/// # let plan: Arc = todo!(); +/// for metrics in VortexMetricsFinder::find_all(plan.as_ref()) { +/// for metric in metrics.aggregate_by_name().sorted_for_display().iter() { +/// println!("{metric}"); +/// } +/// } +/// ``` +/// +/// [`DataSourceExec`]: datafusion_datasource::source::DataSourceExec +/// [`VortexSource::metrics_registry`]: crate::VortexSource::metrics_registry +/// [`MetricsRegistry`]: vortex::metrics::MetricsRegistry #[derive(Default)] pub struct VortexMetricsFinder(Vec); impl VortexMetricsFinder { - /// find all metrics for VortexExec nodes. + /// Collects metrics for each `DataSourceExec` in `plan`, augmenting any + /// Vortex-backed scan with the attached Vortex registry snapshot. pub fn find_all(plan: &dyn ExecutionPlan) -> Vec { let mut finder = Self::default(); match accept(plan, &mut finder) { diff --git a/vortex-datafusion/src/persistent/mod.rs b/vortex-datafusion/src/persistent/mod.rs index c8560cfb7a8..17565937a0f 100644 --- a/vortex-datafusion/src/persistent/mod.rs +++ b/vortex-datafusion/src/persistent/mod.rs @@ -1,7 +1,27 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -//! Persistent implementation of a Vortex table provider. +//! File-backed DataFusion integration for `.vortex` files. +//! +//! Use this module when Vortex data lives in a filesystem or object store and +//! you want DataFusion to discover files, infer schema, and build +//! [`DataSourceExec`] plans for you. +//! +//! The main entry points are: +//! +//! - [`VortexFormatFactory`] to register the `vortex` file format with +//! DataFusion. +//! - [`VortexFormat`] when constructing +//! [`ListingOptions`] directly. +//! - [`VortexSource`] for lower-level `FileScanConfig` construction. +//! - [`VortexAccessPlan`] when external indexing or custom file selection needs +//! to restrict what a scan reads. +//! - [`metrics::VortexMetricsFinder`] to collect Vortex-specific scan metrics +//! from a planned query. +//! +//! [`DataSourceExec`]: datafusion_datasource::source::DataSourceExec +//! [`ListingOptions`]: https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingOptions.html + mod access_plan; mod cache; mod format; @@ -19,261 +39,4 @@ pub use format::VortexTableOptions; pub use source::VortexSource; #[cfg(test)] -mod tests { - - use std::sync::Arc; - - use datafusion::arrow::util::pretty::pretty_format_batches; - use datafusion_physical_plan::display::DisplayableExecutionPlan; - use insta::assert_snapshot; - use rstest::rstest; - use vortex::VortexSessionDefault; - use vortex::array::IntoArray; - use vortex::array::arrays::ChunkedArray; - use vortex::array::arrays::StructArray; - use vortex::array::arrays::VarBinArray; - use vortex::array::validity::Validity; - use vortex::buffer::buffer; - use vortex::file::WriteOptionsSessionExt; - use vortex::io::VortexWrite; - use vortex::io::object_store::ObjectStoreWrite; - use vortex::session::VortexSession; - - use crate::common_tests::TestSessionContext; - - #[rstest] - #[tokio::test] - async fn test_query_file(#[values(Some(1), None)] limit: Option) -> anyhow::Result<()> { - let ctx = TestSessionContext::default(); - - let session = VortexSession::default(); - - let strings = ChunkedArray::from_iter([ - VarBinArray::from(vec!["ab", "foo", "bar", "baz"]).into_array(), - VarBinArray::from(vec!["ab", "foo", "bar", "baz"]).into_array(), - ]) - .into_array(); - - let numbers = ChunkedArray::from_iter([ - buffer![1u32, 2, 3, 4].into_array(), - buffer![5u32, 6, 7, 8].into_array(), - ]) - .into_array(); - - let st = StructArray::try_new( - ["strings", "numbers"].into(), - vec![strings, numbers], - 8, - Validity::NonNullable, - )?; - - let mut writer = - ObjectStoreWrite::new(Arc::clone(&ctx.store), &"test.vortex".into()).await?; - - let summary = session - .write_options() - .write(&mut writer, st.into_array().to_array_stream()) - .await?; - - writer.shutdown().await?; - - assert_eq!(summary.row_count(), 8); - - let read_row_count = ctx - .session - .sql("SELECT * from '/test.vortex'") - .await? - .limit(0, limit)? - .count() - .await?; - - assert_eq!(read_row_count, limit.unwrap_or(8)); - - Ok(()) - } - - #[tokio::test] - async fn test_addition_pushdown() -> anyhow::Result<()> { - let ctx = TestSessionContext::default(); - dbg!(&ctx.store); - - ctx.session - .sql( - "CREATE EXTERNAL TABLE written_data \ - (a TINYINT NOT NULL) \ - STORED AS vortex \ - LOCATION '/test/'", - ) - .await?; - - ctx.session - .sql("INSERT INTO written_data VALUES (0), (1), (2), (3), (4)") - .await? - .collect() - .await?; - - let result = ctx - .session - .sql("SELECT a, a + 5 as five, a + 6 as six FROM written_data WHERE a + 5 > 7") - .await? - .collect() - .await?; - - assert_snapshot!(pretty_format_batches(&result)?, @r" - +---+------+-----+ - | a | five | six | - +---+------+-----+ - | 3 | 8 | 9 | - | 4 | 9 | 10 | - +---+------+-----+ - "); - - Ok(()) - } - - #[tokio::test] - async fn create_table_ordered_by() -> anyhow::Result<()> { - let ctx = TestSessionContext::default(); - - // Vortex - ctx.session - .sql( - "CREATE EXTERNAL TABLE my_tbl_vx \ - (c1 VARCHAR NOT NULL, c2 INT NOT NULL) \ - STORED AS vortex \ - WITH ORDER (c1 ASC) - LOCATION '/test/'", - ) - .await?; - - ctx.session - .sql("INSERT INTO my_tbl_vx VALUES ('air', 5), ('balloon', 42)") - .await? - .collect() - .await?; - - ctx.session - .sql("INSERT INTO my_tbl_vx VALUES ('zebra', 5)") - .await? - .collect() - .await?; - - ctx.session - .sql("INSERT INTO my_tbl_vx VALUES ('texas', 2000), ('alabama', 2000)") - .await? - .collect() - .await?; - - let df = ctx - .session - .sql("SELECT * FROM my_tbl_vx ORDER BY c1 ASC limit 3") - .await?; - - let physical_plan = ctx - .session - .state() - .create_physical_plan(df.logical_plan()) - .await?; - - insta::assert_snapshot!(DisplayableExecutionPlan::new(physical_plan.as_ref()) - .tree_render().to_string(), @r" - ┌───────────────────────────┐ - │ SortPreservingMergeExec │ - │ -------------------- │ - │ c1 ASC NULLS LAST │ - │ │ - │ limit: 3 │ - └─────────────┬─────────────┘ - ┌─────────────┴─────────────┐ - │ DataSourceExec │ - │ -------------------- │ - │ files: 3 │ - │ format: vortex │ - └───────────────────────────┘ - "); - - let r = df.collect().await?; - - insta::assert_snapshot!(pretty_format_batches(&r)?.to_string(), @r" - +---------+------+ - | c1 | c2 | - +---------+------+ - | air | 5 | - | alabama | 2000 | - | balloon | 42 | - +---------+------+ - "); - - Ok(()) - } - - /// Doc example: demonstrates creating, writing, reading, and filtering a Vortex table. - #[tokio::test] - async fn doc_example() -> anyhow::Result<()> { - // [setup] - use std::sync::Arc; - - use datafusion::datasource::provider::DefaultTableFactory; - use datafusion::execution::SessionStateBuilder; - use datafusion::prelude::SessionContext; - use datafusion_common::GetExt; - use object_store::memory::InMemory; - - use crate::VortexFormatFactory; - - let factory = Arc::new(VortexFormatFactory::new()); - let state = SessionStateBuilder::new() - .with_default_features() - .with_table_factory( - factory.get_ext().to_uppercase(), - Arc::new(DefaultTableFactory::new()), - ) - .with_file_formats(vec![factory]) - .build(); - let ctx = SessionContext::new_with_state(state).enable_url_table(); - // [setup] - - // Register an in-memory object store for the test. - let store = Arc::new(InMemory::new()); - ctx.register_object_store(&url::Url::try_from("file://").unwrap(), store); - - // [create] - ctx.sql( - "CREATE EXTERNAL TABLE my_table \ - (name VARCHAR NOT NULL, age INT NOT NULL) \ - STORED AS vortex \ - LOCATION '/demo/'", - ) - .await?; - // [create] - - // [write] - ctx.sql( - "INSERT INTO my_table VALUES \ - ('Alice', 30), ('Bob', 25), ('Charlie', 35), ('Diana', 28)", - ) - .await? - .collect() - .await?; - // [write] - - // [query] - let result = ctx - .sql("SELECT name, age FROM my_table WHERE age > 28 ORDER BY age") - .await? - .collect() - .await?; - // [query] - - assert_snapshot!(pretty_format_batches(&result)?, @r" - +---------+-----+ - | name | age | - +---------+-----+ - | Alice | 30 | - | Charlie | 35 | - +---------+-----+ - "); - - Ok(()) - } -} +mod tests; diff --git a/vortex-datafusion/src/persistent/opener.rs b/vortex-datafusion/src/persistent/opener.rs index db3cc475d75..54ac87bdc67 100644 --- a/vortex-datafusion/src/persistent/opener.rs +++ b/vortex-datafusion/src/persistent/opener.rs @@ -10,7 +10,6 @@ use datafusion_common::DataFusionError; use datafusion_common::Result as DFResult; use datafusion_common::ScalarValue; use datafusion_common::exec_datafusion_err; -use datafusion_datasource::FileRange; use datafusion_datasource::PartitionedFile; use datafusion_datasource::TableSchema; use datafusion_datasource::file_stream::FileOpenFuture; @@ -30,16 +29,19 @@ use futures::FutureExt; use futures::StreamExt; use futures::TryStreamExt; use futures::stream; +use itertools::Itertools; use object_store::path::Path; use tracing::Instrument; -use vortex::array::ArrayRef; use vortex::array::VortexSessionExecute; use vortex::array::arrow::ArrowArrayExecutor; +use vortex::dtype::FieldMask; use vortex::error::VortexError; +use vortex::error::VortexExpect; use vortex::file::OpenOptionsSessionExt; use vortex::io::InstrumentedReadAt; use vortex::layout::LayoutReader; use vortex::layout::scan::scan_builder::ScanBuilder; +use vortex::layout::scan::split_by::SplitBy; use vortex::metrics::Label; use vortex::metrics::MetricsRegistry; use vortex::session::VortexSession; @@ -88,6 +90,8 @@ pub(crate) struct VortexOpener { /// To save on the overhead of reparsing FlatBuffers and rebuilding the layout tree, we cache /// a file reader the first time we read a file. pub layout_readers: Arc>>, + /// Shared full-file natural split ranges keyed by file path. + pub natural_split_ranges: Arc]>>>, /// Whether the query has output ordering specified pub has_output_ordering: bool, @@ -123,6 +127,7 @@ impl FileOpener for VortexOpener { let batch_size = self.batch_size; let limit = self.limit; let layout_reader = Arc::clone(&self.layout_readers); + let natural_split_ranges = Arc::clone(&self.natural_split_ranges); let has_output_ordering = self.has_output_ordering; let scan_concurrency = self.scan_concurrency; @@ -130,7 +135,7 @@ impl FileOpener for VortexOpener { let projection_pushdown = self.projection_pushdown; // Replace column access for partition columns with literals - #[allow(clippy::disallowed_types)] + #[expect(clippy::disallowed_types)] let literal_value_cols = self .table_schema .table_partition_cols() @@ -302,7 +307,7 @@ impl FileOpener for VortexOpener { } }; - let mut scan_builder = ScanBuilder::new(session.clone(), layout_reader); + let mut scan_builder = ScanBuilder::new(session.clone(), Arc::clone(&layout_reader)); if let Some(extensions) = file.extensions && let Some(vortex_plan) = extensions.downcast_ref::() @@ -311,12 +316,31 @@ impl FileOpener for VortexOpener { } if let Some(file_range) = file.range { - scan_builder = apply_byte_range( - file_range, - file.object_meta.size, - vxf.row_count(), - scan_builder, - ); + let byte_range = Range { + start: u64::try_from(file_range.start) + .map_err(|_| exec_datafusion_err!("Vortex file range start is negative"))?, + end: u64::try_from(file_range.end) + .map_err(|_| exec_datafusion_err!("Vortex file range end is negative"))?, + }; + if byte_range.start != 0 || byte_range.end != file.object_meta.size { + // Full-file scans already cover every natural split. Only translate the + // byte range back into row boundaries when DataFusion has trimmed the file. + let natural_split_ranges = natural_split_ranges_for_file( + natural_split_ranges.as_ref(), + &file.object_meta.location, + &layout_reader, + )?; + + let Some(row_range) = split_aligned_row_range( + byte_range, + file.object_meta.size, + natural_split_ranges.as_ref(), + ) else { + return Ok(stream::empty().boxed()); + }; + + scan_builder = scan_builder.with_row_range(row_range); + } } let filter = filter @@ -421,33 +445,94 @@ impl FileOpener for VortexOpener { } } -/// If the file has a [`FileRange`], we translate it into a row range in the file for the scan. -fn apply_byte_range( - file_range: FileRange, - total_size: u64, - row_count: u64, - scan_builder: ScanBuilder, -) -> ScanBuilder { - let row_range = byte_range_to_row_range( - file_range.start as u64..file_range.end as u64, - row_count, - total_size, - ); - - scan_builder.with_row_range(row_range) +fn natural_split_ranges_for_file( + natural_split_ranges: &DashMap]>>, + path: &Path, + layout_reader: &Arc, +) -> DFResult]>> { + if let Some(split_ranges) = natural_split_ranges.get(path) { + return Ok(Arc::clone(split_ranges.value())); + } + + let split_ranges = compute_natural_split_ranges(layout_reader.as_ref())?; + + match natural_split_ranges.entry(path.clone()) { + Entry::Occupied(entry) => Ok(Arc::clone(entry.get())), + Entry::Vacant(entry) => { + entry.insert(Arc::clone(&split_ranges)); + Ok(split_ranges) + } + } } -fn byte_range_to_row_range(byte_range: Range, row_count: u64, total_size: u64) -> Range { - debug_assert!(row_count > 0); // Asserted by an early exit check in VortexOpener::open +fn compute_natural_split_ranges(layout_reader: &dyn LayoutReader) -> DFResult]>> { + let row_count = layout_reader.row_count(); + let row_range = 0..row_count; + let split_points: Vec<_> = SplitBy::Layout + .splits(layout_reader, &row_range, &[FieldMask::All]) + .map_err(|e| exec_datafusion_err!("Failed to compute Vortex natural splits: {e}"))? + .into_iter() + .tuple_windows() + .map(|(s, e)| s..e) + .collect::>(); + + Ok(split_points.into()) +} + +/// Translate a DataFusion byte range to the contiguous natural split ranges it owns. +/// Most splits are assigned by midpoint, but the leading split stays with the range that owns +/// byte 0 so a tiny first byte range still claims the first rows. +fn split_aligned_row_range( + byte_range: Range, + total_size: u64, + split_ranges: &[Range], +) -> Option> { + if byte_range.start >= byte_range.end { + return None; + } + + let row_count = split_ranges.last().map(|split| split.end)?; + if row_count == 0 { + return None; + } + + let mut owned_splits = split_ranges + .iter() + .enumerate() + .filter_map(|(idx, split_range)| { + let assignment_byte = split_assignment_byte(idx, split_range, row_count, total_size); + byte_range.contains(&assignment_byte).then_some(split_range) + }); + + let first_split = owned_splits.next()?; + let mut row_range = first_split.start..first_split.end; + for split_range in owned_splits { + row_range.end = split_range.end; + } + + Some(row_range) +} - let average_row = total_size / row_count; - assert!(average_row > 0, "A row must always have at least one byte"); +fn split_assignment_byte( + idx: usize, + split_range: &Range, + row_count: u64, + total_size: u64, +) -> u64 { + if idx == 0 && split_range.start == 0 { + // Byte 0 is the only stable representative for the leading split. A midpoint can fall + // into the next DataFusion byte range and leave the first range with no rows to read. + 0 + } else { + split_midpoint_to_byte(split_range, row_count, total_size) + } +} - let start_row = byte_range.start / average_row; - let end_row = byte_range.end / average_row; +fn split_midpoint_to_byte(split_range: &Range, row_count: u64, total_size: u64) -> u64 { + let midpoint_row = split_range.start + (split_range.end - split_range.start) / 2; + let midpoint_byte = (u128::from(midpoint_row) * u128::from(total_size)) / u128::from(row_count); - // We take the min here as `end_row` might overshoot - start_row..u64::min(row_count, end_row) + u64::try_from(midpoint_byte).vortex_expect("midpoint byte projection should fit into u64") } #[cfg(test)] @@ -500,43 +585,57 @@ mod tests { static SESSION: LazyLock = LazyLock::new(VortexSession::default); #[rstest] - #[case(0..100, 100, 100, 0..100)] - #[case(0..105, 100, 105, 0..100)] - #[case(0..50, 100, 105, 0..50)] - #[case(50..105, 100, 105, 50..100)] - #[case(0..1, 4, 8, 0..0)] - #[case(1..8, 4, 8, 0..4)] - fn test_range_translation( + #[case(0..3, 10, vec![0..2, 2..5, 5..10], Some(0..2))] + #[case(3..7, 10, vec![0..2, 2..5, 5..10], Some(2..5))] + #[case(1..8, 10, vec![0..1, 1..9, 9..10], Some(1..9))] + #[case(1..4, 16, vec![0..1, 1..2, 2..3, 3..4], None)] + #[case(0..1, 10, vec![0..2, 2..10], Some(0..2))] + fn test_split_aligned_row_range( #[case] byte_range: Range, - #[case] row_count: u64, #[case] total_size: u64, - #[case] expected: Range, + #[case] split_ranges: Vec>, + #[case] expected: Option>, ) { assert_eq!( - byte_range_to_row_range(byte_range, row_count, total_size), + split_aligned_row_range(byte_range, total_size, &split_ranges), expected ); } #[test] - fn test_consecutive_ranges() { - let row_count = 100; - let total_size = 429; - let bytes_a = 0..143; - let bytes_b = 143..286; - let bytes_c = 286..429; - - let rows_a = byte_range_to_row_range(bytes_a, row_count, total_size); - let rows_b = byte_range_to_row_range(bytes_b, row_count, total_size); - let rows_c = byte_range_to_row_range(bytes_c, row_count, total_size); - - assert_eq!(rows_a.end - rows_a.start, 35); - assert_eq!(rows_b.end - rows_b.start, 36); - assert_eq!(rows_c.end - rows_c.start, 29); - - assert_eq!(rows_a.start, 0); - assert_eq!(rows_c.end, 100); - for (left, right) in [rows_a, rows_b, rows_c].iter().tuple_windows() { + fn test_split_aligned_ranges_cover_splits_exactly_once() { + let split_ranges = vec![0..1, 1..4, 4..10, 10..13]; + let byte_ranges = [0..4, 4..8, 8..12, 12..16]; + + let assigned = byte_ranges + .into_iter() + .filter_map(|byte_range| split_aligned_row_range(byte_range, 16, &split_ranges)) + .collect::>(); + + assert_eq!(assigned, vec![0..4, 4..10, 10..13]); + assert_eq!( + assigned + .iter() + .map(|range| range.end - range.start) + .sum::(), + 13 + ); + + let split_starts = split_ranges + .iter() + .map(|range| range.start) + .collect::>(); + let split_ends = split_ranges + .iter() + .map(|range| range.end) + .collect::>(); + + for range in &assigned { + assert!(split_starts.contains(&range.start)); + assert!(split_ends.contains(&range.end)); + } + + for (left, right) in assigned.iter().tuple_windows() { assert_eq!(left.end, right.start); } } @@ -577,6 +676,7 @@ mod tests { limit: None, metrics_registry: Arc::new(DefaultMetricsRegistry::default()), layout_readers: Default::default(), + natural_split_ranges: Default::default(), has_output_ordering: false, expression_convertor: Arc::new(DefaultExpressionConvertor::default()), file_metadata_cache: None, @@ -650,7 +750,7 @@ mod tests { let file_schema = data_batch.schema(); // Parallel scans may attach a byte range even for empty files; the - // opener must not call byte_range_to_row_range when the row_count is 0. + // opener must return early before attempting split-aligned translation. let file = PartitionedFile::new_with_range(file_path.to_string(), file_size, 0, file_size as i64); @@ -708,6 +808,7 @@ mod tests { limit: None, metrics_registry: Arc::new(DefaultMetricsRegistry::default()), layout_readers: Default::default(), + natural_split_ranges: Default::default(), has_output_ordering: false, expression_convertor: Arc::new(DefaultExpressionConvertor::default()), file_metadata_cache: None, @@ -794,6 +895,7 @@ mod tests { limit: None, metrics_registry: Arc::new(DefaultMetricsRegistry::default()), layout_readers: Default::default(), + natural_split_ranges: Default::default(), has_output_ordering: false, expression_convertor: Arc::new(DefaultExpressionConvertor::default()), file_metadata_cache: None, @@ -950,6 +1052,7 @@ mod tests { limit: None, metrics_registry: Arc::new(DefaultMetricsRegistry::default()), layout_readers: Default::default(), + natural_split_ranges: Default::default(), has_output_ordering: false, expression_convertor: Arc::new(DefaultExpressionConvertor::default()), file_metadata_cache: None, @@ -1009,6 +1112,7 @@ mod tests { limit: None, metrics_registry: Arc::new(DefaultMetricsRegistry::default()), layout_readers: Default::default(), + natural_split_ranges: Default::default(), has_output_ordering: false, expression_convertor: Arc::new(DefaultExpressionConvertor::default()), file_metadata_cache: None, @@ -1212,6 +1316,7 @@ mod tests { limit: None, metrics_registry: Arc::new(DefaultMetricsRegistry::default()), layout_readers: Default::default(), + natural_split_ranges: Default::default(), has_output_ordering: false, expression_convertor: Arc::new(DefaultExpressionConvertor::default()), file_metadata_cache: None, diff --git a/vortex-datafusion/src/persistent/reader.rs b/vortex-datafusion/src/persistent/reader.rs index 34ff646722c..de221c28ca7 100644 --- a/vortex-datafusion/src/persistent/reader.rs +++ b/vortex-datafusion/src/persistent/reader.rs @@ -16,7 +16,16 @@ use vortex::io::object_store::ObjectStoreReadAt; use vortex::io::session::RuntimeSessionExt; use vortex::session::VortexSession; -/// Factory to create [`VortexReadAt`] instances to read the target file. +/// Factory to create [`VortexReadAt`] instances for a `PartitionedFile`. +/// +/// Plug a custom implementation into [`VortexSource::with_vortex_reader_factory`] +/// when the default object-store reader is not sufficient, for example to: +/// +/// - reuse an application-level metadata cache, +/// - wrap reads with custom authentication or routing, +/// - coalesce I/O in a remote storage layer. +/// +/// [`VortexSource::with_vortex_reader_factory`]: crate::VortexSource::with_vortex_reader_factory pub trait VortexReaderFactory: Debug + Send + Sync + 'static { /// Create a reader for a target object. fn create_reader( @@ -26,15 +35,30 @@ pub trait VortexReaderFactory: Debug + Send + Sync + 'static { ) -> DFResult>; } -/// Default factory, creates [`ObjectStore`] backed readers for files, -/// works with multiple cloud providers. +/// Default [`VortexReaderFactory`] backed by DataFusion's [`ObjectStore`]. +/// +/// This is the reader used by [`crate::VortexSource`] and +/// [`crate::VortexFormat`] unless a +/// custom factory is supplied. It works with any object store that DataFusion +/// has registered for the scan. #[derive(Debug)] pub struct DefaultVortexReaderFactory { object_store: Arc, } impl DefaultVortexReaderFactory { - /// Creates new instance + /// Creates a new factory from an [`ObjectStore`]. + /// + /// # Example + /// + /// ```rust + /// # use std::sync::Arc; + /// # use object_store::memory::InMemory; + /// use vortex_datafusion::reader::DefaultVortexReaderFactory; + /// + /// let factory = DefaultVortexReaderFactory::new(Arc::new(InMemory::new())); + /// # let _ = factory; + /// ``` pub fn new(object_store: Arc) -> Self { Self { object_store } } diff --git a/vortex-datafusion/src/persistent/source.rs b/vortex-datafusion/src/persistent/source.rs index 537e6162d0b..aed2024d5d9 100644 --- a/vortex-datafusion/src/persistent/source.rs +++ b/vortex-datafusion/src/persistent/source.rs @@ -3,6 +3,7 @@ use std::any::Any; use std::fmt::Formatter; +use std::ops::Range; use std::sync::Arc; use std::sync::Weak; @@ -41,9 +42,133 @@ use crate::convert::exprs::ExpressionConvertor; use crate::persistent::reader::DefaultVortexReaderFactory; use crate::persistent::reader::VortexReaderFactory; -/// Execution plan for reading one or more Vortex files, intended to be consumed by [`DataSourceExec`]. +/// File scan implementation for reading one or more `.vortex` files. /// +/// `VortexSource` is the lower-level read component underneath +/// [`VortexFormat`]. It is the type DataFusion stores in a [`FileScanConfig`], +/// and it is ultimately executed through [`DataSourceExec`]. +/// +/// ```text +/// ▲ +/// │ +/// │ Produce a stream of +/// │ RecordBatches +/// │ +/// ┌───────────────────────┐ +/// │ DataSourceExec │ +/// └───────────────────────┘ +/// ▲ +/// │ uses +/// │ +/// ┌───────────────────────┐ +/// │ VortexSource │ +/// └───────────────────────┘ +/// ▲ +/// │ opens `.vortex` files via +/// │ +/// ObjectStore / VortexReadAt +/// ``` +/// +/// Most applications reach `VortexSource` indirectly through +/// [`VortexFormatFactory`]. Use `VortexSource` directly when you are +/// constructing a `FileScanConfig` yourself or when you need to inject +/// lower-level behavior such as a custom [`VortexReaderFactory`], an external +/// [`VortexAccessPlan`], or a specific [`FileMetadataCache`]. +/// +/// # Example +/// +/// ```rust +/// use std::sync::Arc; +/// +/// use arrow_schema::Schema; +/// use datafusion_datasource::file_scan_config::FileScanConfigBuilder; +/// use datafusion_datasource::source::DataSourceExec; +/// use datafusion_datasource::PartitionedFile; +/// use datafusion_datasource::TableSchema; +/// use datafusion_execution::object_store::ObjectStoreUrl; +/// use vortex::VortexSessionDefault; +/// use vortex::session::VortexSession; +/// use vortex_datafusion::VortexSource; +/// +/// let file_schema = Arc::new(Schema::empty()); +/// let source = Arc::new( +/// VortexSource::new( +/// TableSchema::from_file_schema(file_schema), +/// VortexSession::default(), +/// ) +/// .with_projection_pushdown(true) +/// .with_scan_concurrency(4), +/// ); +/// +/// let config = FileScanConfigBuilder::new(ObjectStoreUrl::local_filesystem(), source) +/// .with_file(PartitionedFile::new("metrics.vortex", 1024)) +/// .build(); +/// +/// let exec = DataSourceExec::from_data_source(config); +/// # let _ = exec; +/// ``` +/// +/// # What `VortexSource` Handles +/// +/// `VortexSource` is responsible for: +/// +/// - translating DataFusion filters into Vortex predicates when possible, +/// - retaining the full predicate for file pruning based on statistics and +/// partition values, +/// - configuring per-file readers and sharing parsed layout readers across +/// partitions within the same scan, +/// - carrying the table schema used for schema evolution and missing-column +/// adaptation, +/// - attaching a Vortex metrics registry to the read path. +/// +/// # Projection And Predicate Behavior +/// +/// `VortexSource` keeps two related predicate forms: +/// +/// - `full_predicate`, which is used by DataFusion's `FilePruner` to skip whole +/// files before they are opened, +/// - `vortex_predicate`, which contains only the expressions Vortex can evaluate +/// during the scan. +/// +/// Projection handling depends on +/// [`VortexTableOptions::projection_pushdown`]: +/// +/// - when disabled, `VortexSource` still prunes unreferenced top-level columns, +/// but DataFusion applies the full projection after the scan, +/// - when enabled, the scan can evaluate a Vortex-native projection and leave +/// only unsupported expressions for DataFusion. +/// +/// # Observability +/// +/// `VortexSource` owns a Vortex metrics registry for the lifetime of a physical +/// scan. The registry is passed to the reader and scan builder so I/O and scan +/// metrics accumulate as the query executes. +/// +/// Use [`VortexMetricsFinder`] to merge those metrics back into DataFusion +/// `MetricsSet` values after the plan has run. +/// +/// # Execution Flow +/// +/// At execution time: +/// +/// 1. DataFusion calls `DataSourceExec`, which delegates file opening to +/// `VortexSource`. +/// 2. `VortexSource` creates a `VortexOpener` configured with the current +/// projection, predicate, options, and metrics. +/// 3. The opener adapts filters and schema for the specific file, applies any +/// [`VortexAccessPlan`], and builds a Vortex scan. +/// 4. Scan results are converted into Arrow `RecordBatch` values for +/// DataFusion. +/// +/// [`VortexFormat`]: crate::VortexFormat +/// [`FileScanConfig`]: datafusion_datasource::file_scan_config::FileScanConfig /// [`DataSourceExec`]: datafusion_datasource::source::DataSourceExec +/// [`VortexFormatFactory`]: crate::VortexFormatFactory +/// [`VortexReaderFactory`]: crate::reader::VortexReaderFactory +/// [`VortexAccessPlan`]: crate::VortexAccessPlan +/// [`FileMetadataCache`]: datafusion_execution::cache::cache_manager::FileMetadataCache +/// [`VortexTableOptions::projection_pushdown`]: crate::VortexTableOptions::projection_pushdown +/// [`VortexMetricsFinder`]: crate::metrics::VortexMetricsFinder #[derive(Clone)] pub struct VortexSource { pub(crate) session: VortexSession, @@ -61,6 +186,8 @@ pub struct VortexSource { /// /// Sharing the readers allows us to only read every layout once from the file, even across partitions. layout_readers: Arc>>, + /// Shared full-file natural split ranges keyed by path. + natural_split_ranges: Arc]>>>, expression_convertor: Arc, pub(crate) vortex_reader_factory: Option>, vx_metrics_registry: Arc, @@ -70,10 +197,14 @@ pub struct VortexSource { } impl VortexSource { - /// Creates a new VortexSource with default configuration and a provided [`VortexSession`]. - /// Meant to be use with a [`FileScanConfig`] to scan a file with the provided schema. + /// Creates a new `VortexSource` for a table schema and [`VortexSession`]. + /// + /// The new source starts with: /// - /// Can be configured using the provided methods. + /// - all top-level columns projected, + /// - no pushed filters, + /// - a default Vortex metrics registry, + /// - default [`VortexTableOptions`]. pub fn new(table_schema: TableSchema, session: VortexSession) -> Self { let full_schema = table_schema.table_schema(); let indices = (0..full_schema.fields().len()).collect::>(); @@ -88,6 +219,7 @@ impl VortexSource { batch_size: None, _unused_df_metrics: Default::default(), layout_readers: Arc::new(DashMap::default()), + natural_split_ranges: Arc::new(DashMap::default()), expression_convertor: Arc::new(DefaultExpressionConvertor::default()), vortex_reader_factory: None, vx_metrics_registry: Arc::new(DefaultMetricsRegistry::default()), @@ -96,13 +228,21 @@ impl VortexSource { } } - /// Enable or disable expression pushdown into the underlying Vortex scan. + /// Enables or disables Vortex-native projection evaluation. + /// + /// This toggles whether `VortexSource` tries to split DataFusion projection + /// expressions into a Vortex scan projection plus a leftover DataFusion + /// projection. pub fn with_projection_pushdown(mut self, enabled: bool) -> Self { self.options.projection_pushdown = enabled; self } - /// Set a [`ExpressionConvertor`] to control how Datafusion expression should be converted and pushed down. + /// Sets the [`ExpressionConvertor`] used to translate DataFusion expressions + /// into Vortex expressions. + /// + /// Override this when the default converter is insufficient for an engine + /// integration or for a custom schema-adaptation strategy. pub fn with_expression_convertor( mut self, expr_convertor: Arc, @@ -111,7 +251,10 @@ impl VortexSource { self } - /// Set a user-defined factory to create the underlying [`VortexReadAt`] + /// Sets a custom factory for the underlying [`VortexReadAt`]. + /// + /// Use this when reads need to go through an application-specific layer + /// rather than the default DataFusion [`ObjectStore`]. /// /// [`VortexReadAt`]: vortex::io::VortexReadAt pub fn with_vortex_reader_factory( @@ -122,12 +265,16 @@ impl VortexSource { self } - /// Returns the [`MetricsRegistry`] attached to this source. + /// Returns the [`MetricsRegistry`] attached to this scan. + /// + /// The registry is populated as files are opened and scanned. In most + /// callers, [`crate::metrics::VortexMetricsFinder`] is the more convenient + /// public API for turning the registry contents into DataFusion metrics. pub fn metrics_registry(&self) -> &Arc { &self.vx_metrics_registry } - /// Override the file metadata cache + /// Overrides the metadata cache used to reuse Vortex footers across scans. pub fn with_file_metadata_cache( mut self, file_metadata_cache: Arc, @@ -136,31 +283,31 @@ impl VortexSource { self } - /// Set the underlying scan concurrency. This limit is used per Vortex scan operations. + /// Sets the per-file Vortex scan concurrency. + /// + /// This is separate from DataFusion's partition-level parallelism. pub fn with_scan_concurrency(mut self, scan_concurrency: usize) -> Self { self.options.scan_concurrency = Some(scan_concurrency); self } - /// Returns the table options for this source. + /// Returns the effective table options for this source. pub fn options(&self) -> &VortexTableOptions { &self.options } - /// Set the table options for this source. + /// Replaces the table options for this source. pub fn with_options(mut self, opts: VortexTableOptions) -> Self { self.options = opts; self } -} -impl FileSource for VortexSource { - fn create_file_opener( + fn create_vortex_opener( &self, object_store: Arc, base_config: &FileScanConfig, partition: usize, - ) -> DFResult> { + ) -> DFResult { let batch_size = self .batch_size .vortex_expect("batch_size must be supplied to VortexSource"); @@ -188,14 +335,30 @@ impl FileSource for VortexSource { limit: base_config.limit.map(|l| l as u64), metrics_registry: Arc::clone(&self.vx_metrics_registry), layout_readers: Arc::clone(&self.layout_readers), + natural_split_ranges: Arc::clone(&self.natural_split_ranges), has_output_ordering: !base_config.output_ordering.is_empty(), - expression_convertor: Arc::new(DefaultExpressionConvertor::default()), + expression_convertor: Arc::clone(&self.expression_convertor), file_metadata_cache: self.file_metadata_cache.clone(), projection_pushdown: self.options.projection_pushdown, scan_concurrency: self.options.scan_concurrency, }; - Ok(Arc::new(opener)) + Ok(opener) + } +} + +impl FileSource for VortexSource { + fn create_file_opener( + &self, + object_store: Arc, + base_config: &FileScanConfig, + partition: usize, + ) -> DFResult> { + Ok(Arc::new(self.create_vortex_opener( + object_store, + base_config, + partition, + )?)) } fn as_any(&self) -> &dyn Any { @@ -223,13 +386,13 @@ impl FileSource for VortexSource { fn fmt_extra(&self, t: DisplayFormatType, f: &mut Formatter) -> std::fmt::Result { match t { DisplayFormatType::Default | DisplayFormatType::Verbose => { - if let Some(ref predicate) = self.vortex_predicate { + if let Some(predicate) = &self.vortex_predicate { write!(f, ", predicate: {predicate}")?; } } // Use TreeRender style key=value formatting to display the predicate DisplayFormatType::TreeRender => { - if let Some(ref predicate) = self.vortex_predicate { + if let Some(predicate) = &self.vortex_predicate { writeln!(f, "predicate={}", fmt_sql(predicate.as_ref()))?; }; } @@ -327,3 +490,83 @@ impl FileSource for VortexSource { &self.table_schema } } + +#[cfg(test)] +mod tests { + use arrow_schema::DataType; + use arrow_schema::Field; + use arrow_schema::Schema; + use datafusion_datasource::file_scan_config::FileScanConfigBuilder; + use datafusion_execution::object_store::ObjectStoreUrl; + use object_store::memory::InMemory; + use vortex::VortexSessionDefault; + + use super::*; + use crate::convert::exprs::ProcessedProjection; + + struct TrackingExpressionConvertor { + inner: DefaultExpressionConvertor, + } + + impl ExpressionConvertor for TrackingExpressionConvertor { + fn can_be_pushed_down(&self, expr: &PhysicalExprRef, schema: &Schema) -> bool { + self.inner.can_be_pushed_down(expr, schema) + } + + fn convert(&self, expr: &dyn PhysicalExpr) -> DFResult { + self.inner.convert(expr) + } + + fn split_projection( + &self, + source_projection: ProjectionExprs, + input_schema: &Schema, + output_schema: &Schema, + ) -> DFResult { + self.inner + .split_projection(source_projection, input_schema, output_schema) + } + + fn no_pushdown_projection( + &self, + source_projection: ProjectionExprs, + input_schema: &Schema, + ) -> DFResult { + self.inner + .no_pushdown_projection(source_projection, input_schema) + } + } + + #[test] + fn create_vortex_opener_preserves_expression_convertor() -> anyhow::Result<()> { + let file_schema = Arc::new(Schema::new(vec![Field::new("a", DataType::Int32, false)])); + let expression_convertor = Arc::new(TrackingExpressionConvertor { + inner: DefaultExpressionConvertor::default(), + }) as Arc; + + let mut source = VortexSource::new( + TableSchema::from_file_schema(file_schema), + VortexSession::default(), + ) + .with_expression_convertor(Arc::clone(&expression_convertor)); + source.batch_size = Some(100); + + let config = FileScanConfigBuilder::new( + ObjectStoreUrl::local_filesystem(), + Arc::new(source.clone()), + ) + .build(); + + let opener = source.create_vortex_opener( + Arc::new(InMemory::new()) as Arc, + &config, + 0, + )?; + + assert!(Arc::ptr_eq( + &opener.expression_convertor, + &expression_convertor + )); + Ok(()) + } +} diff --git a/vortex-datafusion/src/persistent/tests.rs b/vortex-datafusion/src/persistent/tests.rs new file mode 100644 index 00000000000..194184e904e --- /dev/null +++ b/vortex-datafusion/src/persistent/tests.rs @@ -0,0 +1,434 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +use std::sync::Arc; + +use anyhow::anyhow; +use datafusion::arrow::array::Int32Array; +use datafusion::arrow::util::pretty::pretty_format_batches; +use datafusion::datasource::provider::DefaultTableFactory; +use datafusion::execution::SessionStateBuilder; +use datafusion::prelude::SessionConfig; +use datafusion::prelude::SessionContext; +use datafusion_common::GetExt; +use datafusion_physical_plan::display::DisplayableExecutionPlan; +use insta::assert_snapshot; +use object_store::ObjectStore; +use object_store::memory::InMemory; +use rstest::rstest; +use vortex::VortexSessionDefault; +use vortex::array::IntoArray; +use vortex::array::arrays::ChunkedArray; +use vortex::array::arrays::StructArray; +use vortex::array::arrays::VarBinArray; +use vortex::array::validity::Validity; +use vortex::buffer::Buffer; +use vortex::buffer::buffer; +use vortex::file::OpenOptionsSessionExt; +use vortex::file::WriteOptionsSessionExt; +use vortex::io::VortexWrite; +use vortex::io::object_store::ObjectStoreReadAt; +use vortex::io::object_store::ObjectStoreWrite; +use vortex::layout::LayoutStrategy; +use vortex::layout::layouts::chunked::writer::ChunkedLayoutStrategy; +use vortex::layout::layouts::flat::writer::FlatLayoutStrategy; +use vortex::layout::layouts::table::TableStrategy; +use vortex::session::VortexSession; + +use crate::VortexFormatFactory; +use crate::common_tests::TestSessionContext; + +fn make_session( + object_store: Arc, + repartition_file_scans: bool, +) -> SessionContext { + let factory = Arc::new(VortexFormatFactory::new()); + + let config = SessionConfig::new() + .with_target_partitions(4) + .with_repartition_file_scans(repartition_file_scans) + .with_repartition_file_min_size(0); + let mut state = SessionStateBuilder::new() + .with_config(config) + .with_default_features() + .with_table_factory( + factory.get_ext().to_uppercase(), + Arc::new(DefaultTableFactory::new()), + ) + .with_object_store(&url::Url::try_from("file://").unwrap(), object_store); + + if let Some(file_formats) = state.file_formats() { + file_formats.push(factory as _); + } + + SessionContext::new_with_state(state.build()).enable_url_table() +} + +async fn count_query_partitions(ctx: &SessionContext, sql: &str) -> anyhow::Result { + let explain = ctx.sql(&format!("EXPLAIN {sql}")).await?.collect().await?; + let plan = pretty_format_batches(&explain)?.to_string(); + let marker = "DataSourceExec: file_groups={"; + let start = plan + .find(marker) + .ok_or_else(|| anyhow!("EXPLAIN plan did not contain a DataSourceExec"))? + + marker.len(); + let partitions = plan[start..] + .chars() + .take_while(|ch| ch.is_ascii_digit()) + .collect::(); + + Ok(partitions.parse()?) +} + +fn batch_values(batches: &[datafusion::arrow::array::RecordBatch]) -> Vec { + let mut values = Vec::with_capacity(batches.iter().map(|batch| batch.num_rows()).sum()); + + for batch in batches { + let array = batch + .column(0) + .as_any() + .downcast_ref::() + .expect("value column should be Int32"); + values.extend(array.values().iter().copied()); + } + + values +} + +#[rstest] +#[tokio::test] +async fn test_query_file(#[values(Some(1), None)] limit: Option) -> anyhow::Result<()> { + let ctx = TestSessionContext::default(); + + let session = VortexSession::default(); + + let strings = ChunkedArray::from_iter([ + VarBinArray::from(vec!["ab", "foo", "bar", "baz"]).into_array(), + VarBinArray::from(vec!["ab", "foo", "bar", "baz"]).into_array(), + ]) + .into_array(); + + let numbers = ChunkedArray::from_iter([ + buffer![1u32, 2, 3, 4].into_array(), + buffer![5u32, 6, 7, 8].into_array(), + ]) + .into_array(); + + let st = StructArray::try_new( + ["strings", "numbers"].into(), + vec![strings, numbers], + 8, + Validity::NonNullable, + )?; + + let mut writer = ObjectStoreWrite::new(Arc::clone(&ctx.store), &"test.vortex".into()).await?; + + let summary = session + .write_options() + .write(&mut writer, st.into_array().to_array_stream()) + .await?; + + writer.shutdown().await?; + + assert_eq!(summary.row_count(), 8); + + let read_row_count = ctx + .session + .sql("SELECT * from '/test.vortex'") + .await? + .limit(0, limit)? + .count() + .await?; + + assert_eq!(read_row_count, limit.unwrap_or(8)); + + Ok(()) +} + +#[tokio::test] +async fn test_addition_pushdown() -> anyhow::Result<()> { + let ctx = TestSessionContext::default(); + dbg!(&ctx.store); + + ctx.session + .sql( + "CREATE EXTERNAL TABLE written_data \ + (a TINYINT NOT NULL) \ + STORED AS vortex \ + LOCATION '/test/'", + ) + .await?; + + ctx.session + .sql("INSERT INTO written_data VALUES (0), (1), (2), (3), (4)") + .await? + .collect() + .await?; + + let result = ctx + .session + .sql("SELECT a, a + 5 as five, a + 6 as six FROM written_data WHERE a + 5 > 7") + .await? + .collect() + .await?; + + assert_snapshot!(pretty_format_batches(&result)?, @r" + +---+------+-----+ + | a | five | six | + +---+------+-----+ + | 3 | 8 | 9 | + | 4 | 9 | 10 | + +---+------+-----+ + "); + + Ok(()) +} + +#[tokio::test] +async fn create_table_ordered_by() -> anyhow::Result<()> { + let ctx = TestSessionContext::default(); + + // Vortex + ctx.session + .sql( + "CREATE EXTERNAL TABLE my_tbl_vx \ + (c1 VARCHAR NOT NULL, c2 INT NOT NULL) \ + STORED AS vortex \ + WITH ORDER (c1 ASC) + LOCATION '/test/'", + ) + .await?; + + ctx.session + .sql("INSERT INTO my_tbl_vx VALUES ('air', 5), ('balloon', 42)") + .await? + .collect() + .await?; + + ctx.session + .sql("INSERT INTO my_tbl_vx VALUES ('zebra', 5)") + .await? + .collect() + .await?; + + ctx.session + .sql("INSERT INTO my_tbl_vx VALUES ('texas', 2000), ('alabama', 2000)") + .await? + .collect() + .await?; + + let df = ctx + .session + .sql("SELECT * FROM my_tbl_vx ORDER BY c1 ASC limit 3") + .await?; + + let physical_plan = ctx + .session + .state() + .create_physical_plan(df.logical_plan()) + .await?; + + insta::assert_snapshot!(DisplayableExecutionPlan::new(physical_plan.as_ref()) + .tree_render().to_string(), @r" + ┌───────────────────────────┐ + │ SortPreservingMergeExec │ + │ -------------------- │ + │ c1 ASC NULLS LAST │ + │ │ + │ limit: 3 │ + └─────────────┬─────────────┘ + ┌─────────────┴─────────────┐ + │ DataSourceExec │ + │ -------------------- │ + │ files: 3 │ + │ format: vortex │ + └───────────────────────────┘ + "); + + let r = df.collect().await?; + + insta::assert_snapshot!(pretty_format_batches(&r)?.to_string(), @r" + +---------+------+ + | c1 | c2 | + +---------+------+ + | air | 5 | + | alabama | 2000 | + | balloon | 42 | + +---------+------+ + "); + + Ok(()) +} + +/// Doc example: demonstrates creating, writing, reading, and filtering a Vortex table. +#[tokio::test] +async fn doc_example() -> anyhow::Result<()> { + // [setup] + use std::sync::Arc; + + use datafusion::datasource::provider::DefaultTableFactory; + use datafusion::execution::SessionStateBuilder; + use datafusion::prelude::SessionContext; + use datafusion_common::GetExt; + use object_store::memory::InMemory; + + use crate::VortexFormatFactory; + + let factory = Arc::new(VortexFormatFactory::new()); + let state = SessionStateBuilder::new() + .with_default_features() + .with_table_factory( + factory.get_ext().to_uppercase(), + Arc::new(DefaultTableFactory::new()), + ) + .with_file_formats(vec![factory]) + .build(); + let ctx = SessionContext::new_with_state(state).enable_url_table(); + // [setup] + + // Register an in-memory object store for the test. + let store = Arc::new(InMemory::new()); + ctx.register_object_store(&url::Url::try_from("file://").unwrap(), store); + + // [create] + ctx.sql( + "CREATE EXTERNAL TABLE my_table \ + (name VARCHAR NOT NULL, age INT NOT NULL) \ + STORED AS vortex \ + LOCATION '/demo/'", + ) + .await?; + // [create] + + // [write] + ctx.sql( + "INSERT INTO my_table VALUES \ + ('Alice', 30), ('Bob', 25), ('Charlie', 35), ('Diana', 28)", + ) + .await? + .collect() + .await?; + // [write] + + // [query] + let result = ctx + .sql("SELECT name, age FROM my_table WHERE age > 28 ORDER BY age") + .await? + .collect() + .await?; + // [query] + + assert_snapshot!(pretty_format_batches(&result)?, @r" + +---------+-----+ + | name | age | + +---------+-----+ + | Alice | 30 | + | Charlie | 35 | + +---------+-----+ + "); + + Ok(()) +} + +#[tokio::test] +async fn test_repartitioned_scan_matches_non_repartitioned_for_uneven_splits() -> anyhow::Result<()> +{ + let store = Arc::new(InMemory::new()) as _; + let session = VortexSession::default(); + let path = object_store::path::Path::parse("/split-aligned-repartition.vortex")?; + + let chunk_1_len = 2_000; + let chunk_2_len = 5_000; + let chunk_3_len = 6_000; + let row_count = chunk_1_len + chunk_2_len + chunk_3_len; + + let chunk_1 = StructArray::try_new( + ["value"].into(), + vec![Buffer::from_iter(0_i32..chunk_1_len).into_array()], + usize::try_from(chunk_1_len)?, + Validity::NonNullable, + )?; + let chunk_2 = StructArray::try_new( + ["value"].into(), + vec![Buffer::from_iter(chunk_1_len..(chunk_1_len + chunk_2_len)).into_array()], + usize::try_from(chunk_2_len)?, + Validity::NonNullable, + )?; + let chunk_3 = StructArray::try_new( + ["value"].into(), + vec![Buffer::from_iter((chunk_1_len + chunk_2_len)..row_count).into_array()], + usize::try_from(chunk_3_len)?, + Validity::NonNullable, + )?; + let table = ChunkedArray::from_iter([ + chunk_1.into_array(), + chunk_2.into_array(), + chunk_3.into_array(), + ]) + .into_array(); + let flat: Arc = Arc::new(FlatLayoutStrategy::default()); + let strategy: Arc = Arc::new(TableStrategy::new( + Arc::clone(&flat), + Arc::new(ChunkedLayoutStrategy::new(FlatLayoutStrategy::default())), + )); + + let mut writer = ObjectStoreWrite::new(Arc::clone(&store), &path).await?; + let summary = session + .write_options() + .with_strategy(strategy) + .write(&mut writer, table.into_array().to_array_stream()) + .await?; + writer.shutdown().await?; + + let reader = Arc::new(ObjectStoreReadAt::new( + Arc::clone(&store), + path.clone(), + vortex::io::runtime::Handle::find().expect("tokio runtime should be available in tests"), + )); + let vxf = session + .open_options() + .with_file_size(summary.size()) + .open_read(reader) + .await?; + let split_ranges = vxf.splits()?; + let split_lengths = split_ranges + .iter() + .map(|range| range.end - range.start) + .collect::>(); + + assert!(split_ranges.len() > 1); + assert!( + split_lengths + .windows(2) + .any(|window| window[0] != window[1]) + ); + + let serial_ctx = make_session(Arc::clone(&store), false); + let repartitioned_ctx = make_session(Arc::clone(&store), true); + let repartitioned_partitions = count_query_partitions( + &repartitioned_ctx, + "SELECT value FROM '/split-aligned-repartition.vortex'", + ) + .await?; + + assert!(repartitioned_partitions > 1); + + let serial = serial_ctx + .sql("SELECT value FROM '/split-aligned-repartition.vortex' ORDER BY value") + .await? + .collect() + .await?; + let repartitioned = repartitioned_ctx + .sql("SELECT value FROM '/split-aligned-repartition.vortex' ORDER BY value") + .await? + .collect() + .await?; + let serial_values = batch_values(&serial); + let repartitioned_values = batch_values(&repartitioned); + let expected = (0_i32..row_count).collect::>(); + + assert_eq!(serial_values, expected); + assert_eq!(repartitioned_values, serial_values); + + Ok(()) +} diff --git a/vortex-datafusion/src/v2/mod.rs b/vortex-datafusion/src/v2/mod.rs index 1e385ca3c85..5c2e6aa542f 100644 --- a/vortex-datafusion/src/v2/mod.rs +++ b/vortex-datafusion/src/v2/mod.rs @@ -1,10 +1,37 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -//! An experimental implementation of the Vortex Scan API for DataFusion. +//! Direct DataFusion integration for an existing Vortex +//! [`DataSourceRef`]. //! -//! This integration directly implements `TableProvider` + `ExecutionPlan`, bypassing DataFusion's -//! `FileFormat` abstraction. +//! Use this module when some other part of the system has already selected the +//! Vortex source to query and DataFusion only needs an adapter around it. +//! +//! Typical flow: +//! +//! 1. Build or obtain a Vortex [`DataSourceRef`]. +//! 2. Wrap it in [`VortexTable`] to register it with a [`SessionContext`], or +//! build a [`VortexDataSource`] directly when constructing a +//! [`DataSourceExec`]. +//! 3. Let DataFusion apply projection, filter, and limit pushdown through the +//! resulting adapter. +//! +//! The two main types are: +//! +//! - [`VortexTable`], the higher-level +//! [`TableProvider`] for `SessionContext::register_table`. +//! - [`VortexDataSource`], the lower-level +//! [`DataSource`] used when constructing physical plans directly. +//! +//! Compared with [`crate::VortexFormatFactory`], this module starts from an +//! already-constructed Vortex source instead of asking DataFusion to discover +//! `.vortex` files. +//! +//! [`DataSourceRef`]: vortex::scan::DataSourceRef +//! [`SessionContext`]: https://docs.rs/datafusion/latest/datafusion/prelude/struct.SessionContext.html +//! [`DataSourceExec`]: datafusion_datasource::source::DataSourceExec +//! [`TableProvider`]: datafusion_catalog::TableProvider +//! [`DataSource`]: datafusion_datasource::source::DataSource mod source; mod table; diff --git a/vortex-datafusion/src/v2/source.rs b/vortex-datafusion/src/v2/source.rs index b2530c445bb..0996222c2b8 100644 --- a/vortex-datafusion/src/v2/source.rs +++ b/vortex-datafusion/src/v2/source.rs @@ -1,16 +1,75 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -//! [`VortexDataSource`] implements DataFusion's [`DataSource`] trait, deferring scan construction -//! to [`DataSource::open`] so that pushed-down filters and limits are included in the -//! [`ScanRequest`]. A single DataFusion partition is used; Vortex handles internal parallelism -//! by driving splits concurrently via [`TryStreamExt::try_flatten_unordered`]. +//! Use [`VortexDataSource`] to adapt an existing Vortex [`DataSourceRef`] into +//! a DataFusion [`DataSource`] without going through file discovery. +//! +//! [`VortexDataSource`] is responsible for: +//! +//! - exposing an Arrow schema and output statistics to DataFusion, +//! - translating DataFusion projection, filter, and limit pushdown into a +//! Vortex [`ScanRequest`], +//! - executing the Vortex scan and converting the results into Arrow +//! `RecordBatch` values. +//! +//! # Example: Create a `DataSourceExec` +//! +//! ```no_run +//! use std::sync::Arc; +//! +//! use arrow_schema::Schema; +//! use datafusion_datasource::source::DataSourceExec; +//! use vortex::VortexSessionDefault; +//! use vortex::scan::DataSourceRef; +//! use vortex::session::VortexSession; +//! use vortex_datafusion::v2::VortexDataSource; +//! +//! # #[tokio::main] +//! # async fn main() -> Result<(), Box> { +//! # let data_source: DataSourceRef = todo!(); +//! let data_source = VortexDataSource::builder(data_source, VortexSession::default()) +//! .with_arrow_schema(Arc::new(Schema::empty())) +//! .build() +//! .await?; +//! +//! let exec = DataSourceExec::from_data_source(data_source); +//! # let _ = exec; +//! # Ok(()) +//! # } +//! ``` +//! +//! # Execution Flow +//! +//! ```text +//! ▲ +//! │ RecordBatch stream +//! │ +//! ┌───────────────────────┐ +//! │ DataSourceExec │ +//! └───────────────────────┘ +//! ▲ +//! │ DataFusion pushdown +//! │ (projection/filter/limit) +//! ┌───────────────────────┐ +//! │ VortexDataSource │ +//! └───────────────────────┘ +//! ▲ +//! │ final ScanRequest +//! ┌───────────────────────┐ +//! │ DataSourceRef │ +//! └───────────────────────┘ +//! ``` +//! +//! Compared with [`crate::VortexSource`], this path starts from an existing +//! Vortex source rather than from DataFusion-managed file discovery. +//! +//! [`DataSource`]: datafusion_datasource::source::DataSource +//! [`DataSourceRef`]: vortex::scan::DataSourceRef +//! [`ScanRequest`]: vortex::scan::ScanRequest use std::any::Any; use std::fmt; use std::fmt::Formatter; -use std::num::NonZero; -use std::num::NonZeroUsize; use std::sync::Arc; use arrow_schema::DataType; @@ -42,7 +101,6 @@ use vortex::array::arrow::ArrowArrayExecutor; use vortex::dtype::DType; use vortex::dtype::FieldPath; use vortex::dtype::Nullability; -use vortex::error::VortexExpect; use vortex::error::VortexResult; use vortex::error::vortex_bail; use vortex::expr::Expression; @@ -56,6 +114,7 @@ use vortex::io::session::RuntimeSessionExt; use vortex::scan::DataSourceRef; use vortex::scan::ScanRequest; use vortex::session::VortexSession; +use vortex_utils::parallelism::get_available_parallelism; use crate::convert::exprs::DefaultExpressionConvertor; use crate::convert::exprs::ExpressionConvertor; @@ -63,7 +122,45 @@ use crate::convert::exprs::ProcessedProjection; use crate::convert::exprs::make_vortex_predicate; use crate::convert::stats::stats_set_to_df; -/// A builder for a [`VortexDataSource`]. +/// Builder for [`VortexDataSource`]. +/// +/// Use the builder to declare how an existing Vortex +/// [`DataSourceRef`] should appear to DataFusion. +/// In particular, it lets you choose: +/// +/// - the Arrow schema DataFusion should see, +/// - an initial top-level projection if the embedding system already knows +/// which columns are needed. +/// +/// The resulting [`VortexDataSource`] is ready to plug into +/// [`DataSourceExec`] or other DataFusion physical planning code. +/// +/// # Example +/// +/// ```no_run +/// use std::sync::Arc; +/// +/// use arrow_schema::Schema; +/// use vortex::VortexSessionDefault; +/// use vortex::scan::DataSourceRef; +/// use vortex::session::VortexSession; +/// use vortex_datafusion::v2::VortexDataSource; +/// +/// # #[tokio::main] +/// # async fn main() -> Result<(), Box> { +/// # let data_source: DataSourceRef = todo!(); +/// let data_source = VortexDataSource::builder(data_source, VortexSession::default()) +/// .with_arrow_schema(Arc::new(Schema::empty())) +/// .with_projection(vec![0]) +/// .build() +/// .await?; +/// # let _ = data_source; +/// # Ok(()) +/// # } +/// ``` +/// +/// [`DataSourceRef`]: vortex::scan::DataSourceRef +/// [`DataSourceExec`]: datafusion_datasource::source::DataSourceExec pub struct VortexDataSourceBuilder { data_source: DataSourceRef, session: VortexSession, @@ -73,8 +170,10 @@ pub struct VortexDataSourceBuilder { } impl VortexDataSourceBuilder { - /// Manually configure an Arrow schema to use when reading from the Vortex source. - /// If not specified, the data source will infer an Arrow schema from the Vortex DType. + /// Sets the Arrow schema exposed to DataFusion. + /// + /// If not specified, the builder derives an Arrow schema from the Vortex + /// dtype. /// /// Note that this schema is not validated against the Vortex DType so any errors will be /// deferred until read time. @@ -83,24 +182,26 @@ impl VortexDataSourceBuilder { self } - /// Configure an initial projection using top-level field indices. + /// Configures an initial top-level projection. + /// + /// This is useful when the embedding system already knows which columns are + /// needed before DataFusion applies its own optimizer pushdown. pub fn with_projection(mut self, indices: Vec) -> Self { self.projection = Some(indices); self } - /// Configure an initial projection using top-level field indices. + /// Like [`Self::with_projection`], but accepts an optional projection. pub fn with_some_projection(mut self, indices: Option>) -> Self { self.projection = indices; self } - /// Build the [`VortexDataSource`]. + /// Builds the [`VortexDataSource`]. /// - /// FIXME(ngates): Note that due to the DataFusion API, this function eagerly resolves - /// statistics for all projected columns. That said.. we only need to do this for aggregation - /// reductions. Any stats used for pruning are handled internally. We could possibly look - /// at the plan ourselves and decide whether there is any need for the stats? + /// The builder eagerly resolves statistics for the initial projection + /// columns because DataFusion expects the `DataSource` to report output + /// statistics before execution begins. pub async fn build(self) -> VortexResult { // The projection expression let mut projection = root(); @@ -174,9 +275,7 @@ impl VortexDataSourceBuilder { filter: None, limit: None, ordered: false, - num_partitions: std::thread::available_parallelism().unwrap_or_else(|_| { - NonZero::new(1).vortex_expect("available parallelism must be non-zero") - }), + num_partitions: get_available_parallelism().unwrap_or(1), }) } } @@ -193,12 +292,22 @@ impl VortexDataSource { } } -/// A DataFusion [`DataSource`] that defers Vortex scan construction to [`open`](DataSource::open). +/// DataFusion [`DataSource`] backed by a Vortex [`DataSourceRef`]. +/// +/// `VortexDataSource` is the core execution adapter for the `v2` integration. +/// It presents DataFusion with a scanable Arrow data source while preserving the +/// underlying Vortex source until execution time. /// -/// Holds a [`DataSourceRef`] rather than pre-collected splits, so that filters and limits pushed -/// down by DataFusion's optimizer are included in the [`ScanRequest`]. A single DataFusion -/// partition is exposed; Vortex drives splits concurrently via -/// [`TryStreamExt::try_flatten_unordered`]. +/// During planning, it reports the current output schema and column statistics. +/// During execution, it builds the final Vortex [`ScanRequest`] from the +/// current projection, pushed filters, ordering hints, and row limit. +/// +/// This integration intentionally reports a single DataFusion output partition. +/// Vortex then handles split-level concurrency internally by polling multiple +/// split streams concurrently. +/// +/// Use [`crate::VortexSource`] instead when DataFusion should discover and plan +/// `.vortex` files on its own. #[derive(Clone)] pub struct VortexDataSource { /// The Vortex data source. @@ -212,7 +321,7 @@ pub struct VortexDataSource { /// The initial Vortex projection expression (e.g. column selection from the builder). initial_projection: Expression, /// Column statistics for the initial projection columns. - #[allow(dead_code)] + #[expect(dead_code)] initial_statistics: Vec, // --- Phase 2: Projected (pushed into the Vortex scan) --- @@ -247,7 +356,7 @@ pub struct VortexDataSource { /// We use this as a hint for how many splits to execute concurrently in `open()`, but we /// always declare to DataFusion that we only have a single partition so that we can /// internally manage concurrency and fix the problem of partition skew. - num_partitions: NonZeroUsize, + num_partitions: usize, } impl fmt::Debug for VortexDataSource { @@ -315,7 +424,7 @@ impl DataSource for VortexDataSource { let handle = session.handle(); let stream = scan_streams - .try_flatten_unordered(Some(num_partitions.get() * 2)) + .try_flatten_unordered(Some(num_partitions * 2)) .map(move |result| { let session = session.clone(); let schema = Arc::clone(&projected_schema); @@ -324,7 +433,7 @@ impl DataSource for VortexDataSource { result.and_then(|chunk| chunk.execute_record_batch(&schema, &mut ctx)) }) }) - .buffered(num_partitions.get()) + .buffered(num_partitions) .map(|result| result.map_err(|e| DataFusionError::External(Box::new(e)))); // Apply leftover projection (expressions that couldn't be pushed into Vortex). @@ -358,7 +467,7 @@ impl DataSource for VortexDataSource { "VortexScanSource: projection={}", self.projected_projection )?; - if let Some(ref filter) = self.filter { + if let Some(filter) = &self.filter { write!(f, ", filter={filter}")?; } if let Some(limit) = self.limit { @@ -375,8 +484,7 @@ impl DataSource for VortexDataSource { ) -> DFResult>> { // Vortex handles parallelism internally — always use a single partition. let mut this = self.clone(); - this.num_partitions = NonZero::new(target_partitions) - .ok_or_else(|| DataFusionError::Internal("non-zero partitions".to_string()))?; + this.num_partitions = target_partitions; this.ordered |= output_ordering.is_some(); Ok(Some(Arc::new(this))) } @@ -392,10 +500,10 @@ impl DataSource for VortexDataSource { fn partition_statistics(&self, _partition: Option) -> DFResult { // FIXME(ngates): this should be adjusted based on filters. See DuckDB for heuristics, // and in the future, store the selectivity stats in the session. - let num_rows = estimate_to_df_precision(&self.data_source.row_count()); + let num_rows = estimate_to_df_precision(self.data_source.row_count().as_ref()); // FIXME(ngates): byte size should be adjusted for the initial projection... - let total_byte_size = estimate_to_df_precision(&self.data_source.byte_size()); + let total_byte_size = estimate_to_df_precision(self.data_source.byte_size().as_ref()); // Column statistics must match the output schema (leftover_schema), which may differ // from the initial schema after try_swapping_with_projection adds computed columns. @@ -542,8 +650,11 @@ impl DataSource for VortexDataSource { } } -/// Convert a Vortex [`Option`] to a DataFusion [`Precision`](DFPrecision). -fn estimate_to_df_precision(est: &Option>) -> DFPrecision { +/// Convert a Vortex [`Option`] to a DataFusion +/// [`DataFusionPrecision`]. +/// +/// [`DataFusionPrecision`]: datafusion_common::stats::Precision +fn estimate_to_df_precision(est: Option<&Precision>) -> DFPrecision { match est { Some(Precision::Exact(v)) => DFPrecision::Exact(usize::try_from(*v).unwrap_or(usize::MAX)), Some(Precision::Inexact(v)) => { diff --git a/vortex-datafusion/src/v2/table.rs b/vortex-datafusion/src/v2/table.rs index 05edc0f27d8..5ca6829fdb3 100644 --- a/vortex-datafusion/src/v2/table.rs +++ b/vortex-datafusion/src/v2/table.rs @@ -1,8 +1,11 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -//! [`VortexTable`] implements DataFusion's [`TableProvider`] trait, providing a direct -//! integration between a Vortex [`DataSource`] and DataFusion's query engine. +//! [`VortexTable`] adapts a Vortex [`DataSourceRef`] into a DataFusion +//! [`TableProvider`]. +//! +//! [`DataSourceRef`]: vortex::scan::DataSourceRef +//! [`TableProvider`]: datafusion_catalog::TableProvider use std::any::Any; use std::fmt; @@ -26,7 +29,49 @@ use vortex::session::VortexSession; use crate::v2::source::VortexDataSource; -/// A DataFusion [`TableProvider`] backed by a Vortex [`DataSourceRef`]. +/// DataFusion [`TableProvider`] backed by a Vortex +/// [`DataSourceRef`]. +/// +/// `VortexTable` is the usual entry point into [`crate::v2`] when you want to +/// register an existing Vortex source with DataFusion. +/// +/// Use it when another part of the system has already built a Vortex source and +/// you want to expose that source through a +/// [`SessionContext`]. +/// +/// `VortexTable` handles the `TableProvider` side of the integration: +/// +/// - it exposes the table schema and coarse statistics to DataFusion, +/// - it seeds the initial top-level projection during `scan`, +/// - it hands execution off to [`VortexDataSource`] for later pushdown and +/// execution. +/// +/// # Example +/// +/// ```no_run +/// use std::sync::Arc; +/// +/// use arrow_schema::Schema; +/// use datafusion::prelude::SessionContext; +/// use vortex::VortexSessionDefault; +/// use vortex::scan::DataSourceRef; +/// use vortex::session::VortexSession; +/// use vortex_datafusion::v2::VortexTable; +/// +/// # let data_source: DataSourceRef = todo!(); +/// let table = Arc::new(VortexTable::new( +/// data_source, +/// VortexSession::default(), +/// Arc::new(Schema::empty()), +/// )); +/// +/// let ctx = SessionContext::new(); +/// ctx.register_table("vortex_data", table)?; +/// # Ok::<(), datafusion_common::DataFusionError>(()) +/// ``` +/// +/// [`DataSourceRef`]: vortex::scan::DataSourceRef +/// [`SessionContext`]: https://docs.rs/datafusion/latest/datafusion/prelude/struct.SessionContext.html pub struct VortexTable { data_source: DataSourceRef, session: VortexSession, @@ -44,8 +89,8 @@ impl fmt::Debug for VortexTable { impl VortexTable { /// Creates a new [`VortexTable`] from a Vortex data source and session. /// - /// The Arrow schema will be used to emit the correct column names and types to DataFusion. - /// The Vortex DType of the data source should be compatible with this Arrow schema. + /// The Arrow schema is the schema DataFusion will observe for this table. + /// It should be compatible with the Vortex dtype exposed by `data_source`. pub fn new( data_source: DataSourceRef, session: VortexSession, diff --git a/vortex-duckdb/README.md b/vortex-duckdb/README.md index 416bbac7fcb..9230e2420be 100644 --- a/vortex-duckdb/README.md +++ b/vortex-duckdb/README.md @@ -64,8 +64,8 @@ By default, our tests use a precompiled build which means you don't get an 2. If there is an api difference between duckdb-vortex's duckdb submodule and vortex's vortex-duckdb/duckdb submodule, checkout duckdb-vortex to previous - commit. For example, if duckdb-vortex's HEAD uses 1.5 API but vortex's HEAD - uses 1.4.2, checkout duckdb-vortex at 8a41ee6ebd9. + commit. For example, if duckdb-vortex's HEAD uses 1.6 API but vortex's HEAD + uses 1.5.2, checkout duckdb-vortex at 8a41ee6ebd9. 3. Update duckdb-vortex's submodules. Replace vortex/ submodule by a softlink to your local vortex repository. diff --git a/vortex-duckdb/build.rs b/vortex-duckdb/build.rs index a96bd1cbaca..e435e9ba063 100644 --- a/vortex-duckdb/build.rs +++ b/vortex-duckdb/build.rs @@ -1,9 +1,9 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] +#![expect(clippy::unwrap_used)] // exit(1) + cargo:error= doesn't provide a double-traceback like panic!() -#![allow(clippy::exit)] +#![expect(clippy::exit)] use std::env; use std::fs; @@ -20,7 +20,7 @@ const DUCKDB_SOURCE_COMMIT_URL: &str = "https://github.com/duckdb/duckdb/archive const BUILD_ARTIFACTS: [&str; 3] = ["libduckdb.dylib", "libduckdb.so", "libduckdb_static.a"]; -const SOURCE_FILES: [&str; 18] = [ +const SOURCE_FILES: [&str; 17] = [ "cpp/client_context.cpp", "cpp/config.cpp", "cpp/copy_function.cpp", @@ -30,7 +30,6 @@ const SOURCE_FILES: [&str; 18] = [ "cpp/expr.cpp", "cpp/file_system.cpp", "cpp/logical_type.cpp", - "cpp/object_cache.cpp", "cpp/replacement_scan.cpp", "cpp/reusable_dict.cpp", "cpp/scalar_function.cpp", @@ -383,7 +382,7 @@ fn main() { // e.g. reordering fields in C++ structs. let version = env::var("DUCKDB_VERSION") // You can also change this version to a commit hash - .unwrap_or_else(|_| "1.5.0".to_owned()); + .unwrap_or_else(|_| "1.5.2".to_owned()); let version = DuckDBVersion::from(&version); match &version { DuckDBVersion::Release(v) => println!("cargo:info=Using DuckDB release version: {v}"), @@ -392,8 +391,8 @@ fn main() { let crate_dir = PathBuf::from(env::var("CARGO_MANIFEST_DIR").unwrap()); let duckdb_dir = crate_dir.join("duckdb"); - let target_dir = crate_dir.parent().unwrap().join("target"); - let library_dir = target_dir.join(format!("duckdb-lib-{version}")); + let out_dir = PathBuf::from(env::var("OUT_DIR").unwrap()); + let library_dir = out_dir.join(format!("duckdb-lib-{version}")); let library_dir_str = library_dir.display(); println!("cargo:rustc-link-search=native={library_dir_str}"); @@ -411,11 +410,10 @@ fn main() { // println!("cargo:rustc-link-arg=-Wl,-rpath,{duckdb_lib}"); // } // - // Alternatively, set LD_LIBRARY_PATH (Linux) or DYLD_LIBRARY_PATH (macOS) at runtime: - // LD_LIBRARY_PATH=/path/to/target/duckdb-lib-vX.Y.Z cargo run --bin ... + // Alternatively, set LD_LIBRARY_PATH (Linux) or DYLD_LIBRARY_PATH (macOS) at runtime. println!("cargo:lib_dir={library_dir_str}"); - let source_dir = target_dir.join(format!("duckdb-source-{version}")); + let source_dir = out_dir.join(format!("duckdb-source-{version}")); let source_archive_url = match &version { DuckDBVersion::Release(v) => format!("{DUCKDB_SOURCE_RELEASE_URL}/v{v}.zip"), DuckDBVersion::Commit(c) => format!("{DUCKDB_SOURCE_COMMIT_URL}/{c}.zip"), diff --git a/vortex-duckdb/cpp/client_context.cpp b/vortex-duckdb/cpp/client_context.cpp index 181a42ff392..34753181906 100644 --- a/vortex-duckdb/cpp/client_context.cpp +++ b/vortex-duckdb/cpp/client_context.cpp @@ -1,13 +1,11 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#include "duckdb_vx.h" - #include "duckdb_vx/duckdb_diagnostics.h" + DUCKDB_INCLUDES_BEGIN #include #include -#include DUCKDB_INCLUDES_END extern "C" duckdb_client_context duckdb_vx_connection_get_client_context(duckdb_connection conn) { @@ -19,16 +17,6 @@ extern "C" duckdb_client_context duckdb_vx_connection_get_client_context(duckdb_ } } -extern "C" duckdb_vx_object_cache duckdb_client_context_get_object_cache(duckdb_client_context ffi_context) { - try { - auto *context = reinterpret_cast(ffi_context); - // This is okay because this is a ref to the object cache, this lives as long as the database. - return reinterpret_cast(&duckdb::ObjectCache::GetObjectCache(*context)); - } catch (...) { - return nullptr; - } -} - extern "C" duckdb_value duckdb_client_context_try_get_current_setting(duckdb_client_context context, const char *key) { if (!context || !key) { diff --git a/vortex-duckdb/cpp/data_chunk.cpp b/vortex-duckdb/cpp/data_chunk.cpp index 3829a6eaf24..a91e6ce058d 100644 --- a/vortex-duckdb/cpp/data_chunk.cpp +++ b/vortex-duckdb/cpp/data_chunk.cpp @@ -1,13 +1,12 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +#include "duckdb_vx/data_chunk.h" #include "duckdb_vx/duckdb_diagnostics.h" DUCKDB_INCLUDES_BEGIN #include "duckdb/common/types/data_chunk.hpp" DUCKDB_INCLUDES_END -#include "duckdb_vx.h" - const char *duckdb_data_chunk_to_string(duckdb_data_chunk chunk, duckdb_vx_error *err) { try { auto dchunk = reinterpret_cast(chunk); diff --git a/vortex-duckdb/cpp/error.cpp b/vortex-duckdb/cpp/error.cpp index 4e14b34a76d..f00a9a3ecd1 100644 --- a/vortex-duckdb/cpp/error.cpp +++ b/vortex-duckdb/cpp/error.cpp @@ -1,18 +1,15 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#include -#include - +#include +#include "duckdb_vx/error.h" #include "duckdb_vx/duckdb_diagnostics.h" + DUCKDB_INCLUDES_BEGIN -#include "duckdb/common/exception.hpp" -#include "duckdb/common/types/vector_buffer.hpp" -#include "duckdb/common/types/vector.hpp" +#include "duckdb.h" +#include "duckdb/common/assert.hpp" DUCKDB_INCLUDES_END -#include "duckdb_vx.h" - extern "C" duckdb_vx_error duckdb_vx_error_create(const char *message, size_t message_length) { return reinterpret_cast(new std::string(message, message_length)); } @@ -33,26 +30,15 @@ std::string IntoErrString(duckdb_vx_error error) { if (!error) { return {}; } - return *reinterpret_cast(error); + std::string *const error_str = reinterpret_cast(error); + std::string out = std::move(*error_str); + duckdb_vx_error_free(error); + return out; } duckdb_state SetError(duckdb_vx_error *error_out, std::string_view message) { - assert(error_out != nullptr && "SetError called with null error_out"); + D_ASSERT(error_out != nullptr && "SetError called with null error_out"); *error_out = duckdb_vx_error_create(message.data(), message.size()); return DuckDBError; } - -duckdb_state HandleException(std::exception_ptr ex, duckdb_vx_error *error_out) { - if (!ex) { - return SetError(error_out, "Unknown error"); - } - - try { - std::rethrow_exception(ex); - } catch (const std::exception &caught) { - return SetError(error_out, caught.what()); - } catch (...) { - return SetError(error_out, "Unknown error"); - } -} } // namespace vortex diff --git a/vortex-duckdb/cpp/expr.cpp b/vortex-duckdb/cpp/expr.cpp index 85629a904da..1467edc2954 100644 --- a/vortex-duckdb/cpp/expr.cpp +++ b/vortex-duckdb/cpp/expr.cpp @@ -1,9 +1,9 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#include "duckdb_vx.h" - +#include "duckdb_vx/expr.h" #include "duckdb_vx/duckdb_diagnostics.h" + DUCKDB_INCLUDES_BEGIN #include "duckdb/planner/expression/bound_between_expression.hpp" #include "duckdb/planner/expression/bound_columnref_expression.hpp" diff --git a/vortex-duckdb/cpp/file_system.cpp b/vortex-duckdb/cpp/file_system.cpp index 43c2f3e9907..11083ad6b86 100644 --- a/vortex-duckdb/cpp/file_system.cpp +++ b/vortex-duckdb/cpp/file_system.cpp @@ -12,12 +12,9 @@ DUCKDB_INCLUDES_BEGIN #include DUCKDB_INCLUDES_END -#include -#include #include using namespace duckdb; -using vortex::HandleException; using vortex::SetError; extern "C" duckdb_vx_file_handle @@ -153,3 +150,19 @@ extern "C" duckdb_state duckdb_vx_fs_sync(duckdb_vx_file_handle handle, duckdb_v } return DuckDBSuccess; } + +extern "C" duckdb_state +duckdb_vx_fs_remove(duckdb_client_context ctx, const char *path, duckdb_vx_error *error_out) { + if (!ctx || !path) { + return SetError(error_out, "Invalid arguments to fs_remove"); + } + + auto *client_context = reinterpret_cast(ctx); + + try { + FileSystem::GetFileSystem(*client_context).RemoveFile(path); + } catch (const std::exception &e) { + return SetError(error_out, e.what()); + } + return DuckDBSuccess; +} diff --git a/vortex-duckdb/cpp/include/duckdb_vx.h b/vortex-duckdb/cpp/include/duckdb_vx.h index d1396477320..dcad0ae1487 100644 --- a/vortex-duckdb/cpp/include/duckdb_vx.h +++ b/vortex-duckdb/cpp/include/duckdb_vx.h @@ -12,7 +12,6 @@ #include "duckdb_vx/expr.h" #include "duckdb_vx/file_system.h" #include "duckdb_vx/logical_type.h" -#include "duckdb_vx/object_cache.h" #include "duckdb_vx/reusable_dict.h" #include "duckdb_vx/replacement_scan.h" #include "duckdb_vx/scalar_function.h" diff --git a/vortex-duckdb/cpp/include/duckdb_vx/copy_function.h b/vortex-duckdb/cpp/include/duckdb_vx/copy_function.h index 1b253fc11a7..cb9142ed710 100644 --- a/vortex-duckdb/cpp/include/duckdb_vx/copy_function.h +++ b/vortex-duckdb/cpp/include/duckdb_vx/copy_function.h @@ -76,4 +76,4 @@ duckdb_state duckdb_vx_copy_func_register_vtab_one(duckdb_database ffi_db); #ifdef __cplusplus /* End C ABI */ }; -#endif \ No newline at end of file +#endif diff --git a/vortex-duckdb/cpp/include/duckdb_vx/error.hpp b/vortex-duckdb/cpp/include/duckdb_vx/error.hpp index fc4b185ac69..5768421acf1 100644 --- a/vortex-duckdb/cpp/include/duckdb_vx/error.hpp +++ b/vortex-duckdb/cpp/include/duckdb_vx/error.hpp @@ -17,5 +17,4 @@ DUCKDB_INCLUDES_END namespace vortex { std::string IntoErrString(duckdb_vx_error error); duckdb_state SetError(duckdb_vx_error *error_out, std::string_view message); -duckdb_state HandleException(std::exception_ptr ex, duckdb_vx_error *error_out); } // namespace vortex diff --git a/vortex-duckdb/cpp/include/duckdb_vx/file_system.h b/vortex-duckdb/cpp/include/duckdb_vx/file_system.h index bfe31b12aff..567b3f7163f 100644 --- a/vortex-duckdb/cpp/include/duckdb_vx/file_system.h +++ b/vortex-duckdb/cpp/include/duckdb_vx/file_system.h @@ -75,6 +75,9 @@ duckdb_state duckdb_vx_fs_write(duckdb_vx_file_handle handle, // Flush pending writes to storage. duckdb_state duckdb_vx_fs_sync(duckdb_vx_file_handle handle, duckdb_vx_error *error_out); +// Delete a file using DuckDB's filesystem. +duckdb_state duckdb_vx_fs_remove(duckdb_client_context ctx, const char *path, duckdb_vx_error *error_out); + #ifdef __cplusplus /* End C ABI */ } #endif diff --git a/vortex-duckdb/cpp/include/duckdb_vx/object_cache.h b/vortex-duckdb/cpp/include/duckdb_vx/object_cache.h deleted file mode 100644 index a96d62fde9a..00000000000 --- a/vortex-duckdb/cpp/include/duckdb_vx/object_cache.h +++ /dev/null @@ -1,31 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -#include "duckdb_vx/client_context.h" - -#pragma once - -#ifdef __cplusplus /* If compiled as C++, use C ABI */ -extern "C" { -#endif - -typedef struct duckdb_vx_object_cache_ *duckdb_vx_object_cache; - -duckdb_vx_object_cache duckdb_client_context_get_object_cache(duckdb_client_context context); - -// Function pointer type for custom deleter -typedef void (*duckdb_vx_deleter_fn)(void *ptr); - -// Writes the `value` to the object cache with the key `key`, overwriting the current value if it exists. -void duckdb_vx_object_cache_put(duckdb_vx_object_cache object_cache, - const char *key, - void *value, - uint64_t estimated_size, - duckdb_vx_deleter_fn deleter); - -// Fetches the key from the object cache, returning nullptr if the key is not present. -void *duckdb_vx_object_cache_get(duckdb_vx_object_cache object_cache, const char *key); - -#ifdef __cplusplus /* End C ABI */ -} -#endif diff --git a/vortex-duckdb/cpp/include/duckdb_vx/table_function.h b/vortex-duckdb/cpp/include/duckdb_vx/table_function.h index e8f483514a5..4b60207a036 100644 --- a/vortex-duckdb/cpp/include/duckdb_vx/table_function.h +++ b/vortex-duckdb/cpp/include/duckdb_vx/table_function.h @@ -12,9 +12,13 @@ #include "error.h" #include "table_filter.h" #include "duckdb_vx/data.h" -#include "duckdb_vx/client_context.h" +#include -#ifdef __cplusplus /* If compiled as C++, use C ABI */ +#ifdef __cplusplus +static_assert(sizeof(idx_t) == 8); +#endif + +#ifdef __cplusplus extern "C" { #endif @@ -22,43 +26,20 @@ extern "C" { typedef struct duckdb_vx_tfunc_bind_input_ *duckdb_vx_tfunc_bind_input; typedef struct duckdb_vx_tfunc_bind_result_ *duckdb_vx_tfunc_bind_result; -// Fetch the parameter count from the bind info. -size_t duckdb_vx_tfunc_bind_input_get_parameter_count(duckdb_vx_tfunc_bind_input ffi_input); - // Fetch a parameter from the bind info. // The caller is responsible for freeing the value using duckdb_value_free. duckdb_value duckdb_vx_tfunc_bind_input_get_parameter(duckdb_vx_tfunc_bind_input ffi_input, size_t index); -duckdb_value duckdb_vx_tfunc_bind_input_get_named_parameter(duckdb_vx_tfunc_bind_input ffi_input, - const char *name_str); - // Add a result column to the bind info. void duckdb_vx_tfunc_bind_result_add_column(duckdb_vx_tfunc_bind_result ffi_result, const char *name_str, size_t name_len, duckdb_logical_type ffi_type); -// Opaque type for the result of get_virtual_columns -typedef struct duckdb_vx_tfunc_virtual_cols_result_ *duckdb_vx_tfunc_virtual_cols_result; -// Push a column into the get_virtual_columns result. -void duckdb_vx_tfunc_virtual_columns_push(duckdb_vx_tfunc_virtual_cols_result ffi_result, - idx_t column_idx, - const char *name_str, - size_t name_len, - duckdb_logical_type ffi_type); - -// String map for to_string result typedef struct duckdb_vx_string_map_ *duckdb_vx_string_map; - -// Create a new string map -duckdb_vx_string_map duckdb_vx_string_map_create(); - // Add a key-value pair to the string map void duckdb_vx_string_map_insert(duckdb_vx_string_map map, const char *key, const char *value); -// Free the string map -void duckdb_vx_string_map_free(duckdb_vx_string_map map); - // Input data passed into the init_global and init_local callbacks. typedef struct { const void *bind_data; @@ -97,21 +78,33 @@ typedef struct { bool has_max_cardinality; } duckdb_vx_node_statistics; -// A transparent DuckDB table function vtable, which can be used to configure a table function. -// See duckdb/include/function/tfunc.hpp for details on each field. typedef struct { - // The name of the table function. - const char *name; + // Set only for strings and primitive types + duckdb_value min; + duckdb_value max; + // upper bit: "length is set". lower 32 bits: DuckDB's max string length. + // set only for strings + uint64_t max_string_length; + bool has_null; +} duckdb_column_statistics; - // The parameters of the table function. +const idx_t INVALID_IDX = UINT64_MAX; + +typedef struct { + idx_t partition_index; + // Either INVALID_IDX or position of column in output for file_index column + size_t file_index_column_pos; + // File index for the exported partition. + size_t file_index; +} duckdb_vx_partition_data; + +// vtable mimicking subset of TableFunction. +// See duckdb/include/function/tfunc.hpp +typedef struct { + const char *name; const duckdb_logical_type *parameters; size_t parameter_count; - // The named parameters of the table function. - const duckdb_logical_type *named_parameter_types; - const char *const *named_parameter_names; - size_t named_parameter_count; - duckdb_vx_data (*bind)(duckdb_client_context ctx, duckdb_vx_tfunc_bind_input input, duckdb_vx_tfunc_bind_result result, @@ -119,63 +112,33 @@ typedef struct { duckdb_vx_data (*bind_data_clone)(const void *bind_data, duckdb_vx_error *error_out); - // void *bind_replace; - // void *bind_operator; - duckdb_vx_data (*init_global)(const duckdb_vx_tfunc_init_input *input, duckdb_vx_error *error_out); - duckdb_vx_data (*init_local)(const duckdb_vx_tfunc_init_input *input, - void *init_global_data, - duckdb_vx_error *error_out); + duckdb_vx_data (*init_local)(void *init_global_data); - void (*function)(duckdb_client_context ctx, - const void *bind_data, - void *init_global_data, + void (*function)(void *init_global_data, void *init_local_data, duckdb_data_chunk data_chunk_out, duckdb_vx_error *error_out); - // void *in_out_function; - // void *in_out_function_final; - void *statistics; + bool (*statistics)(const void *bind_data, size_t column_index, duckdb_column_statistics *stats_out); - // void *dependency; void (*cardinality)(void *bind_data, duckdb_vx_node_statistics *node_stats_out); bool (*pushdown_complex_filter)(void *bind_data, duckdb_vx_expr expr, duckdb_vx_error *error_out); - void (*get_virtual_columns)(void *bind_data, duckdb_vx_tfunc_virtual_cols_result result_out); - - void *pushdown_expression; - duckdb_vx_string_map (*to_string)(void *bind_data); - // void *dynamic_to_string; - - double (*table_scan_progress)(duckdb_client_context ctx, void *bind_data, void *global_state); - - idx_t (*get_partition_data)(const void *bind_data, - void *init_global_data, - void *init_local_data, - duckdb_vx_error *error_out); - // void *get_bind_info; - // void *type_pushdown; - // void *get_multi_file_reader; - // void *supports_pushdown_type; - // void *get_partition_info; - // void *get_partition_stats; - // void *get_virtual_columns; - // void *get_row_id_columns; - - bool projection_pushdown; - bool filter_pushdown; - bool filter_prune; - bool sampling_pushdown; - bool late_materialization; - idx_t max_threads; + void (*to_string)(void *bind_data, duckdb_vx_string_map map); + + double (*table_scan_progress)(void *global_state); + + void (*get_partition_data)(void *init_global_data, + void *init_local_data, + duckdb_vx_partition_data *partition_data_out); } duckdb_vx_tfunc_vtab_t; // A single function for configuring the DuckDB table function vtable. duckdb_state duckdb_vx_tfunc_register(duckdb_database ffi_db, const duckdb_vx_tfunc_vtab_t *vtab); -#ifdef __cplusplus /* End C ABI */ +#ifdef __cplusplus } #endif diff --git a/vortex-duckdb/cpp/include/duckdb_vx/vector.h b/vortex-duckdb/cpp/include/duckdb_vx/vector.h index ea3e49184f7..036c4aeb313 100644 --- a/vortex-duckdb/cpp/include/duckdb_vx/vector.h +++ b/vortex-duckdb/cpp/include/duckdb_vx/vector.h @@ -47,9 +47,22 @@ void duckdb_vx_string_vector_add_vector_data_buffer(duckdb_vector ffi_vector, du // valid. void duckdb_vx_vector_set_vector_data_buffer(duckdb_vector ffi_vector, duckdb_vx_vector_buffer buffer); +// Reset vector's validity mask to nullptr, making all vector's elements valid. +// vector must not be a DictionaryVector or a SequenceVector +void duckdb_vx_vector_set_all_valid(duckdb_vector ffi_vector); + // Set the data pointer for the vector. This is the start of the values array in the vector. void duckdb_vx_vector_set_data_ptr(duckdb_vector ffi_vector, void *ptr); +// Set the validity pointer for the vector to external data, and store the buffer in auxiliary +// to keep it alive. The validity pointer is derived from data_ptr at the given u64 offset. +// The buffer is attached purely as a keep-alive. This enables zero-copy export of validity masks. +void duckdb_vx_vector_set_validity_data(duckdb_vector ffi_vector, + idx_t u64_offset, + idx_t capacity, + duckdb_vx_vector_buffer buffer, + void *data_ptr); + // Converts a duckdb flat vector into a Sequence vector. void duckdb_vx_sequence_vector(duckdb_vector c_vector, int64_t start, int64_t step, idx_t capacity); diff --git a/vortex-duckdb/cpp/logical_type.cpp b/vortex-duckdb/cpp/logical_type.cpp index f4385af6f42..eb3ac4f3768 100644 --- a/vortex-duckdb/cpp/logical_type.cpp +++ b/vortex-duckdb/cpp/logical_type.cpp @@ -2,12 +2,6 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors #include "duckdb_vx/duckdb_diagnostics.h" - -DUCKDB_INCLUDES_BEGIN -#include "duckdb/common/types.hpp" -DUCKDB_INCLUDES_END - -#include "duckdb_vx.h" #include "duckdb_vx/logical_type.h" DUCKDB_INCLUDES_BEGIN @@ -16,7 +10,7 @@ DUCKDB_INCLUDES_END #include duckdb_logical_type duckdb_vx_logical_type_copy(duckdb_logical_type ty) { - assert(ty != nullptr); + D_ASSERT(ty); auto *src = reinterpret_cast(ty); auto copy = duckdb::make_uniq(*src); return reinterpret_cast(copy.release()); diff --git a/vortex-duckdb/cpp/object_cache.cpp b/vortex-duckdb/cpp/object_cache.cpp deleted file mode 100644 index e2ddb34d510..00000000000 --- a/vortex-duckdb/cpp/object_cache.cpp +++ /dev/null @@ -1,60 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -#include "duckdb_vx.h" -#include "duckdb_vx/duckdb_diagnostics.h" - -DUCKDB_INCLUDES_BEGIN -#include "duckdb/storage/object_cache.hpp" -DUCKDB_INCLUDES_END - -#include - -namespace vortex { - -// Wrapper class to hold opaque pointers in DuckDB's object cache -class OpaqueWrapper : public duckdb::ObjectCacheEntry { -public: - duckdb::unique_ptr ptr; - duckdb::optional_idx estimated_size; - - explicit OpaqueWrapper(void *p, duckdb::optional_idx estimated_size, duckdb_vx_deleter_fn del) - : ptr(p, del), estimated_size(estimated_size) { - } - ~OpaqueWrapper() override = default; - - duckdb::optional_idx GetEstimatedCacheMemory() const override { - return estimated_size; - } - - duckdb::string GetObjectType() override { - return "vortex_opaque_wrapper"; - } - - // Static method required by DuckDB's object cache - static duckdb::string ObjectType() { - return "vortex_opaque_wrapper"; - } -}; - -} // namespace vortex - -extern "C" void duckdb_vx_object_cache_put(duckdb_vx_object_cache cache, - const char *key, - void *value, - uint64_t estimated_size, - duckdb_vx_deleter_fn deleter) { - auto object_cache = reinterpret_cast(cache); - auto wrapper = - duckdb::make_shared_ptr(value, duckdb::optional_idx(estimated_size), deleter); - object_cache->Put(std::string(key), wrapper); -} - -extern "C" void *duckdb_vx_object_cache_get(duckdb_vx_object_cache cache, const char *key) { - auto object_cache = reinterpret_cast(cache); - auto entry = object_cache->Get(std::string(key)); - if (!entry) { - return nullptr; - } - return entry->ptr.get(); -} diff --git a/vortex-duckdb/cpp/table_function.cpp b/vortex-duckdb/cpp/table_function.cpp index fc4ac1b66da..c7bb885e937 100644 --- a/vortex-duckdb/cpp/table_function.cpp +++ b/vortex-duckdb/cpp/table_function.cpp @@ -1,104 +1,206 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +#include "duckdb_vx/data.hpp" #include "duckdb_vx/duckdb_diagnostics.h" +#include "duckdb_vx/error.hpp" +#include "duckdb_vx/table_function.h" DUCKDB_INCLUDES_BEGIN #include "duckdb.h" #include "duckdb/catalog/catalog.hpp" #include "duckdb/common/insertion_order_preserving_map.hpp" +#include "duckdb/common/multi_file/multi_file_reader.hpp" #include "duckdb/function/table_function.hpp" #include "duckdb/main/capi/capi_internal.hpp" #include "duckdb/main/connection.hpp" #include "duckdb/parser/parsed_data/create_table_function_info.hpp" DUCKDB_INCLUDES_END -#include "duckdb_vx.h" -#include "duckdb_vx/data.hpp" -#include "duckdb_vx/error.hpp" - using namespace duckdb; +using vortex::CData; +using vortex::IntoErrString; +constexpr column_t COLUMN_IDENTIFIER_FILE_INDEX = MultiFileReader::COLUMN_IDENTIFIER_FILE_INDEX; +constexpr column_t COLUMN_IDENTIFIER_FILE_ROW_NUMBER = MultiFileReader::COLUMN_IDENTIFIER_FILE_ROW_NUMBER; -namespace vortex { struct CTableFunctionInfo final : TableFunctionInfo { - explicit CTableFunctionInfo(const duckdb_vx_tfunc_vtab_t &vtab) - : vtab(vtab), max_threads(vtab.max_threads) { + explicit CTableFunctionInfo(const duckdb_vx_tfunc_vtab_t &vtab) : vtab(vtab) { } - duckdb_vx_tfunc_vtab_t vtab; - idx_t max_threads; + const duckdb_vx_tfunc_vtab_t vtab; }; -struct CTableBindData final : TableFunctionData { - CTableBindData(unique_ptr info_p, unique_ptr ffi_data_p) - : info(std::move(info_p)), ffi_data(std::move(ffi_data_p)) { +struct CTableBindData final : FunctionData { + CTableBindData(const CTableFunctionInfo &info, + unique_ptr ffi_data_p, + const vector &types) + : info(info), ffi_data(std::move(ffi_data_p)), types(types) { } + unique_ptr Copy() const override; + bool Equals(const FunctionData &other_base) const override; - unique_ptr Copy() const override { - assert(info->vtab.bind_data_clone != nullptr); + // Table function info lives for as long as TableFunction is alive as it's + // stored inside TableFunction, so it's safe to store a reference. + const CTableFunctionInfo &info; + unique_ptr ffi_data; + vector types; +}; - duckdb_vx_error error_out = nullptr; - const auto copied_ffi_data = info->vtab.bind_data_clone(ffi_data->DataPtr(), &error_out); - if (error_out) { - throw BinderException(IntoErrString(error_out)); - } - return make_uniq(make_uniq(info->vtab), - unique_ptr(reinterpret_cast(copied_ffi_data))); +unique_ptr CTableBindData::Copy() const { + duckdb_vx_error error_out = nullptr; + const auto copied_ffi_data = info.vtab.bind_data_clone(ffi_data->DataPtr(), &error_out); + if (error_out) { + throw BinderException(IntoErrString(error_out)); } - unique_ptr info; - unique_ptr ffi_data; -}; + auto ffi_data_p = unique_ptr(reinterpret_cast(copied_ffi_data)); + return make_uniq(info, std::move(ffi_data_p), types); +} + +bool CTableBindData::Equals(const FunctionData &other_base) const { + const CTableBindData &other = other_base.Cast(); + // if "types" are different, "ffi_data" would also be different as it + // contains types inside, so omit "types" from comparison. + return &info == &other.info && ffi_data.get() == other.ffi_data.get(); +} struct CTableGlobalData final : GlobalTableFunctionState { - explicit CTableGlobalData(unique_ptr ffi_data_p, idx_t max_threads_p) - : ffi_data(std::move(ffi_data_p)), max_threads(max_threads_p) { + explicit CTableGlobalData(unique_ptr ffi_data) : ffi_data(std::move(ffi_data)) { } - unique_ptr ffi_data; - idx_t max_threads; - idx_t MaxThreads() const override { - return max_threads; + return GlobalTableFunctionState::MAX_THREADS; } + + unique_ptr ffi_data; }; struct CTableLocalData final : LocalTableFunctionState { - explicit CTableLocalData(unique_ptr ffi_data_p) : ffi_data(std::move(ffi_data_p)) { + explicit CTableLocalData(unique_ptr ffi_data) : ffi_data(std::move(ffi_data)) { } - unique_ptr ffi_data; + unique_ptr ffi_data; }; -/** - * Result of the bind function encapsulates the output schema. - */ +double table_scan_progress(ClientContext &, + const FunctionData *bind_data, + const GlobalTableFunctionState *global_state) { + auto &bind = bind_data->Cast(); + void *const c_global_state = global_state->Cast().ffi_data->DataPtr(); + return bind.info.vtab.table_scan_progress(c_global_state); +} + +static Value &UnwrapValue(duckdb_value value) { + return *(reinterpret_cast(value)); +} + +unique_ptr numeric_stats(duckdb_column_statistics &stats, LogicalType type) { + BaseStatistics out = StringStats::CreateUnknown(type); + if (stats.min) { + NumericStats::SetMin(out, UnwrapValue(stats.min)); + duckdb_destroy_value(&stats.min); + } + if (stats.max) { + NumericStats::SetMax(out, UnwrapValue(stats.max)); + duckdb_destroy_value(&stats.max); + } + if (!stats.has_null) { + out.Set(StatsInfo::CANNOT_HAVE_NULL_VALUES); + } + return out.ToUnique(); +} + +unique_ptr string_stats(duckdb_column_statistics &stats, LogicalType type) { + BaseStatistics out = StringStats::CreateUnknown(type); + if (stats.min) { + StringStats::SetMin(out, StringValue::Get(UnwrapValue(stats.min))); + duckdb_destroy_value(&stats.min); + } + if (stats.max) { + StringStats::SetMax(out, StringValue::Get(UnwrapValue(stats.max))); + duckdb_destroy_value(&stats.max); + } + if (stats.max_string_length >> 63) { + StringStats::SetMaxStringLength(out, uint32_t(stats.max_string_length)); + } + if (!stats.has_null) { + out.Set(StatsInfo::CANNOT_HAVE_NULL_VALUES); + } + + return out.ToUnique(); +} + +unique_ptr base_stats(duckdb_column_statistics &stats, LogicalType type) { + BaseStatistics out = StringStats::CreateUnknown(type); + if (!stats.has_null) { + out.Set(StatsInfo::CANNOT_HAVE_NULL_VALUES); + } + return out.ToUnique(); +} + +unique_ptr statistics(ClientContext &, const FunctionData *bind_data, column_t column_index) { + if (IsVirtualColumn(column_index)) { + return {}; + } + + const auto &bind = bind_data->Cast(); + void *const ffi_bind = bind.ffi_data->DataPtr(); + + duckdb_column_statistics statistics = {}; + if (!bind.info.vtab.statistics(ffi_bind, column_index, &statistics)) { + return {}; + } + + const LogicalType type = bind.types[column_index]; + + switch (type.id()) { + case LogicalTypeId::BOOLEAN: + case LogicalTypeId::TINYINT: + case LogicalTypeId::SMALLINT: + case LogicalTypeId::INTEGER: + case LogicalTypeId::BIGINT: + case LogicalTypeId::FLOAT: + case LogicalTypeId::DOUBLE: + case LogicalTypeId::UTINYINT: + case LogicalTypeId::USMALLINT: + case LogicalTypeId::UINTEGER: + case LogicalTypeId::UBIGINT: + case LogicalTypeId::UHUGEINT: + case LogicalTypeId::HUGEINT: { + return numeric_stats(statistics, type); + } + case LogicalTypeId::VARCHAR: + case LogicalTypeId::BLOB: { + return string_stats(statistics, type); + } + case LogicalTypeId::STRUCT: { + // TODO(myrrc) + // Duckdb's has_null has a different semantics for structs. + // If we propagate our has_null, this breaks Duckdb optimizer. + // You can reproduce it in struct.slt test in vortex-sqllogictests: + return {}; + } + default: + return base_stats(statistics, type); + } +} + struct CTableBindResult { vector &return_types; vector &names; }; -double c_table_scan_progress(ClientContext &context, - const FunctionData *bind_data, - const GlobalTableFunctionState *global_state) { - auto &bind = bind_data->Cast(); - duckdb_client_context c_ctx = reinterpret_cast(&context); - void *const c_bind_data = bind.ffi_data->DataPtr(); - void *const c_global_state = global_state->Cast().ffi_data->DataPtr(); - return bind.info->vtab.table_scan_progress(c_ctx, c_bind_data, c_global_state); -} - +/** + * Called for every new query. For example, if there is a VIEW over *.vortex, + * and after a query another file is added matching the glob, for second query + * bind() will be called again. + */ unique_ptr c_bind(ClientContext &context, TableFunctionBindInput &input, vector &return_types, vector &names) { const auto &info = input.table_function.function_info->Cast(); - - // Setup bind info to pass into the callback. - CTableBindResult result = { - return_types, - names, - }; + CTableBindResult result = {return_types, names}; duckdb_vx_error error_out = nullptr; auto ctx = reinterpret_cast(&context); @@ -110,12 +212,12 @@ unique_ptr c_bind(ClientContext &context, throw BinderException(IntoErrString(error_out)); } - return make_uniq(make_uniq(info.vtab), - unique_ptr(reinterpret_cast(ffi_bind_data))); + auto cdata = unique_ptr(reinterpret_cast(ffi_bind_data)); + return make_uniq(info, std::move(cdata), return_types); } unique_ptr c_init_global(ClientContext &context, TableFunctionInitInput &input) { - const auto &bind = input.bind_data->Cast(); + const CTableBindData &bind = input.bind_data->Cast(); duckdb_vx_tfunc_init_input ffi_input = { .bind_data = bind.ffi_data->DataPtr(), @@ -128,78 +230,51 @@ unique_ptr c_init_global(ClientContext &context, Table }; duckdb_vx_error error_out = nullptr; - auto ffi_global_data = bind.info->vtab.init_global(&ffi_input, &error_out); + duckdb_vx_data ffi_global_data = bind.info.vtab.init_global(&ffi_input, &error_out); if (error_out) { throw BinderException(IntoErrString(error_out)); } - return make_uniq( - unique_ptr(reinterpret_cast(ffi_global_data)), - bind.info->max_threads); + auto cdata = unique_ptr(reinterpret_cast(ffi_global_data)); + return make_uniq(std::move(cdata)); } -unique_ptr c_init_local(ExecutionContext &context, - TableFunctionInitInput &input, - GlobalTableFunctionState *global_state) { +unique_ptr +init_local(ExecutionContext &, TableFunctionInitInput &input, GlobalTableFunctionState *global_state) { const auto &bind = input.bind_data->Cast(); - auto global_data = global_state->Cast().ffi_data->DataPtr(); - - duckdb_vx_tfunc_init_input ffi_input = { - .bind_data = bind.ffi_data->DataPtr(), - .column_ids = input.column_ids.data(), - .column_ids_count = input.column_ids.size(), - .projection_ids = input.projection_ids.data(), - .projection_ids_count = input.projection_ids.size(), - .filters = reinterpret_cast(input.filters.get()), - .client_context = reinterpret_cast(&context), - }; - - duckdb_vx_error error_out = nullptr; - auto ffi_local_data = bind.info->vtab.init_local(&ffi_input, global_data, &error_out); - if (error_out) { - throw BinderException(IntoErrString(error_out)); - } + void *const ffi_global = global_state->Cast().ffi_data->DataPtr(); - return make_uniq( - unique_ptr(reinterpret_cast(ffi_local_data))); + duckdb_vx_data ffi_local_data = bind.info.vtab.init_local(ffi_global); + auto cdata = unique_ptr(reinterpret_cast(ffi_local_data)); + return make_uniq(std::move(cdata)); } -void c_function(ClientContext &context, TableFunctionInput &input, DataChunk &output) { +void function(ClientContext &, TableFunctionInput &input, DataChunk &output) { const auto &bind = input.bind_data->Cast(); - auto ctx = reinterpret_cast(&context); - const auto bind_data = bind.ffi_data->DataPtr(); - auto global_data = input.global_state->Cast().ffi_data->DataPtr(); - auto local_data = input.local_state->Cast().ffi_data->DataPtr(); + void *const ffi_global = input.global_state->Cast().ffi_data->DataPtr(); + void *const ffi_local = input.local_state->Cast().ffi_data->DataPtr(); + duckdb_data_chunk chunk = reinterpret_cast(&output); duckdb_vx_error error_out = nullptr; - bind.info->vtab.function(ctx, - bind_data, - global_data, - local_data, - reinterpret_cast(&output), - &error_out); + bind.info.vtab.function(ffi_global, ffi_local, chunk, &error_out); if (error_out) { throw InvalidInputException(IntoErrString(error_out)); } } -void c_pushdown_complex_filter(ClientContext & /*context*/, - LogicalGet & /*get*/, +void c_pushdown_complex_filter(ClientContext &, + LogicalGet &, FunctionData *bind_data, vector> &filters) { - if (filters.empty()) { - return; - } - auto &bind = bind_data->Cast(); + void *const ffi_bind = bind.ffi_data->DataPtr(); + duckdb_vx_error error_out = nullptr; for (auto iter = filters.begin(); iter != filters.end();) { - duckdb_vx_error error_out = nullptr; - auto pushed = - bind.info->vtab.pushdown_complex_filter(bind_data->Cast().ffi_data->DataPtr(), - reinterpret_cast(iter->get()), - &error_out); + duckdb_vx_expr ffi_expr = reinterpret_cast(iter->get()); + + const bool pushed = bind.info.vtab.pushdown_complex_filter(ffi_bind, ffi_expr, &error_out); if (error_out) { throw BinderException(IntoErrString(error_out)); } @@ -209,205 +284,156 @@ void c_pushdown_complex_filter(ClientContext & /*context*/, } } -unique_ptr c_cardinality(ClientContext & /*context*/, const FunctionData *bind_data) { +unique_ptr c_cardinality(ClientContext &, const FunctionData *bind_data) { auto &bind = bind_data->Cast(); - duckdb_vx_node_statistics node_stats_out = { - .estimated_cardinality = 0, - .max_cardinality = 0, - .has_estimated_cardinality = false, - .has_max_cardinality = false, - }; - bind.info->vtab.cardinality(bind_data->Cast().ffi_data->DataPtr(), &node_stats_out); + duckdb_vx_node_statistics stats = {}; + bind.info.vtab.cardinality(bind.ffi_data->DataPtr(), &stats); - auto stats = make_uniq(); - stats->has_estimated_cardinality = node_stats_out.has_estimated_cardinality; - stats->estimated_cardinality = node_stats_out.estimated_cardinality; - stats->has_max_cardinality = node_stats_out.has_max_cardinality; - stats->max_cardinality = node_stats_out.max_cardinality; + auto out = make_uniq(); + out->has_estimated_cardinality = stats.has_estimated_cardinality; + out->estimated_cardinality = stats.estimated_cardinality; + out->has_max_cardinality = stats.has_max_cardinality; + out->max_cardinality = stats.max_cardinality; - return stats; -} - -extern "C" size_t duckdb_vx_tfunc_bind_input_get_parameter_count(duckdb_vx_tfunc_bind_input ffi_input) { - if (!ffi_input) { - return 0; - } - const auto input = reinterpret_cast(ffi_input); - return input->inputs.size(); + return out; } extern "C" duckdb_value duckdb_vx_tfunc_bind_input_get_parameter(duckdb_vx_tfunc_bind_input ffi_input, size_t index) { - if (!ffi_input || index >= duckdb_vx_tfunc_bind_input_get_parameter_count(ffi_input)) { - return nullptr; - } - const auto info = reinterpret_cast(ffi_input); - return reinterpret_cast(new Value(info->inputs[index])); -} - -extern "C" duckdb_value duckdb_vx_tfunc_bind_input_get_named_parameter(duckdb_vx_tfunc_bind_input ffi_input, - const char *name) { - if (!ffi_input || !name) { - return nullptr; - } - const auto info = reinterpret_cast(ffi_input); - - if (!info->named_parameters.contains(name)) { - return nullptr; - } - auto value = duckdb::make_uniq(info->named_parameters.at(name)); - return reinterpret_cast(value.release()); + D_ASSERT(ffi_input); + const TableFunctionBindInput &input = *reinterpret_cast(ffi_input); + return reinterpret_cast(new Value(input.inputs[index])); } extern "C" void duckdb_vx_tfunc_bind_result_add_column(duckdb_vx_tfunc_bind_result ffi_result, const char *name_str, size_t name_len, duckdb_logical_type ffi_type) { - if (!name_str || !ffi_type) { - return; - } - const auto result = reinterpret_cast(ffi_result); - const auto logical_type = reinterpret_cast(ffi_type); - - result->names.emplace_back(name_str, name_len); - result->return_types.emplace_back(*logical_type); + D_ASSERT(ffi_result); + D_ASSERT(name_str); + D_ASSERT(ffi_type); + const CTableBindResult &result = *reinterpret_cast(ffi_result); + const LogicalType logical_type = *reinterpret_cast(ffi_type); + + result.names.emplace_back(name_str, name_len); + result.return_types.emplace_back(logical_type); } -virtual_column_map_t c_get_virtual_columns(ClientContext & /*context*/, - optional_ptr bind_data) { - auto &bind = bind_data->Cast(); - - auto result = virtual_column_map_t(); - bind.info->vtab.get_virtual_columns(bind_data->Cast().ffi_data->DataPtr(), - reinterpret_cast(&result)); - return result; +/** + * Called at planning time to determine whether data is partitioned by a + * given set of columns. Requested columns are GROUP BY parameters i.e. columns + * over which the query aggregates. + */ +TablePartitionInfo get_partition_info(ClientContext &, TableFunctionPartitionInput &input) { + const vector &ids = input.partition_ids; + // Our data is partitioned by array exporters. Each exporter processes a + // single Array which belongs to a single file. If data is partitioned only + // by file_index, there is one unique value for an Array. Otherwise there + // may be multiple values. + return (ids.size() == 1 && ids[0] == COLUMN_IDENTIFIER_FILE_INDEX) + ? TablePartitionInfo::SINGLE_VALUE_PARTITIONS + : TablePartitionInfo::NOT_PARTITIONED; } -extern "C" void duckdb_vx_tfunc_virtual_columns_push(duckdb_vx_tfunc_virtual_cols_result ffi_result, - idx_t column_idx, - const char *name_str, - size_t name_len, - duckdb_logical_type ffi_type) { - if (!ffi_result || !name_str || !ffi_type) { - return; +/** + * Duckdb requests this function after exporting the chunk. We answer with + * partition_index we have exported as well as information about constant + * columns in this partition. As data is partitioned by array exporters, in + * each partition ~ exported array file_index is constant. + */ +OperatorPartitionData get_partition_data(ClientContext &, TableFunctionGetPartitionInput &input) { + auto &bind = input.bind_data->Cast(); + void *const ffi_global = input.global_state->Cast().ffi_data->DataPtr(); + void *const ffi_local = input.local_state->Cast().ffi_data->DataPtr(); + duckdb_vx_partition_data partition_data; + bind.info.vtab.get_partition_data(ffi_global, ffi_local, &partition_data); + + OperatorPartitionData out(partition_data.partition_index); + + // file_index_column_pos may be INVALID_IDX, but column_index will never + // be INVALID_IDX, so we can compare directly + for (const column_t column_index : input.partition_info.partition_columns) { + if (column_index == partition_data.file_index_column_pos) { + out.partition_data.emplace_back(Value::UBIGINT(partition_data.file_index)); + } else { + throw InternalException(StringUtil::Format( + "get_partition_data: requested column_index %d is not constant for given partition", + column_index)); + } } - - auto result = reinterpret_cast(ffi_result); - const auto logical_type = reinterpret_cast(ffi_type); - const auto name = string(name_str, name_len); - - auto table_col = TableColumn(std::move(name), *logical_type); - result->emplace(column_idx, std::move(table_col)); + return out; } -OperatorPartitionData c_get_partition_data(ClientContext & /*context*/, - TableFunctionGetPartitionInput &input) { - if (input.partition_info.RequiresPartitionColumns()) { - throw InternalException("TableScan::GetPartitionData: partition columns not supported"); - } - auto &bind = input.bind_data->Cast(); - auto &global = input.global_state->Cast(); - auto &local = input.local_state->Cast(); - - duckdb_vx_error error_out = nullptr; - auto index = bind.info->vtab.get_partition_data(bind.ffi_data->DataPtr(), - global.ffi_data->DataPtr(), - local.ffi_data->DataPtr(), - &error_out); - if (error_out) { - throw InvalidInputException(IntoErrString(error_out)); - } - return OperatorPartitionData(index); +extern "C" void duckdb_vx_string_map_insert(duckdb_vx_string_map map, const char *key, const char *value) { + D_ASSERT(map); + D_ASSERT(key); + D_ASSERT(value); + reinterpret_cast *>(map)->insert(key, value); } InsertionOrderPreservingMap c_to_string(TableFunctionToStringInput &input) { InsertionOrderPreservingMap result; - auto &bind = input.bind_data->Cast(); - - // Call the Rust side to get custom string representation if available - if (bind.info->vtab.to_string) { - auto map = bind.info->vtab.to_string(bind.ffi_data->DataPtr()); - if (map) { - // Copy the map contents to the result - auto *cpp_map = reinterpret_cast *>(map); - for (const auto &[key, value] : *cpp_map) { - result[key] = value; - } - // Free the map allocated by Rust - duckdb_vx_string_map_free(map); - } - } - + duckdb_vx_string_map ffi_map = reinterpret_cast(&result); + void *const ffi_bind = input.bind_data->Cast().ffi_data->DataPtr(); + const auto &info = static_cast(*input.table_function.function_info); + info.vtab.to_string(ffi_bind, ffi_map); return result; } extern "C" duckdb_state duckdb_vx_tfunc_register(duckdb_database ffi_db, const duckdb_vx_tfunc_vtab_t *vtab) { - if (!ffi_db || !vtab) { - return DuckDBError; - } + D_ASSERT(ffi_db); + D_ASSERT(vtab); - auto wrapper = reinterpret_cast(ffi_db); - auto db = wrapper->database->instance; - auto tf = TableFunction(vtab->name, {}, c_function, c_bind, c_init_global, c_init_local); + const DatabaseWrapper &wrapper = *reinterpret_cast(ffi_db); + DatabaseInstance &db = *wrapper.database->instance; + TableFunction tf(vtab->name, {}, function, c_bind, c_init_global, init_local); - tf.pushdown_complex_filter = c_pushdown_complex_filter; + tf.projection_pushdown = true; + tf.filter_pushdown = true; + tf.filter_prune = true; + tf.sampling_pushdown = false; - tf.projection_pushdown = vtab->projection_pushdown; - tf.filter_pushdown = vtab->filter_pushdown; - tf.filter_prune = vtab->filter_prune; - tf.sampling_pushdown = vtab->sampling_pushdown; - tf.late_materialization = vtab->late_materialization; + tf.pushdown_complex_filter = c_pushdown_complex_filter; tf.cardinality = c_cardinality; - tf.get_partition_data = c_get_partition_data; - tf.get_virtual_columns = c_get_virtual_columns; + tf.get_partition_info = get_partition_info; + tf.get_partition_data = get_partition_data; tf.to_string = c_to_string; - tf.table_scan_progress = c_table_scan_progress; + tf.table_scan_progress = table_scan_progress; + tf.statistics = statistics; + + tf.late_materialization = true; + // Columns that uniquely identify a row for deferred re-fetch in a multi + // file scan: (file index, row number in file). + tf.get_row_id_columns = [](auto &, auto) -> vector { + return {COLUMN_IDENTIFIER_FILE_INDEX, COLUMN_IDENTIFIER_FILE_ROW_NUMBER}; + }; - // Set up the parameters - tf.arguments.reserve(vtab->parameter_count); + tf.get_virtual_columns = [](auto &, auto) -> virtual_column_map_t { + return { + {COLUMN_IDENTIFIER_EMPTY, {"", LogicalTypeId::BOOLEAN}}, + {COLUMN_IDENTIFIER_FILE_INDEX, {"file_index", LogicalType::UBIGINT}}, + {COLUMN_IDENTIFIER_FILE_ROW_NUMBER, {"file_row_number", LogicalType::BIGINT}}, + }; + }; + + tf.arguments.resize(vtab->parameter_count); for (size_t i = 0; i < vtab->parameter_count; i++) { - auto logical_type = reinterpret_cast(vtab->parameters[i]); - tf.arguments.emplace_back(*logical_type); - } - // And the named parameters - for (size_t i = 0; i < vtab->named_parameter_count; i++) { - auto logical_type = reinterpret_cast(vtab->named_parameter_types[i]); - tf.named_parameters.emplace(vtab->named_parameter_names[i], *logical_type); + tf.arguments[i] = *reinterpret_cast(vtab->parameters[i]); } - // Assign the VTable to the function info so we can access it later to invoke the callbacks. tf.function_info = make_shared_ptr(*vtab); try { - auto &system_catalog = Catalog::GetSystemCatalog(*db); - auto data = CatalogTransaction::GetSystemTransaction(*db); + auto &system_catalog = Catalog::GetSystemCatalog(db); + auto data = CatalogTransaction::GetSystemTransaction(db); CreateTableFunctionInfo tf_info(tf); + tf_info.on_conflict = OnCreateConflict::ALTER_ON_CONFLICT; system_catalog.CreateFunction(data, tf_info); - } catch (...) { + } catch (const std::exception &e) { + ErrorData data(e); + DUCKDB_LOG_ERROR(db, "Failed to create Vortex table function:\t" + data.Message()); return DuckDBError; } return DuckDBSuccess; } - -extern "C" duckdb_vx_string_map duckdb_vx_string_map_create() { - auto map = new InsertionOrderPreservingMap(); - return reinterpret_cast(map); -} - -extern "C" void duckdb_vx_string_map_insert(duckdb_vx_string_map map, const char *key, const char *value) { - if (!map || !key || !value) { - return; - } - auto *cpp_map = reinterpret_cast *>(map); - (*cpp_map)[string(key)] = string(value); -} - -extern "C" void duckdb_vx_string_map_free(duckdb_vx_string_map map) { - if (!map) { - return; - } - auto *cpp_map = reinterpret_cast *>(map); - delete cpp_map; -} -} // namespace vortex diff --git a/vortex-duckdb/cpp/vector.cpp b/vortex-duckdb/cpp/vector.cpp index 3b6439f4c95..e2153886a1b 100644 --- a/vortex-duckdb/cpp/vector.cpp +++ b/vortex-duckdb/cpp/vector.cpp @@ -1,6 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +#include "include/duckdb_vx/vector.h" #include "duckdb_vx/duckdb_diagnostics.h" DUCKDB_INCLUDES_BEGIN @@ -56,6 +57,21 @@ class DataVector : public Vector { inline void SetDataPtr(data_ptr_t ptr) { data = ptr; }; + + inline ValidityMask &GetValidity() { + return validity; + }; +}; + +// Same hack for ValidityMask: access protected fields via inheritance. +class ExternalValidityMask : public ValidityMask { +public: + inline void SetExternal(idx_t u64_offset, idx_t cap, buffer_ptr keeper) { + validity_data = std::move(keeper); + // Derive validity_mask from validity_data so the two stay consistent. + validity_mask = reinterpret_cast(validity_data.get()) + u64_offset; + capacity = cap; + }; }; } // namespace vortex @@ -81,6 +97,29 @@ extern "C" void duckdb_vx_vector_set_data_ptr(duckdb_vector ffi_vector, void *pt dvector->SetDataPtr((data_ptr_t)ptr); } +extern "C" void duckdb_vx_vector_set_validity_data(duckdb_vector ffi_vector, + idx_t u64_offset, + idx_t capacity, + duckdb_vx_vector_buffer buffer, + void *data_ptr) { + auto dvector = reinterpret_cast(ffi_vector); + auto &validity = dvector->GetValidity(); + // ExternalValidityMask adds no members, so this downcast only exposes + // access to ValidityMask's protected fields. + auto ext_validity = static_cast(&validity); + + // Use the shared_ptr aliasing constructor: the control block ref-counts the + // ExternalVectorBuffer (preventing the Rust buffer from being freed), + // while the stored pointer points to the explicit data_ptr. + auto ext_buf = reinterpret_cast *>(buffer); + auto keeper = shared_ptr>( + *ext_buf, + reinterpret_cast *>(data_ptr)); + + // Set validity_data, derive validity_mask from it at u64_offset, and set capacity. + ext_validity->SetExternal(u64_offset, capacity, std::move(keeper)); +} + extern "C" duckdb_value duckdb_vx_vector_get_value(duckdb_vector ffi_vector, idx_t index) { auto vector = reinterpret_cast(ffi_vector); auto value = duckdb::make_uniq(vector->GetValue(index)); @@ -106,3 +145,20 @@ const char *duckdb_vector_to_string(duckdb_vector vector, unsigned long len, duc return nullptr; } } + +void duckdb_vx_vector_set_all_valid(duckdb_vector ffi_vector) { + using enum VectorType; + Vector &vector = *reinterpret_cast(ffi_vector); + const VectorType type = vector.GetVectorType(); + D_ASSERT(type != DICTIONARY_VECTOR && type != SEQUENCE_VECTOR); + switch (type) { + case CONSTANT_VECTOR: + return ConstantVector::Validity(vector).Reset(); + case FLAT_VECTOR: + return FlatVector::Validity(vector).Reset(); + case FSST_VECTOR: + return FSSTVector::Validity(vector).Reset(); + default: + __builtin_unreachable(); + } +} diff --git a/vortex-duckdb/src/convert/dtype.rs b/vortex-duckdb/src/convert/dtype.rs index ad66787f02e..2b7c656be64 100644 --- a/vortex-duckdb/src/convert/dtype.rs +++ b/vortex-duckdb/src/convert/dtype.rs @@ -171,6 +171,7 @@ impl FromLogicalType for DType { DUCKDB_TYPE::DUCKDB_TYPE_BIGNUM => todo!(), DUCKDB_TYPE::DUCKDB_TYPE_STRING_LITERAL => todo!(), DUCKDB_TYPE::DUCKDB_TYPE_INTEGER_LITERAL => todo!(), + DUCKDB_TYPE::DUCKDB_TYPE_GEOMETRY => todo!(), }) } } @@ -583,7 +584,7 @@ mod tests { type NativeValue<'a> = &'a str; fn id(&self) -> ExtId { - ExtId::new_ref("unknown.extension") + ExtId::new("unknown.extension") } fn serialize_metadata(&self, _metadata: &Self::Metadata) -> VortexResult> { diff --git a/vortex-duckdb/src/convert/expr.rs b/vortex-duckdb/src/convert/expr.rs index 80d92ede449..a9f11059805 100644 --- a/vortex-duckdb/src/convert/expr.rs +++ b/vortex-duckdb/src/convert/expr.rs @@ -13,6 +13,7 @@ use vortex::error::vortex_err; use vortex::expr::Expression; use vortex::expr::and_collect; use vortex::expr::col; +use vortex::expr::is_not_null; use vortex::expr::is_null; use vortex::expr::list_contains; use vortex::expr::lit; @@ -43,12 +44,11 @@ fn like_pattern_str(value: &duckdb::ExpressionRef) -> VortexResult VortexResult> { let Some(value) = value.as_class() else { - tracing::debug!("no expression class id {:?}", value.as_class_id()); + debug!("no expression class id {:?}", value.as_class_id()); return Ok(None); }; Ok(Some(match value { @@ -105,7 +105,7 @@ pub fn try_from_bound_expression( DUCKDB_VX_EXPR_TYPE::DUCKDB_VX_EXPR_TYPE_OPERATOR_NOT => not(child), DUCKDB_VX_EXPR_TYPE::DUCKDB_VX_EXPR_TYPE_OPERATOR_IS_NULL => is_null(child), DUCKDB_VX_EXPR_TYPE::DUCKDB_VX_EXPR_TYPE_OPERATOR_IS_NOT_NULL => { - not(is_null(child)) + is_not_null(child) } _ => unreachable!(), } @@ -164,7 +164,7 @@ pub fn try_from_bound_expression( Like.new_expr(LikeOptions::default(), [value, pattern]) } _ => { - tracing::debug!("bound function {}", func.scalar_function.name()); + debug!("bound function {}", func.scalar_function.name()); return Ok(None); } }, diff --git a/vortex-duckdb/src/convert/mod.rs b/vortex-duckdb/src/convert/mod.rs index 8caab12b04c..d61aa81e005 100644 --- a/vortex-duckdb/src/convert/mod.rs +++ b/vortex-duckdb/src/convert/mod.rs @@ -11,4 +11,5 @@ pub use dtype::from_duckdb_table; pub use expr::try_from_bound_expression; pub use scalar::*; pub use table_filter::try_from_table_filter; +pub use table_filter::try_from_virtual_column_filter; pub use vector::data_chunk_to_vortex; diff --git a/vortex-duckdb/src/convert/table_filter.rs b/vortex-duckdb/src/convert/table_filter.rs index 7741fd64d0a..a1c046c678c 100644 --- a/vortex-duckdb/src/convert/table_filter.rs +++ b/vortex-duckdb/src/convert/table_filter.rs @@ -1,30 +1,36 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +use std::ops::Range; use std::sync::Arc; use itertools::Itertools; +use vortex::buffer::Buffer; use vortex::dtype::DType; use vortex::dtype::Nullability; use vortex::error::VortexExpect; use vortex::error::VortexResult; use vortex::error::vortex_bail; +use vortex::error::vortex_err; use vortex::expr::Expression; use vortex::expr::and_collect; use vortex::expr::get_item; +use vortex::expr::is_not_null; use vortex::expr::is_null; use vortex::expr::list_contains; use vortex::expr::lit; -use vortex::expr::not; use vortex::expr::or_collect; use vortex::scalar::Scalar; use vortex::scalar_fn::ScalarFnVTableExt; use vortex::scalar_fn::fns::binary::Binary; use vortex::scalar_fn::fns::operators::CompareOperator; +use vortex::scan::selection::Selection; use crate::cpp::DUCKDB_VX_EXPR_TYPE; +use crate::duckdb::ExtractedValue; use crate::duckdb::TableFilterClass; use crate::duckdb::TableFilterRef; +use crate::duckdb::ValueRef; pub fn try_from_table_filter( value: &TableFilterRef, @@ -61,7 +67,7 @@ pub fn try_from_table_filter( or_collect(children).unwrap_or_else(|| lit(false)) } TableFilterClass::IsNull => is_null(col.clone()), - TableFilterClass::IsNotNull => not(is_null(col.clone())), + TableFilterClass::IsNotNull => is_not_null(col.clone()), TableFilterClass::StructExtract(name, child_filter) => { return try_from_table_filter(child_filter, &get_item(name, col.clone()), scope_dtype); } @@ -125,3 +131,96 @@ pub fn try_from_table_filter( } })) } + +fn nonnegative_number_from_value(value: &ValueRef) -> VortexResult { + match value.extract() { + ExtractedValue::BigInt(i) => { + u64::try_from(i).map_err(|_| vortex_err!("negative value: {i}")) + } + ExtractedValue::Integer(i) => { + u64::try_from(i).map_err(|_| vortex_err!("negative value: {i}")) + } + ExtractedValue::UBigInt(u) => Ok(u), + ExtractedValue::UInteger(u) => Ok(u64::from(u)), + _ => vortex_bail!("unexpected value type"), + } +} + +fn intersect_sorted(left: &[u64], right: &[u64]) -> Vec { + let mut result = Vec::new(); + let (mut i, mut j) = (0, 0); + while i < left.len() && j < right.len() { + match left[i].cmp(&right[j]) { + std::cmp::Ordering::Equal => { + result.push(left[i]); + i += 1; + j += 1; + } + std::cmp::Ordering::Less => i += 1, + std::cmp::Ordering::Greater => j += 1, + } + } + result +} + +/// For constant comparison on IN filters over file_index or file_row_number +/// virtual column, create a selection and a range covering the same range as +/// expressions do. +pub fn try_from_virtual_column_filter( + filter: &TableFilterRef, +) -> VortexResult<(Selection, Option>)> { + match filter.as_class() { + TableFilterClass::InFilter(values) => { + let indices = values + .iter() + .map(nonnegative_number_from_value) + .collect::>>()?; + Ok((Selection::IncludeByIndex(Buffer::from_iter(indices)), None)) + } + TableFilterClass::ConstantComparison(const_) => { + let n = nonnegative_number_from_value(const_.value)?; + let range = match const_.operator { + DUCKDB_VX_EXPR_TYPE::DUCKDB_VX_EXPR_TYPE_COMPARE_EQUAL => Some(n..n + 1), + DUCKDB_VX_EXPR_TYPE::DUCKDB_VX_EXPR_TYPE_COMPARE_GREATERTHANOREQUALTO => { + Some(n..u64::MAX) + } + DUCKDB_VX_EXPR_TYPE::DUCKDB_VX_EXPR_TYPE_COMPARE_GREATERTHAN => { + Some(n.saturating_add(1)..u64::MAX) + } + DUCKDB_VX_EXPR_TYPE::DUCKDB_VX_EXPR_TYPE_COMPARE_LESSTHANOREQUALTO => { + Some(0..n.saturating_add(1)) + } + DUCKDB_VX_EXPR_TYPE::DUCKDB_VX_EXPR_TYPE_COMPARE_LESSTHAN => Some(0..n), + _ => None, + }; + Ok((Selection::All, range)) + } + TableFilterClass::ConjunctionAnd(conj) => { + let mut start = 0u64; + let mut end = u64::MAX; + let mut indices: Option> = None; + for child in conj.children() { + let (sel, range) = try_from_virtual_column_filter(child)?; + if let Selection::IncludeByIndex(buf) = sel { + indices = Some(match indices { + None => buf.iter().copied().collect(), + Some(existing) => intersect_sorted(&existing, buf.as_ref()), + }); + } + if let Some(r) = range { + start = start.max(r.start); + end = end.min(r.end); + } + } + let range = (start < end).then_some(start..end); + let sel = indices + .map(|v| Selection::IncludeByIndex(Buffer::from_iter(v))) + .unwrap_or(Selection::All); + Ok((sel, range)) + } + TableFilterClass::Optional(child) => { + try_from_virtual_column_filter(child).or_else(|_| Ok((Selection::All, None))) + } + _ => Ok((Selection::All, None)), + } +} diff --git a/vortex-duckdb/src/convert/vector.rs b/vortex-duckdb/src/convert/vector.rs index 029a5ae0638..a9a58c752c4 100644 --- a/vortex-duckdb/src/convert/vector.rs +++ b/vortex-duckdb/src/convert/vector.rs @@ -318,7 +318,7 @@ pub fn flat_vector_to_vortex(vector: &VectorRef, len: usize) -> VortexResult { - let validity = vector.validity_ref(len).to_mask(); + let validity = vector.validity_ref(len).execute_mask(); let entries = vector.as_slice_with_len::(len); let (offsets, sizes, child_min_length) = process_duckdb_lists(entries, &validity)?; @@ -375,7 +375,8 @@ pub fn data_chunk_to_vortex( mod tests { use std::ffi::CString; - use vortex::array::ToCanonical; + use vortex::array::LEGACY_SESSION; + use vortex::array::VortexSessionExecute; use vortex::array::arrays::BoolArray; use vortex::array::arrays::fixed_size_list::FixedSizeListArrayExt; use vortex::array::arrays::listview::ListViewArrayExt; @@ -412,6 +413,7 @@ mod tests { #[test] fn test_timestamp_vector_conversion() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values = vec![1_703_980_800_000_000_i64, 0i64, -86_400_000_000_i64]; // microseconds let len = values.len(); @@ -427,7 +429,11 @@ mod tests { // Test conversion let result = flat_vector_to_vortex(&vector, len).unwrap(); let vortex_array = TemporalArray::try_from(result).unwrap(); - let vortex_values = vortex_array.temporal_values().to_primitive(); + let vortex_values = vortex_array + .temporal_values() + .clone() + .execute::(&mut ctx) + .unwrap(); let values_slice = vortex_values.as_slice::(); assert_eq!(values_slice, values); @@ -435,6 +441,7 @@ mod tests { #[test] fn test_timestamp_seconds_vector_conversion() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values = vec![1_703_980_800_i64, 0i64, -86_400_i64]; // seconds let len = values.len(); @@ -450,7 +457,11 @@ mod tests { // Test conversion let result = flat_vector_to_vortex(&vector, len).unwrap(); let vortex_array = TemporalArray::try_from(result).unwrap(); - let vortex_values = vortex_array.temporal_values().to_primitive(); + let vortex_values = vortex_array + .temporal_values() + .clone() + .execute::(&mut ctx) + .unwrap(); let values_slice = vortex_values.as_slice::(); assert_eq!(values_slice, values); @@ -458,6 +469,7 @@ mod tests { #[test] fn test_timestamp_milliseconds_vector_conversion() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values = vec![1_703_980_800_000_i64, 0i64, -86_400_000_i64]; // milliseconds let len = values.len(); @@ -473,7 +485,11 @@ mod tests { // Test conversion let result = flat_vector_to_vortex(&vector, len).unwrap(); let vortex_array = TemporalArray::try_from(result).unwrap(); - let vortex_values = vortex_array.temporal_values().to_primitive(); + let vortex_values = vortex_array + .temporal_values() + .clone() + .execute::(&mut ctx) + .unwrap(); let values_slice = vortex_values.as_slice::(); assert_eq!(values_slice, values); @@ -481,6 +497,7 @@ mod tests { #[test] fn test_timestamp_with_nulls_conversion() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values = vec![1_703_980_800_000_000_i64, 0i64, -86_400_000_000_i64]; let len = values.len(); @@ -501,12 +518,21 @@ mod tests { // Test conversion let result = flat_vector_to_vortex(&vector, len).unwrap(); let vortex_array = TemporalArray::try_from(result).unwrap(); - let vortex_values = vortex_array.temporal_values().to_primitive(); + let vortex_values = vortex_array + .temporal_values() + .clone() + .execute::(&mut ctx) + .unwrap(); let values_slice = vortex_values.as_slice::(); assert_eq!(values_slice, values); assert_eq!( - vortex_values.validity_mask().unwrap(), + vortex_values + .as_ref() + .validity() + .unwrap() + .execute_mask(vortex_values.as_ref().len(), &mut ctx) + .unwrap(), Mask::from_indices(3, vec![0, 2]) ); } @@ -535,7 +561,12 @@ mod tests { // Test conversion let result = flat_vector_to_vortex(&vector, len).unwrap(); let vortex_array = TemporalArray::try_from(result).unwrap(); - let vortex_values = vortex_array.temporal_values().to_primitive(); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let vortex_values = vortex_array + .temporal_values() + .clone() + .execute::(&mut ctx) + .unwrap(); let values_slice = vortex_values.as_slice::(); assert_eq!(values_slice, values); @@ -543,6 +574,7 @@ mod tests { #[test] fn test_timestamp_single_value() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values = vec![1_703_980_800_000_000_i64]; // Single microsecond timestamp let len = values.len(); @@ -558,7 +590,11 @@ mod tests { // Test conversion let result = flat_vector_to_vortex(&vector, len).unwrap(); let vortex_array = TemporalArray::try_from(result).unwrap(); - let vortex_values = vortex_array.temporal_values().to_primitive(); + let vortex_values = vortex_array + .temporal_values() + .clone() + .execute::(&mut ctx) + .unwrap(); let values_slice = vortex_values.as_slice::(); assert_eq!(values_slice, values); @@ -566,6 +602,7 @@ mod tests { #[test] fn test_boolean_vector_conversion() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values = vec![true, false, true, false]; let len = values.len(); @@ -580,13 +617,14 @@ mod tests { // Test conversion let result = flat_vector_to_vortex(&vector, len).unwrap(); - let vortex_array = result.to_bool(); + let vortex_array = result.execute::(&mut ctx).unwrap(); let expected = BoolArray::new(BitBuffer::from(values), Validity::AllValid); assert_arrays_eq!(vortex_array, expected); } #[test] fn test_vector_with_nulls() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values = vec![1i32, 2, 3]; let len = values.len(); @@ -606,12 +644,17 @@ mod tests { // Test conversion let result = flat_vector_to_vortex(&vector, len).unwrap(); - let vortex_array = result.to_primitive(); + let vortex_array = result.execute::(&mut ctx).unwrap(); let vortex_slice = vortex_array.as_slice::(); assert_eq!(vortex_slice, values); assert_eq!( - vortex_array.validity_mask().unwrap(), + vortex_array + .as_ref() + .validity() + .unwrap() + .execute_mask(vortex_array.as_ref().len(), &mut ctx) + .unwrap(), Mask::from_indices(3, vec![0, 2]) ); } @@ -640,7 +683,8 @@ mod tests { // Test conversion let result = flat_vector_to_vortex(&vector, len).unwrap(); - let vortex_array = result.to_listview(); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let vortex_array = result.execute::(&mut ctx).unwrap(); assert_eq!(vortex_array.len(), len); assert_arrays_eq!( @@ -651,6 +695,7 @@ mod tests { #[test] fn test_fixed_sized_list() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values = vec![1i32, 2, 3, 4]; let len = 1; @@ -668,7 +713,7 @@ mod tests { // Test conversion let result = flat_vector_to_vortex(&vector, len).unwrap(); - let vortex_array = result.to_fixed_size_list(); + let vortex_array = result.execute::(&mut ctx).unwrap(); assert_eq!(vortex_array.len(), len); assert_arrays_eq!( @@ -679,6 +724,7 @@ mod tests { #[test] fn test_empty_struct() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let len = 4; let logical_type = LogicalType::struct_type([], []) .vortex_expect("LogicalTypeRef creation should succeed for test data"); @@ -686,7 +732,7 @@ mod tests { // Test conversion let result = flat_vector_to_vortex(&vector, len).unwrap(); - let vortex_array = result.to_struct(); + let vortex_array = result.execute::(&mut ctx).unwrap(); assert_eq!(vortex_array.len(), len); assert_eq!(vortex_array.struct_fields().nfields(), 0); @@ -694,6 +740,7 @@ mod tests { #[test] fn test_struct() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let values1 = vec![1i32, 2, 3, 4]; let values2 = vec![5i32, 6, 7, 8]; let len = values1.len(); @@ -721,7 +768,7 @@ mod tests { // Test conversion let result = flat_vector_to_vortex(&vector, len).unwrap(); - let vortex_array = result.to_struct(); + let vortex_array = result.execute::(&mut ctx).unwrap(); assert_eq!(vortex_array.len(), len); assert_eq!(vortex_array.struct_fields().nfields(), 2); @@ -737,6 +784,7 @@ mod tests { #[test] fn test_list_with_trailing_null() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Regression test: when the last list entry is null, its offset/length may be 0/0, // so we can't use the last entry to compute child vector length. let child_values = vec![1i32, 2, 3, 4]; @@ -770,7 +818,7 @@ mod tests { // Test conversion - the old bug would compute child length as 0+0=0 instead of // max(4,0)=4. let result = flat_vector_to_vortex(&vector, len).unwrap(); - let vortex_array = result.to_listview(); + let vortex_array = result.execute::(&mut ctx).unwrap(); assert_eq!(vortex_array.len(), len); assert_arrays_eq!( @@ -778,13 +826,19 @@ mod tests { PrimitiveArray::from_option_iter([Some(1i32), Some(2), Some(3), Some(4)]) ); assert_eq!( - vortex_array.validity_mask().unwrap(), + vortex_array + .as_ref() + .validity() + .unwrap() + .execute_mask(vortex_array.as_ref().len(), &mut ctx) + .unwrap(), Mask::from_indices(2, vec![0]) ); } #[test] fn test_list_out_of_order() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Regression test: list views can be out of order in DuckDB. The child vector length // must be computed as the maximum end offset, not just the last entry's end offset. let child_values = vec![1i32, 2, 3, 4]; @@ -815,7 +869,7 @@ mod tests { // Test conversion - the old bug would compute child length as 0+2=2 instead of // max(4,2)=4. let result = flat_vector_to_vortex(&vector, len).unwrap(); - let vortex_array = result.to_listview(); + let vortex_array = result.execute::(&mut ctx).unwrap(); assert_eq!(vortex_array.len(), len); assert_arrays_eq!( @@ -830,6 +884,7 @@ mod tests { #[test] fn test_list_null_garbage_data() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); // Test that null list entries with garbage offset/size values don't cause issues. // DuckDB doesn't guarantee valid offset/size for null list views, so we must check // validity before reading the offset/size values. @@ -870,7 +925,7 @@ mod tests { // Test conversion. The old code would compute child_min_length as 9999+9999=19998, which // would panic when trying to read that much data from the child vector. let result = flat_vector_to_vortex(&vector, len).unwrap(); - let vortex_array = result.to_listview(); + let vortex_array = result.execute::(&mut ctx).unwrap(); assert_eq!(vortex_array.len(), len); @@ -885,13 +940,26 @@ mod tests { ); // Verify the null entry has sanitized offset/size (offset=2, size=0) rather than garbage. - let offsets = vortex_array.offsets().to_primitive(); - let sizes = vortex_array.sizes().to_primitive(); + let offsets = vortex_array + .offsets() + .clone() + .execute::(&mut ctx) + .unwrap(); + let sizes = vortex_array + .sizes() + .clone() + .execute::(&mut ctx) + .unwrap(); assert_eq!(offsets.as_slice::()[1], 2); // Previous end (0+2). assert_eq!(sizes.as_slice::()[1], 0); assert_eq!( - vortex_array.validity_mask().unwrap(), + vortex_array + .as_ref() + .validity() + .unwrap() + .execute_mask(vortex_array.as_ref().len(), &mut ctx) + .unwrap(), Mask::from_indices(3, vec![0, 2]) ); } diff --git a/vortex-duckdb/src/copy.rs b/vortex-duckdb/src/copy.rs index c1af6869d39..fa6b25b5ce7 100644 --- a/vortex-duckdb/src/copy.rs +++ b/vortex-duckdb/src/copy.rs @@ -54,7 +54,7 @@ pub struct GlobalState { // Pool of background workers helping to drive the write task. // Note that this is optional and without it, we would only drive the task when DuckDB calls // into us, and we call `RUNTIME.block_on`. - #[allow(dead_code)] + #[expect(dead_code)] worker_pool: CurrentThreadWorkerPool, } diff --git a/vortex-duckdb/src/datasource.rs b/vortex-duckdb/src/datasource.rs index fe6336d3856..1f07155e1f6 100644 --- a/vortex-duckdb/src/datasource.rs +++ b/vortex-duckdb/src/datasource.rs @@ -5,10 +5,10 @@ //! //! Table functions that resolve to a [`DataSourceRef`] can implement [`DataSourceTableFunction`] //! to get a blanket [`TableFunction`] implementation covering init, scan, progress, filter -//! pushdown, cardinality, partitioning, and virtual columns. +//! pushdown, cardinality, and partitioning. -use std::ffi::CString; use std::fmt::Debug; +use std::ops::Range; use std::sync::Arc; use std::sync::atomic::AtomicU64; use std::sync::atomic::Ordering; @@ -21,91 +21,105 @@ use tracing::debug; use vortex::array::ArrayRef; use vortex::array::Canonical; use vortex::array::VortexSessionExecute; -use vortex::array::arrays::ScalarFnVTable; +use vortex::array::arrays::ScalarFn; use vortex::array::arrays::Struct; use vortex::array::arrays::StructArray; use vortex::array::arrays::scalar_fn::ScalarFnArrayExt; use vortex::array::optimizer::ArrayOptimizer; +use vortex::array::stats::StatsSet; use vortex::dtype::DType; use vortex::dtype::FieldNames; +use vortex::dtype::PType; use vortex::error::VortexExpect; use vortex::error::VortexResult; use vortex::error::vortex_err; use vortex::expr::Expression; use vortex::expr::and_collect; +use vortex::expr::cast; use vortex::expr::col; +use vortex::expr::merge; +use vortex::expr::pack; use vortex::expr::root; use vortex::expr::select; use vortex::expr::stats::Precision; +use vortex::expr::stats::Stat; +use vortex::file::v2::FileStatsLayoutReader; use vortex::io::kanal_ext::KanalExt; use vortex::io::runtime::BlockingRuntime; use vortex::io::runtime::current::ThreadSafeIterator; +use vortex::layout::layouts::row_idx::row_idx; +use vortex::layout::scan::multi::MultiLayoutChild; +use vortex::layout::scan::multi::MultiLayoutDataSource; use vortex::metrics::tracing::get_global_labels; +use vortex::scalar::Scalar; +use vortex::scalar::ScalarValue; use vortex::scalar_fn::fns::pack::Pack; -use vortex::scan::DataSourceRef; +use vortex::scan::DataSource; use vortex::scan::ScanRequest; +use vortex::scan::selection::Selection; use vortex_utils::aliases::hash_set::HashSet; +use vortex_utils::parallelism::get_available_parallelism; use crate::RUNTIME; use crate::SESSION; +use crate::convert::ToDuckDBScalar; use crate::convert::try_from_bound_expression; use crate::convert::try_from_table_filter; +use crate::convert::try_from_virtual_column_filter; use crate::duckdb::BindInputRef; use crate::duckdb::BindResultRef; use crate::duckdb::Cardinality; use crate::duckdb::ClientContextRef; +use crate::duckdb::ColumnStatistics; use crate::duckdb::DataChunkRef; +use crate::duckdb::DuckdbStringMapRef; use crate::duckdb::ExpressionRef; use crate::duckdb::LogicalType; +use crate::duckdb::PartitionData; use crate::duckdb::TableFilterSetRef; use crate::duckdb::TableFunction; use crate::duckdb::TableInitInput; -use crate::duckdb::VirtualColumnsResultRef; +use crate::duckdb::Value; use crate::exporter::ArrayExporter; use crate::exporter::ConversionCache; -/// Taken from -/// https://github.com/duckdb/duckdb/blob/dc11eadd8f0a7c600f0034810706605ebe10d5b9/src/include/duckdb/common/constants.hpp#L44 -/// -/// If DuckDB requests a zero-column projection from read_vortex like count(*), -/// its planner tries to get any column: -/// https://github.com/duckdb/duckdb/blob/dc11eadd8f0a7c600f0034810706605ebe10d5b9/src/planner/operator/logical_get.cpp#L149 -/// -/// If you define COLUMN_IDENTIFIER_EMPTY, planner takes it, otherwise the -/// first column. As we don't want to fill the output chunk and we can leave -/// it uninitialized in this case, we define COLUMN_IDENTIFIER_EMPTY as a -/// virtual column in our table function vtab's get_virtual_columns. -/// See vortex-duckdb/cpp/include/duckdb_vx/table_function.h -/// See virtual_columns in this file -static EMPTY_COLUMN_IDX: u64 = 18446744073709551614; -static EMPTY_COLUMN_NAME: &str = ""; +// See MultiFileReader for constants + +/// "file_index" virtual column +static FILE_INDEX_COLUMN_IDX: u64 = 9223372036854775810; +/// "file_row_number" virtual column +static FILE_ROW_NUMBER_COLUMN_IDX: u64 = 9223372036854775809; + +/// See duckdb/src/common/constants.cpp +fn is_virtual_column(id: u64) -> bool { + id >= 9223372036854775808u64 +} /// A trait for table functions that resolve to a [`DataSourceRef`]. /// /// Implementors only need to define how parameters are declared and how binding produces a /// data source. All other [`TableFunction`] methods (init, scan, progress, filter pushdown, -/// cardinality, partitioning, virtual columns) are provided by a blanket implementation. +/// cardinality, partitioning) are provided by a blanket implementation. pub(crate) trait DataSourceTableFunction: Sized + Debug { - /// Returns the positional parameters of the table function. - fn parameters() -> Vec { - vec![] - } - - /// Returns the named parameters of the table function, if any. - fn named_parameters() -> Vec<(CString, LogicalType)> { - vec![] - } + /// Positional parameters + fn parameters() -> Vec; /// Bind the table function and return a [`DataSourceRef`]. - fn bind(ctx: &ClientContextRef, input: &BindInputRef) -> VortexResult; + fn bind(ctx: &ClientContextRef, input: &BindInputRef) -> VortexResult; +} + +#[derive(Debug, Clone)] +struct DuckdbField { + name: String, + logical_type: LogicalType, + dtype: DType, } /// Bind data produced by a [`DataSourceTableFunction`]. pub struct DataSourceBindData { - data_source: DataSourceRef, + data_source: Arc, filter_exprs: Vec, - column_names: Vec, - column_types: Vec, + column_fields: Vec, } impl Clone for DataSourceBindData { @@ -114,8 +128,7 @@ impl Clone for DataSourceBindData { data_source: Arc::clone(&self.data_source), // filter_exprs are consumed once in `init_global`. filter_exprs: vec![], - column_names: self.column_names.clone(), - column_types: self.column_types.clone(), + column_fields: self.column_fields.clone(), } } } @@ -123,8 +136,7 @@ impl Clone for DataSourceBindData { impl Debug for DataSourceBindData { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { f.debug_struct("DataSourceBindData") - .field("column_names", &self.column_names) - .field("column_types", &self.column_types) + .field("column_fields", &self.column_fields) .field( "filter_exprs", &self @@ -145,14 +157,16 @@ pub struct DataSourceGlobal { batch_id: AtomicU64, bytes_total: Arc, bytes_read: AtomicU64, + file_index_column_pos: Option, + file_row_number_column_pos: Option, } /// Per-thread local scan state. pub struct DataSourceLocal { iterator: DataSourceIterator, exporter: Option, - /// The unique batch id of the last chunk exported via scan(). - batch_id: Option, + partition_index: u64, + file_index: usize, } /// Returns scan progress as a percentage (0.0–100.0). @@ -163,45 +177,106 @@ fn progress(bytes_read: &AtomicU64, bytes_total: &AtomicU64) -> f64 { read as f64 / total as f64 * 100. } -// --------------------------------------------------------------------------- -// Blanket TableFunction implementation for any DataSourceTableFunction -// --------------------------------------------------------------------------- +impl ColumnStatistics { + fn from(stats: &ColumnStatisticsAggregate, dtype: DType) -> Self { + let min = stats.min.as_ref().map(|value| { + let value = value.clone(); + Scalar::try_new(dtype.clone(), Some(value)) + .vortex_expect("scalar dtype and value are incompatible") + .try_to_duckdb_scalar() + .vortex_expect("can't convert Scalar to duckdb Value") + }); + let max = stats.max.as_ref().map(|value| { + Scalar::try_new(dtype.clone(), Some(value.clone())) + .vortex_expect("scalar dtype and value are incompatible") + .try_to_duckdb_scalar() + .vortex_expect("can't convert Scalar to duckdb Value") + }); + + let max_string_length = stats + .max_string_length + .map_or(0, |len| (1u64 << 63) | (len as u64)); + + // Useful estimate if we didn't get null count stats + let has_null = stats.has_null && dtype.is_nullable(); + + Self { + min, + max, + max_string_length, + has_null, + } + } +} + +#[derive(Default)] +pub struct ColumnStatisticsAggregate { + pub min: Option, + pub max: Option, + pub max_string_length: Option, + /// May be true if null count stat isn't present + pub has_null: bool, +} + +impl ColumnStatisticsAggregate { + pub fn new(stats: &StatsSet) -> Self { + let min = match stats.get(Stat::Min) { + Some(Precision::Exact(min)) => Some(min), + _ => None, + }; + let max = match stats.get(Stat::Max) { + Some(Precision::Exact(max)) => Some(max), + _ => None, + }; + + let max_string_length = + if let Some(Precision::Exact(value)) = stats.get(Stat::UncompressedSizeInBytes) { + // DuckDB's string length is u32 + #[allow(clippy::cast_possible_truncation)] + Some(value.as_primitive().as_u64().vortex_expect("not a u64") as u32) + } else { + None + }; + + let has_null = match stats.get(Stat::NullCount) { + Some(Precision::Exact(cnt)) => { + cnt.as_primitive().as_u64().vortex_expect("not a u64") > 0 + } + _ => true, + }; + + Self { + min, + max, + max_string_length, + has_null, + } + } +} impl TableFunction for T { type BindData = DataSourceBindData; type GlobalState = DataSourceGlobal; type LocalState = DataSourceLocal; - const PROJECTION_PUSHDOWN: bool = true; - const FILTER_PUSHDOWN: bool = true; - const FILTER_PRUNE: bool = true; - fn parameters() -> Vec { T::parameters() } - fn named_parameters() -> Vec<(CString, LogicalType)> { - T::named_parameters() - } - fn bind( ctx: &ClientContextRef, input: &BindInputRef, result: &mut BindResultRef, ) -> VortexResult { let data_source = T::bind(ctx, input)?; - - let (column_names, column_types) = extract_schema_from_dtype(data_source.dtype())?; - - for (column_name, column_type) in column_names.iter().zip(&column_types) { - result.add_result_column(column_name, column_type); + let column_fields = extract_schema_from_dtype(data_source.dtype())?; + for fields in &column_fields { + result.add_result_column(&fields.name, &fields.logical_type); } - Ok(DataSourceBindData { - data_source, + data_source: Arc::new(data_source), filter_exprs: vec![], - column_names, - column_types, + column_fields, }) } @@ -212,33 +287,49 @@ impl TableFunction for T { let column_ids = init_input.column_ids(); let projection_ids = init_input.projection_ids(); - let projection_expr = - extract_projection_expr(projection_ids, column_ids, &bind_data.column_names); - let filter_expr = extract_table_filter_expr( + let ProjectionWithVirtualColumns { + projection, + file_index_column_pos, + file_row_number_column_pos, + } = extract_projection_expr(projection_ids, column_ids, &bind_data.column_fields); + + let FilterWithVirtualColumns { + filter, + row_selection, + row_range, + file_selection, + file_range, + } = extract_table_filter_expr( init_input.table_filter_set(), column_ids, - &bind_data.column_names, + &bind_data.column_fields, &bind_data.filter_exprs, bind_data.data_source.dtype(), )?; - let filter_expr_str = filter_expr + let filter_expr_str = filter .as_ref() .map_or_else(|| "true".to_string(), |f| f.to_string()); - debug!("Global init Vortex scan SELECT {projection_expr} WHERE {filter_expr_str}"); + debug!( + "Global init Vortex scan SELECT {projection} WHERE {filter_expr_str}\n + row selection: {row_selection:?}, row range: {row_range:?}, + file selection: {file_selection:?}, file range: {file_range:?}" + ); let request = ScanRequest { - projection: projection_expr, - filter: filter_expr, - ordered: false, - ..Default::default() + projection, + filter, + ordered: file_row_number_column_pos.is_some(), + selection: row_selection, + row_range, + partition_selection: file_selection, + partition_range: file_range, + limit: None, }; let scan = RUNTIME.block_on(bind_data.data_source.scan(request))?; - let num_workers = std::thread::available_parallelism() - .map(|n| n.get()) - .unwrap_or(1); + let num_workers = get_available_parallelism().unwrap_or(1); // We create an async bounded channel so that all thread-local workers can pull the next // available array chunk regardless of which partition it came from. @@ -251,13 +342,22 @@ impl TableFunction for T { let stream = scan .partitions() .map(move |partition| { - // We create a new conversion cache scoped to the partition, since there's no point - // caching anything across partitions. - let cache = Arc::new(ConversionCache::default()); let tx = tx.clone(); - RUNTIME.handle().spawn(async move { - let mut stream = match partition.and_then(|p| p.execute()) { + let partition = match partition { + Ok(partition) => partition, + Err(e) => { + let _ = tx.send(Err(e)).await; + return; + } + }; + + let cache = Arc::new(ConversionCache { + file_index: partition.index(), + ..Default::default() + }); + + let mut stream = match partition.execute() { Ok(s) => s, Err(e) => { let _ = tx.send(Err(e)).await; @@ -289,13 +389,12 @@ impl TableFunction for T { batch_id: AtomicU64::new(0), bytes_total: Arc::new(AtomicU64::new(0)), bytes_read: AtomicU64::new(0), + file_index_column_pos, + file_row_number_column_pos, }) } - fn init_local( - _init: &TableInitInput, - global: &Self::GlobalState, - ) -> VortexResult { + fn init_local(global: &Self::GlobalState) -> Self::LocalState { unsafe { use custom_labels::sys; @@ -311,16 +410,15 @@ impl TableFunction for T { CURRENT_LABELSET.set(key, value); } - Ok(DataSourceLocal { + DataSourceLocal { iterator: global.iterator.clone(), exporter: None, - batch_id: None, - }) + partition_index: 0, + file_index: 0, + } } fn scan( - _client_context: &ClientContextRef, - _bind_data: &Self::BindData, local_state: &mut Self::LocalState, global_state: &Self::GlobalState, chunk: &mut DataChunkRef, @@ -332,12 +430,13 @@ impl TableFunction for T { return Ok(()); }; let (array_result, conversion_cache) = result?; - let array_result = array_result.optimize_recursive()?; + let array_result = array_result.optimize_recursive(ctx.session())?; + local_state.file_index = conversion_cache.file_index; let array_result: StructArray = if let Some(array) = array_result.as_opt::() { array.into_owned() - } else if let Some(array) = array_result.as_opt::() + } else if let Some(array) = array_result.as_opt::() && let Some(pack_options) = array.scalar_fn().as_opt::() { StructArray::new( @@ -356,15 +455,19 @@ impl TableFunction for T { ctx, )?); // Relaxed since there is no intra-instruction ordering required. - local_state.batch_id = Some(global_state.batch_id.fetch_add(1, Ordering::Relaxed)); + local_state.partition_index = global_state.batch_id.fetch_add(1, Ordering::Relaxed); } let exporter = local_state .exporter .as_mut() .vortex_expect("error: exporter missing"); + let has_more_data = exporter.export( + chunk, + global_state.file_index_column_pos, + global_state.file_row_number_column_pos, + )?; - let has_more_data = exporter.export(chunk)?; global_state .bytes_read .fetch_add(chunk.len(), Ordering::Relaxed); @@ -372,7 +475,7 @@ impl TableFunction for T { if !has_more_data { // This exporter is fully consumed. local_state.exporter = None; - local_state.batch_id = None; + local_state.partition_index = 0; } else { break; } @@ -380,14 +483,16 @@ impl TableFunction for T { assert!(!chunk.is_empty()); + if let Some(pos) = global_state.file_index_column_pos { + chunk + .get_vector_mut(pos) + .reference_value(&Value::from(local_state.file_index as u64)); + } + Ok(()) } - fn table_scan_progress( - _client_context: &ClientContextRef, - _bind_data: &Self::BindData, - global_state: &Self::GlobalState, - ) -> f64 { + fn table_scan_progress(global_state: &Self::GlobalState) -> f64 { progress(&global_state.bytes_read, &global_state.bytes_total) } @@ -412,6 +517,29 @@ impl TableFunction for T { Ok(false) } + /// Get column-wise statistics. Available only if we're reading a single + /// file. + fn statistics(bind_data: &Self::BindData, column_index: usize) -> Option { + let children = bind_data.data_source.children(); + // Otherwise we'd have to open all files eagerly which is a performance + // regression. Duckdb's Parquet reader only gets metadata for multiple + // files with a UNION BY NAME and we don't support it (yet) + // See duckdb/common/multi_file/multi_file_function.hpp#L691 + if children.len() != 1 { + return None; + } + let MultiLayoutChild::Opened(reader) = &children[0] else { + return None; + }; + let stats_sets = match reader.as_any().downcast_ref::() { + Some(inner) => inner.file_stats().stats_sets(), + None => return None, + }; + let stats_aggregate = ColumnStatisticsAggregate::new(&stats_sets[column_index]); + let dtype = bind_data.column_fields[column_index].dtype.clone(); + Some(ColumnStatistics::from(&stats_aggregate, dtype)) + } + fn cardinality(bind_data: &Self::BindData) -> Cardinality { match bind_data.data_source.row_count() { Some(Precision::Exact(v)) => Cardinality::Maximum(v), @@ -421,105 +549,139 @@ impl TableFunction for T { } fn partition_data( - _bind_data: &Self::BindData, - _global_init_data: &Self::GlobalState, + global_init_data: &Self::GlobalState, local_init_data: &mut Self::LocalState, - ) -> VortexResult { - local_init_data - .batch_id - .ok_or_else(|| vortex_err!("batch id missing, no batches exported")) + ) -> PartitionData { + PartitionData { + partition_index: local_init_data.partition_index, + file_index_column_pos: global_init_data.file_index_column_pos, + file_index: local_init_data.file_index, + } } - fn to_string(bind_data: &Self::BindData) -> Option> { - let mut result = Vec::new(); - - result.push(("Function".to_string(), "Vortex Scan".to_string())); - + fn to_string(bind_data: &Self::BindData, map: &mut DuckdbStringMapRef) { + map.push("Function", "Vortex Scan"); if !bind_data.filter_exprs.is_empty() { let mut filters = bind_data.filter_exprs.iter().map(|f| format!("{}", f)); - result.push(("Filters".to_string(), filters.join(" /\\\n"))); + map.push("Filters", &filters.join(" /\\\n")); } - - Some(result) - } - - fn virtual_columns(_bind_data: &Self::BindData, result: &mut VirtualColumnsResultRef) { - result.register(EMPTY_COLUMN_IDX, EMPTY_COLUMN_NAME, &LogicalType::bool()); } } -// --------------------------------------------------------------------------- -// Helper functions -// --------------------------------------------------------------------------- - /// Extracts DuckDB column names and logical types from a Vortex struct DType. -fn extract_schema_from_dtype(dtype: &DType) -> VortexResult<(Vec, Vec)> { +fn extract_schema_from_dtype(dtype: &DType) -> VortexResult> { let struct_dtype = dtype .as_struct_fields_opt() .ok_or_else(|| vortex_err!("Vortex file must contain a struct array at the top level"))?; - let mut column_names = Vec::new(); - let mut column_types = Vec::new(); + let len = struct_dtype.names().len(); + let mut fields = Vec::with_capacity(len); for (field_name, field_dtype) in struct_dtype.names().iter().zip(struct_dtype.fields()) { let logical_type = LogicalType::try_from(&field_dtype)?; - column_names.push(field_name.to_string()); - column_types.push(logical_type); + fields.push(DuckdbField { + name: field_name.to_string(), + logical_type, + dtype: field_dtype, + }); } + Ok(fields) +} - Ok((column_names, column_types)) +struct ProjectionWithVirtualColumns { + projection: Expression, + file_index_column_pos: Option, + file_row_number_column_pos: Option, } -/// Creates a projection expression from raw projection/column ID slices and column names. +/// Creates a projection expression from raw projection/column ID slices and +/// column names. fn extract_projection_expr( projection_ids: Option<&[u64]>, column_ids: &[u64], - column_names: &[String], -) -> Expression { - // Projection ids may be empty, in which case you need to use projection_ids - // https://github.com/duckdb/duckdb/blob/6e211da91657a94803c465fd0ce585f4c6754b54/src/planner/operator/logical_get.cpp#L168 - let (projection_ids, has_projection_ids) = match projection_ids { + column_fields: &[DuckdbField], +) -> ProjectionWithVirtualColumns { + // If projection ids are empty, use column_ids. + // See duckdb/src/planner/operator/logical_get.cpp#L168 + let (ids, has_projection_ids) = match projection_ids { Some(ids) => (ids, true), None => (column_ids, false), }; - // duckdb index is u64 (size_t) but in Rust u64 and usize are different things. - #[allow(clippy::cast_possible_truncation)] - let names = projection_ids + let mut file_index_column_pos = None; + let mut file_row_number_column_pos = None; + + #[expect(clippy::cast_possible_truncation)] + let names = ids .iter() - .filter(|p| **p != EMPTY_COLUMN_IDX) - .map(|mut idx| { - if has_projection_ids { - idx = &column_ids[*idx as usize]; + .enumerate() + .map(|(column_pos, &column_id)| { + let column_id = if has_projection_ids { + column_ids[column_id as usize] + } else { + column_id + }; + + if column_id == FILE_INDEX_COLUMN_IDX { + file_index_column_pos = Some(column_pos); + } + if column_id == FILE_ROW_NUMBER_COLUMN_IDX { + file_row_number_column_pos = Some(column_pos); } - #[allow(clippy::cast_possible_truncation)] - column_names - .get(*idx as usize) - .vortex_expect("prune idx in column names") + column_id }) - .map(|s| Arc::from(s.as_str())) + .filter(|&col_id| !is_virtual_column(col_id)) + .map(|col_id| Arc::from(column_fields[col_id as usize].name.as_str())) .collect::(); - select(names, root()) + // file_index column will be filled later when exporting the chunk. + let select = select(names, root()); + let projection = if file_row_number_column_pos.is_some() { + // row_idx will be rearranged to correct position in scan(), prepend + // here + let row_idx = cast(row_idx(), DType::Primitive(PType::I64, false.into())); + let row_idx_struct = pack([("file_row_number", row_idx)], false.into()); + merge([row_idx_struct, select]) + } else { + select + }; + + ProjectionWithVirtualColumns { + projection, + file_index_column_pos, + file_row_number_column_pos, + } } -/// Creates a table filter expression from the table filter set, column metadata, additional -/// filter expressions, and the top-level DType. +struct FilterWithVirtualColumns { + filter: Option, + row_selection: Selection, + row_range: Option>, + file_selection: Selection, + file_range: Option>, +} + +/// Creates a table filter expression, row selection, and row range from the table filter set, +/// column metadata, additional filter expressions, and the top-level DType. fn extract_table_filter_expr( table_filter_set: Option<&TableFilterSetRef>, column_ids: &[u64], - column_names: &[String], + column_fields: &[DuckdbField], additional_filters: &[Expression], dtype: &DType, -) -> VortexResult> { +) -> VortexResult { let mut table_filter_exprs: HashSet = if let Some(filter) = table_filter_set { filter .into_iter() + .filter(|(idx, _)| { + let idx_u: usize = idx.as_(); + !is_virtual_column(column_ids[idx_u]) + }) .map(|(idx, ex)| { let idx_u: usize = idx.as_(); let col_idx: usize = column_ids[idx_u].as_(); - let name = column_names.get(col_idx).vortex_expect("exists"); + let name = &column_fields.get(col_idx).vortex_expect("exists").name; try_from_table_filter(ex, &col(name.as_str()), dtype) }) .collect::>>>()? @@ -529,7 +691,31 @@ fn extract_table_filter_expr( }; table_filter_exprs.extend(additional_filters.iter().cloned()); - Ok(and_collect(table_filter_exprs)) + + let mut file_selection = Selection::All; + let mut row_selection = Selection::All; + let mut row_range = None; + let mut file_range = None; + if let Some(filter) = table_filter_set { + for (idx, expression) in filter.into_iter() { + let idx: usize = idx.as_(); + if column_ids[idx] == FILE_ROW_NUMBER_COLUMN_IDX { + (row_selection, row_range) = try_from_virtual_column_filter(expression)?; + } + if column_ids[idx] == FILE_INDEX_COLUMN_IDX { + (file_selection, file_range) = try_from_virtual_column_filter(expression)?; + } + } + }; + + let out = FilterWithVirtualColumns { + filter: and_collect(table_filter_exprs), + row_selection, + row_range, + file_selection, + file_range, + }; + Ok(out) } #[cfg(test)] diff --git a/vortex-duckdb/src/duckdb/client_context.rs b/vortex-duckdb/src/duckdb/client_context.rs index 13b7dd6bdf8..4cb646b82c5 100644 --- a/vortex-duckdb/src/duckdb/client_context.rs +++ b/vortex-duckdb/src/duckdb/client_context.rs @@ -3,11 +3,7 @@ use std::ffi::CStr; -use vortex::error::vortex_panic; - use crate::cpp; -use crate::duckdb::ObjectCache; -use crate::duckdb::ObjectCacheRef; use crate::duckdb::Value; use crate::lifetime_wrapper; @@ -36,17 +32,6 @@ impl ClientContextRef { unsafe { &*(self as *const Self) } } - /// Get the object cache for this client context. - pub fn object_cache(&self) -> &ObjectCacheRef { - unsafe { - let cache = cpp::duckdb_client_context_get_object_cache(self.as_ptr()); - if cache.is_null() { - vortex_panic!("Failed to get object cache from client context"); - } - ObjectCache::borrow(cache) - } - } - /// Try to get the current value of a configuration setting. /// Returns None if the setting doesn't exist. pub fn try_get_current_setting(&self, key: &CStr) -> Option { diff --git a/vortex-duckdb/src/duckdb/connection.rs b/vortex-duckdb/src/duckdb/connection.rs index 7eb566565b5..05734c5e6a4 100644 --- a/vortex-duckdb/src/duckdb/connection.rs +++ b/vortex-duckdb/src/duckdb/connection.rs @@ -273,45 +273,4 @@ mod tests { let result = conn.query("DELETE FROM test WHERE id > 1").unwrap(); assert_eq!(result.row_count(), 2); } - - #[derive(Debug, PartialEq)] - struct TestCacheEntry { - data: String, - value: i32, - } - - #[test] - fn test_object_cache_put_get() { - let conn = test_connection().unwrap(); - let client_context = conn.client_context().unwrap(); - let cache = client_context.object_cache(); - - // Test with a simple struct - let test_entry = TestCacheEntry { - data: "hello world".to_string(), - value: 42, - }; - - // Store the entry in the cache - cache.put("test_key", test_entry); - - // Retrieve it back - let retrieved = cache.get::("test_key"); - assert!(retrieved.is_some()); - - let retrieved_entry = retrieved.unwrap(); - assert_eq!(retrieved_entry.data, "hello world"); - assert_eq!(retrieved_entry.value, 42); - } - - #[test] - fn test_object_cache_get_nonexistent() { - let conn = test_connection().unwrap(); - let client_context = conn.client_context().unwrap(); - let cache = client_context.object_cache(); - - // Try to get a non-existent key - let result = cache.get::("nonexistent_key"); - assert!(result.is_none()); - } } diff --git a/vortex-duckdb/src/duckdb/logical_type.rs b/vortex-duckdb/src/duckdb/logical_type.rs index 851ea2824c3..06f17222a34 100644 --- a/vortex-duckdb/src/duckdb/logical_type.rs +++ b/vortex-duckdb/src/duckdb/logical_type.rs @@ -371,7 +371,6 @@ floating_type!(Float, f32); floating_type!(Double, f64); #[cfg(test)] -#[allow(clippy::cast_possible_truncation)] mod tests { use super::*; @@ -401,7 +400,6 @@ mod tests { fn test_clone_decimal_logical_type() { let decimal_type = LogicalType::decimal_type(10, 2).vortex_expect("Failed to create decimal type"); - #[allow(clippy::redundant_clone)] let cloned = decimal_type.clone(); assert_eq!(decimal_type.as_type_id(), cloned.as_type_id()); @@ -419,7 +417,6 @@ mod tests { let list_type = LogicalType::list_type(int_type).vortex_expect("Failed to create list type"); - #[allow(clippy::redundant_clone)] let cloned = list_type.clone(); assert_eq!(list_type.as_type_id(), cloned.as_type_id()); @@ -440,7 +437,6 @@ mod tests { let array_type = LogicalType::array_type(LogicalType::new(DUCKDB_TYPE::DUCKDB_TYPE_VARCHAR), 5) .vortex_expect("Failed to create array type"); - #[allow(clippy::redundant_clone)] let cloned = array_type.clone(); assert_eq!(array_type.as_type_id(), cloned.as_type_id()); @@ -473,7 +469,6 @@ mod tests { )) }; - #[allow(clippy::redundant_clone)] let cloned = map_type.clone(); assert_eq!(map_type.as_type_id(), cloned.as_type_id()); @@ -504,7 +499,6 @@ mod tests { let struct_type = LogicalType::struct_type(member_types, member_names) .vortex_expect("Failed to create struct type"); - #[allow(clippy::redundant_clone)] let cloned = struct_type.clone(); assert_eq!(struct_type.as_type_id(), cloned.as_type_id()); @@ -548,7 +542,6 @@ mod tests { )) }; - #[allow(clippy::redundant_clone)] let cloned = union_type.clone(); assert_eq!(union_type.as_type_id(), cloned.as_type_id()); diff --git a/vortex-duckdb/src/duckdb/mod.rs b/vortex-duckdb/src/duckdb/mod.rs index b444a3326da..c42fbdaf1e4 100644 --- a/vortex-duckdb/src/duckdb/mod.rs +++ b/vortex-duckdb/src/duckdb/mod.rs @@ -13,7 +13,6 @@ mod expr; mod file_system; mod logical_type; mod macro_; -mod object_cache; mod query_result; mod reusable_dict; mod scalar_function; @@ -38,7 +37,6 @@ pub use ddb_string::*; pub use expr::*; pub use file_system::*; pub use logical_type::*; -pub use object_cache::*; pub use query_result::*; pub use reusable_dict::*; pub use scalar_function::*; diff --git a/vortex-duckdb/src/duckdb/object_cache.rs b/vortex-duckdb/src/duckdb/object_cache.rs deleted file mode 100644 index 4238be061da..00000000000 --- a/vortex-duckdb/src/duckdb/object_cache.rs +++ /dev/null @@ -1,82 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -use std::ffi::CString; -use std::os::raw::c_void; - -use vortex::error::VortexExpect; -use vortex::error::vortex_err; -use vortex::error::vortex_panic; - -use crate::cpp; -use crate::lifetime_wrapper; - -/// Custom deleter function for Box allocated in Rust -unsafe extern "C-unwind" fn rust_box_deleter(ptr: *mut c_void) { - if !ptr.is_null() { - unsafe { - drop(Box::from_raw(ptr as *mut T)); - } - } -} - -lifetime_wrapper!(ObjectCache, cpp::duckdb_vx_object_cache, |_| { - vortex_panic!("ObjectCache is owned by the DatabaseInstance, not by Rust") -}); - -impl ObjectCacheRef { - /// Erases the lifetime of this reference, returning a `&'static ObjectCacheRef`. - /// - /// # Safety - /// - /// The caller must ensure that the underlying `ObjectCache` outlives all uses of the - /// returned reference. In practice, the `ObjectCache` is owned by the `DatabaseInstance` - /// and lives as long as the database, so this is safe as long as the database is kept alive. - pub unsafe fn erase_lifetime(&self) -> &'static Self { - unsafe { &*(self as *const Self) } - } - - /// Store an entry in the object cache with the given key. - /// The entry will be converted to an opaque pointer and stored. - /// Uses a proper deleter to ensure memory is freed when the cache entry is removed. - pub fn put(&self, key: &str, entry: T) -> *mut T { - let key_cstr = CString::new(key) - .map_err(|e| vortex_err!("invalid key: {}", e)) - .vortex_expect("object cache key should be valid C string"); - let opaque_ptr = Box::into_raw(Box::new(entry)); - - // Pass 0 to allow eviction by DuckDB, u64::MAX to prevent eviction, otherwise provide an - // estimate of the size of the object in bytes. - let estimated_size: u64 = 0; - - unsafe { - cpp::duckdb_vx_object_cache_put( - self.as_ptr(), - key_cstr.as_ptr(), - opaque_ptr as *mut c_void, - estimated_size, - Some(rust_box_deleter::), - ); - } - opaque_ptr - } - - /// Retrieve an entry from the object cache with the given key. - /// Returns None if the key is not found. - pub fn get(&self, key: &str) -> Option<&T> { - let key_cstr = CString::new(key) - .map_err(|e| vortex_err!("invalid key: {}", e)) - .vortex_expect("object cache key should be valid C string"); - - unsafe { - let opaque_ptr = cpp::duckdb_vx_object_cache_get(self.as_ptr(), key_cstr.as_ptr()); - (!opaque_ptr.is_null()) - .then_some(opaque_ptr.cast::().as_ref()) - .flatten() - } - } -} - -// SAFETY: Send + Sync since the cache has a mutex wrapper on the C++ side. -unsafe impl Send for ObjectCacheRef {} -unsafe impl Sync for ObjectCacheRef {} diff --git a/vortex-duckdb/src/duckdb/table_function/bind.rs b/vortex-duckdb/src/duckdb/table_function/bind.rs index adbbd028453..f56ca65f974 100644 --- a/vortex-duckdb/src/duckdb/table_function/bind.rs +++ b/vortex-duckdb/src/duckdb/table_function/bind.rs @@ -1,8 +1,6 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -use std::ffi::CStr; - use vortex::error::vortex_err; use crate::cpp; @@ -60,23 +58,6 @@ impl BindInputRef { Some(unsafe { Value::own(value_ptr) }) } } - - /// Returns the named parameter with the given name, if it exists. - pub fn get_named_parameter(&self, name: &CStr) -> Option { - let value_ptr = unsafe { - cpp::duckdb_vx_tfunc_bind_input_get_named_parameter(self.as_ptr(), name.as_ptr()) - }; - if value_ptr.is_null() { - None - } else { - Some(unsafe { Value::own(value_ptr) }) - } - } - - /// Returns the number of parameters bound to this function. - pub fn parameter_count(&self) -> usize { - unsafe { cpp::duckdb_vx_tfunc_bind_input_get_parameter_count(self.as_ptr()) } - } } lifetime_wrapper!(BindResult, cpp::duckdb_vx_tfunc_bind_result, |_| {}); diff --git a/vortex-duckdb/src/duckdb/table_function/init.rs b/vortex-duckdb/src/duckdb/table_function/init.rs index edfd8e324b6..6baa6d6a300 100644 --- a/vortex-duckdb/src/duckdb/table_function/init.rs +++ b/vortex-duckdb/src/duckdb/table_function/init.rs @@ -39,28 +39,14 @@ pub(crate) unsafe extern "C-unwind" fn init_global_callback( } /// Native callback for the local initialization of a table function. -#[allow(deref_nullptr)] pub(crate) unsafe extern "C-unwind" fn init_local_callback( - init_input: *const cpp::duckdb_vx_tfunc_init_input, global_init_data: *mut c_void, - error_out: *mut cpp::duckdb_vx_error, ) -> cpp::duckdb_vx_data { - let init_input = TableInitInput::new( - unsafe { init_input.as_ref() }.vortex_expect("init_input null pointer"), - ); - let global_init_data = unsafe { global_init_data.cast::().as_ref() } .vortex_expect("global_init_data null pointer"); - match T::init_local(&init_input, global_init_data) { - Ok(init_data) => Data::from(Box::new(init_data)).as_ptr(), - Err(e) => { - // Set the error in the error output. - let msg = e.to_string(); - unsafe { error_out.write(cpp::duckdb_vx_error_create(msg.as_ptr().cast(), msg.len())) }; - ptr::null_mut::().cast() - } - } + let init_data = T::init_local(global_init_data); + Data::from(Box::new(init_data)).as_ptr() } /// A typed wrapper for the input to a table function's initialization. diff --git a/vortex-duckdb/src/duckdb/table_function/mod.rs b/vortex-duckdb/src/duckdb/table_function/mod.rs index f20e844d381..986ac64d100 100644 --- a/vortex-duckdb/src/duckdb/table_function/mod.rs +++ b/vortex-duckdb/src/duckdb/table_function/mod.rs @@ -12,32 +12,49 @@ use vortex::error::VortexResult; mod bind; mod cardinality; mod init; -mod partition; -mod pushdown_complex_filter; -mod table_scan_progress; -mod virtual_columns; pub use bind::*; pub use init::*; -pub use virtual_columns::VirtualColumnsResult; -pub use virtual_columns::VirtualColumnsResultRef; use crate::cpp; -use crate::cpp::duckdb_client_context; -use crate::duckdb::ClientContext; use crate::duckdb::DataChunk; use crate::duckdb::DatabaseRef; +use crate::duckdb::Expression; use crate::duckdb::LogicalType; +use crate::duckdb::Value; use crate::duckdb::client_context::ClientContextRef; use crate::duckdb::data_chunk::DataChunkRef; use crate::duckdb::expr::ExpressionRef; use crate::duckdb::table_function::cardinality::cardinality_callback; -use crate::duckdb::table_function::partition::get_partition_data_callback; -use crate::duckdb::table_function::pushdown_complex_filter::pushdown_complex_filter_callback; -use crate::duckdb::table_function::table_scan_progress::table_scan_progress_callback; -use crate::duckdb::table_function::virtual_columns::get_virtual_columns_callback; +use crate::duckdb::try_or; use crate::duckdb_try; +pub struct PartitionData { + pub partition_index: u64, + pub file_index_column_pos: Option, + pub file_index: usize, +} + +#[derive(Debug, Default)] +pub struct ColumnStatistics { + pub min: Option, + pub max: Option, + pub max_string_length: u64, + pub has_null: bool, +} + +// String map lifetime is managed by C++ code +crate::lifetime_wrapper!(DuckdbStringMap, cpp::duckdb_vx_string_map, |_| {}); +impl DuckdbStringMapRef { + pub fn push(&mut self, key: &str, value: &str) { + let key = CString::new(key).unwrap_or_else(|_| CString::default()); + let value = CString::new(value).unwrap_or_else(|_| CString::default()); + unsafe { + cpp::duckdb_vx_string_map_insert(self.as_ptr(), key.as_ptr(), value.as_ptr()); + } + } +} + /// A trait that defines the supported operations for a table function in DuckDB. /// /// This trait does not yet cover the full C++ API, see table_function.hpp. @@ -46,36 +63,12 @@ pub trait TableFunction: Sized + Debug { type GlobalState: Send + Sync; type LocalState; - /// Whether the table function supports projection pushdown. - /// If not supported a projection will be added that filters out unused columns. - const PROJECTION_PUSHDOWN: bool = false; - - /// Whether the table function supports filter pushdown. - /// If not supported a filter will be added that applies the table filter directly. - const FILTER_PUSHDOWN: bool = false; - - /// Whether the table function can immediately prune out filter columns that are unused - /// in the remainder of the query plan. - /// e.g. "SELECT i FROM tbl WHERE j = 42;" - /// - j does not need to leave the table function at all. - const FILTER_PRUNE: bool = false; - - /// Maximum number of threads the table function can use. - /// If not specified, DuckDB will use its default (GlobalTableFunctionState::MAX_THREADS). - const MAX_THREADS: u64 = u64::MAX; - /// Returns the parameters of the table function. fn parameters() -> Vec { // By default, we don't have any parameters. vec![] } - /// Returns the named parameters of the table function, if any. - fn named_parameters() -> Vec<(CString, LogicalType)> { - // By default, we don't have any named parameters. - vec![] - } - /// This function is used for determining the schema of a table producing function and /// returning bind data. fn bind( @@ -84,10 +77,12 @@ pub trait TableFunction: Sized + Debug { result: &mut BindResultRef, ) -> VortexResult; + /// Report column statistics for a file or collections of files e.g. + /// registered as a VIEW. + fn statistics(bind_data: &Self::BindData, column_index: usize) -> Option; + /// The function is called during query execution and is responsible for producing the output fn scan( - client_context: &ClientContextRef, - bind_data: &Self::BindData, init_local: &mut Self::LocalState, init_global: &Self::GlobalState, chunk: &mut DataChunkRef, @@ -103,17 +98,10 @@ pub trait TableFunction: Sized + Debug { /// /// The local operator state is used to keep track of the progress in the table function and /// is thread-local. - fn init_local( - init: &TableInitInput, - global: &Self::GlobalState, - ) -> VortexResult; + fn init_local(global: &Self::GlobalState) -> Self::LocalState; /// Return table scanning progress from 0. to 100. - fn table_scan_progress( - client_context: &ClientContextRef, - bind_data: &Self::BindData, - global_state: &Self::GlobalState, - ) -> f64; + fn table_scan_progress(global_state: &Self::GlobalState) -> f64; /// Pushes down a filter expression to the table function. /// @@ -121,34 +109,22 @@ pub trait TableFunction: Sized + Debug { /// or `false` if the filter could not be pushed down. In which case, the filter will be /// applied later in the query plan. fn pushdown_complex_filter( - _bind_data: &mut Self::BindData, - _expr: &ExpressionRef, - ) -> VortexResult { - Ok(false) - } + bind_data: &mut Self::BindData, + expr: &ExpressionRef, + ) -> VortexResult; /// Returns the cardinality estimate of the table function. - fn cardinality(_bind_data: &Self::BindData) -> Cardinality { - Cardinality::Unknown - } + fn cardinality(bind_data: &Self::BindData) -> Cardinality; /// Returns the idx of the current partition being processed by a local threa. /// This *must* be globally unique. fn partition_data( - _bind_data: &Self::BindData, - _global_init_data: &Self::GlobalState, - _local_init_data: &mut Self::LocalState, - ) -> VortexResult; - - /// Returns the virtual columns of the table function. - fn virtual_columns(_bind_data: &Self::BindData, _result: &mut VirtualColumnsResultRef) {} + global_init_data: &Self::GlobalState, + local_init_data: &mut Self::LocalState, + ) -> PartitionData; /// Returns a vector of key-value pairs for EXPLAIN output - fn to_string(_bind_data: &Self::BindData) -> Option> { - None - } - - // TODO(ngates): there are many more callbacks that can be configured. + fn to_string(bind_data: &Self::BindData, map: &mut DuckdbStringMapRef); } #[derive(Debug)] @@ -170,38 +146,21 @@ impl DatabaseRef { .map(|logical_type| logical_type.as_ptr()) .collect::>(); - let param_names = T::named_parameters(); - let (param_names_ptrs, param_types_ptr) = param_names - .into_iter() - .map(|(name, logical_type)| (name.as_ptr(), logical_type.as_ptr())) - .unzip::<_, _, Vec<_>, Vec<_>>(); - let vtab = cpp::duckdb_vx_tfunc_vtab_t { name: name.as_ptr(), parameters: parameter_ptrs.as_ptr(), parameter_count: parameters.len() as _, - named_parameter_names: param_names_ptrs.as_ptr(), - named_parameter_types: param_types_ptr.as_ptr(), - named_parameter_count: param_names_ptrs.len() as _, bind: Some(bind_callback::), bind_data_clone: Some(bind_data_clone_callback::), init_global: Some(init_global_callback::), init_local: Some(init_local_callback::), function: Some(function::), - statistics: ptr::null_mut::(), + statistics: Some(statistics::), cardinality: Some(cardinality_callback::), pushdown_complex_filter: Some(pushdown_complex_filter_callback::), - pushdown_expression: ptr::null_mut::(), - get_virtual_columns: Some(get_virtual_columns_callback::), to_string: Some(to_string_callback::), table_scan_progress: Some(table_scan_progress_callback::), get_partition_data: Some(get_partition_data_callback::), - projection_pushdown: T::PROJECTION_PUSHDOWN, - filter_pushdown: T::FILTER_PUSHDOWN, - filter_prune: T::FILTER_PRUNE, - sampling_pushdown: false, - late_materialization: false, - max_threads: T::MAX_THREADS, }; duckdb_try!( @@ -214,61 +173,82 @@ impl DatabaseRef { } } -/// The to_string callback for a table function. unsafe extern "C-unwind" fn to_string_callback( bind_data: *mut c_void, -) -> cpp::duckdb_vx_string_map { + map: cpp::duckdb_vx_string_map, +) { let bind_data = unsafe { &*(bind_data as *const T::BindData) }; + let map = unsafe { DuckdbStringMap::borrow_mut(map) }; + T::to_string(bind_data, map); +} - match T::to_string(bind_data) { - Some(map) => { - // Create a new C++ map - let cpp_map = unsafe { cpp::duckdb_vx_string_map_create() }; - - // Fill the map with key-value pairs - for (key, value) in map { - let key_cstr = CString::new(key).unwrap_or_else(|_| CString::default()); - let value_cstr = CString::new(value).unwrap_or_else(|_| CString::default()); - - unsafe { - cpp::duckdb_vx_string_map_insert( - cpp_map, - key_cstr.as_ptr(), - value_cstr.as_ptr(), - ); - } - } - - cpp_map - } - None => ptr::null_mut(), - } +unsafe extern "C-unwind" fn statistics( + bind_data: *const c_void, + column_index: usize, + stats_out: *mut cpp::duckdb_column_statistics, +) -> bool { + let stats_out = unsafe { &mut *stats_out }; + let bind_data = + unsafe { bind_data.cast::().as_ref() }.vortex_expect("bind_data null pointer"); + let Some(stats) = T::statistics(bind_data, column_index) else { + return false; + }; + stats_out.min = stats.min.map_or(ptr::null_mut(), |v| v.into_ptr()); + stats_out.max = stats.max.map_or(ptr::null_mut(), |v| v.into_ptr()); + stats_out.max_string_length = stats.max_string_length; + stats_out.has_null = stats.has_null; + true +} + +unsafe extern "C-unwind" fn table_scan_progress_callback( + global_state: *mut c_void, +) -> f64 { + let global_state = unsafe { global_state.cast::().as_ref() } + .vortex_expect("global_init_data null pointer"); + T::table_scan_progress(global_state) +} + +unsafe extern "C-unwind" fn get_partition_data_callback( + global_init_data: *mut c_void, + local_init_data: *mut c_void, + partition_data_out: *mut cpp::duckdb_vx_partition_data, +) { + let global_init_data = unsafe { global_init_data.cast::().as_ref() } + .vortex_expect("global_init_data null pointer"); + let local_init_data = unsafe { local_init_data.cast::().as_mut() } + .vortex_expect("local_init_data null pointer"); + let data = T::partition_data(global_init_data, local_init_data); + let out = unsafe { &mut *partition_data_out }; + + out.partition_index = data.partition_index; + out.file_index_column_pos = data.file_index_column_pos.unwrap_or(usize::MAX); + out.file_index = data.file_index; +} + +unsafe extern "C-unwind" fn pushdown_complex_filter_callback( + bind_data: *mut c_void, + expr: cpp::duckdb_vx_expr, + error_out: *mut cpp::duckdb_vx_error, +) -> bool { + let bind_data = + unsafe { bind_data.cast::().as_mut() }.vortex_expect("bind_data null pointer"); + let expr = unsafe { Expression::borrow(expr) }; + try_or(error_out, || T::pushdown_complex_filter(bind_data, expr)) } -/// The native function callback for a table function. unsafe extern "C-unwind" fn function( - duckdb_client_context: duckdb_client_context, - bind_data: *const c_void, global_init_data: *mut c_void, local_init_data: *mut c_void, output: cpp::duckdb_data_chunk, error_out: *mut cpp::duckdb_vx_error, ) { - let client_context = unsafe { ClientContext::borrow(duckdb_client_context) }; - let bind_data = unsafe { &*(bind_data as *const T::BindData) }; let global_init_data = unsafe { global_init_data.cast::().as_ref() } .vortex_expect("global_init_data null pointer"); let local_init_data = unsafe { local_init_data.cast::().as_mut() } .vortex_expect("local_init_data null pointer"); let data_chunk = unsafe { DataChunk::borrow_mut(output) }; - match T::scan( - client_context, - bind_data, - local_init_data, - global_init_data, - data_chunk, - ) { + match T::scan(local_init_data, global_init_data, data_chunk) { Ok(()) => { // The data chunk is already filled by the function. // No need to do anything here. diff --git a/vortex-duckdb/src/duckdb/table_function/partition.rs b/vortex-duckdb/src/duckdb/table_function/partition.rs deleted file mode 100644 index 09d0e8fbd94..00000000000 --- a/vortex-duckdb/src/duckdb/table_function/partition.rs +++ /dev/null @@ -1,36 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -use std::ffi::c_void; - -use cpp::duckdb_vx_error; -use vortex::error::VortexExpect; - -use crate::cpp; -use crate::cpp::idx_t; -use crate::duckdb::TableFunction; - -/// Native callback for the cardinality estimate of a table function. -pub(crate) unsafe extern "C-unwind" fn get_partition_data_callback( - bind_data: *const c_void, - global_init_data: *mut c_void, - local_init_data: *mut c_void, - error_out: *mut duckdb_vx_error, -) -> idx_t { - let bind_data = - unsafe { bind_data.cast::().as_ref() }.vortex_expect("bind_data null pointer"); - let global_init_data = unsafe { global_init_data.cast::().as_ref() } - .vortex_expect("global_init_data null pointer"); - let local_init_data = unsafe { local_init_data.cast::().as_mut() } - .vortex_expect("local_init_data null pointer"); - - match T::partition_data(bind_data, global_init_data, local_init_data) { - Ok(batch_id) => batch_id, - Err(e) => { - // Set the error in the error output. - let msg = e.to_string(); - unsafe { error_out.write(cpp::duckdb_vx_error_create(msg.as_ptr().cast(), msg.len())) }; - 0 - } - } -} diff --git a/vortex-duckdb/src/duckdb/table_function/pushdown_complex_filter.rs b/vortex-duckdb/src/duckdb/table_function/pushdown_complex_filter.rs deleted file mode 100644 index bf85d4163a9..00000000000 --- a/vortex-duckdb/src/duckdb/table_function/pushdown_complex_filter.rs +++ /dev/null @@ -1,23 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -use std::ffi::c_void; - -use vortex::error::VortexExpect; - -use crate::cpp; -use crate::duckdb::Expression; -use crate::duckdb::TableFunction; -use crate::duckdb::try_or; - -/// Native callback for the global initialization of a table function. -pub(crate) unsafe extern "C-unwind" fn pushdown_complex_filter_callback( - bind_data: *mut c_void, - expr: cpp::duckdb_vx_expr, - error_out: *mut cpp::duckdb_vx_error, -) -> bool { - let bind_data = - unsafe { bind_data.cast::().as_mut() }.vortex_expect("bind_data null pointer"); - let expr = unsafe { Expression::borrow(expr) }; - try_or(error_out, || T::pushdown_complex_filter(bind_data, expr)) -} diff --git a/vortex-duckdb/src/duckdb/table_function/table_scan_progress.rs b/vortex-duckdb/src/duckdb/table_function/table_scan_progress.rs deleted file mode 100644 index cfe3dd43b45..00000000000 --- a/vortex-duckdb/src/duckdb/table_function/table_scan_progress.rs +++ /dev/null @@ -1,19 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -use vortex::error::VortexExpect; - -use crate::duckdb::TableFunction; - -pub(crate) unsafe extern "C-unwind" fn table_scan_progress_callback( - ctx: crate::cpp::duckdb_client_context, - bind_data: *mut ::std::os::raw::c_void, - global_state: *mut ::std::os::raw::c_void, -) -> f64 { - let ctx = unsafe { crate::duckdb::ClientContext::borrow(ctx) }; - let bind_data = - unsafe { bind_data.cast::().as_ref() }.vortex_expect("bind_data null pointer"); - let global_state = unsafe { global_state.cast::().as_ref() } - .vortex_expect("global_init_data null pointer"); - T::table_scan_progress(ctx, bind_data, global_state) -} diff --git a/vortex-duckdb/src/duckdb/table_function/virtual_columns.rs b/vortex-duckdb/src/duckdb/table_function/virtual_columns.rs deleted file mode 100644 index 7e4b06e48b9..00000000000 --- a/vortex-duckdb/src/duckdb/table_function/virtual_columns.rs +++ /dev/null @@ -1,43 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -use std::ffi::c_void; - -use vortex::error::VortexExpect; - -use crate::cpp; -use crate::duckdb::LogicalTypeRef; -use crate::duckdb::TableFunction; -use crate::lifetime_wrapper; - -/// Native callback for the get_virtual_columns function. -pub(crate) unsafe extern "C-unwind" fn get_virtual_columns_callback( - bind_data: *mut c_void, - result: cpp::duckdb_vx_tfunc_virtual_cols_result, -) { - let bind_data = - unsafe { bind_data.cast::().as_ref() }.vortex_expect("bind_data null pointer"); - let result = unsafe { VirtualColumnsResult::borrow_mut(result) }; - - T::virtual_columns(bind_data, result); -} - -lifetime_wrapper!( - VirtualColumnsResult, - cpp::duckdb_vx_tfunc_virtual_cols_result, - |_| {} -); - -impl VirtualColumnsResultRef { - pub fn register(&self, column_idx: u64, name: &str, logical_type: &LogicalTypeRef) { - unsafe { - cpp::duckdb_vx_tfunc_virtual_columns_push( - self.as_ptr(), - column_idx as _, - name.as_ptr().cast(), - name.len() as _, - logical_type.as_ptr(), - ) - } - } -} diff --git a/vortex-duckdb/src/duckdb/value.rs b/vortex-duckdb/src/duckdb/value.rs index d79c802a625..752f1ccca70 100644 --- a/vortex-duckdb/src/duckdb/value.rs +++ b/vortex-duckdb/src/duckdb/value.rs @@ -378,6 +378,7 @@ impl From<&[u8]> for Value { } /// An enum for extracting the underlying typed value from a `Value`. +#[derive(Debug)] pub enum ExtractedValue { Null, TinyInt(i8), diff --git a/vortex-duckdb/src/duckdb/vector.rs b/vortex-duckdb/src/duckdb/vector.rs index 4a822629980..ac652bed43f 100644 --- a/vortex-duckdb/src/duckdb/vector.rs +++ b/vortex-duckdb/src/duckdb/vector.rs @@ -24,9 +24,20 @@ use crate::duckdb::LogicalTypeRef; use crate::duckdb::SelectionVectorRef; use crate::duckdb::Value; use crate::duckdb::ValueRef; +use crate::duckdb::VectorBuffer; use crate::duckdb::VectorBufferRef; use crate::lifetime_wrapper; +/// External validity data for zero-copy export of validity masks to DuckDB. +/// +/// Holds a [`VectorBuffer`] as a keep-alive and a raw pointer to the validity bitmap. +pub(crate) struct ValidityData { + /// VectorBuffer that keeps the underlying memory alive via DuckDB's ref-counting. + pub(crate) shared_buffer: VectorBuffer, + /// Pointer to the raw validity bitmap data within the buffer. + pub(crate) data_ptr: *const u8, +} + /// Returns the internal vector size used by DuckDB at runtime. #[expect( clippy::cast_possible_truncation, @@ -151,6 +162,31 @@ impl VectorRef { unsafe { cpp::duckdb_vx_vector_set_data_ptr(self.as_ptr(), ptr as *mut c_void) } } + /// Sets the validity data for the vector from a [`ValidityData`]. The buffer is + /// attached purely as a keep-alive, and the data pointer is used as the validity data + /// at the given `u64_offset`. + /// + /// # Safety + /// + /// The data pointer must point to a valid `u64` array with at least + /// `u64_offset + capacity.div_ceil(64)` elements. + pub(crate) unsafe fn set_validity_data( + &self, + u64_offset: usize, + capacity: usize, + zero_copy: &ValidityData, + ) { + unsafe { + cpp::duckdb_vx_vector_set_validity_data( + self.as_ptr(), + u64_offset as idx_t, + capacity as idx_t, + zero_copy.shared_buffer.as_ptr(), + zero_copy.data_ptr as *mut c_void, + ) + } + } + /// Assigns the element at the specified index with a string value. /// FIXME(ngates): remove this. pub fn assign_string_element(&self, idx: usize, value: &CStr) { @@ -207,7 +243,11 @@ impl VectorRef { /// /// The provided capacity *must* be the actual capacity of this vector. pub unsafe fn validity_bitslice_mut(&mut self, capacity: usize) -> Option<&mut BitSlice> { - unsafe { self.validity_slice_mut(capacity) }.map(|slice| slice.view_bits_mut()) + // capacity is always less than BitSlice::MAX_ELTS + unsafe { + self.validity_slice_mut(capacity) + .map(|slice| BitSlice::from_slice_unchecked_mut(slice)) + } } pub fn validity_ref(&self, len: usize) -> ValidityRef<'_> { @@ -324,7 +364,7 @@ impl ValidityRef<'_> { } /// Creates a mask directly from the DuckDB validity mask for optimal performance. - pub fn to_mask(&self) -> Mask { + pub fn execute_mask(&self) -> Mask { let Some(validity) = self.validity else { // All values are valid return Mask::AllTrue(self.len); @@ -337,7 +377,7 @@ impl ValidityRef<'_> { } pub fn to_validity(&self) -> Validity { - Validity::from_mask(self.to_mask(), Nullability::Nullable) + Validity::from_mask(self.execute_mask(), Nullability::Nullable) } } diff --git a/vortex-duckdb/src/e2e_test/mod.rs b/vortex-duckdb/src/e2e_test/mod.rs index 0e5ad05963d..34182bd133d 100644 --- a/vortex-duckdb/src/e2e_test/mod.rs +++ b/vortex-duckdb/src/e2e_test/mod.rs @@ -1,7 +1,5 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#[cfg(test)] -mod object_cache_test; #[cfg(test)] mod vortex_scan_test; diff --git a/vortex-duckdb/src/e2e_test/object_cache_test.rs b/vortex-duckdb/src/e2e_test/object_cache_test.rs deleted file mode 100644 index 712d2a41209..00000000000 --- a/vortex-duckdb/src/e2e_test/object_cache_test.rs +++ /dev/null @@ -1,154 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -//! Test table function that demonstrates object cache usage - -use std::ffi::CString; - -use vortex::error::VortexResult; -use vortex::error::vortex_err; - -use crate::cpp::DUCKDB_TYPE; -use crate::duckdb::BindInputRef; -use crate::duckdb::BindResultRef; -use crate::duckdb::ClientContextRef; -use crate::duckdb::DataChunkRef; -use crate::duckdb::LogicalType; -use crate::duckdb::TableFunction; -use crate::duckdb::TableInitInput; - -#[derive(Debug, Clone)] -pub struct TestTableFunction; - -#[derive(Debug, Clone)] -pub struct TestBindData { - cache_key: String, -} - -#[derive(Debug)] -pub struct TestGlobalState; - -#[derive(Debug)] -pub struct TestLocalState; - -#[derive(Debug, PartialEq)] -struct CachedData { - message: String, - count: i32, -} - -impl TableFunction for TestTableFunction { - type BindData = TestBindData; - type GlobalState = TestGlobalState; - type LocalState = TestLocalState; - - fn bind( - client_context: &ClientContextRef, - _input: &BindInputRef, - result: &mut BindResultRef, - ) -> VortexResult { - let logical_type = LogicalType::new(DUCKDB_TYPE::DUCKDB_TYPE_BIGINT); - result.add_result_column("test_value", &logical_type); - - let cache = client_context.object_cache(); - let cached_data = CachedData { - message: "Hello from bind phase cache!".to_string(), - count: 123, - }; - - // Store data in cache during bind - cache.put("bind_phase_data", cached_data); - - Ok(TestBindData { - cache_key: "test_table_function_data".to_string(), - }) - } - - fn table_scan_progress( - _client_context: &ClientContextRef, - _bind_data: &Self::BindData, - _global_state: &Self::GlobalState, - ) -> f64 { - 100.0 - } - - fn scan( - _client_context: &ClientContextRef, - _bind_data: &Self::BindData, - _local_state: &mut Self::LocalState, - _global_state: &Self::GlobalState, - chunk: &mut DataChunkRef, - ) -> VortexResult<()> { - chunk.set_len(0); - - Ok(()) - } - - fn init_global(input: &TableInitInput) -> VortexResult { - if let Ok(ctx) = input.client_context() { - let cached_data = CachedData { - message: "Hello from table function cache!".to_string(), - count: 42, - }; - let cache = ctx.object_cache(); - - cache.put(&input.bind_data().cache_key, cached_data); - } - - Ok(TestGlobalState) - } - - fn init_local( - _init: &TableInitInput, - _global: &Self::GlobalState, - ) -> VortexResult { - Ok(TestLocalState) - } - - fn partition_data( - _bind_data: &Self::BindData, - _global_init_data: &Self::GlobalState, - _local_init_data: &mut Self::LocalState, - ) -> VortexResult { - Ok(0) - } -} - -use crate::duckdb::Database; - -#[test] -fn test_table_function_with_object_cache() -> VortexResult<()> { - let db = Database::open_in_memory()?; - let conn = db.connect()?; - - // Register our test table function - let name = CString::new("test_cache_func").map_err(|e| vortex_err!("CString error: {}", e))?; - db.register_table_function::(&name)?; - - // Call the table function - this should store data in the cache during init_global - let _result = conn.query("SELECT * FROM test_cache_func()")?; - - // Try to verify that we can access the cached data from outside the table function - // This part is optional since we're not sure if the object cache access is working yet - let ctx = conn.client_context(); - if let Ok(ctx) = ctx { - let cache = ctx.object_cache(); - // Check data from bind phase - let bind_cached_data = cache.get::("bind_phase_data"); - if let Some(data) = bind_cached_data { - assert_eq!(data.message, "Hello from bind phase cache!"); - assert_eq!(data.count, 123); - println!("Successfully retrieved bind phase cached data!"); - } - - // Check data from init_global phase - let cached_data = cache.get::("test_table_function_data"); - if let Some(data) = cached_data { - assert_eq!(data.message, "Hello from table function cache!"); - assert_eq!(data.count, 42); - println!("Successfully retrieved init_global phase cached data!"); - } - } - - Ok(()) -} diff --git a/vortex-duckdb/src/e2e_test/vortex_scan_test.rs b/vortex-duckdb/src/e2e_test/vortex_scan_test.rs index 4e03cd07cd7..8e65d26ed6f 100644 --- a/vortex-duckdb/src/e2e_test/vortex_scan_test.rs +++ b/vortex-duckdb/src/e2e_test/vortex_scan_test.rs @@ -19,6 +19,8 @@ use jiff::tz::TimeZone; use num_traits::AsPrimitive; use tempfile::NamedTempFile; use vortex::array::IntoArray; +use vortex::array::LEGACY_SESSION; +use vortex::array::VortexSessionExecute; use vortex::array::arrays::BoolArray; use vortex::array::arrays::ConstantArray; use vortex::array::arrays::DictArray; @@ -350,6 +352,40 @@ fn test_vortex_scan_multiple_files() { assert_eq!(total_sum, 21); } +#[test] +fn test_vortex_scan_multiple_globs() { + // Test scanning multiple directories using a list of glob patterns. + let (tempdir1, tempdir2, _file1, _file2, _file3) = RUNTIME.block_on(async { + let tempdir1 = tempfile::tempdir().unwrap(); + let tempdir2 = tempfile::tempdir().unwrap(); + + let file1 = write_vortex_file_to_dir(tempdir1.path(), "numbers", buffer![1i32, 2, 3]).await; + let file2 = write_vortex_file_to_dir(tempdir1.path(), "numbers", buffer![4i32, 5, 6]).await; + let file3 = + write_vortex_file_to_dir(tempdir2.path(), "numbers", buffer![7i32, 8, 9, 10]).await; + + (tempdir1, tempdir2, file1, file2, file3) + }); + + // Create glob patterns for each directory. + let glob_pattern1 = format!("{}/*.vortex", tempdir1.path().display()); + let glob_pattern2 = format!("{}/*.vortex", tempdir2.path().display()); + + // Scan files from both directories using a list of globs. + let conn = database_connection(); + let result = conn + .query(&format!( + "SELECT SUM(numbers) FROM read_vortex(['{glob_pattern1}', '{glob_pattern2}'])" + )) + .unwrap(); + let chunk = result.into_iter().next().unwrap(); + let vec = chunk.get_vector(0); + let total_sum = vec.as_slice_with_len::(chunk.len().as_())[0]; + + // 1+2+3 + 4+5+6 + 7+8+9+10 = 55 + assert_eq!(total_sum, 55); +} + #[test] fn test_vortex_scan_over_http() { let file = RUNTIME.block_on(async { @@ -789,7 +825,9 @@ async fn write_vortex_file_with_encodings() -> NamedTempFile { // 4. Run-End let run_ends = buffer![3u32, 5]; let run_values = buffer![100i32, 200]; - let rle_array = RunEnd::try_new(run_ends.into_array(), run_values.into_array()).unwrap(); + let mut rle_ctx = LEGACY_SESSION.create_execution_ctx(); + let rle_array = + RunEnd::try_new(run_ends.into_array(), run_values.into_array(), &mut rle_ctx).unwrap(); // 5. Sequence array let sequence_array = Sequence::try_new( diff --git a/vortex-duckdb/src/exporter/all_invalid.rs b/vortex-duckdb/src/exporter/all_invalid.rs index 873dde3c089..cc5d5da9efb 100644 --- a/vortex-duckdb/src/exporter/all_invalid.rs +++ b/vortex-duckdb/src/exporter/all_invalid.rs @@ -3,39 +3,25 @@ use vortex::array::ExecutionCtx; use vortex::error::VortexResult; -use vortex::error::vortex_ensure; -use crate::duckdb::LogicalTypeRef; -use crate::duckdb::Value; use crate::duckdb::VectorRef; use crate::exporter::ColumnExporter; -struct AllInvalidExporter { - len: usize, - null_value: Value, -} +struct AllInvalidExporter; -pub(crate) fn new_exporter(len: usize, logical_type: &LogicalTypeRef) -> Box { - Box::new(AllInvalidExporter { - len, - null_value: Value::null(logical_type), - }) +pub(crate) fn new_exporter() -> Box { + Box::new(AllInvalidExporter {}) } impl ColumnExporter for AllInvalidExporter { fn export( &self, - offset: usize, - len: usize, + _offset: usize, + _len: usize, vector: &mut VectorRef, _ctx: &mut ExecutionCtx, ) -> VortexResult<()> { - vortex_ensure!( - offset + len <= self.len, - "invalid exporter: offset + len must be less than or equal to len" - ); - - vector.reference_value(&self.null_value); + vector.set_all_false_validity(); Ok(()) } } @@ -43,7 +29,6 @@ impl ColumnExporter for AllInvalidExporter { #[cfg(test)] mod tests { use vortex::array::VortexSessionExecute; - use vortex::array::arrays::PrimitiveArray; use super::*; use crate::SESSION; @@ -52,12 +37,10 @@ mod tests { #[test] fn all_null_array() { - let arr = PrimitiveArray::from_option_iter::([None, None, None]); let ltype = LogicalType::int32(); + let mut chunk = DataChunk::new([ltype]); - let mut chunk = DataChunk::new([ltype.clone()]); - - new_exporter(arr.len(), <ype) + new_exporter() .export( 0, 3, diff --git a/vortex-duckdb/src/exporter/bool.rs b/vortex-duckdb/src/exporter/bool.rs index cb488ce5297..84fd17f0789 100644 --- a/vortex-duckdb/src/exporter/bool.rs +++ b/vortex-duckdb/src/exporter/bool.rs @@ -4,11 +4,11 @@ use vortex::array::ExecutionCtx; use vortex::array::arrays::BoolArray; use vortex::array::arrays::bool::BoolArrayExt; +use vortex::array::validity::Validity; use vortex::buffer::BitBuffer; use vortex::error::VortexResult; use vortex::mask::Mask; -use crate::duckdb::LogicalType; use crate::duckdb::VectorRef; use crate::exporter::ColumnExporter; use crate::exporter::all_invalid; @@ -24,11 +24,12 @@ pub(crate) fn new_exporter( ) -> VortexResult> { let len = array.len(); let bits = array.to_bit_buffer(); - let validity = array.validity()?.to_array(len).execute::(ctx)?; - if validity.all_false() { - return Ok(all_invalid::new_exporter(len, &LogicalType::bool())); + let validity = array.validity()?; + if matches!(validity, Validity::AllInvalid) { + return Ok(all_invalid::new_exporter()); } + let validity = validity.to_array(len).execute::(ctx)?; Ok(validity::new_exporter( validity, diff --git a/vortex-duckdb/src/exporter/cache.rs b/vortex-duckdb/src/exporter/cache.rs index 3b0fd496360..2f495ba9608 100644 --- a/vortex-duckdb/src/exporter/cache.rs +++ b/vortex-duckdb/src/exporter/cache.rs @@ -21,4 +21,5 @@ pub struct ConversionCache { pub dict_cache: DashMap, pub values_cache: DashMap>)>, pub canonical_cache: DashMap, + pub file_index: usize, } diff --git a/vortex-duckdb/src/exporter/constant.rs b/vortex-duckdb/src/exporter/constant.rs index 8bb05ac1edc..587fe9e9472 100644 --- a/vortex-duckdb/src/exporter/constant.rs +++ b/vortex-duckdb/src/exporter/constant.rs @@ -65,14 +65,13 @@ impl ColumnExporter for ConstantExporter { fn export( &self, _offset: usize, - len: usize, + _len: usize, vector: &mut VectorRef, _ctx: &mut ExecutionCtx, ) -> VortexResult<()> { match self.value.as_ref() { None => { - // TODO(ngates): would be good if DuckDB supported constant null vectors. - unsafe { vector.set_validity(&Mask::AllFalse(len), 0, len) }; + vector.set_all_false_validity(); } Some(value) => { vector.reference_value(value); diff --git a/vortex-duckdb/src/exporter/decimal.rs b/vortex-duckdb/src/exporter/decimal.rs index 2fe226b7f31..f6b8d9607e7 100644 --- a/vortex-duckdb/src/exporter/decimal.rs +++ b/vortex-duckdb/src/exporter/decimal.rs @@ -8,9 +8,9 @@ use vortex::array::ExecutionCtx; use vortex::array::arrays::DecimalArray; use vortex::array::arrays::decimal::DecimalDataParts; use vortex::array::match_each_decimal_value_type; +use vortex::array::validity::Validity; use vortex::buffer::Buffer; use vortex::dtype::BigCast; -use vortex::dtype::DType; use vortex::dtype::DecimalDType; use vortex::dtype::DecimalType; use vortex::dtype::NativeDecimalType; @@ -19,7 +19,6 @@ use vortex::error::VortexResult; use vortex::error::vortex_bail; use vortex::mask::Mask; -use crate::duckdb::LogicalType; use crate::duckdb::VectorBuffer; use crate::duckdb::VectorRef; use crate::exporter::ColumnExporter; @@ -49,13 +48,11 @@ pub(crate) fn new_exporter( values, } = array.into_data_parts(); let dest_values_type = precision_to_duckdb_storage_size(&decimal_dtype)?; - let nullability = validity.nullability(); - let validity = validity.to_array(len).execute::(ctx)?; - if validity.all_false() { - let ltype = LogicalType::try_from(DType::Decimal(decimal_dtype, nullability))?; - return Ok(all_invalid::new_exporter(len, <ype)); + if matches!(validity, Validity::AllInvalid) { + return Ok(all_invalid::new_exporter()); } + let validity = validity.to_array(len).execute::(ctx)?; let exporter = if values_type == dest_values_type { match_each_decimal_value_type!(values_type, |D| { @@ -138,6 +135,7 @@ pub fn precision_to_duckdb_storage_size(decimal_dtype: &DecimalDType) -> VortexR #[cfg(test)] mod tests { + use vortex::array::LEGACY_SESSION; use vortex::array::VortexSessionExecute; use vortex::array::arrays::DecimalArray; use vortex::dtype::DecimalDType; @@ -151,7 +149,10 @@ mod tests { pub(crate) fn new_zero_copy_exporter( array: &DecimalArray, ) -> VortexResult> { - let validity = array.validity_mask()?; + let validity = array.as_ref().validity()?.execute_mask( + array.as_ref().len(), + &mut LEGACY_SESSION.create_execution_ctx(), + )?; let dest_values_type = precision_to_duckdb_storage_size(&array.decimal_dtype())?; assert_eq!(array.values_type(), dest_values_type); diff --git a/vortex-duckdb/src/exporter/dict.rs b/vortex-duckdb/src/exporter/dict.rs index 961c7e35b84..da03cd4254c 100644 --- a/vortex-duckdb/src/exporter/dict.rs +++ b/vortex-duckdb/src/exporter/dict.rs @@ -17,7 +17,6 @@ use vortex::dtype::IntegerPType; use vortex::error::VortexResult; use vortex::mask::Mask; -use crate::duckdb::LogicalType; use crate::duckdb::ReusableDict; use crate::duckdb::SelectionVector; use crate::duckdb::VectorRef; @@ -43,21 +42,23 @@ pub(crate) fn new_exporter_with_flatten( ) -> VortexResult> { // Grab the cache dictionary values. let values = array.values(); - let values_type: LogicalType = values.dtype().try_into()?; + let codes = array.codes(); + let codes_len = codes.len(); + if let Some(constant) = values.as_opt::() { return constant::new_exporter_with_mask( - ConstantArray::new(constant.scalar().clone(), array.codes().len()), - array.codes().validity_mask()?, + ConstantArray::new(constant.scalar().clone(), codes_len), + codes.validity()?.execute_mask(codes_len, ctx)?, cache, ctx, ); } - let codes_mask = array.codes().validity_mask()?; + let codes_mask = codes.validity()?.execute_mask(codes_len, ctx)?; match codes_mask { Mask::AllTrue(_) => {} - Mask::AllFalse(len) => return Ok(all_invalid::new_exporter(len, &values_type)), + Mask::AllFalse(_) => return Ok(all_invalid::new_exporter()), Mask::Values(_) => { // duckdb cannot have a dictionary with validity in the codes, so flatten the array and // apply the validity mask there. @@ -76,7 +77,7 @@ pub(crate) fn new_exporter_with_flatten( let canonical = match canonical { Some(c) => c, None => { - let canonical = values.to_canonical()?; + let canonical = values.clone().execute::(ctx)?; cache .canonical_cache .insert(values_key, (values.clone(), canonical.clone())); @@ -191,15 +192,12 @@ mod tests { let mut chunk = DataChunk::new([LogicalType::new(cpp::duckdb_type::DUCKDB_TYPE_INTEGER)]); - new_exporter(&arr, &ConversionCache::default()) - .unwrap() - .export( - 0, - 2, - chunk.get_vector_mut(0), - &mut SESSION.create_execution_ctx(), - ) - .unwrap(); + new_exporter(&arr, &ConversionCache::default())?.export( + 0, + 2, + chunk.get_vector_mut(0), + &mut SESSION.create_execution_ctx(), + )?; chunk.set_len(2); assert_eq!( @@ -222,10 +220,12 @@ mod tests { let mut chunk = DataChunk::new([LogicalType::new(cpp::duckdb_type::DUCKDB_TYPE_INTEGER)]); let mut ctx = ExecutionCtx::new(VortexSession::default()); - new_exporter_with_flatten(&arr, &ConversionCache::default(), &mut ctx, false) - .unwrap() - .export(0, 2, chunk.get_vector_mut(0), &mut ctx) - .unwrap(); + new_exporter_with_flatten(&arr, &ConversionCache::default(), &mut ctx, false)?.export( + 0, + 2, + chunk.get_vector_mut(0), + &mut ctx, + )?; chunk.set_len(2); assert_eq!( @@ -247,15 +247,12 @@ mod tests { let mut chunk = DataChunk::new([LogicalType::new(cpp::duckdb_type::DUCKDB_TYPE_INTEGER)]); - new_exporter(&arr, &ConversionCache::default()) - .unwrap() - .export( - 0, - 3, - chunk.get_vector_mut(0), - &mut SESSION.create_execution_ctx(), - ) - .unwrap(); + new_exporter(&arr, &ConversionCache::default())?.export( + 0, + 3, + chunk.get_vector_mut(0), + &mut SESSION.create_execution_ctx(), + )?; chunk.set_len(3); // some-invalid codes cannot be exported as a dictionary. @@ -270,10 +267,12 @@ mod tests { DataChunk::new([LogicalType::new(cpp::duckdb_type::DUCKDB_TYPE_INTEGER)]); let mut ctx = SESSION.create_execution_ctx(); - new_array_exporter(arr.into_array(), &ConversionCache::default(), &mut ctx) - .unwrap() - .export(0, 3, flat_chunk.get_vector_mut(0), &mut ctx) - .unwrap(); + new_array_exporter(arr.into_array(), &ConversionCache::default(), &mut ctx)?.export( + 0, + 3, + flat_chunk.get_vector_mut(0), + &mut ctx, + )?; flat_chunk.set_len(3); assert_eq!( @@ -296,15 +295,12 @@ mod tests { let mut chunk = DataChunk::new([LogicalType::new(cpp::duckdb_type::DUCKDB_TYPE_INTEGER)]); - new_exporter(&arr, &ConversionCache::default()) - .unwrap() - .export( - 0, - 0, - chunk.get_vector_mut(0), - &mut SESSION.create_execution_ctx(), - ) - .unwrap(); + new_exporter(&arr, &ConversionCache::default())?.export( + 0, + 0, + chunk.get_vector_mut(0), + &mut SESSION.create_execution_ctx(), + )?; chunk.set_len(0); assert_eq!( diff --git a/vortex-duckdb/src/exporter/fixed_size_list.rs b/vortex-duckdb/src/exporter/fixed_size_list.rs index d7fbfa64647..ed93ad2b9c1 100644 --- a/vortex-duckdb/src/exporter/fixed_size_list.rs +++ b/vortex-duckdb/src/exporter/fixed_size_list.rs @@ -11,6 +11,7 @@ use vortex::array::ExecutionCtx; use vortex::array::arrays::FixedSizeListArray; use vortex::array::arrays::fixed_size_list::FixedSizeListArrayExt; +use vortex::array::validity::Validity; use vortex::error::VortexResult; use vortex::mask::Mask; @@ -18,7 +19,6 @@ use super::ConversionCache; use super::all_invalid; use super::new_array_exporter_with_flatten; use super::validity; -use crate::duckdb::LogicalType; use crate::duckdb::VectorRef; use crate::exporter::ColumnExporter; @@ -42,15 +42,14 @@ pub(crate) fn new_exporter( let parts = array.into_data_parts(); let elements = parts.elements; let validity = parts.validity; - let dtype = parts.dtype; - let mask = validity.to_array(len).execute::(ctx)?; - let elements_exporter = new_array_exporter_with_flatten(elements, cache, ctx, true)?; - if mask.all_false() { - let ltype = LogicalType::try_from(dtype)?; - return Ok(all_invalid::new_exporter(len, <ype)); + if matches!(validity, Validity::AllInvalid) { + return Ok(all_invalid::new_exporter()); } + let mask = validity.to_array(len).execute::(ctx)?; + let elements_exporter = new_array_exporter_with_flatten(elements, cache, ctx, true)?; + Ok(validity::new_exporter( mask, Box::new(FixedSizeListExporter { diff --git a/vortex-duckdb/src/exporter/list.rs b/vortex-duckdb/src/exporter/list.rs index 1ae14869908..7e7d024e11c 100644 --- a/vortex-duckdb/src/exporter/list.rs +++ b/vortex-duckdb/src/exporter/list.rs @@ -10,6 +10,7 @@ use vortex::array::arrays::ListArray; use vortex::array::arrays::PrimitiveArray; use vortex::array::arrays::list::ListDataParts; use vortex::array::match_each_integer_ptype; +use vortex::array::validity::Validity; use vortex::dtype::IntegerPType; use vortex::error::VortexResult; use vortex::error::vortex_err; @@ -18,6 +19,7 @@ use vortex::mask::Mask; use super::ConversionCache; use super::all_invalid; use super::new_array_exporter_with_flatten; +use super::validity; use crate::cpp; use crate::duckdb::LogicalType; use crate::duckdb::Vector; @@ -25,7 +27,6 @@ use crate::duckdb::VectorRef; use crate::exporter::ColumnExporter; struct ListExporter { - validity: Mask, /// We cache the child elements of our list array so that we don't have to export it every time, /// and we also share it across any other exporters who want to export this array. /// @@ -49,15 +50,14 @@ pub(crate) fn new_exporter( elements, offsets, validity, - dtype, + dtype: _dtype, } = array.into_data_parts(); let num_elements = elements.len(); - let validity = validity.to_array(array_len).execute::(ctx)?; - if validity.all_false() { - let ltype = LogicalType::try_from(dtype)?; - return Ok(all_invalid::new_exporter(array_len, <ype)); + if matches!(validity, Validity::AllInvalid) { + return Ok(all_invalid::new_exporter()); } + let validity = validity.to_array(array_len).execute::(ctx)?; let values_key = elements.addr(); // Check if we have a cached vector and extract it if we do. @@ -92,7 +92,6 @@ pub(crate) fn new_exporter( let boxed = match_each_integer_ptype!(offsets.ptype(), |O| { Box::new(ListExporter { - validity, duckdb_elements: shared_elements, offsets, num_elements, @@ -100,7 +99,7 @@ pub(crate) fn new_exporter( }) as Box }); - Ok(boxed) + Ok(validity::new_exporter(validity, boxed)) } impl ColumnExporter for ListExporter { @@ -111,21 +110,6 @@ impl ColumnExporter for ListExporter { vector: &mut VectorRef, _ctx: &mut ExecutionCtx, ) -> VortexResult<()> { - // Verify that offset + len doesn't exceed the validity mask length. - assert!( - offset + len <= self.validity.len(), - "Export range [{}, {}) exceeds validity mask length {}", - offset, - offset + len, - self.validity.len() - ); - - // Set validity if necessary. - if unsafe { vector.set_validity(&self.validity, offset, len) } { - // All values are null, so no point copying the data. - return Ok(()); - } - let offsets = &self.offsets.as_slice::()[offset..offset + len + 1]; debug_assert_eq!(offsets.len(), len + 1); diff --git a/vortex-duckdb/src/exporter/list_view.rs b/vortex-duckdb/src/exporter/list_view.rs index dcc7c66ce31..5b2f19c09d7 100644 --- a/vortex-duckdb/src/exporter/list_view.rs +++ b/vortex-duckdb/src/exporter/list_view.rs @@ -10,7 +10,7 @@ use vortex::array::arrays::ListViewArray; use vortex::array::arrays::PrimitiveArray; use vortex::array::arrays::listview::ListViewDataParts; use vortex::array::match_each_integer_ptype; -use vortex::dtype::DType; +use vortex::array::validity::Validity; use vortex::dtype::IntegerPType; use vortex::error::VortexResult; use vortex::error::vortex_err; @@ -19,6 +19,7 @@ use vortex::mask::Mask; use super::ConversionCache; use super::all_invalid; use super::new_array_exporter_with_flatten; +use super::validity; use crate::cpp; use crate::duckdb::LogicalType; use crate::duckdb::Vector; @@ -26,7 +27,6 @@ use crate::duckdb::VectorRef; use crate::exporter::ColumnExporter; struct ListViewExporter { - validity: Mask, /// We cache the child elements of our list array so that we don't have to export it every time, /// and we also share it across any other exporters who want to export this array. /// @@ -56,13 +56,11 @@ pub(crate) fn new_exporter( } = array.into_data_parts(); // Cache an `elements` vector up front so that future exports can reference it. let num_elements = elements.len(); - let nullability = validity.nullability(); - let validity = validity.to_array(len).execute::(ctx)?; - if validity.all_false() { - let ltype = LogicalType::try_from(DType::List(elements_dtype, nullability))?; - return Ok(all_invalid::new_exporter(len, <ype)); + if matches!(validity, Validity::AllInvalid) { + return Ok(all_invalid::new_exporter()); } + let validity = validity.to_array(len).execute::(ctx)?; let values_key = elements.addr(); // Check if we have a cached vector and extract it if we do. @@ -99,7 +97,6 @@ pub(crate) fn new_exporter( let boxed = match_each_integer_ptype!(offsets.ptype(), |O| { match_each_integer_ptype!(sizes.ptype(), |S| { Box::new(ListViewExporter { - validity, duckdb_elements: shared_elements, offsets, sizes, @@ -110,7 +107,7 @@ pub(crate) fn new_exporter( }) }); - Ok(boxed) + Ok(validity::new_exporter(validity, boxed)) } impl ColumnExporter for ListViewExporter { @@ -121,21 +118,6 @@ impl ColumnExporter for ListViewExporter vector: &mut VectorRef, _ctx: &mut ExecutionCtx, ) -> VortexResult<()> { - // Verify that offset + len doesn't exceed the validity mask length. - assert!( - offset + len <= self.validity.len(), - "Export range [{}, {}) exceeds validity mask length {}", - offset, - offset + len, - self.validity.len() - ); - - // Set validity if necessary. - if unsafe { vector.set_validity(&self.validity, offset, len) } { - // All values are null, so no point copying the data. - return Ok(()); - } - let offsets = &self.offsets.as_slice::()[offset..offset + len]; let sizes = &self.sizes.as_slice::()[offset..offset + len]; debug_assert_eq!(offsets.len(), len); diff --git a/vortex-duckdb/src/exporter/mod.rs b/vortex-duckdb/src/exporter/mod.rs index 502038b858c..517776f5521 100644 --- a/vortex-duckdb/src/exporter/mod.rs +++ b/vortex-duckdb/src/exporter/mod.rs @@ -19,8 +19,6 @@ mod validity; mod varbinview; mod vector; -use bitvec::prelude::Lsb0; -use bitvec::view::BitView; pub use cache::ConversionCache; pub use decimal::precision_to_duckdb_storage_size; use vortex::array::ArrayRef; @@ -32,13 +30,14 @@ use vortex::array::arrays::List; use vortex::array::arrays::StructArray; use vortex::array::arrays::TemporalArray; use vortex::array::arrays::struct_::StructArrayExt; +use vortex::buffer::BitChunks; use vortex::encodings::runend::RunEnd; use vortex::encodings::sequence::Sequence; +use vortex::error::VortexExpect; use vortex::error::VortexResult; use vortex::error::vortex_bail; use crate::duckdb::DataChunkRef; -use crate::duckdb::LogicalType; use crate::duckdb::VectorRef; use crate::duckdb::duckdb_vector_size; @@ -76,15 +75,22 @@ impl ArrayExporter { /// Export the data into the next chunk. /// /// Returns `true` if a chunk was exported, `false` if all rows have been exported. - pub fn export(&mut self, chunk: &mut DataChunkRef) -> VortexResult { + pub fn export( + &mut self, + chunk: &mut DataChunkRef, + file_index_column_pos: Option, + file_row_number_column_pos: Option, + ) -> VortexResult { chunk.reset(); if self.remaining == 0 { return Ok(false); } - let expected_cols = self.fields.len(); + let zero_projection = self.fields.is_empty(); + + // file_row_number column is already populated in scan construction + let expected_cols = self.fields.len() + file_index_column_pos.is_some() as usize; let chunk_cols = chunk.column_count(); - let zero_projection = expected_cols == 0; if !zero_projection && chunk_cols != expected_cols { vortex_bail!("Expected {expected_cols} columns in output chunk, got {chunk_cols}"); } @@ -94,14 +100,48 @@ impl ArrayExporter { self.remaining -= chunk_len; chunk.set_len(chunk_len); - // DuckDB asked us for zero columns. This may happen with aggregation - // functions like count(*). In such case we can leave chunk contents - // uninitialized. See EMPTY_COLUMN_IDX comment why this works. + // If DuckDB requests a zero-column projection from read_vortex like count(*), + // its planner tries to get any column: + // See duckdb/src/planner/operator/logical_get.cpp#L149 + // + // If you define COLUMN_IDENTIFIER_EMPTY, planner takes it, otherwise the + // first column. As we don't want to fill the output chunk and we can leave + // it uninitialized in this case, we define COLUMN_IDENTIFIER_EMPTY as a + // virtual column. + // See virtual_columns in vortex-duckdb/cpp/table_function.cpp if zero_projection { return Ok(true); } - for (i, field) in self.fields.iter_mut().enumerate() { + let mut fields = self.fields.iter(); + // file_row_number column is the first one if present. + if let Some(pos) = file_row_number_column_pos { + let field = fields.next().vortex_expect("field column mismatch"); + field.export( + position, + chunk_len, + chunk.get_vector_mut(pos), + &mut self.ctx, + )?; + } + + for i in 0..chunk_cols { + // file_index column: skip index - it will be filled after + // chunk export. + if let Some(pos) = file_index_column_pos + && i == pos + { + continue; + } + + // file_row_number column: skip index, already filled + if let Some(pos) = file_row_number_column_pos + && i == pos + { + continue; + } + + let field = fields.next().vortex_expect("field count mismatch"); field.export(position, chunk_len, chunk.get_vector_mut(i), &mut self.ctx)?; } @@ -167,7 +207,7 @@ fn new_array_exporter_with_flatten( // Otherwise, we fall back to canonical match array.execute::(ctx)? { - Canonical::Null(array) => Ok(all_invalid::new_exporter(array.len(), &LogicalType::null())), + Canonical::Null(_) => Ok(all_invalid::new_exporter()), Canonical::Bool(array) => bool::new_exporter(array, ctx), Canonical::Primitive(array) => primitive::new_exporter(array, ctx), Canonical::Decimal(array) => decimal::new_exporter(array, ctx), @@ -187,17 +227,33 @@ fn new_array_exporter_with_flatten( } } -/// Copy the sliced bits from source into target. +/// Copy the sliced bits from source into target, returning whether all copied bits are zero, +/// all are one, or mixed. /// -/// Offset and length are a _bit_ offset and a _bit_ length into source. +/// offset and len are a _bit_ offset and a _bit_ length into `source`. /// -/// `target.len()` must equal `len`. -fn copy_from_slice(target: &mut [u64], source: &[u8], offset: usize, len: usize) { - let (start, middle, end) = unsafe { target.align_to_mut::() }; - assert!(start.is_empty()); - assert!(end.is_empty()); - let target = &mut middle.view_bits_mut::()[..len]; - target.copy_from_bitslice(&source.view_bits()[offset..][..len]); +/// target must have at least len.div_ceil(64) elements. +/// Returns the number of ones in copied slice. +fn copy_from_slice(target: &mut [u64], source: &[u8], offset: usize, len: usize) -> usize { + if len == 0 { + return 0; + } + + let mut ones = 0usize; + let chunks = BitChunks::new(source, offset, len); + let chunk_len = chunks.chunk_len(); + let remainder_len = chunks.remainder_len(); + let remainder = chunks.remainder_bits(); + for (slot, chunk) in target.iter_mut().zip(chunks) { + *slot = chunk; + ones += chunk.count_ones() as usize; + } + + if remainder_len > 0 { + target[chunk_len] = remainder; + ones += remainder.count_ones() as usize; + } + ones } #[cfg(test)] @@ -359,23 +415,23 @@ mod tests { fn test_copy_from_slice_empty_to_empty() { let target = &mut []; let source = Vec::::new(); - copy_from_slice(target, &source, 0, 0); + assert_eq!(copy_from_slice(target, &source, 0, 0), 0); } #[test] fn test_copy_from_slice_64_to_empty() { let target = &mut []; let source = [1u8, 2, 3, 50, 51, 52, 100, 101]; - copy_from_slice(target, &source, 0, 0); - copy_from_slice(target, &source, 5, 0); - copy_from_slice(target, &source, 8, 0); + assert_eq!(copy_from_slice(target, &source, 0, 0), 0); + assert_eq!(copy_from_slice(target, &source, 5, 0), 0); + assert_eq!(copy_from_slice(target, &source, 8, 0), 0); } #[test] fn test_copy_from_slice_64_to_64() { let mut target = vec![0u64]; let source = [1u8, 2, 3, 50, 51, 52, 100, 101]; - copy_from_slice(&mut target, &source, 0, 64); + assert_eq!(copy_from_slice(&mut target, &source, 0, 64), 21); assert_eq!( target[0], 0x65_64_34_33_32_03_02_01_u64, "{:#08x} == {:#08x}", @@ -383,20 +439,27 @@ mod tests { ); } + #[test] + fn test_copy_from_slice_64_ones() { + let mut target = [0u64]; + let source = [u8::MAX; 8]; + assert_eq!(copy_from_slice(&mut target, &source, 0, 64), 64); + } + #[test] fn test_copy_from_slice_80_to_0() { let target = &mut []; let source = [1u8, 2, 3, 50, 51, 52, 100, 101, 254, 255]; - copy_from_slice(target, &source, 0, 0); - copy_from_slice(target, &source, 8, 0); - copy_from_slice(target, &source, 10, 0); + assert_eq!(copy_from_slice(target, &source, 0, 0), 0); + assert_eq!(copy_from_slice(target, &source, 8, 0), 0); + assert_eq!(copy_from_slice(target, &source, 10, 0), 0,); } #[test] fn test_copy_from_slice_80_to_64_case_1() { let mut target = [0u64]; let source = [1u8, 2, 3, 50, 51, 52, 100, 101, 254, 255]; - copy_from_slice(&mut target, &source, 16, 64); + assert_eq!(copy_from_slice(&mut target, &source, 16, 64), 34); assert_eq!( target[0], 0xff_fe_65_64_34_33_32_03_u64, "{:#08x} == {:#08x}", @@ -408,7 +471,7 @@ mod tests { fn test_copy_from_slice_80_to_64_case_2() { let mut target = [0u64]; let source = [1u8, 2, 3, 50, 51, 52, 100, 101, 254, 255]; - copy_from_slice(&mut target, &source, 8, 64); + assert_eq!(copy_from_slice(&mut target, &source, 8, 64), 27); assert_eq!( target[0], 0xfe_65_64_34_33_32_03_02_u64, "{:#08x} == {:#08x}", @@ -420,7 +483,7 @@ mod tests { fn test_copy_from_slice_80_to_64_case_3() { let mut target = [0u64]; let source = [1u8, 2, 3, 50, 51, 52, 100, 101, 254, 255]; - copy_from_slice(&mut target, &source, 0, 64); + assert_eq!(copy_from_slice(&mut target, &source, 0, 64), 21,); assert_eq!( target[0], 0x65_64_34_33_32_03_02_01_u64, "{:#08x} == {:#08x}", @@ -432,7 +495,7 @@ mod tests { fn test_copy_from_slice_80_to_64_case_4() { let mut target = [0u64]; let source = [1u8, 2, 3, 50, 51, 52, 100, 101, 254, 255]; - copy_from_slice(&mut target, &source, 10, 64); + assert_eq!(copy_from_slice(&mut target, &source, 10, 64), 28,); assert_eq!( target[0], 0xff_99_59_0d_0c_cc_80_c0_u64, // Python: hex(0xff_fe_65_64_34_33_32_03_02 >> 2), then remove the high two hexits @@ -454,7 +517,7 @@ mod tests { let (_, middle, _) = unsafe { source.align_to::() }; assert!(!middle.is_empty()); - copy_from_slice(&mut target, &source, 0, 128); + assert_eq!(copy_from_slice(&mut target, &source, 0, 128), 66,); assert_eq!( target[0], 0xfc_fd_fe_ff_04_03_02_01_u64, "{:#08x} == {:#08x}", @@ -466,7 +529,7 @@ mod tests { target[1], 0xf9_fa_fb_fc_08_07_06_05_u64, ); - copy_from_slice(&mut target, &source, 8, 128); + assert_eq!(copy_from_slice(&mut target, &source, 8, 128), 66); assert_eq!( target[0], 0x05_fc_fd_fe_ff_04_03_02_u64, "{:#08x} == {:#08x}", @@ -478,7 +541,7 @@ mod tests { target[1], 0x01_f9_fa_fb_fc_08_07_06_u64, ); - copy_from_slice(&mut target, &source, 8 * 8, 128); + assert_eq!(copy_from_slice(&mut target, &source, 8 * 8, 128), 66,); assert_eq!( target[0], 0xf9_fa_fb_fc_08_07_06_05_u64, "{:#08x} == {:#08x}", @@ -490,7 +553,7 @@ mod tests { target[1], 0xfc_fd_fe_ff_04_03_02_01_u64, ); - copy_from_slice(&mut target, &source, 8 * 12, 128); + assert_eq!(copy_from_slice(&mut target, &source, 8 * 12, 128), 66,); assert_eq!( target[0], 0x04_03_02_01_f9_fa_fb_fc_u64, "{:#08x} == {:#08x}", @@ -502,7 +565,7 @@ mod tests { target[1], 0x08_07_06_05_fc_fd_fe_ff_u64, ); - copy_from_slice(&mut target, &source, 8 * 12 + 4, 128); + assert_eq!(copy_from_slice(&mut target, &source, 8 * 12 + 4, 128), 66,); // Find the 12th byte, skip the first hexit, take the next 32 hexits (i.e. 16 bytesor 128 // bits). assert_eq!( @@ -517,7 +580,10 @@ mod tests { ); // Take the above and shift one bit towards the right-hand-side. - copy_from_slice(&mut target, &source, 8 * 12 + 4 + 1, 128); + assert_eq!( + copy_from_slice(&mut target, &source, 8 * 12 + 4 + 1, 128), + 66, + ); assert_eq!( target[0], 0xf8_20_18_10_0f_cf_d7_df_u64, "{:#08x} == {:#08x}", diff --git a/vortex-duckdb/src/exporter/primitive.rs b/vortex-duckdb/src/exporter/primitive.rs index 7506cbdb93c..a0b08c80e4b 100644 --- a/vortex-duckdb/src/exporter/primitive.rs +++ b/vortex-duckdb/src/exporter/primitive.rs @@ -6,11 +6,11 @@ use std::marker::PhantomData; use vortex::array::ExecutionCtx; use vortex::array::arrays::PrimitiveArray; use vortex::array::match_each_native_ptype; +use vortex::array::validity::Validity; use vortex::dtype::NativePType; use vortex::error::VortexResult; use vortex::mask::Mask; -use crate::duckdb::LogicalType; use crate::duckdb::VectorBuffer; use crate::duckdb::VectorRef; use crate::exporter::ColumnExporter; @@ -28,15 +28,11 @@ pub fn new_exporter( array: PrimitiveArray, ctx: &mut ExecutionCtx, ) -> VortexResult> { - let validity = array - .validity()? - .to_array(array.len()) - .execute::(ctx)?; - - if validity.all_false() { - let ltype = LogicalType::try_from(array.ptype())?; - return Ok(all_invalid::new_exporter(array.len(), <ype)); - } + let validity = array.validity()?; + if matches!(validity, Validity::AllInvalid) { + return Ok(all_invalid::new_exporter()); + }; + let validity = validity.to_array(array.len()).execute::(ctx)?; match_each_native_ptype!(array.ptype(), |T| { let buffer = array.to_buffer::(); @@ -101,6 +97,78 @@ mod tests { ); } + #[test] + fn test_primitive_exporter_with_nulls() { + let arr = PrimitiveArray::from_option_iter([Some(10i32), None, Some(30), None, Some(50)]); + + let mut chunk = DataChunk::new([LogicalType::new(cpp::duckdb_type::DUCKDB_TYPE_INTEGER)]); + let mut ctx = SESSION.create_execution_ctx(); + + new_exporter(arr, &mut ctx) + .unwrap() + .export(0, 5, chunk.get_vector_mut(0), &mut ctx) + .unwrap(); + chunk.set_len(5); + + assert_eq!( + format!("{}", String::try_from(&*chunk).unwrap()), + r#"Chunk - [1 Columns] +- FLAT INTEGER: 5 = [ 10, NULL, 30, NULL, 50] +"# + ); + } + + /// Export a large nullable primitive array over many chunks to exercise the + /// zero-copy validity path. The non-zero-copy fallback currently panics, + /// so this test proves every chunk goes through the zero-copy branch. + #[test] + fn test_primitive_exporter_with_nulls_zero_copy() { + let vector_size = duckdb_vector_size(); + const NUM_CHUNKS: usize = 8; + let len = vector_size * NUM_CHUNKS; + + // Every 3rd element is null — guarantees mixed validity in every chunk. + #[expect(clippy::cast_possible_truncation, reason = "test data fits in i32")] + let arr = PrimitiveArray::from_option_iter( + (0..len).map(|i| if i % 3 == 1 { None } else { Some(i as i32) }), + ); + + let mut ctx = SESSION.create_execution_ctx(); + let exporter = new_exporter(arr, &mut ctx).unwrap(); + + for chunk_idx in 0..NUM_CHUNKS { + let mut chunk = + DataChunk::new([LogicalType::new(cpp::duckdb_type::DUCKDB_TYPE_INTEGER)]); + + // This will panic if the non-zero-copy path is hit. + exporter + .export( + chunk_idx * vector_size, + vector_size, + chunk.get_vector_mut(0), + &mut ctx, + ) + .unwrap(); + chunk.set_len(vector_size); + + let vec = chunk.get_vector(0); + for i in 0..vector_size { + let global_idx = chunk_idx * vector_size + i; + if global_idx % 3 == 1 { + assert!( + vec.row_is_null(i as u64), + "expected null at global index {global_idx}" + ); + } else { + assert!( + !vec.row_is_null(i as u64), + "expected non-null at global index {global_idx}" + ); + } + } + } + } + #[test] fn test_long_primitive_exporter() { let vector_size = duckdb_vector_size(); diff --git a/vortex-duckdb/src/exporter/run_end.rs b/vortex-duckdb/src/exporter/run_end.rs index fa62388cf57..442e612147c 100644 --- a/vortex-duckdb/src/exporter/run_end.rs +++ b/vortex-duckdb/src/exporter/run_end.rs @@ -86,7 +86,7 @@ impl ColumnExporter for RunEndExporter { if start_run_idx == end_run_idx { // NOTE(ngates): would be great if we could just export and set type == CONSTANT // self.values_exporter.export(start_run_idx, 1, vector, cache); - let constant = self.values.scalar_at(start_run_idx)?; + let constant = self.values.execute_scalar(start_run_idx, ctx)?; let value = constant.try_to_duckdb_scalar()?; vector.reference_value(&value); return Ok(()); diff --git a/vortex-duckdb/src/exporter/sequence.rs b/vortex-duckdb/src/exporter/sequence.rs index a9a569d2a06..bc4b2731b40 100644 --- a/vortex-duckdb/src/exporter/sequence.rs +++ b/vortex-duckdb/src/exporter/sequence.rs @@ -35,6 +35,7 @@ impl ColumnExporter for SequenceExporter { ) -> VortexResult<()> { let offset = offset.as_i64(); let start = (offset * self.step) + self.start; + // TODO why don't we apply validity mask here? vector.to_sequence(start, self.step, len.as_u64()); Ok(()) diff --git a/vortex-duckdb/src/exporter/struct_.rs b/vortex-duckdb/src/exporter/struct_.rs index f53ec437bf2..f18c5495754 100644 --- a/vortex-duckdb/src/exporter/struct_.rs +++ b/vortex-duckdb/src/exporter/struct_.rs @@ -8,9 +8,9 @@ use vortex::array::arrays::StructArray; use vortex::array::arrays::bool::BoolArrayExt; use vortex::array::arrays::struct_::StructDataParts; use vortex::array::builtins::ArrayBuiltins; +use vortex::array::validity::Validity; use vortex::error::VortexResult; -use crate::duckdb::LogicalType; use crate::duckdb::VectorRef; use crate::exporter::ColumnExporter; use crate::exporter::ConversionCache; @@ -30,16 +30,15 @@ pub(crate) fn new_exporter( let len = array.len(); let StructDataParts { validity, - struct_fields, + struct_fields: _struct_fields, fields, .. } = array.into_data_parts(); - let validity = validity.to_array(len).execute::(ctx)?; - if validity.to_bit_buffer().true_count() == 0 { - let ltype = LogicalType::try_from(struct_fields)?; - return Ok(all_invalid::new_exporter(len, <ype)); - } + if matches!(validity, Validity::AllInvalid) { + return Ok(all_invalid::new_exporter()); + }; + let validity = validity.to_array(len).execute::(ctx)?; let children = fields .iter() @@ -57,7 +56,7 @@ pub(crate) fn new_exporter( }) .collect::>>()?; Ok(validity::new_exporter( - validity.to_mask(), + validity.execute_mask(ctx), Box::new(StructExporter { children }), )) } diff --git a/vortex-duckdb/src/exporter/validity.rs b/vortex-duckdb/src/exporter/validity.rs index 76107240212..d1fbf8123cb 100644 --- a/vortex-duckdb/src/exporter/validity.rs +++ b/vortex-duckdb/src/exporter/validity.rs @@ -5,14 +5,42 @@ use vortex::array::ExecutionCtx; use vortex::error::VortexResult; use vortex::mask::Mask; +use crate::duckdb::ValidityData; +use crate::duckdb::VectorBuffer; use crate::duckdb::VectorRef; use crate::exporter::ColumnExporter; struct ValidityExporter { mask: Mask, + /// If the mask's bit buffer is u64-aligned with no sub-byte offset, + /// we can zero-copy it into DuckDB. We hold the ValidityData to keep + /// the underlying memory alive via DuckDB's ref-counting. + zero_copy: Option, exporter: Box, } +/// Returns true if the bit buffer can be zero-copied as a DuckDB validity mask. +/// +/// Requirements: +/// - No sub-byte bit offset (offset == 0) +/// - The underlying byte buffer is u64-aligned +/// - The underlying byte buffer length is a multiple of 8 (so u64 reads are in-bounds) +fn can_zero_copy_validity(mask: &Mask) -> bool { + let Mask::Values(values) = mask else { + return false; + }; + let bit_buf = values.bit_buffer(); + if bit_buf.offset() != 0 { + return false; + } + let inner = bit_buf.inner(); + let slice = inner.as_slice(); + // DuckDB reads validity as u64 words, so the buffer must be u64-aligned and + // its length must be a multiple of 8 bytes to avoid out-of-bounds reads. + (slice.as_ptr() as usize).is_multiple_of(size_of::()) + && slice.len().is_multiple_of(size_of::()) +} + pub(crate) fn new_exporter( mask: Mask, exporter: Box, @@ -20,7 +48,22 @@ pub(crate) fn new_exporter( if mask.all_true() { exporter } else { - Box::new(ValidityExporter { mask, exporter }) + let zero_copy = can_zero_copy_validity(&mask).then(|| { + let Mask::Values(values) = &mask else { + unreachable!() + }; + let buffer = values.bit_buffer().inner().clone(); + let data_ptr = buffer.as_slice().as_ptr(); + ValidityData { + shared_buffer: VectorBuffer::new(buffer), + data_ptr, + } + }); + Box::new(ValidityExporter { + mask, + zero_copy, + exporter, + }) } } @@ -36,7 +79,9 @@ impl ColumnExporter for ValidityExporter { offset + len <= self.mask.len(), "cannot access outside of array" ); - if unsafe { vector.set_validity(&self.mask, offset, len) } { + if unsafe { + vector.set_validity_zero_copy(&self.mask, offset, len, self.zero_copy.as_ref()) + } { // All values are null, so no point copying the data. return Ok(()); } diff --git a/vortex-duckdb/src/exporter/varbinview.rs b/vortex-duckdb/src/exporter/varbinview.rs index 8067b08a856..557795f45f3 100644 --- a/vortex-duckdb/src/exporter/varbinview.rs +++ b/vortex-duckdb/src/exporter/varbinview.rs @@ -9,12 +9,12 @@ use vortex::array::arrays::VarBinViewArray; use vortex::array::arrays::varbinview::BinaryView; use vortex::array::arrays::varbinview::Inlined; use vortex::array::arrays::varbinview::VarBinViewDataParts; +use vortex::array::validity::Validity; use vortex::buffer::Buffer; use vortex::buffer::ByteBuffer; use vortex::error::VortexResult; use vortex::mask::Mask; -use crate::duckdb::LogicalType; use crate::duckdb::VectorBuffer; use crate::duckdb::VectorRef; use crate::exporter::ColumnExporter; @@ -34,15 +34,15 @@ pub(crate) fn new_exporter( let len = array.len(); let VarBinViewDataParts { validity, - dtype, + dtype: _dtype, views, buffers, } = array.into_data_parts(); - let validity = validity.to_array(len).execute::(ctx)?; - if validity.all_false() { - let ltype = LogicalType::try_from(dtype)?; - return Ok(all_invalid::new_exporter(len, <ype)); + + if matches!(validity, Validity::AllInvalid) { + return Ok(all_invalid::new_exporter()); } + let validity = validity.to_array(len).execute::(ctx)?; let buffers: Vec<_> = buffers.iter().cloned().map(|b| b.unwrap_host()).collect(); let buffers: Arc<[ByteBuffer]> = Arc::from(buffers); @@ -161,7 +161,7 @@ mod tests { chunk.set_len(3); assert_eq!( - format!("{}", String::try_from(&*chunk).unwrap()), + format!("{}", String::try_from(&*chunk)?), r#"Chunk - [1 Columns] - CONSTANT VARCHAR: 3 = [ NULL] "# @@ -181,7 +181,7 @@ mod tests { chunk.set_len(3); assert_eq!( - format!("{}", String::try_from(&*chunk).unwrap()), + format!("{}", String::try_from(&*chunk)?), r#"Chunk - [1 Columns] - CONSTANT VARCHAR: 3 = [ NULL] "# @@ -203,7 +203,7 @@ mod tests { chunk.set_len(3); assert_eq!( - format!("{}", String::try_from(&*chunk).unwrap()), + format!("{}", String::try_from(&*chunk)?), r#"Chunk - [1 Columns] - FLAT VARCHAR: 3 = [ NULL, NULL, Hi] "# diff --git a/vortex-duckdb/src/exporter/vector.rs b/vortex-duckdb/src/exporter/vector.rs index 5e4cef97d11..3ab0c484318 100644 --- a/vortex-duckdb/src/exporter/vector.rs +++ b/vortex-duckdb/src/exporter/vector.rs @@ -3,31 +3,58 @@ use vortex::mask::Mask; +use crate::cpp::duckdb_vx_vector_set_all_valid; +use crate::duckdb::ValidityData; use crate::duckdb::Value; use crate::duckdb::VectorRef; use crate::exporter::copy_from_slice; impl VectorRef { - pub(super) unsafe fn set_validity(&mut self, mask: &Mask, offset: usize, len: usize) -> bool { + /// Returns true if all values are null (caller can skip data export). + pub unsafe fn set_validity(&mut self, mask: &Mask, offset: usize, len: usize) -> bool { + unsafe { self.set_validity_zero_copy(mask, offset, len, None) } + } + + /// Like [`set_validity`](Self::set_validity), but attempts a zero-copy path when + /// `zero_copy` is provided and the offset is u64-aligned. + /// + /// Returns true if all values are null (caller can skip data export). + pub(super) unsafe fn set_validity_zero_copy( + &mut self, + mask: &Mask, + offset: usize, + len: usize, + zero_copy: Option<&ValidityData>, + ) -> bool { match mask { Mask::AllTrue(_) => { - // We only need to blank out validity if there is already a slice allocated. - // SAFETY: Caller guarantees this. - unsafe { self.set_all_true_validity(len) } + self.set_all_true_validity(); false } Mask::AllFalse(_) => { - // SAFETY: Caller guarantees this. self.set_all_false_validity(); true } Mask::Values(arr) => { let true_count = arr.bit_buffer().slice(offset..(offset + len)).true_count(); if true_count == len { - unsafe { self.set_all_true_validity(len) } + self.set_all_true_validity() } else if true_count == 0 { self.set_all_false_validity() + } else if let Some(zc) = zero_copy.filter(|_| offset.is_multiple_of(64)) { + let u64_offset = offset / 64; + // SAFETY: the underlying buffer is u64-aligned (checked in + // can_zero_copy_validity) and the VectorBuffer keeps the data alive. + // data_ptr points into the buffer at the start of the validity bitmap. + unsafe { self.set_validity_data(u64_offset, len, zc) }; } else { + // If zero_copy is available and offset is aligned, we should + // have taken the branch above. Assert this invariant. + assert!( + zero_copy.is_none() || !offset.is_multiple_of(64), + "zero-copy validity available and offset {offset} is aligned \ + but copy path was taken" + ); let source = arr.bit_buffer().inner().as_slice(); copy_from_slice( unsafe { self.ensure_validity_slice(len) }, @@ -42,13 +69,11 @@ impl VectorRef { } } - pub(super) unsafe fn set_all_true_validity(&mut self, len: usize) { - if let Some(validity) = unsafe { self.validity_bitslice_mut(len) } { - validity.fill(true); - } + pub fn set_all_true_validity(&mut self) { + unsafe { duckdb_vx_vector_set_all_valid(self.as_ptr()) }; } - pub(super) fn set_all_false_validity(&mut self) { + pub fn set_all_false_validity(&mut self) { self.reference_value(&Value::null(&self.logical_type())); } } diff --git a/vortex-duckdb/src/filesystem.rs b/vortex-duckdb/src/filesystem.rs index e6d5c7c9750..8c09ed6b147 100644 --- a/vortex-duckdb/src/filesystem.rs +++ b/vortex-duckdb/src/filesystem.rs @@ -149,6 +149,27 @@ impl FileSystem for DuckDbFileSystem { let reader = unsafe { DuckDbFsReader::open_url(self.ctx.as_ptr(), &url)? }; Ok(Arc::new(reader)) } + + async fn delete(&self, path: &str) -> VortexResult<()> { + let mut url = self.base_url.clone(); + url.set_path(path); + let c_path = CString::new(url.as_str()).map_err(|e| vortex_err!("Invalid URL: {e}"))?; + let ctx = self.ctx; + + RUNTIME + .handle() + .spawn_blocking(move || { + let mut err: cpp::duckdb_vx_error = ptr::null_mut(); + let status = unsafe { + cpp::duckdb_vx_fs_remove(ctx.as_ptr(), c_path.as_ptr(), &raw mut err) + }; + if status != cpp::duckdb_state::DuckDBSuccess { + return Err(fs_error(err)); + } + Ok::<_, VortexError>(()) + }) + .await + } } /// Recursively list all files under `directory`, stripping `base_path` from each diff --git a/vortex-duckdb/src/lib.rs b/vortex-duckdb/src/lib.rs index 5e321fa2040..413a71c611e 100644 --- a/vortex-duckdb/src/lib.rs +++ b/vortex-duckdb/src/lib.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::missing_safety_doc)] +#![expect(clippy::missing_safety_doc)] use std::ffi::CStr; use std::ffi::c_char; @@ -21,6 +21,7 @@ use crate::duckdb::DatabaseRef; use crate::duckdb::LogicalType; use crate::duckdb::Value; use crate::multi_file::VortexMultiFileScan; +use crate::multi_file::VortexMultiFileScanList; mod convert; mod datasource; @@ -56,6 +57,9 @@ pub fn initialize(db: &DatabaseRef) -> VortexResult<()> { )?; db.register_table_function::(c"vortex_scan")?; db.register_table_function::(c"read_vortex")?; + // Register list overloads for multi-glob scanning (e.g., read_vortex(['a.vortex', 'b.vortex'])) + db.register_table_function::(c"vortex_scan")?; + db.register_table_function::(c"read_vortex")?; db.register_copy_function::(c"vortex", c"vortex") } diff --git a/vortex-duckdb/src/multi_file.rs b/vortex-duckdb/src/multi_file.rs index e0d04d6c2b9..3f99a854a22 100644 --- a/vortex-duckdb/src/multi_file.rs +++ b/vortex-duckdb/src/multi_file.rs @@ -5,18 +5,23 @@ use std::path::Path; use std::path::absolute; use std::sync::Arc; +use itertools::Itertools; use url::Url; use vortex::error::VortexResult; +use vortex::error::vortex_bail; use vortex::error::vortex_err; use vortex::file::multi::MultiFileDataSource; +use vortex::io::filesystem::FileSystemRef; use vortex::io::runtime::BlockingRuntime; -use vortex::scan::DataSourceRef; +use vortex::layout::scan::multi::MultiLayoutDataSource; +use vortex_utils::aliases::hash_map::HashMap; use crate::RUNTIME; use crate::SESSION; use crate::datasource::DataSourceTableFunction; use crate::duckdb::BindInputRef; use crate::duckdb::ClientContextRef; +use crate::duckdb::ExtractedValue; use crate::duckdb::LogicalType; use crate::filesystem::resolve_filesystem; @@ -60,39 +65,98 @@ fn normalize_path(path: std::path::PathBuf) -> std::path::PathBuf { #[derive(Debug)] pub struct VortexMultiFileScan; +/// Variant of [`VortexMultiFileScan`] that accepts a list of globs. +/// +/// This is registered as a separate overload to handle `read_vortex(['a.vortex', 'b.vortex'])`. +#[derive(Debug)] +pub struct VortexMultiFileScanList; + impl DataSourceTableFunction for VortexMultiFileScan { fn parameters() -> Vec { vec![LogicalType::varchar()] } - fn bind(ctx: &ClientContextRef, input: &BindInputRef) -> VortexResult { - let glob_url_parameter = input - .get_parameter(0) - .ok_or_else(|| vortex_err!("Missing file glob parameter"))?; + fn bind(ctx: &ClientContextRef, input: &BindInputRef) -> VortexResult { + bind_multi_file_scan(ctx, input) + } +} + +impl DataSourceTableFunction for VortexMultiFileScanList { + fn parameters() -> Vec { + vec![ + LogicalType::list_type(LogicalType::varchar()) + .unwrap_or_else(|_| unreachable!("creating list type should not fail")), + ] + } + + fn bind(ctx: &ClientContextRef, input: &BindInputRef) -> VortexResult { + bind_multi_file_scan(ctx, input) + } +} + +/// Shared bind logic for both single-glob and multi-glob variants. +fn bind_multi_file_scan( + ctx: &ClientContextRef, + input: &BindInputRef, +) -> VortexResult { + let glob_url_parameter = input + .get_parameter(0) + .ok_or_else(|| vortex_err!("Missing file glob parameter"))?; + + // The input to the table function can either be a single glob, or a List of glob patterns. + let glob_strings: Vec = match glob_url_parameter.extract() { + ExtractedValue::Varchar(glob) => { + vec![glob.to_string()] + } + ExtractedValue::List(globs) => globs + .into_iter() + .map(|glob| { + let ExtractedValue::Varchar(string) = glob.extract() else { + vortex_bail!("list element must be Varchar type") + }; + + Ok(string.to_string()) + }) + .try_collect()?, + _ => vortex_bail!("Invalid argument to read_vortex table function"), + }; - // Parse the URL and separate the base URL (keep scheme, host, etc.) from the path. - let glob_url_string = glob_url_parameter.as_string(); - let glob_url_str = glob_url_string.as_str(); - let glob_url = parse_glob_url(glob_url_str)?; + // Parse each glob URL and resolve its filesystem. + let mut glob_urls: Vec = Vec::with_capacity(glob_strings.len()); + for glob_str in &glob_strings { + glob_urls.push(parse_glob_url(glob_str)?); + } + // Cache filesystems by base URL to avoid resolving the same filesystem multiple times. + let mut fs_cache: HashMap = HashMap::new(); + for glob_url in &glob_urls { let mut base_url = glob_url.clone(); base_url.set_path(""); + if !fs_cache.contains_key(&base_url) { + let fs = resolve_filesystem(&base_url, ctx)?; + fs_cache.insert(base_url, fs); + } + } - let fs = resolve_filesystem(&base_url, ctx)?; + RUNTIME.block_on(async { + let mut builder = MultiFileDataSource::new(SESSION.clone()); - RUNTIME.block_on(async { - let builder = MultiFileDataSource::new(SESSION.clone()) - .with_filesystem(fs) - .with_glob(glob_url.path()); - let ds = builder.build().await?; - VortexResult::Ok(Arc::new(ds) as DataSourceRef) - }) - } + for glob_url in &glob_urls { + let mut base_url = glob_url.clone(); + base_url.set_path(""); + let fs = fs_cache + .get(&base_url) + .map(Arc::clone) + .unwrap_or_else(|| unreachable!("fs should be cached for all base URLs")); + builder = builder.with_glob(glob_url.path(), Some(fs)); + } + + builder.build().await + }) } #[cfg(test)] mod tests { - #[allow(clippy::wildcard_imports)] use rstest::rstest; use super::*; @@ -116,7 +180,7 @@ mod tests { #[test] fn test_parse_glob_url_absolute_glob_path() -> VortexResult<()> { - let tmpdir = tempfile::tempdir().unwrap(); + let tmpdir = tempfile::tempdir()?; let glob = format!("{}/*.vortex", tmpdir.path().display()); let url = parse_glob_url(&glob)?; assert_eq!(url.scheme(), "file"); @@ -126,9 +190,11 @@ mod tests { #[test] fn test_parse_glob_url_absolute_existing_path() -> VortexResult<()> { - let tmpfile = tempfile::NamedTempFile::new().unwrap(); - let canonical = std::fs::canonicalize(tmpfile.path()).unwrap(); - let path_str = canonical.to_str().unwrap(); + let tmpfile = tempfile::NamedTempFile::new()?; + let canonical = std::fs::canonicalize(tmpfile.path())?; + let path_str = canonical + .to_str() + .ok_or_else(|| vortex_err!("canonical path is not valid UTF-8"))?; let url = parse_glob_url(path_str)?; assert_eq!(url.scheme(), "file"); assert_eq!(url.path(), path_str); @@ -139,8 +205,13 @@ mod tests { fn test_parse_glob_url_relative_path() -> VortexResult<()> { // Create a tempfile in the current working directory so we can refer to it // by a relative name (just the filename, without any directory component). - let tmpfile = tempfile::NamedTempFile::new_in(".").unwrap(); - let filename = tmpfile.path().file_name().unwrap().to_str().unwrap(); + let tmpfile = tempfile::NamedTempFile::new_in(".")?; + let filename = tmpfile + .path() + .file_name() + .ok_or_else(|| vortex_err!("temp file missing file name"))? + .to_str() + .ok_or_else(|| vortex_err!("temp file name is not valid UTF-8"))?; let url = parse_glob_url(filename)?; assert_eq!(url.scheme(), "file"); @@ -154,8 +225,13 @@ mod tests { fn test_parse_glob_url_relative_glob_path() -> VortexResult<()> { // A relative path with a glob character (e.g. `./data/*.vortex`) must also resolve // correctly. - let tmpdir = tempfile::tempdir_in(".").unwrap(); - let dir_name = tmpdir.path().file_name().unwrap().to_str().unwrap(); + let tmpdir = tempfile::tempdir_in(".")?; + let dir_name = tmpdir + .path() + .file_name() + .ok_or_else(|| vortex_err!("temp dir missing file name"))? + .to_str() + .ok_or_else(|| vortex_err!("temp dir name is not valid UTF-8"))?; let glob = format!("./{dir_name}/*.vortex"); let url = parse_glob_url(&glob)?; assert_eq!(url.scheme(), "file"); @@ -177,8 +253,13 @@ mod tests { fn test_parse_glob_url_parent_relative_path() -> VortexResult<()> { // A path starting with `..` must be resolved to an absolute path without // percent-encoding the `..` component in the resulting URL. - let tmpfile = tempfile::NamedTempFile::new_in("..").unwrap(); - let filename = tmpfile.path().file_name().unwrap().to_str().unwrap(); + let tmpfile = tempfile::NamedTempFile::new_in("..")?; + let filename = tmpfile + .path() + .file_name() + .ok_or_else(|| vortex_err!("temp file missing file name"))? + .to_str() + .ok_or_else(|| vortex_err!("temp file name is not valid UTF-8"))?; let relative = format!("../{filename}"); let url = parse_glob_url(&relative)?; diff --git a/vortex-error/Cargo.toml b/vortex-error/Cargo.toml index 2856a2724ae..fd85dd770dd 100644 --- a/vortex-error/Cargo.toml +++ b/vortex-error/Cargo.toml @@ -29,7 +29,7 @@ tokio = { workspace = true, features = ["rt"], optional = true } [dev-dependencies] serial_test = { version = "3.3.1" } -temp-env = "0.3" +temp-env = { workspace = true } [lints] workspace = true diff --git a/vortex-error/public-api.lock b/vortex-error/public-api.lock index 757f9a986dc..5624f567c51 100644 --- a/vortex-error/public-api.lock +++ b/vortex-error/public-api.lock @@ -54,59 +54,59 @@ pub vortex_error::VortexError::TryFromInt(core::num::error::TryFromIntError, all impl vortex_error::VortexError -pub fn vortex_error::VortexError::with_context>(self, msg: T) -> Self +pub fn vortex_error::VortexError::with_context>(self, T) -> Self impl core::convert::From<&alloc::sync::Arc> for vortex_error::VortexError -pub fn vortex_error::VortexError::from(e: &alloc::sync::Arc) -> Self +pub fn vortex_error::VortexError::from(&alloc::sync::Arc) -> Self impl core::convert::From> for vortex_error::VortexError -pub fn vortex_error::VortexError::from(value: alloc::sync::Arc) -> Self +pub fn vortex_error::VortexError::from(alloc::sync::Arc) -> Self impl core::convert::From for vortex_error::VortexError -pub fn vortex_error::VortexError::from(value: arrow_schema::error::ArrowError) -> Self +pub fn vortex_error::VortexError::from(arrow_schema::error::ArrowError) -> Self impl core::convert::From for vortex_error::VortexError -pub fn vortex_error::VortexError::from(_: core::convert::Infallible) -> Self +pub fn vortex_error::VortexError::from(core::convert::Infallible) -> Self impl core::convert::From for vortex_error::VortexError -pub fn vortex_error::VortexError::from(value: core::num::error::TryFromIntError) -> Self +pub fn vortex_error::VortexError::from(core::num::error::TryFromIntError) -> Self impl core::convert::From for vortex_error::VortexError -pub fn vortex_error::VortexError::from(value: flatbuffers::verifier::InvalidFlatbuffer) -> Self +pub fn vortex_error::VortexError::from(flatbuffers::verifier::InvalidFlatbuffer) -> Self impl core::convert::From for vortex_error::VortexError -pub fn vortex_error::VortexError::from(value: jiff::error::Error) -> Self +pub fn vortex_error::VortexError::from(jiff::error::Error) -> Self impl core::convert::From for vortex_error::VortexError -pub fn vortex_error::VortexError::from(value: object_store::Error) -> Self +pub fn vortex_error::VortexError::from(object_store::Error) -> Self impl core::convert::From for vortex_error::VortexError -pub fn vortex_error::VortexError::from(value: prost::error::DecodeError) -> Self +pub fn vortex_error::VortexError::from(prost::error::DecodeError) -> Self impl core::convert::From for vortex_error::VortexError -pub fn vortex_error::VortexError::from(value: prost::error::EncodeError) -> Self +pub fn vortex_error::VortexError::from(prost::error::EncodeError) -> Self impl core::convert::From for vortex_error::VortexError -pub fn vortex_error::VortexError::from(value: prost::error::UnknownEnumValue) -> Self +pub fn vortex_error::VortexError::from(prost::error::UnknownEnumValue) -> Self impl core::convert::From for vortex_error::VortexError -pub fn vortex_error::VortexError::from(value: std::io::error::Error) -> Self +pub fn vortex_error::VortexError::from(std::io::error::Error) -> Self impl core::convert::From for vortex_error::VortexError -pub fn vortex_error::VortexError::from(value: tokio::runtime::task::error::JoinError) -> Self +pub fn vortex_error::VortexError::from(tokio::runtime::task::error::JoinError) -> Self impl core::error::Error for vortex_error::VortexError @@ -114,11 +114,11 @@ pub fn vortex_error::VortexError::source(&self) -> core::option::Option<&(dyn co impl core::fmt::Debug for vortex_error::VortexError -pub fn vortex_error::VortexError::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_error::VortexError::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_error::VortexError -pub fn vortex_error::VortexError::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_error::VortexError::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_error::ErrString(_) @@ -128,11 +128,11 @@ pub fn vortex_error::ErrString::as_ref(&self) -> &str impl core::fmt::Debug for vortex_error::ErrString -pub fn vortex_error::ErrString::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_error::ErrString::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_error::ErrString -pub fn vortex_error::ErrString::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_error::ErrString::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::ops::deref::Deref for vortex_error::ErrString @@ -142,25 +142,25 @@ pub fn vortex_error::ErrString::deref(&self) -> &Self::Target impl core::convert::From for vortex_error::ErrString where T: core::convert::Into> -pub fn vortex_error::ErrString::from(msg: T) -> Self +pub fn vortex_error::ErrString::from(T) -> Self pub trait vortex_error::VortexExpect pub type vortex_error::VortexExpect::Output -pub fn vortex_error::VortexExpect::vortex_expect(self, msg: &'static str) -> Self::Output +pub fn vortex_error::VortexExpect::vortex_expect(self, &'static str) -> Self::Output impl vortex_error::VortexExpect for core::result::Result where E: core::convert::Into pub type core::result::Result::Output = T -pub fn core::result::Result::vortex_expect(self, msg: &'static str) -> Self::Output +pub fn core::result::Result::vortex_expect(self, &'static str) -> Self::Output impl vortex_error::VortexExpect for core::option::Option pub type core::option::Option::Output = T -pub fn core::option::Option::vortex_expect(self, msg: &'static str) -> Self::Output +pub fn core::option::Option::vortex_expect(self, &'static str) -> Self::Output pub type vortex_error::SharedVortexResult = core::result::Result> diff --git a/vortex-error/src/lib.rs b/vortex-error/src/lib.rs index 156ab574706..e48dafcf6f9 100644 --- a/vortex-error/src/lib.rs +++ b/vortex-error/src/lib.rs @@ -20,6 +20,7 @@ use std::io; use std::num::TryFromIntError; use std::ops::Deref; use std::sync::Arc; +use std::sync::LazyLock; /// A string that can be used as an error message. #[derive(Debug)] @@ -38,7 +39,7 @@ where reason = "intentionally panic in debug mode when VORTEX_PANIC_ON_ERR is set" )] fn from(msg: T) -> Self { - if env::var("VORTEX_PANIC_ON_ERR").as_deref().unwrap_or("") == "1" { + if panic_on_err() { panic!("{}\nBacktrace:\n{}", msg.into(), Backtrace::capture()); } else { Self(msg.into()) @@ -46,6 +47,12 @@ where } } +fn panic_on_err() -> bool { + static PANIC_ON_ERR: LazyLock = + LazyLock::new(|| env::var("VORTEX_PANIC_ON_ERR").is_ok_and(|v| v == "1")); + *PANIC_ON_ERR +} + impl AsRef for ErrString { fn as_ref(&self) -> &str { &self.0 diff --git a/vortex-error/tests/fmt.rs b/vortex-error/tests/fmt.rs index 43a81508333..8aba428a3ff 100644 --- a/vortex-error/tests/fmt.rs +++ b/vortex-error/tests/fmt.rs @@ -2,7 +2,7 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors // This seems to be a bug in the lint - https://github.com/rust-lang/rust-clippy/issues/11024 -#![allow(clippy::tests_outside_test_module)] +#![expect(clippy::tests_outside_test_module)] use serial_test::serial; use vortex_error::VortexError; diff --git a/vortex-ffi/CMakeLists.txt b/vortex-ffi/CMakeLists.txt index 7bac7537211..68107b0d244 100644 --- a/vortex-ffi/CMakeLists.txt +++ b/vortex-ffi/CMakeLists.txt @@ -2,6 +2,8 @@ # SPDX-FileCopyrightText: Copyright the Vortex contributors cmake_minimum_required(VERSION 3.10) +include(FetchContent) + project(VortexFFI VERSION 0.0.1 LANGUAGES C) @@ -10,6 +12,7 @@ set(CMAKE_C_STANDARD 17) set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wall -Werror -Wextra -Wpedantic") option(BUILD_TESTS "Build tests" OFF) +option(BUILD_EXAMPLES "Build examples" OFF) set(SANITIZER "" CACHE STRING "Build with sanitizers") set(TARGET_TRIPLE "" CACHE STRING "Rust target triple for FFI library") @@ -76,10 +79,10 @@ Static library path ${LIBRARY_PATH} Headers path ${LIBRARY_HEADERS}") if (NOT EXISTS "${LIBRARY_PATH_SHARED}") - message(FATAL_ERROR "Shared library not found") + message(FATAL_ERROR "Shared library not found, run `cargo build --release -p vortex-ffi`") endif() if (NOT EXISTS "${LIBRARY_PATH}") - message(FATAL_ERROR "Static library not found") + message(FATAL_ERROR "Static library not found, run `cargo build --release -p vortex-ffi`") endif() add_library(vortex_ffi STATIC IMPORTED) @@ -95,12 +98,24 @@ set_target_properties(vortex_ffi_shared PROPERTIES INTERFACE_LINK_OPTIONS "LINKER:-rpath,${LIBRARY_DIR}" ) +if (BUILD_TESTS OR BUILD_EXAMPLES) + FetchContent_Declare( + Nanoarrow + GIT_REPOSITORY https://github.com/apache/arrow-nanoarrow + GIT_TAG apache-arrow-nanoarrow-0.8.0 + ) + FetchContent_MakeAvailable(Nanoarrow) +endif() + if (BUILD_TESTS) enable_language(CXX) set(CMAKE_CXX_STANDARD 20) set(CMAKE_CXX_STANDARD_REQUIRED ON) set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Werror -Wextra -Wpedantic") - enable_testing() add_subdirectory(test) endif() + +if (BUILD_EXAMPLES) + add_subdirectory(examples) +endif() diff --git a/vortex-ffi/Cargo.toml b/vortex-ffi/Cargo.toml index 27f2c7b6140..83a0737e2bd 100644 --- a/vortex-ffi/Cargo.toml +++ b/vortex-ffi/Cargo.toml @@ -20,6 +20,8 @@ categories = { workspace = true } all-features = true [dependencies] +arrow-array = { workspace = true, features = ["ffi"] } +arrow-schema = { workspace = true } async-fs = { workspace = true } futures = { workspace = true } itertools = { workspace = true } @@ -33,6 +35,7 @@ url = { workspace = true, features = [] } vortex = { workspace = true, features = ["object_store"] } [dev-dependencies] +rand = { workspace = true } tempfile = { workspace = true } vortex-array = { workspace = true, features = ["_test-harness"] } diff --git a/vortex-ffi/README.md b/vortex-ffi/README.md index 35911ccccc2..34df8cc2a1a 100644 --- a/vortex-ffi/README.md +++ b/vortex-ffi/README.md @@ -1,58 +1,50 @@ -# Foreign Function Interface +# Vortex C interface +## Usage from a CMake project -Vortex is a file format that can be used by any execution engine. Nearly every programming language supports -the C ABI (Application Binary Interface), so by providing an FFI interface to work with Vortex objects we can -make it easy to support a variety of languages. - -Check out the [`examples`](./examples/) directory to see an example of how to use the API to build -a real native application. - -## Design - -The FFI is designed to be very simple and follows a very object-oriented approach: - -- **Constructors** are simple C functions that return opaque pointers -- **Methods** are functions that receive an opaque pointer as the first argument, followed by subsequent arguments. - Methods may return a value or void. -- **Destructors** free native resources (allocations, file handles, network sockets) and must be explicitly called by - the foreign language to avoid leaking resources. - -Constructors will generally allocate rust memory, and destructors free that memory. - -## Documentation - -The FFI API is documented in `docs/api/c` with explicit inclusion of types, enums, and functions, etc. Note that an -item cannot be referenced in the documentation if it does not have a documentation comment. +``` +# in vortex folder +cargo build --release -p vortex-ffi -## Updating Headers +# in your CMakeLists.txt +include_directory(vortex/vortex-ffi) +target_link_libraries(my_target, vortex_ffi_shared) +# or target_link_libraries(my_target, vortex_ffi) +``` -To rebuild the header file: +## Running C examples: ```sh -cargo +nightly build -p vortex-ffi +cmake -Bbuild -DBUILD_EXAMPLES=1 +cmake --build build +./build/examples/dtype +./build/examples/scan +./build/examples/scan_to_arrow +./build/examples/write_sample ``` -The header generation uses cbindgen's macro expansion feature which requires nightly. -Stable builds use the checked-in header file at `cinclude/vortex.h`. +## Updating Headers -### Testing C part +If you're developing FFI and want to rebuild `cinclude/vortex.h`, run +`cargo +nightly build -p vortex-ffi`. -Build the test library +## Testing C part + +Build the test library: ```sh -cmake -Bbuild -cmake --build build -j $(nproc) +cmake -Bbuild -DBUILD_TESTS=1 +cmake --build build ``` -Run the tests +Run the tests: ```sh ctest --test-dir build -j $(nproc) ``` -You would need C++ compiler toolchain to run the tests since they use Catch2. +You will need C++ compiler toolchain to run the tests since they use Catch2. -### Testing Rust part with sanitizers +## Testing Rust part with sanitizers AddressSanitizer: @@ -90,20 +82,20 @@ with sanitizers. - `allow-abi-mismatch` is safe because in our dependency graph only crates like `compiler_builtins` unset sanitization, and they do it on purpose. - Make sure to use `cargo test` and not `cargo nextest` as nextest reports less -leaks. + leaks. - If you want stack trace symbolization, install `llvm-symbolizer`. -### Testing Rust and C with sanitizers +## Testing Rust and C with sanitizers 1. Build FFI library with external sanitizer runtime: ```sh RUSTFLAGS="-Zsanitizer=address -Zexternal-clangrt" \ cargo +nightly build -Zbuild-std --target= \ ---no-default-features -p vortex-ffi + --no-default-features -p vortex-ffi ``` -2. Build tests with target triple +2. Build tests with target triple: ```sh cmake -Bbuild -DWITH_ASAN=1 -DTARGET_TRIPLE= @@ -111,6 +103,6 @@ cmake -Bbuild -DWITH_ASAN=1 -DTARGET_TRIPLE= 3. Run the tests (ctest doesn't output failures in detail): -``` +```sh ./build/test/vortex_ffi_test 2>& 1 | rustfilt -i- ``` diff --git a/vortex-ffi/build.rs b/vortex-ffi/build.rs index c9869522788..16e72db792a 100644 --- a/vortex-ffi/build.rs +++ b/vortex-ffi/build.rs @@ -1,39 +1,48 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] -#![allow(clippy::exit)] +#![expect(clippy::unwrap_used)] use std::env; use std::path::PathBuf; use std::process::Command; use std::process::exit; fn main() { + println!("cargo:rustc-check-cfg=cfg(vortex_asan)"); println!("cargo:rerun-if-changed=src"); println!("cargo:rerun-if-changed=cbindgen.toml"); println!("cargo:rerun-if-changed=Cargo.toml"); println!("cargo:rerun-if-changed=build.rs"); - for env in ["MIRI", "MIRIFLAGS", "RUSTFLAGS"] { + for env in ["MIRI", "MIRIFLAGS", "CARGO_ENCODED_RUSTFLAGS"] { println!("cargo:rerun-if-env-changed={env}"); } - if env::var("MIRI").is_ok() || env::var("MIRIFLAGS").is_ok() { - println!("cargo:info=Skipping header generation under miri (cbindgen incompatible)"); + let is_asan = env::var("CARGO_ENCODED_RUSTFLAGS") + .unwrap_or_default() + .contains("address"); + if is_asan { + println!("cargo:info=building with asan"); + println!("cargo:rustc-cfg=vortex_asan"); + println!("cargo:info=Skipping header generation due to sanitizers"); return; } - if env::var("RUSTFLAGS") + if env::var("CARGO_ENCODED_RUSTFLAGS") .unwrap_or_default() .contains("sanitizer") { println!("cargo:info=Skipping header generation due to sanitizers"); + } + + if env::var("MIRI").is_ok() || env::var("MIRIFLAGS").is_ok() { + println!("cargo:info=Skipping header generation under miri (cbindgen incompatible)"); return; } // cbindgen macro expansion is only available on nightly - let is_nightly = Command::new("rustc") - .arg("-V") - .output() + let rustc = Command::new("rustc").arg("-V").output(); + let is_nightly = rustc + .as_ref() .map(|output| String::from_utf8_lossy(&output.stdout).contains("nightly")) .unwrap_or(false); if !is_nightly { diff --git a/vortex-ffi/cbindgen.toml b/vortex-ffi/cbindgen.toml index 12e484f093e..46d80cd72c7 100644 --- a/vortex-ffi/cbindgen.toml +++ b/vortex-ffi/cbindgen.toml @@ -1,6 +1,6 @@ language = "C" braces = "SameLine" -pragma_once = true +pragma_once = false cpp_compat = true usize_is_size_t = true style = "type" @@ -8,10 +8,57 @@ style = "type" header = """ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +#pragma once +#include // // THIS FILE IS AUTO-GENERATED, DO NOT MAKE EDITS DIRECTLY // + +// https://arrow.apache.org/docs/format/CDataInterface.html#structure-definitions +// If you want to use your own Arrow library like nanoarrow, define this macro +// and typedef your types: +// +// #include "nanoarrow/common/inline_types.h" +// #define USE_OWN_ARROW +// typedef struct ArrowSchema FFI_ArrowSchema; +// typedef struct ArrowArrayStream FFI_ArrowArrayStream; +// #include "vortex.h" +// +#ifndef USE_OWN_ARROW +struct ArrowSchema { + const char* format; + const char* name; + const char* metadata; + int64_t flags; + int64_t n_children; + struct ArrowSchema** children; + struct ArrowSchema* dictionary; + void (*release)(struct ArrowSchema*); + void* private_data; +}; +struct ArrowArray { + int64_t length; + int64_t null_count; + int64_t offset; + int64_t n_buffers; + int64_t n_children; + const void** buffers; + struct ArrowArray** children; + struct ArrowArray* dictionary; + void (*release)(struct ArrowArray*); + void* private_data; +}; +struct ArrowArrayStream { + int (*get_schema)(struct ArrowArrayStream*, struct ArrowSchema* out); + int (*get_next)(struct ArrowArrayStream*, struct ArrowArray* out); + const char* (*get_last_error)(struct ArrowArrayStream*); + void (*release)(struct ArrowArrayStream*); + void* private_data; +}; +typedef struct ArrowSchema FFI_ArrowSchema; +typedef struct ArrowArrayStream FFI_ArrowArrayStream; +#endif """ [export.rename] diff --git a/vortex-ffi/cinclude/vortex.h b/vortex-ffi/cinclude/vortex.h index f1f25491b98..0a1fd5cb0b6 100644 --- a/vortex-ffi/cinclude/vortex.h +++ b/vortex-ffi/cinclude/vortex.h @@ -1,11 +1,56 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +#pragma once +#include // // THIS FILE IS AUTO-GENERATED, DO NOT MAKE EDITS DIRECTLY // -#pragma once +// https://arrow.apache.org/docs/format/CDataInterface.html#structure-definitions +// If you want to use your own Arrow library like nanoarrow, define this macro +// and typedef your types: +// +// #include "nanoarrow/common/inline_types.h" +// #define USE_OWN_ARROW +// typedef struct ArrowSchema FFI_ArrowSchema; +// typedef struct ArrowArrayStream FFI_ArrowArrayStream; +// #include "vortex.h" +// +#ifndef USE_OWN_ARROW +struct ArrowSchema { + const char *format; + const char *name; + const char *metadata; + int64_t flags; + int64_t n_children; + struct ArrowSchema **children; + struct ArrowSchema *dictionary; + void (*release)(struct ArrowSchema *); + void *private_data; +}; +struct ArrowArray { + int64_t length; + int64_t null_count; + int64_t offset; + int64_t n_buffers; + int64_t n_children; + const void **buffers; + struct ArrowArray **children; + struct ArrowArray *dictionary; + void (*release)(struct ArrowArray *); + void *private_data; +}; +struct ArrowArrayStream { + int (*get_schema)(struct ArrowArrayStream *, struct ArrowSchema *out); + int (*get_next)(struct ArrowArrayStream *, struct ArrowArray *out); + const char *(*get_last_error)(struct ArrowArrayStream *); + void (*release)(struct ArrowArrayStream *); + void *private_data; +}; +typedef struct ArrowSchema FFI_ArrowSchema; +typedef struct ArrowArrayStream FFI_ArrowArrayStream; +#endif #include #include @@ -134,6 +179,21 @@ typedef enum { VX_VALIDITY_ARRAY = 3, } vx_validity_type; +typedef enum { + /** + * No estimate is available. + */ + VX_ESTIMATE_UNKNOWN = 0, + /** + * The value in vx_estimate.estimate is exact. + */ + VX_ESTIMATE_EXACT = 1, + /** + * The value in vx_estimate.estimate is an upper bound. + */ + VX_ESTIMATE_INEXACT = 2, +} vx_estimate_type; + /** * Equalities, inequalities, and boolean operations over possibly null values. * For most operations, if either side is null, the result is null. @@ -224,6 +284,18 @@ typedef enum { LOG_LEVEL_TRACE = 5, } vx_log_level; +typedef enum { + VX_SELECTION_INCLUDE_ALL = 0, + /** + * Include rows at the indices in vx_scan_selection.idx. + */ + VX_SELECTION_INCLUDE_RANGE = 1, + /** + * Exclude rows at the indices in vx_scan_selection.idx. + */ + VX_SELECTION_EXCLUDE_RANGE = 2, +} vx_scan_selection_include; + /** * Physical type enum, represents the in-memory physical layout but might represent a different logical type. */ @@ -369,6 +441,14 @@ typedef struct vx_array_sink vx_array_sink; */ typedef struct vx_binary vx_binary; +/** + * A reference to one or more (possibly remote) paths. + * Creating vx_data_source opens the first matched path to read the schema. + * All other I/O is deferred until a scan is requested. Multiple scans may + * be requested from a single data source. + */ +typedef struct vx_data_source vx_data_source; + /** * A Vortex data type. * @@ -403,6 +483,28 @@ typedef struct vx_expression vx_expression; */ typedef struct vx_file vx_file; +/** + * A partition is an independent unit of work. Call vx_partition_next repeatedly to + * retrieve arrays, then free the partition with vx_partition_free. + */ +typedef struct vx_partition vx_partition; + +/** + * A typed scalar value. + * + * A `vx_scalar` represents a single value with an associated `DType`. + * Its value is either null or a `ScalarValue`. Null values are allowed only + * when the associated `DType` allows nulls. Non-null values are represented + * by `ScalarValue` and interpreted using the `DType`. + */ +typedef struct vx_scalar vx_scalar; + +/** + * A scan is a single traversal of a data source with projections and + * filters. A scan can be consumed only once. + */ +typedef struct vx_scan vx_scan; + /** * A handle to a Vortex session. */ @@ -435,6 +537,31 @@ typedef struct { const vx_array *array; } vx_validity; +/** + * Options for creating a data source. + */ +typedef struct { + /** + * Required: paths to files, tables, or layout trees. + * May be a glob pattern like "*.vortex". + * If you want to include multiple paths, concat them with a comma: + * "file1.vortex,../file2.vortex". + */ + const char *paths; +} vx_data_source_options; + +/** + * Used for estimating number of partitions in a data source or number of rows + * in a partition. + */ +typedef struct { + vx_estimate_type type; + /** + * Set only when "type" is not VX_ESTIMATE_UNKNOWN. + */ + uint64_t estimate; +} vx_estimate; + /** * Options supplied for opening a file. */ @@ -497,6 +624,61 @@ typedef struct { unsigned long row_offset; } vx_file_scan_options; +/** + * Scan row selection. + * "idx" is copied while calling vx_data_source_scan and can be freed after. + */ +typedef struct { + /** + * Used only when "include" is not VX_SELECTION_INCLUDE_ALL. + * If set, must point to an array of len "idx_len" row_indices. + */ + const uint64_t *idx; + /** + * Used only when "include" is not VX_SELECTION_INCLUDE_ALL + */ + size_t idx_len; + vx_scan_selection_include include; +} vx_scan_selection; + +/** + * Scan options. All fields are optional. To return everything, + * zero-initialize this struct. + */ +typedef struct { + /** + * What columns to return. NULL means all columns. + */ + const vx_expression *projection; + /** + * Predicate expression. NULL means no filter. + */ + const vx_expression *filter; + /** + * Row range [begin, end). Setting row_range_begin and row_range_end to 0 + * means no limit. + */ + uint64_t row_range_begin; + uint64_t row_range_end; + /** + * Row-index filter applied after row_range. + */ + vx_scan_selection selection; + /** + * Maximum number of rows to return. 0 means no limit. + */ + uint64_t limit; + /** + * Upper limit for parallelism. 0 means no limit. + * Scan will return at most "max_threads" partitions. + */ + uint64_t max_threads; + /** + * If true, return in storage order. + */ + bool ordered; +} vx_scan_options; + #ifdef __cplusplus extern "C" { #endif // __cplusplus @@ -711,6 +893,41 @@ size_t vx_binary_len(const vx_binary *ptr); */ const char *vx_binary_ptr(const vx_binary *ptr); +/** + * Clone a borrowed [`vx_data_source`], returning an owned [`vx_data_source`]. + * + * + * Must be released with [`vx_data_source_free`]. + */ +const vx_data_source *vx_data_source_clone(const vx_data_source *ptr); + +/** + * Free an owned [`vx_data_source`] object. + */ +void vx_data_source_free(const vx_data_source *ptr); + +/** + * Create a data source. + * The first matched file is opened eagerly. to read the schema. All other I/O + * is deferred until a scan is requested. The returned pointer is owned by the + * caller and must be freed with vx_data_source_free. + * + * On error, returns NULL and sets "err". + */ +const vx_data_source * +vx_data_source_new(const vx_session *session, const vx_data_source_options *options, vx_error **err); + +/** + * Return the schema of the data source as a non-owned dtype. + * The returned pointer is valid as long as "ds" is alive. Do not free it. + */ +const vx_dtype *vx_data_source_dtype(const vx_data_source *ds); + +/** + * Write data source's row count estimate into "row_count". + */ +void vx_data_source_get_row_count(const vx_data_source *ds, vx_estimate *row_count); + /** * Clone a borrowed [`vx_dtype`], returning an owned [`vx_dtype`]. * @@ -854,6 +1071,13 @@ uint8_t vx_dtype_time_unit(const DType *dtype); */ const vx_string *vx_dtype_time_zone(const DType *dtype); +/** + * Convert a dtype to ArrowSchema. + * You can use the dtype after conversion + * On success, returns 0. On error, sets err and returns 1. + */ +int vx_dtype_to_arrow_schema(const vx_dtype *dtype, FFI_ArrowSchema *schema, vx_error **err); + /** * Free an owned [`vx_error`] object. */ @@ -891,11 +1115,47 @@ void vx_expression_free(vx_expression *ptr); */ vx_expression *vx_expression_root(void); +/** + * Create a literal expression from a scalar. + * + * Literal expressions are useful for constants in expression trees, especially scan + * predicates. For example, a caller can compare a column expression to a scalar + * threshold and pass the resulting predicate to `vx_data_source_scan`. + * + * Example: + * + * vx_error* error = NULL; + * const vx_data_source* data_source = ...; + * + * vx_expression* root = vx_expression_root(); + * vx_expression* age = vx_expression_get_item("age", root); + * + * vx_scalar* threshold_scalar = vx_scalar_new_u8(50, false); + * vx_expression* threshold = vx_expression_literal(threshold_scalar, &error); + * vx_scalar_free(threshold_scalar); + * + * vx_expression* predicate = vx_expression_binary(VX_OPERATOR_GTE, age, threshold); + * vx_scan_options options = {}; + * options.filter = predicate; + * + * vx_scan* scan = vx_data_source_scan(data_source, &options, NULL, &error); + * + * vx_scan_free(scan); + * vx_expression_free(predicate); + * vx_expression_free(threshold); + * vx_expression_free(age); + * vx_expression_free(root); + * + */ +vx_expression *vx_expression_literal(const vx_scalar *scalar, vx_error **err); + /** * Create an expression that selects (includes) specific fields from a child * expression. Child expression must have a DTYPE_STRUCT dtype. Errors in * vx_array_apply if the child expression doesn't have a specified field. * + * Returns a DTYPE_STRUCT array with selected fields. + * * Example: * * vx_expression* root = vx_expression_root(); @@ -965,7 +1225,9 @@ vx_expression *vx_expression_is_null(const vx_expression *child); * Errors in vx_array_apply if the root array doesn't have a specified field. * * Accesses the specified field from the result of the child expression. - * Equivalent to select(&item, 1, child). + * + * Example: if child is Struct { name=u8, age=u16 } and we do + * vx_expression_get_item("name", child), output type will be DTYPE_U8 */ vx_expression *vx_expression_get_item(const char *item, const vx_expression *child); @@ -1034,6 +1296,328 @@ vx_array_iterator *vx_file_scan(const vx_session *session, */ void vx_set_log_level(vx_log_level level); +/** + * Free an owned [`vx_scalar`] object. + */ +void vx_scalar_free(vx_scalar *ptr); + +/** + * Clone a borrowed scalar handle. + * + * The input scalar handle is not consumed. The returned scalar handle must be + * released with vx_scalar_free. Returns NULL when given a NULL scalar handle. + */ +vx_scalar *vx_scalar_clone(const vx_scalar *scalar); + +/** + * Return the data type of a scalar. + * + * The returned data type handle borrows storage from the scalar handle, so its + * lifetime is bound to the scalar handle. It MUST NOT be freed separately. + * Returns NULL when given a NULL scalar handle. + */ +const vx_dtype *vx_scalar_dtype(const vx_scalar *scalar); + +/** + * Return whether the scalar is a typed null value. + * + * Returns false when given a NULL scalar handle. + */ +bool vx_scalar_is_null(const vx_scalar *scalar); + +/** + * Create a boolean scalar. + */ +vx_scalar *vx_scalar_new_bool(bool value, bool is_nullable); + +/** + * Create an unsigned 8-bit integer scalar. + */ +vx_scalar *vx_scalar_new_u8(uint8_t value, bool is_nullable); + +/** + * Create an unsigned 16-bit integer scalar. + */ +vx_scalar *vx_scalar_new_u16(uint16_t value, bool is_nullable); + +/** + * Create an unsigned 32-bit integer scalar. + */ +vx_scalar *vx_scalar_new_u32(uint32_t value, bool is_nullable); + +/** + * Create an unsigned 64-bit integer scalar. + */ +vx_scalar *vx_scalar_new_u64(uint64_t value, bool is_nullable); + +/** + * Create a signed 8-bit integer scalar. + */ +vx_scalar *vx_scalar_new_i8(int8_t value, bool is_nullable); + +/** + * Create a signed 16-bit integer scalar. + */ +vx_scalar *vx_scalar_new_i16(int16_t value, bool is_nullable); + +/** + * Create a signed 32-bit integer scalar. + */ +vx_scalar *vx_scalar_new_i32(int32_t value, bool is_nullable); + +/** + * Create a signed 64-bit integer scalar. + */ +vx_scalar *vx_scalar_new_i64(int64_t value, bool is_nullable); + +/** + * Create a 32-bit floating point scalar. + */ +vx_scalar *vx_scalar_new_f32(float value, bool is_nullable); + +/** + * Create a 64-bit floating point scalar. + */ +vx_scalar *vx_scalar_new_f64(double value, bool is_nullable); + +/** + * Create a 16-bit floating point scalar. + * + * The value is read from raw half-precision bits because C has no portable + * half-precision floating point ABI. + */ +vx_scalar *vx_scalar_new_f16_bits(uint16_t bits, bool is_nullable); + +/** + * Create a UTF-8 scalar. + * + * The byte range is copied into the scalar. A NULL data pointer is allowed only + * for an empty byte range. Invalid UTF-8 returns NULL and writes the error + * output. + */ +vx_scalar *vx_scalar_new_utf8(const char *ptr, size_t len, bool is_nullable, vx_error **err); + +/** + * Create a binary scalar. + * + * The byte range is copied into the scalar. A NULL data pointer is allowed only + * for an empty byte range. Passing a NULL data pointer for a non-empty byte + * range returns NULL and writes the error output. + */ +vx_scalar *vx_scalar_new_binary(const uint8_t *ptr, size_t len, bool is_nullable, vx_error **err); + +/** + * Create a typed null scalar. + * + * The data type handle is borrowed, not consumed. The returned scalar uses a + * nullable copy of that logical type, regardless of the input type's top-level + * nullability. A NULL data type handle returns NULL and writes the error output. + */ +vx_scalar *vx_scalar_new_null(const vx_dtype *dtype, vx_error **err); + +/** + * Create a decimal scalar. + * + * The unscaled value is provided as a signed 8-bit integer. Decimal precision + * and scale define the logical decimal type. Invalid decimal metadata or value + * overflow returns NULL and writes the error output. + */ +vx_scalar * +vx_scalar_new_decimal_i8(int8_t value, uint8_t precision, int8_t scale, bool is_nullable, vx_error **err); + +/** + * Create a decimal scalar. + * + * The unscaled value is provided as a signed 16-bit integer. Decimal precision + * and scale define the logical decimal type. Invalid decimal metadata or value + * overflow returns NULL and writes the error output. + */ +vx_scalar * +vx_scalar_new_decimal_i16(int16_t value, uint8_t precision, int8_t scale, bool is_nullable, vx_error **err); + +/** + * Create a decimal scalar. + * + * The unscaled value is provided as a signed 32-bit integer. Decimal precision + * and scale define the logical decimal type. Invalid decimal metadata or value + * overflow returns NULL and writes the error output. + */ +vx_scalar * +vx_scalar_new_decimal_i32(int32_t value, uint8_t precision, int8_t scale, bool is_nullable, vx_error **err); + +/** + * Create a decimal scalar. + * + * The unscaled value is provided as a signed 64-bit integer. Decimal precision + * and scale define the logical decimal type. Invalid decimal metadata or value + * overflow returns NULL and writes the error output. + */ +vx_scalar * +vx_scalar_new_decimal_i64(int64_t value, uint8_t precision, int8_t scale, bool is_nullable, vx_error **err); + +/** + * Create a decimal scalar. + * + * The unscaled value is read from a 16-byte little-endian signed integer + * buffer. Decimal precision and scale define the logical decimal type. + * Invalid decimal metadata or value overflow returns NULL and writes the error + * output. + */ +vx_scalar *vx_scalar_new_decimal_i128_le(const uint8_t *bytes16, + uint8_t precision, + int8_t scale, + bool is_nullable, + vx_error **err); + +/** + * Create a decimal scalar. + * + * The unscaled value is read from a 32-byte little-endian signed integer + * buffer. Decimal precision and scale define the logical decimal type. + * Invalid decimal metadata or value overflow returns NULL and writes the error + * output. + */ +vx_scalar *vx_scalar_new_decimal_i256_le(const uint8_t *bytes32, + uint8_t precision, + int8_t scale, + bool is_nullable, + vx_error **err); + +/** + * Create a list scalar. + * + * The element data type handle is borrowed, not consumed. Child scalar handles + * are cloned into the list value, so the caller keeps ownership of the handle + * array and each scalar in it. A NULL child handle array is allowed only for an + * empty list. Child values are validated against the element logical type. + */ +vx_scalar *vx_scalar_new_list(const vx_dtype *element_dtype, + const vx_scalar *const *elements, + size_t len, + bool is_nullable, + vx_error **err); + +/** + * Create a fixed-size list scalar. + * + * The element data type handle is borrowed, not consumed. The number of child + * scalars becomes the fixed-size list width and must fit in a 32-bit unsigned + * integer. Child scalar handles are cloned into the list value, so the caller + * keeps ownership of the handle array and each scalar in it. A NULL child + * handle array is allowed only for an empty list. Child values are validated + * against the element logical type. + */ +vx_scalar *vx_scalar_new_fixed_size_list(const vx_dtype *element_dtype, + const vx_scalar *const *elements, + size_t len, + bool is_nullable, + vx_error **err); + +/** + * Create a struct scalar. + * + * The struct data type handle is borrowed, not consumed. Field scalar handles + * are cloned into the struct value, so the caller keeps ownership of the handle + * array and each scalar in it. Field count and field logical types are validated + * against the struct logical type. A NULL field handle array is allowed only for + * an empty struct value. + */ +vx_scalar *vx_scalar_new_struct(const vx_dtype *struct_dtype, + const vx_scalar *const *fields, + size_t len, + vx_error **err); + +/** + * Free an owned [`vx_scan`] object. + */ +void vx_scan_free(vx_scan *ptr); + +/** + * Free an owned [`vx_partition`] object. + */ +void vx_partition_free(vx_partition *ptr); + +/** + * Scan a data source. + * + * Return an owned scan that must be freed with vx_scan_free. A scan may be + * consumed only once. + * + * "options" and "estimate" may be NULL. + * + * If "options" is NULL, all rows and columns are returned. + * If "estimate" is not NULL, the estimated partition count is written to + * *estimate before returning. + * + * Returns NULL and writes an error to "*err" on failure. + */ +vx_scan *vx_data_source_scan(const vx_data_source *data_source, + const vx_scan_options *options, + vx_estimate *estimate, + vx_error **err); + +/** + * Return borrowed vx_scan's dtype. + * This function will fail if called after vx_scan_next_partition. + * Called must not free the returned pointer as its lifetime is bound to the + * lifetime of the scan. + * On error returns NULL and sets "err". + */ +const vx_dtype *vx_scan_dtype(const vx_scan *scan, vx_error **err); + +/** + * Return an owned partition from a scan. + * The returned partition must be freed with vx_partition_free. + * + * On success returns a partition. + * On exhaustion (no more partitions in scan) returns NULL but doesn't set + * "err". + * On error returns NULL and sets "err". + * + * This function is thread-unsafe. Callers running a multi-threaded pipeline + * should synchronise on calls to this function and dispatch each produced + * partition to a dedicated worker thread. + */ +vx_partition *vx_scan_next_partition(vx_scan *scan, vx_error **err); + +/** + * Get partition's estimated row count. + * Must be called before the first call to vx_partition_next. + * + * On success, returns 0. + * On error, return 1 and sets "error". + */ +int vx_partition_row_count(const vx_partition *partition, vx_estimate *count, vx_error **err); + +/** + * Scan partition to ArrowArrayStream. + * Consumes partition fully: subsequent calls to vx_partition_scan_arrow or + * vx_partition_next are undefined behaviour. + * This call blocks current thread until underlying stream is fully consumed. + * + * Caller must not free partition after calling this function. + * + * On success, sets "stream" and returns 0. + * On error, sets "err" and returns 1, freeing the partition. + */ +int vx_partition_scan_arrow(const vx_session *session, + vx_partition *partition, + FFI_ArrowArrayStream *stream, + vx_error **err); + +/** + * Return an owned owned array from a partition. + * The returned array must be freed with vx_array_free. + * + * On success returns an array. + * On exhaustion (no more arrays in partition) returns NULL but doesn't set + * "err". + * On error return NULL and sets "err". + * + * This function is not thread-safe: call from one thread per partition. + */ +const vx_array *vx_partition_next(vx_partition *partition, vx_error **err); + /** * Free an owned [`vx_session`] object. */ @@ -1115,6 +1699,7 @@ void vx_struct_column_builder_free(vx_struct_column_builder *ptr); /** * Create a new column-wise struct array builder with given validity and a * capacity hint. validity can't be NULL. + * Capacity hint is for the number of columns. * If you don't know capacity, pass 0. * if validity is NULL, returns NULL. */ diff --git a/vortex-ffi/examples/CMakeLists.txt b/vortex-ffi/examples/CMakeLists.txt new file mode 100644 index 00000000000..47228d9a903 --- /dev/null +++ b/vortex-ffi/examples/CMakeLists.txt @@ -0,0 +1,18 @@ +# SPDX-License-Identifier: CC-BY-4.0 +# SPDX-FileCopyrightText: Copyright the Vortex contributors + +# allow linking with vortex_ffi_shared although it's not in current folder +cmake_policy(SET CMP0079 NEW) + +add_executable(scan scan.c) +target_link_libraries(scan PRIVATE vortex_ffi_shared) + +add_executable(scan_to_arrow scan_to_arrow.c) +target_link_libraries(scan_to_arrow PRIVATE + nanoarrow_shared vortex_ffi_shared) + +add_executable(dtype dtype.c) +target_link_libraries(dtype PRIVATE vortex_ffi_shared) + +add_executable(write_sample write_sample.c) +target_link_libraries(write_sample PRIVATE vortex_ffi_shared) diff --git a/vortex-ffi/examples/Makefile b/vortex-ffi/examples/Makefile deleted file mode 100644 index c22056524fd..00000000000 --- a/vortex-ffi/examples/Makefile +++ /dev/null @@ -1,75 +0,0 @@ -# SPDX-License-Identifier: CC-BY-4.0 -# SPDX-FileCopyrightText: Copyright the Vortex contributors - -# Directory containing the Rust crate -ROOT_DIR = ../.. -RUST_CRATE_DIR = .. -# Directory where cargo places the built libraries -RUST_TARGET_DIR = $(ROOT_DIR)/target -# The library name (without lib prefix and .so/.dylib suffix) -LIBRARY_NAME = vortex_ffi - -# Determine platform-specific library extension -UNAME_S := $(shell uname -s) -ifeq ($(UNAME_S),Linux) - LIB_EXT = so -else ifeq ($(UNAME_S),Darwin) - LIB_EXT = dylib -else - # Default to .dll for Windows - LIB_EXT = dll -endif - -# Path to the library depending on build mode -DEBUG_LIB = $(RUST_TARGET_DIR)/debug/lib$(LIBRARY_NAME).$(LIB_EXT) -RELEASE_LIB = $(RUST_TARGET_DIR)/release/lib$(LIBRARY_NAME).$(LIB_EXT) - -# Default to debug mode -LIB = $(DEBUG_LIB) - -# C flags -CFLAGS = -Wall -I$(RUST_CRATE_DIR)/cinclude - -# Linking flags -LDFLAGS = -L$(RUST_TARGET_DIR)/debug -l$(LIBRARY_NAME) - -# Final executable name -EXECUTABLE = hello_vortex - -.PHONY: all clean run release debug - -# Default target -all: debug - -# Debug build -debug: LDFLAGS = -L$(RUST_TARGET_DIR)/debug -l$(LIBRARY_NAME) -debug: build - -# Release build -release: LDFLAGS = -L$(RUST_TARGET_DIR)/release -l$(LIBRARY_NAME) -release: LIB = $(RELEASE_LIB) -release: build - -# Build rule -build: $(LIB) $(EXECUTABLE) - -# Build the Rust library -$(DEBUG_LIB): - cd $(RUST_CRATE_DIR) && cargo build - -$(RELEASE_LIB): - cd $(RUST_CRATE_DIR) && cargo build --release - -# Build the C executable -$(EXECUTABLE): hello-vortex.c - $(CC) $(CFLAGS) -o $@ $< $(LDFLAGS) - -# Run the executable with the correct library path -run: build - @echo "Running example..." - LD_LIBRARY_PATH=$(dir $(LIB)) ./$(EXECUTABLE) - -# Clean build artifacts -clean: - rm -f $(EXECUTABLE) - cd $(RUST_CRATE_DIR) && cargo clean diff --git a/vortex-ffi/examples/README.md b/vortex-ffi/examples/README.md deleted file mode 100644 index 5ebc99f9bfe..00000000000 --- a/vortex-ffi/examples/README.md +++ /dev/null @@ -1,30 +0,0 @@ -# C FFI example - -This example shows how to interface with the FFI of this crate using C code. - -Run `make` to build the `hello_vortex` binary. - -The binary expects a single argument, which is the path to a Vortex file. A new streaming -scan will be created that will materialize file splits as Vortex in-memory arrays, and print -some information about them. - -Here's an example from the TPC-H `partsupp` dataset: - -``` -$ make run file:///tmp/partsupp.vortex -Scanning file: file:///tmp/partsupp.vortex -Chunk 0: 65536 -Chunk 1: 65536 -Chunk 2: 65536 -Chunk 3: 65536 -Chunk 4: 65536 -Chunk 5: 65536 -Chunk 6: 65536 -Chunk 7: 65536 -Chunk 8: 65536 -Chunk 9: 65536 -Chunk 10: 65536 -Chunk 11: 65536 -Chunk 12: 13568 -Scanning complete -``` diff --git a/vortex-ffi/examples/dtype.c b/vortex-ffi/examples/dtype.c new file mode 100644 index 00000000000..007c3993e33 --- /dev/null +++ b/vortex-ffi/examples/dtype.c @@ -0,0 +1,156 @@ +// SPDX-License-Identifier: CC-BY-4.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors +#include "vortex.h" +#include + +const char *usage = "Print dtype of files\n" + "Usage: dtype \n"; + +void print_dtype(const vx_dtype *dtype); + +void print_ptype(const vx_dtype *type) { + const char *ptype = NULL; + + switch (vx_dtype_primitive_ptype(type)) { + case PTYPE_U8: + ptype = "uint8_t"; + break; + case PTYPE_U16: + ptype = "uint16_t"; + break; + case PTYPE_U32: + ptype = "uint32_t"; + break; + case PTYPE_U64: + ptype = "uint64_t"; + break; + case PTYPE_I8: + ptype = "int8_t"; + break; + case PTYPE_I16: + ptype = "int16_t"; + break; + case PTYPE_I32: + ptype = "int32_t"; + break; + case PTYPE_I64: + ptype = "int64_t"; + break; + case PTYPE_F16: + ptype = "float16"; + break; + case PTYPE_F32: + ptype = "float"; + break; + case PTYPE_F64: + ptype = "double"; + break; + default: + __builtin_unreachable(); + } + + printf("primitive(%s)", ptype); +} + +void print_struct_dtype(const vx_dtype *dtype) { + const vx_struct_fields *fields = vx_dtype_struct_dtype(dtype); + + printf("struct(\n"); + for (uint64_t i = 0; i < vx_struct_fields_nfields(fields); ++i) { + const vx_dtype *field_dtype = vx_struct_fields_field_dtype(fields, i); + const vx_string *field_name = vx_struct_fields_field_name(fields, i); + printf(" %.*s = ", (int)vx_string_len(field_name), vx_string_ptr(field_name)); + print_dtype(field_dtype); + vx_dtype_free(field_dtype); + } + printf(")"); +} + +void print_list_dtype(const vx_dtype *dtype) { + printf("list("); + print_dtype(vx_dtype_list_element(dtype)); + printf(")"); +} + +void print_fixed_list_dtype(const vx_dtype *dtype) { + printf("fixed list(size=%d, ", vx_dtype_fixed_size_list_size(dtype)); + print_dtype(vx_dtype_fixed_size_list_element(dtype)); + printf(")"); +} + +void print_decimal_dtype(const vx_dtype *dtype) { + const uint8_t precision = vx_dtype_decimal_precision(dtype); + const int8_t scale = vx_dtype_decimal_scale(dtype); + printf("decimal(precision=%u, scale=%d)", precision, scale); +} + +void print_dtype(const vx_dtype *dtype) { + switch (vx_dtype_get_variant(dtype)) { + case DTYPE_NULL: + printf("null"); + break; + case DTYPE_BOOL: + printf("bool"); + break; + case DTYPE_UTF8: + printf("utf8"); + break; + case DTYPE_BINARY: + printf("binary"); + break; + case DTYPE_EXTENSION: + printf("extension"); + break; + case DTYPE_PRIMITIVE: + print_ptype(dtype); + break; + case DTYPE_STRUCT: + print_struct_dtype(dtype); + break; + case DTYPE_LIST: + print_list_dtype(dtype); + break; + case DTYPE_FIXED_SIZE_LIST: + print_fixed_list_dtype(dtype); + break; + case DTYPE_DECIMAL: + print_decimal_dtype(dtype); + break; + } + printf("%c\n", vx_dtype_is_nullable(dtype) ? '?' : ' '); +} + +void print_error(const char *what, const vx_error *error) { + const vx_string *str = vx_error_get_message(error); + fprintf(stderr, "%s: %.*s\n", what, (int)vx_string_len(str), vx_string_ptr(str)); +} + +int main(int argc, char **argv) { + if (argc != 2) { + fprintf(stderr, "%s", usage); + return 1; + } + + vx_error *error = NULL; + vx_session *const session = vx_session_new(); + if (session == NULL) { + fprintf(stderr, "Failed to create Vortex session\n"); + return 1; + } + + vx_data_source_options ds_options = {.paths = argv[1]}; + const vx_data_source *data_source = vx_data_source_new(session, &ds_options, &error); + if (data_source == NULL) { + print_error("Failed to create data source", error); + vx_error_free(error); + vx_session_free(session); + return 1; + } + + printf("dtype: "); + print_dtype(vx_data_source_dtype(data_source)); + + vx_data_source_free(data_source); + vx_session_free(session); + return 0; +} diff --git a/vortex-ffi/examples/hello-vortex.c b/vortex-ffi/examples/hello-vortex.c deleted file mode 100644 index 192e4d73ae3..00000000000 --- a/vortex-ffi/examples/hello-vortex.c +++ /dev/null @@ -1,116 +0,0 @@ -// SPDX-License-Identifier: CC-BY-4.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -#include "vortex.h" -#include -#include -#include - -int main(int argc, char *argv[]) { - if (argc < 2) { - printf("Usage: %s \n", argv[0]); - return 1; - } - - vx_error *error = NULL; - vx_session *session = vx_session_new(); - if (session == NULL) { - fprintf(stderr, "Failed to create Vortex session\n"); - return -1; - } - - // Open the file - char *uri = argv[1]; - printf("Opening file: %s\n", uri); - - vx_file_open_options open_opts = { - .uri = uri, - .property_keys = NULL, - .property_vals = NULL, - .property_len = 0, - }; - - const vx_file *file = vx_file_open_reader(session, &open_opts, &error); - if (error != NULL) { - fprintf(stderr, "Failed to open file: %s\n%s", uri, vx_string_ptr(vx_error_get_message(error))); - vx_error_free(error); - vx_session_free(session); - return -1; - } - - // Print file metadata - this will satisfy the "File contains" check - uint64_t row_count = vx_file_row_count(file); - printf("File contains %llu total rows\n", (unsigned long long)row_count); - - // Get and display file dtype - const vx_dtype *file_dtype = vx_file_dtype(file); - vx_dtype_variant variant = vx_dtype_get_variant(file_dtype); - bool nullable = vx_dtype_is_nullable(file_dtype); - printf("File DType variant: %d, nullable: %s\n", variant, - nullable ? "true" : "false"); - - // Start scanning - printf("\nScanning file...\n"); - vx_array_iterator *scan = vx_file_scan(session, file, NULL, &error); - if (error != NULL) { - fprintf(stderr, "Failed to create file scan iterator\n"); - vx_error_free(error); - vx_file_free(file); - vx_session_free(session); - return -1; - } - - int chunk_count = 0; - const vx_array *batch = vx_array_iterator_next(scan, &error); - - while (batch != NULL && error == NULL) { - size_t batch_len = vx_array_len(batch); - printf("Chunk %d: %zu rows\n", chunk_count, batch_len); - - // For the first chunk, show additional API coverage including struct introspection - if (chunk_count == 0 && batch_len > 0) { - const vx_dtype *dtype = vx_array_dtype(batch); - vx_dtype_variant batch_variant = vx_dtype_get_variant(dtype); - printf(" First chunk DType variant: %d\n", batch_variant); - - // Test null count API - uint32_t null_count = vx_array_null_count(batch, &error); - if (error == NULL) { - printf(" Null count: %u\n", null_count); - } else { - printf(" Null count check failed (expected for some array types)\n"); - vx_error_free(error); - error = NULL; - } - - // Test struct field count if it's a struct - if (batch_variant == DTYPE_STRUCT) { - const vx_struct_fields *fields = vx_dtype_struct_dtype(dtype); - size_t n_fields = vx_struct_fields_nfields(fields); - printf(" Struct with %zu fields\n", n_fields); - } - } - - vx_array_free(batch); - batch = vx_array_iterator_next(scan, &error); - chunk_count++; - } - - printf("Total chunks processed: %d\n", chunk_count); - - // Clean up resources - vx_array_iterator_free(scan); - vx_file_free(file); - - if (error != NULL) { - fprintf(stderr, "Error during scan operation\n"); - vx_error_free(error); - vx_session_free(session); - return -1; - } - - printf("Scanning completed successfully\n"); - vx_session_free(session); - - return 0; -} diff --git a/vortex-ffi/examples/hello_vortex.rs b/vortex-ffi/examples/hello_vortex.rs index a87d0338292..e33c8e1f2bb 100644 --- a/vortex-ffi/examples/hello_vortex.rs +++ b/vortex-ffi/examples/hello_vortex.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: CC-BY-4.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used, clippy::expect_used, clippy::use_debug)] +#![expect(clippy::unwrap_used, clippy::use_debug)] //! This example shows usage of the Vortex C FFI to read a Vortex file written by a Rust client. //! //! You can invoke this example from a checkout by running diff --git a/vortex-ffi/examples/scan.c b/vortex-ffi/examples/scan.c new file mode 100644 index 00000000000..65479df8f40 --- /dev/null +++ b/vortex-ffi/examples/scan.c @@ -0,0 +1,209 @@ +// SPDX-License-Identifier: CC-BY-4.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors +#include "vortex.h" +#include +#include +#include + +#define MAX_THREADS 64 + +const char *usage = "Multi-threaded file scan\n" + "Usage: scan [-j threads] \n"; + +void print_estimate(const char *what, const vx_estimate *estimate) { + switch (estimate->type) { + case VX_ESTIMATE_UNKNOWN: + printf("%s: unknown\n", what); + return; + case VX_ESTIMATE_EXACT: + printf("%s: %lu\n", what, estimate->estimate); + return; + case VX_ESTIMATE_INEXACT: + printf("%s: approximately %lu\n", what, estimate->estimate); + break; + } +} + +void print_error(const char *what, const vx_error *error) { + const vx_string *str = vx_error_get_message(error); + fprintf(stderr, "%s: %.*s\n", what, (int)vx_string_len(str), vx_string_ptr(str)); +} + +struct scan_thread_info { + pthread_t thread_id; + pthread_mutex_t *mutex; + vx_scan *scan; + size_t partitions, arrays, rows; + vx_error *error; +}; + +void *execute_scan_thread(void *arg) { + struct scan_thread_info *info = arg; + while (true) { + // A partition is an independent unit of work a thread can work on. + pthread_mutex_lock(info->mutex); + vx_partition *partition = vx_scan_next_partition(info->scan, &info->error); + pthread_mutex_unlock(info->mutex); + + if (partition == NULL && info->error == NULL) { + break; // partition iterator exhausted + } + if (partition == NULL && info->error != NULL) { + return NULL; // partition was not scanned due to an error + } + ++info->partitions; + + vx_estimate row_count; + if (vx_partition_row_count(partition, &row_count, &info->error)) { + vx_partition_free(partition); + return NULL; + } + + printf("Thread %lu processing partition %lu, ", info->thread_id + 1, info->partitions); + print_estimate("row count", &row_count); + + // An array is a batch of rows from a partition + const vx_array *array = NULL; + while ((array = vx_partition_next(partition, &info->error)) != NULL) { + ++info->arrays; + info->rows += vx_array_len(array); + vx_array_free(array); + } + + vx_partition_free(partition); + + if (info->error != NULL) { + return NULL; + } + } + + printf("Thread %lu finished, processed %lu partitions, %lu arrays, %lu rows\n", + info->thread_id + 1, + info->partitions, + info->arrays, + info->rows); + return NULL; +} + +vx_error *execute_scan(vx_scan *scan, pthread_t num_threads) { + pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER; + pthread_t threads[MAX_THREADS]; + struct scan_thread_info infos[MAX_THREADS] = {0}; + + printf("Starting scan, using %lu threads\n", num_threads); + for (pthread_t id = 0; id < num_threads; ++id) { + struct scan_thread_info *info = &infos[id]; + info->thread_id = id; + info->mutex = &mutex; + info->scan = scan; + pthread_create(&threads[id], NULL, execute_scan_thread, info); + } + + size_t partitions = 0, arrays = 0, rows = 0; + for (pthread_t id = 0; id < num_threads; ++id) { + pthread_join(threads[id], NULL); + struct scan_thread_info *info = &infos[id]; + + if (info->error != NULL) { + // Don't join other threads as program will be terminated early + return info->error; + } + + partitions += info->partitions; + arrays += info->arrays; + rows += info->rows; + } + + printf("Finished scan, processed %lu partitions, %lu arrays, %lu rows\n", partitions, arrays, rows); + return NULL; +} + +int parse_options(int argc, char *argv[], pthread_t *threads, char **paths) { + int opt; + while ((opt = getopt(argc, argv, "j:")) != -1) { + switch (opt) { + case 'j': + *threads = atoi(optarg); + break; + default: + fprintf(stderr, "%s", usage); + return 1; + } + } + + if (*threads != 0 && (*threads < 1 || *threads > MAX_THREADS)) { + fprintf(stderr, "Invalid thread count %lu, expected [1; 64]\n", *threads); + return 1; + } + + if (optind + 1 != argc) { + fprintf(stderr, "%s", usage); + return 1; + } + + *paths = argv[optind]; + return 0; +} + +int main(int argc, char *argv[]) { + pthread_t threads = 0; + char *paths; + if (parse_options(argc, argv, &threads, &paths)) { + return 1; + } + + vx_session *session = vx_session_new(); + if (session == NULL) { + fprintf(stderr, "Failed to create Vortex session\n"); + return 1; + } + + printf("Opening files: %s\n", paths); + + // A datasource is a reference to some files. + // We can request multiple scans from a data source. + vx_data_source_options ds_options = {.paths = paths}; + vx_error *error = NULL; + const vx_data_source *data_source = vx_data_source_new(session, &ds_options, &error); + if (data_source == NULL) { + print_error("Failed to create data source", error); + // Returned errors are owned and need to be freed + vx_error_free(error); + vx_session_free(session); + return 1; + } + + vx_estimate row_count; + vx_data_source_get_row_count(data_source, &row_count); + print_estimate("Data source row count", &row_count); + + // A scan is a single traversal of a data source. + // Here we request a scan without any filters, projections, or limiting. + vx_scan_options scan_options = {.max_threads = threads}; + vx_estimate partition_estimate; + vx_scan *scan = vx_data_source_scan(data_source, &scan_options, &partition_estimate, &error); + if (scan == NULL) { + print_error("Failed to create scan", error); + vx_error_free(error); + vx_data_source_free(data_source); + vx_session_free(session); + return 1; + } + + // Caller can use partition estimates to schedule worker threads. + print_estimate("Partition count", &partition_estimate); + if (threads == 0) { + threads = partition_estimate.type == VX_ESTIMATE_UNKNOWN ? 1 : partition_estimate.estimate; + } + + error = execute_scan(scan, threads); + if (error != NULL) { + print_error("Failed to scan", error); + vx_error_free(error); + } + + vx_scan_free(scan); + vx_data_source_free(data_source); + vx_session_free(session); + return 0; +} diff --git a/vortex-ffi/examples/scan_to_arrow.c b/vortex-ffi/examples/scan_to_arrow.c new file mode 100644 index 00000000000..7085a8f39b8 --- /dev/null +++ b/vortex-ffi/examples/scan_to_arrow.c @@ -0,0 +1,126 @@ +// SPDX-License-Identifier: CC-BY-4.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors +#include "nanoarrow/common/inline_types.h" +#include "nanoarrow/nanoarrow.h" + +#define USE_OWN_ARROW +typedef struct ArrowSchema FFI_ArrowSchema; +typedef struct ArrowArrayStream FFI_ArrowArrayStream; +#include "vortex.h" + +#include +#include + +const char *usage = "Scan vortex files to Arrow\n" + "Usage: scan_to_arrow \n"; + +void print_error(const char *what, const vx_error *error) { + const vx_string *str = vx_error_get_message(error); + fprintf(stderr, "%s: %.*s\n", what, (int)vx_string_len(str), vx_string_ptr(str)); +} + +void execute_scan(vx_session *session, vx_scan *scan) { + vx_error *error = NULL; + + // Returned dtype is owned and mustn't be freed + const vx_dtype *dtype = vx_scan_dtype(scan, &error); + if (dtype == NULL) { + print_error("Failed to get scan dtype", error); + vx_error_free(error); + return; + } + + struct ArrowSchema schema; + if (vx_dtype_to_arrow_schema(dtype, &schema, &error)) { + print_error("Failed to convert dtype to Arrow schema", error); + vx_error_free(error); + return; + } + + char schema_buf[1024 * 10]; + const int schema_len = ArrowSchemaToString(&schema, schema_buf, sizeof schema_buf, 1); + printf("arrow schema: %.*s\n", schema_len, schema_buf); + if (schema.release) { + schema.release(&schema); + } + + struct ArrowError arrow_error; + ArrowErrorInit(&arrow_error); + + vx_partition *partition; + size_t partitions = 0, arrays = 0, rows = 0; + + while ((partition = vx_scan_next_partition(scan, &error)) != NULL) { + struct ArrowArrayStream stream; + // Partition is consumed, we must not free it or use it after + if (vx_partition_scan_arrow(session, partition, &stream, &error)) { + print_error("Failed to scan partition to Arrow", error); + vx_error_free(error); + error = NULL; + break; + } + + struct ArrowArray array = {0}; + while (ArrowArrayStreamGetNext(&stream, &array, &arrow_error) == NANOARROW_OK && + array.release != NULL) { + rows += array.length; + ++arrays; + array.release(&array); + memset(&array, 0, sizeof(array)); + } + + printf("Read Partition %lu to arrow, %lu arrays, %lu rows\n", partitions, arrays, rows); + rows = 0; + arrays = 0; + + stream.release(&stream); + ++partitions; + } + + if (error) { + print_error("Failed scanning partition", error); + vx_error_free(error); + } +} + +int main(int argc, char *argv[]) { + if (argc != 2) { + fprintf(stderr, "%s", usage); + return 1; + } + const char *paths = argv[1]; + + vx_session *session = vx_session_new(); + if (session == NULL) { + fprintf(stderr, "Failed to create Vortex session\n"); + return -1; + } + + vx_data_source_options ds_options = {.paths = paths}; + vx_error *error = NULL; + const vx_data_source *data_source = vx_data_source_new(session, &ds_options, &error); + if (data_source == NULL) { + print_error("Failed to create data source", error); + // Returned errors are owned and need to be freed + vx_error_free(error); + vx_session_free(session); + return 1; + } + + vx_scan *scan = vx_data_source_scan(data_source, NULL, NULL, &error); + if (scan == NULL) { + print_error("Failed to create scan", error); + vx_error_free(error); + vx_data_source_free(data_source); + vx_session_free(session); + return 1; + } + + execute_scan(session, scan); + + vx_scan_free(scan); + vx_data_source_free(data_source); + vx_session_free(session); + + return 0; +} diff --git a/vortex-ffi/examples/write_sample.c b/vortex-ffi/examples/write_sample.c new file mode 100644 index 00000000000..e807b21b5a9 --- /dev/null +++ b/vortex-ffi/examples/write_sample.c @@ -0,0 +1,143 @@ +// SPDX-License-Identifier: CC-BY-4.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors +#include "vortex.h" +#include +#include +#include + +#define SAMPLE_ROWS 200 + +const char *usage = "Write a sample 200 rows .vortex file\n" + "Usage: write_sample \n"; + +// StructArray { age=u8, height=u16? } +const vx_dtype *sample_dtype(void) { + vx_struct_fields_builder *builder = vx_struct_fields_builder_new(); + + const char *age = "age"; + const vx_string *age_name = vx_string_new(age, strlen(age)); + const vx_dtype *age_type = vx_dtype_new_primitive(PTYPE_U8, false); + vx_struct_fields_builder_add_field(builder, age_name, age_type); + + const char *height = "height"; + const vx_string *height_name = vx_string_new(height, strlen(height)); + const vx_dtype *height_type = vx_dtype_new_primitive(PTYPE_U16, true); + vx_struct_fields_builder_add_field(builder, height_name, height_type); + + vx_struct_fields *fields = vx_struct_fields_builder_finalize(builder); + return vx_dtype_new_struct(fields, false); +} + +void print_error(const char *what, const vx_error *error) { + const vx_string *str = vx_error_get_message(error); + fprintf(stderr, "%s: %.*s\n", what, (int)vx_string_len(str), vx_string_ptr(str)); +} + +const vx_array *sample_array(void) { + vx_validity validity = {.type = VX_VALIDITY_NON_NULLABLE}; + vx_struct_column_builder *builder = vx_struct_column_builder_new(&validity, SAMPLE_ROWS); + + uint8_t age_buffer[SAMPLE_ROWS]; + uint16_t height_buffer[SAMPLE_ROWS]; + for (uint8_t i = 0; i < SAMPLE_ROWS; ++i) { + age_buffer[i] = i; + height_buffer[i] = rand() % (i + 1); + } + + vx_error *error = NULL; + const vx_array *age_array = vx_array_new_primitive(PTYPE_U8, age_buffer, SAMPLE_ROWS, &validity, &error); + if (error != NULL) { + print_error("Error creating age array", error); + vx_error_free(error); + vx_struct_column_builder_free(builder); + return NULL; + } + + vx_struct_column_builder_add_field(builder, "age", age_array, &error); + vx_array_free(age_array); + if (error != NULL) { + print_error("Error adding age array field to root array", error); + vx_error_free(error); + vx_struct_column_builder_free(builder); + return NULL; + } + + validity.type = VX_VALIDITY_ALL_VALID; + const vx_array *height_array = + vx_array_new_primitive(PTYPE_U16, height_buffer, SAMPLE_ROWS, &validity, &error); + if (error != NULL) { + print_error("Error adding height array field to root array", error); + vx_error_free(error); + vx_struct_column_builder_free(builder); + return NULL; + } + + vx_struct_column_builder_add_field(builder, "height", height_array, &error); + vx_array_free(height_array); + if (error != NULL) { + print_error("Error adding height array field to root array", error); + vx_error_free(error); + vx_struct_column_builder_free(builder); + return NULL; + } + + const vx_array *array = vx_struct_column_builder_finalize(builder, &error); + if (error != NULL) { + print_error("Error creating struct array", error); + vx_error_free(error); + return NULL; + } + + return array; +} + +int main(int argc, char *argv[]) { + if (argc != 2) { + fprintf(stderr, "%s", usage); + return 1; + } + const char *output = argv[1]; + + vx_session *const session = vx_session_new(); + if (session == NULL) { + fprintf(stderr, "Failed to create Vortex session\n"); + return 1; + } + + const vx_dtype *dtype = sample_dtype(); + + vx_error *error = NULL; + vx_array_sink *sink = vx_array_sink_open_file(session, output, dtype, &error); + + vx_dtype_free(dtype); + if (error != NULL) { + vx_session_free(session); + return 1; + } + + const vx_array *array = sample_array(); + if (array == NULL) { + // We already have an error, so we can ignore a potential error + // from this operation + vx_array_sink_close(sink, &error); + vx_session_free(session); + return 1; + } + + vx_array_sink_push(sink, array, &error); + if (error != NULL) { + vx_array_sink_close(sink, &error); + vx_session_free(session); + return 1; + } + vx_array_free(array); + + vx_array_sink_close(sink, &error); + if (error != NULL) { + print_error("Error closing output sink", error); + vx_error_free(error); + } + + vx_session_free(session); + return 0; +} diff --git a/vortex-ffi/src/array.rs b/vortex-ffi/src/array.rs index 776a91f3216..bb4e464dea4 100644 --- a/vortex-ffi/src/array.rs +++ b/vortex-ffi/src/array.rs @@ -1,6 +1,6 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(non_camel_case_types)] +#![expect(non_camel_case_types)] //! FFI interface for working with Vortex Arrays. use std::ffi::c_void; @@ -10,9 +10,11 @@ use std::sync::Arc; use paste::paste; use vortex::array::ArrayRef; use vortex::array::IntoArray; -use vortex::array::ToCanonical; +use vortex::array::LEGACY_SESSION; +use vortex::array::VortexSessionExecute; use vortex::array::arrays::NullArray; use vortex::array::arrays::PrimitiveArray; +use vortex::array::arrays::StructArray; use vortex::array::arrays::struct_::StructArrayExt; use vortex::array::validity::Validity; use vortex::buffer::Buffer; @@ -193,6 +195,9 @@ pub unsafe extern "C-unwind" fn vx_array_dtype(array: *const vx_array) -> *const vx_dtype::new_ref(vx_array::as_ref(array).dtype()) } +// Return an owned field for array at index. +// Returns NULL and sets error_out if index is out of bounds or array doesn't +// have dtype DTYPE_STRUCT. #[unsafe(no_mangle)] pub unsafe extern "C-unwind" fn vx_array_get_field( array: *const vx_array, @@ -202,8 +207,10 @@ pub unsafe extern "C-unwind" fn vx_array_get_field( try_or_default(error_out, || { let array = vx_array::as_ref(array); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let field_array = array - .to_struct() + .clone() + .execute::(&mut ctx)? .unmasked_fields() .get(index) .ok_or_else(|| vortex_err!("Field index out of bounds"))? @@ -238,7 +245,7 @@ pub unsafe extern "C-unwind" fn vx_array_element_is_invalid( ) -> bool { try_or_default(error, || { vortex_ensure!(!array.is_null()); - vx_array::as_ref(array).is_invalid(index) + vx_array::as_ref(array).is_invalid(index, &mut LEGACY_SESSION.create_execution_ctx()) }) } @@ -251,7 +258,7 @@ pub unsafe extern "C-unwind" fn vx_array_invalid_count( try_or_default(error_out, || { vortex_ensure!(!array.is_null()); let array = vx_array::as_ref(array); - array.invalid_count() + array.invalid_count(&mut LEGACY_SESSION.create_execution_ctx()) }) } @@ -326,7 +333,9 @@ macro_rules! ffiarray_get_ptype { pub unsafe extern "C-unwind" fn [](array: *const vx_array, index: usize) -> $ptype { let array = vx_array::as_ref(array); // TODO(joe): propagate this error up instead of expecting - let value = array.scalar_at(index).vortex_expect("scalar_at failed"); + let value = array + .execute_scalar(index, &mut LEGACY_SESSION.create_execution_ctx()) + .vortex_expect("scalar_at failed"); // TODO(joe): propagate this error up instead of expecting value.as_primitive() .as_::<$ptype>() @@ -337,7 +346,9 @@ macro_rules! ffiarray_get_ptype { pub unsafe extern "C-unwind" fn [](array: *const vx_array, index: usize) -> $ptype { let array = vx_array::as_ref(array); // TODO(joe): propagate this error up instead of expecting - let value = array.scalar_at(index).vortex_expect("scalar_at failed"); + let value = array + .execute_scalar(index, &mut LEGACY_SESSION.create_execution_ctx()) + .vortex_expect("scalar_at failed"); // TODO(joe): propagate this error up instead of expecting value.as_extension() .to_storage_scalar() @@ -371,7 +382,7 @@ pub unsafe extern "C-unwind" fn vx_array_get_utf8( let array = vx_array::as_ref(array); // TODO(joe): propagate this error up instead of expecting let value = array - .scalar_at(index as usize) + .execute_scalar(index as usize, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at failed"); let utf8_scalar = value.as_utf8(); if let Some(buffer) = utf8_scalar.value() { @@ -391,7 +402,7 @@ pub unsafe extern "C-unwind" fn vx_array_get_binary( let array = vx_array::as_ref(array); // TODO(joe): propagate this error up instead of expecting let value = array - .scalar_at(index as usize) + .execute_scalar(index as usize, &mut LEGACY_SESSION.create_execution_ctx()) .vortex_expect("scalar_at failed"); let binary_scalar = value.as_binary(); if let Some(bytes) = binary_scalar.value() { @@ -424,6 +435,8 @@ mod tests { use std::ptr; use vortex::array::IntoArray; + use vortex::array::LEGACY_SESSION; + use vortex::array::VortexSessionExecute; use vortex::array::arrays::BoolArray; use vortex::array::arrays::PrimitiveArray; use vortex::array::arrays::StructArray; @@ -442,20 +455,9 @@ mod tests { use crate::dtype::vx_dtype_get_variant; use crate::dtype::vx_dtype_variant; use crate::error::vx_error_free; - use crate::error::vx_error_get_message; use crate::expression::vx_expression_free; use crate::string::vx_string_free; - - fn assert_no_error(error: *mut vx_error) { - if !error.is_null() { - let message; - unsafe { - message = vx_string::as_str(vx_error_get_message(error)).to_owned(); - vx_error_free(error); - } - panic!("{message}"); - } - } + use crate::tests::assert_no_error; #[test] // TODO(joe): enable once this is fixed https://github.com/Amanieu/parking_lot/issues/477 @@ -764,7 +766,9 @@ mod tests { assert!(!res.is_null()); { let res = vx_array::as_ref(res); - let buffer = res.to_bool().to_bit_buffer(); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let bool_array = res.clone().execute::(&mut ctx).unwrap(); + let buffer = bool_array.to_bit_buffer(); let expected = BoolArray::from_iter(vec![false, false, true, true]); assert_eq!(buffer, expected.to_bit_buffer()); } diff --git a/vortex-ffi/src/data_source.rs b/vortex-ffi/src/data_source.rs new file mode 100644 index 00000000000..33ebe937b3a --- /dev/null +++ b/vortex-ffi/src/data_source.rs @@ -0,0 +1,220 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors +#![allow(non_camel_case_types)] +#![deny(missing_docs)] + +use std::ffi::c_char; +use std::ptr; +use std::sync::Arc; + +use vortex::error::VortexResult; +use vortex::error::vortex_ensure; +use vortex::expr::stats::Precision::Exact; +use vortex::expr::stats::Precision::Inexact; +use vortex::file::multi::MultiFileDataSource; +use vortex::io::runtime::BlockingRuntime; +use vortex::scan::DataSource; +use vortex::scan::DataSourceRef; + +use crate::RUNTIME; +use crate::dtype::vx_dtype; +use crate::error::try_or; +use crate::error::vx_error; +use crate::scan::vx_estimate; +use crate::scan::vx_estimate_type; +use crate::session::vx_session; +use crate::to_string; + +crate::arc_dyn_wrapper!( + /// A reference to one or more (possibly remote) paths. + /// Creating vx_data_source opens the first matched path to read the schema. + /// All other I/O is deferred until a scan is requested. Multiple scans may + /// be requested from a single data source. + dyn DataSource, + vx_data_source); + +/// Options for creating a data source. +#[repr(C)] +#[cfg_attr(test, derive(Default))] +pub struct vx_data_source_options { + /// Required: paths to files, tables, or layout trees. + /// May be a glob pattern like "*.vortex". + /// If you want to include multiple paths, concat them with a comma: + /// "file1.vortex,../file2.vortex". + pub paths: *const c_char, +} + +#[cfg(vortex_asan)] +unsafe extern "C" { + pub fn __lsan_disable(); + pub fn __lsan_enable(); +} + +unsafe fn data_source_new( + session: *const vx_session, + opts: *const vx_data_source_options, +) -> VortexResult<*const vx_data_source> { + vortex_ensure!(!session.is_null()); + vortex_ensure!(!opts.is_null()); + + let session = vx_session::as_ref(session); + + let opts = unsafe { &*opts }; + vortex_ensure!(!opts.paths.is_null()); + + let glob = unsafe { to_string(opts.paths) }; + let mut data_source = MultiFileDataSource::new(session.clone()); + for glob in glob.split(',') { + data_source = data_source.with_glob(glob, None); + } + + let data_source = RUNTIME.block_on(async { + // TODO(myrrc): see https://github.com/vortex-data/vortex/issues/7324 + #[cfg(vortex_asan)] + unsafe { + __lsan_disable(); + } + let data_source = data_source.build().await; + #[cfg(vortex_asan)] + unsafe { + __lsan_enable(); + } + data_source + })?; + Ok(vx_data_source::new(Arc::new(data_source) as DataSourceRef)) +} + +/// Create a data source. +/// The first matched file is opened eagerly. to read the schema. All other I/O +/// is deferred until a scan is requested. The returned pointer is owned by the +/// caller and must be freed with vx_data_source_free. +/// +/// On error, returns NULL and sets "err". +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_data_source_new( + session: *const vx_session, + options: *const vx_data_source_options, + err: *mut *mut vx_error, +) -> *const vx_data_source { + try_or(err, ptr::null(), || unsafe { + data_source_new(session, options) + }) +} + +/// Return the schema of the data source as a non-owned dtype. +/// The returned pointer is valid as long as "ds" is alive. Do not free it. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_data_source_dtype(ds: *const vx_data_source) -> *const vx_dtype { + vx_dtype::new_ref(vx_data_source::as_ref(ds).dtype()) +} + +/// Write data source's row count estimate into "row_count". +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_data_source_get_row_count( + ds: *const vx_data_source, + row_count: *mut vx_estimate, +) { + let rc = unsafe { &mut *row_count }; + match vx_data_source::as_ref(ds).row_count() { + Some(Exact(rows)) => { + rc.r#type = vx_estimate_type::VX_ESTIMATE_EXACT; + rc.estimate = rows; + } + Some(Inexact(rows)) => { + rc.r#type = vx_estimate_type::VX_ESTIMATE_INEXACT; + rc.estimate = rows; + } + None => { + rc.r#type = vx_estimate_type::VX_ESTIMATE_UNKNOWN; + } + } +} + +// Object store error: Generic LocalFileSystem error: Unable to convert +// URL "file:///C:%255CWindows%255CSystemTemp%255C.tmpRXzX38" to filesystem path +// https://github.com/servo/rust-url/issues/1077 +#[cfg(not(windows))] +#[cfg(test)] +mod tests { + use std::ffi::CString; + use std::ptr; + + use crate::data_source::vx_data_source_dtype; + use crate::data_source::vx_data_source_free; + use crate::data_source::vx_data_source_get_row_count; + use crate::data_source::vx_data_source_new; + use crate::data_source::vx_data_source_options; + use crate::dtype::vx_dtype; + use crate::scan::vx_estimate; + use crate::scan::vx_estimate_type; + use crate::session::vx_session_free; + use crate::session::vx_session_new; + use crate::tests::SAMPLE_ROWS; + use crate::tests::assert_error; + use crate::tests::assert_no_error; + use crate::tests::write_sample; + + #[test] + #[cfg_attr(miri, ignore)] + fn test_create_invalid() { + unsafe { + let session = vx_session_new(); + let mut error = ptr::null_mut(); + + let ds = vx_data_source_new(ptr::null_mut(), ptr::null(), &raw mut error); + assert_error(error); + assert!(ds.is_null()); + + let ds = vx_data_source_new(session, ptr::null(), &raw mut error); + assert_error(error); + assert!(ds.is_null()); + + let mut opts = vx_data_source_options::default(); + let ds = vx_data_source_new(session, &raw const opts, &raw mut error); + assert_error(error); + assert!(ds.is_null()); + + opts.paths = c"test.vortex".as_ptr(); + let ds = vx_data_source_new(session, &raw const opts, &raw mut error); + assert_error(error); + assert!(ds.is_null()); + + opts.paths = c"*.vortex".as_ptr(); + let ds = vx_data_source_new(session, &raw const opts, &raw mut error); + assert_error(error); + assert!(ds.is_null()); + + vx_session_free(session); + } + } + + #[test] + #[cfg_attr(miri, ignore)] + fn test_row_count() { + unsafe { + let session = vx_session_new(); + let (sample, struct_array) = write_sample(session); + + let path = CString::new(sample.path().to_str().unwrap()).unwrap(); + let opts = vx_data_source_options { + paths: path.as_ptr(), + }; + + let mut error = ptr::null_mut(); + let ds = vx_data_source_new(session, &raw const opts, &raw mut error); + assert_no_error(error); + assert!(!ds.is_null()); + + let dtype = vx_dtype::as_ref(vx_data_source_dtype(ds)); + assert_eq!(dtype, struct_array.dtype()); + + let mut row_count = vx_estimate::default(); + vx_data_source_get_row_count(ds, &raw mut row_count); + assert_eq!(row_count.r#type, vx_estimate_type::VX_ESTIMATE_EXACT); + assert_eq!(row_count.estimate, SAMPLE_ROWS as u64); + + vx_data_source_free(ds); + vx_session_free(session); + } + } +} diff --git a/vortex-ffi/src/dtype.rs b/vortex-ffi/src/dtype.rs index f404187bb9f..4e720fc6243 100644 --- a/vortex-ffi/src/dtype.rs +++ b/vortex-ffi/src/dtype.rs @@ -1,9 +1,11 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +use std::ffi::c_int; use std::ptr; use std::sync::Arc; +use arrow_array::ffi::FFI_ArrowSchema; use vortex::dtype::DType; use vortex::dtype::DecimalDType; use vortex::error::VortexExpect; @@ -14,6 +16,8 @@ use vortex::extension::datetime::Time; use vortex::extension::datetime::Timestamp; use crate::arc_wrapper; +use crate::error::try_or; +use crate::error::vx_error; use crate::ptype::vx_ptype; use crate::string::vx_string; use crate::struct_fields::vx_struct_fields; @@ -28,7 +32,7 @@ arc_wrapper!( ); /// The variant tag for a Vortex data type. -#[allow(non_camel_case_types)] +#[expect(non_camel_case_types)] #[non_exhaustive] #[repr(C)] #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -323,8 +327,26 @@ pub unsafe extern "C-unwind" fn vx_dtype_time_zone(dtype: *const DType) -> *cons } } +/// Convert a dtype to ArrowSchema. +/// You can use the dtype after conversion +/// On success, returns 0. On error, sets err and returns 1. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_dtype_to_arrow_schema( + dtype: *const vx_dtype, + schema: *mut FFI_ArrowSchema, + err: *mut *mut vx_error, +) -> c_int { + try_or(err, 1, || { + let dtype = vx_dtype::as_ref(dtype); + let arrow_schema = dtype.to_arrow_schema()?; + let arrow_schema = FFI_ArrowSchema::try_from(&arrow_schema)?; + unsafe { ptr::write(schema, arrow_schema) }; + Ok(0) + }) +} + #[cfg(test)] -#[allow(clippy::cast_possible_truncation)] +#[expect(clippy::cast_possible_truncation)] mod tests { use std::slice; @@ -338,6 +360,7 @@ mod tests { use super::*; use crate::array::vx_array; use crate::array::vx_array_dtype; + use crate::array::vx_array_free; use crate::dtype::vx_dtype; use crate::dtype::vx_dtype_free; use crate::dtype::vx_dtype_get_variant; @@ -652,7 +675,7 @@ mod tests { // Cleanup in reverse order - this is the safest order unsafe { - crate::array::vx_array_free(vx_arr); + vx_array_free(vx_arr); } } @@ -677,7 +700,7 @@ mod tests { // Cleanup in careful order unsafe { // Field name is now a borrowed reference - do not free it - crate::array::vx_array_free(vx_arr); + vx_array_free(vx_arr); } } @@ -710,7 +733,7 @@ mod tests { // Cleanup unsafe { - crate::array::vx_array_free(vx_arr); + vx_array_free(vx_arr); } } } diff --git a/vortex-ffi/src/error.rs b/vortex-ffi/src/error.rs index 79115bad43b..4bc5a1ed4be 100644 --- a/vortex-ffi/src/error.rs +++ b/vortex-ffi/src/error.rs @@ -28,6 +28,7 @@ pub(crate) fn write_error(error: *mut *mut vx_error, message: &str) { unsafe { error.write(err) }; } +#[inline] pub fn try_or_default( error_out: *mut *mut vx_error, function: impl FnOnce() -> VortexResult, @@ -38,15 +39,29 @@ pub fn try_or_default( value } Err(err) => { - let err = vx_error::new(VortexError { - message: err.to_string().into(), - }); - unsafe { error_out.write(err) }; + write_error(error_out, &err.to_string()); T::default() } } } +pub fn try_or( + error_out: *mut *mut vx_error, + error_value: T, + function: impl FnOnce() -> VortexResult, +) -> T { + match function() { + Ok(value) => { + unsafe { error_out.write(ptr::null_mut()) }; + value + } + Err(err) => { + write_error(error_out, &err.to_string()); + error_value + } + } +} + /// Returns the error message from the given Vortex error. /// /// The returned pointer is valid as long as the error is valid. diff --git a/vortex-ffi/src/expression.rs b/vortex-ffi/src/expression.rs index 1f7dcf67195..d5632648e82 100644 --- a/vortex-ffi/src/expression.rs +++ b/vortex-ffi/src/expression.rs @@ -1,6 +1,6 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(non_camel_case_types)] +#![expect(non_camel_case_types)] use std::ffi::CStr; use std::ffi::c_char; @@ -10,11 +10,13 @@ use std::sync::Arc; use vortex::dtype::FieldName; use vortex::error::VortexExpect; +use vortex::error::vortex_ensure; use vortex::expr::Expression; use vortex::expr::and_collect; use vortex::expr::get_item; use vortex::expr::is_null; use vortex::expr::list_contains; +use vortex::expr::lit; use vortex::expr::not; use vortex::expr::or_collect; use vortex::expr::root; @@ -23,6 +25,9 @@ use vortex::scalar_fn::ScalarFnVTableExt; use vortex::scalar_fn::fns::binary::Binary; use vortex::scalar_fn::fns::operators::Operator; +use crate::error::try_or; +use crate::error::vx_error; +use crate::scalar::vx_scalar; use crate::to_field_names; // Expressions are Arc'ed inside @@ -62,10 +67,53 @@ pub unsafe extern "C" fn vx_expression_root() -> *mut vx_expression { vx_expression::new(root()) } +/// Create a literal expression from a scalar. +/// +/// Literal expressions are useful for constants in expression trees, especially scan +/// predicates. For example, a caller can compare a column expression to a scalar +/// threshold and pass the resulting predicate to `vx_data_source_scan`. +/// +/// Example: +/// +/// vx_error* error = NULL; +/// const vx_data_source* data_source = ...; +/// +/// vx_expression* root = vx_expression_root(); +/// vx_expression* age = vx_expression_get_item("age", root); +/// +/// vx_scalar* threshold_scalar = vx_scalar_new_u8(50, false); +/// vx_expression* threshold = vx_expression_literal(threshold_scalar, &error); +/// vx_scalar_free(threshold_scalar); +/// +/// vx_expression* predicate = vx_expression_binary(VX_OPERATOR_GTE, age, threshold); +/// vx_scan_options options = {}; +/// options.filter = predicate; +/// +/// vx_scan* scan = vx_data_source_scan(data_source, &options, NULL, &error); +/// +/// vx_scan_free(scan); +/// vx_expression_free(predicate); +/// vx_expression_free(threshold); +/// vx_expression_free(age); +/// vx_expression_free(root); +/// +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_expression_literal( + scalar: *const vx_scalar, + err: *mut *mut vx_error, +) -> *mut vx_expression { + try_or(err, ptr::null_mut(), || { + vortex_ensure!(!scalar.is_null(), "scalar literal is null"); + Ok(vx_expression::new(lit(vx_scalar::as_ref(scalar).clone()))) + }) +} + /// Create an expression that selects (includes) specific fields from a child /// expression. Child expression must have a DTYPE_STRUCT dtype. Errors in /// vx_array_apply if the child expression doesn't have a specified field. /// +/// Returns a DTYPE_STRUCT array with selected fields. +/// /// Example: /// /// vx_expression* root = vx_expression_root(); @@ -242,7 +290,9 @@ pub unsafe extern "C" fn vx_expression_is_null(child: *const vx_expression) -> * /// Errors in vx_array_apply if the root array doesn't have a specified field. /// /// Accesses the specified field from the result of the child expression. -/// Equivalent to select(&item, 1, child). +/// +/// Example: if child is Struct { name=u8, age=u16 } and we do +/// vx_expression_get_item("name", child), output type will be DTYPE_U8 #[unsafe(no_mangle)] pub unsafe extern "C" fn vx_expression_get_item( item: *const c_char, @@ -290,7 +340,8 @@ mod tests { use std::sync::Arc; use vortex::array::IntoArray; - use vortex::array::ToCanonical; + use vortex::array::LEGACY_SESSION; + use vortex::array::VortexSessionExecute; use vortex::array::arrays::BoolArray; use vortex::array::arrays::ListArray; use vortex::array::arrays::PrimitiveArray; @@ -300,7 +351,8 @@ mod tests { use vortex::array::validity::Validity; use vortex::buffer::Buffer; use vortex::buffer::buffer; - use vortex::expr::lit; + use vortex::dtype::Nullability; + use vortex::scalar::Scalar; use crate::array::vx_array; use crate::array::vx_array_apply; @@ -313,9 +365,12 @@ mod tests { use crate::expression::vx_expression_free; use crate::expression::vx_expression_get_item; use crate::expression::vx_expression_list_contains; + use crate::expression::vx_expression_literal; use crate::expression::vx_expression_or; use crate::expression::vx_expression_root; use crate::expression::vx_expression_select; + use crate::scalar::vx_scalar_free; + use crate::scalar::vx_scalar_new_i32; #[test] #[cfg_attr(miri, ignore)] @@ -342,6 +397,7 @@ mod tests { #[test] #[cfg_attr(miri, ignore)] fn test_get_item() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let (array, names_array, ages_array) = struct_array(); unsafe { let root = vx_expression_root(); @@ -357,7 +413,11 @@ mod tests { { let applied_array = vx_array::as_ref(applied_array); let expected: Buffer = ages_array.to_buffer(); - assert_eq!(applied_array.to_primitive().to_buffer(), expected); + let prim = applied_array + .clone() + .execute::(&mut ctx) + .unwrap(); + assert_eq!(prim.to_buffer(), expected); } vx_array_free(applied_array); @@ -385,6 +445,52 @@ mod tests { } } + #[test] + #[cfg_attr(miri, ignore)] + fn test_literal() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let array = + PrimitiveArray::new(buffer![1i32, 2i32, 3i32], Validity::NonNullable).into_array(); + + unsafe { + let array = vx_array::new(Arc::new(array)); + + let value = 2i32; + let scalar = vx_scalar_new_i32(value, false); + assert!(!scalar.is_null()); + + let mut error = ptr::null_mut(); + let expr = vx_expression_literal(scalar, &raw mut error); + assert!(error.is_null()); + assert!(!expr.is_null()); + vx_scalar_free(scalar); + + let applied = vx_array_apply(array, expr, &raw mut error); + assert!(error.is_null()); + assert!(!applied.is_null()); + + { + let applied = vx_array::as_ref(applied); + let expected = Scalar::primitive(value, Nullability::NonNullable); + assert_eq!(applied.len(), 3); + assert_eq!(applied.execute_scalar(0, &mut ctx).unwrap(), expected); + assert_eq!(applied.execute_scalar(2, &mut ctx).unwrap(), expected); + } + + vx_array_free(applied); + vx_expression_free(expr); + + { + let mut error = ptr::null_mut(); + assert!(vx_expression_literal(ptr::null(), &raw mut error).is_null()); + assert!(!error.is_null()); + vx_error_free(error); + } + + vx_array_free(array); + } + } + #[test] #[cfg_attr(miri, ignore)] fn test_select() { @@ -426,6 +532,7 @@ mod tests { #[test] #[cfg_attr(miri, ignore)] fn test_and_or() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let col1 = BoolArray::from_iter([true, false, true, true]); let col2 = BoolArray::from_iter([false, true, true, false]); let col3 = BoolArray::from_iter([false, true, true, true]); @@ -461,7 +568,10 @@ mod tests { assert!(error.is_null()); assert!(!applied_array.is_null()); { - let array = vx_array::as_ref(applied_array).to_bool(); + let array = vx_array::as_ref(applied_array) + .clone() + .execute::(&mut ctx) + .unwrap(); let expected = BoolArray::from_iter([false, false, true, false]); assert_eq!(array.to_bit_buffer(), expected.to_bit_buffer()); } @@ -474,7 +584,10 @@ mod tests { assert!(error.is_null()); assert!(!applied_array.is_null()); { - let array = vx_array::as_ref(applied_array).to_bool(); + let array = vx_array::as_ref(applied_array) + .clone() + .execute::(&mut ctx) + .unwrap(); let expected = BoolArray::from_iter([true, true, true, false]); assert_eq!(array.to_bit_buffer(), expected.to_bit_buffer()); } @@ -511,6 +624,7 @@ mod tests { #[test] #[cfg_attr(miri, ignore)] fn test_list_contains() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let elements = buffer![1i32, 2, 3, 4, 5].into_array(); let offsets = buffer![0u32, 2, 5, 5].into_array(); let array = ListArray::try_new(elements, offsets, Validity::NonNullable).unwrap(); @@ -518,7 +632,11 @@ mod tests { unsafe { let root = vx_expression_root(); let array = vx_array::new(Arc::new(array.into_array())); - let expression_value = vx_expression::new(lit(1)); + let value = vx_scalar_new_i32(1, false); + let mut error = ptr::null_mut(); + let expression_value = vx_expression_literal(value, &raw mut error); + assert!(error.is_null()); + vx_scalar_free(value); let expression = vx_expression_list_contains(root, expression_value); assert!(!expression.is_null()); @@ -528,7 +646,10 @@ mod tests { assert!(error.is_null()); assert!(!applied.is_null()); { - let applied = vx_array::as_ref(applied).to_bool(); + let applied = vx_array::as_ref(applied) + .clone() + .execute::(&mut ctx) + .unwrap(); let expected = BoolArray::from_iter([true, false, false]); assert_eq!(applied.to_bit_buffer(), expected.to_bit_buffer()); } diff --git a/vortex-ffi/src/file.rs b/vortex-ffi/src/file.rs index 61e49b1d624..4ba04eda836 100644 --- a/vortex-ffi/src/file.rs +++ b/vortex-ffi/src/file.rs @@ -59,7 +59,6 @@ arc_wrapper!( /// Options supplied for opening a file. #[repr(C)] -#[allow(non_camel_case_types)] // FIXME(ngates): we cannot have transparent structs in FFI since we cannot break them. pub struct vx_file_open_options { /// URI for opening the file. @@ -76,7 +75,6 @@ pub struct vx_file_open_options { /// Scan options provided by an FFI client calling the `vx_file_scan` function. #[repr(C)] -#[allow(non_camel_case_types)] // FIXME(ngates): we cannot have transparent structs in FFI since we cannot break them. pub struct vx_file_scan_options { /// Column names to project out in the scan. These must be null-terminated C strings. diff --git a/vortex-ffi/src/lib.rs b/vortex-ffi/src/lib.rs index db387172150..4f7da9fcaf7 100644 --- a/vortex-ffi/src/lib.rs +++ b/vortex-ffi/src/lib.rs @@ -1,7 +1,6 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::missing_safety_doc)] #![deny(missing_docs)] //! Native interface to Vortex arrays, types, files and streams. @@ -9,6 +8,7 @@ mod array; mod array_iterator; mod binary; +mod data_source; mod dtype; mod error; mod expression; @@ -16,6 +16,8 @@ mod file; mod log; mod macros; mod ptype; +mod scalar; +mod scan; mod session; mod sink; mod string; @@ -77,3 +79,102 @@ pub(crate) unsafe fn to_field_names( }) .collect() } + +#[cfg(test)] +mod tests { + use std::ffi::CString; + use std::ptr; + use std::sync::Arc; + + use rand::RngExt; + use tempfile::NamedTempFile; + use vortex_array::IntoArray; + use vortex_array::arrays::PrimitiveArray; + use vortex_array::arrays::StructArray; + use vortex_array::arrays::VarBinViewArray; + use vortex_array::validity::Validity; + + use crate::array::vx_array; + use crate::array::vx_array_free; + use crate::dtype::vx_dtype; + use crate::dtype::vx_dtype_free; + use crate::error::vx_error; + use crate::error::vx_error_free; + use crate::error::vx_error_get_message; + use crate::session::vx_session; + use crate::sink::vx_array_sink_close; + use crate::sink::vx_array_sink_open_file; + use crate::sink::vx_array_sink_push; + use crate::string::vx_string; + + /// Panic if error is NULL. Free the error if it's not + pub(crate) fn assert_error(error: *mut vx_error) { + assert!(!error.is_null(), "Expected error"); + unsafe { vx_error_free(error) }; + } + + /// Panic if error is not NULL. + pub(crate) fn assert_no_error(error: *mut vx_error) { + if !error.is_null() { + let message; + unsafe { + message = vx_string::as_str(vx_error_get_message(error)).to_owned(); + vx_error_free(error); + } + panic!("{message}"); + } + } + + fn random_str(length: usize) -> String { + const CHARSET: &[u8] = b"0123456789"; + let mut rng = rand::rng(); + + (0..length) + .map(|_| { + let idx = rng.random_range(0..CHARSET.len()); + CHARSET[idx] as char + }) + .collect() + } + + pub const SAMPLE_ROWS: usize = 200; + + /// Write 200 rows of Struct { age=i32, height=i32, name=String } into a + /// temporary file + pub(crate) unsafe fn write_sample(session: *const vx_session) -> (NamedTempFile, StructArray) { + let age = (0..SAMPLE_ROWS as u64).map(Some); + let age = PrimitiveArray::from_option_iter(age); + + let height = (0..SAMPLE_ROWS as u64).map(|x| Some(200 * x)); + let height = PrimitiveArray::from_option_iter(height); + + let name = (0..SAMPLE_ROWS).map(random_str); + let name = VarBinViewArray::from_iter_str(name); + + let struct_array = StructArray::try_new( + ["age", "height", "name"].into(), + vec![age.into_array(), height.into_array(), name.into_array()], + SAMPLE_ROWS, + Validity::NonNullable, + ) + .unwrap(); + + let file = NamedTempFile::new().unwrap(); + let path = CString::new(file.path().to_str().unwrap()).unwrap(); + let dtype = struct_array.dtype(); + + unsafe { + let vx_dtype_ptr = vx_dtype::new(Arc::new(dtype.clone())); + let mut error = ptr::null_mut(); + let sink = + vx_array_sink_open_file(session, path.as_ptr(), vx_dtype_ptr, &raw mut error); + let array = vx_array::new(Arc::new(struct_array.clone().into_array())); + vx_array_sink_push(sink, array, &raw mut error); + vx_array_sink_close(sink, &raw mut error); + vx_array_free(array); + vx_dtype_free(vx_dtype_ptr); + } + + (file, struct_array) + } +} diff --git a/vortex-ffi/src/log.rs b/vortex-ffi/src/log.rs index cbd7bb5ae7c..6bde4419715 100644 --- a/vortex-ffi/src/log.rs +++ b/vortex-ffi/src/log.rs @@ -23,7 +23,7 @@ fn to_level_filter(level: vx_log_level) -> LevelFilter { /// Log levels for the Vortex library. #[repr(C)] -#[allow(non_camel_case_types)] +#[expect(non_camel_case_types)] pub enum vx_log_level { /// No logging will be performed. LOG_LEVEL_OFF = 0, diff --git a/vortex-ffi/src/macros.rs b/vortex-ffi/src/macros.rs index 9c4e6f2c342..4b191f1c7f6 100644 --- a/vortex-ffi/src/macros.rs +++ b/vortex-ffi/src/macros.rs @@ -176,10 +176,10 @@ macro_rules! box_dyn_wrapper { ($(#[$meta:meta])* $T:ty, $ffi_ident:ident) => { paste::paste! { $(#[$meta])* - #[allow(non_camel_case_types)] + #[expect(non_camel_case_types)] pub struct $ffi_ident(Box<$T>); - #[allow(dead_code)] + #[expect(dead_code)] impl $ffi_ident { /// Wrap an owned object into a raw pointer. pub(crate) fn new(obj: Box<$T>) -> *mut $ffi_ident { @@ -239,10 +239,10 @@ macro_rules! box_wrapper { ($(#[$meta:meta])* $T:ty, $ffi_ident:ident) => { paste::paste! { $(#[$meta])* - #[allow(non_camel_case_types)] + #[expect(non_camel_case_types)] pub struct $ffi_ident($T); - #[allow(dead_code)] + #[expect(dead_code)] impl $ffi_ident { /// Wrap an owned object into a raw pointer. pub(crate) fn new_box(obj: Box<$T>) -> *mut $ffi_ident { diff --git a/vortex-ffi/src/ptype.rs b/vortex-ffi/src/ptype.rs index 47af686582a..513571e737a 100644 --- a/vortex-ffi/src/ptype.rs +++ b/vortex-ffi/src/ptype.rs @@ -6,7 +6,7 @@ use vortex::dtype::PType; /// Variant enum for Vortex primitive types. #[non_exhaustive] #[repr(C)] -#[allow(non_camel_case_types)] +#[expect(non_camel_case_types)] #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] pub enum vx_ptype { /// Unsigned 8-bit integer diff --git a/vortex-ffi/src/scalar.rs b/vortex-ffi/src/scalar.rs new file mode 100644 index 00000000000..2471c2c4c9f --- /dev/null +++ b/vortex-ffi/src/scalar.rs @@ -0,0 +1,845 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! FFI interface for working with Vortex scalar values. + +use std::ffi::c_char; +use std::ptr; +use std::slice; +use std::str; +use std::sync::Arc; + +use vortex::dtype::DType; +use vortex::dtype::DecimalDType; +use vortex::dtype::Nullability; +use vortex::dtype::half::f16; +use vortex::dtype::i256; +use vortex::error::VortexResult; +use vortex::error::vortex_bail; +use vortex::error::vortex_ensure; +use vortex::error::vortex_err; +use vortex::scalar::DecimalValue; +use vortex::scalar::Scalar; +use vortex::scalar::ScalarValue; + +use crate::dtype::vx_dtype; +use crate::error::try_or; +use crate::error::vx_error; + +crate::box_wrapper!( + /// A typed scalar value. + /// + /// A `vx_scalar` represents a single value with an associated `DType`. + /// Its value is either null or a `ScalarValue`. Null values are allowed only + /// when the associated `DType` allows nulls. Non-null values are represented + /// by `ScalarValue` and interpreted using the `DType`. + Scalar, + vx_scalar +); + +/// Clone a borrowed scalar handle. +/// +/// The input scalar handle is not consumed. The returned scalar handle must be +/// released with vx_scalar_free. Returns NULL when given a NULL scalar handle. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_clone(scalar: *const vx_scalar) -> *mut vx_scalar { + if scalar.is_null() { + return ptr::null_mut(); + } + vx_scalar::new(vx_scalar::as_ref(scalar).clone()) +} + +/// Return the data type of a scalar. +/// +/// The returned data type handle borrows storage from the scalar handle, so its +/// lifetime is bound to the scalar handle. It MUST NOT be freed separately. +/// Returns NULL when given a NULL scalar handle. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_dtype(scalar: *const vx_scalar) -> *const vx_dtype { + if scalar.is_null() { + return ptr::null(); + } + vx_dtype::new_ref(vx_scalar::as_ref(scalar).dtype()) +} + +/// Return whether the scalar is a typed null value. +/// +/// Returns false when given a NULL scalar handle. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_is_null(scalar: *const vx_scalar) -> bool { + if scalar.is_null() { + return false; + } + vx_scalar::as_ref(scalar).is_null() +} + +/// Create a boolean scalar. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_new_bool( + value: bool, + is_nullable: bool, +) -> *mut vx_scalar { + vx_scalar::new(Scalar::bool(value, Nullability::from(is_nullable))) +} + +/// Create an unsigned 8-bit integer scalar. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_new_u8(value: u8, is_nullable: bool) -> *mut vx_scalar { + vx_scalar::new(Scalar::primitive(value, Nullability::from(is_nullable))) +} + +/// Create an unsigned 16-bit integer scalar. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_new_u16(value: u16, is_nullable: bool) -> *mut vx_scalar { + vx_scalar::new(Scalar::primitive(value, Nullability::from(is_nullable))) +} + +/// Create an unsigned 32-bit integer scalar. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_new_u32(value: u32, is_nullable: bool) -> *mut vx_scalar { + vx_scalar::new(Scalar::primitive(value, Nullability::from(is_nullable))) +} + +/// Create an unsigned 64-bit integer scalar. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_new_u64(value: u64, is_nullable: bool) -> *mut vx_scalar { + vx_scalar::new(Scalar::primitive(value, Nullability::from(is_nullable))) +} + +/// Create a signed 8-bit integer scalar. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_new_i8(value: i8, is_nullable: bool) -> *mut vx_scalar { + vx_scalar::new(Scalar::primitive(value, Nullability::from(is_nullable))) +} + +/// Create a signed 16-bit integer scalar. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_new_i16(value: i16, is_nullable: bool) -> *mut vx_scalar { + vx_scalar::new(Scalar::primitive(value, Nullability::from(is_nullable))) +} + +/// Create a signed 32-bit integer scalar. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_new_i32(value: i32, is_nullable: bool) -> *mut vx_scalar { + vx_scalar::new(Scalar::primitive(value, Nullability::from(is_nullable))) +} + +/// Create a signed 64-bit integer scalar. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_new_i64(value: i64, is_nullable: bool) -> *mut vx_scalar { + vx_scalar::new(Scalar::primitive(value, Nullability::from(is_nullable))) +} + +/// Create a 32-bit floating point scalar. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_new_f32(value: f32, is_nullable: bool) -> *mut vx_scalar { + vx_scalar::new(Scalar::primitive(value, Nullability::from(is_nullable))) +} + +/// Create a 64-bit floating point scalar. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_new_f64(value: f64, is_nullable: bool) -> *mut vx_scalar { + vx_scalar::new(Scalar::primitive(value, Nullability::from(is_nullable))) +} + +/// Create a 16-bit floating point scalar. +/// +/// The value is read from raw half-precision bits because C has no portable +/// half-precision floating point ABI. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_new_f16_bits( + bits: u16, + is_nullable: bool, +) -> *mut vx_scalar { + vx_scalar::new(Scalar::primitive( + f16::from_bits(bits), + Nullability::from(is_nullable), + )) +} + +/// Create a UTF-8 scalar. +/// +/// The byte range is copied into the scalar. A NULL data pointer is allowed only +/// for an empty byte range. Invalid UTF-8 returns NULL and writes the error +/// output. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_new_utf8( + ptr: *const c_char, + len: usize, + is_nullable: bool, + err: *mut *mut vx_error, +) -> *mut vx_scalar { + try_or(err, ptr::null_mut(), || { + let bytes = bytes_from_raw(ptr.cast(), len, "utf8")?; + let value = str::from_utf8(bytes).map_err(|e| vortex_err!("invalid utf-8: {e}"))?; + Ok(vx_scalar::new(Scalar::utf8( + value.to_owned(), + Nullability::from(is_nullable), + ))) + }) +} + +/// Create a binary scalar. +/// +/// The byte range is copied into the scalar. A NULL data pointer is allowed only +/// for an empty byte range. Passing a NULL data pointer for a non-empty byte +/// range returns NULL and writes the error output. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_new_binary( + ptr: *const u8, + len: usize, + is_nullable: bool, + err: *mut *mut vx_error, +) -> *mut vx_scalar { + try_or(err, ptr::null_mut(), || { + let bytes = bytes_from_raw(ptr, len, "binary")?; + Ok(vx_scalar::new(Scalar::binary( + bytes.to_vec(), + Nullability::from(is_nullable), + ))) + }) +} + +/// Create a typed null scalar. +/// +/// The data type handle is borrowed, not consumed. The returned scalar uses a +/// nullable copy of that logical type, regardless of the input type's top-level +/// nullability. A NULL data type handle returns NULL and writes the error output. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_new_null( + dtype: *const vx_dtype, + err: *mut *mut vx_error, +) -> *mut vx_scalar { + try_or(err, ptr::null_mut(), || { + vortex_ensure!(!dtype.is_null(), "dtype is null"); + Ok(vx_scalar::new(Scalar::null( + vx_dtype::as_ref(dtype).as_nullable(), + ))) + }) +} + +/// Create a decimal scalar. +/// +/// The unscaled value is provided as a signed 8-bit integer. Decimal precision +/// and scale define the logical decimal type. Invalid decimal metadata or value +/// overflow returns NULL and writes the error output. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_new_decimal_i8( + value: i8, + precision: u8, + scale: i8, + is_nullable: bool, + err: *mut *mut vx_error, +) -> *mut vx_scalar { + try_or(err, ptr::null_mut(), || { + decimal_scalar_from_value(DecimalValue::I8(value), precision, scale, is_nullable) + }) +} + +/// Create a decimal scalar. +/// +/// The unscaled value is provided as a signed 16-bit integer. Decimal precision +/// and scale define the logical decimal type. Invalid decimal metadata or value +/// overflow returns NULL and writes the error output. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_new_decimal_i16( + value: i16, + precision: u8, + scale: i8, + is_nullable: bool, + err: *mut *mut vx_error, +) -> *mut vx_scalar { + try_or(err, ptr::null_mut(), || { + decimal_scalar_from_value(DecimalValue::I16(value), precision, scale, is_nullable) + }) +} + +/// Create a decimal scalar. +/// +/// The unscaled value is provided as a signed 32-bit integer. Decimal precision +/// and scale define the logical decimal type. Invalid decimal metadata or value +/// overflow returns NULL and writes the error output. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_new_decimal_i32( + value: i32, + precision: u8, + scale: i8, + is_nullable: bool, + err: *mut *mut vx_error, +) -> *mut vx_scalar { + try_or(err, ptr::null_mut(), || { + decimal_scalar_from_value(DecimalValue::I32(value), precision, scale, is_nullable) + }) +} + +/// Create a decimal scalar. +/// +/// The unscaled value is provided as a signed 64-bit integer. Decimal precision +/// and scale define the logical decimal type. Invalid decimal metadata or value +/// overflow returns NULL and writes the error output. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_new_decimal_i64( + value: i64, + precision: u8, + scale: i8, + is_nullable: bool, + err: *mut *mut vx_error, +) -> *mut vx_scalar { + try_or(err, ptr::null_mut(), || { + decimal_scalar_from_value(DecimalValue::I64(value), precision, scale, is_nullable) + }) +} + +/// Create a decimal scalar. +/// +/// The unscaled value is read from a 16-byte little-endian signed integer +/// buffer. Decimal precision and scale define the logical decimal type. +/// Invalid decimal metadata or value overflow returns NULL and writes the error +/// output. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_new_decimal_i128_le( + bytes16: *const u8, + precision: u8, + scale: i8, + is_nullable: bool, + err: *mut *mut vx_error, +) -> *mut vx_scalar { + try_or(err, ptr::null_mut(), || { + let bytes = fixed_bytes_from_raw::<16>(bytes16, "decimal i128")?; + decimal_scalar_from_value( + DecimalValue::I128(i128::from_le_bytes(bytes)), + precision, + scale, + is_nullable, + ) + }) +} + +/// Create a decimal scalar. +/// +/// The unscaled value is read from a 32-byte little-endian signed integer +/// buffer. Decimal precision and scale define the logical decimal type. +/// Invalid decimal metadata or value overflow returns NULL and writes the error +/// output. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_new_decimal_i256_le( + bytes32: *const u8, + precision: u8, + scale: i8, + is_nullable: bool, + err: *mut *mut vx_error, +) -> *mut vx_scalar { + try_or(err, ptr::null_mut(), || { + let bytes = fixed_bytes_from_raw::<32>(bytes32, "decimal i256")?; + decimal_scalar_from_value( + DecimalValue::I256(i256::from_le_bytes(bytes)), + precision, + scale, + is_nullable, + ) + }) +} + +/// Create a list scalar. +/// +/// The element data type handle is borrowed, not consumed. Child scalar handles +/// are cloned into the list value, so the caller keeps ownership of the handle +/// array and each scalar in it. A NULL child handle array is allowed only for an +/// empty list. Child values are validated against the element logical type. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_new_list( + element_dtype: *const vx_dtype, + elements: *const *const vx_scalar, + len: usize, + is_nullable: bool, + err: *mut *mut vx_error, +) -> *mut vx_scalar { + try_or(err, ptr::null_mut(), || { + vortex_ensure!(!element_dtype.is_null(), "element dtype is null"); + let dtype = DType::List( + Arc::new(vx_dtype::as_ref(element_dtype).clone()), + Nullability::from(is_nullable), + ); + let values = scalar_values_from_raw(elements, len)?; + Ok(vx_scalar::new(Scalar::try_new( + dtype, + Some(ScalarValue::Tuple(values)), + )?)) + }) +} + +/// Create a fixed-size list scalar. +/// +/// The element data type handle is borrowed, not consumed. The number of child +/// scalars becomes the fixed-size list width and must fit in a 32-bit unsigned +/// integer. Child scalar handles are cloned into the list value, so the caller +/// keeps ownership of the handle array and each scalar in it. A NULL child +/// handle array is allowed only for an empty list. Child values are validated +/// against the element logical type. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_new_fixed_size_list( + element_dtype: *const vx_dtype, + elements: *const *const vx_scalar, + len: usize, + is_nullable: bool, + err: *mut *mut vx_error, +) -> *mut vx_scalar { + try_or(err, ptr::null_mut(), || { + vortex_ensure!(!element_dtype.is_null(), "element dtype is null"); + let size = u32::try_from(len) + .map_err(|_| vortex_err!("fixed-size list length {len} exceeds u32::MAX"))?; + let dtype = DType::FixedSizeList( + Arc::new(vx_dtype::as_ref(element_dtype).clone()), + size, + Nullability::from(is_nullable), + ); + let values = scalar_values_from_raw(elements, len)?; + Ok(vx_scalar::new(Scalar::try_new( + dtype, + Some(ScalarValue::Tuple(values)), + )?)) + }) +} + +/// Create a struct scalar. +/// +/// The struct data type handle is borrowed, not consumed. Field scalar handles +/// are cloned into the struct value, so the caller keeps ownership of the handle +/// array and each scalar in it. Field count and field logical types are validated +/// against the struct logical type. A NULL field handle array is allowed only for +/// an empty struct value. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scalar_new_struct( + struct_dtype: *const vx_dtype, + fields: *const *const vx_scalar, + len: usize, + err: *mut *mut vx_error, +) -> *mut vx_scalar { + try_or(err, ptr::null_mut(), || { + vortex_ensure!(!struct_dtype.is_null(), "struct dtype is null"); + let values = scalar_values_from_raw(fields, len)?; + Ok(vx_scalar::new(Scalar::try_new( + vx_dtype::as_ref(struct_dtype).clone(), + Some(ScalarValue::Tuple(values)), + )?)) + }) +} + +fn decimal_scalar_from_value( + value: DecimalValue, + precision: u8, + scale: i8, + is_nullable: bool, +) -> VortexResult<*mut vx_scalar> { + let decimal_dtype = DecimalDType::try_new(precision, scale)?; + Ok(vx_scalar::new(Scalar::try_new( + DType::Decimal(decimal_dtype, Nullability::from(is_nullable)), + Some(ScalarValue::Decimal(value)), + )?)) +} + +fn scalar_values_from_raw( + values: *const *const vx_scalar, + len: usize, +) -> VortexResult>> { + if len == 0 { + return Ok(Vec::new()); + } + vortex_ensure!(!values.is_null(), "scalar pointer array is null"); + + unsafe { slice::from_raw_parts(values, len) } + .iter() + .enumerate() + .map(|(idx, value)| { + if value.is_null() { + vortex_bail!("scalar pointer at index {idx} is null"); + } + Ok(vx_scalar::as_ref(*value).clone().into_value()) + }) + .collect() +} + +fn bytes_from_raw<'a>(ptr: *const u8, len: usize, label: &str) -> VortexResult<&'a [u8]> { + if len == 0 { + return Ok(&[]); + } + vortex_ensure!(!ptr.is_null(), "{label} data pointer is null"); + Ok(unsafe { slice::from_raw_parts(ptr, len) }) +} + +fn fixed_bytes_from_raw(ptr: *const u8, label: &str) -> VortexResult<[u8; N]> { + vortex_ensure!(!ptr.is_null(), "{label} data pointer is null"); + let mut bytes = [0u8; N]; + bytes.copy_from_slice(unsafe { slice::from_raw_parts(ptr, N) }); + Ok(bytes) +} + +#[cfg(test)] +mod tests { + use std::ptr; + use std::sync::Arc; + + use vortex::dtype::DType; + use vortex::dtype::DecimalDType; + use vortex::dtype::Nullability; + use vortex::dtype::PType; + use vortex::dtype::StructFields; + use vortex::dtype::half::f16; + use vortex::scalar::DecimalValue; + use vortex::scalar::Scalar; + + use crate::dtype::vx_dtype; + use crate::dtype::vx_dtype_free; + use crate::dtype::vx_dtype_new_bool; + use crate::dtype::vx_dtype_new_primitive; + use crate::ptype::vx_ptype; + use crate::scalar::vx_scalar; + use crate::scalar::vx_scalar_clone; + use crate::scalar::vx_scalar_dtype; + use crate::scalar::vx_scalar_free; + use crate::scalar::vx_scalar_is_null; + use crate::scalar::vx_scalar_new_binary; + use crate::scalar::vx_scalar_new_bool; + use crate::scalar::vx_scalar_new_decimal_i8; + use crate::scalar::vx_scalar_new_decimal_i16; + use crate::scalar::vx_scalar_new_decimal_i32; + use crate::scalar::vx_scalar_new_decimal_i64; + use crate::scalar::vx_scalar_new_decimal_i128_le; + use crate::scalar::vx_scalar_new_decimal_i256_le; + use crate::scalar::vx_scalar_new_f16_bits; + use crate::scalar::vx_scalar_new_f32; + use crate::scalar::vx_scalar_new_f64; + use crate::scalar::vx_scalar_new_fixed_size_list; + use crate::scalar::vx_scalar_new_i8; + use crate::scalar::vx_scalar_new_i16; + use crate::scalar::vx_scalar_new_i32; + use crate::scalar::vx_scalar_new_i64; + use crate::scalar::vx_scalar_new_list; + use crate::scalar::vx_scalar_new_null; + use crate::scalar::vx_scalar_new_struct; + use crate::scalar::vx_scalar_new_u8; + use crate::scalar::vx_scalar_new_u16; + use crate::scalar::vx_scalar_new_u32; + use crate::scalar::vx_scalar_new_u64; + use crate::scalar::vx_scalar_new_utf8; + use crate::tests::assert_error; + use crate::tests::assert_no_error; + + fn assert_scalar(ptr: *mut vx_scalar, expected: Scalar) { + assert!(!ptr.is_null()); + assert_eq!(vx_scalar::as_ref(ptr), &expected); + unsafe { vx_scalar_free(ptr) }; + } + + #[test] + fn test_primitive_scalar_constructors() { + unsafe { + assert_scalar( + vx_scalar_new_bool(true, true), + Scalar::bool(true, Nullability::Nullable), + ); + assert_scalar( + vx_scalar_new_u8(42, false), + Scalar::primitive(42u8, Nullability::NonNullable), + ); + assert_scalar( + vx_scalar_new_u16(42, true), + Scalar::primitive(42u16, Nullability::Nullable), + ); + assert_scalar( + vx_scalar_new_u32(42, false), + Scalar::primitive(42u32, Nullability::NonNullable), + ); + assert_scalar( + vx_scalar_new_u64(42, true), + Scalar::primitive(42u64, Nullability::Nullable), + ); + assert_scalar( + vx_scalar_new_i8(-42, false), + Scalar::primitive(-42i8, Nullability::NonNullable), + ); + assert_scalar( + vx_scalar_new_i16(-42, true), + Scalar::primitive(-42i16, Nullability::Nullable), + ); + assert_scalar( + vx_scalar_new_i32(-42, true), + Scalar::primitive(-42i32, Nullability::Nullable), + ); + assert_scalar( + vx_scalar_new_i64(-42, false), + Scalar::primitive(-42i64, Nullability::NonNullable), + ); + + let f16_value = f16::from_f32(1.5); + assert_scalar( + vx_scalar_new_f16_bits(f16_value.to_bits(), false), + Scalar::primitive(f16_value, Nullability::NonNullable), + ); + assert_scalar( + vx_scalar_new_f32(1.5, true), + Scalar::primitive(1.5f32, Nullability::Nullable), + ); + assert_scalar( + vx_scalar_new_f64(1.5, false), + Scalar::primitive(1.5f64, Nullability::NonNullable), + ); + } + } + + #[test] + fn test_utf8_binary_and_null_scalar_constructors() { + unsafe { + let mut error = ptr::null_mut(); + let value = "literal"; + assert_scalar( + vx_scalar_new_utf8(value.as_ptr().cast(), value.len(), false, &raw mut error), + Scalar::utf8(value, Nullability::NonNullable), + ); + assert_no_error(error); + + let invalid_utf8 = [0xffu8]; + let scalar = vx_scalar_new_utf8( + invalid_utf8.as_ptr().cast(), + invalid_utf8.len(), + false, + &raw mut error, + ); + assert!(scalar.is_null()); + assert_error(error); + + let bytes = b"\xde\xad\xbe\xef"; + assert_scalar( + vx_scalar_new_binary(bytes.as_ptr(), bytes.len(), true, &raw mut error), + Scalar::binary(bytes.to_vec(), Nullability::Nullable), + ); + assert_no_error(error); + + let dtype = vx_dtype_new_primitive(vx_ptype::PTYPE_I32, false); + let null_scalar = vx_scalar_new_null(dtype, &raw mut error); + vx_dtype_free(dtype); + assert_no_error(error); + assert!(vx_scalar_is_null(null_scalar)); + assert_eq!( + vx_dtype::as_ref(vx_scalar_dtype(null_scalar)), + &DType::Primitive(PType::I32, Nullability::Nullable) + ); + vx_scalar_free(null_scalar); + } + } + + #[test] + fn test_scalar_clone() { + unsafe { + let scalar = vx_scalar_new_u8(7, false); + let cloned = vx_scalar_clone(scalar); + assert_eq!(vx_scalar::as_ref(cloned), vx_scalar::as_ref(scalar)); + vx_scalar_free(cloned); + vx_scalar_free(scalar); + assert!(vx_scalar_clone(ptr::null()).is_null()); + } + } + + #[test] + fn test_decimal_scalar_constructors() { + unsafe { + let mut error = ptr::null_mut(); + assert_scalar( + vx_scalar_new_decimal_i16(999, 3, 0, false, &raw mut error), + Scalar::decimal( + DecimalValue::I16(999), + DecimalDType::new(3, 0), + Nullability::NonNullable, + ), + ); + assert_no_error(error); + + assert_scalar( + vx_scalar_new_decimal_i32(999, 3, 0, true, &raw mut error), + Scalar::decimal( + DecimalValue::I32(999), + DecimalDType::new(3, 0), + Nullability::Nullable, + ), + ); + assert_no_error(error); + + assert_scalar( + vx_scalar_new_decimal_i64(999, 3, 0, false, &raw mut error), + Scalar::decimal( + DecimalValue::I64(999), + DecimalDType::new(3, 0), + Nullability::NonNullable, + ), + ); + assert_no_error(error); + + let scalar = vx_scalar_new_decimal_i8(100, 2, 0, false, &raw mut error); + assert!(scalar.is_null()); + assert_error(error); + + let i128_value = 12345i128; + assert_scalar( + vx_scalar_new_decimal_i128_le( + i128_value.to_le_bytes().as_ptr(), + 10, + 2, + true, + &raw mut error, + ), + Scalar::decimal( + DecimalValue::I128(i128_value), + DecimalDType::new(10, 2), + Nullability::Nullable, + ), + ); + assert_no_error(error); + + let i256_value = vortex::dtype::i256::from_i128(12345); + assert_scalar( + vx_scalar_new_decimal_i256_le( + i256_value.to_le_bytes().as_ptr(), + 10, + 2, + false, + &raw mut error, + ), + Scalar::decimal( + DecimalValue::I256(i256_value), + DecimalDType::new(10, 2), + Nullability::NonNullable, + ), + ); + assert_no_error(error); + } + } + + #[test] + fn test_nested_scalar_constructors() { + unsafe { + let mut error = ptr::null_mut(); + + let element_dtype = vx_dtype_new_primitive(vx_ptype::PTYPE_I32, false); + let child0 = vx_scalar_new_i32(1, false); + let child1 = vx_scalar_new_i32(2, false); + let children = [child0.cast_const(), child1.cast_const()]; + + assert_scalar( + vx_scalar_new_list( + element_dtype, + children.as_ptr(), + children.len(), + true, + &raw mut error, + ), + Scalar::list( + Arc::new(DType::Primitive(PType::I32, Nullability::NonNullable)), + vec![ + Scalar::primitive(1i32, Nullability::NonNullable), + Scalar::primitive(2i32, Nullability::NonNullable), + ], + Nullability::Nullable, + ), + ); + assert_no_error(error); + + assert_scalar( + vx_scalar_new_fixed_size_list( + element_dtype, + children.as_ptr(), + children.len(), + false, + &raw mut error, + ), + Scalar::fixed_size_list( + Arc::new(DType::Primitive(PType::I32, Nullability::NonNullable)), + vec![ + Scalar::primitive(1i32, Nullability::NonNullable), + Scalar::primitive(2i32, Nullability::NonNullable), + ], + Nullability::NonNullable, + ), + ); + assert_no_error(error); + + let wrong_child = vx_scalar_new_bool(true, false); + let wrong_children = [wrong_child.cast_const()]; + let wrong = vx_scalar_new_list( + element_dtype, + wrong_children.as_ptr(), + wrong_children.len(), + false, + &raw mut error, + ); + assert!(wrong.is_null()); + assert_error(error); + + let struct_dtype = vx_dtype::new(Arc::new(DType::Struct( + StructFields::new( + ["flag", "value"].into(), + vec![ + DType::Bool(Nullability::NonNullable), + DType::Primitive(PType::I32, Nullability::NonNullable), + ], + ), + Nullability::NonNullable, + ))); + let flag = vx_scalar_new_bool(true, false); + let value = vx_scalar_new_i32(10, false); + let fields = [flag.cast_const(), value.cast_const()]; + assert_scalar( + vx_scalar_new_struct(struct_dtype, fields.as_ptr(), fields.len(), &raw mut error), + Scalar::struct_( + DType::Struct( + StructFields::new( + ["flag", "value"].into(), + vec![ + DType::Bool(Nullability::NonNullable), + DType::Primitive(PType::I32, Nullability::NonNullable), + ], + ), + Nullability::NonNullable, + ), + vec![ + Scalar::bool(true, Nullability::NonNullable), + Scalar::primitive(10i32, Nullability::NonNullable), + ], + ), + ); + assert_no_error(error); + + let missing_field = vx_scalar_new_struct( + struct_dtype, + fields.as_ptr(), + fields.len() - 1, + &raw mut error, + ); + assert!(missing_field.is_null()); + assert_error(error); + + vx_dtype_free(element_dtype); + vx_dtype_free(struct_dtype); + vx_scalar_free(child0); + vx_scalar_free(child1); + vx_scalar_free(wrong_child); + vx_scalar_free(flag); + vx_scalar_free(value); + } + } + + #[test] + fn test_nested_null_inputs() { + unsafe { + let mut error = ptr::null_mut(); + let dtype = vx_dtype_new_bool(false); + assert!(vx_scalar_new_list(dtype, ptr::null(), 1, false, &raw mut error).is_null()); + assert_error(error); + + let empty = vx_scalar_new_list(dtype, ptr::null(), 0, false, &raw mut error); + assert_no_error(error); + assert!(!empty.is_null()); + vx_scalar_free(empty); + vx_dtype_free(dtype); + } + } +} diff --git a/vortex-ffi/src/scan.rs b/vortex-ffi/src/scan.rs new file mode 100644 index 00000000000..56182b3c881 --- /dev/null +++ b/vortex-ffi/src/scan.rs @@ -0,0 +1,746 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors +#![allow(non_camel_case_types)] +#![deny(missing_docs)] + +use core::slice; +use std::ffi::c_int; +use std::ops::Range; +use std::ptr; +use std::sync::Arc; + +use arrow_array::RecordBatch; +use arrow_array::cast::AsArray; +use arrow_array::ffi_stream::FFI_ArrowArrayStream; +use arrow_schema::ArrowError; +use arrow_schema::DataType; +use futures::StreamExt; +use vortex::array::ArrayRef; +use vortex::array::ExecutionCtx; +use vortex::array::VortexSessionExecute; +use vortex::array::arrow::ArrowArrayExecutor; +use vortex::array::expr::stats::Precision; +use vortex::array::stream::SendableArrayStream; +use vortex::buffer::Buffer; +use vortex::error::VortexResult; +use vortex::error::vortex_bail; +use vortex::error::vortex_ensure; +use vortex::expr::root; +use vortex::io::runtime::BlockingRuntime; +use vortex::layout::scan::arrow::RecordBatchIteratorAdapter; +use vortex::scan::DataSourceScan; +use vortex::scan::Partition; +use vortex::scan::PartitionStream; +use vortex::scan::ScanRequest; +use vortex::scan::selection::Selection; + +use crate::RUNTIME; +use crate::array::vx_array; +use crate::data_source::vx_data_source; +use crate::dtype::vx_dtype; +use crate::error::try_or; +use crate::error::try_or_default; +use crate::error::vx_error; +use crate::expression::vx_expression; +use crate::session::vx_session; + +pub enum VxScan { + Pending(Box), + Started(PartitionStream), + Finished, +} +crate::box_wrapper!( + /// A scan is a single traversal of a data source with projections and + /// filters. A scan can be consumed only once. + VxScan, + vx_scan); + +pub enum VxPartitionScan { + Pending(Box), + Started(SendableArrayStream), + Finished, +} +crate::box_wrapper!( + /// A partition is an independent unit of work. Call vx_partition_next repeatedly to + /// retrieve arrays, then free the partition with vx_partition_free. + VxPartitionScan, + vx_partition); + +// We parse Selection from vx_scan_selection[_include], so we don't need +// to instantiate VX_SELECTION_* items directly. +#[repr(C)] +#[allow(dead_code)] +#[cfg_attr(test, derive(Default))] +pub enum vx_scan_selection_include { + #[cfg_attr(test, default)] + VX_SELECTION_INCLUDE_ALL = 0, + /// Include rows at the indices in vx_scan_selection.idx. + VX_SELECTION_INCLUDE_RANGE = 1, + /// Exclude rows at the indices in vx_scan_selection.idx. + VX_SELECTION_EXCLUDE_RANGE = 2, +} + +/// Scan row selection. +/// "idx" is copied while calling vx_data_source_scan and can be freed after. +#[repr(C)] +#[cfg_attr(test, derive(Default))] +pub struct vx_scan_selection { + /// Used only when "include" is not VX_SELECTION_INCLUDE_ALL. + /// If set, must point to an array of len "idx_len" row_indices. + pub idx: *const u64, + /// Used only when "include" is not VX_SELECTION_INCLUDE_ALL + pub idx_len: usize, + pub include: vx_scan_selection_include, +} + +/// Scan options. All fields are optional. To return everything, +/// zero-initialize this struct. +#[repr(C)] +#[cfg_attr(test, derive(Default))] +pub struct vx_scan_options { + /// What columns to return. NULL means all columns. + pub projection: *const vx_expression, + /// Predicate expression. NULL means no filter. + pub filter: *const vx_expression, + /// Row range [begin, end). Setting row_range_begin and row_range_end to 0 + /// means no limit. + pub row_range_begin: u64, + pub row_range_end: u64, + /// Row-index filter applied after row_range. + pub selection: vx_scan_selection, + /// Maximum number of rows to return. 0 means no limit. + pub limit: u64, + /// Upper limit for parallelism. 0 means no limit. + /// Scan will return at most "max_threads" partitions. + pub max_threads: u64, + /// If true, return in storage order. + pub ordered: bool, +} + +#[repr(C)] +#[cfg_attr(test, derive(Debug, PartialEq, Eq, Default))] +pub enum vx_estimate_type { + /// No estimate is available. + #[cfg_attr(test, default)] + VX_ESTIMATE_UNKNOWN = 0, + /// The value in vx_estimate.estimate is exact. + VX_ESTIMATE_EXACT = 1, + /// The value in vx_estimate.estimate is an upper bound. + VX_ESTIMATE_INEXACT = 2, +} + +/// Used for estimating number of partitions in a data source or number of rows +/// in a partition. +#[repr(C)] +#[cfg_attr(test, derive(Default))] +pub struct vx_estimate { + pub r#type: vx_estimate_type, + /// Set only when "type" is not VX_ESTIMATE_UNKNOWN. + pub estimate: u64, +} + +fn scan_request(opts: *const vx_scan_options) -> VortexResult { + if opts.is_null() { + return Ok(ScanRequest::default()); + } + let opts = unsafe { &*opts }; + + let projection = if opts.projection.is_null() { + root() + } else { + vx_expression::as_ref(opts.projection).clone() + }; + + let filter = if opts.filter.is_null() { + None + } else { + Some(vx_expression::as_ref(opts.filter).clone()) + }; + + let selection = &opts.selection; + let selection = match selection.include { + vx_scan_selection_include::VX_SELECTION_INCLUDE_ALL => Selection::All, + vx_scan_selection_include::VX_SELECTION_INCLUDE_RANGE => { + vortex_ensure!(!selection.idx.is_null()); + let buf = unsafe { slice::from_raw_parts(selection.idx, selection.idx_len) }; + let buf = Buffer::copy_from(buf); + Selection::IncludeByIndex(buf) + } + vx_scan_selection_include::VX_SELECTION_EXCLUDE_RANGE => { + vortex_ensure!(!selection.idx.is_null()); + let buf = unsafe { slice::from_raw_parts(selection.idx, selection.idx_len) }; + let buf = Buffer::copy_from(buf); + Selection::ExcludeByIndex(buf) + } + }; + + let ordered = opts.ordered; + + let start = opts.row_range_begin; + let end = opts.row_range_end; + let row_range = (start > 0 || end > 0).then_some(Range { start, end }); + + let limit = (opts.limit != 0).then_some(opts.limit); + + Ok(ScanRequest { + projection, + filter, + row_range, + selection, + ordered, + limit, + partition_selection: Selection::All, + partition_range: None, + }) +} + +fn write_estimate>(estimate: Option>, out: &mut vx_estimate) { + match estimate { + Some(Precision::Exact(value)) => { + out.r#type = vx_estimate_type::VX_ESTIMATE_EXACT; + out.estimate = value.into(); + } + Some(Precision::Inexact(value)) => { + out.r#type = vx_estimate_type::VX_ESTIMATE_INEXACT; + out.estimate = value.into(); + } + None => { + out.r#type = vx_estimate_type::VX_ESTIMATE_UNKNOWN; + } + } +} + +/// Scan a data source. +/// +/// Return an owned scan that must be freed with vx_scan_free. A scan may be +/// consumed only once. +/// +/// "options" and "estimate" may be NULL. +/// +/// If "options" is NULL, all rows and columns are returned. +/// If "estimate" is not NULL, the estimated partition count is written to +/// *estimate before returning. +/// +/// Returns NULL and writes an error to "*err" on failure. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_data_source_scan( + data_source: *const vx_data_source, + options: *const vx_scan_options, + estimate: *mut vx_estimate, + err: *mut *mut vx_error, +) -> *mut vx_scan { + try_or(err, ptr::null_mut(), || { + let request = scan_request(options)?; + RUNTIME.block_on(async { + let scan = vx_data_source::as_ref(data_source).scan(request).await?; + if !estimate.is_null() { + write_estimate( + scan.partition_count().map(|x| match x { + Precision::Exact(v) => Precision::Exact(v as u64), + Precision::Inexact(v) => Precision::Inexact(v as u64), + }), + unsafe { &mut *estimate }, + ); + } + Ok(vx_scan::new(VxScan::Pending(scan))) + }) + }) +} + +/// Return borrowed vx_scan's dtype. +/// This function will fail if called after vx_scan_next_partition. +/// Called must not free the returned pointer as its lifetime is bound to the +/// lifetime of the scan. +/// On error returns NULL and sets "err". +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scan_dtype( + scan: *const vx_scan, + err: *mut *mut vx_error, +) -> *const vx_dtype { + try_or(err, ptr::null(), || { + let scan = vx_scan::as_ref(scan); + let VxScan::Pending(scan) = scan else { + vortex_bail!("dtype unavailable: scan already started"); + }; + Ok(vx_dtype::new_ref(scan.dtype())) + }) +} + +/// Return an owned partition from a scan. +/// The returned partition must be freed with vx_partition_free. +/// +/// On success returns a partition. +/// On exhaustion (no more partitions in scan) returns NULL but doesn't set +/// "err". +/// On error returns NULL and sets "err". +/// +/// This function is thread-unsafe. Callers running a multi-threaded pipeline +/// should synchronise on calls to this function and dispatch each produced +/// partition to a dedicated worker thread. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_scan_next_partition( + scan: *mut vx_scan, + err: *mut *mut vx_error, +) -> *mut vx_partition { + let scan = vx_scan::as_mut(scan); + let scan = &mut *scan; + unsafe { + let ptr = scan as *mut VxScan; + + let on_finish = || -> VortexResult<*mut vx_partition> { + ptr::write(ptr, VxScan::Finished); + Ok(ptr::null_mut()) + }; + + let on_stream = |mut stream: PartitionStream| -> VortexResult<*mut vx_partition> { + match RUNTIME.block_on(stream.next()) { + Some(partition) => { + let partition = VxPartitionScan::Pending(partition?); + let partition = vx_partition::new(partition); + ptr::write(ptr, VxScan::Started(stream)); + Ok(partition) + } + None => on_finish(), + } + }; + + let owned = ptr::read(ptr); + try_or_default(err, || match owned { + VxScan::Pending(scan) => on_stream(scan.partitions()), + VxScan::Started(stream) => on_stream(stream), + VxScan::Finished => on_finish(), + }) + } +} + +/// Get partition's estimated row count. +/// Must be called before the first call to vx_partition_next. +/// +/// On success, returns 0. +/// On error, return 1 and sets "error". +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_partition_row_count( + partition: *const vx_partition, + count: *mut vx_estimate, + err: *mut *mut vx_error, +) -> c_int { + try_or(err, 1, || { + let partition = vx_partition::as_ref(partition); + let VxPartitionScan::Pending(partition) = partition else { + vortex_bail!("row count unavailable: partition already started"); + }; + write_estimate(partition.row_count(), unsafe { &mut *count }); + Ok(0) + }) +} + +/// Scan partition to ArrowArrayStream. +/// Consumes partition fully: subsequent calls to vx_partition_scan_arrow or +/// vx_partition_next are undefined behaviour. +/// This call blocks current thread until underlying stream is fully consumed. +/// +/// Caller must not free partition after calling this function. +/// +/// On success, sets "stream" and returns 0. +/// On error, sets "err" and returns 1, freeing the partition. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_partition_scan_arrow( + session: *const vx_session, + partition: *mut vx_partition, + stream: *mut FFI_ArrowArrayStream, + err: *mut *mut vx_error, +) -> c_int { + try_or(err, 1, || { + let partition = match *vx_partition::into_box(partition) { + VxPartitionScan::Pending(partition) => partition, + _ => vortex_bail!( + "Can't consume partition into ArrowArrayStream: partition already being consumed" + ), + }; + let array_stream = partition.execute()?; + let dtype = array_stream.dtype(); + + let schema = dtype.to_arrow_schema()?; + let schema = Arc::new(schema); + let data_type = DataType::Struct(schema.fields().clone()); + + let session = vx_session::as_ref(session); + + let on_chunk = move |chunk: VortexResult| -> VortexResult { + let chunk: ArrayRef = chunk?; + let mut ctx: ExecutionCtx = session.create_execution_ctx(); + let arrow = chunk.execute_arrow(Some(&data_type), &mut ctx)?; + Ok(RecordBatch::from(arrow.as_struct().clone())) + }; + + let iter = RUNTIME + .block_on_stream(array_stream) + .map(on_chunk) + .map(|result| result.map_err(|e| ArrowError::ExternalError(Box::new(e)))); + + let reader = RecordBatchIteratorAdapter::new(iter, schema); + let arrow_stream = FFI_ArrowArrayStream::new(Box::new(reader)); + unsafe { + ptr::write(stream, arrow_stream); + }; + Ok(0) + }) +} + +/// Return an owned owned array from a partition. +/// The returned array must be freed with vx_array_free. +/// +/// On success returns an array. +/// On exhaustion (no more arrays in partition) returns NULL but doesn't set +/// "err". +/// On error return NULL and sets "err". +/// +/// This function is not thread-safe: call from one thread per partition. +#[unsafe(no_mangle)] +pub unsafe extern "C-unwind" fn vx_partition_next( + partition: *mut vx_partition, + err: *mut *mut vx_error, +) -> *const vx_array { + let partition = vx_partition::as_mut(partition); + unsafe { + let ptr = partition as *mut VxPartitionScan; + + let on_finish = || -> VortexResult<*const vx_array> { + ptr::write(ptr, VxPartitionScan::Finished); + Ok(ptr::null_mut()) + }; + + let on_stream = |mut stream: SendableArrayStream| -> VortexResult<*const vx_array> { + match RUNTIME.block_on(stream.next()) { + Some(array) => { + let array = vx_array::new(Arc::new(array?)); + ptr::write(ptr, VxPartitionScan::Started(stream)); + Ok(array) + } + None => on_finish(), + } + }; + + let owned = ptr::read(ptr); + try_or_default(err, || match owned { + VxPartitionScan::Pending(partition) => on_stream(partition.execute()?), + VxPartitionScan::Started(stream) => on_stream(stream), + VxPartitionScan::Finished => on_finish(), + }) + } +} + +// Object store error: Generic LocalFileSystem error: Unable to convert +// URL "file:///C:%255CWindows%255CSystemTemp%255C.tmpRXzX38" to filesystem path +// https://github.com/servo/rust-url/issues/1077 +#[cfg(not(windows))] +#[cfg(test)] +mod tests { + use std::ffi::CString; + use std::ptr; + + use vortex::VortexSessionDefault; + use vortex::array::arrays::StructArray; + use vortex::session::VortexSession; + use vortex_array::ExecutionCtx; + use vortex_array::arrays::struct_::StructArrayExt; + use vortex_array::assert_arrays_eq; + + use crate::array::vx_array; + use crate::array::vx_array_free; + use crate::data_source::vx_data_source_free; + use crate::data_source::vx_data_source_new; + use crate::data_source::vx_data_source_options; + use crate::expression::vx_binary_operator; + use crate::expression::vx_expression_binary; + use crate::expression::vx_expression_free; + use crate::expression::vx_expression_get_item; + use crate::expression::vx_expression_literal; + use crate::expression::vx_expression_root; + use crate::scalar::vx_scalar_free; + use crate::scalar::vx_scalar_new_u64; + use crate::scan::vx_data_source_scan; + use crate::scan::vx_estimate; + use crate::scan::vx_partition_free; + use crate::scan::vx_partition_next; + use crate::scan::vx_partition_row_count; + use crate::scan::vx_scan_free; + use crate::scan::vx_scan_next_partition; + use crate::scan::vx_scan_options; + use crate::scan::vx_scan_selection; + use crate::scan::vx_scan_selection_include; + use crate::session::vx_session_free; + use crate::session::vx_session_new; + use crate::tests::SAMPLE_ROWS; + use crate::tests::assert_no_error; + use crate::tests::write_sample; + + /// Perform a scan with options over a sample file, return owned read array and + /// original generated array for the sample file. + fn scan(options: *const vx_scan_options) -> (*const vx_array, StructArray) { + unsafe { + let session = vx_session_new(); + let (sample, struct_array) = write_sample(session); + let path = CString::new(sample.path().to_str().unwrap()).unwrap(); + let ds_options = vx_data_source_options { + paths: path.as_ptr(), + }; + + let mut error = ptr::null_mut(); + let ds = vx_data_source_new(session, &raw const ds_options, &raw mut error); + assert_no_error(error); + assert!(!ds.is_null()); + + let mut error = ptr::null_mut(); + let scan = vx_data_source_scan(ds, options, ptr::null_mut(), &raw mut error); + assert_no_error(error); + assert!(!scan.is_null()); + + let partition = vx_scan_next_partition(scan, &raw mut error); + assert_no_error(error); + assert!(!partition.is_null()); + + let array = vx_partition_next(partition, &raw mut error); + assert_no_error(error); + assert!(!array.is_null()); + + assert!(vx_partition_next(partition, &raw mut error).is_null()); + assert_no_error(error); + assert!(vx_partition_next(partition, &raw mut error).is_null()); + assert_no_error(error); + + vx_partition_free(partition); + vx_scan_free(scan); + vx_data_source_free(ds); + vx_session_free(session); + + (array, struct_array) + } + } + + #[test] + #[cfg_attr(miri, ignore)] + fn test_no_options() { + let (array, struct_array) = scan(ptr::null()); + assert_arrays_eq!(vx_array::as_ref(array), struct_array); + unsafe { vx_array_free(array) }; + } + + #[test] + #[cfg_attr(miri, ignore)] + fn test_project_all() { + let opts = vx_scan_options::default(); + let (array, struct_array) = scan(&raw const opts); + assert_arrays_eq!(vx_array::as_ref(array), struct_array); + unsafe { vx_array_free(array) }; + } + + #[test] + #[cfg_attr(miri, ignore)] + fn test_project_single_field() { + unsafe { + let root = vx_expression_root(); + let mut opts = vx_scan_options::default(); + + for (field, c_field) in [("age", c"age"), ("height", c"height"), ("name", c"name")] { + let field_expr = vx_expression_get_item(c_field.as_ptr(), root); + assert!(!field_expr.is_null()); + opts.projection = field_expr; + let (array, struct_array) = scan(&raw const opts); + assert_arrays_eq!( + vx_array::as_ref(array), + struct_array.unmasked_field_by_name(field).unwrap() + ); + vx_array_free(array); + vx_expression_free(field_expr); + } + vx_expression_free(root); + } + } + + #[test] + #[cfg_attr(miri, ignore)] + fn test_project_sum() { + let session = VortexSession::default(); + let mut ctx = ExecutionCtx::new(session); + unsafe { + let root = vx_expression_root(); + let mut opts = vx_scan_options::default(); + + let expr_age = vx_expression_get_item(c"age".as_ptr(), root); + let expr_height = vx_expression_get_item(c"height".as_ptr(), root); + let expr_sum = + vx_expression_binary(vx_binary_operator::VX_OPERATOR_ADD, expr_age, expr_height); + + opts.projection = expr_sum; + let (array, _) = scan(&raw const opts); + { + let array = vx_array::as_ref(array); + let stats = array.statistics(); + assert!(stats.compute_is_sorted(&mut ctx).unwrap()); + assert_eq!(stats.compute_min(&mut ctx), Some(0)); + assert_eq!( + stats.compute_max(&mut ctx), + Some(200 * (SAMPLE_ROWS - 1) + 199) + ); + } + vx_array_free(array); + + vx_expression_free(expr_age); + vx_expression_free(expr_height); + vx_expression_free(expr_sum); + vx_expression_free(root); + } + } + + #[test] + #[cfg_attr(miri, ignore)] + fn test_filter() { + unsafe { + let root = vx_expression_root(); + let age_expr = vx_expression_get_item(c"age".as_ptr(), root); + let value = vx_scalar_new_u64(100, false); + let mut error = ptr::null_mut(); + let lit_100 = vx_expression_literal(value, &raw mut error); + assert_no_error(error); + vx_scalar_free(value); + let filter = + vx_expression_binary(vx_binary_operator::VX_OPERATOR_GTE, age_expr, lit_100); + + let opts = vx_scan_options { + filter, + ..Default::default() + }; + let (array, _) = scan(&raw const opts); + assert_eq!(vx_array::as_ref(array).len(), 100); + + vx_array_free(array); + vx_expression_free(filter); + vx_expression_free(age_expr); + vx_expression_free(lit_100); + vx_expression_free(root); + } + } + + #[test] + #[cfg_attr(miri, ignore)] + fn test_filter_project() { + unsafe { + let root = vx_expression_root(); + let age_expr = vx_expression_get_item(c"age".as_ptr(), root); + let value = vx_scalar_new_u64(100, false); + let mut error = ptr::null_mut(); + let lit_100 = vx_expression_literal(value, &raw mut error); + assert_no_error(error); + vx_scalar_free(value); + let filter = + vx_expression_binary(vx_binary_operator::VX_OPERATOR_GTE, age_expr, lit_100); + let projection = vx_expression_get_item(c"age".as_ptr(), root); + + let opts = vx_scan_options { + projection, + filter, + ..Default::default() + }; + let (array, _) = scan(&raw const opts); + assert_eq!(vx_array::as_ref(array).len(), 100); + + vx_array_free(array); + vx_expression_free(filter); + vx_expression_free(age_expr); + vx_expression_free(lit_100); + vx_expression_free(projection); + vx_expression_free(root); + } + } + + #[test] + #[cfg_attr(miri, ignore)] + fn test_row_range() { + let opts = vx_scan_options { + row_range_begin: 50, + row_range_end: 100, + ..Default::default() + }; + let (array, _) = scan(&raw const opts); + assert_eq!(vx_array::as_ref(array).len(), 50); + unsafe { vx_array_free(array) }; + } + + #[test] + #[cfg_attr(miri, ignore)] + fn test_selection() { + let indices = [0u64, 50, 100, 150, 199]; + let opts = vx_scan_options { + selection: vx_scan_selection { + idx: indices.as_ptr(), + idx_len: indices.len(), + include: vx_scan_selection_include::VX_SELECTION_INCLUDE_RANGE, + }, + ..Default::default() + }; + let (array, _) = scan(&raw const opts); + assert_eq!(vx_array::as_ref(array).len(), indices.len()); + unsafe { vx_array_free(array) }; + } + + #[test] + #[cfg_attr(miri, ignore)] + fn test_limit() { + let opts = vx_scan_options { + limit: 50, + ..Default::default() + }; + let (array, _) = scan(&raw const opts); + assert_eq!(vx_array::as_ref(array).len(), 50); + unsafe { vx_array_free(array) }; + } + + #[test] + #[cfg_attr(miri, ignore)] + fn test_ordered() { + let opts = vx_scan_options { + ordered: true, + ..Default::default() + }; + let (array, struct_array) = scan(&raw const opts); + assert_arrays_eq!(vx_array::as_ref(array), struct_array); + unsafe { vx_array_free(array) }; + } + + #[test] + #[cfg_attr(miri, ignore)] + fn test_row_count() { + unsafe { + let session = vx_session_new(); + let (sample, _) = write_sample(session); + let path = CString::new(sample.path().to_str().unwrap()).unwrap(); + let ds_options = vx_data_source_options { + paths: path.as_ptr(), + }; + + let mut error = ptr::null_mut(); + let ds = vx_data_source_new(session, &raw const ds_options, &raw mut error); + assert_no_error(error); + + let mut error = ptr::null_mut(); + let scan_ptr = vx_data_source_scan(ds, ptr::null(), ptr::null_mut(), &raw mut error); + assert_no_error(error); + + let mut error = ptr::null_mut(); + let partition = vx_scan_next_partition(scan_ptr, &raw mut error); + assert_no_error(error); + assert!(!partition.is_null()); + + let mut count: vx_estimate = std::mem::zeroed(); + let result = vx_partition_row_count(partition, &raw mut count, &raw mut error); + assert_no_error(error); + assert_eq!(result, 0); + + vx_partition_free(partition); + vx_scan_free(scan_ptr); + vx_data_source_free(ds); + vx_session_free(session); + } + } +} diff --git a/vortex-ffi/src/session.rs b/vortex-ffi/src/session.rs index 668598852eb..5e8fc4b4909 100644 --- a/vortex-ffi/src/session.rs +++ b/vortex-ffi/src/session.rs @@ -31,3 +31,35 @@ pub unsafe extern "C-unwind" fn vx_session_clone(session: *const vx_session) -> let session = vx_session::as_ref(session); vx_session::new(session.clone()) } + +#[cfg(test)] +mod tests { + use crate::session::vx_session_clone; + use crate::session::vx_session_free; + use crate::session::vx_session_new; + + #[test] + #[cfg_attr(miri, ignore)] + fn test_basic() { + unsafe { + let session = vx_session_new(); + assert!(!session.is_null()); + vx_session_free(session); + } + } + + #[test] + #[cfg_attr(miri, ignore)] + fn test_clone() { + unsafe { + let session = vx_session_new(); + assert!(!session.is_null()); + + let copy = vx_session_clone(session); + assert!(!copy.is_null()); + vx_session_free(session); + + vx_session_free(copy); + } + } +} diff --git a/vortex-ffi/src/sink.rs b/vortex-ffi/src/sink.rs index 7b0bbe69ca0..639d22196e3 100644 --- a/vortex-ffi/src/sink.rs +++ b/vortex-ffi/src/sink.rs @@ -27,7 +27,7 @@ use crate::error::try_or_default; use crate::error::vx_error; use crate::session::vx_session; -#[allow(non_camel_case_types)] +#[expect(non_camel_case_types)] /// The `sink` interface is used to collect array chunks and place them into a resource /// (e.g. an array stream or file (`vx_array_sink_open_file`)). /// diff --git a/vortex-ffi/src/struct_array.rs b/vortex-ffi/src/struct_array.rs index 33409531be0..f3023625a82 100644 --- a/vortex-ffi/src/struct_array.rs +++ b/vortex-ffi/src/struct_array.rs @@ -28,6 +28,7 @@ crate::box_wrapper!(StructBuilder, vx_struct_column_builder); /// Create a new column-wise struct array builder with given validity and a /// capacity hint. validity can't be NULL. +/// Capacity hint is for the number of columns. /// If you don't know capacity, pass 0. /// if validity is NULL, returns NULL. #[unsafe(no_mangle)] @@ -132,7 +133,8 @@ mod tests { use std::sync::Arc; use vortex::array::IntoArray; - use vortex::array::ToCanonical; + use vortex::array::LEGACY_SESSION; + use vortex::array::VortexSessionExecute; use vortex::array::arrays::PrimitiveArray; use vortex::array::arrays::StructArray; use vortex::array::arrays::VarBinViewArray; @@ -169,6 +171,7 @@ mod tests { #[test] #[cfg_attr(miri, ignore)] fn test_many() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let names = ["age", "name"]; let age_field = PrimitiveArray::new(buffer![30u8, 25u8, 35u8], Validity::NonNullable); let name_field = VarBinViewArray::from_iter_str(["Alice", "Bob", "Charlie"]); @@ -236,7 +239,10 @@ mod tests { assert!(!array.is_null()); { - let array = vx_array::as_ref(array).to_struct(); + let array = vx_array::as_ref(array) + .clone() + .execute::(&mut ctx) + .unwrap(); assert_arrays_eq!(array, struct_array); } diff --git a/vortex-ffi/test/CMakeLists.txt b/vortex-ffi/test/CMakeLists.txt index 029a63efffc..be0288ef954 100644 --- a/vortex-ffi/test/CMakeLists.txt +++ b/vortex-ffi/test/CMakeLists.txt @@ -1,8 +1,6 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright the Vortex contributors include(CTest) -include(FetchContent) - FetchContent_Declare( Catch GIT_REPOSITORY https://github.com/catchorg/Catch2.git @@ -10,9 +8,14 @@ FetchContent_Declare( ) FetchContent_MakeAvailable(Catch) include(Catch) +# https://github.com/catchorg/Catch2/issues/1833 +target_compile_definitions(Catch2 PRIVATE CATCH_CONFIG_NO_POSIX_SIGNALS) file(GLOB TEST_FILES "${CMAKE_CURRENT_SOURCE_DIR}/*.cpp") message(NOTICE "Test files ${TEST_FILES}") add_executable(vortex_ffi_test ${TEST_FILES}) -target_link_libraries(vortex_ffi_test PRIVATE vortex_ffi_shared Catch2::Catch2WithMain) +target_link_libraries(vortex_ffi_test PRIVATE + vortex_ffi_shared + Catch2::Catch2WithMain + nanoarrow_shared) catch_discover_tests(vortex_ffi_test) diff --git a/vortex-ffi/test/common.h b/vortex-ffi/test/common.h index dbbcf56d318..29087e7a48d 100644 --- a/vortex-ffi/test/common.h +++ b/vortex-ffi/test/common.h @@ -1,7 +1,6 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors #pragma once - #include #include #include "vortex.h" @@ -19,8 +18,28 @@ inline std::string_view to_string_view(vx_error *err) { return to_string_view(vx_error_get_message(err)); } -inline void require_no_error(vx_error *err) { - if (err) { - FAIL(to_string(err)); +inline void require_no_error(vx_error *error, bool assert = true) { + if (!error) { + return; + } + auto message = to_string(error); + vx_error_free(error); + if (assert) { + FAIL(message); + } else { + throw std::runtime_error(message); } } + +template +struct Defer { + Defer(F &&f) : f(std::move(f)) { + } + ~Defer() { + f(); + } + F f; +}; +#define CONCAT(x, y) x##y +#define CONCAT2(x, y) CONCAT(x, y) +#define defer Defer CONCAT2(defer_, __LINE__) = [&] diff --git a/vortex-ffi/test/scan.cpp b/vortex-ffi/test/scan.cpp new file mode 100644 index 00000000000..a55a98a4bcc --- /dev/null +++ b/vortex-ffi/test/scan.cpp @@ -0,0 +1,878 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors +#include +#include +#include +#include +#include +#include +#include +#include +#include + +using FFI_ArrowArrayStream = ArrowArrayStream; +using FFI_ArrowSchema = ArrowSchema; +#define USE_OWN_ARROW 1 +#include + +#include "common.h" + +namespace fs = std::filesystem; +using namespace std::string_literals; +using namespace std::string_view_literals; +using Catch::Matchers::ContainsSubstring; +using nanoarrow::UniqueArray; +using nanoarrow::UniqueArrayStream; +using nanoarrow::UniqueArrayView; +using nanoarrow::UniqueSchema; + +struct TempPath : fs::path { + TempPath() = default; + explicit TempPath(fs::path p) : fs::path(std::move(p)) { + } + + TempPath(const TempPath &) = delete; + TempPath &operator=(const TempPath &) = delete; + + TempPath(TempPath &&other) noexcept : fs::path(std::move(other)) { + } + TempPath &operator=(TempPath &&other) noexcept { + if (this != &other) { + fs::remove(*this); + fs::path::operator=(std::move(other)); + } + return *this; + } + + ~TempPath() { + if (!empty()) { + fs::remove(*this); + } + } +}; + +// StructArray { age=u8, height=u16? } +[[nodiscard]] const vx_dtype *sample_dtype() { + vx_struct_fields_builder *builder = vx_struct_fields_builder_new(); + + constexpr auto age = "age"sv; + const vx_string *age_name = vx_string_new(age.data(), age.size()); + const vx_dtype *age_type = vx_dtype_new_primitive(PTYPE_U8, false); + vx_struct_fields_builder_add_field(builder, age_name, age_type); + + constexpr auto height = "height"sv; + const vx_string *height_name = vx_string_new(height.data(), height.size()); + const vx_dtype *height_type = vx_dtype_new_primitive(PTYPE_U16, true); + vx_struct_fields_builder_add_field(builder, height_name, height_type); + + vx_struct_fields *fields = vx_struct_fields_builder_finalize(builder); + return vx_dtype_new_struct(fields, false); +} + +constexpr size_t SAMPLE_ROWS = 100; +std::vector sample_age() { + std::vector buf; + for (uint8_t age = 0; age < SAMPLE_ROWS; ++age) { + buf.push_back(age); + } + return buf; +} + +std::vector sample_height() { + std::vector buf; + for (uint16_t height = 0; height < SAMPLE_ROWS; ++height) { + buf.push_back((height + 1) % 200); + } + return buf; +} + +[[nodiscard]] const vx_array *sample_array() { + vx_validity validity = {}; + validity.type = VX_VALIDITY_NON_NULLABLE; + + vx_struct_column_builder *builder = vx_struct_column_builder_new(&validity, SAMPLE_ROWS); + vx_error *error = nullptr; + + std::vector age_buffer = sample_age(); + const vx_array *age_array = + vx_array_new_primitive(PTYPE_U8, age_buffer.data(), age_buffer.size(), &validity, &error); + defer { + vx_array_free(age_array); + }; + require_no_error(error); + + vx_struct_column_builder_add_field(builder, "age", age_array, &error); + require_no_error(error); + + std::vector height_buffer = sample_height(); + validity.type = VX_VALIDITY_ALL_VALID; + const vx_array *height_array = + vx_array_new_primitive(PTYPE_U16, height_buffer.data(), height_buffer.size(), &validity, &error); + defer { + vx_array_free(height_array); + }; + require_no_error(error); + + vx_struct_column_builder_add_field(builder, "height", height_array, &error); + require_no_error(error); + + const vx_array *array = vx_struct_column_builder_finalize(builder, &error); + require_no_error(error); + return array; +} + +UniqueSchema sample_schema() { + UniqueSchema schema; + REQUIRE(ArrowSchemaInitFromType(schema.get(), NANOARROW_TYPE_STRUCT) == NANOARROW_OK); + REQUIRE(ArrowSchemaAllocateChildren(schema.get(), 2) == NANOARROW_OK); + REQUIRE(ArrowSchemaInitFromType(schema->children[0], NANOARROW_TYPE_UINT8) == NANOARROW_OK); + REQUIRE(ArrowSchemaSetName(schema->children[0], "age") == NANOARROW_OK); + REQUIRE(ArrowSchemaInitFromType(schema->children[1], NANOARROW_TYPE_UINT16) == NANOARROW_OK); + REQUIRE(ArrowSchemaSetName(schema->children[1], "height") == NANOARROW_OK); + return schema; +} + +UniqueArrayStream sample_array_stream() { + UniqueSchema schema = sample_schema(); + UniqueArray arr; + + REQUIRE(ArrowArrayInitFromSchema(arr.get(), schema.get(), nullptr) == NANOARROW_OK); + REQUIRE(ArrowArrayStartAppending(arr.get()) == NANOARROW_OK); + + auto ages = sample_age(); + auto heights = sample_height(); + for (size_t i = 0; i < ages.size(); ++i) { + REQUIRE(ArrowArrayAppendInt(arr->children[0], ages[i]) == NANOARROW_OK); + REQUIRE(ArrowArrayAppendInt(arr->children[1], heights[i]) == NANOARROW_OK); + REQUIRE(ArrowArrayFinishElement(arr.get()) == NANOARROW_OK); + } + + REQUIRE(ArrowArrayFinishBuildingDefault(arr.get(), nullptr) == NANOARROW_OK); + + UniqueArrayStream stream; + REQUIRE(ArrowBasicArrayStreamInit(stream.get(), schema.get(), 1) == NANOARROW_OK); + + ArrowBasicArrayStreamSetArray(stream.get(), 0, arr.get()); + return stream; +} + +[[nodiscard]] TempPath write_sample(vx_session *session) { + const fs::path path = std::filesystem::temp_directory_path() / + fs::path("test-" + std::to_string(std::random_device {}()) + ".vortex"); + + const vx_dtype *dtype = sample_dtype(); + defer { + vx_dtype_free(dtype); + }; + + vx_error *error = nullptr; + vx_array_sink *sink = vx_array_sink_open_file(session, path.c_str(), dtype, &error); + REQUIRE(sink != nullptr); + require_no_error(error); + + const vx_array *array = sample_array(); + defer { + vx_array_free(array); + }; + vx_array_sink_push(sink, array, &error); + require_no_error(error); + + vx_array_sink_close(sink, &error); + require_no_error(error); + + INFO("Written vortex file "s + path.generic_string()); + return TempPath {path}; +} + +TEST_CASE("Creating datasources", "[datasource]") { + vx_session *session = vx_session_new(); + defer { + vx_session_free(session); + }; + vx_error *error = nullptr; + + const vx_data_source *ds = vx_data_source_new(session, nullptr, &error); + REQUIRE(ds == nullptr); + REQUIRE(error != nullptr); + vx_error_free(error); + + vx_data_source_options opts = {}; + ds = vx_data_source_new(session, &opts, &error); + REQUIRE(ds == nullptr); + REQUIRE(error != nullptr); + REQUIRE_THAT(to_string(error), ContainsSubstring("opts.paths")); + vx_error_free(error); + + // First file is opened eagerly + opts.paths = "nonexistent"; + ds = vx_data_source_new(session, &opts, &error); + REQUIRE(ds == nullptr); + REQUIRE(error != nullptr); + REQUIRE_THAT(to_string(error), ContainsSubstring("No such file or directory")); + vx_error_free(error); + + opts.paths = "/tmp2/*.vortex"; + ds = vx_data_source_new(session, &opts, &error); + REQUIRE(ds == nullptr); + REQUIRE(error != nullptr); + vx_error_free(error); + + TempPath file = write_sample(session); + opts.paths = file.c_str(); + ds = vx_data_source_new(session, &opts, &error); + require_no_error(error); + REQUIRE(ds != nullptr); + vx_data_source_free(ds); +} + +TEST_CASE("Write file", "[sink]") { + vx_session *session = vx_session_new(); + defer { + vx_session_free(session); + }; + TempPath path = write_sample(session); +} + +TEST_CASE("Write file and read dtypes", "[datasource]") { + vx_session *session = vx_session_new(); + defer { + vx_session_free(session); + }; + TempPath path = write_sample(session); + vx_error *error = nullptr; + + vx_data_source_options opts = {}; + opts.paths = path.c_str(); + + const vx_data_source *ds = vx_data_source_new(session, &opts, &error); + require_no_error(error); + REQUIRE(ds != nullptr); + defer { + vx_data_source_free(ds); + }; + + vx_estimate row_count; + vx_data_source_get_row_count(ds, &row_count); + + CHECK(row_count.type == VX_ESTIMATE_EXACT); + CHECK(row_count.estimate == SAMPLE_ROWS); + + const vx_dtype *data_source_dtype = vx_data_source_dtype(ds); + REQUIRE(vx_dtype_get_variant(data_source_dtype) == DTYPE_STRUCT); + + const vx_struct_fields *fields = vx_dtype_struct_dtype(data_source_dtype); + const size_t len = vx_struct_fields_nfields(fields); + REQUIRE(len == 2); + + const vx_dtype *age_dtype = vx_struct_fields_field_dtype(fields, 0); + defer { + vx_dtype_free(age_dtype); + }; + const vx_string *age_name = vx_struct_fields_field_name(fields, 0); + REQUIRE(vx_dtype_get_variant(age_dtype) == DTYPE_PRIMITIVE); + REQUIRE(vx_dtype_primitive_ptype(age_dtype) == PTYPE_U8); + REQUIRE_FALSE(vx_dtype_is_nullable(age_dtype)); + REQUIRE(to_string_view(age_name) == "age"); + + const vx_dtype *height_dtype = vx_struct_fields_field_dtype(fields, 1); + defer { + vx_dtype_free(height_dtype); + }; + const vx_string *height_name = vx_struct_fields_field_name(fields, 1); + REQUIRE(vx_dtype_get_variant(height_dtype) == DTYPE_PRIMITIVE); + REQUIRE(vx_dtype_primitive_ptype(height_dtype) == PTYPE_U16); + REQUIRE(vx_dtype_is_nullable(height_dtype)); + REQUIRE(to_string_view(height_name) == "height"); +} + +void verify_age_field(const vx_array *age_field) { + REQUIRE(vx_array_has_dtype(age_field, DTYPE_PRIMITIVE)); + REQUIRE(vx_dtype_primitive_ptype(vx_array_dtype(age_field)) == PTYPE_U8); + REQUIRE(vx_array_len(age_field) == SAMPLE_ROWS); + for (size_t i = 0; i < SAMPLE_ROWS; ++i) { + REQUIRE(vx_array_get_u8(age_field, i) == i); + } +} + +void verify_height_field(const vx_array *height_field) { + REQUIRE(vx_array_has_dtype(height_field, DTYPE_PRIMITIVE)); + REQUIRE(vx_dtype_primitive_ptype(vx_array_dtype(height_field)) == PTYPE_U16); + REQUIRE(vx_array_len(height_field) == SAMPLE_ROWS); + for (size_t i = 0; i < SAMPLE_ROWS; ++i) { + REQUIRE(vx_array_get_u16(height_field, i) > 0); + } +} + +void verify_sample_array(const vx_array *array) { + REQUIRE(vx_array_len(array) == SAMPLE_ROWS); + REQUIRE(vx_array_has_dtype(array, DTYPE_STRUCT)); + + const vx_struct_fields *fields = vx_dtype_struct_dtype(vx_array_dtype(array)); + size_t len = vx_struct_fields_nfields(fields); + REQUIRE(len == 2); + + const vx_dtype *age_dtype = vx_struct_fields_field_dtype(fields, 0); + REQUIRE(age_dtype != nullptr); + REQUIRE(vx_dtype_get_variant(age_dtype) == DTYPE_PRIMITIVE); + REQUIRE(vx_dtype_primitive_ptype(age_dtype) == PTYPE_U8); + vx_dtype_free(age_dtype); + const vx_string *age_name = vx_struct_fields_field_name(fields, 0); + REQUIRE(to_string_view(age_name) == "age"); + + const vx_dtype *height_dtype = vx_struct_fields_field_dtype(fields, 1); + REQUIRE(vx_dtype_get_variant(height_dtype) == DTYPE_PRIMITIVE); + REQUIRE(vx_dtype_primitive_ptype(height_dtype) == PTYPE_U16); + vx_dtype_free(height_dtype); + const vx_string *height_name = vx_struct_fields_field_name(fields, 1); + REQUIRE(to_string_view(height_name) == "height"); + + vx_error *error = nullptr; + vx_validity validity = {}; + vx_array_get_validity(array, &validity, &error); + require_no_error(error); + REQUIRE(validity.type == VX_VALIDITY_NON_NULLABLE); + + const vx_array *age_field = vx_array_get_field(array, 0, &error); + require_no_error(error); + verify_age_field(age_field); + vx_array_free(age_field); + + const vx_array *height_field = vx_array_get_field(array, 1, &error); + require_no_error(error); + verify_height_field(height_field); + vx_array_free(height_field); + + REQUIRE(vx_array_get_field(array, 2, &error) == nullptr); + REQUIRE(error != nullptr); + vx_error_free(error); +} + +TEST_CASE("Requesting scans", "[datasource]") { + vx_session *session = vx_session_new(); + defer { + vx_session_free(session); + }; + TempPath path = write_sample(session); + + vx_data_source_options ds_options = {}; + ds_options.paths = path.c_str(); + + vx_error *error = nullptr; + const vx_data_source *ds = vx_data_source_new(session, &ds_options, &error); + require_no_error(error); + REQUIRE(ds != nullptr); + defer { + vx_data_source_free(ds); + }; + + { + vx_scan *scan = vx_data_source_scan(ds, nullptr, nullptr, &error); + require_no_error(error); + REQUIRE(scan != nullptr); + vx_scan_free(scan); + } + + vx_scan_options options = {}; + options.max_threads = 1; + + { + vx_scan *scan = vx_data_source_scan(ds, &options, nullptr, &error); + require_no_error(error); + REQUIRE(scan != nullptr); + vx_scan_free(scan); + } +} + +TEST_CASE("Basic scan", "[datasource]") { + vx_session *session = vx_session_new(); + defer { + vx_session_free(session); + }; + TempPath path = write_sample(session); + vx_error *error = nullptr; + + vx_data_source_options ds_options = {}; + ds_options.paths = path.c_str(); + + const vx_data_source *ds = vx_data_source_new(session, &ds_options, &error); + require_no_error(error); + REQUIRE(ds != nullptr); + defer { + vx_data_source_free(ds); + }; + + vx_estimate estimate = {}; + vx_scan *scan = vx_data_source_scan(ds, nullptr, &estimate, &error); + require_no_error(error); + defer { + vx_scan_free(scan); + }; + REQUIRE(scan != nullptr); + REQUIRE(estimate.type == VX_ESTIMATE_EXACT); + REQUIRE(estimate.estimate == 1); + + vx_partition *partition = vx_scan_next_partition(scan, &error); + require_no_error(error); + defer { + vx_partition_free(partition); + }; + + estimate = {}; + vx_partition_row_count(partition, &estimate, &error); + require_no_error(error); + REQUIRE(estimate.type == VX_ESTIMATE_EXACT); + REQUIRE(estimate.estimate == SAMPLE_ROWS); + + REQUIRE(vx_scan_next_partition(scan, &error) == nullptr); + require_no_error(error); + + const vx_array *array = vx_partition_next(partition, &error); + require_no_error(error); + REQUIRE(array != nullptr); + defer { + vx_array_free(array); + }; + + REQUIRE(vx_partition_next(partition, &error) == nullptr); + require_no_error(error); + + verify_sample_array(array); +} + +TEST_CASE("Multithreaded scan", "[datasource]") { + vx_session *session = vx_session_new(); + defer { + vx_session_free(session); + }; + + constexpr size_t NUM_FILES = 10; + std::vector paths(NUM_FILES); + std::string paths_str; + for (size_t i = 0; i < NUM_FILES; ++i) { + paths[i] = write_sample(session); + if (i == 0) { + paths_str = paths[i].c_str(); + } else { + paths_str += ","s + paths[i].c_str(); + } + } + + vx_data_source_options ds_options = {}; + ds_options.paths = paths_str.c_str(); + + vx_error *error = nullptr; + const vx_data_source *ds = vx_data_source_new(session, &ds_options, &error); + require_no_error(error); + REQUIRE(ds != nullptr); + defer { + vx_data_source_free(ds); + }; + + vx_estimate estimate = {}; + vx_scan *scan = vx_data_source_scan(ds, nullptr, &estimate, &error); + require_no_error(error); + defer { + vx_scan_free(scan); + }; + REQUIRE(scan != nullptr); + REQUIRE(estimate.type == VX_ESTIMATE_INEXACT); + REQUIRE(estimate.estimate == NUM_FILES); + + // Catch doesn't support multithreaded assertions, so we throw here + std::mutex mutex; + std::vector threads(NUM_FILES); + threads.reserve(NUM_FILES); + std::vector arrays(NUM_FILES); + for (size_t i = 0; i < NUM_FILES; ++i) { + threads[i] = std::thread([&, i] { + vx_error *error_tl = nullptr; + vx_partition *partition = nullptr; + { + std::lock_guard _(mutex); + partition = vx_scan_next_partition(scan, &error_tl); + require_no_error(error_tl, false); + if (!partition) { + throw std::runtime_error("partition = nullptr"); + } + } + + defer { + vx_partition_free(partition); + }; + + estimate = {}; + vx_partition_row_count(partition, &estimate, &error_tl); + require_no_error(error_tl, false); + if (estimate.type != VX_ESTIMATE_EXACT) { + throw std::runtime_error("estimate type mismatch"); + } + if (estimate.estimate != SAMPLE_ROWS) { + throw std::runtime_error("estimate mismatch"); + } + + const vx_array *array = vx_partition_next(partition, &error_tl); + require_no_error(error_tl, false); + if (!array) { + throw std::runtime_error("array = nullptr"); + } + + arrays[i] = array; + }); + } + + for (auto &thread : threads) { + thread.join(); + } + + vx_partition *const partition = vx_scan_next_partition(scan, &error); + require_no_error(error); + REQUIRE(partition == nullptr); + + for (const vx_array *array : arrays) { + REQUIRE(array != nullptr); + defer { + vx_array_free(array); + }; + verify_sample_array(array); + } +} + +const vx_array *scan_with_options(vx_scan_options &options) { + vx_session *session = vx_session_new(); + defer { + vx_session_free(session); + }; + TempPath path = write_sample(session); + vx_error *error = nullptr; + + vx_data_source_options ds_options = {}; + ds_options.paths = path.c_str(); + + const vx_data_source *ds = vx_data_source_new(session, &ds_options, &error); + require_no_error(error); + REQUIRE(ds != nullptr); + defer { + vx_data_source_free(ds); + }; + + vx_scan *scan = vx_data_source_scan(ds, &options, nullptr, &error); + require_no_error(error); + REQUIRE(scan != nullptr); + defer { + vx_scan_free(scan); + }; + + vx_partition *partition = vx_scan_next_partition(scan, &error); + require_no_error(error); + REQUIRE(partition != nullptr); + defer { + vx_partition_free(partition); + }; + + const vx_array *array = vx_partition_next(partition, &error); + require_no_error(error); + REQUIRE(array != nullptr); + + return array; +} + +TEST_CASE("Project all fields", "[projection]") { + vx_scan_options opts = {}; + const vx_array *array = scan_with_options(opts); + defer { + vx_array_free(array); + }; + verify_sample_array(array); +} + +TEST_CASE("Project root", "[projection]") { + vx_expression *root = vx_expression_root(); + defer { + vx_expression_free(root); + }; + vx_scan_options opts = {}; + opts.projection = root; + const vx_array *array = scan_with_options(opts); + defer { + vx_array_free(array); + }; + verify_sample_array(array); +} + +TEST_CASE("Project single field", "[projection]") { + vx_expression *root = vx_expression_root(); + defer { + vx_expression_free(root); + }; + vx_scan_options opts = {}; + + vx_expression *age_field = vx_expression_get_item("age", root); + REQUIRE(age_field != nullptr); + defer { + vx_expression_free(age_field); + }; + + { + opts.projection = age_field; + const vx_array *array = scan_with_options(opts); + defer { + vx_array_free(array); + }; + verify_age_field(array); + } + + vx_expression *height_field = vx_expression_get_item("height", root); + REQUIRE(height_field != nullptr); + defer { + vx_expression_free(height_field); + }; + + { + opts.projection = height_field; + const vx_array *array = scan_with_options(opts); + defer { + vx_array_free(array); + }; + verify_height_field(array); + } +} + +TEST_CASE("Filter with literal expression", "[filter]") { + vx_expression *root = vx_expression_root(); + defer { + vx_expression_free(root); + }; + + vx_expression *age_field = vx_expression_get_item("age", root); + REQUIRE(age_field != nullptr); + defer { + vx_expression_free(age_field); + }; + + uint8_t threshold = 50; + vx_scalar *threshold_scalar = vx_scalar_new_u8(threshold, false); + REQUIRE(threshold_scalar != nullptr); + defer { + vx_scalar_free(threshold_scalar); + }; + + vx_error *literal_error = nullptr; + vx_expression *threshold_expr = vx_expression_literal(threshold_scalar, &literal_error); + require_no_error(literal_error); + REQUIRE(threshold_expr != nullptr); + defer { + vx_expression_free(threshold_expr); + }; + + vx_expression *filter = vx_expression_binary(VX_OPERATOR_GTE, age_field, threshold_expr); + REQUIRE(filter != nullptr); + defer { + vx_expression_free(filter); + }; + + vx_scan_options opts = {}; + opts.filter = filter; + const vx_array *array = scan_with_options(opts); + defer { + vx_array_free(array); + }; + + REQUIRE(vx_array_len(array) == SAMPLE_ROWS - threshold); + + vx_error *error = nullptr; + const vx_array *filtered_age = vx_array_get_field(array, 0, &error); + require_no_error(error); + REQUIRE(filtered_age != nullptr); + defer { + vx_array_free(filtered_age); + }; + + for (size_t i = 0; i < vx_array_len(filtered_age); ++i) { + REQUIRE(vx_array_get_u8(filtered_age, i) == static_cast(threshold + i)); + } +} + +TEST_CASE("Project UTF-8 literal expression", "[projection]") { + constexpr auto value = "constant"sv; + vx_error *scalar_error = nullptr; + vx_scalar *literal_scalar = vx_scalar_new_utf8(value.data(), value.size(), false, &scalar_error); + require_no_error(scalar_error); + REQUIRE(literal_scalar != nullptr); + defer { + vx_scalar_free(literal_scalar); + }; + + vx_error *literal_error = nullptr; + vx_expression *literal_expr = vx_expression_literal(literal_scalar, &literal_error); + require_no_error(literal_error); + REQUIRE(literal_expr != nullptr); + defer { + vx_expression_free(literal_expr); + }; + + vx_scan_options opts = {}; + opts.projection = literal_expr; + const vx_array *array = scan_with_options(opts); + defer { + vx_array_free(array); + }; + + REQUIRE(vx_array_len(array) == SAMPLE_ROWS); + + for (size_t i : {size_t {0}, SAMPLE_ROWS - 1}) { + const vx_string *actual = vx_array_get_utf8(array, static_cast(i)); + REQUIRE(actual != nullptr); + defer { + vx_string_free(actual); + }; + REQUIRE(to_string_view(actual) == value); + } +} + +void compare_schemas(const UniqueSchema &left, const UniqueSchema &right) { + REQUIRE(std::string_view {left->format} == std::string_view {right->format}); + REQUIRE(left->n_children == right->n_children); + for (int64_t i = 0; i < left->n_children; i++) { + std::string_view name_left = left->children[i]->name; + std::string_view name_right = right->children[i]->name; + REQUIRE(name_left == name_right); + compare_schemas(left->children[i], right->children[i]); + } +} + +void compare_schema_with_sample(const UniqueSchema &left) { + compare_schemas(left, sample_schema()); +} + +void compare_stream_with_sample(UniqueArrayStream &left) { + UniqueArrayStream right = sample_array_stream(); + UniqueSchema schema_right = sample_schema(); + + ArrowError error; + UniqueSchema schema_left; + ArrowErrorCode res = ArrowArrayStreamGetSchema(left.get(), schema_left.get(), &error); + REQUIRE(res == NANOARROW_OK); + + while (true) { + UniqueArray chunk_left, chunk_right; + int next_left = left->get_next(left.get(), chunk_left.get()); + int next_right = right->get_next(right.get(), chunk_right.get()); + REQUIRE(next_left == next_right); + if (next_left != NANOARROW_OK || next_right != NANOARROW_OK) { + return; + } + + bool done_left = chunk_left->release == nullptr; + bool done_right = chunk_right->release == nullptr; + REQUIRE(done_left == done_right); + if (done_left || done_right) { + return; + } + + REQUIRE(chunk_left->length == chunk_right->length); + + UniqueArrayView view_left, view_right; + res = ArrowArrayViewInitFromSchema(view_left.get(), schema_left.get(), nullptr); + REQUIRE(res == NANOARROW_OK); + res = ArrowArrayViewInitFromSchema(view_right.get(), schema_right.get(), nullptr); + REQUIRE(res == NANOARROW_OK); + + res = ArrowArrayViewSetArray(view_left.get(), chunk_left.get(), nullptr); + REQUIRE(res == NANOARROW_OK); + res = ArrowArrayViewSetArray(view_right.get(), chunk_right.get(), nullptr); + REQUIRE(res == NANOARROW_OK); + + for (int64_t i = 0; i < chunk_left->length; i++) { + auto name_left = ArrowArrayViewGetIntUnsafe(view_left->children[0], i); + auto name_right = ArrowArrayViewGetIntUnsafe(view_right->children[0], i); + REQUIRE(name_left == name_right); + + auto age_left = ArrowArrayViewGetIntUnsafe(view_left->children[1], i); + auto age_right = ArrowArrayViewGetIntUnsafe(view_right->children[1], i); + REQUIRE(age_left == age_right); + } + } +} + +TEST_CASE("Scan Arrow schema", "[scan]") { + vx_session *session = vx_session_new(); + defer { + vx_session_free(session); + }; + + TempPath path = write_sample(session); + vx_error *error = nullptr; + + vx_data_source_options ds_options = {}; + ds_options.paths = path.c_str(); + + const vx_data_source *ds = vx_data_source_new(session, &ds_options, &error); + require_no_error(error); + REQUIRE(ds != nullptr); + defer { + vx_data_source_free(ds); + }; + + vx_scan *scan = vx_data_source_scan(ds, nullptr, nullptr, &error); + require_no_error(error); + REQUIRE(scan != nullptr); + defer { + vx_scan_free(scan); + }; + + ArrowSchema schema; + const vx_dtype *dtype = vx_scan_dtype(scan, &error); + require_no_error(error); + REQUIRE(dtype != nullptr); + defer { + vx_dtype_free(dtype); + }; + + int res = vx_dtype_to_arrow_schema(dtype, &schema, &error); + REQUIRE(res == 0); + require_no_error(error); + + UniqueSchema unique_schema; + ArrowSchemaMove(&schema, unique_schema.get()); + compare_schema_with_sample(unique_schema); +} + +TEST_CASE("Scan to Arrow", "[scan]") { + vx_session *session = vx_session_new(); + defer { + vx_session_free(session); + }; + TempPath path = write_sample(session); + vx_error *error = nullptr; + + vx_data_source_options ds_options = {}; + ds_options.paths = path.c_str(); + + const vx_data_source *ds = vx_data_source_new(session, &ds_options, &error); + require_no_error(error); + REQUIRE(ds != nullptr); + defer { + vx_data_source_free(ds); + }; + + vx_scan *scan = vx_data_source_scan(ds, nullptr, nullptr, &error); + require_no_error(error); + REQUIRE(scan != nullptr); + defer { + vx_scan_free(scan); + }; + + vx_partition *partition = vx_scan_next_partition(scan, &error); + require_no_error(error); + REQUIRE(partition != nullptr); + + UniqueArrayStream unique_stream; + { + ArrowArrayStream stream = {}; + int res = vx_partition_scan_arrow(session, partition, &stream, &error); + REQUIRE(res == 0); + require_no_error(error); + ArrowArrayStreamMove(&stream, unique_stream.get()); + } + compare_stream_with_sample(unique_stream); +} diff --git a/vortex-ffi/tsan_suppressions.txt b/vortex-ffi/tsan_suppressions.txt index 6f9ffaae597..ee8ac6e926d 100644 --- a/vortex-ffi/tsan_suppressions.txt +++ b/vortex-ffi/tsan_suppressions.txt @@ -5,3 +5,4 @@ # This is likely a false positive in TSan for oneshot::channel and kanal # where ordering is correct but uses an explicit fence and not relaxed load race:oneshot-*/src/channel.rs +race:oneshot-*/src/lib.rs diff --git a/vortex-file/Cargo.toml b/vortex-file/Cargo.toml index e812ab228c6..77d664a12cb 100644 --- a/vortex-file/Cargo.toml +++ b/vortex-file/Cargo.toml @@ -21,7 +21,6 @@ async-trait = { workspace = true } bytes = { workspace = true } flatbuffers = { workspace = true } futures = { workspace = true, features = ["std", "async-await"] } -getrandom_v03 = { workspace = true } # Needed to pickup the "wasm_js" feature for wasm targets from the workspace configuration itertools = { workspace = true } kanal = { workspace = true } moka = { workspace = true, features = ["sync"] } @@ -31,7 +30,6 @@ parking_lot = { workspace = true } pin-project-lite = { workspace = true } tokio = { workspace = true, features = ["rt"], optional = true } tracing = { workspace = true } -uuid = { workspace = true } # Needed to pickup the "js" feature for wasm targets from the workspace configuration vortex-alp = { workspace = true } vortex-array = { workspace = true } vortex-btrblocks = { workspace = true } @@ -84,6 +82,3 @@ unstable_encodings = [ "vortex-zstd?/unstable_encodings", "vortex-btrblocks/unstable_encodings", ] - -[package.metadata.cargo-machete] -ignored = ["getrandom_v03", "uuid"] diff --git a/vortex-file/public-api.lock b/vortex-file/public-api.lock index 36d501a926e..ac0af26fff3 100644 --- a/vortex-file/public-api.lock +++ b/vortex-file/public-api.lock @@ -6,15 +6,13 @@ pub struct vortex_file::multi::MultiFileDataSource impl vortex_file::multi::MultiFileDataSource -pub async fn vortex_file::multi::MultiFileDataSource::build(self) -> vortex_error::VortexResult +pub async fn vortex_file::multi::MultiFileDataSource::build(self) -> vortex_error::VortexResult -pub fn vortex_file::multi::MultiFileDataSource::new(session: vortex_session::VortexSession) -> Self +pub fn vortex_file::multi::MultiFileDataSource::new(vortex_session::VortexSession) -> Self -pub fn vortex_file::multi::MultiFileDataSource::with_filesystem(self, fs: vortex_io::filesystem::FileSystemRef) -> Self +pub fn vortex_file::multi::MultiFileDataSource::with_glob(self, impl core::convert::Into, core::option::Option) -> Self -pub fn vortex_file::multi::MultiFileDataSource::with_glob(self, glob: impl core::convert::Into) -> Self - -pub fn vortex_file::multi::MultiFileDataSource::with_open_options(self, f: impl core::ops::function::Fn(vortex_file::VortexOpenOptions) -> vortex_file::VortexOpenOptions + core::marker::Send + core::marker::Sync + 'static) -> Self +pub fn vortex_file::multi::MultiFileDataSource::with_open_options(self, impl core::ops::function::Fn(vortex_file::VortexOpenOptions) -> vortex_file::VortexOpenOptions + core::marker::Send + core::marker::Sync + 'static) -> Self pub mod vortex_file::segments @@ -28,17 +26,17 @@ pub vortex_file::segments::ReadEvent::Request(vortex_file::read::request::ReadRe impl core::fmt::Debug for vortex_file::segments::ReadEvent -pub fn vortex_file::segments::ReadEvent::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_file::segments::ReadEvent::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_file::segments::FileSegmentSource impl vortex_file::segments::FileSegmentSource -pub fn vortex_file::segments::FileSegmentSource::open(segments: alloc::sync::Arc<[vortex_file::SegmentSpec]>, reader: R, handle: vortex_io::runtime::handle::Handle, metrics: vortex_file::segments::RequestMetrics) -> Self +pub fn vortex_file::segments::FileSegmentSource::open(alloc::sync::Arc<[vortex_file::SegmentSpec]>, R, vortex_io::runtime::handle::Handle, vortex_file::segments::RequestMetrics) -> Self impl vortex_layout::segments::source::SegmentSource for vortex_file::segments::FileSegmentSource -pub fn vortex_file::segments::FileSegmentSource::request(&self, id: vortex_layout::segments::SegmentId) -> vortex_layout::segments::source::SegmentFuture +pub fn vortex_file::segments::FileSegmentSource::request(&self, vortex_layout::segments::SegmentId) -> vortex_layout::segments::source::SegmentFuture pub struct vortex_file::segments::InitialReadSegmentCache @@ -48,9 +46,9 @@ pub vortex_file::segments::InitialReadSegmentCache::initial: parking_lot::rwlock impl vortex_layout::segments::cache::SegmentCache for vortex_file::segments::InitialReadSegmentCache -pub fn vortex_file::segments::InitialReadSegmentCache::get<'life0, 'async_trait>(&'life0 self, id: vortex_layout::segments::SegmentId) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait +pub fn vortex_file::segments::InitialReadSegmentCache::get<'life0, 'async_trait>(&'life0 self, vortex_layout::segments::SegmentId) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait -pub fn vortex_file::segments::InitialReadSegmentCache::put<'life0, 'async_trait>(&'life0 self, id: vortex_layout::segments::SegmentId, buffer: vortex_buffer::ByteBuffer) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait +pub fn vortex_file::segments::InitialReadSegmentCache::put<'life0, 'async_trait>(&'life0 self, vortex_layout::segments::SegmentId, vortex_buffer::ByteBuffer) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait pub struct vortex_file::segments::RequestMetrics @@ -62,7 +60,7 @@ pub vortex_file::segments::RequestMetrics::num_requests_coalesced: vortex_metric impl vortex_file::segments::RequestMetrics -pub fn vortex_file::segments::RequestMetrics::new(metrics_registry: &dyn vortex_metrics::MetricsRegistry, labels: alloc::vec::Vec) -> Self +pub fn vortex_file::segments::RequestMetrics::new(&dyn vortex_metrics::MetricsRegistry, alloc::vec::Vec) -> Self pub mod vortex_file::v2 @@ -70,25 +68,29 @@ pub struct vortex_file::v2::FileStatsLayoutReader impl vortex_file::v2::FileStatsLayoutReader -pub fn vortex_file::v2::FileStatsLayoutReader::new(child: vortex_layout::reader::LayoutReaderRef, file_stats: vortex_file::FileStatistics, session: vortex_session::VortexSession) -> Self +pub fn vortex_file::v2::FileStatsLayoutReader::file_stats(&self) -> &vortex_file::FileStatistics + +pub fn vortex_file::v2::FileStatsLayoutReader::new(vortex_layout::reader::LayoutReaderRef, vortex_file::FileStatistics, vortex_session::VortexSession) -> Self impl vortex_array::expr::pruning::StatsCatalog for vortex_file::v2::FileStatsLayoutReader -pub fn vortex_file::v2::FileStatsLayoutReader::stats_ref(&self, field_path: &vortex_array::dtype::field::FieldPath, stat: vortex_array::expr::stats::Stat) -> core::option::Option +pub fn vortex_file::v2::FileStatsLayoutReader::stats_ref(&self, &vortex_array::dtype::field::FieldPath, vortex_array::expr::stats::Stat) -> core::option::Option impl vortex_layout::reader::LayoutReader for vortex_file::v2::FileStatsLayoutReader +pub fn vortex_file::v2::FileStatsLayoutReader::as_any(&self) -> &dyn core::any::Any + pub fn vortex_file::v2::FileStatsLayoutReader::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_file::v2::FileStatsLayoutReader::filter_evaluation(&self, row_range: &core::ops::range::Range, expr: &vortex_array::expr::expression::Expression, mask: vortex_array::mask_future::MaskFuture) -> vortex_error::VortexResult +pub fn vortex_file::v2::FileStatsLayoutReader::filter_evaluation(&self, &core::ops::range::Range, &vortex_array::expr::expression::Expression, vortex_array::mask_future::MaskFuture) -> vortex_error::VortexResult pub fn vortex_file::v2::FileStatsLayoutReader::name(&self) -> &alloc::sync::Arc -pub fn vortex_file::v2::FileStatsLayoutReader::projection_evaluation(&self, row_range: &core::ops::range::Range, expr: &vortex_array::expr::expression::Expression, mask: vortex_array::mask_future::MaskFuture) -> vortex_error::VortexResult +pub fn vortex_file::v2::FileStatsLayoutReader::projection_evaluation(&self, &core::ops::range::Range, &vortex_array::expr::expression::Expression, vortex_array::mask_future::MaskFuture) -> vortex_error::VortexResult -pub fn vortex_file::v2::FileStatsLayoutReader::pruning_evaluation(&self, row_range: &core::ops::range::Range, expr: &vortex_array::expr::expression::Expression, mask: vortex_mask::Mask) -> vortex_error::VortexResult +pub fn vortex_file::v2::FileStatsLayoutReader::pruning_evaluation(&self, &core::ops::range::Range, &vortex_array::expr::expression::Expression, vortex_mask::Mask) -> vortex_error::VortexResult -pub fn vortex_file::v2::FileStatsLayoutReader::register_splits(&self, field_mask: &[vortex_array::dtype::field_mask::FieldMask], row_range: &core::ops::range::Range, splits: &mut alloc::collections::btree::set::BTreeSet) -> vortex_error::VortexResult<()> +pub fn vortex_file::v2::FileStatsLayoutReader::register_splits(&self, &[vortex_array::dtype::field_mask::FieldMask], &core::ops::range::Range, &mut alloc::collections::btree::set::BTreeSet) -> vortex_error::VortexResult<()> pub fn vortex_file::v2::FileStatsLayoutReader::row_count(&self) -> u64 @@ -106,15 +108,15 @@ pub vortex_file::DeserializeStep::NeedMoreData::offset: u64 impl core::fmt::Debug for vortex_file::DeserializeStep -pub fn vortex_file::DeserializeStep::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_file::DeserializeStep::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_file::BlockingWrite<'rt, B: vortex_io::runtime::blocking::BlockingRuntime> impl<'rt, B: vortex_io::runtime::blocking::BlockingRuntime> vortex_file::BlockingWrite<'rt, B> -pub fn vortex_file::BlockingWrite<'rt, B>::write(self, write: W, iter: impl vortex_array::iter::ArrayIterator + core::marker::Send + 'static) -> vortex_error::VortexResult +pub fn vortex_file::BlockingWrite<'rt, B>::write(self, W, impl vortex_array::iter::ArrayIterator + core::marker::Send + 'static) -> vortex_error::VortexResult -pub fn vortex_file::BlockingWrite<'rt, B>::writer<'w, W: std::io::Write + core::marker::Unpin + 'w>(self, write: W, dtype: vortex_array::dtype::DType) -> vortex_file::BlockingWriter<'rt, 'w, B> +pub fn vortex_file::BlockingWrite<'rt, B>::writer<'w, W: std::io::Write + core::marker::Unpin + 'w>(self, W, vortex_array::dtype::DType) -> vortex_file::BlockingWriter<'rt, 'w, B> pub struct vortex_file::BlockingWriter<'rt, 'w, B: vortex_io::runtime::blocking::BlockingRuntime> @@ -126,7 +128,7 @@ pub fn vortex_file::BlockingWriter<'_, '_, B>::bytes_written(&self) -> u64 pub fn vortex_file::BlockingWriter<'_, '_, B>::finish(self) -> vortex_error::VortexResult -pub fn vortex_file::BlockingWriter<'_, '_, B>::push(&mut self, chunk: vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult<()> +pub fn vortex_file::BlockingWriter<'_, '_, B>::push(&mut self, vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult<()> pub struct vortex_file::FileStatistics @@ -134,13 +136,13 @@ impl vortex_file::FileStatistics pub fn vortex_file::FileStatistics::dtypes(&self) -> &alloc::sync::Arc<[vortex_array::dtype::DType]> -pub fn vortex_file::FileStatistics::from_flatbuffer<'a>(fb: &vortex_flatbuffers::footer::FileStatistics<'a>, file_dtype: &vortex_array::dtype::DType, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_file::FileStatistics::from_flatbuffer<'a>(&vortex_flatbuffers::footer::FileStatistics<'a>, &vortex_array::dtype::DType, &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_file::FileStatistics::get(&self, field_idx: usize) -> (&vortex_array::stats::stats_set::StatsSet, &vortex_array::dtype::DType) +pub fn vortex_file::FileStatistics::get(&self, usize) -> (&vortex_array::stats::stats_set::StatsSet, &vortex_array::dtype::DType) -pub fn vortex_file::FileStatistics::new(stats: alloc::sync::Arc<[vortex_array::stats::stats_set::StatsSet]>, dtypes: alloc::sync::Arc<[vortex_array::dtype::DType]>) -> Self +pub fn vortex_file::FileStatistics::new(alloc::sync::Arc<[vortex_array::stats::stats_set::StatsSet]>, alloc::sync::Arc<[vortex_array::dtype::DType]>) -> Self -pub fn vortex_file::FileStatistics::new_with_dtype(stats: alloc::sync::Arc<[vortex_array::stats::stats_set::StatsSet]>, file_dtype: &vortex_array::dtype::DType) -> Self +pub fn vortex_file::FileStatistics::new_with_dtype(alloc::sync::Arc<[vortex_array::stats::stats_set::StatsSet]>, &vortex_array::dtype::DType) -> Self pub fn vortex_file::FileStatistics::stats_sets(&self) -> &alloc::sync::Arc<[vortex_array::stats::stats_set::StatsSet]> @@ -150,7 +152,7 @@ pub fn vortex_file::FileStatistics::clone(&self) -> vortex_file::FileStatistics impl core::fmt::Debug for vortex_file::FileStatistics -pub fn vortex_file::FileStatistics::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_file::FileStatistics::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_flatbuffers::FlatBufferRoot for vortex_file::FileStatistics @@ -158,7 +160,7 @@ impl vortex_flatbuffers::WriteFlatBuffer for vortex_file::FileStatistics pub type vortex_file::FileStatistics::Target<'a> = vortex_flatbuffers::footer::FileStatistics<'a> -pub fn vortex_file::FileStatistics::write_flatbuffer<'fb>(&self, fbb: &mut flatbuffers::builder::FlatBufferBuilder<'fb>) -> vortex_error::VortexResult> +pub fn vortex_file::FileStatistics::write_flatbuffer<'fb>(&self, &mut flatbuffers::builder::FlatBufferBuilder<'fb>) -> vortex_error::VortexResult> impl<'a> core::iter::traits::collect::IntoIterator for &'a vortex_file::FileStatistics @@ -174,7 +176,7 @@ impl vortex_file::Footer pub fn vortex_file::Footer::approx_byte_size(&self) -> core::option::Option -pub fn vortex_file::Footer::deserializer(eof_buffer: vortex_buffer::ByteBuffer, session: vortex_session::VortexSession) -> vortex_file::FooterDeserializer +pub fn vortex_file::Footer::deserializer(vortex_buffer::ByteBuffer, vortex_session::VortexSession) -> vortex_file::FooterDeserializer pub fn vortex_file::Footer::dtype(&self) -> &vortex_array::dtype::DType @@ -194,7 +196,7 @@ pub fn vortex_file::Footer::clone(&self) -> vortex_file::Footer impl core::fmt::Debug for vortex_file::Footer -pub fn vortex_file::Footer::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_file::Footer::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_file::FooterDeserializer @@ -204,15 +206,15 @@ pub fn vortex_file::FooterDeserializer::buffer(&self) -> &vortex_buffer::ByteBuf pub fn vortex_file::FooterDeserializer::deserialize(&mut self) -> vortex_error::VortexResult -pub fn vortex_file::FooterDeserializer::prefix_data(&mut self, more_data: vortex_buffer::ByteBuffer) +pub fn vortex_file::FooterDeserializer::prefix_data(&mut self, vortex_buffer::ByteBuffer) -pub fn vortex_file::FooterDeserializer::with_dtype(self, dtype: vortex_array::dtype::DType) -> Self +pub fn vortex_file::FooterDeserializer::with_dtype(self, vortex_array::dtype::DType) -> Self -pub fn vortex_file::FooterDeserializer::with_size(self, file_size: u64) -> Self +pub fn vortex_file::FooterDeserializer::with_size(self, u64) -> Self -pub fn vortex_file::FooterDeserializer::with_some_dtype(self, dtype: core::option::Option) -> Self +pub fn vortex_file::FooterDeserializer::with_some_dtype(self, core::option::Option) -> Self -pub fn vortex_file::FooterDeserializer::with_some_size(self, file_size: core::option::Option) -> Self +pub fn vortex_file::FooterDeserializer::with_some_size(self, core::option::Option) -> Self pub struct vortex_file::FooterSerializer @@ -222,9 +224,9 @@ pub fn vortex_file::FooterSerializer::exclude_dtype(self) -> Self pub fn vortex_file::FooterSerializer::serialize(self) -> vortex_error::VortexResult> -pub fn vortex_file::FooterSerializer::with_exclude_dtype(self, exclude_dtype: bool) -> Self +pub fn vortex_file::FooterSerializer::with_exclude_dtype(self, bool) -> Self -pub fn vortex_file::FooterSerializer::with_offset(self, offset: u64) -> Self +pub fn vortex_file::FooterSerializer::with_offset(self, u64) -> Self pub struct vortex_file::SegmentSpec @@ -244,17 +246,17 @@ pub fn vortex_file::SegmentSpec::clone(&self) -> vortex_file::SegmentSpec impl core::convert::From<&vortex_file::SegmentSpec> for vortex_flatbuffers::footer::SegmentSpec -pub fn vortex_flatbuffers::footer::SegmentSpec::from(value: &vortex_file::SegmentSpec) -> Self +pub fn vortex_flatbuffers::footer::SegmentSpec::from(&vortex_file::SegmentSpec) -> Self impl core::convert::TryFrom<&vortex_flatbuffers::footer::SegmentSpec> for vortex_file::SegmentSpec pub type vortex_file::SegmentSpec::Error = vortex_error::VortexError -pub fn vortex_file::SegmentSpec::try_from(value: &vortex_flatbuffers::footer::SegmentSpec) -> core::result::Result +pub fn vortex_file::SegmentSpec::try_from(&vortex_flatbuffers::footer::SegmentSpec) -> core::result::Result impl core::fmt::Debug for vortex_file::SegmentSpec -pub fn vortex_file::SegmentSpec::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_file::SegmentSpec::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_file::SegmentSpec @@ -262,7 +264,7 @@ pub struct vortex_file::VortexFile impl vortex_file::VortexFile -pub fn vortex_file::VortexFile::can_prune(&self, filter: &vortex_array::expr::expression::Expression) -> vortex_error::VortexResult +pub fn vortex_file::VortexFile::can_prune(&self, &vortex_array::expr::expression::Expression) -> vortex_error::VortexResult pub fn vortex_file::VortexFile::data_source(&self) -> vortex_error::VortexResult @@ -290,53 +292,53 @@ pub struct vortex_file::VortexOpenOptions impl vortex_file::VortexOpenOptions -pub async fn vortex_file::VortexOpenOptions::open(self, source: alloc::sync::Arc) -> vortex_error::VortexResult +pub async fn vortex_file::VortexOpenOptions::open(self, alloc::sync::Arc) -> vortex_error::VortexResult -pub fn vortex_file::VortexOpenOptions::open_buffer>(self, buffer: B) -> vortex_error::VortexResult +pub fn vortex_file::VortexOpenOptions::open_buffer>(self, B) -> vortex_error::VortexResult -pub async fn vortex_file::VortexOpenOptions::open_path(self, path: impl core::convert::AsRef) -> vortex_error::VortexResult +pub async fn vortex_file::VortexOpenOptions::open_path(self, impl core::convert::AsRef) -> vortex_error::VortexResult -pub async fn vortex_file::VortexOpenOptions::open_read(self, reader: R) -> vortex_error::VortexResult +pub async fn vortex_file::VortexOpenOptions::open_read(self, R) -> vortex_error::VortexResult -pub fn vortex_file::VortexOpenOptions::with_dtype(self, dtype: vortex_array::dtype::DType) -> Self +pub fn vortex_file::VortexOpenOptions::with_dtype(self, vortex_array::dtype::DType) -> Self -pub fn vortex_file::VortexOpenOptions::with_file_size(self, file_size: u64) -> Self +pub fn vortex_file::VortexOpenOptions::with_file_size(self, u64) -> Self -pub fn vortex_file::VortexOpenOptions::with_footer(self, footer: vortex_file::Footer) -> Self +pub fn vortex_file::VortexOpenOptions::with_footer(self, vortex_file::Footer) -> Self -pub fn vortex_file::VortexOpenOptions::with_initial_read_size(self, initial_read_size: usize) -> Self +pub fn vortex_file::VortexOpenOptions::with_initial_read_size(self, usize) -> Self -pub fn vortex_file::VortexOpenOptions::with_labels(self, labels: alloc::vec::Vec) -> Self +pub fn vortex_file::VortexOpenOptions::with_labels(self, alloc::vec::Vec) -> Self -pub fn vortex_file::VortexOpenOptions::with_metrics_registry(self, metrics: alloc::sync::Arc) -> Self +pub fn vortex_file::VortexOpenOptions::with_metrics_registry(self, alloc::sync::Arc) -> Self -pub fn vortex_file::VortexOpenOptions::with_segment_cache(self, segment_cache: alloc::sync::Arc) -> Self +pub fn vortex_file::VortexOpenOptions::with_segment_cache(self, alloc::sync::Arc) -> Self -pub fn vortex_file::VortexOpenOptions::with_some_file_size(self, file_size: core::option::Option) -> Self +pub fn vortex_file::VortexOpenOptions::with_some_file_size(self, core::option::Option) -> Self impl vortex_file::VortexOpenOptions -pub async fn vortex_file::VortexOpenOptions::open_object_store(self, object_store: &alloc::sync::Arc, path: &str) -> vortex_error::VortexResult +pub async fn vortex_file::VortexOpenOptions::open_object_store(self, &alloc::sync::Arc, &str) -> vortex_error::VortexResult pub struct vortex_file::VortexWriteOptions impl vortex_file::VortexWriteOptions -pub fn vortex_file::VortexWriteOptions::blocking(self, runtime: &B) -> vortex_file::BlockingWrite<'_, B> +pub fn vortex_file::VortexWriteOptions::blocking(self, &B) -> vortex_file::BlockingWrite<'_, B> -pub async fn vortex_file::VortexWriteOptions::write(self, write: W, stream: S) -> vortex_error::VortexResult +pub async fn vortex_file::VortexWriteOptions::write(self, W, S) -> vortex_error::VortexResult -pub fn vortex_file::VortexWriteOptions::writer<'w, W: vortex_io::write::VortexWrite + core::marker::Unpin + 'w>(self, write: W, dtype: vortex_array::dtype::DType) -> vortex_file::Writer<'w> +pub fn vortex_file::VortexWriteOptions::writer<'w, W: vortex_io::write::VortexWrite + core::marker::Unpin + 'w>(self, W, vortex_array::dtype::DType) -> vortex_file::Writer<'w> impl vortex_file::VortexWriteOptions pub fn vortex_file::VortexWriteOptions::exclude_dtype(self) -> Self -pub fn vortex_file::VortexWriteOptions::new(session: vortex_session::VortexSession) -> Self +pub fn vortex_file::VortexWriteOptions::new(vortex_session::VortexSession) -> Self -pub fn vortex_file::VortexWriteOptions::with_file_statistics(self, file_statistics: alloc::vec::Vec) -> Self +pub fn vortex_file::VortexWriteOptions::with_file_statistics(self, alloc::vec::Vec) -> Self -pub fn vortex_file::VortexWriteOptions::with_strategy(self, strategy: alloc::sync::Arc) -> Self +pub fn vortex_file::VortexWriteOptions::with_strategy(self, alloc::sync::Arc) -> Self pub struct vortex_file::WriteStrategyBuilder @@ -344,17 +346,17 @@ impl vortex_file::WriteStrategyBuilder pub fn vortex_file::WriteStrategyBuilder::build(self) -> alloc::sync::Arc -pub fn vortex_file::WriteStrategyBuilder::with_allow_encodings(self, allow_encodings: vortex_utils::aliases::hash_set::HashSet) -> Self +pub fn vortex_file::WriteStrategyBuilder::with_allow_encodings(self, vortex_utils::aliases::hash_set::HashSet) -> Self -pub fn vortex_file::WriteStrategyBuilder::with_btrblocks_builder(self, builder: vortex_btrblocks::builder::BtrBlocksCompressorBuilder) -> Self +pub fn vortex_file::WriteStrategyBuilder::with_btrblocks_builder(self, vortex_btrblocks::builder::BtrBlocksCompressorBuilder) -> Self -pub fn vortex_file::WriteStrategyBuilder::with_compressor(self, compressor: C) -> Self +pub fn vortex_file::WriteStrategyBuilder::with_compressor(self, C) -> Self -pub fn vortex_file::WriteStrategyBuilder::with_field_writer(self, field: impl core::convert::Into, writer: alloc::sync::Arc) -> Self +pub fn vortex_file::WriteStrategyBuilder::with_field_writer(self, impl core::convert::Into, alloc::sync::Arc) -> Self -pub fn vortex_file::WriteStrategyBuilder::with_flat_strategy(self, flat: alloc::sync::Arc) -> Self +pub fn vortex_file::WriteStrategyBuilder::with_flat_strategy(self, alloc::sync::Arc) -> Self -pub fn vortex_file::WriteStrategyBuilder::with_row_block_size(self, row_block_size: usize) -> Self +pub fn vortex_file::WriteStrategyBuilder::with_row_block_size(self, usize) -> Self impl core::clone::Clone for vortex_file::WriteStrategyBuilder @@ -366,7 +368,13 @@ pub fn vortex_file::WriteStrategyBuilder::default() -> Self impl core::fmt::Debug for vortex_file::WriteStrategyBuilder -pub fn vortex_file::WriteStrategyBuilder::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_file::WriteStrategyBuilder::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +impl vortex_session::SessionVar for vortex_file::WriteStrategyBuilder + +pub fn vortex_file::WriteStrategyBuilder::as_any(&self) -> &dyn core::any::Any + +pub fn vortex_file::WriteStrategyBuilder::as_any_mut(&mut self) -> &mut dyn core::any::Any pub struct vortex_file::WriteSummary @@ -388,9 +396,9 @@ pub fn vortex_file::Writer<'_>::bytes_written(&self) -> u64 pub async fn vortex_file::Writer<'_>::finish(self) -> vortex_error::VortexResult -pub async fn vortex_file::Writer<'_>::push(&mut self, chunk: vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult<()> +pub async fn vortex_file::Writer<'_>::push(&mut self, vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult<()> -pub async fn vortex_file::Writer<'_>::push_stream(&mut self, stream: vortex_array::stream::SendableArrayStream) -> vortex_error::VortexResult<()> +pub async fn vortex_file::Writer<'_>::push_stream(&mut self, vortex_array::stream::SendableArrayStream) -> vortex_error::VortexResult<()> pub const vortex_file::EOF_SIZE: usize @@ -422,4 +430,4 @@ impl vortex_file::WriteOptionsSessionExt for S pub fn S::write_options(&self) -> vortex_file::VortexWriteOptions -pub fn vortex_file::register_default_encodings(session: &vortex_session::VortexSession) +pub fn vortex_file::register_default_encodings(&vortex_session::VortexSession) diff --git a/vortex-file/src/file.rs b/vortex-file/src/file.rs index 7257676b05c..f321d774197 100644 --- a/vortex-file/src/file.rs +++ b/vortex-file/src/file.rs @@ -12,7 +12,9 @@ use std::sync::Arc; use itertools::Itertools; use vortex_array::ArrayRef; use vortex_array::Columnar; +use vortex_array::IntoArray; use vortex_array::VortexSessionExecute; +use vortex_array::arrays::ConstantArray; use vortex_array::dtype::DType; use vortex_array::dtype::Field; use vortex_array::dtype::FieldMask; @@ -20,6 +22,7 @@ use vortex_array::dtype::FieldPath; use vortex_array::dtype::FieldPathSet; use vortex_array::expr::Expression; use vortex_array::expr::pruning::checked_pruning_expr; +use vortex_array::scalar_fn::internal::row_count::substitute_row_count; use vortex_error::VortexResult; use vortex_layout::LayoutReader; use vortex_layout::scan::layout::LayoutReaderDataSource; @@ -117,7 +120,11 @@ impl VortexFile { )) } - /// Returns true if the expression will never match any rows in the file. + /// Returns `true` if file-level statistics prove the expression cannot + /// match any rows in this file. + /// + /// Row-count-aware pruning predicates are evaluated with the file's total + /// row count as their scope. pub fn can_prune(&self, filter: &Expression) -> VortexResult { let Some((stats, fields)) = self .footer @@ -162,16 +169,18 @@ impl VortexFile { return Ok(false); }; + // Apply the predicate, then substitute any row_count placeholders in the resulting array + // tree with a ConstantArray carrying the file-level row count. + let applied = file_stats.apply(&predicate)?; + let row_count_replacement = + ConstantArray::new(self.footer.row_count(), applied.len()).into_array(); + let applied = substitute_row_count(applied, &row_count_replacement)?; + let mut ctx = self.session.create_execution_ctx(); - Ok( - match file_stats - .apply(&predicate)? - .execute::(&mut ctx)? - { - Columnar::Constant(s) => s.scalar().as_bool().value() == Some(true), - Columnar::Canonical(_) => false, - }, - ) + Ok(match applied.execute::(&mut ctx)? { + Columnar::Constant(s) => s.scalar().as_bool().value() == Some(true), + Columnar::Canonical(_) => false, + }) } pub fn splits(&self) -> VortexResult>> { diff --git a/vortex-file/src/footer/deserializer.rs b/vortex-file/src/footer/deserializer.rs index 9b594e31fd6..4d0085d43de 100644 --- a/vortex-file/src/footer/deserializer.rs +++ b/vortex-file/src/footer/deserializer.rs @@ -81,7 +81,7 @@ impl FooterDeserializer { } pub fn deserialize(&mut self) -> VortexResult { - let postscript = if let Some(ref postscript) = self.postscript { + let postscript = if let Some(postscript) = &self.postscript { postscript } else { self.postscript = Some(self.parse_postscript(&self.buffer)?); diff --git a/vortex-file/src/footer/mod.rs b/vortex-file/src/footer/mod.rs index e0d08568108..ac2153f7cd1 100644 --- a/vortex-file/src/footer/mod.rs +++ b/vortex-file/src/footer/mod.rs @@ -88,7 +88,7 @@ impl Footer { let layout_ids: Arc<[_]> = layout_specs .iter() .flat_map(|e| e.iter()) - .map(|encoding| LayoutEncodingId::new_arc(Arc::from(encoding.id()))) + .map(|encoding| LayoutEncodingId::new(encoding.id())) .collect(); let layout_read_ctx = ReadContext::new(layout_ids); @@ -97,7 +97,7 @@ impl Footer { let array_ids: Arc<[_]> = array_specs .iter() .flat_map(|e| e.iter()) - .map(|encoding| ArrayId::new_arc(Arc::from(encoding.id()))) + .map(|encoding| ArrayId::new(encoding.id())) .collect(); let array_read_ctx = ReadContext::new(array_ids); diff --git a/vortex-file/src/lib.rs b/vortex-file/src/lib.rs index 000df4a7bcb..ce6598173a6 100644 --- a/vortex-file/src/lib.rs +++ b/vortex-file/src/lib.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::cast_possible_truncation)] +#![expect(clippy::cast_possible_truncation)] #![doc(html_logo_url = "/vortex/docs/_static/vortex_spiral_logo.svg")] //! Read and write Vortex layouts, a serialization of Vortex arrays. //! @@ -110,6 +110,8 @@ pub use forever_constant::*; pub use open::*; pub use strategy::*; use vortex_array::arrays::Dict; +use vortex_array::arrays::Patched; +use vortex_array::arrays::patched::use_experimental_patches; use vortex_array::session::ArraySessionExt; use vortex_bytebool::ByteBool; use vortex_fsst::FSST; @@ -168,6 +170,9 @@ pub fn register_default_encodings(session: &VortexSession) { arrays.register(vortex_zstd::Zstd); #[cfg(all(feature = "zstd", feature = "unstable_encodings"))] arrays.register(vortex_zstd::ZstdBuffers); + if use_experimental_patches() { + arrays.register(Patched); + } } // Eventually all encodings crates should expose an initialize function. For now it's only diff --git a/vortex-file/src/multi/mod.rs b/vortex-file/src/multi/mod.rs index 2e0c3735499..0b97376c362 100644 --- a/vortex-file/src/multi/mod.rs +++ b/vortex-file/src/multi/mod.rs @@ -13,7 +13,6 @@ use session::MultiFileSessionExt; use tracing::debug; use vortex_error::VortexResult; use vortex_error::vortex_bail; -use vortex_error::vortex_err; use vortex_io::filesystem::FileListing; use vortex_io::filesystem::FileSystemRef; use vortex_layout::LayoutReaderRef; @@ -26,33 +25,41 @@ use crate::OpenOptionsSessionExt; use crate::VortexOpenOptions; use crate::v2::FileStatsLayoutReader; -/// A builder that discovers multiple Vortex files from a glob pattern and constructs a +/// A builder that discovers multiple Vortex files from glob patterns and constructs a /// [`MultiLayoutDataSource`] to scan them as a single data source. /// -/// The primary interface is [`Self::with_glob`], which accepts a glob -/// pattern (optionally prefixed with `file://`). For non-local filesystems (S3, GCS, etc.), -/// callers must also provide a [`FileSystemRef`] via [`Self::with_filesystem`]). +/// The primary interface is [`Self::with_glob`], which accepts a glob pattern and an optional +/// filesystem. For non-local filesystems (S3, GCS, etc.), callers must provide a [`FileSystemRef`]. +/// For local files, pass `None` and a local filesystem will be created automatically. /// /// # Examples /// /// ```ignore /// // Local files — filesystem is auto-created: /// let ds = MultiFileDataSource::new(session) -/// .with_glob("/data/warehouse/*.vortex") +/// .with_glob("/data/warehouse/*.vortex", None) /// .build() /// .await?; /// /// // S3 — caller provides the filesystem: /// let ds = MultiFileDataSource::new(session) -/// .with_filesystem(s3_fs) -/// .with_glob("prefix/*.vortex") +/// .with_glob("prefix/*.vortex", Some(s3_fs)) +/// .build() +/// .await?; +/// +/// // Mixed filesystems — multiple globs with different filesystems: +/// let ds = MultiFileDataSource::new(session) +/// .with_glob("bucket-a/*.vortex", Some(s3_fs.clone())) +/// .with_glob("bucket-b/*.vortex", Some(s3_fs)) +/// .with_glob("gcs-bucket/*.vortex", Some(gcs_fs)) /// .build() /// .await?; /// ``` pub struct MultiFileDataSource { session: VortexSession, - fs: Option, - glob: Option, + /// List of (glob, optional filesystem) pairs to resolve. + /// When the filesystem is None, a local filesystem will be created in build(). + glob_sources: Vec<(String, Option)>, open_options_fn: Arc VortexOpenOptions + Send + Sync>, } @@ -61,26 +68,29 @@ impl MultiFileDataSource { pub fn new(session: VortexSession) -> Self { Self { session, - fs: None, - glob: None, + glob_sources: Vec::new(), open_options_fn: Arc::new(|opts| opts), } } - /// Set the path glob for file discovery. + /// Add a path glob for file discovery. /// - /// This path should be relative to the filesystem's base URL. - pub fn with_glob(mut self, glob: impl Into) -> Self { - self.glob = Some(glob.into().trim_start_matches("/").to_string()); - self - } - - /// Set the filesystem to use for file discovery and reading. + /// The glob path should be relative to the filesystem's base URL. Pass `None` for the + /// filesystem to use the local filesystem (auto-created in [`Self::build`]). /// - /// Required for non-local URLs (S3, GCS, etc.). For `file://` or bare path URLs, - /// a local filesystem is created automatically if none is provided. - pub fn with_filesystem(mut self, fs: FileSystemRef) -> Self { - self.fs = Some(fs); + /// Relative paths are resolved against the process working directory. + pub fn with_glob(mut self, glob: impl Into, fs: Option) -> Self { + let glob = glob.into(); + let glob = if fs.is_none() && std::path::Path::new(&glob).is_relative() { + std::env::current_dir() + .map(|cwd| cwd.join(&glob).to_string_lossy().into_owned()) + .unwrap_or(glob) + .trim_start_matches('/') + .to_string() + } else { + glob.trim_start_matches('/').to_string() + }; + self.glob_sources.push((glob, fs)); self } @@ -99,36 +109,58 @@ impl MultiFileDataSource { /// /// Discovers files via glob, opens the first file eagerly to determine the schema, /// and creates lazy factories for the remaining files. - pub async fn build(mut self) -> VortexResult { - let glob = self - .glob - .take() - .ok_or_else(|| vortex_err!("MultiFileDataSource requires a glob URL"))?; - - let fs = match self.fs { - Some(fs) => fs, - None => create_local_filesystem(&self.session)?, - }; - let files: Vec = fs.glob(&glob)?.try_collect().await?; + pub async fn build(self) -> VortexResult { + if self.glob_sources.is_empty() { + vortex_bail!("MultiFileDataSource requires at least one glob pattern"); + } + + // Create local filesystem lazily if needed (only if any glob lacks a filesystem). + let local_fs: Option = self + .glob_sources + .iter() + .any(|(_, fs)| fs.is_none()) + .then(|| create_local_filesystem(&self.session)) + .transpose()?; + + // Collect files from all glob sources. + let mut all_files: Vec<(FileListing, FileSystemRef)> = Vec::new(); + for (glob, maybe_fs) in &self.glob_sources { + // Use the provided filesystem, or fall back to the local filesystem. + // We know local_fs is Some when maybe_fs is None (by construction above). + let fs = maybe_fs + .as_ref() + .or(local_fs.as_ref()) + .map(Arc::clone) + .unwrap_or_else(|| { + unreachable!("local_fs is set when any glob lacks a filesystem") + }); + let files: Vec = fs.glob(glob)?.try_collect().await?; + for file in files { + all_files.push((file, Arc::clone(&fs))); + } + } - if files.is_empty() { - vortex_bail!("No files matched the glob pattern '{}'", glob); + if all_files.is_empty() { + let globs: Vec<_> = self.glob_sources.iter().map(|(g, _)| g.as_str()).collect(); + vortex_bail!("No files matched the glob pattern(s): {:?}", globs); } - let file_count = files.len(); - debug!(file_count, glob = %glob, "discovered files"); + let file_count = all_files.len(); + let globs: Vec<_> = self.glob_sources.iter().map(|(g, _)| g.as_str()).collect(); + debug!(file_count, glob = ?globs, "discovered files"); // Open first file eagerly for dtype. - let first_file = - open_file(&fs, &files[0], &self.session, self.open_options_fn.as_ref()).await?; + let (first_file_listing, first_fs) = &all_files[0]; + let open_fn = self.open_options_fn.as_ref(); + let first_file = open_file(first_fs, first_file_listing, &self.session, open_fn).await?; let first_reader = layout_reader_with_stats(&first_file)?; - let factories: Vec> = files[1..] + let factories: Vec> = all_files[1..] .iter() - .map(|f| { + .map(|(file, fs)| { Arc::new(VortexFileReaderFactory { - fs: Arc::clone(&fs), - file: f.clone(), + fs: Arc::clone(fs), + file: file.clone(), session: self.session.clone(), open_options_fn: Arc::clone(&self.open_options_fn), }) as Arc diff --git a/vortex-file/src/multi/session.rs b/vortex-file/src/multi/session.rs index d7705006d37..7023d3cd020 100644 --- a/vortex-file/src/multi/session.rs +++ b/vortex-file/src/multi/session.rs @@ -3,10 +3,12 @@ //! Session extension for multi-file scanning, providing a shared footer cache. +use std::any::Any; use std::fmt; use std::fmt::Debug; use vortex_session::SessionExt; +use vortex_session::SessionVar; use crate::footer::Footer; @@ -61,6 +63,16 @@ impl MultiFileSession { } } +impl SessionVar for MultiFileSession { + fn as_any(&self) -> &dyn Any { + self + } + + fn as_any_mut(&mut self) -> &mut dyn Any { + self + } +} + /// Extension trait for accessing the [`MultiFileSession`] from a session. pub(super) trait MultiFileSessionExt: SessionExt { /// Returns a reference to the [`MultiFileSession`] state. diff --git a/vortex-file/src/strategy.rs b/vortex-file/src/strategy.rs index 544d2b03470..ea2add5ef13 100644 --- a/vortex-file/src/strategy.rs +++ b/vortex-file/src/strategy.rs @@ -3,6 +3,7 @@ //! This module defines the default layout strategy for a Vortex file. +use std::any::Any; use std::sync::Arc; use std::sync::LazyLock; @@ -26,6 +27,7 @@ use vortex_array::arrays::Primitive; use vortex_array::arrays::Struct; use vortex_array::arrays::VarBin; use vortex_array::arrays::VarBinView; +use vortex_array::arrays::patched::use_experimental_patches; use vortex_array::dtype::FieldPath; use vortex_btrblocks::BtrBlocksCompressorBuilder; use vortex_btrblocks::SchemeExt; @@ -54,9 +56,8 @@ use vortex_layout::layouts::zoned::writer::ZonedStrategy; use vortex_pco::Pco; use vortex_runend::RunEnd; use vortex_sequence::Sequence; +use vortex_session::SessionVar; use vortex_sparse::Sparse; -#[cfg(feature = "unstable_encodings")] -use vortex_tensor::encodings::turboquant::TurboQuant; use vortex_utils::aliases::hash_map::HashMap; use vortex_utils::aliases::hash_set::HashSet; use vortex_zigzag::ZigZag; @@ -91,10 +92,6 @@ pub static ALLOWED_ENCODINGS: LazyLock> = LazyLock::new(|| { allowed.insert(Masked.id()); allowed.insert(Dict.id()); - if *vortex_fastlanes::USE_EXPERIMENTAL_PATCHES { - allowed.insert(Patched.id()); - } - // Compressed encodings from encoding crates allowed.insert(ALP.id()); allowed.insert(ALPRD.id()); @@ -110,10 +107,14 @@ pub static ALLOWED_ENCODINGS: LazyLock> = LazyLock::new(|| { allowed.insert(RunEnd.id()); allowed.insert(Sequence.id()); allowed.insert(Sparse.id()); - #[cfg(feature = "unstable_encodings")] - allowed.insert(TurboQuant.id()); allowed.insert(ZigZag.id()); + // Experimental encodings + + if use_experimental_patches() { + allowed.insert(Patched.id()); + } + #[cfg(feature = "zstd")] allowed.insert(Zstd.id()); #[cfg(all(feature = "zstd", feature = "unstable_encodings"))] @@ -170,6 +171,16 @@ impl std::fmt::Debug for WriteStrategyBuilder { } } +impl SessionVar for WriteStrategyBuilder { + fn as_any(&self) -> &dyn Any { + self + } + + fn as_any_mut(&mut self) -> &mut dyn Any { + self + } +} + impl Default for WriteStrategyBuilder { /// Create a new empty builder. It can be further configured, /// and then finally built yielding the [`LayoutStrategy`]. diff --git a/vortex-file/src/tests.rs b/vortex-file/src/tests.rs index 84840b2f5bf..2ba10d96684 100644 --- a/vortex-file/src/tests.rs +++ b/vortex-file/src/tests.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::cast_possible_truncation)] +#![expect(clippy::cast_possible_truncation)] use std::iter; use std::sync::Arc; use std::sync::LazyLock; @@ -12,7 +12,7 @@ use futures::TryStreamExt; use futures::pin_mut; use vortex_array::ArrayRef; use vortex_array::IntoArray; -use vortex_array::ToCanonical; +use vortex_array::VortexSessionExecute; use vortex_array::accessor::ArrayAccessor; use vortex_array::arrays::ChunkedArray; use vortex_array::arrays::ConstantArray; @@ -262,6 +262,7 @@ async fn test_read_simple_with_spawn() { #[tokio::test] #[cfg_attr(miri, ignore)] async fn test_read_projection() { + let mut ctx = SESSION.create_execution_ctx(); let strings_expected = ["ab", "foo", "bar", "baz", "ab", "foo", "bar", "baz"]; let strings = ChunkedArray::from_iter([ VarBinArray::from(strings_expected[..4].to_vec()).into_array(), @@ -306,7 +307,11 @@ async fn test_read_projection() { ) ); - let actual = array.to_struct().unmasked_field(0).clone(); + let actual = array + .execute::(&mut ctx) + .unwrap() + .unmasked_field(0) + .clone(); let expected = VarBinArray::from(strings_expected.to_vec()).into_array(); assert_arrays_eq!(actual, expected); @@ -328,7 +333,11 @@ async fn test_read_projection() { ) ); - let actual = array.to_struct().unmasked_field(0).clone(); + let actual = array + .execute::(&mut ctx) + .unwrap() + .unmasked_field(0) + .clone(); let expected = Buffer::copy_from(numbers_expected).into_array(); assert_arrays_eq!(actual, expected); } @@ -336,6 +345,7 @@ async fn test_read_projection() { #[tokio::test] #[cfg_attr(miri, ignore)] async fn unequal_batches() { + let mut ctx = SESSION.create_execution_ctx(); let strings = ChunkedArray::from_iter([ VarBinArray::from(vec!["ab", "foo", "bar", "bob"]).into_array(), VarBinArray::from(vec!["baz", "ab", "foo", "bar", "baz", "alice"]).into_array(), @@ -373,10 +383,13 @@ async fn unequal_batches() { item_count += array.len(); let numbers = array - .to_struct() + .execute::(&mut ctx) + .unwrap() .unmasked_field_by_name("numbers") .unwrap() - .to_primitive(); + .clone() + .execute::(&mut ctx) + .unwrap(); assert_eq!(numbers.ptype(), PType::U32); } assert_eq!(item_count, 10); @@ -505,6 +518,7 @@ async fn issue_5385_filter_casted_column() { #[tokio::test] #[cfg_attr(miri, ignore)] async fn filter_string() { + let mut ctx = SESSION.create_execution_ctx(); let names_orig = VarBinArray::from_iter( vec![Some("Joseph"), None, Some("Angela"), Some("Mikhail"), None], DType::Utf8(Nullability::Nullable), @@ -541,13 +555,23 @@ async fn filter_string() { .unwrap(); assert_eq!(result.len(), 1); - let names_actual = result[0].to_struct().unmasked_field(0).clone(); + let names_actual = result[0] + .clone() + .execute::(&mut ctx) + .unwrap() + .unmasked_field(0) + .clone(); let names_expected = VarBinArray::from_iter(vec![Some("Joseph")], DType::Utf8(Nullability::Nullable)) .into_array(); assert_arrays_eq!(names_actual, names_expected); - let ages_actual = result[0].to_struct().unmasked_field(1).clone(); + let ages_actual = result[0] + .clone() + .execute::(&mut ctx) + .unwrap() + .unmasked_field(1) + .clone(); let ages_expected = PrimitiveArray::from_option_iter([Some(25i32)]).into_array(); assert_arrays_eq!(ages_actual, ages_expected); } @@ -555,6 +579,7 @@ async fn filter_string() { #[tokio::test] #[cfg_attr(miri, ignore)] async fn filter_or() { + let mut ctx = SESSION.create_execution_ctx(); let names = VarBinArray::from_iter( vec![Some("Joseph"), None, Some("Angela"), Some("Mikhail"), None], DType::Utf8(Nullability::Nullable), @@ -596,7 +621,12 @@ async fn filter_or() { .unwrap(); assert_eq!(result.len(), 1); - let names_actual = result[0].to_struct().unmasked_field(0).clone(); + let names_actual = result[0] + .clone() + .execute::(&mut ctx) + .unwrap() + .unmasked_field(0) + .clone(); let names_expected = VarBinArray::from_iter( vec![Some("Joseph"), Some("Angela")], DType::Utf8(Nullability::Nullable), @@ -604,7 +634,12 @@ async fn filter_or() { .into_array(); assert_arrays_eq!(names_actual, names_expected); - let ages_actual = result[0].to_struct().unmasked_field(1).clone(); + let ages_actual = result[0] + .clone() + .execute::(&mut ctx) + .unwrap() + .unmasked_field(1) + .clone(); let ages_expected = PrimitiveArray::from_option_iter([Some(25i32), None]).into_array(); assert_arrays_eq!(ages_actual, ages_expected); } @@ -612,6 +647,7 @@ async fn filter_or() { #[tokio::test] #[cfg_attr(miri, ignore)] async fn filter_and() { + let mut ctx = SESSION.create_execution_ctx(); let names = VarBinArray::from_iter( vec![Some("Joseph"), None, Some("Angela"), Some("Mikhail"), None], DType::Utf8(Nullability::Nullable), @@ -650,7 +686,12 @@ async fn filter_and() { .unwrap(); assert_eq!(result.len(), 1); - let names_actual = result[0].to_struct().unmasked_field(0).clone(); + let names_actual = result[0] + .clone() + .execute::(&mut ctx) + .unwrap() + .unmasked_field(0) + .clone(); let names_expected = VarBinArray::from_iter( vec![Some("Joseph"), None], DType::Utf8(Nullability::Nullable), @@ -658,7 +699,12 @@ async fn filter_and() { .into_array(); assert_arrays_eq!(names_actual, names_expected); - let ages_actual = result[0].to_struct().unmasked_field(1).clone(); + let ages_actual = result[0] + .clone() + .execute::(&mut ctx) + .unwrap() + .unmasked_field(1) + .clone(); let ages_expected = PrimitiveArray::from_option_iter([Some(25i32), Some(31i32)]).into_array(); assert_arrays_eq!(ages_actual, ages_expected); } @@ -666,6 +712,7 @@ async fn filter_and() { #[tokio::test] #[cfg_attr(miri, ignore)] async fn test_with_indices_simple() { + let mut ctx = SESSION.create_execution_ctx(); let expected_numbers_split: Vec> = (0..5).map(|_| (0_i16..100).collect()).collect(); let expected_array = StructArray::from_fields(&[( "numbers", @@ -699,7 +746,8 @@ async fn test_with_indices_simple() { .read_all() .await .unwrap() - .to_struct(); + .execute::(&mut ctx) + .unwrap(); assert_eq!(actual_kept_array.len(), 0); @@ -715,8 +763,13 @@ async fn test_with_indices_simple() { .read_all() .await .unwrap() - .to_struct(); - let actual_kept_numbers_array = actual_kept_array.unmasked_field(0).to_primitive(); + .execute::(&mut ctx) + .unwrap(); + let actual_kept_numbers_array = actual_kept_array + .unmasked_field(0) + .clone() + .execute::(&mut ctx) + .unwrap(); let expected_kept_numbers: Vec = kept_indices .iter() @@ -735,7 +788,8 @@ async fn test_with_indices_simple() { .read_all() .await .unwrap() - .to_struct(); + .execute::(&mut ctx) + .unwrap(); let actual_numbers_array = actual_array.unmasked_field(0).clone(); let expected_array = Buffer::copy_from(&expected_numbers).into_array(); assert_arrays_eq!(actual_numbers_array, expected_array); @@ -744,6 +798,7 @@ async fn test_with_indices_simple() { #[tokio::test] #[cfg_attr(miri, ignore)] async fn test_with_indices_on_two_columns() { + let mut ctx = SESSION.create_execution_ctx(); let strings_expected = ["ab", "foo", "bar", "baz", "ab", "foo", "bar", "baz"]; let strings = ChunkedArray::from_iter([ VarBinArray::from(strings_expected[..4].to_vec()).into_array(), @@ -778,7 +833,8 @@ async fn test_with_indices_on_two_columns() { .read_all() .await .unwrap() - .to_struct(); + .execute::(&mut ctx) + .unwrap(); let strings_actual = array.unmasked_field(0).clone(); let strings_expected_vec: Vec<&str> = kept_indices @@ -800,6 +856,7 @@ async fn test_with_indices_on_two_columns() { #[tokio::test] #[cfg_attr(miri, ignore)] async fn test_with_indices_and_with_row_filter_simple() { + let mut ctx = SESSION.create_execution_ctx(); let expected_numbers_split: Vec> = (0..5).map(|_| (0_i16..100).collect()).collect(); let expected_array = StructArray::from_fields(&[( "numbers", @@ -833,7 +890,8 @@ async fn test_with_indices_and_with_row_filter_simple() { .read_all() .await .unwrap() - .to_struct(); + .execute::(&mut ctx) + .unwrap(); assert_eq!(actual_kept_array.len(), 0); @@ -850,9 +908,14 @@ async fn test_with_indices_and_with_row_filter_simple() { .read_all() .await .unwrap() - .to_struct(); + .execute::(&mut ctx) + .unwrap(); - let actual_kept_numbers_array = actual_kept_array.unmasked_field(0).to_primitive(); + let actual_kept_numbers_array = actual_kept_array + .unmasked_field(0) + .clone() + .execute::(&mut ctx) + .unwrap(); let expected_kept_numbers: Buffer = kept_indices .iter() @@ -873,7 +936,8 @@ async fn test_with_indices_and_with_row_filter_simple() { .read_all() .await .unwrap() - .to_struct(); + .execute::(&mut ctx) + .unwrap(); let actual_numbers_array = actual_array.unmasked_field(0).clone(); let expected_filtered: Buffer = expected_numbers @@ -888,6 +952,7 @@ async fn test_with_indices_and_with_row_filter_simple() { #[tokio::test] #[cfg_attr(miri, ignore)] async fn filter_string_chunked() { + let mut ctx = SESSION.create_execution_ctx(); let name_chunk1 = VarBinViewArray::from_iter_nullable_str([Some("Joseph"), Some("James"), Some("Angela")]) .into_array(); @@ -932,7 +997,8 @@ async fn filter_string_chunked() { .read_all() .await .unwrap() - .to_struct(); + .execute::(&mut ctx) + .unwrap(); assert_eq!(actual_array.len(), 1); let names_actual = actual_array.unmasked_field(0).clone(); @@ -949,6 +1015,7 @@ async fn filter_string_chunked() { #[tokio::test] #[cfg_attr(miri, ignore)] async fn test_pruning_with_or() { + let mut ctx = SESSION.create_execution_ctx(); let letter_chunk1 = VarBinViewArray::from_iter_nullable_str([ Some("A".to_owned()), Some("B".to_owned()), @@ -1023,7 +1090,8 @@ async fn test_pruning_with_or() { .read_all() .await .unwrap() - .to_struct(); + .execute::(&mut ctx) + .unwrap(); assert_eq!(actual_array.len(), 10); let letters_actual = actual_array.unmasked_field(0).clone(); @@ -1061,6 +1129,7 @@ async fn test_pruning_with_or() { #[tokio::test] async fn test_repeated_projection() { + let mut ctx = SESSION.create_execution_ctx(); let strings = ChunkedArray::from_iter([ VarBinArray::from(vec!["ab", "foo", "bar", "baz"]).into_array(), VarBinArray::from(vec!["ab", "foo", "bar", "baz"]).into_array(), @@ -1093,7 +1162,8 @@ async fn test_repeated_projection() { .read_all() .await .unwrap() - .to_struct(); + .execute::(&mut ctx) + .unwrap(); assert_arrays_eq!(actual, expected); } @@ -1240,16 +1310,22 @@ async fn write_nullable_nested_struct() -> VortexResult<()> { )? .into_array(); - let result = round_trip(&array, Ok).await?.to_struct(); + let mut ctx = SESSION.create_execution_ctx(); + let result = round_trip(&array, Ok) + .await? + .execute::(&mut ctx)?; assert_eq!(result.len(), 3); assert_eq!(result.struct_fields().nfields(), 1); - assert!(result.all_valid()?); + assert!(result.all_valid(&mut ctx)?); - let nested_struct = result.unmasked_field_by_name("struct")?.to_struct(); + let nested_struct = result + .unmasked_field_by_name("struct")? + .clone() + .execute::(&mut ctx)?; assert_eq!(nested_struct.dtype(), &nested_dtype); assert_eq!(nested_struct.len(), 3); - assert!(nested_struct.all_invalid()?); + assert!(nested_struct.all_invalid(&mut ctx)?); Ok(()) } @@ -1356,6 +1432,7 @@ async fn test_writer_basic_push() -> VortexResult<()> { #[tokio::test] async fn test_writer_multiple_pushes() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); let chunk1 = StructArray::from_fields(&[("numbers", buffer![1u32, 2, 3].into_array())])?.into_array(); let chunk2 = @@ -1380,7 +1457,7 @@ async fn test_writer_multiple_pushes() -> VortexResult<()> { assert_eq!(result.len(), 9); let numbers = result - .to_struct() + .execute::(&mut ctx)? .unmasked_field_by_name("numbers")? .clone(); let expected = buffer![1u32, 2, 3, 4, 5, 6, 7, 8, 9].into_array(); @@ -1391,6 +1468,7 @@ async fn test_writer_multiple_pushes() -> VortexResult<()> { #[tokio::test] async fn test_writer_push_stream() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); let chunk1 = StructArray::from_fields(&[("numbers", buffer![1u32, 2, 3].into_array())])?.into_array(); let chunk2 = @@ -1414,7 +1492,7 @@ async fn test_writer_push_stream() -> VortexResult<()> { assert_eq!(result.len(), 6); let numbers = result - .to_struct() + .execute::(&mut ctx)? .unmasked_field_by_name("numbers")? .clone(); let expected = buffer![1u32, 2, 3, 4, 5, 6].into_array(); @@ -1453,6 +1531,7 @@ async fn test_writer_bytes_written() -> VortexResult<()> { #[tokio::test] async fn test_writer_empty_chunks() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); let empty = StructArray::from_fields(&[( "numbers", PrimitiveArray::new::(buffer![], Validity::NonNullable).into_array(), @@ -1478,7 +1557,7 @@ async fn test_writer_empty_chunks() -> VortexResult<()> { assert_eq!(result.len(), 2); let numbers = result - .to_struct() + .execute::(&mut ctx)? .unmasked_field_by_name("numbers")? .clone(); let expected = buffer![1u32, 2].into_array(); @@ -1489,6 +1568,7 @@ async fn test_writer_empty_chunks() -> VortexResult<()> { #[tokio::test] async fn test_writer_mixed_push_and_stream() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); let chunk1 = StructArray::from_fields(&[("numbers", buffer![1u32, 2].into_array())])?.into_array(); let chunk2 = @@ -1516,7 +1596,7 @@ async fn test_writer_mixed_push_and_stream() -> VortexResult<()> { assert_eq!(result.len(), 6); let numbers = result - .to_struct() + .execute::(&mut ctx)? .unmasked_field_by_name("numbers")? .clone(); let expected = buffer![1u32, 2, 3, 4, 5, 6].into_array(); @@ -1527,6 +1607,7 @@ async fn test_writer_mixed_push_and_stream() -> VortexResult<()> { #[tokio::test] async fn test_writer_with_complex_types() -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); let strings = VarBinArray::from(vec!["hello", "world", "test"]).into_array(); let numbers = buffer![100i32, 200, 300].into_array(); let lists = ListArray::from_iter_slow::( @@ -1558,13 +1639,15 @@ async fn test_writer_with_complex_types() -> VortexResult<()> { assert_eq!(result.dtype(), &dtype); let strings_field = result - .to_struct() + .execute::(&mut ctx)? .unmasked_field_by_name("strings") .cloned()?; - let strings = strings_field.to_varbinview().with_iterator(|iter| { - iter.map(|s| s.map(|st| unsafe { String::from_utf8_unchecked(st.to_vec()) })) - .collect::>() - }); + let strings = strings_field + .execute::(&mut ctx)? + .with_iterator(|iter| { + iter.map(|s| s.map(|st| unsafe { String::from_utf8_unchecked(st.to_vec()) })) + .collect::>() + }); assert_eq!( strings, vec![ diff --git a/vortex-file/src/v2/file_stats_reader.rs b/vortex-file/src/v2/file_stats_reader.rs index fb31b28d61f..f197fea32b6 100644 --- a/vortex-file/src/v2/file_stats_reader.rs +++ b/vortex-file/src/v2/file_stats_reader.rs @@ -15,6 +15,7 @@ use vortex_array::Canonical; use vortex_array::IntoArray; use vortex_array::MaskFuture; use vortex_array::VortexSessionExecute; +use vortex_array::arrays::ConstantArray; use vortex_array::arrays::NullArray; use vortex_array::dtype::DType; use vortex_array::dtype::FieldMask; @@ -26,6 +27,7 @@ use vortex_array::expr::lit; use vortex_array::expr::stats::Stat; use vortex_array::scalar::Scalar; use vortex_array::scalar_fn::fns::literal::Literal; +use vortex_array::scalar_fn::internal::row_count::substitute_row_count; use vortex_error::VortexResult; use vortex_layout::ArrayFuture; use vortex_layout::LayoutReader; @@ -75,18 +77,19 @@ impl FileStatsLayoutReader { } } - /// Evaluates whether the file can be fully pruned for the given expression. + /// Evaluates whether file-level statistics prove `expr` cannot match. /// - /// Returns `true` if file-level stats prove no rows can match, `false` otherwise. + /// Row-count placeholders are resolved against the full file row count, + /// independent of the requested row range. fn evaluate_file_stats(&self, expr: &Expression) -> VortexResult { let Some(pruning_expr) = expr.stat_falsification(self) else { // If there is no pruning expression, we can't prune. return Ok(false); }; - // Given how we implemented the StatsCatalog, we know the expression must be all literals. - // We can therefore optimize with a null scope since there are no field references that - // need to be resolved. + // Given how we implemented the StatsCatalog, we know the expression must be all literals + // or row_count placeholders. We can therefore optimize with a null scope since there are + // no field references that need to be resolved. let simplified = pruning_expr.optimize_recursive(&DType::Null)?; if let Some(result) = simplified.as_opt::() { // Can prune if the result is non-nullable and true @@ -94,18 +97,26 @@ impl FileStatsLayoutReader { } // Sometimes expressions don't implement constant folding to literals... In this case, - // we just execute the expression over a null array. + // we apply the expression over a null array and substitute any row_count placeholders + // in the resulting array tree with the file's row count. let pruning = NullArray::new(1).into_array().apply(&pruning_expr)?; + let row_count_replacement = + ConstantArray::new(self.child.row_count(), pruning.len()).into_array(); + let pruning = substitute_row_count(pruning, &row_count_replacement)?; let mut ctx = self.session.create_execution_ctx(); let result = pruning .execute::(&mut ctx)? .into_bool() .into_array() - .scalar_at(0)?; + .execute_scalar(0, &mut ctx)?; Ok(result.as_bool().value() == Some(true)) } + + pub fn file_stats(&self) -> &FileStatistics { + &self.file_stats + } } /// Implements [`StatsCatalog`] to provide file-level stats to expressions during pruning evaluation. @@ -122,7 +133,8 @@ impl StatsCatalog for FileStatsLayoutReader { let stat_value = field_stats.get(stat)?.as_exact()?; let field_dtype = self.struct_fields.field_by_index(field_idx)?; - let stat_scalar = Scalar::try_new(field_dtype, Some(stat_value)).ok()?; + let stat_dtype = stat.dtype(&field_dtype)?; + let stat_scalar = Scalar::try_new(stat_dtype, Some(stat_value)).ok()?; Some(lit(stat_scalar)) } @@ -192,6 +204,10 @@ impl LayoutReader for FileStatsLayoutReader { ) -> VortexResult { self.child.projection_evaluation(row_range, expr, mask) } + + fn as_any(&self) -> &dyn std::any::Any { + self + } } #[cfg(test)] @@ -201,16 +217,21 @@ mod tests { use vortex_array::ArrayContext; use vortex_array::IntoArray as _; + use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::StructArray; + use vortex_array::arrays::datetime::TemporalData; use vortex_array::dtype::DType; use vortex_array::dtype::Nullability; use vortex_array::dtype::PType; use vortex_array::expr::get_item; use vortex_array::expr::gt; + use vortex_array::expr::is_not_null; + use vortex_array::expr::is_null; use vortex_array::expr::lit; use vortex_array::expr::root; use vortex_array::expr::stats::Precision; use vortex_array::expr::stats::Stat; + use vortex_array::extension::datetime::TimeUnit; use vortex_array::scalar::ScalarValue; use vortex_array::scalar_fn::session::ScalarFnSession; use vortex_array::session::ArraySession; @@ -224,6 +245,7 @@ mod tests { use vortex_layout::LayoutStrategy; use vortex_layout::layouts::flat::writer::FlatLayoutStrategy; use vortex_layout::layouts::table::TableStrategy; + use vortex_layout::segments::SegmentSink; use vortex_layout::segments::TestSegments; use vortex_layout::sequence::SequenceId; use vortex_layout::sequence::SequentialArrayStreamExt; @@ -251,6 +273,18 @@ mod tests { ) } + fn test_file_null_count_stats(null_count: u64) -> FileStatistics { + let mut stats = StatsSet::default(); + stats.set( + Stat::NullCount, + Precision::exact(ScalarValue::from(null_count)), + ); + FileStatistics::new( + Arc::from([stats]), + Arc::from([DType::Primitive(PType::I32, Nullability::Nullable)]), + ) + } + #[test] fn pruning_when_filter_out_of_range() -> VortexResult<()> { block_on(|handle| async { @@ -329,4 +363,102 @@ mod tests { Ok(()) }) } + + /// Regression test: `IS NULL` on a nullable timestamp column must not fail with a + /// dtype mismatch. The bug was that `stats_ref` used the *field* dtype (timestamp) + /// for the `NullCount` stat scalar instead of the stat's own dtype (u64). + #[test] + fn is_null_pruning_on_nullable_timestamp_column() -> VortexResult<()> { + block_on(|handle| async { + let session = SESSION.clone().with_handle(handle); + let ctx = ArrayContext::empty(); + let segments = Arc::new(TestSegments::default()); + let (ptr, eof) = SequenceId::root().split(); + + // Build a struct with a nullable timestamp column containing some nulls. + let prim_array = + PrimitiveArray::from_option_iter([Some(1_000_000i64), None, Some(3_000_000)]) + .into_array(); + let ts_data = TemporalData::new_timestamp(prim_array, TimeUnit::Microseconds, None); + let ts_dtype = ts_data.dtype().clone(); + let ts_array = ts_data.into_array(); + + let struct_array = StructArray::from_fields([("deleted_at", ts_array)].as_slice())?; + + let strategy = TableStrategy::new( + Arc::new(FlatLayoutStrategy::default()), + Arc::new(FlatLayoutStrategy::default()), + ); + let layout = strategy + .write_stream( + ctx, + Arc::clone(&segments) as Arc, + struct_array.into_array().to_array_stream().sequenced(ptr), + eof, + &session, + ) + .await?; + + let child = layout.new_reader("".into(), segments, &SESSION)?; + + // File-level stats: 1 null in deleted_at. + let mut stats = StatsSet::default(); + stats.set(Stat::NullCount, Precision::exact(ScalarValue::from(1u64))); + let file_stats = FileStatistics::new(Arc::from([stats]), Arc::from([ts_dtype])); + + let reader = FileStatsLayoutReader::new(child, file_stats, SESSION.clone()); + + // `is_null(deleted_at)` — should NOT panic or error due to dtype mismatch. + let expr = is_null(get_item("deleted_at", root())); + let mask = Mask::new_true(3); + let result = reader.pruning_evaluation(&(0..3), &expr, mask)?.await?; + // null_count is 1 (non-zero), so is_null is not falsified => not pruned. + assert_eq!(result, Mask::new_true(3)); + + Ok(()) + }) + } + + #[test] + fn pruning_is_not_null_when_file_is_all_null() -> VortexResult<()> { + block_on(|handle| async { + let session = SESSION.clone().with_handle(handle); + let ctx = ArrayContext::empty(); + let segments = Arc::new(TestSegments::default()); + let (ptr, eof) = SequenceId::root().split(); + let struct_array = StructArray::from_fields( + [( + "col", + PrimitiveArray::from_option_iter([None::, None, None, None, None]) + .into_array(), + )] + .as_slice(), + )?; + let strategy = TableStrategy::new( + Arc::new(FlatLayoutStrategy::default()), + Arc::new(FlatLayoutStrategy::default()), + ); + let layout = strategy + .write_stream( + ctx, + Arc::clone(&segments) as Arc, + struct_array.into_array().to_array_stream().sequenced(ptr), + eof, + &session, + ) + .await?; + + let child = layout.new_reader("".into(), segments, &SESSION)?; + + let reader = + FileStatsLayoutReader::new(child, test_file_null_count_stats(5), SESSION.clone()); + + let expr = is_not_null(get_item("col", root())); + let mask = Mask::new_true(5); + let result = reader.pruning_evaluation(&(0..5), &expr, mask)?.await?; + assert_eq!(result, Mask::new_false(5)); + + Ok(()) + }) + } } diff --git a/vortex-file/src/writer.rs b/vortex-file/src/writer.rs index 63f30cd102e..0580c47d8fc 100644 --- a/vortex-file/src/writer.rs +++ b/vortex-file/src/writer.rs @@ -47,6 +47,7 @@ use vortex_session::SessionExt; use vortex_session::VortexSession; use vortex_session::registry::ReadContext; +use crate::ALLOWED_ENCODINGS; use crate::Footer; use crate::MAGIC_BYTES; use crate::WriteStrategyBuilder; @@ -157,7 +158,7 @@ impl VortexWriteOptions { let arrays = self.session.arrays(); arrays.registry().clone() }; - let ctx = ArrayContext::new(registry.ids().sorted().collect()) + let ctx = ArrayContext::new(ALLOWED_ENCODINGS.iter().cloned().sorted().collect()) // Configure a registry just to ensure only known encodings are interned. .with_registry(registry); let dtype = stream.dtype().clone(); @@ -175,6 +176,7 @@ impl VortexWriteOptions { stream, self.file_statistics.clone().into(), self.max_variable_length_statistics_size, + &self.session, ); // First, write the magic bytes. diff --git a/vortex-file/tests/test_write_table.rs b/vortex-file/tests/test_write_table.rs index d0522c5a78c..8b051250e2e 100644 --- a/vortex-file/tests/test_write_table.rs +++ b/vortex-file/tests/test_write_table.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::tests_outside_test_module)] +#![expect(clippy::tests_outside_test_module)] use std::sync::Arc; use std::sync::LazyLock; @@ -9,7 +9,7 @@ use std::sync::LazyLock; use futures::StreamExt; use futures::pin_mut; use vortex_array::IntoArray; -use vortex_array::ToCanonical; +use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; use vortex_array::arrays::DictArray; use vortex_array::arrays::ListViewArray; @@ -46,6 +46,7 @@ static SESSION: LazyLock = LazyLock::new(|| { #[tokio::test] async fn test_file_roundtrip() { + let mut ctx = SESSION.create_execution_ctx(); // Create a simple roundtrip let nums = PrimitiveArray::from_iter((0..1024).cycle().take(16_384)).into_array(); @@ -102,8 +103,13 @@ async fn test_file_roundtrip() { while let Some(next) = stream.next().await { let next = next.expect("next"); - let next = next.to_struct(); - let a = next.unmasked_field_by_name("a").unwrap().to_struct(); + let next = next.execute::(&mut ctx).unwrap(); + let a = next + .unmasked_field_by_name("a") + .unwrap() + .clone() + .execute::(&mut ctx) + .unwrap(); let b = next.unmasked_field_by_name("b").unwrap(); let raw = a.unmasked_field_by_name("raw").unwrap(); diff --git a/vortex-flatbuffers/public-api.lock b/vortex-flatbuffers/public-api.lock index 18d0dae07cf..4e96887be42 100644 --- a/vortex-flatbuffers/public-api.lock +++ b/vortex-flatbuffers/public-api.lock @@ -20,19 +20,19 @@ pub const vortex_flatbuffers::array::Array<'a>::VT_ROOT: flatbuffers::primitives pub fn vortex_flatbuffers::array::Array<'a>::buffers(&self) -> core::option::Option> -pub fn vortex_flatbuffers::array::Array<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, args: &'args vortex_flatbuffers::array::ArrayArgs<'args>) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::array::Array<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::array::ArrayArgs<'args>) -> flatbuffers::primitives::WIPOffset> -pub unsafe fn vortex_flatbuffers::array::Array<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::array::Array<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self pub fn vortex_flatbuffers::array::Array<'a>::root(&self) -> core::option::Option> impl core::fmt::Debug for vortex_flatbuffers::array::Array<'_> -pub fn vortex_flatbuffers::array::Array<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::array::Array<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::array::Array<'_> -pub fn vortex_flatbuffers::array::Array<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::array::Array<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::array::Array<'a> @@ -40,7 +40,7 @@ pub fn vortex_flatbuffers::array::Array<'a>::clone(&self) -> vortex_flatbuffers: impl<'a> core::cmp::PartialEq for vortex_flatbuffers::array::Array<'a> -pub fn vortex_flatbuffers::array::Array<'a>::eq(&self, other: &vortex_flatbuffers::array::Array<'a>) -> bool +pub fn vortex_flatbuffers::array::Array<'a>::eq(&self, &vortex_flatbuffers::array::Array<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::array::Array<'a> @@ -50,7 +50,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::array::Array<'a pub type vortex_flatbuffers::array::Array<'a>::Inner = vortex_flatbuffers::array::Array<'a> -pub unsafe fn vortex_flatbuffers::array::Array<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::array::Array<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::array::ArrayArgs<'a> @@ -66,13 +66,13 @@ pub struct vortex_flatbuffers::array::ArrayBuilder<'a: 'b, 'b, A: flatbuffers::b impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::array::ArrayBuilder<'a, 'b, A> -pub fn vortex_flatbuffers::array::ArrayBuilder<'a, 'b, A>::add_buffers(&mut self, buffers: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::array::ArrayBuilder<'a, 'b, A>::add_buffers(&mut self, flatbuffers::primitives::WIPOffset>) -pub fn vortex_flatbuffers::array::ArrayBuilder<'a, 'b, A>::add_root(&mut self, root: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::array::ArrayBuilder<'a, 'b, A>::add_root(&mut self, flatbuffers::primitives::WIPOffset>) pub fn vortex_flatbuffers::array::ArrayBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::array::ArrayBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::array::ArrayBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::array::ArrayBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::array::ArrayBuilder<'a, 'b, A> pub struct vortex_flatbuffers::array::ArrayNode<'a> @@ -94,11 +94,11 @@ pub fn vortex_flatbuffers::array::ArrayNode<'a>::buffers(&self) -> core::option: pub fn vortex_flatbuffers::array::ArrayNode<'a>::children(&self) -> core::option::Option>>> -pub fn vortex_flatbuffers::array::ArrayNode<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, args: &'args vortex_flatbuffers::array::ArrayNodeArgs<'args>) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::array::ArrayNode<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::array::ArrayNodeArgs<'args>) -> flatbuffers::primitives::WIPOffset> pub fn vortex_flatbuffers::array::ArrayNode<'a>::encoding(&self) -> u16 -pub unsafe fn vortex_flatbuffers::array::ArrayNode<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::array::ArrayNode<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self pub fn vortex_flatbuffers::array::ArrayNode<'a>::metadata(&self) -> core::option::Option> @@ -106,11 +106,11 @@ pub fn vortex_flatbuffers::array::ArrayNode<'a>::stats(&self) -> core::option::O impl core::fmt::Debug for vortex_flatbuffers::array::ArrayNode<'_> -pub fn vortex_flatbuffers::array::ArrayNode<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::array::ArrayNode<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::array::ArrayNode<'_> -pub fn vortex_flatbuffers::array::ArrayNode<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::array::ArrayNode<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::array::ArrayNode<'a> @@ -118,7 +118,7 @@ pub fn vortex_flatbuffers::array::ArrayNode<'a>::clone(&self) -> vortex_flatbuff impl<'a> core::cmp::PartialEq for vortex_flatbuffers::array::ArrayNode<'a> -pub fn vortex_flatbuffers::array::ArrayNode<'a>::eq(&self, other: &vortex_flatbuffers::array::ArrayNode<'a>) -> bool +pub fn vortex_flatbuffers::array::ArrayNode<'a>::eq(&self, &vortex_flatbuffers::array::ArrayNode<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::array::ArrayNode<'a> @@ -128,7 +128,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::array::ArrayNod pub type vortex_flatbuffers::array::ArrayNode<'a>::Inner = vortex_flatbuffers::array::ArrayNode<'a> -pub unsafe fn vortex_flatbuffers::array::ArrayNode<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::array::ArrayNode<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::array::ArrayNodeArgs<'a> @@ -150,19 +150,19 @@ pub struct vortex_flatbuffers::array::ArrayNodeBuilder<'a: 'b, 'b, A: flatbuffer impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::array::ArrayNodeBuilder<'a, 'b, A> -pub fn vortex_flatbuffers::array::ArrayNodeBuilder<'a, 'b, A>::add_buffers(&mut self, buffers: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::array::ArrayNodeBuilder<'a, 'b, A>::add_buffers(&mut self, flatbuffers::primitives::WIPOffset>) -pub fn vortex_flatbuffers::array::ArrayNodeBuilder<'a, 'b, A>::add_children(&mut self, children: flatbuffers::primitives::WIPOffset>>>) +pub fn vortex_flatbuffers::array::ArrayNodeBuilder<'a, 'b, A>::add_children(&mut self, flatbuffers::primitives::WIPOffset>>>) -pub fn vortex_flatbuffers::array::ArrayNodeBuilder<'a, 'b, A>::add_encoding(&mut self, encoding: u16) +pub fn vortex_flatbuffers::array::ArrayNodeBuilder<'a, 'b, A>::add_encoding(&mut self, u16) -pub fn vortex_flatbuffers::array::ArrayNodeBuilder<'a, 'b, A>::add_metadata(&mut self, metadata: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::array::ArrayNodeBuilder<'a, 'b, A>::add_metadata(&mut self, flatbuffers::primitives::WIPOffset>) -pub fn vortex_flatbuffers::array::ArrayNodeBuilder<'a, 'b, A>::add_stats(&mut self, stats: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::array::ArrayNodeBuilder<'a, 'b, A>::add_stats(&mut self, flatbuffers::primitives::WIPOffset>) pub fn vortex_flatbuffers::array::ArrayNodeBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::array::ArrayNodeBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::array::ArrayNodeBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::array::ArrayNodeBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::array::ArrayNodeBuilder<'a, 'b, A> pub struct vortex_flatbuffers::array::ArrayStats<'a> @@ -192,9 +192,9 @@ pub const vortex_flatbuffers::array::ArrayStats<'a>::VT_SUM: flatbuffers::primit pub const vortex_flatbuffers::array::ArrayStats<'a>::VT_UNCOMPRESSED_SIZE_IN_BYTES: flatbuffers::primitives::VOffsetT -pub fn vortex_flatbuffers::array::ArrayStats<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, args: &'args vortex_flatbuffers::array::ArrayStatsArgs<'args>) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::array::ArrayStats<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::array::ArrayStatsArgs<'args>) -> flatbuffers::primitives::WIPOffset> -pub unsafe fn vortex_flatbuffers::array::ArrayStats<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::array::ArrayStats<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self pub fn vortex_flatbuffers::array::ArrayStats<'a>::is_constant(&self) -> core::option::Option @@ -220,11 +220,11 @@ pub fn vortex_flatbuffers::array::ArrayStats<'a>::uncompressed_size_in_bytes(&se impl core::fmt::Debug for vortex_flatbuffers::array::ArrayStats<'_> -pub fn vortex_flatbuffers::array::ArrayStats<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::array::ArrayStats<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::array::ArrayStats<'_> -pub fn vortex_flatbuffers::array::ArrayStats<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::array::ArrayStats<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::array::ArrayStats<'a> @@ -232,7 +232,7 @@ pub fn vortex_flatbuffers::array::ArrayStats<'a>::clone(&self) -> vortex_flatbuf impl<'a> core::cmp::PartialEq for vortex_flatbuffers::array::ArrayStats<'a> -pub fn vortex_flatbuffers::array::ArrayStats<'a>::eq(&self, other: &vortex_flatbuffers::array::ArrayStats<'a>) -> bool +pub fn vortex_flatbuffers::array::ArrayStats<'a>::eq(&self, &vortex_flatbuffers::array::ArrayStats<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::array::ArrayStats<'a> @@ -242,7 +242,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::array::ArraySta pub type vortex_flatbuffers::array::ArrayStats<'a>::Inner = vortex_flatbuffers::array::ArrayStats<'a> -pub unsafe fn vortex_flatbuffers::array::ArrayStats<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::array::ArrayStats<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::array::ArrayStatsArgs<'a> @@ -276,31 +276,31 @@ pub struct vortex_flatbuffers::array::ArrayStatsBuilder<'a: 'b, 'b, A: flatbuffe impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A> -pub fn vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A>::add_is_constant(&mut self, is_constant: bool) +pub fn vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A>::add_is_constant(&mut self, bool) -pub fn vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A>::add_is_sorted(&mut self, is_sorted: bool) +pub fn vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A>::add_is_sorted(&mut self, bool) -pub fn vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A>::add_is_strict_sorted(&mut self, is_strict_sorted: bool) +pub fn vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A>::add_is_strict_sorted(&mut self, bool) -pub fn vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A>::add_max(&mut self, max: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A>::add_max(&mut self, flatbuffers::primitives::WIPOffset>) -pub fn vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A>::add_max_precision(&mut self, max_precision: vortex_flatbuffers::array::Precision) +pub fn vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A>::add_max_precision(&mut self, vortex_flatbuffers::array::Precision) -pub fn vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A>::add_min(&mut self, min: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A>::add_min(&mut self, flatbuffers::primitives::WIPOffset>) -pub fn vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A>::add_min_precision(&mut self, min_precision: vortex_flatbuffers::array::Precision) +pub fn vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A>::add_min_precision(&mut self, vortex_flatbuffers::array::Precision) -pub fn vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A>::add_nan_count(&mut self, nan_count: u64) +pub fn vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A>::add_nan_count(&mut self, u64) -pub fn vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A>::add_null_count(&mut self, null_count: u64) +pub fn vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A>::add_null_count(&mut self, u64) -pub fn vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A>::add_sum(&mut self, sum: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A>::add_sum(&mut self, flatbuffers::primitives::WIPOffset>) -pub fn vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A>::add_uncompressed_size_in_bytes(&mut self, uncompressed_size_in_bytes: u64) +pub fn vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A>::add_uncompressed_size_in_bytes(&mut self, u64) pub fn vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::array::ArrayStatsBuilder<'a, 'b, A> #[repr(transparent)] pub struct vortex_flatbuffers::array::Buffer(pub [u8; 8]) @@ -312,17 +312,17 @@ pub fn vortex_flatbuffers::array::Buffer::compression(&self) -> vortex_flatbuffe pub fn vortex_flatbuffers::array::Buffer::length(&self) -> u32 -pub fn vortex_flatbuffers::array::Buffer::new(padding: u16, alignment_exponent: u8, compression: vortex_flatbuffers::array::Compression, length: u32) -> Self +pub fn vortex_flatbuffers::array::Buffer::new(u16, u8, vortex_flatbuffers::array::Compression, u32) -> Self pub fn vortex_flatbuffers::array::Buffer::padding(&self) -> u16 -pub fn vortex_flatbuffers::array::Buffer::set_alignment_exponent(&mut self, x: u8) +pub fn vortex_flatbuffers::array::Buffer::set_alignment_exponent(&mut self, u8) -pub fn vortex_flatbuffers::array::Buffer::set_compression(&mut self, x: vortex_flatbuffers::array::Compression) +pub fn vortex_flatbuffers::array::Buffer::set_compression(&mut self, vortex_flatbuffers::array::Compression) -pub fn vortex_flatbuffers::array::Buffer::set_length(&mut self, x: u32) +pub fn vortex_flatbuffers::array::Buffer::set_length(&mut self, u32) -pub fn vortex_flatbuffers::array::Buffer::set_padding(&mut self, x: u16) +pub fn vortex_flatbuffers::array::Buffer::set_padding(&mut self, u16) impl core::clone::Clone for vortex_flatbuffers::array::Buffer @@ -330,7 +330,7 @@ pub fn vortex_flatbuffers::array::Buffer::clone(&self) -> vortex_flatbuffers::ar impl core::cmp::PartialEq for vortex_flatbuffers::array::Buffer -pub fn vortex_flatbuffers::array::Buffer::eq(&self, other: &vortex_flatbuffers::array::Buffer) -> bool +pub fn vortex_flatbuffers::array::Buffer::eq(&self, &vortex_flatbuffers::array::Buffer) -> bool impl core::default::Default for vortex_flatbuffers::array::Buffer @@ -338,7 +338,7 @@ pub fn vortex_flatbuffers::array::Buffer::default() -> Self impl core::fmt::Debug for vortex_flatbuffers::array::Buffer -pub fn vortex_flatbuffers::array::Buffer::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::array::Buffer::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_flatbuffers::array::Buffer @@ -350,17 +350,17 @@ impl<'a> flatbuffers::follow::Follow<'a> for &'a vortex_flatbuffers::array::Buff pub type &'a vortex_flatbuffers::array::Buffer::Inner = &'a vortex_flatbuffers::array::Buffer -pub unsafe fn &'a vortex_flatbuffers::array::Buffer::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn &'a vortex_flatbuffers::array::Buffer::follow(&'a [u8], usize) -> Self::Inner impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::array::Buffer pub type vortex_flatbuffers::array::Buffer::Inner = &'a vortex_flatbuffers::array::Buffer -pub unsafe fn vortex_flatbuffers::array::Buffer::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::array::Buffer::follow(&'a [u8], usize) -> Self::Inner impl<'a> flatbuffers::verifier::Verifiable for vortex_flatbuffers::array::Buffer -pub fn vortex_flatbuffers::array::Buffer::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::array::Buffer::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'b> flatbuffers::push::Push for vortex_flatbuffers::array::Buffer @@ -368,7 +368,7 @@ pub type vortex_flatbuffers::array::Buffer::Output = vortex_flatbuffers::array:: pub fn vortex_flatbuffers::array::Buffer::alignment() -> flatbuffers::push::PushAlignment -pub unsafe fn vortex_flatbuffers::array::Buffer::push(&self, dst: &mut [u8], _written_len: usize) +pub unsafe fn vortex_flatbuffers::array::Buffer::push(&self, &mut [u8], usize) #[repr(transparent)] pub struct vortex_flatbuffers::array::Compression(pub u8) @@ -394,15 +394,15 @@ impl core::cmp::Eq for vortex_flatbuffers::array::Compression impl core::cmp::Ord for vortex_flatbuffers::array::Compression -pub fn vortex_flatbuffers::array::Compression::cmp(&self, other: &vortex_flatbuffers::array::Compression) -> core::cmp::Ordering +pub fn vortex_flatbuffers::array::Compression::cmp(&self, &vortex_flatbuffers::array::Compression) -> core::cmp::Ordering impl core::cmp::PartialEq for vortex_flatbuffers::array::Compression -pub fn vortex_flatbuffers::array::Compression::eq(&self, other: &vortex_flatbuffers::array::Compression) -> bool +pub fn vortex_flatbuffers::array::Compression::eq(&self, &vortex_flatbuffers::array::Compression) -> bool impl core::cmp::PartialOrd for vortex_flatbuffers::array::Compression -pub fn vortex_flatbuffers::array::Compression::partial_cmp(&self, other: &vortex_flatbuffers::array::Compression) -> core::option::Option +pub fn vortex_flatbuffers::array::Compression::partial_cmp(&self, &vortex_flatbuffers::array::Compression) -> core::option::Option impl core::default::Default for vortex_flatbuffers::array::Compression @@ -410,11 +410,11 @@ pub fn vortex_flatbuffers::array::Compression::default() -> vortex_flatbuffers:: impl core::fmt::Debug for vortex_flatbuffers::array::Compression -pub fn vortex_flatbuffers::array::Compression::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::array::Compression::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_flatbuffers::array::Compression -pub fn vortex_flatbuffers::array::Compression::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_flatbuffers::array::Compression::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_flatbuffers::array::Compression @@ -424,7 +424,7 @@ impl flatbuffers::endian_scalar::EndianScalar for vortex_flatbuffers::array::Com pub type vortex_flatbuffers::array::Compression::Scalar = u8 -pub fn vortex_flatbuffers::array::Compression::from_little_endian(v: u8) -> Self +pub fn vortex_flatbuffers::array::Compression::from_little_endian(u8) -> Self pub fn vortex_flatbuffers::array::Compression::to_little_endian(self) -> u8 @@ -432,7 +432,7 @@ impl flatbuffers::push::Push for vortex_flatbuffers::array::Compression pub type vortex_flatbuffers::array::Compression::Output = vortex_flatbuffers::array::Compression -pub unsafe fn vortex_flatbuffers::array::Compression::push(&self, dst: &mut [u8], _written_len: usize) +pub unsafe fn vortex_flatbuffers::array::Compression::push(&self, &mut [u8], usize) impl flatbuffers::verifier::SimpleToVerifyInSlice for vortex_flatbuffers::array::Compression @@ -440,11 +440,11 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::array::Compress pub type vortex_flatbuffers::array::Compression::Inner = vortex_flatbuffers::array::Compression -pub unsafe fn vortex_flatbuffers::array::Compression::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::array::Compression::follow(&'a [u8], usize) -> Self::Inner impl<'a> flatbuffers::verifier::Verifiable for vortex_flatbuffers::array::Compression -pub fn vortex_flatbuffers::array::Compression::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::array::Compression::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> #[repr(transparent)] pub struct vortex_flatbuffers::array::Precision(pub u8) @@ -470,15 +470,15 @@ impl core::cmp::Eq for vortex_flatbuffers::array::Precision impl core::cmp::Ord for vortex_flatbuffers::array::Precision -pub fn vortex_flatbuffers::array::Precision::cmp(&self, other: &vortex_flatbuffers::array::Precision) -> core::cmp::Ordering +pub fn vortex_flatbuffers::array::Precision::cmp(&self, &vortex_flatbuffers::array::Precision) -> core::cmp::Ordering impl core::cmp::PartialEq for vortex_flatbuffers::array::Precision -pub fn vortex_flatbuffers::array::Precision::eq(&self, other: &vortex_flatbuffers::array::Precision) -> bool +pub fn vortex_flatbuffers::array::Precision::eq(&self, &vortex_flatbuffers::array::Precision) -> bool impl core::cmp::PartialOrd for vortex_flatbuffers::array::Precision -pub fn vortex_flatbuffers::array::Precision::partial_cmp(&self, other: &vortex_flatbuffers::array::Precision) -> core::option::Option +pub fn vortex_flatbuffers::array::Precision::partial_cmp(&self, &vortex_flatbuffers::array::Precision) -> core::option::Option impl core::default::Default for vortex_flatbuffers::array::Precision @@ -486,11 +486,11 @@ pub fn vortex_flatbuffers::array::Precision::default() -> vortex_flatbuffers::ar impl core::fmt::Debug for vortex_flatbuffers::array::Precision -pub fn vortex_flatbuffers::array::Precision::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::array::Precision::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_flatbuffers::array::Precision -pub fn vortex_flatbuffers::array::Precision::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_flatbuffers::array::Precision::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_flatbuffers::array::Precision @@ -500,7 +500,7 @@ impl flatbuffers::endian_scalar::EndianScalar for vortex_flatbuffers::array::Pre pub type vortex_flatbuffers::array::Precision::Scalar = u8 -pub fn vortex_flatbuffers::array::Precision::from_little_endian(v: u8) -> Self +pub fn vortex_flatbuffers::array::Precision::from_little_endian(u8) -> Self pub fn vortex_flatbuffers::array::Precision::to_little_endian(self) -> u8 @@ -508,7 +508,7 @@ impl flatbuffers::push::Push for vortex_flatbuffers::array::Precision pub type vortex_flatbuffers::array::Precision::Output = vortex_flatbuffers::array::Precision -pub unsafe fn vortex_flatbuffers::array::Precision::push(&self, dst: &mut [u8], _written_len: usize) +pub unsafe fn vortex_flatbuffers::array::Precision::push(&self, &mut [u8], usize) impl flatbuffers::verifier::SimpleToVerifyInSlice for vortex_flatbuffers::array::Precision @@ -516,11 +516,11 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::array::Precisio pub type vortex_flatbuffers::array::Precision::Inner = vortex_flatbuffers::array::Precision -pub unsafe fn vortex_flatbuffers::array::Precision::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::array::Precision::follow(&'a [u8], usize) -> Self::Inner impl<'a> flatbuffers::verifier::Verifiable for vortex_flatbuffers::array::Precision -pub fn vortex_flatbuffers::array::Precision::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::array::Precision::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> pub const vortex_flatbuffers::array::ENUM_MAX_COMPRESSION: u8 @@ -534,21 +534,21 @@ pub const vortex_flatbuffers::array::ENUM_VALUES_COMPRESSION: [vortex_flatbuffer pub const vortex_flatbuffers::array::ENUM_VALUES_PRECISION: [vortex_flatbuffers::array::Precision; 2] -pub fn vortex_flatbuffers::array::finish_array_buffer<'a, 'b, A: flatbuffers::builder::Allocator + 'a>(fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>, root: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::array::finish_array_buffer<'a, 'b, A: flatbuffers::builder::Allocator + 'a>(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>, flatbuffers::primitives::WIPOffset>) -pub fn vortex_flatbuffers::array::finish_size_prefixed_array_buffer<'a, 'b, A: flatbuffers::builder::Allocator + 'a>(fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>, root: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::array::finish_size_prefixed_array_buffer<'a, 'b, A: flatbuffers::builder::Allocator + 'a>(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>, flatbuffers::primitives::WIPOffset>) -pub fn vortex_flatbuffers::array::root_as_array(buf: &[u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::array::root_as_array(&[u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> -pub unsafe fn vortex_flatbuffers::array::root_as_array_unchecked(buf: &[u8]) -> vortex_flatbuffers::array::Array<'_> +pub unsafe fn vortex_flatbuffers::array::root_as_array_unchecked(&[u8]) -> vortex_flatbuffers::array::Array<'_> -pub fn vortex_flatbuffers::array::root_as_array_with_opts<'b, 'o>(opts: &'o flatbuffers::verifier::VerifierOptions, buf: &'b [u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::array::root_as_array_with_opts<'b, 'o>(&'o flatbuffers::verifier::VerifierOptions, &'b [u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> -pub fn vortex_flatbuffers::array::size_prefixed_root_as_array(buf: &[u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::array::size_prefixed_root_as_array(&[u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> -pub unsafe fn vortex_flatbuffers::array::size_prefixed_root_as_array_unchecked(buf: &[u8]) -> vortex_flatbuffers::array::Array<'_> +pub unsafe fn vortex_flatbuffers::array::size_prefixed_root_as_array_unchecked(&[u8]) -> vortex_flatbuffers::array::Array<'_> -pub fn vortex_flatbuffers::array::size_prefixed_root_as_array_with_opts<'b, 'o>(opts: &'o flatbuffers::verifier::VerifierOptions, buf: &'b [u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::array::size_prefixed_root_as_array_with_opts<'b, 'o>(&'o flatbuffers::verifier::VerifierOptions, &'b [u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> pub mod vortex_flatbuffers::dtype @@ -584,19 +584,19 @@ impl<'a> vortex_flatbuffers::dtype::Binary<'a> pub const vortex_flatbuffers::dtype::Binary<'a>::VT_NULLABLE: flatbuffers::primitives::VOffsetT -pub fn vortex_flatbuffers::dtype::Binary<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, args: &'args vortex_flatbuffers::dtype::BinaryArgs) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::dtype::Binary<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::dtype::BinaryArgs) -> flatbuffers::primitives::WIPOffset> -pub unsafe fn vortex_flatbuffers::dtype::Binary<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::dtype::Binary<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self pub fn vortex_flatbuffers::dtype::Binary<'a>::nullable(&self) -> bool impl core::fmt::Debug for vortex_flatbuffers::dtype::Binary<'_> -pub fn vortex_flatbuffers::dtype::Binary<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::dtype::Binary<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::dtype::Binary<'_> -pub fn vortex_flatbuffers::dtype::Binary<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::dtype::Binary<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::dtype::Binary<'a> @@ -604,7 +604,7 @@ pub fn vortex_flatbuffers::dtype::Binary<'a>::clone(&self) -> vortex_flatbuffers impl<'a> core::cmp::PartialEq for vortex_flatbuffers::dtype::Binary<'a> -pub fn vortex_flatbuffers::dtype::Binary<'a>::eq(&self, other: &vortex_flatbuffers::dtype::Binary<'a>) -> bool +pub fn vortex_flatbuffers::dtype::Binary<'a>::eq(&self, &vortex_flatbuffers::dtype::Binary<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::dtype::Binary<'a> @@ -614,7 +614,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::dtype::Binary<' pub type vortex_flatbuffers::dtype::Binary<'a>::Inner = vortex_flatbuffers::dtype::Binary<'a> -pub unsafe fn vortex_flatbuffers::dtype::Binary<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::dtype::Binary<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::dtype::BinaryArgs @@ -628,11 +628,11 @@ pub struct vortex_flatbuffers::dtype::BinaryBuilder<'a: 'b, 'b, A: flatbuffers:: impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::dtype::BinaryBuilder<'a, 'b, A> -pub fn vortex_flatbuffers::dtype::BinaryBuilder<'a, 'b, A>::add_nullable(&mut self, nullable: bool) +pub fn vortex_flatbuffers::dtype::BinaryBuilder<'a, 'b, A>::add_nullable(&mut self, bool) pub fn vortex_flatbuffers::dtype::BinaryBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::dtype::BinaryBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::dtype::BinaryBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::dtype::BinaryBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::dtype::BinaryBuilder<'a, 'b, A> pub struct vortex_flatbuffers::dtype::Bool<'a> @@ -642,19 +642,19 @@ impl<'a> vortex_flatbuffers::dtype::Bool<'a> pub const vortex_flatbuffers::dtype::Bool<'a>::VT_NULLABLE: flatbuffers::primitives::VOffsetT -pub fn vortex_flatbuffers::dtype::Bool<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, args: &'args vortex_flatbuffers::dtype::BoolArgs) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::dtype::Bool<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::dtype::BoolArgs) -> flatbuffers::primitives::WIPOffset> -pub unsafe fn vortex_flatbuffers::dtype::Bool<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::dtype::Bool<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self pub fn vortex_flatbuffers::dtype::Bool<'a>::nullable(&self) -> bool impl core::fmt::Debug for vortex_flatbuffers::dtype::Bool<'_> -pub fn vortex_flatbuffers::dtype::Bool<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::dtype::Bool<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::dtype::Bool<'_> -pub fn vortex_flatbuffers::dtype::Bool<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::dtype::Bool<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::dtype::Bool<'a> @@ -662,7 +662,7 @@ pub fn vortex_flatbuffers::dtype::Bool<'a>::clone(&self) -> vortex_flatbuffers:: impl<'a> core::cmp::PartialEq for vortex_flatbuffers::dtype::Bool<'a> -pub fn vortex_flatbuffers::dtype::Bool<'a>::eq(&self, other: &vortex_flatbuffers::dtype::Bool<'a>) -> bool +pub fn vortex_flatbuffers::dtype::Bool<'a>::eq(&self, &vortex_flatbuffers::dtype::Bool<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::dtype::Bool<'a> @@ -672,7 +672,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::dtype::Bool<'a> pub type vortex_flatbuffers::dtype::Bool<'a>::Inner = vortex_flatbuffers::dtype::Bool<'a> -pub unsafe fn vortex_flatbuffers::dtype::Bool<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::dtype::Bool<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::dtype::BoolArgs @@ -686,11 +686,11 @@ pub struct vortex_flatbuffers::dtype::BoolBuilder<'a: 'b, 'b, A: flatbuffers::bu impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::dtype::BoolBuilder<'a, 'b, A> -pub fn vortex_flatbuffers::dtype::BoolBuilder<'a, 'b, A>::add_nullable(&mut self, nullable: bool) +pub fn vortex_flatbuffers::dtype::BoolBuilder<'a, 'b, A>::add_nullable(&mut self, bool) pub fn vortex_flatbuffers::dtype::BoolBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::dtype::BoolBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::dtype::BoolBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::dtype::BoolBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::dtype::BoolBuilder<'a, 'b, A> pub struct vortex_flatbuffers::dtype::DType<'a> @@ -702,9 +702,9 @@ pub const vortex_flatbuffers::dtype::DType<'a>::VT_TYPE_: flatbuffers::primitive pub const vortex_flatbuffers::dtype::DType<'a>::VT_TYPE_TYPE: flatbuffers::primitives::VOffsetT -pub fn vortex_flatbuffers::dtype::DType<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, args: &'args vortex_flatbuffers::dtype::DTypeArgs) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::dtype::DType<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::dtype::DTypeArgs) -> flatbuffers::primitives::WIPOffset> -pub unsafe fn vortex_flatbuffers::dtype::DType<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::dtype::DType<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self pub fn vortex_flatbuffers::dtype::DType<'a>::type_(&self) -> core::option::Option> @@ -734,11 +734,11 @@ pub fn vortex_flatbuffers::dtype::DType<'a>::type_type(&self) -> vortex_flatbuff impl core::fmt::Debug for vortex_flatbuffers::dtype::DType<'_> -pub fn vortex_flatbuffers::dtype::DType<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::dtype::DType<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::dtype::DType<'_> -pub fn vortex_flatbuffers::dtype::DType<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::dtype::DType<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::dtype::DType<'a> @@ -746,7 +746,7 @@ pub fn vortex_flatbuffers::dtype::DType<'a>::clone(&self) -> vortex_flatbuffers: impl<'a> core::cmp::PartialEq for vortex_flatbuffers::dtype::DType<'a> -pub fn vortex_flatbuffers::dtype::DType<'a>::eq(&self, other: &vortex_flatbuffers::dtype::DType<'a>) -> bool +pub fn vortex_flatbuffers::dtype::DType<'a>::eq(&self, &vortex_flatbuffers::dtype::DType<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::dtype::DType<'a> @@ -756,7 +756,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::dtype::DType<'a pub type vortex_flatbuffers::dtype::DType<'a>::Inner = vortex_flatbuffers::dtype::DType<'a> -pub unsafe fn vortex_flatbuffers::dtype::DType<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::dtype::DType<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::dtype::DTypeArgs @@ -772,13 +772,13 @@ pub struct vortex_flatbuffers::dtype::DTypeBuilder<'a: 'b, 'b, A: flatbuffers::b impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::dtype::DTypeBuilder<'a, 'b, A> -pub fn vortex_flatbuffers::dtype::DTypeBuilder<'a, 'b, A>::add_type_(&mut self, type_: flatbuffers::primitives::WIPOffset) +pub fn vortex_flatbuffers::dtype::DTypeBuilder<'a, 'b, A>::add_type_(&mut self, flatbuffers::primitives::WIPOffset) -pub fn vortex_flatbuffers::dtype::DTypeBuilder<'a, 'b, A>::add_type_type(&mut self, type_type: vortex_flatbuffers::dtype::Type) +pub fn vortex_flatbuffers::dtype::DTypeBuilder<'a, 'b, A>::add_type_type(&mut self, vortex_flatbuffers::dtype::Type) pub fn vortex_flatbuffers::dtype::DTypeBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::dtype::DTypeBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::dtype::DTypeBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::dtype::DTypeBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::dtype::DTypeBuilder<'a, 'b, A> pub struct vortex_flatbuffers::dtype::Decimal<'a> @@ -792,9 +792,9 @@ pub const vortex_flatbuffers::dtype::Decimal<'a>::VT_PRECISION: flatbuffers::pri pub const vortex_flatbuffers::dtype::Decimal<'a>::VT_SCALE: flatbuffers::primitives::VOffsetT -pub fn vortex_flatbuffers::dtype::Decimal<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, args: &'args vortex_flatbuffers::dtype::DecimalArgs) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::dtype::Decimal<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::dtype::DecimalArgs) -> flatbuffers::primitives::WIPOffset> -pub unsafe fn vortex_flatbuffers::dtype::Decimal<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::dtype::Decimal<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self pub fn vortex_flatbuffers::dtype::Decimal<'a>::nullable(&self) -> bool @@ -804,11 +804,11 @@ pub fn vortex_flatbuffers::dtype::Decimal<'a>::scale(&self) -> i8 impl core::fmt::Debug for vortex_flatbuffers::dtype::Decimal<'_> -pub fn vortex_flatbuffers::dtype::Decimal<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::dtype::Decimal<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::dtype::Decimal<'_> -pub fn vortex_flatbuffers::dtype::Decimal<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::dtype::Decimal<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::dtype::Decimal<'a> @@ -816,7 +816,7 @@ pub fn vortex_flatbuffers::dtype::Decimal<'a>::clone(&self) -> vortex_flatbuffer impl<'a> core::cmp::PartialEq for vortex_flatbuffers::dtype::Decimal<'a> -pub fn vortex_flatbuffers::dtype::Decimal<'a>::eq(&self, other: &vortex_flatbuffers::dtype::Decimal<'a>) -> bool +pub fn vortex_flatbuffers::dtype::Decimal<'a>::eq(&self, &vortex_flatbuffers::dtype::Decimal<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::dtype::Decimal<'a> @@ -826,7 +826,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::dtype::Decimal< pub type vortex_flatbuffers::dtype::Decimal<'a>::Inner = vortex_flatbuffers::dtype::Decimal<'a> -pub unsafe fn vortex_flatbuffers::dtype::Decimal<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::dtype::Decimal<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::dtype::DecimalArgs @@ -844,15 +844,15 @@ pub struct vortex_flatbuffers::dtype::DecimalBuilder<'a: 'b, 'b, A: flatbuffers: impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::dtype::DecimalBuilder<'a, 'b, A> -pub fn vortex_flatbuffers::dtype::DecimalBuilder<'a, 'b, A>::add_nullable(&mut self, nullable: bool) +pub fn vortex_flatbuffers::dtype::DecimalBuilder<'a, 'b, A>::add_nullable(&mut self, bool) -pub fn vortex_flatbuffers::dtype::DecimalBuilder<'a, 'b, A>::add_precision(&mut self, precision: u8) +pub fn vortex_flatbuffers::dtype::DecimalBuilder<'a, 'b, A>::add_precision(&mut self, u8) -pub fn vortex_flatbuffers::dtype::DecimalBuilder<'a, 'b, A>::add_scale(&mut self, scale: i8) +pub fn vortex_flatbuffers::dtype::DecimalBuilder<'a, 'b, A>::add_scale(&mut self, i8) pub fn vortex_flatbuffers::dtype::DecimalBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::dtype::DecimalBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::dtype::DecimalBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::dtype::DecimalBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::dtype::DecimalBuilder<'a, 'b, A> pub struct vortex_flatbuffers::dtype::Extension<'a> @@ -866,11 +866,11 @@ pub const vortex_flatbuffers::dtype::Extension<'a>::VT_METADATA: flatbuffers::pr pub const vortex_flatbuffers::dtype::Extension<'a>::VT_STORAGE_DTYPE: flatbuffers::primitives::VOffsetT -pub fn vortex_flatbuffers::dtype::Extension<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, args: &'args vortex_flatbuffers::dtype::ExtensionArgs<'args>) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::dtype::Extension<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::dtype::ExtensionArgs<'args>) -> flatbuffers::primitives::WIPOffset> pub fn vortex_flatbuffers::dtype::Extension<'a>::id(&self) -> core::option::Option<&'a str> -pub unsafe fn vortex_flatbuffers::dtype::Extension<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::dtype::Extension<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self pub fn vortex_flatbuffers::dtype::Extension<'a>::metadata(&self) -> core::option::Option> @@ -878,11 +878,11 @@ pub fn vortex_flatbuffers::dtype::Extension<'a>::storage_dtype(&self) -> core::o impl core::fmt::Debug for vortex_flatbuffers::dtype::Extension<'_> -pub fn vortex_flatbuffers::dtype::Extension<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::dtype::Extension<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::dtype::Extension<'_> -pub fn vortex_flatbuffers::dtype::Extension<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::dtype::Extension<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::dtype::Extension<'a> @@ -890,7 +890,7 @@ pub fn vortex_flatbuffers::dtype::Extension<'a>::clone(&self) -> vortex_flatbuff impl<'a> core::cmp::PartialEq for vortex_flatbuffers::dtype::Extension<'a> -pub fn vortex_flatbuffers::dtype::Extension<'a>::eq(&self, other: &vortex_flatbuffers::dtype::Extension<'a>) -> bool +pub fn vortex_flatbuffers::dtype::Extension<'a>::eq(&self, &vortex_flatbuffers::dtype::Extension<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::dtype::Extension<'a> @@ -900,7 +900,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::dtype::Extensio pub type vortex_flatbuffers::dtype::Extension<'a>::Inner = vortex_flatbuffers::dtype::Extension<'a> -pub unsafe fn vortex_flatbuffers::dtype::Extension<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::dtype::Extension<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::dtype::ExtensionArgs<'a> @@ -918,15 +918,15 @@ pub struct vortex_flatbuffers::dtype::ExtensionBuilder<'a: 'b, 'b, A: flatbuffer impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::dtype::ExtensionBuilder<'a, 'b, A> -pub fn vortex_flatbuffers::dtype::ExtensionBuilder<'a, 'b, A>::add_id(&mut self, id: flatbuffers::primitives::WIPOffset<&'b str>) +pub fn vortex_flatbuffers::dtype::ExtensionBuilder<'a, 'b, A>::add_id(&mut self, flatbuffers::primitives::WIPOffset<&'b str>) -pub fn vortex_flatbuffers::dtype::ExtensionBuilder<'a, 'b, A>::add_metadata(&mut self, metadata: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::dtype::ExtensionBuilder<'a, 'b, A>::add_metadata(&mut self, flatbuffers::primitives::WIPOffset>) -pub fn vortex_flatbuffers::dtype::ExtensionBuilder<'a, 'b, A>::add_storage_dtype(&mut self, storage_dtype: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::dtype::ExtensionBuilder<'a, 'b, A>::add_storage_dtype(&mut self, flatbuffers::primitives::WIPOffset>) pub fn vortex_flatbuffers::dtype::ExtensionBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::dtype::ExtensionBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::dtype::ExtensionBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::dtype::ExtensionBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::dtype::ExtensionBuilder<'a, 'b, A> pub struct vortex_flatbuffers::dtype::FixedSizeList<'a> @@ -940,11 +940,11 @@ pub const vortex_flatbuffers::dtype::FixedSizeList<'a>::VT_NULLABLE: flatbuffers pub const vortex_flatbuffers::dtype::FixedSizeList<'a>::VT_SIZE: flatbuffers::primitives::VOffsetT -pub fn vortex_flatbuffers::dtype::FixedSizeList<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, args: &'args vortex_flatbuffers::dtype::FixedSizeListArgs<'args>) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::dtype::FixedSizeList<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::dtype::FixedSizeListArgs<'args>) -> flatbuffers::primitives::WIPOffset> pub fn vortex_flatbuffers::dtype::FixedSizeList<'a>::element_type(&self) -> core::option::Option> -pub unsafe fn vortex_flatbuffers::dtype::FixedSizeList<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::dtype::FixedSizeList<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self pub fn vortex_flatbuffers::dtype::FixedSizeList<'a>::nullable(&self) -> bool @@ -952,11 +952,11 @@ pub fn vortex_flatbuffers::dtype::FixedSizeList<'a>::size(&self) -> u32 impl core::fmt::Debug for vortex_flatbuffers::dtype::FixedSizeList<'_> -pub fn vortex_flatbuffers::dtype::FixedSizeList<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::dtype::FixedSizeList<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::dtype::FixedSizeList<'_> -pub fn vortex_flatbuffers::dtype::FixedSizeList<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::dtype::FixedSizeList<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::dtype::FixedSizeList<'a> @@ -964,7 +964,7 @@ pub fn vortex_flatbuffers::dtype::FixedSizeList<'a>::clone(&self) -> vortex_flat impl<'a> core::cmp::PartialEq for vortex_flatbuffers::dtype::FixedSizeList<'a> -pub fn vortex_flatbuffers::dtype::FixedSizeList<'a>::eq(&self, other: &vortex_flatbuffers::dtype::FixedSizeList<'a>) -> bool +pub fn vortex_flatbuffers::dtype::FixedSizeList<'a>::eq(&self, &vortex_flatbuffers::dtype::FixedSizeList<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::dtype::FixedSizeList<'a> @@ -974,7 +974,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::dtype::FixedSiz pub type vortex_flatbuffers::dtype::FixedSizeList<'a>::Inner = vortex_flatbuffers::dtype::FixedSizeList<'a> -pub unsafe fn vortex_flatbuffers::dtype::FixedSizeList<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::dtype::FixedSizeList<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::dtype::FixedSizeListArgs<'a> @@ -992,15 +992,15 @@ pub struct vortex_flatbuffers::dtype::FixedSizeListBuilder<'a: 'b, 'b, A: flatbu impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::dtype::FixedSizeListBuilder<'a, 'b, A> -pub fn vortex_flatbuffers::dtype::FixedSizeListBuilder<'a, 'b, A>::add_element_type(&mut self, element_type: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::dtype::FixedSizeListBuilder<'a, 'b, A>::add_element_type(&mut self, flatbuffers::primitives::WIPOffset>) -pub fn vortex_flatbuffers::dtype::FixedSizeListBuilder<'a, 'b, A>::add_nullable(&mut self, nullable: bool) +pub fn vortex_flatbuffers::dtype::FixedSizeListBuilder<'a, 'b, A>::add_nullable(&mut self, bool) -pub fn vortex_flatbuffers::dtype::FixedSizeListBuilder<'a, 'b, A>::add_size(&mut self, size: u32) +pub fn vortex_flatbuffers::dtype::FixedSizeListBuilder<'a, 'b, A>::add_size(&mut self, u32) pub fn vortex_flatbuffers::dtype::FixedSizeListBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::dtype::FixedSizeListBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::dtype::FixedSizeListBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::dtype::FixedSizeListBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::dtype::FixedSizeListBuilder<'a, 'b, A> pub struct vortex_flatbuffers::dtype::List<'a> @@ -1012,21 +1012,21 @@ pub const vortex_flatbuffers::dtype::List<'a>::VT_ELEMENT_TYPE: flatbuffers::pri pub const vortex_flatbuffers::dtype::List<'a>::VT_NULLABLE: flatbuffers::primitives::VOffsetT -pub fn vortex_flatbuffers::dtype::List<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, args: &'args vortex_flatbuffers::dtype::ListArgs<'args>) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::dtype::List<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::dtype::ListArgs<'args>) -> flatbuffers::primitives::WIPOffset> pub fn vortex_flatbuffers::dtype::List<'a>::element_type(&self) -> core::option::Option> -pub unsafe fn vortex_flatbuffers::dtype::List<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::dtype::List<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self pub fn vortex_flatbuffers::dtype::List<'a>::nullable(&self) -> bool impl core::fmt::Debug for vortex_flatbuffers::dtype::List<'_> -pub fn vortex_flatbuffers::dtype::List<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::dtype::List<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::dtype::List<'_> -pub fn vortex_flatbuffers::dtype::List<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::dtype::List<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::dtype::List<'a> @@ -1034,7 +1034,7 @@ pub fn vortex_flatbuffers::dtype::List<'a>::clone(&self) -> vortex_flatbuffers:: impl<'a> core::cmp::PartialEq for vortex_flatbuffers::dtype::List<'a> -pub fn vortex_flatbuffers::dtype::List<'a>::eq(&self, other: &vortex_flatbuffers::dtype::List<'a>) -> bool +pub fn vortex_flatbuffers::dtype::List<'a>::eq(&self, &vortex_flatbuffers::dtype::List<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::dtype::List<'a> @@ -1044,7 +1044,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::dtype::List<'a> pub type vortex_flatbuffers::dtype::List<'a>::Inner = vortex_flatbuffers::dtype::List<'a> -pub unsafe fn vortex_flatbuffers::dtype::List<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::dtype::List<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::dtype::ListArgs<'a> @@ -1060,13 +1060,13 @@ pub struct vortex_flatbuffers::dtype::ListBuilder<'a: 'b, 'b, A: flatbuffers::bu impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::dtype::ListBuilder<'a, 'b, A> -pub fn vortex_flatbuffers::dtype::ListBuilder<'a, 'b, A>::add_element_type(&mut self, element_type: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::dtype::ListBuilder<'a, 'b, A>::add_element_type(&mut self, flatbuffers::primitives::WIPOffset>) -pub fn vortex_flatbuffers::dtype::ListBuilder<'a, 'b, A>::add_nullable(&mut self, nullable: bool) +pub fn vortex_flatbuffers::dtype::ListBuilder<'a, 'b, A>::add_nullable(&mut self, bool) pub fn vortex_flatbuffers::dtype::ListBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::dtype::ListBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::dtype::ListBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::dtype::ListBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::dtype::ListBuilder<'a, 'b, A> pub struct vortex_flatbuffers::dtype::Null<'a> @@ -1074,17 +1074,17 @@ pub vortex_flatbuffers::dtype::Null::_tab: flatbuffers::table::Table<'a> impl<'a> vortex_flatbuffers::dtype::Null<'a> -pub fn vortex_flatbuffers::dtype::Null<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, _args: &'args vortex_flatbuffers::dtype::NullArgs) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::dtype::Null<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::dtype::NullArgs) -> flatbuffers::primitives::WIPOffset> -pub unsafe fn vortex_flatbuffers::dtype::Null<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::dtype::Null<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self impl core::fmt::Debug for vortex_flatbuffers::dtype::Null<'_> -pub fn vortex_flatbuffers::dtype::Null<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::dtype::Null<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::dtype::Null<'_> -pub fn vortex_flatbuffers::dtype::Null<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::dtype::Null<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::dtype::Null<'a> @@ -1092,7 +1092,7 @@ pub fn vortex_flatbuffers::dtype::Null<'a>::clone(&self) -> vortex_flatbuffers:: impl<'a> core::cmp::PartialEq for vortex_flatbuffers::dtype::Null<'a> -pub fn vortex_flatbuffers::dtype::Null<'a>::eq(&self, other: &vortex_flatbuffers::dtype::Null<'a>) -> bool +pub fn vortex_flatbuffers::dtype::Null<'a>::eq(&self, &vortex_flatbuffers::dtype::Null<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::dtype::Null<'a> @@ -1102,7 +1102,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::dtype::Null<'a> pub type vortex_flatbuffers::dtype::Null<'a>::Inner = vortex_flatbuffers::dtype::Null<'a> -pub unsafe fn vortex_flatbuffers::dtype::Null<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::dtype::Null<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::dtype::NullArgs @@ -1116,7 +1116,7 @@ impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::dt pub fn vortex_flatbuffers::dtype::NullBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::dtype::NullBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::dtype::NullBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::dtype::NullBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::dtype::NullBuilder<'a, 'b, A> #[repr(transparent)] pub struct vortex_flatbuffers::dtype::PType(pub u8) @@ -1160,15 +1160,15 @@ impl core::cmp::Eq for vortex_flatbuffers::dtype::PType impl core::cmp::Ord for vortex_flatbuffers::dtype::PType -pub fn vortex_flatbuffers::dtype::PType::cmp(&self, other: &vortex_flatbuffers::dtype::PType) -> core::cmp::Ordering +pub fn vortex_flatbuffers::dtype::PType::cmp(&self, &vortex_flatbuffers::dtype::PType) -> core::cmp::Ordering impl core::cmp::PartialEq for vortex_flatbuffers::dtype::PType -pub fn vortex_flatbuffers::dtype::PType::eq(&self, other: &vortex_flatbuffers::dtype::PType) -> bool +pub fn vortex_flatbuffers::dtype::PType::eq(&self, &vortex_flatbuffers::dtype::PType) -> bool impl core::cmp::PartialOrd for vortex_flatbuffers::dtype::PType -pub fn vortex_flatbuffers::dtype::PType::partial_cmp(&self, other: &vortex_flatbuffers::dtype::PType) -> core::option::Option +pub fn vortex_flatbuffers::dtype::PType::partial_cmp(&self, &vortex_flatbuffers::dtype::PType) -> core::option::Option impl core::default::Default for vortex_flatbuffers::dtype::PType @@ -1176,11 +1176,11 @@ pub fn vortex_flatbuffers::dtype::PType::default() -> vortex_flatbuffers::dtype: impl core::fmt::Debug for vortex_flatbuffers::dtype::PType -pub fn vortex_flatbuffers::dtype::PType::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::dtype::PType::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_flatbuffers::dtype::PType -pub fn vortex_flatbuffers::dtype::PType::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_flatbuffers::dtype::PType::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_flatbuffers::dtype::PType @@ -1190,7 +1190,7 @@ impl flatbuffers::endian_scalar::EndianScalar for vortex_flatbuffers::dtype::PTy pub type vortex_flatbuffers::dtype::PType::Scalar = u8 -pub fn vortex_flatbuffers::dtype::PType::from_little_endian(v: u8) -> Self +pub fn vortex_flatbuffers::dtype::PType::from_little_endian(u8) -> Self pub fn vortex_flatbuffers::dtype::PType::to_little_endian(self) -> u8 @@ -1198,7 +1198,7 @@ impl flatbuffers::push::Push for vortex_flatbuffers::dtype::PType pub type vortex_flatbuffers::dtype::PType::Output = vortex_flatbuffers::dtype::PType -pub unsafe fn vortex_flatbuffers::dtype::PType::push(&self, dst: &mut [u8], _written_len: usize) +pub unsafe fn vortex_flatbuffers::dtype::PType::push(&self, &mut [u8], usize) impl flatbuffers::verifier::SimpleToVerifyInSlice for vortex_flatbuffers::dtype::PType @@ -1206,11 +1206,11 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::dtype::PType pub type vortex_flatbuffers::dtype::PType::Inner = vortex_flatbuffers::dtype::PType -pub unsafe fn vortex_flatbuffers::dtype::PType::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::dtype::PType::follow(&'a [u8], usize) -> Self::Inner impl<'a> flatbuffers::verifier::Verifiable for vortex_flatbuffers::dtype::PType -pub fn vortex_flatbuffers::dtype::PType::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::dtype::PType::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> pub struct vortex_flatbuffers::dtype::Primitive<'a> @@ -1222,9 +1222,9 @@ pub const vortex_flatbuffers::dtype::Primitive<'a>::VT_NULLABLE: flatbuffers::pr pub const vortex_flatbuffers::dtype::Primitive<'a>::VT_PTYPE: flatbuffers::primitives::VOffsetT -pub fn vortex_flatbuffers::dtype::Primitive<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, args: &'args vortex_flatbuffers::dtype::PrimitiveArgs) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::dtype::Primitive<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::dtype::PrimitiveArgs) -> flatbuffers::primitives::WIPOffset> -pub unsafe fn vortex_flatbuffers::dtype::Primitive<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::dtype::Primitive<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self pub fn vortex_flatbuffers::dtype::Primitive<'a>::nullable(&self) -> bool @@ -1232,11 +1232,11 @@ pub fn vortex_flatbuffers::dtype::Primitive<'a>::ptype(&self) -> vortex_flatbuff impl core::fmt::Debug for vortex_flatbuffers::dtype::Primitive<'_> -pub fn vortex_flatbuffers::dtype::Primitive<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::dtype::Primitive<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::dtype::Primitive<'_> -pub fn vortex_flatbuffers::dtype::Primitive<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::dtype::Primitive<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::dtype::Primitive<'a> @@ -1244,7 +1244,7 @@ pub fn vortex_flatbuffers::dtype::Primitive<'a>::clone(&self) -> vortex_flatbuff impl<'a> core::cmp::PartialEq for vortex_flatbuffers::dtype::Primitive<'a> -pub fn vortex_flatbuffers::dtype::Primitive<'a>::eq(&self, other: &vortex_flatbuffers::dtype::Primitive<'a>) -> bool +pub fn vortex_flatbuffers::dtype::Primitive<'a>::eq(&self, &vortex_flatbuffers::dtype::Primitive<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::dtype::Primitive<'a> @@ -1254,7 +1254,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::dtype::Primitiv pub type vortex_flatbuffers::dtype::Primitive<'a>::Inner = vortex_flatbuffers::dtype::Primitive<'a> -pub unsafe fn vortex_flatbuffers::dtype::Primitive<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::dtype::Primitive<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::dtype::PrimitiveArgs @@ -1270,13 +1270,13 @@ pub struct vortex_flatbuffers::dtype::PrimitiveBuilder<'a: 'b, 'b, A: flatbuffer impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::dtype::PrimitiveBuilder<'a, 'b, A> -pub fn vortex_flatbuffers::dtype::PrimitiveBuilder<'a, 'b, A>::add_nullable(&mut self, nullable: bool) +pub fn vortex_flatbuffers::dtype::PrimitiveBuilder<'a, 'b, A>::add_nullable(&mut self, bool) -pub fn vortex_flatbuffers::dtype::PrimitiveBuilder<'a, 'b, A>::add_ptype(&mut self, ptype: vortex_flatbuffers::dtype::PType) +pub fn vortex_flatbuffers::dtype::PrimitiveBuilder<'a, 'b, A>::add_ptype(&mut self, vortex_flatbuffers::dtype::PType) pub fn vortex_flatbuffers::dtype::PrimitiveBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::dtype::PrimitiveBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::dtype::PrimitiveBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::dtype::PrimitiveBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::dtype::PrimitiveBuilder<'a, 'b, A> pub struct vortex_flatbuffers::dtype::Struct_<'a> @@ -1290,11 +1290,11 @@ pub const vortex_flatbuffers::dtype::Struct_<'a>::VT_NAMES: flatbuffers::primiti pub const vortex_flatbuffers::dtype::Struct_<'a>::VT_NULLABLE: flatbuffers::primitives::VOffsetT -pub fn vortex_flatbuffers::dtype::Struct_<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, args: &'args vortex_flatbuffers::dtype::Struct_Args<'args>) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::dtype::Struct_<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::dtype::Struct_Args<'args>) -> flatbuffers::primitives::WIPOffset> pub fn vortex_flatbuffers::dtype::Struct_<'a>::dtypes(&self) -> core::option::Option>>> -pub unsafe fn vortex_flatbuffers::dtype::Struct_<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::dtype::Struct_<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self pub fn vortex_flatbuffers::dtype::Struct_<'a>::names(&self) -> core::option::Option>> @@ -1302,11 +1302,11 @@ pub fn vortex_flatbuffers::dtype::Struct_<'a>::nullable(&self) -> bool impl core::fmt::Debug for vortex_flatbuffers::dtype::Struct_<'_> -pub fn vortex_flatbuffers::dtype::Struct_<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::dtype::Struct_<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::dtype::Struct_<'_> -pub fn vortex_flatbuffers::dtype::Struct_<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::dtype::Struct_<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::dtype::Struct_<'a> @@ -1314,7 +1314,7 @@ pub fn vortex_flatbuffers::dtype::Struct_<'a>::clone(&self) -> vortex_flatbuffer impl<'a> core::cmp::PartialEq for vortex_flatbuffers::dtype::Struct_<'a> -pub fn vortex_flatbuffers::dtype::Struct_<'a>::eq(&self, other: &vortex_flatbuffers::dtype::Struct_<'a>) -> bool +pub fn vortex_flatbuffers::dtype::Struct_<'a>::eq(&self, &vortex_flatbuffers::dtype::Struct_<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::dtype::Struct_<'a> @@ -1324,7 +1324,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::dtype::Struct_< pub type vortex_flatbuffers::dtype::Struct_<'a>::Inner = vortex_flatbuffers::dtype::Struct_<'a> -pub unsafe fn vortex_flatbuffers::dtype::Struct_<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::dtype::Struct_<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::dtype::Struct_Args<'a> @@ -1342,15 +1342,15 @@ pub struct vortex_flatbuffers::dtype::Struct_Builder<'a: 'b, 'b, A: flatbuffers: impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::dtype::Struct_Builder<'a, 'b, A> -pub fn vortex_flatbuffers::dtype::Struct_Builder<'a, 'b, A>::add_dtypes(&mut self, dtypes: flatbuffers::primitives::WIPOffset>>>) +pub fn vortex_flatbuffers::dtype::Struct_Builder<'a, 'b, A>::add_dtypes(&mut self, flatbuffers::primitives::WIPOffset>>>) -pub fn vortex_flatbuffers::dtype::Struct_Builder<'a, 'b, A>::add_names(&mut self, names: flatbuffers::primitives::WIPOffset>>) +pub fn vortex_flatbuffers::dtype::Struct_Builder<'a, 'b, A>::add_names(&mut self, flatbuffers::primitives::WIPOffset>>) -pub fn vortex_flatbuffers::dtype::Struct_Builder<'a, 'b, A>::add_nullable(&mut self, nullable: bool) +pub fn vortex_flatbuffers::dtype::Struct_Builder<'a, 'b, A>::add_nullable(&mut self, bool) pub fn vortex_flatbuffers::dtype::Struct_Builder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::dtype::Struct_Builder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::dtype::Struct_Builder<'a, 'b, A> +pub fn vortex_flatbuffers::dtype::Struct_Builder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::dtype::Struct_Builder<'a, 'b, A> #[repr(transparent)] pub struct vortex_flatbuffers::dtype::Type(pub u8) @@ -1396,15 +1396,15 @@ impl core::cmp::Eq for vortex_flatbuffers::dtype::Type impl core::cmp::Ord for vortex_flatbuffers::dtype::Type -pub fn vortex_flatbuffers::dtype::Type::cmp(&self, other: &vortex_flatbuffers::dtype::Type) -> core::cmp::Ordering +pub fn vortex_flatbuffers::dtype::Type::cmp(&self, &vortex_flatbuffers::dtype::Type) -> core::cmp::Ordering impl core::cmp::PartialEq for vortex_flatbuffers::dtype::Type -pub fn vortex_flatbuffers::dtype::Type::eq(&self, other: &vortex_flatbuffers::dtype::Type) -> bool +pub fn vortex_flatbuffers::dtype::Type::eq(&self, &vortex_flatbuffers::dtype::Type) -> bool impl core::cmp::PartialOrd for vortex_flatbuffers::dtype::Type -pub fn vortex_flatbuffers::dtype::Type::partial_cmp(&self, other: &vortex_flatbuffers::dtype::Type) -> core::option::Option +pub fn vortex_flatbuffers::dtype::Type::partial_cmp(&self, &vortex_flatbuffers::dtype::Type) -> core::option::Option impl core::default::Default for vortex_flatbuffers::dtype::Type @@ -1412,11 +1412,11 @@ pub fn vortex_flatbuffers::dtype::Type::default() -> vortex_flatbuffers::dtype:: impl core::fmt::Debug for vortex_flatbuffers::dtype::Type -pub fn vortex_flatbuffers::dtype::Type::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::dtype::Type::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_flatbuffers::dtype::Type -pub fn vortex_flatbuffers::dtype::Type::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_flatbuffers::dtype::Type::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_flatbuffers::dtype::Type @@ -1426,7 +1426,7 @@ impl flatbuffers::endian_scalar::EndianScalar for vortex_flatbuffers::dtype::Typ pub type vortex_flatbuffers::dtype::Type::Scalar = u8 -pub fn vortex_flatbuffers::dtype::Type::from_little_endian(v: u8) -> Self +pub fn vortex_flatbuffers::dtype::Type::from_little_endian(u8) -> Self pub fn vortex_flatbuffers::dtype::Type::to_little_endian(self) -> u8 @@ -1434,7 +1434,7 @@ impl flatbuffers::push::Push for vortex_flatbuffers::dtype::Type pub type vortex_flatbuffers::dtype::Type::Output = vortex_flatbuffers::dtype::Type -pub unsafe fn vortex_flatbuffers::dtype::Type::push(&self, dst: &mut [u8], _written_len: usize) +pub unsafe fn vortex_flatbuffers::dtype::Type::push(&self, &mut [u8], usize) impl flatbuffers::verifier::SimpleToVerifyInSlice for vortex_flatbuffers::dtype::Type @@ -1442,11 +1442,11 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::dtype::Type pub type vortex_flatbuffers::dtype::Type::Inner = vortex_flatbuffers::dtype::Type -pub unsafe fn vortex_flatbuffers::dtype::Type::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::dtype::Type::follow(&'a [u8], usize) -> Self::Inner impl<'a> flatbuffers::verifier::Verifiable for vortex_flatbuffers::dtype::Type -pub fn vortex_flatbuffers::dtype::Type::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::dtype::Type::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> pub struct vortex_flatbuffers::dtype::TypeUnionTableOffset @@ -1458,19 +1458,19 @@ impl<'a> vortex_flatbuffers::dtype::Utf8<'a> pub const vortex_flatbuffers::dtype::Utf8<'a>::VT_NULLABLE: flatbuffers::primitives::VOffsetT -pub fn vortex_flatbuffers::dtype::Utf8<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, args: &'args vortex_flatbuffers::dtype::Utf8Args) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::dtype::Utf8<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::dtype::Utf8Args) -> flatbuffers::primitives::WIPOffset> -pub unsafe fn vortex_flatbuffers::dtype::Utf8<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::dtype::Utf8<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self pub fn vortex_flatbuffers::dtype::Utf8<'a>::nullable(&self) -> bool impl core::fmt::Debug for vortex_flatbuffers::dtype::Utf8<'_> -pub fn vortex_flatbuffers::dtype::Utf8<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::dtype::Utf8<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::dtype::Utf8<'_> -pub fn vortex_flatbuffers::dtype::Utf8<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::dtype::Utf8<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::dtype::Utf8<'a> @@ -1478,7 +1478,7 @@ pub fn vortex_flatbuffers::dtype::Utf8<'a>::clone(&self) -> vortex_flatbuffers:: impl<'a> core::cmp::PartialEq for vortex_flatbuffers::dtype::Utf8<'a> -pub fn vortex_flatbuffers::dtype::Utf8<'a>::eq(&self, other: &vortex_flatbuffers::dtype::Utf8<'a>) -> bool +pub fn vortex_flatbuffers::dtype::Utf8<'a>::eq(&self, &vortex_flatbuffers::dtype::Utf8<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::dtype::Utf8<'a> @@ -1488,7 +1488,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::dtype::Utf8<'a> pub type vortex_flatbuffers::dtype::Utf8<'a>::Inner = vortex_flatbuffers::dtype::Utf8<'a> -pub unsafe fn vortex_flatbuffers::dtype::Utf8<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::dtype::Utf8<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::dtype::Utf8Args @@ -1502,11 +1502,11 @@ pub struct vortex_flatbuffers::dtype::Utf8Builder<'a: 'b, 'b, A: flatbuffers::bu impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::dtype::Utf8Builder<'a, 'b, A> -pub fn vortex_flatbuffers::dtype::Utf8Builder<'a, 'b, A>::add_nullable(&mut self, nullable: bool) +pub fn vortex_flatbuffers::dtype::Utf8Builder<'a, 'b, A>::add_nullable(&mut self, bool) pub fn vortex_flatbuffers::dtype::Utf8Builder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::dtype::Utf8Builder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::dtype::Utf8Builder<'a, 'b, A> +pub fn vortex_flatbuffers::dtype::Utf8Builder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::dtype::Utf8Builder<'a, 'b, A> pub struct vortex_flatbuffers::dtype::Variant<'a> @@ -1516,19 +1516,19 @@ impl<'a> vortex_flatbuffers::dtype::Variant<'a> pub const vortex_flatbuffers::dtype::Variant<'a>::VT_NULLABLE: flatbuffers::primitives::VOffsetT -pub fn vortex_flatbuffers::dtype::Variant<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, args: &'args vortex_flatbuffers::dtype::VariantArgs) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::dtype::Variant<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::dtype::VariantArgs) -> flatbuffers::primitives::WIPOffset> -pub unsafe fn vortex_flatbuffers::dtype::Variant<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::dtype::Variant<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self pub fn vortex_flatbuffers::dtype::Variant<'a>::nullable(&self) -> bool impl core::fmt::Debug for vortex_flatbuffers::dtype::Variant<'_> -pub fn vortex_flatbuffers::dtype::Variant<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::dtype::Variant<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::dtype::Variant<'_> -pub fn vortex_flatbuffers::dtype::Variant<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::dtype::Variant<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::dtype::Variant<'a> @@ -1536,7 +1536,7 @@ pub fn vortex_flatbuffers::dtype::Variant<'a>::clone(&self) -> vortex_flatbuffer impl<'a> core::cmp::PartialEq for vortex_flatbuffers::dtype::Variant<'a> -pub fn vortex_flatbuffers::dtype::Variant<'a>::eq(&self, other: &vortex_flatbuffers::dtype::Variant<'a>) -> bool +pub fn vortex_flatbuffers::dtype::Variant<'a>::eq(&self, &vortex_flatbuffers::dtype::Variant<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::dtype::Variant<'a> @@ -1546,7 +1546,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::dtype::Variant< pub type vortex_flatbuffers::dtype::Variant<'a>::Inner = vortex_flatbuffers::dtype::Variant<'a> -pub unsafe fn vortex_flatbuffers::dtype::Variant<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::dtype::Variant<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::dtype::VariantArgs @@ -1560,11 +1560,11 @@ pub struct vortex_flatbuffers::dtype::VariantBuilder<'a: 'b, 'b, A: flatbuffers: impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::dtype::VariantBuilder<'a, 'b, A> -pub fn vortex_flatbuffers::dtype::VariantBuilder<'a, 'b, A>::add_nullable(&mut self, nullable: bool) +pub fn vortex_flatbuffers::dtype::VariantBuilder<'a, 'b, A>::add_nullable(&mut self, bool) pub fn vortex_flatbuffers::dtype::VariantBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::dtype::VariantBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::dtype::VariantBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::dtype::VariantBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::dtype::VariantBuilder<'a, 'b, A> pub const vortex_flatbuffers::dtype::ENUM_MAX_PTYPE: u8 @@ -1578,21 +1578,21 @@ pub const vortex_flatbuffers::dtype::ENUM_VALUES_PTYPE: [vortex_flatbuffers::dty pub const vortex_flatbuffers::dtype::ENUM_VALUES_TYPE: [vortex_flatbuffers::dtype::Type; 12] -pub fn vortex_flatbuffers::dtype::finish_dtype_buffer<'a, 'b, A: flatbuffers::builder::Allocator + 'a>(fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>, root: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::dtype::finish_dtype_buffer<'a, 'b, A: flatbuffers::builder::Allocator + 'a>(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>, flatbuffers::primitives::WIPOffset>) -pub fn vortex_flatbuffers::dtype::finish_size_prefixed_dtype_buffer<'a, 'b, A: flatbuffers::builder::Allocator + 'a>(fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>, root: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::dtype::finish_size_prefixed_dtype_buffer<'a, 'b, A: flatbuffers::builder::Allocator + 'a>(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>, flatbuffers::primitives::WIPOffset>) -pub fn vortex_flatbuffers::dtype::root_as_dtype(buf: &[u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::dtype::root_as_dtype(&[u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> -pub unsafe fn vortex_flatbuffers::dtype::root_as_dtype_unchecked(buf: &[u8]) -> vortex_flatbuffers::dtype::DType<'_> +pub unsafe fn vortex_flatbuffers::dtype::root_as_dtype_unchecked(&[u8]) -> vortex_flatbuffers::dtype::DType<'_> -pub fn vortex_flatbuffers::dtype::root_as_dtype_with_opts<'b, 'o>(opts: &'o flatbuffers::verifier::VerifierOptions, buf: &'b [u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::dtype::root_as_dtype_with_opts<'b, 'o>(&'o flatbuffers::verifier::VerifierOptions, &'b [u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> -pub fn vortex_flatbuffers::dtype::size_prefixed_root_as_dtype(buf: &[u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::dtype::size_prefixed_root_as_dtype(&[u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> -pub unsafe fn vortex_flatbuffers::dtype::size_prefixed_root_as_dtype_unchecked(buf: &[u8]) -> vortex_flatbuffers::dtype::DType<'_> +pub unsafe fn vortex_flatbuffers::dtype::size_prefixed_root_as_dtype_unchecked(&[u8]) -> vortex_flatbuffers::dtype::DType<'_> -pub fn vortex_flatbuffers::dtype::size_prefixed_root_as_dtype_with_opts<'b, 'o>(opts: &'o flatbuffers::verifier::VerifierOptions, buf: &'b [u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::dtype::size_prefixed_root_as_dtype_with_opts<'b, 'o>(&'o flatbuffers::verifier::VerifierOptions, &'b [u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> pub mod vortex_flatbuffers::footer @@ -1620,19 +1620,19 @@ impl<'a> vortex_flatbuffers::footer::ArraySpec<'a> pub const vortex_flatbuffers::footer::ArraySpec<'a>::VT_ID: flatbuffers::primitives::VOffsetT -pub fn vortex_flatbuffers::footer::ArraySpec<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, args: &'args vortex_flatbuffers::footer::ArraySpecArgs<'args>) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::footer::ArraySpec<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::footer::ArraySpecArgs<'args>) -> flatbuffers::primitives::WIPOffset> pub fn vortex_flatbuffers::footer::ArraySpec<'a>::id(&self) -> &'a str -pub unsafe fn vortex_flatbuffers::footer::ArraySpec<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::footer::ArraySpec<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self impl core::fmt::Debug for vortex_flatbuffers::footer::ArraySpec<'_> -pub fn vortex_flatbuffers::footer::ArraySpec<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::footer::ArraySpec<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::footer::ArraySpec<'_> -pub fn vortex_flatbuffers::footer::ArraySpec<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::footer::ArraySpec<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::footer::ArraySpec<'a> @@ -1640,7 +1640,7 @@ pub fn vortex_flatbuffers::footer::ArraySpec<'a>::clone(&self) -> vortex_flatbuf impl<'a> core::cmp::PartialEq for vortex_flatbuffers::footer::ArraySpec<'a> -pub fn vortex_flatbuffers::footer::ArraySpec<'a>::eq(&self, other: &vortex_flatbuffers::footer::ArraySpec<'a>) -> bool +pub fn vortex_flatbuffers::footer::ArraySpec<'a>::eq(&self, &vortex_flatbuffers::footer::ArraySpec<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::footer::ArraySpec<'a> @@ -1650,7 +1650,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::footer::ArraySp pub type vortex_flatbuffers::footer::ArraySpec<'a>::Inner = vortex_flatbuffers::footer::ArraySpec<'a> -pub unsafe fn vortex_flatbuffers::footer::ArraySpec<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::footer::ArraySpec<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::footer::ArraySpecArgs<'a> @@ -1664,11 +1664,11 @@ pub struct vortex_flatbuffers::footer::ArraySpecBuilder<'a: 'b, 'b, A: flatbuffe impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::footer::ArraySpecBuilder<'a, 'b, A> -pub fn vortex_flatbuffers::footer::ArraySpecBuilder<'a, 'b, A>::add_id(&mut self, id: flatbuffers::primitives::WIPOffset<&'b str>) +pub fn vortex_flatbuffers::footer::ArraySpecBuilder<'a, 'b, A>::add_id(&mut self, flatbuffers::primitives::WIPOffset<&'b str>) pub fn vortex_flatbuffers::footer::ArraySpecBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::footer::ArraySpecBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::footer::ArraySpecBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::footer::ArraySpecBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::footer::ArraySpecBuilder<'a, 'b, A> #[repr(transparent)] pub struct vortex_flatbuffers::footer::CompressionScheme(pub u8) @@ -1698,15 +1698,15 @@ impl core::cmp::Eq for vortex_flatbuffers::footer::CompressionScheme impl core::cmp::Ord for vortex_flatbuffers::footer::CompressionScheme -pub fn vortex_flatbuffers::footer::CompressionScheme::cmp(&self, other: &vortex_flatbuffers::footer::CompressionScheme) -> core::cmp::Ordering +pub fn vortex_flatbuffers::footer::CompressionScheme::cmp(&self, &vortex_flatbuffers::footer::CompressionScheme) -> core::cmp::Ordering impl core::cmp::PartialEq for vortex_flatbuffers::footer::CompressionScheme -pub fn vortex_flatbuffers::footer::CompressionScheme::eq(&self, other: &vortex_flatbuffers::footer::CompressionScheme) -> bool +pub fn vortex_flatbuffers::footer::CompressionScheme::eq(&self, &vortex_flatbuffers::footer::CompressionScheme) -> bool impl core::cmp::PartialOrd for vortex_flatbuffers::footer::CompressionScheme -pub fn vortex_flatbuffers::footer::CompressionScheme::partial_cmp(&self, other: &vortex_flatbuffers::footer::CompressionScheme) -> core::option::Option +pub fn vortex_flatbuffers::footer::CompressionScheme::partial_cmp(&self, &vortex_flatbuffers::footer::CompressionScheme) -> core::option::Option impl core::default::Default for vortex_flatbuffers::footer::CompressionScheme @@ -1714,11 +1714,11 @@ pub fn vortex_flatbuffers::footer::CompressionScheme::default() -> vortex_flatbu impl core::fmt::Debug for vortex_flatbuffers::footer::CompressionScheme -pub fn vortex_flatbuffers::footer::CompressionScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::footer::CompressionScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_flatbuffers::footer::CompressionScheme -pub fn vortex_flatbuffers::footer::CompressionScheme::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_flatbuffers::footer::CompressionScheme::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_flatbuffers::footer::CompressionScheme @@ -1728,7 +1728,7 @@ impl flatbuffers::endian_scalar::EndianScalar for vortex_flatbuffers::footer::Co pub type vortex_flatbuffers::footer::CompressionScheme::Scalar = u8 -pub fn vortex_flatbuffers::footer::CompressionScheme::from_little_endian(v: u8) -> Self +pub fn vortex_flatbuffers::footer::CompressionScheme::from_little_endian(u8) -> Self pub fn vortex_flatbuffers::footer::CompressionScheme::to_little_endian(self) -> u8 @@ -1736,7 +1736,7 @@ impl flatbuffers::push::Push for vortex_flatbuffers::footer::CompressionScheme pub type vortex_flatbuffers::footer::CompressionScheme::Output = vortex_flatbuffers::footer::CompressionScheme -pub unsafe fn vortex_flatbuffers::footer::CompressionScheme::push(&self, dst: &mut [u8], _written_len: usize) +pub unsafe fn vortex_flatbuffers::footer::CompressionScheme::push(&self, &mut [u8], usize) impl flatbuffers::verifier::SimpleToVerifyInSlice for vortex_flatbuffers::footer::CompressionScheme @@ -1744,11 +1744,11 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::footer::Compres pub type vortex_flatbuffers::footer::CompressionScheme::Inner = vortex_flatbuffers::footer::CompressionScheme -pub unsafe fn vortex_flatbuffers::footer::CompressionScheme::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::footer::CompressionScheme::follow(&'a [u8], usize) -> Self::Inner impl<'a> flatbuffers::verifier::Verifiable for vortex_flatbuffers::footer::CompressionScheme -pub fn vortex_flatbuffers::footer::CompressionScheme::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::footer::CompressionScheme::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> pub struct vortex_flatbuffers::footer::CompressionSpec<'a> @@ -1758,19 +1758,19 @@ impl<'a> vortex_flatbuffers::footer::CompressionSpec<'a> pub const vortex_flatbuffers::footer::CompressionSpec<'a>::VT_SCHEME: flatbuffers::primitives::VOffsetT -pub fn vortex_flatbuffers::footer::CompressionSpec<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, args: &'args vortex_flatbuffers::footer::CompressionSpecArgs) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::footer::CompressionSpec<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::footer::CompressionSpecArgs) -> flatbuffers::primitives::WIPOffset> -pub unsafe fn vortex_flatbuffers::footer::CompressionSpec<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::footer::CompressionSpec<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self pub fn vortex_flatbuffers::footer::CompressionSpec<'a>::scheme(&self) -> vortex_flatbuffers::footer::CompressionScheme impl core::fmt::Debug for vortex_flatbuffers::footer::CompressionSpec<'_> -pub fn vortex_flatbuffers::footer::CompressionSpec<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::footer::CompressionSpec<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::footer::CompressionSpec<'_> -pub fn vortex_flatbuffers::footer::CompressionSpec<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::footer::CompressionSpec<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::footer::CompressionSpec<'a> @@ -1778,7 +1778,7 @@ pub fn vortex_flatbuffers::footer::CompressionSpec<'a>::clone(&self) -> vortex_f impl<'a> core::cmp::PartialEq for vortex_flatbuffers::footer::CompressionSpec<'a> -pub fn vortex_flatbuffers::footer::CompressionSpec<'a>::eq(&self, other: &vortex_flatbuffers::footer::CompressionSpec<'a>) -> bool +pub fn vortex_flatbuffers::footer::CompressionSpec<'a>::eq(&self, &vortex_flatbuffers::footer::CompressionSpec<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::footer::CompressionSpec<'a> @@ -1788,7 +1788,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::footer::Compres pub type vortex_flatbuffers::footer::CompressionSpec<'a>::Inner = vortex_flatbuffers::footer::CompressionSpec<'a> -pub unsafe fn vortex_flatbuffers::footer::CompressionSpec<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::footer::CompressionSpec<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::footer::CompressionSpecArgs @@ -1802,11 +1802,11 @@ pub struct vortex_flatbuffers::footer::CompressionSpecBuilder<'a: 'b, 'b, A: fla impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::footer::CompressionSpecBuilder<'a, 'b, A> -pub fn vortex_flatbuffers::footer::CompressionSpecBuilder<'a, 'b, A>::add_scheme(&mut self, scheme: vortex_flatbuffers::footer::CompressionScheme) +pub fn vortex_flatbuffers::footer::CompressionSpecBuilder<'a, 'b, A>::add_scheme(&mut self, vortex_flatbuffers::footer::CompressionScheme) pub fn vortex_flatbuffers::footer::CompressionSpecBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::footer::CompressionSpecBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::footer::CompressionSpecBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::footer::CompressionSpecBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::footer::CompressionSpecBuilder<'a, 'b, A> pub struct vortex_flatbuffers::footer::EncryptionSpec<'a> @@ -1814,17 +1814,17 @@ pub vortex_flatbuffers::footer::EncryptionSpec::_tab: flatbuffers::table::Table< impl<'a> vortex_flatbuffers::footer::EncryptionSpec<'a> -pub fn vortex_flatbuffers::footer::EncryptionSpec<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, _args: &'args vortex_flatbuffers::footer::EncryptionSpecArgs) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::footer::EncryptionSpec<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::footer::EncryptionSpecArgs) -> flatbuffers::primitives::WIPOffset> -pub unsafe fn vortex_flatbuffers::footer::EncryptionSpec<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::footer::EncryptionSpec<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self impl core::fmt::Debug for vortex_flatbuffers::footer::EncryptionSpec<'_> -pub fn vortex_flatbuffers::footer::EncryptionSpec<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::footer::EncryptionSpec<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::footer::EncryptionSpec<'_> -pub fn vortex_flatbuffers::footer::EncryptionSpec<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::footer::EncryptionSpec<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::footer::EncryptionSpec<'a> @@ -1832,7 +1832,7 @@ pub fn vortex_flatbuffers::footer::EncryptionSpec<'a>::clone(&self) -> vortex_fl impl<'a> core::cmp::PartialEq for vortex_flatbuffers::footer::EncryptionSpec<'a> -pub fn vortex_flatbuffers::footer::EncryptionSpec<'a>::eq(&self, other: &vortex_flatbuffers::footer::EncryptionSpec<'a>) -> bool +pub fn vortex_flatbuffers::footer::EncryptionSpec<'a>::eq(&self, &vortex_flatbuffers::footer::EncryptionSpec<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::footer::EncryptionSpec<'a> @@ -1842,7 +1842,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::footer::Encrypt pub type vortex_flatbuffers::footer::EncryptionSpec<'a>::Inner = vortex_flatbuffers::footer::EncryptionSpec<'a> -pub unsafe fn vortex_flatbuffers::footer::EncryptionSpec<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::footer::EncryptionSpec<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::footer::EncryptionSpecArgs @@ -1856,7 +1856,7 @@ impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::fo pub fn vortex_flatbuffers::footer::EncryptionSpecBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::footer::EncryptionSpecBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::footer::EncryptionSpecBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::footer::EncryptionSpecBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::footer::EncryptionSpecBuilder<'a, 'b, A> pub struct vortex_flatbuffers::footer::FileStatistics<'a> @@ -1866,19 +1866,19 @@ impl<'a> vortex_flatbuffers::footer::FileStatistics<'a> pub const vortex_flatbuffers::footer::FileStatistics<'a>::VT_FIELD_STATS: flatbuffers::primitives::VOffsetT -pub fn vortex_flatbuffers::footer::FileStatistics<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, args: &'args vortex_flatbuffers::footer::FileStatisticsArgs<'args>) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::footer::FileStatistics<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::footer::FileStatisticsArgs<'args>) -> flatbuffers::primitives::WIPOffset> pub fn vortex_flatbuffers::footer::FileStatistics<'a>::field_stats(&self) -> core::option::Option>>> -pub unsafe fn vortex_flatbuffers::footer::FileStatistics<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::footer::FileStatistics<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self impl core::fmt::Debug for vortex_flatbuffers::footer::FileStatistics<'_> -pub fn vortex_flatbuffers::footer::FileStatistics<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::footer::FileStatistics<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::footer::FileStatistics<'_> -pub fn vortex_flatbuffers::footer::FileStatistics<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::footer::FileStatistics<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::footer::FileStatistics<'a> @@ -1886,7 +1886,7 @@ pub fn vortex_flatbuffers::footer::FileStatistics<'a>::clone(&self) -> vortex_fl impl<'a> core::cmp::PartialEq for vortex_flatbuffers::footer::FileStatistics<'a> -pub fn vortex_flatbuffers::footer::FileStatistics<'a>::eq(&self, other: &vortex_flatbuffers::footer::FileStatistics<'a>) -> bool +pub fn vortex_flatbuffers::footer::FileStatistics<'a>::eq(&self, &vortex_flatbuffers::footer::FileStatistics<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::footer::FileStatistics<'a> @@ -1896,7 +1896,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::footer::FileSta pub type vortex_flatbuffers::footer::FileStatistics<'a>::Inner = vortex_flatbuffers::footer::FileStatistics<'a> -pub unsafe fn vortex_flatbuffers::footer::FileStatistics<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::footer::FileStatistics<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::footer::FileStatisticsArgs<'a> @@ -1910,11 +1910,11 @@ pub struct vortex_flatbuffers::footer::FileStatisticsBuilder<'a: 'b, 'b, A: flat impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::footer::FileStatisticsBuilder<'a, 'b, A> -pub fn vortex_flatbuffers::footer::FileStatisticsBuilder<'a, 'b, A>::add_field_stats(&mut self, field_stats: flatbuffers::primitives::WIPOffset>>>) +pub fn vortex_flatbuffers::footer::FileStatisticsBuilder<'a, 'b, A>::add_field_stats(&mut self, flatbuffers::primitives::WIPOffset>>>) pub fn vortex_flatbuffers::footer::FileStatisticsBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::footer::FileStatisticsBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::footer::FileStatisticsBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::footer::FileStatisticsBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::footer::FileStatisticsBuilder<'a, 'b, A> pub struct vortex_flatbuffers::footer::Footer<'a> @@ -1936,11 +1936,11 @@ pub fn vortex_flatbuffers::footer::Footer<'a>::array_specs(&self) -> core::optio pub fn vortex_flatbuffers::footer::Footer<'a>::compression_specs(&self) -> core::option::Option>>> -pub fn vortex_flatbuffers::footer::Footer<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, args: &'args vortex_flatbuffers::footer::FooterArgs<'args>) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::footer::Footer<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::footer::FooterArgs<'args>) -> flatbuffers::primitives::WIPOffset> pub fn vortex_flatbuffers::footer::Footer<'a>::encryption_specs(&self) -> core::option::Option>>> -pub unsafe fn vortex_flatbuffers::footer::Footer<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::footer::Footer<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self pub fn vortex_flatbuffers::footer::Footer<'a>::layout_specs(&self) -> core::option::Option>>> @@ -1948,11 +1948,11 @@ pub fn vortex_flatbuffers::footer::Footer<'a>::segment_specs(&self) -> core::opt impl core::fmt::Debug for vortex_flatbuffers::footer::Footer<'_> -pub fn vortex_flatbuffers::footer::Footer<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::footer::Footer<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::footer::Footer<'_> -pub fn vortex_flatbuffers::footer::Footer<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::footer::Footer<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::footer::Footer<'a> @@ -1960,7 +1960,7 @@ pub fn vortex_flatbuffers::footer::Footer<'a>::clone(&self) -> vortex_flatbuffer impl<'a> core::cmp::PartialEq for vortex_flatbuffers::footer::Footer<'a> -pub fn vortex_flatbuffers::footer::Footer<'a>::eq(&self, other: &vortex_flatbuffers::footer::Footer<'a>) -> bool +pub fn vortex_flatbuffers::footer::Footer<'a>::eq(&self, &vortex_flatbuffers::footer::Footer<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::footer::Footer<'a> @@ -1970,7 +1970,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::footer::Footer< pub type vortex_flatbuffers::footer::Footer<'a>::Inner = vortex_flatbuffers::footer::Footer<'a> -pub unsafe fn vortex_flatbuffers::footer::Footer<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::footer::Footer<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::footer::FooterArgs<'a> @@ -1992,19 +1992,19 @@ pub struct vortex_flatbuffers::footer::FooterBuilder<'a: 'b, 'b, A: flatbuffers: impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::footer::FooterBuilder<'a, 'b, A> -pub fn vortex_flatbuffers::footer::FooterBuilder<'a, 'b, A>::add_array_specs(&mut self, array_specs: flatbuffers::primitives::WIPOffset>>>) +pub fn vortex_flatbuffers::footer::FooterBuilder<'a, 'b, A>::add_array_specs(&mut self, flatbuffers::primitives::WIPOffset>>>) -pub fn vortex_flatbuffers::footer::FooterBuilder<'a, 'b, A>::add_compression_specs(&mut self, compression_specs: flatbuffers::primitives::WIPOffset>>>) +pub fn vortex_flatbuffers::footer::FooterBuilder<'a, 'b, A>::add_compression_specs(&mut self, flatbuffers::primitives::WIPOffset>>>) -pub fn vortex_flatbuffers::footer::FooterBuilder<'a, 'b, A>::add_encryption_specs(&mut self, encryption_specs: flatbuffers::primitives::WIPOffset>>>) +pub fn vortex_flatbuffers::footer::FooterBuilder<'a, 'b, A>::add_encryption_specs(&mut self, flatbuffers::primitives::WIPOffset>>>) -pub fn vortex_flatbuffers::footer::FooterBuilder<'a, 'b, A>::add_layout_specs(&mut self, layout_specs: flatbuffers::primitives::WIPOffset>>>) +pub fn vortex_flatbuffers::footer::FooterBuilder<'a, 'b, A>::add_layout_specs(&mut self, flatbuffers::primitives::WIPOffset>>>) -pub fn vortex_flatbuffers::footer::FooterBuilder<'a, 'b, A>::add_segment_specs(&mut self, segment_specs: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::footer::FooterBuilder<'a, 'b, A>::add_segment_specs(&mut self, flatbuffers::primitives::WIPOffset>) pub fn vortex_flatbuffers::footer::FooterBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::footer::FooterBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::footer::FooterBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::footer::FooterBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::footer::FooterBuilder<'a, 'b, A> pub struct vortex_flatbuffers::footer::LayoutSpec<'a> @@ -2014,19 +2014,19 @@ impl<'a> vortex_flatbuffers::footer::LayoutSpec<'a> pub const vortex_flatbuffers::footer::LayoutSpec<'a>::VT_ID: flatbuffers::primitives::VOffsetT -pub fn vortex_flatbuffers::footer::LayoutSpec<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, args: &'args vortex_flatbuffers::footer::LayoutSpecArgs<'args>) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::footer::LayoutSpec<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::footer::LayoutSpecArgs<'args>) -> flatbuffers::primitives::WIPOffset> pub fn vortex_flatbuffers::footer::LayoutSpec<'a>::id(&self) -> &'a str -pub unsafe fn vortex_flatbuffers::footer::LayoutSpec<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::footer::LayoutSpec<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self impl core::fmt::Debug for vortex_flatbuffers::footer::LayoutSpec<'_> -pub fn vortex_flatbuffers::footer::LayoutSpec<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::footer::LayoutSpec<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::footer::LayoutSpec<'_> -pub fn vortex_flatbuffers::footer::LayoutSpec<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::footer::LayoutSpec<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::footer::LayoutSpec<'a> @@ -2034,7 +2034,7 @@ pub fn vortex_flatbuffers::footer::LayoutSpec<'a>::clone(&self) -> vortex_flatbu impl<'a> core::cmp::PartialEq for vortex_flatbuffers::footer::LayoutSpec<'a> -pub fn vortex_flatbuffers::footer::LayoutSpec<'a>::eq(&self, other: &vortex_flatbuffers::footer::LayoutSpec<'a>) -> bool +pub fn vortex_flatbuffers::footer::LayoutSpec<'a>::eq(&self, &vortex_flatbuffers::footer::LayoutSpec<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::footer::LayoutSpec<'a> @@ -2044,7 +2044,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::footer::LayoutS pub type vortex_flatbuffers::footer::LayoutSpec<'a>::Inner = vortex_flatbuffers::footer::LayoutSpec<'a> -pub unsafe fn vortex_flatbuffers::footer::LayoutSpec<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::footer::LayoutSpec<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::footer::LayoutSpecArgs<'a> @@ -2058,11 +2058,11 @@ pub struct vortex_flatbuffers::footer::LayoutSpecBuilder<'a: 'b, 'b, A: flatbuff impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::footer::LayoutSpecBuilder<'a, 'b, A> -pub fn vortex_flatbuffers::footer::LayoutSpecBuilder<'a, 'b, A>::add_id(&mut self, id: flatbuffers::primitives::WIPOffset<&'b str>) +pub fn vortex_flatbuffers::footer::LayoutSpecBuilder<'a, 'b, A>::add_id(&mut self, flatbuffers::primitives::WIPOffset<&'b str>) pub fn vortex_flatbuffers::footer::LayoutSpecBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::footer::LayoutSpecBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::footer::LayoutSpecBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::footer::LayoutSpecBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::footer::LayoutSpecBuilder<'a, 'b, A> pub struct vortex_flatbuffers::footer::Postscript<'a> @@ -2078,13 +2078,13 @@ pub const vortex_flatbuffers::footer::Postscript<'a>::VT_LAYOUT: flatbuffers::pr pub const vortex_flatbuffers::footer::Postscript<'a>::VT_STATISTICS: flatbuffers::primitives::VOffsetT -pub fn vortex_flatbuffers::footer::Postscript<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, args: &'args vortex_flatbuffers::footer::PostscriptArgs<'args>) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::footer::Postscript<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::footer::PostscriptArgs<'args>) -> flatbuffers::primitives::WIPOffset> pub fn vortex_flatbuffers::footer::Postscript<'a>::dtype(&self) -> core::option::Option> pub fn vortex_flatbuffers::footer::Postscript<'a>::footer(&self) -> core::option::Option> -pub unsafe fn vortex_flatbuffers::footer::Postscript<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::footer::Postscript<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self pub fn vortex_flatbuffers::footer::Postscript<'a>::layout(&self) -> core::option::Option> @@ -2092,11 +2092,11 @@ pub fn vortex_flatbuffers::footer::Postscript<'a>::statistics(&self) -> core::op impl core::fmt::Debug for vortex_flatbuffers::footer::Postscript<'_> -pub fn vortex_flatbuffers::footer::Postscript<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::footer::Postscript<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::footer::Postscript<'_> -pub fn vortex_flatbuffers::footer::Postscript<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::footer::Postscript<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::footer::Postscript<'a> @@ -2104,7 +2104,7 @@ pub fn vortex_flatbuffers::footer::Postscript<'a>::clone(&self) -> vortex_flatbu impl<'a> core::cmp::PartialEq for vortex_flatbuffers::footer::Postscript<'a> -pub fn vortex_flatbuffers::footer::Postscript<'a>::eq(&self, other: &vortex_flatbuffers::footer::Postscript<'a>) -> bool +pub fn vortex_flatbuffers::footer::Postscript<'a>::eq(&self, &vortex_flatbuffers::footer::Postscript<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::footer::Postscript<'a> @@ -2114,7 +2114,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::footer::Postscr pub type vortex_flatbuffers::footer::Postscript<'a>::Inner = vortex_flatbuffers::footer::Postscript<'a> -pub unsafe fn vortex_flatbuffers::footer::Postscript<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::footer::Postscript<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::footer::PostscriptArgs<'a> @@ -2134,17 +2134,17 @@ pub struct vortex_flatbuffers::footer::PostscriptBuilder<'a: 'b, 'b, A: flatbuff impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::footer::PostscriptBuilder<'a, 'b, A> -pub fn vortex_flatbuffers::footer::PostscriptBuilder<'a, 'b, A>::add_dtype(&mut self, dtype: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::footer::PostscriptBuilder<'a, 'b, A>::add_dtype(&mut self, flatbuffers::primitives::WIPOffset>) -pub fn vortex_flatbuffers::footer::PostscriptBuilder<'a, 'b, A>::add_footer(&mut self, footer: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::footer::PostscriptBuilder<'a, 'b, A>::add_footer(&mut self, flatbuffers::primitives::WIPOffset>) -pub fn vortex_flatbuffers::footer::PostscriptBuilder<'a, 'b, A>::add_layout(&mut self, layout: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::footer::PostscriptBuilder<'a, 'b, A>::add_layout(&mut self, flatbuffers::primitives::WIPOffset>) -pub fn vortex_flatbuffers::footer::PostscriptBuilder<'a, 'b, A>::add_statistics(&mut self, statistics: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::footer::PostscriptBuilder<'a, 'b, A>::add_statistics(&mut self, flatbuffers::primitives::WIPOffset>) pub fn vortex_flatbuffers::footer::PostscriptBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::footer::PostscriptBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::footer::PostscriptBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::footer::PostscriptBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::footer::PostscriptBuilder<'a, 'b, A> pub struct vortex_flatbuffers::footer::PostscriptSegment<'a> @@ -2168,9 +2168,9 @@ pub fn vortex_flatbuffers::footer::PostscriptSegment<'a>::_encryption(&self) -> pub fn vortex_flatbuffers::footer::PostscriptSegment<'a>::alignment_exponent(&self) -> u8 -pub fn vortex_flatbuffers::footer::PostscriptSegment<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, args: &'args vortex_flatbuffers::footer::PostscriptSegmentArgs<'args>) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::footer::PostscriptSegment<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::footer::PostscriptSegmentArgs<'args>) -> flatbuffers::primitives::WIPOffset> -pub unsafe fn vortex_flatbuffers::footer::PostscriptSegment<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::footer::PostscriptSegment<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self pub fn vortex_flatbuffers::footer::PostscriptSegment<'a>::length(&self) -> u32 @@ -2178,11 +2178,11 @@ pub fn vortex_flatbuffers::footer::PostscriptSegment<'a>::offset(&self) -> u64 impl core::fmt::Debug for vortex_flatbuffers::footer::PostscriptSegment<'_> -pub fn vortex_flatbuffers::footer::PostscriptSegment<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::footer::PostscriptSegment<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::footer::PostscriptSegment<'_> -pub fn vortex_flatbuffers::footer::PostscriptSegment<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::footer::PostscriptSegment<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::footer::PostscriptSegment<'a> @@ -2190,7 +2190,7 @@ pub fn vortex_flatbuffers::footer::PostscriptSegment<'a>::clone(&self) -> vortex impl<'a> core::cmp::PartialEq for vortex_flatbuffers::footer::PostscriptSegment<'a> -pub fn vortex_flatbuffers::footer::PostscriptSegment<'a>::eq(&self, other: &vortex_flatbuffers::footer::PostscriptSegment<'a>) -> bool +pub fn vortex_flatbuffers::footer::PostscriptSegment<'a>::eq(&self, &vortex_flatbuffers::footer::PostscriptSegment<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::footer::PostscriptSegment<'a> @@ -2200,7 +2200,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::footer::Postscr pub type vortex_flatbuffers::footer::PostscriptSegment<'a>::Inner = vortex_flatbuffers::footer::PostscriptSegment<'a> -pub unsafe fn vortex_flatbuffers::footer::PostscriptSegment<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::footer::PostscriptSegment<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::footer::PostscriptSegmentArgs<'a> @@ -2222,19 +2222,19 @@ pub struct vortex_flatbuffers::footer::PostscriptSegmentBuilder<'a: 'b, 'b, A: f impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::footer::PostscriptSegmentBuilder<'a, 'b, A> -pub fn vortex_flatbuffers::footer::PostscriptSegmentBuilder<'a, 'b, A>::add__compression(&mut self, _compression: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::footer::PostscriptSegmentBuilder<'a, 'b, A>::add__compression(&mut self, flatbuffers::primitives::WIPOffset>) -pub fn vortex_flatbuffers::footer::PostscriptSegmentBuilder<'a, 'b, A>::add__encryption(&mut self, _encryption: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::footer::PostscriptSegmentBuilder<'a, 'b, A>::add__encryption(&mut self, flatbuffers::primitives::WIPOffset>) -pub fn vortex_flatbuffers::footer::PostscriptSegmentBuilder<'a, 'b, A>::add_alignment_exponent(&mut self, alignment_exponent: u8) +pub fn vortex_flatbuffers::footer::PostscriptSegmentBuilder<'a, 'b, A>::add_alignment_exponent(&mut self, u8) -pub fn vortex_flatbuffers::footer::PostscriptSegmentBuilder<'a, 'b, A>::add_length(&mut self, length: u32) +pub fn vortex_flatbuffers::footer::PostscriptSegmentBuilder<'a, 'b, A>::add_length(&mut self, u32) -pub fn vortex_flatbuffers::footer::PostscriptSegmentBuilder<'a, 'b, A>::add_offset(&mut self, offset: u64) +pub fn vortex_flatbuffers::footer::PostscriptSegmentBuilder<'a, 'b, A>::add_offset(&mut self, u64) pub fn vortex_flatbuffers::footer::PostscriptSegmentBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::footer::PostscriptSegmentBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::footer::PostscriptSegmentBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::footer::PostscriptSegmentBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::footer::PostscriptSegmentBuilder<'a, 'b, A> #[repr(transparent)] pub struct vortex_flatbuffers::footer::SegmentSpec(pub [u8; 16]) @@ -2248,19 +2248,19 @@ pub fn vortex_flatbuffers::footer::SegmentSpec::alignment_exponent(&self) -> u8 pub fn vortex_flatbuffers::footer::SegmentSpec::length(&self) -> u32 -pub fn vortex_flatbuffers::footer::SegmentSpec::new(offset: u64, length: u32, alignment_exponent: u8, _compression: u8, _encryption: u16) -> Self +pub fn vortex_flatbuffers::footer::SegmentSpec::new(u64, u32, u8, u8, u16) -> Self pub fn vortex_flatbuffers::footer::SegmentSpec::offset(&self) -> u64 -pub fn vortex_flatbuffers::footer::SegmentSpec::set__compression(&mut self, x: u8) +pub fn vortex_flatbuffers::footer::SegmentSpec::set__compression(&mut self, u8) -pub fn vortex_flatbuffers::footer::SegmentSpec::set__encryption(&mut self, x: u16) +pub fn vortex_flatbuffers::footer::SegmentSpec::set__encryption(&mut self, u16) -pub fn vortex_flatbuffers::footer::SegmentSpec::set_alignment_exponent(&mut self, x: u8) +pub fn vortex_flatbuffers::footer::SegmentSpec::set_alignment_exponent(&mut self, u8) -pub fn vortex_flatbuffers::footer::SegmentSpec::set_length(&mut self, x: u32) +pub fn vortex_flatbuffers::footer::SegmentSpec::set_length(&mut self, u32) -pub fn vortex_flatbuffers::footer::SegmentSpec::set_offset(&mut self, x: u64) +pub fn vortex_flatbuffers::footer::SegmentSpec::set_offset(&mut self, u64) impl core::clone::Clone for vortex_flatbuffers::footer::SegmentSpec @@ -2268,7 +2268,7 @@ pub fn vortex_flatbuffers::footer::SegmentSpec::clone(&self) -> vortex_flatbuffe impl core::cmp::PartialEq for vortex_flatbuffers::footer::SegmentSpec -pub fn vortex_flatbuffers::footer::SegmentSpec::eq(&self, other: &vortex_flatbuffers::footer::SegmentSpec) -> bool +pub fn vortex_flatbuffers::footer::SegmentSpec::eq(&self, &vortex_flatbuffers::footer::SegmentSpec) -> bool impl core::default::Default for vortex_flatbuffers::footer::SegmentSpec @@ -2276,7 +2276,7 @@ pub fn vortex_flatbuffers::footer::SegmentSpec::default() -> Self impl core::fmt::Debug for vortex_flatbuffers::footer::SegmentSpec -pub fn vortex_flatbuffers::footer::SegmentSpec::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::footer::SegmentSpec::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_flatbuffers::footer::SegmentSpec @@ -2288,17 +2288,17 @@ impl<'a> flatbuffers::follow::Follow<'a> for &'a vortex_flatbuffers::footer::Seg pub type &'a vortex_flatbuffers::footer::SegmentSpec::Inner = &'a vortex_flatbuffers::footer::SegmentSpec -pub unsafe fn &'a vortex_flatbuffers::footer::SegmentSpec::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn &'a vortex_flatbuffers::footer::SegmentSpec::follow(&'a [u8], usize) -> Self::Inner impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::footer::SegmentSpec pub type vortex_flatbuffers::footer::SegmentSpec::Inner = &'a vortex_flatbuffers::footer::SegmentSpec -pub unsafe fn vortex_flatbuffers::footer::SegmentSpec::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::footer::SegmentSpec::follow(&'a [u8], usize) -> Self::Inner impl<'a> flatbuffers::verifier::Verifiable for vortex_flatbuffers::footer::SegmentSpec -pub fn vortex_flatbuffers::footer::SegmentSpec::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::footer::SegmentSpec::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'b> flatbuffers::push::Push for vortex_flatbuffers::footer::SegmentSpec @@ -2306,7 +2306,7 @@ pub type vortex_flatbuffers::footer::SegmentSpec::Output = vortex_flatbuffers::f pub fn vortex_flatbuffers::footer::SegmentSpec::alignment() -> flatbuffers::push::PushAlignment -pub unsafe fn vortex_flatbuffers::footer::SegmentSpec::push(&self, dst: &mut [u8], _written_len: usize) +pub unsafe fn vortex_flatbuffers::footer::SegmentSpec::push(&self, &mut [u8], usize) pub const vortex_flatbuffers::footer::ENUM_MAX_COMPRESSION_SCHEME: u8 @@ -2314,21 +2314,21 @@ pub const vortex_flatbuffers::footer::ENUM_MIN_COMPRESSION_SCHEME: u8 pub const vortex_flatbuffers::footer::ENUM_VALUES_COMPRESSION_SCHEME: [vortex_flatbuffers::footer::CompressionScheme; 4] -pub fn vortex_flatbuffers::footer::finish_postscript_buffer<'a, 'b, A: flatbuffers::builder::Allocator + 'a>(fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>, root: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::footer::finish_postscript_buffer<'a, 'b, A: flatbuffers::builder::Allocator + 'a>(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>, flatbuffers::primitives::WIPOffset>) -pub fn vortex_flatbuffers::footer::finish_size_prefixed_postscript_buffer<'a, 'b, A: flatbuffers::builder::Allocator + 'a>(fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>, root: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::footer::finish_size_prefixed_postscript_buffer<'a, 'b, A: flatbuffers::builder::Allocator + 'a>(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>, flatbuffers::primitives::WIPOffset>) -pub fn vortex_flatbuffers::footer::root_as_postscript(buf: &[u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::footer::root_as_postscript(&[u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> -pub unsafe fn vortex_flatbuffers::footer::root_as_postscript_unchecked(buf: &[u8]) -> vortex_flatbuffers::footer::Postscript<'_> +pub unsafe fn vortex_flatbuffers::footer::root_as_postscript_unchecked(&[u8]) -> vortex_flatbuffers::footer::Postscript<'_> -pub fn vortex_flatbuffers::footer::root_as_postscript_with_opts<'b, 'o>(opts: &'o flatbuffers::verifier::VerifierOptions, buf: &'b [u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::footer::root_as_postscript_with_opts<'b, 'o>(&'o flatbuffers::verifier::VerifierOptions, &'b [u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> -pub fn vortex_flatbuffers::footer::size_prefixed_root_as_postscript(buf: &[u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::footer::size_prefixed_root_as_postscript(&[u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> -pub unsafe fn vortex_flatbuffers::footer::size_prefixed_root_as_postscript_unchecked(buf: &[u8]) -> vortex_flatbuffers::footer::Postscript<'_> +pub unsafe fn vortex_flatbuffers::footer::size_prefixed_root_as_postscript_unchecked(&[u8]) -> vortex_flatbuffers::footer::Postscript<'_> -pub fn vortex_flatbuffers::footer::size_prefixed_root_as_postscript_with_opts<'b, 'o>(opts: &'o flatbuffers::verifier::VerifierOptions, buf: &'b [u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::footer::size_prefixed_root_as_postscript_with_opts<'b, 'o>(&'o flatbuffers::verifier::VerifierOptions, &'b [u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> pub mod vortex_flatbuffers::layout @@ -2352,11 +2352,11 @@ pub const vortex_flatbuffers::layout::Layout<'a>::VT_SEGMENTS: flatbuffers::prim pub fn vortex_flatbuffers::layout::Layout<'a>::children(&self) -> core::option::Option>>> -pub fn vortex_flatbuffers::layout::Layout<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, args: &'args vortex_flatbuffers::layout::LayoutArgs<'args>) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::layout::Layout<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::layout::LayoutArgs<'args>) -> flatbuffers::primitives::WIPOffset> pub fn vortex_flatbuffers::layout::Layout<'a>::encoding(&self) -> u16 -pub unsafe fn vortex_flatbuffers::layout::Layout<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::layout::Layout<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self pub fn vortex_flatbuffers::layout::Layout<'a>::metadata(&self) -> core::option::Option> @@ -2366,11 +2366,11 @@ pub fn vortex_flatbuffers::layout::Layout<'a>::segments(&self) -> core::option:: impl core::fmt::Debug for vortex_flatbuffers::layout::Layout<'_> -pub fn vortex_flatbuffers::layout::Layout<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::layout::Layout<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::layout::Layout<'_> -pub fn vortex_flatbuffers::layout::Layout<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::layout::Layout<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::layout::Layout<'a> @@ -2378,7 +2378,7 @@ pub fn vortex_flatbuffers::layout::Layout<'a>::clone(&self) -> vortex_flatbuffer impl<'a> core::cmp::PartialEq for vortex_flatbuffers::layout::Layout<'a> -pub fn vortex_flatbuffers::layout::Layout<'a>::eq(&self, other: &vortex_flatbuffers::layout::Layout<'a>) -> bool +pub fn vortex_flatbuffers::layout::Layout<'a>::eq(&self, &vortex_flatbuffers::layout::Layout<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::layout::Layout<'a> @@ -2388,7 +2388,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::layout::Layout< pub type vortex_flatbuffers::layout::Layout<'a>::Inner = vortex_flatbuffers::layout::Layout<'a> -pub unsafe fn vortex_flatbuffers::layout::Layout<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::layout::Layout<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::layout::LayoutArgs<'a> @@ -2410,35 +2410,35 @@ pub struct vortex_flatbuffers::layout::LayoutBuilder<'a: 'b, 'b, A: flatbuffers: impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::layout::LayoutBuilder<'a, 'b, A> -pub fn vortex_flatbuffers::layout::LayoutBuilder<'a, 'b, A>::add_children(&mut self, children: flatbuffers::primitives::WIPOffset>>>) +pub fn vortex_flatbuffers::layout::LayoutBuilder<'a, 'b, A>::add_children(&mut self, flatbuffers::primitives::WIPOffset>>>) -pub fn vortex_flatbuffers::layout::LayoutBuilder<'a, 'b, A>::add_encoding(&mut self, encoding: u16) +pub fn vortex_flatbuffers::layout::LayoutBuilder<'a, 'b, A>::add_encoding(&mut self, u16) -pub fn vortex_flatbuffers::layout::LayoutBuilder<'a, 'b, A>::add_metadata(&mut self, metadata: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::layout::LayoutBuilder<'a, 'b, A>::add_metadata(&mut self, flatbuffers::primitives::WIPOffset>) -pub fn vortex_flatbuffers::layout::LayoutBuilder<'a, 'b, A>::add_row_count(&mut self, row_count: u64) +pub fn vortex_flatbuffers::layout::LayoutBuilder<'a, 'b, A>::add_row_count(&mut self, u64) -pub fn vortex_flatbuffers::layout::LayoutBuilder<'a, 'b, A>::add_segments(&mut self, segments: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::layout::LayoutBuilder<'a, 'b, A>::add_segments(&mut self, flatbuffers::primitives::WIPOffset>) pub fn vortex_flatbuffers::layout::LayoutBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::layout::LayoutBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::layout::LayoutBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::layout::LayoutBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::layout::LayoutBuilder<'a, 'b, A> -pub fn vortex_flatbuffers::layout::finish_layout_buffer<'a, 'b, A: flatbuffers::builder::Allocator + 'a>(fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>, root: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::layout::finish_layout_buffer<'a, 'b, A: flatbuffers::builder::Allocator + 'a>(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>, flatbuffers::primitives::WIPOffset>) -pub fn vortex_flatbuffers::layout::finish_size_prefixed_layout_buffer<'a, 'b, A: flatbuffers::builder::Allocator + 'a>(fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>, root: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::layout::finish_size_prefixed_layout_buffer<'a, 'b, A: flatbuffers::builder::Allocator + 'a>(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>, flatbuffers::primitives::WIPOffset>) -pub fn vortex_flatbuffers::layout::root_as_layout(buf: &[u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::layout::root_as_layout(&[u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> -pub unsafe fn vortex_flatbuffers::layout::root_as_layout_unchecked(buf: &[u8]) -> vortex_flatbuffers::layout::Layout<'_> +pub unsafe fn vortex_flatbuffers::layout::root_as_layout_unchecked(&[u8]) -> vortex_flatbuffers::layout::Layout<'_> -pub fn vortex_flatbuffers::layout::root_as_layout_with_opts<'b, 'o>(opts: &'o flatbuffers::verifier::VerifierOptions, buf: &'b [u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::layout::root_as_layout_with_opts<'b, 'o>(&'o flatbuffers::verifier::VerifierOptions, &'b [u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> -pub fn vortex_flatbuffers::layout::size_prefixed_root_as_layout(buf: &[u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::layout::size_prefixed_root_as_layout(&[u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> -pub unsafe fn vortex_flatbuffers::layout::size_prefixed_root_as_layout_unchecked(buf: &[u8]) -> vortex_flatbuffers::layout::Layout<'_> +pub unsafe fn vortex_flatbuffers::layout::size_prefixed_root_as_layout_unchecked(&[u8]) -> vortex_flatbuffers::layout::Layout<'_> -pub fn vortex_flatbuffers::layout::size_prefixed_root_as_layout_with_opts<'b, 'o>(opts: &'o flatbuffers::verifier::VerifierOptions, buf: &'b [u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::layout::size_prefixed_root_as_layout_with_opts<'b, 'o>(&'o flatbuffers::verifier::VerifierOptions, &'b [u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> pub mod vortex_flatbuffers::message @@ -2460,21 +2460,21 @@ pub const vortex_flatbuffers::message::ArrayMessage<'a>::VT_ENCODINGS: flatbuffe pub const vortex_flatbuffers::message::ArrayMessage<'a>::VT_ROW_COUNT: flatbuffers::primitives::VOffsetT -pub fn vortex_flatbuffers::message::ArrayMessage<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, args: &'args vortex_flatbuffers::message::ArrayMessageArgs<'args>) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::message::ArrayMessage<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::message::ArrayMessageArgs<'args>) -> flatbuffers::primitives::WIPOffset> pub fn vortex_flatbuffers::message::ArrayMessage<'a>::encodings(&self) -> core::option::Option>> -pub unsafe fn vortex_flatbuffers::message::ArrayMessage<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::message::ArrayMessage<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self pub fn vortex_flatbuffers::message::ArrayMessage<'a>::row_count(&self) -> u32 impl core::fmt::Debug for vortex_flatbuffers::message::ArrayMessage<'_> -pub fn vortex_flatbuffers::message::ArrayMessage<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::message::ArrayMessage<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::message::ArrayMessage<'_> -pub fn vortex_flatbuffers::message::ArrayMessage<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::message::ArrayMessage<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::message::ArrayMessage<'a> @@ -2482,7 +2482,7 @@ pub fn vortex_flatbuffers::message::ArrayMessage<'a>::clone(&self) -> vortex_fla impl<'a> core::cmp::PartialEq for vortex_flatbuffers::message::ArrayMessage<'a> -pub fn vortex_flatbuffers::message::ArrayMessage<'a>::eq(&self, other: &vortex_flatbuffers::message::ArrayMessage<'a>) -> bool +pub fn vortex_flatbuffers::message::ArrayMessage<'a>::eq(&self, &vortex_flatbuffers::message::ArrayMessage<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::message::ArrayMessage<'a> @@ -2492,7 +2492,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::message::ArrayM pub type vortex_flatbuffers::message::ArrayMessage<'a>::Inner = vortex_flatbuffers::message::ArrayMessage<'a> -pub unsafe fn vortex_flatbuffers::message::ArrayMessage<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::message::ArrayMessage<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::message::ArrayMessageArgs<'a> @@ -2508,13 +2508,13 @@ pub struct vortex_flatbuffers::message::ArrayMessageBuilder<'a: 'b, 'b, A: flatb impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::message::ArrayMessageBuilder<'a, 'b, A> -pub fn vortex_flatbuffers::message::ArrayMessageBuilder<'a, 'b, A>::add_encodings(&mut self, encodings: flatbuffers::primitives::WIPOffset>>) +pub fn vortex_flatbuffers::message::ArrayMessageBuilder<'a, 'b, A>::add_encodings(&mut self, flatbuffers::primitives::WIPOffset>>) -pub fn vortex_flatbuffers::message::ArrayMessageBuilder<'a, 'b, A>::add_row_count(&mut self, row_count: u32) +pub fn vortex_flatbuffers::message::ArrayMessageBuilder<'a, 'b, A>::add_row_count(&mut self, u32) pub fn vortex_flatbuffers::message::ArrayMessageBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::message::ArrayMessageBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::message::ArrayMessageBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::message::ArrayMessageBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::message::ArrayMessageBuilder<'a, 'b, A> pub struct vortex_flatbuffers::message::BufferMessage<'a> @@ -2526,17 +2526,17 @@ pub const vortex_flatbuffers::message::BufferMessage<'a>::VT_ALIGNMENT_EXPONENT: pub fn vortex_flatbuffers::message::BufferMessage<'a>::alignment_exponent(&self) -> u8 -pub fn vortex_flatbuffers::message::BufferMessage<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, args: &'args vortex_flatbuffers::message::BufferMessageArgs) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::message::BufferMessage<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::message::BufferMessageArgs) -> flatbuffers::primitives::WIPOffset> -pub unsafe fn vortex_flatbuffers::message::BufferMessage<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::message::BufferMessage<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self impl core::fmt::Debug for vortex_flatbuffers::message::BufferMessage<'_> -pub fn vortex_flatbuffers::message::BufferMessage<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::message::BufferMessage<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::message::BufferMessage<'_> -pub fn vortex_flatbuffers::message::BufferMessage<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::message::BufferMessage<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::message::BufferMessage<'a> @@ -2544,7 +2544,7 @@ pub fn vortex_flatbuffers::message::BufferMessage<'a>::clone(&self) -> vortex_fl impl<'a> core::cmp::PartialEq for vortex_flatbuffers::message::BufferMessage<'a> -pub fn vortex_flatbuffers::message::BufferMessage<'a>::eq(&self, other: &vortex_flatbuffers::message::BufferMessage<'a>) -> bool +pub fn vortex_flatbuffers::message::BufferMessage<'a>::eq(&self, &vortex_flatbuffers::message::BufferMessage<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::message::BufferMessage<'a> @@ -2554,7 +2554,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::message::Buffer pub type vortex_flatbuffers::message::BufferMessage<'a>::Inner = vortex_flatbuffers::message::BufferMessage<'a> -pub unsafe fn vortex_flatbuffers::message::BufferMessage<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::message::BufferMessage<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::message::BufferMessageArgs @@ -2568,11 +2568,11 @@ pub struct vortex_flatbuffers::message::BufferMessageBuilder<'a: 'b, 'b, A: flat impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::message::BufferMessageBuilder<'a, 'b, A> -pub fn vortex_flatbuffers::message::BufferMessageBuilder<'a, 'b, A>::add_alignment_exponent(&mut self, alignment_exponent: u8) +pub fn vortex_flatbuffers::message::BufferMessageBuilder<'a, 'b, A>::add_alignment_exponent(&mut self, u8) pub fn vortex_flatbuffers::message::BufferMessageBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::message::BufferMessageBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::message::BufferMessageBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::message::BufferMessageBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::message::BufferMessageBuilder<'a, 'b, A> pub struct vortex_flatbuffers::message::DTypeMessage<'a> @@ -2580,17 +2580,17 @@ pub vortex_flatbuffers::message::DTypeMessage::_tab: flatbuffers::table::Table<' impl<'a> vortex_flatbuffers::message::DTypeMessage<'a> -pub fn vortex_flatbuffers::message::DTypeMessage<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, _args: &'args vortex_flatbuffers::message::DTypeMessageArgs) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::message::DTypeMessage<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::message::DTypeMessageArgs) -> flatbuffers::primitives::WIPOffset> -pub unsafe fn vortex_flatbuffers::message::DTypeMessage<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::message::DTypeMessage<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self impl core::fmt::Debug for vortex_flatbuffers::message::DTypeMessage<'_> -pub fn vortex_flatbuffers::message::DTypeMessage<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::message::DTypeMessage<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::message::DTypeMessage<'_> -pub fn vortex_flatbuffers::message::DTypeMessage<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::message::DTypeMessage<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::message::DTypeMessage<'a> @@ -2598,7 +2598,7 @@ pub fn vortex_flatbuffers::message::DTypeMessage<'a>::clone(&self) -> vortex_fla impl<'a> core::cmp::PartialEq for vortex_flatbuffers::message::DTypeMessage<'a> -pub fn vortex_flatbuffers::message::DTypeMessage<'a>::eq(&self, other: &vortex_flatbuffers::message::DTypeMessage<'a>) -> bool +pub fn vortex_flatbuffers::message::DTypeMessage<'a>::eq(&self, &vortex_flatbuffers::message::DTypeMessage<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::message::DTypeMessage<'a> @@ -2608,7 +2608,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::message::DTypeM pub type vortex_flatbuffers::message::DTypeMessage<'a>::Inner = vortex_flatbuffers::message::DTypeMessage<'a> -pub unsafe fn vortex_flatbuffers::message::DTypeMessage<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::message::DTypeMessage<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::message::DTypeMessageArgs @@ -2622,7 +2622,7 @@ impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::me pub fn vortex_flatbuffers::message::DTypeMessageBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::message::DTypeMessageBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::message::DTypeMessageBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::message::DTypeMessageBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::message::DTypeMessageBuilder<'a, 'b, A> pub struct vortex_flatbuffers::message::Message<'a> @@ -2640,7 +2640,7 @@ pub const vortex_flatbuffers::message::Message<'a>::VT_VERSION: flatbuffers::pri pub fn vortex_flatbuffers::message::Message<'a>::body_size(&self) -> u64 -pub fn vortex_flatbuffers::message::Message<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(_fbb: &'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, args: &'args vortex_flatbuffers::message::MessageArgs) -> flatbuffers::primitives::WIPOffset> +pub fn vortex_flatbuffers::message::Message<'a>::create<'bldr: 'args, 'args: 'mut_bldr, 'mut_bldr, A: flatbuffers::builder::Allocator + 'bldr>(&'mut_bldr mut flatbuffers::builder::FlatBufferBuilder<'bldr, A>, &'args vortex_flatbuffers::message::MessageArgs) -> flatbuffers::primitives::WIPOffset> pub fn vortex_flatbuffers::message::Message<'a>::header(&self) -> core::option::Option> @@ -2652,17 +2652,17 @@ pub fn vortex_flatbuffers::message::Message<'a>::header_as_dtype_message(&self) pub fn vortex_flatbuffers::message::Message<'a>::header_type(&self) -> vortex_flatbuffers::message::MessageHeader -pub unsafe fn vortex_flatbuffers::message::Message<'a>::init_from_table(table: flatbuffers::table::Table<'a>) -> Self +pub unsafe fn vortex_flatbuffers::message::Message<'a>::init_from_table(flatbuffers::table::Table<'a>) -> Self pub fn vortex_flatbuffers::message::Message<'a>::version(&self) -> vortex_flatbuffers::message::MessageVersion impl core::fmt::Debug for vortex_flatbuffers::message::Message<'_> -pub fn vortex_flatbuffers::message::Message<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::message::Message<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl flatbuffers::verifier::Verifiable for vortex_flatbuffers::message::Message<'_> -pub fn vortex_flatbuffers::message::Message<'_>::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::message::Message<'_>::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> impl<'a> core::clone::Clone for vortex_flatbuffers::message::Message<'a> @@ -2670,7 +2670,7 @@ pub fn vortex_flatbuffers::message::Message<'a>::clone(&self) -> vortex_flatbuff impl<'a> core::cmp::PartialEq for vortex_flatbuffers::message::Message<'a> -pub fn vortex_flatbuffers::message::Message<'a>::eq(&self, other: &vortex_flatbuffers::message::Message<'a>) -> bool +pub fn vortex_flatbuffers::message::Message<'a>::eq(&self, &vortex_flatbuffers::message::Message<'a>) -> bool impl<'a> core::marker::Copy for vortex_flatbuffers::message::Message<'a> @@ -2680,7 +2680,7 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::message::Messag pub type vortex_flatbuffers::message::Message<'a>::Inner = vortex_flatbuffers::message::Message<'a> -pub unsafe fn vortex_flatbuffers::message::Message<'a>::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::message::Message<'a>::follow(&'a [u8], usize) -> Self::Inner pub struct vortex_flatbuffers::message::MessageArgs @@ -2700,17 +2700,17 @@ pub struct vortex_flatbuffers::message::MessageBuilder<'a: 'b, 'b, A: flatbuffer impl<'a: 'b, 'b, A: flatbuffers::builder::Allocator + 'a> vortex_flatbuffers::message::MessageBuilder<'a, 'b, A> -pub fn vortex_flatbuffers::message::MessageBuilder<'a, 'b, A>::add_body_size(&mut self, body_size: u64) +pub fn vortex_flatbuffers::message::MessageBuilder<'a, 'b, A>::add_body_size(&mut self, u64) -pub fn vortex_flatbuffers::message::MessageBuilder<'a, 'b, A>::add_header(&mut self, header: flatbuffers::primitives::WIPOffset) +pub fn vortex_flatbuffers::message::MessageBuilder<'a, 'b, A>::add_header(&mut self, flatbuffers::primitives::WIPOffset) -pub fn vortex_flatbuffers::message::MessageBuilder<'a, 'b, A>::add_header_type(&mut self, header_type: vortex_flatbuffers::message::MessageHeader) +pub fn vortex_flatbuffers::message::MessageBuilder<'a, 'b, A>::add_header_type(&mut self, vortex_flatbuffers::message::MessageHeader) -pub fn vortex_flatbuffers::message::MessageBuilder<'a, 'b, A>::add_version(&mut self, version: vortex_flatbuffers::message::MessageVersion) +pub fn vortex_flatbuffers::message::MessageBuilder<'a, 'b, A>::add_version(&mut self, vortex_flatbuffers::message::MessageVersion) pub fn vortex_flatbuffers::message::MessageBuilder<'a, 'b, A>::finish(self) -> flatbuffers::primitives::WIPOffset> -pub fn vortex_flatbuffers::message::MessageBuilder<'a, 'b, A>::new(_fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::message::MessageBuilder<'a, 'b, A> +pub fn vortex_flatbuffers::message::MessageBuilder<'a, 'b, A>::new(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>) -> vortex_flatbuffers::message::MessageBuilder<'a, 'b, A> #[repr(transparent)] pub struct vortex_flatbuffers::message::MessageHeader(pub u8) @@ -2740,15 +2740,15 @@ impl core::cmp::Eq for vortex_flatbuffers::message::MessageHeader impl core::cmp::Ord for vortex_flatbuffers::message::MessageHeader -pub fn vortex_flatbuffers::message::MessageHeader::cmp(&self, other: &vortex_flatbuffers::message::MessageHeader) -> core::cmp::Ordering +pub fn vortex_flatbuffers::message::MessageHeader::cmp(&self, &vortex_flatbuffers::message::MessageHeader) -> core::cmp::Ordering impl core::cmp::PartialEq for vortex_flatbuffers::message::MessageHeader -pub fn vortex_flatbuffers::message::MessageHeader::eq(&self, other: &vortex_flatbuffers::message::MessageHeader) -> bool +pub fn vortex_flatbuffers::message::MessageHeader::eq(&self, &vortex_flatbuffers::message::MessageHeader) -> bool impl core::cmp::PartialOrd for vortex_flatbuffers::message::MessageHeader -pub fn vortex_flatbuffers::message::MessageHeader::partial_cmp(&self, other: &vortex_flatbuffers::message::MessageHeader) -> core::option::Option +pub fn vortex_flatbuffers::message::MessageHeader::partial_cmp(&self, &vortex_flatbuffers::message::MessageHeader) -> core::option::Option impl core::default::Default for vortex_flatbuffers::message::MessageHeader @@ -2756,11 +2756,11 @@ pub fn vortex_flatbuffers::message::MessageHeader::default() -> vortex_flatbuffe impl core::fmt::Debug for vortex_flatbuffers::message::MessageHeader -pub fn vortex_flatbuffers::message::MessageHeader::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::message::MessageHeader::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_flatbuffers::message::MessageHeader -pub fn vortex_flatbuffers::message::MessageHeader::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_flatbuffers::message::MessageHeader::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_flatbuffers::message::MessageHeader @@ -2770,7 +2770,7 @@ impl flatbuffers::endian_scalar::EndianScalar for vortex_flatbuffers::message::M pub type vortex_flatbuffers::message::MessageHeader::Scalar = u8 -pub fn vortex_flatbuffers::message::MessageHeader::from_little_endian(v: u8) -> Self +pub fn vortex_flatbuffers::message::MessageHeader::from_little_endian(u8) -> Self pub fn vortex_flatbuffers::message::MessageHeader::to_little_endian(self) -> u8 @@ -2778,7 +2778,7 @@ impl flatbuffers::push::Push for vortex_flatbuffers::message::MessageHeader pub type vortex_flatbuffers::message::MessageHeader::Output = vortex_flatbuffers::message::MessageHeader -pub unsafe fn vortex_flatbuffers::message::MessageHeader::push(&self, dst: &mut [u8], _written_len: usize) +pub unsafe fn vortex_flatbuffers::message::MessageHeader::push(&self, &mut [u8], usize) impl flatbuffers::verifier::SimpleToVerifyInSlice for vortex_flatbuffers::message::MessageHeader @@ -2786,11 +2786,11 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::message::Messag pub type vortex_flatbuffers::message::MessageHeader::Inner = vortex_flatbuffers::message::MessageHeader -pub unsafe fn vortex_flatbuffers::message::MessageHeader::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::message::MessageHeader::follow(&'a [u8], usize) -> Self::Inner impl<'a> flatbuffers::verifier::Verifiable for vortex_flatbuffers::message::MessageHeader -pub fn vortex_flatbuffers::message::MessageHeader::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::message::MessageHeader::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> pub struct vortex_flatbuffers::message::MessageHeaderUnionTableOffset @@ -2816,15 +2816,15 @@ impl core::cmp::Eq for vortex_flatbuffers::message::MessageVersion impl core::cmp::Ord for vortex_flatbuffers::message::MessageVersion -pub fn vortex_flatbuffers::message::MessageVersion::cmp(&self, other: &vortex_flatbuffers::message::MessageVersion) -> core::cmp::Ordering +pub fn vortex_flatbuffers::message::MessageVersion::cmp(&self, &vortex_flatbuffers::message::MessageVersion) -> core::cmp::Ordering impl core::cmp::PartialEq for vortex_flatbuffers::message::MessageVersion -pub fn vortex_flatbuffers::message::MessageVersion::eq(&self, other: &vortex_flatbuffers::message::MessageVersion) -> bool +pub fn vortex_flatbuffers::message::MessageVersion::eq(&self, &vortex_flatbuffers::message::MessageVersion) -> bool impl core::cmp::PartialOrd for vortex_flatbuffers::message::MessageVersion -pub fn vortex_flatbuffers::message::MessageVersion::partial_cmp(&self, other: &vortex_flatbuffers::message::MessageVersion) -> core::option::Option +pub fn vortex_flatbuffers::message::MessageVersion::partial_cmp(&self, &vortex_flatbuffers::message::MessageVersion) -> core::option::Option impl core::default::Default for vortex_flatbuffers::message::MessageVersion @@ -2832,11 +2832,11 @@ pub fn vortex_flatbuffers::message::MessageVersion::default() -> vortex_flatbuff impl core::fmt::Debug for vortex_flatbuffers::message::MessageVersion -pub fn vortex_flatbuffers::message::MessageVersion::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_flatbuffers::message::MessageVersion::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_flatbuffers::message::MessageVersion -pub fn vortex_flatbuffers::message::MessageVersion::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_flatbuffers::message::MessageVersion::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_flatbuffers::message::MessageVersion @@ -2846,7 +2846,7 @@ impl flatbuffers::endian_scalar::EndianScalar for vortex_flatbuffers::message::M pub type vortex_flatbuffers::message::MessageVersion::Scalar = u8 -pub fn vortex_flatbuffers::message::MessageVersion::from_little_endian(v: u8) -> Self +pub fn vortex_flatbuffers::message::MessageVersion::from_little_endian(u8) -> Self pub fn vortex_flatbuffers::message::MessageVersion::to_little_endian(self) -> u8 @@ -2854,7 +2854,7 @@ impl flatbuffers::push::Push for vortex_flatbuffers::message::MessageVersion pub type vortex_flatbuffers::message::MessageVersion::Output = vortex_flatbuffers::message::MessageVersion -pub unsafe fn vortex_flatbuffers::message::MessageVersion::push(&self, dst: &mut [u8], _written_len: usize) +pub unsafe fn vortex_flatbuffers::message::MessageVersion::push(&self, &mut [u8], usize) impl flatbuffers::verifier::SimpleToVerifyInSlice for vortex_flatbuffers::message::MessageVersion @@ -2862,11 +2862,11 @@ impl<'a> flatbuffers::follow::Follow<'a> for vortex_flatbuffers::message::Messag pub type vortex_flatbuffers::message::MessageVersion::Inner = vortex_flatbuffers::message::MessageVersion -pub unsafe fn vortex_flatbuffers::message::MessageVersion::follow(buf: &'a [u8], loc: usize) -> Self::Inner +pub unsafe fn vortex_flatbuffers::message::MessageVersion::follow(&'a [u8], usize) -> Self::Inner impl<'a> flatbuffers::verifier::Verifiable for vortex_flatbuffers::message::MessageVersion -pub fn vortex_flatbuffers::message::MessageVersion::run_verifier(v: &mut flatbuffers::verifier::Verifier<'_, '_>, pos: usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::message::MessageVersion::run_verifier(&mut flatbuffers::verifier::Verifier<'_, '_>, usize) -> core::result::Result<(), flatbuffers::verifier::InvalidFlatbuffer> pub const vortex_flatbuffers::message::ENUM_MAX_MESSAGE_HEADER: u8 @@ -2880,21 +2880,21 @@ pub const vortex_flatbuffers::message::ENUM_VALUES_MESSAGE_HEADER: [vortex_flatb pub const vortex_flatbuffers::message::ENUM_VALUES_MESSAGE_VERSION: [vortex_flatbuffers::message::MessageVersion; 1] -pub fn vortex_flatbuffers::message::finish_message_buffer<'a, 'b, A: flatbuffers::builder::Allocator + 'a>(fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>, root: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::message::finish_message_buffer<'a, 'b, A: flatbuffers::builder::Allocator + 'a>(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>, flatbuffers::primitives::WIPOffset>) -pub fn vortex_flatbuffers::message::finish_size_prefixed_message_buffer<'a, 'b, A: flatbuffers::builder::Allocator + 'a>(fbb: &'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>, root: flatbuffers::primitives::WIPOffset>) +pub fn vortex_flatbuffers::message::finish_size_prefixed_message_buffer<'a, 'b, A: flatbuffers::builder::Allocator + 'a>(&'b mut flatbuffers::builder::FlatBufferBuilder<'a, A>, flatbuffers::primitives::WIPOffset>) -pub fn vortex_flatbuffers::message::root_as_message(buf: &[u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::message::root_as_message(&[u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> -pub unsafe fn vortex_flatbuffers::message::root_as_message_unchecked(buf: &[u8]) -> vortex_flatbuffers::message::Message<'_> +pub unsafe fn vortex_flatbuffers::message::root_as_message_unchecked(&[u8]) -> vortex_flatbuffers::message::Message<'_> -pub fn vortex_flatbuffers::message::root_as_message_with_opts<'b, 'o>(opts: &'o flatbuffers::verifier::VerifierOptions, buf: &'b [u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::message::root_as_message_with_opts<'b, 'o>(&'o flatbuffers::verifier::VerifierOptions, &'b [u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> -pub fn vortex_flatbuffers::message::size_prefixed_root_as_message(buf: &[u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::message::size_prefixed_root_as_message(&[u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> -pub unsafe fn vortex_flatbuffers::message::size_prefixed_root_as_message_unchecked(buf: &[u8]) -> vortex_flatbuffers::message::Message<'_> +pub unsafe fn vortex_flatbuffers::message::size_prefixed_root_as_message_unchecked(&[u8]) -> vortex_flatbuffers::message::Message<'_> -pub fn vortex_flatbuffers::message::size_prefixed_root_as_message_with_opts<'b, 'o>(opts: &'o flatbuffers::verifier::VerifierOptions, buf: &'b [u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> +pub fn vortex_flatbuffers::message::size_prefixed_root_as_message_with_opts<'b, 'o>(&'o flatbuffers::verifier::VerifierOptions, &'b [u8]) -> core::result::Result, flatbuffers::verifier::InvalidFlatbuffer> pub trait vortex_flatbuffers::FlatBufferRoot @@ -2904,15 +2904,15 @@ pub type vortex_flatbuffers::ReadFlatBuffer::Error: core::convert::From: flatbuffers::verifier::Verifiable + flatbuffers::follow::Follow<'a> -pub fn vortex_flatbuffers::ReadFlatBuffer::read_flatbuffer<'buf>(fb: &>::Inner) -> core::result::Result +pub fn vortex_flatbuffers::ReadFlatBuffer::read_flatbuffer<'buf>(&>::Inner) -> core::result::Result -pub fn vortex_flatbuffers::ReadFlatBuffer::read_flatbuffer_bytes<'buf>(bytes: &'buf [u8]) -> core::result::Result where ::Source: 'buf +pub fn vortex_flatbuffers::ReadFlatBuffer::read_flatbuffer_bytes<'buf>(&'buf [u8]) -> core::result::Result where ::Source: 'buf pub trait vortex_flatbuffers::WriteFlatBuffer pub type vortex_flatbuffers::WriteFlatBuffer::Target<'a> -pub fn vortex_flatbuffers::WriteFlatBuffer::write_flatbuffer<'fb>(&self, fbb: &mut flatbuffers::builder::FlatBufferBuilder<'fb>) -> vortex_error::VortexResult> +pub fn vortex_flatbuffers::WriteFlatBuffer::write_flatbuffer<'fb>(&self, &mut flatbuffers::builder::FlatBufferBuilder<'fb>) -> vortex_error::VortexResult> pub trait vortex_flatbuffers::WriteFlatBufferExt: vortex_flatbuffers::WriteFlatBuffer + vortex_flatbuffers::FlatBufferRoot diff --git a/vortex-io/Cargo.toml b/vortex-io/Cargo.toml index da8053af471..ee1a5e76ffb 100644 --- a/vortex-io/Cargo.toml +++ b/vortex-io/Cargo.toml @@ -22,9 +22,7 @@ async-stream = { workspace = true } async-trait = { workspace = true } bytes = { workspace = true } futures = { workspace = true, features = ["std", "executor"] } -getrandom_v03 = { workspace = true } # Needed to pickup the "wasm_js" feature for wasm targets from the workspace configuration glob = { workspace = true } -handle = "1.0.2" kanal = { workspace = true } log = { workspace = true } object_store = { workspace = true, optional = true, features = ["fs"] } @@ -42,6 +40,7 @@ vortex-buffer = { workspace = true } vortex-error = { workspace = true } vortex-metrics = { workspace = true } vortex-session = { workspace = true } +vortex-utils = { workspace = true } [target.'cfg(unix)'.dependencies] custom-labels = { workspace = true } @@ -68,6 +67,3 @@ tokio = ["tokio/fs", "tokio/rt-multi-thread"] [lints] workspace = true - -[package.metadata.cargo-machete] -ignored = ["getrandom_v03"] diff --git a/vortex-io/public-api.lock b/vortex-io/public-api.lock index 08d74ffc830..0c5170828e7 100644 --- a/vortex-io/public-api.lock +++ b/vortex-io/public-api.lock @@ -6,15 +6,17 @@ pub struct vortex_io::compat::Compat impl vortex_io::compat::Compat -pub fn vortex_io::compat::Compat::new(inner: T) -> Self +pub fn vortex_io::compat::Compat::new(T) -> Self impl<'__pin, T> core::marker::Unpin for vortex_io::compat::Compat where pin_project_lite::__private::PinnedFieldsOf<__Origin<'__pin, T>>: core::marker::Unpin impl vortex_io::filesystem::FileSystem for vortex_io::compat::Compat -pub fn vortex_io::compat::Compat::list(&self, prefix: &str) -> futures_core::stream::BoxStream<'_, vortex_error::VortexResult> +pub fn vortex_io::compat::Compat::delete<'life0, 'life1, 'async_trait>(&'life0 self, &'life1 str) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait -pub fn vortex_io::compat::Compat::open_read<'life0, 'life1, 'async_trait>(&'life0 self, path: &'life1 str) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait +pub fn vortex_io::compat::Compat::list(&self, &str) -> futures_core::stream::BoxStream<'_, vortex_error::VortexResult> + +pub fn vortex_io::compat::Compat::open_read<'life0, 'life1, 'async_trait>(&'life0 self, &'life1 str) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait impl vortex_io::VortexReadAt for vortex_io::compat::Compat @@ -22,7 +24,7 @@ pub fn vortex_io::compat::Compat::coalesce_config(&self) -> core::option::Opt pub fn vortex_io::compat::Compat::concurrency(&self) -> usize -pub fn vortex_io::compat::Compat::read_at(&self, offset: u64, length: usize, alignment: vortex_buffer::alignment::Alignment) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> +pub fn vortex_io::compat::Compat::read_at(&self, u64, usize, vortex_buffer::alignment::Alignment) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> pub fn vortex_io::compat::Compat::size(&self) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> @@ -32,7 +34,7 @@ impl futures_core::stream::Stream for vortex_io pub type vortex_io::compat::Compat::Item = ::Item -pub fn vortex_io::compat::Compat::poll_next(self: core::pin::Pin<&mut Self>, cx: &mut core::task::wake::Context<'_>) -> core::task::poll::Poll> +pub fn vortex_io::compat::Compat::poll_next(core::pin::Pin<&mut Self>, &mut core::task::wake::Context<'_>) -> core::task::poll::Poll> pub fn vortex_io::compat::Compat::size_hint(&self) -> (usize, core::option::Option) @@ -42,13 +44,47 @@ pub fn vortex_io::compat::Compat::clone(&self) -> vortex_io::compat::Compat core::fmt::Debug for vortex_io::compat::Compat -pub fn vortex_io::compat::Compat::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_io::compat::Compat::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::future::future::Future for vortex_io::compat::Compat pub type vortex_io::compat::Compat::Output = ::Output -pub fn vortex_io::compat::Compat::poll(self: core::pin::Pin<&mut Self>, cx: &mut core::task::wake::Context<'_>) -> core::task::poll::Poll +pub fn vortex_io::compat::Compat::poll(core::pin::Pin<&mut Self>, &mut core::task::wake::Context<'_>) -> core::task::poll::Poll + +impl core::fmt::Display for vortex_io::compat::Compat + +pub fn vortex_io::compat::Compat::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +impl object_store::ObjectStore for vortex_io::compat::Compat + +pub fn vortex_io::compat::Compat::copy_opts<'life0, 'life1, 'life2, 'async_trait>(&'life0 self, &'life1 object_store::path::Path, &'life2 object_store::path::Path, object_store::CopyOptions) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait + +pub fn vortex_io::compat::Compat::delete_stream(&self, futures_core::stream::BoxStream<'static, object_store::Result>) -> futures_core::stream::BoxStream<'static, object_store::Result> + +pub fn vortex_io::compat::Compat::get_opts<'life0, 'life1, 'async_trait>(&'life0 self, &'life1 object_store::path::Path, object_store::GetOptions) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait + +pub fn vortex_io::compat::Compat::get_ranges<'life0, 'life1, 'life2, 'async_trait>(&'life0 self, &'life1 object_store::path::Path, &'life2 [core::ops::range::Range]) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait + +pub fn vortex_io::compat::Compat::list(&self, core::option::Option<&object_store::path::Path>) -> futures_core::stream::BoxStream<'static, object_store::Result> + +pub fn vortex_io::compat::Compat::list_with_delimiter<'life0, 'life1, 'async_trait>(&'life0 self, core::option::Option<&'life1 object_store::path::Path>) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait + +pub fn vortex_io::compat::Compat::list_with_offset(&self, core::option::Option<&object_store::path::Path>, &object_store::path::Path) -> futures_core::stream::BoxStream<'static, object_store::Result> + +pub fn vortex_io::compat::Compat::put_multipart_opts<'life0, 'life1, 'async_trait>(&'life0 self, &'life1 object_store::path::Path, object_store::PutMultipartOptions) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait + +pub fn vortex_io::compat::Compat::put_opts<'life0, 'life1, 'async_trait>(&'life0 self, &'life1 object_store::path::Path, object_store::payload::PutPayload, object_store::PutOptions) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait + +pub fn vortex_io::compat::Compat::rename_opts<'life0, 'life1, 'life2, 'async_trait>(&'life0 self, &'life1 object_store::path::Path, &'life2 object_store::path::Path, object_store::RenameOptions) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait + +impl object_store::upload::MultipartUpload for vortex_io::compat::Compat + +pub fn vortex_io::compat::Compat::abort<'life0, 'async_trait>(&'life0 mut self) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait + +pub fn vortex_io::compat::Compat::complete<'life0, 'async_trait>(&'life0 mut self) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait + +pub fn vortex_io::compat::Compat::put_part(&mut self, object_store::payload::PutPayload) -> object_store::upload::UploadPart impl core::ops::drop::Drop for vortex_io::compat::Compat @@ -60,7 +96,7 @@ pub fn vortex_io::compat::Compat::flush(&mut self) -> impl core::future::futu pub fn vortex_io::compat::Compat::shutdown(&mut self) -> impl core::future::future::Future> -pub fn vortex_io::compat::Compat::write_all(&mut self, buffer: B) -> impl core::future::future::Future> +pub fn vortex_io::compat::Compat::write_all(&mut self, B) -> impl core::future::future::Future> pub mod vortex_io::filesystem @@ -78,39 +114,45 @@ impl core::cmp::Eq for vortex_io::filesystem::FileListing impl core::cmp::Ord for vortex_io::filesystem::FileListing -pub fn vortex_io::filesystem::FileListing::cmp(&self, other: &vortex_io::filesystem::FileListing) -> core::cmp::Ordering +pub fn vortex_io::filesystem::FileListing::cmp(&self, &vortex_io::filesystem::FileListing) -> core::cmp::Ordering impl core::cmp::PartialEq for vortex_io::filesystem::FileListing -pub fn vortex_io::filesystem::FileListing::eq(&self, other: &vortex_io::filesystem::FileListing) -> bool +pub fn vortex_io::filesystem::FileListing::eq(&self, &vortex_io::filesystem::FileListing) -> bool impl core::cmp::PartialOrd for vortex_io::filesystem::FileListing -pub fn vortex_io::filesystem::FileListing::partial_cmp(&self, other: &vortex_io::filesystem::FileListing) -> core::option::Option +pub fn vortex_io::filesystem::FileListing::partial_cmp(&self, &vortex_io::filesystem::FileListing) -> core::option::Option impl core::fmt::Debug for vortex_io::filesystem::FileListing -pub fn vortex_io::filesystem::FileListing::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_io::filesystem::FileListing::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::StructuralPartialEq for vortex_io::filesystem::FileListing pub trait vortex_io::filesystem::FileSystem: core::fmt::Debug + core::marker::Send + core::marker::Sync -pub fn vortex_io::filesystem::FileSystem::list(&self, prefix: &str) -> futures_core::stream::BoxStream<'_, vortex_error::VortexResult> +pub fn vortex_io::filesystem::FileSystem::delete<'life0, 'life1, 'async_trait>(&'life0 self, &'life1 str) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait + +pub fn vortex_io::filesystem::FileSystem::list(&self, &str) -> futures_core::stream::BoxStream<'_, vortex_error::VortexResult> -pub fn vortex_io::filesystem::FileSystem::open_read<'life0, 'life1, 'async_trait>(&'life0 self, path: &'life1 str) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait +pub fn vortex_io::filesystem::FileSystem::open_read<'life0, 'life1, 'async_trait>(&'life0 self, &'life1 str) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait impl vortex_io::filesystem::FileSystem for vortex_io::object_store::ObjectStoreFileSystem -pub fn vortex_io::object_store::ObjectStoreFileSystem::list(&self, prefix: &str) -> futures_core::stream::BoxStream<'_, vortex_error::VortexResult> +pub fn vortex_io::object_store::ObjectStoreFileSystem::delete<'life0, 'life1, 'async_trait>(&'life0 self, &'life1 str) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait -pub fn vortex_io::object_store::ObjectStoreFileSystem::open_read<'life0, 'life1, 'async_trait>(&'life0 self, path: &'life1 str) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait +pub fn vortex_io::object_store::ObjectStoreFileSystem::list(&self, &str) -> futures_core::stream::BoxStream<'_, vortex_error::VortexResult> + +pub fn vortex_io::object_store::ObjectStoreFileSystem::open_read<'life0, 'life1, 'async_trait>(&'life0 self, &'life1 str) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait impl vortex_io::filesystem::FileSystem for vortex_io::compat::Compat -pub fn vortex_io::compat::Compat::list(&self, prefix: &str) -> futures_core::stream::BoxStream<'_, vortex_error::VortexResult> +pub fn vortex_io::compat::Compat::delete<'life0, 'life1, 'async_trait>(&'life0 self, &'life1 str) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait + +pub fn vortex_io::compat::Compat::list(&self, &str) -> futures_core::stream::BoxStream<'_, vortex_error::VortexResult> -pub fn vortex_io::compat::Compat::open_read<'life0, 'life1, 'async_trait>(&'life0 self, path: &'life1 str) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait +pub fn vortex_io::compat::Compat::open_read<'life0, 'life1, 'async_trait>(&'life0 self, &'life1 str) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait pub type vortex_io::filesystem::FileSystemRef = alloc::sync::Arc @@ -130,31 +172,33 @@ pub struct vortex_io::object_store::ObjectStoreFileSystem impl vortex_io::object_store::ObjectStoreFileSystem -pub fn vortex_io::object_store::ObjectStoreFileSystem::local(handle: vortex_io::runtime::Handle) -> Self +pub fn vortex_io::object_store::ObjectStoreFileSystem::local(vortex_io::runtime::Handle) -> Self -pub fn vortex_io::object_store::ObjectStoreFileSystem::new(store: alloc::sync::Arc, handle: vortex_io::runtime::Handle) -> Self +pub fn vortex_io::object_store::ObjectStoreFileSystem::new(alloc::sync::Arc, vortex_io::runtime::Handle) -> Self impl core::fmt::Debug for vortex_io::object_store::ObjectStoreFileSystem -pub fn vortex_io::object_store::ObjectStoreFileSystem::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_io::object_store::ObjectStoreFileSystem::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_io::filesystem::FileSystem for vortex_io::object_store::ObjectStoreFileSystem -pub fn vortex_io::object_store::ObjectStoreFileSystem::list(&self, prefix: &str) -> futures_core::stream::BoxStream<'_, vortex_error::VortexResult> +pub fn vortex_io::object_store::ObjectStoreFileSystem::delete<'life0, 'life1, 'async_trait>(&'life0 self, &'life1 str) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait + +pub fn vortex_io::object_store::ObjectStoreFileSystem::list(&self, &str) -> futures_core::stream::BoxStream<'_, vortex_error::VortexResult> -pub fn vortex_io::object_store::ObjectStoreFileSystem::open_read<'life0, 'life1, 'async_trait>(&'life0 self, path: &'life1 str) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait +pub fn vortex_io::object_store::ObjectStoreFileSystem::open_read<'life0, 'life1, 'async_trait>(&'life0 self, &'life1 str) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait pub struct vortex_io::object_store::ObjectStoreReadAt impl vortex_io::object_store::ObjectStoreReadAt -pub fn vortex_io::object_store::ObjectStoreReadAt::new(store: alloc::sync::Arc, path: object_store::path::Path, handle: vortex_io::runtime::Handle) -> Self +pub fn vortex_io::object_store::ObjectStoreReadAt::new(alloc::sync::Arc, object_store::path::Path, vortex_io::runtime::Handle) -> Self -pub fn vortex_io::object_store::ObjectStoreReadAt::new_with_allocator(store: alloc::sync::Arc, path: object_store::path::Path, handle: vortex_io::runtime::Handle, allocator: vortex_array::memory::HostAllocatorRef) -> Self +pub fn vortex_io::object_store::ObjectStoreReadAt::new_with_allocator(alloc::sync::Arc, object_store::path::Path, vortex_io::runtime::Handle, vortex_array::memory::HostAllocatorRef) -> Self -pub fn vortex_io::object_store::ObjectStoreReadAt::with_coalesce_config(self, config: vortex_io::CoalesceConfig) -> Self +pub fn vortex_io::object_store::ObjectStoreReadAt::with_coalesce_config(self, vortex_io::CoalesceConfig) -> Self -pub fn vortex_io::object_store::ObjectStoreReadAt::with_concurrency(self, concurrency: usize) -> Self +pub fn vortex_io::object_store::ObjectStoreReadAt::with_concurrency(self, usize) -> Self impl vortex_io::VortexReadAt for vortex_io::object_store::ObjectStoreReadAt @@ -162,7 +206,7 @@ pub fn vortex_io::object_store::ObjectStoreReadAt::coalesce_config(&self) -> cor pub fn vortex_io::object_store::ObjectStoreReadAt::concurrency(&self) -> usize -pub fn vortex_io::object_store::ObjectStoreReadAt::read_at(&self, offset: u64, length: usize, alignment: vortex_buffer::alignment::Alignment) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> +pub fn vortex_io::object_store::ObjectStoreReadAt::read_at(&self, u64, usize, vortex_buffer::alignment::Alignment) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> pub fn vortex_io::object_store::ObjectStoreReadAt::size(&self) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> @@ -172,7 +216,7 @@ pub struct vortex_io::object_store::ObjectStoreWrite impl vortex_io::object_store::ObjectStoreWrite -pub async fn vortex_io::object_store::ObjectStoreWrite::new(object_store: alloc::sync::Arc, location: &object_store::path::Path) -> vortex_error::VortexResult +pub async fn vortex_io::object_store::ObjectStoreWrite::new(alloc::sync::Arc, &object_store::path::Path) -> vortex_error::VortexResult pub fn vortex_io::object_store::ObjectStoreWrite::put_result(&self) -> core::option::Option<&object_store::PutResult> @@ -182,7 +226,7 @@ pub async fn vortex_io::object_store::ObjectStoreWrite::flush(&mut self) -> std: pub async fn vortex_io::object_store::ObjectStoreWrite::shutdown(&mut self) -> std::io::error::Result<()> -pub async fn vortex_io::object_store::ObjectStoreWrite::write_all(&mut self, buffer: B) -> std::io::error::Result +pub async fn vortex_io::object_store::ObjectStoreWrite::write_all(&mut self, B) -> std::io::error::Result pub const vortex_io::object_store::DEFAULT_CONCURRENCY: usize @@ -202,7 +246,7 @@ pub struct vortex_io::runtime::current::CurrentThreadRuntime impl vortex_io::runtime::current::CurrentThreadRuntime -pub fn vortex_io::runtime::current::CurrentThreadRuntime::block_on_stream_thread_safe(&self, f: F) -> vortex_io::runtime::current::ThreadSafeIterator where F: core::ops::function::FnOnce(vortex_io::runtime::Handle) -> S, S: futures_core::stream::Stream + core::marker::Send + 'static, R: core::marker::Send + 'static +pub fn vortex_io::runtime::current::CurrentThreadRuntime::block_on_stream_thread_safe(&self, F) -> vortex_io::runtime::current::ThreadSafeIterator where F: core::ops::function::FnOnce(vortex_io::runtime::Handle) -> S, S: futures_core::stream::Stream + core::marker::Send + 'static, R: core::marker::Send + 'static pub fn vortex_io::runtime::current::CurrentThreadRuntime::new() -> Self @@ -220,9 +264,9 @@ impl vortex_io::runtime::BlockingRuntime for vortex_io::runtime::current::Curren pub type vortex_io::runtime::current::CurrentThreadRuntime::BlockingIterator<'a, R: 'a> = vortex_io::runtime::current::CurrentThreadIterator<'a, R> -pub fn vortex_io::runtime::current::CurrentThreadRuntime::block_on(&self, fut: Fut) -> R where Fut: core::future::future::Future +pub fn vortex_io::runtime::current::CurrentThreadRuntime::block_on(&self, Fut) -> R where Fut: core::future::future::Future -pub fn vortex_io::runtime::current::CurrentThreadRuntime::block_on_stream<'a, S, R>(&self, stream: S) -> Self::BlockingIterator where S: futures_core::stream::Stream + core::marker::Send + 'a, R: core::marker::Send + 'a +pub fn vortex_io::runtime::current::CurrentThreadRuntime::block_on_stream<'a, S, R>(&self, S) -> Self::BlockingIterator where S: futures_core::stream::Stream + core::marker::Send + 'a, R: core::marker::Send + 'a pub fn vortex_io::runtime::current::CurrentThreadRuntime::handle(&self) -> vortex_io::runtime::Handle @@ -230,7 +274,7 @@ pub struct vortex_io::runtime::current::CurrentThreadWorkerPool impl vortex_io::runtime::current::CurrentThreadWorkerPool -pub fn vortex_io::runtime::current::CurrentThreadWorkerPool::set_workers(&self, n: usize) +pub fn vortex_io::runtime::current::CurrentThreadWorkerPool::set_workers(&self, usize) pub fn vortex_io::runtime::current::CurrentThreadWorkerPool::set_workers_to_available_parallelism(&self) @@ -276,15 +320,15 @@ impl vortex_io::runtime::BlockingRuntime for vortex_io::runtime::single::SingleT pub type vortex_io::runtime::single::SingleThreadRuntime::BlockingIterator<'a, R: 'a> = vortex_io::runtime::single::SingleThreadIterator<'a, R> -pub fn vortex_io::runtime::single::SingleThreadRuntime::block_on(&self, fut: Fut) -> R where Fut: core::future::future::Future +pub fn vortex_io::runtime::single::SingleThreadRuntime::block_on(&self, Fut) -> R where Fut: core::future::future::Future -pub fn vortex_io::runtime::single::SingleThreadRuntime::block_on_stream<'a, S, R>(&self, stream: S) -> Self::BlockingIterator where S: futures_core::stream::Stream + core::marker::Send + 'a, R: core::marker::Send + 'a +pub fn vortex_io::runtime::single::SingleThreadRuntime::block_on_stream<'a, S, R>(&self, S) -> Self::BlockingIterator where S: futures_core::stream::Stream + core::marker::Send + 'a, R: core::marker::Send + 'a pub fn vortex_io::runtime::single::SingleThreadRuntime::handle(&self) -> vortex_io::runtime::Handle -pub fn vortex_io::runtime::single::block_on(f: F) -> R where F: core::ops::function::FnOnce(vortex_io::runtime::Handle) -> Fut, Fut: core::future::future::Future +pub fn vortex_io::runtime::single::block_on(F) -> R where F: core::ops::function::FnOnce(vortex_io::runtime::Handle) -> Fut, Fut: core::future::future::Future -pub fn vortex_io::runtime::single::block_on_stream<'a, F, S, R>(f: F) -> vortex_io::runtime::single::SingleThreadIterator<'a, R> where F: core::ops::function::FnOnce(vortex_io::runtime::Handle) -> S, S: futures_core::stream::Stream + core::marker::Send + core::marker::Unpin + 'a, R: core::marker::Send + 'a +pub fn vortex_io::runtime::single::block_on_stream<'a, F, S, R>(F) -> vortex_io::runtime::single::SingleThreadIterator<'a, R> where F: core::ops::function::FnOnce(vortex_io::runtime::Handle) -> S, S: futures_core::stream::Stream + core::marker::Send + core::marker::Unpin + 'a, R: core::marker::Send + 'a pub mod vortex_io::runtime::tokio @@ -302,23 +346,23 @@ impl vortex_io::runtime::tokio::TokioRuntime pub fn vortex_io::runtime::tokio::TokioRuntime::current() -> vortex_io::runtime::Handle -pub fn vortex_io::runtime::tokio::TokioRuntime::new(handle: tokio::runtime::handle::Handle) -> Self +pub fn vortex_io::runtime::tokio::TokioRuntime::new(tokio::runtime::handle::Handle) -> Self impl core::convert::From<&tokio::runtime::handle::Handle> for vortex_io::runtime::tokio::TokioRuntime -pub fn vortex_io::runtime::tokio::TokioRuntime::from(value: &tokio::runtime::handle::Handle) -> Self +pub fn vortex_io::runtime::tokio::TokioRuntime::from(&tokio::runtime::handle::Handle) -> Self impl core::convert::From for vortex_io::runtime::tokio::TokioRuntime -pub fn vortex_io::runtime::tokio::TokioRuntime::from(value: tokio::runtime::handle::Handle) -> Self +pub fn vortex_io::runtime::tokio::TokioRuntime::from(tokio::runtime::handle::Handle) -> Self impl vortex_io::runtime::BlockingRuntime for vortex_io::runtime::tokio::TokioRuntime pub type vortex_io::runtime::tokio::TokioRuntime::BlockingIterator<'a, R: 'a> = vortex_io::runtime::tokio::TokioBlockingIterator<'a, R> -pub fn vortex_io::runtime::tokio::TokioRuntime::block_on(&self, fut: Fut) -> R where Fut: core::future::future::Future +pub fn vortex_io::runtime::tokio::TokioRuntime::block_on(&self, Fut) -> R where Fut: core::future::future::Future -pub fn vortex_io::runtime::tokio::TokioRuntime::block_on_stream<'a, S, R>(&self, stream: S) -> Self::BlockingIterator where S: futures_core::stream::Stream + core::marker::Send + 'a, R: core::marker::Send + 'a +pub fn vortex_io::runtime::tokio::TokioRuntime::block_on_stream<'a, S, R>(&self, S) -> Self::BlockingIterator where S: futures_core::stream::Stream + core::marker::Send + 'a, R: core::marker::Send + 'a pub fn vortex_io::runtime::tokio::TokioRuntime::handle(&self) -> vortex_io::runtime::Handle @@ -328,15 +372,15 @@ impl vortex_io::runtime::Handle pub fn vortex_io::runtime::Handle::find() -> core::option::Option -pub fn vortex_io::runtime::Handle::new(runtime: alloc::sync::Weak) -> Self +pub fn vortex_io::runtime::Handle::new(alloc::sync::Weak) -> Self -pub fn vortex_io::runtime::Handle::spawn(&self, f: Fut) -> vortex_io::runtime::Task where Fut: core::future::future::Future + core::marker::Send + 'static, R: core::marker::Send + 'static +pub fn vortex_io::runtime::Handle::spawn(&self, Fut) -> vortex_io::runtime::Task where Fut: core::future::future::Future + core::marker::Send + 'static, R: core::marker::Send + 'static -pub fn vortex_io::runtime::Handle::spawn_blocking(&self, f: F) -> vortex_io::runtime::Task where F: core::ops::function::FnOnce() -> R + core::marker::Send + 'static, R: core::marker::Send + 'static +pub fn vortex_io::runtime::Handle::spawn_blocking(&self, F) -> vortex_io::runtime::Task where F: core::ops::function::FnOnce() -> R + core::marker::Send + 'static, R: core::marker::Send + 'static -pub fn vortex_io::runtime::Handle::spawn_cpu(&self, f: F) -> vortex_io::runtime::Task where F: core::ops::function::FnOnce() -> R + core::marker::Send + 'static, R: core::marker::Send + 'static +pub fn vortex_io::runtime::Handle::spawn_cpu(&self, F) -> vortex_io::runtime::Task where F: core::ops::function::FnOnce() -> R + core::marker::Send + 'static, R: core::marker::Send + 'static -pub fn vortex_io::runtime::Handle::spawn_nested(&self, f: F) -> vortex_io::runtime::Task where F: core::ops::function::FnOnce(vortex_io::runtime::Handle) -> Fut, Fut: core::future::future::Future + core::marker::Send + 'static, R: core::marker::Send + 'static +pub fn vortex_io::runtime::Handle::spawn_nested(&self, F) -> vortex_io::runtime::Task where F: core::ops::function::FnOnce(vortex_io::runtime::Handle) -> Fut, Fut: core::future::future::Future + core::marker::Send + 'static, R: core::marker::Send + 'static impl core::clone::Clone for vortex_io::runtime::Handle @@ -352,7 +396,7 @@ impl core::future::future::Future for vortex_io::runtime::Task pub type vortex_io::runtime::Task::Output = T -pub fn vortex_io::runtime::Task::poll(self: core::pin::Pin<&mut Self>, cx: &mut core::task::wake::Context<'_>) -> core::task::poll::Poll +pub fn vortex_io::runtime::Task::poll(core::pin::Pin<&mut Self>, &mut core::task::wake::Context<'_>) -> core::task::poll::Poll impl core::ops::drop::Drop for vortex_io::runtime::Task @@ -360,19 +404,19 @@ pub fn vortex_io::runtime::Task::drop(&mut self) pub trait vortex_io::runtime::AbortHandle: core::marker::Send + core::marker::Sync -pub fn vortex_io::runtime::AbortHandle::abort(self: alloc::boxed::Box) +pub fn vortex_io::runtime::AbortHandle::abort(alloc::boxed::Box) impl vortex_io::runtime::AbortHandle for tokio::runtime::task::abort::AbortHandle -pub fn tokio::runtime::task::abort::AbortHandle::abort(self: alloc::boxed::Box) +pub fn tokio::runtime::task::abort::AbortHandle::abort(alloc::boxed::Box) pub trait vortex_io::runtime::BlockingRuntime pub type vortex_io::runtime::BlockingRuntime::BlockingIterator<'a, R: 'a>: core::iter::traits::iterator::Iterator + 'a -pub fn vortex_io::runtime::BlockingRuntime::block_on(&self, fut: Fut) -> R where Fut: core::future::future::Future +pub fn vortex_io::runtime::BlockingRuntime::block_on(&self, Fut) -> R where Fut: core::future::future::Future -pub fn vortex_io::runtime::BlockingRuntime::block_on_stream<'a, S, R>(&self, stream: S) -> Self::BlockingIterator where S: futures_core::stream::Stream + core::marker::Send + 'a, R: core::marker::Send + 'a +pub fn vortex_io::runtime::BlockingRuntime::block_on_stream<'a, S, R>(&self, S) -> Self::BlockingIterator where S: futures_core::stream::Stream + core::marker::Send + 'a, R: core::marker::Send + 'a pub fn vortex_io::runtime::BlockingRuntime::handle(&self) -> vortex_io::runtime::Handle @@ -380,9 +424,9 @@ impl vortex_io::runtime::BlockingRuntime for vortex_io::runtime::current::Curren pub type vortex_io::runtime::current::CurrentThreadRuntime::BlockingIterator<'a, R: 'a> = vortex_io::runtime::current::CurrentThreadIterator<'a, R> -pub fn vortex_io::runtime::current::CurrentThreadRuntime::block_on(&self, fut: Fut) -> R where Fut: core::future::future::Future +pub fn vortex_io::runtime::current::CurrentThreadRuntime::block_on(&self, Fut) -> R where Fut: core::future::future::Future -pub fn vortex_io::runtime::current::CurrentThreadRuntime::block_on_stream<'a, S, R>(&self, stream: S) -> Self::BlockingIterator where S: futures_core::stream::Stream + core::marker::Send + 'a, R: core::marker::Send + 'a +pub fn vortex_io::runtime::current::CurrentThreadRuntime::block_on_stream<'a, S, R>(&self, S) -> Self::BlockingIterator where S: futures_core::stream::Stream + core::marker::Send + 'a, R: core::marker::Send + 'a pub fn vortex_io::runtime::current::CurrentThreadRuntime::handle(&self) -> vortex_io::runtime::Handle @@ -390,9 +434,9 @@ impl vortex_io::runtime::BlockingRuntime for vortex_io::runtime::single::SingleT pub type vortex_io::runtime::single::SingleThreadRuntime::BlockingIterator<'a, R: 'a> = vortex_io::runtime::single::SingleThreadIterator<'a, R> -pub fn vortex_io::runtime::single::SingleThreadRuntime::block_on(&self, fut: Fut) -> R where Fut: core::future::future::Future +pub fn vortex_io::runtime::single::SingleThreadRuntime::block_on(&self, Fut) -> R where Fut: core::future::future::Future -pub fn vortex_io::runtime::single::SingleThreadRuntime::block_on_stream<'a, S, R>(&self, stream: S) -> Self::BlockingIterator where S: futures_core::stream::Stream + core::marker::Send + 'a, R: core::marker::Send + 'a +pub fn vortex_io::runtime::single::SingleThreadRuntime::block_on_stream<'a, S, R>(&self, S) -> Self::BlockingIterator where S: futures_core::stream::Stream + core::marker::Send + 'a, R: core::marker::Send + 'a pub fn vortex_io::runtime::single::SingleThreadRuntime::handle(&self) -> vortex_io::runtime::Handle @@ -400,35 +444,35 @@ impl vortex_io::runtime::BlockingRuntime for vortex_io::runtime::tokio::TokioRun pub type vortex_io::runtime::tokio::TokioRuntime::BlockingIterator<'a, R: 'a> = vortex_io::runtime::tokio::TokioBlockingIterator<'a, R> -pub fn vortex_io::runtime::tokio::TokioRuntime::block_on(&self, fut: Fut) -> R where Fut: core::future::future::Future +pub fn vortex_io::runtime::tokio::TokioRuntime::block_on(&self, Fut) -> R where Fut: core::future::future::Future -pub fn vortex_io::runtime::tokio::TokioRuntime::block_on_stream<'a, S, R>(&self, stream: S) -> Self::BlockingIterator where S: futures_core::stream::Stream + core::marker::Send + 'a, R: core::marker::Send + 'a +pub fn vortex_io::runtime::tokio::TokioRuntime::block_on_stream<'a, S, R>(&self, S) -> Self::BlockingIterator where S: futures_core::stream::Stream + core::marker::Send + 'a, R: core::marker::Send + 'a pub fn vortex_io::runtime::tokio::TokioRuntime::handle(&self) -> vortex_io::runtime::Handle pub trait vortex_io::runtime::Executor: core::marker::Send + core::marker::Sync -pub fn vortex_io::runtime::Executor::spawn(&self, fut: futures_core::future::BoxFuture<'static, ()>) -> vortex_io::runtime::AbortHandleRef +pub fn vortex_io::runtime::Executor::spawn(&self, futures_core::future::BoxFuture<'static, ()>) -> vortex_io::runtime::AbortHandleRef -pub fn vortex_io::runtime::Executor::spawn_blocking_io(&self, task: alloc::boxed::Box<(dyn core::ops::function::FnOnce() + core::marker::Send + 'static)>) -> vortex_io::runtime::AbortHandleRef +pub fn vortex_io::runtime::Executor::spawn_blocking_io(&self, alloc::boxed::Box<(dyn core::ops::function::FnOnce() + core::marker::Send + 'static)>) -> vortex_io::runtime::AbortHandleRef -pub fn vortex_io::runtime::Executor::spawn_cpu(&self, task: alloc::boxed::Box<(dyn core::ops::function::FnOnce() + core::marker::Send + 'static)>) -> vortex_io::runtime::AbortHandleRef +pub fn vortex_io::runtime::Executor::spawn_cpu(&self, alloc::boxed::Box<(dyn core::ops::function::FnOnce() + core::marker::Send + 'static)>) -> vortex_io::runtime::AbortHandleRef impl vortex_io::runtime::Executor for async_executor::Executor<'static> -pub fn async_executor::Executor<'static>::spawn(&self, fut: futures_core::future::BoxFuture<'static, ()>) -> vortex_io::runtime::AbortHandleRef +pub fn async_executor::Executor<'static>::spawn(&self, futures_core::future::BoxFuture<'static, ()>) -> vortex_io::runtime::AbortHandleRef -pub fn async_executor::Executor<'static>::spawn_blocking_io(&self, task: alloc::boxed::Box<(dyn core::ops::function::FnOnce() + core::marker::Send + 'static)>) -> vortex_io::runtime::AbortHandleRef +pub fn async_executor::Executor<'static>::spawn_blocking_io(&self, alloc::boxed::Box<(dyn core::ops::function::FnOnce() + core::marker::Send + 'static)>) -> vortex_io::runtime::AbortHandleRef -pub fn async_executor::Executor<'static>::spawn_cpu(&self, task: alloc::boxed::Box<(dyn core::ops::function::FnOnce() + core::marker::Send + 'static)>) -> vortex_io::runtime::AbortHandleRef +pub fn async_executor::Executor<'static>::spawn_cpu(&self, alloc::boxed::Box<(dyn core::ops::function::FnOnce() + core::marker::Send + 'static)>) -> vortex_io::runtime::AbortHandleRef impl vortex_io::runtime::Executor for tokio::runtime::handle::Handle -pub fn tokio::runtime::handle::Handle::spawn(&self, fut: futures_core::future::BoxFuture<'static, ()>) -> vortex_io::runtime::AbortHandleRef +pub fn tokio::runtime::handle::Handle::spawn(&self, futures_core::future::BoxFuture<'static, ()>) -> vortex_io::runtime::AbortHandleRef -pub fn tokio::runtime::handle::Handle::spawn_blocking_io(&self, task: alloc::boxed::Box<(dyn core::ops::function::FnOnce() + core::marker::Send + 'static)>) -> vortex_io::runtime::AbortHandleRef +pub fn tokio::runtime::handle::Handle::spawn_blocking_io(&self, alloc::boxed::Box<(dyn core::ops::function::FnOnce() + core::marker::Send + 'static)>) -> vortex_io::runtime::AbortHandleRef -pub fn tokio::runtime::handle::Handle::spawn_cpu(&self, cpu: alloc::boxed::Box<(dyn core::ops::function::FnOnce() + core::marker::Send + 'static)>) -> vortex_io::runtime::AbortHandleRef +pub fn tokio::runtime::handle::Handle::spawn_cpu(&self, alloc::boxed::Box<(dyn core::ops::function::FnOnce() + core::marker::Send + 'static)>) -> vortex_io::runtime::AbortHandleRef pub type vortex_io::runtime::AbortHandleRef = alloc::boxed::Box @@ -442,13 +486,19 @@ pub fn vortex_io::session::RuntimeSession::default() -> Self impl core::fmt::Debug for vortex_io::session::RuntimeSession -pub fn vortex_io::session::RuntimeSession::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_io::session::RuntimeSession::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +impl vortex_session::SessionVar for vortex_io::session::RuntimeSession + +pub fn vortex_io::session::RuntimeSession::as_any(&self) -> &dyn core::any::Any + +pub fn vortex_io::session::RuntimeSession::as_any_mut(&mut self) -> &mut dyn core::any::Any pub trait vortex_io::session::RuntimeSessionExt: vortex_session::SessionExt pub fn vortex_io::session::RuntimeSessionExt::handle(&self) -> vortex_io::runtime::Handle -pub fn vortex_io::session::RuntimeSessionExt::with_handle(self, handle: vortex_io::runtime::Handle) -> Self +pub fn vortex_io::session::RuntimeSessionExt::with_handle(self, vortex_io::runtime::Handle) -> Self pub fn vortex_io::session::RuntimeSessionExt::with_tokio(self) -> Self @@ -456,7 +506,7 @@ impl vortex_io::session::RuntimeSessionExt for S pub fn S::handle(&self) -> vortex_io::runtime::Handle -pub fn S::with_handle(self, handle: vortex_io::runtime::Handle) -> Self +pub fn S::with_handle(self, vortex_io::runtime::Handle) -> Self pub fn S::with_tokio(self) -> Self @@ -466,9 +516,9 @@ pub struct vortex_io::std_file::FileReadAt impl vortex_io::std_file::FileReadAt -pub fn vortex_io::std_file::FileReadAt::open(path: impl core::convert::AsRef, handle: vortex_io::runtime::Handle) -> vortex_error::VortexResult +pub fn vortex_io::std_file::FileReadAt::open(impl core::convert::AsRef, vortex_io::runtime::Handle) -> vortex_error::VortexResult -pub fn vortex_io::std_file::FileReadAt::open_with_allocator(path: impl core::convert::AsRef, handle: vortex_io::runtime::Handle, allocator: vortex_array::memory::HostAllocatorRef) -> vortex_error::VortexResult +pub fn vortex_io::std_file::FileReadAt::open_with_allocator(impl core::convert::AsRef, vortex_io::runtime::Handle, vortex_array::memory::HostAllocatorRef) -> vortex_error::VortexResult impl vortex_io::VortexReadAt for vortex_io::std_file::FileReadAt @@ -476,7 +526,7 @@ pub fn vortex_io::std_file::FileReadAt::coalesce_config(&self) -> core::option:: pub fn vortex_io::std_file::FileReadAt::concurrency(&self) -> usize -pub fn vortex_io::std_file::FileReadAt::read_at(&self, offset: u64, length: usize, alignment: vortex_buffer::alignment::Alignment) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> +pub fn vortex_io::std_file::FileReadAt::read_at(&self, u64, usize, vortex_buffer::alignment::Alignment) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> pub fn vortex_io::std_file::FileReadAt::size(&self) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> @@ -484,7 +534,7 @@ pub fn vortex_io::std_file::FileReadAt::uri(&self) -> core::option::Option<&allo pub const vortex_io::std_file::DEFAULT_CONCURRENCY: usize -pub fn vortex_io::std_file::read_exact_at(file: &std::fs::File, buffer: &mut [u8], offset: u64) -> std::io::error::Result<()> +pub fn vortex_io::std_file::read_exact_at(&std::fs::File, &mut [u8], u64) -> std::io::error::Result<()> pub struct vortex_io::AsyncWriteAdapter(pub W) @@ -494,7 +544,7 @@ pub async fn vortex_io::AsyncWriteAdapter::flush(&mut self) -> std::io::error pub async fn vortex_io::AsyncWriteAdapter::shutdown(&mut self) -> std::io::error::Result<()> -pub async fn vortex_io::AsyncWriteAdapter::write_all(&mut self, buffer: B) -> std::io::error::Result +pub async fn vortex_io::AsyncWriteAdapter::write_all(&mut self, B) -> std::io::error::Result pub struct vortex_io::CoalesceConfig @@ -508,7 +558,7 @@ pub const fn vortex_io::CoalesceConfig::file() -> Self pub const fn vortex_io::CoalesceConfig::in_memory() -> Self -pub const fn vortex_io::CoalesceConfig::new(distance: u64, max_size: u64) -> Self +pub const fn vortex_io::CoalesceConfig::new(u64, u64) -> Self pub const fn vortex_io::CoalesceConfig::object_storage() -> Self @@ -518,7 +568,7 @@ pub fn vortex_io::CoalesceConfig::clone(&self) -> vortex_io::CoalesceConfig impl core::fmt::Debug for vortex_io::CoalesceConfig -pub fn vortex_io::CoalesceConfig::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_io::CoalesceConfig::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_io::CoalesceConfig @@ -526,9 +576,9 @@ pub struct vortex_io::InstrumentedReadAt vortex_io::InstrumentedReadAt -pub fn vortex_io::InstrumentedReadAt::new(read: T, metrics_registry: &dyn vortex_metrics::MetricsRegistry) -> Self +pub fn vortex_io::InstrumentedReadAt::new(T, &dyn vortex_metrics::MetricsRegistry) -> Self -pub fn vortex_io::InstrumentedReadAt::new_with_labels(read: T, metrics_registry: &dyn vortex_metrics::MetricsRegistry, labels: I) -> Self where I: core::iter::traits::collect::IntoIterator, L: core::convert::Into +pub fn vortex_io::InstrumentedReadAt::new_with_labels(T, &dyn vortex_metrics::MetricsRegistry, I) -> Self where I: core::iter::traits::collect::IntoIterator, L: core::convert::Into impl core::clone::Clone for vortex_io::InstrumentedReadAt @@ -540,7 +590,7 @@ pub fn vortex_io::InstrumentedReadAt::coalesce_config(&self) -> core::option: pub fn vortex_io::InstrumentedReadAt::concurrency(&self) -> usize -pub fn vortex_io::InstrumentedReadAt::read_at(&self, offset: u64, length: usize, alignment: vortex_buffer::alignment::Alignment) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> +pub fn vortex_io::InstrumentedReadAt::read_at(&self, u64, usize, vortex_buffer::alignment::Alignment) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> pub fn vortex_io::InstrumentedReadAt::size(&self) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> @@ -560,21 +610,21 @@ pub fn vortex_io::OwnedSlice::bytes_init(&self) -> usize pub fn vortex_io::OwnedSlice::read_ptr(&self) -> *const u8 -pub fn vortex_io::OwnedSlice::slice_owned(self, range: core::ops::range::Range) -> vortex_io::OwnedSlice where Self: core::marker::Sized +pub fn vortex_io::OwnedSlice::slice_owned(self, core::ops::range::Range) -> vortex_io::OwnedSlice where Self: core::marker::Sized pub struct vortex_io::SizeLimitedStream impl vortex_io::SizeLimitedStream where Fut: core::future::future::Future -pub async fn vortex_io::SizeLimitedStream::push(&self, fut: Fut, bytes: usize) +pub async fn vortex_io::SizeLimitedStream::push(&self, Fut, usize) -pub fn vortex_io::SizeLimitedStream::try_push(&self, fut: Fut, bytes: usize) -> core::result::Result<(), Fut> +pub fn vortex_io::SizeLimitedStream::try_push(&self, Fut, usize) -> core::result::Result<(), Fut> impl vortex_io::SizeLimitedStream pub fn vortex_io::SizeLimitedStream::bytes_available(&self) -> usize -pub fn vortex_io::SizeLimitedStream::new(max_bytes: usize) -> Self +pub fn vortex_io::SizeLimitedStream::new(usize) -> Self impl<'__pin, Fut> core::marker::Unpin for vortex_io::SizeLimitedStream where pin_project_lite::__private::PinnedFieldsOf<__Origin<'__pin, Fut>>: core::marker::Unpin @@ -582,7 +632,7 @@ impl futures_core::stream::Stream for vortex_io::SizeLimitedStream whe pub type vortex_io::SizeLimitedStream::Item = ::Output -pub fn vortex_io::SizeLimitedStream::poll_next(self: core::pin::Pin<&mut Self>, cx: &mut core::task::wake::Context<'_>) -> core::task::poll::Poll> +pub fn vortex_io::SizeLimitedStream::poll_next(core::pin::Pin<&mut Self>, &mut core::task::wake::Context<'_>) -> core::task::poll::Poll> pub unsafe trait vortex_io::IoBuf: core::marker::Unpin + core::marker::Send + 'static @@ -592,7 +642,7 @@ pub fn vortex_io::IoBuf::bytes_init(&self) -> usize pub fn vortex_io::IoBuf::read_ptr(&self) -> *const u8 -pub fn vortex_io::IoBuf::slice_owned(self, range: core::ops::range::Range) -> vortex_io::OwnedSlice where Self: core::marker::Sized +pub fn vortex_io::IoBuf::slice_owned(self, core::ops::range::Range) -> vortex_io::OwnedSlice where Self: core::marker::Sized impl vortex_io::IoBuf for &'static [u8] @@ -602,7 +652,7 @@ pub fn &'static [u8]::bytes_init(&self) -> usize pub fn &'static [u8]::read_ptr(&self) -> *const u8 -pub fn &'static [u8]::slice_owned(self, range: core::ops::range::Range) -> vortex_io::OwnedSlice where Self: core::marker::Sized +pub fn &'static [u8]::slice_owned(self, core::ops::range::Range) -> vortex_io::OwnedSlice where Self: core::marker::Sized impl vortex_io::IoBuf for alloc::vec::Vec @@ -612,7 +662,7 @@ pub fn alloc::vec::Vec::bytes_init(&self) -> usize pub fn alloc::vec::Vec::read_ptr(&self) -> *const u8 -pub fn alloc::vec::Vec::slice_owned(self, range: core::ops::range::Range) -> vortex_io::OwnedSlice where Self: core::marker::Sized +pub fn alloc::vec::Vec::slice_owned(self, core::ops::range::Range) -> vortex_io::OwnedSlice where Self: core::marker::Sized impl vortex_io::IoBuf for bytes::bytes::Bytes @@ -622,7 +672,7 @@ pub fn bytes::bytes::Bytes::bytes_init(&self) -> usize pub fn bytes::bytes::Bytes::read_ptr(&self) -> *const u8 -pub fn bytes::bytes::Bytes::slice_owned(self, range: core::ops::range::Range) -> vortex_io::OwnedSlice where Self: core::marker::Sized +pub fn bytes::bytes::Bytes::slice_owned(self, core::ops::range::Range) -> vortex_io::OwnedSlice where Self: core::marker::Sized impl vortex_io::IoBuf for vortex_buffer::buffer::Buffer @@ -632,7 +682,7 @@ pub fn vortex_buffer::buffer::Buffer::bytes_init(&self) -> usize pub fn vortex_buffer::buffer::Buffer::read_ptr(&self) -> *const u8 -pub fn vortex_buffer::buffer::Buffer::slice_owned(self, range: core::ops::range::Range) -> vortex_io::OwnedSlice where Self: core::marker::Sized +pub fn vortex_buffer::buffer::Buffer::slice_owned(self, core::ops::range::Range) -> vortex_io::OwnedSlice where Self: core::marker::Sized impl vortex_io::IoBuf for vortex_io::OwnedSlice @@ -642,7 +692,7 @@ pub fn vortex_io::OwnedSlice::bytes_init(&self) -> usize pub fn vortex_io::OwnedSlice::read_ptr(&self) -> *const u8 -pub fn vortex_io::OwnedSlice::slice_owned(self, range: core::ops::range::Range) -> vortex_io::OwnedSlice where Self: core::marker::Sized +pub fn vortex_io::OwnedSlice::slice_owned(self, core::ops::range::Range) -> vortex_io::OwnedSlice where Self: core::marker::Sized impl vortex_io::IoBuf for vortex_buffer::ConstByteBuffer @@ -652,7 +702,7 @@ pub fn vortex_buffer::ConstByteBuffer::bytes_init(&self) -> usize pub fn vortex_buffer::ConstByteBuffer::read_ptr(&self) -> *const u8 -pub fn vortex_buffer::ConstByteBuffer::slice_owned(self, range: core::ops::range::Range) -> vortex_io::OwnedSlice where Self: core::marker::Sized +pub fn vortex_buffer::ConstByteBuffer::slice_owned(self, core::ops::range::Range) -> vortex_io::OwnedSlice where Self: core::marker::Sized impl vortex_io::IoBuf for [u8; N] @@ -662,7 +712,7 @@ pub fn [u8; N]::bytes_init(&self) -> usize pub fn [u8; N]::read_ptr(&self) -> *const u8 -pub fn [u8; N]::slice_owned(self, range: core::ops::range::Range) -> vortex_io::OwnedSlice where Self: core::marker::Sized +pub fn [u8; N]::slice_owned(self, core::ops::range::Range) -> vortex_io::OwnedSlice where Self: core::marker::Sized pub trait vortex_io::VortexReadAt: core::marker::Send + core::marker::Sync + 'static @@ -670,7 +720,7 @@ pub fn vortex_io::VortexReadAt::coalesce_config(&self) -> core::option::Option usize -pub fn vortex_io::VortexReadAt::read_at(&self, offset: u64, length: usize, alignment: vortex_buffer::alignment::Alignment) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> +pub fn vortex_io::VortexReadAt::read_at(&self, u64, usize, vortex_buffer::alignment::Alignment) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> pub fn vortex_io::VortexReadAt::size(&self) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> @@ -682,7 +732,7 @@ pub fn alloc::sync::Arc::coalesce_config(&self) -> pub fn alloc::sync::Arc::concurrency(&self) -> usize -pub fn alloc::sync::Arc::read_at(&self, offset: u64, length: usize, alignment: vortex_buffer::alignment::Alignment) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> +pub fn alloc::sync::Arc::read_at(&self, u64, usize, vortex_buffer::alignment::Alignment) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> pub fn alloc::sync::Arc::size(&self) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> @@ -694,7 +744,7 @@ pub fn vortex_buffer::ByteBuffer::coalesce_config(&self) -> core::option::Option pub fn vortex_buffer::ByteBuffer::concurrency(&self) -> usize -pub fn vortex_buffer::ByteBuffer::read_at(&self, offset: u64, length: usize, alignment: vortex_buffer::alignment::Alignment) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> +pub fn vortex_buffer::ByteBuffer::read_at(&self, u64, usize, vortex_buffer::alignment::Alignment) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> pub fn vortex_buffer::ByteBuffer::size(&self) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> @@ -706,7 +756,7 @@ pub fn vortex_io::object_store::ObjectStoreReadAt::coalesce_config(&self) -> cor pub fn vortex_io::object_store::ObjectStoreReadAt::concurrency(&self) -> usize -pub fn vortex_io::object_store::ObjectStoreReadAt::read_at(&self, offset: u64, length: usize, alignment: vortex_buffer::alignment::Alignment) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> +pub fn vortex_io::object_store::ObjectStoreReadAt::read_at(&self, u64, usize, vortex_buffer::alignment::Alignment) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> pub fn vortex_io::object_store::ObjectStoreReadAt::size(&self) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> @@ -718,7 +768,7 @@ pub fn vortex_io::std_file::FileReadAt::coalesce_config(&self) -> core::option:: pub fn vortex_io::std_file::FileReadAt::concurrency(&self) -> usize -pub fn vortex_io::std_file::FileReadAt::read_at(&self, offset: u64, length: usize, alignment: vortex_buffer::alignment::Alignment) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> +pub fn vortex_io::std_file::FileReadAt::read_at(&self, u64, usize, vortex_buffer::alignment::Alignment) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> pub fn vortex_io::std_file::FileReadAt::size(&self) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> @@ -730,7 +780,7 @@ pub fn alloc::sync::Arc::coalesce_config(&self) -> core::option::Option::concurrency(&self) -> usize -pub fn alloc::sync::Arc::read_at(&self, offset: u64, length: usize, alignment: vortex_buffer::alignment::Alignment) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> +pub fn alloc::sync::Arc::read_at(&self, u64, usize, vortex_buffer::alignment::Alignment) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> pub fn alloc::sync::Arc::size(&self) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> @@ -742,7 +792,7 @@ pub fn vortex_io::compat::Compat::coalesce_config(&self) -> core::option::Opt pub fn vortex_io::compat::Compat::concurrency(&self) -> usize -pub fn vortex_io::compat::Compat::read_at(&self, offset: u64, length: usize, alignment: vortex_buffer::alignment::Alignment) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> +pub fn vortex_io::compat::Compat::read_at(&self, u64, usize, vortex_buffer::alignment::Alignment) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> pub fn vortex_io::compat::Compat::size(&self) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> @@ -754,7 +804,7 @@ pub fn vortex_io::InstrumentedReadAt::coalesce_config(&self) -> core::option: pub fn vortex_io::InstrumentedReadAt::concurrency(&self) -> usize -pub fn vortex_io::InstrumentedReadAt::read_at(&self, offset: u64, length: usize, alignment: vortex_buffer::alignment::Alignment) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> +pub fn vortex_io::InstrumentedReadAt::read_at(&self, u64, usize, vortex_buffer::alignment::Alignment) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> pub fn vortex_io::InstrumentedReadAt::size(&self) -> futures_core::future::BoxFuture<'static, vortex_error::VortexResult> @@ -766,7 +816,7 @@ pub fn vortex_io::VortexWrite::flush(&mut self) -> impl core::future::future::Fu pub fn vortex_io::VortexWrite::shutdown(&mut self) -> impl core::future::future::Future> -pub fn vortex_io::VortexWrite::write_all(&mut self, buffer: B) -> impl core::future::future::Future> +pub fn vortex_io::VortexWrite::write_all(&mut self, B) -> impl core::future::future::Future> impl vortex_io::VortexWrite for alloc::vec::Vec @@ -774,7 +824,7 @@ pub fn alloc::vec::Vec::flush(&mut self) -> impl core::future::future::Futur pub fn alloc::vec::Vec::shutdown(&mut self) -> impl core::future::future::Future> -pub fn alloc::vec::Vec::write_all(&mut self, buffer: B) -> impl core::future::future::Future> +pub fn alloc::vec::Vec::write_all(&mut self, B) -> impl core::future::future::Future> impl vortex_io::VortexWrite for async_fs::File @@ -782,7 +832,7 @@ pub fn async_fs::File::flush(&mut self) -> impl core::future::future::Future impl core::future::future::Future> -pub async fn async_fs::File::write_all(&mut self, buffer: B) -> std::io::error::Result +pub async fn async_fs::File::write_all(&mut self, B) -> std::io::error::Result impl vortex_io::VortexWrite for tokio::fs::file::File @@ -790,7 +840,7 @@ pub async fn tokio::fs::file::File::flush(&mut self) -> std::io::error::Result<( pub async fn tokio::fs::file::File::shutdown(&mut self) -> std::io::error::Result<()> -pub async fn tokio::fs::file::File::write_all(&mut self, buffer: B) -> std::io::error::Result +pub async fn tokio::fs::file::File::write_all(&mut self, B) -> std::io::error::Result impl vortex_io::VortexWrite for vortex_buffer::ByteBufferMut @@ -798,7 +848,7 @@ pub fn vortex_buffer::ByteBufferMut::flush(&mut self) -> impl core::future::futu pub fn vortex_buffer::ByteBufferMut::shutdown(&mut self) -> impl core::future::future::Future> -pub fn vortex_buffer::ByteBufferMut::write_all(&mut self, buffer: B) -> impl core::future::future::Future> +pub fn vortex_buffer::ByteBufferMut::write_all(&mut self, B) -> impl core::future::future::Future> impl vortex_io::VortexWrite for vortex_io::object_store::ObjectStoreWrite @@ -806,15 +856,15 @@ pub async fn vortex_io::object_store::ObjectStoreWrite::flush(&mut self) -> std: pub async fn vortex_io::object_store::ObjectStoreWrite::shutdown(&mut self) -> std::io::error::Result<()> -pub async fn vortex_io::object_store::ObjectStoreWrite::write_all(&mut self, buffer: B) -> std::io::error::Result +pub async fn vortex_io::object_store::ObjectStoreWrite::write_all(&mut self, B) -> std::io::error::Result -impl vortex_io::VortexWrite for std::io::cursor::Cursor where std::io::cursor::Cursor: std::io::Write +impl vortex_io::VortexWrite for core::io::cursor::Cursor where core::io::cursor::Cursor: std::io::Write -pub fn std::io::cursor::Cursor::flush(&mut self) -> impl core::future::future::Future> +pub fn core::io::cursor::Cursor::flush(&mut self) -> impl core::future::future::Future> -pub fn std::io::cursor::Cursor::shutdown(&mut self) -> impl core::future::future::Future> +pub fn core::io::cursor::Cursor::shutdown(&mut self) -> impl core::future::future::Future> -pub fn std::io::cursor::Cursor::write_all(&mut self, buffer: B) -> impl core::future::future::Future> +pub fn core::io::cursor::Cursor::write_all(&mut self, B) -> impl core::future::future::Future> impl vortex_io::VortexWrite for vortex_io::AsyncWriteAdapter @@ -822,7 +872,7 @@ pub async fn vortex_io::AsyncWriteAdapter::flush(&mut self) -> std::io::error pub async fn vortex_io::AsyncWriteAdapter::shutdown(&mut self) -> std::io::error::Result<()> -pub async fn vortex_io::AsyncWriteAdapter::write_all(&mut self, buffer: B) -> std::io::error::Result +pub async fn vortex_io::AsyncWriteAdapter::write_all(&mut self, B) -> std::io::error::Result impl vortex_io::VortexWrite for &mut W @@ -830,7 +880,7 @@ pub fn &mut W::flush(&mut self) -> impl core::future::future::Future impl core::future::future::Future> -pub fn &mut W::write_all(&mut self, buffer: B) -> impl core::future::future::Future> +pub fn &mut W::write_all(&mut self, B) -> impl core::future::future::Future> impl vortex_io::VortexWrite for futures_util::io::cursor::Cursor @@ -838,7 +888,7 @@ pub fn futures_util::io::cursor::Cursor::flush(&mut self) -> impl core::futur pub fn futures_util::io::cursor::Cursor::shutdown(&mut self) -> impl core::future::future::Future> -pub fn futures_util::io::cursor::Cursor::write_all(&mut self, buffer: B) -> impl core::future::future::Future> +pub fn futures_util::io::cursor::Cursor::write_all(&mut self, B) -> impl core::future::future::Future> impl vortex_io::VortexWrite for vortex_io::compat::Compat @@ -846,4 +896,4 @@ pub fn vortex_io::compat::Compat::flush(&mut self) -> impl core::future::futu pub fn vortex_io::compat::Compat::shutdown(&mut self) -> impl core::future::future::Future> -pub fn vortex_io::compat::Compat::write_all(&mut self, buffer: B) -> impl core::future::future::Future> +pub fn vortex_io::compat::Compat::write_all(&mut self, B) -> impl core::future::future::Future> diff --git a/vortex-io/src/compat/filesystem.rs b/vortex-io/src/compat/filesystem.rs index e9f9dbf292f..9f9902b6a08 100644 --- a/vortex-io/src/compat/filesystem.rs +++ b/vortex-io/src/compat/filesystem.rs @@ -25,4 +25,8 @@ impl FileSystem for Compat { let read_at = Compat::new(self.inner().open_read(path)).await?; Ok(Arc::new(Compat::new(read_at))) } + + async fn delete(&self, path: &str) -> VortexResult<()> { + Compat::new(self.inner().delete(path)).await + } } diff --git a/vortex-io/src/compat/mod.rs b/vortex-io/src/compat/mod.rs index eb31d05b2ff..843c4cfdcb7 100644 --- a/vortex-io/src/compat/mod.rs +++ b/vortex-io/src/compat/mod.rs @@ -11,6 +11,8 @@ //! and async-compat only supports the latter. mod filesystem; +#[cfg(feature = "object_store")] +mod obj_store; mod read_at; mod write; diff --git a/vortex-io/src/compat/obj_store.rs b/vortex-io/src/compat/obj_store.rs new file mode 100644 index 00000000000..8ff9cd5e94b --- /dev/null +++ b/vortex-io/src/compat/obj_store.rs @@ -0,0 +1,111 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +use std::fmt::Display; +use std::fmt::Formatter; +use std::ops::Range; + +use async_trait::async_trait; +use bytes::Bytes; +use futures::stream::BoxStream; +use object_store::CopyOptions; +use object_store::GetOptions; +use object_store::GetResult; +use object_store::ListResult; +use object_store::MultipartUpload; +use object_store::ObjectMeta; +use object_store::ObjectStore; +use object_store::PutMultipartOptions; +use object_store::PutOptions; +use object_store::PutPayload; +use object_store::PutResult; +use object_store::RenameOptions; +use object_store::Result; +use object_store::UploadPart; +use object_store::path::Path; +use smol::future::FutureExt; +use smol::stream::StreamExt; + +use crate::compat::Compat; + +impl Display for Compat { + fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { + write!(f, "Compat<{}>", self.inner()) + } +} + +#[async_trait] +impl ObjectStore for Compat { + async fn put_opts( + &self, + location: &Path, + payload: PutPayload, + opts: PutOptions, + ) -> Result { + Compat::new(self.inner().put_opts(location, payload, opts)).await + } + + async fn put_multipart_opts( + &self, + location: &Path, + opts: PutMultipartOptions, + ) -> Result> { + Ok(Box::new(Compat::new( + Compat::new(self.inner().put_multipart_opts(location, opts)).await?, + ))) + } + + async fn get_opts(&self, location: &Path, options: GetOptions) -> Result { + Compat::new(self.inner().get_opts(location, options)).await + } + + async fn get_ranges(&self, location: &Path, ranges: &[Range]) -> Result> { + Compat::new(self.inner().get_ranges(location, ranges)).await + } + + fn delete_stream( + &self, + locations: BoxStream<'static, Result>, + ) -> BoxStream<'static, Result> { + Compat::new(self.inner().delete_stream(locations)).boxed() + } + + fn list(&self, prefix: Option<&Path>) -> BoxStream<'static, Result> { + Compat::new(self.inner().list(prefix)).boxed() + } + + fn list_with_offset( + &self, + prefix: Option<&Path>, + offset: &Path, + ) -> BoxStream<'static, Result> { + Compat::new(self.inner().list_with_offset(prefix, offset)).boxed() + } + + async fn list_with_delimiter(&self, prefix: Option<&Path>) -> Result { + Compat::new(self.inner().list_with_delimiter(prefix)).await + } + + async fn copy_opts(&self, from: &Path, to: &Path, options: CopyOptions) -> Result<()> { + Compat::new(self.inner().copy_opts(from, to, options)).await + } + + async fn rename_opts(&self, from: &Path, to: &Path, options: RenameOptions) -> Result<()> { + Compat::new(self.inner().rename_opts(from, to, options)).await + } +} + +#[async_trait] +impl MultipartUpload for Compat { + fn put_part(&mut self, data: PutPayload) -> UploadPart { + Compat::new(self.inner_mut().put_part(data)).boxed() + } + + async fn complete(&mut self) -> Result { + Compat::new(self.inner_mut().complete()).await + } + + async fn abort(&mut self) -> Result<()> { + Compat::new(self.inner_mut().abort()).await + } +} diff --git a/vortex-io/src/compat/read_at.rs b/vortex-io/src/compat/read_at.rs index 03f47bf58b3..4fc49785d28 100644 --- a/vortex-io/src/compat/read_at.rs +++ b/vortex-io/src/compat/read_at.rs @@ -29,7 +29,7 @@ impl VortexReadAt for Compat { } fn size(&self) -> BoxFuture<'static, VortexResult> { - self.inner().size() + Compat::new(self.inner().size()).boxed() } fn read_at( diff --git a/vortex-io/src/filesystem/glob.rs b/vortex-io/src/filesystem/glob.rs index adb6da3c394..6bbe26fa5dc 100644 --- a/vortex-io/src/filesystem/glob.rs +++ b/vortex-io/src/filesystem/glob.rs @@ -106,6 +106,10 @@ mod tests { async fn open_read(&self, _path: &str) -> VortexResult> { vortex_panic!("open_read() should not be called") } + + async fn delete(&self, _path: &str) -> VortexResult<()> { + vortex_panic!("delete() should not be called") + } } #[tokio::test] diff --git a/vortex-io/src/filesystem/mod.rs b/vortex-io/src/filesystem/mod.rs index 8749d87d8ae..97535909d28 100644 --- a/vortex-io/src/filesystem/mod.rs +++ b/vortex-io/src/filesystem/mod.rs @@ -53,4 +53,6 @@ pub trait FileSystem: Debug + Send + Sync { /// Open a file for reading at the given path. async fn open_read(&self, path: &str) -> VortexResult>; + + async fn delete(&self, path: &str) -> VortexResult<()>; } diff --git a/vortex-io/src/filesystem/prefix.rs b/vortex-io/src/filesystem/prefix.rs index d194b30960d..5f6304430aa 100644 --- a/vortex-io/src/filesystem/prefix.rs +++ b/vortex-io/src/filesystem/prefix.rs @@ -57,6 +57,12 @@ impl FileSystem for PrefixFileSystem { .open_read(&format!("{}{}", self.prefix, path.trim_start_matches('/'))) .await } + + async fn delete(&self, path: &str) -> VortexResult<()> { + self.inner + .delete(&format!("{}{}", self.prefix, path.trim_start_matches('/'))) + .await + } } impl dyn FileSystem + 'static { diff --git a/vortex-io/src/limit.rs b/vortex-io/src/limit.rs index f49ea7a3661..cd6928f8aa9 100644 --- a/vortex-io/src/limit.rs +++ b/vortex-io/src/limit.rs @@ -277,7 +277,7 @@ mod tests { // Push many small items for i in 0..10 { - #[allow(clippy::cast_possible_truncation)] + #[expect(clippy::cast_possible_truncation)] stream.push(async move { vec![i as u8; 5] }, 5).await; } diff --git a/vortex-io/src/object_store/filesystem.rs b/vortex-io/src/object_store/filesystem.rs index 6fdbc23caa3..b8c2aa39ce2 100644 --- a/vortex-io/src/object_store/filesystem.rs +++ b/vortex-io/src/object_store/filesystem.rs @@ -11,8 +11,10 @@ use async_trait::async_trait; use futures::StreamExt; use futures::stream::BoxStream; use object_store::ObjectStore; +use object_store::ObjectStoreExt; use object_store::path::Path; use vortex_error::VortexResult; +use vortex_error::vortex_err; use crate::VortexReadAt; use crate::filesystem::FileListing; @@ -81,4 +83,14 @@ impl FileSystem for ObjectStoreFileSystem { self.handle.clone(), ))) } + + async fn delete(&self, path: &str) -> VortexResult<()> { + self.store + .delete( + &Path::from_url_path(path) + .map_err(|_| vortex_err!("invalid path for url {path}"))?, + ) + .await?; + Ok(()) + } } diff --git a/vortex-io/src/read_at.rs b/vortex-io/src/read_at.rs index 28cc5bf244b..aa9a8a03abf 100644 --- a/vortex-io/src/read_at.rs +++ b/vortex-io/src/read_at.rs @@ -229,10 +229,8 @@ impl InstrumentedReadAt { } } -// We implement drop for `InnerMetrics` so this will be logged only when we eventually drop the final instance of `InstrumentedRead` -impl Drop for InnerMetrics { - #[allow(clippy::cognitive_complexity)] - fn drop(&mut self) { +impl InnerMetrics { + fn log_sizes(&self) { tracing::debug!("Reads: {}", self.sizes.count()); if !self.sizes.is_empty() { tracing::debug!( @@ -245,10 +243,10 @@ impl Drop for InnerMetrics { .vortex_expect("must not be empty"), ); } + tracing::debug!("Total read size: {}", self.total_size.value()); + } - let total_size = self.total_size.value(); - tracing::debug!("Total read size: {total_size}"); - + fn log_durations(&self) { if !self.durations.is_empty() { tracing::debug!( "Read duration: p50={}ms p95={}ms p99={}ms p999={}ms", @@ -273,6 +271,14 @@ impl Drop for InnerMetrics { } } +// We implement drop for `InnerMetrics` so this will be logged only when we eventually drop the final instance of `InstrumentedRead` +impl Drop for InnerMetrics { + fn drop(&mut self) { + self.log_sizes(); + self.log_durations(); + } +} + impl VortexReadAt for InstrumentedReadAt { fn uri(&self) -> Option<&Arc> { self.read.uri() diff --git a/vortex-io/src/runtime/current.rs b/vortex-io/src/runtime/current.rs index c49d9e010a5..c8ff91b4aec 100644 --- a/vortex-io/src/runtime/current.rs +++ b/vortex-io/src/runtime/current.rs @@ -152,7 +152,7 @@ impl Iterator for ThreadSafeIterator { } } -#[allow(clippy::if_then_some_else_none)] // Clippy is wrong when if/else has await. +#[expect(clippy::if_then_some_else_none)] // Clippy is wrong when if/else has await. #[cfg(test)] mod tests { use std::sync::Arc; diff --git a/vortex-io/src/runtime/handle.rs b/vortex-io/src/runtime/handle.rs index f04b897b29e..dfbdc7fa77f 100644 --- a/vortex-io/src/runtime/handle.rs +++ b/vortex-io/src/runtime/handle.rs @@ -165,7 +165,7 @@ impl Task { impl Future for Task { type Output = T; - #[allow(clippy::panic)] + #[expect(clippy::panic)] fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { match ready!(self.recv.poll_unpin(cx)) { Ok(result) => Poll::Ready(result), diff --git a/vortex-io/src/runtime/pool.rs b/vortex-io/src/runtime/pool.rs index a8d1f67a149..71ffb54cae6 100644 --- a/vortex-io/src/runtime/pool.rs +++ b/vortex-io/src/runtime/pool.rs @@ -9,6 +9,7 @@ use std::time::Duration; use parking_lot::Mutex; use smol::block_on; use vortex_error::VortexExpect; +use vortex_utils::parallelism::get_available_parallelism; #[derive(Clone)] pub struct CurrentThreadWorkerPool { @@ -25,10 +26,10 @@ impl CurrentThreadWorkerPool { } /// Set the number of worker threads to the available system parallelism as reported by - /// `std::thread::available_parallelism()` minus 1, to leave a slot open for the calling thread. + /// [`get_available_parallelism()`] minus 1, to leave a slot open for the calling thread. pub fn set_workers_to_available_parallelism(&self) { - let n = std::thread::available_parallelism() - .map(|n| n.get().saturating_sub(1).max(1)) + let n = get_available_parallelism() + .map(|n| n.saturating_sub(1).max(1)) .unwrap_or(1); self.set_workers(n); } diff --git a/vortex-io/src/runtime/tests.rs b/vortex-io/src/runtime/tests.rs index 8b342f4242d..e35d9db6f78 100644 --- a/vortex-io/src/runtime/tests.rs +++ b/vortex-io/src/runtime/tests.rs @@ -2,7 +2,7 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors #![cfg(feature = "tokio")] -#![allow(clippy::cast_possible_truncation)] +#![expect(clippy::cast_possible_truncation)] use std::sync::Arc; use std::sync::atomic::AtomicUsize; diff --git a/vortex-io/src/session.rs b/vortex-io/src/session.rs index 38500d248a0..42975ca70de 100644 --- a/vortex-io/src/session.rs +++ b/vortex-io/src/session.rs @@ -1,10 +1,12 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +use std::any::Any; use std::fmt::Debug; use vortex_error::VortexExpect; use vortex_session::SessionExt; +use vortex_session::SessionVar; use crate::runtime::Handle; @@ -13,6 +15,16 @@ pub struct RuntimeSession { handle: Option, } +impl SessionVar for RuntimeSession { + fn as_any(&self) -> &dyn Any { + self + } + + fn as_any_mut(&mut self) -> &mut dyn Any { + self + } +} + impl Default for RuntimeSession { fn default() -> Self { Self { diff --git a/vortex-ipc/public-api.lock b/vortex-ipc/public-api.lock index 5eba6f17f38..397667c2c83 100644 --- a/vortex-ipc/public-api.lock +++ b/vortex-ipc/public-api.lock @@ -18,7 +18,7 @@ pub struct vortex_ipc::iterator::SyncIPCReader impl vortex_ipc::iterator::SyncIPCReader -pub fn vortex_ipc::iterator::SyncIPCReader::try_new(read: R, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_ipc::iterator::SyncIPCReader::try_new(R, &vortex_session::VortexSession) -> vortex_error::VortexResult impl core::iter::traits::iterator::Iterator for vortex_ipc::iterator::SyncIPCReader @@ -32,15 +32,15 @@ pub fn vortex_ipc::iterator::SyncIPCReader::dtype(&self) -> &vortex_array::dt pub trait vortex_ipc::iterator::ArrayIteratorIPC -pub fn vortex_ipc::iterator::ArrayIteratorIPC::into_ipc(self, session: &vortex_session::VortexSession) -> vortex_error::VortexResult where Self: core::marker::Sized +pub fn vortex_ipc::iterator::ArrayIteratorIPC::into_ipc(self, &vortex_session::VortexSession) -> vortex_error::VortexResult where Self: core::marker::Sized -pub fn vortex_ipc::iterator::ArrayIteratorIPC::write_ipc(self, write: W, session: &vortex_session::VortexSession) -> vortex_error::VortexResult where Self: core::marker::Sized +pub fn vortex_ipc::iterator::ArrayIteratorIPC::write_ipc(self, W, &vortex_session::VortexSession) -> vortex_error::VortexResult where Self: core::marker::Sized impl vortex_ipc::iterator::ArrayIteratorIPC for I -pub fn I::into_ipc(self, session: &vortex_session::VortexSession) -> vortex_error::VortexResult where Self: core::marker::Sized +pub fn I::into_ipc(self, &vortex_session::VortexSession) -> vortex_error::VortexResult where Self: core::marker::Sized -pub fn I::write_ipc(self, write: W, session: &vortex_session::VortexSession) -> vortex_error::VortexResult where Self: core::marker::Sized +pub fn I::write_ipc(self, W, &vortex_session::VortexSession) -> vortex_error::VortexResult where Self: core::marker::Sized pub mod vortex_ipc::messages @@ -54,7 +54,7 @@ pub vortex_ipc::messages::DecoderMessage::DType(vortex_flatbuffers::FlatBuffer) impl core::fmt::Debug for vortex_ipc::messages::DecoderMessage -pub fn vortex_ipc::messages::DecoderMessage::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_ipc::messages::DecoderMessage::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub enum vortex_ipc::messages::EncoderMessage<'a> @@ -72,13 +72,13 @@ pub vortex_ipc::messages::PollRead::Some(vortex_ipc::messages::DecoderMessage) impl core::fmt::Debug for vortex_ipc::messages::PollRead -pub fn vortex_ipc::messages::PollRead::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_ipc::messages::PollRead::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_ipc::messages::AsyncMessageReader impl vortex_ipc::messages::AsyncMessageReader -pub fn vortex_ipc::messages::AsyncMessageReader::new(read: R) -> Self +pub fn vortex_ipc::messages::AsyncMessageReader::new(R) -> Self impl<'__pin, R> core::marker::Unpin for vortex_ipc::messages::AsyncMessageReader where pin_project_lite::__private::PinnedFieldsOf<__Origin<'__pin, R>>: core::marker::Unpin @@ -86,7 +86,7 @@ impl futures_core::stream::Stream for vortex_i pub type vortex_ipc::messages::AsyncMessageReader::Item = core::result::Result -pub fn vortex_ipc::messages::AsyncMessageReader::poll_next(self: core::pin::Pin<&mut Self>, cx: &mut core::task::wake::Context<'_>) -> core::task::poll::Poll> +pub fn vortex_ipc::messages::AsyncMessageReader::poll_next(core::pin::Pin<&mut Self>, &mut core::task::wake::Context<'_>) -> core::task::poll::Poll> pub struct vortex_ipc::messages::AsyncMessageWriter @@ -96,15 +96,15 @@ pub fn vortex_ipc::messages::AsyncMessageWriter::inner(&self) -> &W pub fn vortex_ipc::messages::AsyncMessageWriter::into_inner(self) -> W -pub fn vortex_ipc::messages::AsyncMessageWriter::new(write: W, session: &vortex_session::VortexSession) -> Self +pub fn vortex_ipc::messages::AsyncMessageWriter::new(W, &vortex_session::VortexSession) -> Self -pub async fn vortex_ipc::messages::AsyncMessageWriter::write_message(&mut self, message: vortex_ipc::messages::EncoderMessage<'_>) -> vortex_error::VortexResult<()> +pub async fn vortex_ipc::messages::AsyncMessageWriter::write_message(&mut self, vortex_ipc::messages::EncoderMessage<'_>) -> vortex_error::VortexResult<()> pub struct vortex_ipc::messages::BufMessageReader impl vortex_ipc::messages::BufMessageReader -pub fn vortex_ipc::messages::BufMessageReader::new(buffer: B) -> Self +pub fn vortex_ipc::messages::BufMessageReader::new(B) -> Self impl core::iter::traits::iterator::Iterator for vortex_ipc::messages::BufMessageReader @@ -116,7 +116,7 @@ pub struct vortex_ipc::messages::MessageDecoder impl vortex_ipc::messages::MessageDecoder -pub fn vortex_ipc::messages::MessageDecoder::read_next(&mut self, bytes: &mut B) -> vortex_error::VortexResult +pub fn vortex_ipc::messages::MessageDecoder::read_next(&mut self, &mut B) -> vortex_error::VortexResult impl core::default::Default for vortex_ipc::messages::MessageDecoder @@ -126,15 +126,15 @@ pub struct vortex_ipc::messages::MessageEncoder impl vortex_ipc::messages::MessageEncoder -pub fn vortex_ipc::messages::MessageEncoder::encode(&mut self, message: vortex_ipc::messages::EncoderMessage<'_>) -> vortex_error::VortexResult> +pub fn vortex_ipc::messages::MessageEncoder::encode(&mut self, vortex_ipc::messages::EncoderMessage<'_>) -> vortex_error::VortexResult> -pub fn vortex_ipc::messages::MessageEncoder::new(session: vortex_session::VortexSession) -> Self +pub fn vortex_ipc::messages::MessageEncoder::new(vortex_session::VortexSession) -> Self pub struct vortex_ipc::messages::SyncMessageReader impl vortex_ipc::messages::SyncMessageReader -pub fn vortex_ipc::messages::SyncMessageReader::new(read: R) -> Self +pub fn vortex_ipc::messages::SyncMessageReader::new(R) -> Self impl core::iter::traits::iterator::Iterator for vortex_ipc::messages::SyncMessageReader @@ -146,9 +146,9 @@ pub struct vortex_ipc::messages::SyncMessageWriter impl vortex_ipc::messages::SyncMessageWriter -pub fn vortex_ipc::messages::SyncMessageWriter::new(write: W, session: &vortex_session::VortexSession) -> Self +pub fn vortex_ipc::messages::SyncMessageWriter::new(W, &vortex_session::VortexSession) -> Self -pub fn vortex_ipc::messages::SyncMessageWriter::write_message(&mut self, message: vortex_ipc::messages::EncoderMessage<'_>) -> vortex_error::VortexResult<()> +pub fn vortex_ipc::messages::SyncMessageWriter::write_message(&mut self, vortex_ipc::messages::EncoderMessage<'_>) -> vortex_error::VortexResult<()> pub mod vortex_ipc::stream @@ -162,13 +162,13 @@ impl futures_core::stream::Stream for vortex_ipc::stream::ArrayStreamIPCBytes pub type vortex_ipc::stream::ArrayStreamIPCBytes::Item = core::result::Result -pub fn vortex_ipc::stream::ArrayStreamIPCBytes::poll_next(self: core::pin::Pin<&mut Self>, cx: &mut core::task::wake::Context<'_>) -> core::task::poll::Poll> +pub fn vortex_ipc::stream::ArrayStreamIPCBytes::poll_next(core::pin::Pin<&mut Self>, &mut core::task::wake::Context<'_>) -> core::task::poll::Poll> pub struct vortex_ipc::stream::AsyncIPCReader impl vortex_ipc::stream::AsyncIPCReader -pub async fn vortex_ipc::stream::AsyncIPCReader::try_new(read: R, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub async fn vortex_ipc::stream::AsyncIPCReader::try_new(R, &vortex_session::VortexSession) -> vortex_error::VortexResult impl<'__pin, R> core::marker::Unpin for vortex_ipc::stream::AsyncIPCReader where pin_project_lite::__private::PinnedFieldsOf<__Origin<'__pin, R>>: core::marker::Unpin @@ -176,7 +176,7 @@ impl futures_core::stream::Stream for vortex_i pub type vortex_ipc::stream::AsyncIPCReader::Item = core::result::Result -pub fn vortex_ipc::stream::AsyncIPCReader::poll_next(self: core::pin::Pin<&mut Self>, cx: &mut core::task::wake::Context<'_>) -> core::task::poll::Poll> +pub fn vortex_ipc::stream::AsyncIPCReader::poll_next(core::pin::Pin<&mut Self>, &mut core::task::wake::Context<'_>) -> core::task::poll::Poll> impl vortex_array::stream::ArrayStream for vortex_ipc::stream::AsyncIPCReader @@ -184,12 +184,12 @@ pub fn vortex_ipc::stream::AsyncIPCReader::dtype(&self) -> &vortex_array::dty pub trait vortex_ipc::stream::ArrayStreamIPC -pub fn vortex_ipc::stream::ArrayStreamIPC::into_ipc(self, session: &vortex_session::VortexSession) -> vortex_ipc::stream::ArrayStreamIPCBytes where Self: core::marker::Sized +pub fn vortex_ipc::stream::ArrayStreamIPC::into_ipc(self, &vortex_session::VortexSession) -> vortex_ipc::stream::ArrayStreamIPCBytes where Self: core::marker::Sized -pub fn vortex_ipc::stream::ArrayStreamIPC::write_ipc(self, write: W, session: &vortex_session::VortexSession) -> impl core::future::future::Future> where Self: core::marker::Sized +pub fn vortex_ipc::stream::ArrayStreamIPC::write_ipc(self, W, &vortex_session::VortexSession) -> impl core::future::future::Future> where Self: core::marker::Sized impl vortex_ipc::stream::ArrayStreamIPC for S -pub fn S::into_ipc(self, session: &vortex_session::VortexSession) -> vortex_ipc::stream::ArrayStreamIPCBytes where Self: core::marker::Sized +pub fn S::into_ipc(self, &vortex_session::VortexSession) -> vortex_ipc::stream::ArrayStreamIPCBytes where Self: core::marker::Sized -pub async fn S::write_ipc(self, write: W, session: &vortex_session::VortexSession) -> vortex_error::VortexResult where Self: core::marker::Sized +pub async fn S::write_ipc(self, W, &vortex_session::VortexSession) -> vortex_error::VortexResult where Self: core::marker::Sized diff --git a/vortex-ipc/src/messages/decoder.rs b/vortex-ipc/src/messages/decoder.rs index d5f2cce336d..190461df96f 100644 --- a/vortex-ipc/src/messages/decoder.rs +++ b/vortex-ipc/src/messages/decoder.rs @@ -125,7 +125,7 @@ impl MessageDecoder { .encodings() .iter() .flat_map(|e| e.iter()) - .map(|id| ArrayId::new_arc(Arc::from(id.to_string()))) + .map(ArrayId::new) .collect(); let ctx = ReadContext::new(encoding_ids); diff --git a/vortex-jni/Cargo.toml b/vortex-jni/Cargo.toml index d6901514b49..66dd5f6254b 100644 --- a/vortex-jni/Cargo.toml +++ b/vortex-jni/Cargo.toml @@ -18,22 +18,19 @@ categories = { workspace = true } [dependencies] arrow-array = { workspace = true, features = ["ffi"] } -arrow-ipc = { workspace = true } arrow-schema = { workspace = true } futures = { workspace = true } -jni = "0.21.1" +jni = { workspace = true } object_store = { workspace = true, features = ["aws", "azure", "gcp"] } parking_lot = { workspace = true } -prost = { workspace = true } thiserror = { workspace = true } -tokio = { workspace = true, features = ["rt-multi-thread"] } tracing = { workspace = true, features = ["std", "log"] } tracing-subscriber = { workspace = true, features = ["env-filter"] } url = { workspace = true } -vortex = { workspace = true, features = ["object_store", "files", "tokio"] } +vortex = { workspace = true, features = ["object_store", "files"] } [dev-dependencies] -jni = { version = "0.21.1", features = ["invocation"] } +jni = { workspace = true, features = ["invocation"] } [lib] crate-type = ["staticlib", "cdylib"] diff --git a/vortex-jni/src/array.rs b/vortex-jni/src/array.rs deleted file mode 100644 index c45f232b920..00000000000 --- a/vortex-jni/src/array.rs +++ /dev/null @@ -1,508 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -use std::sync::Arc; - -use arrow_array::ffi::FFI_ArrowArray; -use arrow_array::ffi::FFI_ArrowSchema; -use arrow_schema::DataType; -use arrow_schema::FieldRef; -use arrow_schema::Fields; -use jni::JNIEnv; -use jni::objects::JClass; -use jni::objects::JIntArray; -use jni::objects::JLongArray; -use jni::objects::JObject; -use jni::objects::JValue; -use jni::sys::JNI_FALSE; -use jni::sys::JNI_TRUE; -use jni::sys::jboolean; -use jni::sys::jbyte; -use jni::sys::jbyteArray; -use jni::sys::jdouble; -use jni::sys::jfloat; -use jni::sys::jint; -use jni::sys::jlong; -use jni::sys::jobject; -use jni::sys::jshort; -use jni::sys::jstring; -use vortex::array::ArrayRef; -use vortex::array::ArrayView; -use vortex::array::ToCanonical; -use vortex::array::arrays::VarBin; -use vortex::array::arrays::VarBinView; -use vortex::array::arrays::extension::ExtensionArrayExt; -use vortex::array::arrays::struct_::StructArrayExt; -use vortex::array::arrays::varbin::VarBinArrayExt; -use vortex::array::arrow::IntoArrowArray; -use vortex::dtype::DType; -use vortex::dtype::i256; -use vortex::error::VortexError; -use vortex::error::VortexExpect; -use vortex::error::vortex_err; -use vortex::scalar::DecimalValue; - -use crate::errors::JNIError; -use crate::errors::try_or_throw; - -pub struct NativeArray { - inner: ArrayRef, - is_extension: bool, -} - -impl NativeArray { - pub fn new(array_ref: ArrayRef) -> Box { - Box::new(NativeArray { - is_extension: array_ref.dtype().is_extension(), - inner: array_ref, - }) - } - - pub fn into_raw(self: Box) -> jlong { - Box::into_raw(self) as jlong - } - - /// Reconstruct a boxed `NativeArray` from a raw heap pointer. - pub unsafe fn from_raw(pointer: jlong) -> Box { - // SAFETY: caller must ensure that the pointer is valid and points to a `NativeArray`. - unsafe { Box::from_raw(pointer as *mut NativeArray) } - } - - #[expect( - clippy::expect_used, - reason = "JNI contract guarantees non-null pointer" - )] - pub unsafe fn from_ptr<'a>(pointer: jlong) -> &'a Self { - unsafe { - (pointer as *const NativeArray) - .as_ref() - .expect("Pointer should never be null") - } - } -} - -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeArrayMethods_free( - _env: JNIEnv, - _class: JClass, - array_ptr: jlong, -) { - drop(unsafe { NativeArray::from_raw(array_ptr) }); -} - -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeArrayMethods_nbytes( - _env: JNIEnv, - _class: JClass, - array_ptr: jlong, -) -> jlong { - let array_ref = unsafe { NativeArray::from_ptr(array_ptr) }; - array_ref.inner.nbytes() as jlong -} - -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeArrayMethods_exportToArrow<'local>( - mut env: JNIEnv<'local>, - _class: JClass, - array_ptr: jlong, - arrow_schema_ptr: JLongArray<'local>, - arrow_array_ptr: JLongArray<'local>, -) { - let array_ref = unsafe { NativeArray::from_ptr(array_ptr) }; - - try_or_throw(&mut env, |env| { - let preferred_arrow_type = array_ref.inner.dtype().to_arrow_dtype()?; - let viewless_arrow_type = data_type_no_views(preferred_arrow_type); - - let arrow_array = array_ref.inner.clone().into_arrow(&viewless_arrow_type)?; - let (ffi_array, ffi_schema) = - arrow_array::ffi::to_ffi(&arrow_array.to_data()).map_err(VortexError::from)?; - - let ffi_schema_ptr = Box::into_raw(Box::new(ffi_schema)); - let ffi_array_ptr = Box::into_raw(Box::new(ffi_array)); - - // Return native Arrow FFI pointers to caller. - env.set_long_array_region(arrow_schema_ptr, 0, &[ffi_schema_ptr as jlong])?; - env.set_long_array_region(arrow_array_ptr, 0, &[ffi_array_ptr as jlong])?; - Ok(()) - }); -} - -/// Visit the potentially nested DataType, replacing all instances of Utf8View and BinaryView -/// with non-Viewable equivalents. This is necessary because Spark and Iceberg do not support -/// Utf8View. -fn data_type_no_views(data_type: DataType) -> DataType { - match data_type { - DataType::BinaryView => DataType::Binary, - DataType::Utf8View => DataType::Utf8, - // List - DataType::List(inner) | DataType::ListView(inner) => { - let new_inner = (*inner) - .clone() - .with_data_type(data_type_no_views(inner.data_type().clone())); - DataType::List(FieldRef::new(new_inner)) - } - // LargeList - DataType::LargeList(inner) | DataType::LargeListView(inner) => { - let new_inner = (*inner) - .clone() - .with_data_type(data_type_no_views(inner.data_type().clone())); - DataType::LargeList(FieldRef::new(new_inner)) - } - DataType::Struct(fields) => { - let viewless_fields: Vec = fields - .iter() - .map(|field_ref| { - let field = (*Arc::clone(field_ref)).clone(); - let data_type = field.data_type().clone(); - let field = field.with_data_type(data_type_no_views(data_type)); - FieldRef::new(field) - }) - .collect(); - DataType::Struct(Fields::from(viewless_fields)) - } - DataType::Decimal128(precision, scale) => DataType::Decimal128(precision, scale), - DataType::Decimal256(precision, scale) => DataType::Decimal256(precision, scale), - DataType::FixedSizeList(inner, size) => { - let new_inner = (*inner) - .clone() - .with_data_type(data_type_no_views(inner.data_type().clone())); - DataType::FixedSizeList(FieldRef::new(new_inner), size) - } - DataType::Union(..) => unreachable!("Vortex never returns Union"), - DataType::Dictionary(..) => unreachable!("Vortex never returns Dictionary"), - DataType::Map(..) => unreachable!("Vortex never returns Map"), - DataType::RunEndEncoded(..) => unreachable!("Vortex never returns RunEndEncoded"), - // The non-nested non-view types stay the same. - dt => dt, - } -} - -/// Drop the native memory holding an Arrow FFI Schema behind the pointer. -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeArrayMethods_dropArrowSchema( - _env: JNIEnv, - _class: JClass, - schema_ptr: jlong, -) { - drop(unsafe { Box::from_raw(schema_ptr as *mut FFI_ArrowSchema) }); -} - -/// Drop FFI_ArrowArray behind the pointer. -/// -/// Note that this doesn't not free the memory backing the arrow data buffers, those -/// are still backed by Vortex Buffers. -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeArrayMethods_dropArrowArray( - _env: JNIEnv, - _class: JClass, - array_ptr: jlong, -) { - drop(unsafe { Box::from_raw(array_ptr as *mut FFI_ArrowArray) }); -} - -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeArrayMethods_getLen( - _env: JNIEnv, - _class: JClass, - array_ptr: jlong, -) -> jlong { - unsafe { NativeArray::from_ptr(array_ptr) }.inner.len() as jlong -} - -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeArrayMethods_getDataType( - _env: JNIEnv, - _class: JClass, - array_ptr: jlong, -) -> jlong { - let array_ref = unsafe { NativeArray::from_ptr(array_ptr) }; - let dtype_ptr = array_ref.inner.dtype(); - // Return a pointer to the DType. - (dtype_ptr as *const DType).addr() as jlong -} - -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeArrayMethods_getField( - mut env: JNIEnv, - _class: JClass, - array_ptr: jlong, - index: jint, -) -> jlong { - let array_ref = unsafe { NativeArray::from_ptr(array_ptr) }; - - try_or_throw(&mut env, |_| { - let struct_array = array_ref.inner.to_struct(); - let idx = index as usize; - if idx >= struct_array.struct_fields().nfields() { - return Err(vortex_err!("Field index out of bounds").into()); - } - let field = struct_array.unmasked_field(idx).clone(); - Ok(NativeArray::new(field).into_raw()) - }) -} - -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeArrayMethods_slice( - mut env: JNIEnv, - _class: JClass, - array_ptr: jlong, - start: jint, - end: jint, -) -> jlong { - let array_ref = unsafe { NativeArray::from_ptr(array_ptr) }; - - try_or_throw(&mut env, |_| { - let sliced_array = array_ref.inner.slice(start as usize..end as usize)?; - Ok(NativeArray::new(sliced_array).into_raw()) - }) -} - -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeArrayMethods_getNull( - mut env: JNIEnv, - _class: JClass, - array_ptr: jlong, - index: jint, -) -> jboolean { - let array_ref = unsafe { NativeArray::from_ptr(array_ptr) }; - try_or_throw(&mut env, |_| { - let is_null = array_ref.inner.is_invalid(index as usize)?; - if is_null { Ok(JNI_TRUE) } else { Ok(JNI_FALSE) } - }) -} - -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeArrayMethods_getNullCount( - mut env: JNIEnv, - _class: JClass, - array_ptr: jlong, -) -> jint { - let array_ref = unsafe { NativeArray::from_ptr(array_ptr) }; - try_or_throw(&mut env, |_| { - let count = array_ref.inner.invalid_count()?; - Ok(jint::try_from(count).unwrap_or(-1)) - }) -} - -macro_rules! get_primitive { - ($name:ident, $native:ty, $jtype:ty) => { - #[unsafe(no_mangle)] - pub extern "system" fn $name( - mut env: JNIEnv, - _class: JClass, - array_ptr: jlong, - index: jint, - ) -> $jtype { - let array_ref = unsafe { NativeArray::from_ptr(array_ptr) }; - try_or_throw(&mut env, |_| { - let scalar_value = if array_ref.is_extension { - array_ref - .inner - .to_extension() - .storage_array() - .scalar_at(index as usize)? - } else { - array_ref.inner.scalar_at(index as usize)? - }; - - Ok(scalar_value - .as_primitive() - .as_::<$native>() - .unwrap_or_default()) - }) - } - }; -} - -get_primitive!(Java_dev_vortex_jni_NativeArrayMethods_getByte, i8, jbyte); -get_primitive!(Java_dev_vortex_jni_NativeArrayMethods_getShort, i16, jshort); -get_primitive!(Java_dev_vortex_jni_NativeArrayMethods_getInt, i32, jint); -get_primitive!(Java_dev_vortex_jni_NativeArrayMethods_getLong, i64, jlong); -get_primitive!(Java_dev_vortex_jni_NativeArrayMethods_getFloat, f32, jfloat); -get_primitive!( - Java_dev_vortex_jni_NativeArrayMethods_getDouble, - f64, - jdouble -); - -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeArrayMethods_getBigDecimal( - mut env: JNIEnv, - _class: JClass, - array_ptr: jlong, - index: jint, -) -> jobject { - let array_ref = unsafe { NativeArray::from_ptr(array_ptr) }; - try_or_throw(&mut env, |env| { - let scalar_value = if array_ref.is_extension { - array_ref - .inner - .to_extension() - .storage_array() - .scalar_at(index as usize)? - } else { - array_ref.inner.scalar_at(index as usize)? - }; - - let decimal_scalar = scalar_value.as_decimal(); - let DType::Decimal(decimal_type, ..) = decimal_scalar.dtype() else { - return Err(vortex_err!("Expected Decimal type").into()); - }; - let scale = decimal_type.scale(); - if let Some(v) = decimal_scalar.decimal_value() { - match v { - DecimalValue::I8(v) => bigdecimal_i8(env, v, scale), - DecimalValue::I16(v) => bigdecimal_i16(env, v, scale), - DecimalValue::I32(v) => bigdecimal_i32(env, v, scale), - DecimalValue::I64(v) => bigdecimal_i64(env, v, scale), - DecimalValue::I128(v) => bigdecimal_i128(env, v, scale), - DecimalValue::I256(v) => bigdecimal_i256(env, v, scale), - } - } else { - Ok(JObject::null().into_raw()) - } - }) -} - -static BIGDECIMAL_CLASS: &str = "java/math/BigDecimal"; -static BIGINT_CLASS: &str = "java/math/BigInteger"; - -macro_rules! bigdecimal_from_bytes { - ($typ:ty, $name:ident) => { - fn $name(env: &mut JNIEnv, value: $typ, scale: i8) -> Result { - // NOTE: BigInteger constructor expects big-endian bytes. - let be_bytes = value.to_be_bytes(); - let bytearray = env.byte_array_from_slice(&be_bytes)?; - let bigint = env.new_object(BIGINT_CLASS, "([B)V", &[JValue::from(&bytearray)])?; - - // Create the BigDecimal from a BigInteger + scale - let bigdecimal = env.new_object( - BIGDECIMAL_CLASS, - "(Ljava/math/BigInteger;I)V", - &[JValue::from(&bigint), JValue::from(scale as jint)], - )?; - - Ok(bigdecimal.into_raw()) - } - }; -} - -bigdecimal_from_bytes!(i8, bigdecimal_i8); -bigdecimal_from_bytes!(i16, bigdecimal_i16); -bigdecimal_from_bytes!(i32, bigdecimal_i32); -bigdecimal_from_bytes!(i64, bigdecimal_i64); -bigdecimal_from_bytes!(i128, bigdecimal_i128); -bigdecimal_from_bytes!(i256, bigdecimal_i256); - -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeArrayMethods_getBool( - mut env: JNIEnv, - _class: JClass, - array_ptr: jlong, - index: jint, -) -> jboolean { - let array_ref = unsafe { NativeArray::from_ptr(array_ptr) }; - try_or_throw(&mut env, |_| { - let value = array_ref.inner.scalar_at(index as usize)?; - match value.as_bool().value() { - None => Ok(JNI_FALSE), - Some(b) => { - if b { - Ok(JNI_TRUE) - } else { - Ok(JNI_FALSE) - } - } - } - }) -} - -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeArrayMethods_getUTF8<'local>( - mut env: JNIEnv<'local>, - _class: JClass<'local>, - array_ptr: jlong, - index: jint, -) -> jstring { - let array_ref = unsafe { NativeArray::from_ptr(array_ptr) }; - try_or_throw(&mut env, |env| { - let value = array_ref.inner.scalar_at(index as usize)?; - match value.as_utf8().value() { - None => Ok(JObject::null().into_raw()), - Some(buf_str) => Ok(env.new_string(buf_str.as_str())?.into_raw()), - } - }) -} - -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeArrayMethods_getUTF8_1ptr_1len<'local>( - mut env: JNIEnv<'local>, - _class: JClass<'local>, - array_ptr: jlong, - index: jint, - out_ptr: JLongArray<'local>, - out_len: JIntArray<'local>, -) { - let array_ref = unsafe { NativeArray::from_ptr(array_ptr) }; - - try_or_throw(&mut env, |env| { - if !array_ref.inner.dtype().is_utf8() { - throw_runtime!("getUTF8_ptr_len expected UTF8 array"); - } - - if let Some(varbin) = array_ref.inner.as_opt::() { - let (ptr, len) = get_ptr_len_varbin(index, varbin); - env.set_long_array_region(&out_ptr, 0, &[ptr as jlong])?; - env.set_int_array_region(&out_len, 0, &[len as jint])?; - } else if let Some(varbinview) = array_ref.inner.as_opt::() { - let (ptr, len) = get_ptr_len_view(index, varbinview); - env.set_long_array_region(&out_ptr, 0, &[ptr as jlong])?; - env.set_int_array_region(&out_len, 0, &[len as jint])?; - } else { - throw_runtime!("getUTF8_ptr_len expected VarBin or VarBinView"); - } - Ok(()) - }) -} - -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeArrayMethods_getBinary<'local>( - mut env: JNIEnv<'local>, - _class: JClass<'local>, - array_ptr: jlong, - index: jint, -) -> jbyteArray { - let array_ref = unsafe { NativeArray::from_ptr(array_ptr) }; - try_or_throw(&mut env, |env| { - let value = array_ref.inner.scalar_at(index as usize)?; - match value.as_binary().value() { - None => Ok(JObject::null().into_raw()), - Some(buf) => Ok(env.byte_array_from_slice(buf.as_slice())?.into_raw()), - } - }) -} - -/// Get a raw pointer + len to pass back to Java to avoid copying across the boundary. -/// -/// Panics if the index is out of bounds. -fn get_ptr_len_varbin(index: jint, array: ArrayView) -> (*const u8, u32) { - // TODO: propagate this error up instead of expecting - let bytes = array.bytes_at(usize::try_from(index).vortex_expect("index must fit in usize")); - ( - bytes.as_ptr(), - // TODO: propagate this error up instead of expecting - u32::try_from(bytes.len()).vortex_expect("string length must fit in u32"), - ) -} - -/// Get a raw pointer + len to pass back to Java to avoid copying across the boundary. -fn get_ptr_len_view(index: jint, array: ArrayView) -> (*const u8, u32) { - // TODO: propagate this error up instead of expecting - let bytes = array.bytes_at(usize::try_from(index).vortex_expect("index must fit in usize")); - ( - bytes.as_ptr(), - // TODO: propagate this error up instead of expecting - u32::try_from(bytes.len()).vortex_expect("string length must fit in u32"), - ) -} diff --git a/vortex-jni/src/array_iter.rs b/vortex-jni/src/array_iter.rs deleted file mode 100644 index 0306fed8950..00000000000 --- a/vortex-jni/src/array_iter.rs +++ /dev/null @@ -1,122 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -use jni::JNIEnv; -use jni::objects::JClass; -use jni::sys::jlong; -use vortex::array::iter::ArrayIterator; -use vortex::dtype::DType; - -use crate::array::NativeArray; -use crate::errors::try_or_throw; - -/// Blocking JNI bridge to a Vortex [`ArrayIterator`]. -pub struct NativeArrayIterator { - inner: Option>, -} - -impl NativeArrayIterator { - pub fn new(stream: Box) -> Box { - Box::new(Self { - inner: Some(stream), - }) - } - - pub fn into_raw(self: Box) -> jlong { - Box::into_raw(self) as jlong - } - - #[expect( - clippy::expect_used, - reason = "JNI contract guarantees non-null pointer" - )] - pub unsafe fn from_ptr<'a>(pointer: jlong) -> &'a Self { - unsafe { - (pointer as *const NativeArrayIterator) - .as_ref() - .expect("Pointer should never be null") - } - } - - #[expect( - clippy::expect_used, - reason = "JNI contract guarantees non-null pointer" - )] - pub unsafe fn from_ptr_mut<'a>(pointer: jlong) -> &'a mut Self { - unsafe { - (pointer as *mut NativeArrayIterator) - .as_mut() - .expect("Pointer should never be null") - } - } - - pub unsafe fn from_raw(pointer: jlong) -> Box { - unsafe { Box::from_raw(pointer as *mut NativeArrayIterator) } - } -} - -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeArrayIteratorMethods_free( - _env: JNIEnv, - _class: JClass, - pointer: jlong, -) { - drop(unsafe { NativeArrayIterator::from_raw(pointer) }); -} - -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeArrayIteratorMethods_take( - mut env: JNIEnv, - _class: JClass, - pointer: jlong, -) -> jlong { - let iter = unsafe { NativeArrayIterator::from_ptr_mut(pointer) }; - - try_or_throw(&mut env, |_| { - if let Some(mut inner) = iter.inner.take() { - let result = match inner.next() { - Some(result) => { - match result { - Ok(array_ref) => { - // Successfully got the next array - let ptr = NativeArray::new(array_ref).into_raw(); - Ok(ptr) - } - Err(e) => { - // Error occurred, but we still need to restore the iterator - Err(e.into()) - } - } - } - None => { - // Iterator is exhausted - Ok(-1) - } - }; - - // Always restore the iterator, even on error - iter.inner = Some(inner); - result - } else { - throw_runtime!("attempted to take() on a closed ArrayIter"); - } - }) -} - -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeArrayIteratorMethods_getDType( - mut env: JNIEnv, - _class: JClass, - pointer: jlong, -) -> jlong { - let iter = unsafe { NativeArrayIterator::from_ptr(pointer) }; - - try_or_throw(&mut env, |_| { - if let Some(ref inner) = iter.inner { - let dtype = inner.dtype(); - Ok(dtype as *const DType as jlong) - } else { - throw_runtime!("NativeArrayMethods.getDType: closed stream"); - } - }) -} diff --git a/vortex-jni/src/data_source.rs b/vortex-jni/src/data_source.rs new file mode 100644 index 00000000000..d62b583a4a9 --- /dev/null +++ b/vortex-jni/src/data_source.rs @@ -0,0 +1,267 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! JNI bindings for [`vortex::scan::DataSource`] (see the equivalent types in +//! `vortex-ffi/src/data_source.rs`). +//! +//! Glob handling mirrors `vortex-duckdb`'s `VortexMultiFileScan`: +//! * full URLs (`s3://...`, `file:///...`) are used as-is, +//! * bare file paths are made absolute and have `.`/`..` components normalized, +//! * filesystems are cached per base URL so repeated globs against the same bucket share +//! a single client. + +use std::path::Component; +use std::path::Path; +use std::path::PathBuf; +use std::path::absolute; +use std::sync::Arc; + +use jni::EnvUnowned; +use jni::objects::JClass; +use jni::objects::JLongArray; +use jni::objects::JObject; +use jni::objects::JObjectArray; +use jni::objects::JString; +use jni::sys::jlong; +use url::Url; +use vortex::error::VortexResult; +use vortex::error::vortex_err; +use vortex::expr::stats::Precision; +use vortex::file::multi::MultiFileDataSource; +use vortex::io::filesystem::FileSystemRef; +use vortex::io::runtime::BlockingRuntime; +use vortex::io::session::RuntimeSessionExt; +use vortex::scan::DataSourceRef; +use vortex::utils::aliases::hash_map::HashMap; + +use crate::RUNTIME; +use crate::dtype::export_dtype_to_arrow; +use crate::errors::try_or_throw; +use crate::file::extract_properties; +use crate::object_store::object_store_fs; +use crate::session::session_ref; + +/// Wraps an `Arc` behind a single pointer. +pub(crate) struct NativeDataSource { + inner: DataSourceRef, +} + +impl NativeDataSource { + fn into_raw(self: Box) -> jlong { + Box::into_raw(self) as jlong + } + + /// SAFETY: pointer must have been returned from [`Self::into_raw`]. + pub(crate) unsafe fn from_ptr<'a>(ptr: jlong) -> &'a Self { + debug_assert!(ptr != 0, "null data source pointer"); + unsafe { &*(ptr as *const Self) } + } + + pub(crate) fn inner(&self) -> &DataSourceRef { + &self.inner + } +} + +#[unsafe(no_mangle)] +pub extern "system" fn Java_dev_vortex_jni_NativeDataSource_open( + mut env: EnvUnowned, + _class: JClass, + session_ptr: jlong, + uris: JObjectArray, + options: JObject, +) -> jlong { + try_or_throw(&mut env, |env| { + let session = unsafe { session_ref(session_ptr) }; + let properties = extract_properties(env, &options)?; + + let mut glob_strings = Vec::new(); + let uri_count = uris.len(env)?; + for idx in 0..uri_count { + let uri = uris.get_element(env, idx)?; + let uri = env.cast_local::(uri)?; + let uri: String = uri.try_to_string(env)?; + let uri = uri.trim(); + if !uri.is_empty() { + glob_strings.push(uri.to_owned()); + } + } + if glob_strings.is_empty() { + return Err(vortex_err!("no paths provided").into()); + } + + let glob_urls: Vec = glob_strings + .iter() + .map(|g| parse_glob_url(g.as_str())) + .collect::>()?; + + let mut fs_cache: HashMap = HashMap::new(); + for glob_url in &glob_urls { + let base = base_url(glob_url); + if !fs_cache.contains_key(&base) { + let fs = object_store_fs(glob_url, &properties, session.handle())?; + fs_cache.insert(base, fs); + } + } + + let mut builder = MultiFileDataSource::new(session.clone()); + for glob_url in &glob_urls { + let base = base_url(glob_url); + let fs = fs_cache + .get(&base) + .cloned() + .unwrap_or_else(|| unreachable!("fs cached for every base url")); + builder = builder.with_glob(glob_url.path(), Some(fs)); + } + + let inner = RUNTIME + .block_on(builder.build()) + .map(|ds| Arc::new(ds) as DataSourceRef)?; + Ok(Box::new(NativeDataSource { inner }).into_raw()) + }) +} + +/// Parse a glob string into a [`Url`]. Accepts full URLs and bare (relative or absolute) +/// file paths — see the module docs for details. +fn parse_glob_url(glob: &str) -> VortexResult { + // `Url::parse` accepts Windows absolute paths like `C:\foo` as a URL with a + // single-letter scheme (`c`). No real URL scheme is one character, so treat any + // single-letter scheme as a filesystem path instead. + if let Ok(url) = Url::parse(glob) + && url.scheme().len() > 1 + { + return Ok(url); + } + let path = + absolute(Path::new(glob)).map_err(|e| vortex_err!("failed to absolutize {glob}: {e}"))?; + let path = normalize_path(path); + Url::from_file_path(path).map_err(|_| vortex_err!("neither URL nor path: {glob}")) +} + +/// Normalize `.` and `..` without touching the filesystem. +fn normalize_path(path: PathBuf) -> PathBuf { + let mut out = PathBuf::new(); + for component in path.components() { + match component { + Component::CurDir => {} + Component::ParentDir => { + out.pop(); + } + c => out.push(c), + } + } + out +} + +/// URL with the path cleared, used as a cache key for filesystem reuse. +fn base_url(url: &Url) -> Url { + let mut base = url.clone(); + base.set_path(""); + base +} + +#[unsafe(no_mangle)] +pub extern "system" fn Java_dev_vortex_jni_NativeDataSource_free( + _env: EnvUnowned, + _class: JClass, + pointer: jlong, +) { + if pointer == 0 { + return; + } + drop(unsafe { Box::from_raw(pointer as *mut NativeDataSource) }); +} + +/// Export the data source's schema into the Arrow C Data Interface schema struct at +/// `schema_addr`. +#[unsafe(no_mangle)] +pub extern "system" fn Java_dev_vortex_jni_NativeDataSource_arrowSchema( + mut env: EnvUnowned, + _class: JClass, + pointer: jlong, + schema_addr: jlong, +) { + try_or_throw(&mut env, |_| { + if schema_addr == 0 { + throw_runtime!("null arrow schema address"); + } + let ds = unsafe { NativeDataSource::from_ptr(pointer) }; + export_dtype_to_arrow(ds.inner.dtype(), schema_addr)?; + Ok(()) + }); +} + +/// Write the row count into the two-slot jlong pair `out`: +/// `out[0]` receives the row count (0 when unknown), `out[1]` the cardinality (0=unknown, 1=estimate, 2=exact). +#[unsafe(no_mangle)] +pub extern "system" fn Java_dev_vortex_jni_NativeDataSource_rowCount( + mut env: EnvUnowned, + _class: JClass, + pointer: jlong, + out: JLongArray, +) { + try_or_throw(&mut env, |env| { + let ds = unsafe { NativeDataSource::from_ptr(pointer) }; + let (rows, cardinality) = match ds.inner.row_count() { + Some(Precision::Exact(r)) => (r as jlong, 2), + Some(Precision::Inexact(r)) => (r as jlong, 1), + None => (0, 0), + }; + out.set_region(env, 0, &[rows, cardinality])?; + Ok(()) + }); +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn test_parse_glob_url_full_url() { + let url = parse_glob_url("s3://bucket/prefix/*.vortex").unwrap(); + assert_eq!(url.scheme(), "s3"); + assert_eq!(url.host_str(), Some("bucket")); + assert_eq!(url.path(), "/prefix/*.vortex"); + } + + #[test] + fn test_parse_glob_url_absolute_path() { + // Use a drive-prefixed input on Windows so `absolute()` doesn't inject the cwd drive + // and the expected URL path is predictable. + #[cfg(unix)] + let (input, expected_path) = ("/tmp/data/*.vortex", "/tmp/data/*.vortex"); + #[cfg(windows)] + let (input, expected_path) = (r"C:\tmp\data\*.vortex", "/C:/tmp/data/*.vortex"); + let url = parse_glob_url(input).unwrap(); + assert_eq!(url.scheme(), "file"); + assert_eq!(url.path(), expected_path); + } + + #[test] + fn test_parse_glob_url_normalizes_dots() { + #[cfg(unix)] + let (input, expected_path) = ("/a/b/../c/./d", "/a/c/d"); + #[cfg(windows)] + let (input, expected_path) = (r"C:\a\b\..\c\.\d", "/C:/a/c/d"); + let url = parse_glob_url(input).unwrap(); + assert_eq!(url.path(), expected_path); + } + + #[test] + fn test_parse_glob_url_single_letter_scheme_is_path() { + // Regression: `Url::parse("C:\\tmp")` succeeds with scheme="c"; the function must + // treat that as a filesystem path, not a URL. Exercised on all platforms because + // the check lives in `parse_glob_url`, not in an OS-specific branch. + let url = parse_glob_url(r"C:\tmp\data\*.vortex").unwrap(); + assert_eq!(url.scheme(), "file"); + assert_ne!(url.scheme(), "c"); + } + + #[test] + fn test_base_url_strips_path() { + let url = Url::parse("s3://bucket/a/b/c").unwrap(); + let base = base_url(&url); + assert_eq!(base.scheme(), "s3"); + assert_eq!(base.host_str(), Some("bucket")); + assert_eq!(base.path(), ""); + } +} diff --git a/vortex-jni/src/dtype.rs b/vortex-jni/src/dtype.rs index 3c8666127ae..f748135ade1 100644 --- a/vortex-jni/src/dtype.rs +++ b/vortex-jni/src/dtype.rs @@ -1,658 +1,78 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -use std::sync::Arc; +//! Bridge functions between a Vortex [`DType`] and Arrow's C Data Interface +//! [`FFI_ArrowSchema`]. Java receives DType information exclusively as Arrow schema. -use jni::JNIEnv; -use jni::objects::JClass; -use jni::objects::JLongArray; -use jni::objects::JObject; -use jni::objects::JObjectArray; -use jni::objects::JString; -use jni::objects::JValue; -use jni::sys::JNI_FALSE; -use jni::sys::JNI_TRUE; -use jni::sys::jboolean; -use jni::sys::jbyte; -use jni::sys::jint; -use jni::sys::jlong; -use jni::sys::jobject; -use jni::sys::jstring; -use vortex::dtype::DType; -use vortex::dtype::DecimalDType; -use vortex::dtype::Nullability; -use vortex::dtype::PType; -use vortex::dtype::StructFields; -use vortex::error::vortex_err; -use vortex::extension::datetime::AnyTemporal; -use vortex::extension::datetime::Date; -use vortex::extension::datetime::Time; -use vortex::extension::datetime::TimeUnit; -use vortex::extension::datetime::Timestamp; - -use crate::errors::JNIError; -use crate::errors::try_or_throw; - -pub const DTYPE_NULL: jbyte = 0; -pub const DTYPE_BOOL: jbyte = 1; -pub const DTYPE_PRIMITIVE_U8: jbyte = 2; -pub const DTYPE_PRIMITIVE_U16: jbyte = 3; -pub const DTYPE_PRIMITIVE_U32: jbyte = 4; -pub const DTYPE_PRIMITIVE_U64: jbyte = 5; -pub const DTYPE_PRIMITIVE_I8: jbyte = 6; -pub const DTYPE_PRIMITIVE_I16: jbyte = 7; -pub const DTYPE_PRIMITIVE_I32: jbyte = 8; -pub const DTYPE_PRIMITIVE_I64: jbyte = 9; -pub const DTYPE_PRIMITIVE_F16: jbyte = 10; -pub const DTYPE_PRIMITIVE_F32: jbyte = 11; -pub const DTYPE_PRIMITIVE_F64: jbyte = 12; -pub const DTYPE_UTF8: jbyte = 13; -pub const DTYPE_BINARY: jbyte = 14; -pub const DTYPE_STRUCT: jbyte = 15; -pub const DTYPE_LIST: jbyte = 16; -pub const DTYPE_EXTENSION: jbyte = 17; -pub const DTYPE_DECIMAL: jbyte = 18; -pub const DTYPE_FIXED_SIZE_LIST: jbyte = 19; - -static LONG_CLASS: &str = "java/lang/Long"; - -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeDTypeMethods_free( - _env: JNIEnv, - _class: JClass, - dtype_ptr: jlong, -) { - // SAFETY: caller must ensure that the pointer is valid and points to a `DType`. - drop(unsafe { Box::from_raw(dtype_ptr as *mut DType) }); -} - -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeDTypeMethods_getVariant( - _env: JNIEnv, - _class: JClass, - dtype_ptr: jlong, -) -> jbyte { - // SAFETY: caller must ensure that the pointer is valid and points to a `DType`. - let dtype = unsafe { &*(dtype_ptr as *const DType) }; - match dtype { - DType::Null => DTYPE_NULL, - DType::Bool(_) => DTYPE_BOOL, - DType::Primitive(ptype, _) => match ptype { - PType::U8 => DTYPE_PRIMITIVE_U8, - PType::U16 => DTYPE_PRIMITIVE_U16, - PType::U32 => DTYPE_PRIMITIVE_U32, - PType::U64 => DTYPE_PRIMITIVE_U64, - PType::I8 => DTYPE_PRIMITIVE_I8, - PType::I16 => DTYPE_PRIMITIVE_I16, - PType::I32 => DTYPE_PRIMITIVE_I32, - PType::I64 => DTYPE_PRIMITIVE_I64, - PType::F16 => DTYPE_PRIMITIVE_F16, - PType::F32 => DTYPE_PRIMITIVE_F32, - PType::F64 => DTYPE_PRIMITIVE_F64, - }, - DType::Decimal(..) => DTYPE_DECIMAL, - DType::Utf8(_) => DTYPE_UTF8, - DType::Binary(_) => DTYPE_BINARY, - DType::Struct(..) => DTYPE_STRUCT, - DType::List(..) => DTYPE_LIST, - DType::FixedSizeList(..) => DTYPE_FIXED_SIZE_LIST, - DType::Extension(_) => DTYPE_EXTENSION, - DType::Variant(_) => unimplemented!("Variant DType is not supported in JNI yet"), - } -} +use std::ptr; -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeDTypeMethods_isNullable( - _env: JNIEnv, - _class: JClass, - dtype_ptr: jlong, -) -> jboolean { - // SAFETY: caller must ensure that the pointer is valid and points to a `DType`. - let dtype = unsafe { &*(dtype_ptr as *const DType) }; - if dtype.is_nullable() { - JNI_TRUE - } else { - JNI_FALSE +use arrow_array::ffi::FFI_ArrowSchema; +use arrow_schema::DataType; +use arrow_schema::FieldRef; +use arrow_schema::Fields; +use vortex::dtype::DType; +use vortex::dtype::arrow::FromArrowType; +use vortex::error::VortexResult; + +/// Export a Vortex [`DType`] to the Arrow C Data Interface struct at `schema_addr`. Views +/// (Utf8View/BinaryView) are downgraded to regular Utf8/Binary so Spark and other consumers +/// without view support can read them. +pub(crate) fn export_dtype_to_arrow(dtype: &DType, schema_addr: i64) -> VortexResult<()> { + let arrow_schema = dtype.to_arrow_schema()?; + let viewless = strip_views(DataType::Struct(arrow_schema.fields().clone())); + let fields = match viewless { + DataType::Struct(fields) => fields, + _ => unreachable!("Vortex DType always exports as a struct"), + }; + let schema = arrow_schema::Schema::new(fields); + let ffi_schema = FFI_ArrowSchema::try_from(&schema)?; + unsafe { + ptr::write(schema_addr as *mut FFI_ArrowSchema, ffi_schema); } -} - -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeDTypeMethods_getFieldNames( - mut env: JNIEnv, - _class: JClass, - dtype_ptr: jlong, -) -> jobject { - let dtype = unsafe { &*(dtype_ptr as *const DType) }; - - try_or_throw(&mut env, |env| { - let array_list = env.new_object("java/util/ArrayList", "()V", &[])?; - let field_names = env.get_list(&array_list)?; - let Some(struct_dtype) = dtype.as_struct_fields_opt() else { - throw_runtime!("DType should be STRUCT, was {dtype}"); - }; - - for name in struct_dtype.names().iter() { - let field = env.new_string(name)?; - field_names.add(env, field.as_ref())?; + Ok(()) +} + +/// Replace view-based Arrow types with their non-view counterparts throughout the tree. +pub(crate) fn strip_views(data_type: DataType) -> DataType { + match data_type { + DataType::BinaryView => DataType::Binary, + DataType::Utf8View => DataType::Utf8, + DataType::List(inner) | DataType::ListView(inner) => { + let new_inner = (*inner) + .clone() + .with_data_type(strip_views(inner.data_type().clone())); + DataType::List(FieldRef::new(new_inner)) } - - Ok::(array_list.into_raw()) - }) -} - -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeDTypeMethods_getFieldTypes( - mut env: JNIEnv, - _class: JClass, - dtype_ptr: jlong, -) -> jobject { - let dtype = unsafe { &*(dtype_ptr as *const DType) }; - - try_or_throw(&mut env, |env| { - let array_list = env - .new_object("java/util/ArrayList", "()V", &[]) - .map_err(|e| JNIError::Vortex(vortex_err!("failure constructing ArrayList: {e}")))?; - let field_types = env.get_list(&array_list)?; - let Some(struct_dtype) = dtype.as_struct_fields_opt() else { - throw_runtime!("DType should be STRUCT, was {dtype}"); - }; - - for field_dtype in struct_dtype.fields() { - let ptr: *mut DType = Box::into_raw(Box::new(field_dtype)); - let boxed = env - .call_static_method( - LONG_CLASS, - "valueOf", - "(J)Ljava/lang/Long;", - &[JValue::Long(ptr.addr() as jlong)], - )? - .l()?; - field_types.add(env, &boxed)?; + DataType::LargeList(inner) | DataType::LargeListView(inner) => { + let new_inner = (*inner) + .clone() + .with_data_type(strip_views(inner.data_type().clone())); + DataType::LargeList(FieldRef::new(new_inner)) } - - Ok(array_list.into_raw()) - }) -} - -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeDTypeMethods_getElementType( - mut env: JNIEnv, - _class: JClass, - dtype_ptr: jlong, -) -> jlong { - let dtype = unsafe { &*(dtype_ptr as *const DType) }; - - try_or_throw(&mut env, |_| { - let element_type = dtype - .as_list_element_opt() - .or_else(|| dtype.as_fixed_size_list_element_opt()); - let Some(element_type) = element_type else { - throw_runtime!("DType should be LIST or FIXED_SIZE_LIST, was {dtype}"); - }; - - Ok(element_type.as_ref() as *const DType as jlong) - }) -} - -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeDTypeMethods_isDate( - mut env: JNIEnv, - _class: JClass, - dtype_ptr: jlong, -) -> jboolean { - let dtype = unsafe { &*(dtype_ptr as *const DType) }; - - try_or_throw(&mut env, |_| { - let DType::Extension(ext_dtype) = dtype else { - throw_runtime!("DType should be an EXTENSION, was {dtype}"); - }; - - if ext_dtype.is::() { - Ok(JNI_TRUE) - } else { - Ok(JNI_FALSE) + DataType::Struct(fields) => { + let viewless_fields: Vec = fields + .iter() + .map(|field_ref| { + let field = (**field_ref).clone(); + let data_type = field.data_type().clone(); + FieldRef::new(field.with_data_type(strip_views(data_type))) + }) + .collect(); + DataType::Struct(Fields::from(viewless_fields)) } - }) -} - -#[unsafe(no_mangle)] -pub extern "system" fn Java_dev_vortex_jni_NativeDTypeMethods_isTime( - mut env: JNIEnv, - _class: JClass, - dtype_ptr: jlong, -) -> jboolean { - let dtype = unsafe { &*(dtype_ptr as *const DType) }; - - try_or_throw(&mut env, |_| { - let DType::Extension(ext_dtype) = dtype else { - throw_runtime!("DType should be an EXTENSION, was {dtype}"); - }; - - if ext_dtype.is:: -pub fn vortex_layout::scan::repeated_scan::RepeatedScan::execute(&self, row_range: core::option::Option>) -> vortex_error::VortexResult>>>> +pub fn vortex_layout::scan::repeated_scan::RepeatedScan::execute(&self, core::option::Option>) -> vortex_error::VortexResult>>>> -pub fn vortex_layout::scan::repeated_scan::RepeatedScan::execute_stream(&self, row_range: core::option::Option>) -> vortex_error::VortexResult> + core::marker::Send + 'static + use> +pub fn vortex_layout::scan::repeated_scan::RepeatedScan::execute_stream(&self, core::option::Option>) -> vortex_error::VortexResult> + core::marker::Send + 'static + use> -pub fn vortex_layout::scan::repeated_scan::RepeatedScan::new(session: vortex_session::VortexSession, layout_reader: vortex_layout::LayoutReaderRef, projection: vortex_array::expr::expression::Expression, filter: core::option::Option, ordered: bool, row_range: core::option::Option>, selection: vortex_scan::selection::Selection, splits: vortex_layout::scan::splits::Splits, concurrency: usize, map_fn: alloc::sync::Arc<(dyn core::ops::function::Fn(vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult + core::marker::Send + core::marker::Sync)>, limit: core::option::Option, dtype: vortex_array::dtype::DType) -> Self +pub fn vortex_layout::scan::repeated_scan::RepeatedScan::new(vortex_session::VortexSession, vortex_layout::LayoutReaderRef, vortex_array::expr::expression::Expression, core::option::Option, bool, core::option::Option>, vortex_scan::selection::Selection, vortex_layout::scan::splits::Splits, usize, alloc::sync::Arc<(dyn core::ops::function::Fn(vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult + core::marker::Send + core::marker::Sync)>, core::option::Option, vortex_array::dtype::DType) -> Self pub mod vortex_layout::scan::scan_builder @@ -1054,17 +1048,17 @@ pub struct vortex_layout::scan::scan_builder::ScanBuilder impl vortex_layout::scan::scan_builder::ScanBuilder -pub fn vortex_layout::scan::scan_builder::ScanBuilder::into_array_iter(self, runtime: &B) -> vortex_error::VortexResult +pub fn vortex_layout::scan::scan_builder::ScanBuilder::into_array_iter(self, &B) -> vortex_error::VortexResult pub fn vortex_layout::scan::scan_builder::ScanBuilder::into_array_stream(self) -> vortex_error::VortexResult -pub fn vortex_layout::scan::scan_builder::ScanBuilder::new(session: vortex_session::VortexSession, layout_reader: alloc::sync::Arc) -> Self +pub fn vortex_layout::scan::scan_builder::ScanBuilder::new(vortex_session::VortexSession, alloc::sync::Arc) -> Self impl vortex_layout::scan::scan_builder::ScanBuilder -pub fn vortex_layout::scan::scan_builder::ScanBuilder::into_record_batch_reader(self, schema: arrow_schema::schema::SchemaRef, runtime: &B) -> vortex_error::VortexResult +pub fn vortex_layout::scan::scan_builder::ScanBuilder::into_record_batch_reader(self, arrow_schema::schema::SchemaRef, &B) -> vortex_error::VortexResult -pub fn vortex_layout::scan::scan_builder::ScanBuilder::into_record_batch_stream(self, schema: arrow_schema::schema::SchemaRef) -> vortex_error::VortexResult> + core::marker::Send + 'static> +pub fn vortex_layout::scan::scan_builder::ScanBuilder::into_record_batch_stream(self, arrow_schema::schema::SchemaRef) -> vortex_error::VortexResult> + core::marker::Send + 'static> impl vortex_layout::scan::scan_builder::ScanBuilder @@ -1074,11 +1068,11 @@ pub fn vortex_layout::scan::scan_builder::ScanBuilder::concurrency(&self) -> pub fn vortex_layout::scan::scan_builder::ScanBuilder::dtype(&self) -> vortex_error::VortexResult -pub fn vortex_layout::scan::scan_builder::ScanBuilder::into_iter(self, runtime: &B) -> vortex_error::VortexResult> + 'static> +pub fn vortex_layout::scan::scan_builder::ScanBuilder::into_iter(self, &B) -> vortex_error::VortexResult> + 'static> pub fn vortex_layout::scan::scan_builder::ScanBuilder::into_stream(self) -> vortex_error::VortexResult> + core::marker::Send + 'static + use> -pub fn vortex_layout::scan::scan_builder::ScanBuilder::map(self, map_fn: impl core::ops::function::Fn(A) -> vortex_error::VortexResult + 'static + core::marker::Send + core::marker::Sync) -> vortex_layout::scan::scan_builder::ScanBuilder +pub fn vortex_layout::scan::scan_builder::ScanBuilder::map(self, impl core::ops::function::Fn(A) -> vortex_error::VortexResult + 'static + core::marker::Send + core::marker::Sync) -> vortex_layout::scan::scan_builder::ScanBuilder pub fn vortex_layout::scan::scan_builder::ScanBuilder::ordered(&self) -> bool @@ -1086,35 +1080,35 @@ pub fn vortex_layout::scan::scan_builder::ScanBuilder::prepare(self) -> vorte pub fn vortex_layout::scan::scan_builder::ScanBuilder::session(&self) -> &vortex_session::VortexSession -pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_concurrency(self, concurrency: usize) -> Self +pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_concurrency(self, usize) -> Self -pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_filter(self, filter: vortex_array::expr::expression::Expression) -> Self +pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_filter(self, vortex_array::expr::expression::Expression) -> Self -pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_limit(self, limit: u64) -> Self +pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_limit(self, u64) -> Self -pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_metrics_registry(self, metrics: alloc::sync::Arc) -> Self +pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_metrics_registry(self, alloc::sync::Arc) -> Self -pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_ordered(self, ordered: bool) -> Self +pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_ordered(self, bool) -> Self -pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_projection(self, projection: vortex_array::expr::expression::Expression) -> Self +pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_projection(self, vortex_array::expr::expression::Expression) -> Self -pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_row_indices(self, row_indices: vortex_buffer::buffer::Buffer) -> Self +pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_row_indices(self, vortex_buffer::buffer::Buffer) -> Self -pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_row_offset(self, row_offset: u64) -> Self +pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_row_offset(self, u64) -> Self -pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_row_range(self, row_range: core::ops::range::Range) -> Self +pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_row_range(self, core::ops::range::Range) -> Self -pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_selection(self, selection: vortex_scan::selection::Selection) -> Self +pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_selection(self, vortex_scan::selection::Selection) -> Self -pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_some_filter(self, filter: core::option::Option) -> Self +pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_some_filter(self, core::option::Option) -> Self -pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_some_limit(self, limit: core::option::Option) -> Self +pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_some_limit(self, core::option::Option) -> Self -pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_some_metrics_registry(self, metrics: core::option::Option>) -> Self +pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_some_metrics_registry(self, core::option::Option>) -> Self -pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_split_by(self, split_by: vortex_layout::scan::split_by::SplitBy) -> Self +pub fn vortex_layout::scan::scan_builder::ScanBuilder::with_split_by(self, vortex_layout::scan::split_by::SplitBy) -> Self -pub fn vortex_layout::scan::scan_builder::filter_and_projection_masks(projection: &vortex_array::expr::expression::Expression, filter: core::option::Option<&vortex_array::expr::expression::Expression>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult<(alloc::vec::Vec, alloc::vec::Vec)> +pub fn vortex_layout::scan::scan_builder::filter_and_projection_masks(&vortex_array::expr::expression::Expression, core::option::Option<&vortex_array::expr::expression::Expression>, &vortex_array::dtype::DType) -> vortex_error::VortexResult<(alloc::vec::Vec, alloc::vec::Vec)> pub mod vortex_layout::scan::split_by @@ -1126,7 +1120,7 @@ pub vortex_layout::scan::split_by::SplitBy::RowCount(usize) impl vortex_layout::scan::split_by::SplitBy -pub fn vortex_layout::scan::split_by::SplitBy::splits(&self, layout_reader: &dyn vortex_layout::LayoutReader, row_range: &core::ops::range::Range, field_mask: &[vortex_array::dtype::field_mask::FieldMask]) -> vortex_error::VortexResult> +pub fn vortex_layout::scan::split_by::SplitBy::splits(&self, &dyn vortex_layout::LayoutReader, &core::ops::range::Range, &[vortex_array::dtype::field_mask::FieldMask]) -> vortex_error::VortexResult> impl core::clone::Clone for vortex_layout::scan::split_by::SplitBy @@ -1138,7 +1132,7 @@ pub fn vortex_layout::scan::split_by::SplitBy::default() -> vortex_layout::scan: impl core::fmt::Debug for vortex_layout::scan::split_by::SplitBy -pub fn vortex_layout::scan::split_by::SplitBy::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_layout::scan::split_by::SplitBy::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_layout::scan::split_by::SplitBy @@ -1148,43 +1142,43 @@ pub struct vortex_layout::segments::InstrumentedSegmentCache impl vortex_layout::segments::InstrumentedSegmentCache -pub fn vortex_layout::segments::InstrumentedSegmentCache::new(segment_cache: C, metrics_registry: &dyn vortex_metrics::MetricsRegistry, labels: alloc::vec::Vec) -> Self +pub fn vortex_layout::segments::InstrumentedSegmentCache::new(C, &dyn vortex_metrics::MetricsRegistry, alloc::vec::Vec) -> Self impl vortex_layout::segments::SegmentCache for vortex_layout::segments::InstrumentedSegmentCache -pub fn vortex_layout::segments::InstrumentedSegmentCache::get<'life0, 'async_trait>(&'life0 self, id: vortex_layout::segments::SegmentId) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait +pub fn vortex_layout::segments::InstrumentedSegmentCache::get<'life0, 'async_trait>(&'life0 self, vortex_layout::segments::SegmentId) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait -pub fn vortex_layout::segments::InstrumentedSegmentCache::put<'life0, 'async_trait>(&'life0 self, id: vortex_layout::segments::SegmentId, buffer: vortex_buffer::ByteBuffer) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait +pub fn vortex_layout::segments::InstrumentedSegmentCache::put<'life0, 'async_trait>(&'life0 self, vortex_layout::segments::SegmentId, vortex_buffer::ByteBuffer) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait pub struct vortex_layout::segments::MokaSegmentCache(_) impl vortex_layout::segments::MokaSegmentCache -pub fn vortex_layout::segments::MokaSegmentCache::new(max_capacity_bytes: u64) -> Self +pub fn vortex_layout::segments::MokaSegmentCache::new(u64) -> Self impl vortex_layout::segments::SegmentCache for vortex_layout::segments::MokaSegmentCache -pub fn vortex_layout::segments::MokaSegmentCache::get<'life0, 'async_trait>(&'life0 self, id: vortex_layout::segments::SegmentId) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait +pub fn vortex_layout::segments::MokaSegmentCache::get<'life0, 'async_trait>(&'life0 self, vortex_layout::segments::SegmentId) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait -pub fn vortex_layout::segments::MokaSegmentCache::put<'life0, 'async_trait>(&'life0 self, id: vortex_layout::segments::SegmentId, buffer: vortex_buffer::ByteBuffer) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait +pub fn vortex_layout::segments::MokaSegmentCache::put<'life0, 'async_trait>(&'life0 self, vortex_layout::segments::SegmentId, vortex_buffer::ByteBuffer) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait pub struct vortex_layout::segments::NoOpSegmentCache impl vortex_layout::segments::SegmentCache for vortex_layout::segments::NoOpSegmentCache -pub fn vortex_layout::segments::NoOpSegmentCache::get<'life0, 'async_trait>(&'life0 self, _id: vortex_layout::segments::SegmentId) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait +pub fn vortex_layout::segments::NoOpSegmentCache::get<'life0, 'async_trait>(&'life0 self, vortex_layout::segments::SegmentId) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait -pub fn vortex_layout::segments::NoOpSegmentCache::put<'life0, 'async_trait>(&'life0 self, _id: vortex_layout::segments::SegmentId, _buffer: vortex_buffer::ByteBuffer) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait +pub fn vortex_layout::segments::NoOpSegmentCache::put<'life0, 'async_trait>(&'life0 self, vortex_layout::segments::SegmentId, vortex_buffer::ByteBuffer) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait pub struct vortex_layout::segments::SegmentCacheSourceAdapter impl vortex_layout::segments::SegmentCacheSourceAdapter -pub fn vortex_layout::segments::SegmentCacheSourceAdapter::new(cache: alloc::sync::Arc, source: alloc::sync::Arc) -> Self +pub fn vortex_layout::segments::SegmentCacheSourceAdapter::new(alloc::sync::Arc, alloc::sync::Arc) -> Self impl vortex_layout::segments::SegmentSource for vortex_layout::segments::SegmentCacheSourceAdapter -pub fn vortex_layout::segments::SegmentCacheSourceAdapter::request(&self, id: vortex_layout::segments::SegmentId) -> vortex_layout::segments::SegmentFuture +pub fn vortex_layout::segments::SegmentCacheSourceAdapter::request(&self, vortex_layout::segments::SegmentId) -> vortex_layout::segments::SegmentFuture pub struct vortex_layout::segments::SegmentId(_) @@ -1196,25 +1190,25 @@ impl core::cmp::Eq for vortex_layout::segments::SegmentId impl core::cmp::Ord for vortex_layout::segments::SegmentId -pub fn vortex_layout::segments::SegmentId::cmp(&self, other: &vortex_layout::segments::SegmentId) -> core::cmp::Ordering +pub fn vortex_layout::segments::SegmentId::cmp(&self, &vortex_layout::segments::SegmentId) -> core::cmp::Ordering impl core::cmp::PartialEq for vortex_layout::segments::SegmentId -pub fn vortex_layout::segments::SegmentId::eq(&self, other: &vortex_layout::segments::SegmentId) -> bool +pub fn vortex_layout::segments::SegmentId::eq(&self, &vortex_layout::segments::SegmentId) -> bool impl core::cmp::PartialOrd for vortex_layout::segments::SegmentId -pub fn vortex_layout::segments::SegmentId::partial_cmp(&self, other: &vortex_layout::segments::SegmentId) -> core::option::Option +pub fn vortex_layout::segments::SegmentId::partial_cmp(&self, &vortex_layout::segments::SegmentId) -> core::option::Option impl core::convert::From for vortex_layout::segments::SegmentId -pub fn vortex_layout::segments::SegmentId::from(value: u32) -> Self +pub fn vortex_layout::segments::SegmentId::from(u32) -> Self impl core::convert::TryFrom for vortex_layout::segments::SegmentId pub type vortex_layout::segments::SegmentId::Error = vortex_error::VortexError -pub fn vortex_layout::segments::SegmentId::try_from(value: usize) -> core::result::Result +pub fn vortex_layout::segments::SegmentId::try_from(usize) -> core::result::Result impl core::default::Default for vortex_layout::segments::SegmentId @@ -1222,15 +1216,15 @@ pub fn vortex_layout::segments::SegmentId::default() -> vortex_layout::segments: impl core::fmt::Debug for vortex_layout::segments::SegmentId -pub fn vortex_layout::segments::SegmentId::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_layout::segments::SegmentId::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_layout::segments::SegmentId -pub fn vortex_layout::segments::SegmentId::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_layout::segments::SegmentId::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_layout::segments::SegmentId -pub fn vortex_layout::segments::SegmentId::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_layout::segments::SegmentId::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_layout::segments::SegmentId @@ -1246,51 +1240,51 @@ pub struct vortex_layout::segments::SharedSegmentSource impl vortex_layout::segments::SharedSegmentSource -pub fn vortex_layout::segments::SharedSegmentSource::new(inner: S) -> Self +pub fn vortex_layout::segments::SharedSegmentSource::new(S) -> Self impl vortex_layout::segments::SegmentSource for vortex_layout::segments::SharedSegmentSource -pub fn vortex_layout::segments::SharedSegmentSource::request(&self, id: vortex_layout::segments::SegmentId) -> vortex_layout::segments::SegmentFuture +pub fn vortex_layout::segments::SharedSegmentSource::request(&self, vortex_layout::segments::SegmentId) -> vortex_layout::segments::SegmentFuture pub trait vortex_layout::segments::SegmentCache: core::marker::Send + core::marker::Sync -pub fn vortex_layout::segments::SegmentCache::get<'life0, 'async_trait>(&'life0 self, id: vortex_layout::segments::SegmentId) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait +pub fn vortex_layout::segments::SegmentCache::get<'life0, 'async_trait>(&'life0 self, vortex_layout::segments::SegmentId) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait -pub fn vortex_layout::segments::SegmentCache::put<'life0, 'async_trait>(&'life0 self, id: vortex_layout::segments::SegmentId, buffer: vortex_buffer::ByteBuffer) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait +pub fn vortex_layout::segments::SegmentCache::put<'life0, 'async_trait>(&'life0 self, vortex_layout::segments::SegmentId, vortex_buffer::ByteBuffer) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait impl vortex_layout::segments::SegmentCache for vortex_layout::segments::MokaSegmentCache -pub fn vortex_layout::segments::MokaSegmentCache::get<'life0, 'async_trait>(&'life0 self, id: vortex_layout::segments::SegmentId) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait +pub fn vortex_layout::segments::MokaSegmentCache::get<'life0, 'async_trait>(&'life0 self, vortex_layout::segments::SegmentId) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait -pub fn vortex_layout::segments::MokaSegmentCache::put<'life0, 'async_trait>(&'life0 self, id: vortex_layout::segments::SegmentId, buffer: vortex_buffer::ByteBuffer) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait +pub fn vortex_layout::segments::MokaSegmentCache::put<'life0, 'async_trait>(&'life0 self, vortex_layout::segments::SegmentId, vortex_buffer::ByteBuffer) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait impl vortex_layout::segments::SegmentCache for vortex_layout::segments::NoOpSegmentCache -pub fn vortex_layout::segments::NoOpSegmentCache::get<'life0, 'async_trait>(&'life0 self, _id: vortex_layout::segments::SegmentId) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait +pub fn vortex_layout::segments::NoOpSegmentCache::get<'life0, 'async_trait>(&'life0 self, vortex_layout::segments::SegmentId) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait -pub fn vortex_layout::segments::NoOpSegmentCache::put<'life0, 'async_trait>(&'life0 self, _id: vortex_layout::segments::SegmentId, _buffer: vortex_buffer::ByteBuffer) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait +pub fn vortex_layout::segments::NoOpSegmentCache::put<'life0, 'async_trait>(&'life0 self, vortex_layout::segments::SegmentId, vortex_buffer::ByteBuffer) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait impl vortex_layout::segments::SegmentCache for vortex_layout::segments::InstrumentedSegmentCache -pub fn vortex_layout::segments::InstrumentedSegmentCache::get<'life0, 'async_trait>(&'life0 self, id: vortex_layout::segments::SegmentId) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait +pub fn vortex_layout::segments::InstrumentedSegmentCache::get<'life0, 'async_trait>(&'life0 self, vortex_layout::segments::SegmentId) -> core::pin::Pin>> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait -pub fn vortex_layout::segments::InstrumentedSegmentCache::put<'life0, 'async_trait>(&'life0 self, id: vortex_layout::segments::SegmentId, buffer: vortex_buffer::ByteBuffer) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait +pub fn vortex_layout::segments::InstrumentedSegmentCache::put<'life0, 'async_trait>(&'life0 self, vortex_layout::segments::SegmentId, vortex_buffer::ByteBuffer) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait pub trait vortex_layout::segments::SegmentSink: core::marker::Send + core::marker::Sync -pub fn vortex_layout::segments::SegmentSink::write<'life0, 'async_trait>(&'life0 self, sequence_id: vortex_layout::sequence::SequenceId, buffers: alloc::vec::Vec) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait +pub fn vortex_layout::segments::SegmentSink::write<'life0, 'async_trait>(&'life0 self, vortex_layout::sequence::SequenceId, alloc::vec::Vec) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait pub trait vortex_layout::segments::SegmentSource: 'static + core::marker::Send + core::marker::Sync -pub fn vortex_layout::segments::SegmentSource::request(&self, id: vortex_layout::segments::SegmentId) -> vortex_layout::segments::SegmentFuture +pub fn vortex_layout::segments::SegmentSource::request(&self, vortex_layout::segments::SegmentId) -> vortex_layout::segments::SegmentFuture impl vortex_layout::segments::SegmentSource for vortex_layout::segments::SegmentCacheSourceAdapter -pub fn vortex_layout::segments::SegmentCacheSourceAdapter::request(&self, id: vortex_layout::segments::SegmentId) -> vortex_layout::segments::SegmentFuture +pub fn vortex_layout::segments::SegmentCacheSourceAdapter::request(&self, vortex_layout::segments::SegmentId) -> vortex_layout::segments::SegmentFuture impl vortex_layout::segments::SegmentSource for vortex_layout::segments::SharedSegmentSource -pub fn vortex_layout::segments::SharedSegmentSource::request(&self, id: vortex_layout::segments::SegmentId) -> vortex_layout::segments::SegmentFuture +pub fn vortex_layout::segments::SharedSegmentSource::request(&self, vortex_layout::segments::SegmentId) -> vortex_layout::segments::SegmentFuture pub type vortex_layout::segments::SegmentFuture = futures_core::future::BoxFuture<'static, vortex_error::VortexResult> @@ -1312,23 +1306,23 @@ impl core::cmp::Eq for vortex_layout::sequence::SequenceId impl core::cmp::Ord for vortex_layout::sequence::SequenceId -pub fn vortex_layout::sequence::SequenceId::cmp(&self, other: &Self) -> core::cmp::Ordering +pub fn vortex_layout::sequence::SequenceId::cmp(&self, &Self) -> core::cmp::Ordering impl core::cmp::PartialEq for vortex_layout::sequence::SequenceId -pub fn vortex_layout::sequence::SequenceId::eq(&self, other: &Self) -> bool +pub fn vortex_layout::sequence::SequenceId::eq(&self, &Self) -> bool impl core::cmp::PartialOrd for vortex_layout::sequence::SequenceId -pub fn vortex_layout::sequence::SequenceId::partial_cmp(&self, other: &Self) -> core::option::Option +pub fn vortex_layout::sequence::SequenceId::partial_cmp(&self, &Self) -> core::option::Option impl core::fmt::Debug for vortex_layout::sequence::SequenceId -pub fn vortex_layout::sequence::SequenceId::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_layout::sequence::SequenceId::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_layout::sequence::SequenceId -pub fn vortex_layout::sequence::SequenceId::hash(&self, state: &mut H) +pub fn vortex_layout::sequence::SequenceId::hash(&self, &mut H) impl core::ops::drop::Drop for vortex_layout::sequence::SequenceId @@ -1348,13 +1342,13 @@ pub fn vortex_layout::sequence::SequencePointer::split_off(&mut self) -> vortex_ impl core::fmt::Debug for vortex_layout::sequence::SequencePointer -pub fn vortex_layout::sequence::SequencePointer::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_layout::sequence::SequencePointer::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_layout::sequence::SequentialStreamAdapter impl vortex_layout::sequence::SequentialStreamAdapter -pub fn vortex_layout::sequence::SequentialStreamAdapter::new(dtype: vortex_array::dtype::DType, inner: S) -> Self +pub fn vortex_layout::sequence::SequentialStreamAdapter::new(vortex_array::dtype::DType, S) -> Self impl<'__pin, S> core::marker::Unpin for vortex_layout::sequence::SequentialStreamAdapter where pin_project_lite::__private::PinnedFieldsOf<__Origin<'__pin, S>>: core::marker::Unpin @@ -1362,7 +1356,7 @@ impl futures_core::stream::Stream for vortex_layout::sequence::SequentialStre pub type vortex_layout::sequence::SequentialStreamAdapter::Item = core::result::Result<(vortex_layout::sequence::SequenceId, vortex_array::array::erased::ArrayRef), vortex_error::VortexError> -pub fn vortex_layout::sequence::SequentialStreamAdapter::poll_next(self: core::pin::Pin<&mut Self>, cx: &mut core::task::wake::Context<'_>) -> core::task::poll::Poll> +pub fn vortex_layout::sequence::SequentialStreamAdapter::poll_next(core::pin::Pin<&mut Self>, &mut core::task::wake::Context<'_>) -> core::task::poll::Poll> pub fn vortex_layout::sequence::SequentialStreamAdapter::size_hint(&self) -> (usize, core::option::Option) @@ -1372,11 +1366,11 @@ pub fn vortex_layout::sequence::SequentialStreamAdapter::dtype(&self) -> &vor pub trait vortex_layout::sequence::SequentialArrayStreamExt: vortex_array::stream::ArrayStream -pub fn vortex_layout::sequence::SequentialArrayStreamExt::sequenced(self, pointer: vortex_layout::sequence::SequencePointer) -> vortex_layout::sequence::SendableSequentialStream where Self: core::marker::Sized + core::marker::Send + 'static +pub fn vortex_layout::sequence::SequentialArrayStreamExt::sequenced(self, vortex_layout::sequence::SequencePointer) -> vortex_layout::sequence::SendableSequentialStream where Self: core::marker::Sized + core::marker::Send + 'static impl vortex_layout::sequence::SequentialArrayStreamExt for S -pub fn S::sequenced(self, pointer: vortex_layout::sequence::SequencePointer) -> vortex_layout::sequence::SendableSequentialStream where Self: core::marker::Sized + core::marker::Send + 'static +pub fn S::sequenced(self, vortex_layout::sequence::SequencePointer) -> vortex_layout::sequence::SendableSequentialStream where Self: core::marker::Sized + core::marker::Send + 'static pub trait vortex_layout::sequence::SequentialStream: futures_core::stream::Stream> @@ -1406,9 +1400,9 @@ pub struct vortex_layout::session::LayoutSession impl vortex_layout::session::LayoutSession -pub fn vortex_layout::session::LayoutSession::register(&self, layout: vortex_layout::LayoutEncodingRef) +pub fn vortex_layout::session::LayoutSession::register(&self, vortex_layout::LayoutEncodingRef) -pub fn vortex_layout::session::LayoutSession::register_many(&self, layouts: impl core::iter::traits::collect::IntoIterator) +pub fn vortex_layout::session::LayoutSession::register_many(&self, impl core::iter::traits::collect::IntoIterator) pub fn vortex_layout::session::LayoutSession::registry(&self) -> &vortex_layout::session::LayoutRegistry @@ -1418,7 +1412,13 @@ pub fn vortex_layout::session::LayoutSession::default() -> Self impl core::fmt::Debug for vortex_layout::session::LayoutSession -pub fn vortex_layout::session::LayoutSession::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_layout::session::LayoutSession::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +impl vortex_session::SessionVar for vortex_layout::session::LayoutSession + +pub fn vortex_layout::session::LayoutSession::as_any(&self) -> &dyn core::any::Any + +pub fn vortex_layout::session::LayoutSession::as_any_mut(&mut self) -> &mut dyn core::any::Any pub trait vortex_layout::session::LayoutSessionExt: vortex_session::SessionExt @@ -1440,29 +1440,29 @@ pub type vortex_layout::vtable::VTable::Layout: 'static + core::marker::Send + c pub type vortex_layout::vtable::VTable::Metadata: vortex_array::metadata::SerializeMetadata + vortex_array::metadata::DeserializeMetadata + core::fmt::Debug -pub fn vortex_layout::vtable::VTable::build(encoding: &Self::Encoding, dtype: &vortex_array::dtype::DType, row_count: u64, metadata: &::Output, segment_ids: alloc::vec::Vec, children: &dyn vortex_layout::LayoutChildren, ctx: &vortex_session::registry::ReadContext) -> vortex_error::VortexResult +pub fn vortex_layout::vtable::VTable::build(&Self::Encoding, &vortex_array::dtype::DType, u64, &::Output, alloc::vec::Vec, &dyn vortex_layout::LayoutChildren, &vortex_session::registry::ReadContext) -> vortex_error::VortexResult -pub fn vortex_layout::vtable::VTable::child(layout: &Self::Layout, idx: usize) -> vortex_error::VortexResult +pub fn vortex_layout::vtable::VTable::child(&Self::Layout, usize) -> vortex_error::VortexResult -pub fn vortex_layout::vtable::VTable::child_type(layout: &Self::Layout, idx: usize) -> vortex_layout::LayoutChildType +pub fn vortex_layout::vtable::VTable::child_type(&Self::Layout, usize) -> vortex_layout::LayoutChildType -pub fn vortex_layout::vtable::VTable::dtype(layout: &Self::Layout) -> &vortex_array::dtype::DType +pub fn vortex_layout::vtable::VTable::dtype(&Self::Layout) -> &vortex_array::dtype::DType -pub fn vortex_layout::vtable::VTable::encoding(layout: &Self::Layout) -> vortex_layout::LayoutEncodingRef +pub fn vortex_layout::vtable::VTable::encoding(&Self::Layout) -> vortex_layout::LayoutEncodingRef -pub fn vortex_layout::vtable::VTable::id(encoding: &Self::Encoding) -> vortex_layout::LayoutId +pub fn vortex_layout::vtable::VTable::id(&Self::Encoding) -> vortex_layout::LayoutId -pub fn vortex_layout::vtable::VTable::metadata(layout: &Self::Layout) -> Self::Metadata +pub fn vortex_layout::vtable::VTable::metadata(&Self::Layout) -> Self::Metadata -pub fn vortex_layout::vtable::VTable::nchildren(layout: &Self::Layout) -> usize +pub fn vortex_layout::vtable::VTable::nchildren(&Self::Layout) -> usize -pub fn vortex_layout::vtable::VTable::new_reader(layout: &Self::Layout, name: alloc::sync::Arc, segment_source: alloc::sync::Arc, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_layout::vtable::VTable::new_reader(&Self::Layout, alloc::sync::Arc, alloc::sync::Arc, &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_layout::vtable::VTable::row_count(layout: &Self::Layout) -> u64 +pub fn vortex_layout::vtable::VTable::row_count(&Self::Layout) -> u64 -pub fn vortex_layout::vtable::VTable::segment_ids(layout: &Self::Layout) -> alloc::vec::Vec +pub fn vortex_layout::vtable::VTable::segment_ids(&Self::Layout) -> alloc::vec::Vec -pub fn vortex_layout::vtable::VTable::with_children(_layout: &mut Self::Layout, _children: alloc::vec::Vec) -> vortex_error::VortexResult<()> +pub fn vortex_layout::vtable::VTable::with_children(&mut Self::Layout, alloc::vec::Vec) -> vortex_error::VortexResult<()> impl vortex_layout::VTable for vortex_layout::layouts::chunked::Chunked @@ -1472,29 +1472,29 @@ pub type vortex_layout::layouts::chunked::Chunked::Layout = vortex_layout::layou pub type vortex_layout::layouts::chunked::Chunked::Metadata = vortex_array::metadata::EmptyMetadata -pub fn vortex_layout::layouts::chunked::Chunked::build(_encoding: &Self::Encoding, dtype: &vortex_array::dtype::DType, row_count: u64, _metadata: &::Output, _segment_ids: alloc::vec::Vec, children: &dyn vortex_layout::LayoutChildren, _ctx: &vortex_session::registry::ReadContext) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::chunked::Chunked::build(&Self::Encoding, &vortex_array::dtype::DType, u64, &::Output, alloc::vec::Vec, &dyn vortex_layout::LayoutChildren, &vortex_session::registry::ReadContext) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::chunked::Chunked::child(layout: &Self::Layout, idx: usize) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::chunked::Chunked::child(&Self::Layout, usize) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::chunked::Chunked::child_type(layout: &Self::Layout, idx: usize) -> vortex_layout::LayoutChildType +pub fn vortex_layout::layouts::chunked::Chunked::child_type(&Self::Layout, usize) -> vortex_layout::LayoutChildType -pub fn vortex_layout::layouts::chunked::Chunked::dtype(layout: &Self::Layout) -> &vortex_array::dtype::DType +pub fn vortex_layout::layouts::chunked::Chunked::dtype(&Self::Layout) -> &vortex_array::dtype::DType -pub fn vortex_layout::layouts::chunked::Chunked::encoding(_layout: &Self::Layout) -> vortex_layout::LayoutEncodingRef +pub fn vortex_layout::layouts::chunked::Chunked::encoding(&Self::Layout) -> vortex_layout::LayoutEncodingRef -pub fn vortex_layout::layouts::chunked::Chunked::id(_encoding: &Self::Encoding) -> vortex_layout::LayoutId +pub fn vortex_layout::layouts::chunked::Chunked::id(&Self::Encoding) -> vortex_layout::LayoutId -pub fn vortex_layout::layouts::chunked::Chunked::metadata(_layout: &Self::Layout) -> Self::Metadata +pub fn vortex_layout::layouts::chunked::Chunked::metadata(&Self::Layout) -> Self::Metadata -pub fn vortex_layout::layouts::chunked::Chunked::nchildren(layout: &Self::Layout) -> usize +pub fn vortex_layout::layouts::chunked::Chunked::nchildren(&Self::Layout) -> usize -pub fn vortex_layout::layouts::chunked::Chunked::new_reader(layout: &Self::Layout, name: alloc::sync::Arc, segment_source: alloc::sync::Arc, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::chunked::Chunked::new_reader(&Self::Layout, alloc::sync::Arc, alloc::sync::Arc, &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::chunked::Chunked::row_count(layout: &Self::Layout) -> u64 +pub fn vortex_layout::layouts::chunked::Chunked::row_count(&Self::Layout) -> u64 -pub fn vortex_layout::layouts::chunked::Chunked::segment_ids(_layout: &Self::Layout) -> alloc::vec::Vec +pub fn vortex_layout::layouts::chunked::Chunked::segment_ids(&Self::Layout) -> alloc::vec::Vec -pub fn vortex_layout::layouts::chunked::Chunked::with_children(layout: &mut Self::Layout, children: alloc::vec::Vec) -> vortex_error::VortexResult<()> +pub fn vortex_layout::layouts::chunked::Chunked::with_children(&mut Self::Layout, alloc::vec::Vec) -> vortex_error::VortexResult<()> impl vortex_layout::VTable for vortex_layout::layouts::dict::Dict @@ -1504,29 +1504,29 @@ pub type vortex_layout::layouts::dict::Dict::Layout = vortex_layout::layouts::di pub type vortex_layout::layouts::dict::Dict::Metadata = vortex_array::metadata::ProstMetadata -pub fn vortex_layout::layouts::dict::Dict::build(_encoding: &Self::Encoding, dtype: &vortex_array::dtype::DType, _row_count: u64, metadata: &::Output, _segment_ids: alloc::vec::Vec, children: &dyn vortex_layout::LayoutChildren, _ctx: &vortex_session::registry::ReadContext) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::dict::Dict::build(&Self::Encoding, &vortex_array::dtype::DType, u64, &::Output, alloc::vec::Vec, &dyn vortex_layout::LayoutChildren, &vortex_session::registry::ReadContext) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::dict::Dict::child(layout: &Self::Layout, idx: usize) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::dict::Dict::child(&Self::Layout, usize) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::dict::Dict::child_type(_layout: &Self::Layout, idx: usize) -> vortex_layout::LayoutChildType +pub fn vortex_layout::layouts::dict::Dict::child_type(&Self::Layout, usize) -> vortex_layout::LayoutChildType -pub fn vortex_layout::layouts::dict::Dict::dtype(layout: &Self::Layout) -> &vortex_array::dtype::DType +pub fn vortex_layout::layouts::dict::Dict::dtype(&Self::Layout) -> &vortex_array::dtype::DType -pub fn vortex_layout::layouts::dict::Dict::encoding(_layout: &Self::Layout) -> vortex_layout::LayoutEncodingRef +pub fn vortex_layout::layouts::dict::Dict::encoding(&Self::Layout) -> vortex_layout::LayoutEncodingRef -pub fn vortex_layout::layouts::dict::Dict::id(_encoding: &Self::Encoding) -> vortex_layout::LayoutId +pub fn vortex_layout::layouts::dict::Dict::id(&Self::Encoding) -> vortex_layout::LayoutId -pub fn vortex_layout::layouts::dict::Dict::metadata(layout: &Self::Layout) -> Self::Metadata +pub fn vortex_layout::layouts::dict::Dict::metadata(&Self::Layout) -> Self::Metadata -pub fn vortex_layout::layouts::dict::Dict::nchildren(_layout: &Self::Layout) -> usize +pub fn vortex_layout::layouts::dict::Dict::nchildren(&Self::Layout) -> usize -pub fn vortex_layout::layouts::dict::Dict::new_reader(layout: &Self::Layout, name: alloc::sync::Arc, segment_source: alloc::sync::Arc, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::dict::Dict::new_reader(&Self::Layout, alloc::sync::Arc, alloc::sync::Arc, &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::dict::Dict::row_count(layout: &Self::Layout) -> u64 +pub fn vortex_layout::layouts::dict::Dict::row_count(&Self::Layout) -> u64 -pub fn vortex_layout::layouts::dict::Dict::segment_ids(_layout: &Self::Layout) -> alloc::vec::Vec +pub fn vortex_layout::layouts::dict::Dict::segment_ids(&Self::Layout) -> alloc::vec::Vec -pub fn vortex_layout::layouts::dict::Dict::with_children(layout: &mut Self::Layout, children: alloc::vec::Vec) -> vortex_error::VortexResult<()> +pub fn vortex_layout::layouts::dict::Dict::with_children(&mut Self::Layout, alloc::vec::Vec) -> vortex_error::VortexResult<()> impl vortex_layout::VTable for vortex_layout::layouts::flat::Flat @@ -1536,29 +1536,29 @@ pub type vortex_layout::layouts::flat::Flat::Layout = vortex_layout::layouts::fl pub type vortex_layout::layouts::flat::Flat::Metadata = vortex_array::metadata::ProstMetadata -pub fn vortex_layout::layouts::flat::Flat::build(_encoding: &Self::Encoding, dtype: &vortex_array::dtype::DType, row_count: u64, metadata: &::Output, segment_ids: alloc::vec::Vec, _children: &dyn vortex_layout::LayoutChildren, ctx: &vortex_session::registry::ReadContext) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::flat::Flat::build(&Self::Encoding, &vortex_array::dtype::DType, u64, &::Output, alloc::vec::Vec, &dyn vortex_layout::LayoutChildren, &vortex_session::registry::ReadContext) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::flat::Flat::child(_layout: &Self::Layout, _idx: usize) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::flat::Flat::child(&Self::Layout, usize) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::flat::Flat::child_type(_layout: &Self::Layout, _idx: usize) -> vortex_layout::LayoutChildType +pub fn vortex_layout::layouts::flat::Flat::child_type(&Self::Layout, usize) -> vortex_layout::LayoutChildType -pub fn vortex_layout::layouts::flat::Flat::dtype(layout: &Self::Layout) -> &vortex_array::dtype::DType +pub fn vortex_layout::layouts::flat::Flat::dtype(&Self::Layout) -> &vortex_array::dtype::DType -pub fn vortex_layout::layouts::flat::Flat::encoding(_layout: &Self::Layout) -> vortex_layout::LayoutEncodingRef +pub fn vortex_layout::layouts::flat::Flat::encoding(&Self::Layout) -> vortex_layout::LayoutEncodingRef -pub fn vortex_layout::layouts::flat::Flat::id(_encoding: &Self::Encoding) -> vortex_layout::LayoutId +pub fn vortex_layout::layouts::flat::Flat::id(&Self::Encoding) -> vortex_layout::LayoutId -pub fn vortex_layout::layouts::flat::Flat::metadata(layout: &Self::Layout) -> Self::Metadata +pub fn vortex_layout::layouts::flat::Flat::metadata(&Self::Layout) -> Self::Metadata -pub fn vortex_layout::layouts::flat::Flat::nchildren(_layout: &Self::Layout) -> usize +pub fn vortex_layout::layouts::flat::Flat::nchildren(&Self::Layout) -> usize -pub fn vortex_layout::layouts::flat::Flat::new_reader(layout: &Self::Layout, name: alloc::sync::Arc, segment_source: alloc::sync::Arc, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::flat::Flat::new_reader(&Self::Layout, alloc::sync::Arc, alloc::sync::Arc, &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::flat::Flat::row_count(layout: &Self::Layout) -> u64 +pub fn vortex_layout::layouts::flat::Flat::row_count(&Self::Layout) -> u64 -pub fn vortex_layout::layouts::flat::Flat::segment_ids(layout: &Self::Layout) -> alloc::vec::Vec +pub fn vortex_layout::layouts::flat::Flat::segment_ids(&Self::Layout) -> alloc::vec::Vec -pub fn vortex_layout::layouts::flat::Flat::with_children(_layout: &mut Self::Layout, children: alloc::vec::Vec) -> vortex_error::VortexResult<()> +pub fn vortex_layout::layouts::flat::Flat::with_children(&mut Self::Layout, alloc::vec::Vec) -> vortex_error::VortexResult<()> impl vortex_layout::VTable for vortex_layout::layouts::struct_::Struct @@ -1568,29 +1568,29 @@ pub type vortex_layout::layouts::struct_::Struct::Layout = vortex_layout::layout pub type vortex_layout::layouts::struct_::Struct::Metadata = vortex_array::metadata::EmptyMetadata -pub fn vortex_layout::layouts::struct_::Struct::build(_encoding: &Self::Encoding, dtype: &vortex_array::dtype::DType, row_count: u64, _metadata: &::Output, _segment_ids: alloc::vec::Vec, children: &dyn vortex_layout::LayoutChildren, _ctx: &vortex_session::registry::ReadContext) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::struct_::Struct::build(&Self::Encoding, &vortex_array::dtype::DType, u64, &::Output, alloc::vec::Vec, &dyn vortex_layout::LayoutChildren, &vortex_session::registry::ReadContext) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::struct_::Struct::child(layout: &Self::Layout, index: usize) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::struct_::Struct::child(&Self::Layout, usize) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::struct_::Struct::child_type(layout: &Self::Layout, idx: usize) -> vortex_layout::LayoutChildType +pub fn vortex_layout::layouts::struct_::Struct::child_type(&Self::Layout, usize) -> vortex_layout::LayoutChildType -pub fn vortex_layout::layouts::struct_::Struct::dtype(layout: &Self::Layout) -> &vortex_array::dtype::DType +pub fn vortex_layout::layouts::struct_::Struct::dtype(&Self::Layout) -> &vortex_array::dtype::DType -pub fn vortex_layout::layouts::struct_::Struct::encoding(_layout: &Self::Layout) -> vortex_layout::LayoutEncodingRef +pub fn vortex_layout::layouts::struct_::Struct::encoding(&Self::Layout) -> vortex_layout::LayoutEncodingRef -pub fn vortex_layout::layouts::struct_::Struct::id(_encoding: &Self::Encoding) -> vortex_layout::LayoutId +pub fn vortex_layout::layouts::struct_::Struct::id(&Self::Encoding) -> vortex_layout::LayoutId -pub fn vortex_layout::layouts::struct_::Struct::metadata(_layout: &Self::Layout) -> Self::Metadata +pub fn vortex_layout::layouts::struct_::Struct::metadata(&Self::Layout) -> Self::Metadata -pub fn vortex_layout::layouts::struct_::Struct::nchildren(layout: &Self::Layout) -> usize +pub fn vortex_layout::layouts::struct_::Struct::nchildren(&Self::Layout) -> usize -pub fn vortex_layout::layouts::struct_::Struct::new_reader(layout: &Self::Layout, name: alloc::sync::Arc, segment_source: alloc::sync::Arc, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::struct_::Struct::new_reader(&Self::Layout, alloc::sync::Arc, alloc::sync::Arc, &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::struct_::Struct::row_count(layout: &Self::Layout) -> u64 +pub fn vortex_layout::layouts::struct_::Struct::row_count(&Self::Layout) -> u64 -pub fn vortex_layout::layouts::struct_::Struct::segment_ids(_layout: &Self::Layout) -> alloc::vec::Vec +pub fn vortex_layout::layouts::struct_::Struct::segment_ids(&Self::Layout) -> alloc::vec::Vec -pub fn vortex_layout::layouts::struct_::Struct::with_children(layout: &mut Self::Layout, children: alloc::vec::Vec) -> vortex_error::VortexResult<()> +pub fn vortex_layout::layouts::struct_::Struct::with_children(&mut Self::Layout, alloc::vec::Vec) -> vortex_error::VortexResult<()> impl vortex_layout::VTable for vortex_layout::layouts::zoned::Zoned @@ -1600,29 +1600,29 @@ pub type vortex_layout::layouts::zoned::Zoned::Layout = vortex_layout::layouts:: pub type vortex_layout::layouts::zoned::Zoned::Metadata = vortex_layout::layouts::zoned::ZonedMetadata -pub fn vortex_layout::layouts::zoned::Zoned::build(_encoding: &Self::Encoding, dtype: &vortex_array::dtype::DType, _row_count: u64, metadata: &vortex_layout::layouts::zoned::ZonedMetadata, _segment_ids: alloc::vec::Vec, children: &dyn vortex_layout::LayoutChildren, _ctx: &vortex_session::registry::ReadContext) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::zoned::Zoned::build(&Self::Encoding, &vortex_array::dtype::DType, u64, &vortex_layout::layouts::zoned::ZonedMetadata, alloc::vec::Vec, &dyn vortex_layout::LayoutChildren, &vortex_session::registry::ReadContext) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::zoned::Zoned::child(layout: &Self::Layout, idx: usize) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::zoned::Zoned::child(&Self::Layout, usize) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::zoned::Zoned::child_type(_layout: &Self::Layout, idx: usize) -> vortex_layout::LayoutChildType +pub fn vortex_layout::layouts::zoned::Zoned::child_type(&Self::Layout, usize) -> vortex_layout::LayoutChildType -pub fn vortex_layout::layouts::zoned::Zoned::dtype(layout: &Self::Layout) -> &vortex_array::dtype::DType +pub fn vortex_layout::layouts::zoned::Zoned::dtype(&Self::Layout) -> &vortex_array::dtype::DType -pub fn vortex_layout::layouts::zoned::Zoned::encoding(_layout: &Self::Layout) -> vortex_layout::LayoutEncodingRef +pub fn vortex_layout::layouts::zoned::Zoned::encoding(&Self::Layout) -> vortex_layout::LayoutEncodingRef -pub fn vortex_layout::layouts::zoned::Zoned::id(_encoding: &Self::Encoding) -> vortex_layout::LayoutId +pub fn vortex_layout::layouts::zoned::Zoned::id(&Self::Encoding) -> vortex_layout::LayoutId -pub fn vortex_layout::layouts::zoned::Zoned::metadata(layout: &Self::Layout) -> Self::Metadata +pub fn vortex_layout::layouts::zoned::Zoned::metadata(&Self::Layout) -> Self::Metadata -pub fn vortex_layout::layouts::zoned::Zoned::nchildren(_layout: &Self::Layout) -> usize +pub fn vortex_layout::layouts::zoned::Zoned::nchildren(&Self::Layout) -> usize -pub fn vortex_layout::layouts::zoned::Zoned::new_reader(layout: &Self::Layout, name: alloc::sync::Arc, segment_source: alloc::sync::Arc, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::zoned::Zoned::new_reader(&Self::Layout, alloc::sync::Arc, alloc::sync::Arc, &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::zoned::Zoned::row_count(layout: &Self::Layout) -> u64 +pub fn vortex_layout::layouts::zoned::Zoned::row_count(&Self::Layout) -> u64 -pub fn vortex_layout::layouts::zoned::Zoned::segment_ids(_layout: &Self::Layout) -> alloc::vec::Vec +pub fn vortex_layout::layouts::zoned::Zoned::segment_ids(&Self::Layout) -> alloc::vec::Vec -pub fn vortex_layout::layouts::zoned::Zoned::with_children(layout: &mut Self::Layout, children: alloc::vec::Vec) -> vortex_error::VortexResult<()> +pub fn vortex_layout::layouts::zoned::Zoned::with_children(&mut Self::Layout, alloc::vec::Vec) -> vortex_error::VortexResult<()> pub macro vortex_layout::vtable! @@ -1650,11 +1650,11 @@ impl core::cmp::Eq for vortex_layout::LayoutChildType impl core::cmp::PartialEq for vortex_layout::LayoutChildType -pub fn vortex_layout::LayoutChildType::eq(&self, other: &vortex_layout::LayoutChildType) -> bool +pub fn vortex_layout::LayoutChildType::eq(&self, &vortex_layout::LayoutChildType) -> bool impl core::fmt::Debug for vortex_layout::LayoutChildType -pub fn vortex_layout::LayoutChildType::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_layout::LayoutChildType::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::StructuralPartialEq for vortex_layout::LayoutChildType @@ -1662,17 +1662,17 @@ impl core::marker::StructuralPartialEq for vortex_layout::LayoutChildType impl core::fmt::Debug for vortex_layout::LayoutAdapter -pub fn vortex_layout::LayoutAdapter::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_layout::LayoutAdapter::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_layout::Layout for vortex_layout::LayoutAdapter pub fn vortex_layout::LayoutAdapter::as_any(&self) -> &dyn core::any::Any -pub fn vortex_layout::LayoutAdapter::as_any_arc(self: alloc::sync::Arc) -> alloc::sync::Arc<(dyn core::any::Any + core::marker::Send + core::marker::Sync)> +pub fn vortex_layout::LayoutAdapter::as_any_arc(alloc::sync::Arc) -> alloc::sync::Arc<(dyn core::any::Any + core::marker::Send + core::marker::Sync)> -pub fn vortex_layout::LayoutAdapter::child(&self, idx: usize) -> vortex_error::VortexResult +pub fn vortex_layout::LayoutAdapter::child(&self, usize) -> vortex_error::VortexResult -pub fn vortex_layout::LayoutAdapter::child_type(&self, idx: usize) -> vortex_layout::LayoutChildType +pub fn vortex_layout::LayoutAdapter::child_type(&self, usize) -> vortex_layout::LayoutChildType pub fn vortex_layout::LayoutAdapter::dtype(&self) -> &vortex_array::dtype::DType @@ -1682,7 +1682,7 @@ pub fn vortex_layout::LayoutAdapter::metadata(&self) -> alloc::vec::Vec pub fn vortex_layout::LayoutAdapter::nchildren(&self) -> usize -pub fn vortex_layout::LayoutAdapter::new_reader(&self, name: alloc::sync::Arc, segment_source: alloc::sync::Arc, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_layout::LayoutAdapter::new_reader(&self, alloc::sync::Arc, alloc::sync::Arc, &vortex_session::VortexSession) -> vortex_error::VortexResult pub fn vortex_layout::LayoutAdapter::row_count(&self) -> u64 @@ -1694,13 +1694,13 @@ pub fn vortex_layout::LayoutAdapter::to_layout(&self) -> vortex_layout::Layou impl core::fmt::Debug for vortex_layout::LayoutEncodingAdapter -pub fn vortex_layout::LayoutEncodingAdapter::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_layout::LayoutEncodingAdapter::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_layout::LayoutEncoding for vortex_layout::LayoutEncodingAdapter pub fn vortex_layout::LayoutEncodingAdapter::as_any(&self) -> &dyn core::any::Any -pub fn vortex_layout::LayoutEncodingAdapter::build(&self, dtype: &vortex_array::dtype::DType, row_count: u64, metadata: &[u8], segment_ids: alloc::vec::Vec, children: &dyn vortex_layout::LayoutChildren, ctx: &vortex_session::registry::ReadContext) -> vortex_error::VortexResult +pub fn vortex_layout::LayoutEncodingAdapter::build(&self, &vortex_array::dtype::DType, u64, &[u8], alloc::vec::Vec, &dyn vortex_layout::LayoutChildren, &vortex_session::registry::ReadContext) -> vortex_error::VortexResult pub fn vortex_layout::LayoutEncodingAdapter::id(&self) -> vortex_layout::LayoutEncodingId @@ -1708,17 +1708,17 @@ pub struct vortex_layout::LazyReaderChildren impl vortex_layout::LazyReaderChildren -pub fn vortex_layout::LazyReaderChildren::get(&self, idx: usize) -> vortex_error::VortexResult<&vortex_layout::LayoutReaderRef> +pub fn vortex_layout::LazyReaderChildren::get(&self, usize) -> vortex_error::VortexResult<&vortex_layout::LayoutReaderRef> -pub fn vortex_layout::LazyReaderChildren::new(children: alloc::sync::Arc, dtypes: alloc::vec::Vec, names: alloc::vec::Vec>, segment_source: alloc::sync::Arc, session: vortex_session::VortexSession) -> Self +pub fn vortex_layout::LazyReaderChildren::new(alloc::sync::Arc, alloc::vec::Vec, alloc::vec::Vec>, alloc::sync::Arc, vortex_session::VortexSession) -> Self pub trait vortex_layout::ArrayFutureExt -pub fn vortex_layout::ArrayFutureExt::masked(self, mask: vortex_array::mask_future::MaskFuture) -> Self +pub fn vortex_layout::ArrayFutureExt::masked(self, vortex_array::mask_future::MaskFuture) -> Self impl vortex_layout::ArrayFutureExt for vortex_layout::ArrayFuture -pub fn vortex_layout::ArrayFuture::masked(self, mask: vortex_array::mask_future::MaskFuture) -> Self +pub fn vortex_layout::ArrayFuture::masked(self, vortex_array::mask_future::MaskFuture) -> Self pub trait vortex_layout::IntoLayout @@ -1748,11 +1748,11 @@ pub trait vortex_layout::Layout: 'static + core::marker::Send + core::marker::Sy pub fn vortex_layout::Layout::as_any(&self) -> &dyn core::any::Any -pub fn vortex_layout::Layout::as_any_arc(self: alloc::sync::Arc) -> alloc::sync::Arc<(dyn core::any::Any + core::marker::Send + core::marker::Sync)> +pub fn vortex_layout::Layout::as_any_arc(alloc::sync::Arc) -> alloc::sync::Arc<(dyn core::any::Any + core::marker::Send + core::marker::Sync)> -pub fn vortex_layout::Layout::child(&self, idx: usize) -> vortex_error::VortexResult +pub fn vortex_layout::Layout::child(&self, usize) -> vortex_error::VortexResult -pub fn vortex_layout::Layout::child_type(&self, idx: usize) -> vortex_layout::LayoutChildType +pub fn vortex_layout::Layout::child_type(&self, usize) -> vortex_layout::LayoutChildType pub fn vortex_layout::Layout::dtype(&self) -> &vortex_array::dtype::DType @@ -1762,7 +1762,7 @@ pub fn vortex_layout::Layout::metadata(&self) -> alloc::vec::Vec pub fn vortex_layout::Layout::nchildren(&self) -> usize -pub fn vortex_layout::Layout::new_reader(&self, name: alloc::sync::Arc, segment_source: alloc::sync::Arc, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_layout::Layout::new_reader(&self, alloc::sync::Arc, alloc::sync::Arc, &vortex_session::VortexSession) -> vortex_error::VortexResult pub fn vortex_layout::Layout::row_count(&self) -> u64 @@ -1774,11 +1774,11 @@ impl vortex_layout::Layout for vortex_layout::LayoutAd pub fn vortex_layout::LayoutAdapter::as_any(&self) -> &dyn core::any::Any -pub fn vortex_layout::LayoutAdapter::as_any_arc(self: alloc::sync::Arc) -> alloc::sync::Arc<(dyn core::any::Any + core::marker::Send + core::marker::Sync)> +pub fn vortex_layout::LayoutAdapter::as_any_arc(alloc::sync::Arc) -> alloc::sync::Arc<(dyn core::any::Any + core::marker::Send + core::marker::Sync)> -pub fn vortex_layout::LayoutAdapter::child(&self, idx: usize) -> vortex_error::VortexResult +pub fn vortex_layout::LayoutAdapter::child(&self, usize) -> vortex_error::VortexResult -pub fn vortex_layout::LayoutAdapter::child_type(&self, idx: usize) -> vortex_layout::LayoutChildType +pub fn vortex_layout::LayoutAdapter::child_type(&self, usize) -> vortex_layout::LayoutChildType pub fn vortex_layout::LayoutAdapter::dtype(&self) -> &vortex_array::dtype::DType @@ -1788,7 +1788,7 @@ pub fn vortex_layout::LayoutAdapter::metadata(&self) -> alloc::vec::Vec pub fn vortex_layout::LayoutAdapter::nchildren(&self) -> usize -pub fn vortex_layout::LayoutAdapter::new_reader(&self, name: alloc::sync::Arc, segment_source: alloc::sync::Arc, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_layout::LayoutAdapter::new_reader(&self, alloc::sync::Arc, alloc::sync::Arc, &vortex_session::VortexSession) -> vortex_error::VortexResult pub fn vortex_layout::LayoutAdapter::row_count(&self) -> u64 @@ -1798,9 +1798,9 @@ pub fn vortex_layout::LayoutAdapter::to_layout(&self) -> vortex_layout::Layou pub trait vortex_layout::LayoutChildren: 'static + core::marker::Send + core::marker::Sync -pub fn vortex_layout::LayoutChildren::child(&self, idx: usize, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_layout::LayoutChildren::child(&self, usize, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_layout::LayoutChildren::child_row_count(&self, idx: usize) -> u64 +pub fn vortex_layout::LayoutChildren::child_row_count(&self, usize) -> u64 pub fn vortex_layout::LayoutChildren::nchildren(&self) -> usize @@ -1808,9 +1808,9 @@ pub fn vortex_layout::LayoutChildren::to_arc(&self) -> alloc::sync::Arc -pub fn alloc::sync::Arc::child(&self, idx: usize, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn alloc::sync::Arc::child(&self, usize, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn alloc::sync::Arc::child_row_count(&self, idx: usize) -> u64 +pub fn alloc::sync::Arc::child_row_count(&self, usize) -> u64 pub fn alloc::sync::Arc::nchildren(&self) -> usize @@ -1820,7 +1820,7 @@ pub trait vortex_layout::LayoutEncoding: 'static + core::marker::Send + core::ma pub fn vortex_layout::LayoutEncoding::as_any(&self) -> &dyn core::any::Any -pub fn vortex_layout::LayoutEncoding::build(&self, dtype: &vortex_array::dtype::DType, row_count: u64, metadata: &[u8], segment_ids: alloc::vec::Vec, children: &dyn vortex_layout::LayoutChildren, ctx: &vortex_session::registry::ReadContext) -> vortex_error::VortexResult +pub fn vortex_layout::LayoutEncoding::build(&self, &vortex_array::dtype::DType, u64, &[u8], alloc::vec::Vec, &dyn vortex_layout::LayoutChildren, &vortex_session::registry::ReadContext) -> vortex_error::VortexResult pub fn vortex_layout::LayoutEncoding::id(&self) -> vortex_layout::LayoutEncodingId @@ -1828,39 +1828,43 @@ impl vortex_layout::LayoutEncoding for vortex_layout:: pub fn vortex_layout::LayoutEncodingAdapter::as_any(&self) -> &dyn core::any::Any -pub fn vortex_layout::LayoutEncodingAdapter::build(&self, dtype: &vortex_array::dtype::DType, row_count: u64, metadata: &[u8], segment_ids: alloc::vec::Vec, children: &dyn vortex_layout::LayoutChildren, ctx: &vortex_session::registry::ReadContext) -> vortex_error::VortexResult +pub fn vortex_layout::LayoutEncodingAdapter::build(&self, &vortex_array::dtype::DType, u64, &[u8], alloc::vec::Vec, &dyn vortex_layout::LayoutChildren, &vortex_session::registry::ReadContext) -> vortex_error::VortexResult pub fn vortex_layout::LayoutEncodingAdapter::id(&self) -> vortex_layout::LayoutEncodingId pub trait vortex_layout::LayoutReader: 'static + core::marker::Send + core::marker::Sync +pub fn vortex_layout::LayoutReader::as_any(&self) -> &dyn core::any::Any + pub fn vortex_layout::LayoutReader::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_layout::LayoutReader::filter_evaluation(&self, row_range: &core::ops::range::Range, expr: &vortex_array::expr::expression::Expression, mask: vortex_array::mask_future::MaskFuture) -> vortex_error::VortexResult +pub fn vortex_layout::LayoutReader::filter_evaluation(&self, &core::ops::range::Range, &vortex_array::expr::expression::Expression, vortex_array::mask_future::MaskFuture) -> vortex_error::VortexResult pub fn vortex_layout::LayoutReader::name(&self) -> &alloc::sync::Arc -pub fn vortex_layout::LayoutReader::projection_evaluation(&self, row_range: &core::ops::range::Range, expr: &vortex_array::expr::expression::Expression, mask: vortex_array::mask_future::MaskFuture) -> vortex_error::VortexResult +pub fn vortex_layout::LayoutReader::projection_evaluation(&self, &core::ops::range::Range, &vortex_array::expr::expression::Expression, vortex_array::mask_future::MaskFuture) -> vortex_error::VortexResult -pub fn vortex_layout::LayoutReader::pruning_evaluation(&self, row_range: &core::ops::range::Range, expr: &vortex_array::expr::expression::Expression, mask: vortex_mask::Mask) -> vortex_error::VortexResult +pub fn vortex_layout::LayoutReader::pruning_evaluation(&self, &core::ops::range::Range, &vortex_array::expr::expression::Expression, vortex_mask::Mask) -> vortex_error::VortexResult -pub fn vortex_layout::LayoutReader::register_splits(&self, field_mask: &[vortex_array::dtype::field_mask::FieldMask], row_range: &core::ops::range::Range, splits: &mut alloc::collections::btree::set::BTreeSet) -> vortex_error::VortexResult<()> +pub fn vortex_layout::LayoutReader::register_splits(&self, &[vortex_array::dtype::field_mask::FieldMask], &core::ops::range::Range, &mut alloc::collections::btree::set::BTreeSet) -> vortex_error::VortexResult<()> pub fn vortex_layout::LayoutReader::row_count(&self) -> u64 impl vortex_layout::LayoutReader for vortex_layout::layouts::row_idx::RowIdxLayoutReader +pub fn vortex_layout::layouts::row_idx::RowIdxLayoutReader::as_any(&self) -> &dyn core::any::Any + pub fn vortex_layout::layouts::row_idx::RowIdxLayoutReader::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_layout::layouts::row_idx::RowIdxLayoutReader::filter_evaluation(&self, row_range: &core::ops::range::Range, expr: &vortex_array::expr::expression::Expression, mask: vortex_array::mask_future::MaskFuture) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::row_idx::RowIdxLayoutReader::filter_evaluation(&self, &core::ops::range::Range, &vortex_array::expr::expression::Expression, vortex_array::mask_future::MaskFuture) -> vortex_error::VortexResult pub fn vortex_layout::layouts::row_idx::RowIdxLayoutReader::name(&self) -> &alloc::sync::Arc -pub fn vortex_layout::layouts::row_idx::RowIdxLayoutReader::projection_evaluation(&self, row_range: &core::ops::range::Range, expr: &vortex_array::expr::expression::Expression, mask: vortex_array::mask_future::MaskFuture) -> vortex_error::VortexResult>> +pub fn vortex_layout::layouts::row_idx::RowIdxLayoutReader::projection_evaluation(&self, &core::ops::range::Range, &vortex_array::expr::expression::Expression, vortex_array::mask_future::MaskFuture) -> vortex_error::VortexResult>> -pub fn vortex_layout::layouts::row_idx::RowIdxLayoutReader::pruning_evaluation(&self, row_range: &core::ops::range::Range, expr: &vortex_array::expr::expression::Expression, mask: vortex_mask::Mask) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::row_idx::RowIdxLayoutReader::pruning_evaluation(&self, &core::ops::range::Range, &vortex_array::expr::expression::Expression, vortex_mask::Mask) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::row_idx::RowIdxLayoutReader::register_splits(&self, field_mask: &[vortex_array::dtype::field_mask::FieldMask], row_range: &core::ops::range::Range, splits: &mut alloc::collections::btree::set::BTreeSet) -> vortex_error::VortexResult<()> +pub fn vortex_layout::layouts::row_idx::RowIdxLayoutReader::register_splits(&self, &[vortex_array::dtype::field_mask::FieldMask], &core::ops::range::Range, &mut alloc::collections::btree::set::BTreeSet) -> vortex_error::VortexResult<()> pub fn vortex_layout::layouts::row_idx::RowIdxLayoutReader::row_count(&self) -> u64 @@ -1868,67 +1872,67 @@ pub trait vortex_layout::LayoutStrategy: 'static + core::marker::Send + core::ma pub fn vortex_layout::LayoutStrategy::buffered_bytes(&self) -> u64 -pub fn vortex_layout::LayoutStrategy::write_stream<'life0, 'life1, 'async_trait>(&'life0 self, ctx: vortex_array::ArrayContext, segment_sink: vortex_layout::segments::SegmentSinkRef, stream: vortex_layout::sequence::SendableSequentialStream, eof: vortex_layout::sequence::SequencePointer, session: &'life1 vortex_session::VortexSession) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait +pub fn vortex_layout::LayoutStrategy::write_stream<'life0, 'life1, 'async_trait>(&'life0 self, vortex_array::ArrayContext, vortex_layout::segments::SegmentSinkRef, vortex_layout::sequence::SendableSequentialStream, vortex_layout::sequence::SequencePointer, &'life1 vortex_session::VortexSession) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait impl vortex_layout::LayoutStrategy for alloc::sync::Arc pub fn alloc::sync::Arc::buffered_bytes(&self) -> u64 -pub fn alloc::sync::Arc::write_stream<'life0, 'life1, 'async_trait>(&'life0 self, ctx: vortex_array::ArrayContext, segment_sink: vortex_layout::segments::SegmentSinkRef, stream: vortex_layout::sequence::SendableSequentialStream, eof: vortex_layout::sequence::SequencePointer, session: &'life1 vortex_session::VortexSession) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait +pub fn alloc::sync::Arc::write_stream<'life0, 'life1, 'async_trait>(&'life0 self, vortex_array::ArrayContext, vortex_layout::segments::SegmentSinkRef, vortex_layout::sequence::SendableSequentialStream, vortex_layout::sequence::SequencePointer, &'life1 vortex_session::VortexSession) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait impl vortex_layout::LayoutStrategy for vortex_layout::layouts::buffered::BufferedStrategy pub fn vortex_layout::layouts::buffered::BufferedStrategy::buffered_bytes(&self) -> u64 -pub fn vortex_layout::layouts::buffered::BufferedStrategy::write_stream<'life0, 'life1, 'async_trait>(&'life0 self, ctx: vortex_array::ArrayContext, segment_sink: vortex_layout::segments::SegmentSinkRef, stream: vortex_layout::sequence::SendableSequentialStream, eof: vortex_layout::sequence::SequencePointer, session: &'life1 vortex_session::VortexSession) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait +pub fn vortex_layout::layouts::buffered::BufferedStrategy::write_stream<'life0, 'life1, 'async_trait>(&'life0 self, vortex_array::ArrayContext, vortex_layout::segments::SegmentSinkRef, vortex_layout::sequence::SendableSequentialStream, vortex_layout::sequence::SequencePointer, &'life1 vortex_session::VortexSession) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait impl vortex_layout::LayoutStrategy for vortex_layout::layouts::chunked::writer::ChunkedLayoutStrategy pub fn vortex_layout::layouts::chunked::writer::ChunkedLayoutStrategy::buffered_bytes(&self) -> u64 -pub fn vortex_layout::layouts::chunked::writer::ChunkedLayoutStrategy::write_stream<'life0, 'life1, 'async_trait>(&'life0 self, ctx: vortex_array::ArrayContext, segment_sink: vortex_layout::segments::SegmentSinkRef, stream: vortex_layout::sequence::SendableSequentialStream, eof: vortex_layout::sequence::SequencePointer, session: &'life1 vortex_session::VortexSession) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait +pub fn vortex_layout::layouts::chunked::writer::ChunkedLayoutStrategy::write_stream<'life0, 'life1, 'async_trait>(&'life0 self, vortex_array::ArrayContext, vortex_layout::segments::SegmentSinkRef, vortex_layout::sequence::SendableSequentialStream, vortex_layout::sequence::SequencePointer, &'life1 vortex_session::VortexSession) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait impl vortex_layout::LayoutStrategy for vortex_layout::layouts::collect::CollectStrategy pub fn vortex_layout::layouts::collect::CollectStrategy::buffered_bytes(&self) -> u64 -pub fn vortex_layout::layouts::collect::CollectStrategy::write_stream<'life0, 'life1, 'async_trait>(&'life0 self, ctx: vortex_array::ArrayContext, segment_sink: vortex_layout::segments::SegmentSinkRef, stream: vortex_layout::sequence::SendableSequentialStream, eof: vortex_layout::sequence::SequencePointer, session: &'life1 vortex_session::VortexSession) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait +pub fn vortex_layout::layouts::collect::CollectStrategy::write_stream<'life0, 'life1, 'async_trait>(&'life0 self, vortex_array::ArrayContext, vortex_layout::segments::SegmentSinkRef, vortex_layout::sequence::SendableSequentialStream, vortex_layout::sequence::SequencePointer, &'life1 vortex_session::VortexSession) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait impl vortex_layout::LayoutStrategy for vortex_layout::layouts::compressed::CompressingStrategy pub fn vortex_layout::layouts::compressed::CompressingStrategy::buffered_bytes(&self) -> u64 -pub fn vortex_layout::layouts::compressed::CompressingStrategy::write_stream<'life0, 'life1, 'async_trait>(&'life0 self, ctx: vortex_array::ArrayContext, segment_sink: vortex_layout::segments::SegmentSinkRef, stream: vortex_layout::sequence::SendableSequentialStream, eof: vortex_layout::sequence::SequencePointer, session: &'life1 vortex_session::VortexSession) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait +pub fn vortex_layout::layouts::compressed::CompressingStrategy::write_stream<'life0, 'life1, 'async_trait>(&'life0 self, vortex_array::ArrayContext, vortex_layout::segments::SegmentSinkRef, vortex_layout::sequence::SendableSequentialStream, vortex_layout::sequence::SequencePointer, &'life1 vortex_session::VortexSession) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait impl vortex_layout::LayoutStrategy for vortex_layout::layouts::dict::writer::DictStrategy pub fn vortex_layout::layouts::dict::writer::DictStrategy::buffered_bytes(&self) -> u64 -pub fn vortex_layout::layouts::dict::writer::DictStrategy::write_stream<'life0, 'life1, 'async_trait>(&'life0 self, ctx: vortex_array::ArrayContext, segment_sink: vortex_layout::segments::SegmentSinkRef, stream: vortex_layout::sequence::SendableSequentialStream, eof: vortex_layout::sequence::SequencePointer, session: &'life1 vortex_session::VortexSession) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait +pub fn vortex_layout::layouts::dict::writer::DictStrategy::write_stream<'life0, 'life1, 'async_trait>(&'life0 self, vortex_array::ArrayContext, vortex_layout::segments::SegmentSinkRef, vortex_layout::sequence::SendableSequentialStream, vortex_layout::sequence::SequencePointer, &'life1 vortex_session::VortexSession) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait impl vortex_layout::LayoutStrategy for vortex_layout::layouts::flat::writer::FlatLayoutStrategy pub fn vortex_layout::layouts::flat::writer::FlatLayoutStrategy::buffered_bytes(&self) -> u64 -pub fn vortex_layout::layouts::flat::writer::FlatLayoutStrategy::write_stream<'life0, 'life1, 'async_trait>(&'life0 self, ctx: vortex_array::ArrayContext, segment_sink: vortex_layout::segments::SegmentSinkRef, stream: vortex_layout::sequence::SendableSequentialStream, _eof: vortex_layout::sequence::SequencePointer, session: &'life1 vortex_session::VortexSession) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait +pub fn vortex_layout::layouts::flat::writer::FlatLayoutStrategy::write_stream<'life0, 'life1, 'async_trait>(&'life0 self, vortex_array::ArrayContext, vortex_layout::segments::SegmentSinkRef, vortex_layout::sequence::SendableSequentialStream, vortex_layout::sequence::SequencePointer, &'life1 vortex_session::VortexSession) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait impl vortex_layout::LayoutStrategy for vortex_layout::layouts::repartition::RepartitionStrategy pub fn vortex_layout::layouts::repartition::RepartitionStrategy::buffered_bytes(&self) -> u64 -pub fn vortex_layout::layouts::repartition::RepartitionStrategy::write_stream<'life0, 'life1, 'async_trait>(&'life0 self, ctx: vortex_array::ArrayContext, segment_sink: vortex_layout::segments::SegmentSinkRef, stream: vortex_layout::sequence::SendableSequentialStream, eof: vortex_layout::sequence::SequencePointer, session: &'life1 vortex_session::VortexSession) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait +pub fn vortex_layout::layouts::repartition::RepartitionStrategy::write_stream<'life0, 'life1, 'async_trait>(&'life0 self, vortex_array::ArrayContext, vortex_layout::segments::SegmentSinkRef, vortex_layout::sequence::SendableSequentialStream, vortex_layout::sequence::SequencePointer, &'life1 vortex_session::VortexSession) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait impl vortex_layout::LayoutStrategy for vortex_layout::layouts::table::TableStrategy pub fn vortex_layout::layouts::table::TableStrategy::buffered_bytes(&self) -> u64 -pub fn vortex_layout::layouts::table::TableStrategy::write_stream<'life0, 'life1, 'async_trait>(&'life0 self, ctx: vortex_array::ArrayContext, segment_sink: vortex_layout::segments::SegmentSinkRef, stream: vortex_layout::sequence::SendableSequentialStream, eof: vortex_layout::sequence::SequencePointer, session: &'life1 vortex_session::VortexSession) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait +pub fn vortex_layout::layouts::table::TableStrategy::write_stream<'life0, 'life1, 'async_trait>(&'life0 self, vortex_array::ArrayContext, vortex_layout::segments::SegmentSinkRef, vortex_layout::sequence::SendableSequentialStream, vortex_layout::sequence::SequencePointer, &'life1 vortex_session::VortexSession) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait impl vortex_layout::LayoutStrategy for vortex_layout::layouts::zoned::writer::ZonedStrategy pub fn vortex_layout::layouts::zoned::writer::ZonedStrategy::buffered_bytes(&self) -> u64 -pub fn vortex_layout::layouts::zoned::writer::ZonedStrategy::write_stream<'life0, 'life1, 'async_trait>(&'life0 self, ctx: vortex_array::ArrayContext, segment_sink: vortex_layout::segments::SegmentSinkRef, stream: vortex_layout::sequence::SendableSequentialStream, eof: vortex_layout::sequence::SequencePointer, session: &'life1 vortex_session::VortexSession) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait +pub fn vortex_layout::layouts::zoned::writer::ZonedStrategy::write_stream<'life0, 'life1, 'async_trait>(&'life0 self, vortex_array::ArrayContext, vortex_layout::segments::SegmentSinkRef, vortex_layout::sequence::SendableSequentialStream, vortex_layout::sequence::SequencePointer, &'life1 vortex_session::VortexSession) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait pub trait vortex_layout::VTable: 'static + core::marker::Sized + core::marker::Send + core::marker::Sync + core::fmt::Debug @@ -1938,29 +1942,29 @@ pub type vortex_layout::VTable::Layout: 'static + core::marker::Send + core::mar pub type vortex_layout::VTable::Metadata: vortex_array::metadata::SerializeMetadata + vortex_array::metadata::DeserializeMetadata + core::fmt::Debug -pub fn vortex_layout::VTable::build(encoding: &Self::Encoding, dtype: &vortex_array::dtype::DType, row_count: u64, metadata: &::Output, segment_ids: alloc::vec::Vec, children: &dyn vortex_layout::LayoutChildren, ctx: &vortex_session::registry::ReadContext) -> vortex_error::VortexResult +pub fn vortex_layout::VTable::build(&Self::Encoding, &vortex_array::dtype::DType, u64, &::Output, alloc::vec::Vec, &dyn vortex_layout::LayoutChildren, &vortex_session::registry::ReadContext) -> vortex_error::VortexResult -pub fn vortex_layout::VTable::child(layout: &Self::Layout, idx: usize) -> vortex_error::VortexResult +pub fn vortex_layout::VTable::child(&Self::Layout, usize) -> vortex_error::VortexResult -pub fn vortex_layout::VTable::child_type(layout: &Self::Layout, idx: usize) -> vortex_layout::LayoutChildType +pub fn vortex_layout::VTable::child_type(&Self::Layout, usize) -> vortex_layout::LayoutChildType -pub fn vortex_layout::VTable::dtype(layout: &Self::Layout) -> &vortex_array::dtype::DType +pub fn vortex_layout::VTable::dtype(&Self::Layout) -> &vortex_array::dtype::DType -pub fn vortex_layout::VTable::encoding(layout: &Self::Layout) -> vortex_layout::LayoutEncodingRef +pub fn vortex_layout::VTable::encoding(&Self::Layout) -> vortex_layout::LayoutEncodingRef -pub fn vortex_layout::VTable::id(encoding: &Self::Encoding) -> vortex_layout::LayoutId +pub fn vortex_layout::VTable::id(&Self::Encoding) -> vortex_layout::LayoutId -pub fn vortex_layout::VTable::metadata(layout: &Self::Layout) -> Self::Metadata +pub fn vortex_layout::VTable::metadata(&Self::Layout) -> Self::Metadata -pub fn vortex_layout::VTable::nchildren(layout: &Self::Layout) -> usize +pub fn vortex_layout::VTable::nchildren(&Self::Layout) -> usize -pub fn vortex_layout::VTable::new_reader(layout: &Self::Layout, name: alloc::sync::Arc, segment_source: alloc::sync::Arc, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_layout::VTable::new_reader(&Self::Layout, alloc::sync::Arc, alloc::sync::Arc, &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_layout::VTable::row_count(layout: &Self::Layout) -> u64 +pub fn vortex_layout::VTable::row_count(&Self::Layout) -> u64 -pub fn vortex_layout::VTable::segment_ids(layout: &Self::Layout) -> alloc::vec::Vec +pub fn vortex_layout::VTable::segment_ids(&Self::Layout) -> alloc::vec::Vec -pub fn vortex_layout::VTable::with_children(_layout: &mut Self::Layout, _children: alloc::vec::Vec) -> vortex_error::VortexResult<()> +pub fn vortex_layout::VTable::with_children(&mut Self::Layout, alloc::vec::Vec) -> vortex_error::VortexResult<()> impl vortex_layout::VTable for vortex_layout::layouts::chunked::Chunked @@ -1970,29 +1974,29 @@ pub type vortex_layout::layouts::chunked::Chunked::Layout = vortex_layout::layou pub type vortex_layout::layouts::chunked::Chunked::Metadata = vortex_array::metadata::EmptyMetadata -pub fn vortex_layout::layouts::chunked::Chunked::build(_encoding: &Self::Encoding, dtype: &vortex_array::dtype::DType, row_count: u64, _metadata: &::Output, _segment_ids: alloc::vec::Vec, children: &dyn vortex_layout::LayoutChildren, _ctx: &vortex_session::registry::ReadContext) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::chunked::Chunked::build(&Self::Encoding, &vortex_array::dtype::DType, u64, &::Output, alloc::vec::Vec, &dyn vortex_layout::LayoutChildren, &vortex_session::registry::ReadContext) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::chunked::Chunked::child(layout: &Self::Layout, idx: usize) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::chunked::Chunked::child(&Self::Layout, usize) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::chunked::Chunked::child_type(layout: &Self::Layout, idx: usize) -> vortex_layout::LayoutChildType +pub fn vortex_layout::layouts::chunked::Chunked::child_type(&Self::Layout, usize) -> vortex_layout::LayoutChildType -pub fn vortex_layout::layouts::chunked::Chunked::dtype(layout: &Self::Layout) -> &vortex_array::dtype::DType +pub fn vortex_layout::layouts::chunked::Chunked::dtype(&Self::Layout) -> &vortex_array::dtype::DType -pub fn vortex_layout::layouts::chunked::Chunked::encoding(_layout: &Self::Layout) -> vortex_layout::LayoutEncodingRef +pub fn vortex_layout::layouts::chunked::Chunked::encoding(&Self::Layout) -> vortex_layout::LayoutEncodingRef -pub fn vortex_layout::layouts::chunked::Chunked::id(_encoding: &Self::Encoding) -> vortex_layout::LayoutId +pub fn vortex_layout::layouts::chunked::Chunked::id(&Self::Encoding) -> vortex_layout::LayoutId -pub fn vortex_layout::layouts::chunked::Chunked::metadata(_layout: &Self::Layout) -> Self::Metadata +pub fn vortex_layout::layouts::chunked::Chunked::metadata(&Self::Layout) -> Self::Metadata -pub fn vortex_layout::layouts::chunked::Chunked::nchildren(layout: &Self::Layout) -> usize +pub fn vortex_layout::layouts::chunked::Chunked::nchildren(&Self::Layout) -> usize -pub fn vortex_layout::layouts::chunked::Chunked::new_reader(layout: &Self::Layout, name: alloc::sync::Arc, segment_source: alloc::sync::Arc, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::chunked::Chunked::new_reader(&Self::Layout, alloc::sync::Arc, alloc::sync::Arc, &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::chunked::Chunked::row_count(layout: &Self::Layout) -> u64 +pub fn vortex_layout::layouts::chunked::Chunked::row_count(&Self::Layout) -> u64 -pub fn vortex_layout::layouts::chunked::Chunked::segment_ids(_layout: &Self::Layout) -> alloc::vec::Vec +pub fn vortex_layout::layouts::chunked::Chunked::segment_ids(&Self::Layout) -> alloc::vec::Vec -pub fn vortex_layout::layouts::chunked::Chunked::with_children(layout: &mut Self::Layout, children: alloc::vec::Vec) -> vortex_error::VortexResult<()> +pub fn vortex_layout::layouts::chunked::Chunked::with_children(&mut Self::Layout, alloc::vec::Vec) -> vortex_error::VortexResult<()> impl vortex_layout::VTable for vortex_layout::layouts::dict::Dict @@ -2002,29 +2006,29 @@ pub type vortex_layout::layouts::dict::Dict::Layout = vortex_layout::layouts::di pub type vortex_layout::layouts::dict::Dict::Metadata = vortex_array::metadata::ProstMetadata -pub fn vortex_layout::layouts::dict::Dict::build(_encoding: &Self::Encoding, dtype: &vortex_array::dtype::DType, _row_count: u64, metadata: &::Output, _segment_ids: alloc::vec::Vec, children: &dyn vortex_layout::LayoutChildren, _ctx: &vortex_session::registry::ReadContext) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::dict::Dict::build(&Self::Encoding, &vortex_array::dtype::DType, u64, &::Output, alloc::vec::Vec, &dyn vortex_layout::LayoutChildren, &vortex_session::registry::ReadContext) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::dict::Dict::child(layout: &Self::Layout, idx: usize) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::dict::Dict::child(&Self::Layout, usize) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::dict::Dict::child_type(_layout: &Self::Layout, idx: usize) -> vortex_layout::LayoutChildType +pub fn vortex_layout::layouts::dict::Dict::child_type(&Self::Layout, usize) -> vortex_layout::LayoutChildType -pub fn vortex_layout::layouts::dict::Dict::dtype(layout: &Self::Layout) -> &vortex_array::dtype::DType +pub fn vortex_layout::layouts::dict::Dict::dtype(&Self::Layout) -> &vortex_array::dtype::DType -pub fn vortex_layout::layouts::dict::Dict::encoding(_layout: &Self::Layout) -> vortex_layout::LayoutEncodingRef +pub fn vortex_layout::layouts::dict::Dict::encoding(&Self::Layout) -> vortex_layout::LayoutEncodingRef -pub fn vortex_layout::layouts::dict::Dict::id(_encoding: &Self::Encoding) -> vortex_layout::LayoutId +pub fn vortex_layout::layouts::dict::Dict::id(&Self::Encoding) -> vortex_layout::LayoutId -pub fn vortex_layout::layouts::dict::Dict::metadata(layout: &Self::Layout) -> Self::Metadata +pub fn vortex_layout::layouts::dict::Dict::metadata(&Self::Layout) -> Self::Metadata -pub fn vortex_layout::layouts::dict::Dict::nchildren(_layout: &Self::Layout) -> usize +pub fn vortex_layout::layouts::dict::Dict::nchildren(&Self::Layout) -> usize -pub fn vortex_layout::layouts::dict::Dict::new_reader(layout: &Self::Layout, name: alloc::sync::Arc, segment_source: alloc::sync::Arc, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::dict::Dict::new_reader(&Self::Layout, alloc::sync::Arc, alloc::sync::Arc, &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::dict::Dict::row_count(layout: &Self::Layout) -> u64 +pub fn vortex_layout::layouts::dict::Dict::row_count(&Self::Layout) -> u64 -pub fn vortex_layout::layouts::dict::Dict::segment_ids(_layout: &Self::Layout) -> alloc::vec::Vec +pub fn vortex_layout::layouts::dict::Dict::segment_ids(&Self::Layout) -> alloc::vec::Vec -pub fn vortex_layout::layouts::dict::Dict::with_children(layout: &mut Self::Layout, children: alloc::vec::Vec) -> vortex_error::VortexResult<()> +pub fn vortex_layout::layouts::dict::Dict::with_children(&mut Self::Layout, alloc::vec::Vec) -> vortex_error::VortexResult<()> impl vortex_layout::VTable for vortex_layout::layouts::flat::Flat @@ -2034,29 +2038,29 @@ pub type vortex_layout::layouts::flat::Flat::Layout = vortex_layout::layouts::fl pub type vortex_layout::layouts::flat::Flat::Metadata = vortex_array::metadata::ProstMetadata -pub fn vortex_layout::layouts::flat::Flat::build(_encoding: &Self::Encoding, dtype: &vortex_array::dtype::DType, row_count: u64, metadata: &::Output, segment_ids: alloc::vec::Vec, _children: &dyn vortex_layout::LayoutChildren, ctx: &vortex_session::registry::ReadContext) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::flat::Flat::build(&Self::Encoding, &vortex_array::dtype::DType, u64, &::Output, alloc::vec::Vec, &dyn vortex_layout::LayoutChildren, &vortex_session::registry::ReadContext) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::flat::Flat::child(_layout: &Self::Layout, _idx: usize) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::flat::Flat::child(&Self::Layout, usize) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::flat::Flat::child_type(_layout: &Self::Layout, _idx: usize) -> vortex_layout::LayoutChildType +pub fn vortex_layout::layouts::flat::Flat::child_type(&Self::Layout, usize) -> vortex_layout::LayoutChildType -pub fn vortex_layout::layouts::flat::Flat::dtype(layout: &Self::Layout) -> &vortex_array::dtype::DType +pub fn vortex_layout::layouts::flat::Flat::dtype(&Self::Layout) -> &vortex_array::dtype::DType -pub fn vortex_layout::layouts::flat::Flat::encoding(_layout: &Self::Layout) -> vortex_layout::LayoutEncodingRef +pub fn vortex_layout::layouts::flat::Flat::encoding(&Self::Layout) -> vortex_layout::LayoutEncodingRef -pub fn vortex_layout::layouts::flat::Flat::id(_encoding: &Self::Encoding) -> vortex_layout::LayoutId +pub fn vortex_layout::layouts::flat::Flat::id(&Self::Encoding) -> vortex_layout::LayoutId -pub fn vortex_layout::layouts::flat::Flat::metadata(layout: &Self::Layout) -> Self::Metadata +pub fn vortex_layout::layouts::flat::Flat::metadata(&Self::Layout) -> Self::Metadata -pub fn vortex_layout::layouts::flat::Flat::nchildren(_layout: &Self::Layout) -> usize +pub fn vortex_layout::layouts::flat::Flat::nchildren(&Self::Layout) -> usize -pub fn vortex_layout::layouts::flat::Flat::new_reader(layout: &Self::Layout, name: alloc::sync::Arc, segment_source: alloc::sync::Arc, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::flat::Flat::new_reader(&Self::Layout, alloc::sync::Arc, alloc::sync::Arc, &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::flat::Flat::row_count(layout: &Self::Layout) -> u64 +pub fn vortex_layout::layouts::flat::Flat::row_count(&Self::Layout) -> u64 -pub fn vortex_layout::layouts::flat::Flat::segment_ids(layout: &Self::Layout) -> alloc::vec::Vec +pub fn vortex_layout::layouts::flat::Flat::segment_ids(&Self::Layout) -> alloc::vec::Vec -pub fn vortex_layout::layouts::flat::Flat::with_children(_layout: &mut Self::Layout, children: alloc::vec::Vec) -> vortex_error::VortexResult<()> +pub fn vortex_layout::layouts::flat::Flat::with_children(&mut Self::Layout, alloc::vec::Vec) -> vortex_error::VortexResult<()> impl vortex_layout::VTable for vortex_layout::layouts::struct_::Struct @@ -2066,29 +2070,29 @@ pub type vortex_layout::layouts::struct_::Struct::Layout = vortex_layout::layout pub type vortex_layout::layouts::struct_::Struct::Metadata = vortex_array::metadata::EmptyMetadata -pub fn vortex_layout::layouts::struct_::Struct::build(_encoding: &Self::Encoding, dtype: &vortex_array::dtype::DType, row_count: u64, _metadata: &::Output, _segment_ids: alloc::vec::Vec, children: &dyn vortex_layout::LayoutChildren, _ctx: &vortex_session::registry::ReadContext) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::struct_::Struct::build(&Self::Encoding, &vortex_array::dtype::DType, u64, &::Output, alloc::vec::Vec, &dyn vortex_layout::LayoutChildren, &vortex_session::registry::ReadContext) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::struct_::Struct::child(layout: &Self::Layout, index: usize) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::struct_::Struct::child(&Self::Layout, usize) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::struct_::Struct::child_type(layout: &Self::Layout, idx: usize) -> vortex_layout::LayoutChildType +pub fn vortex_layout::layouts::struct_::Struct::child_type(&Self::Layout, usize) -> vortex_layout::LayoutChildType -pub fn vortex_layout::layouts::struct_::Struct::dtype(layout: &Self::Layout) -> &vortex_array::dtype::DType +pub fn vortex_layout::layouts::struct_::Struct::dtype(&Self::Layout) -> &vortex_array::dtype::DType -pub fn vortex_layout::layouts::struct_::Struct::encoding(_layout: &Self::Layout) -> vortex_layout::LayoutEncodingRef +pub fn vortex_layout::layouts::struct_::Struct::encoding(&Self::Layout) -> vortex_layout::LayoutEncodingRef -pub fn vortex_layout::layouts::struct_::Struct::id(_encoding: &Self::Encoding) -> vortex_layout::LayoutId +pub fn vortex_layout::layouts::struct_::Struct::id(&Self::Encoding) -> vortex_layout::LayoutId -pub fn vortex_layout::layouts::struct_::Struct::metadata(_layout: &Self::Layout) -> Self::Metadata +pub fn vortex_layout::layouts::struct_::Struct::metadata(&Self::Layout) -> Self::Metadata -pub fn vortex_layout::layouts::struct_::Struct::nchildren(layout: &Self::Layout) -> usize +pub fn vortex_layout::layouts::struct_::Struct::nchildren(&Self::Layout) -> usize -pub fn vortex_layout::layouts::struct_::Struct::new_reader(layout: &Self::Layout, name: alloc::sync::Arc, segment_source: alloc::sync::Arc, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::struct_::Struct::new_reader(&Self::Layout, alloc::sync::Arc, alloc::sync::Arc, &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::struct_::Struct::row_count(layout: &Self::Layout) -> u64 +pub fn vortex_layout::layouts::struct_::Struct::row_count(&Self::Layout) -> u64 -pub fn vortex_layout::layouts::struct_::Struct::segment_ids(_layout: &Self::Layout) -> alloc::vec::Vec +pub fn vortex_layout::layouts::struct_::Struct::segment_ids(&Self::Layout) -> alloc::vec::Vec -pub fn vortex_layout::layouts::struct_::Struct::with_children(layout: &mut Self::Layout, children: alloc::vec::Vec) -> vortex_error::VortexResult<()> +pub fn vortex_layout::layouts::struct_::Struct::with_children(&mut Self::Layout, alloc::vec::Vec) -> vortex_error::VortexResult<()> impl vortex_layout::VTable for vortex_layout::layouts::zoned::Zoned @@ -2098,43 +2102,43 @@ pub type vortex_layout::layouts::zoned::Zoned::Layout = vortex_layout::layouts:: pub type vortex_layout::layouts::zoned::Zoned::Metadata = vortex_layout::layouts::zoned::ZonedMetadata -pub fn vortex_layout::layouts::zoned::Zoned::build(_encoding: &Self::Encoding, dtype: &vortex_array::dtype::DType, _row_count: u64, metadata: &vortex_layout::layouts::zoned::ZonedMetadata, _segment_ids: alloc::vec::Vec, children: &dyn vortex_layout::LayoutChildren, _ctx: &vortex_session::registry::ReadContext) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::zoned::Zoned::build(&Self::Encoding, &vortex_array::dtype::DType, u64, &vortex_layout::layouts::zoned::ZonedMetadata, alloc::vec::Vec, &dyn vortex_layout::LayoutChildren, &vortex_session::registry::ReadContext) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::zoned::Zoned::child(layout: &Self::Layout, idx: usize) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::zoned::Zoned::child(&Self::Layout, usize) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::zoned::Zoned::child_type(_layout: &Self::Layout, idx: usize) -> vortex_layout::LayoutChildType +pub fn vortex_layout::layouts::zoned::Zoned::child_type(&Self::Layout, usize) -> vortex_layout::LayoutChildType -pub fn vortex_layout::layouts::zoned::Zoned::dtype(layout: &Self::Layout) -> &vortex_array::dtype::DType +pub fn vortex_layout::layouts::zoned::Zoned::dtype(&Self::Layout) -> &vortex_array::dtype::DType -pub fn vortex_layout::layouts::zoned::Zoned::encoding(_layout: &Self::Layout) -> vortex_layout::LayoutEncodingRef +pub fn vortex_layout::layouts::zoned::Zoned::encoding(&Self::Layout) -> vortex_layout::LayoutEncodingRef -pub fn vortex_layout::layouts::zoned::Zoned::id(_encoding: &Self::Encoding) -> vortex_layout::LayoutId +pub fn vortex_layout::layouts::zoned::Zoned::id(&Self::Encoding) -> vortex_layout::LayoutId -pub fn vortex_layout::layouts::zoned::Zoned::metadata(layout: &Self::Layout) -> Self::Metadata +pub fn vortex_layout::layouts::zoned::Zoned::metadata(&Self::Layout) -> Self::Metadata -pub fn vortex_layout::layouts::zoned::Zoned::nchildren(_layout: &Self::Layout) -> usize +pub fn vortex_layout::layouts::zoned::Zoned::nchildren(&Self::Layout) -> usize -pub fn vortex_layout::layouts::zoned::Zoned::new_reader(layout: &Self::Layout, name: alloc::sync::Arc, segment_source: alloc::sync::Arc, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_layout::layouts::zoned::Zoned::new_reader(&Self::Layout, alloc::sync::Arc, alloc::sync::Arc, &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_layout::layouts::zoned::Zoned::row_count(layout: &Self::Layout) -> u64 +pub fn vortex_layout::layouts::zoned::Zoned::row_count(&Self::Layout) -> u64 -pub fn vortex_layout::layouts::zoned::Zoned::segment_ids(_layout: &Self::Layout) -> alloc::vec::Vec +pub fn vortex_layout::layouts::zoned::Zoned::segment_ids(&Self::Layout) -> alloc::vec::Vec -pub fn vortex_layout::layouts::zoned::Zoned::with_children(layout: &mut Self::Layout, children: alloc::vec::Vec) -> vortex_error::VortexResult<()> +pub fn vortex_layout::layouts::zoned::Zoned::with_children(&mut Self::Layout, alloc::vec::Vec) -> vortex_error::VortexResult<()> -pub fn vortex_layout::layout_from_flatbuffer(flatbuffer: vortex_flatbuffers::FlatBuffer, dtype: &vortex_array::dtype::DType, layout_ctx: &vortex_session::registry::ReadContext, ctx: &vortex_session::registry::ReadContext, layouts: &vortex_layout::session::LayoutRegistry) -> vortex_error::VortexResult +pub fn vortex_layout::layout_from_flatbuffer(vortex_flatbuffers::FlatBuffer, &vortex_array::dtype::DType, &vortex_session::registry::ReadContext, &vortex_session::registry::ReadContext, &vortex_layout::session::LayoutRegistry) -> vortex_error::VortexResult -pub fn vortex_layout::layout_from_flatbuffer_with_options(flatbuffer: vortex_flatbuffers::FlatBuffer, dtype: &vortex_array::dtype::DType, layout_ctx: &vortex_session::registry::ReadContext, ctx: &vortex_session::registry::ReadContext, layouts: &vortex_layout::session::LayoutRegistry, allow_unknown: bool) -> vortex_error::VortexResult +pub fn vortex_layout::layout_from_flatbuffer_with_options(vortex_flatbuffers::FlatBuffer, &vortex_array::dtype::DType, &vortex_session::registry::ReadContext, &vortex_session::registry::ReadContext, &vortex_layout::session::LayoutRegistry, bool) -> vortex_error::VortexResult pub type vortex_layout::ArrayFuture = futures_core::future::BoxFuture<'static, vortex_error::VortexResult> pub type vortex_layout::LayoutContext = vortex_session::registry::Context -pub type vortex_layout::LayoutEncodingId = arcref::ArcRef +pub type vortex_layout::LayoutEncodingId = vortex_session::registry::Id pub type vortex_layout::LayoutEncodingRef = arcref::ArcRef -pub type vortex_layout::LayoutId = arcref::ArcRef +pub type vortex_layout::LayoutId = vortex_session::registry::Id pub type vortex_layout::LayoutReaderRef = alloc::sync::Arc diff --git a/vortex-layout/src/display.rs b/vortex-layout/src/display.rs index 39605dfb5d1..5acf685e946 100644 --- a/vortex-layout/src/display.rs +++ b/vortex-layout/src/display.rs @@ -231,7 +231,6 @@ mod tests { use vortex_buffer::buffer; use vortex_io::runtime::single::block_on; use vortex_io::session::RuntimeSessionExt; - use vortex_utils::env::EnvVarGuard; use crate::IntoLayout; use crate::OwnedLayoutChildren; @@ -248,231 +247,244 @@ mod tests { /// Test display_tree with inline array_tree metadata (no segment source needed). #[test] fn test_display_tree_inline_array_tree() { - let _guard = EnvVarGuard::set("FLAT_LAYOUT_INLINE_ARRAY_NODE", "1"); - block_on(|handle| async move { - let session = SESSION.clone().with_handle(handle); - let ctx = ArrayContext::empty(); - let segments = Arc::new(TestSegments::default()); - - // Create nullable i64 array (2 buffers: data + validity) - let (ptr1, eof1) = SequenceId::root().split(); - let mut validity_builder = BitBufferMut::with_capacity(5); - for b in [true, false, true, true, false] { - validity_builder.append(b); - } - let validity = Validity::Array( - BoolArray::new(validity_builder.freeze(), Validity::NonNullable).into_array(), - ); - let array1 = PrimitiveArray::new(buffer![1i64, 2, 3, 4, 5], validity); - let layout1 = FlatLayoutStrategy::default() - .write_stream( - ctx.clone(), - Arc::::clone(&segments), - array1.into_array().to_array_stream().sequenced(ptr1), - eof1, - &session, - ) - .await - .unwrap(); - - // Create utf8 array (2 buffers: views + data) - let (ptr2, eof2) = SequenceId::root().split(); - let mut builder = VarBinViewBuilder::with_capacity(DType::Utf8(NonNullable), 5); - for s in [ - "hello world this is long", - "another long string", - "short", - "medium str", - "x", - ] { - builder.append_value(s); - } - let layout2 = FlatLayoutStrategy::default() - .write_stream( - ctx.clone(), - Arc::::clone(&segments), - builder - .finish() - .into_array() - .to_array_stream() - .sequenced(ptr2), - eof2, - &session, - ) - .await - .unwrap(); - - // Create struct layout - let struct_layout = StructLayout::new( - 5, - DType::Struct( - StructFields::new( - vec![FieldName::from("numbers"), FieldName::from("strings")].into(), + // LazyLock caches the env var on first read, so only nextest (separate processes) can isolate it. + if std::env::var("NEXTEST_RUN_ID").is_ok() { + temp_env::with_var("FLAT_LAYOUT_INLINE_ARRAY_NODE", Some("1"), || { + block_on(|handle| async move { + let session = SESSION.clone().with_handle(handle); + let ctx = ArrayContext::empty(); + let segments = Arc::new(TestSegments::default()); + + // Create nullable i64 array (2 buffers: data + validity) + let (ptr1, eof1) = SequenceId::root().split(); + let mut validity_builder = BitBufferMut::with_capacity(5); + for b in [true, false, true, true, false] { + validity_builder.append(b); + } + let validity = Validity::Array( + BoolArray::new(validity_builder.freeze(), Validity::NonNullable) + .into_array(), + ); + let array1 = PrimitiveArray::new(buffer![1i64, 2, 3, 4, 5], validity); + let layout1 = FlatLayoutStrategy::default() + .write_stream( + ctx.clone(), + Arc::::clone(&segments), + array1.into_array().to_array_stream().sequenced(ptr1), + eof1, + &session, + ) + .await + .unwrap(); + + // Create utf8 array (2 buffers: views + data) + let (ptr2, eof2) = SequenceId::root().split(); + let mut builder = VarBinViewBuilder::with_capacity(DType::Utf8(NonNullable), 5); + for s in [ + "hello world this is long", + "another long string", + "short", + "medium str", + "x", + ] { + builder.append_value(s); + } + let layout2 = FlatLayoutStrategy::default() + .write_stream( + ctx.clone(), + Arc::::clone(&segments), + builder + .finish() + .into_array() + .to_array_stream() + .sequenced(ptr2), + eof2, + &session, + ) + .await + .unwrap(); + + // Create struct layout + let struct_layout = StructLayout::new( + 5, + DType::Struct( + StructFields::new( + vec![FieldName::from("numbers"), FieldName::from("strings")].into(), + vec![ + DType::Primitive(PType::I64, Nullability::Nullable), + DType::Utf8(NonNullable), + ], + ), + NonNullable, + ), vec![ - DType::Primitive(PType::I64, Nullability::Nullable), - DType::Utf8(NonNullable), + ChunkedLayout::new( + 5, + DType::Primitive(PType::I64, Nullability::Nullable), + OwnedLayoutChildren::layout_children(vec![layout1]), + ) + .into_layout(), + layout2, ], - ), - NonNullable, - ), - vec![ - ChunkedLayout::new( - 5, - DType::Primitive(PType::I64, Nullability::Nullable), - OwnedLayoutChildren::layout_children(vec![layout1]), ) - .into_layout(), - layout2, - ], - ) - .into_layout(); + .into_layout(); - let output = format!("{}", struct_layout.display_tree_verbose(true)); + let output = format!("{}", struct_layout.display_tree_verbose(true)); - let expected = "\ + let expected = "\ vortex.struct, dtype: {numbers=i64?, strings=utf8}, children: 2, rows: 5 ├── numbers: vortex.chunked, dtype: i64?, children: 1, rows: 5 │ └── [0]: vortex.flat, dtype: i64?, metadata: 171 bytes, rows: 5, segment 0, buffers=[40B, 1B], total=41B └── strings: vortex.flat, dtype: utf8, metadata: 110 bytes, rows: 5, segment 1, buffers=[43B, 80B], total=123B "; - assert_eq!(output, expected); - }) + assert_eq!(output, expected); + }) + }) + } } /// Test display_tree_with_segments using async segment source to fetch buffer sizes. #[test] fn test_display_tree_with_segment_source() { - // Ensure inline array node is disabled for this test - let _guard = EnvVarGuard::remove("FLAT_LAYOUT_INLINE_ARRAY_NODE"); - block_on(|handle| async move { - let session = SESSION.clone().with_handle(handle); - let ctx = ArrayContext::empty(); - let segments = Arc::new(TestSegments::default()); - - // Create simple i32 array - let (ptr1, eof1) = SequenceId::root().split(); - let array1 = PrimitiveArray::new(buffer![1i32, 2, 3, 4, 5], Validity::NonNullable); - let layout1 = FlatLayoutStrategy::default() - .write_stream( - ctx.clone(), - Arc::::clone(&segments), - array1.into_array().to_array_stream().sequenced(ptr1), - eof1, - &session, - ) - .await - .unwrap(); - - // Create another i32 array - let (ptr2, eof2) = SequenceId::root().split(); - let array2 = PrimitiveArray::new(buffer![6i32, 7, 8, 9, 10], Validity::NonNullable); - let layout2 = FlatLayoutStrategy::default() - .write_stream( - ctx.clone(), - Arc::::clone(&segments), - array2.into_array().to_array_stream().sequenced(ptr2), - eof2, - &session, - ) - .await - .unwrap(); - - // Create chunked layout - let chunked_layout = ChunkedLayout::new( - 10, - DType::Primitive(PType::I32, NonNullable), - OwnedLayoutChildren::layout_children(vec![layout1, layout2]), - ) - .into_layout(); - - let output = chunked_layout - .display_tree_with_segments(segments) - .await - .unwrap(); - - let expected = "\ + if std::env::var("NEXTEST_RUN_ID").is_ok() { + temp_env::with_var("FLAT_LAYOUT_INLINE_ARRAY_NODE", None::<&str>, || { + block_on(|handle| async move { + let session = SESSION.clone().with_handle(handle); + let ctx = ArrayContext::empty(); + let segments = Arc::new(TestSegments::default()); + + // Create simple i32 array + let (ptr1, eof1) = SequenceId::root().split(); + let array1 = + PrimitiveArray::new(buffer![1i32, 2, 3, 4, 5], Validity::NonNullable); + let layout1 = FlatLayoutStrategy::default() + .write_stream( + ctx.clone(), + Arc::::clone(&segments), + array1.into_array().to_array_stream().sequenced(ptr1), + eof1, + &session, + ) + .await + .unwrap(); + + // Create another i32 array + let (ptr2, eof2) = SequenceId::root().split(); + let array2 = + PrimitiveArray::new(buffer![6i32, 7, 8, 9, 10], Validity::NonNullable); + let layout2 = FlatLayoutStrategy::default() + .write_stream( + ctx.clone(), + Arc::::clone(&segments), + array2.into_array().to_array_stream().sequenced(ptr2), + eof2, + &session, + ) + .await + .unwrap(); + + // Create chunked layout + let chunked_layout = ChunkedLayout::new( + 10, + DType::Primitive(PType::I32, NonNullable), + OwnedLayoutChildren::layout_children(vec![layout1, layout2]), + ) + .into_layout(); + + let output = chunked_layout + .display_tree_with_segments(segments) + .await + .unwrap(); + + let expected = "\ vortex.chunked, dtype: i32, children: 2, rows: 10 ├── [0]: vortex.flat, dtype: i32, rows: 5, segment 0, buffers=[20B], total=20B └── [1]: vortex.flat, dtype: i32, rows: 5, segment 1, buffers=[20B], total=20B "; - assert_eq!(output.to_string(), expected); - }) + assert_eq!(output.to_string(), expected); + }) + }) + } } /// Test display_array_tree with inline array node metadata. #[test] fn test_display_array_tree_with_inline_node() { - let _guard = EnvVarGuard::set("FLAT_LAYOUT_INLINE_ARRAY_NODE", "1"); - - let ctx = ArrayContext::empty(); - let segments = Arc::new(TestSegments::default()); - let (ptr, eof) = SequenceId::root().split(); - - // Create a simple primitive array - let array = PrimitiveArray::new(buffer![1i32, 2, 3, 4, 5], Validity::AllValid); - let layout = block_on(|handle| async { - let session = SESSION.clone().with_handle(handle); - FlatLayoutStrategy::default() - .write_stream( - ctx.clone(), - Arc::::clone(&segments), - array.into_array().to_array_stream().sequenced(ptr), - eof, - &session, - ) - .await - .unwrap() - }); - - let flat_layout = layout.as_::(); - - let array_tree = flat_layout - .array_tree() - .expect("array_tree should be populated when FLAT_LAYOUT_INLINE_ARRAY_NODE is set"); - - let parts = SerializedArray::from_array_tree(array_tree.as_ref().to_vec()) - .expect("should parse array_tree"); - assert_eq!(parts.buffer_lengths(), vec![20]); // 5 i32 values = 20 bytes - - assert_eq!( - layout.display_tree().to_string(), - "\ + if std::env::var("NEXTEST_RUN_ID").is_ok() { + temp_env::with_var("FLAT_LAYOUT_INLINE_ARRAY_NODE", Some("1"), || { + let ctx = ArrayContext::empty(); + let segments = Arc::new(TestSegments::default()); + let (ptr, eof) = SequenceId::root().split(); + + // Create a simple primitive array + let array = PrimitiveArray::new(buffer![1i32, 2, 3, 4, 5], Validity::AllValid); + let layout = block_on(|handle| async { + let session = SESSION.clone().with_handle(handle); + FlatLayoutStrategy::default() + .write_stream( + ctx.clone(), + Arc::::clone(&segments), + array.into_array().to_array_stream().sequenced(ptr), + eof, + &session, + ) + .await + .unwrap() + }); + + let flat_layout = layout.as_::(); + + let array_tree = flat_layout.array_tree().expect( + "array_tree should be populated when FLAT_LAYOUT_INLINE_ARRAY_NODE is set", + ); + + let parts = SerializedArray::from_array_tree(array_tree.as_ref().to_vec()) + .expect("should parse array_tree"); + assert_eq!(parts.buffer_lengths(), vec![20]); // 5 i32 values = 20 bytes + + assert_eq!( + layout.display_tree().to_string(), + "\ vortex.flat, dtype: i32?, segment 0, buffers=[20B], total=20B " - ); + ); + }) + } } /// Test display_tree without inline array node (shows segment ID). #[test] fn test_display_tree_without_inline_node() { - let _guard = EnvVarGuard::set("FLAT_LAYOUT_INLINE_ARRAY_NODE", "1"); - - let ctx = ArrayContext::empty(); - let segments = Arc::new(TestSegments::default()); - let (ptr, eof) = SequenceId::root().split(); - - // Create a simple primitive array - let array = PrimitiveArray::new(buffer![10i64, 20, 30], Validity::NonNullable); - let layout = block_on(|handle| async { - let session = SESSION.clone().with_handle(handle); - FlatLayoutStrategy::default() - .write_stream( - ctx, - Arc::::clone(&segments), - array.into_array().to_array_stream().sequenced(ptr), - eof, - &session, - ) - .await - .unwrap() - }); - - // Test display_tree exact output (with inline array_tree enabled by env var from other test) - assert_eq!( - layout.display_tree().to_string(), - "\ + if std::env::var("NEXTEST_RUN_ID").is_ok() { + temp_env::with_var("FLAT_LAYOUT_INLINE_ARRAY_NODE", Some("1"), || { + let ctx = ArrayContext::empty(); + let segments = Arc::new(TestSegments::default()); + let (ptr, eof) = SequenceId::root().split(); + + // Create a simple primitive array + let array = PrimitiveArray::new(buffer![10i64, 20, 30], Validity::NonNullable); + let layout = block_on(|handle| async { + let session = SESSION.clone().with_handle(handle); + FlatLayoutStrategy::default() + .write_stream( + ctx, + Arc::::clone(&segments), + array.into_array().to_array_stream().sequenced(ptr), + eof, + &session, + ) + .await + .unwrap() + }); + + // Test display_tree exact output (with inline array_tree enabled by env var from other test) + assert_eq!( + layout.display_tree().to_string(), + "\ vortex.flat, dtype: i64, segment 0, buffers=[24B], total=24B " - ); + ); + }) + } } } diff --git a/vortex-layout/src/encoding.rs b/vortex-layout/src/encoding.rs index 271a94f9059..828634a8902 100644 --- a/vortex-layout/src/encoding.rs +++ b/vortex-layout/src/encoding.rs @@ -12,6 +12,7 @@ use vortex_array::dtype::DType; use vortex_error::VortexExpect; use vortex_error::VortexResult; use vortex_error::vortex_panic; +use vortex_session::registry::Id; use vortex_session::registry::ReadContext; use crate::IntoLayout; @@ -20,7 +21,8 @@ use crate::LayoutRef; use crate::VTable; use crate::segments::SegmentId; -pub type LayoutEncodingId = ArcRef; +/// A unique identifier for a layout encoding. +pub type LayoutEncodingId = Id; pub type LayoutEncodingRef = ArcRef; pub trait LayoutEncoding: 'static + Send + Sync + Debug + private::Sealed { diff --git a/vortex-layout/src/flatbuffers.rs b/vortex-layout/src/flatbuffers.rs index 02e05b199c8..4063e23eff7 100644 --- a/vortex-layout/src/flatbuffers.rs +++ b/vortex-layout/src/flatbuffers.rs @@ -259,8 +259,8 @@ mod tests { ); let layout_ctx = ReadContext::new([ - LayoutEncodingId::new_ref("vortex.test.foreign_layout"), - LayoutEncodingId::new_ref("vortex.test.foreign_child_layout"), + LayoutEncodingId::new("vortex.test.foreign_layout"), + LayoutEncodingId::new("vortex.test.foreign_child_layout"), ]); let array_ctx = ReadContext::new([]); let layouts = LayoutSession::default().registry().clone(); diff --git a/vortex-layout/src/layout.rs b/vortex-layout/src/layout.rs index 0d805623c0f..11f0afd629c 100644 --- a/vortex-layout/src/layout.rs +++ b/vortex-layout/src/layout.rs @@ -7,7 +7,6 @@ use std::fmt::Display; use std::fmt::Formatter; use std::sync::Arc; -use arcref::ArcRef; use itertools::Itertools; use vortex_array::SerializeMetadata; use vortex_array::dtype::DType; @@ -16,6 +15,7 @@ use vortex_error::VortexExpect; use vortex_error::VortexResult; use vortex_error::vortex_err; use vortex_session::VortexSession; +use vortex_session::registry::Id; use crate::LayoutEncodingId; use crate::LayoutEncodingRef; @@ -26,7 +26,8 @@ use crate::display::display_tree_with_segment_sizes; use crate::segments::SegmentId; use crate::segments::SegmentSource; -pub type LayoutId = ArcRef; +/// A unique identifier for a layout. +pub type LayoutId = Id; pub type LayoutRef = Arc; diff --git a/vortex-layout/src/layouts/chunked/mod.rs b/vortex-layout/src/layouts/chunked/mod.rs index af3c8bef4fa..82fcb650607 100644 --- a/vortex-layout/src/layouts/chunked/mod.rs +++ b/vortex-layout/src/layouts/chunked/mod.rs @@ -34,7 +34,7 @@ impl VTable for Chunked { type Metadata = EmptyMetadata; fn id(_encoding: &Self::Encoding) -> LayoutId { - LayoutId::new_ref("vortex.chunked") + LayoutId::new("vortex.chunked") } fn encoding(_layout: &Self::Layout) -> LayoutEncodingRef { diff --git a/vortex-layout/src/layouts/chunked/reader.rs b/vortex-layout/src/layouts/chunked/reader.rs index 756e5573704..376b6654707 100644 --- a/vortex-layout/src/layouts/chunked/reader.rs +++ b/vortex-layout/src/layouts/chunked/reader.rs @@ -324,6 +324,10 @@ impl LayoutReader for ChunkedReader { } .boxed()) } + + fn as_any(&self) -> &dyn std::any::Any { + self + } } #[cfg(test)] diff --git a/vortex-layout/src/layouts/compressed.rs b/vortex-layout/src/layouts/compressed.rs index 24dc2a02874..efd85db9e5d 100644 --- a/vortex-layout/src/layouts/compressed.rs +++ b/vortex-layout/src/layouts/compressed.rs @@ -7,11 +7,14 @@ use async_trait::async_trait; use futures::StreamExt as _; use vortex_array::ArrayContext; use vortex_array::ArrayRef; +use vortex_array::ExecutionCtx; +use vortex_array::VortexSessionExecute; use vortex_array::expr::stats::Stat; use vortex_btrblocks::BtrBlocksCompressor; use vortex_error::VortexResult; use vortex_io::session::RuntimeSessionExt; use vortex_session::VortexSession; +use vortex_utils::parallelism::get_available_parallelism; use crate::LayoutRef; use crate::LayoutStrategy; @@ -25,27 +28,27 @@ use crate::sequence::SequentialStreamExt; /// /// API consumers are free to implement this trait to provide new plugin compressors. pub trait CompressorPlugin: Send + Sync + 'static { - fn compress_chunk(&self, chunk: &ArrayRef) -> VortexResult; + fn compress_chunk(&self, chunk: &ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult; } impl CompressorPlugin for Arc { - fn compress_chunk(&self, chunk: &ArrayRef) -> VortexResult { - self.as_ref().compress_chunk(chunk) + fn compress_chunk(&self, chunk: &ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + self.as_ref().compress_chunk(chunk, ctx) } } impl CompressorPlugin for F where - F: Fn(&ArrayRef) -> VortexResult + Send + Sync + 'static, + F: Fn(&ArrayRef, &mut ExecutionCtx) -> VortexResult + Send + Sync + 'static, { - fn compress_chunk(&self, chunk: &ArrayRef) -> VortexResult { - self(chunk) + fn compress_chunk(&self, chunk: &ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + self(chunk, ctx) } } impl CompressorPlugin for BtrBlocksCompressor { - fn compress_chunk(&self, chunk: &ArrayRef) -> VortexResult { - self.compress(chunk) + fn compress_chunk(&self, chunk: &ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult { + self.compress(chunk, ctx) } } @@ -54,6 +57,7 @@ impl CompressorPlugin for BtrBlocksCompressor { pub struct CompressingStrategy { child: Arc, compressor: Arc, + stats: Arc<[Stat]>, concurrency: usize, } @@ -63,9 +67,8 @@ impl CompressingStrategy { Self { child: Arc::new(child), compressor: Arc::new(compressor), - concurrency: std::thread::available_parallelism() - .map(|v| v.get()) - .unwrap_or(1), + stats: Stat::all().collect(), + concurrency: get_available_parallelism().unwrap_or(1), } } @@ -73,6 +76,13 @@ impl CompressingStrategy { self.concurrency = concurrency; self } + + /// Override the set of statistics computed on each chunk before compression. + /// Defaults to `Stat::all()`. + pub fn with_stats(mut self, stats: &[Stat]) -> Self { + self.stats = stats.into(); + self + } } #[async_trait] @@ -87,18 +97,22 @@ impl LayoutStrategy for CompressingStrategy { ) -> VortexResult { let dtype = stream.dtype().clone(); let compressor = Arc::clone(&self.compressor); + let stats = Arc::clone(&self.stats); + let session = session.clone(); + let compute_session = session.clone(); let handle = session.handle(); let stream = stream .map(move |chunk| { let compressor = Arc::clone(&compressor); + let stats = Arc::clone(&stats); + let session = compute_session.clone(); handle.spawn_cpu(move || { let (sequence_id, chunk) = chunk?; + let mut ctx = session.create_execution_ctx(); // Compute the stats for the chunk prior to compression - chunk - .statistics() - .compute_all(&Stat::all().collect::>())?; - Ok((sequence_id, compressor.compress_chunk(&chunk)?)) + chunk.statistics().compute_all(&stats, &mut ctx)?; + Ok((sequence_id, compressor.compress_chunk(&chunk, &mut ctx)?)) }) }) .buffered(self.concurrency); @@ -109,7 +123,7 @@ impl LayoutStrategy for CompressingStrategy { segment_sink, SequentialStreamAdapter::new(dtype, stream).sendable(), eof, - session, + &session, ) .await } diff --git a/vortex-layout/src/layouts/dict/mod.rs b/vortex-layout/src/layouts/dict/mod.rs index 866cf6587e9..7928b447fa9 100644 --- a/vortex-layout/src/layouts/dict/mod.rs +++ b/vortex-layout/src/layouts/dict/mod.rs @@ -40,7 +40,7 @@ impl VTable for Dict { type Metadata = ProstMetadata; fn id(_encoding: &Self::Encoding) -> LayoutId { - LayoutId::new_ref("vortex.dict") + LayoutId::new("vortex.dict") } fn encoding(_layout: &Self::Layout) -> LayoutEncodingRef { diff --git a/vortex-layout/src/layouts/dict/reader.rs b/vortex-layout/src/layouts/dict/reader.rs index 486bbab14c2..dd57926ac63 100644 --- a/vortex-layout/src/layouts/dict/reader.rs +++ b/vortex-layout/src/layouts/dict/reader.rs @@ -251,6 +251,10 @@ impl LayoutReader for DictReader { } .boxed()) } + + fn as_any(&self) -> &dyn std::any::Any { + self + } } #[cfg(test)] @@ -259,8 +263,11 @@ mod tests { use rstest::rstest; use vortex_array::ArrayContext; + use vortex_array::Canonical; use vortex_array::IntoArray as _; + use vortex_array::LEGACY_SESSION; use vortex_array::MaskFuture; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; use vortex_array::arrays::StructArray; use vortex_array::arrays::VarBinArray; @@ -270,9 +277,8 @@ mod tests { use vortex_array::dtype::FieldNames; use vortex_array::dtype::Nullability; use vortex_array::expr::eq; - use vortex_array::expr::is_null; + use vortex_array::expr::is_not_null; use vortex_array::expr::lit; - use vortex_array::expr::not; use vortex_array::expr::pack; use vortex_array::expr::root; use vortex_array::scalar_fn::session::ScalarFnSession; @@ -361,7 +367,7 @@ mod tests { )], Nullability::NonNullable, ); - assert!(layout.encoding_id() == LayoutId::new_ref("vortex.dict")); + assert!(layout.encoding_id() == LayoutId::new("vortex.dict")); let actual = layout .new_reader("".into(), segments, &session) .unwrap() @@ -461,6 +467,7 @@ mod tests { #[test] fn reading_is_null_works() { block_on(|handle| async move { + let mut ctx_exec = LEGACY_SESSION.create_execution_ctx(); let session = session_with_handle(handle); let strategy = DictStrategy::new( FlatLayoutStrategy::default(), @@ -504,8 +511,8 @@ mod tests { .await .unwrap(); - let expression = not(is_null(root())); // easier to test not_is_null b/c that's the validity array - assert_eq!(layout.encoding_id(), LayoutId::new_ref("vortex.dict")); + let expression = is_not_null(root()); + assert_eq!(layout.encoding_id(), LayoutId::new("vortex.dict")); let actual = layout .new_reader("".into(), segments, &session) .unwrap() @@ -517,14 +524,17 @@ mod tests { .unwrap() .await .unwrap(); - let expected = array.validity_mask().unwrap().into_array(); - assert_arrays_eq!( - actual - .to_canonical() - .vortex_expect("to_canonical failed") - .into_array(), - expected - ); + let expected = array + .validity() + .unwrap() + .execute_mask(array.len(), &mut ctx_exec) + .unwrap() + .into_array(); + let actual_canonical = actual + .execute::(&mut ctx_exec) + .vortex_expect("to_canonical failed") + .into_array(); + assert_arrays_eq!(actual_canonical, expected); }) } } diff --git a/vortex-layout/src/layouts/dict/writer.rs b/vortex-layout/src/layouts/dict/writer.rs index a1ca9a2b9df..7013becd4df 100644 --- a/vortex-layout/src/layouts/dict/writer.rs +++ b/vortex-layout/src/layouts/dict/writer.rs @@ -20,6 +20,7 @@ use futures::stream::once; use futures::try_join; use vortex_array::ArrayContext; use vortex_array::ArrayRef; +use vortex_array::VortexSessionExecute; use vortex_array::arrays::Dict; use vortex_array::builders::dict::DictConstraints; use vortex_array::builders::dict::DictEncoder; @@ -151,7 +152,8 @@ impl LayoutStrategy for DictStrategy { let should_fallback = match first_chunk { None => true, // empty stream Some(chunk) => { - let compressed = BtrBlocksCompressor::default().compress(&chunk)?; + let mut exec_ctx = session.create_execution_ctx(); + let compressed = BtrBlocksCompressor::default().compress(&chunk, &mut exec_ctx)?; !compressed.is::() } }; diff --git a/vortex-layout/src/layouts/file_stats.rs b/vortex-layout/src/layouts/file_stats.rs index f6b779b431d..0b3205e216f 100644 --- a/vortex-layout/src/layouts/file_stats.rs +++ b/vortex-layout/src/layouts/file_stats.rs @@ -8,9 +8,9 @@ use futures::StreamExt; use itertools::Itertools; use parking_lot::Mutex; use vortex_array::ArrayRef; -use vortex_array::LEGACY_SESSION; -use vortex_array::ToCanonical as _; +use vortex_array::ExecutionCtx; use vortex_array::VortexSessionExecute; +use vortex_array::arrays::StructArray; use vortex_array::arrays::struct_::StructArrayExt; use vortex_array::dtype::DType; use vortex_array::dtype::Nullability; @@ -19,8 +19,9 @@ use vortex_array::stats::StatsSet; use vortex_error::VortexExpect; use vortex_error::VortexResult; use vortex_error::vortex_panic; +use vortex_session::VortexSession; -use crate::layouts::zoned::zone_map::StatsAccumulator; +use crate::layouts::zoned::StatsAccumulator; use crate::sequence::SendableSequentialStream; use crate::sequence::SequenceId; use crate::sequence::SequentialStreamAdapter; @@ -30,9 +31,14 @@ pub fn accumulate_stats( stream: SendableSequentialStream, stats: Arc<[Stat]>, max_variable_length_statistics_size: usize, + session: &VortexSession, ) -> (FileStatsAccumulator, SendableSequentialStream) { - let accumulator = - FileStatsAccumulator::new(stream.dtype(), stats, max_variable_length_statistics_size); + let accumulator = FileStatsAccumulator::new( + stream.dtype(), + stats, + max_variable_length_statistics_size, + session, + ); let stream = SequentialStreamAdapter::new( stream.dtype().clone(), stream.scan(accumulator.clone(), |acc, item| { @@ -50,10 +56,16 @@ pub fn accumulate_stats( pub struct FileStatsAccumulator { stats: Arc<[Stat]>, accumulators: Arc>>, + ctx: Arc>, } impl FileStatsAccumulator { - fn new(dtype: &DType, stats: Arc<[Stat]>, max_variable_length_statistics_size: usize) -> Self { + fn new( + dtype: &DType, + stats: Arc<[Stat]>, + max_variable_length_statistics_size: usize, + session: &VortexSession, + ) -> Self { let accumulators = Arc::new(Mutex::new(match dtype.as_struct_fields_opt() { Some(struct_dtype) => { if dtype.nullability() == Nullability::Nullable { @@ -86,6 +98,7 @@ impl FileStatsAccumulator { Self { stats, accumulators, + ctx: Arc::new(Mutex::new(session.create_execution_ctx())), } } @@ -94,36 +107,31 @@ impl FileStatsAccumulator { chunk: VortexResult<(SequenceId, ArrayRef)>, ) -> VortexResult<(SequenceId, ArrayRef)> { let (sequence_id, chunk) = chunk?; + let mut ctx = self.ctx.lock(); if chunk.dtype().is_struct() { - let chunk = chunk.to_struct(); + let struct_chunk = chunk.clone().execute::(&mut ctx)?; for (acc, field) in self .accumulators .lock() .iter_mut() - .zip_eq(chunk.iter_unmasked_fields()) + .zip_eq(struct_chunk.iter_unmasked_fields()) { - acc.push_chunk(field)?; + acc.push_chunk(field, &mut ctx)?; } } else { - self.accumulators.lock()[0].push_chunk(&chunk)?; + self.accumulators.lock()[0].push_chunk(&chunk, &mut ctx)?; } Ok((sequence_id, chunk)) } pub fn stats_sets(&self) -> Vec { - let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let mut ctx = self.ctx.lock(); self.accumulators .lock() .iter_mut() .map(|acc| { - acc.as_stats_table() + acc.as_stats_set(&self.stats, &mut ctx) .vortex_expect("as_stats_table should not fail") - .map(|table| { - table - .to_stats_set(&self.stats, &mut ctx) - .vortex_expect("shouldn't fail to convert table we just created") - }) - .unwrap_or_default() }) .collect() } diff --git a/vortex-layout/src/layouts/flat/mod.rs b/vortex-layout/src/layouts/flat/mod.rs index 6fa95a5e256..b167e633170 100644 --- a/vortex-layout/src/layouts/flat/mod.rs +++ b/vortex-layout/src/layouts/flat/mod.rs @@ -6,6 +6,7 @@ pub mod writer; use std::env; use std::sync::Arc; +use std::sync::LazyLock; use vortex_array::DeserializeMetadata; use vortex_array::ProstMetadata; @@ -30,9 +31,10 @@ use crate::segments::SegmentSource; use crate::vtable; /// Check if inline array node is enabled. -/// This checks the env var each time to allow tests to toggle the behavior. pub(super) fn flat_layout_inline_array_node() -> bool { - env::var("FLAT_LAYOUT_INLINE_ARRAY_NODE").is_ok() + static FLAT_LAYOUT_INLINE_ARRAY_NODE: LazyLock = + LazyLock::new(|| env::var("FLAT_LAYOUT_INLINE_ARRAY_NODE").is_ok_and(|v| v == "1")); + *FLAT_LAYOUT_INLINE_ARRAY_NODE } vtable!(Flat); @@ -43,7 +45,7 @@ impl VTable for Flat { type Metadata = ProstMetadata; fn id(_encoding: &Self::Encoding) -> LayoutId { - LayoutId::new_ref("vortex.flat") + LayoutId::new("vortex.flat") } fn encoding(_layout: &Self::Layout) -> LayoutEncodingRef { diff --git a/vortex-layout/src/layouts/flat/reader.rs b/vortex-layout/src/layouts/flat/reader.rs index 29627e2b4fd..66f9553c0de 100644 --- a/vortex-layout/src/layouts/flat/reader.rs +++ b/vortex-layout/src/layouts/flat/reader.rs @@ -216,6 +216,10 @@ impl LayoutReader for FlatReader { } .boxed()) } + + fn as_any(&self) -> &dyn std::any::Any { + self + } } #[cfg(test)] diff --git a/vortex-layout/src/layouts/flat/writer.rs b/vortex-layout/src/layouts/flat/writer.rs index 0d804ee6fb8..da250414951 100644 --- a/vortex-layout/src/layouts/flat/writer.rs +++ b/vortex-layout/src/layouts/flat/writer.rs @@ -199,8 +199,9 @@ mod tests { use vortex_array::ArrayContext; use vortex_array::ArrayRef; use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; use vortex_array::MaskFuture; - use vortex_array::ToCanonical; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; use vortex_array::arrays::Dict; use vortex_array::arrays::DictArray; @@ -289,10 +290,11 @@ mod tests { builder.append_value("Long value to test that the statistics are actually truncated, it needs a bit of extra padding though"); builder.append_value("Another string that's meant to be smaller than the previous value, though still need extra padding"); let array = builder.finish(); + let mut stats_ctx = session.create_execution_ctx(); array.statistics().set_iter( array .statistics() - .compute_all(&Stat::all().collect::>()) + .compute_all(&Stat::all().collect::>(), &mut stats_ctx) .vortex_expect("stats computation should succeed for test array") .into_iter(), ); @@ -340,6 +342,7 @@ mod tests { #[test] fn struct_array_round_trip() { block_on(|handle| async { + let mut ctx_exec = LEGACY_SESSION.create_execution_ctx(); let session = SESSION.clone().with_handle(handle); let mut validity_builder = BitBufferMut::with_capacity(2); validity_builder.append(true); @@ -392,28 +395,34 @@ mod tests { .await .unwrap(); - assert_eq!( - result.validity_mask().unwrap().bit_buffer(), - AllOr::Some(&validity_boolean_buffer) - ); assert_eq!( result - .to_struct() - .unmasked_field_by_name("a") + .validity() .unwrap() - .to_primitive() - .as_slice::(), - &[1, 2] - ); - assert_eq!( - result - .to_struct() - .unmasked_field_by_name("b") + .execute_mask(result.len(), &mut ctx_exec) .unwrap() - .to_primitive() - .as_slice::(), - &[3, 4] + .bit_buffer(), + AllOr::Some(&validity_boolean_buffer) ); + let result_struct = result + .clone() + .execute::(&mut ctx_exec) + .unwrap(); + let field_a = result_struct + .unmasked_field_by_name("a") + .unwrap() + .clone() + .execute::(&mut ctx_exec) + .unwrap(); + assert_eq!(field_a.as_slice::(), &[1, 2]); + let result_struct_b = result.execute::(&mut ctx_exec).unwrap(); + let field_b = result_struct_b + .unmasked_field_by_name("b") + .unwrap() + .clone() + .execute::(&mut ctx_exec) + .unwrap(); + assert_eq!(field_b.as_slice::(), &[3, 4]); }) } diff --git a/vortex-layout/src/layouts/foreign/mod.rs b/vortex-layout/src/layouts/foreign/mod.rs index 358e2a42be1..9abacc8b26e 100644 --- a/vortex-layout/src/layouts/foreign/mod.rs +++ b/vortex-layout/src/layouts/foreign/mod.rs @@ -40,7 +40,7 @@ impl LayoutEncoding for ForeignLayoutEncoding { } fn id(&self) -> LayoutEncodingId { - self.id.clone() + self.id } fn build( @@ -57,7 +57,7 @@ impl LayoutEncoding for ForeignLayoutEncoding { .collect::>>()?; Ok(new_foreign_layout( - self.id.clone(), + self.id, dtype.clone(), row_count, metadata.to_vec(), diff --git a/vortex-layout/src/layouts/repartition.rs b/vortex-layout/src/layouts/repartition.rs index 03c7e15d722..c810078e55b 100644 --- a/vortex-layout/src/layouts/repartition.rs +++ b/vortex-layout/src/layouts/repartition.rs @@ -10,7 +10,9 @@ use futures::StreamExt as _; use futures::pin_mut; use vortex_array::ArrayContext; use vortex_array::ArrayRef; +use vortex_array::Canonical; use vortex_array::IntoArray; +use vortex_array::VortexSessionExecute; use vortex_array::arrays::ChunkedArray; use vortex_array::dtype::DType; use vortex_error::VortexExpect; @@ -103,11 +105,14 @@ impl LayoutStrategy for RepartitionStrategy { // canon_stream = stream.map(async {to_canonical}).map(spawn).buffered(parallelism) let dtype = stream.dtype().clone(); let stream = if self.options.canonicalize { + let canonicalize_session = session.clone(); SequentialStreamAdapter::new( dtype.clone(), - stream.map(|chunk| { + stream.map(move |chunk| { let (sequence_id, chunk) = chunk?; - VortexResult::Ok((sequence_id, chunk.to_canonical()?.into_array())) + let mut ctx = canonicalize_session.create_execution_ctx(); + let canonical = chunk.execute::(&mut ctx)?.into_array(); + VortexResult::Ok((sequence_id, canonical)) }), ) .sendable() @@ -123,11 +128,13 @@ impl LayoutStrategy for RepartitionStrategy { // segments. let block_len = options.effective_block_len(&dtype); let block_size_minimum = options.block_size_minimum; + let repartition_session = session.clone(); let repartitioned_stream = try_stream! { let canonical_stream = stream.peekable(); pin_mut!(canonical_stream); + let mut ctx = repartition_session.create_execution_ctx(); let mut chunks = ChunksBuffer::new(block_size_minimum, block_len); while let Some(chunk) = canonical_stream.as_mut().next().await { let (sequence_id, chunk) = chunk?; @@ -145,9 +152,10 @@ impl LayoutStrategy for RepartitionStrategy { let chunked = ChunkedArray::try_new(output_chunks, dtype_clone.clone())?; if !chunked.is_empty() { + let canonical = chunked.into_array().execute::(&mut ctx)?.into_array(); yield ( sequence_pointer.advance(), - chunked.into_array().to_canonical()?.into_array(), + canonical, ) } } @@ -158,9 +166,10 @@ impl LayoutStrategy for RepartitionStrategy { dtype_clone.clone(), )?; if !to_flush.is_empty() { + let canonical = to_flush.into_array().execute::(&mut ctx)?.into_array(); yield ( sequence_pointer.advance(), - to_flush.into_array().to_canonical()?.into_array(), + canonical, ) } } @@ -267,6 +276,7 @@ mod tests { use vortex_array::ArrayContext; use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; use vortex_array::arrays::ConstantArray; use vortex_array::arrays::FixedSizeListArray; use vortex_array::arrays::PrimitiveArray; @@ -479,6 +489,7 @@ mod tests { /// `pop_front` subtracted the larger Cached-era values. #[test] fn chunks_buffer_pop_front_no_panic_after_shared_execution() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let n = 20_000usize; let block_len = 10_000usize; @@ -498,7 +509,8 @@ mod tests { // Transition SharedState from Source to Cached for ALL slices sharing this Arc. use vortex_array::arrays::shared::SharedArrayExt; - shared_handle.get_or_compute(|source| source.to_canonical())?; + let _canonical = + shared_handle.get_or_compute(|source| source.clone().execute::(&mut ctx))?; // Before the fix this panicked with "attempt to subtract with overflow". let _s2 = buf.pop_front().unwrap(); diff --git a/vortex-layout/src/layouts/row_idx/expr.rs b/vortex-layout/src/layouts/row_idx/expr.rs index c235193ac6c..2d85d6e43f5 100644 --- a/vortex-layout/src/layouts/row_idx/expr.rs +++ b/vortex-layout/src/layouts/row_idx/expr.rs @@ -25,7 +25,7 @@ impl ScalarFnVTable for RowIdx { type Options = EmptyOptions; fn id(&self) -> ScalarFnId { - ScalarFnId::from("vortex.row_idx") + ScalarFnId::new("vortex.row_idx") } fn arity(&self, _options: &Self::Options) -> Arity { diff --git a/vortex-layout/src/layouts/row_idx/mod.rs b/vortex-layout/src/layouts/row_idx/mod.rs index a4cabee8e45..33078617722 100644 --- a/vortex-layout/src/layouts/row_idx/mod.rs +++ b/vortex-layout/src/layouts/row_idx/mod.rs @@ -16,6 +16,7 @@ pub use expr::*; use futures::FutureExt; use futures::future::BoxFuture; use vortex_array::ArrayRef; +use vortex_array::Canonical; use vortex_array::IntoArray; use vortex_array::MaskFuture; use vortex_array::VortexSessionExecute; @@ -218,9 +219,13 @@ impl LayoutReader for RowIdxLayoutReader { Partition::Child => self.child.filter_evaluation(row_range, expr, mask), }, |annotation, expr, mask| match annotation { - Partition::RowIdx => { - Ok(row_idx_array_future(self.row_offset, row_range, expr, mask)) - } + Partition::RowIdx => Ok(row_idx_array_future( + self.row_offset, + row_range, + expr, + mask, + self.session.clone(), + )), Partition::Child => self.child.projection_evaluation(row_range, expr, mask), }, self.session.clone(), @@ -235,20 +240,32 @@ impl LayoutReader for RowIdxLayoutReader { mask: MaskFuture, ) -> VortexResult>> { match &self.partition_expr(expr) { - Partitioning::RowIdx(expr) => { - Ok(row_idx_array_future(self.row_offset, row_range, expr, mask)) - } + Partitioning::RowIdx(expr) => Ok(row_idx_array_future( + self.row_offset, + row_range, + expr, + mask, + self.session.clone(), + )), Partitioning::Child(expr) => self.child.projection_evaluation(row_range, expr, mask), Partitioning::Partitioned(p) => { Arc::clone(p).into_array_future(mask, |annotation, expr, mask| match annotation { - Partition::RowIdx => { - Ok(row_idx_array_future(self.row_offset, row_range, expr, mask)) - } + Partition::RowIdx => Ok(row_idx_array_future( + self.row_offset, + row_range, + expr, + mask, + self.session.clone(), + )), Partition::Child => self.child.projection_evaluation(row_range, expr, mask), }) } } } + + fn as_any(&self) -> &dyn std::any::Any { + self + } } // Returns a SequenceArray representing the row indices for the given row range, @@ -288,12 +305,15 @@ fn row_idx_array_future( row_range: &Range, expr: &Expression, mask: MaskFuture, + session: VortexSession, ) -> ArrayFuture { let row_range = row_range.clone(); let expr = expr.clone(); async move { let array = idx_array(row_offset, &row_range).into_array(); - let array = array.filter(mask.await?)?.to_canonical()?.into_array(); + let filtered = array.filter(mask.await?)?; + let mut ctx = session.create_execution_ctx(); + let array = filtered.execute::(&mut ctx)?.into_array(); array.apply(&expr) } .boxed() diff --git a/vortex-layout/src/layouts/struct_/mod.rs b/vortex-layout/src/layouts/struct_/mod.rs index 3924394ff11..29843f096d8 100644 --- a/vortex-layout/src/layouts/struct_/mod.rs +++ b/vortex-layout/src/layouts/struct_/mod.rs @@ -42,7 +42,7 @@ impl VTable for Struct { type Metadata = EmptyMetadata; fn id(_encoding: &Self::Encoding) -> LayoutId { - LayoutId::new_ref("vortex.struct") + LayoutId::new("vortex.struct") } fn encoding(_layout: &Self::Layout) -> LayoutEncodingRef { diff --git a/vortex-layout/src/layouts/struct_/reader.rs b/vortex-layout/src/layouts/struct_/reader.rs index 5d0b2cf0a8d..94cf534ea16 100644 --- a/vortex-layout/src/layouts/struct_/reader.rs +++ b/vortex-layout/src/layouts/struct_/reader.rs @@ -11,7 +11,7 @@ use itertools::Itertools; use vortex_array::ArrayRef; use vortex_array::IntoArray; use vortex_array::MaskFuture; -use vortex_array::ToCanonical; +use vortex_array::VortexSessionExecute; use vortex_array::arrays::StructArray; use vortex_array::arrays::struct_::StructArrayExt; use vortex_array::builtins::ArrayBuiltins; @@ -353,13 +353,15 @@ impl LayoutReader for StructReader { ), }; + let session = self.session.clone(); Ok(Box::pin(async move { if let Some(validity_fut) = validity_fut { let (array, validity) = try_join!(projected, validity_fut)?; // If root expression was a pack, then we apply the validity to each child field if is_pack_merge { - let struct_array = array.to_struct(); + let mut ctx = session.create_execution_ctx(); + let struct_array = array.execute::(&mut ctx)?; let masked_fields: Vec = struct_array .iter_unmasked_fields() .map(|a| a.clone().mask(validity.clone())) @@ -382,6 +384,10 @@ impl LayoutReader for StructReader { } })) } + + fn as_any(&self) -> &dyn std::any::Any { + self + } } #[cfg(test)] @@ -392,8 +398,9 @@ mod tests { use rstest::rstest; use vortex_array::ArrayContext; use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; use vortex_array::MaskFuture; - use vortex_array::ToCanonical; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::BoolArray; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::StructArray; @@ -666,6 +673,7 @@ mod tests { fn test_struct_layout_select( #[from(struct_layout)] (segments, layout): (Arc, LayoutRef), ) { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let reader = layout.new_reader("".into(), segments, &SESSION).unwrap(); let expr = pack( [("a", get_item("a", root())), ("b", get_item("b", root()))], @@ -686,14 +694,16 @@ mod tests { assert_eq!(result.len(), 2); let expected_a = PrimitiveArray::from_iter([7i32, 2]); + let result_struct_a = result.clone().execute::(&mut ctx).unwrap(); assert_arrays_eq!( - result.to_struct().unmasked_field_by_name("a").unwrap(), + result_struct_a.unmasked_field_by_name("a").unwrap(), expected_a ); let expected_b = PrimitiveArray::from_iter([4i32, 5]); + let result_struct_b = result.execute::(&mut ctx).unwrap(); assert_arrays_eq!( - result.to_struct().unmasked_field_by_name("b").unwrap(), + result_struct_b.unmasked_field_by_name("b").unwrap(), expected_b ); } @@ -718,7 +728,9 @@ mod tests { // ...and the result is masked with the validity of the parent StructArray assert_eq!( - result.scalar_at(0).unwrap(), + result + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap(), Scalar::null(result.dtype().clone()), ); assert_nth_scalar!(result, 1, 2); @@ -760,7 +772,7 @@ mod tests { // Row 0: struct is valid, field "c" is 4. assert_eq!( result - .scalar_at(0) + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx()) .unwrap() .as_struct() .field_by_idx(0) @@ -769,12 +781,18 @@ mod tests { ); // Row 1: struct is null (because root.a.b was null at this row). - assert!(result.scalar_at(1).unwrap().as_struct().is_null()); + assert!( + result + .execute_scalar(1, &mut LEGACY_SESSION.create_execution_ctx()) + .unwrap() + .as_struct() + .is_null() + ); // Row 2: struct is valid, field "c" is 6. assert_eq!( result - .scalar_at(2) + .execute_scalar(2, &mut LEGACY_SESSION.create_execution_ctx()) .unwrap() .as_struct() .field_by_idx(0) diff --git a/vortex-layout/src/layouts/table.rs b/vortex-layout/src/layouts/table.rs index f8a0cee3581..28b3af52c87 100644 --- a/vortex-layout/src/layouts/table.rs +++ b/vortex-layout/src/layouts/table.rs @@ -16,7 +16,8 @@ use itertools::Itertools; use vortex_array::ArrayContext; use vortex_array::ArrayRef; use vortex_array::IntoArray; -use vortex_array::ToCanonical; +use vortex_array::VortexSessionExecute; +use vortex_array::arrays::StructArray; use vortex_array::arrays::struct_::StructArrayExt; use vortex_array::dtype::DType; use vortex_array::dtype::Field; @@ -231,15 +232,20 @@ impl LayoutStrategy for TableStrategy { } // stream -> stream> + let columns_session = session.clone(); let columns_vec_stream = stream.map(move |chunk| { let (sequence_id, chunk) = chunk?; let mut sequence_pointer = sequence_id.descend(); - let struct_chunk = chunk.to_struct(); + let mut ctx = columns_session.create_execution_ctx(); + let struct_chunk = chunk.clone().execute::(&mut ctx)?; let mut columns: Vec<(SequenceId, ArrayRef)> = Vec::new(); if is_nullable { columns.push(( sequence_pointer.advance(), - chunk.validity_mask()?.into_array(), + chunk + .validity()? + .execute_mask(chunk.len(), &mut ctx)? + .into_array(), )); } diff --git a/vortex-layout/src/layouts/zoned/builder.rs b/vortex-layout/src/layouts/zoned/builder.rs index c8fd3686281..25a6e89f9db 100644 --- a/vortex-layout/src/layouts/zoned/builder.rs +++ b/vortex-layout/src/layouts/zoned/builder.rs @@ -1,30 +1,171 @@ +//! Write-time accumulation and builders for zoned layout stats tables. + // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors use std::marker::PhantomData; +use std::sync::Arc; +use itertools::Itertools; use vortex_array::ArrayRef; +use vortex_array::ExecutionCtx; use vortex_array::IntoArray; +use vortex_array::LEGACY_SESSION; +use vortex_array::VortexSessionExecute; +use vortex_array::aggregate_fn::fns::sum::sum; use vortex_array::arrays::ConstantArray; +use vortex_array::arrays::StructArray; +use vortex_array::arrays::struct_::StructArrayExt; use vortex_array::builders::ArrayBuilder; use vortex_array::builders::BoolBuilder; use vortex_array::builders::builder_with_capacity; use vortex_array::dtype::DType; use vortex_array::dtype::FieldName; use vortex_array::dtype::Nullability; +use vortex_array::dtype::PType; +use vortex_array::expr::stats::Precision; use vortex_array::expr::stats::Stat; +use vortex_array::expr::stats::StatsProvider; use vortex_array::scalar::Scalar; use vortex_array::scalar::ScalarTruncation; use vortex_array::scalar::lower_bound; use vortex_array::scalar::upper_bound; +use vortex_array::stats::StatsSet; +use vortex_array::validity::Validity; use vortex_buffer::BufferString; use vortex_buffer::ByteBuffer; use vortex_error::VortexResult; -pub const MAX_IS_TRUNCATED: &str = "max_is_truncated"; -pub const MIN_IS_TRUNCATED: &str = "min_is_truncated"; +use crate::layouts::zoned::schema::MAX_IS_TRUNCATED; +use crate::layouts::zoned::schema::MIN_IS_TRUNCATED; + +/// Accumulates write-time statistics for each logical zone. +pub struct StatsAccumulator { + builders: Vec>, + length: usize, +} + +impl StatsAccumulator { + pub fn new(dtype: &DType, stats: &[Stat], max_variable_length_statistics_size: usize) -> Self { + let builders = stats + .iter() + .filter_map(|&stat| { + stat.dtype(dtype).map(|stat_dtype| { + stats_builder_with_capacity( + stat, + &stat_dtype.as_nullable(), + 1024, + max_variable_length_statistics_size, + ) + }) + }) + .collect::>(); + + Self { + builders, + length: 0, + } + } + + pub fn push_chunk_without_compute(&mut self, array: &ArrayRef) -> VortexResult<()> { + for builder in &mut self.builders { + if let Some(Precision::Exact(value)) = array.statistics().get(builder.stat()) { + builder.append_scalar(value.cast(&value.dtype().as_nullable())?)?; + } else { + builder.append_null(); + } + } + self.length += 1; + Ok(()) + } + + pub fn push_chunk(&mut self, array: &ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult<()> { + for builder in &mut self.builders { + if let Some(value) = array.statistics().compute_stat(builder.stat(), ctx)? { + builder.append_scalar(value.cast(&value.dtype().as_nullable())?)?; + } else { + builder.append_null(); + } + } + self.length += 1; + Ok(()) + } + + pub fn as_array(&mut self) -> VortexResult)>> { + let mut names = Vec::new(); + let mut fields = Vec::new(); + let mut stats = Vec::new(); + + for builder in self + .builders + .iter_mut() + // We sort the stats so the DType is deterministic based on which stats are present. + .sorted_unstable_by_key(|builder| builder.stat()) + { + let values = builder.finish(); + + // We drop any all-null stats columns. + if values.all_invalid()? { + continue; + } -pub fn stats_builder_with_capacity( + stats.push(builder.stat()); + names.extend(values.names); + fields.extend(values.arrays); + } + + if names.is_empty() { + return Ok(None); + } + + let array = StructArray::try_new(names.into(), fields, self.length, Validity::NonNullable)?; + Ok(Some((array, stats.into()))) + } + + /// Returns an aggregated stats set for the table. + pub fn as_stats_set( + &mut self, + stats: &[Stat], + ctx: &mut ExecutionCtx, + ) -> VortexResult { + let mut stats_set = StatsSet::default(); + let Some((array, _)) = self.as_array()? else { + return Ok(stats_set); + }; + + for &stat in stats { + let Some(array) = array.unmasked_field_by_name_opt(stat.name()) else { + continue; + }; + + // Different stats need different aggregations + match stat { + // For stats that are associative, we can just compute them over the stat column + Stat::Min | Stat::Max | Stat::Sum => { + if let Some(s) = array.statistics().compute_stat(stat, ctx)? + && let Some(v) = s.into_value() + { + stats_set.set(stat, Precision::exact(v)) + } + } + // These stats sum up + Stat::NullCount | Stat::NaNCount | Stat::UncompressedSizeInBytes => { + if let Some(sum_value) = sum(array, ctx)? + .cast(&DType::Primitive(PType::U64, Nullability::Nullable))? + .into_value() + { + stats_set.set(stat, Precision::exact(sum_value)); + } + } + // We could implement these aggregations in the future, but for now they're unused + Stat::IsConstant | Stat::IsSorted | Stat::IsStrictSorted => {} + } + } + Ok(stats_set) + } +} + +fn stats_builder_with_capacity( stat: Stat, dtype: &DType, capacity: usize, @@ -62,21 +203,20 @@ pub fn stats_builder_with_capacity( } } -/// Arrays with their associated names, reduced version of a StructArray -pub struct NamedArrays { - pub names: Vec, - pub arrays: Vec, +/// Arrays with their associated names, reduced version of a `StructArray`. +struct NamedArrays { + names: Vec, + arrays: Vec, } impl NamedArrays { - pub fn all_invalid(&self) -> VortexResult { - // by convention we assume that the first array is the one we care about for logical validity - self.arrays[0].all_invalid() + fn all_invalid(&self) -> VortexResult { + // By convention the first array is the logical validity signal for the stat column. + self.arrays[0].all_invalid(&mut LEGACY_SESSION.create_execution_ctx()) } } -/// Minimal array builder interface for use by StatsTable for building stats arrays -pub trait StatsArrayBuilder: Send { +trait StatsArrayBuilder: Send { fn stat(&self) -> Stat; fn append_scalar(&mut self, value: Scalar) -> VortexResult<()>; @@ -86,13 +226,13 @@ pub trait StatsArrayBuilder: Send { fn finish(&mut self) -> NamedArrays; } -pub struct StatNameArrayBuilder { +struct StatNameArrayBuilder { stat: Stat, builder: Box, } impl StatNameArrayBuilder { - pub fn new(stat: Stat, builder: Box) -> Self { + fn new(stat: Stat, builder: Box) -> Self { Self { stat, builder } } } @@ -138,7 +278,7 @@ struct TruncatedMaxBinaryStatsBuilder { } impl TruncatedMaxBinaryStatsBuilder { - pub fn new( + fn new( values: Box, is_truncated: BoolBuilder, max_value_length: usize, @@ -160,7 +300,7 @@ struct TruncatedMinBinaryStatsBuilder { } impl TruncatedMinBinaryStatsBuilder { - pub fn new( + fn new( values: Box, is_truncated: BoolBuilder, max_value_length: usize, @@ -192,7 +332,6 @@ impl StatsArrayBuilder for TruncatedMaxBinaryStatsBuilder StatsArrayBuilder for TruncatedMinBinaryStatsBuilder StatsArrayBuilder for TruncatedMinBinaryStatsBuilder(&mut ctx) + .unwrap(); + assert_eq!( + field1_bool.to_bit_buffer(), + BitBuffer::from(vec![false, true]) + ); + let field3_bool = stats_table + .unmasked_field(3) + .clone() + .execute::(&mut ctx) + .unwrap(); + assert_eq!( + field3_bool.to_bit_buffer(), + BitBuffer::from(vec![true, false]) + ); + } + + #[test] + fn always_adds_is_truncated_column() { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let array = buffer![0, 1, 2].into_array(); + let mut acc = StatsAccumulator::new(array.dtype(), &[Stat::Max, Stat::Min, Stat::Sum], 12); + acc.push_chunk(&array, &mut ctx) + .vortex_expect("push_chunk should succeed for test array"); + let (stats_table, _) = acc.as_array().unwrap().expect("Must have stats table"); + assert_eq!( + stats_table.names().as_ref(), + &[ + Stat::Max.name(), + MAX_IS_TRUNCATED, + Stat::Min.name(), + MIN_IS_TRUNCATED, + Stat::Sum.name(), + ] + ); + let field1_bool = stats_table + .unmasked_field(1) + .clone() + .execute::(&mut ctx) + .unwrap(); + assert_eq!(field1_bool.to_bit_buffer(), BitBuffer::from(vec![false])); + let field3_bool = stats_table + .unmasked_field(3) + .clone() + .execute::(&mut ctx) + .unwrap(); + assert_eq!(field3_bool.to_bit_buffer(), BitBuffer::from(vec![false])); + } +} diff --git a/vortex-layout/src/layouts/zoned/mod.rs b/vortex-layout/src/layouts/zoned/mod.rs index 5167025ff3a..697e55e968c 100644 --- a/vortex-layout/src/layouts/zoned/mod.rs +++ b/vortex-layout/src/layouts/zoned/mod.rs @@ -1,15 +1,28 @@ +//! Zoned layouts wrap a data layout with an auxiliary per-zone statistics layout. +//! +//! The zoned layout tree has exactly two children: +//! - a transparent `data` child containing the underlying column data +//! - an auxiliary `zones` child containing one row of aggregate statistics per zone +//! +//! Metadata stores the logical zone length in rows plus the sorted list of statistics present in +//! the auxiliary table. During scans, pruning first evaluates a falsification predicate against +//! the `zones` child and only forwards surviving rows to the underlying `data` child. + // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors mod builder; +mod pruning; mod reader; +mod schema; pub mod writer; pub mod zone_map; use std::sync::Arc; -pub use builder::MAX_IS_TRUNCATED; -pub use builder::MIN_IS_TRUNCATED; +pub(crate) use builder::StatsAccumulator; +pub use schema::MAX_IS_TRUNCATED; +pub use schema::MIN_IS_TRUNCATED; use vortex_array::DeserializeMetadata; use vortex_array::SerializeMetadata; use vortex_array::dtype::DType; @@ -20,6 +33,8 @@ use vortex_array::stats::stats_from_bitset_bytes; use vortex_error::VortexExpect; use vortex_error::VortexResult; use vortex_error::vortex_bail; +use vortex_error::vortex_ensure; +use vortex_error::vortex_ensure_eq; use vortex_error::vortex_panic; use vortex_session::VortexSession; use vortex_session::registry::ReadContext; @@ -33,7 +48,7 @@ use crate::VTable; use crate::children::LayoutChildren; use crate::children::OwnedLayoutChildren; use crate::layouts::zoned::reader::ZonedReader; -use crate::layouts::zoned::zone_map::ZoneMap; +use crate::layouts::zoned::schema::stats_table_dtype; use crate::segments::SegmentId; use crate::segments::SegmentSource; use crate::vtable; @@ -46,7 +61,8 @@ impl VTable for Zoned { type Metadata = ZonedMetadata; fn id(_encoding: &Self::Encoding) -> LayoutId { - LayoutId::new_ref("vortex.stats") // For legacy reasons, this is called stats + // For legacy reasons the serialized layout encoding ID is still `vortex.stats`. + LayoutId::new("vortex.stats") } fn encoding(_layout: &Self::Layout) -> LayoutEncodingRef { @@ -79,10 +95,9 @@ impl VTable for Zoned { fn child(layout: &Self::Layout, idx: usize) -> VortexResult { match idx { 0 => layout.children.child(0, layout.dtype()), - 1 => layout.children.child( - 1, - &ZoneMap::dtype_for_stats_table(layout.dtype(), &layout.present_stats), - ), + 1 => layout + .children + .child(1, &stats_table_dtype(layout.dtype(), &layout.present_stats)), _ => vortex_bail!("Invalid child index: {}", idx), } } @@ -118,6 +133,11 @@ impl VTable for Zoned { children: &dyn LayoutChildren, _ctx: &ReadContext, ) -> VortexResult { + vortex_ensure_eq!( + children.nchildren(), + 2, + "ZonedLayout expects exactly 2 children (data, zones)" + ); Ok(ZonedLayout { dtype: dtype.clone(), children: children.to_arc(), @@ -138,12 +158,15 @@ impl VTable for Zoned { } } +/// Encoding marker for the zoned layout. #[derive(Debug)] pub struct ZonedLayoutEncoding; -/// Annotates a data layout with per-zone aggregate statistics (e.g. min, max, null count). +/// A layout that annotates a data child with one row of aggregate statistics per zone. /// -/// During reads, zone maps allow entire zones to be skipped when a filter predicate cannot match. +/// The first child is the underlying data layout. The second child is an auxiliary stats table +/// whose rows align with logical row zones of length `zone_len`, except for the final partial zone. +/// During reads, pruning uses the stats table to skip zones whose rows cannot satisfy a filter. #[derive(Clone, Debug)] pub struct ZonedLayout { dtype: DType, @@ -162,7 +185,7 @@ impl ZonedLayout { if zone_len == 0 { vortex_panic!("Zone length must be greater than 0"); } - let expected_dtype = ZoneMap::dtype_for_stats_table(data.dtype(), &present_stats); + let expected_dtype = stats_table_dtype(data.dtype(), &present_stats); if zones.dtype() != &expected_dtype { vortex_panic!("Invalid zone map layout: zones dtype does not match expected dtype"); } @@ -178,12 +201,20 @@ impl ZonedLayout { usize::try_from(self.children.child_row_count(1)).vortex_expect("Invalid number of zones") } + pub fn zone_len(&self) -> usize { + self.zone_len + } + /// Returns an array of stats that exist in the layout's data, must be sorted. pub fn present_stats(&self) -> &Arc<[Stat]> { &self.present_stats } } +/// Serialized zoned-layout metadata. +/// +/// `zone_len` is the logical row length of each zone. `present_stats` is the sorted list of +/// statistics stored in the auxiliary stats-table child. #[derive(Debug, PartialEq, Eq, Clone)] pub struct ZonedMetadata { pub(super) zone_len: u32, @@ -194,8 +225,18 @@ impl DeserializeMetadata for ZonedMetadata { type Output = Self; fn deserialize(metadata: &[u8]) -> VortexResult { + vortex_ensure!( + metadata.len() >= 4, + "Zoned metadata must contain at least 4 bytes for zone length, got {}", + metadata.len() + ); + + // Backward compat: older files may encode `zone_len == 0`. Preserve the raw metadata on + // read and let the reader disable zoned pruning for those layouts instead of rejecting + // deserialization outright. let zone_len = u32::try_from_le_bytes(&metadata[0..4])?; let present_stats: Arc<[Stat]> = stats_from_bitset_bytes(&metadata[4..]).into(); + Ok(Self { zone_len, present_stats, @@ -216,19 +257,25 @@ impl SerializeMetadata for ZonedMetadata { #[cfg(test)] mod tests { + use std::panic; + use rstest::rstest; + use vortex_array::dtype::DType; + use vortex_array::dtype::Nullability; + use vortex_array::dtype::PType; + use vortex_session::registry::ReadContext; use super::*; + use crate::IntoLayout; + use crate::children::OwnedLayoutChildren; + use crate::layouts::flat::FlatLayout; + use crate::segments::SegmentId; #[rstest] #[case(ZonedMetadata { zone_len: u32::MAX, present_stats: Arc::new([]), })] - #[case(ZonedMetadata { - zone_len: 0, - present_stats: Arc::new([Stat::IsConstant]), - })] #[case::all_sorted(ZonedMetadata { zone_len: 314, present_stats: Arc::new([Stat::IsConstant, Stat::IsSorted, Stat::IsStrictSorted, Stat::Max, Stat::Min, Stat::Sum, Stat::NullCount, Stat::UncompressedSizeInBytes, Stat::NaNCount]), @@ -258,4 +305,84 @@ mod tests { ); assert_ne!(deserialized.present_stats, metadata.present_stats); } + + #[rstest] + #[case(vec![])] + #[case(vec![0])] + #[case(vec![0, 0])] + #[case(vec![0, 0, 0])] + fn test_deserialize_short_metadata_errors(#[case] metadata: Vec) { + assert!(ZonedMetadata::deserialize(&metadata).is_err()); + } + + #[test] + fn test_deserialize_short_metadata_returns_error_not_panic() { + let result = panic::catch_unwind(|| ZonedMetadata::deserialize(&[])); + assert!( + result.is_ok(), + "deserialize should return an error, not panic" + ); + assert!(result.unwrap().is_err()); + } + + #[test] + fn test_deserialize_zero_zone_len_is_allowed_for_backcompat() { + let metadata = 0u32.to_le_bytes(); + let deserialized = ZonedMetadata::deserialize(&metadata).unwrap(); + assert_eq!(deserialized.zone_len, 0); + assert!(deserialized.present_stats.is_empty()); + } + + #[test] + fn test_build_allows_zero_zone_len_for_backcompat() -> VortexResult<()> { + let dtype = DType::Primitive(PType::I32, Nullability::NonNullable); + let read_ctx = ReadContext::new([]); + let children = OwnedLayoutChildren::layout_children(vec![ + FlatLayout::new(0, dtype.clone(), SegmentId::from(0), read_ctx.clone()).into_layout(), + FlatLayout::new( + 0, + stats_table_dtype(&dtype, &[]), + SegmentId::from(1), + read_ctx, + ) + .into_layout(), + ]); + + let layout = ::build( + &ZonedLayoutEncoding, + &dtype, + 0, + &ZonedMetadata { + zone_len: 0, + present_stats: Arc::new([]), + }, + vec![], + children.as_ref(), + &ReadContext::new([]), + )?; + + assert_eq!(layout.zone_len, 0); + Ok(()) + } + + #[test] + fn test_build_rejects_invalid_child_count() { + let metadata = ZonedMetadata { + zone_len: 3, + present_stats: Arc::new([]), + }; + let children = OwnedLayoutChildren::layout_children(vec![]); + + let result = ::build( + &ZonedLayoutEncoding, + &DType::Primitive(PType::I32, Nullability::NonNullable), + 0, + &metadata, + vec![], + children.as_ref(), + &ReadContext::new([]), + ); + + assert!(result.is_err()); + } } diff --git a/vortex-layout/src/layouts/zoned/pruning.rs b/vortex-layout/src/layouts/zoned/pruning.rs new file mode 100644 index 00000000000..99526363813 --- /dev/null +++ b/vortex-layout/src/layouts/zoned/pruning.rs @@ -0,0 +1,213 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Read-time pruning support for zoned layouts. + +use std::sync::Arc; +use std::sync::LazyLock; +use std::sync::OnceLock; + +use futures::FutureExt; +use futures::TryFutureExt; +use futures::future::BoxFuture; +use futures::future::Shared; +use parking_lot::RwLock; +use vortex_array::MaskFuture; +use vortex_array::VortexSessionExecute; +use vortex_array::arrays::StructArray; +use vortex_array::dtype::FieldPath; +use vortex_array::dtype::FieldPathSet; +use vortex_array::expr::Expression; +use vortex_array::expr::pruning::checked_pruning_expr; +use vortex_array::expr::root; +use vortex_array::expr::stats::Stat; +use vortex_array::scalar_fn::fns::dynamic::DynamicExprUpdates; +use vortex_error::SharedVortexResult; +use vortex_error::VortexExpect; +use vortex_error::VortexResult; +use vortex_mask::Mask; +use vortex_session::VortexSession; +use vortex_utils::aliases::dash_map::DashMap; + +use crate::LazyReaderChildren; +use crate::layouts::zoned::ZonedLayout; +use crate::layouts::zoned::zone_map::ZoneMap; + +type SharedZoneMap = Shared>>; +pub(super) type SharedPruningResult = + Shared>>>; +type PredicateCache = Arc>>; + +pub(super) struct PruningState { + zone_count: usize, + row_count: u64, + zone_len: u64, + present_stats: Arc<[Stat]>, + lazy_children: Arc, + session: VortexSession, + + pruning_result: LazyLock>>, + zone_map: OnceLock, + pruning_predicates: LazyLock>>, +} + +impl PruningState { + pub(super) fn new( + layout: &ZonedLayout, + lazy_children: Arc, + session: VortexSession, + ) -> Self { + Self { + zone_count: layout.nzones(), + row_count: layout.row_count(), + zone_len: layout.zone_len() as u64, + present_stats: Arc::clone(layout.present_stats()), + lazy_children, + session, + pruning_result: Default::default(), + zone_map: Default::default(), + pruning_predicates: Default::default(), + } + } + + pub(super) fn pruning_mask_future(&self, expr: Expression) -> Option { + if let Some(result) = self.pruning_result.get(&expr) { + return result.value().clone(); + } + + self.pruning_result + .entry(expr.clone()) + .or_insert_with(|| match self.pruning_predicate(expr.clone()) { + None => { + tracing::debug!("No pruning predicate for expr: {expr}"); + None + } + Some(predicate) => { + tracing::debug!( + "Constructed pruning predicate for expr: {expr}: {predicate:?}" + ); + let zone_map = self.zone_map(); + let dynamic_updates = DynamicExprUpdates::new(&expr); + let session = self.session.clone(); + + Some( + async move { + let zone_map = zone_map.await?; + let initial_mask = + zone_map.prune(&predicate, &session).map_err(|err| { + err.with_context(format!( + "While evaluating pruning predicate {} (derived from {})", + predicate, expr + )) + })?; + Ok(Arc::new(PruningResult { + zone_map, + predicate, + dynamic_updates, + latest_result: RwLock::new((0, initial_mask)), + session, + })) + } + .boxed() + .shared(), + ) + } + }) + .clone() + } + + fn pruning_predicate(&self, expr: Expression) -> Option { + self.pruning_predicates + .entry(expr.clone()) + .or_default() + .get_or_init(move || { + let available_stats = FieldPathSet::from_iter( + self.present_stats + .iter() + .map(|stat| FieldPath::from_name(stat.name())), + ); + checked_pruning_expr(&expr, &available_stats).map(|(expr, _)| expr) + }) + .clone() + } + + fn zone_map(&self) -> SharedZoneMap { + self.zone_map + .get_or_init(move || { + let zone_count = self.zone_count; + let zones_eval = self + .lazy_children + .get(1) + .vortex_expect("failed to get zone child") + .projection_evaluation( + &(0..zone_count as u64), + &root(), + MaskFuture::new_true(zone_count), + ) + .vortex_expect("Failed construct zone map evaluation"); + let session = self.session.clone(); + let zone_len = self.zone_len; + let row_count = self.row_count; + + async move { + let mut ctx = session.create_execution_ctx(); + let zones_array = zones_eval.await?.execute::(&mut ctx)?; + // SAFETY: zoned layout validation ensures the zones child matches the expected + // stats-table schema for `present_stats`. + Ok(unsafe { ZoneMap::new_unchecked(zones_array, zone_len, row_count) }) + } + .map_err(Arc::new) + .boxed() + .shared() + }) + .clone() + } +} + +pub(super) struct PruningResult { + zone_map: ZoneMap, + predicate: Expression, + dynamic_updates: Option, + latest_result: RwLock<(u64, Mask)>, + session: VortexSession, +} + +impl PruningResult { + pub(super) fn mask(&self) -> VortexResult { + let Some(dynamic_updates) = &self.dynamic_updates else { + return Ok(self.latest_result.read().1.clone()); + }; + + let version = dynamic_updates.version(); + + { + let read_guard = self.latest_result.read(); + if read_guard.0 >= version { + return Ok(read_guard.1.clone()); + } + } + + let mut guard = self.latest_result.write(); + if guard.0 >= version { + return Ok(guard.1.clone()); + } + + tracing::debug!( + "Re-computing pruning mask for version {version} on {}", + self.predicate + ); + + let next_mask = self + .zone_map + .prune(&self.predicate, &self.session) + .map_err(|err| { + err.with_context(format!( + "While evaluating pruning predicate {}", + self.predicate + )) + })?; + *guard = (version, next_mask.clone()); + + Ok(next_mask) + } +} diff --git a/vortex-layout/src/layouts/zoned/reader.rs b/vortex-layout/src/layouts/zoned/reader.rs index 23118e8ce5c..e3dc2bf2e00 100644 --- a/vortex-layout/src/layouts/zoned/reader.rs +++ b/vortex-layout/src/layouts/zoned/reader.rs @@ -5,61 +5,33 @@ use std::collections::BTreeSet; use std::ops::BitAnd; use std::ops::Range; use std::sync::Arc; -use std::sync::LazyLock; -use std::sync::OnceLock; -use futures::FutureExt; -use futures::TryFutureExt; use futures::future::BoxFuture; -use futures::future::Shared; use itertools::Itertools; -use parking_lot::RwLock; use vortex_array::ArrayRef; use vortex_array::MaskFuture; -use vortex_array::ToCanonical; use vortex_array::dtype::DType; use vortex_array::dtype::FieldMask; -use vortex_array::dtype::FieldPath; -use vortex_array::dtype::FieldPathSet; use vortex_array::expr::Expression; -use vortex_array::expr::pruning::checked_pruning_expr; -use vortex_array::expr::root; -use vortex_array::scalar_fn::fns::dynamic::DynamicExprUpdates; use vortex_buffer::BitBufferMut; -use vortex_error::SharedVortexResult; use vortex_error::VortexError; -use vortex_error::VortexExpect; use vortex_error::VortexResult; use vortex_mask::Mask; use vortex_session::VortexSession; -use vortex_utils::aliases::dash_map::DashMap; use crate::LayoutReader; use crate::LayoutReaderRef; use crate::LazyReaderChildren; use crate::layouts::zoned::ZonedLayout; -use crate::layouts::zoned::zone_map::ZoneMap; +use crate::layouts::zoned::pruning::PruningState; +use crate::layouts::zoned::schema::stats_table_dtype; use crate::segments::SegmentSource; -type SharedZoneMap = Shared>>; -type SharedPruningResult = Shared>>>; -type PredicateCache = Arc>>; - pub struct ZonedReader { layout: ZonedLayout, name: Arc, - lazy_children: LazyReaderChildren, - session: VortexSession, - - /// A cache of expr -> optional pruning result (applying the pruning expr to the zone map) - pruning_result: LazyLock>>, - - /// Shared zone map - zone_map: OnceLock, - - /// A cache of expr -> optional pruning predicate. - /// This also uses the present_stats from the `ZonedLayout` - pruning_predicates: LazyLock>>, + lazy_children: Arc, + pruning: PruningState, } impl ZonedReader { @@ -71,25 +43,22 @@ impl ZonedReader { ) -> VortexResult { let dtypes = vec![ layout.dtype.clone(), - ZoneMap::dtype_for_stats_table(layout.dtype(), layout.present_stats()), + stats_table_dtype(layout.dtype(), layout.present_stats()), ]; let names = vec![Arc::clone(&name), format!("{}.zones", name).into()]; - let lazy_children = LazyReaderChildren::new( + let lazy_children = Arc::new(LazyReaderChildren::new( Arc::clone(&layout.children), dtypes, names, Arc::clone(&segment_source), session.clone(), - ); + )); Ok(Self { + pruning: PruningState::new(&layout, Arc::clone(&lazy_children), session), layout, name, lazy_children, - session, - pruning_result: Default::default(), - zone_map: Default::default(), - pruning_predicates: Default::default(), }) } @@ -97,108 +66,12 @@ impl ZonedReader { self.lazy_children.get(0) } - /// Get or create the pruning predicate for a given expression. - fn pruning_predicate(&self, expr: Expression) -> Option { - self.pruning_predicates - .entry(expr.clone()) - .or_default() - .get_or_init(move || { - let field_path_set = FieldPathSet::from_iter( - self.layout - .present_stats - .iter() - .map(|s| FieldPath::from_name(s.name())), - ); - checked_pruning_expr(&expr, &field_path_set).map(|(expr, _)| expr) - }) - .clone() - } - - /// Get or initialize the zone map. - /// - /// Only the first successful caller will initialize the zone map, all other callers will - /// resolve to the same result. - fn zone_map(&self) -> SharedZoneMap { - self.zone_map - .get_or_init(move || { - let nzones = self.layout.nzones(); - let present_stats = Arc::clone(&self.layout.present_stats); - - let zones_eval = self - .lazy_children - .get(1) - .vortex_expect("failed to get zone child") - .projection_evaluation( - &(0..nzones as u64), - &root(), - MaskFuture::new_true(nzones), - ) - .vortex_expect("Failed construct zone map evaluation"); - - async move { - let zones_array = zones_eval.await?.to_struct(); - // SAFETY: This is only fine to call because we perform validation above - Ok(unsafe { ZoneMap::new_unchecked(zones_array, present_stats) }) - } - .map_err(Arc::new) - .boxed() - .shared() - }) - .clone() - } - - /// Returns a pruning mask where `true` means the chunk _can be pruned_. - fn pruning_mask_future(&self, expr: Expression) -> Option { - // Check cache first with read-only lock - if let Some(result) = self.pruning_result.get(&expr) { - return result.value().clone(); - } - - self.pruning_result - .entry(expr.clone()) - .or_insert_with(|| match self.pruning_predicate(expr.clone()) { - None => { - tracing::debug!("No pruning predicate for expr: {expr}"); - None - } - Some(predicate) => { - tracing::debug!( - "Constructed pruning predicate for expr: {expr}: {predicate:?}" - ); - let zone_map = self.zone_map(); - let dynamic_updates = DynamicExprUpdates::new(&expr); - let session = self.session.clone(); - - Some( - async move { - let zone_map = zone_map.await?; - let initial_mask = - zone_map.prune(&predicate, &session).map_err(|err| { - err.with_context(format!( - "While evaluating pruning predicate {} (derived from {})", - predicate, expr - )) - })?; - Ok(Arc::new(PruningResult { - zone_map, - predicate, - dynamic_updates, - latest_result: RwLock::new((0, initial_mask)), - session, - })) - } - .boxed() - .shared(), - ) - } - }) - .clone() - } - /// Get the range of zone IDs containing a row range. pub(crate) fn zone_range(&self, row_range: &Range) -> Range { - // Zone length is guaranteed to be > 0 by ZonedLayout::new validation + // Caller must ensure zone_len > 0. Legacy files may deserialize with zone_len == 0, but + // pruning_evaluation disables zoned pruning for those layouts before calling this helper. debug_assert!(self.layout.zone_len > 0, "zone_len must be > 0"); + let zone_len_u64 = self.layout.zone_len as u64; let zone_start = row_range.start / zone_len_u64; let zone_end = row_range.end.div_ceil(zone_len_u64); @@ -218,6 +91,10 @@ impl LayoutReader for ZonedReader { &self.name } + fn as_any(&self) -> &dyn std::any::Any { + self + } + fn dtype(&self) -> &DType { self.layout.dtype() } @@ -247,7 +124,14 @@ impl LayoutReader for ZonedReader { .data_child()? .pruning_evaluation(row_range, expr, mask.clone())?; - let Some(pruning_mask_future) = self.pruning_mask_future(expr.clone()) else { + if self.layout.zone_len == 0 { + tracing::debug!( + "Stats pruning evaluation: skipping zoned pruning for legacy zero-length zones" + ); + return Ok(data_eval); + } + + let Some(pruning_mask_future) = self.pruning.pruning_mask_future(expr.clone()) else { tracing::debug!("Stats pruning evaluation: not prune-able {expr}"); return Ok(data_eval); }; @@ -330,67 +214,6 @@ impl LayoutReader for ZonedReader { } } -/// A wrapper for the result of pruning an expression against a zone map such that we can refresh -/// it each time the dynamic expressions are updated. -struct PruningResult { - zone_map: ZoneMap, - predicate: Expression, - dynamic_updates: Option, - latest_result: RwLock<(u64, Mask)>, - session: VortexSession, -} - -impl PruningResult { - /// Return the pruning mask, computed for _at least_ the given version. - /// - /// The version typically comes from the dynamic expression updates, but zero can be passed - /// to fetch any version. - fn mask(&self) -> VortexResult { - // If we're not dynamic, then the result is always the latest result. - let Some(dynamic_updates) = &self.dynamic_updates else { - return Ok(self.latest_result.read().1.clone()); - }; - - // Compute the latest version of the dynamic expression values. - let version = dynamic_updates.version(); - - { - let read_guard = self.latest_result.read(); - if read_guard.0 >= version { - // We're up to date, so we can return the cached result. - return Ok(read_guard.1.clone()); - } - } - - // Otherwise, we re-compute the mask for the given version number. - let mut guard = self.latest_result.write(); - - // Once we've taken the write lock, we check again in case another thread has already - // beaten us to it. - if guard.0 >= version { - return Ok(guard.1.clone()); - } - - tracing::debug!( - "Re-computing pruning mask for version {version} on {}", - self.predicate - ); - - let next_mask = self - .zone_map - .prune(&self.predicate, &self.session) - .map_err(|err| { - err.with_context(format!( - "While evaluating pruning predicate {}", - self.predicate - )) - })?; - *guard = (version, next_mask.clone()); - - Ok(next_mask) - } -} - #[cfg(test)] mod test { use std::sync::Arc; @@ -405,22 +228,44 @@ mod test { use vortex_array::expr::gt; use vortex_array::expr::lit; use vortex_array::expr::root; + use vortex_array::scalar_fn::session::ScalarFnSession; + use vortex_array::session::ArraySession; use vortex_buffer::buffer; + use vortex_error::VortexResult; + use vortex_io::runtime::Handle; use vortex_io::runtime::single::block_on; + use vortex_io::session::RuntimeSession; use vortex_io::session::RuntimeSessionExt; use vortex_mask::Mask; + use vortex_session::VortexSession; + use vortex_session::registry::ReadContext; + use crate::IntoLayout; use crate::LayoutRef; use crate::LayoutStrategy; + use crate::VTable; + use crate::children::OwnedLayoutChildren; use crate::layouts::chunked::writer::ChunkedLayoutStrategy; use crate::layouts::flat::writer::FlatLayoutStrategy; + use crate::layouts::zoned::Zoned; + use crate::layouts::zoned::ZonedLayoutEncoding; + use crate::layouts::zoned::ZonedMetadata; use crate::layouts::zoned::writer::ZonedLayoutOptions; use crate::layouts::zoned::writer::ZonedStrategy; use crate::segments::SegmentSource; use crate::segments::TestSegments; use crate::sequence::SequenceId; use crate::sequence::SequentialArrayStreamExt; - use crate::test::SESSION; + use crate::session::LayoutSession; + + fn session_with_handle(handle: Handle) -> VortexSession { + VortexSession::empty() + .with::() + .with::() + .with::() + .with::() + .with_handle(handle) + } #[fixture] /// Create a stats layout with three chunks of primitive arrays. @@ -446,7 +291,7 @@ mod test { .sequenced(ptr); let segments2 = Arc::::clone(&segments); let layout = block_on(|handle| async move { - let session = SESSION.clone().with_handle(handle); + let session = session_with_handle(handle); strategy .write_stream(ctx, segments2, array_stream, eof, &session) .await @@ -459,9 +304,10 @@ mod test { fn test_stats_evaluator( #[from(stats_layout)] (segments, layout): (Arc, LayoutRef), ) { - block_on(|_| async { + block_on(|handle| async { + let session = session_with_handle(handle); let result = layout - .new_reader("".into(), segments, &SESSION) + .new_reader("".into(), segments, &session) .unwrap() .projection_evaluation( &(0..layout.row_count()), @@ -481,9 +327,10 @@ mod test { fn test_stats_pruning_mask( #[from(stats_layout)] (segments, layout): (Arc, LayoutRef), ) { - block_on(|_| async { + block_on(|handle| async { let row_count = layout.row_count(); - let reader = layout.new_reader("".into(), segments, &SESSION).unwrap(); + let session = session_with_handle(handle); + let reader = layout.new_reader("".into(), segments, &session).unwrap(); // Choose a prune-able expression let expr = gt(root(), lit(7)); @@ -504,4 +351,72 @@ mod test { ); }) } + + #[rstest] + fn test_legacy_zero_zone_len_skips_zoned_pruning( + #[from(stats_layout)] (segments, layout): (Arc, LayoutRef), + ) -> VortexResult<()> { + let zoned_layout = layout.as_::(); + let children = + OwnedLayoutChildren::layout_children(vec![layout.child(0)?, layout.child(1)?]); + let legacy_layout = ::build( + &ZonedLayoutEncoding, + layout.dtype(), + layout.row_count(), + &ZonedMetadata { + zone_len: 0, + present_stats: Arc::clone(zoned_layout.present_stats()), + }, + vec![], + children.as_ref(), + &ReadContext::new([]), + )? + .into_layout(); + + block_on(|handle| async { + let row_count = legacy_layout.row_count(); + let session = session_with_handle(handle); + let reader = legacy_layout.new_reader("".into(), segments, &session)?; + + let result = reader + .pruning_evaluation( + &(0..row_count), + >(root(), lit(7)), + Mask::new_true(row_count.try_into().unwrap()), + )? + .await?; + + assert!(result.all_true()); + Ok(()) + }) + } + + #[test] + fn test_writer_rejects_zero_block_size() { + let ctx = ArrayContext::empty(); + let segments = Arc::new(TestSegments::default()); + let (ptr, eof) = SequenceId::root().split(); + let strategy = ZonedStrategy::new( + ChunkedLayoutStrategy::new(FlatLayoutStrategy::default()), + FlatLayoutStrategy::default(), + ZonedLayoutOptions { + block_size: 0, + ..Default::default() + }, + ); + let array_stream = ChunkedArray::from_iter([buffer![1, 2, 3].into_array()]) + .into_array() + .to_array_stream() + .sequenced(ptr); + let segments2 = Arc::::clone(&segments); + + let result = block_on(|handle| async move { + let session = session_with_handle(handle); + strategy + .write_stream(ctx, segments2, array_stream, eof, &session) + .await + }); + + assert!(result.is_err()); + } } diff --git a/vortex-layout/src/layouts/zoned/schema.rs b/vortex-layout/src/layouts/zoned/schema.rs new file mode 100644 index 00000000000..e14388ba379 --- /dev/null +++ b/vortex-layout/src/layouts/zoned/schema.rs @@ -0,0 +1,94 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Shared helpers for the zoned layout's auxiliary stats-table schema. + +use vortex_array::dtype::DType; +use vortex_array::dtype::Nullability; +use vortex_array::dtype::StructFields; +use vortex_array::expr::stats::Stat; + +pub const MAX_IS_TRUNCATED: &str = "max_is_truncated"; +pub const MIN_IS_TRUNCATED: &str = "min_is_truncated"; + +/// Return the auxiliary stats-table schema for a zoned layout. +pub(crate) fn stats_table_dtype(column_dtype: &DType, present_stats: &[Stat]) -> DType { + assert!(present_stats.is_sorted(), "Stats must be sorted"); + DType::Struct( + StructFields::from_iter( + present_stats + .iter() + .filter_map(|stat| { + stat.dtype(column_dtype) + .or_else(|| { + // Backward compat: older files may have stored stats (e.g. Sum) + // for extension types by resolving through the storage dtype. + if let DType::Extension(ext) = column_dtype { + stat.dtype(ext.storage_dtype()) + } else { + None + } + }) + .map(|dtype| (stat, dtype.as_nullable())) + }) + .flat_map(|(stat, dtype)| match stat { + Stat::Max => vec![ + (stat.name(), dtype), + (MAX_IS_TRUNCATED, DType::Bool(Nullability::NonNullable)), + ], + Stat::Min => vec![ + (stat.name(), dtype), + (MIN_IS_TRUNCATED, DType::Bool(Nullability::NonNullable)), + ], + _ => vec![(stat.name(), dtype)], + }), + ), + Nullability::NonNullable, + ) +} + +#[cfg(test)] +mod tests { + use vortex_array::dtype::DType; + use vortex_array::dtype::Nullability; + use vortex_array::dtype::PType; + use vortex_array::extension::datetime::Date; + use vortex_array::extension::datetime::TimeUnit; + + use super::*; + + #[test] + fn stats_table_dtype_adds_truncation_flags() { + let dtype = stats_table_dtype( + &DType::Primitive(PType::I32, Nullability::NonNullable), + &[Stat::Max, Stat::Min, Stat::Sum], + ); + + assert_eq!( + dtype.as_struct_fields().names().as_ref(), + &[ + Stat::Max.name(), + MAX_IS_TRUNCATED, + Stat::Min.name(), + MIN_IS_TRUNCATED, + Stat::Sum.name(), + ] + ); + } + + #[test] + fn stats_table_dtype_uses_storage_dtype_for_extensions() { + let dtype = DType::Extension(Date::new(TimeUnit::Days, Nullability::NonNullable).erased()); + let stats_dtype = stats_table_dtype(&dtype, &[Stat::Max, Stat::Min]); + + assert_eq!( + stats_dtype.as_struct_fields().names().as_ref(), + &[ + Stat::Max.name(), + MAX_IS_TRUNCATED, + Stat::Min.name(), + MIN_IS_TRUNCATED, + ] + ); + } +} diff --git a/vortex-layout/src/layouts/zoned/writer.rs b/vortex-layout/src/layouts/zoned/writer.rs index 3e1d61098b0..7308a62f044 100644 --- a/vortex-layout/src/layouts/zoned/writer.rs +++ b/vortex-layout/src/layouts/zoned/writer.rs @@ -1,3 +1,5 @@ +//! Write-time assembly for zoned layouts. + // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors @@ -8,17 +10,20 @@ use futures::StreamExt as _; use parking_lot::Mutex; use vortex_array::ArrayContext; use vortex_array::IntoArray; +use vortex_array::VortexSessionExecute; use vortex_array::expr::stats::Stat; use vortex_array::stats::PRUNING_STATS; use vortex_error::VortexResult; +use vortex_error::vortex_ensure; use vortex_io::session::RuntimeSessionExt; use vortex_session::VortexSession; +use vortex_utils::parallelism::get_available_parallelism; use crate::IntoLayout; use crate::LayoutRef; use crate::LayoutStrategy; +use crate::layouts::zoned::StatsAccumulator; use crate::layouts::zoned::ZonedLayout; -use crate::layouts::zoned::zone_map::StatsAccumulator; use crate::segments::SegmentSinkRef; use crate::sequence::SendableSequentialStream; use crate::sequence::SequencePointer; @@ -26,6 +31,10 @@ use crate::sequence::SequentialArrayStreamExt; use crate::sequence::SequentialStreamAdapter; use crate::sequence::SequentialStreamExt; +/// Configuration for building zoned layouts. +/// +/// The input stream is assumed to already be partitioned into one chunk per zone, except +/// possibly the final partial zone. pub struct ZonedLayoutOptions { /// The size of a statistics block pub block_size: usize, @@ -43,9 +52,7 @@ impl Default for ZonedLayoutOptions { block_size: 8192, stats: PRUNING_STATS.into(), max_variable_length_statistics_size: 64, - concurrency: std::thread::available_parallelism() - .map(|v| v.get()) - .unwrap_or(1), + concurrency: get_available_parallelism().unwrap_or(1), } } } @@ -57,6 +64,7 @@ pub struct ZonedStrategy { } impl ZonedStrategy { + /// Create a writer that emits a data child plus an auxiliary per-zone stats child. pub fn new( child: Child, stats: Stats, @@ -80,7 +88,14 @@ impl LayoutStrategy for ZonedStrategy { mut eof: SequencePointer, session: &VortexSession, ) -> VortexResult { + vortex_ensure!( + self.options.block_size > 0, + "ZonedStrategy requires block_size > 0 when writing" + ); + let stats = Arc::clone(&self.options.stats); + let session = session.clone(); + let compute_session = session.clone(); let handle = session.handle(); let handle2 = handle.clone(); @@ -96,10 +111,13 @@ impl LayoutStrategy for ZonedStrategy { stream .map(move |chunk| { let stats = Arc::clone(&stats); + let session = compute_session.clone(); handle2.spawn_cpu(move || { let (sequence_id, chunk) = chunk?; - chunk.statistics().compute_all(&stats)?; - VortexResult::Ok((sequence_id, chunk)) + chunk + .statistics() + .compute_all(&stats, &mut session.create_execution_ctx())?; + Ok((sequence_id, chunk)) }) }) .buffered(self.options.concurrency), @@ -133,11 +151,11 @@ impl LayoutStrategy for ZonedStrategy { Arc::clone(&segment_sink), stream, data_eof, - session, + &session, ) .await?; - let Some(stats_table) = stats_accumulator.lock().as_stats_table()? else { + let Some((stats_array, stats)) = stats_accumulator.lock().as_array()? else { // If we have no stats (e.g. the DType doesn't support them), then we just return the // child layout. return Ok(data_layout); @@ -145,24 +163,16 @@ impl LayoutStrategy for ZonedStrategy { // We must defer creating the stats table LayoutWriter until now, because the DType of // the table depends on which stats were successfully computed. - let stats_stream = stats_table - .array() - .clone() + let stats_stream = stats_array .into_array() .to_array_stream() .sequenced(eof.split_off()); let zones_layout = self .stats - .write_stream(ctx, Arc::clone(&segment_sink), stats_stream, eof, session) + .write_stream(ctx, Arc::clone(&segment_sink), stats_stream, eof, &session) .await?; - Ok(ZonedLayout::new( - data_layout, - zones_layout, - block_size, - Arc::clone(stats_table.present_stats()), - ) - .into_layout()) + Ok(ZonedLayout::new(data_layout, zones_layout, block_size, stats).into_layout()) } fn buffered_bytes(&self) -> u64 { diff --git a/vortex-layout/src/layouts/zoned/zone_map.rs b/vortex-layout/src/layouts/zoned/zone_map.rs index 39ab7179236..16360ff287d 100644 --- a/vortex-layout/src/layouts/zoned/zone_map.rs +++ b/vortex-layout/src/layouts/zoned/zone_map.rs @@ -1,36 +1,30 @@ +//! Runtime view of a zoned layout's auxiliary per-zone statistics table. + // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors use std::sync::Arc; -use itertools::Itertools; use vortex_array::ArrayRef; -use vortex_array::ExecutionCtx; use vortex_array::IntoArray; use vortex_array::VortexSessionExecute; -use vortex_array::aggregate_fn::fns::sum::sum; +use vortex_array::arrays::ConstantArray; +use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::StructArray; -use vortex_array::arrays::struct_::StructArrayExt; use vortex_array::dtype::DType; -use vortex_array::dtype::Nullability; -use vortex_array::dtype::PType; -use vortex_array::dtype::StructFields; use vortex_array::expr::Expression; -use vortex_array::expr::stats::Precision; use vortex_array::expr::stats::Stat; -use vortex_array::expr::stats::StatsProvider; -use vortex_array::stats::StatsSet; +use vortex_array::scalar_fn::internal::row_count::contains_row_count; +use vortex_array::scalar_fn::internal::row_count::substitute_row_count; use vortex_array::validity::Validity; -use vortex_error::VortexExpect; +use vortex_buffer::buffer; use vortex_error::VortexResult; use vortex_error::vortex_bail; use vortex_mask::Mask; +use vortex_runend::RunEnd; use vortex_session::VortexSession; -use crate::layouts::zoned::builder::MAX_IS_TRUNCATED; -use crate::layouts::zoned::builder::MIN_IS_TRUNCATED; -use crate::layouts::zoned::builder::StatsArrayBuilder; -use crate::layouts::zoned::builder::stats_builder_with_capacity; +use crate::layouts::zoned::schema::stats_table_dtype; /// A zone map containing statistics for a column. /// Each row of the zone map corresponds to a chunk of the column. @@ -40,8 +34,10 @@ use crate::layouts::zoned::builder::stats_builder_with_capacity; pub struct ZoneMap { // The struct array backing the zone map array: StructArray, - // The statistics that are included in the table. - stats: Arc<[Stat]>, + // The length of each zone in the zone map. + zone_len: u64, + // Number of rows that the zone map covers + row_count: u64, } impl ZoneMap { @@ -51,14 +47,16 @@ impl ZoneMap { column_dtype: DType, array: StructArray, stats: Arc<[Stat]>, + zone_len: u64, + row_count: u64, ) -> VortexResult { - let expected_dtype = Self::dtype_for_stats_table(&column_dtype, &stats); + let expected_dtype = stats_table_dtype(&column_dtype, &stats); if &expected_dtype != array.dtype() { vortex_bail!("Array dtype does not match expected zone map dtype: {expected_dtype}"); } - // SAFETY: We checked that the - Ok(unsafe { Self::new_unchecked(array, stats) }) + // SAFETY: We checked that the array matches the expected stats-table schema. + Ok(unsafe { Self::new_unchecked(array, zone_len, row_count) }) } /// Creates [`ZoneMap`] without validating return array against expected stats. @@ -66,333 +64,105 @@ impl ZoneMap { /// # Safety /// /// Assumes that the input struct array has the correct statistics as fields. Or in other words, - /// the [`DType`] of the input array is equal to the result of [`Self::dtype_for_stats_table`]. - pub unsafe fn new_unchecked(array: StructArray, stats: Arc<[Stat]>) -> Self { - Self { array, stats } + pub unsafe fn new_unchecked(array: StructArray, zone_len: u64, row_count: u64) -> Self { + Self { + array, + zone_len, + row_count, + } } /// Returns the [`DType`] of the statistics table given a set of statistics and column [`DType`]. + /// + /// This remains as a compatibility wrapper around the zoned schema helper. + #[deprecated(note = "use `stats_table_dtype` from `crate::layouts::zoned::schema` instead")] pub fn dtype_for_stats_table(column_dtype: &DType, present_stats: &[Stat]) -> DType { - assert!(present_stats.is_sorted(), "Stats must be sorted"); - DType::Struct( - StructFields::from_iter( - present_stats - .iter() - .filter_map(|stat| { - stat.dtype(column_dtype) - .or_else(|| { - // Backward compat: older files may have stored stats (e.g. Sum) - // for extension types by resolving through the storage dtype. - if let DType::Extension(ext) = column_dtype { - stat.dtype(ext.storage_dtype()) - } else { - None - } - }) - .map(|dtype| (stat, dtype.as_nullable())) - }) - .flat_map(|(s, dt)| match s { - Stat::Max => vec![ - (s.name(), dt), - (MAX_IS_TRUNCATED, DType::Bool(Nullability::NonNullable)), - ], - Stat::Min => vec![ - (s.name(), dt), - (MIN_IS_TRUNCATED, DType::Bool(Nullability::NonNullable)), - ], - _ => vec![(s.name(), dt)], - }), - ), - Nullability::NonNullable, - ) + stats_table_dtype(column_dtype, present_stats) } - /// Returns the underlying [`StructArray`] backing the zone map - pub fn array(&self) -> &StructArray { - &self.array - } - - /// Returns the list of stats included in the zone map. - pub fn present_stats(&self) -> &Arc<[Stat]> { - &self.stats - } - - /// Returns an aggregated stats set for the table. - pub fn to_stats_set(&self, stats: &[Stat], ctx: &mut ExecutionCtx) -> VortexResult { - let mut stats_set = StatsSet::default(); - for &stat in stats { - let Some(array) = self.get_stat(stat)? else { - continue; - }; - - // Different stats need different aggregations - match stat { - // For stats that are associative, we can just compute them over the stat column - Stat::Min | Stat::Max | Stat::Sum => { - if let Some(s) = array.statistics().compute_stat(stat)? - && let Some(v) = s.into_value() - { - stats_set.set(stat, Precision::exact(v)) - } - } - // These stats sum up - Stat::NullCount | Stat::NaNCount | Stat::UncompressedSizeInBytes => { - if let Some(sum_value) = sum(&array, ctx)? - .cast(&DType::Primitive(PType::U64, Nullability::Nullable))? - .into_value() - { - stats_set.set(stat, Precision::exact(sum_value)); - } - } - // We could implement these aggregations in the future, but for now they're unused - Stat::IsConstant | Stat::IsSorted | Stat::IsStrictSorted => {} - } - } - Ok(stats_set) - } - - /// Returns the array for a given stat. - pub fn get_stat(&self, stat: Stat) -> VortexResult> { - Ok(self.array.unmasked_field_by_name_opt(stat.name()).cloned()) - } - - /// Apply a pruning predicate against the ZoneMap, yielding a mask indicating which zones can - /// be pruned. + /// Apply a pruning predicate to this zone map. /// - /// The expression provided should be the result of converting an existing `VortexExpr` via - /// [`checked_pruning_expr`][vortex_array::expr::pruning::checked_pruning_expr] into a prunable - /// expression that can be evaluated on a zone map. + /// `predicate` should be the result of converting a filter with + /// [`checked_pruning_expr`]. The returned mask has one value per zone, where + /// `true` means the zone cannot contain matching rows and can be skipped. /// - /// All zones where the predicate evaluates to `true` can be skipped entirely. + /// If the predicate contains [`row_count`][vortex_array::scalar_fn::internal::row_count] + /// placeholders, they are replaced after [`ArrayRef::apply`] with per-zone + /// counts derived from `zone_len` and `row_count`. Uniform zones use a + /// [`ConstantArray`]; a short final zone uses a run-end encoded array. + /// + /// [`checked_pruning_expr`]: vortex_array::expr::pruning::checked_pruning_expr pub fn prune(&self, predicate: &Expression, session: &VortexSession) -> VortexResult { let mut ctx = session.create_execution_ctx(); - self.array - .clone() - .into_array() - .apply(predicate)? - .execute::(&mut ctx) - } -} + let num_zones = self.array.len(); -// TODO(ngates): we should make it such that the zone map stores a mirror of the DType -// underneath each stats column. For example, `min: i32` for an `i32` array. -// Or `min: {a: i32, b: i32}` for a struct array of type `{a: i32, b: i32}`. -// See: -/// Accumulates statistics for a column. -pub struct StatsAccumulator { - builders: Vec>, - length: usize, -} + let applied = self.array.clone().into_array().apply(predicate)?; -impl StatsAccumulator { - pub fn new(dtype: &DType, stats: &[Stat], max_variable_length_statistics_size: usize) -> Self { - let builders = stats - .iter() - .filter_map(|&s| { - s.dtype(dtype).map(|stat_dtype| { - stats_builder_with_capacity( - s, - &stat_dtype.as_nullable(), - 1024, - max_variable_length_statistics_size, - ) - }) - }) - .collect::>(); - - Self { - builders, - length: 0, + if num_zones == 0 || !contains_row_count(&applied) { + return applied.execute::(&mut ctx); } - } - pub fn push_chunk_without_compute(&mut self, array: &ArrayRef) -> VortexResult<()> { - for builder in self.builders.iter_mut() { - if let Some(Precision::Exact(v)) = array.statistics().get(builder.stat()) { - builder.append_scalar(v.cast(&v.dtype().as_nullable())?)?; - } else { - builder.append_null(); - } - } - self.length += 1; - Ok(()) + let row_count_array = row_count_array(self.zone_len, self.row_count, num_zones)?; + let substituted = substitute_row_count(applied, &row_count_array)?; + substituted.execute::(&mut ctx) } +} - pub fn push_chunk(&mut self, array: &ArrayRef) -> VortexResult<()> { - for builder in self.builders.iter_mut() { - if let Some(v) = array.statistics().compute_stat(builder.stat())? { - builder.append_scalar(v.cast(&v.dtype().as_nullable())?)?; - } else { - builder.append_null(); - } - } - self.length += 1; - Ok(()) +/// Build per-zone row counts for a zone map. +/// +/// `zone_len` is the nominal zone size; only the final zone may be shorter. The +/// result is a [`ConstantArray`] for uniform zone sizes, otherwise a two-run +/// run-end encoded array whose trailing run carries the final zone length. +fn row_count_array(zone_len: u64, row_count: u64, num_zones: usize) -> VortexResult { + let last_zone_len = row_count - zone_len.saturating_mul((num_zones as u64) - 1); + if num_zones == 1 || last_zone_len == zone_len { + return Ok(ConstantArray::new(last_zone_len, num_zones).into_array()); } - /// Finishes the accumulator into a [`ZoneMap`]. - /// - /// Returns `None` if none of the requested statistics can be computed, for example they are - /// not applicable to the column's data type. - pub fn as_stats_table(&mut self) -> VortexResult> { - let mut names = Vec::new(); - let mut fields = Vec::new(); - let mut stats = Vec::new(); - - for builder in self - .builders - .iter_mut() - // We sort the stats so the DType is deterministic based on which stats are present. - .sorted_unstable_by_key(|b| b.stat()) - { - let values = builder.finish(); - - // We drop any all-null stats columns - if values.all_invalid()? { - continue; - } - - stats.push(builder.stat()); - names.extend(values.names); - fields.extend(values.arrays); - } - - if names.is_empty() { - return Ok(None); - } - - Ok(Some(ZoneMap { - array: StructArray::try_new(names.into(), fields, self.length, Validity::NonNullable) - .vortex_expect("Failed to create zone map"), - stats: stats.into(), - })) + let ends = unsafe { + PrimitiveArray::new_unchecked( + buffer![num_zones as u64 - 1, num_zones as u64], + Validity::NonNullable, + ) } + .into_array(); + let values = unsafe { + PrimitiveArray::new_unchecked(buffer![zone_len, last_zone_len], Validity::NonNullable) + } + .into_array(); + + // SAFETY: `ends` are strictly increasing, terminate at `num_zones`, and align one-to-one + // with the non-null run values. + Ok(unsafe { RunEnd::new_unchecked(ends, values, 0, num_zones) }.into_array()) } #[cfg(test)] mod tests { use std::sync::Arc; - use rstest::rstest; use vortex_array::IntoArray; - use vortex_array::ToCanonical; use vortex_array::arrays::BoolArray; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::StructArray; - use vortex_array::arrays::bool::BoolArrayExt; - use vortex_array::arrays::struct_::StructArrayExt; use vortex_array::assert_arrays_eq; - use vortex_array::builders::ArrayBuilder; - use vortex_array::builders::VarBinViewBuilder; - use vortex_array::dtype::DType; use vortex_array::dtype::FieldPath; use vortex_array::dtype::FieldPathSet; - use vortex_array::dtype::Nullability; use vortex_array::dtype::PType; use vortex_array::expr::gt; use vortex_array::expr::gt_eq; + use vortex_array::expr::is_not_null; use vortex_array::expr::lit; use vortex_array::expr::lt; use vortex_array::expr::pruning::checked_pruning_expr; use vortex_array::expr::root; use vortex_array::expr::stats::Stat; use vortex_array::validity::Validity; - use vortex_buffer::BitBuffer; use vortex_buffer::buffer; - use vortex_error::VortexExpect; - use crate::layouts::zoned::MAX_IS_TRUNCATED; - use crate::layouts::zoned::MIN_IS_TRUNCATED; - use crate::layouts::zoned::zone_map::StatsAccumulator; use crate::layouts::zoned::zone_map::ZoneMap; use crate::test::SESSION; - #[rstest] - #[case(DType::Utf8(Nullability::NonNullable))] - #[case(DType::Binary(Nullability::NonNullable))] - fn truncates_accumulated_stats(#[case] dtype: DType) { - let mut builder = VarBinViewBuilder::with_capacity(dtype.clone(), 2); - builder.append_value("Value to be truncated"); - builder.append_value("untruncated"); - let mut builder2 = VarBinViewBuilder::with_capacity(dtype, 2); - builder2.append_value("Another"); - builder2.append_value("wait a minute"); - let mut acc = - StatsAccumulator::new(builder.dtype(), &[Stat::Max, Stat::Min, Stat::Sum], 12); - acc.push_chunk(&builder.finish()) - .vortex_expect("push_chunk should succeed for test data"); - acc.push_chunk(&builder2.finish()) - .vortex_expect("push_chunk should succeed for test data"); - let stats_table = acc - .as_stats_table() - .unwrap() - .expect("Must have stats table"); - assert_eq!( - stats_table.array.names().as_ref(), - &[ - Stat::Max.name(), - MAX_IS_TRUNCATED, - Stat::Min.name(), - MIN_IS_TRUNCATED, - ] - ); - assert_eq!( - stats_table - .array - .unmasked_field(1) - .to_bool() - .to_bit_buffer(), - BitBuffer::from(vec![false, true]) - ); - assert_eq!( - stats_table - .array - .unmasked_field(3) - .to_bool() - .to_bit_buffer(), - BitBuffer::from(vec![true, false]) - ); - } - #[test] - fn always_adds_is_truncated_column() { - let array = buffer![0, 1, 2].into_array(); - let mut acc = StatsAccumulator::new(array.dtype(), &[Stat::Max, Stat::Min, Stat::Sum], 12); - acc.push_chunk(&array) - .vortex_expect("push_chunk should succeed for test array"); - let stats_table = acc - .as_stats_table() - .unwrap() - .expect("Must have stats table"); - assert_eq!( - stats_table.array.names().as_ref(), - &[ - Stat::Max.name(), - MAX_IS_TRUNCATED, - Stat::Min.name(), - MIN_IS_TRUNCATED, - Stat::Sum.name(), - ] - ); - assert_eq!( - stats_table - .array - .unmasked_field(1) - .to_bool() - .to_bit_buffer(), - BitBuffer::from(vec![false]) - ); - assert_eq!( - stats_table - .array - .unmasked_field(3) - .to_bool() - .to_bit_buffer(), - BitBuffer::from(vec![false]) - ); - } - - #[rstest] fn test_zone_map_prunes() { // All stats that are known at pruning time. let stats = FieldPathSet::from_iter([ @@ -433,6 +203,8 @@ mod tests { ]) .unwrap(), Arc::new([Stat::Max, Stat::Min]), + 3, + 10, ) .unwrap(); @@ -463,4 +235,59 @@ mod tests { let mask = zone_map.prune(&pruning_expr, &SESSION).unwrap(); assert_arrays_eq!(mask.into_array(), BoolArray::from_iter([false, true, true])); } + + #[test] + fn row_count_prunes_short_trailing_zone() { + let zone_map = ZoneMap::try_new( + PType::U64.into(), + StructArray::from_fields(&[( + "null_count", + PrimitiveArray::new(buffer![0u64, 0, 2], Validity::AllValid).into_array(), + )]) + .unwrap(), + Arc::new([Stat::NullCount]), + 4, + 10, + ) + .unwrap(); + + let available_stats = + FieldPathSet::from_iter([FieldPath::from_iter([Stat::NullCount.name().into()])]); + let expr = is_not_null(root()); + let (pruning_expr, _) = checked_pruning_expr(&expr, &available_stats).unwrap(); + + let mask = zone_map.prune(&pruning_expr, &SESSION).unwrap(); + assert_arrays_eq!( + mask.into_array(), + BoolArray::from_iter([false, false, true]) + ); + } + + #[test] + fn row_count_prunes_all_null_uniform_zones() { + let zone_map = ZoneMap::try_new( + PType::U64.into(), + StructArray::from_fields(&[( + "null_count", + PrimitiveArray::new(buffer![0u64, 4, 0], Validity::AllValid).into_array(), + )]) + .unwrap(), + Arc::new([Stat::NullCount]), + 4, + 12, + ) + .unwrap(); + + let available_stats = + FieldPathSet::from_iter([FieldPath::from_iter([Stat::NullCount.name().into()])]); + let expr = is_not_null(root()); + let (pruning_expr, _) = checked_pruning_expr(&expr, &available_stats).unwrap(); + + // All three zones have length 4 (total rows = 12). + let mask = zone_map.prune(&pruning_expr, &SESSION).unwrap(); + assert_arrays_eq!( + mask.into_array(), + BoolArray::from_iter([false, true, false]) + ); + } } diff --git a/vortex-layout/src/reader.rs b/vortex-layout/src/reader.rs index e103c163efe..f1467212c38 100644 --- a/vortex-layout/src/reader.rs +++ b/vortex-layout/src/reader.rs @@ -1,6 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +use std::any::Any; use std::collections::BTreeSet; use std::ops::Range; use std::sync::Arc; @@ -31,6 +32,8 @@ pub trait LayoutReader: 'static + Send + Sync { /// Returns the name of the layout reader for debugging. fn name(&self) -> &Arc; + fn as_any(&self) -> &dyn Any; + /// Returns the un-projected dtype of the layout reader. fn dtype(&self) -> &DType; diff --git a/vortex-layout/src/scan/layout.rs b/vortex-layout/src/scan/layout.rs index 4ffdadab2e6..0b0a7d94f3b 100644 --- a/vortex-layout/src/scan/layout.rs +++ b/vortex-layout/src/scan/layout.rs @@ -139,7 +139,7 @@ impl DataSource for LayoutReaderDataSource { // Check file-level pruning: if the filter can be proven false for the entire row range // using file-level statistics (e.g. via FileStatsLayoutReader), skip the scan entirely. - if let Some(ref filter) = scan_request.filter { + if let Some(filter) = &scan_request.filter { let mask = Mask::new_true( usize::try_from(row_range.end - row_range.start).unwrap_or(usize::MAX), ); @@ -289,6 +289,12 @@ impl Partition for LayoutReaderSplit { self } + #[expect(clippy::cast_possible_truncation)] + fn index(&self) -> usize { + // Row range is unique per split + self.row_range.start as usize + } + fn row_count(&self) -> Option> { let row_count = self.row_range.end - self.row_range.start; let row_count = self.selection.row_count(row_count); @@ -351,6 +357,10 @@ impl Partition for Empty { self } + fn index(&self) -> usize { + 0 + } + fn row_count(&self) -> Option> { Some(Precision::exact(self.row_count)) } diff --git a/vortex-layout/src/scan/multi.rs b/vortex-layout/src/scan/multi.rs index 307aca187cc..38503c77a63 100644 --- a/vortex-layout/src/scan/multi.rs +++ b/vortex-layout/src/scan/multi.rs @@ -51,7 +51,9 @@ use vortex_scan::Partition; use vortex_scan::PartitionRef; use vortex_scan::PartitionStream; use vortex_scan::ScanRequest; +use vortex_scan::selection::Selection; use vortex_session::VortexSession; +use vortex_utils::parallelism::get_available_parallelism; use crate::LayoutReaderRef; use crate::scan::scan_builder::ScanBuilder; @@ -84,7 +86,7 @@ pub struct MultiLayoutDataSource { concurrency: usize, } -enum MultiLayoutChild { +pub enum MultiLayoutChild { Opened(LayoutReaderRef), Deferred(Arc), } @@ -100,9 +102,7 @@ impl MultiLayoutDataSource { session: &VortexSession, ) -> Self { let dtype = first.dtype().clone(); - let concurrency = std::thread::available_parallelism() - .map(|v| v.get()) - .unwrap_or(DEFAULT_CONCURRENCY); + let concurrency = get_available_parallelism().unwrap_or(DEFAULT_CONCURRENCY); let mut children = Vec::with_capacity(1 + remaining.len()); children.push(MultiLayoutChild::Opened(first)); @@ -126,9 +126,7 @@ impl MultiLayoutDataSource { factories: Vec>, session: &VortexSession, ) -> Self { - let concurrency = std::thread::available_parallelism() - .map(|v| v.get()) - .unwrap_or(DEFAULT_CONCURRENCY); + let concurrency = get_available_parallelism().unwrap_or(DEFAULT_CONCURRENCY); Self { dtype, @@ -141,6 +139,10 @@ impl MultiLayoutDataSource { } } + pub fn children(&self) -> &Vec { + &self.children + } + /// Sets the concurrency for opening deferred readers. /// /// Controls how many file opens run in parallel via `buffer_unordered`. @@ -308,8 +310,9 @@ impl DataSourceScan for MultiLayoutScan { // `flat_map` is appropriate here. The real I/O work happens when `execute()` is called. ready_stream .chain(deferred_stream) - .flat_map(move |reader_result| match reader_result { - Ok(reader) => reader_partition(reader, session.clone(), request.clone()), + .enumerate() + .flat_map(move |(i, reader_result)| match reader_result { + Ok(reader) => reader_partition(i, reader, session.clone(), request.clone()), Err(e) => stream::once(async move { Err(e) }).boxed(), }) .boxed() @@ -322,6 +325,7 @@ impl DataSourceScan for MultiLayoutScan { /// can match, returns an empty stream. Otherwise, yields a single partition covering the /// reader's full row range. fn reader_partition( + partition_idx: usize, reader: LayoutReaderRef, session: VortexSession, request: ScanRequest, @@ -329,9 +333,29 @@ fn reader_partition( let row_count = reader.row_count(); let row_range = request.row_range.clone().unwrap_or(0..row_count); + let partition_idx_u64: u64 = partition_idx as u64; + if let Some(range) = &request.partition_range + && !range.contains(&partition_idx_u64) + { + return stream::empty().boxed(); + }; + match &request.partition_selection { + Selection::IncludeByIndex(buffer) => { + if buffer.as_slice().binary_search(&partition_idx_u64).is_err() { + return stream::empty().boxed(); + } + } + Selection::ExcludeByIndex(buffer) => { + if buffer.as_slice().binary_search(&partition_idx_u64).is_ok() { + return stream::empty().boxed(); + } + } + _ => {} + }; + // Check file-level pruning: if the filter can be proven false for the entire row range // using file-level statistics, skip this reader entirely. - if let Some(ref filter) = request.filter { + if let Some(filter) = &request.filter { let mask_len = usize::try_from(row_range.end - row_range.start).unwrap_or(usize::MAX); let mask = Mask::new_true(mask_len); if let Ok(pruning_future) = reader.pruning_evaluation(&row_range, filter, mask) @@ -350,6 +374,7 @@ fn reader_partition( row_range: Some(row_range), ..request }, + index: partition_idx, }) as PartitionRef) }) .boxed() @@ -363,6 +388,7 @@ struct MultiLayoutPartition { reader: LayoutReaderRef, session: VortexSession, request: ScanRequest, + index: usize, } impl Partition for MultiLayoutPartition { @@ -370,6 +396,10 @@ impl Partition for MultiLayoutPartition { self } + fn index(&self) -> usize { + self.index + } + fn row_count(&self) -> Option> { let row_range = self.request.row_range.as_ref()?; let row_count = row_range.end - row_range.start; diff --git a/vortex-layout/src/scan/repeated_scan.rs b/vortex-layout/src/scan/repeated_scan.rs index 1d5caaa66cf..a0f2101556c 100644 --- a/vortex-layout/src/scan/repeated_scan.rs +++ b/vortex-layout/src/scan/repeated_scan.rs @@ -22,6 +22,7 @@ use vortex_io::runtime::BlockingRuntime; use vortex_io::session::RuntimeSessionExt; use vortex_scan::selection::Selection; use vortex_session::VortexSession; +use vortex_utils::parallelism::get_available_parallelism; use crate::LayoutReaderRef; use crate::scan::filter::FilterExpr; @@ -187,9 +188,7 @@ impl RepeatedScan { row_range: Option>, ) -> VortexResult> + Send + 'static + use> { use futures::StreamExt; - let num_workers = std::thread::available_parallelism() - .map(|n| n.get()) - .unwrap_or(1); + let num_workers = get_available_parallelism().unwrap_or(1); let concurrency = self.concurrency * num_workers; let handle = self.session.handle(); diff --git a/vortex-layout/src/scan/scan_builder.rs b/vortex-layout/src/scan/scan_builder.rs index d53a38163fa..bdf5d0bfb11 100644 --- a/vortex-layout/src/scan/scan_builder.rs +++ b/vortex-layout/src/scan/scan_builder.rs @@ -38,6 +38,7 @@ use vortex_io::session::RuntimeSessionExt; use vortex_metrics::MetricsRegistry; use vortex_scan::selection::Selection; use vortex_session::VortexSession; +use vortex_utils::parallelism::get_available_parallelism; use crate::LayoutReader; use crate::LayoutReaderRef; @@ -367,9 +368,7 @@ impl Stream for LazyScanStream { LazyScanState::Builder(builder) => { let builder = builder.take().vortex_expect("polled after completion"); let ordered = builder.ordered; - let num_workers = std::thread::available_parallelism() - .map(|n| n.get()) - .unwrap_or(1); + let num_workers = get_available_parallelism().unwrap_or(1); let concurrency = builder.concurrency * num_workers; let handle = builder.session.handle(); let task = handle.spawn_blocking(move || { @@ -465,8 +464,9 @@ mod test { use futures::task::noop_waker_ref; use parking_lot::Mutex; use vortex_array::IntoArray; + use vortex_array::LEGACY_SESSION; use vortex_array::MaskFuture; - use vortex_array::ToCanonical; + use vortex_array::VortexSessionExecute; use vortex_array::arrays::PrimitiveArray; use vortex_array::dtype::DType; use vortex_array::dtype::FieldMask; @@ -554,6 +554,10 @@ mod test { unreachable!("scan should not be polled in this test") })) } + + fn as_any(&self) -> &dyn std::any::Any { + self + } } #[test] @@ -649,22 +653,28 @@ mod test { let array = PrimitiveArray::from_iter(values?).into_array(); Ok(Box::pin(async move { Ok(array) })) } + + fn as_any(&self) -> &dyn std::any::Any { + self + } } #[test] fn into_stream_executes_after_prepare() -> VortexResult<()> { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let calls = Arc::new(AtomicUsize::new(0)); let reader = Arc::new(SplittingLayoutReader::new(Arc::clone(&calls))); let runtime = SingleThreadRuntime::default(); let session = crate::scan::test::session_with_handle(runtime.handle()); - let stream = ScanBuilder::new(session, reader).into_stream().unwrap(); + let stream = ScanBuilder::new(session, reader).into_stream()?; let mut iter = runtime.block_on_stream(stream); let mut values = Vec::new(); for chunk in &mut iter { - values.push(chunk?.to_primitive().into_buffer::()[0]); + let prim = chunk?.execute::(&mut ctx)?; + values.push(prim.into_buffer::()[0]); } assert_eq!(calls.load(Ordering::Relaxed), 1); @@ -747,6 +757,10 @@ mod test { unreachable!("scan should not be polled in this test") })) } + + fn as_any(&self) -> &dyn std::any::Any { + self + } } #[test] diff --git a/vortex-layout/src/session.rs b/vortex-layout/src/session.rs index 131115950e4..370fe391ca0 100644 --- a/vortex-layout/src/session.rs +++ b/vortex-layout/src/session.rs @@ -1,8 +1,11 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +use std::any::Any; + use vortex_session::Ref; use vortex_session::SessionExt; +use vortex_session::SessionVar; use vortex_session::registry::Registry; use crate::LayoutEncodingRef; @@ -54,6 +57,15 @@ impl Default for LayoutSession { } } +impl SessionVar for LayoutSession { + fn as_any(&self) -> &dyn Any { + self + } + fn as_any_mut(&mut self) -> &mut dyn Any { + self + } +} + /// Extension trait for accessing layout session data. pub trait LayoutSessionExt: SessionExt { /// Returns the layout encoding registry. diff --git a/vortex-mask/benches/intersect_by_rank.rs b/vortex-mask/benches/intersect_by_rank.rs index fd02d5ece7b..e7f495d6be7 100644 --- a/vortex-mask/benches/intersect_by_rank.rs +++ b/vortex-mask/benches/intersect_by_rank.rs @@ -3,8 +3,6 @@ //! Benchmarks for `intersect_by_rank`. -#![allow(clippy::unwrap_used, clippy::cast_possible_truncation)] - use divan::Bencher; use vortex_buffer::BitBuffer; use vortex_mask::Mask; @@ -39,6 +37,7 @@ const DENSITY_MATRIX_ARGS: &[(f64, f64, &str)] = &[ fn create_random_mask(len: usize, selectivity: f64) -> Mask { Mask::from_buffer(BitBuffer::from_iter((0..len).map(|i| { + #[expect(clippy::cast_possible_truncation, clippy::cast_sign_loss)] let threshold = (selectivity * 1000.0) as usize; (i * 7 + 13) % 1000 < threshold }))) diff --git a/vortex-mask/public-api.lock b/vortex-mask/public-api.lock index a986381bcff..d61e008ae72 100644 --- a/vortex-mask/public-api.lock +++ b/vortex-mask/public-api.lock @@ -14,17 +14,17 @@ pub fn vortex_mask::AllOr<&T>::cloned(self) -> vortex_mask::AllOr where T: co impl vortex_mask::AllOr -pub fn vortex_mask::AllOr::unwrap_or_else(self, all_true: F, all_false: G) -> T where F: core::ops::function::FnOnce() -> T, G: core::ops::function::FnOnce() -> T +pub fn vortex_mask::AllOr::unwrap_or_else(self, F, G) -> T where F: core::ops::function::FnOnce() -> T, G: core::ops::function::FnOnce() -> T impl core::cmp::Eq for vortex_mask::AllOr where T: core::cmp::Eq impl core::cmp::PartialEq for vortex_mask::AllOr where T: core::cmp::PartialEq -pub fn vortex_mask::AllOr::eq(&self, other: &Self) -> bool +pub fn vortex_mask::AllOr::eq(&self, &Self) -> bool impl core::fmt::Debug for vortex_mask::AllOr where T: core::fmt::Debug -pub fn vortex_mask::AllOr::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_mask::AllOr::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub enum vortex_mask::Mask @@ -44,7 +44,7 @@ pub fn vortex_mask::Mask::bit_buffer(&self) -> vortex_mask::AllOr<&vortex_buffer pub fn vortex_mask::Mask::clear(&mut self) -pub fn vortex_mask::Mask::concat<'a>(masks: impl core::iter::traits::iterator::Iterator) -> vortex_error::VortexResult +pub fn vortex_mask::Mask::concat<'a>(impl core::iter::traits::iterator::Iterator) -> vortex_error::VortexResult pub fn vortex_mask::Mask::density(&self) -> f64 @@ -52,15 +52,15 @@ pub fn vortex_mask::Mask::false_count(&self) -> usize pub fn vortex_mask::Mask::first(&self) -> core::option::Option -pub fn vortex_mask::Mask::from_buffer(buffer: vortex_buffer::bit::buf::BitBuffer) -> Self +pub fn vortex_mask::Mask::from_buffer(vortex_buffer::bit::buf::BitBuffer) -> Self -pub fn vortex_mask::Mask::from_excluded_indices(len: usize, indices: impl core::iter::traits::collect::IntoIterator) -> Self +pub fn vortex_mask::Mask::from_excluded_indices(usize, impl core::iter::traits::collect::IntoIterator) -> Self -pub fn vortex_mask::Mask::from_indices(len: usize, indices: alloc::vec::Vec) -> Self +pub fn vortex_mask::Mask::from_indices(usize, alloc::vec::Vec) -> Self -pub fn vortex_mask::Mask::from_intersection_indices(len: usize, lhs: impl core::iter::traits::iterator::Iterator, rhs: impl core::iter::traits::iterator::Iterator) -> Self +pub fn vortex_mask::Mask::from_intersection_indices(usize, impl core::iter::traits::iterator::Iterator, impl core::iter::traits::iterator::Iterator) -> Self -pub fn vortex_mask::Mask::from_slices(len: usize, vec: alloc::vec::Vec<(usize, usize)>) -> Self +pub fn vortex_mask::Mask::from_slices(usize, alloc::vec::Vec<(usize, usize)>) -> Self pub fn vortex_mask::Mask::indices(&self) -> vortex_mask::AllOr<&[usize]> @@ -72,49 +72,39 @@ pub fn vortex_mask::Mask::last(&self) -> core::option::Option pub fn vortex_mask::Mask::len(&self) -> usize -pub fn vortex_mask::Mask::limit(self, limit: usize) -> Self +pub fn vortex_mask::Mask::limit(self, usize) -> Self -pub fn vortex_mask::Mask::new(length: usize, value: bool) -> Self +pub fn vortex_mask::Mask::new(usize, bool) -> Self -pub fn vortex_mask::Mask::new_false(length: usize) -> Self +pub fn vortex_mask::Mask::new_false(usize) -> Self -pub fn vortex_mask::Mask::new_true(length: usize) -> Self +pub fn vortex_mask::Mask::new_true(usize) -> Self -pub fn vortex_mask::Mask::rank(&self, n: usize) -> usize +pub fn vortex_mask::Mask::rank(&self, usize) -> usize -pub fn vortex_mask::Mask::slice(&self, range: impl core::ops::range::RangeBounds) -> Self +pub fn vortex_mask::Mask::slice(&self, impl core::ops::range::RangeBounds) -> Self pub fn vortex_mask::Mask::slices(&self) -> vortex_mask::AllOr<&[(usize, usize)]> -pub fn vortex_mask::Mask::threshold_iter(&self, threshold: f64) -> vortex_mask::AllOr> +pub fn vortex_mask::Mask::threshold_iter(&self, f64) -> vortex_mask::AllOr> pub fn vortex_mask::Mask::to_bit_buffer(&self) -> vortex_buffer::bit::buf::BitBuffer pub fn vortex_mask::Mask::true_count(&self) -> usize -pub fn vortex_mask::Mask::valid_counts_for_indices(&self, indices: &[usize]) -> alloc::vec::Vec +pub fn vortex_mask::Mask::valid_counts_for_indices(&self, &[usize]) -> alloc::vec::Vec -pub fn vortex_mask::Mask::value(&self, idx: usize) -> bool +pub fn vortex_mask::Mask::value(&self, usize) -> bool pub fn vortex_mask::Mask::values(&self) -> core::option::Option<&vortex_mask::MaskValues> impl vortex_mask::Mask -pub fn vortex_mask::Mask::bitand_not(self, rhs: &vortex_mask::Mask) -> vortex_mask::Mask +pub fn vortex_mask::Mask::bitand_not(self, &vortex_mask::Mask) -> vortex_mask::Mask impl vortex_mask::Mask -pub fn vortex_mask::Mask::intersect_by_rank(&self, mask: &vortex_mask::Mask) -> vortex_mask::Mask - -impl vortex_mask::Mask - -pub fn vortex_mask::Mask::into_mut(self) -> vortex_mask::MaskMut - -pub fn vortex_mask::Mask::try_into_mut(self) -> core::result::Result - -impl vortex_mask::Mask - -pub fn vortex_mask::Mask::iter_bools(&self, f: F) -> T where F: core::ops::function::FnMut(&mut dyn core::iter::traits::iterator::Iterator) -> T +pub fn vortex_mask::Mask::intersect_by_rank(&self, &vortex_mask::Mask) -> vortex_mask::Mask impl core::clone::Clone for vortex_mask::Mask @@ -124,11 +114,11 @@ impl core::cmp::Eq for vortex_mask::Mask impl core::cmp::PartialEq for vortex_mask::Mask -pub fn vortex_mask::Mask::eq(&self, other: &Self) -> bool +pub fn vortex_mask::Mask::eq(&self, &Self) -> bool impl core::convert::From for vortex_mask::Mask -pub fn vortex_mask::Mask::from(value: vortex_buffer::bit::buf::BitBuffer) -> Self +pub fn vortex_mask::Mask::from(vortex_buffer::bit::buf::BitBuffer) -> Self impl core::default::Default for vortex_mask::Mask @@ -136,27 +126,27 @@ pub fn vortex_mask::Mask::default() -> Self impl core::fmt::Debug for vortex_mask::Mask -pub fn vortex_mask::Mask::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_mask::Mask::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::iter::traits::collect::FromIterator for vortex_mask::Mask -pub fn vortex_mask::Mask::from_iter>(iter: T) -> Self +pub fn vortex_mask::Mask::from_iter>(T) -> Self impl core::iter::traits::collect::FromIterator for vortex_mask::Mask -pub fn vortex_mask::Mask::from_iter>(iter: T) -> Self +pub fn vortex_mask::Mask::from_iter>(T) -> Self impl core::ops::bit::BitAnd for &vortex_mask::Mask pub type &vortex_mask::Mask::Output = vortex_mask::Mask -pub fn &vortex_mask::Mask::bitand(self, rhs: Self) -> Self::Output +pub fn &vortex_mask::Mask::bitand(self, Self) -> Self::Output impl core::ops::bit::BitOr for &vortex_mask::Mask pub type &vortex_mask::Mask::Output = vortex_mask::Mask -pub fn &vortex_mask::Mask::bitor(self, rhs: Self) -> Self::Output +pub fn &vortex_mask::Mask::bitor(self, Self) -> Self::Output impl core::ops::bit::Not for &vortex_mask::Mask @@ -176,78 +166,6 @@ pub vortex_mask::MaskIter::Indices(&'a [usize]) pub vortex_mask::MaskIter::Slices(&'a [(usize, usize)]) -pub struct vortex_mask::MaskMut(_) - -impl vortex_mask::MaskMut - -pub fn vortex_mask::MaskMut::all_false(&self) -> bool - -pub fn vortex_mask::MaskMut::all_true(&self) -> bool - -pub fn vortex_mask::MaskMut::append_mask(&mut self, other: &vortex_mask::Mask) - -pub fn vortex_mask::MaskMut::append_n(&mut self, new_value: bool, n: usize) - -pub fn vortex_mask::MaskMut::as_bit_buffer_mut(&mut self) -> core::option::Option<&mut vortex_buffer::bit::buf_mut::BitBufferMut> - -pub fn vortex_mask::MaskMut::capacity(&self) -> usize - -pub fn vortex_mask::MaskMut::clear(&mut self) - -pub fn vortex_mask::MaskMut::empty() -> Self - -pub fn vortex_mask::MaskMut::freeze(self) -> vortex_mask::Mask - -pub fn vortex_mask::MaskMut::from_buffer(bit_buffer: vortex_buffer::bit::buf_mut::BitBufferMut) -> Self - -pub fn vortex_mask::MaskMut::is_empty(&self) -> bool - -pub fn vortex_mask::MaskMut::len(&self) -> usize - -pub fn vortex_mask::MaskMut::new(len: usize, value: bool) -> Self - -pub fn vortex_mask::MaskMut::new_false(len: usize) -> Self - -pub fn vortex_mask::MaskMut::new_true(len: usize) -> Self - -pub fn vortex_mask::MaskMut::reserve(&mut self, additional: usize) - -pub fn vortex_mask::MaskMut::set(&mut self, index: usize) - -pub unsafe fn vortex_mask::MaskMut::set_len(&mut self, new_len: usize) - -pub fn vortex_mask::MaskMut::set_to(&mut self, index: usize, value: bool) - -pub unsafe fn vortex_mask::MaskMut::set_to_unchecked(&mut self, index: usize, value: bool) - -pub unsafe fn vortex_mask::MaskMut::set_unchecked(&mut self, index: usize) - -pub fn vortex_mask::MaskMut::split_off(&mut self, at: usize) -> Self - -pub fn vortex_mask::MaskMut::truncate(&mut self, len: usize) - -pub fn vortex_mask::MaskMut::unset(&mut self, index: usize) - -pub unsafe fn vortex_mask::MaskMut::unset_unchecked(&mut self, index: usize) - -pub fn vortex_mask::MaskMut::unsplit(&mut self, other: Self) - -pub fn vortex_mask::MaskMut::value(&self, index: usize) -> bool - -pub fn vortex_mask::MaskMut::with_capacity(capacity: usize) -> Self - -impl core::clone::Clone for vortex_mask::MaskMut - -pub fn vortex_mask::MaskMut::clone(&self) -> vortex_mask::MaskMut - -impl core::default::Default for vortex_mask::MaskMut - -pub fn vortex_mask::MaskMut::default() -> Self - -impl core::fmt::Debug for vortex_mask::MaskMut - -pub fn vortex_mask::MaskMut::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result - pub struct vortex_mask::MaskValues impl vortex_mask::MaskValues @@ -266,12 +184,12 @@ pub fn vortex_mask::MaskValues::len(&self) -> usize pub fn vortex_mask::MaskValues::slices(&self) -> &[(usize, usize)] -pub fn vortex_mask::MaskValues::threshold_iter(&self, threshold: f64) -> vortex_mask::MaskIter<'_> +pub fn vortex_mask::MaskValues::threshold_iter(&self, f64) -> vortex_mask::MaskIter<'_> pub fn vortex_mask::MaskValues::true_count(&self) -> usize -pub fn vortex_mask::MaskValues::value(&self, index: usize) -> bool +pub fn vortex_mask::MaskValues::value(&self, usize) -> bool impl core::fmt::Debug for vortex_mask::MaskValues -pub fn vortex_mask::MaskValues::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_mask::MaskValues::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result diff --git a/vortex-mask/src/arrow.rs b/vortex-mask/src/arrow.rs deleted file mode 100644 index 1fb8d98eb89..00000000000 --- a/vortex-mask/src/arrow.rs +++ /dev/null @@ -1,24 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -use arrow_buffer::NullBuffer; - -use crate::Mask; - -impl From for Option { - fn from(value: Mask) -> Self { - match value { - Mask::AllTrue(_) => None, - Mask::AllFalse(len) => Some(NullBuffer::new_null(len)), - Mask::Values(values) => { - // SAFETY: we maintain our own validated true count. - Some(unsafe { - NullBuffer::new_unchecked( - values.bit_buffer().clone().into(), - values.len() - values.true_count(), - ) - }) - } - } - } -} diff --git a/vortex-mask/src/bitops.rs b/vortex-mask/src/bitops.rs index 3ad681db161..94d7ef02980 100644 --- a/vortex-mask/src/bitops.rs +++ b/vortex-mask/src/bitops.rs @@ -82,7 +82,7 @@ impl Not for &Mask { } #[cfg(test)] -#[allow(clippy::many_single_char_names)] +#[expect(clippy::many_single_char_names)] mod tests { use vortex_buffer::BitBuffer; diff --git a/vortex-mask/src/iter_bools.rs b/vortex-mask/src/iter_bools.rs deleted file mode 100644 index c3d982db45a..00000000000 --- a/vortex-mask/src/iter_bools.rs +++ /dev/null @@ -1,87 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -use std::iter; - -use crate::AllOr; -use crate::Mask; - -impl Mask { - /// Provides a closure with an iterator over the boolean values of the mask. - /// - /// This allows us to provide different implementations of the iterator based on the underlying - /// representation of the mask, while avoiding a heap allocation to return a boxed iterator. - /// - /// Note that bool iteration might not be the fastest way to achieve whatever is it you're - /// trying to do! - #[inline] - pub fn iter_bools(&self, mut f: F) -> T - where - F: FnMut(&mut dyn Iterator) -> T, - { - match self.bit_buffer() { - AllOr::All => f(&mut iter::repeat_n(true, self.len())), - AllOr::None => f(&mut iter::repeat_n(false, self.len())), - AllOr::Some(buffer) => f(&mut buffer.iter()), - } - } -} - -#[cfg(test)] -mod test { - use itertools::Itertools; - - use crate::Mask; - - #[test] - fn iter_bools_all_true() { - let mask = Mask::new_true(10); - assert_eq!(mask.iter_bools(|iter| iter.collect_vec()), vec![true; 10]); - } - - #[test] - fn iter_bools_all_false() { - let mask = Mask::new_false(10); - assert_eq!(mask.iter_bools(|iter| iter.collect_vec()), vec![false; 10]); - } - - #[test] - fn iter_bools_indices() { - assert_eq!( - Mask::from_indices(5, vec![]).iter_bools(|iter| iter.collect_vec()), - vec![false; 5], - ); - assert_eq!( - Mask::from_indices(5, vec![0, 1, 2, 3, 4]).iter_bools(|iter| iter.collect_vec()), - vec![true; 5], - ); - assert_eq!( - Mask::from_indices(5, vec![0, 4]).iter_bools(|iter| iter.collect_vec()), - vec![true, false, false, false, true], - ); - assert_eq!( - Mask::from_indices(5, vec![1, 2, 3]).iter_bools(|iter| iter.collect_vec()), - vec![false, true, true, true, false], - ); - } - - #[test] - fn iter_bools_slices() { - assert_eq!( - Mask::from_slices(5, vec![]).iter_bools(|iter| iter.collect_vec()), - vec![false; 5], - ); - assert_eq!( - Mask::from_slices(5, vec![(0, 5)]).iter_bools(|iter| iter.collect_vec()), - vec![true; 5], - ); - assert_eq!( - Mask::from_slices(5, vec![(0, 1), (4, 5)]).iter_bools(|iter| iter.collect_vec()), - vec![true, false, false, false, true], - ); - assert_eq!( - Mask::from_slices(5, vec![(1, 4)]).iter_bools(|iter| iter.collect_vec()), - vec![false, true, true, true, false], - ); - } -} diff --git a/vortex-mask/src/lib.rs b/vortex-mask/src/lib.rs index 9c551837de1..898aa767a3a 100644 --- a/vortex-mask/src/lib.rs +++ b/vortex-mask/src/lib.rs @@ -7,11 +7,7 @@ mod bitops; mod eq; mod intersect_by_rank; -mod iter_bools; -mod mask_mut; -#[cfg(feature = "arrow")] -mod arrow; #[cfg(test)] mod tests; @@ -24,10 +20,8 @@ use std::sync::Arc; use std::sync::OnceLock; use itertools::Itertools; -pub use mask_mut::*; use vortex_buffer::BitBuffer; use vortex_buffer::BitBufferMut; -use vortex_buffer::set_bit_unchecked; use vortex_error::VortexResult; use vortex_error::vortex_panic; @@ -620,12 +614,12 @@ impl Mask { return self; } - match self { + match &self { Mask::AllTrue(len) => { Self::from_iter([Self::new_true(limit), Self::new_false(len - limit)]) } Mask::AllFalse(_) => self, - Mask::Values(ref mask_values) => { + Mask::Values(mask_values) => { if limit >= mask_values.true_count() { return self; } @@ -635,11 +629,10 @@ impl Mask { let mut new_buffer_builder = BitBufferMut::new_unset(mask_values.len()); debug_assert!(limit < mask_values.len()); - let ptr = new_buffer_builder.as_mut_ptr(); for index in existing_buffer.set_indices().take(limit) { // SAFETY: We checked that `limit` was less than the mask values length, // therefore `index` must be within the bounds of the bit buffer. - unsafe { set_bit_unchecked(ptr, index) } + unsafe { new_buffer_builder.set_unchecked(index) } } Self::from(new_buffer_builder.freeze()) @@ -765,11 +758,6 @@ impl MaskValues { MaskIter::Indices(self.indices()) } } - - /// Extracts the internal [`BitBuffer`]. - pub(crate) fn into_buffer(self) -> BitBuffer { - self.buffer - } } /// Iterator over the indices or slices of a mask. diff --git a/vortex-mask/src/mask_mut.rs b/vortex-mask/src/mask_mut.rs deleted file mode 100644 index 91232ec9327..00000000000 --- a/vortex-mask/src/mask_mut.rs +++ /dev/null @@ -1,874 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -use std::sync::Arc; - -use vortex_buffer::BitBufferMut; -use vortex_error::vortex_panic; - -use crate::Mask; - -/// A mutable mask, used for lazily allocating the bit buffer as required. -#[derive(Debug, Clone)] -pub struct MaskMut(Inner); - -impl Default for MaskMut { - fn default() -> Self { - Self::empty() - } -} - -#[derive(Debug, Clone)] -enum Inner { - /// Initially, the mask is empty but may have some capacity. - Empty { capacity: usize }, - /// When the first value is pushed, the mask becomes constant. - Constant { - value: bool, - len: usize, - capacity: usize, - }, - /// When the first non-constant value is written, we allocate the bit buffer and switch - /// into the builder state. - Builder(BitBufferMut), -} - -impl MaskMut { - /// Creates a new empty mask. - pub fn empty() -> Self { - Self::with_capacity(0) - } - - /// Creates a new empty mask with the default capacity. - pub fn with_capacity(capacity: usize) -> Self { - Self(Inner::Empty { capacity }) - } - - /// Creates a new mask with the specified capacity. - pub fn new(len: usize, value: bool) -> Self { - Self(Inner::Constant { - value, - len, - capacity: len, - }) - } - - /// Creates a new mask with all values set to `true`. - pub fn new_true(len: usize) -> Self { - Self(Inner::Constant { - value: true, - len, - capacity: len, - }) - } - - /// Creates a new mask with all values set to `false`. - pub fn new_false(len: usize) -> Self { - Self(Inner::Constant { - value: false, - len, - capacity: len, - }) - } - - /// Creates a new mask from an existing bit buffer. - pub fn from_buffer(bit_buffer: BitBufferMut) -> Self { - Self(Inner::Builder(bit_buffer)) - } - - /// Returns the boolean value at a given index. - /// - /// # Panics - /// - /// Panics if the index is out of bounds. - pub fn value(&self, index: usize) -> bool { - match &self.0 { - Inner::Empty { .. } => { - vortex_panic!("index out of bounds: the length is 0 but the index is {index}") - } - Inner::Constant { value, len, .. } => { - assert!( - index < *len, - "index out of bounds: the length is {} but the index is {index}", - *len - ); - - *value - } - Inner::Builder(bit_buffer) => bit_buffer.value(index), - } - } - - /// Reserve capacity for at least `additional` more values to be appended. - pub fn reserve(&mut self, additional: usize) { - match &mut self.0 { - Inner::Empty { capacity } => { - *capacity += additional; - } - Inner::Constant { capacity, .. } => { - *capacity += additional; - } - Inner::Builder(bits) => { - bits.reserve(additional); - } - } - } - - /// Set the length of the mask. - /// - /// # Safety - /// - /// - `new_len` must be less than or equal to [`capacity()`]. - /// - The elements at `old_len..new_len` must be initialized. - /// - /// [`capacity()`]: Self::capacity - pub unsafe fn set_len(&mut self, new_len: usize) { - debug_assert!(new_len < self.capacity()); - match &mut self.0 { - Inner::Empty { capacity, .. } => { - self.0 = Inner::Constant { - value: false, // Pick any value - len: new_len, - capacity: *capacity, - } - } - Inner::Constant { len, .. } => { - *len = new_len; - } - Inner::Builder(bits) => { - unsafe { bits.set_len(new_len) }; - } - } - } - - /// Returns the capacity of the mask. - pub fn capacity(&self) -> usize { - match &self.0 { - Inner::Empty { capacity } => *capacity, - Inner::Constant { capacity, .. } => *capacity, - Inner::Builder(bits) => bits.capacity(), - } - } - - /// Clears the mask. - /// - /// Note that this method has no effect on the allocated capacity of the mask. - pub fn clear(&mut self) { - match &mut self.0 { - Inner::Empty { .. } => {} - Inner::Constant { capacity, .. } => { - self.0 = Inner::Empty { - capacity: *capacity, - } - } - Inner::Builder(bit_buffer) => bit_buffer.clear(), - }; - } - - /// Shortens the mask, keeping the first `len` bits. - /// - /// If `len` is greater or equal to the vector’s current length, this has no effect. - /// - /// Note that this method has no effect on the allocated capacity of the mask. - pub fn truncate(&mut self, len: usize) { - let truncated_len = len; - if truncated_len > self.len() { - return; - } - - match &mut self.0 { - Inner::Empty { .. } => {} - Inner::Constant { len, .. } => *len = truncated_len.min(*len), - Inner::Builder(bit_buffer) => bit_buffer.truncate(truncated_len), - }; - } - - /// Append n values to the mask. - pub fn append_n(&mut self, new_value: bool, n: usize) { - match &mut self.0 { - Inner::Empty { capacity } => { - self.0 = Inner::Constant { - value: new_value, - len: n, - capacity: (*capacity).max(n), - } - } - Inner::Constant { - value, - len, - capacity, - } => { - if *value == new_value { - // Same value, just increase length. - self.0 = Inner::Constant { - value: *value, - len: *len + n, - capacity: (*capacity).max(*len + n), - } - } else { - // Different value, need to allocate the bit buffer. - // Note: materialize() already appends the existing constant values - let bits = self.materialize(); - bits.append_n(new_value, n); - } - } - Inner::Builder(bits) => { - bits.append_n(new_value, n); - } - } - } - - /// Append a [`Mask`] to this mutable mask. - pub fn append_mask(&mut self, other: &Mask) { - match other { - Mask::AllTrue(len) => self.append_n(true, *len), - Mask::AllFalse(len) => self.append_n(false, *len), - Mask::Values(values) => { - let bitbuffer = values.buffer.clone(); - self.materialize().append_buffer(&bitbuffer); - } - } - } - - /// Ensures that the internal bit buffer is allocated and returns a mutable reference to it. - fn materialize(&mut self) -> &mut BitBufferMut { - let needs_materialization = !matches!(self.0, Inner::Builder(_)); - - if needs_materialization { - let new_builder = match &self.0 { - Inner::Empty { capacity } => BitBufferMut::with_capacity(*capacity), - Inner::Constant { - value, - len, - capacity, - } => { - let required_capacity = (*capacity).max(*len); - let mut bits = BitBufferMut::with_capacity(required_capacity); - bits.append_n(*value, *len); - bits - } - Inner::Builder(_) => unreachable!(), - }; - self.0 = Inner::Builder(new_builder); - } - - match &mut self.0 { - Inner::Builder(bits) => bits, - _ => unreachable!(), - } - } - - /// Split-off the mask at the given index, returning a new mask with the - /// values from `at` to the end, and leaving `self` with the values from - /// the start to `at`. - pub fn split_off(&mut self, at: usize) -> Self { - assert!(at <= self.capacity(), "split_off index out of bounds"); - match &mut self.0 { - Inner::Empty { capacity } => { - let new_capacity = *capacity - at; - *capacity = at; - Self(Inner::Empty { - capacity: new_capacity, - }) - } - Inner::Constant { - value, - len, - capacity, - } => { - // Adjust the lengths, given that length may be < at - let new_len = len.saturating_sub(at); - let new_capacity = *capacity - at; - *len = (*len).min(at); - *capacity = at; - - Self(Inner::Constant { - value: *value, - len: new_len, - capacity: new_capacity, - }) - } - Inner::Builder(bits) => { - let new_bits = bits.split_off(at); - Self(Inner::Builder(new_bits)) - } - } - } - - /// Absorb another mask into this one, appending its values. - pub fn unsplit(&mut self, other: Self) { - match other.0 { - Inner::Empty { .. } => { - // No work to do - } - Inner::Constant { value, len, .. } => { - self.append_n(value, len); - } - Inner::Builder(bits) => { - self.materialize().unsplit(bits); - } - } - } - - /// Freezes the mutable mask into an immutable one. - pub fn freeze(self) -> Mask { - match self.0 { - Inner::Empty { .. } => Mask::new_true(0), - Inner::Constant { value, len, .. } => { - if value { - Mask::new_true(len) - } else { - Mask::new_false(len) - } - } - Inner::Builder(bits) => Mask::from_buffer(bits.freeze()), - } - } - - /// Returns the logical length of the mask. - pub fn len(&self) -> usize { - match &self.0 { - Inner::Empty { .. } => 0, - Inner::Constant { len, .. } => *len, - Inner::Builder(bits) => bits.len(), - } - } - - /// Returns true if the mask is empty. - pub fn is_empty(&self) -> bool { - self.len() == 0 - } - - /// Returns true if all values in the mask are true. - pub fn all_true(&self) -> bool { - match &self.0 { - Inner::Empty { .. } => true, - Inner::Constant { value, .. } => *value, - Inner::Builder(bits) => bits.true_count() == bits.len(), - } - } - - /// Returns true if all values in the mask are false. - pub fn all_false(&self) -> bool { - match &self.0 { - Inner::Empty { .. } => true, - Inner::Constant { value, .. } => !*value, - Inner::Builder(bits) => !bits.is_empty() && bits.true_count() == 0, - } - } - - /// Returns the internal bit buffer if it exists. - pub fn as_bit_buffer_mut(&mut self) -> Option<&mut BitBufferMut> { - match &mut self.0 { - Inner::Builder(bits) => Some(bits), - _ => None, - } - } - - /// Set the value at the given index to true. - /// - /// # Panics - /// - /// Panics if the index is out of bounds. - pub fn set(&mut self, index: usize) { - self.set_to(index, true); - } - - /// Set the value at the given index to false. - /// - /// # Panics - /// - /// Panics if the index is out of bounds. - pub fn unset(&mut self, index: usize) { - self.set_to(index, false); - } - - /// Set the value at the given index to the specified boolean value. - /// - /// # Panics - /// - /// Panics if the index is out of bounds. - pub fn set_to(&mut self, index: usize, value: bool) { - match &mut self.0 { - Inner::Empty { .. } => { - vortex_panic!("index out of bounds: the length is 0 but the index is {index}") - } - Inner::Constant { - value: current_value, - len, - .. - } => { - assert!( - index < *len, - "index out of bounds: the length is {} but the index is {index}", - *len - ); - - if *current_value != value { - // Need to materialize the buffer since we're changing from constant. - self.materialize().set_to(index, value); - } - // If the value is the same as the constant, no action needed. - } - Inner::Builder(bit_buffer) => { - bit_buffer.set_to(index, value); - } - } - } - - /// Set the value at the given index to true without bounds checking. - /// - /// # Safety - /// - /// The caller must ensure that `index < self.len()`. - pub unsafe fn set_unchecked(&mut self, index: usize) { - unsafe { self.set_to_unchecked(index, true) } - } - - /// Set the value at the given index to false without bounds checking. - /// - /// # Safety - /// - /// The caller must ensure that `index < self.len()`. - pub unsafe fn unset_unchecked(&mut self, index: usize) { - unsafe { self.set_to_unchecked(index, false) } - } - - /// Set the value at the given index to the specified boolean value without bounds checking. - /// - /// # Safety - /// - /// The caller must ensure that `index < self.len()`. - pub unsafe fn set_to_unchecked(&mut self, index: usize, value: bool) { - unsafe { - match &mut self.0 { - Inner::Empty { .. } => { - // In debug mode, we still want to catch this error. - debug_assert!(false, "cannot set value in empty mask"); - } - Inner::Constant { - value: current_value, - len, - .. - } => { - debug_assert!( - index < *len, - "index out of bounds: the length is {} but the index is {index}", - *len - ); - - if *current_value != value { - // Need to materialize the buffer since we're changing from constant. - self.materialize().set_to_unchecked(index, value); - } - // If the value is the same as the constant, no action needed. - } - Inner::Builder(bit_buffer) => { - bit_buffer.set_to_unchecked(index, value); - } - } - } - } -} - -impl Mask { - /// Attempts to convert an immutable mask into a mutable one, returning an error of `Self` if - /// the underlying [`BitBuffer`](crate::BitBuffer) data if there are any other references. - pub fn try_into_mut(self) -> Result { - match self { - Mask::AllTrue(len) => Ok(MaskMut::new_true(len)), - Mask::AllFalse(len) => Ok(MaskMut::new_false(len)), - Mask::Values(values) => { - // We need to check for uniqueness twice, first for the `Arc` with `try_unwrap`, - // then for the internal `BitBuffer` with `try_into_mut`. - let owned_values = Arc::try_unwrap(values).map_err(Mask::Values)?; - let bit_buffer = owned_values.into_buffer(); - let mut_buffer = bit_buffer.try_into_mut().map_err(Mask::from_buffer)?; - - Ok(MaskMut(Inner::Builder(mut_buffer))) - } - } - } - - /// Convert an immutable mask into a mutable one, cloning the underlying - /// [`BitBuffer`](crate::BitBuffer) data if there are any other references. - pub fn into_mut(self) -> MaskMut { - match self { - Mask::AllTrue(len) => MaskMut::new_true(len), - Mask::AllFalse(len) => MaskMut::new_false(len), - Mask::Values(values) => { - let bit_buffer_mut = match Arc::try_unwrap(values) { - Ok(mask_values) => mask_values - .into_buffer() - .try_into_mut() - .unwrap_or_else(|bb| BitBufferMut::copy_from(&bb)), - Err(arc_mask_values) => { - let bit_buffer = arc_mask_values.bit_buffer(); - BitBufferMut::copy_from(bit_buffer) - } - }; - - MaskMut(Inner::Builder(bit_buffer_mut)) - } - } - } -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn test_split_off_empty() { - let mut mask = MaskMut::with_capacity(10); - assert_eq!(mask.len(), 0); - - let other = mask.split_off(0); - assert_eq!(mask.len(), 0); - assert_eq!(other.len(), 0); - } - - #[test] - fn test_split_off_constant_true_at_zero() { - let mut mask = MaskMut::new_true(10); - let other = mask.split_off(0); - - assert_eq!(mask.len(), 0); - assert_eq!(other.len(), 10); - - let frozen = other.freeze(); - assert_eq!(frozen.true_count(), 10); - } - - #[test] - fn test_split_off_constant_true_at_end() { - let mut mask = MaskMut::new_true(10); - let other = mask.split_off(10); - - assert_eq!(mask.len(), 10); - assert_eq!(other.len(), 0); - - let frozen = mask.freeze(); - assert_eq!(frozen.true_count(), 10); - } - - #[test] - fn test_split_off_constant_true_in_middle() { - let mut mask = MaskMut::new_true(10); - let other = mask.split_off(6); - - assert_eq!(mask.len(), 6); - assert_eq!(other.len(), 4); - - let frozen_first = mask.freeze(); - assert_eq!(frozen_first.true_count(), 6); - - let frozen_second = other.freeze(); - assert_eq!(frozen_second.true_count(), 4); - } - - #[test] - fn test_split_off_constant_false() { - let mut mask = MaskMut::new_false(20); - let other = mask.split_off(12); - - assert_eq!(mask.len(), 12); - assert_eq!(other.len(), 8); - - let frozen_first = mask.freeze(); - assert_eq!(frozen_first.true_count(), 0); - - let frozen_second = other.freeze(); - assert_eq!(frozen_second.true_count(), 0); - } - - // Note: Tests using BitBuffer operations are marked as ignored under miri - // because bitvec uses raw pointer operations that miri cannot verify. - #[test] - fn test_split_off_builder_at_byte_boundary() { - let mut mask = MaskMut::with_capacity(16); - // Create a pattern: 8 true, 8 false - mask.append_n(true, 8); - mask.append_n(false, 8); - - let mask_ptr = match &mask.0 { - Inner::Builder(bits) => bits.as_slice().as_ptr(), - _ => unreachable!(), - }; - - let other = mask.split_off(8); - - assert_eq!(mask.len(), 8); - assert_eq!(other.len(), 8); - - // Ensure the unsplit was zero-copy. - mask.unsplit(other); - let new_mask_ptr = match &mask.0 { - Inner::Builder(bits) => bits.as_slice().as_ptr(), - _ => unreachable!(), - }; - assert_eq!(mask_ptr, new_mask_ptr); - } - - #[test] - fn test_split_off_builder_not_byte_aligned() { - let mut mask = MaskMut::with_capacity(20); - // Create a pattern: 10 true, 10 false - mask.append_n(true, 10); - mask.append_n(false, 10); - - let other = mask.split_off(10); - - assert_eq!(mask.len(), 10); - assert_eq!(other.len(), 10); - - let frozen_first = mask.freeze(); - assert_eq!(frozen_first.true_count(), 10); - - let frozen_second = other.freeze(); - assert_eq!(frozen_second.true_count(), 0); - } - - #[test] - fn test_split_off_builder_mixed_pattern() { - let mut mask = MaskMut::with_capacity(15); - // Create pattern: TFTFTFTFTFTFTFT (alternating) - for i in 0..15 { - mask.append_n(i % 2 == 0, 1); - } - - let other = mask.split_off(7); - - assert_eq!(mask.len(), 7); - assert_eq!(other.len(), 8); - - let frozen_first = mask.freeze(); - assert_eq!(frozen_first.true_count(), 4); // positions 0,2,4,6 - - let frozen_second = other.freeze(); - assert_eq!(frozen_second.true_count(), 4); // positions 7,9,11,13 => 0,2,4,6 in split - } - - #[test] - fn test_unsplit_empty_with_empty() { - let mut mask = MaskMut::with_capacity(10); - let other = MaskMut::with_capacity(10); - - mask.unsplit(other); - assert_eq!(mask.len(), 0); - } - - #[test] - fn test_unsplit_empty_with_constant() { - let mut mask = MaskMut::with_capacity(10); - let other = MaskMut::new_true(5); - - mask.unsplit(other); - assert_eq!(mask.len(), 5); - - let frozen = mask.freeze(); - assert_eq!(frozen.true_count(), 5); - } - - #[test] - fn test_unsplit_constant_with_constant_same() { - let mut mask = MaskMut::new_true(5); - let other = MaskMut::new_true(5); - - mask.unsplit(other); - assert_eq!(mask.len(), 10); - - let frozen = mask.freeze(); - assert_eq!(frozen.true_count(), 10); - } - - #[test] - fn test_unsplit_constant_with_constant_different() { - let mut mask = MaskMut::new_true(5); - let other = MaskMut::new_false(5); - - mask.unsplit(other); - assert_eq!(mask.len(), 10); - - let frozen = mask.freeze(); - assert_eq!(frozen.true_count(), 5); - } - - #[test] - fn test_unsplit_constant_with_builder() { - let mut mask = MaskMut::new_true(5); - - let mut other = MaskMut::with_capacity(10); - other.append_n(true, 3); - other.append_n(false, 2); - - mask.unsplit(other); - assert_eq!(mask.len(), 10); - - let frozen = mask.freeze(); - assert_eq!(frozen.true_count(), 8); // 5 from first + 3 from second - } - - #[test] - fn test_unsplit_builder_with_constant() { - let mut mask = MaskMut::with_capacity(10); - mask.append_n(true, 3); - mask.append_n(false, 2); - - let other = MaskMut::new_true(5); - - mask.unsplit(other); - assert_eq!(mask.len(), 10); - - let frozen = mask.freeze(); - assert_eq!(frozen.true_count(), 8); // 3 from first + 5 from second - } - - #[test] - fn test_unsplit_builder_with_builder() { - let mut mask = MaskMut::with_capacity(10); - mask.append_n(true, 3); - mask.append_n(false, 2); - - let mut other = MaskMut::with_capacity(10); - other.append_n(false, 3); - other.append_n(true, 2); - - mask.unsplit(other); - assert_eq!(mask.len(), 10); - - let frozen = mask.freeze(); - assert_eq!(frozen.true_count(), 5); // 3 from first + 2 from second - } - - #[test] - fn test_round_trip_split_unsplit() { - let mut original = MaskMut::with_capacity(20); - // Pattern: 10 true, 10 false - original.append_n(true, 10); - original.append_n(false, 10); - - let original_frozen = original.freeze(); - let original_true_count = original_frozen.true_count(); - - // Convert back to mutable for split - let mut mask = original_frozen.try_into_mut().unwrap(); - - // Split at 10 - let other = mask.split_off(10); - - // Unsplit back together - mask.unsplit(other); - - assert_eq!(mask.len(), 20); - let frozen = mask.freeze(); - assert_eq!(frozen.true_count(), original_true_count); - } - - #[test] - #[should_panic(expected = "split_off index out of bounds")] - fn test_split_off_out_of_bounds() { - let mut mask = MaskMut::new_true(10); - mask.split_off(11); - } - - #[test] - fn test_split_off_builder_at_bit_1() { - let mut mask = MaskMut::with_capacity(16); - mask.append_n(true, 16); - - let other = mask.split_off(1); - - assert_eq!(mask.len(), 1); - assert_eq!(other.len(), 15); - - let frozen_first = mask.freeze(); - assert_eq!(frozen_first.true_count(), 1); - - let frozen_second = other.freeze(); - assert_eq!(frozen_second.true_count(), 15); - } - - #[test] - fn test_multiple_split_unsplit() { - let mut mask = MaskMut::new_true(30); - - // Split into 3 parts - let third = mask.split_off(20); // 20-30 - let second = mask.split_off(10); // 10-20 - // first is 0-10 - - assert_eq!(mask.len(), 10); - assert_eq!(second.len(), 10); - assert_eq!(third.len(), 10); - - // Recombine in order - mask.unsplit(second); - mask.unsplit(third); - - assert_eq!(mask.len(), 30); - let frozen = mask.freeze(); - assert_eq!(frozen.true_count(), 30); - } - - #[test] - fn test_try_into_mut_all_variants() { - // Test AllTrue and AllFalse variants. - let mask_true = Mask::new_true(100); - let mut_mask_true = mask_true.try_into_mut().unwrap(); - assert_eq!(mut_mask_true.len(), 100); - assert_eq!(mut_mask_true.freeze().true_count(), 100); - - let mask_false = Mask::new_false(50); - let mut_mask_false = mask_false.try_into_mut().unwrap(); - assert_eq!(mut_mask_false.len(), 50); - assert_eq!(mut_mask_false.freeze().true_count(), 0); - } - - #[test] - fn test_try_into_mut_with_references() { - // Create a MaskValues variant. - let mut mask_mut = MaskMut::with_capacity(10); - mask_mut.append_n(true, 5); - mask_mut.append_n(false, 5); - let mask = mask_mut.freeze(); - - // Should succeed with unique reference (no clones). - let mask2 = { - let mut mask_mut2 = MaskMut::with_capacity(10); - mask_mut2.append_n(true, 5); - mask_mut2.append_n(false, 5); - mask_mut2.freeze() - }; - let result = mask2.try_into_mut(); - assert!(result.is_ok()); - assert_eq!(result.unwrap().len(), 10); - - // Should fail with shared references. - let _cloned = mask.clone(); - let result = mask.try_into_mut(); - assert!(result.is_err()); - if let Err(returned_mask) = result { - assert_eq!(returned_mask.len(), 10); - assert_eq!(returned_mask.true_count(), 5); - } - } - - #[test] - fn test_try_into_mut_round_trip() { - // Test freeze -> try_into_mut -> modify -> freeze cycle. - let mut original = MaskMut::with_capacity(20); - original.append_n(true, 10); - original.append_n(false, 10); - - let frozen = original.freeze(); - assert_eq!(frozen.true_count(), 10); - - let mut mut_mask = frozen.try_into_mut().unwrap(); - mut_mask.append_n(true, 5); - assert_eq!(mut_mask.len(), 25); - - let frozen_again = mut_mask.freeze(); - assert_eq!(frozen_again.true_count(), 15); - } -} diff --git a/vortex-mask/src/tests.rs b/vortex-mask/src/tests.rs index a13827abbc3..ff0bf04f76a 100644 --- a/vortex-mask/src/tests.rs +++ b/vortex-mask/src/tests.rs @@ -1,9 +1,6 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::panic)] -#![allow(clippy::many_single_char_names)] - use rstest::rstest; use vortex_buffer::BitBuffer; diff --git a/vortex-metrics/Cargo.toml b/vortex-metrics/Cargo.toml index fe8a783a4b5..a9d98eb448c 100644 --- a/vortex-metrics/Cargo.toml +++ b/vortex-metrics/Cargo.toml @@ -14,12 +14,8 @@ rust-version = { workspace = true } version = { workspace = true } [dependencies] -getrandom_v03 = { workspace = true } parking_lot = { workspace = true } sketches-ddsketch = { workspace = true } [lints] workspace = true - -[package.metadata.cargo-machete] -ignored = ["getrandom_v03"] diff --git a/vortex-metrics/public-api.lock b/vortex-metrics/public-api.lock index cf2d9077b34..337559cfbf8 100644 --- a/vortex-metrics/public-api.lock +++ b/vortex-metrics/public-api.lock @@ -4,7 +4,7 @@ pub mod vortex_metrics::tracing pub fn vortex_metrics::tracing::get_global_labels() -> alloc::vec::Vec<(&'static str, alloc::string::String)> -pub fn vortex_metrics::tracing::set_global_labels(i: I) where I: core::iter::traits::collect::IntoIterator +pub fn vortex_metrics::tracing::set_global_labels(I) where I: core::iter::traits::collect::IntoIterator pub enum vortex_metrics::MetricValue @@ -22,29 +22,29 @@ pub fn vortex_metrics::MetricValue::clone(&self) -> vortex_metrics::MetricValue impl core::convert::From for vortex_metrics::MetricValue -pub fn vortex_metrics::MetricValue::from(value: vortex_metrics::Counter) -> Self +pub fn vortex_metrics::MetricValue::from(vortex_metrics::Counter) -> Self impl core::convert::From for vortex_metrics::MetricValue -pub fn vortex_metrics::MetricValue::from(value: vortex_metrics::Gauge) -> Self +pub fn vortex_metrics::MetricValue::from(vortex_metrics::Gauge) -> Self impl core::convert::From for vortex_metrics::MetricValue -pub fn vortex_metrics::MetricValue::from(value: vortex_metrics::Histogram) -> Self +pub fn vortex_metrics::MetricValue::from(vortex_metrics::Histogram) -> Self impl core::convert::From for vortex_metrics::MetricValue -pub fn vortex_metrics::MetricValue::from(value: vortex_metrics::Timer) -> Self +pub fn vortex_metrics::MetricValue::from(vortex_metrics::Timer) -> Self impl core::fmt::Debug for vortex_metrics::MetricValue -pub fn vortex_metrics::MetricValue::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_metrics::MetricValue::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_metrics::Counter(_) impl vortex_metrics::Counter -pub fn vortex_metrics::Counter::add(&self, value: u64) +pub fn vortex_metrics::Counter::add(&self, u64) pub fn vortex_metrics::Counter::value(&self) -> u64 @@ -54,11 +54,11 @@ pub fn vortex_metrics::Counter::clone(&self) -> vortex_metrics::Counter impl core::convert::From for vortex_metrics::MetricValue -pub fn vortex_metrics::MetricValue::from(value: vortex_metrics::Counter) -> Self +pub fn vortex_metrics::MetricValue::from(vortex_metrics::Counter) -> Self impl core::fmt::Debug for vortex_metrics::Counter -pub fn vortex_metrics::Counter::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_metrics::Counter::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_metrics::DefaultMetricsRegistry @@ -72,13 +72,13 @@ pub fn vortex_metrics::DefaultMetricsRegistry::default() -> vortex_metrics::Defa impl vortex_metrics::MetricsRegistry for vortex_metrics::DefaultMetricsRegistry -pub fn vortex_metrics::DefaultMetricsRegistry::register_counter(&self, name: alloc::borrow::Cow<'static, str>, labels: alloc::vec::Vec) -> vortex_metrics::Counter +pub fn vortex_metrics::DefaultMetricsRegistry::register_counter(&self, alloc::borrow::Cow<'static, str>, alloc::vec::Vec) -> vortex_metrics::Counter -pub fn vortex_metrics::DefaultMetricsRegistry::register_gauge(&self, name: alloc::borrow::Cow<'static, str>, labels: alloc::vec::Vec) -> vortex_metrics::Gauge +pub fn vortex_metrics::DefaultMetricsRegistry::register_gauge(&self, alloc::borrow::Cow<'static, str>, alloc::vec::Vec) -> vortex_metrics::Gauge -pub fn vortex_metrics::DefaultMetricsRegistry::register_histogram(&self, name: alloc::borrow::Cow<'static, str>, labels: alloc::vec::Vec) -> vortex_metrics::Histogram +pub fn vortex_metrics::DefaultMetricsRegistry::register_histogram(&self, alloc::borrow::Cow<'static, str>, alloc::vec::Vec) -> vortex_metrics::Histogram -pub fn vortex_metrics::DefaultMetricsRegistry::register_timer(&self, name: alloc::borrow::Cow<'static, str>, labels: alloc::vec::Vec) -> vortex_metrics::Timer +pub fn vortex_metrics::DefaultMetricsRegistry::register_timer(&self, alloc::borrow::Cow<'static, str>, alloc::vec::Vec) -> vortex_metrics::Timer pub fn vortex_metrics::DefaultMetricsRegistry::snapshot(&self) -> alloc::vec::Vec @@ -86,11 +86,11 @@ pub struct vortex_metrics::Gauge(_) impl vortex_metrics::Gauge -pub fn vortex_metrics::Gauge::decrement(&self, value: f64) +pub fn vortex_metrics::Gauge::decrement(&self, f64) -pub fn vortex_metrics::Gauge::increment(&self, value: f64) +pub fn vortex_metrics::Gauge::increment(&self, f64) -pub fn vortex_metrics::Gauge::set(&self, value: f64) +pub fn vortex_metrics::Gauge::set(&self, f64) pub fn vortex_metrics::Gauge::value(&self) -> f64 @@ -100,11 +100,11 @@ pub fn vortex_metrics::Gauge::clone(&self) -> vortex_metrics::Gauge impl core::convert::From for vortex_metrics::MetricValue -pub fn vortex_metrics::MetricValue::from(value: vortex_metrics::Gauge) -> Self +pub fn vortex_metrics::MetricValue::from(vortex_metrics::Gauge) -> Self impl core::fmt::Debug for vortex_metrics::Gauge -pub fn vortex_metrics::Gauge::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_metrics::Gauge::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_metrics::Histogram(_) @@ -114,11 +114,11 @@ pub fn vortex_metrics::Histogram::count(&self) -> usize pub fn vortex_metrics::Histogram::is_empty(&self) -> bool -pub fn vortex_metrics::Histogram::quantile(&self, quantile: f64) -> core::option::Option +pub fn vortex_metrics::Histogram::quantile(&self, f64) -> core::option::Option pub fn vortex_metrics::Histogram::total(&self) -> f64 -pub fn vortex_metrics::Histogram::update(&self, value: f64) +pub fn vortex_metrics::Histogram::update(&self, f64) impl core::clone::Clone for vortex_metrics::Histogram @@ -126,7 +126,7 @@ pub fn vortex_metrics::Histogram::clone(&self) -> vortex_metrics::Histogram impl core::convert::From for vortex_metrics::MetricValue -pub fn vortex_metrics::MetricValue::from(value: vortex_metrics::Histogram) -> Self +pub fn vortex_metrics::MetricValue::from(vortex_metrics::Histogram) -> Self impl core::default::Default for vortex_metrics::Histogram @@ -134,7 +134,7 @@ pub fn vortex_metrics::Histogram::default() -> vortex_metrics::Histogram impl core::fmt::Debug for vortex_metrics::Histogram -pub fn vortex_metrics::Histogram::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_metrics::Histogram::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_metrics::Label @@ -142,7 +142,7 @@ impl vortex_metrics::Label pub fn vortex_metrics::Label::key(&self) -> &str -pub fn vortex_metrics::Label::new(key: impl core::convert::Into>, value: impl core::convert::Into>) -> Self +pub fn vortex_metrics::Label::new(impl core::convert::Into>, impl core::convert::Into>) -> Self pub fn vortex_metrics::Label::value(&self) -> &str @@ -152,15 +152,15 @@ pub fn vortex_metrics::Label::clone(&self) -> vortex_metrics::Label impl core::fmt::Debug for vortex_metrics::Label -pub fn vortex_metrics::Label::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_metrics::Label::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_metrics::Label -pub fn vortex_metrics::Label::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_metrics::Label::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::convert::From<(K, V)> for vortex_metrics::Label where K: core::convert::Into>, V: core::convert::Into> -pub fn vortex_metrics::Label::from(value: (K, V)) -> Self +pub fn vortex_metrics::Label::from((K, V)) -> Self pub struct vortex_metrics::Metric @@ -178,25 +178,25 @@ pub fn vortex_metrics::Metric::clone(&self) -> vortex_metrics::Metric impl core::fmt::Debug for vortex_metrics::Metric -pub fn vortex_metrics::Metric::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_metrics::Metric::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_metrics::MetricBuilder<'s> impl<'r> vortex_metrics::MetricBuilder<'r> -pub fn vortex_metrics::MetricBuilder<'r>::add_label(self, key: K, value: V) -> Self where K: core::convert::Into>, V: core::convert::Into> +pub fn vortex_metrics::MetricBuilder<'r>::add_label(self, K, V) -> Self where K: core::convert::Into>, V: core::convert::Into> -pub fn vortex_metrics::MetricBuilder<'r>::add_labels(self, labels: I) -> Self where I: core::iter::traits::collect::IntoIterator, L: core::convert::Into +pub fn vortex_metrics::MetricBuilder<'r>::add_labels(self, I) -> Self where I: core::iter::traits::collect::IntoIterator, L: core::convert::Into -pub fn vortex_metrics::MetricBuilder<'r>::counter(self, name: impl core::convert::Into>) -> vortex_metrics::Counter +pub fn vortex_metrics::MetricBuilder<'r>::counter(self, impl core::convert::Into>) -> vortex_metrics::Counter -pub fn vortex_metrics::MetricBuilder<'r>::gauge(self, name: impl core::convert::Into>) -> vortex_metrics::Gauge +pub fn vortex_metrics::MetricBuilder<'r>::gauge(self, impl core::convert::Into>) -> vortex_metrics::Gauge -pub fn vortex_metrics::MetricBuilder<'r>::histogram(self, name: impl core::convert::Into>) -> vortex_metrics::Histogram +pub fn vortex_metrics::MetricBuilder<'r>::histogram(self, impl core::convert::Into>) -> vortex_metrics::Histogram -pub fn vortex_metrics::MetricBuilder<'r>::new(registry: &'r dyn vortex_metrics::MetricsRegistry) -> Self +pub fn vortex_metrics::MetricBuilder<'r>::new(&'r dyn vortex_metrics::MetricsRegistry) -> Self -pub fn vortex_metrics::MetricBuilder<'r>::timer(self, name: impl core::convert::Into>) -> vortex_metrics::Timer +pub fn vortex_metrics::MetricBuilder<'r>::timer(self, impl core::convert::Into>) -> vortex_metrics::Timer pub struct vortex_metrics::TimeGuard<'a> @@ -212,13 +212,13 @@ pub fn vortex_metrics::Timer::count(&self) -> usize pub fn vortex_metrics::Timer::is_empty(&self) -> bool -pub fn vortex_metrics::Timer::quantile(&self, quantile: f64) -> core::option::Option +pub fn vortex_metrics::Timer::quantile(&self, f64) -> core::option::Option pub fn vortex_metrics::Timer::time(&self) -> vortex_metrics::TimeGuard<'_> pub fn vortex_metrics::Timer::total(&self) -> core::time::Duration -pub fn vortex_metrics::Timer::update(&self, duration: core::time::Duration) +pub fn vortex_metrics::Timer::update(&self, core::time::Duration) impl core::clone::Clone for vortex_metrics::Timer @@ -226,7 +226,7 @@ pub fn vortex_metrics::Timer::clone(&self) -> vortex_metrics::Timer impl core::convert::From for vortex_metrics::MetricValue -pub fn vortex_metrics::MetricValue::from(value: vortex_metrics::Timer) -> Self +pub fn vortex_metrics::MetricValue::from(vortex_metrics::Timer) -> Self impl core::default::Default for vortex_metrics::Timer @@ -234,28 +234,28 @@ pub fn vortex_metrics::Timer::default() -> vortex_metrics::Timer impl core::fmt::Debug for vortex_metrics::Timer -pub fn vortex_metrics::Timer::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_metrics::Timer::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub trait vortex_metrics::MetricsRegistry: core::marker::Send + core::marker::Sync -pub fn vortex_metrics::MetricsRegistry::register_counter(&self, name: alloc::borrow::Cow<'static, str>, labels: alloc::vec::Vec) -> vortex_metrics::Counter +pub fn vortex_metrics::MetricsRegistry::register_counter(&self, alloc::borrow::Cow<'static, str>, alloc::vec::Vec) -> vortex_metrics::Counter -pub fn vortex_metrics::MetricsRegistry::register_gauge(&self, name: alloc::borrow::Cow<'static, str>, labels: alloc::vec::Vec) -> vortex_metrics::Gauge +pub fn vortex_metrics::MetricsRegistry::register_gauge(&self, alloc::borrow::Cow<'static, str>, alloc::vec::Vec) -> vortex_metrics::Gauge -pub fn vortex_metrics::MetricsRegistry::register_histogram(&self, name: alloc::borrow::Cow<'static, str>, labels: alloc::vec::Vec) -> vortex_metrics::Histogram +pub fn vortex_metrics::MetricsRegistry::register_histogram(&self, alloc::borrow::Cow<'static, str>, alloc::vec::Vec) -> vortex_metrics::Histogram -pub fn vortex_metrics::MetricsRegistry::register_timer(&self, name: alloc::borrow::Cow<'static, str>, labels: alloc::vec::Vec) -> vortex_metrics::Timer +pub fn vortex_metrics::MetricsRegistry::register_timer(&self, alloc::borrow::Cow<'static, str>, alloc::vec::Vec) -> vortex_metrics::Timer pub fn vortex_metrics::MetricsRegistry::snapshot(&self) -> alloc::vec::Vec impl vortex_metrics::MetricsRegistry for vortex_metrics::DefaultMetricsRegistry -pub fn vortex_metrics::DefaultMetricsRegistry::register_counter(&self, name: alloc::borrow::Cow<'static, str>, labels: alloc::vec::Vec) -> vortex_metrics::Counter +pub fn vortex_metrics::DefaultMetricsRegistry::register_counter(&self, alloc::borrow::Cow<'static, str>, alloc::vec::Vec) -> vortex_metrics::Counter -pub fn vortex_metrics::DefaultMetricsRegistry::register_gauge(&self, name: alloc::borrow::Cow<'static, str>, labels: alloc::vec::Vec) -> vortex_metrics::Gauge +pub fn vortex_metrics::DefaultMetricsRegistry::register_gauge(&self, alloc::borrow::Cow<'static, str>, alloc::vec::Vec) -> vortex_metrics::Gauge -pub fn vortex_metrics::DefaultMetricsRegistry::register_histogram(&self, name: alloc::borrow::Cow<'static, str>, labels: alloc::vec::Vec) -> vortex_metrics::Histogram +pub fn vortex_metrics::DefaultMetricsRegistry::register_histogram(&self, alloc::borrow::Cow<'static, str>, alloc::vec::Vec) -> vortex_metrics::Histogram -pub fn vortex_metrics::DefaultMetricsRegistry::register_timer(&self, name: alloc::borrow::Cow<'static, str>, labels: alloc::vec::Vec) -> vortex_metrics::Timer +pub fn vortex_metrics::DefaultMetricsRegistry::register_timer(&self, alloc::borrow::Cow<'static, str>, alloc::vec::Vec) -> vortex_metrics::Timer pub fn vortex_metrics::DefaultMetricsRegistry::snapshot(&self) -> alloc::vec::Vec diff --git a/vortex-metrics/src/histogram.rs b/vortex-metrics/src/histogram.rs index 5172369fe08..ea4728ce8d7 100644 --- a/vortex-metrics/src/histogram.rs +++ b/vortex-metrics/src/histogram.rs @@ -31,8 +31,7 @@ impl Histogram { /// Returns the estimated quantile value, which must be in the [0.0, 1.0] range, will panic otherwise. /// Returns `None` if the histogram is empty. - #[allow(clippy::expect_used)] - #[allow(clippy::unwrap_in_result)] + #[expect(clippy::expect_used)] pub fn quantile(&self, quantile: f64) -> Option { assert!( (0.0..=1.0).contains(&quantile), diff --git a/vortex-metrics/src/timer.rs b/vortex-metrics/src/timer.rs index 0bf5545ec77..619d7ceb178 100644 --- a/vortex-metrics/src/timer.rs +++ b/vortex-metrics/src/timer.rs @@ -42,8 +42,7 @@ impl Timer { /// Returns the estimated quantile value, which must be in the [0.0, 1.0] range, will panic otherwise. /// Returns `None` if the timer is empty. - #[allow(clippy::expect_used)] - #[allow(clippy::unwrap_in_result)] + #[expect(clippy::expect_used)] pub fn quantile(&self, quantile: f64) -> Option { assert!( (0.0..=1.0).contains(&quantile), diff --git a/vortex-proto/public-api.lock b/vortex-proto/public-api.lock index 045b53d17eb..5b0beef0153 100644 --- a/vortex-proto/public-api.lock +++ b/vortex-proto/public-api.lock @@ -30,11 +30,11 @@ pub vortex_proto::dtype::d_type::DtypeType::Variant(vortex_proto::dtype::Variant impl vortex_proto::dtype::d_type::DtypeType -pub fn vortex_proto::dtype::d_type::DtypeType::encode(&self, buf: &mut impl bytes::buf::buf_mut::BufMut) +pub fn vortex_proto::dtype::d_type::DtypeType::encode(&self, &mut impl bytes::buf::buf_mut::BufMut) pub fn vortex_proto::dtype::d_type::DtypeType::encoded_len(&self) -> usize -pub fn vortex_proto::dtype::d_type::DtypeType::merge(field: &mut core::option::Option, tag: u32, wire_type: prost::encoding::wire_type::WireType, buf: &mut impl bytes::buf::buf_impl::Buf, ctx: prost::encoding::DecodeContext) -> core::result::Result<(), prost::error::DecodeError> +pub fn vortex_proto::dtype::d_type::DtypeType::merge(&mut core::option::Option, u32, prost::encoding::wire_type::WireType, &mut impl bytes::buf::buf_impl::Buf, prost::encoding::DecodeContext) -> core::result::Result<(), prost::error::DecodeError> impl core::clone::Clone for vortex_proto::dtype::d_type::DtypeType @@ -42,11 +42,11 @@ pub fn vortex_proto::dtype::d_type::DtypeType::clone(&self) -> vortex_proto::dty impl core::cmp::PartialEq for vortex_proto::dtype::d_type::DtypeType -pub fn vortex_proto::dtype::d_type::DtypeType::eq(&self, other: &vortex_proto::dtype::d_type::DtypeType) -> bool +pub fn vortex_proto::dtype::d_type::DtypeType::eq(&self, &vortex_proto::dtype::d_type::DtypeType) -> bool impl core::fmt::Debug for vortex_proto::dtype::d_type::DtypeType -pub fn vortex_proto::dtype::d_type::DtypeType::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::dtype::d_type::DtypeType::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::StructuralPartialEq for vortex_proto::dtype::d_type::DtypeType @@ -58,11 +58,11 @@ pub vortex_proto::dtype::field::FieldType::Name(alloc::string::String) impl vortex_proto::dtype::field::FieldType -pub fn vortex_proto::dtype::field::FieldType::encode(&self, buf: &mut impl bytes::buf::buf_mut::BufMut) +pub fn vortex_proto::dtype::field::FieldType::encode(&self, &mut impl bytes::buf::buf_mut::BufMut) pub fn vortex_proto::dtype::field::FieldType::encoded_len(&self) -> usize -pub fn vortex_proto::dtype::field::FieldType::merge(field: &mut core::option::Option, tag: u32, wire_type: prost::encoding::wire_type::WireType, buf: &mut impl bytes::buf::buf_impl::Buf, ctx: prost::encoding::DecodeContext) -> core::result::Result<(), prost::error::DecodeError> +pub fn vortex_proto::dtype::field::FieldType::merge(&mut core::option::Option, u32, prost::encoding::wire_type::WireType, &mut impl bytes::buf::buf_impl::Buf, prost::encoding::DecodeContext) -> core::result::Result<(), prost::error::DecodeError> impl core::clone::Clone for vortex_proto::dtype::field::FieldType @@ -72,15 +72,15 @@ impl core::cmp::Eq for vortex_proto::dtype::field::FieldType impl core::cmp::PartialEq for vortex_proto::dtype::field::FieldType -pub fn vortex_proto::dtype::field::FieldType::eq(&self, other: &vortex_proto::dtype::field::FieldType) -> bool +pub fn vortex_proto::dtype::field::FieldType::eq(&self, &vortex_proto::dtype::field::FieldType) -> bool impl core::fmt::Debug for vortex_proto::dtype::field::FieldType -pub fn vortex_proto::dtype::field::FieldType::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::dtype::field::FieldType::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_proto::dtype::field::FieldType -pub fn vortex_proto::dtype::field::FieldType::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_proto::dtype::field::FieldType::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_proto::dtype::field::FieldType @@ -112,13 +112,13 @@ impl vortex_proto::dtype::PType pub fn vortex_proto::dtype::PType::as_str_name(&self) -> &'static str -pub fn vortex_proto::dtype::PType::from_str_name(value: &str) -> core::option::Option +pub fn vortex_proto::dtype::PType::from_str_name(&str) -> core::option::Option impl vortex_proto::dtype::PType -pub fn vortex_proto::dtype::PType::from_i32(value: i32) -> core::option::Option +pub fn vortex_proto::dtype::PType::from_i32(i32) -> core::option::Option -pub fn vortex_proto::dtype::PType::is_valid(value: i32) -> bool +pub fn vortex_proto::dtype::PType::is_valid(i32) -> bool impl core::clone::Clone for vortex_proto::dtype::PType @@ -128,25 +128,25 @@ impl core::cmp::Eq for vortex_proto::dtype::PType impl core::cmp::Ord for vortex_proto::dtype::PType -pub fn vortex_proto::dtype::PType::cmp(&self, other: &vortex_proto::dtype::PType) -> core::cmp::Ordering +pub fn vortex_proto::dtype::PType::cmp(&self, &vortex_proto::dtype::PType) -> core::cmp::Ordering impl core::cmp::PartialEq for vortex_proto::dtype::PType -pub fn vortex_proto::dtype::PType::eq(&self, other: &vortex_proto::dtype::PType) -> bool +pub fn vortex_proto::dtype::PType::eq(&self, &vortex_proto::dtype::PType) -> bool impl core::cmp::PartialOrd for vortex_proto::dtype::PType -pub fn vortex_proto::dtype::PType::partial_cmp(&self, other: &vortex_proto::dtype::PType) -> core::option::Option +pub fn vortex_proto::dtype::PType::partial_cmp(&self, &vortex_proto::dtype::PType) -> core::option::Option impl core::convert::From for i32 -pub fn i32::from(value: vortex_proto::dtype::PType) -> i32 +pub fn i32::from(vortex_proto::dtype::PType) -> i32 impl core::convert::TryFrom for vortex_proto::dtype::PType pub type vortex_proto::dtype::PType::Error = prost::error::UnknownEnumValue -pub fn vortex_proto::dtype::PType::try_from(value: i32) -> core::result::Result +pub fn vortex_proto::dtype::PType::try_from(i32) -> core::result::Result impl core::default::Default for vortex_proto::dtype::PType @@ -154,11 +154,11 @@ pub fn vortex_proto::dtype::PType::default() -> vortex_proto::dtype::PType impl core::fmt::Debug for vortex_proto::dtype::PType -pub fn vortex_proto::dtype::PType::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::dtype::PType::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_proto::dtype::PType -pub fn vortex_proto::dtype::PType::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_proto::dtype::PType::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_proto::dtype::PType @@ -176,7 +176,7 @@ impl core::cmp::Eq for vortex_proto::dtype::Binary impl core::cmp::PartialEq for vortex_proto::dtype::Binary -pub fn vortex_proto::dtype::Binary::eq(&self, other: &vortex_proto::dtype::Binary) -> bool +pub fn vortex_proto::dtype::Binary::eq(&self, &vortex_proto::dtype::Binary) -> bool impl core::default::Default for vortex_proto::dtype::Binary @@ -184,11 +184,11 @@ pub fn vortex_proto::dtype::Binary::default() -> Self impl core::fmt::Debug for vortex_proto::dtype::Binary -pub fn vortex_proto::dtype::Binary::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::dtype::Binary::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_proto::dtype::Binary -pub fn vortex_proto::dtype::Binary::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_proto::dtype::Binary::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_proto::dtype::Binary @@ -212,7 +212,7 @@ impl core::cmp::Eq for vortex_proto::dtype::Bool impl core::cmp::PartialEq for vortex_proto::dtype::Bool -pub fn vortex_proto::dtype::Bool::eq(&self, other: &vortex_proto::dtype::Bool) -> bool +pub fn vortex_proto::dtype::Bool::eq(&self, &vortex_proto::dtype::Bool) -> bool impl core::default::Default for vortex_proto::dtype::Bool @@ -220,11 +220,11 @@ pub fn vortex_proto::dtype::Bool::default() -> Self impl core::fmt::Debug for vortex_proto::dtype::Bool -pub fn vortex_proto::dtype::Bool::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::dtype::Bool::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_proto::dtype::Bool -pub fn vortex_proto::dtype::Bool::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_proto::dtype::Bool::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_proto::dtype::Bool @@ -246,7 +246,7 @@ pub fn vortex_proto::dtype::DType::clone(&self) -> vortex_proto::dtype::DType impl core::cmp::PartialEq for vortex_proto::dtype::DType -pub fn vortex_proto::dtype::DType::eq(&self, other: &vortex_proto::dtype::DType) -> bool +pub fn vortex_proto::dtype::DType::eq(&self, &vortex_proto::dtype::DType) -> bool impl core::default::Default for vortex_proto::dtype::DType @@ -254,7 +254,7 @@ pub fn vortex_proto::dtype::DType::default() -> Self impl core::fmt::Debug for vortex_proto::dtype::DType -pub fn vortex_proto::dtype::DType::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::dtype::DType::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::StructuralPartialEq for vortex_proto::dtype::DType @@ -280,7 +280,7 @@ impl core::cmp::Eq for vortex_proto::dtype::Decimal impl core::cmp::PartialEq for vortex_proto::dtype::Decimal -pub fn vortex_proto::dtype::Decimal::eq(&self, other: &vortex_proto::dtype::Decimal) -> bool +pub fn vortex_proto::dtype::Decimal::eq(&self, &vortex_proto::dtype::Decimal) -> bool impl core::default::Default for vortex_proto::dtype::Decimal @@ -288,11 +288,11 @@ pub fn vortex_proto::dtype::Decimal::default() -> Self impl core::fmt::Debug for vortex_proto::dtype::Decimal -pub fn vortex_proto::dtype::Decimal::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::dtype::Decimal::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_proto::dtype::Decimal -pub fn vortex_proto::dtype::Decimal::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_proto::dtype::Decimal::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_proto::dtype::Decimal @@ -322,7 +322,7 @@ pub fn vortex_proto::dtype::Extension::clone(&self) -> vortex_proto::dtype::Exte impl core::cmp::PartialEq for vortex_proto::dtype::Extension -pub fn vortex_proto::dtype::Extension::eq(&self, other: &vortex_proto::dtype::Extension) -> bool +pub fn vortex_proto::dtype::Extension::eq(&self, &vortex_proto::dtype::Extension) -> bool impl core::default::Default for vortex_proto::dtype::Extension @@ -330,7 +330,7 @@ pub fn vortex_proto::dtype::Extension::default() -> Self impl core::fmt::Debug for vortex_proto::dtype::Extension -pub fn vortex_proto::dtype::Extension::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::dtype::Extension::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::StructuralPartialEq for vortex_proto::dtype::Extension @@ -352,7 +352,7 @@ impl core::cmp::Eq for vortex_proto::dtype::Field impl core::cmp::PartialEq for vortex_proto::dtype::Field -pub fn vortex_proto::dtype::Field::eq(&self, other: &vortex_proto::dtype::Field) -> bool +pub fn vortex_proto::dtype::Field::eq(&self, &vortex_proto::dtype::Field) -> bool impl core::default::Default for vortex_proto::dtype::Field @@ -360,11 +360,11 @@ pub fn vortex_proto::dtype::Field::default() -> Self impl core::fmt::Debug for vortex_proto::dtype::Field -pub fn vortex_proto::dtype::Field::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::dtype::Field::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_proto::dtype::Field -pub fn vortex_proto::dtype::Field::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_proto::dtype::Field::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_proto::dtype::Field @@ -384,7 +384,7 @@ pub fn vortex_proto::dtype::FieldPath::clone(&self) -> vortex_proto::dtype::Fiel impl core::cmp::PartialEq for vortex_proto::dtype::FieldPath -pub fn vortex_proto::dtype::FieldPath::eq(&self, other: &vortex_proto::dtype::FieldPath) -> bool +pub fn vortex_proto::dtype::FieldPath::eq(&self, &vortex_proto::dtype::FieldPath) -> bool impl core::default::Default for vortex_proto::dtype::FieldPath @@ -392,7 +392,7 @@ pub fn vortex_proto::dtype::FieldPath::default() -> Self impl core::fmt::Debug for vortex_proto::dtype::FieldPath -pub fn vortex_proto::dtype::FieldPath::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::dtype::FieldPath::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::StructuralPartialEq for vortex_proto::dtype::FieldPath @@ -416,7 +416,7 @@ pub fn vortex_proto::dtype::FixedSizeList::clone(&self) -> vortex_proto::dtype:: impl core::cmp::PartialEq for vortex_proto::dtype::FixedSizeList -pub fn vortex_proto::dtype::FixedSizeList::eq(&self, other: &vortex_proto::dtype::FixedSizeList) -> bool +pub fn vortex_proto::dtype::FixedSizeList::eq(&self, &vortex_proto::dtype::FixedSizeList) -> bool impl core::default::Default for vortex_proto::dtype::FixedSizeList @@ -424,7 +424,7 @@ pub fn vortex_proto::dtype::FixedSizeList::default() -> Self impl core::fmt::Debug for vortex_proto::dtype::FixedSizeList -pub fn vortex_proto::dtype::FixedSizeList::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::dtype::FixedSizeList::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::StructuralPartialEq for vortex_proto::dtype::FixedSizeList @@ -446,7 +446,7 @@ pub fn vortex_proto::dtype::List::clone(&self) -> vortex_proto::dtype::List impl core::cmp::PartialEq for vortex_proto::dtype::List -pub fn vortex_proto::dtype::List::eq(&self, other: &vortex_proto::dtype::List) -> bool +pub fn vortex_proto::dtype::List::eq(&self, &vortex_proto::dtype::List) -> bool impl core::default::Default for vortex_proto::dtype::List @@ -454,7 +454,7 @@ pub fn vortex_proto::dtype::List::default() -> Self impl core::fmt::Debug for vortex_proto::dtype::List -pub fn vortex_proto::dtype::List::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::dtype::List::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::StructuralPartialEq for vortex_proto::dtype::List @@ -474,7 +474,7 @@ impl core::cmp::Eq for vortex_proto::dtype::Null impl core::cmp::PartialEq for vortex_proto::dtype::Null -pub fn vortex_proto::dtype::Null::eq(&self, other: &vortex_proto::dtype::Null) -> bool +pub fn vortex_proto::dtype::Null::eq(&self, &vortex_proto::dtype::Null) -> bool impl core::default::Default for vortex_proto::dtype::Null @@ -482,11 +482,11 @@ pub fn vortex_proto::dtype::Null::default() -> Self impl core::fmt::Debug for vortex_proto::dtype::Null -pub fn vortex_proto::dtype::Null::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::dtype::Null::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_proto::dtype::Null -pub fn vortex_proto::dtype::Null::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_proto::dtype::Null::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_proto::dtype::Null @@ -506,7 +506,7 @@ pub vortex_proto::dtype::Primitive::type: i32 impl vortex_proto::dtype::Primitive -pub fn vortex_proto::dtype::Primitive::set_type(&mut self, value: vortex_proto::dtype::PType) +pub fn vortex_proto::dtype::Primitive::set_type(&mut self, vortex_proto::dtype::PType) pub fn vortex_proto::dtype::Primitive::type(&self) -> vortex_proto::dtype::PType @@ -518,7 +518,7 @@ impl core::cmp::Eq for vortex_proto::dtype::Primitive impl core::cmp::PartialEq for vortex_proto::dtype::Primitive -pub fn vortex_proto::dtype::Primitive::eq(&self, other: &vortex_proto::dtype::Primitive) -> bool +pub fn vortex_proto::dtype::Primitive::eq(&self, &vortex_proto::dtype::Primitive) -> bool impl core::default::Default for vortex_proto::dtype::Primitive @@ -526,11 +526,11 @@ pub fn vortex_proto::dtype::Primitive::default() -> Self impl core::fmt::Debug for vortex_proto::dtype::Primitive -pub fn vortex_proto::dtype::Primitive::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::dtype::Primitive::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_proto::dtype::Primitive -pub fn vortex_proto::dtype::Primitive::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_proto::dtype::Primitive::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_proto::dtype::Primitive @@ -556,7 +556,7 @@ pub fn vortex_proto::dtype::Struct::clone(&self) -> vortex_proto::dtype::Struct impl core::cmp::PartialEq for vortex_proto::dtype::Struct -pub fn vortex_proto::dtype::Struct::eq(&self, other: &vortex_proto::dtype::Struct) -> bool +pub fn vortex_proto::dtype::Struct::eq(&self, &vortex_proto::dtype::Struct) -> bool impl core::default::Default for vortex_proto::dtype::Struct @@ -564,7 +564,7 @@ pub fn vortex_proto::dtype::Struct::default() -> Self impl core::fmt::Debug for vortex_proto::dtype::Struct -pub fn vortex_proto::dtype::Struct::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::dtype::Struct::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::StructuralPartialEq for vortex_proto::dtype::Struct @@ -586,7 +586,7 @@ impl core::cmp::Eq for vortex_proto::dtype::Utf8 impl core::cmp::PartialEq for vortex_proto::dtype::Utf8 -pub fn vortex_proto::dtype::Utf8::eq(&self, other: &vortex_proto::dtype::Utf8) -> bool +pub fn vortex_proto::dtype::Utf8::eq(&self, &vortex_proto::dtype::Utf8) -> bool impl core::default::Default for vortex_proto::dtype::Utf8 @@ -594,11 +594,11 @@ pub fn vortex_proto::dtype::Utf8::default() -> Self impl core::fmt::Debug for vortex_proto::dtype::Utf8 -pub fn vortex_proto::dtype::Utf8::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::dtype::Utf8::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_proto::dtype::Utf8 -pub fn vortex_proto::dtype::Utf8::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_proto::dtype::Utf8::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_proto::dtype::Utf8 @@ -622,7 +622,7 @@ impl core::cmp::Eq for vortex_proto::dtype::Variant impl core::cmp::PartialEq for vortex_proto::dtype::Variant -pub fn vortex_proto::dtype::Variant::eq(&self, other: &vortex_proto::dtype::Variant) -> bool +pub fn vortex_proto::dtype::Variant::eq(&self, &vortex_proto::dtype::Variant) -> bool impl core::default::Default for vortex_proto::dtype::Variant @@ -630,11 +630,11 @@ pub fn vortex_proto::dtype::Variant::default() -> Self impl core::fmt::Debug for vortex_proto::dtype::Variant -pub fn vortex_proto::dtype::Variant::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::dtype::Variant::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_proto::dtype::Variant -pub fn vortex_proto::dtype::Variant::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_proto::dtype::Variant::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_proto::dtype::Variant @@ -680,13 +680,13 @@ impl vortex_proto::expr::binary_opts::BinaryOp pub fn vortex_proto::expr::binary_opts::BinaryOp::as_str_name(&self) -> &'static str -pub fn vortex_proto::expr::binary_opts::BinaryOp::from_str_name(value: &str) -> core::option::Option +pub fn vortex_proto::expr::binary_opts::BinaryOp::from_str_name(&str) -> core::option::Option impl vortex_proto::expr::binary_opts::BinaryOp -pub fn vortex_proto::expr::binary_opts::BinaryOp::from_i32(value: i32) -> core::option::Option +pub fn vortex_proto::expr::binary_opts::BinaryOp::from_i32(i32) -> core::option::Option -pub fn vortex_proto::expr::binary_opts::BinaryOp::is_valid(value: i32) -> bool +pub fn vortex_proto::expr::binary_opts::BinaryOp::is_valid(i32) -> bool impl core::clone::Clone for vortex_proto::expr::binary_opts::BinaryOp @@ -696,25 +696,25 @@ impl core::cmp::Eq for vortex_proto::expr::binary_opts::BinaryOp impl core::cmp::Ord for vortex_proto::expr::binary_opts::BinaryOp -pub fn vortex_proto::expr::binary_opts::BinaryOp::cmp(&self, other: &vortex_proto::expr::binary_opts::BinaryOp) -> core::cmp::Ordering +pub fn vortex_proto::expr::binary_opts::BinaryOp::cmp(&self, &vortex_proto::expr::binary_opts::BinaryOp) -> core::cmp::Ordering impl core::cmp::PartialEq for vortex_proto::expr::binary_opts::BinaryOp -pub fn vortex_proto::expr::binary_opts::BinaryOp::eq(&self, other: &vortex_proto::expr::binary_opts::BinaryOp) -> bool +pub fn vortex_proto::expr::binary_opts::BinaryOp::eq(&self, &vortex_proto::expr::binary_opts::BinaryOp) -> bool impl core::cmp::PartialOrd for vortex_proto::expr::binary_opts::BinaryOp -pub fn vortex_proto::expr::binary_opts::BinaryOp::partial_cmp(&self, other: &vortex_proto::expr::binary_opts::BinaryOp) -> core::option::Option +pub fn vortex_proto::expr::binary_opts::BinaryOp::partial_cmp(&self, &vortex_proto::expr::binary_opts::BinaryOp) -> core::option::Option impl core::convert::From for i32 -pub fn i32::from(value: vortex_proto::expr::binary_opts::BinaryOp) -> i32 +pub fn i32::from(vortex_proto::expr::binary_opts::BinaryOp) -> i32 impl core::convert::TryFrom for vortex_proto::expr::binary_opts::BinaryOp pub type vortex_proto::expr::binary_opts::BinaryOp::Error = prost::error::UnknownEnumValue -pub fn vortex_proto::expr::binary_opts::BinaryOp::try_from(value: i32) -> core::result::Result +pub fn vortex_proto::expr::binary_opts::BinaryOp::try_from(i32) -> core::result::Result impl core::default::Default for vortex_proto::expr::binary_opts::BinaryOp @@ -722,11 +722,11 @@ pub fn vortex_proto::expr::binary_opts::BinaryOp::default() -> vortex_proto::exp impl core::fmt::Debug for vortex_proto::expr::binary_opts::BinaryOp -pub fn vortex_proto::expr::binary_opts::BinaryOp::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::expr::binary_opts::BinaryOp::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_proto::expr::binary_opts::BinaryOp -pub fn vortex_proto::expr::binary_opts::BinaryOp::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_proto::expr::binary_opts::BinaryOp::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_proto::expr::binary_opts::BinaryOp @@ -742,11 +742,11 @@ pub vortex_proto::expr::select_opts::Opts::Include(vortex_proto::expr::FieldName impl vortex_proto::expr::select_opts::Opts -pub fn vortex_proto::expr::select_opts::Opts::encode(&self, buf: &mut impl bytes::buf::buf_mut::BufMut) +pub fn vortex_proto::expr::select_opts::Opts::encode(&self, &mut impl bytes::buf::buf_mut::BufMut) pub fn vortex_proto::expr::select_opts::Opts::encoded_len(&self) -> usize -pub fn vortex_proto::expr::select_opts::Opts::merge(field: &mut core::option::Option, tag: u32, wire_type: prost::encoding::wire_type::WireType, buf: &mut impl bytes::buf::buf_impl::Buf, ctx: prost::encoding::DecodeContext) -> core::result::Result<(), prost::error::DecodeError> +pub fn vortex_proto::expr::select_opts::Opts::merge(&mut core::option::Option, u32, prost::encoding::wire_type::WireType, &mut impl bytes::buf::buf_impl::Buf, prost::encoding::DecodeContext) -> core::result::Result<(), prost::error::DecodeError> impl core::clone::Clone for vortex_proto::expr::select_opts::Opts @@ -756,15 +756,15 @@ impl core::cmp::Eq for vortex_proto::expr::select_opts::Opts impl core::cmp::PartialEq for vortex_proto::expr::select_opts::Opts -pub fn vortex_proto::expr::select_opts::Opts::eq(&self, other: &vortex_proto::expr::select_opts::Opts) -> bool +pub fn vortex_proto::expr::select_opts::Opts::eq(&self, &vortex_proto::expr::select_opts::Opts) -> bool impl core::fmt::Debug for vortex_proto::expr::select_opts::Opts -pub fn vortex_proto::expr::select_opts::Opts::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::expr::select_opts::Opts::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_proto::expr::select_opts::Opts -pub fn vortex_proto::expr::select_opts::Opts::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_proto::expr::select_opts::Opts::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_proto::expr::select_opts::Opts @@ -786,7 +786,7 @@ impl core::cmp::Eq for vortex_proto::expr::AggregateFn impl core::cmp::PartialEq for vortex_proto::expr::AggregateFn -pub fn vortex_proto::expr::AggregateFn::eq(&self, other: &vortex_proto::expr::AggregateFn) -> bool +pub fn vortex_proto::expr::AggregateFn::eq(&self, &vortex_proto::expr::AggregateFn) -> bool impl core::default::Default for vortex_proto::expr::AggregateFn @@ -794,11 +794,11 @@ pub fn vortex_proto::expr::AggregateFn::default() -> Self impl core::fmt::Debug for vortex_proto::expr::AggregateFn -pub fn vortex_proto::expr::AggregateFn::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::expr::AggregateFn::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_proto::expr::AggregateFn -pub fn vortex_proto::expr::AggregateFn::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_proto::expr::AggregateFn::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_proto::expr::AggregateFn @@ -822,7 +822,7 @@ impl core::cmp::Eq for vortex_proto::expr::BetweenOpts impl core::cmp::PartialEq for vortex_proto::expr::BetweenOpts -pub fn vortex_proto::expr::BetweenOpts::eq(&self, other: &vortex_proto::expr::BetweenOpts) -> bool +pub fn vortex_proto::expr::BetweenOpts::eq(&self, &vortex_proto::expr::BetweenOpts) -> bool impl core::default::Default for vortex_proto::expr::BetweenOpts @@ -830,11 +830,11 @@ pub fn vortex_proto::expr::BetweenOpts::default() -> Self impl core::fmt::Debug for vortex_proto::expr::BetweenOpts -pub fn vortex_proto::expr::BetweenOpts::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::expr::BetweenOpts::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_proto::expr::BetweenOpts -pub fn vortex_proto::expr::BetweenOpts::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_proto::expr::BetweenOpts::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_proto::expr::BetweenOpts @@ -854,7 +854,7 @@ impl vortex_proto::expr::BinaryOpts pub fn vortex_proto::expr::BinaryOpts::op(&self) -> vortex_proto::expr::binary_opts::BinaryOp -pub fn vortex_proto::expr::BinaryOpts::set_op(&mut self, value: vortex_proto::expr::binary_opts::BinaryOp) +pub fn vortex_proto::expr::BinaryOpts::set_op(&mut self, vortex_proto::expr::binary_opts::BinaryOp) impl core::clone::Clone for vortex_proto::expr::BinaryOpts @@ -864,7 +864,7 @@ impl core::cmp::Eq for vortex_proto::expr::BinaryOpts impl core::cmp::PartialEq for vortex_proto::expr::BinaryOpts -pub fn vortex_proto::expr::BinaryOpts::eq(&self, other: &vortex_proto::expr::BinaryOpts) -> bool +pub fn vortex_proto::expr::BinaryOpts::eq(&self, &vortex_proto::expr::BinaryOpts) -> bool impl core::default::Default for vortex_proto::expr::BinaryOpts @@ -872,11 +872,11 @@ pub fn vortex_proto::expr::BinaryOpts::default() -> Self impl core::fmt::Debug for vortex_proto::expr::BinaryOpts -pub fn vortex_proto::expr::BinaryOpts::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::expr::BinaryOpts::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_proto::expr::BinaryOpts -pub fn vortex_proto::expr::BinaryOpts::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_proto::expr::BinaryOpts::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_proto::expr::BinaryOpts @@ -900,7 +900,7 @@ impl core::cmp::Eq for vortex_proto::expr::CaseWhenOpts impl core::cmp::PartialEq for vortex_proto::expr::CaseWhenOpts -pub fn vortex_proto::expr::CaseWhenOpts::eq(&self, other: &vortex_proto::expr::CaseWhenOpts) -> bool +pub fn vortex_proto::expr::CaseWhenOpts::eq(&self, &vortex_proto::expr::CaseWhenOpts) -> bool impl core::default::Default for vortex_proto::expr::CaseWhenOpts @@ -908,11 +908,11 @@ pub fn vortex_proto::expr::CaseWhenOpts::default() -> Self impl core::fmt::Debug for vortex_proto::expr::CaseWhenOpts -pub fn vortex_proto::expr::CaseWhenOpts::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::expr::CaseWhenOpts::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_proto::expr::CaseWhenOpts -pub fn vortex_proto::expr::CaseWhenOpts::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_proto::expr::CaseWhenOpts::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_proto::expr::CaseWhenOpts @@ -934,7 +934,7 @@ pub fn vortex_proto::expr::CastOpts::clone(&self) -> vortex_proto::expr::CastOpt impl core::cmp::PartialEq for vortex_proto::expr::CastOpts -pub fn vortex_proto::expr::CastOpts::eq(&self, other: &vortex_proto::expr::CastOpts) -> bool +pub fn vortex_proto::expr::CastOpts::eq(&self, &vortex_proto::expr::CastOpts) -> bool impl core::default::Default for vortex_proto::expr::CastOpts @@ -942,7 +942,7 @@ pub fn vortex_proto::expr::CastOpts::default() -> Self impl core::fmt::Debug for vortex_proto::expr::CastOpts -pub fn vortex_proto::expr::CastOpts::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::expr::CastOpts::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::StructuralPartialEq for vortex_proto::expr::CastOpts @@ -970,7 +970,7 @@ pub fn vortex_proto::expr::Expr::clone(&self) -> vortex_proto::expr::Expr impl core::cmp::PartialEq for vortex_proto::expr::Expr -pub fn vortex_proto::expr::Expr::eq(&self, other: &vortex_proto::expr::Expr) -> bool +pub fn vortex_proto::expr::Expr::eq(&self, &vortex_proto::expr::Expr) -> bool impl core::default::Default for vortex_proto::expr::Expr @@ -978,7 +978,7 @@ pub fn vortex_proto::expr::Expr::default() -> Self impl core::fmt::Debug for vortex_proto::expr::Expr -pub fn vortex_proto::expr::Expr::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::expr::Expr::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::StructuralPartialEq for vortex_proto::expr::Expr @@ -1000,7 +1000,7 @@ impl core::cmp::Eq for vortex_proto::expr::FieldNames impl core::cmp::PartialEq for vortex_proto::expr::FieldNames -pub fn vortex_proto::expr::FieldNames::eq(&self, other: &vortex_proto::expr::FieldNames) -> bool +pub fn vortex_proto::expr::FieldNames::eq(&self, &vortex_proto::expr::FieldNames) -> bool impl core::default::Default for vortex_proto::expr::FieldNames @@ -1008,11 +1008,11 @@ pub fn vortex_proto::expr::FieldNames::default() -> Self impl core::fmt::Debug for vortex_proto::expr::FieldNames -pub fn vortex_proto::expr::FieldNames::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::expr::FieldNames::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_proto::expr::FieldNames -pub fn vortex_proto::expr::FieldNames::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_proto::expr::FieldNames::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_proto::expr::FieldNames @@ -1034,7 +1034,7 @@ impl core::cmp::Eq for vortex_proto::expr::GetItemOpts impl core::cmp::PartialEq for vortex_proto::expr::GetItemOpts -pub fn vortex_proto::expr::GetItemOpts::eq(&self, other: &vortex_proto::expr::GetItemOpts) -> bool +pub fn vortex_proto::expr::GetItemOpts::eq(&self, &vortex_proto::expr::GetItemOpts) -> bool impl core::default::Default for vortex_proto::expr::GetItemOpts @@ -1042,11 +1042,11 @@ pub fn vortex_proto::expr::GetItemOpts::default() -> Self impl core::fmt::Debug for vortex_proto::expr::GetItemOpts -pub fn vortex_proto::expr::GetItemOpts::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::expr::GetItemOpts::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_proto::expr::GetItemOpts -pub fn vortex_proto::expr::GetItemOpts::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_proto::expr::GetItemOpts::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_proto::expr::GetItemOpts @@ -1070,7 +1070,7 @@ impl core::cmp::Eq for vortex_proto::expr::LikeOpts impl core::cmp::PartialEq for vortex_proto::expr::LikeOpts -pub fn vortex_proto::expr::LikeOpts::eq(&self, other: &vortex_proto::expr::LikeOpts) -> bool +pub fn vortex_proto::expr::LikeOpts::eq(&self, &vortex_proto::expr::LikeOpts) -> bool impl core::default::Default for vortex_proto::expr::LikeOpts @@ -1078,11 +1078,11 @@ pub fn vortex_proto::expr::LikeOpts::default() -> Self impl core::fmt::Debug for vortex_proto::expr::LikeOpts -pub fn vortex_proto::expr::LikeOpts::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::expr::LikeOpts::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_proto::expr::LikeOpts -pub fn vortex_proto::expr::LikeOpts::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_proto::expr::LikeOpts::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_proto::expr::LikeOpts @@ -1104,7 +1104,7 @@ pub fn vortex_proto::expr::LiteralOpts::clone(&self) -> vortex_proto::expr::Lite impl core::cmp::PartialEq for vortex_proto::expr::LiteralOpts -pub fn vortex_proto::expr::LiteralOpts::eq(&self, other: &vortex_proto::expr::LiteralOpts) -> bool +pub fn vortex_proto::expr::LiteralOpts::eq(&self, &vortex_proto::expr::LiteralOpts) -> bool impl core::default::Default for vortex_proto::expr::LiteralOpts @@ -1112,7 +1112,7 @@ pub fn vortex_proto::expr::LiteralOpts::default() -> Self impl core::fmt::Debug for vortex_proto::expr::LiteralOpts -pub fn vortex_proto::expr::LiteralOpts::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::expr::LiteralOpts::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::StructuralPartialEq for vortex_proto::expr::LiteralOpts @@ -1136,7 +1136,7 @@ impl core::cmp::Eq for vortex_proto::expr::PackOpts impl core::cmp::PartialEq for vortex_proto::expr::PackOpts -pub fn vortex_proto::expr::PackOpts::eq(&self, other: &vortex_proto::expr::PackOpts) -> bool +pub fn vortex_proto::expr::PackOpts::eq(&self, &vortex_proto::expr::PackOpts) -> bool impl core::default::Default for vortex_proto::expr::PackOpts @@ -1144,11 +1144,11 @@ pub fn vortex_proto::expr::PackOpts::default() -> Self impl core::fmt::Debug for vortex_proto::expr::PackOpts -pub fn vortex_proto::expr::PackOpts::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::expr::PackOpts::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_proto::expr::PackOpts -pub fn vortex_proto::expr::PackOpts::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_proto::expr::PackOpts::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_proto::expr::PackOpts @@ -1170,7 +1170,7 @@ impl core::cmp::Eq for vortex_proto::expr::SelectOpts impl core::cmp::PartialEq for vortex_proto::expr::SelectOpts -pub fn vortex_proto::expr::SelectOpts::eq(&self, other: &vortex_proto::expr::SelectOpts) -> bool +pub fn vortex_proto::expr::SelectOpts::eq(&self, &vortex_proto::expr::SelectOpts) -> bool impl core::default::Default for vortex_proto::expr::SelectOpts @@ -1178,11 +1178,11 @@ pub fn vortex_proto::expr::SelectOpts::default() -> Self impl core::fmt::Debug for vortex_proto::expr::SelectOpts -pub fn vortex_proto::expr::SelectOpts::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::expr::SelectOpts::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_proto::expr::SelectOpts -pub fn vortex_proto::expr::SelectOpts::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_proto::expr::SelectOpts::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_proto::expr::SelectOpts @@ -1222,11 +1222,11 @@ pub vortex_proto::scalar::scalar_value::Kind::VariantValue(alloc::boxed::Box usize -pub fn vortex_proto::scalar::scalar_value::Kind::merge(field: &mut core::option::Option, tag: u32, wire_type: prost::encoding::wire_type::WireType, buf: &mut impl bytes::buf::buf_impl::Buf, ctx: prost::encoding::DecodeContext) -> core::result::Result<(), prost::error::DecodeError> +pub fn vortex_proto::scalar::scalar_value::Kind::merge(&mut core::option::Option, u32, prost::encoding::wire_type::WireType, &mut impl bytes::buf::buf_impl::Buf, prost::encoding::DecodeContext) -> core::result::Result<(), prost::error::DecodeError> impl core::clone::Clone for vortex_proto::scalar::scalar_value::Kind @@ -1234,11 +1234,11 @@ pub fn vortex_proto::scalar::scalar_value::Kind::clone(&self) -> vortex_proto::s impl core::cmp::PartialEq for vortex_proto::scalar::scalar_value::Kind -pub fn vortex_proto::scalar::scalar_value::Kind::eq(&self, other: &vortex_proto::scalar::scalar_value::Kind) -> bool +pub fn vortex_proto::scalar::scalar_value::Kind::eq(&self, &vortex_proto::scalar::scalar_value::Kind) -> bool impl core::fmt::Debug for vortex_proto::scalar::scalar_value::Kind -pub fn vortex_proto::scalar::scalar_value::Kind::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::scalar::scalar_value::Kind::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::StructuralPartialEq for vortex_proto::scalar::scalar_value::Kind @@ -1252,7 +1252,7 @@ pub fn vortex_proto::scalar::ListValue::clone(&self) -> vortex_proto::scalar::Li impl core::cmp::PartialEq for vortex_proto::scalar::ListValue -pub fn vortex_proto::scalar::ListValue::eq(&self, other: &vortex_proto::scalar::ListValue) -> bool +pub fn vortex_proto::scalar::ListValue::eq(&self, &vortex_proto::scalar::ListValue) -> bool impl core::default::Default for vortex_proto::scalar::ListValue @@ -1260,7 +1260,7 @@ pub fn vortex_proto::scalar::ListValue::default() -> Self impl core::fmt::Debug for vortex_proto::scalar::ListValue -pub fn vortex_proto::scalar::ListValue::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::scalar::ListValue::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::StructuralPartialEq for vortex_proto::scalar::ListValue @@ -1282,7 +1282,7 @@ pub fn vortex_proto::scalar::Scalar::clone(&self) -> vortex_proto::scalar::Scala impl core::cmp::PartialEq for vortex_proto::scalar::Scalar -pub fn vortex_proto::scalar::Scalar::eq(&self, other: &vortex_proto::scalar::Scalar) -> bool +pub fn vortex_proto::scalar::Scalar::eq(&self, &vortex_proto::scalar::Scalar) -> bool impl core::default::Default for vortex_proto::scalar::Scalar @@ -1290,7 +1290,7 @@ pub fn vortex_proto::scalar::Scalar::default() -> Self impl core::fmt::Debug for vortex_proto::scalar::Scalar -pub fn vortex_proto::scalar::Scalar::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::scalar::Scalar::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::StructuralPartialEq for vortex_proto::scalar::Scalar @@ -1310,7 +1310,7 @@ pub fn vortex_proto::scalar::ScalarValue::clone(&self) -> vortex_proto::scalar:: impl core::cmp::PartialEq for vortex_proto::scalar::ScalarValue -pub fn vortex_proto::scalar::ScalarValue::eq(&self, other: &vortex_proto::scalar::ScalarValue) -> bool +pub fn vortex_proto::scalar::ScalarValue::eq(&self, &vortex_proto::scalar::ScalarValue) -> bool impl core::default::Default for vortex_proto::scalar::ScalarValue @@ -1318,7 +1318,7 @@ pub fn vortex_proto::scalar::ScalarValue::default() -> Self impl core::fmt::Debug for vortex_proto::scalar::ScalarValue -pub fn vortex_proto::scalar::ScalarValue::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_proto::scalar::ScalarValue::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::StructuralPartialEq for vortex_proto::scalar::ScalarValue diff --git a/vortex-python/pyproject.toml b/vortex-python/pyproject.toml index e90bae5a41b..fbb86f53e44 100644 --- a/vortex-python/pyproject.toml +++ b/vortex-python/pyproject.toml @@ -50,13 +50,15 @@ Issues = "https://github.com/vortex-data/vortex/issues" Benchmarks = "https://bench.vortex.dev" [build-system] -requires = ["maturin>=1.7.2,<2.0"] +requires = ["maturin>=1.10.2,<2.0"] build-backend = "maturin" [tool.uv] managed = true [tool.maturin] +profile = "release" +editable-profile = "dev" python-source = "python" module-name = "vortex._lib" features = ["pyo3/extension-module", "mimalloc"] @@ -70,7 +72,7 @@ include = [ [dependency-groups] dev = [ "duckdb>=1.1.2", - "maturin>=1.7.2", + "maturin>=1.10.2", "numpy>=2.2.2", "pandas-stubs>=2.2.3.241126", "pandas[output-formatting]>=2.2.3", diff --git a/vortex-python/python/vortex/_lib/expr.pyi b/vortex-python/python/vortex/_lib/expr.pyi index c1777095743..c69307266de 100644 --- a/vortex-python/python/vortex/_lib/expr.pyi +++ b/vortex-python/python/vortex/_lib/expr.pyi @@ -35,3 +35,4 @@ def not_(child: Expr) -> Expr: ... def and_(left: Expr, right: Expr) -> Expr: ... def cast(child: Expr, dtype: DType) -> Expr: ... def is_null(child: Expr) -> Expr: ... +def is_not_null(child: Expr) -> Expr: ... diff --git a/vortex-python/python/vortex/ray/datasource.py b/vortex-python/python/vortex/ray/datasource.py index b3516490026..966c33b3eaf 100644 --- a/vortex-python/python/vortex/ray/datasource.py +++ b/vortex-python/python/vortex/ray/datasource.py @@ -133,7 +133,7 @@ def _read_task( num_rows=num_rows, size_bytes=None, exec_stats=None, - input_files=paths, + input_files=tuple(paths), ) def read() -> Iterable[pandas.DataFrame]: diff --git a/vortex-python/python/vortex/substrait.py b/vortex-python/python/vortex/substrait.py index 28a144f62ef..1bab4b87d2b 100644 --- a/vortex-python/python/vortex/substrait.py +++ b/vortex-python/python/vortex/substrait.py @@ -498,7 +498,7 @@ def extension_function( case "is_null": return _expr.is_null case "is_not_null": - return _is_not_null + return _expr.is_not_null case name: raise NotImplementedError(f"Function name {name} not supported") case "https://github.com/substrait-io/substrait/blob/main/extensions/functions_arithmetic.yaml": @@ -517,13 +517,6 @@ def extension_function( raise NotImplementedError(f"Extension URI {uri} not supported") -def _is_not_null(e: _expr.Expr) -> _expr.Expr: - """ - Helper function to have a well-typed callable to return - """ - return _expr.not_(_expr.is_null(e)) - - def expression( substrait_object: Expression, functions: list[Callable[..., _expr.Expr]], diff --git a/vortex-python/src/arrays/compressed.rs b/vortex-python/src/arrays/compressed.rs index 96abfa4f1a1..2bb7c1dd7a8 100644 --- a/vortex-python/src/arrays/compressed.rs +++ b/vortex-python/src/arrays/compressed.rs @@ -3,8 +3,10 @@ use pyo3::prelude::*; use vortex::array::IntoArray; -use vortex::array::ToCanonical; +use vortex::array::LEGACY_SESSION; +use vortex::array::VortexSessionExecute; use vortex::array::arrays::Dict; +use vortex::array::arrays::PrimitiveArray; use vortex::encodings::alp::ALP; use vortex::encodings::alp::ALPRD; use vortex::encodings::datetime_parts::DateTimeParts; @@ -89,9 +91,12 @@ impl EncodingSubclass for PyZigZagArray { impl PyZigZagArray { #[staticmethod] pub fn encode(array: PyArrayRef) -> PyVortexResult { - Ok(PyVortex( - zigzag_encode(array.inner().clone().to_primitive())?.into_array(), - )) + // PyZigZagArray (and PyArrayRef) do not currently carry a VortexSession; + // threading one through would change the FromPyObject contract. Use + // LEGACY_SESSION until the wrappers are refactored. + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let primitive = array.inner().clone().execute::(&mut ctx)?; + Ok(PyVortex(zigzag_encode(primitive.as_view())?.into_array())) } } diff --git a/vortex-python/src/arrays/mod.rs b/vortex-python/src/arrays/mod.rs index baabe693614..3ad3d530ef1 100644 --- a/vortex-python/src/arrays/mod.rs +++ b/vortex-python/src/arrays/mod.rs @@ -23,12 +23,15 @@ use pyo3::types::PyRange; use pyo3::types::PyRangeMethods; use pyo3_bytes::PyBytes; use vortex::array::ArrayRef; +use vortex::array::Canonical; use vortex::array::IntoArray; -use vortex::array::ToCanonical; +use vortex::array::LEGACY_SESSION; +use vortex::array::VortexSessionExecute; +use vortex::array::arrays::BoolArray; use vortex::array::arrays::Chunked; use vortex::array::arrays::bool::BoolArrayExt; use vortex::array::arrays::chunked::ChunkedArrayExt; -use vortex::array::arrow::IntoArrowArray; +use vortex::array::arrow::ArrowArrayExecutor; use vortex::array::builtins::ArrayBuiltins; use vortex::array::match_each_integer_ptype; use vortex::dtype::DType; @@ -203,7 +206,7 @@ pub struct PyArray; impl PyArray { #[new] #[pyo3(signature = (*args, **kwargs))] - #[allow(unused_variables)] + #[expect(unused_variables)] fn new(args: &Bound<'_, PyAny>, kwargs: Option<&Bound<'_, PyAny>>) -> Self { Self } @@ -334,7 +337,12 @@ impl PyArray { let chunks = chunked_array .iter_chunks() - .map(|chunk| -> PyVortexResult<_> { Ok(chunk.clone().into_arrow(&arrow_dtype)?) }) + .map(|chunk| -> PyVortexResult<_> { + Ok(chunk.clone().execute_arrow( + Some(&arrow_dtype), + &mut LEGACY_SESSION.create_execution_ctx(), + )?) + }) .collect::, _>>()?; let pa_data_type = arrow_dtype.clone().to_pyarrow(py)?; @@ -354,7 +362,7 @@ impl PyArray { )?) } else { Ok(array - .into_arrow_preferred()? + .execute_arrow(None, &mut LEGACY_SESSION.create_execution_ctx())? .into_data() .to_pyarrow(py)? .into_bound(py)) @@ -524,9 +532,17 @@ impl PyArray { /// ] /// ``` fn filter(slf: Bound, mask: PyArrayRef) -> PyVortexResult { + // PyArray/PyArrayRef do not currently carry a VortexSession; threading one + // through would change the FromPyObject contract. Use LEGACY_SESSION until + // the wrappers are refactored. + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let slf = PyArrayRef::extract(slf.as_any().as_borrowed())?.into_inner(); - let mask = (&*mask as &ArrayRef).to_bool().to_mask_fill_null_false(); - let inner = slf.filter(mask)?.to_canonical()?.into_array(); + let mask_bool = (&*mask as &ArrayRef) + .clone() + .execute::(&mut ctx)?; + let mask = mask_bool.to_mask_fill_null_false(&mut ctx); + let canonical = slf.filter(mask)?.execute::(&mut ctx)?; + let inner = canonical.into_array(); Ok(PyArrayRef::from(inner)) } @@ -601,6 +617,10 @@ impl PyArray { /// ``` // TODO(ngates): return a vortex.Scalar fn scalar_at(slf: Bound, index: usize) -> PyVortexResult> { + // PyArray/PyArrayRef do not currently carry a VortexSession; threading one + // through would change the FromPyObject contract. Use LEGACY_SESSION until + // the wrappers are refactored. + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let py = slf.py(); let slf = PyArrayRef::extract(slf.as_any().as_borrowed())?.into_inner(); if index >= slf.len() { @@ -610,7 +630,7 @@ impl PyArray { )) .into()); } - Ok(PyScalar::init(py, slf.scalar_at(index)?)?) + Ok(PyScalar::init(py, slf.execute_scalar(index, &mut ctx)?)?) } /// Filter, permute, and/or repeat elements by their index. diff --git a/vortex-python/src/arrays/py/array.rs b/vortex-python/src/arrays/py/array.rs index b73fc9c6fb0..e18972bdaf2 100644 --- a/vortex-python/src/arrays/py/array.rs +++ b/vortex-python/src/arrays/py/array.rs @@ -49,7 +49,7 @@ impl<'py> FromPyObject<'_, 'py> for PythonArray { let python_array = ob_cast.get(); Ok(Self { vtable: PythonVTable { - id: python_array.id.clone(), + id: python_array.id, }, object: Arc::new(ob.to_owned().unbind()), len: python_array.len, diff --git a/vortex-python/src/arrays/py/mod.rs b/vortex-python/src/arrays/py/mod.rs index 761d8a3d7df..c0aab2cb98c 100644 --- a/vortex-python/src/arrays/py/mod.rs +++ b/vortex-python/src/arrays/py/mod.rs @@ -9,7 +9,6 @@ pub(crate) use array::*; use pyo3::Bound; use pyo3::PyAny; use pyo3::exceptions::PyValueError; -use pyo3::intern; use pyo3::prelude::PyAnyMethods; pub(crate) use python::*; use vortex::array::ArrayId; @@ -19,15 +18,14 @@ use crate::error::PyVortexResult; /// Extract the array id from a Python class `id` attribute. pub fn id_from_obj(cls: &Bound) -> PyVortexResult { - Ok(ArrayId::new_arc( - cls.getattr(intern!(cls.py(), "id")) - .map_err(|_| { - PyValueError::new_err(format!( - "PyEncoding subclass {cls:?} must have an 'id' attribute" - )) - })? - .extract::() - .map_err(|_| PyValueError::new_err("'id' attribute must be a string"))? - .into(), - )) + let id_str: String = cls + .getattr("id") + .map_err(|_| { + PyValueError::new_err(format!( + "PyEncoding subclass {cls:?} must have an 'id' attribute" + )) + })? + .extract() + .map_err(|_| PyValueError::new_err("'id' attribute must be a string"))?; + Ok(ArrayId::new(&id_str)) } diff --git a/vortex-python/src/arrays/py/vtable.rs b/vortex-python/src/arrays/py/vtable.rs index aa646e971fe..4329c19261f 100644 --- a/vortex-python/src/arrays/py/vtable.rs +++ b/vortex-python/src/arrays/py/vtable.rs @@ -54,7 +54,7 @@ impl VTable for PythonVTable { type ValidityVTable = Self; fn id(&self) -> ArrayId { - self.id.clone() + self.id } fn validate( diff --git a/vortex-python/src/arrow.rs b/vortex-python/src/arrow.rs index 1ca4be5981f..fefd174c176 100644 --- a/vortex-python/src/arrow.rs +++ b/vortex-python/src/arrow.rs @@ -3,7 +3,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileComment: Derived from upstream file arrow-pyarrow/src/main at commit 549709fb at https://github.com/apache/arrow-rs // SPDX-FileNotice: https://github.com/apache/arrow-rs/blob/549709fbdf91cd1f6c263a7e4540c542b6fecf6b/NOTICE.txt -#![allow(clippy::same_name_method)] +#![expect(clippy::same_name_method)] use std::convert::From; use std::convert::TryFrom; diff --git a/vortex-python/src/compress.rs b/vortex-python/src/compress.rs index ef63426acae..fa77cbb1800 100644 --- a/vortex-python/src/compress.rs +++ b/vortex-python/src/compress.rs @@ -2,8 +2,10 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors use pyo3::prelude::*; +use vortex::array::VortexSessionExecute; use vortex::compressor::BtrBlocksCompressor; +use crate::SESSION; use crate::arrays::PyArrayRef; use crate::error::PyVortexResult; use crate::install_module; @@ -51,6 +53,7 @@ pub(crate) fn init(py: Python, parent: &Bound) -> PyResult<()> { /// 'vortex.alp(f64?, len=1000)' #[pyfunction] pub fn compress(array: PyArrayRef) -> PyVortexResult { - let compressed = BtrBlocksCompressor::default().compress(array.inner())?; + let compressed = BtrBlocksCompressor::default() + .compress(array.inner(), &mut SESSION.create_execution_ctx())?; Ok(PyArrayRef::from(compressed)) } diff --git a/vortex-python/src/dataset.rs b/vortex-python/src/dataset.rs index dccb103037d..bb1a99b8b48 100644 --- a/vortex-python/src/dataset.rs +++ b/vortex-python/src/dataset.rs @@ -11,7 +11,9 @@ use pyo3::exceptions::PyValueError; use pyo3::prelude::*; use pyo3::types::PyString; use vortex::array::ArrayRef; -use vortex::array::ToCanonical; +use vortex::array::ExecutionCtx; +use vortex::array::VortexSessionExecute; +use vortex::array::arrays::PrimitiveArray; use vortex::array::iter::ArrayIteratorExt; use vortex::dtype::FieldName; use vortex::dtype::FieldNames; @@ -22,6 +24,7 @@ use vortex::expr::select; use vortex::file::OpenOptionsSessionExt; use vortex::file::VortexFile; use vortex::layout::scan::split_by::SplitBy; +use vortex::session::VortexSession; use crate::RUNTIME; use crate::SESSION; @@ -53,6 +56,7 @@ pub fn read_array_from_reader( filter: Option, indices: Option, row_range: Option<(u64, u64)>, + ctx: &mut ExecutionCtx, ) -> VortexResult { let mut scan = vortex_file.scan()?.with_projection(projection); @@ -61,7 +65,8 @@ pub fn read_array_from_reader( } if let Some(indices) = indices { - let indices = indices.to_primitive().into_buffer(); + let primitive = indices.execute::(ctx)?; + let indices = primitive.into_buffer(); scan = scan.with_row_indices(indices); } @@ -103,12 +108,17 @@ fn filter_from_python(row_filter: Option<&Bound>) -> Option pub struct PyVortexDataset { vxf: VortexFile, schema: SchemaRef, + session: VortexSession, } impl PyVortexDataset { - pub fn try_new(vxf: VortexFile) -> VortexResult { + pub fn try_new(vxf: VortexFile, session: VortexSession) -> VortexResult { let schema = Arc::new(vxf.dtype().to_arrow_schema()?); - Ok(Self { vxf, schema }) + Ok(Self { + vxf, + schema, + session, + }) } pub async fn from_url( @@ -124,7 +134,7 @@ impl PyVortexDataset { } ResolvedStore::Path(path) => SESSION.open_options().open_path(path).await?, }; - PyVortexDataset::try_new(vxf) + PyVortexDataset::try_new(vxf, SESSION.clone()) } } @@ -142,12 +152,14 @@ impl PyVortexDataset { indices: Option, row_range: Option<(u64, u64)>, ) -> PyVortexResult { + let mut ctx = self.session.create_execution_ctx(); let array = read_array_from_reader( &self.vxf, projection_from_python(columns)?, filter_from_python(row_filter), indices.map(|i| i.into_inner()), row_range, + &mut ctx, )?; Ok(PyArrayRef::from(array)) } diff --git a/vortex-python/src/expr/mod.rs b/vortex-python/src/expr/mod.rs index 2daaefe4f4d..17b342d7db5 100644 --- a/vortex-python/src/expr/mod.rs +++ b/vortex-python/src/expr/mod.rs @@ -35,6 +35,7 @@ pub(crate) fn init(py: Python, parent: &Bound) -> PyResult<()> { m.add_function(wrap_pyfunction!(and_, &m)?)?; m.add_function(wrap_pyfunction!(cast, &m)?)?; m.add_function(wrap_pyfunction!(is_null, &m)?)?; + m.add_function(wrap_pyfunction!(is_not_null, &m)?)?; m.add_class::()?; Ok(()) @@ -422,3 +423,19 @@ pub fn is_null(child: PyExpr) -> PyResult { inner: expr::is_null(child.into_inner()), }) } + +/// Creates an expression that checks for non-null values. +/// +/// Parameters +/// ---------- +/// child : :class:`vortex.Expr` +/// +/// Returns +/// ------- +/// :class:`vortex.Expr` +#[pyfunction] +pub fn is_not_null(child: PyExpr) -> PyResult { + Ok(PyExpr { + inner: expr::is_not_null(child.into_inner()), + }) +} diff --git a/vortex-python/src/file.rs b/vortex-python/src/file.rs index 06cee7590c1..0ba334ae7d4 100644 --- a/vortex-python/src/file.rs +++ b/vortex-python/src/file.rs @@ -9,7 +9,9 @@ use pyo3::prelude::*; use pyo3::types::PyList; use pyo3_object_store::PyObjectStore; use vortex::array::ArrayRef; -use vortex::array::ToCanonical; +use vortex::array::ExecutionCtx; +use vortex::array::VortexSessionExecute; +use vortex::array::arrays::PrimitiveArray; use vortex::array::builtins::ArrayBuiltins; use vortex::dtype::DType; use vortex::dtype::FieldNames; @@ -24,6 +26,7 @@ use vortex::file::VortexFile; use vortex::layout::scan::scan_builder::ScanBuilder; use vortex::layout::scan::split_by::SplitBy; use vortex::layout::segments::MokaSegmentCache; +use vortex::session::VortexSession; use crate::RUNTIME; use crate::SESSION; @@ -80,12 +83,16 @@ pub fn open( }) })?; - Ok(PyVortexFile { vxf }) + Ok(PyVortexFile { + vxf, + session: SESSION.clone(), + }) } #[pyclass(name = "VortexFile", module = "vortex", frozen)] pub struct PyVortexFile { vxf: VortexFile, + session: VortexSession, } #[pymethods] @@ -108,12 +115,14 @@ impl PyVortexFile { indices: Option, batch_size: Option, ) -> PyVortexResult { + let mut ctx = slf.get().session.create_execution_ctx(); let builder = slf.get().scan_builder( projection.map(|p| p.0), expr.map(|e| e.into_inner()), limit, indices.map(|i| i.into_inner()), batch_size, + &mut ctx, )?; Ok(PyArrayIterator::new(Box::new( @@ -130,12 +139,14 @@ impl PyVortexFile { indices: Option, batch_size: Option, ) -> PyVortexResult { + let mut ctx = slf.get().session.create_execution_ctx(); let builder = slf.get().scan_builder( projection.map(|p| p.0), expr.map(|e| e.into_inner()), limit, indices.map(|i| i.into_inner()), batch_size, + &mut ctx, )?; let scan = builder.prepare()?; @@ -179,7 +190,10 @@ impl PyVortexFile { } fn to_dataset(slf: Bound) -> PyVortexResult { - Ok(PyVortexDataset::try_new(slf.get().vxf.clone())?) + Ok(PyVortexDataset::try_new( + slf.get().vxf.clone(), + slf.get().session.clone(), + )?) } #[pyo3(signature = (*))] @@ -201,6 +215,7 @@ impl PyVortexFile { limit: Option, indices: Option, batch_size: Option, + ctx: &mut ExecutionCtx, ) -> VortexResult> { let mut builder = self .vxf @@ -213,10 +228,8 @@ impl PyVortexFile { } if let Some(indices) = indices { - let indices = indices - .cast(DType::Primitive(PType::U64, NonNullable))? - .to_primitive() - .into_buffer::(); + let casted = indices.cast(DType::Primitive(PType::U64, NonNullable))?; + let indices = casted.execute::(ctx)?.into_buffer::(); builder = builder.with_row_indices(indices); } diff --git a/vortex-python/src/io.rs b/vortex-python/src/io.rs index edf1061188d..81bfdc750f6 100644 --- a/vortex-python/src/io.rs +++ b/vortex-python/src/io.rs @@ -280,7 +280,7 @@ impl PyVortexWriteOptions { /// >>> vx.io.VortexWriteOptions.default().write(sprl, "chonky.vortex") /// >>> import os /// >>> os.path.getsize('chonky.vortex') - /// 216004 + /// 215972 /// ``` /// /// Wow, Vortex manages to use about two bytes per integer! So advanced. So tiny. @@ -292,7 +292,7 @@ impl PyVortexWriteOptions { /// ```python /// >>> vx.io.VortexWriteOptions.compact().write(sprl, "tiny.vortex") /// >>> os.path.getsize('tiny.vortex') - /// 55120 + /// 55088 /// ``` /// /// Random numbers are not (usually) composed of random bytes! diff --git a/vortex-python/src/iter/mod.rs b/vortex-python/src/iter/mod.rs index c878ae33e93..8e40fdb8259 100644 --- a/vortex-python/src/iter/mod.rs +++ b/vortex-python/src/iter/mod.rs @@ -20,7 +20,9 @@ use pyo3::prelude::*; use pyo3::types::PyIterator; use vortex::array::Canonical; use vortex::array::IntoArray; -use vortex::array::arrow::IntoArrowArray; +use vortex::array::LEGACY_SESSION; +use vortex::array::VortexSessionExecute; +use vortex::array::arrow::ArrowArrayExecutor; use vortex::array::iter::ArrayIterator; use vortex::array::iter::ArrayIteratorAdapter; use vortex::array::iter::ArrayIteratorExt; @@ -126,7 +128,8 @@ impl PyArrayIterator { Box::new(RecordBatchIterator::new( iter.map(move |chunk| { let data_type = data_type.clone(); - chunk?.into_arrow(&data_type) + chunk? + .execute_arrow(Some(&data_type), &mut LEGACY_SESSION.create_execution_ctx()) }) .map(|chunk| chunk.map_err(|e| ArrowError::ExternalError(Box::new(e)))) .map(|array| array.map(|a| RecordBatch::from(a.as_struct().clone()))), diff --git a/vortex-python/src/object_store/registry.rs b/vortex-python/src/object_store/registry.rs index 13a16e3c36c..c9544e0558e 100644 --- a/vortex-python/src/object_store/registry.rs +++ b/vortex-python/src/object_store/registry.rs @@ -7,7 +7,7 @@ //! //! See also -#![allow(clippy::disallowed_types)] +#![expect(clippy::disallowed_types)] use std::collections::HashMap; use std::sync::Arc; diff --git a/vortex-python/src/scalar/factory.rs b/vortex-python/src/scalar/factory.rs index ea07d3494a9..90e1c2031f7 100644 --- a/vortex-python/src/scalar/factory.rs +++ b/vortex-python/src/scalar/factory.rs @@ -26,7 +26,6 @@ use crate::error::PyVortexResult; use crate::scalar::PyScalar; use crate::scalar::bool; -#[allow(unused_variables)] #[pyfunction(name = "scalar")] #[pyo3(signature = (value, *, dtype=None))] pub fn scalar<'py>( diff --git a/vortex-python/src/scan.rs b/vortex-python/src/scan.rs index 11773588add..1fd53d96131 100644 --- a/vortex-python/src/scan.rs +++ b/vortex-python/src/scan.rs @@ -4,6 +4,8 @@ use pyo3::exceptions::PyIndexError; use pyo3::prelude::*; use vortex::array::ArrayRef; +use vortex::array::LEGACY_SESSION; +use vortex::array::VortexSessionExecute; use vortex::layout::scan::repeated_scan::RepeatedScan; use crate::RUNTIME; @@ -68,7 +70,7 @@ impl PyRepeatedScan { if array.is_empty() { continue; } - let scalar = array.scalar_at(0)?; + let scalar = array.execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx())?; return Ok(PyScalar::init(slf.py(), scalar)?); } diff --git a/vortex-python/src/serde/context.rs b/vortex-python/src/serde/context.rs index 6a4f6d5d3a2..9e88a523749 100644 --- a/vortex-python/src/serde/context.rs +++ b/vortex-python/src/serde/context.rs @@ -74,9 +74,7 @@ impl PyReadContext { #[new] fn new(ids: Vec) -> Self { Self(ReadContext::new( - ids.into_iter() - .map(|i| Id::new_arc(Arc::from(i))) - .collect::>(), + ids.into_iter().map(|i| Id::new(&i)).collect::>(), )) } diff --git a/vortex-python/test/conftest.py b/vortex-python/test/conftest.py index d5111a6c336..f2c1e6dd646 100644 --- a/vortex-python/test/conftest.py +++ b/vortex-python/test/conftest.py @@ -2,19 +2,5 @@ # SPDX-FileCopyrightText: Copyright the Vortex contributors import logging -import os -import pathlib -import subprocess logging.basicConfig(level=logging.DEBUG) - - -def pytest_sessionstart(): - """Pytest plugin to trigger maturin builds before running tests.""" - if os.environ.get("CI") is None: - # Running maturin develop --skip-install builds a "linux" wheel which PyPI rejects - # (https://peps.python.org/pep-0513/#rationale). When testing an already built wheel, we - # neither want to rebuild nor pollute the target/wheels directory with a wheel that PyPI - # will reject. - working_dir = pathlib.Path(__file__).parent.parent - _ = subprocess.check_call(["maturin", "develop", "--skip-install"], cwd=working_dir) diff --git a/vortex-python/test/test_datasource.py b/vortex-python/test/test_datasource.py index eece2e7e7de..fc7fc0f492c 100644 --- a/vortex-python/test/test_datasource.py +++ b/vortex-python/test/test_datasource.py @@ -14,13 +14,15 @@ @pytest.fixture(scope="module") def ray_init(): - # https://github.com/ray-project/ray/issues/53848#issuecomment-3056271943 - ray.init( # pyright: ignore[reportUnknownMemberType] - runtime_env={ - "working_dir": None, - "excludes": [".git", ".venv"], - } - ) + # Ray's uv_runtime_env_hook would auto-upload the working directory to + # workers, but vortex-python's compiled _lib extension exceeds Ray's + # 512 MiB upload limit. Disable the hook for these local-mode tests. + # (Ray 2.55 added a string-type validation that broke the previous + # `working_dir: None` workaround from ray-project/ray#53848.) + import ray._private.ray_constants as ray_constants + + ray_constants.RAY_ENABLE_UV_RUN_RUNTIME_ENV = False + _ = ray.init() # pyright: ignore[reportUnknownMemberType] yield None ray.shutdown() # pyright: ignore[reportUnknownMemberType] @@ -53,7 +55,7 @@ def test_vortex_datasource(ray_init, tmpdir_factory): # pyright: ignore[reportU # Without an explicit sort, Ray may reorder rows *even within a single record batch*. ds = ds.sort("index") - tbl = pa.concat_tables(pa.Table.from_pydict(x) for x in ds.iter_batches()) # pyright: ignore[reportArgumentType] + tbl = pa.concat_tables(pa.Table.from_pydict(x) for x in ds.iter_batches()) # pyright: ignore[reportArgumentType, reportUnknownMemberType, reportUnknownVariableType] expected = pa.Table.from_pylist([record(x) for x in range(0, 10)], schema=tbl.schema) assert tbl == expected diff --git a/vortex-scan/README.md b/vortex-scan/README.md index 2473604853f..31621370f50 100644 --- a/vortex-scan/README.md +++ b/vortex-scan/README.md @@ -121,7 +121,6 @@ Filters are automatically optimized using: All concurrent code has been verified using: -- **Loom Testing**: Exhaustive verification of all possible thread interleavings - **Address Sanitizer**: Memory safety verification in CI - **Debug Assertions**: Runtime checks for invariants in debug builds @@ -135,23 +134,6 @@ Run the standard test suite: cargo test -p vortex-scan --all-features ``` -### Loom Concurrency Tests - -The crate includes comprehensive Loom tests that exhaustively verify concurrent behavior. -These tests run by default but can be disabled if need be: - -```rust -# Skip Loom tests when using incompatible tools like address sanitizer -RUSTFLAGS="--cfg disable_loom" cargo test -p vortex-scan -``` - -Loom tests verify: - -- Memory ordering correctness in the work-stealing queue -- Absence of data races in filter expression evaluation -- Proper synchronization in concurrent task factories -- Thread termination conditions and cleanup - ## Performance Considerations ### Concurrency Level diff --git a/vortex-scan/public-api.lock b/vortex-scan/public-api.lock index 2748e5cc037..5c25cf4409e 100644 --- a/vortex-scan/public-api.lock +++ b/vortex-scan/public-api.lock @@ -8,7 +8,7 @@ impl vortex_scan::row_mask::RowMask pub fn vortex_scan::row_mask::RowMask::mask(&self) -> &vortex_mask::Mask -pub fn vortex_scan::row_mask::RowMask::new(row_offset: u64, mask: vortex_mask::Mask) -> Self +pub fn vortex_scan::row_mask::RowMask::new(u64, vortex_mask::Mask) -> Self pub fn vortex_scan::row_mask::RowMask::row_range(&self) -> core::ops::range::Range @@ -18,7 +18,7 @@ pub fn vortex_scan::row_mask::RowMask::clone(&self) -> vortex_scan::row_mask::Ro impl core::fmt::Debug for vortex_scan::row_mask::RowMask -pub fn vortex_scan::row_mask::RowMask::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_scan::row_mask::RowMask::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub mod vortex_scan::selection @@ -36,9 +36,9 @@ pub vortex_scan::selection::Selection::IncludeRoaring(roaring::treemap::RoaringT impl vortex_scan::selection::Selection -pub fn vortex_scan::selection::Selection::row_count(&self, total_rows: u64) -> u64 +pub fn vortex_scan::selection::Selection::row_count(&self, u64) -> u64 -pub fn vortex_scan::selection::Selection::row_mask(&self, range: &core::ops::range::Range) -> vortex_scan::row_mask::RowMask +pub fn vortex_scan::selection::Selection::row_mask(&self, &core::ops::range::Range) -> vortex_scan::row_mask::RowMask impl core::clone::Clone for vortex_scan::selection::Selection @@ -50,7 +50,7 @@ pub fn vortex_scan::selection::Selection::default() -> vortex_scan::selection::S impl core::fmt::Debug for vortex_scan::selection::Selection -pub fn vortex_scan::selection::Selection::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_scan::selection::Selection::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_scan::ScanRequest @@ -60,6 +60,10 @@ pub vortex_scan::ScanRequest::limit: core::option::Option pub vortex_scan::ScanRequest::ordered: bool +pub vortex_scan::ScanRequest::partition_range: core::option::Option> + +pub vortex_scan::ScanRequest::partition_selection: vortex_scan::selection::Selection + pub vortex_scan::ScanRequest::projection: vortex_array::expr::expression::Expression pub vortex_scan::ScanRequest::row_range: core::option::Option> @@ -76,31 +80,31 @@ pub fn vortex_scan::ScanRequest::default() -> Self impl core::fmt::Debug for vortex_scan::ScanRequest -pub fn vortex_scan::ScanRequest::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_scan::ScanRequest::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub trait vortex_scan::DataSource: 'static + core::marker::Send + core::marker::Sync pub fn vortex_scan::DataSource::byte_size(&self) -> core::option::Option> -pub fn vortex_scan::DataSource::deserialize_partition(&self, data: &[u8], session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_scan::DataSource::deserialize_partition(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult pub fn vortex_scan::DataSource::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_scan::DataSource::field_statistics<'life0, 'life1, 'async_trait>(&'life0 self, field_path: &'life1 vortex_array::dtype::field::FieldPath) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait +pub fn vortex_scan::DataSource::field_statistics<'life0, 'life1, 'async_trait>(&'life0 self, &'life1 vortex_array::dtype::field::FieldPath) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait pub fn vortex_scan::DataSource::row_count(&self) -> core::option::Option> -pub fn vortex_scan::DataSource::scan<'life0, 'async_trait>(&'life0 self, scan_request: vortex_scan::ScanRequest) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait +pub fn vortex_scan::DataSource::scan<'life0, 'async_trait>(&'life0 self, vortex_scan::ScanRequest) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait pub fn vortex_scan::DataSource::serialize(&self) -> vortex_error::VortexResult>> pub trait vortex_scan::DataSourceOpener: 'static -pub fn vortex_scan::DataSourceOpener::open<'life0, 'life1, 'async_trait>(&'life0 self, uri: alloc::string::String, session: &'life1 vortex_session::VortexSession) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait +pub fn vortex_scan::DataSourceOpener::open<'life0, 'life1, 'async_trait>(&'life0 self, alloc::string::String, &'life1 vortex_session::VortexSession) -> core::pin::Pin> + core::marker::Send + 'async_trait)>> where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait pub trait vortex_scan::DataSourceRemote: 'static -pub fn vortex_scan::DataSourceRemote::deserialize_data_source(&self, data: &[u8], session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_scan::DataSourceRemote::deserialize_data_source(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult pub trait vortex_scan::DataSourceScan: 'static + core::marker::Send @@ -108,7 +112,7 @@ pub fn vortex_scan::DataSourceScan::dtype(&self) -> &vortex_array::dtype::DType pub fn vortex_scan::DataSourceScan::partition_count(&self) -> core::option::Option> -pub fn vortex_scan::DataSourceScan::partitions(self: alloc::boxed::Box) -> vortex_scan::PartitionStream +pub fn vortex_scan::DataSourceScan::partitions(alloc::boxed::Box) -> vortex_scan::PartitionStream pub trait vortex_scan::Partition: 'static + core::marker::Send @@ -116,7 +120,9 @@ pub fn vortex_scan::Partition::as_any(&self) -> &dyn core::any::Any pub fn vortex_scan::Partition::byte_size(&self) -> core::option::Option> -pub fn vortex_scan::Partition::execute(self: alloc::boxed::Box) -> vortex_error::VortexResult +pub fn vortex_scan::Partition::execute(alloc::boxed::Box) -> vortex_error::VortexResult + +pub fn vortex_scan::Partition::index(&self) -> usize pub fn vortex_scan::Partition::row_count(&self) -> core::option::Option> diff --git a/vortex-scan/src/lib.rs b/vortex-scan/src/lib.rs index edcb5e29979..a9521728a05 100644 --- a/vortex-scan/src/lib.rs +++ b/vortex-scan/src/lib.rs @@ -76,6 +76,14 @@ pub type DataSourceRef = Arc; /// /// The DataSource may be used multiple times to create multiple scans, whereas each scan and each /// partition of a scan can only be consumed once. +/// +/// Partitions have indices. Partition index is stable throughout DataSource's +/// lifetime. For every scan requested on the DataSource, the index will stay +/// the same. +/// However, this means you should create another instance of a DataSource if +/// your environment changes e.g. you have a glob and another file is added to +/// the filesystem this glob references. +/// See MultiFileDataSource in vortex-file/src/multi/mod.rs #[async_trait] pub trait DataSource: 'static + Send + Sync { /// Returns the dtype of the source. @@ -120,11 +128,16 @@ pub struct ScanRequest { pub projection: Expression, /// Filter expression, `None` implies no filter. pub filter: Option, - /// The row range to read. + /// The per-partition row range to read. Row range will be applied + /// over every partition you scan. pub row_range: Option>, - /// A row selection to apply to the scan. The selection identifies rows within the specified - /// row range. + /// The per-partition row selection to read. Row selection will be applied + /// over every partition you scan. pub selection: Selection, + /// Partition selection to scan, which allows readers to skip unwanted partitions. + pub partition_selection: Selection, + /// Partition range to scan, which allows readers to skip unwanted partitions. + pub partition_range: Option>, /// Whether the scan should preserve row order. If false, the scan may produce rows in any /// order, for example to enable parallel execution across partitions. pub ordered: bool, @@ -140,8 +153,10 @@ impl Default for ScanRequest { filter: None, row_range: None, selection: Selection::default(), + partition_selection: Selection::default(), ordered: false, limit: None, + partition_range: None, } } } @@ -169,6 +184,11 @@ pub trait Partition: 'static + Send { /// Downcast the partition to a concrete type. fn as_any(&self) -> &dyn Any; + /// Some unique identifier for partition. + /// If you have an instance of a DataSource, the indices of emitted + /// partitions will stay stable for every scan in this DataSource. + fn index(&self) -> usize; + /// Returns an estimate of the row count for this partition. fn row_count(&self) -> Option>; diff --git a/vortex-session/Cargo.toml b/vortex-session/Cargo.toml index f6af94e1260..d4e047223c1 100644 --- a/vortex-session/Cargo.toml +++ b/vortex-session/Cargo.toml @@ -20,8 +20,8 @@ all-features = true workspace = true [dependencies] -arcref = { workspace = true } dashmap = { workspace = true } +lasso = { workspace = true } parking_lot = { workspace = true } vortex-error = { workspace = true } vortex-utils = { workspace = true, features = ["dashmap"] } diff --git a/vortex-session/public-api.lock b/vortex-session/public-api.lock index c59b40fd037..58896611d60 100644 --- a/vortex-session/public-api.lock +++ b/vortex-session/public-api.lock @@ -2,19 +2,31 @@ pub mod vortex_session pub mod vortex_session::registry +pub struct vortex_session::registry::CachedId + +impl vortex_session::registry::CachedId + +pub const fn vortex_session::registry::CachedId::new(&'static str) -> Self + +impl core::ops::deref::Deref for vortex_session::registry::CachedId + +pub type vortex_session::registry::CachedId::Target = vortex_session::registry::Id + +pub fn vortex_session::registry::CachedId::deref(&self) -> &vortex_session::registry::Id + pub struct vortex_session::registry::Context impl vortex_session::registry::Context pub fn vortex_session::registry::Context::empty() -> Self -pub fn vortex_session::registry::Context::intern(&self, id: &vortex_session::registry::Id) -> core::option::Option +pub fn vortex_session::registry::Context::intern(&self, &vortex_session::registry::Id) -> core::option::Option -pub fn vortex_session::registry::Context::new(ids: alloc::vec::Vec) -> Self +pub fn vortex_session::registry::Context::new(alloc::vec::Vec) -> Self pub fn vortex_session::registry::Context::to_ids(&self) -> alloc::vec::Vec -pub fn vortex_session::registry::Context::with_registry(self, registry: vortex_session::registry::Registry) -> Self +pub fn vortex_session::registry::Context::with_registry(self, vortex_session::registry::Registry) -> Self impl core::clone::Clone for vortex_session::registry::Context @@ -22,21 +34,81 @@ pub fn vortex_session::registry::Context::clone(&self) -> vortex_session::reg impl core::fmt::Debug for vortex_session::registry::Context -pub fn vortex_session::registry::Context::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_session::registry::Context::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::default::Default for vortex_session::registry::Context pub fn vortex_session::registry::Context::default() -> Self +pub struct vortex_session::registry::Id(_) + +impl vortex_session::registry::Id + +pub fn vortex_session::registry::Id::as_str(&self) -> &str + +pub fn vortex_session::registry::Id::new(&str) -> Self + +pub fn vortex_session::registry::Id::new_static(&'static str) -> Self + +impl core::clone::Clone for vortex_session::registry::Id + +pub fn vortex_session::registry::Id::clone(&self) -> vortex_session::registry::Id + +impl core::cmp::Eq for vortex_session::registry::Id + +impl core::cmp::Ord for vortex_session::registry::Id + +pub fn vortex_session::registry::Id::cmp(&self, &Self) -> core::cmp::Ordering + +impl core::cmp::PartialEq for vortex_session::registry::Id + +pub fn vortex_session::registry::Id::eq(&self, &vortex_session::registry::Id) -> bool + +impl core::cmp::PartialEq<&vortex_session::registry::Id> for vortex_session::registry::Id + +pub fn vortex_session::registry::Id::eq(&self, &&vortex_session::registry::Id) -> bool + +impl core::cmp::PartialEq for &vortex_session::registry::Id + +pub fn &vortex_session::registry::Id::eq(&self, &vortex_session::registry::Id) -> bool + +impl core::cmp::PartialOrd for vortex_session::registry::Id + +pub fn vortex_session::registry::Id::partial_cmp(&self, &Self) -> core::option::Option + +impl core::convert::AsRef for vortex_session::registry::Id + +pub fn vortex_session::registry::Id::as_ref(&self) -> &str + +impl core::convert::From<&str> for vortex_session::registry::Id + +pub fn vortex_session::registry::Id::from(&str) -> Self + +impl core::fmt::Debug for vortex_session::registry::Id + +pub fn vortex_session::registry::Id::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +impl core::fmt::Display for vortex_session::registry::Id + +pub fn vortex_session::registry::Id::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +impl core::hash::Hash for vortex_session::registry::Id + +pub fn vortex_session::registry::Id::hash<__H: core::hash::Hasher>(&self, &mut __H) + +impl core::marker::Copy for vortex_session::registry::Id + +impl core::marker::StructuralPartialEq for vortex_session::registry::Id + pub struct vortex_session::registry::ReadContext impl vortex_session::registry::ReadContext pub fn vortex_session::registry::ReadContext::ids(&self) -> &[vortex_session::registry::Id] -pub fn vortex_session::registry::ReadContext::new(ids: impl core::convert::Into>) -> Self +pub fn vortex_session::registry::ReadContext::new(impl core::convert::Into>) -> Self -pub fn vortex_session::registry::ReadContext::resolve(&self, idx: u16) -> core::option::Option +pub fn vortex_session::registry::ReadContext::resolve(&self, u16) -> core::option::Option impl core::clone::Clone for vortex_session::registry::ReadContext @@ -44,7 +116,7 @@ pub fn vortex_session::registry::ReadContext::clone(&self) -> vortex_session::re impl core::fmt::Debug for vortex_session::registry::ReadContext -pub fn vortex_session::registry::ReadContext::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_session::registry::ReadContext::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_session::registry::Registry(_) @@ -52,17 +124,17 @@ impl vortex_session::registry::Registry pub fn vortex_session::registry::Registry::empty() -> Self -pub fn vortex_session::registry::Registry::find(&self, id: &vortex_session::registry::Id) -> core::option::Option +pub fn vortex_session::registry::Registry::find(&self, &vortex_session::registry::Id) -> core::option::Option -pub fn vortex_session::registry::Registry::find_many<'a>(&self, ids: impl core::iter::traits::collect::IntoIterator) -> impl core::iter::traits::iterator::Iterator>> +pub fn vortex_session::registry::Registry::find_many<'a>(&self, impl core::iter::traits::collect::IntoIterator) -> impl core::iter::traits::iterator::Iterator>> pub fn vortex_session::registry::Registry::ids(&self) -> impl core::iter::traits::iterator::Iterator + '_ pub fn vortex_session::registry::Registry::items(&self) -> impl core::iter::traits::iterator::Iterator + '_ -pub fn vortex_session::registry::Registry::register(&self, id: impl core::convert::Into, item: impl core::convert::Into) +pub fn vortex_session::registry::Registry::register(&self, impl core::convert::Into, impl core::convert::Into) -pub fn vortex_session::registry::Registry::with(self, id: impl core::convert::Into, item: impl core::convert::Into) -> Self +pub fn vortex_session::registry::Registry::with(self, impl core::convert::Into, impl core::convert::Into) -> Self impl core::clone::Clone for vortex_session::registry::Registry @@ -70,19 +142,17 @@ pub fn vortex_session::registry::Registry::clone(&self) -> vortex_session::re impl core::fmt::Debug for vortex_session::registry::Registry -pub fn vortex_session::registry::Registry::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_session::registry::Registry::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::default::Default for vortex_session::registry::Registry pub fn vortex_session::registry::Registry::default() -> Self -pub type vortex_session::registry::Id = arcref::ArcRef - pub struct vortex_session::Ref<'a, T>(_) impl<'a, T> vortex_session::Ref<'a, T> -pub fn vortex_session::Ref<'a, T>::map(self, f: F) -> vortex_session::Ref<'a, U> where F: core::ops::function::FnOnce(&T) -> &U +pub fn vortex_session::Ref<'a, T>::map(self, F) -> vortex_session::Ref<'a, U> where F: core::ops::function::FnOnce(&T) -> &U impl<'a, T> core::ops::deref::Deref for vortex_session::Ref<'a, T> @@ -94,7 +164,7 @@ pub struct vortex_session::RefMut<'a, T>(_) impl<'a, T> vortex_session::RefMut<'a, T> -pub fn vortex_session::RefMut<'a, T>::map(self, f: F) -> vortex_session::RefMut<'a, U> where F: core::ops::function::FnOnce(&mut T) -> &mut U +pub fn vortex_session::RefMut<'a, T>::map(self, F) -> vortex_session::RefMut<'a, U> where F: core::ops::function::FnOnce(&mut T) -> &mut U impl<'a, T> core::ops::deref::Deref for vortex_session::RefMut<'a, T> @@ -116,11 +186,11 @@ pub fn vortex_session::VortexSession::allows_unknown(&self) -> bool pub fn vortex_session::VortexSession::empty() -> Self -pub fn vortex_session::VortexSession::set(self, val: V) -> Self +pub fn vortex_session::VortexSession::set(self, V) -> Self pub fn vortex_session::VortexSession::with(self) -> Self -pub fn vortex_session::VortexSession::with_some(self, var: V) -> Self +pub fn vortex_session::VortexSession::with_some(self, V) -> Self impl core::clone::Clone for vortex_session::VortexSession @@ -128,7 +198,7 @@ pub fn vortex_session::VortexSession::clone(&self) -> vortex_session::VortexSess impl core::fmt::Debug for vortex_session::VortexSession -pub fn vortex_session::VortexSession::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_session::VortexSession::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_session::SessionExt for vortex_session::VortexSession @@ -171,9 +241,3 @@ pub trait vortex_session::SessionVar: core::any::Any + core::marker::Send + core pub fn vortex_session::SessionVar::as_any(&self) -> &dyn core::any::Any pub fn vortex_session::SessionVar::as_any_mut(&mut self) -> &mut dyn core::any::Any - -impl vortex_session::SessionVar for T - -pub fn T::as_any(&self) -> &dyn core::any::Any - -pub fn T::as_any_mut(&mut self) -> &mut dyn core::any::Any diff --git a/vortex-session/src/lib.rs b/vortex-session/src/lib.rs index 2cd6e7585d0..0c2384d2861 100644 --- a/vortex-session/src/lib.rs +++ b/vortex-session/src/lib.rs @@ -81,7 +81,9 @@ impl VortexSession { /// Returns whether unknown plugins should deserialize as foreign placeholders. pub fn allows_unknown(&self) -> bool { - ::get::(self).allow_unknown + ::get_opt::(self) + .map(|p| p.allow_unknown) + .unwrap_or(false) } } @@ -90,6 +92,16 @@ struct UnknownPluginPolicy { allow_unknown: bool, } +impl SessionVar for UnknownPluginPolicy { + fn as_any(&self) -> &dyn Any { + self + } + + fn as_any_mut(&mut self) -> &mut dyn Any { + self + } +} + /// Trait for accessing and modifying the state of a Vortex session. pub trait SessionExt: Sized + private::Sealed { /// Returns the [`VortexSession`]. @@ -203,21 +215,13 @@ impl Hasher for IdHasher { } /// This trait defines variables that can be stored against a Vortex session. +/// +/// Users should implement this trait for anything that you want to store on a `VortexSession`. pub trait SessionVar: Any + Send + Sync + Debug + 'static { fn as_any(&self) -> &dyn Any; fn as_any_mut(&mut self) -> &mut dyn Any; } -impl SessionVar for T { - fn as_any(&self) -> &dyn Any { - self - } - - fn as_any_mut(&mut self) -> &mut dyn Any { - self - } -} - // NOTE(ngates): we don't want to expose that the internals of a session is a DashMap, so we have // our own wrapped Ref type. pub struct Ref<'a, T>(dashmap::mapref::one::MappedRef<'a, TypeId, Box, T>); diff --git a/vortex-session/src/registry.rs b/vortex-session/src/registry.rs index 35a65b086a3..36c03828844 100644 --- a/vortex-session/src/registry.rs +++ b/vortex-session/src/registry.rs @@ -4,17 +4,140 @@ //! Many session types use a registry of objects that can be looked up by name to construct //! contexts. This module provides a generic registry type for that purpose. +use std::cmp::Ordering; +use std::fmt; use std::fmt::Debug; +use std::fmt::Display; +use std::fmt::Formatter; +use std::hash::Hash; use std::ops::Deref; use std::sync::Arc; +use std::sync::LazyLock; +use std::sync::OnceLock; -use arcref::ArcRef; +use lasso::Spur; +use lasso::ThreadedRodeo; use parking_lot::RwLock; use vortex_error::VortexExpect; use vortex_utils::aliases::dash_map::DashMap; -/// An identifier for an item in a registry. -pub type Id = ArcRef; +/// Global string interner for [`Id`] values. +static INTERNER: LazyLock = LazyLock::new(ThreadedRodeo::new); + +/// A lightweight, copyable identifier backed by a global string interner. +/// +/// Used for array encoding IDs, scalar function IDs, layout IDs, and similar +/// globally-unique string identifiers throughout Vortex. Equality and hashing +/// are O(1) symbol comparisons. +#[derive(Clone, Copy, PartialEq, Eq, Hash)] +pub struct Id(Spur); + +impl Id { + /// Intern a string and return its `Id`. + pub fn new(s: &str) -> Self { + Self(INTERNER.get_or_intern(s)) + } + + /// Intern a string and return its `Id`. + pub fn new_static(s: &'static str) -> Self { + Self(INTERNER.get_or_intern_static(s)) + } + + /// Returns the interned string. + pub fn as_str(&self) -> &str { + let s = INTERNER.resolve(&self.0); + // SAFETY: INTERNER is 'static and its arena is append-only, so resolved string + // pointers are stable for the lifetime of the program. + unsafe { &*(s as *const str) } + } +} + +impl From<&str> for Id { + fn from(s: &str) -> Self { + Self::new(s) + } +} + +impl Display for Id { + fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { + f.write_str(self.as_str()) + } +} + +impl Debug for Id { + fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { + write!(f, "Id(\"{}\")", self.as_str()) + } +} + +impl PartialOrd for Id { + fn partial_cmp(&self, other: &Self) -> Option { + Some(self.cmp(other)) + } +} + +impl Ord for Id { + fn cmp(&self, other: &Self) -> Ordering { + self.as_str().cmp(other.as_str()) + } +} + +impl AsRef for Id { + fn as_ref(&self) -> &str { + self.as_str() + } +} + +impl PartialEq<&Id> for Id { + fn eq(&self, other: &&Id) -> bool { + self == *other + } +} + +impl PartialEq for &Id { + fn eq(&self, other: &Id) -> bool { + *self == other + } +} + +/// A lazily-initialized, cached [`Id`] for use as a `static`. +/// +/// Avoids repeated interner write-lock acquisition by storing the interned [`Id`] +/// on first access and returning the cached copy on all subsequent calls. +/// +/// # Example +/// +/// ``` +/// use vortex_session::registry::{CachedId, Id}; +/// +/// static MY_ID: CachedId = CachedId::new("my.encoding"); +/// +/// fn get_id() -> Id { +/// *MY_ID +/// } +/// ``` +pub struct CachedId { + s: &'static str, + cached: OnceLock, +} + +impl CachedId { + /// Create a new `CachedId` that will intern `s` on first access. + pub const fn new(s: &'static str) -> Self { + Self { + s, + cached: OnceLock::new(), + } + } +} + +impl Deref for CachedId { + type Target = Id; + + fn deref(&self) -> &Id { + self.cached.get_or_init(|| Id::new(self.s)) + } +} /// A registry of items that are keyed by a string identifier. #[derive(Clone, Debug)] @@ -33,7 +156,7 @@ impl Registry { /// List the IDs in the registry. pub fn ids(&self) -> impl Iterator + '_ { - self.0.iter().map(|i| i.key().clone()) + self.0.iter().map(|i| *i.key()) } /// List the items in the registry. @@ -156,7 +279,7 @@ impl Context { idx < u16::MAX as usize, "Cannot have more than u16::MAX items" ); - ids.push(id.clone()); + ids.push(*id); Some(u16::try_from(idx).vortex_expect("checked already")) } diff --git a/vortex-sqllogictest/bin/sqllogictests-runner.rs b/vortex-sqllogictest/bin/sqllogictests-runner.rs index d20b851677f..1dde50ada35 100644 --- a/vortex-sqllogictest/bin/sqllogictests-runner.rs +++ b/vortex-sqllogictest/bin/sqllogictests-runner.rs @@ -17,6 +17,7 @@ use indicatif::MultiProgress; use indicatif::ProgressBar; use indicatif::ProgressDrawTarget; use indicatif::ProgressStyle; +use sqllogictest::Normalizer; use sqllogictest::Record; use sqllogictest::Runner; use sqllogictest::parse_file; @@ -28,6 +29,18 @@ use vortex_sqllogictest::duckdb::DuckDB; use vortex_sqllogictest::duckdb::DuckDBTestError; use vortex_sqllogictest::utils::list_files; +fn duckdb_validator(normalizer: Normalizer, actual: &[Vec], expected: &[String]) -> bool { + let actual = actual.iter().flat_map(|strings| { + strings + .join(" ") + .trim_end() + .split('\n') + .map(|line| line.trim_end().to_string()) + .collect::>() + }); + Iterator::eq(actual, expected.iter().map(normalizer)) +} + #[tokio::main] async fn main() -> anyhow::Result<()> { let args = Args::parse(); @@ -127,6 +140,7 @@ async fn main() -> anyhow::Result<()> { duckdb_runner.add_label("duckdb"); duckdb_runner.with_column_validator(strict_column_validator); duckdb_runner.with_normalizer(value_normalizer); + duckdb_runner.with_validator(duckdb_validator); for record in records.iter() { if let Record::Halt { .. } = record { diff --git a/vortex-sqllogictest/build.rs b/vortex-sqllogictest/build.rs index a62b3e39458..177aadb6cd5 100644 --- a/vortex-sqllogictest/build.rs +++ b/vortex-sqllogictest/build.rs @@ -1,7 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] +#![expect(clippy::unwrap_used)] fn main() { // Propagate DuckDB rpath from vortex-duckdb diff --git a/vortex-sqllogictest/slt/duckdb/explain.slt b/vortex-sqllogictest/slt/duckdb/explain.slt new file mode 100644 index 00000000000..1917d9f0494 --- /dev/null +++ b/vortex-sqllogictest/slt/duckdb/explain.slt @@ -0,0 +1,24 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright the Vortex contributors + +include ../setup.slt + +query I +COPY (SELECT * FROM (VALUES ('Hello'), ('Hi'), ('Hey')) AS t(str)) TO '$__TEST_DIR__/explain.vortex'; +---- +3 + +query TT +EXPLAIN (FORMAT json) SELECT * FROM '$__TEST_DIR__/explain.vortex'; +---- +physical_plan [ + { + "name": "READ_VORTEX", + "children": [], + "extra_info": { + "Function": "Vortex Scan", + "Projections": "str", + "Estimated Cardinality": "3" + } + } +] diff --git a/vortex-sqllogictest/slt/duckdb/file_index.slt b/vortex-sqllogictest/slt/duckdb/file_index.slt new file mode 100644 index 00000000000..ea2693e81a3 --- /dev/null +++ b/vortex-sqllogictest/slt/duckdb/file_index.slt @@ -0,0 +1,50 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright the Vortex contributors + +include ../setup.slt + +query I +COPY (SELECT * FROM (VALUES ('Hello'), ('Hi'), ) AS t(str)) TO '$__TEST_DIR__/file_index_1.vortex'; +---- +2 + +query I +COPY (SELECT * FROM (VALUES ('1'), ('2'), ('3')) AS t(str)) TO '$__TEST_DIR__/file_index_2.vortex'; +---- +3 + +query TI +SELECT str, file_index FROM '$__TEST_DIR__/file_index_1.vortex'; +---- +Hello 0 +Hi 0 + +query IT +SELECT file_index, str FROM '$__TEST_DIR__/file_index_2.vortex'; +---- +0 1 +0 2 +0 3 + +query TB +SELECT *, file_index < 2 FROM '$__TEST_DIR__/*.vortex' +ORDER BY str; +---- +1 true +2 true +3 true +Hello true +Hi true + +query IB +SELECT count(*) AS cnt, sum(file_index) <= 3 FROM '$__TEST_DIR__/*.vortex' +ORDER BY cnt; +---- +5 true + +query B +SELECT file_index < 2 FROM '$__TEST_DIR__/*.vortex' +WHERE len(str) > 1; +---- +true +true diff --git a/vortex-sqllogictest/slt/duckdb/file_row_number.slt b/vortex-sqllogictest/slt/duckdb/file_row_number.slt new file mode 100644 index 00000000000..4cb0a459480 --- /dev/null +++ b/vortex-sqllogictest/slt/duckdb/file_row_number.slt @@ -0,0 +1,56 @@ +# SPDX-License-Identifier: Apache-2.0 +# SPDX-FileCopyrightText: Copyright the Vortex contributors + +include ../setup.slt + +query I +COPY (SELECT * FROM (VALUES ('Hello'), ('Hi')) AS t(str)) TO '$__TEST_DIR__/file_row_1.vortex'; +---- +2 + +query I +COPY (SELECT * FROM (VALUES ('1'), ('2'), ('3'), ('45')) AS t(str)) TO '$__TEST_DIR__/file_row_2.vortex'; +---- +4 + +query TI +SELECT str, file_row_number FROM '$__TEST_DIR__/file_row_1.vortex'; +---- +Hello 0 +Hi 1 + +query IT +SELECT file_row_number, str FROM '$__TEST_DIR__/file_row_2.vortex'; +---- +0 1 +1 2 +2 3 +3 45 + +query IT +SELECT file_row_number, str FROM '$__TEST_DIR__/file_row_2.vortex' +WHERE len(str) = 1; +---- +0 1 +1 2 +2 3 + +query TI +SELECT *, file_row_number FROM '$__TEST_DIR__/*.vortex' +ORDER BY str; +---- +1 0 +2 1 +3 2 +45 3 +Hello 0 +Hi 1 + +query I +SELECT file_row_number FROM '$__TEST_DIR__/*.vortex' +WHERE len(str) > 1 +ORDER BY file_row_number; +---- +0 +1 +3 diff --git a/vortex-sqllogictest/src/utils.rs b/vortex-sqllogictest/src/utils.rs index 5d4e1bf2d93..f5ee0e6b12c 100644 --- a/vortex-sqllogictest/src/utils.rs +++ b/vortex-sqllogictest/src/utils.rs @@ -34,7 +34,7 @@ fn list_files_impl(file_paths: &mut Vec, path: impl AsRef) -> any Ok(()) } -#[allow(clippy::unwrap_used)] +#[expect(clippy::unwrap_used)] pub fn pb_style() -> ProgressStyle { ProgressStyle::with_template("[{elapsed_precise}] {wide_bar} {pos:>7}/{len:7} {msg}") .unwrap() diff --git a/vortex-tensor/Cargo.toml b/vortex-tensor/Cargo.toml index 9f94a0c2d3d..2f92ce5a107 100644 --- a/vortex-tensor/Cargo.toml +++ b/vortex-tensor/Cargo.toml @@ -29,8 +29,11 @@ half = { workspace = true } itertools = { workspace = true } num-traits = { workspace = true } prost = { workspace = true } -rand = { workspace = true } [dev-dependencies] +divan = { workspace = true } +mimalloc = { workspace = true } +rand = { workspace = true } rand_distr = { workspace = true } rstest = { workspace = true } +vortex-btrblocks = { path = "../vortex-btrblocks" } diff --git a/vortex-tensor/public-api.lock b/vortex-tensor/public-api.lock index 8a14e2204f0..1ebb5962a90 100644 --- a/vortex-tensor/public-api.lock +++ b/vortex-tensor/public-api.lock @@ -2,79 +2,25 @@ pub mod vortex_tensor pub mod vortex_tensor::encodings -pub mod vortex_tensor::encodings::turboquant - -pub struct vortex_tensor::encodings::turboquant::TurboQuant - -impl vortex_tensor::encodings::turboquant::TurboQuant - -pub const vortex_tensor::encodings::turboquant::TurboQuant::ID: vortex_array::array::ArrayId - -pub const vortex_tensor::encodings::turboquant::TurboQuant::MAX_BIT_WIDTH: u8 - -pub const vortex_tensor::encodings::turboquant::TurboQuant::MAX_CENTROIDS: usize - -pub const vortex_tensor::encodings::turboquant::TurboQuant::MIN_DIMENSION: u32 - -pub unsafe fn vortex_tensor::encodings::turboquant::TurboQuant::new_array_unchecked(dtype: vortex_array::dtype::DType, codes: vortex_array::array::erased::ArrayRef, centroids: vortex_array::array::erased::ArrayRef, rotation_signs: vortex_array::array::erased::ArrayRef) -> vortex_tensor::encodings::turboquant::TurboQuantArray - -pub fn vortex_tensor::encodings::turboquant::TurboQuant::try_new_array(dtype: vortex_array::dtype::DType, codes: vortex_array::array::erased::ArrayRef, centroids: vortex_array::array::erased::ArrayRef, rotation_signs: vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult - -pub fn vortex_tensor::encodings::turboquant::TurboQuant::validate_dtype(dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult - -impl core::clone::Clone for vortex_tensor::encodings::turboquant::TurboQuant - -pub fn vortex_tensor::encodings::turboquant::TurboQuant::clone(&self) -> vortex_tensor::encodings::turboquant::TurboQuant - -impl core::fmt::Debug for vortex_tensor::encodings::turboquant::TurboQuant - -pub fn vortex_tensor::encodings::turboquant::TurboQuant::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result - -impl vortex_array::array::vtable::VTable for vortex_tensor::encodings::turboquant::TurboQuant - -pub type vortex_tensor::encodings::turboquant::TurboQuant::ArrayData = vortex_tensor::encodings::turboquant::TurboQuantData - -pub type vortex_tensor::encodings::turboquant::TurboQuant::OperationsVTable = vortex_tensor::encodings::turboquant::TurboQuant - -pub type vortex_tensor::encodings::turboquant::TurboQuant::ValidityVTable = vortex_tensor::encodings::turboquant::TurboQuant - -pub fn vortex_tensor::encodings::turboquant::TurboQuant::buffer(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle - -pub fn vortex_tensor::encodings::turboquant::TurboQuant::buffer_name(_array: vortex_array::array::view::ArrayView<'_, Self>, _idx: usize) -> core::option::Option - -pub fn vortex_tensor::encodings::turboquant::TurboQuant::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> - -pub fn vortex_tensor::encodings::turboquant::TurboQuant::execute(array: vortex_array::array::typed::Array, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult - -pub fn vortex_tensor::encodings::turboquant::TurboQuant::execute_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> +pub mod vortex_tensor::encodings::l2_denorm -pub fn vortex_tensor::encodings::turboquant::TurboQuant::id(&self) -> vortex_array::array::ArrayId +pub struct vortex_tensor::encodings::l2_denorm::L2DenormScheme -pub fn vortex_tensor::encodings::turboquant::TurboQuant::nbuffers(_array: vortex_array::array::view::ArrayView<'_, Self>) -> usize +impl core::fmt::Debug for vortex_tensor::encodings::l2_denorm::L2DenormScheme -pub fn vortex_tensor::encodings::turboquant::TurboQuant::reduce_parent(array: vortex_array::array::view::ArrayView<'_, Self>, parent: &vortex_array::array::erased::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_tensor::encodings::l2_denorm::L2DenormScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result -pub fn vortex_tensor::encodings::turboquant::TurboQuant::serialize(array: vortex_array::array::view::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +impl vortex_compressor::scheme::Scheme for vortex_tensor::encodings::l2_denorm::L2DenormScheme -pub fn vortex_tensor::encodings::turboquant::TurboQuant::slot_name(_array: vortex_array::array::view::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_tensor::encodings::l2_denorm::L2DenormScheme::compress(&self, &vortex_compressor::compressor::CascadingCompressor, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_tensor::encodings::turboquant::TurboQuant::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_tensor::encodings::l2_denorm::L2DenormScheme::expected_compression_ratio(&self, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -impl vortex_array::array::vtable::operations::OperationsVTable for vortex_tensor::encodings::turboquant::TurboQuant +pub fn vortex_tensor::encodings::l2_denorm::L2DenormScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool -pub fn vortex_tensor::encodings::turboquant::TurboQuant::scalar_at(array: vortex_array::array::view::ArrayView<'_, vortex_tensor::encodings::turboquant::TurboQuant>, index: usize, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_tensor::encodings::l2_denorm::L2DenormScheme::scheme_name(&self) -> &'static str -impl vortex_array::array::vtable::validity::ValidityVTable for vortex_tensor::encodings::turboquant::TurboQuant - -pub fn vortex_tensor::encodings::turboquant::TurboQuant::validity(_array: vortex_array::array::view::ArrayView<'_, vortex_tensor::encodings::turboquant::TurboQuant>) -> vortex_error::VortexResult - -impl vortex_array::arrays::dict::take::TakeExecute for vortex_tensor::encodings::turboquant::TurboQuant - -pub fn vortex_tensor::encodings::turboquant::TurboQuant::take(array: vortex_array::array::view::ArrayView<'_, vortex_tensor::encodings::turboquant::TurboQuant>, indices: &vortex_array::array::erased::ArrayRef, _ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult> - -impl vortex_array::arrays::slice::SliceReduce for vortex_tensor::encodings::turboquant::TurboQuant - -pub fn vortex_tensor::encodings::turboquant::TurboQuant::slice(array: vortex_array::array::view::ArrayView<'_, vortex_tensor::encodings::turboquant::TurboQuant>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub mod vortex_tensor::encodings::turboquant pub struct vortex_tensor::encodings::turboquant::TurboQuantConfig @@ -82,7 +28,7 @@ pub vortex_tensor::encodings::turboquant::TurboQuantConfig::bit_width: u8 pub vortex_tensor::encodings::turboquant::TurboQuantConfig::num_rounds: u8 -pub vortex_tensor::encodings::turboquant::TurboQuantConfig::seed: core::option::Option +pub vortex_tensor::encodings::turboquant::TurboQuantConfig::seed: u64 impl core::clone::Clone for vortex_tensor::encodings::turboquant::TurboQuantConfig @@ -94,45 +40,7 @@ pub fn vortex_tensor::encodings::turboquant::TurboQuantConfig::default() -> Self impl core::fmt::Debug for vortex_tensor::encodings::turboquant::TurboQuantConfig -pub fn vortex_tensor::encodings::turboquant::TurboQuantConfig::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result - -pub struct vortex_tensor::encodings::turboquant::TurboQuantData - -impl vortex_tensor::encodings::turboquant::TurboQuantData - -pub fn vortex_tensor::encodings::turboquant::TurboQuantData::bit_width(&self) -> u8 - -pub fn vortex_tensor::encodings::turboquant::TurboQuantData::dimension(&self) -> u32 - -pub unsafe fn vortex_tensor::encodings::turboquant::TurboQuantData::new_unchecked(dimension: u32, bit_width: u8, num_rounds: u8) -> Self - -pub fn vortex_tensor::encodings::turboquant::TurboQuantData::num_rounds(&self) -> u8 - -pub fn vortex_tensor::encodings::turboquant::TurboQuantData::padded_dim(&self) -> u32 - -pub fn vortex_tensor::encodings::turboquant::TurboQuantData::try_new(dimension: u32, bit_width: u8, num_rounds: u8) -> vortex_error::VortexResult - -pub fn vortex_tensor::encodings::turboquant::TurboQuantData::validate(dtype: &vortex_array::dtype::DType, codes: &vortex_array::array::erased::ArrayRef, centroids: &vortex_array::array::erased::ArrayRef, rotation_signs: &vortex_array::array::erased::ArrayRef) -> vortex_error::VortexResult<()> - -impl core::clone::Clone for vortex_tensor::encodings::turboquant::TurboQuantData - -pub fn vortex_tensor::encodings::turboquant::TurboQuantData::clone(&self) -> vortex_tensor::encodings::turboquant::TurboQuantData - -impl core::fmt::Debug for vortex_tensor::encodings::turboquant::TurboQuantData - -pub fn vortex_tensor::encodings::turboquant::TurboQuantData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result - -impl core::fmt::Display for vortex_tensor::encodings::turboquant::TurboQuantData - -pub fn vortex_tensor::encodings::turboquant::TurboQuantData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result - -impl vortex_array::hash::ArrayEq for vortex_tensor::encodings::turboquant::TurboQuantData - -pub fn vortex_tensor::encodings::turboquant::TurboQuantData::array_eq(&self, other: &Self, _precision: vortex_array::hash::Precision) -> bool - -impl vortex_array::hash::ArrayHash for vortex_tensor::encodings::turboquant::TurboQuantData - -pub fn vortex_tensor::encodings::turboquant::TurboQuantData::array_hash(&self, state: &mut H, _precision: vortex_array::hash::Precision) +pub fn vortex_tensor::encodings::turboquant::TurboQuantConfig::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_tensor::encodings::turboquant::TurboQuantScheme @@ -144,11 +52,11 @@ impl core::cmp::Eq for vortex_tensor::encodings::turboquant::TurboQuantScheme impl core::cmp::PartialEq for vortex_tensor::encodings::turboquant::TurboQuantScheme -pub fn vortex_tensor::encodings::turboquant::TurboQuantScheme::eq(&self, other: &vortex_tensor::encodings::turboquant::TurboQuantScheme) -> bool +pub fn vortex_tensor::encodings::turboquant::TurboQuantScheme::eq(&self, &vortex_tensor::encodings::turboquant::TurboQuantScheme) -> bool impl core::fmt::Debug for vortex_tensor::encodings::turboquant::TurboQuantScheme -pub fn vortex_tensor::encodings::turboquant::TurboQuantScheme::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_tensor::encodings::turboquant::TurboQuantScheme::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_tensor::encodings::turboquant::TurboQuantScheme @@ -156,167 +64,159 @@ impl core::marker::StructuralPartialEq for vortex_tensor::encodings::turboquant: impl vortex_compressor::scheme::Scheme for vortex_tensor::encodings::turboquant::TurboQuantScheme -pub fn vortex_tensor::encodings::turboquant::TurboQuantScheme::compress(&self, compressor: &vortex_compressor::compressor::CascadingCompressor, data: &mut vortex_compressor::stats::cache::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_error::VortexResult +pub fn vortex_tensor::encodings::turboquant::TurboQuantScheme::compress(&self, &vortex_compressor::compressor::CascadingCompressor, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_tensor::encodings::turboquant::TurboQuantScheme::expected_compression_ratio(&self, data: &mut vortex_compressor::stats::cache::ArrayAndStats, _ctx: vortex_compressor::ctx::CompressorContext) -> vortex_compressor::estimate::CompressionEstimate +pub fn vortex_tensor::encodings::turboquant::TurboQuantScheme::expected_compression_ratio(&self, &vortex_compressor::stats::cache::ArrayAndStats, vortex_compressor::ctx::CompressorContext, &mut vortex_array::executor::ExecutionCtx) -> vortex_compressor::estimate::CompressionEstimate -pub fn vortex_tensor::encodings::turboquant::TurboQuantScheme::matches(&self, canonical: &vortex_array::canonical::Canonical) -> bool +pub fn vortex_tensor::encodings::turboquant::TurboQuantScheme::matches(&self, &vortex_array::canonical::Canonical) -> bool pub fn vortex_tensor::encodings::turboquant::TurboQuantScheme::scheme_name(&self) -> &'static str -pub trait vortex_tensor::encodings::turboquant::TurboQuantArrayExt: vortex_array::array::typed::TypedArrayRef - -pub fn vortex_tensor::encodings::turboquant::TurboQuantArrayExt::centroids(&self) -> &vortex_array::array::erased::ArrayRef - -pub fn vortex_tensor::encodings::turboquant::TurboQuantArrayExt::codes(&self) -> &vortex_array::array::erased::ArrayRef - -pub fn vortex_tensor::encodings::turboquant::TurboQuantArrayExt::rotation_signs(&self) -> &vortex_array::array::erased::ArrayRef - -impl> vortex_tensor::encodings::turboquant::TurboQuantArrayExt for T +pub const vortex_tensor::encodings::turboquant::MAX_BIT_WIDTH: u8 -pub fn T::centroids(&self) -> &vortex_array::array::erased::ArrayRef +pub const vortex_tensor::encodings::turboquant::MAX_CENTROIDS: usize -pub fn T::codes(&self) -> &vortex_array::array::erased::ArrayRef +pub const vortex_tensor::encodings::turboquant::MIN_DIMENSION: u32 -pub fn T::rotation_signs(&self) -> &vortex_array::array::erased::ArrayRef +pub fn vortex_tensor::encodings::turboquant::tq_validate_vector_dtype(&vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_tensor::encodings::turboquant::turboquant_encode(ext: vortex_array::array::view::ArrayView<'_, vortex_array::arrays::extension::vtable::Extension>, config: &vortex_tensor::encodings::turboquant::TurboQuantConfig, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_tensor::encodings::turboquant::turboquant_encode(vortex_array::array::erased::ArrayRef, &vortex_tensor::encodings::turboquant::TurboQuantConfig, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub unsafe fn vortex_tensor::encodings::turboquant::turboquant_encode_unchecked(ext: vortex_array::array::view::ArrayView<'_, vortex_array::arrays::extension::vtable::Extension>, config: &vortex_tensor::encodings::turboquant::TurboQuantConfig, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub unsafe fn vortex_tensor::encodings::turboquant::turboquant_encode_unchecked(vortex_array::array::view::ArrayView<'_, vortex_array::arrays::extension::vtable::Extension>, &vortex_tensor::encodings::turboquant::TurboQuantConfig, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub type vortex_tensor::encodings::turboquant::TurboQuantArray = vortex_array::array::typed::Array +pub mod vortex_tensor::fixed_shape_tensor -pub mod vortex_tensor::fixed_shape +pub struct vortex_tensor::fixed_shape_tensor::AnyFixedShapeTensor -pub struct vortex_tensor::fixed_shape::AnyFixedShapeTensor +impl vortex_array::dtype::extension::matcher::Matcher for vortex_tensor::fixed_shape_tensor::AnyFixedShapeTensor -impl vortex_array::dtype::extension::matcher::Matcher for vortex_tensor::fixed_shape::AnyFixedShapeTensor +pub type vortex_tensor::fixed_shape_tensor::AnyFixedShapeTensor::Match<'a> = vortex_tensor::fixed_shape_tensor::FixedShapeTensorMatcherMetadata<'a> -pub type vortex_tensor::fixed_shape::AnyFixedShapeTensor::Match<'a> = vortex_tensor::fixed_shape::FixedShapeTensorMatcherMetadata<'a> +pub fn vortex_tensor::fixed_shape_tensor::AnyFixedShapeTensor::try_match<'a>(&'a vortex_array::dtype::extension::erased::ExtDTypeRef) -> core::option::Option -pub fn vortex_tensor::fixed_shape::AnyFixedShapeTensor::try_match<'a>(ext_dtype: &'a vortex_array::dtype::extension::erased::ExtDTypeRef) -> core::option::Option +pub struct vortex_tensor::fixed_shape_tensor::FixedShapeTensor -pub struct vortex_tensor::fixed_shape::FixedShapeTensor +impl core::clone::Clone for vortex_tensor::fixed_shape_tensor::FixedShapeTensor -impl core::clone::Clone for vortex_tensor::fixed_shape::FixedShapeTensor +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensor::clone(&self) -> vortex_tensor::fixed_shape_tensor::FixedShapeTensor -pub fn vortex_tensor::fixed_shape::FixedShapeTensor::clone(&self) -> vortex_tensor::fixed_shape::FixedShapeTensor +impl core::cmp::Eq for vortex_tensor::fixed_shape_tensor::FixedShapeTensor -impl core::cmp::Eq for vortex_tensor::fixed_shape::FixedShapeTensor +impl core::cmp::PartialEq for vortex_tensor::fixed_shape_tensor::FixedShapeTensor -impl core::cmp::PartialEq for vortex_tensor::fixed_shape::FixedShapeTensor +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensor::eq(&self, &vortex_tensor::fixed_shape_tensor::FixedShapeTensor) -> bool -pub fn vortex_tensor::fixed_shape::FixedShapeTensor::eq(&self, other: &vortex_tensor::fixed_shape::FixedShapeTensor) -> bool +impl core::default::Default for vortex_tensor::fixed_shape_tensor::FixedShapeTensor -impl core::default::Default for vortex_tensor::fixed_shape::FixedShapeTensor +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensor::default() -> vortex_tensor::fixed_shape_tensor::FixedShapeTensor -pub fn vortex_tensor::fixed_shape::FixedShapeTensor::default() -> vortex_tensor::fixed_shape::FixedShapeTensor +impl core::fmt::Debug for vortex_tensor::fixed_shape_tensor::FixedShapeTensor -impl core::fmt::Debug for vortex_tensor::fixed_shape::FixedShapeTensor +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result -pub fn vortex_tensor::fixed_shape::FixedShapeTensor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +impl core::hash::Hash for vortex_tensor::fixed_shape_tensor::FixedShapeTensor -impl core::hash::Hash for vortex_tensor::fixed_shape::FixedShapeTensor +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensor::hash<__H: core::hash::Hasher>(&self, &mut __H) -pub fn vortex_tensor::fixed_shape::FixedShapeTensor::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +impl core::marker::StructuralPartialEq for vortex_tensor::fixed_shape_tensor::FixedShapeTensor -impl core::marker::StructuralPartialEq for vortex_tensor::fixed_shape::FixedShapeTensor +impl vortex_array::dtype::extension::vtable::ExtVTable for vortex_tensor::fixed_shape_tensor::FixedShapeTensor -impl vortex_array::dtype::extension::vtable::ExtVTable for vortex_tensor::fixed_shape::FixedShapeTensor +pub type vortex_tensor::fixed_shape_tensor::FixedShapeTensor::Metadata = vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata -pub type vortex_tensor::fixed_shape::FixedShapeTensor::Metadata = vortex_tensor::fixed_shape::FixedShapeTensorMetadata +pub type vortex_tensor::fixed_shape_tensor::FixedShapeTensor::NativeValue<'a> = &'a vortex_array::scalar::scalar_value::ScalarValue -pub type vortex_tensor::fixed_shape::FixedShapeTensor::NativeValue<'a> = &'a vortex_array::scalar::scalar_value::ScalarValue +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensor::deserialize_metadata(&self, &[u8]) -> vortex_error::VortexResult -pub fn vortex_tensor::fixed_shape::FixedShapeTensor::deserialize_metadata(&self, metadata: &[u8]) -> vortex_error::VortexResult +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensor::id(&self) -> vortex_array::dtype::extension::ExtId -pub fn vortex_tensor::fixed_shape::FixedShapeTensor::id(&self) -> vortex_array::dtype::extension::ExtId +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensor::least_supertype(&vortex_array::dtype::extension::typed::ExtDType, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_tensor::fixed_shape::FixedShapeTensor::serialize_metadata(&self, metadata: &Self::Metadata) -> vortex_error::VortexResult> +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensor::serialize_metadata(&self, &Self::Metadata) -> vortex_error::VortexResult> -pub fn vortex_tensor::fixed_shape::FixedShapeTensor::unpack_native<'a>(_ext_dtype: &'a vortex_array::dtype::extension::typed::ExtDType, storage_value: &'a vortex_array::scalar::scalar_value::ScalarValue) -> vortex_error::VortexResult +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensor::unpack_native<'a>(&'a vortex_array::dtype::extension::typed::ExtDType, &'a vortex_array::scalar::scalar_value::ScalarValue) -> vortex_error::VortexResult -pub fn vortex_tensor::fixed_shape::FixedShapeTensor::validate_dtype(ext_dtype: &vortex_array::dtype::extension::typed::ExtDType) -> vortex_error::VortexResult<()> +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensor::validate_dtype(&vortex_array::dtype::extension::typed::ExtDType) -> vortex_error::VortexResult<()> -pub struct vortex_tensor::fixed_shape::FixedShapeTensorMatcherMetadata<'a> +pub struct vortex_tensor::fixed_shape_tensor::FixedShapeTensorMatcherMetadata<'a> -impl vortex_tensor::fixed_shape::FixedShapeTensorMatcherMetadata<'_> +impl vortex_tensor::fixed_shape_tensor::FixedShapeTensorMatcherMetadata<'_> -pub fn vortex_tensor::fixed_shape::FixedShapeTensorMatcherMetadata<'_>::element_ptype(&self) -> vortex_array::dtype::ptype::PType +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensorMatcherMetadata<'_>::element_ptype(&self) -> vortex_array::dtype::ptype::PType -pub fn vortex_tensor::fixed_shape::FixedShapeTensorMatcherMetadata<'_>::list_size(&self) -> usize +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensorMatcherMetadata<'_>::flat_list_size(&self) -> u32 -pub fn vortex_tensor::fixed_shape::FixedShapeTensorMatcherMetadata<'_>::metadata(&self) -> &vortex_tensor::fixed_shape::FixedShapeTensorMetadata +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensorMatcherMetadata<'_>::metadata(&self) -> &vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata -impl<'a> core::clone::Clone for vortex_tensor::fixed_shape::FixedShapeTensorMatcherMetadata<'a> +impl<'a> core::clone::Clone for vortex_tensor::fixed_shape_tensor::FixedShapeTensorMatcherMetadata<'a> -pub fn vortex_tensor::fixed_shape::FixedShapeTensorMatcherMetadata<'a>::clone(&self) -> vortex_tensor::fixed_shape::FixedShapeTensorMatcherMetadata<'a> +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensorMatcherMetadata<'a>::clone(&self) -> vortex_tensor::fixed_shape_tensor::FixedShapeTensorMatcherMetadata<'a> -impl<'a> core::cmp::Eq for vortex_tensor::fixed_shape::FixedShapeTensorMatcherMetadata<'a> +impl<'a> core::cmp::Eq for vortex_tensor::fixed_shape_tensor::FixedShapeTensorMatcherMetadata<'a> -impl<'a> core::cmp::PartialEq for vortex_tensor::fixed_shape::FixedShapeTensorMatcherMetadata<'a> +impl<'a> core::cmp::PartialEq for vortex_tensor::fixed_shape_tensor::FixedShapeTensorMatcherMetadata<'a> -pub fn vortex_tensor::fixed_shape::FixedShapeTensorMatcherMetadata<'a>::eq(&self, other: &vortex_tensor::fixed_shape::FixedShapeTensorMatcherMetadata<'a>) -> bool +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensorMatcherMetadata<'a>::eq(&self, &vortex_tensor::fixed_shape_tensor::FixedShapeTensorMatcherMetadata<'a>) -> bool -impl<'a> core::fmt::Debug for vortex_tensor::fixed_shape::FixedShapeTensorMatcherMetadata<'a> +impl<'a> core::fmt::Debug for vortex_tensor::fixed_shape_tensor::FixedShapeTensorMatcherMetadata<'a> -pub fn vortex_tensor::fixed_shape::FixedShapeTensorMatcherMetadata<'a>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensorMatcherMetadata<'a>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result -impl<'a> core::marker::Copy for vortex_tensor::fixed_shape::FixedShapeTensorMatcherMetadata<'a> +impl<'a> core::marker::Copy for vortex_tensor::fixed_shape_tensor::FixedShapeTensorMatcherMetadata<'a> -impl<'a> core::marker::StructuralPartialEq for vortex_tensor::fixed_shape::FixedShapeTensorMatcherMetadata<'a> +impl<'a> core::marker::StructuralPartialEq for vortex_tensor::fixed_shape_tensor::FixedShapeTensorMatcherMetadata<'a> -pub struct vortex_tensor::fixed_shape::FixedShapeTensorMetadata +pub struct vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata -impl vortex_tensor::fixed_shape::FixedShapeTensorMetadata +impl vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata -pub fn vortex_tensor::fixed_shape::FixedShapeTensorMetadata::dim_names(&self) -> core::option::Option<&[alloc::string::String]> +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata::dim_names(&self) -> core::option::Option<&[alloc::string::String]> -pub fn vortex_tensor::fixed_shape::FixedShapeTensorMetadata::logical_shape(&self) -> &[usize] +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata::logical_shape(&self) -> &[usize] -pub fn vortex_tensor::fixed_shape::FixedShapeTensorMetadata::ndim(&self) -> usize +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata::ndim(&self) -> usize -pub fn vortex_tensor::fixed_shape::FixedShapeTensorMetadata::new(shape: alloc::vec::Vec) -> Self +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata::new(alloc::vec::Vec) -> Self -pub fn vortex_tensor::fixed_shape::FixedShapeTensorMetadata::permutation(&self) -> core::option::Option<&[usize]> +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata::permutation(&self) -> core::option::Option<&[usize]> -pub fn vortex_tensor::fixed_shape::FixedShapeTensorMetadata::physical_shape(&self) -> impl core::iter::traits::iterator::Iterator + '_ +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata::physical_shape(&self) -> impl core::iter::traits::iterator::Iterator + '_ -pub fn vortex_tensor::fixed_shape::FixedShapeTensorMetadata::strides(&self) -> impl core::iter::traits::iterator::Iterator + '_ +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata::strides(&self) -> impl core::iter::traits::iterator::Iterator + '_ -pub fn vortex_tensor::fixed_shape::FixedShapeTensorMetadata::with_dim_names(self, names: alloc::vec::Vec) -> vortex_error::VortexResult +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata::with_dim_names(self, alloc::vec::Vec) -> vortex_error::VortexResult -pub fn vortex_tensor::fixed_shape::FixedShapeTensorMetadata::with_permutation(self, permutation: alloc::vec::Vec) -> vortex_error::VortexResult +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata::with_permutation(self, alloc::vec::Vec) -> vortex_error::VortexResult -impl core::clone::Clone for vortex_tensor::fixed_shape::FixedShapeTensorMetadata +impl core::clone::Clone for vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata -pub fn vortex_tensor::fixed_shape::FixedShapeTensorMetadata::clone(&self) -> vortex_tensor::fixed_shape::FixedShapeTensorMetadata +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata::clone(&self) -> vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata -impl core::cmp::Eq for vortex_tensor::fixed_shape::FixedShapeTensorMetadata +impl core::cmp::Eq for vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata -impl core::cmp::PartialEq for vortex_tensor::fixed_shape::FixedShapeTensorMetadata +impl core::cmp::PartialEq for vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata -pub fn vortex_tensor::fixed_shape::FixedShapeTensorMetadata::eq(&self, other: &vortex_tensor::fixed_shape::FixedShapeTensorMetadata) -> bool +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata::eq(&self, &vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata) -> bool -impl core::fmt::Debug for vortex_tensor::fixed_shape::FixedShapeTensorMetadata +impl core::fmt::Debug for vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata -pub fn vortex_tensor::fixed_shape::FixedShapeTensorMetadata::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result -impl core::fmt::Display for vortex_tensor::fixed_shape::FixedShapeTensorMetadata +impl core::fmt::Display for vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata -pub fn vortex_tensor::fixed_shape::FixedShapeTensorMetadata::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result -impl core::hash::Hash for vortex_tensor::fixed_shape::FixedShapeTensorMetadata +impl core::hash::Hash for vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata -pub fn vortex_tensor::fixed_shape::FixedShapeTensorMetadata::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata::hash<__H: core::hash::Hasher>(&self, &mut __H) -impl core::marker::StructuralPartialEq for vortex_tensor::fixed_shape::FixedShapeTensorMetadata +impl core::marker::StructuralPartialEq for vortex_tensor::fixed_shape_tensor::FixedShapeTensorMetadata pub mod vortex_tensor::matcher pub enum vortex_tensor::matcher::TensorMatch<'a> -pub vortex_tensor::matcher::TensorMatch::FixedShapeTensor(vortex_tensor::fixed_shape::FixedShapeTensorMatcherMetadata<'a>) +pub vortex_tensor::matcher::TensorMatch::FixedShapeTensor(vortex_tensor::fixed_shape_tensor::FixedShapeTensorMatcherMetadata<'a>) pub vortex_tensor::matcher::TensorMatch::Vector(vortex_tensor::vector::VectorMatcherMetadata) @@ -324,7 +224,7 @@ impl vortex_tensor::matcher::TensorMatch<'_> pub fn vortex_tensor::matcher::TensorMatch<'_>::element_ptype(self) -> vortex_array::dtype::ptype::PType -pub fn vortex_tensor::matcher::TensorMatch<'_>::list_size(self) -> usize +pub fn vortex_tensor::matcher::TensorMatch<'_>::list_size(self) -> u32 impl<'a> core::clone::Clone for vortex_tensor::matcher::TensorMatch<'a> @@ -334,11 +234,11 @@ impl<'a> core::cmp::Eq for vortex_tensor::matcher::TensorMatch<'a> impl<'a> core::cmp::PartialEq for vortex_tensor::matcher::TensorMatch<'a> -pub fn vortex_tensor::matcher::TensorMatch<'a>::eq(&self, other: &vortex_tensor::matcher::TensorMatch<'a>) -> bool +pub fn vortex_tensor::matcher::TensorMatch<'a>::eq(&self, &vortex_tensor::matcher::TensorMatch<'a>) -> bool impl<'a> core::fmt::Debug for vortex_tensor::matcher::TensorMatch<'a> -pub fn vortex_tensor::matcher::TensorMatch<'a>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_tensor::matcher::TensorMatch<'a>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl<'a> core::marker::Copy for vortex_tensor::matcher::TensorMatch<'a> @@ -350,7 +250,7 @@ impl vortex_array::dtype::extension::matcher::Matcher for vortex_tensor::matcher pub type vortex_tensor::matcher::AnyTensor::Match<'a> = vortex_tensor::matcher::TensorMatch<'a> -pub fn vortex_tensor::matcher::AnyTensor::try_match<'a>(ext_dtype: &'a vortex_array::dtype::extension::erased::ExtDTypeRef) -> core::option::Option +pub fn vortex_tensor::matcher::AnyTensor::try_match<'a>(&'a vortex_array::dtype::extension::erased::ExtDTypeRef) -> core::option::Option pub mod vortex_tensor::scalar_fns @@ -360,35 +260,41 @@ pub struct vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity impl vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity -pub fn vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::new(options: &vortex_tensor::scalar_fns::ApproxOptions) -> vortex_array::scalar_fn::typed::ScalarFn +pub fn vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::new() -> vortex_array::scalar_fn::typed::TypedScalarFnInstance -pub fn vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::try_new_array(options: &vortex_tensor::scalar_fns::ApproxOptions, lhs: vortex_array::array::erased::ArrayRef, rhs: vortex_array::array::erased::ArrayRef, len: usize) -> vortex_error::VortexResult +pub fn vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::try_new_array(vortex_array::array::erased::ArrayRef, vortex_array::array::erased::ArrayRef, usize) -> vortex_error::VortexResult impl core::clone::Clone for vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity pub fn vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::clone(&self) -> vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity +impl vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayVTable for vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity + +pub fn vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> + +pub fn vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::serialize(&self, &vortex_array::arrays::scalar_fn::vtable::ScalarFnArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> + impl vortex_array::scalar_fn::vtable::ScalarFnVTable for vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity -pub type vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::Options = vortex_tensor::scalar_fns::ApproxOptions +pub type vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::Options = vortex_array::scalar_fn::vtable::EmptyOptions -pub fn vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::vtable::Arity +pub fn vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::arity(&self, &Self::Options) -> vortex_array::scalar_fn::vtable::Arity -pub fn vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::child_name(&self, _options: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::vtable::ChildName +pub fn vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::vtable::ChildName -pub fn vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::execute(&self, options: &Self::Options, args: &dyn vortex_array::scalar_fn::vtable::ExecutionArgs, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::vtable::ExecutionArgs, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::fmt_sql(&self, _options: &Self::Options, expr: &vortex_array::expr::expression::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::fmt_sql(&self, &Self::Options, &vortex_array::expr::expression::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::is_fallible(&self, _options: &Self::Options) -> bool +pub fn vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::is_null_sensitive(&self, _options: &Self::Options) -> bool +pub fn vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::return_dtype(&self, _options: &Self::Options, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::validity(&self, _options: &Self::Options, expression: &vortex_array::expr::expression::Expression) -> vortex_error::VortexResult> +pub fn vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity::validity(&self, &Self::Options, &vortex_array::expr::expression::Expression) -> vortex_error::VortexResult> pub mod vortex_tensor::scalar_fns::inner_product @@ -396,35 +302,41 @@ pub struct vortex_tensor::scalar_fns::inner_product::InnerProduct impl vortex_tensor::scalar_fns::inner_product::InnerProduct -pub fn vortex_tensor::scalar_fns::inner_product::InnerProduct::new(options: &vortex_tensor::scalar_fns::ApproxOptions) -> vortex_array::scalar_fn::typed::ScalarFn +pub fn vortex_tensor::scalar_fns::inner_product::InnerProduct::new() -> vortex_array::scalar_fn::typed::TypedScalarFnInstance -pub fn vortex_tensor::scalar_fns::inner_product::InnerProduct::try_new_array(options: &vortex_tensor::scalar_fns::ApproxOptions, lhs: vortex_array::array::erased::ArrayRef, rhs: vortex_array::array::erased::ArrayRef, len: usize) -> vortex_error::VortexResult +pub fn vortex_tensor::scalar_fns::inner_product::InnerProduct::try_new_array(vortex_array::array::erased::ArrayRef, vortex_array::array::erased::ArrayRef, usize) -> vortex_error::VortexResult impl core::clone::Clone for vortex_tensor::scalar_fns::inner_product::InnerProduct pub fn vortex_tensor::scalar_fns::inner_product::InnerProduct::clone(&self) -> vortex_tensor::scalar_fns::inner_product::InnerProduct +impl vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayVTable for vortex_tensor::scalar_fns::inner_product::InnerProduct + +pub fn vortex_tensor::scalar_fns::inner_product::InnerProduct::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> + +pub fn vortex_tensor::scalar_fns::inner_product::InnerProduct::serialize(&self, &vortex_array::arrays::scalar_fn::vtable::ScalarFnArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> + impl vortex_array::scalar_fn::vtable::ScalarFnVTable for vortex_tensor::scalar_fns::inner_product::InnerProduct -pub type vortex_tensor::scalar_fns::inner_product::InnerProduct::Options = vortex_tensor::scalar_fns::ApproxOptions +pub type vortex_tensor::scalar_fns::inner_product::InnerProduct::Options = vortex_array::scalar_fn::vtable::EmptyOptions -pub fn vortex_tensor::scalar_fns::inner_product::InnerProduct::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::vtable::Arity +pub fn vortex_tensor::scalar_fns::inner_product::InnerProduct::arity(&self, &Self::Options) -> vortex_array::scalar_fn::vtable::Arity -pub fn vortex_tensor::scalar_fns::inner_product::InnerProduct::child_name(&self, _options: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::vtable::ChildName +pub fn vortex_tensor::scalar_fns::inner_product::InnerProduct::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::vtable::ChildName -pub fn vortex_tensor::scalar_fns::inner_product::InnerProduct::execute(&self, options: &Self::Options, args: &dyn vortex_array::scalar_fn::vtable::ExecutionArgs, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_tensor::scalar_fns::inner_product::InnerProduct::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::vtable::ExecutionArgs, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_tensor::scalar_fns::inner_product::InnerProduct::fmt_sql(&self, _options: &Self::Options, expr: &vortex_array::expr::expression::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_tensor::scalar_fns::inner_product::InnerProduct::fmt_sql(&self, &Self::Options, &vortex_array::expr::expression::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_tensor::scalar_fns::inner_product::InnerProduct::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_tensor::scalar_fns::inner_product::InnerProduct::is_fallible(&self, _options: &Self::Options) -> bool +pub fn vortex_tensor::scalar_fns::inner_product::InnerProduct::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_tensor::scalar_fns::inner_product::InnerProduct::is_null_sensitive(&self, _options: &Self::Options) -> bool +pub fn vortex_tensor::scalar_fns::inner_product::InnerProduct::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_tensor::scalar_fns::inner_product::InnerProduct::return_dtype(&self, _options: &Self::Options, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_tensor::scalar_fns::inner_product::InnerProduct::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_tensor::scalar_fns::inner_product::InnerProduct::validity(&self, _options: &Self::Options, expression: &vortex_array::expr::expression::Expression) -> vortex_error::VortexResult> +pub fn vortex_tensor::scalar_fns::inner_product::InnerProduct::validity(&self, &Self::Options, &vortex_array::expr::expression::Expression) -> vortex_error::VortexResult> pub mod vortex_tensor::scalar_fns::l2_denorm @@ -432,41 +344,47 @@ pub struct vortex_tensor::scalar_fns::l2_denorm::L2Denorm impl vortex_tensor::scalar_fns::l2_denorm::L2Denorm -pub fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::new(options: &vortex_tensor::scalar_fns::ApproxOptions) -> vortex_array::scalar_fn::typed::ScalarFn +pub fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::new() -> vortex_array::scalar_fn::typed::TypedScalarFnInstance -pub unsafe fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::new_array_unchecked(options: &vortex_tensor::scalar_fns::ApproxOptions, normalized: vortex_array::array::erased::ArrayRef, norms: vortex_array::array::erased::ArrayRef, len: usize) -> vortex_error::VortexResult +pub unsafe fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::new_array_unchecked(vortex_array::array::erased::ArrayRef, vortex_array::array::erased::ArrayRef, usize) -> vortex_error::VortexResult -pub fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::try_new_array(options: &vortex_tensor::scalar_fns::ApproxOptions, normalized: vortex_array::array::erased::ArrayRef, norms: vortex_array::array::erased::ArrayRef, len: usize, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::try_new_array(vortex_array::array::erased::ArrayRef, vortex_array::array::erased::ArrayRef, usize, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult impl core::clone::Clone for vortex_tensor::scalar_fns::l2_denorm::L2Denorm pub fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::clone(&self) -> vortex_tensor::scalar_fns::l2_denorm::L2Denorm +impl vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayVTable for vortex_tensor::scalar_fns::l2_denorm::L2Denorm + +pub fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> + +pub fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::serialize(&self, &vortex_array::arrays::scalar_fn::vtable::ScalarFnArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> + impl vortex_array::scalar_fn::vtable::ScalarFnVTable for vortex_tensor::scalar_fns::l2_denorm::L2Denorm -pub type vortex_tensor::scalar_fns::l2_denorm::L2Denorm::Options = vortex_tensor::scalar_fns::ApproxOptions +pub type vortex_tensor::scalar_fns::l2_denorm::L2Denorm::Options = vortex_array::scalar_fn::vtable::EmptyOptions -pub fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::vtable::Arity +pub fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::arity(&self, &Self::Options) -> vortex_array::scalar_fn::vtable::Arity -pub fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::child_name(&self, _options: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::vtable::ChildName +pub fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::vtable::ChildName -pub fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::execute(&self, _options: &Self::Options, args: &dyn vortex_array::scalar_fn::vtable::ExecutionArgs, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::vtable::ExecutionArgs, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::fmt_sql(&self, _options: &Self::Options, expr: &vortex_array::expr::expression::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::fmt_sql(&self, &Self::Options, &vortex_array::expr::expression::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::is_fallible(&self, _options: &Self::Options) -> bool +pub fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::is_null_sensitive(&self, _options: &Self::Options) -> bool +pub fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::return_dtype(&self, _options: &Self::Options, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::validity(&self, _options: &Self::Options, expression: &vortex_array::expr::expression::Expression) -> vortex_error::VortexResult> +pub fn vortex_tensor::scalar_fns::l2_denorm::L2Denorm::validity(&self, &Self::Options, &vortex_array::expr::expression::Expression) -> vortex_error::VortexResult> -pub fn vortex_tensor::scalar_fns::l2_denorm::normalize_as_l2_denorm(options: &vortex_tensor::scalar_fns::ApproxOptions, input: vortex_array::array::erased::ArrayRef, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_tensor::scalar_fns::l2_denorm::normalize_as_l2_denorm(vortex_array::array::erased::ArrayRef, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_tensor::scalar_fns::l2_denorm::validate_l2_normalized_rows(input: vortex_array::array::erased::ArrayRef, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_tensor::scalar_fns::l2_denorm::validate_l2_normalized_rows_against_norms(&vortex_array::array::erased::ArrayRef, core::option::Option<&vortex_array::array::erased::ArrayRef>, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult<()> pub mod vortex_tensor::scalar_fns::l2_norm @@ -474,75 +392,129 @@ pub struct vortex_tensor::scalar_fns::l2_norm::L2Norm impl vortex_tensor::scalar_fns::l2_norm::L2Norm -pub fn vortex_tensor::scalar_fns::l2_norm::L2Norm::new(options: &vortex_tensor::scalar_fns::ApproxOptions) -> vortex_array::scalar_fn::typed::ScalarFn +pub fn vortex_tensor::scalar_fns::l2_norm::L2Norm::new() -> vortex_array::scalar_fn::typed::TypedScalarFnInstance -pub fn vortex_tensor::scalar_fns::l2_norm::L2Norm::try_new_array(options: &vortex_tensor::scalar_fns::ApproxOptions, child: vortex_array::array::erased::ArrayRef, len: usize) -> vortex_error::VortexResult +pub fn vortex_tensor::scalar_fns::l2_norm::L2Norm::try_new_array(vortex_array::array::erased::ArrayRef, usize) -> vortex_error::VortexResult impl core::clone::Clone for vortex_tensor::scalar_fns::l2_norm::L2Norm pub fn vortex_tensor::scalar_fns::l2_norm::L2Norm::clone(&self) -> vortex_tensor::scalar_fns::l2_norm::L2Norm +impl vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayVTable for vortex_tensor::scalar_fns::l2_norm::L2Norm + +pub fn vortex_tensor::scalar_fns::l2_norm::L2Norm::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> + +pub fn vortex_tensor::scalar_fns::l2_norm::L2Norm::serialize(&self, &vortex_array::arrays::scalar_fn::vtable::ScalarFnArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> + impl vortex_array::scalar_fn::vtable::ScalarFnVTable for vortex_tensor::scalar_fns::l2_norm::L2Norm -pub type vortex_tensor::scalar_fns::l2_norm::L2Norm::Options = vortex_tensor::scalar_fns::ApproxOptions +pub type vortex_tensor::scalar_fns::l2_norm::L2Norm::Options = vortex_array::scalar_fn::vtable::EmptyOptions -pub fn vortex_tensor::scalar_fns::l2_norm::L2Norm::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::vtable::Arity +pub fn vortex_tensor::scalar_fns::l2_norm::L2Norm::arity(&self, &Self::Options) -> vortex_array::scalar_fn::vtable::Arity -pub fn vortex_tensor::scalar_fns::l2_norm::L2Norm::child_name(&self, _options: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::vtable::ChildName +pub fn vortex_tensor::scalar_fns::l2_norm::L2Norm::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::vtable::ChildName -pub fn vortex_tensor::scalar_fns::l2_norm::L2Norm::execute(&self, _options: &Self::Options, args: &dyn vortex_array::scalar_fn::vtable::ExecutionArgs, ctx: &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_tensor::scalar_fns::l2_norm::L2Norm::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::vtable::ExecutionArgs, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_tensor::scalar_fns::l2_norm::L2Norm::fmt_sql(&self, _options: &Self::Options, expr: &vortex_array::expr::expression::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_tensor::scalar_fns::l2_norm::L2Norm::fmt_sql(&self, &Self::Options, &vortex_array::expr::expression::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_tensor::scalar_fns::l2_norm::L2Norm::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_tensor::scalar_fns::l2_norm::L2Norm::is_fallible(&self, _options: &Self::Options) -> bool +pub fn vortex_tensor::scalar_fns::l2_norm::L2Norm::is_fallible(&self, &Self::Options) -> bool + +pub fn vortex_tensor::scalar_fns::l2_norm::L2Norm::is_null_sensitive(&self, &Self::Options) -> bool + +pub fn vortex_tensor::scalar_fns::l2_norm::L2Norm::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult + +pub fn vortex_tensor::scalar_fns::l2_norm::L2Norm::validity(&self, &Self::Options, &vortex_array::expr::expression::Expression) -> vortex_error::VortexResult> + +pub mod vortex_tensor::scalar_fns::sorf_transform + +pub struct vortex_tensor::scalar_fns::sorf_transform::SorfMatrix -pub fn vortex_tensor::scalar_fns::l2_norm::L2Norm::is_null_sensitive(&self, _options: &Self::Options) -> bool +impl vortex_tensor::scalar_fns::sorf_transform::SorfMatrix -pub fn vortex_tensor::scalar_fns::l2_norm::L2Norm::return_dtype(&self, _options: &Self::Options, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_tensor::scalar_fns::sorf_transform::SorfMatrix::inverse_rotate(&self, &[f32], &mut [f32]) -pub fn vortex_tensor::scalar_fns::l2_norm::L2Norm::validity(&self, _options: &Self::Options, expression: &vortex_array::expr::expression::Expression) -> vortex_error::VortexResult> +pub fn vortex_tensor::scalar_fns::sorf_transform::SorfMatrix::padded_dim(&self) -> usize -pub enum vortex_tensor::scalar_fns::ApproxOptions +pub fn vortex_tensor::scalar_fns::sorf_transform::SorfMatrix::rotate(&self, &[f32], &mut [f32]) -pub vortex_tensor::scalar_fns::ApproxOptions::Approximate +pub fn vortex_tensor::scalar_fns::sorf_transform::SorfMatrix::try_new(u64, usize, usize) -> vortex_error::VortexResult -pub vortex_tensor::scalar_fns::ApproxOptions::Exact +pub struct vortex_tensor::scalar_fns::sorf_transform::SorfOptions -impl vortex_tensor::scalar_fns::ApproxOptions +pub vortex_tensor::scalar_fns::sorf_transform::SorfOptions::dimensions: u32 -pub fn vortex_tensor::scalar_fns::ApproxOptions::is_approx(&self) -> bool +pub vortex_tensor::scalar_fns::sorf_transform::SorfOptions::element_ptype: vortex_array::dtype::ptype::PType -pub fn vortex_tensor::scalar_fns::ApproxOptions::is_exact(&self) -> bool +pub vortex_tensor::scalar_fns::sorf_transform::SorfOptions::num_rounds: u8 -impl core::clone::Clone for vortex_tensor::scalar_fns::ApproxOptions +pub vortex_tensor::scalar_fns::sorf_transform::SorfOptions::seed: u64 -pub fn vortex_tensor::scalar_fns::ApproxOptions::clone(&self) -> vortex_tensor::scalar_fns::ApproxOptions +impl core::clone::Clone for vortex_tensor::scalar_fns::sorf_transform::SorfOptions -impl core::cmp::Eq for vortex_tensor::scalar_fns::ApproxOptions +pub fn vortex_tensor::scalar_fns::sorf_transform::SorfOptions::clone(&self) -> vortex_tensor::scalar_fns::sorf_transform::SorfOptions -impl core::cmp::PartialEq for vortex_tensor::scalar_fns::ApproxOptions +impl core::cmp::Eq for vortex_tensor::scalar_fns::sorf_transform::SorfOptions -pub fn vortex_tensor::scalar_fns::ApproxOptions::eq(&self, other: &vortex_tensor::scalar_fns::ApproxOptions) -> bool +impl core::cmp::PartialEq for vortex_tensor::scalar_fns::sorf_transform::SorfOptions -impl core::default::Default for vortex_tensor::scalar_fns::ApproxOptions +pub fn vortex_tensor::scalar_fns::sorf_transform::SorfOptions::eq(&self, &vortex_tensor::scalar_fns::sorf_transform::SorfOptions) -> bool -pub fn vortex_tensor::scalar_fns::ApproxOptions::default() -> vortex_tensor::scalar_fns::ApproxOptions +impl core::fmt::Debug for vortex_tensor::scalar_fns::sorf_transform::SorfOptions -impl core::fmt::Debug for vortex_tensor::scalar_fns::ApproxOptions +pub fn vortex_tensor::scalar_fns::sorf_transform::SorfOptions::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result -pub fn vortex_tensor::scalar_fns::ApproxOptions::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +impl core::fmt::Display for vortex_tensor::scalar_fns::sorf_transform::SorfOptions -impl core::fmt::Display for vortex_tensor::scalar_fns::ApproxOptions +pub fn vortex_tensor::scalar_fns::sorf_transform::SorfOptions::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result -pub fn vortex_tensor::scalar_fns::ApproxOptions::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +impl core::hash::Hash for vortex_tensor::scalar_fns::sorf_transform::SorfOptions -impl core::hash::Hash for vortex_tensor::scalar_fns::ApproxOptions +pub fn vortex_tensor::scalar_fns::sorf_transform::SorfOptions::hash<__H: core::hash::Hasher>(&self, &mut __H) -pub fn vortex_tensor::scalar_fns::ApproxOptions::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +impl core::marker::StructuralPartialEq for vortex_tensor::scalar_fns::sorf_transform::SorfOptions -impl core::marker::StructuralPartialEq for vortex_tensor::scalar_fns::ApproxOptions +pub struct vortex_tensor::scalar_fns::sorf_transform::SorfTransform + +impl vortex_tensor::scalar_fns::sorf_transform::SorfTransform + +pub fn vortex_tensor::scalar_fns::sorf_transform::SorfTransform::new(&vortex_tensor::scalar_fns::sorf_transform::SorfOptions) -> vortex_array::scalar_fn::typed::TypedScalarFnInstance + +pub fn vortex_tensor::scalar_fns::sorf_transform::SorfTransform::try_new_array(&vortex_tensor::scalar_fns::sorf_transform::SorfOptions, vortex_array::array::erased::ArrayRef, usize) -> vortex_error::VortexResult + +impl core::clone::Clone for vortex_tensor::scalar_fns::sorf_transform::SorfTransform + +pub fn vortex_tensor::scalar_fns::sorf_transform::SorfTransform::clone(&self) -> vortex_tensor::scalar_fns::sorf_transform::SorfTransform + +impl vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayVTable for vortex_tensor::scalar_fns::sorf_transform::SorfTransform + +pub fn vortex_tensor::scalar_fns::sorf_transform::SorfTransform::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> + +pub fn vortex_tensor::scalar_fns::sorf_transform::SorfTransform::serialize(&self, &vortex_array::arrays::scalar_fn::vtable::ScalarFnArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> + +impl vortex_array::scalar_fn::vtable::ScalarFnVTable for vortex_tensor::scalar_fns::sorf_transform::SorfTransform + +pub type vortex_tensor::scalar_fns::sorf_transform::SorfTransform::Options = vortex_tensor::scalar_fns::sorf_transform::SorfOptions + +pub fn vortex_tensor::scalar_fns::sorf_transform::SorfTransform::arity(&self, &Self::Options) -> vortex_array::scalar_fn::vtable::Arity + +pub fn vortex_tensor::scalar_fns::sorf_transform::SorfTransform::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::vtable::ChildName + +pub fn vortex_tensor::scalar_fns::sorf_transform::SorfTransform::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::vtable::ExecutionArgs, &mut vortex_array::executor::ExecutionCtx) -> vortex_error::VortexResult + +pub fn vortex_tensor::scalar_fns::sorf_transform::SorfTransform::fmt_sql(&self, &Self::Options, &vortex_array::expr::expression::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +pub fn vortex_tensor::scalar_fns::sorf_transform::SorfTransform::id(&self) -> vortex_array::scalar_fn::ScalarFnId + +pub fn vortex_tensor::scalar_fns::sorf_transform::SorfTransform::is_fallible(&self, &Self::Options) -> bool + +pub fn vortex_tensor::scalar_fns::sorf_transform::SorfTransform::is_null_sensitive(&self, &Self::Options) -> bool + +pub fn vortex_tensor::scalar_fns::sorf_transform::SorfTransform::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult + +pub fn vortex_tensor::scalar_fns::sorf_transform::SorfTransform::validity(&self, &Self::Options, &vortex_array::expr::expression::Expression) -> vortex_error::VortexResult> pub mod vortex_tensor::vector @@ -552,7 +524,7 @@ impl vortex_array::dtype::extension::matcher::Matcher for vortex_tensor::vector: pub type vortex_tensor::vector::AnyVector::Match<'a> = vortex_tensor::vector::VectorMatcherMetadata -pub fn vortex_tensor::vector::AnyVector::try_match<'a>(ext_dtype: &'a vortex_array::dtype::extension::erased::ExtDTypeRef) -> core::option::Option +pub fn vortex_tensor::vector::AnyVector::try_match<'a>(&'a vortex_array::dtype::extension::erased::ExtDTypeRef) -> core::option::Option pub struct vortex_tensor::vector::Vector @@ -564,7 +536,7 @@ impl core::cmp::Eq for vortex_tensor::vector::Vector impl core::cmp::PartialEq for vortex_tensor::vector::Vector -pub fn vortex_tensor::vector::Vector::eq(&self, other: &vortex_tensor::vector::Vector) -> bool +pub fn vortex_tensor::vector::Vector::eq(&self, &vortex_tensor::vector::Vector) -> bool impl core::default::Default for vortex_tensor::vector::Vector @@ -572,11 +544,11 @@ pub fn vortex_tensor::vector::Vector::default() -> vortex_tensor::vector::Vector impl core::fmt::Debug for vortex_tensor::vector::Vector -pub fn vortex_tensor::vector::Vector::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_tensor::vector::Vector::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_tensor::vector::Vector -pub fn vortex_tensor::vector::Vector::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_tensor::vector::Vector::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_tensor::vector::Vector @@ -586,15 +558,17 @@ pub type vortex_tensor::vector::Vector::Metadata = vortex_array::extension::Empt pub type vortex_tensor::vector::Vector::NativeValue<'a> = &'a vortex_array::scalar::scalar_value::ScalarValue -pub fn vortex_tensor::vector::Vector::deserialize_metadata(&self, _metadata: &[u8]) -> vortex_error::VortexResult +pub fn vortex_tensor::vector::Vector::deserialize_metadata(&self, &[u8]) -> vortex_error::VortexResult pub fn vortex_tensor::vector::Vector::id(&self) -> vortex_array::dtype::extension::ExtId -pub fn vortex_tensor::vector::Vector::serialize_metadata(&self, _metadata: &Self::Metadata) -> vortex_error::VortexResult> +pub fn vortex_tensor::vector::Vector::least_supertype(&vortex_array::dtype::extension::typed::ExtDType, &vortex_array::dtype::DType) -> core::option::Option + +pub fn vortex_tensor::vector::Vector::serialize_metadata(&self, &Self::Metadata) -> vortex_error::VortexResult> -pub fn vortex_tensor::vector::Vector::unpack_native<'a>(_ext_dtype: &'a vortex_array::dtype::extension::typed::ExtDType, storage_value: &'a vortex_array::scalar::scalar_value::ScalarValue) -> vortex_error::VortexResult +pub fn vortex_tensor::vector::Vector::unpack_native<'a>(&'a vortex_array::dtype::extension::typed::ExtDType, &'a vortex_array::scalar::scalar_value::ScalarValue) -> vortex_error::VortexResult -pub fn vortex_tensor::vector::Vector::validate_dtype(ext_dtype: &vortex_array::dtype::extension::typed::ExtDType) -> vortex_error::VortexResult<()> +pub fn vortex_tensor::vector::Vector::validate_dtype(&vortex_array::dtype::extension::typed::ExtDType) -> vortex_error::VortexResult<()> pub struct vortex_tensor::vector::VectorMatcherMetadata @@ -604,7 +578,7 @@ pub fn vortex_tensor::vector::VectorMatcherMetadata::dimensions(&self) -> u32 pub fn vortex_tensor::vector::VectorMatcherMetadata::element_ptype(&self) -> vortex_array::dtype::ptype::PType -pub fn vortex_tensor::vector::VectorMatcherMetadata::try_new(element_ptype: vortex_array::dtype::ptype::PType, dimensions: u32) -> vortex_error::VortexResult +pub fn vortex_tensor::vector::VectorMatcherMetadata::try_new(vortex_array::dtype::ptype::PType, u32) -> vortex_error::VortexResult impl core::clone::Clone for vortex_tensor::vector::VectorMatcherMetadata @@ -614,18 +588,24 @@ impl core::cmp::Eq for vortex_tensor::vector::VectorMatcherMetadata impl core::cmp::PartialEq for vortex_tensor::vector::VectorMatcherMetadata -pub fn vortex_tensor::vector::VectorMatcherMetadata::eq(&self, other: &vortex_tensor::vector::VectorMatcherMetadata) -> bool +pub fn vortex_tensor::vector::VectorMatcherMetadata::eq(&self, &vortex_tensor::vector::VectorMatcherMetadata) -> bool impl core::fmt::Debug for vortex_tensor::vector::VectorMatcherMetadata -pub fn vortex_tensor::vector::VectorMatcherMetadata::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_tensor::vector::VectorMatcherMetadata::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_tensor::vector::VectorMatcherMetadata -pub fn vortex_tensor::vector::VectorMatcherMetadata::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_tensor::vector::VectorMatcherMetadata::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_tensor::vector::VectorMatcherMetadata impl core::marker::StructuralPartialEq for vortex_tensor::vector::VectorMatcherMetadata -pub fn vortex_tensor::initialize(session: &vortex_session::VortexSession) +pub mod vortex_tensor::vector_search + +pub fn vortex_tensor::vector_search::build_similarity_search_tree>(vortex_array::array::erased::ArrayRef, &[T], T) -> vortex_error::VortexResult + +pub const vortex_tensor::SCALAR_FN_ARRAY_TENSOR_PLUGIN_ENV: &str + +pub fn vortex_tensor::initialize(&vortex_session::VortexSession) diff --git a/vortex-tensor/src/encodings/l2_denorm.rs b/vortex-tensor/src/encodings/l2_denorm.rs new file mode 100644 index 00000000000..6cb4fcb0626 --- /dev/null +++ b/vortex-tensor/src/encodings/l2_denorm.rs @@ -0,0 +1,54 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +use vortex_array::ArrayRef; +use vortex_array::Canonical; +use vortex_array::ExecutionCtx; +use vortex_array::IntoArray; +use vortex_array::arrays::extension::ExtensionArrayExt; +use vortex_compressor::CascadingCompressor; +use vortex_compressor::ctx::CompressorContext; +use vortex_compressor::estimate::CompressionEstimate; +use vortex_compressor::estimate::EstimateVerdict; +use vortex_compressor::scheme::Scheme; +use vortex_compressor::stats::ArrayAndStats; +use vortex_error::VortexResult; + +use crate::matcher::AnyTensor; +use crate::scalar_fns::l2_denorm::normalize_as_l2_denorm; + +#[derive(Debug)] +pub struct L2DenormScheme; + +impl Scheme for L2DenormScheme { + fn scheme_name(&self) -> &'static str { + "vortex.tensor.l2_denorm" + } + + fn matches(&self, canonical: &Canonical) -> bool { + matches!( + canonical, + Canonical::Extension(ext) if ext.ext_dtype().is::() + ) + } + + fn expected_compression_ratio( + &self, + _data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, + ) -> CompressionEstimate { + CompressionEstimate::Verdict(EstimateVerdict::AlwaysUse) + } + + fn compress( + &self, + _compressor: &CascadingCompressor, + data: &ArrayAndStats, + _compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, + ) -> VortexResult { + let l2_denorm = normalize_as_l2_denorm(data.array().clone(), exec_ctx)?; + Ok(l2_denorm.into_array()) + } +} diff --git a/vortex-tensor/src/encodings/mod.rs b/vortex-tensor/src/encodings/mod.rs index 7c75269b632..22e57763171 100644 --- a/vortex-tensor/src/encodings/mod.rs +++ b/vortex-tensor/src/encodings/mod.rs @@ -4,7 +4,7 @@ //! Encodings for the different tensor types. // TODO(connor): -// pub mod norm; // Unit-normalized vectors. // pub mod spherical; // Spherical transform on unit-normalized vectors. +pub mod l2_denorm; pub mod turboquant; diff --git a/vortex-tensor/src/encodings/turboquant/array/data.rs b/vortex-tensor/src/encodings/turboquant/array/data.rs deleted file mode 100644 index dd1867eb598..00000000000 --- a/vortex-tensor/src/encodings/turboquant/array/data.rs +++ /dev/null @@ -1,261 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -use std::fmt::Display; -use std::fmt::Formatter; -use std::sync::Arc; - -use vortex_array::ArrayRef; -use vortex_array::TypedArrayRef; -use vortex_array::dtype::DType; -use vortex_array::dtype::Nullability; -use vortex_array::dtype::PType; -use vortex_error::VortexExpect; -use vortex_error::VortexResult; -use vortex_error::vortex_ensure; -use vortex_error::vortex_ensure_eq; - -use crate::encodings::turboquant::array::slots::Slot; -use crate::encodings::turboquant::vtable::TurboQuant; - -/// TurboQuant array data. -/// -/// TurboQuant is a lossy vector quantization encoding for [`Vector`](crate::vector::Vector) -/// extension arrays. It stores quantized coordinate codes for unit-norm vectors, along with shared -/// codebook centroids and the parameters of the current structured rotation. -/// -/// Norms should be stored externally in the [`L2Denorm`](crate::scalar_fns::l2_denorm::L2Denorm) -/// `ScalarFnArray` wrapper. -/// -/// See the [module docs](crate::encodings::turboquant) for algorithmic details. -/// -/// Note that degenerate TurboQuant arrays have zero rows and `bit_width == 0`, with all slots -/// empty. -#[derive(Clone, Debug)] -pub struct TurboQuantData { - /// The vector dimension `d`, cached from the `FixedSizeList` storage dtype's list size. - /// - /// Stored as a convenience field to avoid repeatedly extracting it from `dtype`. - pub(crate) dimension: u32, - - /// The number of bits per coordinate (0-8), derived from `log2(centroids.len())`. - /// - /// This is 0 for degenerate empty arrays. - pub(crate) bit_width: u8, - - /// The number of sign-diagonal + WHT rounds in the structured rotation. - /// - /// This is 0 for degenerate empty arrays. - pub(crate) num_rounds: u8, -} - -impl Display for TurboQuantData { - fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result { - write!( - f, - "dimension: {}, bit_width: {}, num_rounds: {}", - self.dimension, self.bit_width, self.num_rounds - ) - } -} - -impl TurboQuantData { - /// Build a `TurboQuantData` with validation. - /// - /// # Errors - /// - /// Returns an error if: - /// - `dimension` is less than [`MIN_DIMENSION`](TurboQuant::MIN_DIMENSION). - /// - `bit_width` is greater than [`MAX_BIT_WIDTH`](TurboQuant::MAX_BIT_WIDTH). - pub fn try_new(dimension: u32, bit_width: u8, num_rounds: u8) -> VortexResult { - vortex_ensure!( - dimension >= TurboQuant::MIN_DIMENSION, - "TurboQuant requires dimension >= {}, got {dimension}", - TurboQuant::MIN_DIMENSION - ); - vortex_ensure!( - bit_width <= TurboQuant::MAX_BIT_WIDTH, - "bit_width is expected to be between 0 and {}, got {bit_width}", - TurboQuant::MAX_BIT_WIDTH - ); - - Ok(Self { - dimension, - bit_width, - num_rounds, - }) - } - - /// Build a `TurboQuantData` without validation. - /// - /// # Safety - /// - /// The caller must ensure: - /// - /// - `dimension` is >= [`MIN_DIMENSION`](TurboQuant::MIN_DIMENSION). - /// - `bit_width` is in the range `[0, MAX_BIT_WIDTH]`. - /// - `num_rounds` is >= 1 (or 0 for degenerate empty arrays). - /// - /// Violating these invariants may produce incorrect results during decompression. - pub unsafe fn new_unchecked(dimension: u32, bit_width: u8, num_rounds: u8) -> Self { - Self { - dimension, - bit_width, - num_rounds, - } - } - - /// Validates the components that would be used to create a `TurboQuantData`. - /// - /// This function checks all the invariants required by [`new_unchecked`](Self::new_unchecked). - pub fn validate( - dtype: &DType, - codes: &ArrayRef, - centroids: &ArrayRef, - rotation_signs: &ArrayRef, - ) -> VortexResult<()> { - let vector_metadata = TurboQuant::validate_dtype(dtype)?; - let dimension = vector_metadata.dimensions(); - let padded_dim = dimension.next_power_of_two(); - - // TurboQuant arrays are always non-nullable. Nullability should be handled by the external - // L2Denorm ScalarFnArray wrapper. - vortex_ensure!( - !dtype.is_nullable(), - "TurboQuant dtype must be non-nullable, got {dtype}", - ); - - // Codes must be a non-nullable FixedSizeList with list_size == padded_dim. - let expected_codes_dtype = DType::FixedSizeList( - Arc::new(DType::Primitive(PType::U8, Nullability::NonNullable)), - padded_dim, - Nullability::NonNullable, - ); - vortex_ensure_eq!( - *codes.dtype(), - expected_codes_dtype, - "codes dtype does not match expected {expected_codes_dtype}", - ); - - // Centroids are always f32 regardless of element type. - let centroids_dtype = DType::Primitive(PType::F32, Nullability::NonNullable); - vortex_ensure_eq!( - *centroids.dtype(), - centroids_dtype, - "centroids dtype must be non-nullable f32", - ); - - // Rotation signs must be a FixedSizeList with list_size == padded_dim. The FSL length - // is the number of rotation rounds. - let expected_signs_dtype = DType::FixedSizeList( - Arc::new(DType::Primitive(PType::U8, Nullability::NonNullable)), - padded_dim, - Nullability::NonNullable, - ); - vortex_ensure_eq!( - *rotation_signs.dtype(), - expected_signs_dtype, - "rotation_signs dtype does not match expected {expected_signs_dtype}", - ); - // Degenerate (empty) case: all children must be empty, and bit_width is 0. - let num_rows = codes.len(); - if num_rows == 0 { - vortex_ensure!( - centroids.is_empty(), - "degenerate TurboQuant must have empty centroids, got length {}", - centroids.len() - ); - vortex_ensure!( - rotation_signs.is_empty(), - "degenerate TurboQuant must have empty rotation_signs, got length {}", - rotation_signs.len() - ); - return Ok(()); - } - - vortex_ensure!( - !rotation_signs.is_empty(), - "rotation_signs must have at least 1 round" - ); - - // Non-degenerate: derive and validate bit_width from centroids. - let num_centroids = centroids.len(); - vortex_ensure!( - num_centroids.is_power_of_two() - && (2..=TurboQuant::MAX_CENTROIDS).contains(&num_centroids), - "centroids length must be a power of 2 in [2, {}], got {num_centroids}", - TurboQuant::MAX_CENTROIDS - ); - - #[expect( - clippy::cast_possible_truncation, - reason = "Guaranteed to be [1,8] by the preceding power-of-2 and range checks." - )] - let bit_width = num_centroids.trailing_zeros() as u8; - vortex_ensure!( - (1..=TurboQuant::MAX_BIT_WIDTH).contains(&bit_width), - "derived bit_width must be 1-{}, got {bit_width}", - TurboQuant::MAX_BIT_WIDTH - ); - - Ok(()) - } - - pub(crate) fn make_slots( - codes: ArrayRef, - centroids: ArrayRef, - rotation_signs: ArrayRef, - ) -> Vec> { - let mut slots = vec![None; Slot::COUNT]; - slots[Slot::Codes as usize] = Some(codes); - slots[Slot::Centroids as usize] = Some(centroids); - slots[Slot::RotationSigns as usize] = Some(rotation_signs); - slots - } - - /// The vector dimension `d`, as stored in the [`Vector`](crate::vector::Vector) extension - /// dtype's `FixedSizeList` storage. - pub fn dimension(&self) -> u32 { - self.dimension - } - - /// MSE bits per coordinate (1-MAX_BIT_WIDTH for non-empty arrays, 0 for degenerate empty arrays). - pub fn bit_width(&self) -> u8 { - self.bit_width - } - - /// The number of sign-diagonal + WHT rounds in the structured rotation. - pub fn num_rounds(&self) -> u8 { - self.num_rounds - } - - /// Padded dimension (next power of 2 >= [`dimension`](Self::dimension)). - /// - /// The current Walsh-Hadamard-based structured rotation requires power-of-2 input, so - /// non-power-of-2 dimensions are zero-padded to this value. - pub fn padded_dim(&self) -> u32 { - self.dimension.next_power_of_two() - } -} - -pub trait TurboQuantArrayExt: TypedArrayRef { - fn codes(&self) -> &ArrayRef { - self.as_ref().slots()[Slot::Codes as usize] - .as_ref() - .vortex_expect("TurboQuantArray codes slot") - } - - fn centroids(&self) -> &ArrayRef { - self.as_ref().slots()[Slot::Centroids as usize] - .as_ref() - .vortex_expect("TurboQuantArray centroids slot") - } - - fn rotation_signs(&self) -> &ArrayRef { - self.as_ref().slots()[Slot::RotationSigns as usize] - .as_ref() - .vortex_expect("TurboQuantArray rotation_signs slot") - } -} - -impl> TurboQuantArrayExt for T {} diff --git a/vortex-tensor/src/encodings/turboquant/array/mod.rs b/vortex-tensor/src/encodings/turboquant/array/mod.rs deleted file mode 100644 index e82313f1dc6..00000000000 --- a/vortex-tensor/src/encodings/turboquant/array/mod.rs +++ /dev/null @@ -1,11 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -//! TurboQuant array definition: stores quantized coordinate codes, norms, centroids (codebook), -//! and rotation signs. - -pub(crate) mod data; -pub(crate) mod slots; - -pub(crate) mod centroids; -pub(crate) mod rotation; diff --git a/vortex-tensor/src/encodings/turboquant/array/slots.rs b/vortex-tensor/src/encodings/turboquant/array/slots.rs deleted file mode 100644 index c4fe7e0c5bf..00000000000 --- a/vortex-tensor/src/encodings/turboquant/array/slots.rs +++ /dev/null @@ -1,38 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -/// Slot positions for TurboQuantArray children. -/// -/// Norms are not stored in the TurboQuantArray. They live in the external [`L2Denorm`] -/// ScalarFnArray wrapper returned by [`turboquant_encode`]. -/// -/// [`L2Denorm`]: crate::scalar_fns::l2_denorm::L2Denorm -/// [`turboquant_encode`]: crate::encodings::turboquant::turboquant_encode -#[repr(usize)] -#[derive(Clone, Copy, Debug)] -pub(crate) enum Slot { - Codes = 0, - Centroids = 1, - RotationSigns = 2, -} - -impl Slot { - pub(crate) const COUNT: usize = 3; - - pub(crate) fn name(self) -> &'static str { - match self { - Self::Codes => "codes", - Self::Centroids => "centroids", - Self::RotationSigns => "rotation_signs", - } - } - - pub(crate) fn from_index(idx: usize) -> Self { - match idx { - 0 => Self::Codes, - 1 => Self::Centroids, - 2 => Self::RotationSigns, - _ => vortex_error::vortex_panic!("invalid slot index {idx}"), - } - } -} diff --git a/vortex-tensor/src/encodings/turboquant/array/centroids.rs b/vortex-tensor/src/encodings/turboquant/centroids.rs similarity index 87% rename from vortex-tensor/src/encodings/turboquant/array/centroids.rs rename to vortex-tensor/src/encodings/turboquant/centroids.rs index e3027d0a58a..1af86c79d85 100644 --- a/vortex-tensor/src/encodings/turboquant/array/centroids.rs +++ b/vortex-tensor/src/encodings/turboquant/centroids.rs @@ -11,11 +11,13 @@ use std::sync::LazyLock; +use vortex_buffer::Buffer; use vortex_error::VortexResult; use vortex_error::vortex_ensure; use vortex_utils::aliases::dash_map::DashMap; -use crate::encodings::turboquant::TurboQuant; +use crate::encodings::turboquant::MAX_BIT_WIDTH; +use crate::encodings::turboquant::MIN_DIMENSION; /// The maximum iterations for Max-Lloyd algorithm when computing centroids. const MAX_ITERATIONS: usize = 200; @@ -26,25 +28,24 @@ const CONVERGENCE_EPSILON: f64 = 1e-12; /// Number of numerical integration points for computing conditional expectations. const INTEGRATION_POINTS: usize = 1000; -// TODO(connor): Maybe we should just store an `ArrayRef` here? /// Global centroid cache keyed by (dimension, bit_width). -static CENTROID_CACHE: LazyLock>> = LazyLock::new(DashMap::default); +static CENTROID_CACHE: LazyLock>> = LazyLock::new(DashMap::default); /// Get or compute cached centroids for the given dimension and bit width. /// /// Returns `2^bit_width` centroids sorted in ascending order, representing optimal scalar /// quantization levels for the coordinate distribution after random rotation in /// `dimension`-dimensional space. -pub fn get_centroids(dimension: u32, bit_width: u8) -> VortexResult> { +pub fn compute_or_get_centroids(dimension: u32, bit_width: u8) -> VortexResult> { vortex_ensure!( - (1..=TurboQuant::MAX_BIT_WIDTH).contains(&bit_width), + (1..=MAX_BIT_WIDTH).contains(&bit_width), "TurboQuant bit_width must be 1-{}, got {bit_width}", - TurboQuant::MAX_BIT_WIDTH + MAX_BIT_WIDTH ); vortex_ensure!( - dimension >= TurboQuant::MIN_DIMENSION, + dimension >= MIN_DIMENSION, "TurboQuant dimension must be >= {}, got {dimension}", - TurboQuant::MIN_DIMENSION + MIN_DIMENSION ); if let Some(centroids) = CENTROID_CACHE.get(&(dimension, bit_width)) { @@ -91,8 +92,8 @@ impl HalfIntExponent { /// The probability distribution function is: /// `f(x) = C_d * (1 - x^2)^((d-3)/2)` on `[-1, 1]` /// where `C_d` is the normalizing constant. -fn max_lloyd_centroids(dimension: u32, bit_width: u8) -> Vec { - debug_assert!((1..=TurboQuant::MAX_BIT_WIDTH).contains(&bit_width)); +fn max_lloyd_centroids(dimension: u32, bit_width: u8) -> Buffer { + debug_assert!((1..=MAX_BIT_WIDTH).contains(&bit_width)); let num_centroids = 1usize << bit_width; // For the marginal distribution on [-1, 1], we use the exponent (d-3)/2. @@ -175,7 +176,6 @@ fn mean_between_centroids(lo: f64, hi: f64, exponent: HalfIntExponent) -> f64 { /// Uses `powi` + `sqrt` instead of `powf` for the half-integer exponents that arise from `(d-3)/2`. /// This is significantly faster than the general `powf` which goes through /// `exp(exponent * ln(base))`. -#[inline] fn pdf_unnormalized(x_val: f64, exponent: HalfIntExponent) -> f64 { let base = (1.0 - x_val * x_val).max(0.0); @@ -199,7 +199,7 @@ pub fn compute_centroid_boundaries(centroids: &[f32]) -> Vec { /// Find the index of the nearest centroid using precomputed decision boundaries. /// -/// `boundaries` must be the output of [`compute_boundaries`] for the corresponding +/// `boundaries` must be the output of [`compute_centroid_boundaries`] for the corresponding /// centroids. Uses binary search on the midpoints, avoiding distance comparisons /// in the inner loop. #[inline] @@ -210,7 +210,7 @@ pub fn find_nearest_centroid(value: f32, boundaries: &[f32]) -> u8 { ); debug_assert!( boundaries.len() <= 256, // 1 << 8 - "boundaries must be sorted" + "too many boundaries" ); #[expect( @@ -239,7 +239,7 @@ mod tests { #[case] bits: u8, #[case] expected: usize, ) -> VortexResult<()> { - let centroids = get_centroids(dim, bits)?; + let centroids = compute_or_get_centroids(dim, bits)?; assert_eq!(centroids.len(), expected); Ok(()) } @@ -251,7 +251,7 @@ mod tests { #[case(128, 4)] #[case(768, 2)] fn centroids_are_sorted(#[case] dim: u32, #[case] bits: u8) -> VortexResult<()> { - let centroids = get_centroids(dim, bits)?; + let centroids = compute_or_get_centroids(dim, bits)?; for window in centroids.windows(2) { assert!( window[0] < window[1], @@ -268,7 +268,7 @@ mod tests { #[case(256, 2)] #[case(768, 2)] fn centroids_are_symmetric(#[case] dim: u32, #[case] bits: u8) -> VortexResult<()> { - let centroids = get_centroids(dim, bits)?; + let centroids = compute_or_get_centroids(dim, bits)?; let count = centroids.len(); for idx in 0..count / 2 { let diff = (centroids[idx] + centroids[count - 1 - idx]).abs(); @@ -287,8 +287,8 @@ mod tests { #[case(128, 1)] #[case(128, 4)] fn centroids_within_bounds(#[case] dim: u32, #[case] bits: u8) -> VortexResult<()> { - let centroids = get_centroids(dim, bits)?; - for &val in ¢roids { + let centroids = compute_or_get_centroids(dim, bits)?; + for &val in centroids.iter() { assert!( (-1.0..=1.0).contains(&val), "centroid out of [-1, 1]: {val}", @@ -299,15 +299,15 @@ mod tests { #[test] fn centroids_cached() -> VortexResult<()> { - let c1 = get_centroids(128, 2)?; - let c2 = get_centroids(128, 2)?; + let c1 = compute_or_get_centroids(128, 2)?; + let c2 = compute_or_get_centroids(128, 2)?; assert_eq!(c1, c2); Ok(()) } #[test] fn find_nearest_basic() -> VortexResult<()> { - let centroids = get_centroids(128, 2)?; + let centroids = compute_or_get_centroids(128, 2)?; let boundaries = compute_centroid_boundaries(¢roids); assert_eq!(find_nearest_centroid(-1.0, &boundaries), 0); @@ -324,9 +324,9 @@ mod tests { #[test] fn rejects_invalid_params() { - assert!(get_centroids(128, 0).is_err()); - assert!(get_centroids(128, 9).is_err()); - assert!(get_centroids(1, 2).is_err()); - assert!(get_centroids(127, 2).is_err()); + assert!(compute_or_get_centroids(128, 0).is_err()); + assert!(compute_or_get_centroids(128, 9).is_err()); + assert!(compute_or_get_centroids(1, 2).is_err()); + assert!(compute_or_get_centroids(127, 2).is_err()); } } diff --git a/vortex-tensor/src/encodings/turboquant/compress.rs b/vortex-tensor/src/encodings/turboquant/compress.rs new file mode 100644 index 00000000000..ca32faa6ec9 --- /dev/null +++ b/vortex-tensor/src/encodings/turboquant/compress.rs @@ -0,0 +1,270 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! TurboQuant encoding (quantization) logic. +//! +//! The input to [`turboquant_encode`] must be a non-nullable [`Vector`](crate::vector::Vector) +//! extension array whose rows are already L2-normalized (unit norm). Normalization is handled +//! externally by [`normalize_as_l2_denorm`](crate::scalar_fns::l2_denorm::normalize_as_l2_denorm), +//! which the [`TurboQuantScheme`] calls before invoking this function. +//! +//! [`TurboQuantScheme`]: crate::encodings::turboquant::TurboQuantScheme + +use vortex_array::ArrayRef; +use vortex_array::ArrayView; +use vortex_array::ExecutionCtx; +use vortex_array::IntoArray; +use vortex_array::arrays::Extension; +use vortex_array::arrays::FixedSizeListArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::dict::DictArray; +use vortex_array::arrays::extension::ExtensionArrayExt; +use vortex_array::arrays::fixed_size_list::FixedSizeListArrayExt; +use vortex_array::arrays::scalar_fn::ScalarFnArrayExt; +use vortex_array::dtype::Nullability; +use vortex_array::validity::Validity; +use vortex_buffer::Buffer; +use vortex_buffer::BufferMut; +use vortex_error::VortexExpect; +use vortex_error::VortexResult; +use vortex_error::vortex_ensure; + +use crate::encodings::turboquant::MAX_BIT_WIDTH; +use crate::encodings::turboquant::MIN_DIMENSION; +use crate::encodings::turboquant::centroids::compute_centroid_boundaries; +use crate::encodings::turboquant::centroids::compute_or_get_centroids; +use crate::encodings::turboquant::centroids::find_nearest_centroid; +use crate::scalar_fns::l2_denorm::L2Denorm; +use crate::scalar_fns::l2_denorm::normalize_as_l2_denorm; +use crate::scalar_fns::sorf_transform::SorfMatrix; +use crate::scalar_fns::sorf_transform::SorfOptions; +use crate::scalar_fns::sorf_transform::SorfTransform; +use crate::types::vector::AnyVector; +use crate::types::vector::Vector; +use crate::utils::cast_to_f32; + +/// Configuration for TurboQuant encoding. +#[derive(Clone, Debug)] +pub struct TurboQuantConfig { + /// Bits per coordinate (1-8). + pub bit_width: u8, + /// Seed for the rotation matrix. + pub seed: u64, + /// Number of sign-diagonal + WHT rounds in the structured rotation (default 3). + pub num_rounds: u8, +} + +impl Default for TurboQuantConfig { + fn default() -> Self { + Self { + bit_width: MAX_BIT_WIDTH, + seed: 42, + num_rounds: 3, + } + } +} + +/// Apply the full TurboQuant compression pipeline to a [`Vector`](crate::vector::Vector) +/// extension array: normalize the rows via [`normalize_as_l2_denorm`], quantize the normalized +/// child via [`turboquant_encode_unchecked`], and reattach the stored norms as the outer +/// [`L2Denorm`] wrapper. +/// +/// The returned array has the canonical TurboQuant shape: +/// +/// ```text +/// ScalarFnArray(L2Denorm, [ +/// ScalarFnArray(SorfTransform, [FSL(Dict(codes, centroids))]), +/// norms, +/// ]) +/// ``` +/// +/// # Errors +/// +/// Returns an error if `input` is not a tensor-like extension array, if normalization fails, or +/// if [`turboquant_encode_unchecked`] rejects the input shape. +pub fn turboquant_encode( + input: ArrayRef, + config: &TurboQuantConfig, + ctx: &mut ExecutionCtx, +) -> VortexResult { + // We must normalize the array before we can encode it with TurboQuant. + let l2_denorm = normalize_as_l2_denorm(input, ctx)?; + let normalized = l2_denorm.child_at(0).clone(); + let norms = l2_denorm.child_at(1).clone(); + let num_rows = l2_denorm.len(); + + let normalized_ext = normalized + .as_opt::() + .vortex_expect("normalize_as_l2_denorm always produces an Extension array child"); + + // SAFETY: `normalize_as_l2_denorm` guarantees every row is unit-norm (or zero for null rows). + let tq = unsafe { turboquant_encode_unchecked(normalized_ext, config, ctx) }?; + + // SAFETY: TurboQuant is a lossy approximation of the normalized child, so we intentionally + // bypass the strict normalized-row validation when reattaching the stored norms. + Ok(unsafe { L2Denorm::new_array_unchecked(tq, norms, num_rows) }?.into_array()) +} + +/// Encode a non-nullable, L2-normalized [`Vector`](crate::vector::Vector) extension array into a +/// `ScalarFnArray(SorfTransform, [FSL(Dict(codes, centroids))])`, without validating the unit-norm +/// precondition. +/// +/// # Safety +/// +/// The caller must ensure: +/// +/// - The input dtype is non-nullable. +/// - Every row is L2-normalized (unit norm) or is a zero vector. +/// +/// Passing non-unit-norm vectors will not cause memory unsafety, but will produce silently +/// incorrect quantization results. +pub unsafe fn turboquant_encode_unchecked( + ext: ArrayView, + config: &TurboQuantConfig, + ctx: &mut ExecutionCtx, +) -> VortexResult { + let ext_dtype = ext.dtype().clone(); + let storage = ext.storage_array(); + let fsl = storage.clone().execute::(ctx)?; + + vortex_ensure!( + config.bit_width >= 1 && config.bit_width <= MAX_BIT_WIDTH, + "bit_width must be 1-{MAX_BIT_WIDTH}, got {}", + config.bit_width + ); + let dimension = fsl.list_size(); + vortex_ensure!( + dimension >= MIN_DIMENSION, + "TurboQuant requires dimension >= {MIN_DIMENSION}, got {dimension}", + ); + + let vector_metadata = ext_dtype.as_extension().metadata::(); + let element_ptype = vector_metadata.element_ptype(); + + let seed = config.seed; + let num_rows = fsl.len(); + + if fsl.is_empty() { + let padded_dim = dimension.next_power_of_two(); + let empty_codes = PrimitiveArray::empty::(Nullability::NonNullable); + let empty_centroids = PrimitiveArray::empty::(Nullability::NonNullable); + let empty_dict = + DictArray::try_new(empty_codes.into_array(), empty_centroids.into_array())?; + let empty_fsl = FixedSizeListArray::try_new( + empty_dict.into_array(), + padded_dim, + Validity::NonNullable, + 0, + )?; + let empty_padded_vector = Vector::try_new_vector_array(empty_fsl.into_array())?; + + let sorf_options = SorfOptions { + seed, + num_rounds: config.num_rounds, + dimensions: dimension, + element_ptype, + }; + return Ok( + SorfTransform::try_new_array(&sorf_options, empty_padded_vector, 0)?.into_array(), + ); + } + + let core = turboquant_quantize_core(&fsl, seed, config.bit_width, config.num_rounds, ctx)?; + let quantized_fsl = + build_quantized_fsl(num_rows, core.all_indices, core.centroids, core.padded_dim)?; + let padded_vector = Vector::try_new_vector_array(quantized_fsl)?; + + let sorf_options = SorfOptions { + seed, + num_rounds: config.num_rounds, + dimensions: dimension, + element_ptype, + }; + Ok(SorfTransform::try_new_array(&sorf_options, padded_vector, num_rows)?.into_array()) +} + +/// Shared intermediate results from the quantization loop. +struct QuantizationResult { + centroids: Buffer, + all_indices: Buffer, + padded_dim: usize, +} + +/// Core quantization: rotate and quantize already-normalized rows. +/// +/// The input `fsl` must contain non-nullable, unit-norm vectors (already L2-normalized). Null +/// vectors are not supported and must be zeroed out before reaching this function. The rotation +/// and centroid lookup happen in f32. +fn turboquant_quantize_core( + fsl: &FixedSizeListArray, + seed: u64, + bit_width: u8, + num_rounds: u8, + ctx: &mut ExecutionCtx, +) -> VortexResult { + let dimension = fsl.list_size() as usize; + let num_rows = fsl.len(); + + let rotation = SorfMatrix::try_new(seed, dimension, num_rounds as usize)?; + let padded_dim = rotation.padded_dim(); + let padded_dim_u32 = + u32::try_from(padded_dim).vortex_expect("padded_dim stays representable as u32"); + + let elements_prim: PrimitiveArray = fsl.elements().clone().execute(ctx)?; + let f32_elements = cast_to_f32(elements_prim)?; + + let centroids = compute_or_get_centroids(padded_dim_u32, bit_width)?; + let boundaries = compute_centroid_boundaries(¢roids); + + let mut all_indices = BufferMut::::with_capacity(num_rows * padded_dim); + let mut padded = vec![0.0f32; padded_dim]; + let mut rotated = vec![0.0f32; padded_dim]; + + let f32_slice = f32_elements.as_slice(); + for row in 0..num_rows { + let x = &f32_slice[row * dimension..(row + 1) * dimension]; + + // Zero-pad to the next power of 2. + padded[..dimension].copy_from_slice(x); + padded[dimension..].fill(0.0); + + rotation.rotate(&padded, &mut rotated); + + for j in 0..padded_dim { + all_indices.push(find_nearest_centroid(rotated[j], &boundaries)); + } + } + + Ok(QuantizationResult { + centroids, + all_indices: all_indices.freeze(), + padded_dim, + }) +} + +/// Build a quantized representation: `FSL(DictArray(codes, centroids), padded_dim)`. +/// +/// This is a Dict-encoded FixedSizeList where each row of `padded_dim` u8 codes indexes into the +/// centroid codebook. The Dict can be independently sliced, taken, or executed (dequantized) +/// without knowledge of the rotation. +fn build_quantized_fsl( + num_rows: usize, + all_indices: Buffer, + centroids: Buffer, + padded_dim: usize, +) -> VortexResult { + let codes = PrimitiveArray::new::(all_indices, Validity::NonNullable); + let centroids_array = PrimitiveArray::new::(centroids, Validity::NonNullable); + + let dict = DictArray::try_new(codes.into_array(), centroids_array.into_array())?; + + let padded_dim_u32 = + u32::try_from(padded_dim).vortex_expect("padded_dim stays representable as u32"); + Ok(FixedSizeListArray::try_new( + dict.into_array(), + padded_dim_u32, + Validity::NonNullable, + num_rows, + )? + .into_array()) +} diff --git a/vortex-tensor/src/encodings/turboquant/compute/mod.rs b/vortex-tensor/src/encodings/turboquant/compute/mod.rs deleted file mode 100644 index eab759ef0b9..00000000000 --- a/vortex-tensor/src/encodings/turboquant/compute/mod.rs +++ /dev/null @@ -1,22 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -//! Compute pushdown implementations for TurboQuant. - -mod ops; -mod slice; -mod take; - -pub(crate) mod rules; - -use num_traits::Float; -use num_traits::FromPrimitive; -use vortex_error::VortexExpect; - -/// Convert an f32 value to a float type `T`. -/// -/// `FromPrimitive::from_f32` is infallible for all Vortex float types: f16 saturates via the -/// inherent `f16::from_f32()`, f32 is identity, f64 is lossless widening. -pub(crate) fn float_from_f32(v: f32) -> T { - FromPrimitive::from_f32(v).vortex_expect("f32-to-float conversion is infallible") -} diff --git a/vortex-tensor/src/encodings/turboquant/compute/ops.rs b/vortex-tensor/src/encodings/turboquant/compute/ops.rs deleted file mode 100644 index 4999816319b..00000000000 --- a/vortex-tensor/src/encodings/turboquant/compute/ops.rs +++ /dev/null @@ -1,28 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -use vortex_array::ArrayView; -use vortex_array::ExecutionCtx; -use vortex_array::arrays::ExtensionArray; -use vortex_array::arrays::slice::SliceReduce; -use vortex_array::scalar::Scalar; -use vortex_array::vtable::OperationsVTable; -use vortex_error::VortexResult; -use vortex_error::vortex_bail; - -use crate::encodings::turboquant::TurboQuant; - -impl OperationsVTable for TurboQuant { - fn scalar_at( - array: ArrayView<'_, TurboQuant>, - index: usize, - ctx: &mut ExecutionCtx, - ) -> VortexResult { - // Slice to single row, decompress that one row. - let Some(sliced) = ::slice(array, index..index + 1)? else { - vortex_bail!("slice returned None for index {index}") - }; - let decoded = sliced.execute::(ctx)?; - decoded.scalar_at(0) - } -} diff --git a/vortex-tensor/src/encodings/turboquant/compute/rules.rs b/vortex-tensor/src/encodings/turboquant/compute/rules.rs deleted file mode 100644 index 39919a8c1ec..00000000000 --- a/vortex-tensor/src/encodings/turboquant/compute/rules.rs +++ /dev/null @@ -1,15 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -use vortex_array::arrays::dict::TakeExecuteAdaptor; -use vortex_array::arrays::slice::SliceReduceAdaptor; -use vortex_array::kernel::ParentKernelSet; -use vortex_array::optimizer::rules::ParentRuleSet; - -use crate::encodings::turboquant::TurboQuant; - -pub(crate) static RULES: ParentRuleSet = - ParentRuleSet::new(&[ParentRuleSet::lift(&SliceReduceAdaptor(TurboQuant))]); - -pub(crate) static PARENT_KERNELS: ParentKernelSet = - ParentKernelSet::new(&[ParentKernelSet::lift(&TakeExecuteAdaptor(TurboQuant))]); diff --git a/vortex-tensor/src/encodings/turboquant/compute/slice.rs b/vortex-tensor/src/encodings/turboquant/compute/slice.rs deleted file mode 100644 index c19f604e36a..00000000000 --- a/vortex-tensor/src/encodings/turboquant/compute/slice.rs +++ /dev/null @@ -1,32 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -use std::ops::Range; - -use vortex_array::ArrayRef; -use vortex_array::ArrayView; -use vortex_array::IntoArray; -use vortex_array::arrays::slice::SliceReduce; -use vortex_error::VortexResult; - -use crate::encodings::turboquant::TurboQuant; -use crate::encodings::turboquant::TurboQuantArrayExt; - -impl SliceReduce for TurboQuant { - fn slice( - array: ArrayView<'_, TurboQuant>, - range: Range, - ) -> VortexResult> { - let sliced_codes = array.codes().slice(range)?; - - Ok(Some( - TurboQuant::try_new_array( - array.dtype().clone(), - sliced_codes, - array.centroids().clone(), - array.rotation_signs().clone(), - )? - .into_array(), - )) - } -} diff --git a/vortex-tensor/src/encodings/turboquant/compute/take.rs b/vortex-tensor/src/encodings/turboquant/compute/take.rs deleted file mode 100644 index 19a2e65e393..00000000000 --- a/vortex-tensor/src/encodings/turboquant/compute/take.rs +++ /dev/null @@ -1,33 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -use vortex_array::ArrayRef; -use vortex_array::ArrayView; -use vortex_array::ExecutionCtx; -use vortex_array::IntoArray; -use vortex_array::arrays::dict::TakeExecute; -use vortex_error::VortexResult; - -use crate::encodings::turboquant::TurboQuant; -use crate::encodings::turboquant::TurboQuantArrayExt; - -impl TakeExecute for TurboQuant { - fn take( - array: ArrayView<'_, TurboQuant>, - indices: &ArrayRef, - _ctx: &mut ExecutionCtx, - ) -> VortexResult> { - // FSL children handle per-row take natively. - let taken_codes = array.codes().take(indices.clone())?; - - Ok(Some( - TurboQuant::try_new_array( - array.dtype().clone(), - taken_codes, - array.centroids().clone(), - array.rotation_signs().clone(), - )? - .into_array(), - )) - } -} diff --git a/vortex-tensor/src/encodings/turboquant/metadata.rs b/vortex-tensor/src/encodings/turboquant/metadata.rs deleted file mode 100644 index b4a7a8aef08..00000000000 --- a/vortex-tensor/src/encodings/turboquant/metadata.rs +++ /dev/null @@ -1,89 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -//! Protobuf-backed metadata for TurboQuant encoding. - -use prost::Message; -use vortex_error::VortexResult; -use vortex_error::vortex_ensure; -use vortex_error::vortex_err; - -use crate::encodings::turboquant::TurboQuant; - -/// Serialized metadata for TurboQuant arrays. -#[derive(Clone, PartialEq, Message)] -pub(super) struct TurboQuantMetadata { - /// The number of bits per coordinate, which must be <= [`TurboQuant::MAX_BIT_WIDTH`]. - #[prost(uint32, required, tag = "1")] - bit_width: u32, - - /// The number of sign-diagonal + WHT rounds in the structured rotation. - #[prost(uint32, required, tag = "2")] - num_rounds: u32, -} - -impl TurboQuantMetadata { - /// Creates metadata for the given bit width and number of rotation rounds. - pub(super) fn new(bit_width: u8, num_rounds: u8) -> Self { - Self { - bit_width: u32::from(bit_width), - num_rounds: u32::from(num_rounds), - } - } - - /// Returns the validated TurboQuant bit width. - pub(super) fn bit_width(&self) -> VortexResult { - let bit_width = u8::try_from(self.bit_width).map_err(|_| { - vortex_err!( - "TurboQuant bit_width must fit into u8, got {}", - self.bit_width - ) - })?; - vortex_ensure!( - bit_width <= TurboQuant::MAX_BIT_WIDTH, - "bit_width is expected to be between 0 and {}, got {bit_width}", - TurboQuant::MAX_BIT_WIDTH - ); - - Ok(bit_width) - } - - /// Returns the validated number of rotation rounds. - /// - /// Returns 0 for degenerate (empty) arrays, which is validated at a higher level. - pub(super) fn num_rounds(&self) -> VortexResult { - u8::try_from(self.num_rounds).map_err(|_| { - vortex_err!( - "TurboQuant num_rounds must fit into u8, got {}", - self.num_rounds - ) - }) - } -} - -#[cfg(test)] -mod tests { - use prost::Message; - use rstest::rstest; - use vortex_error::VortexResult; - - use super::TurboQuantMetadata; - - #[rstest] - #[case(0, 0)] - #[case(0, 3)] - #[case(3, 1)] - #[case(8, 3)] - #[case(8, 5)] - fn protobuf_metadata_roundtrip( - #[case] bit_width: u8, - #[case] num_rounds: u8, - ) -> VortexResult<()> { - let bytes = TurboQuantMetadata::new(bit_width, num_rounds).encode_to_vec(); - let decoded = TurboQuantMetadata::decode(bytes.as_slice())?; - assert_eq!(decoded.bit_width()?, bit_width); - assert_eq!(decoded.num_rounds()?, num_rounds); - - Ok(()) - } -} diff --git a/vortex-tensor/src/encodings/turboquant/mod.rs b/vortex-tensor/src/encodings/turboquant/mod.rs index ff3259788e9..50cef7b721e 100644 --- a/vortex-tensor/src/encodings/turboquant/mod.rs +++ b/vortex-tensor/src/encodings/turboquant/mod.rs @@ -17,25 +17,34 @@ //! TurboQuant minimizes mean-squared reconstruction error (1-8 bits per coordinate) //! using MSE-optimal scalar quantization on coordinates of a rotated unit vector. //! -//! The `TurboQuantArray` stores only the quantized unit-norm vector data (codes, centroids, -//! rotation signs). Per-vector L2 norms are stored separately in an [`L2Denorm`] ScalarFnArray -//! wrapper. The [`turboquant_encode`] function returns this wrapper: +//! The encoding is decomposed into independently swappable layers: +//! +//! - **Normalization**: [`L2Denorm`] stores per-vector norms and wraps the compressed child. +//! - **Orthogonal transform**: [`SorfTransform`] records the SORF structured orthogonal +//! transform and applies the inverse at decode time. +//! - **Quantization**: `DictArray(codes, centroids)` wrapped in `FixedSizeListArray` stores +//! the per-coordinate codebook indices. +//! +//! The full encoded tree is: //! //! ```text -//! ScalarFnArray(L2Denorm, [TurboQuantArray, norms]) +//! ScalarFnArray(L2Denorm, [ +//! ScalarFnArray(SorfTransform, [FSL(Dict(codes, centroids))]), +//! norms +//! ]) //! ``` //! -//! When executed, the TQ array decompresses to unit-norm vectors, and the [`L2Denorm`] function -//! lazily re-applies the stored norms to reconstruct the original magnitudes. +//! When executed, the tree automatically decompresses: Dict dequantizes codes → SorfTransform +//! inverse-rotates → L2Denorm re-applies norms → original vectors (approximately). //! //! [`L2Denorm`]: crate::scalar_fns::l2_denorm::L2Denorm -//! [`turboquant_encode`]: crate::encodings::turboquant::turboquant_encode +//! [`SorfTransform`]: crate::scalar_fns::sorf_transform::SorfTransform //! //! The TurboQuant paper analyzes a full random orthogonal rotation. The current Vortex //! implementation instead uses a fixed 3-round Walsh-Hadamard-based structured transform with -//! random sign diagonals. This is a practical approximation chosen for encode/decode efficiency, -//! and should be understood as an implementation choice rather than the exact construction used in -//! the paper's proofs. +//! random sign diagonals generated by Vortex's frozen local SplitMix64 stream. This is a practical +//! approximation chosen for encode/decode efficiency, and should be understood as an +//! implementation choice rather than the exact construction used in the paper's proofs. //! //! The current encoding is also intentionally MSE-only. It does not yet implement the paper's QJL //! residual correction for unbiased inner-product estimation, and it still uses internal @@ -86,17 +95,12 @@ //! use vortex_array::arrays::ExtensionArray; //! use vortex_array::arrays::FixedSizeListArray; //! use vortex_array::arrays::PrimitiveArray; -//! use vortex_array::arrays::Extension; -//! use vortex_array::arrays::scalar_fn::ScalarFnArrayExt; -//! use vortex_array::dtype::extension::ExtDType; //! use vortex_array::extension::EmptyMetadata; +//! use vortex_array::session::ArraySession; //! use vortex_array::validity::Validity; //! use vortex_buffer::BufferMut; -//! use vortex_array::session::ArraySession; //! use vortex_session::VortexSession; -//! use vortex_tensor::encodings::turboquant::{TurboQuantConfig, turboquant_encode_unchecked}; -//! use vortex_tensor::scalar_fns::ApproxOptions; -//! use vortex_tensor::scalar_fns::l2_denorm::normalize_as_l2_denorm; +//! use vortex_tensor::encodings::turboquant::{TurboQuantConfig, turboquant_encode}; //! use vortex_tensor::vector::Vector; //! //! // Create a Vector extension array of 100 random 128-d vectors. @@ -110,47 +114,69 @@ //! let fsl = FixedSizeListArray::try_new( //! elements.into_array(), dim, Validity::NonNullable, num_rows, //! ).unwrap(); -//! let ext_dtype = ExtDType::::try_new(EmptyMetadata, fsl.dtype().clone()) -//! .unwrap().erased(); -//! let ext = ExtensionArray::new(ext_dtype, fsl.into_array()); +//! let vector = ExtensionArray::try_new_from_vtable(Vector, EmptyMetadata, fsl.into_array()) +//! .map(|ext| ext.into_array()) +//! .unwrap(); //! -//! // Normalize, then quantize the normalized child at 2 bits per coordinate. +//! // Normalize and quantize at 2 bits per coordinate in one pass. //! let session = VortexSession::empty().with::(); //! let mut ctx = session.create_execution_ctx(); -//! let l2_denorm = normalize_as_l2_denorm( -//! &ApproxOptions::Exact, ext.into_array(), &mut ctx, -//! ).unwrap(); -//! let normalized = l2_denorm.child_at(0).clone(); -//! -//! let normalized_ext = normalized.as_opt::().unwrap(); -//! let config = TurboQuantConfig { bit_width: 2, seed: Some(42), num_rounds: 3 }; -//! // SAFETY: We just normalized the input. -//! let tq = unsafe { -//! turboquant_encode_unchecked(normalized_ext, &config, &mut ctx).unwrap() -//! }; +//! let config = TurboQuantConfig { bit_width: 2, seed: 42, num_rounds: 3 }; +//! let tq = turboquant_encode(vector, &config, &mut ctx).unwrap(); //! //! // Verify compression: 100 vectors x 128 dims x 4 bytes = 51200 bytes input. //! assert!(tq.nbytes() < 51200); //! ``` -mod array; -pub use array::data::TurboQuantArrayExt; -pub use array::data::TurboQuantData; +pub(crate) mod centroids; +pub(crate) mod compress; + +mod scheme; +pub use compress::TurboQuantConfig; +pub use compress::turboquant_encode; +pub use compress::turboquant_encode_unchecked; +pub use scheme::TurboQuantScheme; -pub(crate) mod compute; +/// Minimum vector dimension for TurboQuant encoding. +/// +/// Note that this is not a theoretical minimum, it is mostly a practical one to limit the total +/// amount of distortion. +pub const MIN_DIMENSION: u32 = 128; -mod metadata; +/// Maximum supported number of bits per quantized coordinate. +pub const MAX_BIT_WIDTH: u8 = 8; -mod vtable; +/// Maximum supported number of centroids in the scalar quantizer codebook. +pub const MAX_CENTROIDS: usize = 1usize << (MAX_BIT_WIDTH as usize); -pub use vtable::TurboQuant; -pub use vtable::TurboQuantArray; +use vortex_array::dtype::DType; +use vortex_error::VortexResult; +use vortex_error::vortex_ensure; +use vortex_error::vortex_err; -mod scheme; -pub use scheme::TurboQuantScheme; -pub use scheme::compress::TurboQuantConfig; -pub use scheme::compress::turboquant_encode; -pub use scheme::compress::turboquant_encode_unchecked; +use crate::types::vector::AnyVector; +use crate::types::vector::VectorMatcherMetadata; + +/// Validates that `dtype` is a [`Vector`](crate::vector::Vector) extension type with +/// dimension >= [`MIN_DIMENSION`]. +/// +/// Returns the validated vector metadata on success. +pub fn tq_validate_vector_dtype(dtype: &DType) -> VortexResult { + let vector_metadata = dtype + .as_extension_opt() + .and_then(|ext| ext.metadata_opt::()) + .ok_or_else(|| { + vortex_err!("TurboQuant dtype must be a Vector extension type, got {dtype}") + })?; + + let dimensions = vector_metadata.dimensions(); + vortex_ensure!( + dimensions >= MIN_DIMENSION, + "TurboQuant requires dimension >= {MIN_DIMENSION}, got {dimensions}", + ); + + Ok(vector_metadata) +} #[cfg(test)] mod tests; diff --git a/vortex-tensor/src/encodings/turboquant/scheme.rs b/vortex-tensor/src/encodings/turboquant/scheme.rs new file mode 100644 index 00000000000..d4362096bd2 --- /dev/null +++ b/vortex-tensor/src/encodings/turboquant/scheme.rs @@ -0,0 +1,221 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! TurboQuant compression scheme. +//! +//! The scheme is a thin [`Scheme`] adapter over [`turboquant_encode`], which produces: +//! +//! ```text +//! ScalarFnArray(L2Denorm, [ +//! ScalarFnArray( +//! SorfTransform, +//! FSL(Dict(codes, centroids)) +//! ), +//! norms +//! ]) +//! ``` +//! +//! Decompression is automatic: executing the outer array walks the ScalarFn tree. +//! +//! [`turboquant_encode`]: crate::encodings::turboquant::turboquant_encode + +use vortex_array::ArrayRef; +use vortex_array::Canonical; +use vortex_array::ExecutionCtx; +use vortex_compressor::CascadingCompressor; +use vortex_compressor::ctx::CompressorContext; +use vortex_compressor::estimate::CompressionEstimate; +use vortex_compressor::estimate::EstimateVerdict; +use vortex_compressor::scheme::Scheme; +use vortex_compressor::stats::ArrayAndStats; +use vortex_error::VortexExpect; +use vortex_error::VortexResult; + +use crate::encodings::turboquant::MAX_CENTROIDS; +use crate::encodings::turboquant::TurboQuantConfig; +use crate::encodings::turboquant::tq_validate_vector_dtype; +use crate::encodings::turboquant::turboquant_encode; + +/// TurboQuant compression scheme for [`Vector`] extension types. +/// +/// Applies lossy vector quantization to [`Vector`] extension arrays using the TurboQuant algorithm +/// with MSE-optimal encoding. +/// +/// Register this scheme with the compressor builder via `with_scheme`: +/// +/// ```ignore +/// use vortex_btrblocks::BtrBlocksCompressorBuilder; +/// use vortex_tensor::encodings::turboquant::TurboQuantScheme; +/// +/// let compressor = BtrBlocksCompressorBuilder::default() +/// .with_new_scheme(&TurboQuantScheme) +/// .build(); +/// ``` +/// +/// [`Vector`]: crate::vector::Vector +#[derive(Debug, Copy, Clone, PartialEq, Eq)] +pub struct TurboQuantScheme; + +impl Scheme for TurboQuantScheme { + fn scheme_name(&self) -> &'static str { + "vortex.tensor.turboquant" + } + + fn matches(&self, canonical: &Canonical) -> bool { + let Canonical::Extension(ext) = canonical else { + return false; + }; + + tq_validate_vector_dtype(ext.dtype()).is_ok() + } + + fn expected_compression_ratio( + &self, + data: &ArrayAndStats, + _compress_ctx: CompressorContext, + _exec_ctx: &mut ExecutionCtx, + ) -> CompressionEstimate { + let len = data.array().len(); + let dtype = data.array().dtype(); + + let vector_metadata = + tq_validate_vector_dtype(dtype).vortex_expect("invalid dtype for TurboQuant"); + let element_ptype = vector_metadata.element_ptype(); + let element_bit_width: u8 = element_ptype + .bit_width() + .try_into() + .vortex_expect("invalid bit width for TurboQuant"); + let dimension = vector_metadata.dimensions(); + + CompressionEstimate::Verdict(EstimateVerdict::Ratio(estimate_compression_ratio( + element_bit_width, + dimension, + len, + ))) + } + + fn compress( + &self, + _compressor: &CascadingCompressor, + data: &ArrayAndStats, + _compress_ctx: CompressorContext, + exec_ctx: &mut ExecutionCtx, + ) -> VortexResult { + turboquant_encode(data.array().clone(), &TurboQuantConfig::default(), exec_ctx) + } +} + +// TODO(connor): If we ever add scheme vtables with metadata, we would need to pass in the config as +// a parameter here. +/// Estimate the compression ratio for TurboQuant MSE encoding with the default config. +fn estimate_compression_ratio(element_bit_width: u8, dimensions: u32, num_vectors: usize) -> f64 { + let config = TurboQuantConfig::default(); + let padded_dim = dimensions.next_power_of_two() as usize; + let element_bits = usize::from(element_bit_width); + + // Get the size of the fully uncompressed vector data. + let uncompressed_size_bits = element_bits * dimensions as usize * num_vectors; + + // Per-vector: MSE codes per padded coordinate, plus one stored norm in the input element float + // width. + let norm_bits = element_bits; + let compressed_bits_per_vector = usize::from(config.bit_width) * padded_dim; + let total_bits_per_vector = norm_bits + compressed_bits_per_vector; + + // Shared overhead: codebook centroids (2^bit_width f32 values). + let num_centroids = 1usize << config.bit_width; + debug_assert!(num_centroids <= MAX_CENTROIDS); + let overhead_bits = num_centroids * 32; // centroids are always f32 + + // This includes the quantized vectors, norms, and centroid codebook. + let compressed_size_bits = total_bits_per_vector * num_vectors + overhead_bits; + + uncompressed_size_bits as f64 / compressed_size_bits as f64 +} + +#[cfg(test)] +mod tests { + use rstest::rstest; + + use super::*; + + /// Verify compression ratio for typical embedding dimensions. + /// + /// f32 input at 768-d (padded to 1024) with 1000 vectors should give ~3x. + /// f32 input at 1024-d (no padding) should give ~4x since no padding waste. + #[rstest] + #[case::f32_768d(32, 768, 1000, 2.5, 4.5)] + #[case::f32_1024d(32, 1024, 1000, 3.5, 5.0)] + #[case::f32_1536d(32, 1536, 1000, 2.5, 4.5)] + #[case::f32_128d(32, 128, 1000, 3.0, 5.0)] + #[case::f64_768d(64, 768, 1000, 5.0, 9.0)] + #[case::f16_768d(16, 768, 1000, 1.2, 2.5)] + fn compression_ratio_in_expected_range( + #[case] element_bit_width: u8, + #[case] dim: u32, + #[case] num_vectors: usize, + #[case] min_ratio: f64, + #[case] max_ratio: f64, + ) { + let ratio = estimate_compression_ratio(element_bit_width, dim, num_vectors); + assert!( + ratio > min_ratio && ratio < max_ratio, + "ratio {ratio:.2} not in [{min_ratio}, {max_ratio}] for \ + {element_bit_width}-bit elements, dim={dim}, n={num_vectors}" + ); + } + + /// Compression ratio must always be > 1 for reasonable inputs, + /// otherwise TurboQuant makes things bigger and should not be selected. + #[rstest] + #[case(32, 128, 100)] + #[case(32, 768, 10)] + #[case(64, 256, 50)] + fn ratio_always_greater_than_one( + #[case] element_bit_width: u8, + #[case] dim: u32, + #[case] num_vectors: usize, + ) { + let ratio = estimate_compression_ratio(element_bit_width, dim, num_vectors); + assert!( + ratio > 1.0, + "ratio {ratio:.4} <= 1.0 for {element_bit_width}-bit, dim={dim}, n={num_vectors}" + ); + } + + #[rstest] + #[case(16)] + #[case(32)] + #[case(64)] + fn ratio_accounts_for_norm_storage_width(#[case] element_bit_width: u8) { + let dim = 128u32; + let num_vectors = 1usize; + let padded_dim = dim.next_power_of_two() as usize; + let config = TurboQuantConfig::default(); + let num_centroids = 1usize << config.bit_width; + + let expected_compressed_bits = usize::from(element_bit_width) + + usize::from(config.bit_width) * padded_dim + + num_centroids * 32; + let expected_uncompressed_bits = + usize::from(element_bit_width) * dim as usize * num_vectors; + let expected = expected_uncompressed_bits as f64 / expected_compressed_bits as f64; + + assert_eq!( + estimate_compression_ratio(element_bit_width, dim, num_vectors), + expected + ); + } + + /// Power-of-2 dimensions should have better ratios than their non-power-of-2 + /// predecessors due to no padding waste. + #[test] + fn power_of_two_has_better_ratio() { + let ratio_768 = estimate_compression_ratio(32, 768, 1000); + let ratio_1024 = estimate_compression_ratio(32, 1024, 1000); + assert!( + ratio_1024 > ratio_768, + "1024-d ratio ({ratio_1024:.2}) should exceed 768-d ({ratio_768:.2})" + ); + } +} diff --git a/vortex-tensor/src/encodings/turboquant/scheme/compress.rs b/vortex-tensor/src/encodings/turboquant/scheme/compress.rs deleted file mode 100644 index f787a80cbf7..00000000000 --- a/vortex-tensor/src/encodings/turboquant/scheme/compress.rs +++ /dev/null @@ -1,312 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -//! TurboQuant encoding (quantization) logic. -//! -//! The input to [`turboquant_encode`] must be a non-nullable [`Vector`](crate::vector::Vector) -//! extension array whose rows are already L2-normalized (unit norm). Normalization is handled -//! externally by [`normalize_as_l2_denorm`](crate::scalar_fns::l2_denorm::normalize_as_l2_denorm), -//! which the [`TurboQuantScheme`](super::TurboQuantScheme) calls before invoking this function. - -use vortex_array::ArrayRef; -use vortex_array::ArrayView; -use vortex_array::ExecutionCtx; -use vortex_array::IntoArray; -use vortex_array::arrays::Extension; -use vortex_array::arrays::FixedSizeListArray; -use vortex_array::arrays::PrimitiveArray; -use vortex_array::arrays::extension::ExtensionArrayExt; -use vortex_array::arrays::fixed_size_list::FixedSizeListArrayExt; -use vortex_array::dtype::DType; -use vortex_array::dtype::Nullability; -use vortex_array::dtype::PType; -use vortex_array::validity::Validity; -use vortex_buffer::BufferMut; -use vortex_error::VortexExpect; -use vortex_error::VortexResult; -use vortex_error::vortex_bail; -use vortex_error::vortex_ensure; -use vortex_fastlanes::bitpack_compress::bitpack_encode; - -use crate::encodings::turboquant::TurboQuant; -use crate::encodings::turboquant::array::centroids::compute_centroid_boundaries; -use crate::encodings::turboquant::array::centroids::find_nearest_centroid; -use crate::encodings::turboquant::array::centroids::get_centroids; -use crate::encodings::turboquant::array::rotation::RotationMatrix; -use crate::encodings::turboquant::vtable::TurboQuantArray; -use crate::scalar_fns::l2_denorm::validate_l2_normalized_rows; - -/// Configuration for TurboQuant encoding. -#[derive(Clone, Debug)] -pub struct TurboQuantConfig { - /// Bits per coordinate (1-8). - pub bit_width: u8, - /// Optional seed for the rotation matrix. If None, the default seed is used. - pub seed: Option, - /// Number of sign-diagonal + WHT rounds in the structured rotation (default 3). - pub num_rounds: u8, -} - -impl Default for TurboQuantConfig { - fn default() -> Self { - Self { - bit_width: TurboQuant::MAX_BIT_WIDTH, - seed: Some(42), - num_rounds: 3, - } - } -} - -/// Extract elements from a FixedSizeListArray as a flat f32 PrimitiveArray for quantization. -/// -/// All quantization (rotation, centroid lookup) happens in f32. f16 is upcast; f64 is truncated. -fn extract_f32_elements( - fsl: &FixedSizeListArray, - ctx: &mut ExecutionCtx, -) -> VortexResult { - let elements = fsl.elements(); - let primitive = elements.clone().execute::(ctx)?; - let ptype = primitive.ptype(); - - match ptype { - PType::F16 => Ok(primitive - .as_slice::() - .iter() - .map(|&v| f32::from(v)) - .collect()), - PType::F32 => Ok(primitive), - PType::F64 => Ok(primitive - .as_slice::() - .iter() - .map(|&v| { - #[expect( - clippy::cast_possible_truncation, - reason = "TurboQuant quantization operates in f32, so f64 inputs are intentionally downcast" - )] - let v = v as f32; - v - }) - .collect()), - _ => vortex_bail!("TurboQuant requires float elements, got {ptype:?}"), - } -} - -/// Shared intermediate results from the quantization loop. -struct QuantizationResult { - rotation: RotationMatrix, - centroids: Vec, - all_indices: BufferMut, - padded_dim: usize, -} - -/// Core quantization: rotate and quantize already-normalized rows. -/// -/// The input `fsl` must contain non-nullable, unit-norm vectors (already L2-normalized). Null -/// vectors are not supported and must be zeroed out before reaching this function. The rotation -/// and centroid lookup happen in f32. -fn turboquant_quantize_core( - fsl: &FixedSizeListArray, - seed: u64, - bit_width: u8, - num_rounds: u8, - ctx: &mut ExecutionCtx, -) -> VortexResult { - let dimension = - usize::try_from(fsl.list_size()).vortex_expect("u32 FixedSizeList dimension fits in usize"); - let num_rows = fsl.len(); - - let rotation = RotationMatrix::try_new(seed, dimension, num_rounds as usize)?; - let padded_dim = rotation.padded_dim(); - let padded_dim_u32 = - u32::try_from(padded_dim).vortex_expect("padded_dim stays representable as u32"); - - let f32_elements = extract_f32_elements(fsl, ctx)?; - - let centroids = get_centroids(padded_dim_u32, bit_width)?; - let boundaries = compute_centroid_boundaries(¢roids); - - let mut all_indices = BufferMut::::with_capacity(num_rows * padded_dim); - let mut padded = vec![0.0f32; padded_dim]; - let mut rotated = vec![0.0f32; padded_dim]; - - let f32_slice = f32_elements.as_slice::(); - for row in 0..num_rows { - let x = &f32_slice[row * dimension..(row + 1) * dimension]; - - // Zero-pad to the next power of 2. - padded[..dimension].copy_from_slice(x); - padded[dimension..].fill(0.0); - - rotation.rotate(&padded, &mut rotated); - - for j in 0..padded_dim { - all_indices.push(find_nearest_centroid(rotated[j], &boundaries)); - } - } - - Ok(QuantizationResult { - rotation, - centroids, - all_indices, - padded_dim, - }) -} - -/// Build a `TurboQuantArray` from quantization results. -/// -/// The `ext_dtype` must be a non-nullable [`Vector`](crate::vector::Vector) extension dtype. -fn build_turboquant( - num_rows: usize, - core: QuantizationResult, - ext_dtype: &DType, -) -> VortexResult { - let padded_dim = core.padded_dim; - let padded_dim_u32 = - u32::try_from(padded_dim).vortex_expect("padded_dim stays representable as u32"); - let codes_elements = - PrimitiveArray::new::(core.all_indices.freeze(), Validity::NonNullable); - let codes = FixedSizeListArray::try_new( - codes_elements.into_array(), - padded_dim_u32, - Validity::NonNullable, - num_rows, - )? - .into_array(); - - // TODO(perf): `get_centroids` returns Vec; could avoid the copy by - // supporting Buffer::from(Vec) or caching as Buffer directly. - let mut centroids_buf = BufferMut::::with_capacity(core.centroids.len()); - centroids_buf.extend_from_slice(&core.centroids); - let centroids_array = - PrimitiveArray::new::(centroids_buf.freeze(), Validity::NonNullable).into_array(); - - let rotation_signs = bitpack_rotation_signs(&core.rotation)?; - - TurboQuant::try_new_array(ext_dtype.clone(), codes, centroids_array, rotation_signs) -} - -/// Encode a non-nullable, L2-normalized [`Vector`](crate::vector::Vector) extension array into a -/// [`TurboQuantArray`]. -/// -/// The input must be a non-nullable Vector extension array whose rows are already unit-norm. -/// **Null vectors are not supported.** The caller must normalize and strip nullability before -/// calling this function, for example via [`normalize_as_l2_denorm`]. -/// -/// This function validates that every row is L2-normalized (or is exactly 0.0). Use -/// [`turboquant_encode_unchecked`] to skip this check when the caller has just performed -/// normalization. -/// -/// The returned array is a plain [`TurboQuantArray`] that decompresses to unit-norm vectors. -/// The caller is responsible for wrapping it in an [`L2Denorm`] ScalarFnArray if the original -/// magnitudes need to be restored. -/// -/// [`normalize_as_l2_denorm`]: crate::scalar_fns::l2_denorm::normalize_as_l2_denorm -/// [`L2Denorm`]: crate::scalar_fns::l2_denorm::L2Denorm -pub fn turboquant_encode( - ext: ArrayView, - config: &TurboQuantConfig, - ctx: &mut ExecutionCtx, -) -> VortexResult { - let ext_dtype = ext.dtype().clone(); - - vortex_ensure!( - !ext_dtype.is_nullable(), - "TurboQuant input must be non-nullable (normalize first via L2Denorm), got {ext_dtype}", - ); - - validate_l2_normalized_rows(ext.as_ref().clone(), ctx)?; - - // SAFETY: We just validated that the input is non-nullable and all rows are unit-norm. - unsafe { turboquant_encode_unchecked(ext, config, ctx) } -} - -/// Encode a non-nullable, L2-normalized [`Vector`](crate::vector::Vector) extension array into a -/// [`TurboQuantArray`], without validating the unit-norm precondition. -/// -/// # Safety -/// -/// The caller must ensure: -/// -/// - The input dtype is non-nullable. -/// - Every row is L2-normalized (unit norm) or is a zero vector. -/// -/// Passing non-unit-norm vectors will not cause memory unsafety, but will produce silently -/// incorrect quantization results. -pub unsafe fn turboquant_encode_unchecked( - ext: ArrayView, - config: &TurboQuantConfig, - ctx: &mut ExecutionCtx, -) -> VortexResult { - let ext_dtype = ext.dtype().clone(); - let storage = ext.storage_array(); - let fsl = storage.clone().execute::(ctx)?; - - vortex_ensure!( - config.bit_width >= 1 && config.bit_width <= TurboQuant::MAX_BIT_WIDTH, - "bit_width must be 1-{}, got {}", - TurboQuant::MAX_BIT_WIDTH, - config.bit_width - ); - let dimension = fsl.list_size(); - vortex_ensure!( - dimension >= TurboQuant::MIN_DIMENSION, - "TurboQuant requires dimension >= {}, got {dimension}", - TurboQuant::MIN_DIMENSION - ); - - if fsl.is_empty() { - let padded_dim = dimension.next_power_of_two(); - let empty_codes = FixedSizeListArray::try_new( - PrimitiveArray::empty::(Nullability::NonNullable).into_array(), - padded_dim, - Validity::NonNullable, - 0, - )?; - - let empty_centroids = PrimitiveArray::empty::(Nullability::NonNullable); - let empty_signs = FixedSizeListArray::try_new( - PrimitiveArray::empty::(Nullability::NonNullable).into_array(), - padded_dim, - Validity::NonNullable, - 0, - )?; - - return Ok(TurboQuant::try_new_array( - ext_dtype, - empty_codes.into_array(), - empty_centroids.into_array(), - empty_signs.into_array(), - )? - .into_array()); - } - - let seed = config.seed.unwrap_or(42); - let num_rows = fsl.len(); - let core = turboquant_quantize_core(&fsl, seed, config.bit_width, config.num_rounds, ctx)?; - - Ok(build_turboquant(num_rows, core, &ext_dtype)?.into_array()) -} - -/// Export rotation signs as a `FixedSizeListArray` wrapping a 1-bit [`BitPackedArray`]. -/// -/// The rotation matrix's `num_rounds * padded_dim` sign values are exported as 0/1 u8 values in -/// inverse application order, bitpacked to 1 bit per sign, then wrapped in a -/// `FixedSizeListArray` with `list_size = padded_dim` and `len = num_rounds`. -fn bitpack_rotation_signs(rotation: &RotationMatrix) -> VortexResult { - let signs_u8 = rotation.export_inverse_signs_u8(); - let num_rounds = rotation.num_rounds(); - let padded_dim = u32::try_from(rotation.padded_dim()).vortex_expect("padded_dim fits in u32"); - - let mut buf = BufferMut::::with_capacity(signs_u8.len()); - buf.extend_from_slice(&signs_u8); - let prim = PrimitiveArray::new::(buf.freeze(), Validity::NonNullable); - let bitpacked = bitpack_encode(&prim, 1, None)?; - - let fsl = FixedSizeListArray::try_new( - bitpacked.into_array(), - padded_dim, - Validity::NonNullable, - num_rounds, - )?; - Ok(fsl.into_array()) -} diff --git a/vortex-tensor/src/encodings/turboquant/scheme/decompress.rs b/vortex-tensor/src/encodings/turboquant/scheme/decompress.rs deleted file mode 100644 index 2e92ced88e1..00000000000 --- a/vortex-tensor/src/encodings/turboquant/scheme/decompress.rs +++ /dev/null @@ -1,141 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -//! TurboQuant decoding (dequantization) logic. -//! -//! Decompression produces unit-norm vectors. The original magnitudes are restored externally -//! by the [`L2Denorm`](crate::scalar_fns::l2_denorm::L2Denorm) ScalarFnArray wrapper. - -use num_traits::Float; -use num_traits::FromPrimitive; -use vortex_array::Array; -use vortex_array::ArrayRef; -use vortex_array::ExecutionCtx; -use vortex_array::IntoArray; -use vortex_array::arrays::ExtensionArray; -use vortex_array::arrays::FixedSizeListArray; -use vortex_array::arrays::PrimitiveArray; -use vortex_array::arrays::fixed_size_list::FixedSizeListArrayExt; -use vortex_array::dtype::NativePType; -use vortex_array::dtype::Nullability; -use vortex_array::match_each_float_ptype; -use vortex_array::validity::Validity; -use vortex_buffer::BufferMut; -use vortex_error::VortexResult; - -use crate::encodings::turboquant::TurboQuant; -use crate::encodings::turboquant::TurboQuantArrayExt; -use crate::encodings::turboquant::array::rotation::RotationMatrix; -use crate::encodings::turboquant::compute::float_from_f32; -use crate::vector::AnyVector; - -/// Decompress a `TurboQuantArray` into a unit-norm [`Vector`] extension array. -/// -/// The returned array is an [`ExtensionArray`] with the (non-nullable) Vector dtype wrapping a -/// `FixedSizeListArray` of the original vector element type. Each vector has unit L2 norm; the -/// original magnitudes are restored by the [`L2Denorm`](crate::scalar_fns::l2_denorm::L2Denorm) -/// ScalarFnArray wrapper. -/// -/// [`Vector`]: crate::vector::Vector -pub fn execute_decompress( - array: Array, - ctx: &mut ExecutionCtx, -) -> VortexResult { - let dim = array.dimension() as usize; - let padded_dim = array.padded_dim() as usize; - let num_rows = array.len(); - let ext_dtype = array.dtype().as_extension().clone(); - let element_ptype = ext_dtype.metadata::().element_ptype(); - - if num_rows == 0 { - match_each_float_ptype!(element_ptype, |T| { - let elements = PrimitiveArray::empty::(Nullability::NonNullable); - let fsl = FixedSizeListArray::try_new( - elements.into_array(), - array.dimension(), - Validity::NonNullable, - 0, - )?; - - return Ok(ExtensionArray::new(ext_dtype, fsl.into_array()).into_array()); - }) - } - - // Read stored centroids (always f32). - let centroids_prim = array.centroids().clone().execute::(ctx)?; - let centroids = centroids_prim.as_slice::(); - - // The rotation signs are stored as a FixedSizeListArray wrapping bitpacked u8 values. - // We unwrap to the flat elements, then FastLanes SIMD-unpacks the 1-bit values into u8 0/1. - // These are expanded to u32 XOR masks once (amortized over all rows), enabling branchless - // XOR-based sign application in the per-row structured-rotation hot loop. - let num_rounds = array.num_rounds() as usize; - let signs_fsl = array - .rotation_signs() - .clone() - .execute::(ctx)?; - let signs_prim = signs_fsl - .elements() - .clone() - .execute::(ctx)?; - let rotation = RotationMatrix::from_u8_slice(signs_prim.as_slice::(), dim, num_rounds)?; - - // Unpack codes from FixedSizeListArray -> flat u8 elements. - let codes_fsl = array.codes().clone().execute::(ctx)?; - let codes_prim = codes_fsl - .elements() - .clone() - .execute::(ctx)?; - let indices = codes_prim.as_slice::(); - - // MSE decode: dequantize (f32) -> inverse rotate (f32) -> cast to T. - // The rotation and centroid lookup always happen in f32. The final output is cast to the - // Vector's element type to match the original storage dtype. No norm scaling is applied here; - // that is handled by the external L2Denorm wrapper. - match_each_float_ptype!(element_ptype, |T| { - decompress_typed::(centroids, &rotation, indices, dim, padded_dim, num_rows).and_then( - |elements| { - let fsl = FixedSizeListArray::try_new( - elements.into_array(), - array.dimension(), - Validity::NonNullable, - num_rows, - )?; - Ok(ExtensionArray::new(ext_dtype, fsl.into_array()).into_array()) - }, - ) - }) -} - -/// Typed decompress: dequantizes in f32 and produces unit-norm output as `T`. -fn decompress_typed( - centroids: &[f32], - rotation: &RotationMatrix, - indices: &[u8], - dim: usize, - padded_dim: usize, - num_rows: usize, -) -> VortexResult { - let mut output = BufferMut::::with_capacity(num_rows * dim); - let mut dequantized = vec![0.0f32; padded_dim]; - let mut unrotated = vec![0.0f32; padded_dim]; - - for row in 0..num_rows { - let row_indices = &indices[row * padded_dim..(row + 1) * padded_dim]; - - for idx in 0..padded_dim { - dequantized[idx] = centroids[row_indices[idx] as usize]; - } - - rotation.inverse_rotate(&dequantized, &mut unrotated); - - for idx in 0..dim { - output.push(float_from_f32::(unrotated[idx])); - } - } - - Ok(PrimitiveArray::new::( - output.freeze(), - Validity::NonNullable, - )) -} diff --git a/vortex-tensor/src/encodings/turboquant/scheme/mod.rs b/vortex-tensor/src/encodings/turboquant/scheme/mod.rs deleted file mode 100644 index c3beb2a6993..00000000000 --- a/vortex-tensor/src/encodings/turboquant/scheme/mod.rs +++ /dev/null @@ -1,211 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -//! TurboQuant compression scheme and decompression. -//! -//! The scheme first normalizes the input via [`normalize_as_l2_denorm`], then encodes the -//! normalized child via [`turboquant_encode`]. The result is: -//! -//! ```text -//! ScalarFnArray(L2Denorm, [TurboQuantArray, norms]) -//! ``` -//! -//! [`normalize_as_l2_denorm`]: crate::scalar_fns::l2_denorm::normalize_as_l2_denorm - -use vortex_array::ArrayRef; -use vortex_array::Canonical; -use vortex_array::IntoArray; -use vortex_array::arrays::Extension; -use vortex_array::arrays::scalar_fn::ScalarFnArrayExt; -use vortex_compressor::CascadingCompressor; -use vortex_compressor::ctx::CompressorContext; -use vortex_compressor::estimate::CompressionEstimate; -use vortex_compressor::scheme::Scheme; -use vortex_compressor::stats::ArrayAndStats; -use vortex_error::VortexExpect; -use vortex_error::VortexResult; - -use crate::encodings::turboquant::TurboQuant; -use crate::encodings::turboquant::TurboQuantConfig; -use crate::encodings::turboquant::turboquant_encode_unchecked; -use crate::scalar_fns::ApproxOptions; -use crate::scalar_fns::l2_denorm::L2Denorm; -use crate::scalar_fns::l2_denorm::normalize_as_l2_denorm; - -pub(super) mod compress; -pub(super) mod decompress; - -/// TurboQuant compression scheme for [`Vector`] extension types. -/// -/// Applies lossy vector quantization to [`Vector`] extension arrays using the TurboQuant -/// algorithm with MSE-optimal encoding. -/// -/// Register this scheme with the compressor builder via `with_scheme`: -/// ```ignore -/// use vortex_btrblocks::BtrBlocksCompressorBuilder; -/// use vortex_tensor::encodings::turboquant::TurboQuantScheme; -/// -/// let compressor = BtrBlocksCompressorBuilder::default() -/// .with_new_scheme(&TurboQuantScheme) -/// .build(); -/// ``` -/// -/// [`Vector`]: crate::vector::Vector -#[derive(Debug, Copy, Clone, PartialEq, Eq)] -pub struct TurboQuantScheme; - -impl Scheme for TurboQuantScheme { - fn scheme_name(&self) -> &'static str { - "vortex.tensor.turboquant" - } - - fn matches(&self, canonical: &Canonical) -> bool { - let Canonical::Extension(ext) = canonical else { - return false; - }; - - TurboQuant::validate_dtype(ext.dtype()).is_ok() - } - - fn expected_compression_ratio( - &self, - data: &mut ArrayAndStats, - _ctx: CompressorContext, - ) -> CompressionEstimate { - let len = data.array().len(); - let dtype = data.array().dtype(); - - let vector_metadata = - TurboQuant::validate_dtype(dtype).vortex_expect("invalid dtype for TurboQuant"); - let element_ptype = vector_metadata.element_ptype(); - let bit_width: u8 = element_ptype - .bit_width() - .try_into() - .vortex_expect("invalid bit width for TurboQuant"); - let dimension = vector_metadata.dimensions(); - - CompressionEstimate::Ratio(estimate_compression_ratio(bit_width, dimension, len)) - } - - fn compress( - &self, - compressor: &CascadingCompressor, - data: &mut ArrayAndStats, - _ctx: CompressorContext, - ) -> VortexResult { - let ext_array = data - .array() - .as_opt::() - .vortex_expect("expected an extension array"); - - let mut ctx = compressor.execution_ctx(); - - // Normalize first: produces L2Denorm(normalized_vectors, norms). - let l2_denorm = - normalize_as_l2_denorm(&ApproxOptions::Exact, ext_array.as_ref().clone(), &mut ctx)?; - let normalized = l2_denorm.child_at(0).clone(); - let norms = l2_denorm.child_at(1).clone(); - let num_rows = l2_denorm.len(); - - // Quantize the normalized child. - let normalized_ext = normalized - .as_opt::() - .vortex_expect("normalized child should be an Extension array"); - let config = TurboQuantConfig::default(); - // SAFETY: We just normalized the input via `normalize_as_l2_denorm`, so all rows are - // guaranteed to be unit-norm (or zero for originally-null rows). - let tq = unsafe { turboquant_encode_unchecked(normalized_ext, &config, &mut ctx)? }; - - // SAFETY: TurboQuant is a lossy approximation of the normalized child, so we intentionally - // bypass the strict normalized-row validation when reattaching the stored norms. - Ok( - unsafe { L2Denorm::new_array_unchecked(&ApproxOptions::Exact, tq, norms, num_rows) }? - .into_array(), - ) - } -} - -/// Estimate the compression ratio for TurboQuant MSE encoding with the default config. -fn estimate_compression_ratio(bits_per_element: u8, dimensions: u32, num_vectors: usize) -> f64 { - let config = TurboQuantConfig::default(); - let padded_dim = dimensions.next_power_of_two() as usize; - - // Per-vector: MSE codes per padded coordinate, plus one f32 norm. - let compressed_bits_per_vector = 32 // norm is always f32 - + (config.bit_width as usize) * padded_dim; // MSE codes - - // Shared overhead: codebook centroids (2^bit_width f32 values) and - // rotation signs (num_rounds * padded_dim bits). - let num_centroids = 1usize << config.bit_width; - debug_assert!(num_centroids <= TurboQuant::MAX_CENTROIDS); - let overhead_bits = num_centroids * 32 // centroids are always f32 - + config.num_rounds as usize * padded_dim; // rotation signs, 1 bit each - - let compressed_size_bits = compressed_bits_per_vector * num_vectors + overhead_bits; - - let uncompressed_size_bits = bits_per_element as usize * dimensions as usize * num_vectors; - uncompressed_size_bits as f64 / compressed_size_bits as f64 -} - -#[cfg(test)] -mod tests { - use rstest::rstest; - - use super::*; - - /// Verify compression ratio for typical embedding dimensions. - /// - /// f32 input at 768-d (padded to 1024) with 1000 vectors should give ~4-6x. - /// f32 input at 1024-d (no padding) should give higher ratio since no waste. - #[rstest] - #[case::f32_768d(32, 768, 1000, 2.5, 4.0)] - #[case::f32_1024d(32, 1024, 1000, 3.5, 5.0)] - #[case::f32_1536d(32, 1536, 1000, 2.5, 4.0)] - #[case::f32_128d(32, 128, 1000, 3.0, 5.0)] - #[case::f64_768d(64, 768, 1000, 5.0, 7.0)] - #[case::f16_768d(16, 768, 1000, 1.2, 2.0)] - fn compression_ratio_in_expected_range( - #[case] bits_per_element: u8, - #[case] dim: u32, - #[case] num_vectors: usize, - #[case] min_ratio: f64, - #[case] max_ratio: f64, - ) { - let ratio = estimate_compression_ratio(bits_per_element, dim, num_vectors); - assert!( - ratio > min_ratio && ratio < max_ratio, - "ratio {ratio:.2} not in [{min_ratio}, {max_ratio}] for \ - {bits_per_element}-bit elements, dim={dim}, n={num_vectors}" - ); - } - - /// Compression ratio must always be > 1 for reasonable inputs, - /// otherwise TurboQuant makes things bigger and should not be selected. - #[rstest] - #[case(32, 128, 100)] - #[case(32, 768, 10)] - #[case(64, 256, 50)] - fn ratio_always_greater_than_one( - #[case] bits_per_element: u8, - #[case] dim: u32, - #[case] num_vectors: usize, - ) { - let ratio = estimate_compression_ratio(bits_per_element, dim, num_vectors); - assert!( - ratio > 1.0, - "ratio {ratio:.4} <= 1.0 for {bits_per_element}-bit, dim={dim}, n={num_vectors}" - ); - } - - /// Power-of-2 dimensions should have better ratios than their non-power-of-2 - /// predecessors due to no padding waste. - #[test] - fn power_of_two_has_better_ratio() { - let ratio_768 = estimate_compression_ratio(32, 768, 1000); - let ratio_1024 = estimate_compression_ratio(32, 1024, 1000); - assert!( - ratio_1024 > ratio_768, - "1024-d ratio ({ratio_1024:.2}) should exceed 768-d ({ratio_768:.2})" - ); - } -} diff --git a/vortex-tensor/src/encodings/turboquant/tests.rs b/vortex-tensor/src/encodings/turboquant/tests.rs deleted file mode 100644 index 8e633469a3d..00000000000 --- a/vortex-tensor/src/encodings/turboquant/tests.rs +++ /dev/null @@ -1,1260 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -use std::sync::LazyLock; - -use rand::SeedableRng; -use rand::rngs::StdRng; -use rand_distr::Distribution; -use rand_distr::Normal; -use rstest::rstest; -use vortex_array::ArrayRef; -use vortex_array::IntoArray; -use vortex_array::VortexSessionExecute; -use vortex_array::arrays::Extension; -use vortex_array::arrays::ExtensionArray; -use vortex_array::arrays::FixedSizeListArray; -use vortex_array::arrays::PrimitiveArray; -use vortex_array::arrays::ScalarFnVTable; -use vortex_array::arrays::extension::ExtensionArrayExt; -use vortex_array::arrays::fixed_size_list::FixedSizeListArrayExt; -use vortex_array::arrays::scalar_fn::ScalarFnArrayExt; -use vortex_array::dtype::Nullability; -use vortex_array::dtype::extension::ExtDType; -use vortex_array::extension::EmptyMetadata; -use vortex_array::session::ArraySession; -use vortex_array::validity::Validity; -use vortex_buffer::BufferMut; -use vortex_error::VortexExpect; -use vortex_error::VortexResult; -use vortex_session::VortexSession; - -use crate::encodings::turboquant::TurboQuant; -use crate::encodings::turboquant::TurboQuantArrayExt; -use crate::encodings::turboquant::TurboQuantConfig; -use crate::encodings::turboquant::array::rotation::RotationMatrix; -use crate::encodings::turboquant::turboquant_encode; -use crate::encodings::turboquant::turboquant_encode_unchecked; -use crate::scalar_fns::ApproxOptions; -use crate::scalar_fns::l2_denorm::L2Denorm; -use crate::scalar_fns::l2_denorm::normalize_as_l2_denorm; -use crate::scalar_fns::l2_norm::L2Norm; -use crate::vector::Vector; - -static SESSION: LazyLock = - LazyLock::new(|| VortexSession::empty().with::()); - -/// Create a FixedSizeListArray of random f32 vectors (i.i.d. standard normal) with the given -/// validity. -fn make_fsl_with_validity( - num_rows: usize, - dim: usize, - seed: u64, - validity: Validity, -) -> FixedSizeListArray { - let mut rng = StdRng::seed_from_u64(seed); - let normal = Normal::new(0.0f32, 1.0).unwrap(); - - let mut buf = BufferMut::::with_capacity(num_rows * dim); - for _ in 0..(num_rows * dim) { - buf.push(normal.sample(&mut rng)); - } - - let elements = PrimitiveArray::new::(buf.freeze(), Validity::NonNullable); - FixedSizeListArray::try_new( - elements.into_array(), - dim.try_into() - .expect("somehow got dimension greater than u32::MAX"), - validity, - num_rows, - ) - .unwrap() -} - -/// Create a non-nullable FixedSizeListArray of random f32 vectors (i.i.d. standard normal). -fn make_fsl(num_rows: usize, dim: usize, seed: u64) -> FixedSizeListArray { - let mut rng = StdRng::seed_from_u64(seed); - let normal = Normal::new(0.0f32, 1.0).unwrap(); - - let mut buf = BufferMut::::with_capacity(num_rows * dim); - for _ in 0..(num_rows * dim) { - buf.push(normal.sample(&mut rng)); - } - - let elements = PrimitiveArray::new::(buf.freeze(), Validity::NonNullable); - FixedSizeListArray::try_new( - elements.into_array(), - dim.try_into() - .expect("somehow got dimension greater than u32::MAX"), - Validity::NonNullable, - num_rows, - ) - .unwrap() -} - -/// Wrap a `FixedSizeListArray` in a `Vector` extension array. -fn make_vector_ext(fsl: &FixedSizeListArray) -> ExtensionArray { - let ext_dtype = ExtDType::::try_new(EmptyMetadata, fsl.dtype().clone()) - .unwrap() - .erased(); - ExtensionArray::new(ext_dtype, fsl.clone().into_array()) -} - -/// Full encode pipeline: normalize, then TQ-encode, then wrap in L2Denorm. -/// -/// This mirrors what `TurboQuantScheme::compress()` does: normalize via `normalize_as_l2_denorm`, -/// then quantize the normalized child via `turboquant_encode_unchecked`, then reassemble. -fn normalize_and_encode( - ext: &ExtensionArray, - config: &TurboQuantConfig, - ctx: &mut vortex_array::ExecutionCtx, -) -> VortexResult { - let l2_denorm = normalize_as_l2_denorm(&ApproxOptions::Exact, ext.as_ref().clone(), ctx)?; - let normalized = l2_denorm.child_at(0).clone(); - let norms = l2_denorm.child_at(1).clone(); - let num_rows = l2_denorm.len(); - - let normalized_ext = normalized - .as_opt::() - .vortex_expect("normalized child should be an Extension array"); - // SAFETY: We just normalized the input via `normalize_as_l2_denorm`. - let tq = unsafe { turboquant_encode_unchecked(normalized_ext, config, ctx)? }; - - Ok( - unsafe { L2Denorm::new_array_unchecked(&ApproxOptions::Exact, tq, norms, num_rows) }? - .into_array(), - ) -} - -/// Unwrap an L2Denorm ScalarFnArray into its TQ child and norms child. -fn unwrap_l2denorm(encoded: &ArrayRef) -> (ArrayRef, ArrayRef) { - let sfn = encoded - .as_opt::() - .expect("expected ScalarFnArray"); - let tq_child = sfn.child_at(0).clone(); - let norms_child = sfn.child_at(1).clone(); - (tq_child, norms_child) -} - -fn theoretical_mse_bound(bit_width: u8) -> f32 { - let sqrt3_pi_over_2 = (3.0f32).sqrt() * std::f32::consts::PI / 2.0; - sqrt3_pi_over_2 / (4.0f32).powi(bit_width as i32) -} - -fn per_vector_normalized_mse( - original: &[f32], - reconstructed: &[f32], - dim: usize, - num_rows: usize, -) -> f32 { - let mut total = 0.0f32; - for row in 0..num_rows { - let orig = &original[row * dim..(row + 1) * dim]; - let recon = &reconstructed[row * dim..(row + 1) * dim]; - let norm_sq: f32 = orig.iter().map(|&v| v * v).sum(); - if norm_sq < 1e-10 { - continue; - } - let err_sq: f32 = orig - .iter() - .zip(recon.iter()) - .map(|(&a, &b)| (a - b) * (a - b)) - .sum(); - total += err_sq / norm_sq; - } - total / num_rows as f32 -} - -/// Normalize, encode, and decode, returning (original, decoded) flat f32 slices. -fn encode_decode( - fsl: &FixedSizeListArray, - config: &TurboQuantConfig, -) -> VortexResult<(Vec, Vec)> { - let mut ctx = SESSION.create_execution_ctx(); - let original: Vec = { - let prim = fsl.elements().clone().execute::(&mut ctx)?; - prim.as_slice::().to_vec() - }; - let ext = make_vector_ext(fsl); - let encoded = normalize_and_encode(&ext, config, &mut ctx)?; - let decoded_ext = encoded.execute::(&mut ctx)?; - let decoded_fsl = decoded_ext - .storage_array() - .clone() - .execute::(&mut ctx)?; - let decoded_elements: Vec = { - let prim = decoded_fsl - .elements() - .clone() - .execute::(&mut ctx)?; - prim.as_slice::().to_vec() - }; - Ok((original, decoded_elements)) -} - -fn empty_turboquant_parts( - dim: u32, -) -> VortexResult<(vortex_array::dtype::DType, ArrayRef, ArrayRef, ArrayRef)> { - let fsl = make_fsl(0, dim as usize, 42); - let ext = make_vector_ext(&fsl); - - let codes = FixedSizeListArray::try_new( - PrimitiveArray::empty::(Nullability::NonNullable).into_array(), - dim, - Validity::NonNullable, - 0, - )? - .into_array(); - let centroids = PrimitiveArray::empty::(Nullability::NonNullable).into_array(); - let rotation_signs = FixedSizeListArray::try_new( - PrimitiveArray::empty::(Nullability::NonNullable).into_array(), - dim, - Validity::NonNullable, - 0, - )? - .into_array(); - - // TQ dtype is non-nullable. - Ok(( - ext.dtype().as_nonnullable(), - codes, - centroids, - rotation_signs, - )) -} - -fn normalized_child( - ext: &ExtensionArray, - ctx: &mut vortex_array::ExecutionCtx, -) -> VortexResult { - Ok( - normalize_as_l2_denorm(&ApproxOptions::Exact, ext.as_ref().clone(), ctx)? - .child_at(0) - .clone(), - ) -} - -// ----------------------------------------------------------------------- -// Roundtrip tests -// ----------------------------------------------------------------------- - -#[rstest] -#[case(128, 1)] -#[case(128, 2)] -#[case(128, 3)] -#[case(128, 4)] -#[case(128, 6)] -#[case(128, 8)] -#[case(256, 2)] -fn roundtrip(#[case] dim: usize, #[case] bit_width: u8) -> VortexResult<()> { - let fsl = make_fsl(10, dim, 42); - let config = TurboQuantConfig { - bit_width, - seed: Some(123), - num_rounds: 3, - }; - let (original, decoded) = encode_decode(&fsl, &config)?; - assert_eq!(decoded.len(), original.len()); - Ok(()) -} - -#[test] -fn empty_try_new_rejects_invalid_centroids_dtype() -> VortexResult<()> { - let (dtype, codes, _centroids, rotation_signs) = empty_turboquant_parts(128)?; - let wrong_centroids = PrimitiveArray::empty::(Nullability::NonNullable).into_array(); - - let err = TurboQuant::try_new_array(dtype, codes, wrong_centroids, rotation_signs).unwrap_err(); - - assert!( - err.to_string() - .contains("centroids dtype must be non-nullable f32") - ); - Ok(()) -} - -#[test] -fn empty_try_new_rejects_invalid_rotation_signs_dtype() -> VortexResult<()> { - let (dtype, codes, centroids, _rotation_signs) = empty_turboquant_parts(128)?; - let wrong_rotation_signs = PrimitiveArray::empty::(Nullability::NonNullable).into_array(); - - let err = TurboQuant::try_new_array(dtype, codes, centroids, wrong_rotation_signs).unwrap_err(); - - assert!( - err.to_string() - .contains("rotation_signs dtype does not match") - ); - Ok(()) -} - -// ----------------------------------------------------------------------- -// MSE quality tests -// ----------------------------------------------------------------------- - -#[rstest] -#[case(128, 1)] -#[case(128, 2)] -#[case(128, 3)] -#[case(128, 4)] -#[case(256, 2)] -#[case(256, 4)] -fn mse_within_theoretical_bound(#[case] dim: usize, #[case] bit_width: u8) -> VortexResult<()> { - let num_rows = 200; - let fsl = make_fsl(num_rows, dim, 42); - let config = TurboQuantConfig { - bit_width, - seed: Some(123), - num_rounds: 3, - }; - let (original, decoded) = encode_decode(&fsl, &config)?; - - let normalized_mse = per_vector_normalized_mse(&original, &decoded, dim, num_rows); - let bound = theoretical_mse_bound(bit_width); - - assert!( - normalized_mse < bound, - "Normalized MSE {normalized_mse:.6} exceeds bound {bound:.6} \ - for dim={dim}, bits={bit_width}", - ); - Ok(()) -} - -#[rstest] -#[case(128, 6)] -#[case(128, 8)] -#[case(256, 6)] -#[case(256, 8)] -fn high_bitwidth_mse_is_small(#[case] dim: usize, #[case] bit_width: u8) -> VortexResult<()> { - let num_rows = 200; - let fsl = make_fsl(num_rows, dim, 42); - - let config_4bit = TurboQuantConfig { - bit_width: 4, - seed: Some(123), - num_rounds: 3, - }; - let (original_4, decoded_4) = encode_decode(&fsl, &config_4bit)?; - let mse_4bit = per_vector_normalized_mse(&original_4, &decoded_4, dim, num_rows); - - let config = TurboQuantConfig { - bit_width, - seed: Some(123), - num_rounds: 3, - }; - let (original, decoded) = encode_decode(&fsl, &config)?; - let mse = per_vector_normalized_mse(&original, &decoded, dim, num_rows); - - assert!( - mse < mse_4bit, - "{bit_width}-bit MSE ({mse:.6}) should be < 4-bit MSE ({mse_4bit:.6})" - ); - assert!(mse < 0.01, "{bit_width}-bit MSE ({mse:.6}) should be < 1%"); - Ok(()) -} - -#[test] -fn mse_decreases_with_bits() -> VortexResult<()> { - let dim = 128; - let num_rows = 50; - let fsl = make_fsl(num_rows, dim, 99); - - let mut prev_mse = f32::MAX; - for bit_width in 1..=8u8 { - let config = TurboQuantConfig { - bit_width, - seed: Some(123), - num_rounds: 3, - }; - let (original, decoded) = encode_decode(&fsl, &config)?; - let mse = per_vector_normalized_mse(&original, &decoded, dim, num_rows); - assert!( - mse <= prev_mse * 1.01, - "MSE should decrease: {bit_width}-bit={mse:.6} > prev={prev_mse:.6}" - ); - prev_mse = mse; - } - Ok(()) -} - -// ----------------------------------------------------------------------- -// Edge cases -// ----------------------------------------------------------------------- - -#[rstest] -#[case(0)] -#[case(1)] -fn roundtrip_edge_cases(#[case] num_rows: usize) -> VortexResult<()> { - let fsl = make_fsl(num_rows, 128, 42); - let ext = make_vector_ext(&fsl); - let config = TurboQuantConfig { - bit_width: 2, - seed: Some(123), - num_rounds: 3, - }; - let mut ctx = SESSION.create_execution_ctx(); - let encoded = normalize_and_encode(&ext, &config, &mut ctx)?; - let decoded = encoded.execute::(&mut ctx)?; - assert_eq!(decoded.len(), num_rows); - Ok(()) -} - -#[rstest] -#[case(1)] -#[case(64)] -#[case(127)] -fn rejects_dimension_below_128(#[case] dim: usize) { - let fsl = make_fsl_small(dim); - let ext = make_vector_ext(&fsl); - let config = TurboQuantConfig { - bit_width: 2, - seed: Some(0), - num_rounds: 3, - }; - let mut ctx = SESSION.create_execution_ctx(); - assert!(turboquant_encode(ext.as_view(), &config, &mut ctx).is_err()); -} - -#[test] -fn checked_encode_accepts_normalized_f16_input() -> VortexResult<()> { - let num_rows = 10; - let dim = 128; - let mut rng = StdRng::seed_from_u64(99); - let normal = Normal::new(0.0f32, 1.0).unwrap(); - - let mut buf = BufferMut::::with_capacity(num_rows * dim); - for _ in 0..(num_rows * dim) { - buf.push(half::f16::from_f32(normal.sample(&mut rng))); - } - let elements = PrimitiveArray::new::(buf.freeze(), Validity::NonNullable); - let fsl = FixedSizeListArray::try_new( - elements.into_array(), - dim.try_into() - .expect("somehow got dimension greater than u32::MAX"), - Validity::NonNullable, - num_rows, - )?; - - let ext = make_vector_ext(&fsl); - let config = TurboQuantConfig { - bit_width: 3, - seed: Some(42), - num_rounds: 3, - }; - - let mut ctx = SESSION.create_execution_ctx(); - let normalized = normalized_child(&ext, &mut ctx)?; - let normalized_ext = normalized - .as_opt::() - .vortex_expect("normalized child should be an Extension array"); - - let encoded = turboquant_encode(normalized_ext, &config, &mut ctx)?; - assert_eq!(encoded.len(), num_rows); - Ok(()) -} - -fn make_fsl_small(dim: usize) -> FixedSizeListArray { - let mut buf = BufferMut::::with_capacity(dim); - for i in 0..dim { - buf.push(i as f32 + 1.0); - } - let elements = PrimitiveArray::new::(buf.freeze(), Validity::NonNullable); - FixedSizeListArray::try_new( - elements.into_array(), - dim.try_into() - .expect("somehow got dimension greater than u32::MAX"), - Validity::NonNullable, - 1, - ) - .unwrap() -} - -/// Verify that all-zero vectors roundtrip correctly (norm == 0 branch). -#[test] -fn all_zero_vectors_roundtrip() -> VortexResult<()> { - let num_rows = 10; - let dim = 128; - let buf = BufferMut::::full(0.0f32, num_rows * dim); - let elements = PrimitiveArray::new::(buf.freeze(), Validity::NonNullable); - let fsl = FixedSizeListArray::try_new( - elements.into_array(), - dim.try_into() - .expect("somehow got dimension greater than u32::MAX"), - Validity::NonNullable, - num_rows, - )?; - - let config = TurboQuantConfig { - bit_width: 3, - seed: Some(42), - num_rounds: 3, - }; - let (original, decoded) = encode_decode(&fsl, &config)?; - // All-zero vectors should decode to all-zero (norm=0 -> 0 * anything = 0). - for (i, (&o, &d)) in original.iter().zip(decoded.iter()).enumerate() { - assert_eq!(o, 0.0, "original[{i}] not zero"); - assert_eq!(d, 0.0, "decoded[{i}] not zero for all-zero input"); - } - Ok(()) -} - -/// Verify that f64 input is accepted and encoded (converted to f32 internally). -#[test] -fn f64_input_encodes_successfully() -> VortexResult<()> { - let num_rows = 10; - let dim = 128; - let mut rng = StdRng::seed_from_u64(99); - let normal = Normal::new(0.0f64, 1.0).unwrap(); - - let mut buf = BufferMut::::with_capacity(num_rows * dim); - for _ in 0..(num_rows * dim) { - buf.push(normal.sample(&mut rng)); - } - let elements = PrimitiveArray::new::(buf.freeze(), Validity::NonNullable); - let fsl = FixedSizeListArray::try_new( - elements.into_array(), - dim.try_into() - .expect("somehow got dimension greater than u32::MAX"), - Validity::NonNullable, - num_rows, - )?; - - let ext = make_vector_ext(&fsl); - let config = TurboQuantConfig { - bit_width: 3, - seed: Some(42), - num_rounds: 3, - }; - // Verify encoding succeeds with f64 input (f64->f32 conversion). - let mut ctx = SESSION.create_execution_ctx(); - let encoded = normalize_and_encode(&ext, &config, &mut ctx)?; - let (tq_child, norms_child) = unwrap_l2denorm(&encoded); - let tq = tq_child.as_opt::().unwrap(); - assert_eq!(norms_child.len(), num_rows); - assert_eq!(tq.dimension() as usize, dim); - Ok(()) -} - -/// Verify that f16 input is accepted and encoded (upcast to f32 internally). -#[test] -fn f16_input_encodes_successfully() -> VortexResult<()> { - let num_rows = 10; - let dim = 128; - let mut rng = StdRng::seed_from_u64(99); - let normal = Normal::new(0.0f32, 1.0).unwrap(); - - let mut buf = BufferMut::::with_capacity(num_rows * dim); - for _ in 0..(num_rows * dim) { - buf.push(half::f16::from_f32(normal.sample(&mut rng))); - } - let elements = PrimitiveArray::new::(buf.freeze(), Validity::NonNullable); - let fsl = FixedSizeListArray::try_new( - elements.into_array(), - dim.try_into() - .expect("somehow got dimension greater than u32::MAX"), - Validity::NonNullable, - num_rows, - )?; - - let ext = make_vector_ext(&fsl); - let config = TurboQuantConfig { - bit_width: 3, - seed: Some(42), - num_rounds: 3, - }; - let mut ctx = SESSION.create_execution_ctx(); - let encoded = normalize_and_encode(&ext, &config, &mut ctx)?; - let (tq_child, norms_child) = unwrap_l2denorm(&encoded); - let tq = tq_child.as_opt::().unwrap(); - assert_eq!(norms_child.len(), num_rows); - assert_eq!(tq.dimension() as usize, dim); - - // Verify roundtrip: decode and check reconstruction is reasonable. - let decoded_ext = encoded.execute::(&mut ctx)?; - let decoded_fsl = decoded_ext - .storage_array() - .clone() - .execute::(&mut ctx)?; - assert_eq!(decoded_fsl.len(), num_rows); - Ok(()) -} - -// ----------------------------------------------------------------------- -// Verification tests for stored metadata -// ----------------------------------------------------------------------- - -/// Verify that the centroids stored in the array match what `get_centroids()` computes. -#[test] -fn stored_centroids_match_computed() -> VortexResult<()> { - let fsl = make_fsl(10, 128, 42); - let ext = make_vector_ext(&fsl); - let config = TurboQuantConfig { - bit_width: 3, - seed: Some(123), - num_rounds: 3, - }; - let mut ctx = SESSION.create_execution_ctx(); - let encoded = normalize_and_encode(&ext, &config, &mut ctx)?; - let (tq_child, _norms) = unwrap_l2denorm(&encoded); - let tq = tq_child.as_opt::().unwrap(); - - let mut ctx = SESSION.create_execution_ctx(); - let stored_centroids_prim = tq.centroids().clone().execute::(&mut ctx)?; - let stored = stored_centroids_prim.as_slice::(); - - let padded_dim = tq.padded_dim(); - let computed = crate::encodings::turboquant::array::centroids::get_centroids(padded_dim, 3)?; - - assert_eq!(stored.len(), computed.len()); - for i in 0..stored.len() { - assert_eq!(stored[i], computed[i], "Centroid mismatch at {i}"); - } - Ok(()) -} - -/// Verify that stored rotation signs produce identical decode to seed-based decode. -#[test] -fn stored_rotation_signs_produce_correct_decode() -> VortexResult<()> { - let fsl = make_fsl(20, 128, 42); - let ext = make_vector_ext(&fsl); - let config = TurboQuantConfig { - bit_width: 3, - seed: Some(123), - num_rounds: 4, - }; - let mut ctx = SESSION.create_execution_ctx(); - let encoded = normalize_and_encode(&ext, &config, &mut ctx)?; - let (tq_child, _norms) = unwrap_l2denorm(&encoded); - let tq = tq_child.as_opt::().unwrap(); - - // Decode via the full L2Denorm path (TQ decompress + norm scaling). - let mut ctx = SESSION.create_execution_ctx(); - let decoded_ext = encoded.execute::(&mut ctx)?; - let decoded_fsl = decoded_ext - .storage_array() - .clone() - .execute::(&mut ctx)?; - let decoded = decoded_fsl - .elements() - .clone() - .execute::(&mut ctx)?; - let decoded_slice = decoded.as_slice::(); - - // Verify stored signs match seed-derived signs. - let rot_from_seed = RotationMatrix::try_new(123, 128, 4)?; - let expected_u8 = rot_from_seed.export_inverse_signs_u8(); - let stored_signs_fsl = tq - .rotation_signs() - .clone() - .execute::(&mut ctx)?; - let stored_signs = stored_signs_fsl - .elements() - .clone() - .execute::(&mut ctx)?; - let stored_u8 = stored_signs.as_slice::(); - - assert_eq!(expected_u8.len(), stored_u8.len()); - for i in 0..expected_u8.len() { - assert_eq!(expected_u8[i], stored_u8[i], "Sign mismatch at index {i}"); - } - - // Also verify decode output is non-empty and has expected size. - assert_eq!(decoded_slice.len(), 20 * 128); - Ok(()) -} - -// ----------------------------------------------------------------------- -// Compute pushdown tests -// ----------------------------------------------------------------------- - -#[test] -fn slice_preserves_data() -> VortexResult<()> { - let fsl = make_fsl(20, 128, 42); - let ext = make_vector_ext(&fsl); - let config = TurboQuantConfig { - bit_width: 3, - seed: Some(123), - num_rounds: 4, - }; - let mut ctx = SESSION.create_execution_ctx(); - let encoded = normalize_and_encode(&ext, &config, &mut ctx)?; - - // Full decompress then slice. - let mut ctx = SESSION.create_execution_ctx(); - let full_decoded = encoded.clone().execute::(&mut ctx)?; - let full_fsl = full_decoded - .storage_array() - .clone() - .execute::(&mut ctx)?; - let expected = full_fsl.slice(5..10)?; - let expected_fsl = expected.execute::(&mut ctx)?; - let expected_elements = expected_fsl - .elements() - .clone() - .execute::(&mut ctx)?; - - // Slice then decompress. - let sliced = encoded.slice(5..10)?; - let sliced_decoded = sliced.execute::(&mut ctx)?; - let sliced_fsl = sliced_decoded - .storage_array() - .clone() - .execute::(&mut ctx)?; - let actual_elements = sliced_fsl - .elements() - .clone() - .execute::(&mut ctx)?; - - assert_eq!( - expected_elements.as_slice::(), - actual_elements.as_slice::() - ); - Ok(()) -} - -#[test] -fn scalar_at_matches_decompress() -> VortexResult<()> { - let fsl = make_fsl(10, 128, 42); - let ext = make_vector_ext(&fsl); - let config = TurboQuantConfig { - bit_width: 3, - seed: Some(123), - num_rounds: 2, - }; - let mut ctx = SESSION.create_execution_ctx(); - let encoded = normalize_and_encode(&ext, &config, &mut ctx)?; - - let full_decoded = encoded.clone().execute::(&mut ctx)?; - - for i in [0, 1, 5, 9] { - let expected = full_decoded.scalar_at(i)?; - let actual = encoded.scalar_at(i)?; - assert_eq!(expected, actual, "scalar_at mismatch at index {i}"); - } - Ok(()) -} - -#[test] -fn l2_norm_readthrough() -> VortexResult<()> { - let fsl = make_fsl(10, 128, 42); - let ext = make_vector_ext(&fsl); - let config = TurboQuantConfig { - bit_width: 3, - seed: Some(123), - num_rounds: 5, - }; - let mut ctx = SESSION.create_execution_ctx(); - let encoded = normalize_and_encode(&ext, &config, &mut ctx)?; - let (_tq_child, norms_child) = unwrap_l2denorm(&encoded); - - // Stored norms should match the actual L2 norms of the input. - let norms_prim = norms_child.execute::(&mut ctx)?; - let stored_norms = norms_prim.as_slice::(); - - let input_prim = fsl.elements().clone().execute::(&mut ctx)?; - let input_f32 = input_prim.as_slice::(); - for row in 0..10 { - let vec = &input_f32[row * 128..(row + 1) * 128]; - let actual_norm: f32 = vec.iter().map(|&v| v * v).sum::().sqrt(); - assert!( - (stored_norms[row] - actual_norm).abs() < 1e-5, - "norm mismatch at row {row}: stored={}, actual={}", - stored_norms[row], - actual_norm - ); - } - Ok(()) -} - -#[test] -fn cosine_similarity_quantized_accuracy() -> VortexResult<()> { - let fsl = make_fsl(20, 128, 42); - let ext = make_vector_ext(&fsl); - let config = TurboQuantConfig { - bit_width: 4, - seed: Some(123), - num_rounds: 3, - }; - let mut ctx = SESSION.create_execution_ctx(); - let encoded = normalize_and_encode(&ext, &config, &mut ctx)?; - let (tq_child, norms_child) = unwrap_l2denorm(&encoded); - let tq = tq_child.as_opt::().unwrap(); - - // Compute exact cosine similarity from original data. - let input_prim = fsl.elements().clone().execute::(&mut ctx)?; - let input_f32 = input_prim.as_slice::(); - - // Read quantized codes, norms, and centroids for approximate computation. - let mut ctx = SESSION.create_execution_ctx(); - let pd = tq.padded_dim() as usize; - let norms_prim = norms_child.execute::(&mut ctx)?; - let norms = norms_prim.as_slice::(); - let codes_fsl = tq.codes().clone().execute::(&mut ctx)?; - let codes_prim = codes_fsl - .elements() - .clone() - .execute::(&mut ctx)?; - let all_codes = codes_prim.as_slice::(); - let centroids_prim = tq.centroids().clone().execute::(&mut ctx)?; - let centroid_vals = centroids_prim.as_slice::(); - - for (row_a, row_b) in [(0, 1), (5, 10), (0, 19)] { - let vec_a = &input_f32[row_a * 128..(row_a + 1) * 128]; - let vec_b = &input_f32[row_b * 128..(row_b + 1) * 128]; - - let dot: f32 = vec_a.iter().zip(vec_b.iter()).map(|(&x, &y)| x * y).sum(); - let norm_a: f32 = vec_a.iter().map(|&v| v * v).sum::().sqrt(); - let norm_b: f32 = vec_b.iter().map(|&v| v * v).sum::().sqrt(); - let exact_cos = dot / (norm_a * norm_b); - - // Approximate cosine similarity in quantized domain. - let approx_cos = if norms[row_a] == 0.0 || norms[row_b] == 0.0 { - 0.0 - } else { - let codes_a = &all_codes[row_a * pd..(row_a + 1) * pd]; - let codes_b = &all_codes[row_b * pd..(row_b + 1) * pd]; - codes_a - .iter() - .zip(codes_b.iter()) - .map(|(&ca, &cb)| centroid_vals[ca as usize] * centroid_vals[cb as usize]) - .sum::() - }; - - // At 4-bit, the theoretical MSE bound per coordinate is ~0.0106 (Theorem 1). For cosine - // similarity (bounded [-1, 1]), the error is bounded roughly by 2*sqrt(MSE) ~ 0.2. We use - // 0.15 as a tighter empirical bound. - let error = (exact_cos - approx_cos).abs(); - assert!( - error < 0.15, - "cosine similarity error too large for ({row_a}, {row_b}): \ - exact={exact_cos:.4}, approx={approx_cos:.4}, error={error:.4}" - ); - } - Ok(()) -} - -/// Verify approximate dot product in the quantized domain. -/// -/// NOTE: The MSE quantizer (TurboQuant_mse) has inherent **multiplicative bias** for inner -/// products — the quantized dot product systematically over- or under-estimates the true value. -/// This is a fundamental property: the paper's `TurboQuant_prod` variant adds QJL specifically -/// to debias inner products, but we only implement the MSE-only variant. -/// -/// Even at 8-bit (near-lossless reconstruction, MSE ~4e-5), the quantized-domain dot product -/// can have ~10-15% relative error due to this bias. This tolerance is therefore intentionally -/// loose — we're testing that the approximation is in the right ballpark, not that it's precise. -/// -/// TODO(connor): Revisit these tolerances when we have TurboQuant_prod (QJL debiasing). -#[test] -fn dot_product_quantized_accuracy() -> VortexResult<()> { - let fsl = make_fsl(20, 128, 42); - let ext = make_vector_ext(&fsl); - let config = TurboQuantConfig { - bit_width: 8, - seed: Some(123), - num_rounds: 3, - }; - let mut ctx = SESSION.create_execution_ctx(); - let encoded = normalize_and_encode(&ext, &config, &mut ctx)?; - let (tq_child, norms_child) = unwrap_l2denorm(&encoded); - let tq = tq_child.as_opt::().unwrap(); - - let input_prim = fsl.elements().clone().execute::(&mut ctx)?; - let input_f32 = input_prim.as_slice::(); - - let mut ctx = SESSION.create_execution_ctx(); - let pd = tq.padded_dim() as usize; - let norms_prim = norms_child.execute::(&mut ctx)?; - let norms = norms_prim.as_slice::(); - let codes_fsl = tq.codes().clone().execute::(&mut ctx)?; - let codes_prim = codes_fsl - .elements() - .clone() - .execute::(&mut ctx)?; - let all_codes = codes_prim.as_slice::(); - let centroids_prim = tq.centroids().clone().execute::(&mut ctx)?; - let centroid_vals = centroids_prim.as_slice::(); - - for (row_a, row_b) in [(0, 1), (5, 10), (0, 19)] { - let vec_a = &input_f32[row_a * 128..(row_a + 1) * 128]; - let vec_b = &input_f32[row_b * 128..(row_b + 1) * 128]; - - let exact_dot: f32 = vec_a.iter().zip(vec_b.iter()).map(|(&x, &y)| x * y).sum(); - - let codes_a = &all_codes[row_a * pd..(row_a + 1) * pd]; - let codes_b = &all_codes[row_b * pd..(row_b + 1) * pd]; - let unit_dot: f32 = codes_a - .iter() - .zip(codes_b.iter()) - .map(|(&ca, &cb)| centroid_vals[ca as usize] * centroid_vals[cb as usize]) - .sum(); - let approx_dot = norms[row_a] * norms[row_b] * unit_dot; - - // See doc comment above: 15% relative error is expected due to MSE quantizer bias. - let scale = exact_dot.abs().max(1.0); - let rel_error = (exact_dot - approx_dot).abs() / scale; - assert!( - rel_error < 0.15, - "dot product error too large for ({row_a}, {row_b}): \ - exact={exact_dot:.4}, approx={approx_dot:.4}, rel_error={rel_error:.4}" - ); - } - Ok(()) -} - -/// Roundtrip at large embedding dimensions to validate padding and SRHT at common sizes. -/// -/// NOTE: The theoretical MSE bound (Theorem 1) is proved for Haar-distributed random orthogonal -/// matrices, not SRHT. The SRHT is a practical O(d log d) approximation that doesn't exactly -/// satisfy the Haar assumption, so empirical MSE can slightly exceed the theoretical bound. We -/// use a 2x multiplier to account for this gap. -/// -/// The 1024-d case uses 5-bit instead of 4-bit because at 4-bit the SRHT approximation error -/// at d=1024 pushes MSE ~20% above the 1x theoretical bound (0.0127 vs bound 0.0106). -/// -/// TODO(connor): Revisit after Stage 2 block decomposition — at d=768 with block_size=256, -/// the per-block SRHT will be lower-dimensional and may have different error characteristics. -#[rstest] -#[case(768, 4)] -#[case(1024, 5)] -fn large_dimension_roundtrip(#[case] dim: usize, #[case] bit_width: u8) -> VortexResult<()> { - let num_rows = 10; - let fsl = make_fsl(num_rows, dim, 42); - let config = TurboQuantConfig { - bit_width, - seed: Some(123), - num_rounds: 3, - }; - let (original, decoded) = encode_decode(&fsl, &config)?; - assert_eq!(decoded.len(), original.len()); - - let normalized_mse = per_vector_normalized_mse(&original, &decoded, dim, num_rows); - // 2x slack for the SRHT-vs-Haar gap (see doc comment above). - let bound = 2.0 * theoretical_mse_bound(bit_width); - assert!( - normalized_mse < bound, - "Normalized MSE {normalized_mse:.6} exceeds 2x bound {bound:.6} for dim={dim}, bits={bit_width}", - ); - Ok(()) -} - -/// Verify that the encoded array's dtype is a Vector extension type. -#[test] -fn encoded_dtype_is_vector_extension() -> VortexResult<()> { - let fsl = make_fsl(10, 128, 42); - let ext = make_vector_ext(&fsl); - let config = TurboQuantConfig { - bit_width: 3, - seed: Some(123), - num_rounds: 2, - }; - let mut ctx = SESSION.create_execution_ctx(); - let encoded = normalize_and_encode(&ext, &config, &mut ctx)?; - - // The encoded TurboQuant array should claim a Vector extension dtype. - assert!( - encoded.dtype().is_extension(), - "TurboQuant dtype should be an extension type, got {}", - encoded.dtype() - ); - assert!( - encoded.dtype().as_extension().is::(), - "TurboQuant dtype should be a Vector extension type" - ); - Ok(()) -} - -// ----------------------------------------------------------------------- -// Nullable vector tests -// ----------------------------------------------------------------------- - -/// Encode a nullable Vector array and verify roundtrip preserves validity and non-null values. -#[test] -fn nullable_vectors_roundtrip() -> VortexResult<()> { - // Rows 2, 5, 7 are null. - let validity = Validity::from_iter([ - true, true, false, true, true, false, true, false, true, true, - ]); - let fsl = make_fsl_with_validity(10, 128, 42, validity); - let ext = make_vector_ext(&fsl); - - let config = TurboQuantConfig { - bit_width: 3, - seed: Some(123), - num_rounds: 4, - }; - let mut ctx = SESSION.create_execution_ctx(); - let encoded = normalize_and_encode(&ext, &config, &mut ctx)?; - - assert_eq!(encoded.len(), 10); - assert!(encoded.dtype().is_nullable()); - - // Check validity of the encoded array. - let encoded_validity = encoded.validity()?; - for i in 0..10 { - let expected = ![2, 5, 7].contains(&i); - assert_eq!( - encoded_validity.is_valid(i)?, - expected, - "validity mismatch at row {i}" - ); - } - - // Decode and verify non-null rows have correct data. - let decoded_ext = encoded.execute::(&mut ctx)?; - assert_eq!(decoded_ext.len(), 10); - - let decoded_fsl = decoded_ext - .storage_array() - .clone() - .execute::(&mut ctx)?; - let decoded_prim = decoded_fsl - .elements() - .clone() - .execute::(&mut ctx)?; - let decoded_f32 = decoded_prim.as_slice::(); - - // Original f32 elements for non-null row comparison. - let orig_prim = fsl.elements().clone().execute::(&mut ctx)?; - let orig_f32 = orig_prim.as_slice::(); - - // Non-null rows should have reasonable reconstruction (within MSE bounds). - for row in [0, 1, 3, 4, 6, 8, 9] { - let orig_vec = &orig_f32[row * 128..(row + 1) * 128]; - let dec_vec = &decoded_f32[row * 128..(row + 1) * 128]; - let norm_sq: f32 = orig_vec.iter().map(|&v| v * v).sum(); - let err_sq: f32 = orig_vec - .iter() - .zip(dec_vec.iter()) - .map(|(&a, &b)| (a - b) * (a - b)) - .sum(); - // 3-bit normalized MSE should be well under the theoretical bound. - assert!( - err_sq / norm_sq < 0.1, - "non-null row {row} has excessive reconstruction error" - ); - } - Ok(()) -} - -/// Verify that norms carry the validity: null vectors have null norms. -#[test] -fn nullable_norms_match_validity() -> VortexResult<()> { - let validity = Validity::from_iter([true, false, true, false, true]); - let fsl = make_fsl_with_validity(5, 128, 42, validity); - let ext = make_vector_ext(&fsl); - - let config = TurboQuantConfig { - bit_width: 2, - seed: Some(123), - num_rounds: 3, - }; - let mut ctx = SESSION.create_execution_ctx(); - let encoded = normalize_and_encode(&ext, &config, &mut ctx)?; - let (_tq_child, norms_child) = unwrap_l2denorm(&encoded); - - let norms_validity = norms_child.validity()?; - for i in 0..5 { - let expected = i % 2 == 0; // rows 0, 2, 4 are valid - assert_eq!( - norms_validity.is_valid(i)?, - expected, - "norms validity mismatch at row {i}" - ); - } - Ok(()) -} - -/// Verify that L2Norm readthrough works correctly on nullable TurboQuant arrays. -#[test] -fn nullable_l2_norm_readthrough() -> VortexResult<()> { - let validity = Validity::from_iter([true, false, true, false, true]); - let fsl = make_fsl_with_validity(5, 128, 42, validity); - let ext = make_vector_ext(&fsl); - - let config = TurboQuantConfig { - bit_width: 3, - seed: Some(123), - num_rounds: 3, - }; - let mut ctx = SESSION.create_execution_ctx(); - let encoded = normalize_and_encode(&ext, &config, &mut ctx)?; - - // Compute L2Norm on the encoded array. - let norm_sfn = L2Norm::try_new_array(&ApproxOptions::Exact, encoded, 5)?; - let norms: PrimitiveArray = norm_sfn.into_array().execute(&mut ctx)?; - - // Null rows should have null norms, valid rows should have correct norms. - let orig_prim = fsl.elements().clone().execute::(&mut ctx)?; - let orig_f32 = orig_prim.as_slice::(); - for row in 0..5 { - if row % 2 == 0 { - assert!(norms.is_valid(row)?, "row {row} should be valid"); - let expected: f32 = orig_f32[row * 128..(row + 1) * 128] - .iter() - .map(|&v| v * v) - .sum::() - .sqrt(); - let actual = norms.as_slice::()[row]; - assert!( - (actual - expected).abs() < 1e-5, - "norm mismatch at valid row {row}: actual={actual}, expected={expected}" - ); - } else { - assert!(!norms.is_valid(row)?, "row {row} should be null"); - } - } - Ok(()) -} - -/// Verify that slicing a nullable TurboQuant array preserves validity. -#[test] -fn nullable_slice_preserves_validity() -> VortexResult<()> { - // Rows 2, 5, 7 are null. - let validity = Validity::from_iter([ - true, true, false, true, true, false, true, false, true, true, - ]); - let fsl = make_fsl_with_validity(10, 128, 42, validity); - let ext = make_vector_ext(&fsl); - - let config = TurboQuantConfig { - bit_width: 3, - seed: Some(123), - num_rounds: 2, - }; - let mut ctx = SESSION.create_execution_ctx(); - let encoded = normalize_and_encode(&ext, &config, &mut ctx)?; - - // Slice rows 1..6 -> [true, false, true, true, false]. - let sliced = encoded.slice(1..6)?; - assert_eq!(sliced.len(), 5); - - let sliced_validity = sliced.validity()?; - let expected = [true, false, true, true, false]; - for (i, &exp) in expected.iter().enumerate() { - assert_eq!( - sliced_validity.is_valid(i)?, - exp, - "sliced validity mismatch at index {i}" - ); - } - Ok(()) -} - -// ----------------------------------------------------------------------- -// Serde roundtrip tests -// ----------------------------------------------------------------------- - -/// Verify that a TurboQuant array (extracted from the L2Denorm wrapper) survives -/// serialize/deserialize. -/// -/// TODO(connor): ScalarFnArray cannot be serialized yet, so we test the TQ child directly. -#[test] -fn serde_roundtrip() -> VortexResult<()> { - use vortex_array::ArrayContext; - use vortex_array::ArrayEq; - use vortex_array::Precision; - use vortex_array::serde::SerializeOptions; - use vortex_array::serde::SerializedArray; - use vortex_array::session::ArraySessionExt; - use vortex_buffer::ByteBufferMut; - use vortex_fastlanes::BitPacked; - use vortex_session::registry::ReadContext; - - let fsl = make_fsl(20, 128, 42); - let ext = make_vector_ext(&fsl); - let config = TurboQuantConfig { - bit_width: 3, - seed: Some(123), - num_rounds: 5, - }; - let mut ctx = SESSION.create_execution_ctx(); - let encoded = normalize_and_encode(&ext, &config, &mut ctx)?; - let (tq_child, _norms) = unwrap_l2denorm(&encoded); - - let dtype = tq_child.dtype().clone(); - let len = tq_child.len(); - - // Serialize the TQ child. - let array_ctx = ArrayContext::empty(); - let serde_session = VortexSession::empty().with::(); - serde_session.arrays().register(TurboQuant); - let serialized = - tq_child.serialize(&array_ctx, &serde_session, &SerializeOptions::default())?; - - let mut concat = ByteBufferMut::empty(); - for buf in serialized { - concat.extend_from_slice(buf.as_ref()); - } - - // Deserialize. The session needs TurboQuant and BitPacked (for rotation signs) registered. - serde_session.arrays().register(BitPacked); - - let parts = SerializedArray::try_from(concat.freeze())?; - let decoded = parts.decode( - &dtype, - len, - &ReadContext::new(array_ctx.to_ids()), - &serde_session, - )?; - - assert!( - decoded.array_eq(&tq_child, Precision::Value), - "serde roundtrip did not preserve array equality" - ); - Ok(()) -} - -/// Verify that a degenerate (empty) TurboQuant array survives serialize/deserialize. -#[test] -fn serde_roundtrip_empty() -> VortexResult<()> { - use vortex_array::ArrayContext; - use vortex_array::ArrayEq; - use vortex_array::Precision; - use vortex_array::serde::SerializeOptions; - use vortex_array::serde::SerializedArray; - use vortex_array::session::ArraySessionExt; - use vortex_buffer::ByteBufferMut; - use vortex_fastlanes::BitPacked; - use vortex_session::registry::ReadContext; - - let fsl = make_fsl(0, 128, 42); - let ext = make_vector_ext(&fsl); - let config = TurboQuantConfig { - bit_width: 2, - seed: Some(123), - num_rounds: 3, - }; - let mut ctx = SESSION.create_execution_ctx(); - let encoded = normalize_and_encode(&ext, &config, &mut ctx)?; - assert_eq!(encoded.len(), 0); - let (tq_child, _norms) = unwrap_l2denorm(&encoded); - - let dtype = tq_child.dtype().clone(); - let len = tq_child.len(); - - let serde_session = VortexSession::empty().with::(); - serde_session.arrays().register(TurboQuant); - serde_session.arrays().register(BitPacked); - - let array_ctx = ArrayContext::empty(); - let serialized = - tq_child.serialize(&array_ctx, &serde_session, &SerializeOptions::default())?; - - let mut concat = ByteBufferMut::empty(); - for buf in serialized { - concat.extend_from_slice(buf.as_ref()); - } - - let parts = SerializedArray::try_from(concat.freeze())?; - let decoded = parts.decode( - &dtype, - len, - &ReadContext::new(array_ctx.to_ids()), - &serde_session, - )?; - - assert!( - decoded.array_eq(&tq_child, Precision::Value), - "serde roundtrip did not preserve array equality" - ); - Ok(()) -} diff --git a/vortex-tensor/src/encodings/turboquant/tests/compute.rs b/vortex-tensor/src/encodings/turboquant/tests/compute.rs new file mode 100644 index 00000000000..4d670695eaf --- /dev/null +++ b/vortex-tensor/src/encodings/turboquant/tests/compute.rs @@ -0,0 +1,216 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +use vortex_array::ArrayRef; +use vortex_array::IntoArray; +use vortex_array::LEGACY_SESSION; +use vortex_array::VortexSessionExecute; +use vortex_array::arrays::ExtensionArray; +use vortex_array::arrays::FixedSizeListArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::fixed_size_list::FixedSizeListArrayExt; +use vortex_error::VortexResult; + +use super::*; +use crate::scalar_fns::cosine_similarity::CosineSimilarity; +use crate::scalar_fns::l2_norm::L2Norm; + +fn execute_l2_norm( + input: ArrayRef, + len: usize, + ctx: &mut vortex_array::ExecutionCtx, +) -> VortexResult { + L2Norm::try_new_array(input, len)?.into_array().execute(ctx) +} + +fn execute_cosine_similarity( + lhs: ArrayRef, + rhs: ArrayRef, + len: usize, + ctx: &mut vortex_array::ExecutionCtx, +) -> VortexResult { + CosineSimilarity::try_new_array(lhs, rhs, len)? + .into_array() + .execute(ctx) +} + +#[test] +fn slice_preserves_data() -> VortexResult<()> { + let fsl = make_fsl(20, 128, 42); + let ext = make_vector_ext(&fsl); + let config = TurboQuantConfig { + bit_width: 3, + seed: 123, + num_rounds: 4, + }; + let mut ctx = SESSION.create_execution_ctx(); + let encoded = turboquant_encode(ext, &config, &mut ctx)?; + + // Full decompress then slice. + let mut ctx = SESSION.create_execution_ctx(); + let full_decoded = encoded.clone().execute::(&mut ctx)?; + let full_fsl = full_decoded + .storage_array() + .clone() + .execute::(&mut ctx)?; + let expected = full_fsl.slice(5..10)?; + let expected_fsl = expected.execute::(&mut ctx)?; + let expected_elements = expected_fsl + .elements() + .clone() + .execute::(&mut ctx)?; + + // Slice then decompress. + let sliced = encoded.slice(5..10)?; + let sliced_decoded = sliced.execute::(&mut ctx)?; + let sliced_fsl = sliced_decoded + .storage_array() + .clone() + .execute::(&mut ctx)?; + let actual_elements = sliced_fsl + .elements() + .clone() + .execute::(&mut ctx)?; + + assert_eq!( + expected_elements.as_slice::(), + actual_elements.as_slice::() + ); + Ok(()) +} + +#[test] +fn scalar_at_matches_decompress() -> VortexResult<()> { + let fsl = make_fsl(10, 128, 42); + let ext = make_vector_ext(&fsl); + let config = TurboQuantConfig { + bit_width: 3, + seed: 123, + num_rounds: 2, + }; + let mut ctx = SESSION.create_execution_ctx(); + let encoded = turboquant_encode(ext, &config, &mut ctx)?; + + let full_decoded = encoded.clone().execute::(&mut ctx)?; + + for i in [0, 1, 5, 9] { + let expected = + full_decoded.execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx())?; + let actual = encoded.execute_scalar(i, &mut LEGACY_SESSION.create_execution_ctx())?; + assert_eq!(expected, actual, "scalar_at mismatch at index {i}"); + } + Ok(()) +} + +#[test] +fn l2_norm_readthrough() -> VortexResult<()> { + let fsl = make_fsl(10, 128, 42); + let ext = make_vector_ext(&fsl); + let config = TurboQuantConfig { + bit_width: 3, + seed: 123, + num_rounds: 5, + }; + let mut ctx = SESSION.create_execution_ctx(); + let encoded = turboquant_encode(ext, &config, &mut ctx)?; + let (_sorf_child, norms_child) = unwrap_l2denorm(&encoded); + + // Stored norms should match the actual L2 norms of the input. + let norms_prim = norms_child.execute::(&mut ctx)?; + let stored_norms = norms_prim.as_slice::(); + + let input_prim = fsl.elements().clone().execute::(&mut ctx)?; + let input_f32 = input_prim.as_slice::(); + for row in 0..10 { + let vec = &input_f32[row * 128..(row + 1) * 128]; + let actual_norm: f32 = vec.iter().map(|&v| v * v).sum::().sqrt(); + assert!( + (stored_norms[row] - actual_norm).abs() < 1e-5, + "norm mismatch at row {row}: stored={}, actual={}", + stored_norms[row], + actual_norm + ); + } + + // Also verify L2Norm readthrough shortcut works. + let norms = execute_l2_norm(encoded, 10, &mut ctx)?; + assert_eq!(norms.as_slice::(), stored_norms); + assert_eq!(norms.len(), 10); + Ok(()) +} + +#[test] +fn l2_norm_readthrough_is_authoritative_for_lossy_storage() -> VortexResult<()> { + let num_rows = 12; + let fsl = make_fsl(num_rows, 128, 7); + let ext = make_vector_ext(&fsl); + let config = TurboQuantConfig { + bit_width: 1, + seed: 123, + num_rounds: 3, + }; + let mut ctx = SESSION.create_execution_ctx(); + let encoded = turboquant_encode(ext, &config, &mut ctx)?; + let (_sorf_child, norms_child) = unwrap_l2denorm(&encoded); + + let stored_norms: PrimitiveArray = norms_child.execute(&mut ctx)?; + let encoded_norms = execute_l2_norm(encoded.clone(), num_rows, &mut ctx)?; + assert_eq!( + encoded_norms.as_slice::(), + stored_norms.as_slice::() + ); + + let decoded = encoded.execute::(&mut ctx)?.into_array(); + let decoded_norms = execute_l2_norm(decoded, num_rows, &mut ctx)?; + let max_gap = stored_norms + .as_slice::() + .iter() + .zip(decoded_norms.as_slice::().iter()) + .map(|(&stored, &decoded)| (stored - decoded).abs()) + .fold(0.0f32, f32::max); + + assert!( + max_gap > 1e-3, + "expected at least one decoded norm to drift from the authoritative stored norms, got max gap {max_gap:.6}", + ); + Ok(()) +} + +#[test] +fn cosine_similarity_readthrough_is_authoritative_for_lossy_storage() -> VortexResult<()> { + let num_rows = 12; + let fsl = make_fsl(num_rows, 128, 11); + let ext = make_vector_ext(&fsl); + let config = TurboQuantConfig { + bit_width: 1, + seed: 123, + num_rounds: 3, + }; + let mut ctx = SESSION.create_execution_ctx(); + let encoded = turboquant_encode(ext, &config, &mut ctx)?; + + let encoded_cos = + execute_cosine_similarity(encoded.clone(), encoded.clone(), num_rows, &mut ctx)?; + let decoded = encoded.execute::(&mut ctx)?.into_array(); + let decoded_cos = execute_cosine_similarity(decoded.clone(), decoded, num_rows, &mut ctx)?; + + let decoded_values = decoded_cos.as_slice::(); + assert!( + decoded_values + .iter() + .all(|&value| (value - 1.0).abs() < 1e-5), + "decoded cosine(x, x) should stay at 1.0", + ); + + let max_gap = encoded_cos + .as_slice::() + .iter() + .zip(decoded_values.iter()) + .map(|(&encoded, &decoded)| (encoded - decoded).abs()) + .fold(0.0f32, f32::max); + assert!( + max_gap > 1e-3, + "expected encoded cosine readthrough to differ from decoded recomputation, got max gap {max_gap:.6}", + ); + Ok(()) +} diff --git a/vortex-tensor/src/encodings/turboquant/tests/mod.rs b/vortex-tensor/src/encodings/turboquant/tests/mod.rs new file mode 100644 index 00000000000..ec4182dcc3d --- /dev/null +++ b/vortex-tensor/src/encodings/turboquant/tests/mod.rs @@ -0,0 +1,161 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Tests for TurboQuant encoding with decomposed SorfTransform + DictArray tree. + +mod compute; +mod nullable; +mod roundtrip; +mod structural; + +use rand::SeedableRng; +use rand::rngs::StdRng; +use rand_distr::Distribution; +use rand_distr::Normal; +use vortex_array::ArrayRef; +use vortex_array::IntoArray; +use vortex_array::VortexSessionExecute; +use vortex_array::arrays::Dict; +use vortex_array::arrays::ExtensionArray; +use vortex_array::arrays::FixedSizeListArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::ScalarFn; +use vortex_array::arrays::dict::DictArraySlotsExt; +use vortex_array::arrays::extension::ExtensionArrayExt; +use vortex_array::arrays::fixed_size_list::FixedSizeListArrayExt; +use vortex_array::arrays::scalar_fn::ScalarFnArrayExt; +use vortex_array::validity::Validity; +use vortex_buffer::BufferMut; +use vortex_error::VortexExpect; +use vortex_error::VortexResult; + +use crate::encodings::turboquant::TurboQuantConfig; +use crate::encodings::turboquant::turboquant_encode; +use crate::tests::SESSION; +use crate::types::vector::Vector; + +/// Create a FixedSizeListArray of random f32 vectors with the given validity. +fn make_fsl_with_validity( + num_rows: usize, + dim: usize, + seed: u64, + validity: Validity, +) -> FixedSizeListArray { + let mut rng = StdRng::seed_from_u64(seed); + let normal = Normal::new(0.0f32, 1.0).unwrap(); + + let mut buf = BufferMut::::with_capacity(num_rows * dim); + for _ in 0..(num_rows * dim) { + buf.push(normal.sample(&mut rng)); + } + + let elements = PrimitiveArray::new::(buf.freeze(), Validity::NonNullable); + FixedSizeListArray::try_new( + elements.into_array(), + dim.try_into() + .expect("somehow got dimension greater than u32::MAX"), + validity, + num_rows, + ) + .unwrap() +} + +/// Create a non-nullable FixedSizeListArray of random f32 vectors. +fn make_fsl(num_rows: usize, dim: usize, seed: u64) -> FixedSizeListArray { + make_fsl_with_validity(num_rows, dim, seed, Validity::NonNullable) +} + +/// Wrap a `FixedSizeListArray` in a `Vector` extension array. +fn make_vector_ext(fsl: &FixedSizeListArray) -> ArrayRef { + Vector::try_new_vector_array(fsl.clone().into_array()) + .vortex_expect("test FSL satisfies Vector storage constraints") +} + +/// Unwrap an L2Denorm ScalarFnArray into (sorf_child, norms_child). +fn unwrap_l2denorm(encoded: &ArrayRef) -> (ArrayRef, ArrayRef) { + let sfn = encoded + .as_opt::() + .expect("expected ScalarFnArray (L2Denorm)"); + (sfn.child_at(0).clone(), sfn.child_at(1).clone()) +} + +/// Navigate the full tree to get (codes, centroids, norms) as flat arrays. +fn unwrap_codes_centroids_norms( + encoded: &ArrayRef, + ctx: &mut vortex_array::ExecutionCtx, +) -> VortexResult<(PrimitiveArray, PrimitiveArray, PrimitiveArray)> { + let (sorf_child, norms_child) = unwrap_l2denorm(encoded); + let padded_vector_child = sorf_child + .as_opt::() + .expect("expected SorfTransform ScalarFnArray") + .child_at(0) + .clone(); + + // Vector wrapping FSL(Dict(codes, centroids)) + let padded_vector: ExtensionArray = padded_vector_child.execute(ctx)?; + let fsl: FixedSizeListArray = padded_vector.storage_array().clone().execute(ctx)?; + let dict = fsl + .elements() + .as_opt::() + .vortex_expect("FSL elements should be a DictArray"); + let codes: PrimitiveArray = dict.codes().clone().execute(ctx)?; + let centroids: PrimitiveArray = dict.values().clone().execute(ctx)?; + let norms: PrimitiveArray = norms_child.execute(ctx)?; + + Ok((codes, centroids, norms)) +} + +fn theoretical_mse_bound(bit_width: u8) -> f32 { + let sqrt3_pi_over_2 = (3.0f32).sqrt() * std::f32::consts::PI / 2.0; + sqrt3_pi_over_2 / (4.0f32).powi(bit_width as i32) +} + +fn per_vector_normalized_mse( + original: &[f32], + reconstructed: &[f32], + dim: usize, + num_rows: usize, +) -> f32 { + let mut total = 0.0f32; + for row in 0..num_rows { + let orig = &original[row * dim..(row + 1) * dim]; + let recon = &reconstructed[row * dim..(row + 1) * dim]; + let norm_sq: f32 = orig.iter().map(|&v| v * v).sum(); + if norm_sq < 1e-10 { + continue; + } + let err_sq: f32 = orig + .iter() + .zip(recon.iter()) + .map(|(&a, &b)| (a - b) * (a - b)) + .sum(); + total += err_sq / norm_sq; + } + total / num_rows as f32 +} + +/// Normalize, encode, and decode, returning (original, decoded) flat f32 slices. +fn encode_decode( + fsl: &FixedSizeListArray, + config: &TurboQuantConfig, +) -> VortexResult<(Vec, Vec)> { + let mut ctx = SESSION.create_execution_ctx(); + let original: Vec = { + let prim = fsl.elements().clone().execute::(&mut ctx)?; + prim.as_slice::().to_vec() + }; + let encoded = turboquant_encode(make_vector_ext(fsl), config, &mut ctx)?; + let decoded_ext = encoded.execute::(&mut ctx)?; + let decoded_fsl = decoded_ext + .storage_array() + .clone() + .execute::(&mut ctx)?; + let decoded_elements: Vec = { + let prim = decoded_fsl + .elements() + .clone() + .execute::(&mut ctx)?; + prim.as_slice::().to_vec() + }; + Ok((original, decoded_elements)) +} diff --git a/vortex-tensor/src/encodings/turboquant/tests/nullable.rs b/vortex-tensor/src/encodings/turboquant/tests/nullable.rs new file mode 100644 index 00000000000..e3febbaab7b --- /dev/null +++ b/vortex-tensor/src/encodings/turboquant/tests/nullable.rs @@ -0,0 +1,178 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +use vortex_array::VortexSessionExecute; +use vortex_array::arrays::ExtensionArray; +use vortex_array::arrays::FixedSizeListArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::extension::ExtensionArrayExt; +use vortex_array::arrays::fixed_size_list::FixedSizeListArrayExt; +use vortex_array::validity::Validity; +use vortex_error::VortexResult; + +use super::*; + +/// Encode a nullable Vector array and verify roundtrip preserves validity and non-null values. +#[test] +fn nullable_vectors_roundtrip() -> VortexResult<()> { + let validity = Validity::from_iter([ + true, true, false, true, true, false, true, false, true, true, + ]); + let fsl = make_fsl_with_validity(10, 128, 42, validity); + let ext = make_vector_ext(&fsl); + + let config = TurboQuantConfig { + bit_width: 3, + seed: 123, + num_rounds: 4, + }; + let mut ctx = SESSION.create_execution_ctx(); + let encoded = turboquant_encode(ext, &config, &mut ctx)?; + + assert_eq!(encoded.len(), 10); + assert!(encoded.dtype().is_nullable()); + + let encoded_validity = encoded.validity()?; + for i in 0..10 { + let expected = ![2, 5, 7].contains(&i); + assert_eq!( + encoded_validity.is_valid(i)?, + expected, + "validity mismatch at row {i}" + ); + } + + let decoded_ext = encoded.execute::(&mut ctx)?; + assert_eq!(decoded_ext.len(), 10); + + let decoded_fsl = decoded_ext + .storage_array() + .clone() + .execute::(&mut ctx)?; + let decoded_prim = decoded_fsl + .elements() + .clone() + .execute::(&mut ctx)?; + let decoded_f32 = decoded_prim.as_slice::(); + + let orig_prim = fsl.elements().clone().execute::(&mut ctx)?; + let orig_f32 = orig_prim.as_slice::(); + + for row in [0, 1, 3, 4, 6, 8, 9] { + let orig_vec = &orig_f32[row * 128..(row + 1) * 128]; + let dec_vec = &decoded_f32[row * 128..(row + 1) * 128]; + let norm_sq: f32 = orig_vec.iter().map(|&v| v * v).sum(); + let err_sq: f32 = orig_vec + .iter() + .zip(dec_vec.iter()) + .map(|(&a, &b)| (a - b) * (a - b)) + .sum(); + assert!( + err_sq / norm_sq < 0.1, + "non-null row {row} has excessive reconstruction error" + ); + } + Ok(()) +} + +/// Verify that norms carry the validity: null vectors have null norms. +#[test] +fn nullable_norms_match_validity() -> VortexResult<()> { + let validity = Validity::from_iter([true, false, true, false, true]); + let fsl = make_fsl_with_validity(5, 128, 42, validity); + let ext = make_vector_ext(&fsl); + + let config = TurboQuantConfig { + bit_width: 2, + seed: 123, + num_rounds: 3, + }; + let mut ctx = SESSION.create_execution_ctx(); + let encoded = turboquant_encode(ext, &config, &mut ctx)?; + let (_sorf_child, norms_child) = unwrap_l2denorm(&encoded); + + let norms_validity = norms_child.validity()?; + for i in 0..5 { + let expected = i % 2 == 0; + assert_eq!( + norms_validity.is_valid(i)?, + expected, + "norms validity mismatch at row {i}" + ); + } + Ok(()) +} + +/// Verify that L2Norm readthrough works correctly on nullable TurboQuant arrays. +#[test] +fn nullable_l2_norm_readthrough() -> VortexResult<()> { + use crate::scalar_fns::l2_norm::L2Norm; + + let validity = Validity::from_iter([true, false, true, false, true]); + let fsl = make_fsl_with_validity(5, 128, 42, validity); + let ext = make_vector_ext(&fsl); + + let config = TurboQuantConfig { + bit_width: 3, + seed: 123, + num_rounds: 3, + }; + let mut ctx = SESSION.create_execution_ctx(); + let encoded = turboquant_encode(ext, &config, &mut ctx)?; + + let norm_sfn = L2Norm::try_new_array(encoded, 5)?; + let norms: PrimitiveArray = norm_sfn.into_array().execute(&mut ctx)?; + + let orig_prim = fsl.elements().clone().execute::(&mut ctx)?; + let orig_f32 = orig_prim.as_slice::(); + for row in 0..5 { + if row % 2 == 0 { + assert!(norms.is_valid(row, &mut ctx)?, "row {row} should be valid"); + let expected: f32 = orig_f32[row * 128..(row + 1) * 128] + .iter() + .map(|&v| v * v) + .sum::() + .sqrt(); + let actual = norms.as_slice::()[row]; + assert!( + (actual - expected).abs() < 1e-5, + "norm mismatch at valid row {row}: actual={actual}, expected={expected}" + ); + } else { + assert!(!norms.is_valid(row, &mut ctx)?, "row {row} should be null"); + } + } + Ok(()) +} + +/// Verify that slicing a nullable TurboQuant array preserves validity. +#[test] +fn nullable_slice_preserves_validity() -> VortexResult<()> { + let validity = Validity::from_iter([ + true, true, false, true, true, false, true, false, true, true, + ]); + let fsl = make_fsl_with_validity(10, 128, 42, validity); + let ext = make_vector_ext(&fsl); + + let config = TurboQuantConfig { + bit_width: 3, + seed: 123, + num_rounds: 2, + }; + let mut ctx = SESSION.create_execution_ctx(); + let encoded = turboquant_encode(ext, &config, &mut ctx)?; + + let sliced = encoded.slice(1..6)?; + assert_eq!(sliced.len(), 5); + + let sliced_validity = sliced.validity()?; + let expected = [true, false, true, true, false]; + for (i, &exp) in expected.iter().enumerate() { + assert_eq!( + sliced_validity.is_valid(i)?, + exp, + "sliced validity mismatch at index {i}" + ); + } + Ok(()) +} diff --git a/vortex-tensor/src/encodings/turboquant/tests/roundtrip.rs b/vortex-tensor/src/encodings/turboquant/tests/roundtrip.rs new file mode 100644 index 00000000000..d82be3cf714 --- /dev/null +++ b/vortex-tensor/src/encodings/turboquant/tests/roundtrip.rs @@ -0,0 +1,313 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +use rstest::rstest; +use vortex_array::IntoArray; +use vortex_array::VortexSessionExecute; +use vortex_array::arrays::Extension; +use vortex_array::arrays::ExtensionArray; +use vortex_array::arrays::FixedSizeListArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::validity::Validity; +use vortex_buffer::BufferMut; +use vortex_error::VortexResult; +use vortex_error::vortex_err; + +use super::*; +use crate::encodings::turboquant::turboquant_encode_unchecked; +use crate::scalar_fns::l2_denorm::normalize_as_l2_denorm; + +#[rstest] +#[case(128, 1)] +#[case(128, 2)] +#[case(128, 3)] +#[case(128, 4)] +#[case(128, 6)] +#[case(128, 8)] +#[case(256, 2)] +fn roundtrip(#[case] dim: usize, #[case] bit_width: u8) -> VortexResult<()> { + let fsl = make_fsl(10, dim, 42); + let config = TurboQuantConfig { + bit_width, + seed: 123, + num_rounds: 3, + }; + let (original, decoded) = encode_decode(&fsl, &config)?; + assert_eq!(decoded.len(), original.len()); + Ok(()) +} + +#[rstest] +#[case(128, 1)] +#[case(128, 2)] +#[case(128, 3)] +#[case(128, 4)] +#[case(256, 2)] +#[case(256, 4)] +fn mse_within_theoretical_bound(#[case] dim: usize, #[case] bit_width: u8) -> VortexResult<()> { + let num_rows = 200; + let fsl = make_fsl(num_rows, dim, 42); + let config = TurboQuantConfig { + bit_width, + seed: 123, + num_rounds: 3, + }; + let (original, decoded) = encode_decode(&fsl, &config)?; + + let normalized_mse = per_vector_normalized_mse(&original, &decoded, dim, num_rows); + let bound = theoretical_mse_bound(bit_width); + + assert!( + normalized_mse < bound, + "Normalized MSE {normalized_mse:.6} exceeds bound {bound:.6} \ + for dim={dim}, bits={bit_width}", + ); + Ok(()) +} + +#[rstest] +#[case(128, 6)] +#[case(128, 8)] +#[case(256, 6)] +#[case(256, 8)] +fn high_bitwidth_mse_is_small(#[case] dim: usize, #[case] bit_width: u8) -> VortexResult<()> { + let num_rows = 200; + let fsl = make_fsl(num_rows, dim, 42); + + let config_4bit = TurboQuantConfig { + bit_width: 4, + seed: 123, + num_rounds: 3, + }; + let (original_4, decoded_4) = encode_decode(&fsl, &config_4bit)?; + let mse_4bit = per_vector_normalized_mse(&original_4, &decoded_4, dim, num_rows); + + let config = TurboQuantConfig { + bit_width, + seed: 123, + num_rounds: 3, + }; + let (original, decoded) = encode_decode(&fsl, &config)?; + let mse = per_vector_normalized_mse(&original, &decoded, dim, num_rows); + + assert!( + mse < mse_4bit, + "{bit_width}-bit MSE ({mse:.6}) should be < 4-bit MSE ({mse_4bit:.6})" + ); + assert!(mse < 0.01, "{bit_width}-bit MSE ({mse:.6}) should be < 1%"); + Ok(()) +} + +#[test] +fn mse_decreases_with_bits() -> VortexResult<()> { + let dim = 128; + let num_rows = 50; + let fsl = make_fsl(num_rows, dim, 99); + + let mut prev_mse = f32::MAX; + for bit_width in 1..=8u8 { + let config = TurboQuantConfig { + bit_width, + seed: 123, + num_rounds: 3, + }; + let (original, decoded) = encode_decode(&fsl, &config)?; + let mse = per_vector_normalized_mse(&original, &decoded, dim, num_rows); + assert!( + mse <= prev_mse * 1.01, + "MSE should decrease: {bit_width}-bit={mse:.6} > prev={prev_mse:.6}" + ); + prev_mse = mse; + } + Ok(()) +} + +#[rstest] +#[case(0)] +#[case(1)] +fn roundtrip_edge_cases(#[case] num_rows: usize) -> VortexResult<()> { + let fsl = make_fsl(num_rows, 128, 42); + let ext = make_vector_ext(&fsl); + let config = TurboQuantConfig { + bit_width: 2, + seed: 123, + num_rounds: 3, + }; + let mut ctx = SESSION.create_execution_ctx(); + let encoded = turboquant_encode(ext, &config, &mut ctx)?; + let decoded = encoded.execute::(&mut ctx)?; + assert_eq!(decoded.len(), num_rows); + Ok(()) +} + +#[rstest] +#[case(1)] +#[case(64)] +#[case(127)] +fn rejects_dimension_below_128(#[case] dim: usize) { + let elements = PrimitiveArray::new::( + BufferMut::from_iter((0..dim).map(|i| i as f32 + 1.0)).freeze(), + Validity::NonNullable, + ); + let fsl = FixedSizeListArray::try_new( + elements.into_array(), + dim.try_into().expect("dim fits u32"), + Validity::NonNullable, + 1, + ) + .unwrap(); + let ext = make_vector_ext(&fsl); + let config = TurboQuantConfig { + bit_width: 2, + seed: 0, + num_rounds: 3, + }; + let mut ctx = SESSION.create_execution_ctx(); + assert!(turboquant_encode(ext, &config, &mut ctx).is_err()); +} + +#[rstest] +#[case(0)] +#[case(9)] +fn rejects_invalid_bit_width(#[case] bit_width: u8) { + let fsl = make_fsl(10, 128, 42); + let ext = make_vector_ext(&fsl); + let config = TurboQuantConfig { + bit_width, + seed: 0, + num_rounds: 3, + }; + let mut ctx = SESSION.create_execution_ctx(); + let normalized = normalize_as_l2_denorm(ext, &mut ctx) + .unwrap() + .child_at(0) + .clone(); + let normalized_ext = normalized + .as_opt::() + .expect("normalized child should be Extension"); + assert!(unsafe { turboquant_encode_unchecked(normalized_ext, &config, &mut ctx) }.is_err()); +} + +#[test] +fn all_zero_vectors_roundtrip() -> VortexResult<()> { + let num_rows = 10; + let dim = 128; + let buf = BufferMut::::full(0.0f32, num_rows * dim); + let elements = PrimitiveArray::new::(buf.freeze(), Validity::NonNullable); + let fsl = FixedSizeListArray::try_new( + elements.into_array(), + dim.try_into().map_err(|e| vortex_err!("{e}"))?, + Validity::NonNullable, + num_rows, + )?; + + let config = TurboQuantConfig { + bit_width: 3, + seed: 42, + num_rounds: 3, + }; + let (original, decoded) = encode_decode(&fsl, &config)?; + for (i, (&o, &d)) in original.iter().zip(decoded.iter()).enumerate() { + assert_eq!(o, 0.0, "original[{i}] not zero"); + assert_eq!(d, 0.0, "decoded[{i}] not zero for all-zero input"); + } + Ok(()) +} + +/// Roundtrip at large embedding dimensions. +#[rstest] +#[case(768, 4)] +#[case(1024, 5)] +fn large_dimension_roundtrip(#[case] dim: usize, #[case] bit_width: u8) -> VortexResult<()> { + let num_rows = 10; + let fsl = make_fsl(num_rows, dim, 42); + let config = TurboQuantConfig { + bit_width, + seed: 123, + num_rounds: 3, + }; + let (original, decoded) = encode_decode(&fsl, &config)?; + assert_eq!(decoded.len(), original.len()); + + let normalized_mse = per_vector_normalized_mse(&original, &decoded, dim, num_rows); + // 2x slack for the SRHT-vs-Haar gap. + let bound = 2.0 * theoretical_mse_bound(bit_width); + assert!( + normalized_mse < bound, + "Normalized MSE {normalized_mse:.6} exceeds 2x bound {bound:.6} for dim={dim}, bits={bit_width}", + ); + Ok(()) +} + +/// Verify that f64 input is accepted and encoded. +#[test] +fn f64_input_encodes_successfully() -> VortexResult<()> { + let num_rows = 10; + let dim = 128; + let mut rng = StdRng::seed_from_u64(99); + let normal = Normal::new(0.0f64, 1.0).map_err(|e| vortex_err!("{e}"))?; + + let mut buf = BufferMut::::with_capacity(num_rows * dim); + for _ in 0..(num_rows * dim) { + buf.push(normal.sample(&mut rng)); + } + let elements = PrimitiveArray::new::(buf.freeze(), Validity::NonNullable); + let fsl = FixedSizeListArray::try_new( + elements.into_array(), + dim.try_into().map_err(|e| vortex_err!("{e}"))?, + Validity::NonNullable, + num_rows, + )?; + + let ext = make_vector_ext(&fsl); + let config = TurboQuantConfig { + bit_width: 3, + seed: 42, + num_rounds: 3, + }; + let mut ctx = SESSION.create_execution_ctx(); + let encoded = turboquant_encode(ext, &config, &mut ctx)?; + let (_sorf_child, norms_child) = unwrap_l2denorm(&encoded); + assert_eq!(norms_child.len(), num_rows); + Ok(()) +} + +/// Verify that f16 input is accepted and encoded. +#[test] +fn f16_input_encodes_successfully() -> VortexResult<()> { + let num_rows = 10; + let dim = 128; + let mut rng = StdRng::seed_from_u64(99); + let normal = Normal::new(0.0f32, 1.0).map_err(|e| vortex_err!("{e}"))?; + + let mut buf = BufferMut::::with_capacity(num_rows * dim); + for _ in 0..(num_rows * dim) { + buf.push(half::f16::from_f32(normal.sample(&mut rng))); + } + let elements = PrimitiveArray::new::(buf.freeze(), Validity::NonNullable); + let fsl = FixedSizeListArray::try_new( + elements.into_array(), + dim.try_into().map_err(|e| vortex_err!("{e}"))?, + Validity::NonNullable, + num_rows, + )?; + + let ext = make_vector_ext(&fsl); + let config = TurboQuantConfig { + bit_width: 3, + seed: 42, + num_rounds: 3, + }; + let mut ctx = SESSION.create_execution_ctx(); + let encoded = turboquant_encode(ext, &config, &mut ctx)?; + let (_sorf_child, norms_child) = unwrap_l2denorm(&encoded); + assert_eq!(norms_child.len(), num_rows); + + let decoded_ext = encoded.execute::(&mut ctx)?; + let decoded_fsl = decoded_ext + .storage_array() + .clone() + .execute::(&mut ctx)?; + assert_eq!(decoded_fsl.len(), num_rows); + Ok(()) +} diff --git a/vortex-tensor/src/encodings/turboquant/tests/structural.rs b/vortex-tensor/src/encodings/turboquant/tests/structural.rs new file mode 100644 index 00000000000..3913cf3d8fe --- /dev/null +++ b/vortex-tensor/src/encodings/turboquant/tests/structural.rs @@ -0,0 +1,333 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Tests that verify the internal structure of the encoded tree. + +use vortex_array::VortexSessionExecute; +use vortex_array::arrays::ExtensionArray; +use vortex_array::arrays::FixedSizeListArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::extension::ExtensionArrayExt; +use vortex_array::arrays::fixed_size_list::FixedSizeListArrayExt; +use vortex_error::VortexResult; + +use super::*; + +/// Verify that the centroids stored in the DictArray match what `compute_or_get_centroids()` computes. +#[test] +fn stored_centroids_match_computed() -> VortexResult<()> { + let fsl = make_fsl(10, 128, 42); + let ext = make_vector_ext(&fsl); + let config = TurboQuantConfig { + bit_width: 3, + seed: 123, + num_rounds: 3, + }; + let mut ctx = SESSION.create_execution_ctx(); + let encoded = turboquant_encode(ext, &config, &mut ctx)?; + + let (_codes, centroids, _norms) = unwrap_codes_centroids_norms(&encoded, &mut ctx)?; + let stored = centroids.as_slice::(); + + // padded_dim for dim=128 is 128. + let computed = crate::encodings::turboquant::centroids::compute_or_get_centroids(128, 3)?; + + assert_eq!(stored.len(), computed.len()); + for i in 0..stored.len() { + assert_eq!(stored[i], computed[i], "Centroid mismatch at {i}"); + } + Ok(()) +} + +/// Verify that the rotation is deterministic from seed by checking decode output. +#[test] +fn seed_deterministic_rotation_produces_correct_decode() -> VortexResult<()> { + let fsl = make_fsl(20, 128, 42); + let ext = make_vector_ext(&fsl); + let config = TurboQuantConfig { + bit_width: 3, + seed: 123, + num_rounds: 4, + }; + + // Encode twice with the same seed → should produce identical results. + let mut ctx = SESSION.create_execution_ctx(); + let encoded1 = turboquant_encode(ext.clone(), &config, &mut ctx)?; + let decoded1 = encoded1.execute::(&mut ctx)?; + let fsl1 = decoded1 + .storage_array() + .clone() + .execute::(&mut ctx)?; + let elems1 = fsl1 + .elements() + .clone() + .execute::(&mut ctx)?; + + let mut ctx = SESSION.create_execution_ctx(); + let encoded2 = turboquant_encode(ext, &config, &mut ctx)?; + let decoded2 = encoded2.execute::(&mut ctx)?; + let fsl2 = decoded2 + .storage_array() + .clone() + .execute::(&mut ctx)?; + let elems2 = fsl2 + .elements() + .clone() + .execute::(&mut ctx)?; + + assert_eq!( + elems1.as_slice::(), + elems2.as_slice::(), + "Two encodes with same seed should produce identical decode output" + ); + Ok(()) +} + +/// Verify that the encoded array's dtype is a Vector extension type. +#[test] +fn encoded_dtype_is_vector_extension() -> VortexResult<()> { + let fsl = make_fsl(10, 128, 42); + let ext = make_vector_ext(&fsl); + let config = TurboQuantConfig { + bit_width: 3, + seed: 123, + num_rounds: 2, + }; + let mut ctx = SESSION.create_execution_ctx(); + let encoded = turboquant_encode(ext, &config, &mut ctx)?; + + assert!( + encoded.dtype().is_extension(), + "TurboQuant dtype should be an extension type, got {}", + encoded.dtype() + ); + assert!( + encoded.dtype().as_extension().is::(), + "TurboQuant dtype should be a Vector extension type" + ); + Ok(()) +} + +/// Verify approximate cosine similarity in the quantized domain. +#[test] +fn cosine_similarity_quantized_accuracy() -> VortexResult<()> { + let fsl = make_fsl(20, 128, 42); + let ext = make_vector_ext(&fsl); + let config = TurboQuantConfig { + bit_width: 4, + seed: 123, + num_rounds: 3, + }; + let mut ctx = SESSION.create_execution_ctx(); + let encoded = turboquant_encode(ext, &config, &mut ctx)?; + + let input_prim = fsl.elements().clone().execute::(&mut ctx)?; + let input_f32 = input_prim.as_slice::(); + + // Navigate tree to get codes, centroids, norms. + let (codes_prim, centroids_prim, norms_prim) = + unwrap_codes_centroids_norms(&encoded, &mut ctx)?; + let all_codes = codes_prim.as_slice::(); + let centroid_vals = centroids_prim.as_slice::(); + let norms = norms_prim.as_slice::(); + + // padded_dim for dim=128. + let pd = 128usize; + + for (row_a, row_b) in [(0, 1), (5, 10), (0, 19)] { + let vec_a = &input_f32[row_a * 128..(row_a + 1) * 128]; + let vec_b = &input_f32[row_b * 128..(row_b + 1) * 128]; + + let dot: f32 = vec_a.iter().zip(vec_b.iter()).map(|(&x, &y)| x * y).sum(); + let norm_a: f32 = vec_a.iter().map(|&v| v * v).sum::().sqrt(); + let norm_b: f32 = vec_b.iter().map(|&v| v * v).sum::().sqrt(); + let exact_cos = dot / (norm_a * norm_b); + + let approx_cos = if norms[row_a] == 0.0 || norms[row_b] == 0.0 { + 0.0 + } else { + let codes_a = &all_codes[row_a * pd..(row_a + 1) * pd]; + let codes_b = &all_codes[row_b * pd..(row_b + 1) * pd]; + codes_a + .iter() + .zip(codes_b.iter()) + .map(|(&ca, &cb)| centroid_vals[ca as usize] * centroid_vals[cb as usize]) + .sum::() + }; + + let error = (exact_cos - approx_cos).abs(); + assert!( + error < 0.15, + "cosine similarity error too large for ({row_a}, {row_b}): \ + exact={exact_cos:.4}, approx={approx_cos:.4}, error={error:.4}" + ); + } + Ok(()) +} + +/// Verify approximate dot product in the quantized domain. +#[test] +fn dot_product_quantized_accuracy() -> VortexResult<()> { + let fsl = make_fsl(20, 128, 42); + let ext = make_vector_ext(&fsl); + let config = TurboQuantConfig { + bit_width: 8, + seed: 123, + num_rounds: 3, + }; + let mut ctx = SESSION.create_execution_ctx(); + let encoded = turboquant_encode(ext, &config, &mut ctx)?; + + let input_prim = fsl.elements().clone().execute::(&mut ctx)?; + let input_f32 = input_prim.as_slice::(); + + let (codes_prim, centroids_prim, norms_prim) = + unwrap_codes_centroids_norms(&encoded, &mut ctx)?; + let all_codes = codes_prim.as_slice::(); + let centroid_vals = centroids_prim.as_slice::(); + let norms = norms_prim.as_slice::(); + + let pd = 128usize; + + for (row_a, row_b) in [(0, 1), (5, 10), (0, 19)] { + let vec_a = &input_f32[row_a * 128..(row_a + 1) * 128]; + let vec_b = &input_f32[row_b * 128..(row_b + 1) * 128]; + + let exact_dot: f32 = vec_a.iter().zip(vec_b.iter()).map(|(&x, &y)| x * y).sum(); + + let codes_a = &all_codes[row_a * pd..(row_a + 1) * pd]; + let codes_b = &all_codes[row_b * pd..(row_b + 1) * pd]; + let unit_dot: f32 = codes_a + .iter() + .zip(codes_b.iter()) + .map(|(&ca, &cb)| centroid_vals[ca as usize] * centroid_vals[cb as usize]) + .sum(); + let approx_dot = norms[row_a] * norms[row_b] * unit_dot; + + let scale = exact_dot.abs().max(1.0); + let rel_error = (exact_dot - approx_dot).abs() / scale; + assert!( + rel_error < 0.15, + "dot product error too large for ({row_a}, {row_b}): \ + exact={exact_dot:.4}, approx={approx_dot:.4}, rel_error={rel_error:.4}" + ); + } + Ok(()) +} + +/// Verify SorfTransform in isolation: manually forward-rotate known data, wrap in +/// FSL(Dict), execute SorfTransform, and check inverse rotation recovers the original. +#[test] +#[expect( + clippy::cast_possible_truncation, + reason = "test uses known small dimensions" +)] +fn sorf_transform_roundtrip_isolation() -> VortexResult<()> { + use vortex_array::IntoArray; + use vortex_array::arrays::dict::DictArray; + use vortex_array::dtype::extension::ExtDType; + use vortex_array::extension::EmptyMetadata; + use vortex_array::validity::Validity; + use vortex_buffer::BufferMut; + + use crate::encodings::turboquant::centroids::compute_centroid_boundaries; + use crate::encodings::turboquant::centroids::compute_or_get_centroids; + use crate::encodings::turboquant::centroids::find_nearest_centroid; + use crate::scalar_fns::sorf_transform::SorfMatrix; + use crate::scalar_fns::sorf_transform::SorfOptions; + use crate::scalar_fns::sorf_transform::SorfTransform; + use crate::types::vector::Vector; + + let dim = 128usize; + let seed = 99u64; + let num_rounds = 3u8; + let num_rows = 5; + + // Build a known input: simple increasing values, then normalize each row to unit norm. + let mut input_f32 = vec![0.0f32; num_rows * dim]; + for row in 0..num_rows { + let mut norm_sq = 0.0f32; + for i in 0..dim { + let val = ((row * dim + i) as f32 + 1.0) * 0.01; + input_f32[row * dim + i] = val; + norm_sq += val * val; + } + let norm = norm_sq.sqrt(); + for i in 0..dim { + input_f32[row * dim + i] /= norm; + } + } + + // Forward transform + quantize (mimicking what turboquant_quantize_core does). + let rotation = SorfMatrix::try_new(seed, dim, num_rounds as usize)?; + let padded_dim = rotation.padded_dim(); + let centroids = compute_or_get_centroids(padded_dim as u32, 8)?; + let boundaries = compute_centroid_boundaries(¢roids); + + let mut all_indices = BufferMut::::with_capacity(num_rows * padded_dim); + let mut padded = vec![0.0f32; padded_dim]; + let mut rotated = vec![0.0f32; padded_dim]; + + for row in 0..num_rows { + padded[..dim].copy_from_slice(&input_f32[row * dim..(row + 1) * dim]); + padded[dim..].fill(0.0); + rotation.rotate(&padded, &mut rotated); + for j in 0..padded_dim { + all_indices.push(find_nearest_centroid(rotated[j], &boundaries)); + } + } + + // Build FSL(Dict(codes, centroids)). + let codes = PrimitiveArray::new::(all_indices.freeze(), Validity::NonNullable); + let mut centroids_buf = BufferMut::::with_capacity(centroids.len()); + centroids_buf.extend_from_slice(¢roids); + let centroids_arr = PrimitiveArray::new::(centroids_buf.freeze(), Validity::NonNullable); + let dict = DictArray::try_new(codes.into_array(), centroids_arr.into_array())?; + let fsl = FixedSizeListArray::try_new( + dict.into_array(), + padded_dim as u32, + Validity::NonNullable, + num_rows, + )?; + + // Wrap the padded FSL in a Vector extension so it can be the SorfTransform child. + let padded_vector_dtype = + ExtDType::::try_new(EmptyMetadata, fsl.dtype().clone())?.erased(); + let padded_vector = ExtensionArray::new(padded_vector_dtype, fsl.into_array()); + + // Wrap in SorfTransform and execute. + let sorf_options = SorfOptions { + seed, + num_rounds, + dimensions: dim as u32, + element_ptype: vortex_array::dtype::PType::F32, + }; + let sorf_array = + SorfTransform::try_new_array(&sorf_options, padded_vector.into_array(), num_rows)?; + + let mut ctx = SESSION.create_execution_ctx(); + let result: ExtensionArray = sorf_array.into_array().execute(&mut ctx)?; + let result_fsl: FixedSizeListArray = result.storage_array().clone().execute(&mut ctx)?; + let result_prim: PrimitiveArray = result_fsl.elements().clone().execute(&mut ctx)?; + let result_f32 = result_prim.as_slice::(); + + assert_eq!(result_f32.len(), num_rows * dim); + + // At 8-bit quantization, reconstruction should be very close to input. + for row in 0..num_rows { + let orig = &input_f32[row * dim..(row + 1) * dim]; + let recon = &result_f32[row * dim..(row + 1) * dim]; + let err_sq: f32 = orig + .iter() + .zip(recon) + .map(|(&a, &b)| (a - b) * (a - b)) + .sum(); + let norm_sq: f32 = orig.iter().map(|&v| v * v).sum(); + assert!( + err_sq / norm_sq < 1e-3, + "SorfTransform isolation: row {row} MSE too high: {:.6}", + err_sq / norm_sq + ); + } + Ok(()) +} diff --git a/vortex-tensor/src/encodings/turboquant/vtable.rs b/vortex-tensor/src/encodings/turboquant/vtable.rs deleted file mode 100644 index b673dea6ba5..00000000000 --- a/vortex-tensor/src/encodings/turboquant/vtable.rs +++ /dev/null @@ -1,382 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -//! VTable implementation for TurboQuant encoding. - -use std::hash::Hash; -use std::hash::Hasher; -use std::sync::Arc; - -use prost::Message; -use vortex_array::Array; -use vortex_array::ArrayEq; -use vortex_array::ArrayHash; -use vortex_array::ArrayId; -use vortex_array::ArrayParts; -use vortex_array::ArrayRef; -use vortex_array::ArrayView; -use vortex_array::ExecutionCtx; -use vortex_array::ExecutionResult; -use vortex_array::Precision; -use vortex_array::buffer::BufferHandle; -use vortex_array::dtype::DType; -use vortex_array::dtype::Nullability; -use vortex_array::dtype::PType; -use vortex_array::serde::ArrayChildren; -use vortex_array::validity::Validity; -use vortex_array::vtable::VTable; -use vortex_array::vtable::ValidityVTable; -use vortex_error::VortexExpect; -use vortex_error::VortexResult; -use vortex_error::vortex_ensure; -use vortex_error::vortex_ensure_eq; -use vortex_error::vortex_err; -use vortex_error::vortex_panic; -use vortex_session::VortexSession; - -use crate::encodings::turboquant::TurboQuantData; -use crate::encodings::turboquant::array::slots::Slot; -use crate::encodings::turboquant::compute::rules::PARENT_KERNELS; -use crate::encodings::turboquant::compute::rules::RULES; -use crate::encodings::turboquant::metadata::TurboQuantMetadata; -use crate::encodings::turboquant::scheme::decompress::execute_decompress; -use crate::vector::AnyVector; -use crate::vector::VectorMatcherMetadata; - -/// Encoding marker type for TurboQuant. -#[derive(Clone, Debug)] -pub struct TurboQuant; - -impl TurboQuant { - pub const ID: ArrayId = ArrayId::new_ref("vortex.turboquant"); - - /// Minimum vector dimension for TurboQuant encoding. - /// - /// Note that this is not a theoretical minimum, it is mostly a practical one to limit the total - /// amount of distortion. - pub const MIN_DIMENSION: u32 = 128; - - /// Maximum supported number of bits per quantized coordinate. - pub const MAX_BIT_WIDTH: u8 = 8; - - /// Maximum supported number of centroids in the scalar quantizer codebook. - pub const MAX_CENTROIDS: usize = 1usize << (Self::MAX_BIT_WIDTH as usize); - - /// Validates that `dtype` is a [`Vector`](crate::vector::Vector) extension type with - /// dimension >= [`MIN_DIMENSION`](Self::MIN_DIMENSION). - /// - /// Returns the validated vector metadata on success. - pub fn validate_dtype(dtype: &DType) -> VortexResult { - let vector_metadata = dtype - .as_extension_opt() - .and_then(|ext| ext.metadata_opt::()) - .ok_or_else(|| { - vortex_err!("TurboQuant dtype must be a Vector extension type, got {dtype}") - })?; - - let dimensions = vector_metadata.dimensions(); - vortex_ensure!( - dimensions >= Self::MIN_DIMENSION, - "TurboQuant requires dimension >= {}, got {dimensions}", - Self::MIN_DIMENSION - ); - - Ok(vector_metadata) - } - - /// Creates a new [`TurboQuantArray`]. - /// - /// The `dtype` must be a non-nullable [`Vector`](crate::vector::Vector) extension type. - /// Nullability is handled externally by the [`L2Denorm`](crate::scalar_fns::l2_denorm::L2Denorm) - /// ScalarFnArray wrapper. - /// - /// Internally calls [`TurboQuantData::validate`] and [`TurboQuantData::try_new`], then - /// delegates to [`new_array_unchecked`](Self::new_array_unchecked). - pub fn try_new_array( - dtype: DType, - codes: ArrayRef, - centroids: ArrayRef, - rotation_signs: ArrayRef, - ) -> VortexResult { - TurboQuantData::validate(&dtype, &codes, ¢roids, &rotation_signs)?; - - Ok(unsafe { Self::new_array_unchecked(dtype, codes, centroids, rotation_signs) }) - } - - /// Creates a new [`TurboQuantArray`] without validation. - /// - /// # Safety - /// - /// The caller must ensure all invariants required by [`TurboQuantData::validate`] hold: - /// - /// - `dtype` is a non-nullable [`Vector`](crate::vector::Vector) extension type with - /// dimension >= [`MIN_DIMENSION`](Self::MIN_DIMENSION). - /// - `codes` is a non-nullable `FixedSizeList` with `list_size == padded_dim`. - /// - `centroids` is a non-nullable `Primitive` with a power-of-2 length in - /// `[2, MAX_CENTROIDS]` (or empty for degenerate arrays). - /// - `rotation_signs` is a non-nullable `FixedSizeList` with `list_size == padded_dim`. - /// - /// Violating these invariants may produce incorrect results during decompression or panics - /// during array access. - pub unsafe fn new_array_unchecked( - dtype: DType, - codes: ArrayRef, - centroids: ArrayRef, - rotation_signs: ArrayRef, - ) -> TurboQuantArray { - #[cfg(debug_assertions)] - TurboQuantData::validate(&dtype, &codes, ¢roids, &rotation_signs) - .vortex_expect("[DEBUG ASSERTION]: TurboQuantData arrays are invalid"); - - let len = codes.len(); - - let dimension = dtype - .as_extension_opt() - .vortex_expect("we validated the dtype") - .metadata_opt::() - .vortex_expect("we validated that this is a vector") - .dimensions(); - - let bit_width = if centroids.is_empty() { - 0 - } else { - #[expect( - clippy::cast_possible_truncation, - reason = "bit_width is guaranteed <= 8" - )] - (centroids.len().trailing_zeros() as u8) - }; - - #[expect( - clippy::cast_possible_truncation, - reason = "num_rounds fits in u8 by the caller's invariants" - )] - let num_rounds = rotation_signs.len() as u8; - - // SAFETY: The caller guarantees that dimension, bit_width, and num_rounds satisfy the - // invariants documented on `TurboQuantData::new_unchecked`. - let data = unsafe { TurboQuantData::new_unchecked(dimension, bit_width, num_rounds) }; - let parts = ArrayParts::new(TurboQuant, dtype, len, data) - .with_slots(TurboQuantData::make_slots(codes, centroids, rotation_signs)); - - // SAFETY: The caller guarantees the parts are logically consistent. - unsafe { Array::from_parts_unchecked(parts) } - } -} - -/// A [`TurboQuant`]-encoded Vortex array. -pub type TurboQuantArray = Array; - -impl VTable for TurboQuant { - type ArrayData = TurboQuantData; - type OperationsVTable = TurboQuant; - type ValidityVTable = TurboQuant; - - fn id(&self) -> ArrayId { - Self::ID - } - - fn validate( - &self, - data: &Self::ArrayData, - dtype: &DType, - len: usize, - slots: &[Option], - ) -> VortexResult<()> { - vortex_ensure_eq!( - slots.len(), - Slot::COUNT, - "TurboQuantArray got incorrect amount of slots", - ); - - // Even if the array is degenerate (empty), the arrays still have to exist - // (they will be empty). - let codes = slots[Slot::Codes as usize] - .as_ref() - .ok_or_else(|| vortex_err!("TurboQuantArray missing codes slot"))?; - let centroids = slots[Slot::Centroids as usize] - .as_ref() - .ok_or_else(|| vortex_err!("TurboQuantArray missing centroids slot"))?; - let rotation_signs = slots[Slot::RotationSigns as usize] - .as_ref() - .ok_or_else(|| vortex_err!("TurboQuantArray missing rotation_signs slot"))?; - - vortex_ensure_eq!( - codes.len(), - len, - "TurboQuant codes length does not match outer length", - ); - - TurboQuantData::validate(dtype, codes, centroids, rotation_signs)?; - - vortex_ensure_eq!(data.dimension, Self::validate_dtype(dtype)?.dimensions()); - - let expected_bit_width = if centroids.is_empty() { - 0 - } else { - u8::try_from(centroids.len().trailing_zeros()) - .map_err(|_| vortex_err!("centroids bit_width does not fit in u8"))? - }; - vortex_ensure_eq!( - data.bit_width, - expected_bit_width, - "TurboQuant bit_width does not match centroids slot", - ); - - // Verify num_rounds matches the rotation_signs FSL length. - let expected_num_rounds = u8::try_from(rotation_signs.len()) - .map_err(|_| vortex_err!("rotation_signs num_rounds does not fit in u8"))?; - vortex_ensure_eq!( - data.num_rounds, - expected_num_rounds, - "TurboQuant num_rounds does not match rotation_signs slot", - ); - - Ok(()) - } - - fn nbuffers(_array: ArrayView) -> usize { - 0 - } - - fn buffer(_array: ArrayView, idx: usize) -> BufferHandle { - vortex_panic!("TurboQuantArray buffer index {idx} out of bounds") - } - - fn buffer_name(_array: ArrayView, _idx: usize) -> Option { - None - } - - fn serialize( - array: ArrayView<'_, Self>, - _session: &VortexSession, - ) -> VortexResult>> { - Ok(Some( - TurboQuantMetadata::new(array.bit_width, array.num_rounds).encode_to_vec(), - )) - } - - fn deserialize( - &self, - dtype: &DType, - len: usize, - metadata: &[u8], - _buffers: &[BufferHandle], - children: &dyn ArrayChildren, - _session: &VortexSession, - ) -> VortexResult> { - let metadata = TurboQuantMetadata::decode(metadata)?; - let bit_width = metadata.bit_width()?; - let num_rounds = metadata.num_rounds()?; - - // bit_width == 0 and num_rounds == 0 are only valid for degenerate (empty) arrays. - vortex_ensure!( - bit_width > 0 || len == 0, - "bit_width == 0 is only valid for empty arrays, got len={len}" - ); - vortex_ensure!( - num_rounds > 0 || len == 0, - "num_rounds == 0 is only valid for empty arrays, got len={len}" - ); - - // Validate and derive dimension from the Vector extension dtype. - let vector_metadata = TurboQuant::validate_dtype(dtype)?; - let dimensions = vector_metadata.dimensions(); - - // TurboQuant arrays are always non-nullable. - vortex_ensure!( - !dtype.is_nullable(), - "TurboQuant dtype must be non-nullable during deserialization" - ); - - let padded_dim = dimensions.next_power_of_two(); - - // Get the codes array (indices into the codebook). Codes are always non-nullable. - let codes_ptype = DType::Primitive(PType::U8, Nullability::NonNullable); - let codes_dtype = - DType::FixedSizeList(Arc::new(codes_ptype), padded_dim, Nullability::NonNullable); - let codes_array = children.get(0, &codes_dtype, len)?; - - // Get the centroids array (codebook). - let num_centroids = if bit_width == 0 { - 0 // A degenerate TQ array. - } else { - 1usize << bit_width - }; - let centroids_dtype = DType::Primitive(PType::F32, Nullability::NonNullable); - let centroids = children.get(1, ¢roids_dtype, num_centroids)?; - - // Get the rotation signs array (FixedSizeList with list_size = padded_dim). - let signs_len = if len == 0 { 0 } else { num_rounds as usize }; - let signs_dtype = DType::FixedSizeList( - Arc::new(DType::Primitive(PType::U8, Nullability::NonNullable)), - padded_dim, - Nullability::NonNullable, - ); - let rotation_signs = children.get(2, &signs_dtype, signs_len)?; - - Ok(ArrayParts::new( - TurboQuant, - dtype.clone(), - len, - TurboQuantData { - dimension: dimensions, - bit_width, - num_rounds, - }, - ) - .with_slots(TurboQuantData::make_slots( - codes_array, - centroids, - rotation_signs, - ))) - } - - fn slot_name(_array: ArrayView, idx: usize) -> String { - Slot::from_index(idx).name().to_string() - } - fn execute(array: Array, ctx: &mut ExecutionCtx) -> VortexResult { - Ok(ExecutionResult::done(execute_decompress(array, ctx)?)) - } - - fn execute_parent( - array: ArrayView, - parent: &ArrayRef, - child_idx: usize, - ctx: &mut ExecutionCtx, - ) -> VortexResult> { - PARENT_KERNELS.execute(array, parent, child_idx, ctx) - } - - fn reduce_parent( - array: ArrayView, - parent: &ArrayRef, - child_idx: usize, - ) -> VortexResult> { - RULES.evaluate(array, parent, child_idx) - } -} - -impl ValidityVTable for TurboQuant { - fn validity(_array: ArrayView<'_, TurboQuant>) -> VortexResult { - // TurboQuant arrays are always non-nullable. This method is only called when the dtype is - // nullable, which should never happen for TQ arrays. - Ok(Validity::NonNullable) - } -} - -impl ArrayHash for TurboQuantData { - fn array_hash(&self, state: &mut H, _precision: Precision) { - self.dimension.hash(state); - self.bit_width.hash(state); - self.num_rounds.hash(state); - } -} - -impl ArrayEq for TurboQuantData { - fn array_eq(&self, other: &Self, _precision: Precision) -> bool { - self.dimension == other.dimension - && self.bit_width == other.bit_width - && self.num_rounds == other.num_rounds - } -} diff --git a/vortex-tensor/src/lib.rs b/vortex-tensor/src/lib.rs index a4577f6d262..6be34c0c41d 100644 --- a/vortex-tensor/src/lib.rs +++ b/vortex-tensor/src/lib.rs @@ -5,38 +5,84 @@ //! including unit vectors, spherical coordinates, and similarity measures such as cosine //! similarity. +#![cfg_attr( + test, + allow(clippy::unwrap_used, clippy::expect_used, clippy::unwrap_in_result) +)] + +use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin; use vortex_array::dtype::session::DTypeSessionExt; use vortex_array::scalar_fn::session::ScalarFnSessionExt; use vortex_array::session::ArraySessionExt; use vortex_session::VortexSession; -use crate::encodings::turboquant::TurboQuant; -use crate::fixed_shape::FixedShapeTensor; use crate::scalar_fns::cosine_similarity::CosineSimilarity; use crate::scalar_fns::inner_product::InnerProduct; use crate::scalar_fns::l2_denorm::L2Denorm; use crate::scalar_fns::l2_norm::L2Norm; -use crate::vector::Vector; +use crate::scalar_fns::sorf_transform::SorfTransform; +use crate::types::fixed_shape_tensor::FixedShapeTensor; +use crate::types::vector::Vector; pub mod matcher; pub mod scalar_fns; -pub mod fixed_shape; -pub mod vector; +mod types; + +pub use types::fixed_shape_tensor; +pub use types::vector; pub mod encodings; +pub mod vector_search; + mod utils; +/// Environment variable that gates registration of the tensor scalar-fn array plugins (the array +/// encodings that let [`CosineSimilarity`], [`InnerProduct`], [`L2Denorm`], [`L2Norm`], and +/// [`SorfTransform`] persist in a Vortex file). When unset, only the scalar functions themselves +/// are registered; readers of files containing serialized tensor scalar-fn arrays will fail to +/// deserialize. Opt-in by setting the variable to any non-empty value. +pub const SCALAR_FN_ARRAY_TENSOR_PLUGIN_ENV: &str = "VX_SCALAR_FN_ARRAY_TENSOR_PLUGIN"; + /// Initialize the Vortex tensor library with a Vortex session. pub fn initialize(session: &VortexSession) { session.dtypes().register(Vector); session.dtypes().register(FixedShapeTensor); - session.arrays().register(TurboQuant); + let session_fns = session.scalar_fns(); + + session_fns.register(CosineSimilarity); + session_fns.register(InnerProduct); + session_fns.register(L2Denorm); + session_fns.register(L2Norm); + session_fns.register(SorfTransform); + + // Registering the scalar-fn array plugins lets the tensor scalar fns be serialized as array + // encodings inside Vortex files. Gate this on an env var so applications that do not intend + // to persist these encodings do not pay the registry cost or widen their stable-encoding + // surface unintentionally. + if std::env::var_os(SCALAR_FN_ARRAY_TENSOR_PLUGIN_ENV).is_some_and(|v| !v.is_empty()) { + let session_arrays = session.arrays(); + + session_arrays.register(ScalarFnArrayPlugin::new(CosineSimilarity)); + session_arrays.register(ScalarFnArrayPlugin::new(InnerProduct)); + session_arrays.register(ScalarFnArrayPlugin::new(L2Denorm)); + session_arrays.register(ScalarFnArrayPlugin::new(L2Norm)); + session_arrays.register(ScalarFnArrayPlugin::new(SorfTransform)); + } +} + +#[cfg(test)] +mod tests { + use std::sync::LazyLock; + + use vortex_array::session::ArraySession; + use vortex_session::VortexSession; - session.scalar_fns().register(CosineSimilarity); - session.scalar_fns().register(InnerProduct); - session.scalar_fns().register(L2Denorm); - session.scalar_fns().register(L2Norm); + pub static SESSION: LazyLock = LazyLock::new(|| { + let session = VortexSession::empty().with::(); + crate::initialize(&session); + session + }); } diff --git a/vortex-tensor/src/matcher.rs b/vortex-tensor/src/matcher.rs index 973562be3da..10aa2581d03 100644 --- a/vortex-tensor/src/matcher.rs +++ b/vortex-tensor/src/matcher.rs @@ -7,10 +7,10 @@ use vortex_array::dtype::PType; use vortex_array::dtype::extension::ExtDTypeRef; use vortex_array::dtype::extension::Matcher; -use crate::fixed_shape::AnyFixedShapeTensor; -use crate::fixed_shape::FixedShapeTensorMatcherMetadata; -use crate::vector::AnyVector; -use crate::vector::VectorMatcherMetadata; +use crate::types::fixed_shape_tensor::AnyFixedShapeTensor; +use crate::types::fixed_shape_tensor::FixedShapeTensorMatcherMetadata; +use crate::types::vector::AnyVector; +use crate::types::vector::VectorMatcherMetadata; /// Matcher for any tensor-like extension type. /// @@ -23,7 +23,7 @@ pub struct AnyTensor; /// The matched variant of a tensor-like extension type. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum TensorMatch<'a> { - /// A [`FixedShapeTensor`](crate::fixed_shape::FixedShapeTensor) extension type. + /// A [`FixedShapeTensor`](crate::fixed_shape_tensor::FixedShapeTensor) extension type. FixedShapeTensor(FixedShapeTensorMatcherMetadata<'a>), /// A [`Vector`](crate::vector::Vector) extension type. @@ -42,10 +42,10 @@ impl TensorMatch<'_> { } /// Returns the flattened element count for each logical tensor row. - pub fn list_size(self) -> usize { + pub fn list_size(self) -> u32 { match self { - Self::FixedShapeTensor(metadata) => metadata.list_size(), - Self::Vector(metadata) => metadata.dimensions() as usize, + Self::FixedShapeTensor(metadata) => metadata.flat_list_size(), + Self::Vector(metadata) => metadata.dimensions(), } } } diff --git a/vortex-tensor/src/scalar_fns/cosine_similarity.rs b/vortex-tensor/src/scalar_fns/cosine_similarity.rs index 39c8b135e43..87f8c27ae6e 100644 --- a/vortex-tensor/src/scalar_fns/cosine_similarity.rs +++ b/vortex-tensor/src/scalar_fns/cosine_similarity.rs @@ -11,8 +11,9 @@ use vortex_array::ExecutionCtx; use vortex_array::IntoArray; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::ScalarFnArray; -use vortex_array::arrays::scalar_fn::ExactScalarFn; -use vortex_array::builtins::ArrayBuiltins; +use vortex_array::arrays::scalar_fn::ScalarFnArrayView; +use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayParts; +use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayVTable; use vortex_array::dtype::DType; use vortex_array::dtype::Nullability; use vortex_array::expr::Expression; @@ -20,21 +21,23 @@ use vortex_array::expr::and; use vortex_array::match_each_float_ptype; use vortex_array::scalar_fn::Arity; use vortex_array::scalar_fn::ChildName; +use vortex_array::scalar_fn::EmptyOptions; use vortex_array::scalar_fn::ExecutionArgs; -use vortex_array::scalar_fn::ScalarFn; use vortex_array::scalar_fn::ScalarFnId; use vortex_array::scalar_fn::ScalarFnVTable; -use vortex_array::validity::Validity; +use vortex_array::scalar_fn::TypedScalarFnInstance; +use vortex_array::serde::ArrayChildren; use vortex_buffer::Buffer; use vortex_error::VortexResult; -use vortex_error::vortex_ensure; +use vortex_session::VortexSession; -use crate::scalar_fns::ApproxOptions; use crate::scalar_fns::inner_product::InnerProduct; -use crate::scalar_fns::l2_denorm::L2Denorm; +use crate::scalar_fns::l2_denorm::DenormOrientation; +use crate::scalar_fns::l2_denorm::try_build_constant_l2_denorm; use crate::scalar_fns::l2_norm::L2Norm; +use crate::utils::BinaryTensorOpMetadata; use crate::utils::extract_l2_denorm_children; -use crate::utils::validate_tensor_float_input; +use crate::utils::validate_binary_tensor_float_inputs; /// Cosine similarity between two columns. /// @@ -45,16 +48,21 @@ use crate::utils::validate_tensor_float_input; /// Both inputs must be tensor-like extension arrays ([`FixedShapeTensor`] or [`Vector`]) with the /// same dtype and a float element type. The output is a float column of the same float type. /// -/// [`FixedShapeTensor`]: crate::fixed_shape::FixedShapeTensor +/// When either input is wrapped in [`L2Denorm`], this operator treats the stored norms and +/// normalized children as authoritative. For lossy encodings such as TurboQuant, that means the +/// optimized readthrough path may intentionally differ slightly from decoding both sides to dense +/// coordinates and recomputing cosine from scratch. +/// +/// [`FixedShapeTensor`]: crate::fixed_shape_tensor::FixedShapeTensor /// [`Vector`]: crate::vector::Vector +/// [`L2Denorm`]: crate::scalar_fns::l2_denorm::L2Denorm #[derive(Clone)] pub struct CosineSimilarity; impl CosineSimilarity { - /// Creates a new [`ScalarFn`] wrapping the cosine similarity operation with the given - /// [`ApproxOptions`] controlling approximation behavior. - pub fn new(options: &ApproxOptions) -> ScalarFn { - ScalarFn::new(CosineSimilarity, options.clone()) + /// Creates a new [`TypedScalarFnInstance`] wrapping the cosine similarity operation. + pub fn new() -> TypedScalarFnInstance { + TypedScalarFnInstance::new(CosineSimilarity, EmptyOptions) } /// Constructs a [`ScalarFnArray`] that lazily computes the cosine similarity between `lhs` and @@ -64,21 +72,16 @@ impl CosineSimilarity { /// /// Returns an error if the [`ScalarFnArray`] cannot be constructed (e.g. due to dtype /// mismatches). - pub fn try_new_array( - options: &ApproxOptions, - lhs: ArrayRef, - rhs: ArrayRef, - len: usize, - ) -> VortexResult { - ScalarFnArray::try_new(CosineSimilarity::new(options).erased(), vec![lhs, rhs], len) + pub fn try_new_array(lhs: ArrayRef, rhs: ArrayRef, len: usize) -> VortexResult { + ScalarFnArray::try_new(CosineSimilarity::new().erased(), vec![lhs, rhs], len) } } impl ScalarFnVTable for CosineSimilarity { - type Options = ApproxOptions; + type Options = EmptyOptions; fn id(&self) -> ScalarFnId { - ScalarFnId::from("vortex.tensor.cosine_similarity") + ScalarFnId::new("vortex.tensor.cosine_similarity") } fn arity(&self, _options: &Self::Options) -> Arity { @@ -110,23 +113,15 @@ impl ScalarFnVTable for CosineSimilarity { let lhs = &arg_dtypes[0]; let rhs = &arg_dtypes[1]; - // Both must have the same dtype (ignoring top-level nullability). - vortex_ensure!( - lhs.eq_ignore_nullability(rhs), - "CosineSimilarity requires both inputs to have the same dtype, got {lhs} and {rhs}" - ); - - // We don't need to look at rhs anymore since we know lhs and rhs are equal. - let tensor_match = validate_tensor_float_input(lhs)?; + let tensor_match = validate_binary_tensor_float_inputs(lhs, rhs)?; let ptype = tensor_match.element_ptype(); - let nullability = Nullability::from(lhs.is_nullable() || rhs.is_nullable()); Ok(DType::Primitive(ptype, nullability)) } fn execute( &self, - options: &Self::Options, + _options: &Self::Options, args: &dyn ExecutionArgs, ctx: &mut ExecutionCtx, ) -> VortexResult { @@ -134,28 +129,34 @@ impl ScalarFnVTable for CosineSimilarity { let mut rhs_ref = args.get(1)?; let len = args.row_count(); - // Check if any of our children have be already normalized. - { - let lhs_is_denorm = lhs_ref.is::>(); - let rhs_is_denorm = rhs_ref.is::>(); - - if lhs_is_denorm && rhs_is_denorm { - return self.execute_both_denorm(options, &lhs_ref, &rhs_ref, len, ctx); - } else if lhs_is_denorm || rhs_is_denorm { - if rhs_is_denorm { - (lhs_ref, rhs_ref) = (rhs_ref, lhs_ref); - } - return self.execute_one_denorm(options, &lhs_ref, &rhs_ref, len, ctx); + // If either side is a constant tensor-like extension array, eagerly normalize the single + // stored row and re-wrap it as an `L2Denorm` whose children are both `ConstantArray`s. + // The L2Denorm fast path below then picks it up. + if let Some(sfn) = try_build_constant_l2_denorm(&lhs_ref, len, ctx)? { + lhs_ref = sfn.into_array(); + } + if let Some(sfn) = try_build_constant_l2_denorm(&rhs_ref, len, ctx)? { + rhs_ref = sfn.into_array(); + } + + // Take any L2Denorm-wrapped fast path that applies. + match DenormOrientation::classify(&lhs_ref, &rhs_ref) { + DenormOrientation::Both { lhs, rhs } => { + return self.execute_both_denorm(lhs, rhs, len, ctx); + } + DenormOrientation::One { denorm, plain } => { + return self.execute_one_denorm(denorm, plain, len, ctx); } + DenormOrientation::Neither => {} } // Compute combined validity. let validity = lhs_ref.validity()?.and(rhs_ref.validity()?)?; // Compute inner product and norms as columnar operations, and propagate the options. - let norm_lhs_arr = L2Norm::try_new_array(options, lhs_ref.clone(), len)?; - let norm_rhs_arr = L2Norm::try_new_array(options, rhs_ref.clone(), len)?; - let dot_arr = InnerProduct::try_new_array(options, lhs_ref, rhs_ref, len)?; + let norm_lhs_arr = L2Norm::try_new_array(lhs_ref.clone(), len)?; + let norm_rhs_arr = L2Norm::try_new_array(rhs_ref.clone(), len)?; + let dot_arr = InnerProduct::try_new_array(lhs_ref, rhs_ref, len)?; // Execute to get the inner product and norms of the arrays. We only fully decompress // because we need to perform special logic (guard against 0) during division. @@ -207,39 +208,81 @@ impl ScalarFnVTable for CosineSimilarity { } } +impl ScalarFnArrayVTable for CosineSimilarity { + fn serialize( + &self, + view: &ScalarFnArrayView, + _session: &VortexSession, + ) -> VortexResult>> { + Ok(Some(BinaryTensorOpMetadata::encode_from_view(view)?)) + } + + fn deserialize( + &self, + _dtype: &DType, + len: usize, + metadata: &[u8], + children: &dyn ArrayChildren, + session: &VortexSession, + ) -> VortexResult> { + let reconstructed = + BinaryTensorOpMetadata::decode_children(metadata, len, children, session)?; + Ok(ScalarFnArrayParts { + options: EmptyOptions, + children: reconstructed, + }) + } +} + impl CosineSimilarity { - /// Both sides are `L2Denorm`: norms cancel, so `cosine_similarity = dot(n_l, n_r)`. + /// Both sides are `L2Denorm`: treat the normalized children as authoritative, so + /// `cosine_similarity = dot(n_l, n_r)`. fn execute_both_denorm( &self, - options: &ApproxOptions, lhs_ref: &ArrayRef, rhs_ref: &ArrayRef, len: usize, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { let validity = lhs_ref.validity()?.and(rhs_ref.validity()?)?; - let (normalized_l, _) = extract_l2_denorm_children(lhs_ref); - let (normalized_r, _) = extract_l2_denorm_children(rhs_ref); + let (normalized_l, norms_l) = extract_l2_denorm_children(lhs_ref); + let (normalized_r, norms_r) = extract_l2_denorm_children(rhs_ref); - // Dot product of already-normalized children IS the cosine similarity. - let dot = - InnerProduct::try_new_array(options, normalized_l, normalized_r, len)?.into_array(); + // `L2Denorm` makes the normalized children authoritative, so their dot product is the + // cosine similarity even for lossy storage wrappers, except that a zero stored norm still + // represents a zero vector. + let dot: PrimitiveArray = InnerProduct::try_new_array(normalized_l, normalized_r, len)? + .into_array() + .execute(ctx)?; + let norms_l: PrimitiveArray = norms_l.execute(ctx)?; + let norms_r: PrimitiveArray = norms_r.execute(ctx)?; - if !matches!(validity, Validity::NonNullable) { - // Masking always changes the nullability to nullable. - dot.mask(validity.to_array(len)) - } else { - Ok(dot) - } + match_each_float_ptype!(dot.ptype(), |T| { + let dots = dot.as_slice::(); + let norms_l = norms_l.as_slice::(); + let norms_r = norms_r.as_slice::(); + let buffer: Buffer = (0..len) + .map(|i| { + if norms_l[i] == T::zero() || norms_r[i] == T::zero() { + T::zero() + } else { + dots[i] + } + }) + .collect(); + + // SAFETY: The buffer length equals `len`, which matches the source validity length. + Ok(unsafe { PrimitiveArray::new_unchecked(buffer, validity) }.into_array()) + }) } - /// One side is `L2Denorm`: `cosine_similarity = dot(n, b) / ||b||`. + /// One side is `L2Denorm`: treat the normalized child as authoritative, so + /// `cosine_similarity = dot(n, b) / ||b||`. /// /// The caller must pass the denorm array as `denorm_ref` and the plain array as `plain_ref`. fn execute_one_denorm( &self, - options: &ApproxOptions, denorm_ref: &ArrayRef, plain_ref: &ArrayRef, len: usize, @@ -247,24 +290,28 @@ impl CosineSimilarity { ) -> VortexResult { let validity = denorm_ref.validity()?.and(plain_ref.validity()?)?; - let (normalized, _) = extract_l2_denorm_children(denorm_ref); + let (normalized, denorm_norms) = extract_l2_denorm_children(denorm_ref); - let dot_arr = InnerProduct::try_new_array(options, normalized, plain_ref.clone(), len)?; - let norm_arr = L2Norm::try_new_array(options, plain_ref.clone(), len)?; + let dot_arr = InnerProduct::try_new_array(normalized, plain_ref.clone(), len)?; let dot: PrimitiveArray = dot_arr.into_array().execute(ctx)?; + + let denorm_norms: PrimitiveArray = denorm_norms.execute(ctx)?; + + let norm_arr = L2Norm::try_new_array(plain_ref.clone(), len)?; let plain_norm: PrimitiveArray = norm_arr.into_array().execute(ctx)?; // TODO(connor): Ideally we would have a `SafeDiv` binary numeric operation. // TODO(connor): This can be written in a more SIMD-friendly manner. match_each_float_ptype!(dot.ptype(), |T| { let dots = dot.as_slice::(); - let norms = plain_norm.as_slice::(); + let denorm_norms = denorm_norms.as_slice::(); + let plain_norms = plain_norm.as_slice::(); let buffer: Buffer = (0..len) .map(|i| { - if norms[i] == T::zero() { + if denorm_norms[i] == T::zero() || plain_norms[i] == T::zero() { T::zero() } else { - dots[i] / norms[i] + dots[i] / plain_norms[i] } }) .collect(); @@ -277,36 +324,32 @@ impl CosineSimilarity { #[cfg(test)] mod tests { - use std::sync::LazyLock; use rstest::rstest; + use vortex_array::ArrayPlugin; use vortex_array::ArrayRef; use vortex_array::IntoArray; use vortex_array::VortexSessionExecute; use vortex_array::arrays::MaskedArray; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::ScalarFnArray; - use vortex_array::scalar_fn::ScalarFn; - use vortex_array::session::ArraySession; + use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin; use vortex_array::validity::Validity; use vortex_error::VortexResult; - use vortex_session::VortexSession; - use crate::scalar_fns::ApproxOptions; use crate::scalar_fns::cosine_similarity::CosineSimilarity; use crate::scalar_fns::l2_denorm::L2Denorm; + use crate::tests::SESSION; + use crate::types::vector::Vector; use crate::utils::test_helpers::assert_close; use crate::utils::test_helpers::constant_tensor_array; - use crate::utils::test_helpers::constant_vector_array; + use crate::utils::test_helpers::l2_denorm_array; use crate::utils::test_helpers::tensor_array; use crate::utils::test_helpers::vector_array; - static SESSION: LazyLock = - LazyLock::new(|| VortexSession::empty().with::()); - /// Evaluates cosine similarity between two tensor arrays and returns the result as `Vec`. fn eval_cosine_similarity(lhs: ArrayRef, rhs: ArrayRef, len: usize) -> VortexResult> { - let scalar_fn = ScalarFn::new(CosineSimilarity, ApproxOptions::Exact).erased(); + let scalar_fn = CosineSimilarity::new().erased(); let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs], len)?; let mut ctx = SESSION.create_execution_ctx(); let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; @@ -461,7 +504,7 @@ mod tests { 1.0, 0.0, 0.0, // vector 3 ], )?; - let query = constant_vector_array(&[1.0, 0.0, 0.0], 4)?; + let query = Vector::constant_array(&[1.0, 0.0, 0.0], 4)?; assert_close( &eval_cosine_similarity(data, query, 4)?, @@ -477,40 +520,25 @@ mod tests { let rhs = tensor_array(&[2], &[3.0, 4.0, 0.0, 1.0])?; let rhs = MaskedArray::try_new(rhs, Validity::from_iter([true, false]))?.into_array(); - let scalar_fn = ScalarFn::new(CosineSimilarity, ApproxOptions::Exact).erased(); + let scalar_fn = CosineSimilarity::new().erased(); let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs], 2)?; let mut ctx = SESSION.create_execution_ctx(); let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; // Row 0: self-similarity = 1.0, row 1: null. - assert!(prim.is_valid(0)?); - assert!(!prim.is_valid(1)?); + assert!(prim.is_valid(0, &mut ctx)?); + assert!(!prim.is_valid(1, &mut ctx)?); assert_close(&[prim.as_slice::()[0]], &[1.0]); Ok(()) } - /// Creates an `L2Denorm` scalar function array from pre-normalized elements and norms. - fn l2_denorm_array( - shape: &[usize], - normalized_elements: &[f64], - norms: &[f64], - ) -> VortexResult { - let len = norms.len(); - let normalized = tensor_array(shape, normalized_elements)?; - let norms = PrimitiveArray::from_iter(norms.iter().copied()).into_array(); - let mut ctx = SESSION.create_execution_ctx(); - Ok( - L2Denorm::try_new_array(&ApproxOptions::Exact, normalized, norms, len, &mut ctx)? - .into_array(), - ) - } - #[test] fn both_denorm_self_similarity() -> VortexResult<()> { // [3.0, 4.0] has norm 5.0, normalized [0.6, 0.8]. // [1.0, 0.0] has norm 1.0, normalized [1.0, 0.0]. - let lhs = l2_denorm_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0])?; - let rhs = l2_denorm_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0])?; + let mut ctx = SESSION.create_execution_ctx(); + let lhs = l2_denorm_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; + let rhs = l2_denorm_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; // Self-similarity should always be 1.0. assert_close(&eval_cosine_similarity(lhs, rhs, 2)?, &[1.0, 1.0]); @@ -521,8 +549,9 @@ mod tests { fn both_denorm_orthogonal() -> VortexResult<()> { // [3.0, 0.0] normalized [1.0, 0.0], norm 3.0. // [0.0, 4.0] normalized [0.0, 1.0], norm 4.0. - let lhs = l2_denorm_array(&[2], &[1.0, 0.0], &[3.0])?; - let rhs = l2_denorm_array(&[2], &[0.0, 1.0], &[4.0])?; + let mut ctx = SESSION.create_execution_ctx(); + let lhs = l2_denorm_array(&[2], &[1.0, 0.0], &[3.0], &mut ctx)?; + let rhs = l2_denorm_array(&[2], &[0.0, 1.0], &[4.0], &mut ctx)?; assert_close(&eval_cosine_similarity(lhs, rhs, 1)?, &[0.0]); Ok(()) @@ -531,8 +560,9 @@ mod tests { #[test] fn both_denorm_zero_norm() -> VortexResult<()> { // Zero-norm row: normalized is [0.0, 0.0], norm is 0.0. - let lhs = l2_denorm_array(&[2], &[0.6, 0.8, 0.0, 0.0], &[5.0, 0.0])?; - let rhs = l2_denorm_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0])?; + let mut ctx = SESSION.create_execution_ctx(); + let lhs = l2_denorm_array(&[2], &[0.6, 0.8, 0.0, 0.0], &[5.0, 0.0], &mut ctx)?; + let rhs = l2_denorm_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; // Row 0: dot([0.6, 0.8], [0.6, 0.8]) = 1.0, row 1: dot([0,0], [1,0]) = 0.0. assert_close(&eval_cosine_similarity(lhs, rhs, 2)?, &[1.0, 0.0]); @@ -544,7 +574,8 @@ mod tests { // LHS is L2Denorm([0.6, 0.8], 5.0) representing [3.0, 4.0]. // RHS is plain [3.0, 4.0]. // cosine_similarity([3.0, 4.0], [3.0, 4.0]) = 1.0. - let lhs = l2_denorm_array(&[2], &[0.6, 0.8], &[5.0])?; + let mut ctx = SESSION.create_execution_ctx(); + let lhs = l2_denorm_array(&[2], &[0.6, 0.8], &[5.0], &mut ctx)?; let rhs = tensor_array(&[2], &[3.0, 4.0])?; assert_close(&eval_cosine_similarity(lhs, rhs, 1)?, &[1.0]); @@ -555,8 +586,9 @@ mod tests { fn one_side_denorm_rhs() -> VortexResult<()> { // LHS is plain [1.0, 0.0], RHS is L2Denorm([0.6, 0.8], 5.0) representing [3.0, 4.0]. // cosine_similarity([1.0, 0.0], [3.0, 4.0]) = 3.0 / (1.0 * 5.0) = 0.6. + let mut ctx = SESSION.create_execution_ctx(); let lhs = tensor_array(&[2], &[1.0, 0.0])?; - let rhs = l2_denorm_array(&[2], &[0.6, 0.8], &[5.0])?; + let rhs = l2_denorm_array(&[2], &[0.6, 0.8], &[5.0], &mut ctx)?; assert_close(&eval_cosine_similarity(lhs, rhs, 1)?, &[0.6]); Ok(()) @@ -565,22 +597,218 @@ mod tests { #[test] fn both_denorm_null_norms() -> VortexResult<()> { // Row 0: valid, row 1: null (via nullable norms on rhs). - let lhs = l2_denorm_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0])?; + let mut ctx = SESSION.create_execution_ctx(); + let lhs = l2_denorm_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; let normalized_r = tensor_array(&[2], &[0.6, 0.8, 1.0, 0.0])?; let norms_r = PrimitiveArray::from_option_iter([Some(5.0f64), None]).into_array(); - let mut ctx = SESSION.create_execution_ctx(); - let rhs = - L2Denorm::try_new_array(&ApproxOptions::Exact, normalized_r, norms_r, 2, &mut ctx)? - .into_array(); + let rhs = L2Denorm::try_new_array(normalized_r, norms_r, 2, &mut ctx)?.into_array(); - let scalar_fn = ScalarFn::new(CosineSimilarity, ApproxOptions::Exact).erased(); + let scalar_fn = CosineSimilarity::new().erased(); let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs], 2)?; let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; - assert!(prim.is_valid(0)?); - assert!(!prim.is_valid(1)?); + assert!(prim.is_valid(0, &mut ctx)?); + assert!(!prim.is_valid(1, &mut ctx)?); assert_close(&[prim.as_slice::()[0]], &[1.0]); Ok(()) } + + #[test] + fn both_denorm_lossy_zero_stored_norm_returns_zero() -> VortexResult<()> { + // Mimics a lossy encoding (e.g. TurboQuant) where the stored norm is authoritative but + // the decoded normalized child is physically nonzero. With a stored norm of `0.0`, cosine + // similarity for that row must be `0.0` even though the dot product of the normalized + // children is nonzero. + let normalized_l = tensor_array(&[2], &[0.6, 0.8])?; + let norms_l = PrimitiveArray::from_iter([0.0f64]).into_array(); + // SAFETY: This is a focused test that intentionally violates the unit-norm invariant by + // pairing a nonzero normalized row with a stored norm of `0.0`, mimicking lossy storage. + let lhs = unsafe { L2Denorm::new_array_unchecked(normalized_l, norms_l, 1)? }.into_array(); + + let normalized_r = tensor_array(&[2], &[0.6, 0.8])?; + let norms_r = PrimitiveArray::from_iter([0.0f64]).into_array(); + // SAFETY: Same as above for the rhs operand. + let rhs = unsafe { L2Denorm::new_array_unchecked(normalized_r, norms_r, 1)? }.into_array(); + + // `dot(normalized_l, normalized_r) = 1.0`, but the authoritative stored norms are both + // `0.0`, so cosine similarity must be `0.0`. + assert_close(&eval_cosine_similarity(lhs, rhs, 1)?, &[0.0]); + Ok(()) + } + + #[test] + fn one_side_denorm_lossy_zero_stored_norm_returns_zero() -> VortexResult<()> { + // Mimics a lossy encoding (e.g. TurboQuant) where the stored norm is authoritative but + // the decoded normalized child is physically nonzero. The plain side is a normal nonzero + // tensor with positive norm. cosine similarity must still be `0.0` because the + // authoritative stored norm on the denorm side is `0.0`. + let normalized = tensor_array(&[2], &[0.6, 0.8])?; + let norms = PrimitiveArray::from_iter([0.0f64]).into_array(); + // SAFETY: This is a focused test that intentionally pairs a nonzero normalized row with a + // stored norm of `0.0`, mimicking lossy storage where the stored norm is authoritative. + let denorm = unsafe { L2Denorm::new_array_unchecked(normalized, norms, 1)? }.into_array(); + + let plain = tensor_array(&[2], &[1.0, 0.0])?; + + // Denorm on the lhs: `One { denorm: lhs, plain: rhs }`. + assert_close( + &eval_cosine_similarity(denorm.clone(), plain.clone(), 1)?, + &[0.0], + ); + + // Denorm on the rhs: `One { denorm: rhs, plain: lhs }`. The same zero-norm guard must + // fire regardless of operand order. + assert_close(&eval_cosine_similarity(plain, denorm, 1)?, &[0.0]); + Ok(()) + } + + #[test] + fn constant_lhs_matches_plain_tensor() -> VortexResult<()> { + // The constant query `[1, 2, 2]` has norm 3, so its normalized form is `[1/3, 2/3, 2/3]`. + // Expected cosine similarity against each row is `dot([1, 2, 2], row) / (3 * ||row||)`. + let lhs = constant_tensor_array(&[3], &[1.0, 2.0, 2.0], 4)?; + let rhs = tensor_array( + &[3], + &[ + 1.0, 0.0, 0.0, // dot=1, ||rhs||=1, expected=1/3 + 1.0, 2.0, 2.0, // dot=9, ||rhs||=3, expected=1 + 0.0, 0.0, 1.0, // dot=2, ||rhs||=1, expected=2/3 + 2.0, 1.0, 2.0, // dot=8, ||rhs||=3, expected=8/9 + ], + )?; + assert_close( + &eval_cosine_similarity(lhs, rhs, 4)?, + &[1.0 / 3.0, 1.0, 2.0 / 3.0, 8.0 / 9.0], + ); + Ok(()) + } + + #[test] + fn constant_rhs_matches_plain_tensor() -> VortexResult<()> { + // Mirror of `constant_lhs_matches_plain_tensor` with the constant on the right. + let lhs = tensor_array( + &[3], + &[ + 1.0, 0.0, 0.0, // + 1.0, 2.0, 2.0, // + 0.0, 0.0, 1.0, // + 2.0, 1.0, 2.0, // + ], + )?; + let rhs = constant_tensor_array(&[3], &[1.0, 2.0, 2.0], 4)?; + assert_close( + &eval_cosine_similarity(lhs, rhs, 4)?, + &[1.0 / 3.0, 1.0, 2.0 / 3.0, 8.0 / 9.0], + ); + Ok(()) + } + + #[test] + fn both_constant_tensors() -> VortexResult<()> { + // `[1, 0, 0]` vs `[1, 1, 0]`. dot=1, ||lhs||=1, ||rhs||=sqrt(2), expected=1/sqrt(2). + let lhs = constant_tensor_array(&[3], &[1.0, 0.0, 0.0], 3)?; + let rhs = constant_tensor_array(&[3], &[1.0, 1.0, 0.0], 3)?; + let expected = 1.0 / 2.0_f64.sqrt(); + assert_close( + &eval_cosine_similarity(lhs, rhs, 3)?, + &[expected, expected, expected], + ); + Ok(()) + } + + #[test] + fn constant_zero_norm_query() -> VortexResult<()> { + // A zero-norm constant query must produce `0.0` for every row via the zero-norm guard in + // `execute_one_denorm` and `execute_both_denorm`. + let lhs = constant_tensor_array(&[3], &[0.0, 0.0, 0.0], 3)?; + let rhs = tensor_array( + &[3], + &[ + 1.0, 2.0, 3.0, // + 4.0, 5.0, 6.0, // + 7.0, 8.0, 9.0, // + ], + )?; + assert_close(&eval_cosine_similarity(lhs, rhs, 3)?, &[0.0, 0.0, 0.0]); + Ok(()) + } + + #[test] + fn constant_self_similarity_nonunit() -> VortexResult<()> { + // A non-unit constant query compared to itself must produce `1.0`. This exercises the + // helper's division: after normalization, both sides must be exactly unit so the + // L2Denorm fast path's inner product yields 1. + let lhs = constant_tensor_array(&[3], &[3.0, 4.0, 0.0], 5)?; + let rhs = constant_tensor_array(&[3], &[3.0, 4.0, 0.0], 5)?; + assert_close(&eval_cosine_similarity(lhs, rhs, 5)?, &[1.0; 5]); + Ok(()) + } + + #[test] + fn vector_constant_matches_plain() -> VortexResult<()> { + // Exercise the `Vector` extension variant through the new pre-pass. + let lhs = Vector::constant_array(&[1.0, 2.0, 2.0], 4)?; + let rhs = vector_array( + 3, + &[ + 1.0, 0.0, 0.0, // + 1.0, 2.0, 2.0, // + 0.0, 0.0, 1.0, // + 2.0, 1.0, 2.0, // + ], + )?; + assert_close( + &eval_cosine_similarity(lhs, rhs, 4)?, + &[1.0 / 3.0, 1.0, 2.0 / 3.0, 8.0 / 9.0], + ); + Ok(()) + } + + #[rstest] + #[case::vector(cosine_vector_lhs(), cosine_vector_rhs(), 2)] + #[case::fixed_shape_tensor(cosine_tensor_lhs(), cosine_tensor_rhs(), 2)] + fn serde_round_trip( + #[case] lhs: ArrayRef, + #[case] rhs: ArrayRef, + #[case] len: usize, + ) -> VortexResult<()> { + let original = CosineSimilarity::try_new_array(lhs.clone(), rhs.clone(), len)?.into_array(); + + let plugin = ScalarFnArrayPlugin::new(CosineSimilarity); + let metadata = plugin + .serialize(&original, &SESSION)? + .expect("CosineSimilarity serialize must produce metadata"); + + let children = vec![lhs, rhs]; + let recovered = plugin.deserialize( + original.dtype(), + original.len(), + &metadata, + &[], + &children, + &SESSION, + )?; + + assert_eq!(recovered.dtype(), original.dtype()); + assert_eq!(recovered.len(), original.len()); + assert_eq!(recovered.encoding_id(), original.encoding_id()); + Ok(()) + } + + fn cosine_vector_lhs() -> ArrayRef { + vector_array(3, &[1.0, 0.0, 0.0, 3.0, 4.0, 0.0]).expect("valid vector array") + } + + fn cosine_vector_rhs() -> ArrayRef { + vector_array(3, &[0.0, 1.0, 0.0, 3.0, 4.0, 0.0]).expect("valid vector array") + } + + fn cosine_tensor_lhs() -> ArrayRef { + tensor_array(&[2], &[1.0, 0.0, 3.0, 4.0]).expect("valid tensor array") + } + + fn cosine_tensor_rhs() -> ArrayRef { + tensor_array(&[2], &[0.0, 1.0, 3.0, 4.0]).expect("valid tensor array") + } } diff --git a/vortex-tensor/src/scalar_fns/inner_product.rs b/vortex-tensor/src/scalar_fns/inner_product.rs index da1b62e6ca1..f34cb257dbd 100644 --- a/vortex-tensor/src/scalar_fns/inner_product.rs +++ b/vortex-tensor/src/scalar_fns/inner_product.rs @@ -9,34 +9,52 @@ use num_traits::Float; use vortex_array::ArrayRef; use vortex_array::ExecutionCtx; use vortex_array::IntoArray; +use vortex_array::arrays::Constant; +use vortex_array::arrays::ConstantArray; +use vortex_array::arrays::Dict; +use vortex_array::arrays::Extension; use vortex_array::arrays::ExtensionArray; +use vortex_array::arrays::FixedSizeList; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::ScalarFnArray; +use vortex_array::arrays::dict::DictArraySlotsExt; use vortex_array::arrays::extension::ExtensionArrayExt; +use vortex_array::arrays::fixed_size_list::FixedSizeListArrayExt; use vortex_array::arrays::scalar_fn::ExactScalarFn; +use vortex_array::arrays::scalar_fn::ScalarFnArrayView; +use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayParts; +use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayVTable; use vortex_array::dtype::DType; use vortex_array::dtype::NativePType; use vortex_array::dtype::Nullability; +use vortex_array::dtype::PType; use vortex_array::expr::Expression; use vortex_array::expr::and; use vortex_array::match_each_float_ptype; use vortex_array::scalar_fn::Arity; use vortex_array::scalar_fn::ChildName; +use vortex_array::scalar_fn::EmptyOptions; use vortex_array::scalar_fn::ExecutionArgs; -use vortex_array::scalar_fn::ScalarFn; use vortex_array::scalar_fn::ScalarFnId; use vortex_array::scalar_fn::ScalarFnVTable; +use vortex_array::scalar_fn::TypedScalarFnInstance; +use vortex_array::serde::ArrayChildren; use vortex_buffer::Buffer; +use vortex_buffer::BufferMut; use vortex_error::VortexExpect; use vortex_error::VortexResult; -use vortex_error::vortex_ensure; -use vortex_error::vortex_err; +use vortex_session::VortexSession; use crate::matcher::AnyTensor; -use crate::scalar_fns::ApproxOptions; -use crate::scalar_fns::l2_denorm::L2Denorm; +use crate::scalar_fns::l2_denorm::DenormOrientation; +use crate::scalar_fns::sorf_transform::SorfMatrix; +use crate::scalar_fns::sorf_transform::SorfTransform; +use crate::types::vector::Vector; +use crate::utils::BinaryTensorOpMetadata; +use crate::utils::extract_constant_flat_row; use crate::utils::extract_flat_elements; use crate::utils::extract_l2_denorm_children; +use crate::utils::validate_binary_tensor_float_inputs; /// Inner product (dot product) between two columns. /// @@ -47,16 +65,15 @@ use crate::utils::extract_l2_denorm_children; /// Both inputs must be tensor-like extension arrays ([`FixedShapeTensor`] or [`Vector`]) with the /// same dtype and a float element type. The output is a float column of the same float type. /// -/// [`FixedShapeTensor`]: crate::fixed_shape::FixedShapeTensor +/// [`FixedShapeTensor`]: crate::fixed_shape_tensor::FixedShapeTensor /// [`Vector`]: crate::vector::Vector #[derive(Clone)] pub struct InnerProduct; impl InnerProduct { - /// Creates a new [`ScalarFn`] wrapping the inner product operation with the given - /// [`ApproxOptions`] controlling approximation behavior. - pub fn new(options: &ApproxOptions) -> ScalarFn { - ScalarFn::new(InnerProduct, options.clone()) + /// Creates a new [`TypedScalarFnInstance`] wrapping the inner product operation. + pub fn new() -> TypedScalarFnInstance { + TypedScalarFnInstance::new(InnerProduct, EmptyOptions) } /// Constructs a [`ScalarFnArray`] that lazily computes the inner product between `lhs` and @@ -66,21 +83,16 @@ impl InnerProduct { /// /// Returns an error if the [`ScalarFnArray`] cannot be constructed (e.g. due to dtype /// mismatches). - pub fn try_new_array( - options: &ApproxOptions, - lhs: ArrayRef, - rhs: ArrayRef, - len: usize, - ) -> VortexResult { - ScalarFnArray::try_new(InnerProduct::new(options).erased(), vec![lhs, rhs], len) + pub fn try_new_array(lhs: ArrayRef, rhs: ArrayRef, len: usize) -> VortexResult { + ScalarFnArray::try_new(InnerProduct::new().erased(), vec![lhs, rhs], len) } } impl ScalarFnVTable for InnerProduct { - type Options = ApproxOptions; + type Options = EmptyOptions; fn id(&self) -> ScalarFnId { - ScalarFnId::from("vortex.tensor.inner_product") + ScalarFnId::new("vortex.tensor.inner_product") } fn arity(&self, _options: &Self::Options) -> Arity { @@ -112,59 +124,46 @@ impl ScalarFnVTable for InnerProduct { let lhs = &arg_dtypes[0]; let rhs = &arg_dtypes[1]; - // Both must have the same dtype (ignoring top-level nullability). - vortex_ensure!( - lhs.eq_ignore_nullability(rhs), - "InnerProduct requires both inputs to have the same dtype, got {lhs} and {rhs}" - ); - - // Both inputs must be tensor-like extension types. - let lhs_ext = lhs - .as_extension_opt() - .ok_or_else(|| vortex_err!("InnerProduct lhs must be an extension type, got {lhs}"))?; - - vortex_ensure!( - lhs_ext.is::(), - "InnerProduct inputs must be an `AnyTensor`, got {lhs}" - ); - - let tensor_match = lhs_ext - .metadata_opt::() - .ok_or_else(|| vortex_err!("InnerProduct inputs must be an `AnyTensor`, got {lhs}"))?; + // TODO(connor): relax the float-only gate once integer tensors are supported. + let tensor_match = validate_binary_tensor_float_inputs(lhs, rhs)?; let ptype = tensor_match.element_ptype(); - // TODO(connor): This should support integer tensors! - vortex_ensure!( - ptype.is_float(), - "InnerProduct element dtype must be a float primitive, got {ptype}" - ); - let nullability = Nullability::from(lhs.is_nullable() || rhs.is_nullable()); Ok(DType::Primitive(ptype, nullability)) } fn execute( &self, - options: &Self::Options, + _options: &Self::Options, args: &dyn ExecutionArgs, ctx: &mut ExecutionCtx, ) -> VortexResult { - let mut lhs_ref = args.get(0)?; - let mut rhs_ref = args.get(1)?; + let lhs_ref = args.get(0)?; + let rhs_ref = args.get(1)?; let len = args.row_count(); - // Check if any of our children have be already normalized. - { - let lhs_is_denorm = lhs_ref.is::>(); - let rhs_is_denorm = rhs_ref.is::>(); - - if lhs_is_denorm && rhs_is_denorm { - return self.execute_both_denorm(options, &lhs_ref, &rhs_ref, len, ctx); - } else if lhs_is_denorm || rhs_is_denorm { - if rhs_is_denorm { - (lhs_ref, rhs_ref) = (rhs_ref, lhs_ref); - } - return self.execute_one_denorm(options, &lhs_ref, &rhs_ref, len, ctx); + // Take any L2Denorm-wrapped fast path that applies. + match DenormOrientation::classify(&lhs_ref, &rhs_ref) { + DenormOrientation::Both { lhs, rhs } => { + return self.execute_both_denorm(lhs, rhs, len, ctx); } + DenormOrientation::One { denorm, plain } => { + return self.execute_one_denorm(denorm, plain, len, ctx); + } + DenormOrientation::Neither => {} + } + + // Reduction case 1: `InnerProduct(SorfTransform(x), const)` rewrites to + // `InnerProduct(x, forward_rotate(zero_pad(const)))`. Re-executes recursively so + // case 2 can fire on the rewritten tree. + if let Some(rewritten) = self.try_execute_sorf_constant(&lhs_ref, &rhs_ref, len, ctx)? { + return Ok(rewritten); + } + + // Reduction case 2: `InnerProduct(Vector[FSL(Dict(u8, f32))], const)` is computed by + // gather-summing `q[j] * values[codes[j] as usize]` per row, reading the codebook + // directly instead of decoding the column into dense vectors. + if let Some(result) = self.try_execute_dict_constant(&lhs_ref, &rhs_ref, len, ctx)? { + return Ok(result); } // Compute combined validity. @@ -179,7 +178,7 @@ impl ScalarFnVTable for InnerProduct { let tensor_match = ext .metadata_opt::() .vortex_expect("we already validated this in `return_dtype`"); - let dimensions = tensor_match.list_size(); + let dimensions = tensor_match.list_size() as usize; // Extract the storage array from each extension input. We pass the storage (FSL) rather // than the extension array to avoid canonicalizing the extension wrapper. @@ -221,11 +220,36 @@ impl ScalarFnVTable for InnerProduct { } } +impl ScalarFnArrayVTable for InnerProduct { + fn serialize( + &self, + view: &ScalarFnArrayView, + _session: &VortexSession, + ) -> VortexResult>> { + Ok(Some(BinaryTensorOpMetadata::encode_from_view(view)?)) + } + + fn deserialize( + &self, + _dtype: &DType, + len: usize, + metadata: &[u8], + children: &dyn ArrayChildren, + session: &VortexSession, + ) -> VortexResult> { + let reconstructed = + BinaryTensorOpMetadata::decode_children(metadata, len, children, session)?; + Ok(ScalarFnArrayParts { + options: EmptyOptions, + children: reconstructed, + }) + } +} + impl InnerProduct { /// Both sides are `L2Denorm`: `inner_product = s_l * s_r * dot(n_l, n_r)`. fn execute_both_denorm( &self, - options: &ApproxOptions, lhs_ref: &ArrayRef, rhs_ref: &ArrayRef, len: usize, @@ -239,10 +263,9 @@ impl InnerProduct { let norms_l: PrimitiveArray = norms_l.execute(ctx)?; let norms_r: PrimitiveArray = norms_r.execute(ctx)?; - let dot: PrimitiveArray = - InnerProduct::try_new_array(options, normalized_l, normalized_r, len)? - .into_array() - .execute(ctx)?; + let dot: PrimitiveArray = InnerProduct::try_new_array(normalized_l, normalized_r, len)? + .into_array() + .execute(ctx)?; match_each_float_ptype!(dot.ptype(), |T| { let dots = dot.as_slice::(); @@ -260,7 +283,6 @@ impl InnerProduct { /// The caller must pass the denorm array as `denorm_ref` and the plain array as `plain_ref`. fn execute_one_denorm( &self, - options: &ApproxOptions, denorm_ref: &ArrayRef, plain_ref: &ArrayRef, len: usize, @@ -271,10 +293,9 @@ impl InnerProduct { let (normalized, norms) = extract_l2_denorm_children(denorm_ref); let denorm_norms: PrimitiveArray = norms.execute(ctx)?; - let dot: PrimitiveArray = - InnerProduct::try_new_array(options, normalized, plain_ref.clone(), len)? - .into_array() - .execute(ctx)?; + let dot: PrimitiveArray = InnerProduct::try_new_array(normalized, plain_ref.clone(), len)? + .into_array() + .execute(ctx)?; match_each_float_ptype!(dot.ptype(), |T| { let dots = dot.as_slice::(); @@ -285,6 +306,206 @@ impl InnerProduct { Ok(unsafe { PrimitiveArray::new_unchecked(buffer, validity) }.into_array()) }) } + + /// Fast path when one side is `ExactScalarFn` and the other side is a + /// constant-backed tensor-like extension. Rewrites to + /// `InnerProduct(sorf_child, forward_rotate(zero_pad(const_query)))` because SORF is + /// orthogonal, so ` = ` where `T` is the truncation from + /// `padded_dim` to `dim` applied by `SorfTransform` and `R` is the SORF forward matrix. See the + /// proof in the crate-level docs and in the plan file. + /// + /// Returns `Ok(None)` if neither side matches, when the operand element type is not `F32`, or + /// when the constant side is not a constant-backed tensor extension. The caller is expected to + /// fall through to the standard path in that case. + /// + /// # F32-only + /// + /// TODO(connor): this rewrite is only sound for `PType::F32` because `SorfTransform` applies an + /// `f32 -> element_ptype` cast at the end of its `execute`. For `F16`/`F64` the cast changes + /// the inner product's rounding and the rewrite would not be semantically equivalent. Until we + /// push the cast through `InnerProduct`, both the SorfTransform output ptype and the + /// constant-side element ptype must be `F32` here. + fn try_execute_sorf_constant( + &self, + lhs_ref: &ArrayRef, + rhs_ref: &ArrayRef, + len: usize, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + // Identify which side is the SorfTransform, if any. + let (sorf_view, const_ref) = + if let Some(view) = lhs_ref.as_opt::>() { + (view, rhs_ref) + } else if let Some(view) = rhs_ref.as_opt::>() { + (view, lhs_ref) + } else { + return Ok(None); + }; + + if sorf_view.options.element_ptype != PType::F32 { + return Ok(None); + } + + // The other side must be a constant tensor. + let Some(const_storage) = constant_tensor_storage(const_ref) else { + return Ok(None); + }; + + let dim = sorf_view.options.dimensions as usize; + let num_rounds = sorf_view.options.num_rounds as usize; + let seed = sorf_view.options.seed; + let padded_dim = dim.next_power_of_two(); + + // Extract the single stored row of the constant. + let flat = extract_constant_flat_row(&const_storage, ctx)?; + if flat.ptype() != PType::F32 { + return Ok(None); + } + + // Zero-pad the query from `dim` to `padded_dim` and forward-rotate. + let mut padded_query = vec![0.0f32; padded_dim]; + padded_query[..dim].copy_from_slice(flat.as_slice::()); + + let rotation = SorfMatrix::try_new(seed, dim, num_rounds)?; + let mut rotated_query = vec![0.0f32; padded_dim]; + rotation.rotate(&padded_query, &mut rotated_query); + + // Wrap the rotated query as a `Vector` constant broadcast to `len` + // rows. The new extension dtype has `padded_dim` instead of `dim`, matching the + // SorfTransform child we are about to dot it with. + let new_constant = Vector::constant_array(&rotated_query, len)?; + + // Extract the SorfTransform child (the already-padded Vector). + let sorf_child = sorf_view + .nth_child(0) + .vortex_expect("SorfTransform must have exactly one child"); + + // Recursively execute the rewritten inner product. This allows case 2 to fire on + // the rewritten tree if the sorf child is `Vector[FSL(Dict)]`. Termination is + // guaranteed because the rewrite strictly removes a `SorfTransform` scalar-fn node + // from the tree and SORFs cannot be nested. + let rewritten = InnerProduct::try_new_array(sorf_child, new_constant, len)? + .into_array() + .execute(ctx)?; + Ok(Some(rewritten)) + } + + /// Fast path when one side is an extension whose storage is `FSL(Dict(u8, f32))` and + /// the other side is a constant-backed tensor extension with an F32 element ptype. + /// + /// Computes each row's inner product as + /// `out[i] = sum_{j in 0..padded_dim} q[j] * values[codes[i * padded_dim + j] as usize]` + /// using a direct codebook lookup in the hot loop. An explicit product table + /// `P[j, k] = q[j] * values[k]` (size `padded_dim * num_centroids * 4B`, ~1 MiB for the + /// common 1024/256 case) was tried and measured ~10% *slower* on the + /// `similarity_search` bench because the 1 KiB `values` table stays in L1 across all + /// rows, while the 1 MiB product table does not. + /// + /// Returns `Ok(None)` when the pattern doesn't match; the caller should fall through to + /// the standard path. + fn try_execute_dict_constant( + &self, + lhs_ref: &ArrayRef, + rhs_ref: &ArrayRef, + len: usize, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + // Try each orientation. The oriented helper navigates each side exactly once, so + // the only redundant work here is the failed navigation of the first side when the + // dict happens to be on the right. + if let Some(result) = self.try_execute_dict_constant_oriented(lhs_ref, rhs_ref, len, ctx)? { + return Ok(Some(result)); + } + self.try_execute_dict_constant_oriented(rhs_ref, lhs_ref, len, ctx) + } + + /// Orientation-specific helper for [`Self::try_execute_dict_constant`]. `dict_candidate` + /// is tried as `Extension[FSL[Dict]]`; `const_candidate` is tried as a constant-backed + /// tensor extension. Returns `Ok(None)` if either navigation fails or any gate rejects. + fn try_execute_dict_constant_oriented( + &self, + dict_candidate: &ArrayRef, + const_candidate: &ArrayRef, + len: usize, + ctx: &mut ExecutionCtx, + ) -> VortexResult> { + // Navigate the dict side. + let Some(dict_ext) = dict_candidate.as_opt::() else { + return Ok(None); + }; + let Some(fsl) = dict_ext.storage_array().as_opt::() else { + return Ok(None); + }; + let Some(dict) = fsl.elements().as_opt::() else { + return Ok(None); + }; + + // Navigate the constant side and require its scalar be non-null. + let Some(const_storage) = constant_tensor_storage(const_candidate) else { + return Ok(None); + }; + + // Canonicalize codes and values. Codes may be e.g. BitPacked; executing is cheaper + // than falling through to the standard path (which would also canonicalize). + let codes_prim: PrimitiveArray = dict.codes().clone().execute(ctx)?; + let values_prim: PrimitiveArray = dict.values().clone().execute(ctx)?; + + // Gate: u8 codes and f32 centroids. + if codes_prim.ptype() != PType::U8 { + // TODO(connor): Should we support wider codes? + return Ok(None); + } + if values_prim.ptype() != PType::F32 { + // TODO(connor): direct-lookup path only supports f32 centroids. SorfTransform + // forces f32 anyway, so this is the only shape we need for now. + return Ok(None); + } + + let padded_dim = usize::try_from(fsl.list_size()).vortex_expect("fsl list_size fits usize"); + + let flat = extract_constant_flat_row(&const_storage, ctx)?; + if flat.ptype() != PType::F32 { + // TODO(connor): case 2 is f32-only. For f16/f64 we fall through to the standard + // path, which computes the inner product with the correct element type. + return Ok(None); + } + + // Combine the input validities up front; the per-row arithmetic may write garbage + // into null rows but the validity mask hides it (matching the standard path). + let validity = dict_candidate + .validity()? + .and(const_candidate.validity()?)?; + + // Fast path for the empty case: skip allocating and touching the codes buffer. + if len == 0 { + let empty = PrimitiveArray::empty::(validity.nullability()); + return Ok(Some(empty.into_array())); + } + + let q: &[f32] = flat.as_slice::(); + debug_assert_eq!(q.len(), padded_dim); + let codes: &[u8] = codes_prim.as_slice::(); + let values: &[f32] = values_prim.as_slice::(); + debug_assert_eq!(codes.len(), len * padded_dim); + + // The hot loop is extracted into [`execute_dict_constant_inner_product`] so the compiler + // can prove the chunked indices stay in bounds and vectorize the inner gather-accumulate. + let out = execute_dict_constant_inner_product(q, values, codes, len, padded_dim); + + // SAFETY: the buffer length equals `len`, which matches the validity length. + let result = unsafe { PrimitiveArray::new_unchecked(out.freeze(), validity) }.into_array(); + Ok(Some(result)) + } +} + +/// Return the storage constant for a canonical tensor-like constant query. +fn constant_tensor_storage(array: &ArrayRef) -> Option { + let constant = array.as_opt::()?; + if constant.scalar().is_null() { + return None; + } + let ext_scalar = constant.scalar().as_extension_opt()?; + Some(ConstantArray::new(ext_scalar.to_storage_scalar(), array.len()).into_array()) } /// Computes the inner product (dot product) of two equal-length float slices. @@ -297,36 +518,75 @@ fn inner_product_row(a: &[T], b: &[T]) -> T { .fold(T::zero(), |acc, v| acc + v) } +/// Compute inner products between a constant query vector and dictionary-encoded rows. +/// +/// For each row, computes `sum(q[j] * values[codes[row * dim + j]])` using the codebook `values` +/// directly instead of decoding the dictionary into dense vectors. +/// +/// The inner loop uses `PARTIAL_SUMS` independent accumulators so the CPU can pipeline FP additions +/// instead of waiting for each `fadd` to retire before starting the next. +fn execute_dict_constant_inner_product( + q: &[f32], + values: &[f32], + codes: &[u8], + num_rows: usize, + dim: usize, +) -> BufferMut { + let mut out = BufferMut::::with_capacity(num_rows); + + const PARTIAL_SUMS: usize = 8; + + for row_codes in codes.chunks_exact(dim) { + let mut acc = [0.0f32; PARTIAL_SUMS]; + + let code_chunks = row_codes.chunks_exact(PARTIAL_SUMS); + let q_chunks = q.chunks_exact(PARTIAL_SUMS); + let code_rem = code_chunks.remainder(); + let q_rem = q_chunks.remainder(); + + for (cc, qd) in code_chunks.zip(q_chunks) { + for i in 0..PARTIAL_SUMS { + acc[i] = qd[i].mul_add(values[cc[i] as usize], acc[i]); + } + } + + for (&code, &q_val) in code_rem.iter().zip(q_rem.iter()) { + acc[0] = q_val.mul_add(values[code as usize], acc[0]); + } + + // SAFETY: we reserved `num_rows` slots and push exactly once per row. + unsafe { out.push_unchecked(acc.iter().sum::()) }; + } + + out +} + #[cfg(test)] mod tests { - use std::sync::LazyLock; use rstest::rstest; + use vortex_array::ArrayPlugin; use vortex_array::ArrayRef; use vortex_array::IntoArray; use vortex_array::VortexSessionExecute; use vortex_array::arrays::MaskedArray; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::ScalarFnArray; - use vortex_array::scalar_fn::ScalarFn; - use vortex_array::session::ArraySession; + use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin; use vortex_array::validity::Validity; use vortex_error::VortexResult; - use vortex_session::VortexSession; - use crate::scalar_fns::ApproxOptions; use crate::scalar_fns::inner_product::InnerProduct; use crate::scalar_fns::l2_denorm::L2Denorm; + use crate::tests::SESSION; use crate::utils::test_helpers::assert_close; + use crate::utils::test_helpers::l2_denorm_array; use crate::utils::test_helpers::tensor_array; use crate::utils::test_helpers::vector_array; - static SESSION: LazyLock = - LazyLock::new(|| VortexSession::empty().with::()); - /// Evaluates inner product between two tensor arrays and returns the result as `Vec`. fn eval_inner_product(lhs: ArrayRef, rhs: ArrayRef, len: usize) -> VortexResult> { - let scalar_fn = ScalarFn::new(InnerProduct, ApproxOptions::Exact).erased(); + let scalar_fn = InnerProduct::new().erased(); let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs], len)?; let mut ctx = SESSION.create_execution_ctx(); let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; @@ -404,15 +664,15 @@ mod tests { let rhs = tensor_array(&[2], &[7.0, 8.0, 9.0, 10.0, 11.0, 12.0])?; let lhs = MaskedArray::try_new(lhs, Validity::from_iter([true, false, true]))?.into_array(); - let scalar_fn = ScalarFn::new(InnerProduct, ApproxOptions::Exact).erased(); + let scalar_fn = InnerProduct::new().erased(); let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs], 3)?; let mut ctx = SESSION.create_execution_ctx(); let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; // Row 0: 1*7 + 2*8 = 23, row 1: null, row 2: 5*11 + 6*12 = 127. - assert!(prim.is_valid(0)?); - assert!(!prim.is_valid(1)?); - assert!(prim.is_valid(2)?); + assert!(prim.is_valid(0, &mut ctx)?); + assert!(!prim.is_valid(1, &mut ctx)?); + assert!(prim.is_valid(2, &mut ctx)?); assert_close(&[prim.as_slice::()[0]], &[23.0]); assert_close(&[prim.as_slice::()[2]], &[127.0]); Ok(()) @@ -422,7 +682,7 @@ mod tests { fn rejects_non_extension_dtype() { let lhs = PrimitiveArray::from_iter([1.0_f64, 2.0]).into_array(); let rhs = PrimitiveArray::from_iter([3.0_f64, 4.0]).into_array(); - let result = InnerProduct::try_new_array(&ApproxOptions::Exact, lhs, rhs, 2); + let result = InnerProduct::try_new_array(lhs, rhs, 2); assert!(result.is_err()); } @@ -430,36 +690,19 @@ mod tests { fn rejects_mismatched_dtypes() -> VortexResult<()> { let lhs = tensor_array(&[2], &[1.0_f64, 2.0])?; let rhs = vector_array(2, &[3.0_f64, 4.0])?; - let result = InnerProduct::try_new_array(&ApproxOptions::Exact, lhs, rhs, 1); + let result = InnerProduct::try_new_array(lhs, rhs, 1); assert!(result.is_err()); Ok(()) } - /// Creates an `L2Denorm` scalar function array from pre-normalized elements and norms. - fn l2_denorm_array( - shape: &[usize], - normalized_elements: &[f64], - norms: &[f64], - ) -> VortexResult { - use vortex_array::IntoArray; - - let len = norms.len(); - let normalized = tensor_array(shape, normalized_elements)?; - let norms = PrimitiveArray::from_iter(norms.iter().copied()).into_array(); - let mut ctx = SESSION.create_execution_ctx(); - Ok( - L2Denorm::try_new_array(&ApproxOptions::Exact, normalized, norms, len, &mut ctx)? - .into_array(), - ) - } - #[test] fn both_denorm() -> VortexResult<()> { // LHS: [3.0, 4.0] = L2Denorm([0.6, 0.8], 5.0). // RHS: [1.0, 0.0] = L2Denorm([1.0, 0.0], 1.0). // dot([3.0, 4.0], [1.0, 0.0]) = 3.0. - let lhs = l2_denorm_array(&[2], &[0.6, 0.8], &[5.0])?; - let rhs = l2_denorm_array(&[2], &[1.0, 0.0], &[1.0])?; + let mut ctx = SESSION.create_execution_ctx(); + let lhs = l2_denorm_array(&[2], &[0.6, 0.8], &[5.0], &mut ctx)?; + let rhs = l2_denorm_array(&[2], &[1.0, 0.0], &[1.0], &mut ctx)?; // Expected: 5.0 * 1.0 * dot([0.6, 0.8], [1.0, 0.0]) = 5.0 * 0.6 = 3.0. assert_close(&eval_inner_product(lhs, rhs, 1)?, &[3.0]); @@ -470,8 +713,9 @@ mod tests { fn both_denorm_multiple_rows() -> VortexResult<()> { // Row 0: [3.0, 4.0] dot [3.0, 4.0] = 25.0. // Row 1: [1.0, 0.0] dot [0.0, 1.0] = 0.0. - let lhs = l2_denorm_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0])?; - let rhs = l2_denorm_array(&[2], &[0.6, 0.8, 0.0, 1.0], &[5.0, 1.0])?; + let mut ctx = SESSION.create_execution_ctx(); + let lhs = l2_denorm_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; + let rhs = l2_denorm_array(&[2], &[0.6, 0.8, 0.0, 1.0], &[5.0, 1.0], &mut ctx)?; assert_close(&eval_inner_product(lhs, rhs, 2)?, &[25.0, 0.0]); Ok(()) @@ -482,7 +726,8 @@ mod tests { // LHS: L2Denorm([0.6, 0.8], 5.0) representing [3.0, 4.0]. // RHS: plain [1.0, 2.0]. // dot([3.0, 4.0], [1.0, 2.0]) = 3.0 + 8.0 = 11.0. - let lhs = l2_denorm_array(&[2], &[0.6, 0.8], &[5.0])?; + let mut ctx = SESSION.create_execution_ctx(); + let lhs = l2_denorm_array(&[2], &[0.6, 0.8], &[5.0], &mut ctx)?; let rhs = tensor_array(&[2], &[1.0, 2.0])?; assert_close(&eval_inner_product(lhs, rhs, 1)?, &[11.0]); @@ -494,8 +739,9 @@ mod tests { // LHS: plain [1.0, 2.0]. // RHS: L2Denorm([0.6, 0.8], 5.0) representing [3.0, 4.0]. // dot([1.0, 2.0], [3.0, 4.0]) = 3.0 + 8.0 = 11.0. + let mut ctx = SESSION.create_execution_ctx(); let lhs = tensor_array(&[2], &[1.0, 2.0])?; - let rhs = l2_denorm_array(&[2], &[0.6, 0.8], &[5.0])?; + let rhs = l2_denorm_array(&[2], &[0.6, 0.8], &[5.0], &mut ctx)?; assert_close(&eval_inner_product(lhs, rhs, 1)?, &[11.0]); Ok(()) @@ -508,19 +754,847 @@ mod tests { let norms_l = PrimitiveArray::from_option_iter([Some(5.0f64), None]).into_array(); let mut ctx = SESSION.create_execution_ctx(); - let lhs = - L2Denorm::try_new_array(&ApproxOptions::Exact, normalized_l, norms_l, 2, &mut ctx)? - .into_array(); - let rhs = l2_denorm_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0])?; + let lhs = L2Denorm::try_new_array(normalized_l, norms_l, 2, &mut ctx)?.into_array(); + let rhs = l2_denorm_array(&[2], &[0.6, 0.8, 1.0, 0.0], &[5.0, 1.0], &mut ctx)?; - let scalar_fn = ScalarFn::new(InnerProduct, ApproxOptions::Exact).erased(); + let scalar_fn = InnerProduct::new().erased(); let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs], 2)?; let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; // Row 0: 5.0 * 5.0 * dot([0.6, 0.8], [0.6, 0.8]) = 25.0, row 1: null. - assert!(prim.is_valid(0)?); - assert!(!prim.is_valid(1)?); + assert!(prim.is_valid(0, &mut ctx)?); + assert!(!prim.is_valid(1, &mut ctx)?); assert_close(&[prim.as_slice::()[0]], &[25.0]); Ok(()) } + + #[rstest] + #[case::vector(inner_product_vector_lhs(), inner_product_vector_rhs(), 2)] + #[case::fixed_shape_tensor(inner_product_tensor_lhs(), inner_product_tensor_rhs(), 2)] + fn serde_round_trip( + #[case] lhs: ArrayRef, + #[case] rhs: ArrayRef, + #[case] len: usize, + ) -> VortexResult<()> { + let original = InnerProduct::try_new_array(lhs.clone(), rhs.clone(), len)?.into_array(); + + let plugin = ScalarFnArrayPlugin::new(InnerProduct); + let metadata = plugin + .serialize(&original, &SESSION)? + .expect("InnerProduct serialize must produce metadata"); + + let children = vec![lhs, rhs]; + let recovered = plugin.deserialize( + original.dtype(), + original.len(), + &metadata, + &[], + &children, + &SESSION, + )?; + + assert_eq!(recovered.dtype(), original.dtype()); + assert_eq!(recovered.len(), original.len()); + assert_eq!(recovered.encoding_id(), original.encoding_id()); + Ok(()) + } + + fn inner_product_vector_lhs() -> ArrayRef { + vector_array(3, &[1.0, 2.0, 3.0, 4.0, 5.0, 6.0]).expect("valid vector array") + } + + fn inner_product_vector_rhs() -> ArrayRef { + vector_array(3, &[7.0, 8.0, 9.0, 10.0, 11.0, 12.0]).expect("valid vector array") + } + + fn inner_product_tensor_lhs() -> ArrayRef { + tensor_array(&[2], &[1.0, 2.0, 3.0, 4.0]).expect("valid tensor array") + } + + fn inner_product_tensor_rhs() -> ArrayRef { + tensor_array(&[2], &[5.0, 6.0, 7.0, 8.0]).expect("valid tensor array") + } + + // ---- Tests for the `SorfTransform + constant` and `Dict + constant` fast paths ---- + + #[allow( + clippy::cast_possible_truncation, + reason = "tests build small fixtures with deterministic in-range indices" + )] + mod constant_query_optimizations { + use rstest::rstest; + use vortex_array::ArrayRef; + use vortex_array::IntoArray; + use vortex_array::VortexSessionExecute; + use vortex_array::arrays::Constant; + use vortex_array::arrays::FixedSizeListArray; + use vortex_array::arrays::PrimitiveArray; + use vortex_array::arrays::ScalarFnArray; + use vortex_array::arrays::dict::DictArray; + use vortex_array::dtype::DType; + use vortex_array::dtype::Nullability; + use vortex_array::dtype::PType; + use vortex_array::validity::Validity; + use vortex_buffer::Buffer; + use vortex_error::VortexResult; + + use crate::scalar_fns::inner_product::InnerProduct; + use crate::scalar_fns::inner_product::constant_tensor_storage; + use crate::scalar_fns::sorf_transform::SorfMatrix; + use crate::scalar_fns::sorf_transform::SorfOptions; + use crate::scalar_fns::sorf_transform::SorfTransform; + use crate::tests::SESSION; + use crate::types::vector::Vector; + use crate::utils::extract_flat_elements; + use crate::utils::test_helpers::literal_vector_array; + use crate::utils::test_helpers::vector_array; + + /// Build a `Vector` whose storage is `FSL(DictArray(codes: u8, values: + /// f32))`. This mirrors the shape that TurboQuant produces as the SorfTransform child. + fn dict_vector_f32(list_size: u32, codes: &[u8], values: &[f32]) -> VortexResult { + let num_rows = codes.len() / list_size as usize; + let codes_arr = + PrimitiveArray::new::(Buffer::copy_from(codes), Validity::NonNullable) + .into_array(); + let values_arr = + PrimitiveArray::new::(Buffer::copy_from(values), Validity::NonNullable) + .into_array(); + let dict = DictArray::try_new(codes_arr, values_arr)?; + let fsl = FixedSizeListArray::try_new( + dict.into_array(), + list_size, + Validity::NonNullable, + num_rows, + )?; + Vector::try_new_vector_array(fsl.into_array()) + } + + /// Execute an inner product and return the flat `f32` results. + fn eval_ip_f32(lhs: ArrayRef, rhs: ArrayRef, len: usize) -> VortexResult> { + let scalar_fn = InnerProduct::new().erased(); + let result = ScalarFnArray::try_new(scalar_fn, vec![lhs, rhs], len)?; + let mut ctx = SESSION.create_execution_ctx(); + let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; + Ok(prim.as_slice::().to_vec()) + } + + fn assert_close_f32(actual: &[f32], expected: &[f32], tol: f32) { + assert_eq!(actual.len(), expected.len(), "length mismatch"); + for (i, (a, e)) in actual.iter().zip(expected).enumerate() { + assert!( + (a - e).abs() < tol, + "row {i}: got {a}, expected {e} (diff = {})", + (a - e).abs() + ); + } + } + + /// Build a SorfTransform ScalarFnArray whose child is a `Vector` + /// wrapping `FSL(Dict(codes, values))`. Returns `(sorf_array, codes, values, + /// padded_dim)`. + fn build_sorf_with_dict_child( + dim: u32, + num_rows: usize, + seed: u64, + num_rounds: u8, + ) -> VortexResult<(ArrayRef, Vec, Vec, usize)> { + let padded_dim = (dim as usize).next_power_of_two(); + // Small hand-picked codebook of 8 f32 centroids. + let values: Vec = vec![-1.5, -1.0, -0.5, -0.1, 0.1, 0.5, 1.0, 1.5]; + // Deterministic codes in 0..values.len() covering every position. + let codes: Vec = (0..num_rows * padded_dim) + .map(|i| (i as u8) % (values.len() as u8)) + .collect(); + + let padded_vector = dict_vector_f32(padded_dim as u32, &codes, &values)?; + let sorf_options = SorfOptions { + seed, + num_rounds, + dimensions: dim, + element_ptype: PType::F32, + }; + let sorf = + SorfTransform::try_new_array(&sorf_options, padded_vector, num_rows)?.into_array(); + Ok((sorf, codes, values, padded_dim)) + } + + /// Decode a SorfTransform-wrapped dict-vector to a flat `Vec` of `num_rows * + /// dim` post-rotation, post-truncation values. This is the ground truth against + /// which we compare the fast-path result. + fn decode_sorf_dict( + codes: &[u8], + values: &[f32], + padded_dim: usize, + dim: usize, + num_rows: usize, + seed: u64, + num_rounds: u8, + ) -> VortexResult> { + let rotation = SorfMatrix::try_new(seed, dim, num_rounds as usize)?; + let mut padded = vec![0.0f32; padded_dim]; + let mut rotated = vec![0.0f32; padded_dim]; + let mut out = Vec::with_capacity(num_rows * dim); + for row in 0..num_rows { + for j in 0..padded_dim { + padded[j] = values[codes[row * padded_dim + j] as usize]; + } + rotation.inverse_rotate(&padded, &mut rotated); + out.extend_from_slice(&rotated[..dim]); + } + Ok(out) + } + + fn naive_dot(a: &[f32], b: &[f32]) -> f32 { + a.iter().zip(b.iter()).map(|(&x, &y)| x * y).sum() + } + + // ---- Case 1: SorfTransform + Constant pull-through ---- + + #[test] + fn constant_tensor_storage_accepts_extension_scalar_literal() -> VortexResult<()> { + let literal = literal_vector_array(&[1.0f32, 2.0, 3.0], 5); + let storage = + constant_tensor_storage(&literal).expect("literal vector should be recognized"); + + assert_eq!(storage.len(), 5); + let const_storage = storage + .as_opt::() + .expect("storage should remain constant-backed"); + assert!(matches!( + const_storage.scalar().dtype(), + DType::FixedSizeList(_, 3, Nullability::NonNullable) + )); + + let mut ctx = SESSION.create_execution_ctx(); + let flat = extract_flat_elements(&storage, 3, &mut ctx)?; + assert_eq!(flat.row::(0), &[1.0, 2.0, 3.0]); + Ok(()) + } + + /// Case 1: SorfTransform on LHS, constant query on RHS, with `dim < padded_dim` + /// so the zero-padding branch is exercised. + #[test] + fn case1_sorf_lhs_constant_rhs_padded_gt_dim() -> VortexResult<()> { + let dim: u32 = 100; + let num_rows = 7usize; + let seed = 42u64; + let num_rounds = 3u8; + let padded_dim = (dim as usize).next_power_of_two(); + assert!(padded_dim > dim as usize, "test must exercise padding"); + + let (sorf_lhs, codes, values, padded_dim_computed) = + build_sorf_with_dict_child(dim, num_rows, seed, num_rounds)?; + assert_eq!(padded_dim_computed, padded_dim); + + // Query has `dim` elements. + let query_elems: Vec = (0..dim).map(|i| (i as f32 * 0.1).sin()).collect(); + let const_rhs = Vector::constant_array(&query_elems, num_rows)?; + + // Ground truth: decode LHS to plain f32 vectors, dot each with the query. + let decoded = decode_sorf_dict( + &codes, + &values, + padded_dim, + dim as usize, + num_rows, + seed, + num_rounds, + )?; + let expected: Vec = (0..num_rows) + .map(|i| { + naive_dot( + &decoded[i * dim as usize..(i + 1) * dim as usize], + &query_elems, + ) + }) + .collect(); + + let actual = eval_ip_f32(sorf_lhs, const_rhs, num_rows)?; + assert_close_f32(&actual, &expected, 1e-3); + Ok(()) + } + + /// Case 1: SorfTransform on RHS, constant query on LHS (mirrored). + #[test] + fn case1_constant_lhs_sorf_rhs_mirrored() -> VortexResult<()> { + let dim: u32 = 100; + let num_rows = 5usize; + let seed = 7u64; + let num_rounds = 3u8; + + let (sorf, codes, values, padded_dim) = + build_sorf_with_dict_child(dim, num_rows, seed, num_rounds)?; + + let query_elems: Vec = (0..dim).map(|i| (i as f32 * 0.2).cos()).collect(); + let const_lhs = Vector::constant_array(&query_elems, num_rows)?; + + let decoded = decode_sorf_dict( + &codes, + &values, + padded_dim, + dim as usize, + num_rows, + seed, + num_rounds, + )?; + let expected: Vec = (0..num_rows) + .map(|i| { + naive_dot( + &decoded[i * dim as usize..(i + 1) * dim as usize], + &query_elems, + ) + }) + .collect(); + + let actual = eval_ip_f32(const_lhs, sorf, num_rows)?; + assert_close_f32(&actual, &expected, 1e-3); + Ok(()) + } + + /// Case 1: `dim == padded_dim` (power-of-two, no zero padding). + #[test] + fn case1_padded_equals_dim() -> VortexResult<()> { + let dim: u32 = 128; + let num_rows = 4usize; + let seed = 11u64; + let num_rounds = 3u8; + + let (sorf, codes, values, padded_dim) = + build_sorf_with_dict_child(dim, num_rows, seed, num_rounds)?; + assert_eq!(padded_dim, dim as usize); + + let query_elems: Vec = (0..dim).map(|i| i as f32 * 0.01 - 0.5).collect(); + let const_rhs = Vector::constant_array(&query_elems, num_rows)?; + + let decoded = decode_sorf_dict( + &codes, + &values, + padded_dim, + dim as usize, + num_rows, + seed, + num_rounds, + )?; + let expected: Vec = (0..num_rows) + .map(|i| { + naive_dot( + &decoded[i * dim as usize..(i + 1) * dim as usize], + &query_elems, + ) + }) + .collect(); + + let actual = eval_ip_f32(sorf, const_rhs, num_rows)?; + assert_close_f32(&actual, &expected, 1e-3); + Ok(()) + } + + /// Case 1: empty `len == 0`. The fast path should handle this without exploding. + #[test] + fn case1_empty_len_zero() -> VortexResult<()> { + let dim: u32 = 100; + let num_rows = 0usize; + let seed = 42u64; + let num_rounds = 3u8; + + let (sorf, _codes, _values, _padded_dim) = + build_sorf_with_dict_child(dim, num_rows, seed, num_rounds)?; + + let query_elems: Vec = vec![0.0; dim as usize]; + let const_rhs = Vector::constant_array(&query_elems, num_rows)?; + + let actual = eval_ip_f32(sorf, const_rhs, num_rows)?; + assert_eq!(actual.len(), 0); + Ok(()) + } + + // ---- Case 2: Dict + Constant direct-lookup path ---- + + /// Case 2: Vector[FSL[Dict(u8, f32)]] on LHS, constant query on RHS. + #[test] + fn case2_dict_lhs_constant_rhs_matches_naive() -> VortexResult<()> { + let list_size: u32 = 8; + let num_rows = 10usize; + // 8 centroids, tiny table. + let values: Vec = vec![-1.0, -0.5, -0.25, -0.1, 0.1, 0.25, 0.5, 1.0]; + // Deterministic codes. + let codes: Vec = (0..num_rows * list_size as usize) + .map(|i| (i as u8) % (values.len() as u8)) + .collect(); + let dict_lhs = dict_vector_f32(list_size, &codes, &values)?; + + let query: Vec = (0..list_size).map(|i| (i as f32 + 1.0) * 0.3).collect(); + let const_rhs = Vector::constant_array(&query, num_rows)?; + + let expected: Vec = (0..num_rows) + .map(|row| { + let mut acc = 0.0f32; + for j in 0..list_size as usize { + let k = codes[row * list_size as usize + j] as usize; + acc += query[j] * values[k]; + } + acc + }) + .collect(); + + let actual = eval_ip_f32(dict_lhs, const_rhs, num_rows)?; + assert_close_f32(&actual, &expected, 1e-5); + Ok(()) + } + + /// Case 2: constant query on LHS, dict column on RHS (mirrored). + #[test] + fn case2_constant_lhs_dict_rhs_mirrored() -> VortexResult<()> { + let list_size: u32 = 4; + let num_rows = 6usize; + let values: Vec = vec![0.1, 0.4, 0.7, 1.0]; + let codes: Vec = (0..num_rows * list_size as usize) + .map(|i| ((i * 3) as u8) % (values.len() as u8)) + .collect(); + let dict_rhs = dict_vector_f32(list_size, &codes, &values)?; + + let query: Vec = vec![0.5, -1.0, 2.5, -0.25]; + let const_lhs = Vector::constant_array(&query, num_rows)?; + + let expected: Vec = (0..num_rows) + .map(|row| { + let mut acc = 0.0f32; + for j in 0..list_size as usize { + let k = codes[row * list_size as usize + j] as usize; + acc += query[j] * values[k]; + } + acc + }) + .collect(); + + let actual = eval_ip_f32(const_lhs, dict_rhs, num_rows)?; + assert_close_f32(&actual, &expected, 1e-5); + Ok(()) + } + + /// Case 2: dict with `u16` codes (and hence more than 256 values) falls through to + /// the standard path but still produces the correct result. The direct-lookup path + /// only handles `u8` codes today. + #[test] + fn case2_u16_codes_falls_through() -> VortexResult<()> { + let list_size: u32 = 4; + let num_rows = 3usize; + let num_values = 300usize; + let values: Vec = (0..num_values).map(|i| i as f32 * 0.01).collect(); + // Codes must be u16 because 300 > 255. dict_vector_f32 only supports u8 so we + // build the dict by hand here. + let codes_u16: Vec = (0..(num_rows * 4)) + .map(|i| (i % num_values) as u16) + .collect(); + let codes_arr = + PrimitiveArray::new::(Buffer::copy_from(codes_u16), Validity::NonNullable) + .into_array(); + let values_arr = + PrimitiveArray::new::(Buffer::copy_from(&values), Validity::NonNullable) + .into_array(); + let dict = DictArray::try_new(codes_arr, values_arr)?; + let fsl = FixedSizeListArray::try_new( + dict.into_array(), + list_size, + Validity::NonNullable, + num_rows, + )?; + let dict_lhs = Vector::try_new_vector_array(fsl.into_array())?; + + let query: Vec = vec![1.0, 2.0, 3.0, 4.0]; + let const_rhs = Vector::constant_array(&query, num_rows)?; + + // Build expected by decoding by hand. + let expected: Vec = (0..num_rows) + .map(|row| { + let mut acc = 0.0f32; + for j in 0..4 { + let code = (row * 4 + j) % num_values; + acc += query[j] * values[code]; + } + acc + }) + .collect(); + + let actual = eval_ip_f32(dict_lhs, const_rhs, num_rows)?; + assert_close_f32(&actual, &expected, 1e-5); + Ok(()) + } + + /// Case 2: plain (non-dict) FSL with a constant RHS falls through to the standard + /// path and produces the correct result. + #[test] + fn case2_plain_fsl_falls_through() -> VortexResult<()> { + let dim: u32 = 4; + let num_rows = 3usize; + let lhs_elems: Vec = (0..num_rows * dim as usize) + .map(|i| i as f32 * 0.25) + .collect(); + let plain_lhs = vector_array(dim, &lhs_elems)?; + + let query: Vec = vec![1.0, 2.0, 3.0, 4.0]; + let const_rhs = Vector::constant_array(&query, num_rows)?; + + let expected: Vec = (0..num_rows) + .map(|row| { + naive_dot( + &lhs_elems[row * dim as usize..(row + 1) * dim as usize], + &query, + ) + }) + .collect(); + + let actual = eval_ip_f32(plain_lhs, const_rhs, num_rows)?; + assert_close_f32(&actual, &expected, 1e-5); + Ok(()) + } + + /// Case 2: empty `len == 0` fast path returns an empty primitive array without + /// touching the codes buffer. + #[test] + fn case2_empty_len_zero() -> VortexResult<()> { + let list_size: u32 = 4; + let num_rows = 0usize; + let values: Vec = vec![0.0, 1.0, 2.0, 3.0]; + let codes: Vec = Vec::new(); + let dict_lhs = dict_vector_f32(list_size, &codes, &values)?; + + let query: Vec = vec![0.0; 4]; + let const_rhs = Vector::constant_array(&query, num_rows)?; + + let actual = eval_ip_f32(dict_lhs, const_rhs, num_rows)?; + assert_eq!(actual.len(), 0); + Ok(()) + } + + /// Case 1 + Case 2 end-to-end: the SorfTransform-wrapped dict column hits Case 1 + /// then Case 2 via recursive execution. + #[test] + fn end_to_end_sorf_plus_dict_cosine_path() -> VortexResult<()> { + let dim: u32 = 100; + let num_rows = 9usize; + let seed = 99u64; + let num_rounds = 3u8; + + let (sorf, codes, values, padded_dim) = + build_sorf_with_dict_child(dim, num_rows, seed, num_rounds)?; + + let query_elems: Vec = (0..dim).map(|i| ((i as f32) * 0.15).sin() * 0.4).collect(); + let const_rhs = Vector::constant_array(&query_elems, num_rows)?; + + // Ground truth via full decode + naive dot. + let decoded = decode_sorf_dict( + &codes, + &values, + padded_dim, + dim as usize, + num_rows, + seed, + num_rounds, + )?; + let expected: Vec = (0..num_rows) + .map(|i| { + naive_dot( + &decoded[i * dim as usize..(i + 1) * dim as usize], + &query_elems, + ) + }) + .collect(); + + let actual = eval_ip_f32(sorf, const_rhs, num_rows)?; + assert_close_f32(&actual, &expected, 1e-3); + Ok(()) + } + + // ---- Additional correctness / stress tests (all with loose tolerances) ---- + + /// A tiny in-place xorshift64 PRNG so these tests don't depend on `rand`. Producing + /// deterministic pseudo-random f32 values lets the correctness checks exercise + /// realistic data instead of smooth sin/cos patterns. + struct XorShift64(u64); + + impl XorShift64 { + fn new(seed: u64) -> Self { + // Any nonzero seed is fine; xorshift fixed-points at 0. + Self(seed.wrapping_add(0x9E37_79B9_7F4A_7C15)) + } + + fn next_u64(&mut self) -> u64 { + let mut x = self.0; + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + self.0 = x; + x + } + + /// Uniform f32 in `[-1.0, 1.0)`. + fn next_f32(&mut self) -> f32 { + // Top 24 bits -> mantissa in [0, 1), then shift to [-1, 1). + let bits = (self.next_u64() >> 40) as u32; // 24 bits + (bits as f32) / (1u32 << 24) as f32 * 2.0 - 1.0 + } + } + + /// Case 2 stress: u8-coded dict with 200 centroids (formerly blocked by the + /// `values.len() <= 256` gate). The direct-lookup path must now handle it. + #[test] + fn case2_large_u8_codebook_direct_lookup() -> VortexResult<()> { + let list_size: u32 = 16; + let num_rows = 20usize; + let num_centroids = 200usize; + assert!(num_centroids > 8 && num_centroids <= 256); + + let mut rng = XorShift64::new(0xDEAD_BEEF); + let values: Vec = (0..num_centroids).map(|_| rng.next_f32()).collect(); + let codes: Vec = (0..num_rows * list_size as usize) + .map(|_| (rng.next_u64() % num_centroids as u64) as u8) + .collect(); + + let dict_lhs = dict_vector_f32(list_size, &codes, &values)?; + let query: Vec = (0..list_size).map(|_| rng.next_f32()).collect(); + let const_rhs = Vector::constant_array(&query, num_rows)?; + + let expected: Vec = (0..num_rows) + .map(|row| { + let mut acc = 0.0f32; + for j in 0..list_size as usize { + let k = codes[row * list_size as usize + j] as usize; + acc += query[j] * values[k]; + } + acc + }) + .collect(); + + let actual = eval_ip_f32(dict_lhs, const_rhs, num_rows)?; + assert_close_f32(&actual, &expected, 1e-4); + Ok(()) + } + + /// Parameterized sweep over the full `InnerProduct(SorfTransform(Vector[FSL(Dict)]), + /// ConstantArray)` tree, exercising the case 1 + case 2 chain for a realistic mix + /// of dimensions, row counts, seeds, and number of SORF rounds. Tolerance is + /// deliberately loose because the rewrite introduces an f32-domain rotation that + /// accumulates a small numerical drift versus a naive decode. + #[rstest] + #[case::small_no_pad(128, 11, 1, 1)] + #[case::small_no_pad_rounds3(128, 23, 1_234, 3)] + #[case::small_padded(100, 17, 42, 3)] + #[case::mid_padded(200, 13, 2024, 3)] + #[case::mid_power_of_two(256, 31, 7, 3)] + #[case::larger_padded(300, 9, 99, 3)] + #[case::max_rounds(128, 5, 31_415, 5)] + fn case1_sorf_random_sweep( + #[case] dim: u32, + #[case] num_rows: usize, + #[case] seed: u64, + #[case] num_rounds: u8, + ) -> VortexResult<()> { + let (sorf, codes, values, padded_dim) = + build_sorf_with_dict_child(dim, num_rows, seed, num_rounds)?; + + // Use a pseudo-random query with both positive and negative entries so the sum + // has cancellation. + let mut rng = XorShift64::new(seed ^ 0xABCD_1234); + let query: Vec = (0..dim).map(|_| rng.next_f32()).collect(); + let const_rhs = Vector::constant_array(&query, num_rows)?; + + let decoded = decode_sorf_dict( + &codes, + &values, + padded_dim, + dim as usize, + num_rows, + seed, + num_rounds, + )?; + let expected: Vec = (0..num_rows) + .map(|i| naive_dot(&decoded[i * dim as usize..(i + 1) * dim as usize], &query)) + .collect(); + + // Loose tolerance: the sorf transform works in f32 with a k-round butterfly, so + // the rewrite path and the decoded path accumulate slightly different rounding + // even though the math is equivalent in exact arithmetic. + let actual = eval_ip_f32(sorf, const_rhs, num_rows)?; + assert_close_f32(&actual, &expected, 1e-2); + Ok(()) + } + + /// Parameterized sweep over plain `Vector[FSL(Dict(u8, f32))]` + constant query, + /// without SorfTransform in the mix. This directly exercises case 2 across a + /// variety of list sizes, num_rows, and codebook sizes including large ones that + /// the old `<= 256` gate would have rejected. + #[rstest] + #[case::small(4, 7, 8)] + #[case::medium(16, 50, 64)] + #[case::larger(32, 100, 150)] + #[case::very_large_codebook(8, 25, 250)] + fn case2_random_sweep( + #[case] list_size: u32, + #[case] num_rows: usize, + #[case] num_centroids: usize, + ) -> VortexResult<()> { + let mut rng = XorShift64::new((list_size as u64) * 31 + num_rows as u64); + let values: Vec = (0..num_centroids).map(|_| rng.next_f32()).collect(); + assert!(num_centroids <= 256, "u8 codes cap at 256 centroids"); + let codes: Vec = (0..num_rows * list_size as usize) + .map(|_| (rng.next_u64() % num_centroids as u64) as u8) + .collect(); + + let dict_lhs = dict_vector_f32(list_size, &codes, &values)?; + let query: Vec = (0..list_size).map(|_| rng.next_f32()).collect(); + let const_rhs = Vector::constant_array(&query, num_rows)?; + + let expected: Vec = (0..num_rows) + .map(|row| { + let mut acc = 0.0f32; + for j in 0..list_size as usize { + let k = codes[row * list_size as usize + j] as usize; + acc += query[j] * values[k]; + } + acc + }) + .collect(); + + // Tight tolerance here because no SorfTransform rotation is involved — the + // arithmetic should agree bit-for-bit up to float reassociation. + let actual = eval_ip_f32(dict_lhs, const_rhs, num_rows)?; + assert_close_f32(&actual, &expected, 1e-4); + Ok(()) + } + + /// End-to-end regression: for a plausible vector-search configuration (SORF rounds + /// = 3, dim = 128, num_rows = 64, u8 codes, 64 centroids), the fast-path result + /// must track a fully naive computation within 1e-2. + #[test] + fn end_to_end_dim128_rows64_bit6_regression() -> VortexResult<()> { + let dim: u32 = 128; + let num_rows = 64usize; + let seed = 0xFACE_F00D; + let num_rounds = 3u8; + + // Use 64 centroids (6 bits), a typical TurboQuant configuration. + let num_centroids = 64usize; + let padded_dim = (dim as usize).next_power_of_two(); + let mut rng = XorShift64::new(seed); + let values: Vec = (0..num_centroids).map(|_| rng.next_f32()).collect(); + let codes: Vec = (0..num_rows * padded_dim) + .map(|_| (rng.next_u64() % num_centroids as u64) as u8) + .collect(); + + let padded_vector = dict_vector_f32(padded_dim as u32, &codes, &values)?; + let sorf_options = SorfOptions { + seed, + num_rounds, + dimensions: dim, + element_ptype: PType::F32, + }; + let sorf = + SorfTransform::try_new_array(&sorf_options, padded_vector, num_rows)?.into_array(); + + let query: Vec = (0..dim).map(|_| rng.next_f32()).collect(); + let const_rhs = Vector::constant_array(&query, num_rows)?; + + let decoded = decode_sorf_dict( + &codes, + &values, + padded_dim, + dim as usize, + num_rows, + seed, + num_rounds, + )?; + let expected: Vec = (0..num_rows) + .map(|i| naive_dot(&decoded[i * dim as usize..(i + 1) * dim as usize], &query)) + .collect(); + + let actual = eval_ip_f32(sorf, const_rhs, num_rows)?; + assert_close_f32(&actual, &expected, 1e-2); + + // Also verify the max relative error is small. The SORF rotation does not + // amplify error, so both measures should be bounded. + for (i, (a, e)) in actual.iter().zip(expected.iter()).enumerate() { + let denom = e.abs().max(1.0); + let rel = (a - e).abs() / denom; + assert!( + rel < 1e-3, + "row {i}: rel err {rel} too large (a={a}, e={e})" + ); + } + Ok(()) + } + + /// Case 1 + Case 2 end-to-end with varying `num_rounds`. The rotation becomes + /// progressively more chaotic as rounds increase, so this catches any off-by-one + /// bug in the round-indexing that would not show up in the 3-round default. + #[rstest] + #[case(1)] + #[case(2)] + #[case(3)] + #[case(4)] + #[case(5)] + fn case1_various_num_rounds(#[case] num_rounds: u8) -> VortexResult<()> { + let dim: u32 = 128; + let num_rows = 8usize; + let seed = 0x1234_5678; + + let (sorf, codes, values, padded_dim) = + build_sorf_with_dict_child(dim, num_rows, seed, num_rounds)?; + + let mut rng = XorShift64::new(seed ^ (num_rounds as u64)); + let query: Vec = (0..dim).map(|_| rng.next_f32()).collect(); + let const_rhs = Vector::constant_array(&query, num_rows)?; + + let decoded = decode_sorf_dict( + &codes, + &values, + padded_dim, + dim as usize, + num_rows, + seed, + num_rounds, + )?; + let expected: Vec = (0..num_rows) + .map(|i| naive_dot(&decoded[i * dim as usize..(i + 1) * dim as usize], &query)) + .collect(); + + let actual = eval_ip_f32(sorf, const_rhs, num_rows)?; + assert_close_f32(&actual, &expected, 1e-2); + Ok(()) + } + + /// Swap LHS and RHS on the full tree to prove the side-detection and the scalar + /// argument-order handling are symmetric for both cases simultaneously. + #[test] + fn end_to_end_constant_lhs_sorf_rhs_mirrored() -> VortexResult<()> { + let dim: u32 = 256; + let num_rows = 12usize; + let seed = 0xBEEF_CAFE; + let num_rounds = 3u8; + + let (sorf, codes, values, padded_dim) = + build_sorf_with_dict_child(dim, num_rows, seed, num_rounds)?; + + let mut rng = XorShift64::new(seed); + let query: Vec = (0..dim).map(|_| rng.next_f32()).collect(); + let const_lhs = Vector::constant_array(&query, num_rows)?; + + let decoded = decode_sorf_dict( + &codes, + &values, + padded_dim, + dim as usize, + num_rows, + seed, + num_rounds, + )?; + let expected: Vec = (0..num_rows) + .map(|i| naive_dot(&decoded[i * dim as usize..(i + 1) * dim as usize], &query)) + .collect(); + + let actual = eval_ip_f32(const_lhs, sorf, num_rows)?; + assert_close_f32(&actual, &expected, 1e-2); + Ok(()) + } + } } diff --git a/vortex-tensor/src/scalar_fns/l2_denorm.rs b/vortex-tensor/src/scalar_fns/l2_denorm.rs index 04a348ff7d8..1bdd81833d9 100644 --- a/vortex-tensor/src/scalar_fns/l2_denorm.rs +++ b/vortex-tensor/src/scalar_fns/l2_denorm.rs @@ -5,28 +5,47 @@ use std::fmt::Formatter; +use num_traits::Float; use num_traits::ToPrimitive; use num_traits::Zero; +use prost::Message; use vortex_array::ArrayRef; use vortex_array::ExecutionCtx; use vortex_array::IntoArray; +use vortex_array::arrays::Constant; +use vortex_array::arrays::ConstantArray; +use vortex_array::arrays::Extension; use vortex_array::arrays::ExtensionArray; use vortex_array::arrays::FixedSizeListArray; use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::ScalarFn as ScalarFnArrayEncoding; use vortex_array::arrays::ScalarFnArray; use vortex_array::arrays::extension::ExtensionArrayExt; +use vortex_array::arrays::fixed_size_list::FixedSizeListArrayExt; +use vortex_array::arrays::scalar_fn::ExactScalarFn; +use vortex_array::arrays::scalar_fn::ScalarFnArrayExt; +use vortex_array::arrays::scalar_fn::ScalarFnArrayView; +use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayParts; +use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayVTable; +use vortex_array::builtins::ArrayBuiltins; use vortex_array::dtype::DType; use vortex_array::dtype::NativePType; -use vortex_array::dtype::PType; +use vortex_array::dtype::Nullability; +use vortex_array::dtype::proto::dtype as pb; use vortex_array::expr::Expression; use vortex_array::expr::and; use vortex_array::match_each_float_ptype; +use vortex_array::scalar::Scalar; +use vortex_array::scalar::ScalarValue; use vortex_array::scalar_fn::Arity; use vortex_array::scalar_fn::ChildName; +use vortex_array::scalar_fn::EmptyOptions; use vortex_array::scalar_fn::ExecutionArgs; -use vortex_array::scalar_fn::ScalarFn; use vortex_array::scalar_fn::ScalarFnId; use vortex_array::scalar_fn::ScalarFnVTable; +use vortex_array::scalar_fn::TypedScalarFnInstance; +use vortex_array::scalar_fn::fns::operators::Operator; +use vortex_array::serde::ArrayChildren; use vortex_array::validity::Validity; use vortex_buffer::Buffer; use vortex_buffer::BufferMut; @@ -35,14 +54,17 @@ use vortex_error::VortexResult; use vortex_error::vortex_bail; use vortex_error::vortex_ensure; use vortex_error::vortex_ensure_eq; +use vortex_error::vortex_err; +use vortex_session::VortexSession; use crate::matcher::AnyTensor; -use crate::scalar_fns::ApproxOptions; use crate::scalar_fns::l2_norm::L2Norm; +use crate::utils::extract_constant_flat_row; use crate::utils::extract_flat_elements; +use crate::utils::unit_norm_tolerance; use crate::utils::validate_tensor_float_input; -/// Re-applies L2 norms to a normalized tensor column. +/// Re-applies authoritative L2 norms to a normalized tensor column. /// /// Computes `normalized * norm` on each row over the flat backing buffer of each tensor-like type. /// @@ -51,17 +73,28 @@ use crate::utils::validate_tensor_float_input; /// /// The norms input must be a primitive float column with the same element type as the normalized /// tensor elements. +/// +/// [`L2Denorm`] is the norm-splitting wrapper used throughout the tensor crate. Callers that build +/// it through [`try_new_array`](Self::try_new_array) get an exact unit-norm invariant on the +/// `normalized` child. +/// +/// Advanced callers can also use [`new_array_unchecked`](Self::new_array_unchecked) to attach +/// authoritative stored norms to a lossy approximation of that child, such as quantized normalized +/// vectors. +/// +/// Downstream readthrough rules intentionally treat the stored norms and normalized child as the +/// encoding contract, even when that differs slightly from recomputing over fully decoded +/// coordinates. #[derive(Clone)] pub struct L2Denorm; impl L2Denorm { - /// Creates a new [`ScalarFn`] wrapping the L2 denormalization operation with the given - /// [`ApproxOptions`] controlling approximation behavior. + /// Creates a new [`TypedScalarFnInstance`] wrapping the L2 denormalization operation. /// /// This is a low-level scalar-function descriptor constructor. To build a semantically valid /// [`L2Denorm`] array, prefer [`try_new_array`](Self::try_new_array). - pub fn new(options: &ApproxOptions) -> ScalarFn { - ScalarFn::new(L2Denorm, options.clone()) + pub fn new() -> TypedScalarFnInstance { + TypedScalarFnInstance::new(L2Denorm, EmptyOptions) } /// Constructs a validated [`ScalarFnArray`] that lazily re-applies `norms` to `normalized`. @@ -77,21 +110,15 @@ impl L2Denorm { /// Returns an error if the [`ScalarFnArray`] cannot be constructed (e.g. due to dtype /// mismatches) or if the `normalized` child is not row-wise L2-normalized. pub fn try_new_array( - options: &ApproxOptions, normalized: ArrayRef, norms: ArrayRef, len: usize, ctx: &mut ExecutionCtx, ) -> VortexResult { - let result = ScalarFnArray::try_new( - L2Denorm::new(options).erased(), - vec![normalized.clone(), norms.clone()], - len, - )?; - - validate_l2_denorm_children(normalized, norms, ctx)?; + validate_l2_normalized_rows_against_norms(&normalized, Some(&norms), ctx)?; - Ok(result) + // SAFETY: We just validated that it is normalized. + unsafe { Self::new_array_unchecked(normalized, norms, len) } } /// Constructs an [`L2Denorm`] array without validating that the `normalized` child is actually @@ -104,27 +131,27 @@ impl L2Denorm { /// # Safety /// /// The caller must ensure the `normalized` child is semantically suitable for L2 - /// denormalization, which typically means every valid row is unit-norm or zero. Violating this - /// invariant will not cause memory unsafety, but may produce incorrect results. + /// denormalization. For exact wrappers, that means every valid row is unit-norm or zero. + /// + /// Lossy encodings may deliberately relax that invariant while still treating the stored norms + /// as authoritative. + /// + /// Violating the intended contract will not cause memory unsafety, but may produce incorrect + /// results. pub unsafe fn new_array_unchecked( - options: &ApproxOptions, normalized: ArrayRef, norms: ArrayRef, len: usize, ) -> VortexResult { - ScalarFnArray::try_new( - L2Denorm::new(options).erased(), - vec![normalized, norms], - len, - ) + ScalarFnArray::try_new(L2Denorm::new().erased(), vec![normalized, norms], len) } } impl ScalarFnVTable for L2Denorm { - type Options = ApproxOptions; + type Options = EmptyOptions; fn id(&self) -> ScalarFnId { - ScalarFnId::new_ref("vortex.tensor.l2_denorm") + ScalarFnId::new("vortex.tensor.l2_denorm") } fn arity(&self, _options: &Self::Options) -> Arity { @@ -178,8 +205,35 @@ impl ScalarFnVTable for L2Denorm { args: &dyn ExecutionArgs, ctx: &mut ExecutionCtx, ) -> VortexResult { - let normalized: ExtensionArray = args.get(0)?.execute(ctx)?; - let norms: PrimitiveArray = args.get(1)?.execute(ctx)?; + let normalized_ref = args.get(0)?; + let norms_ref = args.get(1)?; + let output_dtype = normalized_ref + .dtype() + .union_nullability(norms_ref.dtype().nullability()); + let validity = normalized_ref.validity()?.and(norms_ref.validity()?)?; + + if let Some(const_norms) = norms_ref.as_opt::() { + let norm_scalar = const_norms.scalar(); + vortex_ensure!( + norm_scalar.dtype().is_float(), + "L2Denorm constant norms must be a float scalar, got {}", + norm_scalar.dtype(), + ); + + if let Some(norm_value) = norm_scalar.value() { + return execute_l2_denorm_constant_norms( + normalized_ref, + norm_scalar, + norm_value, + output_dtype, + validity, + ctx, + ); + } + } + + let normalized: ExtensionArray = normalized_ref.execute(ctx)?; + let norms: PrimitiveArray = norms_ref.execute(ctx)?; let row_count = args.row_count(); let tensor_match = normalized @@ -187,17 +241,11 @@ impl ScalarFnVTable for L2Denorm { .as_extension() .metadata_opt::() .vortex_expect("we already validated this in `return_dtype`"); - let tensor_flat_size = tensor_match.list_size(); + let tensor_flat_size = tensor_match.list_size() as usize; - let validity = normalized.as_ref().validity()?.and(norms.validity()?)?; - let output_dtype = normalized - .dtype() - .union_nullability(norms.dtype().nullability()); let flat = extract_flat_elements(normalized.storage_array(), tensor_flat_size, ctx)?; - // TODO(connor): Theoretically we could model this as a multiplication between the - // normalized array and a `RunEnd(Sequence(0, dimensions), norms)`. But since we have - // already canonicalized the array, it is probably not faster to do that. + // TODO(connor): Do we want a "broadcast" expression for the List types, or is this fine? match_each_float_ptype!(flat.ptype(), |T| { let norms = norms.as_slice::(); @@ -238,6 +286,127 @@ impl ScalarFnVTable for L2Denorm { } } +/// Metadata for a serialized [`L2Denorm`] array: both children's full [`DType`]s. The parent's +/// dtype is `normalized.union_nullability(norms.nullability())`, which loses both children's +/// individual nullabilities, so we persist them directly. +#[derive(Clone, prost::Message)] +pub(super) struct L2DenormMetadata { + #[prost(message, optional, tag = "1")] + normalized_dtype: Option, + #[prost(message, optional, tag = "2")] + norms_dtype: Option, +} + +impl ScalarFnArrayVTable for L2Denorm { + fn serialize( + &self, + view: &ScalarFnArrayView, + _session: &VortexSession, + ) -> VortexResult>> { + let scalar_fn_array = view.as_::(); + let normalized_dtype = Some(scalar_fn_array.child_at(0).dtype().try_into()?); + let norms_dtype = Some(scalar_fn_array.child_at(1).dtype().try_into()?); + Ok(Some( + L2DenormMetadata { + normalized_dtype, + norms_dtype, + } + .encode_to_vec(), + )) + } + + fn deserialize( + &self, + _dtype: &DType, + len: usize, + metadata: &[u8], + children: &dyn ArrayChildren, + session: &VortexSession, + ) -> VortexResult> { + let metadata = L2DenormMetadata::decode(metadata) + .map_err(|e| vortex_err!("Failed to decode L2DenormMetadata: {e}"))?; + let normalized_pb = metadata + .normalized_dtype + .as_ref() + .ok_or_else(|| vortex_err!("L2DenormMetadata missing normalized_dtype"))?; + let norms_pb = metadata + .norms_dtype + .as_ref() + .ok_or_else(|| vortex_err!("L2DenormMetadata missing norms_dtype"))?; + let normalized_dtype = DType::from_proto(normalized_pb, session)?; + let norms_dtype = DType::from_proto(norms_pb, session)?; + let normalized = children.get(0, &normalized_dtype, len)?; + let norms = children.get(1, &norms_dtype, len)?; + Ok(ScalarFnArrayParts { + options: EmptyOptions, + children: vec![normalized, norms], + }) + } +} + +/// Optimized execution when the norms array is constant. +fn execute_l2_denorm_constant_norms( + normalized_ref: ArrayRef, + norm_scalar: &Scalar, + norm_value: &ScalarValue, + output_dtype: DType, + new_validity: Validity, + ctx: &mut ExecutionCtx, +) -> VortexResult { + // If the norms are all equal to 1 then we don't need to do anything. + let err = norm_value + .as_primitive() + .as_f64() + .vortex_expect("we know that this is a float, so it must fit in f64") + - 1.0f64; + + let tensor_match = normalized_ref + .dtype() + .as_extension_opt() + .and_then(|ext| ext.metadata_opt::()) + .ok_or_else(|| { + vortex_err!( + "L2Denorm normalized child must be a tensor-like extension, got {}", + normalized_ref.dtype(), + ) + })?; + + let tolerance = unit_norm_tolerance( + norm_scalar.dtype().as_ptype(), + tensor_match.list_size() as usize, + ); + if err.abs() < tolerance { + return Ok(normalized_ref); + } + + // Even if the norms are not all 1, if they are all the same then we can multiply + // the entire elements array by the same number. + let normalized: ExtensionArray = normalized_ref.execute(ctx)?; + let storage_fsl: FixedSizeListArray = normalized.storage_array().clone().execute(ctx)?; + + // Replace the elements array with an array that multiplies it by the constant + // norms array (with length multiplied by the dimensions of the vectors). + let const_array = + ConstantArray::new(norm_scalar.clone(), storage_fsl.elements().len()).into_array(); + let mult_elements = storage_fsl + .elements() + .clone() + .binary(const_array, Operator::Mul)?; + + // SAFETY: We just updated the elements of the array with a scalar fn, so all + // invariants still hold. + let new_fsl = unsafe { + FixedSizeListArray::new_unchecked( + mult_elements, + storage_fsl.list_size(), + new_validity, + storage_fsl.len(), + ) + }; + + Ok(ExtensionArray::new(output_dtype.as_extension().clone(), new_fsl.into_array()).into_array()) +} + /// Builds an unexecuted [`L2Denorm`] expression by normalizing `input` and reattaching the exact /// norms as the norms child. /// @@ -258,26 +427,36 @@ impl ScalarFnVTable for L2Denorm { /// [`L2Norm`]'s validity propagation. When the [`L2Denorm`] wrapper is executed, its validity is /// `and(normalized_validity, norms_validity)`, which correctly identifies originally-null rows /// since the normalized child is all-valid and the norms child carries the original nulls. +/// +/// Because this helper computes exact norms first and then divides by those norms, the returned +/// `normalized` child satisfies the strict unit-norm invariant required by [`L2Denorm`]. pub fn normalize_as_l2_denorm( - options: &ApproxOptions, input: ArrayRef, ctx: &mut ExecutionCtx, ) -> VortexResult { let row_count = input.len(); let tensor_match = validate_tensor_float_input(input.dtype())?; - let tensor_flat_size = tensor_match.list_size(); + let tensor_flat_size = tensor_match.list_size() as usize; - let norms_sfn = L2Norm::try_new_array(options, input.clone(), row_count)?; + // Constant fast path: if the input is a constant-backed extension, normalize the single + // stored row once and return an `L2Denorm` whose children are both `ConstantArray`s. + if let Some(wrapped) = try_build_constant_l2_denorm(&input, row_count, ctx)? { + return Ok(wrapped); + } + + // Calculate the norms of the vectors. + let norms_sfn = L2Norm::try_new_array(input.clone(), row_count)?; let norms_array: ArrayRef = norms_sfn.into_array().execute(ctx)?; - let norms: PrimitiveArray = norms_array.clone().execute(ctx)?; - let norms_validity = norms.validity()?; + let primitive_norms: PrimitiveArray = norms_array.clone().execute(ctx)?; + let norms_validity = primitive_norms.validity()?; let input: ExtensionArray = input.execute(ctx)?; let normalized_dtype = input.dtype().as_nonnullable(); let flat = extract_flat_elements(input.storage_array(), tensor_flat_size, ctx)?; + // Normalize all of the vectors. let normalized = match_each_float_ptype!(flat.ptype(), |T| { - let norm_values = norms.as_slice::(); + let norm_values = primitive_norms.as_slice::(); let total_elements = row_count * tensor_flat_size; let mut elements = BufferMut::::with_capacity(total_elements); @@ -298,6 +477,8 @@ pub fn normalize_as_l2_denorm( } } + // Since L2Denorm's validity is the `and` of its child validities, we can make the + // normalized array non-nullable. build_tensor_array( normalized_dtype, tensor_flat_size, @@ -307,8 +488,99 @@ pub fn normalize_as_l2_denorm( ) })?; - // TODO(connor): Need to figure out a way to not run validation. - L2Denorm::try_new_array(options, normalized, norms_array, row_count, ctx) + // SAFETY: + // - `norms_array` was produced by `L2Norm(input)`, so every stored norm is non-negative and + // null rows already carry null validity through that child. + // - For every valid row, we either emit all zeros when the norm is zero or divide every + // element by the exact stored norm, so the normalized child is unit-norm (or zero) by + // construction. + // - Null rows are zeroed out above to avoid propagating arbitrary physical storage values into + // downstream lossy encodings. + unsafe { L2Denorm::new_array_unchecked(normalized, norms_array, row_count) } +} + +/// Attempts to build an [`L2Denorm`] whose two children are both [`ConstantArray`]s by eagerly +/// normalizing `input`'s single stored row. +/// +/// Returns `Ok(None)` when `input` is not a tensor-like extension array whose storage is a +/// [`ConstantArray`] with a non-null fixed-size-list scalar. +/// +/// When `input` matches, the returned [`ScalarFnArray`] is equivalent to [`normalize_as_l2_denorm`] +/// but runs in `O(list_size)` time instead of `O(row_count * list_size)`. +/// +/// This is helpful in some of the reduction steps for cosine similarity execution into inner +/// product execution. +pub(crate) fn try_build_constant_l2_denorm( + input: &ArrayRef, + len: usize, + ctx: &mut ExecutionCtx, +) -> VortexResult> { + let Some(ext) = input.as_opt::() else { + return Ok(None); + }; + let storage = ext.storage_array(); + let Some(const_storage) = storage.as_opt::() else { + return Ok(None); + }; + if const_storage.scalar().is_null() { + return Ok(None); + } + + // The caller is expected to have already validated that `input` is an `AnyTensor` + // extension dtype. + let tensor_match = input + .dtype() + .as_extension() + .metadata_opt::() + .vortex_expect("caller validated input has AnyTensor metadata"); + let list_size = tensor_match.list_size() as usize; + let original_nullability = input.dtype().nullability(); + let ext_dtype = input.dtype().as_extension().clone(); + let storage_fsl_nullability = storage.dtype().nullability(); + + // Materialize just the single stored row; this does not expand the constant to the full + // column length. + let flat = extract_constant_flat_row(storage, ctx)?; + + let (normalized_fsl_scalar, norms_scalar) = match_each_float_ptype!(flat.ptype(), |T| { + let row = flat.as_slice::(); + + let mut sum_sq = T::zero(); + for &x in row { + sum_sq += x * x; + } + let norm_t: T = sum_sq.sqrt(); + + // Zero-norm rows must be stored as all-zeros so [`L2Denorm`]'s unit-norm-or-zero + // invariant holds. This mirrors the per-row logic in `normalize_as_l2_denorm`. + let element_dtype = DType::Primitive(T::PTYPE, Nullability::NonNullable); + let children: Vec = if norm_t == T::zero() { + (0..list_size) + .map(|_| Scalar::zero_value(&element_dtype)) + .collect() + } else { + row.iter() + .map(|&v| Scalar::primitive(v / norm_t, Nullability::NonNullable)) + .collect() + }; + + // The rebuilt FSL scalar preserves the original storage FSL's nullability so the + // resulting `ExtensionArray::new` call accepts the same extension dtype. + let fsl_scalar = Scalar::fixed_size_list(element_dtype, children, storage_fsl_nullability); + let norms_scalar = Scalar::primitive(norm_t, original_nullability); + (fsl_scalar, norms_scalar) + }); + + let normalized_storage = ConstantArray::new(normalized_fsl_scalar, len).into_array(); + let normalized_ext = ExtensionArray::new(ext_dtype, normalized_storage).into_array(); + let norms_array = ConstantArray::new(norms_scalar, len).into_array(); + + // SAFETY: Each row of `normalized_ext` is either `v / ||v||` (unit norm within floating + // point tolerance) or all zeros when `||v|| == 0`. Stored norms are non-negative by + // construction (`sqrt`). These are exactly the invariants required by + // [`L2Denorm::new_array_unchecked`]. + let wrapped = unsafe { L2Denorm::new_array_unchecked(normalized_ext, norms_array, len)? }; + Ok(Some(wrapped)) } /// Rebuilds a tensor-like extension array from flat primitive elements. @@ -336,37 +608,16 @@ fn build_tensor_array( Ok(ExtensionArray::new(dtype.as_extension().clone(), storage.into_array()).into_array()) } -/// Returns the acceptable unit-norm drift for the given element precision. -fn unit_norm_tolerance(element_ptype: PType) -> f64 { - match element_ptype { - PType::F16 => 2e-3, - PType::F32 => 2e-6, - PType::F64 => 1e-10, - _ => unreachable!("L2Denorm requires float elements, got {element_ptype:?}"), - } -} - -/// Validates that every valid row of `input` is already L2-normalized (either length 1 or 0). -pub fn validate_l2_normalized_rows(input: ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult<()> { - validate_l2_normalized_rows_impl(input, None, ctx) -} - -/// Validates that the `normalized` and `norms` children jointly satisfy the [`L2Denorm`] -/// invariants, which are: +/// Validates that `normalized` and (when supplied) the matching `norms` jointly satisfy the +/// [`L2Denorm`] invariants: /// -/// - All vectors in the normalized array have length 1 or 0. -/// - If the vector has a norm of 0, then the vector must be all 0s. -fn validate_l2_denorm_children( - normalized: ArrayRef, - norms: ArrayRef, - ctx: &mut ExecutionCtx, -) -> VortexResult<()> { - validate_l2_normalized_rows_impl(normalized, Some(norms), ctx) -} - -fn validate_l2_normalized_rows_impl( - normalized: ArrayRef, - norms: Option, +/// - Every valid row of `normalized` has L2 norm `1.0` or `0.0` (within element-precision +/// tolerance). +/// - When `norms` is supplied, every stored norm is non-negative and any row whose stored norm is +/// `0.0` is exactly the zero vector in `normalized`. +pub fn validate_l2_normalized_rows_against_norms( + normalized: &ArrayRef, + norms: Option<&ArrayRef>, ctx: &mut ExecutionCtx, ) -> VortexResult<()> { let row_count = normalized.len(); @@ -376,14 +627,23 @@ fn validate_l2_normalized_rows_impl( let tensor_match = validate_tensor_float_input(normalized.dtype())?; let element_ptype = tensor_match.element_ptype(); - let tolerance = unit_norm_tolerance(element_ptype); - let tensor_flat_size = tensor_match.list_size(); + let tensor_flat_size = tensor_match.list_size() as usize; + let tolerance = unit_norm_tolerance(element_ptype, tensor_flat_size); - let normalized: ExtensionArray = normalized.execute(ctx)?; + if let Some(norms) = norms { + vortex_ensure_eq!( + norms.dtype().as_ptype(), + element_ptype, + "L2Denorm norms ptype must match normalized element ptype" + ); + } + + let normalized: ExtensionArray = normalized.clone().execute(ctx)?; let normalized_validity = normalized.as_ref().validity()?; + let flat = extract_flat_elements(normalized.storage_array(), tensor_flat_size, ctx)?; let norms = norms - .map(|norms| norms.execute::(ctx)) + .map(|norms| norms.clone().execute::(ctx)) .transpose()?; let combined_validity = match &norms { @@ -434,13 +694,62 @@ fn validate_l2_normalized_rows_impl( Ok(()) } +/// Classification of a binary operand pair by which side (if any) is wrapped in [`L2Denorm`]. +/// +/// Symmetric binary tensor operators (e.g. [`CosineSimilarity`], [`InnerProduct`]) have identical +/// fast paths for "only the lhs is denormalized" and "only the rhs is denormalized", and a separate +/// fast path for "both are denormalized". Rather than hand-rolling the commutative swap at every +/// call site, callers classify their operands with [`Self::classify`] and pattern-match on the +/// returned variant. +/// +/// [`CosineSimilarity`]: crate::scalar_fns::cosine_similarity::CosineSimilarity +/// [`InnerProduct`]: crate::scalar_fns::inner_product::InnerProduct +pub(crate) enum DenormOrientation<'a> { + /// Both operands are [`ExactScalarFn`] arrays. + Both { + lhs: &'a ArrayRef, + rhs: &'a ArrayRef, + }, + /// Exactly one operand is an [`ExactScalarFn`]; the other is plain. + One { + denorm: &'a ArrayRef, + plain: &'a ArrayRef, + }, + /// Neither operand is an [`ExactScalarFn`]. + Neither, +} + +impl<'a> DenormOrientation<'a> { + /// Classify `(lhs, rhs)` by which side (if any) is wrapped in [`L2Denorm`]. + pub(crate) fn classify(lhs: &'a ArrayRef, rhs: &'a ArrayRef) -> Self { + let lhs_denorm = lhs.is::>(); + let rhs_denorm = rhs.is::>(); + match (lhs_denorm, rhs_denorm) { + (true, true) => Self::Both { lhs, rhs }, + (true, false) => Self::One { + denorm: lhs, + plain: rhs, + }, + (false, true) => Self::One { + denorm: rhs, + plain: lhs, + }, + (false, false) => Self::Neither, + } + } +} + #[cfg(test)] mod tests { - use std::sync::LazyLock; + use rstest::rstest; + use vortex_array::ArrayPlugin; use vortex_array::ArrayRef; use vortex_array::IntoArray; use vortex_array::VortexSessionExecute; + use vortex_array::arrays::Constant; + use vortex_array::arrays::ConstantArray; + use vortex_array::arrays::Extension; use vortex_array::arrays::ExtensionArray; use vortex_array::arrays::FixedSizeListArray; use vortex_array::arrays::MaskedArray; @@ -448,55 +757,33 @@ mod tests { use vortex_array::arrays::extension::ExtensionArrayExt; use vortex_array::arrays::fixed_size_list::FixedSizeListArrayExt; use vortex_array::arrays::scalar_fn::ScalarFnArrayExt; + use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin; use vortex_array::dtype::DType; + use vortex_array::dtype::Nullability; use vortex_array::dtype::extension::ExtDType; - use vortex_array::extension::EmptyMetadata; use vortex_array::extension::datetime::Date; use vortex_array::extension::datetime::TimeUnit; - use vortex_array::session::ArraySession; + use vortex_array::scalar::Scalar; use vortex_array::validity::Validity; - use vortex_buffer::Buffer; use vortex_error::VortexResult; - use vortex_session::VortexSession; - use crate::fixed_shape::FixedShapeTensor; - use crate::fixed_shape::FixedShapeTensorMetadata; - use crate::scalar_fns::ApproxOptions; use crate::scalar_fns::l2_denorm::L2Denorm; use crate::scalar_fns::l2_denorm::normalize_as_l2_denorm; - use crate::scalar_fns::l2_denorm::validate_l2_normalized_rows; + use crate::scalar_fns::l2_denorm::validate_l2_normalized_rows_against_norms; + use crate::tests::SESSION; + use crate::types::vector::Vector; use crate::utils::test_helpers::assert_close; use crate::utils::test_helpers::constant_tensor_array; - use crate::utils::test_helpers::constant_vector_array; use crate::utils::test_helpers::tensor_array; use crate::utils::test_helpers::vector_array; - use crate::vector::Vector; - - static SESSION: LazyLock = - LazyLock::new(|| VortexSession::empty().with::()); /// Evaluates L2 denorm on a tensor/vector array and returns the executed array. fn eval_l2_denorm(normalized: ArrayRef, norms: ArrayRef, len: usize) -> VortexResult { let mut ctx = SESSION.create_execution_ctx(); - let result = - L2Denorm::try_new_array(&ApproxOptions::Exact, normalized, norms, len, &mut ctx)?; + let result = L2Denorm::try_new_array(normalized, norms, len, &mut ctx)?; result.into_array().execute(&mut ctx) } - fn integer_tensor_array(shape: &[usize], elements: &[i32]) -> VortexResult { - let list_size: u32 = shape.iter().product::().max(1).try_into().unwrap(); - let row_count = elements.len() / list_size as usize; - - let elems: ArrayRef = Buffer::copy_from(elements).into_array(); - let fsl = FixedSizeListArray::new(elems, list_size, Validity::NonNullable, row_count); - - let metadata = FixedShapeTensorMetadata::new(shape.to_vec()); - let ext_dtype = - ExtDType::::try_new(metadata, fsl.dtype().clone())?.erased(); - - Ok(ExtensionArray::new(ext_dtype, fsl.into_array()).into_array()) - } - fn non_tensor_extension_array() -> VortexResult { let storage = PrimitiveArray::from_iter([1i32, 2]).into_array(); let ext_dtype = @@ -504,21 +791,11 @@ mod tests { Ok(ExtensionArray::new(ext_dtype, storage).into_array()) } - fn f16_vector_array(dim: u32, elements: &[f32]) -> VortexResult { - let row_count = elements.len() / dim as usize; - let values: Vec<_> = elements.iter().copied().map(half::f16::from_f32).collect(); - let elems: ArrayRef = Buffer::copy_from(values.as_slice()).into_array(); - let fsl = FixedSizeListArray::new(elems, dim, Validity::NonNullable, row_count); - - let ext_dtype = ExtDType::::try_new(EmptyMetadata, fsl.dtype().clone())?.erased(); - Ok(ExtensionArray::new(ext_dtype, fsl.into_array()).into_array()) - } - fn tensor_snapshot(array: ArrayRef) -> VortexResult<(DType, Vec, Vec)> { let mut ctx = SESSION.create_execution_ctx(); let ext: ExtensionArray = array.execute(&mut ctx)?; let validity = (0..ext.len()) - .map(|i| ext.is_valid(i)) + .map(|i| ext.is_valid(i, &mut ctx)) .collect::>>()?; let storage: FixedSizeListArray = ext.storage_array().clone().execute(&mut ctx)?; let elements: PrimitiveArray = storage.elements().clone().execute(&mut ctx)?; @@ -572,9 +849,9 @@ mod tests { let storage: FixedSizeListArray = actual.storage_array().clone().execute(&mut ctx)?; let elements: PrimitiveArray = storage.elements().clone().execute(&mut ctx)?; - assert!(actual.is_valid(0)?); - assert!(!actual.is_valid(1)?); - assert!(!actual.is_valid(2)?); + assert!(actual.is_valid(0, &mut ctx)?); + assert!(!actual.is_valid(1, &mut ctx)?); + assert!(!actual.is_valid(2, &mut ctx)?); assert_close(&elements.as_slice::()[..2], &[3.0, 4.0]); Ok(()) } @@ -585,7 +862,7 @@ mod tests { let rhs = PrimitiveArray::from_iter([1.0f64, 1.0]).into_array(); let mut ctx = SESSION.create_execution_ctx(); - let result = L2Denorm::try_new_array(&ApproxOptions::Exact, lhs, rhs, 2, &mut ctx); + let result = L2Denorm::try_new_array(lhs, rhs, 2, &mut ctx); assert!(result.is_err()); } @@ -595,18 +872,18 @@ mod tests { let rhs = PrimitiveArray::from_iter([1.0f64, 1.0]).into_array(); let mut ctx = SESSION.create_execution_ctx(); - let result = L2Denorm::try_new_array(&ApproxOptions::Exact, lhs, rhs, 2, &mut ctx); + let result = L2Denorm::try_new_array(lhs, rhs, 2, &mut ctx); assert!(result.is_err()); Ok(()) } #[test] fn l2_denorm_rejects_integer_tensor_lhs() -> VortexResult<()> { - let lhs = integer_tensor_array(&[2], &[1, 2, 3, 4])?; + let lhs = tensor_array(&[2], &[1i32, 2, 3, 4])?; let rhs = PrimitiveArray::from_iter([1.0f64, 1.0]).into_array(); let mut ctx = SESSION.create_execution_ctx(); - let result = L2Denorm::try_new_array(&ApproxOptions::Exact, lhs, rhs, 2, &mut ctx); + let result = L2Denorm::try_new_array(lhs, rhs, 2, &mut ctx); assert!(result.is_err()); Ok(()) } @@ -617,17 +894,17 @@ mod tests { let rhs = PrimitiveArray::from_iter([1.0f32, 1.0]).into_array(); let mut ctx = SESSION.create_execution_ctx(); - let result = L2Denorm::try_new_array(&ApproxOptions::Exact, lhs, rhs, 2, &mut ctx); + let result = L2Denorm::try_new_array(lhs, rhs, 2, &mut ctx); assert!(result.is_err()); Ok(()) } #[test] fn validate_l2_normalized_rows_accepts_normalized_f16_input() -> VortexResult<()> { - let input = f16_vector_array(2, &[3.0, 4.0, 0.0, 0.0])?; + let input = vector_array(2, &[3.0f32, 4.0, 0.0, 0.0].map(half::f16::from_f32))?; let mut ctx = SESSION.create_execution_ctx(); - let roundtrip = normalize_as_l2_denorm(&ApproxOptions::Exact, input, &mut ctx)?; - validate_l2_normalized_rows(roundtrip.child_at(0).clone(), &mut ctx)?; + let roundtrip = normalize_as_l2_denorm(input, &mut ctx)?; + validate_l2_normalized_rows_against_norms(&roundtrip.child_at(0).clone(), None, &mut ctx)?; Ok(()) } @@ -635,7 +912,7 @@ mod tests { fn validate_l2_normalized_rows_rejects_unnormalized_input() -> VortexResult<()> { let input = vector_array(2, &[3.0, 4.0, 1.0, 0.0])?; let mut ctx = SESSION.create_execution_ctx(); - let result = validate_l2_normalized_rows(input, &mut ctx); + let result = validate_l2_normalized_rows_against_norms(&input, None, &mut ctx); assert!(result.is_err()); Ok(()) } @@ -646,7 +923,7 @@ mod tests { let norms = PrimitiveArray::from_iter([5.0f64, 1.0]).into_array(); let mut ctx = SESSION.create_execution_ctx(); - let result = L2Denorm::try_new_array(&ApproxOptions::Exact, normalized, norms, 2, &mut ctx); + let result = L2Denorm::try_new_array(normalized, norms, 2, &mut ctx); assert!(result.is_err()); Ok(()) } @@ -657,7 +934,7 @@ mod tests { let norms = PrimitiveArray::from_iter([0.0f64, 0.0]).into_array(); let mut ctx = SESSION.create_execution_ctx(); - let result = L2Denorm::try_new_array(&ApproxOptions::Exact, normalized, norms, 2, &mut ctx); + let result = L2Denorm::try_new_array(normalized, norms, 2, &mut ctx); assert!(result.is_err()); Ok(()) } @@ -668,7 +945,7 @@ mod tests { let norms = PrimitiveArray::from_iter([1.0f64, -1.0]).into_array(); let mut ctx = SESSION.create_execution_ctx(); - let result = L2Denorm::try_new_array(&ApproxOptions::Exact, normalized, norms, 2, &mut ctx); + let result = L2Denorm::try_new_array(normalized, norms, 2, &mut ctx); assert!(result.is_err()); Ok(()) } @@ -678,8 +955,7 @@ mod tests { let normalized = vector_array(2, &[3.0, 4.0, 1.0, 0.0])?; let norms = PrimitiveArray::from_iter([5.0f64, 1.0]).into_array(); - let result = - unsafe { L2Denorm::new_array_unchecked(&ApproxOptions::Exact, normalized, norms, 2) }; + let result = unsafe { L2Denorm::new_array_unchecked(normalized, norms, 2) }; assert!(result.is_ok()); Ok(()) } @@ -688,7 +964,7 @@ mod tests { fn normalize_as_l2_denorm_roundtrips_vectors() -> VortexResult<()> { let input = vector_array(3, &[3.0, 4.0, 0.0, 0.0, 0.0, 0.0])?; let mut ctx = SESSION.create_execution_ctx(); - let roundtrip = normalize_as_l2_denorm(&ApproxOptions::Exact, input.clone(), &mut ctx)?; + let roundtrip = normalize_as_l2_denorm(input.clone(), &mut ctx)?; let actual = roundtrip.into_array().execute(&mut ctx)?; assert_tensor_arrays_eq(actual, input)?; @@ -699,7 +975,7 @@ mod tests { fn normalize_as_l2_denorm_roundtrips_fixed_shape_tensors() -> VortexResult<()> { let input = tensor_array(&[2, 2], &[1.0, 2.0, 3.0, 4.0, 0.0, 0.0, 0.0, 0.0])?; let mut ctx = SESSION.create_execution_ctx(); - let roundtrip = normalize_as_l2_denorm(&ApproxOptions::Exact, input.clone(), &mut ctx)?; + let roundtrip = normalize_as_l2_denorm(input.clone(), &mut ctx)?; let actual = roundtrip.into_array().execute(&mut ctx)?; assert_tensor_arrays_eq(actual, input)?; @@ -710,7 +986,7 @@ mod tests { fn normalize_as_l2_denorm_supports_constant_tensors() -> VortexResult<()> { let input = constant_tensor_array(&[2], &[3.0, 4.0], 3)?; let mut ctx = SESSION.create_execution_ctx(); - let roundtrip = normalize_as_l2_denorm(&ApproxOptions::Exact, input.clone(), &mut ctx)?; + let roundtrip = normalize_as_l2_denorm(input.clone(), &mut ctx)?; let actual = roundtrip.into_array().execute(&mut ctx)?; assert_tensor_arrays_eq(actual, input)?; @@ -719,20 +995,58 @@ mod tests { #[test] fn normalize_as_l2_denorm_supports_constant_vectors() -> VortexResult<()> { - let input = constant_vector_array(&[3.0, 4.0], 2)?; + let input = Vector::constant_array(&[3.0, 4.0], 2)?; let mut ctx = SESSION.create_execution_ctx(); - let roundtrip = normalize_as_l2_denorm(&ApproxOptions::Exact, input.clone(), &mut ctx)?; + let roundtrip = normalize_as_l2_denorm(input.clone(), &mut ctx)?; let actual = roundtrip.into_array().execute(&mut ctx)?; assert_tensor_arrays_eq(actual, input)?; Ok(()) } + #[test] + fn normalize_as_l2_denorm_constant_input_has_constant_children() -> VortexResult<()> { + // The constant fast path in `normalize_as_l2_denorm` must produce an `L2Denorm` whose + // normalized storage and norms child are both still `ConstantArray`s. This is what + // allows downstream ops (cosine similarity, inner product) to short-circuit. + let input = Vector::constant_array(&[3.0, 4.0], 16)?; + let mut ctx = SESSION.create_execution_ctx(); + let roundtrip = normalize_as_l2_denorm(input, &mut ctx)?; + + // The normalized child must be an extension array whose storage is still constant. + let normalized = roundtrip.child_at(0).clone(); + let normalized_ext = normalized + .as_opt::() + .expect("normalized child should be an Extension array"); + assert!( + normalized_ext + .storage_array() + .as_opt::() + .is_some(), + "normalized storage should stay constant after the fast path" + ); + + // The norms child must itself be a ConstantArray with the exact precomputed norm. + let norms = roundtrip.child_at(1).clone(); + let norms_const = norms + .as_opt::() + .expect("norms child should be a ConstantArray"); + assert_close( + &[norms_const + .scalar() + .as_primitive() + .typed_value::() + .expect("norms scalar")], + &[5.0], + ); + Ok(()) + } + #[test] fn normalize_as_l2_denorm_uses_zero_rows_for_zero_norms() -> VortexResult<()> { let input = vector_array(2, &[0.0, 0.0, 3.0, 4.0])?; let mut ctx = SESSION.create_execution_ctx(); - let roundtrip = normalize_as_l2_denorm(&ApproxOptions::Exact, input.clone(), &mut ctx)?; + let roundtrip = normalize_as_l2_denorm(input.clone(), &mut ctx)?; let normalized: ExtensionArray = roundtrip.child_at(0).clone().execute(&mut ctx)?; let storage: FixedSizeListArray = normalized.storage_array().clone().execute(&mut ctx)?; let elements: PrimitiveArray = storage.elements().clone().execute(&mut ctx)?; @@ -742,4 +1056,102 @@ mod tests { assert_tensor_arrays_eq(actual, input)?; Ok(()) } + + /// Builds a non-nullable constant f64 norms array of length `len`. + fn constant_f64_norms(value: f64, len: usize) -> ArrayRef { + ConstantArray::new(Scalar::primitive(value, Nullability::NonNullable), len).into_array() + } + + #[test] + fn l2_denorm_constant_unit_norms_is_noop() -> VortexResult<()> { + // Every stored norm is exactly 1.0, so the constant fast path must short-circuit and + // return the normalized child unchanged. + let normalized = vector_array(3, &[1.0, 0.0, 0.0, 0.0, 1.0, 0.0])?; + let norms = constant_f64_norms(1.0, 2); + + let actual = eval_l2_denorm(normalized.clone(), norms, 2)?; + assert_tensor_arrays_eq(actual, normalized)?; + Ok(()) + } + + #[test] + fn l2_denorm_constant_near_unit_norms_is_noop() -> VortexResult<()> { + // A norm that differs from 1.0 by less than the f64 unit-norm tolerance must still + // hit the fast path and return the normalized child unchanged. + let normalized = vector_array(3, &[1.0, 0.0, 0.0, 0.0, 1.0, 0.0])?; + let norms = constant_f64_norms(1.0 + 1e-12, 2); + + let actual = eval_l2_denorm(normalized.clone(), norms, 2)?; + assert_tensor_arrays_eq(actual, normalized)?; + Ok(()) + } + + #[test] + fn l2_denorm_constant_nonunit_norms_scales_vectors() -> VortexResult<()> { + // A constant norm that is not 1.0 must scale every element of every row by the same + // factor via the backing elements multiplication path. + let normalized = vector_array(3, &[0.6, 0.8, 0.0, 1.0, 0.0, 0.0])?; + let norms = constant_f64_norms(5.0, 2); + + let actual = eval_l2_denorm(normalized, norms, 2)?; + let expected = vector_array(3, &[3.0, 4.0, 0.0, 5.0, 0.0, 0.0])?; + assert_tensor_arrays_eq(actual, expected)?; + Ok(()) + } + + #[test] + fn l2_denorm_constant_nonunit_norms_scales_fixed_shape_tensors() -> VortexResult<()> { + // The same constant-scaling fast path must also cover multi-dimensional fixed-shape + // tensors, where the backing elements buffer spans more than one slot per row. + let normalized = tensor_array(&[2, 2], &[0.5, 0.5, 0.5, 0.5, 1.0, 0.0, 0.0, 0.0])?; + let norms = constant_f64_norms(4.0, 2); + + let actual = eval_l2_denorm(normalized, norms, 2)?; + let expected = tensor_array(&[2, 2], &[2.0, 2.0, 2.0, 2.0, 4.0, 0.0, 0.0, 0.0])?; + assert_tensor_arrays_eq(actual, expected)?; + Ok(()) + } + + /// Build an `L2Denorm` array from a raw input (which may have nullable storage) by running + /// `normalize_as_l2_denorm`. The normalized child ends up non-nullable, and the norms child + /// inherits the input's nullability, giving us two different per-child nullabilities to + /// round-trip. + #[rstest] + #[case::vector(l2_denorm_vector_input())] + #[case::fixed_shape_tensor(l2_denorm_tensor_input())] + fn serde_round_trip(#[case] input: ArrayRef) -> VortexResult<()> { + let mut ctx = SESSION.create_execution_ctx(); + let original = normalize_as_l2_denorm(input, &mut ctx)?.into_array(); + + let scalar_fn_array = original.as_::(); + let children = scalar_fn_array.children(); + + let plugin = ScalarFnArrayPlugin::new(L2Denorm); + let metadata = plugin + .serialize(&original, &SESSION)? + .expect("L2Denorm serialize must produce metadata"); + + let recovered = plugin.deserialize( + original.dtype(), + original.len(), + &metadata, + &[], + &children, + &SESSION, + )?; + + assert_eq!(recovered.dtype(), original.dtype()); + assert_eq!(recovered.len(), original.len()); + assert_eq!(recovered.encoding_id(), original.encoding_id()); + Ok(()) + } + + fn l2_denorm_vector_input() -> ArrayRef { + vector_array(3, &[3.0, 4.0, 0.0, 0.0, 0.0, 0.0]).expect("valid vector array") + } + + fn l2_denorm_tensor_input() -> ArrayRef { + tensor_array(&[2, 2], &[1.0, 2.0, 3.0, 4.0, 0.0, 0.0, 0.0, 0.0]) + .expect("valid tensor array") + } } diff --git a/vortex-tensor/src/scalar_fns/l2_norm.rs b/vortex-tensor/src/scalar_fns/l2_norm.rs index 2632c07cffd..5d741eef55e 100644 --- a/vortex-tensor/src/scalar_fns/l2_norm.rs +++ b/vortex-tensor/src/scalar_fns/l2_norm.rs @@ -6,35 +6,48 @@ use std::fmt::Formatter; use num_traits::Float; +use prost::Message; use vortex_array::ArrayRef; use vortex_array::ExecutionCtx; use vortex_array::IntoArray; +use vortex_array::arrays::Constant; +use vortex_array::arrays::ConstantArray; use vortex_array::arrays::ExtensionArray; use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::ScalarFn as ScalarFnArrayEncoding; use vortex_array::arrays::ScalarFnArray; -use vortex_array::arrays::ScalarFnVTable as ScalarFnArrayVTable; use vortex_array::arrays::extension::ExtensionArrayExt; +use vortex_array::arrays::scalar_fn::ExactScalarFn; use vortex_array::arrays::scalar_fn::ScalarFnArrayExt; +use vortex_array::arrays::scalar_fn::ScalarFnArrayView; +use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayParts; +use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayVTable; use vortex_array::dtype::DType; use vortex_array::dtype::NativePType; use vortex_array::dtype::Nullability; +use vortex_array::dtype::proto::dtype as pb; use vortex_array::expr::Expression; use vortex_array::match_each_float_ptype; +use vortex_array::scalar::Scalar; use vortex_array::scalar_fn::Arity; use vortex_array::scalar_fn::ChildName; +use vortex_array::scalar_fn::EmptyOptions; use vortex_array::scalar_fn::ExecutionArgs; -use vortex_array::scalar_fn::ScalarFn; use vortex_array::scalar_fn::ScalarFnId; use vortex_array::scalar_fn::ScalarFnVTable; +use vortex_array::scalar_fn::TypedScalarFnInstance; +use vortex_array::serde::ArrayChildren; use vortex_buffer::Buffer; use vortex_error::VortexExpect; use vortex_error::VortexResult; use vortex_error::vortex_ensure_eq; +use vortex_error::vortex_err; +use vortex_session::VortexSession; use crate::matcher::AnyTensor; -use crate::scalar_fns::ApproxOptions; use crate::scalar_fns::l2_denorm::L2Denorm; use crate::utils::extract_flat_elements; +use crate::utils::extract_l2_denorm_children; use crate::utils::validate_tensor_float_input; /// L2 norm (Euclidean norm) of a tensor or vector column. @@ -43,14 +56,18 @@ use crate::utils::validate_tensor_float_input; /// /// The input must be a tensor-like extension array with a float element type. The output is a float /// column of the same float type. +/// +/// When the input is wrapped in [`L2Denorm`], this operator treats the stored norms as +/// authoritative. For lossy encodings such as TurboQuant, that means `L2Norm` may intentionally +/// read the stored norms instead of re-deriving them from fully decoded coordinates. That behavior +/// is part of the lossy storage contract, not a separate lossy-compute mode. #[derive(Clone)] pub struct L2Norm; impl L2Norm { - /// Creates a new [`ScalarFn`] wrapping the L2 norm operation with the given [`ApproxOptions`] - /// controlling approximation behavior. - pub fn new(options: &ApproxOptions) -> ScalarFn { - ScalarFn::new(L2Norm, options.clone()) + /// Creates a new [`TypedScalarFnInstance`] wrapping the L2 norm operation. + pub fn new() -> TypedScalarFnInstance { + TypedScalarFnInstance::new(L2Norm, EmptyOptions) } /// Constructs a [`ScalarFnArray`] that lazily computes the L2 norm over `child`. @@ -59,20 +76,16 @@ impl L2Norm { /// /// Returns an error if the [`ScalarFnArray`] cannot be constructed (e.g. due to dtype /// mismatches). - pub fn try_new_array( - options: &ApproxOptions, - child: ArrayRef, - len: usize, - ) -> VortexResult { - ScalarFnArray::try_new(L2Norm::new(options).erased(), vec![child], len) + pub fn try_new_array(child: ArrayRef, len: usize) -> VortexResult { + ScalarFnArray::try_new(L2Norm::new().erased(), vec![child], len) } } impl ScalarFnVTable for L2Norm { - type Options = ApproxOptions; + type Options = EmptyOptions; fn id(&self) -> ScalarFnId { - ScalarFnId::from("vortex.tensor.l2_norm") + ScalarFnId::new("vortex.tensor.l2_norm") } fn arity(&self, _options: &Self::Options) -> Arity { @@ -119,23 +132,44 @@ impl ScalarFnVTable for L2Norm { let tensor_match = ext .metadata_opt::() .vortex_expect("we already validated this in `return_dtype`"); - let tensor_flat_size = tensor_match.list_size(); + let tensor_flat_size = tensor_match.list_size() as usize; let element_ptype = tensor_match.element_ptype(); - // L2Norm(L2Denorm(normalized, norms)) == norms, since normalized vectors have unit norm - // and L2 norms are non-negative. This avoids decompressing the TQ child just to recompute - // norms that are already stored. - if let Some(sfn) = input_ref.as_opt::() - && sfn.scalar_fn().as_opt::().is_some() - { - let norms: PrimitiveArray = sfn.child_at(1).clone().execute(ctx)?; + let norm_dtype = DType::Primitive(element_ptype, ext.nullability()); - vortex_ensure_eq!( - norms.dtype(), - &DType::Primitive(element_ptype, input_ref.dtype().nullability()) - ); + // L2Norm(L2Denorm(normalized, norms)) is defined to read back the authoritative stored + // norms. Exact callers of lossy encodings like TurboQuant opt into that storage semantics + // instead of forcing a decode-and-recompute path here. + if input_ref.is::>() { + let (_, norms) = extract_l2_denorm_children(&input_ref); + vortex_ensure_eq!(norms.dtype(), &norm_dtype); + return Ok(norms); + } - return Ok(norms.into_array()); + // Optimize for the constant array case. + if let Some(array) = input_ref.as_opt::() { + let scalar = array.scalar().as_extension().to_storage_scalar(); + + let Some(elements) = scalar.as_list().elements() else { + return Ok(ConstantArray::new(Scalar::null(norm_dtype), row_count).into_array()); + }; + + let norm_scalar = match_each_float_ptype!(element_ptype, |T| { + let values: Vec = elements + .iter() + .map(|s| { + s.as_primitive() + .as_::() + .vortex_expect("element was somehow not the correct float") + }) + .collect(); + let norm = l2_norm_row::(&values); + + Scalar::try_new(norm_dtype, Some(norm.into())) + })?; + + let norms = ConstantArray::new(norm_scalar, row_count).into_array(); + return Ok(norms); } let input: ExtensionArray = input_ref.execute(ctx)?; @@ -173,6 +207,49 @@ impl ScalarFnVTable for L2Norm { } } +/// Metadata for a serialized [`L2Norm`] array: the single `input` child's [`DType`], which carries +/// the extension type (`FixedShapeTensor` vs `Vector`), dimension, and nullability that are not +/// recoverable from the parent's primitive-float output. +#[derive(Clone, prost::Message)] +pub(super) struct L2NormMetadata { + #[prost(message, optional, tag = "1")] + input_dtype: Option, +} + +impl ScalarFnArrayVTable for L2Norm { + fn serialize( + &self, + view: &ScalarFnArrayView, + _session: &VortexSession, + ) -> VortexResult>> { + let scalar_fn_array = view.as_::(); + let input_dtype = Some(scalar_fn_array.child_at(0).dtype().try_into()?); + Ok(Some(L2NormMetadata { input_dtype }.encode_to_vec())) + } + + fn deserialize( + &self, + _dtype: &DType, + len: usize, + metadata: &[u8], + children: &dyn ArrayChildren, + session: &VortexSession, + ) -> VortexResult> { + let metadata = L2NormMetadata::decode(metadata) + .map_err(|e| vortex_err!("Failed to decode L2NormMetadata: {e}"))?; + let input_pb = metadata + .input_dtype + .as_ref() + .ok_or_else(|| vortex_err!("L2NormMetadata missing input_dtype"))?; + let input_dtype = DType::from_proto(input_pb, session)?; + let child = children.get(0, &input_dtype, len)?; + Ok(ScalarFnArrayParts { + options: EmptyOptions, + children: vec![child], + }) + } +} + /// Computes the L2 norm (Euclidean norm) of a float slice. /// /// Returns `sqrt(sum(v_i^2))`. A zero-length or all-zero input produces `0.0`. @@ -186,33 +263,38 @@ fn l2_norm_row(v: &[T]) -> T { #[cfg(test)] mod tests { - use std::sync::LazyLock; use rstest::rstest; + use vortex_array::ArrayPlugin; use vortex_array::ArrayRef; use vortex_array::IntoArray; use vortex_array::VortexSessionExecute; + use vortex_array::arrays::Constant; + use vortex_array::arrays::ConstantArray; use vortex_array::arrays::MaskedArray; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::ScalarFnArray; - use vortex_array::scalar_fn::ScalarFn; - use vortex_array::session::ArraySession; + use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin; + use vortex_array::dtype::DType; + use vortex_array::dtype::Nullability; + use vortex_array::dtype::PType; + use vortex_array::dtype::extension::ExtDType; + use vortex_array::extension::EmptyMetadata; + use vortex_array::scalar::Scalar; use vortex_array::validity::Validity; use vortex_error::VortexResult; - use vortex_session::VortexSession; - use crate::scalar_fns::ApproxOptions; use crate::scalar_fns::l2_norm::L2Norm; + use crate::tests::SESSION; + use crate::types::vector::Vector; use crate::utils::test_helpers::assert_close; + use crate::utils::test_helpers::literal_vector_array; use crate::utils::test_helpers::tensor_array; use crate::utils::test_helpers::vector_array; - static SESSION: LazyLock = - LazyLock::new(|| VortexSession::empty().with::()); - /// Evaluates L2 norm on a tensor/vector array and returns the result as `Vec`. fn eval_l2_norm(input: ArrayRef, len: usize) -> VortexResult> { - let scalar_fn = ScalarFn::new(L2Norm, ApproxOptions::Exact).erased(); + let scalar_fn = L2Norm::new().erased(); let result = ScalarFnArray::try_new(scalar_fn, vec![input], len)?; let mut ctx = SESSION.create_execution_ctx(); let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; @@ -267,15 +349,106 @@ mod tests { let arr = tensor_array(&[2], &[3.0, 4.0, 0.0, 0.0])?; let arr = MaskedArray::try_new(arr, Validity::from_iter([true, false]))?.into_array(); - let scalar_fn = ScalarFn::new(L2Norm, ApproxOptions::Exact).erased(); + let scalar_fn = L2Norm::new().erased(); let result = ScalarFnArray::try_new(scalar_fn, vec![arr], 2)?; let mut ctx = SESSION.create_execution_ctx(); let prim: PrimitiveArray = result.into_array().execute(&mut ctx)?; // Row 0: norm = 5.0, row 1: null. - assert!(prim.is_valid(0)?); - assert!(!prim.is_valid(1)?); + assert!(prim.is_valid(0, &mut ctx)?); + assert!(!prim.is_valid(1, &mut ctx)?); assert_close(&[prim.as_slice::()[0]], &[5.0]); Ok(()) } + + /// A constant input whose scalar is a non-null tensor should short-circuit to a + /// [`ConstantArray`] output whose scalar is the precomputed norm. Uses [`execute_until`] so + /// execution stops at the [`Constant`] encoding instead of canonicalizing into a + /// [`PrimitiveArray`]. + #[test] + fn constant_non_null_input_yields_constant_output() -> VortexResult<()> { + let input = literal_vector_array(&[3.0f64, 4.0], 4); + + let scalar_fn = L2Norm::new().erased(); + let result = ScalarFnArray::try_new(scalar_fn, vec![input], 4)?.into_array(); + let mut ctx = SESSION.create_execution_ctx(); + let output = result.execute_until::(&mut ctx)?; + + let constant = output + .as_opt::() + .expect("L2Norm over a constant input must produce a constant output"); + assert_eq!(constant.len(), 4); + let norm = constant + .scalar() + .as_primitive() + .as_::() + .expect("norm scalar must be a non-null primitive"); + assert_close(&[norm], &[5.0]); + Ok(()) + } + + /// A constant input whose scalar is null should short-circuit to a null [`ConstantArray`] of + /// the correct primitive dtype and length. + #[test] + fn constant_null_input_yields_null_constant_output() -> VortexResult<()> { + let storage_dtype = DType::FixedSizeList( + DType::Primitive(PType::F64, Nullability::NonNullable).into(), + 2, + Nullability::Nullable, + ); + let ext_dtype = ExtDType::::try_new(EmptyMetadata, storage_dtype)?.erased(); + let null_scalar = Scalar::null(DType::Extension(ext_dtype)); + let input = ConstantArray::new(null_scalar, 3).into_array(); + + let scalar_fn = L2Norm::new().erased(); + let result = ScalarFnArray::try_new(scalar_fn, vec![input], 3)?.into_array(); + let mut ctx = SESSION.create_execution_ctx(); + let output = result.execute_until::(&mut ctx)?; + + let constant = output + .as_opt::() + .expect("null constant input must produce a constant output"); + assert_eq!(constant.len(), 3); + assert!(constant.scalar().is_null()); + assert_eq!( + constant.dtype(), + &DType::Primitive(PType::F64, Nullability::Nullable) + ); + Ok(()) + } + + #[rstest] + #[case::fixed_shape_tensor(l2_norm_tensor_child(), 2)] + #[case::vector(l2_norm_vector_child(), 2)] + fn serde_round_trip(#[case] child: ArrayRef, #[case] len: usize) -> VortexResult<()> { + let original = L2Norm::try_new_array(child.clone(), len)?.into_array(); + + let plugin = ScalarFnArrayPlugin::new(L2Norm); + let metadata = plugin + .serialize(&original, &SESSION)? + .expect("L2Norm serialize must produce metadata"); + + let children = vec![child]; + let recovered = plugin.deserialize( + original.dtype(), + original.len(), + &metadata, + &[], + &children, + &SESSION, + )?; + + assert_eq!(recovered.dtype(), original.dtype()); + assert_eq!(recovered.len(), original.len()); + assert_eq!(recovered.encoding_id(), original.encoding_id()); + Ok(()) + } + + fn l2_norm_tensor_child() -> ArrayRef { + tensor_array(&[3], &[1.0, 2.0, 3.0, 4.0, 5.0, 6.0]).expect("valid tensor array") + } + + fn l2_norm_vector_child() -> ArrayRef { + vector_array(3, &[1.0, 2.0, 3.0, 4.0, 5.0, 6.0]).expect("valid vector array") + } } diff --git a/vortex-tensor/src/scalar_fns/mod.rs b/vortex-tensor/src/scalar_fns/mod.rs index 32db845a6e2..1d1a362c8af 100644 --- a/vortex-tensor/src/scalar_fns/mod.rs +++ b/vortex-tensor/src/scalar_fns/mod.rs @@ -3,40 +3,8 @@ //! Scalar function expressions defined on tensor and tensor-like extension types. -use std::fmt; - pub mod cosine_similarity; pub mod inner_product; pub mod l2_denorm; pub mod l2_norm; - -/// Options for tensor-related expressions that might have error. -#[derive(Debug, Default, Clone, PartialEq, Eq, Hash)] -pub enum ApproxOptions { - /// Computes the exact result. - #[default] - Exact, - /// Allows approximate results. - Approximate, -} - -impl ApproxOptions { - /// Returns `true` if the option is [`Exact`](Self::Exact). - pub fn is_exact(&self) -> bool { - matches!(self, Self::Exact) - } - - /// Returns `true` if the option is [`Approximate`](Self::Approximate). - pub fn is_approx(&self) -> bool { - matches!(self, Self::Approximate) - } -} - -impl fmt::Display for ApproxOptions { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - match self { - Self::Exact => write!(f, "Exact"), - Self::Approximate => write!(f, "Approximate"), - } - } -} +pub mod sorf_transform; diff --git a/vortex-tensor/src/scalar_fns/sorf_transform/mod.rs b/vortex-tensor/src/scalar_fns/sorf_transform/mod.rs new file mode 100644 index 00000000000..26d38e87a1e --- /dev/null +++ b/vortex-tensor/src/scalar_fns/sorf_transform/mod.rs @@ -0,0 +1,146 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! SORF inverse transform scalar function. +//! +//! SORF (Structured Orthogonal Random Features, [Yu et al. 2016][sorf-paper]) is a fast structured +//! approximation to a random orthogonal matrix. It composes random sign diagonals with the +//! Walsh-Hadamard transform to achieve O(d log d) matrix-vector products instead of the O(d^2) cost +//! of a dense orthogonal matrix. +//! +//! This module wraps a [`Vector`] extension array whose dimension is the padded SORF dimension +//! (e.g. a `Vector` wrapping `FSL(Dict(codes, centroids))`) and applies the inverse SORF transform +//! at execution time, producing a [`Vector`] extension array with the original (pre-padding) +//! dimensionality. +//! +//! The transform parameters are stored as a deterministic seed in [`SorfOptions`], so the +//! [`SorfMatrix`] is reconstructed cheaply at decode time. Sign diagonals are defined by Vortex's +//! frozen local SplitMix64 stream contract rather than by an external RNG crate. +//! +//! # Input element type: `f32` only (TODO(connor): for now...) +//! +//! The child [`Vector`] **must** have `f32` storage elements. This is a hard constraint that is +//! enforced by `SorfTransform`'s `return_dtype` check. Callers with `f16` or `f64` source data need +//! to cast to `f32` before wrapping in a [`Vector`] and handing it to SorfTransform. +//! +//! The reason for this constraint is that TurboQuant (the only production caller today) stores its +//! dictionary centroids as `f32`, and the SORF transform itself operates internally in `f32`. +//! +//! Supporting other float storage types would require an implicit up-/down-cast that we do not yet +//! want to bake into SorfTransform. This restriction is intentional and may be relaxed in the +//! future, but today it is load-bearing. +//! +//! # Output element type +//! +//! The output [`Vector`]'s element type is whatever [`SorfOptions::element_ptype`] is set to. It +//! does **not** have to match the child's `f32` storage: we apply an explicit `f32 -> T` cast +//! while materializing the output. This lets SorfTransform hand its result directly to a +//! downstream consumer (e.g. [`L2Denorm`](crate::scalar_fns::l2_denorm::L2Denorm)) whose +//! element-type expectation may differ from the `f32` the transform operated on internally. +//! +//! [sorf-paper]: https://proceedings.neurips.cc/paper_files/paper/2016/file/53adaf494dc89ef7196d73636eb2451b-Paper.pdf +//! [`Vector`]: crate::vector::Vector + +use std::fmt; +use std::fmt::Formatter; + +use vortex_array::ArrayRef; +use vortex_array::arrays::ScalarFnArray; +use vortex_array::dtype::PType; +use vortex_array::scalar_fn::TypedScalarFnInstance; +use vortex_error::VortexResult; +use vortex_error::vortex_ensure; + +mod rotation; +mod splitmix64; +pub use rotation::SorfMatrix; + +mod vtable; + +/// Inverse SORF orthogonal transform scalar function. +/// +/// Takes a [`Vector`](crate::vector::Vector) extension child at the padded dimension with `f32` +/// storage, applies the inverse structured Walsh-Hadamard orthogonal transform, truncates to the +/// original (pre-padding) dimension, casts element-wise to [`SorfOptions::element_ptype`], and +/// wraps the result in a new [`Vector`](crate::vector::Vector) extension array. +/// +/// See the [module-level docs](crate::scalar_fns::sorf_transform) for the rationale behind the +/// `f32`-only input constraint. +#[derive(Clone)] +pub struct SorfTransform; + +/// Options for the SORF inverse transform scalar function. +/// +/// Stored in the [`ScalarFnArray`] and used to deterministically reconstruct the +/// [`SorfMatrix`] at decode time. +#[derive(Clone, Debug, PartialEq, Eq, Hash)] +pub struct SorfOptions { + /// Seed used to generate the structured sign diagonals via Vortex's frozen SplitMix64 stream. + pub seed: u64, + /// Number of sign-diagonal + WHT rounds in the structured orthogonal transform. + pub num_rounds: u8, + /// Original vector dimension (before power-of-2 padding). The output + /// [`Vector`](crate::vector::Vector) has this dimension. + pub dimensions: u32, + /// Element type of the output [`Vector`](crate::vector::Vector). The child input must always + /// be `f32`, but the output can be any float type (`F16`, `F32`, `F64`); the final + /// `f32 -> element_ptype` cast happens while building the output. + pub element_ptype: PType, +} + +impl SorfTransform { + /// Creates a new [`TypedScalarFnInstance`] wrapping the SORF inverse transform with the given options. + pub fn new(options: &SorfOptions) -> TypedScalarFnInstance { + TypedScalarFnInstance::new(SorfTransform, options.clone()) + } + + /// Constructs a validated [`ScalarFnArray`] that lazily applies the inverse SORF transform. + /// + /// The `child` must be a [`Vector`] extension array (or an array that executes to one) with: + /// + /// - dimension equal to `padded_dim` (i.e. `options.dimension.next_power_of_two()`), and + /// - `f32` storage elements. This is a hard requirement today; see the + /// [module-level docs](crate::scalar_fns::sorf_transform) for the rationale. + /// + /// The output [`Vector`] has dimension `options.dimension` and element type + /// `options.element_ptype`. + /// + /// [`Vector`]: crate::vector::Vector + pub fn try_new_array( + options: &SorfOptions, + child: ArrayRef, + len: usize, + ) -> VortexResult { + validate_sorf_options(options)?; + + ScalarFnArray::try_new(SorfTransform::new(options).erased(), vec![child], len) + } +} + +/// Checks that the SORF configuration is valid. +pub(crate) fn validate_sorf_options(options: &SorfOptions) -> VortexResult<()> { + vortex_ensure!( + options.num_rounds >= 1, + "SorfTransform num_rounds must be >= 1, got {}", + options.num_rounds + ); + vortex_ensure!( + options.element_ptype.is_float(), + "SorfTransform element_ptype must be a float type, got {}", + options.element_ptype + ); + Ok(()) +} + +impl fmt::Display for SorfOptions { + fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { + write!( + f, + "SorfOptions(seed={}, rounds={}, dim={}, ptype={})", + self.seed, self.num_rounds, self.dimensions, self.element_ptype + ) + } +} + +#[cfg(test)] +mod tests; diff --git a/vortex-tensor/src/encodings/turboquant/array/rotation.rs b/vortex-tensor/src/scalar_fns/sorf_transform/rotation.rs similarity index 65% rename from vortex-tensor/src/encodings/turboquant/array/rotation.rs rename to vortex-tensor/src/scalar_fns/sorf_transform/rotation.rs index 57c236c821a..ff8aebd0f11 100644 --- a/vortex-tensor/src/encodings/turboquant/array/rotation.rs +++ b/vortex-tensor/src/scalar_fns/sorf_transform/rotation.rs @@ -1,15 +1,31 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -//! Deterministic random rotation for TurboQuant. +//! SORF (Structured Orthogonal Random Features) orthogonal transform. //! -//! The TurboQuant paper analyzes a full random orthogonal rotation. The current implementation -//! uses a cheaper structured Walsh-Hadamard-based surrogate instead of a dense d x d matrix. +//! Implements the SORF construction from [Yu et al. 2016][sorf-paper]: a fast structured +//! approximation to a random orthogonal matrix using random sign diagonals interleaved with the +//! Fast Walsh-Hadamard Transform (FWHT). //! -//! Concretely, this applies three rounds of random sign diagonals interleaved with the Fast -//! Walsh-Hadamard Transform (FWHT): `D3 * H * D2 * H * D1 * H`, followed by normalization. This is -//! a SORF-style structured approximation to a random orthogonal matrix, chosen for O(d log d) -//! encode/decode cost and compact serialized parameters. +//! For `k` rounds, the transform is `norm * H * D_k * ... * H * D_1 * x`, where `D_1` is the +//! first sign diagonal applied. The number of rounds is configurable (typically 3). Each round +//! applies a random sign diagonal `D_i` and then the Hadamard matrix `H`, giving O(d log d) cost +//! per matrix-vector product instead of the O(d^2) cost of a dense orthogonal matrix. +//! +//! Vortex defines those sign diagonals using a frozen local SplitMix64 stream rather than an +//! external RNG crate. The contract is: +//! +//! - state is a single `u64` seed, +//! - each `next_u64()` call uses the SplitMix64 reference algorithm with wrapping `u64` +//! arithmetic, +//! - signs are generated in round-major, block-major order, +//! - each generated `u64` contributes 64 signs in least-significant-bit-first order, +//! - bit `1` means `+1` and bit `0` means `-1`. +//! +//! This makes SORF sign generation stable as a Vortex format contract even if external RNG +//! implementations change. +//! +//! [sorf-paper]: https://proceedings.neurips.cc/paper_files/paper/2016/file/53adaf494dc89ef7196d73636eb2451b-Paper.pdf //! //! The FWHT exploits the Kronecker product structure of the Hadamard matrix (`H_n = H_2 (x) H_2 //! (x) ... (x) H_2`, with `log2(n)` factors) to compute the matrix-vector product in O(n log n) @@ -26,18 +42,22 @@ //! floating-point multiply, which avoids FP dependency chains and auto-vectorizes into //! `vpxor`/`veor`. -use rand::RngExt; -use rand::SeedableRng; -use rand::rngs::StdRng; use vortex_error::VortexResult; use vortex_error::vortex_ensure; +use super::splitmix64::SplitMix64; + /// IEEE 754 sign bit mask for f32. const F32_SIGN_BIT: u32 = 0x8000_0000; -/// A Walsh-Hadamard-based structured surrogate for a random orthogonal rotation. -pub struct RotationMatrix { - /// Flat XOR masks for all `num_rounds` diagonal matrices, total length `num_rounds * padded_dim`. +/// A Walsh-Hadamard-based structured orthogonal transform matrix. +/// +/// All computation is done in f32. The sign diagonals are stored as IEEE 754 XOR masks on +/// f32 bit patterns, and the Walsh-Hadamard butterfly operates on `&mut [f32]` slices. +pub struct SorfMatrix { + /// Flat XOR masks for all `num_rounds` diagonal matrices, total length + /// `num_rounds * padded_dim`. + /// /// Indexed as `round * padded_dim + i`. `0x00000000` = multiply by +1 (no-op), `0x80000000` = /// multiply by -1 (flip sign bit). sign_masks: Vec, @@ -49,22 +69,25 @@ pub struct RotationMatrix { norm_factor: f32, } -impl RotationMatrix { - /// Create a new structured Walsh-Hadamard-based rotation from a deterministic seed. - pub fn try_new(seed: u64, dimension: usize, num_rounds: usize) -> VortexResult { +impl SorfMatrix { + /// Create a new structured Walsh-Hadamard-based orthogonal transform from a deterministic + /// seed. + /// + /// The seed is expanded using Vortex's frozen local SplitMix64 stream. Signs are generated in + /// round-major, block-major order, with each `u64` contributing 64 sign bits in + /// least-significant-bit-first order. + pub fn try_new(seed: u64, dimensions: usize, num_rounds: usize) -> VortexResult { vortex_ensure!(num_rounds >= 1, "num_rounds must be >= 1, got {num_rounds}"); - let padded_dim = dimension.next_power_of_two(); - let mut rng = StdRng::seed_from_u64(seed); - - let mut sign_masks = Vec::with_capacity(num_rounds * padded_dim); - for _ in 0..num_rounds { - sign_masks.extend(gen_random_sign_masks(&mut rng, padded_dim)); - } + let padded_dim = dimensions.next_power_of_two(); + let sign_masks = gen_sign_masks_from_seed(seed, padded_dim, num_rounds); + // Compute in f64 for precision, then store as f32 since the WHT operates on f32 buffers. + // The result is always in (0, 1] for any valid padded_dim >= 2 and num_rounds >= 1, so + // the f64 -> f32 cast is a precision loss only (it cannot overflow to infinity). #[expect( clippy::cast_possible_truncation, - reason = "Intentional f64 -> f32 truncation for normalization factor." + reason = "the norm factor is in (0, 1] so the f64 -> f32 cast cannot overflow" )] let norm_factor = (padded_dim as f64).powf(-(num_rounds as f64) / 2.0) as f32; @@ -76,7 +99,14 @@ impl RotationMatrix { }) } - /// Apply forward rotation: `output = R(input)`. + /// Returns the padded dimension (next power of 2 >= dim). + /// + /// All `rotate`/`inverse_rotate` buffers must be this length. + pub fn padded_dim(&self) -> usize { + self.padded_dim + } + + /// Apply the forward orthogonal transform: `output = R(input)`. /// /// Both `input` and `output` must have length [`padded_dim()`](Self::padded_dim). The caller is /// responsible for zero-padding input beyond `dim` positions. @@ -88,7 +118,7 @@ impl RotationMatrix { self.apply_srht(output); } - /// Apply inverse rotation: `output = R⁻¹(input)`. + /// Apply the inverse orthogonal transform: `output = R⁻¹(input)`. /// /// Both `input` and `output` must have length `padded_dim()`. pub fn inverse_rotate(&self, input: &[f32], output: &mut [f32]) { @@ -99,19 +129,7 @@ impl RotationMatrix { self.apply_inverse_srht(output); } - /// Returns the number of sign-diagonal + WHT rounds. - pub fn num_rounds(&self) -> usize { - self.num_rounds - } - - /// Returns the padded dimension (next power of 2 >= dim). - /// - /// All rotate/inverse_rotate buffers must be this length. - pub fn padded_dim(&self) -> usize { - self.padded_dim - } - - /// Apply the structured rotation: `D_k · H · ... · D₁ · H · x`, with normalization. + /// Apply the forward structured transform: `norm · H · D_k · ... · H · D₁ · x`. fn apply_srht(&self, buf: &mut [f32]) { for round in 0..self.num_rounds { let offset = round * self.padded_dim; @@ -123,9 +141,9 @@ impl RotationMatrix { buf.iter_mut().for_each(|val| *val *= norm); } - /// Apply the inverse structured rotation. + /// Apply the inverse structured transform. /// - /// Forward is: `norm · H · D_k · H · ... · D₁ · H`. + /// Forward is: `norm · H · D_k · ... · H · D₁`. /// Inverse is: `norm · D₁ · H · ... · D_k · H`. fn apply_inverse_srht(&self, buf: &mut [f32]) { for round in (0..self.num_rounds).rev() { @@ -144,6 +162,7 @@ impl RotationMatrix { /// Convention: `1` = positive (+1), `0` = negative (-1). The output has length /// `num_rounds * padded_dim` and is suitable for bitpacking via FastLanes /// `bitpack_encode(..., 1, None)`. + #[cfg(test)] pub fn export_inverse_signs_u8(&self) -> Vec { let total = self.num_rounds * self.padded_dim; let mut out = Vec::with_capacity(total); @@ -158,7 +177,7 @@ impl RotationMatrix { out } - /// Reconstruct a [`RotationMatrix`] from unpacked `u8` 0/1 values. + /// Reconstruct a [`SorfMatrix`] from unpacked `u8` 0/1 values. /// /// The input must have length `num_rounds * padded_dim` with signs in inverse application /// order `[D_k | ... | D₁]` (as produced by [`export_inverse_signs_u8`]). Convention: @@ -166,6 +185,7 @@ impl RotationMatrix { /// /// This is the decode-time reconstruction path: FastLanes SIMD-unpacks the stored /// [`BitPackedArray`] into `&[u8]`, which is passed here. + #[cfg(test)] pub fn from_u8_slice( signs_u8: &[u8], dimension: usize, @@ -196,9 +216,11 @@ impl RotationMatrix { } } + // Same norm factor computation as `try_new`. See the comment there for why this cast + // cannot overflow. #[expect( clippy::cast_possible_truncation, - reason = "Intentional f64 -> f32 truncation for normalization factor." + reason = "the norm factor is in (0, 1] so the f64 -> f32 cast cannot overflow" )] let norm_factor = (padded_dim as f64).powf(-(num_rounds as f64) / 2.0) as f32; @@ -211,24 +233,40 @@ impl RotationMatrix { } } -/// Generate a vector of random XOR sign masks. -fn gen_random_sign_masks(rng: &mut StdRng, len: usize) -> Vec { - (0..len) - .map(|_| { - if rng.random_bool(0.5) { - 0u32 // +1: no-op - } else { - F32_SIGN_BIT // -1: flip sign bit - } - }) - .collect() +/// Generate XOR sign masks from the frozen local SplitMix64 stream. +/// +/// Signs are produced in round-major, block-major order. For each block we call +/// [`SplitMix64::next_u64`] exactly once and unpack its bits from least significant to most +/// significant. Bit `1` means positive sign / `0x00000000`; bit `0` means negative sign / +/// [`F32_SIGN_BIT`]. +fn gen_sign_masks_from_seed(seed: u64, padded_dim: usize, num_rounds: usize) -> Vec { + let mut rng = SplitMix64::new(seed); + let mut sign_masks = Vec::with_capacity(num_rounds * padded_dim); + + for _round in 0..num_rounds { + for base_idx in (0..padded_dim).step_by(64) { + let word = rng.next_u64(); + let bits_in_block = (padded_dim - base_idx).min(64); + sign_masks.extend((0..bits_in_block).map(|bit_idx| sign_mask_from_word(word, bit_idx))); + } + } + + sign_masks +} + +/// Convert one bit from a SplitMix64 output word into an XOR sign mask. +fn sign_mask_from_word(word: u64, bit_idx: usize) -> u32 { + if ((word >> bit_idx) & 1) != 0 { + 0u32 + } else { + F32_SIGN_BIT + } } /// Apply sign masks via XOR on the IEEE 754 sign bit. /// /// This is branchless and auto-vectorizes into `vpxor` (x86) / `veor` (ARM). Equivalent to /// multiplying each element by +/-1.0, but avoids FP dependency chains. -#[inline] fn apply_signs_xor(buf: &mut [f32], masks: &[u32]) { for (val, &mask) in buf.iter_mut().zip(masks.iter()) { *val = f32::from_bits(val.to_bits() ^ mask); @@ -265,7 +303,6 @@ fn walsh_hadamard_transform(buf: &mut [f32]) { /// This is multiplication by the 2x2 Hadamard kernel `H_2 = [[1, 1], [1, -1]]` on each element /// pair. Factored into a separate function so LLVM can see the slice lengths match and /// auto-vectorize. -#[inline(always)] fn butterfly(lo: &mut [f32], hi: &mut [f32]) { debug_assert_eq!(lo.len(), hi.len()); for (a, b) in lo.iter_mut().zip(hi.iter_mut()) { @@ -282,11 +319,18 @@ mod tests { use vortex_error::VortexResult; use super::*; + use crate::scalar_fns::sorf_transform::splitmix64::SplitMix64; + + fn unpack_sign_bits(word: u64, count: usize) -> Vec { + (0..count) + .map(|bit_idx| u8::from(((word >> bit_idx) & 1) != 0)) + .collect() + } #[test] fn deterministic_from_seed() -> VortexResult<()> { - let r1 = RotationMatrix::try_new(42, 64, 3)?; - let r2 = RotationMatrix::try_new(42, 64, 3)?; + let r1 = SorfMatrix::try_new(42, 64, 3)?; + let r2 = SorfMatrix::try_new(42, 64, 3)?; let pd = r1.padded_dim(); let mut input = vec![0.0f32; pd]; @@ -303,6 +347,48 @@ mod tests { Ok(()) } + #[test] + fn export_inverse_signs_matches_golden_words() -> VortexResult<()> { + let rot = SorfMatrix::try_new(42, 64, 2)?; + let actual = rot.export_inverse_signs_u8(); + let mut rng = SplitMix64::new(42); + let round0_word = rng.next_u64(); + let round1_word = rng.next_u64(); + + let mut expected = Vec::with_capacity(128); + expected.extend(unpack_sign_bits(round1_word, 64)); + expected.extend(unpack_sign_bits(round0_word, 64)); + + assert_eq!(actual, expected); + Ok(()) + } + + #[test] + fn one_word_generates_64_signs_lsb_first() { + let masks = gen_sign_masks_from_seed(42, 64, 1); + assert_eq!(masks.len(), 64); + + let mut rng = SplitMix64::new(42); + let word = rng.next_u64(); + let expected: Vec<_> = (0..64) + .map(|bit_idx| sign_mask_from_word(word, bit_idx)) + .collect(); + assert_eq!(masks, expected); + } + + #[test] + fn tail_block_uses_only_required_bits() { + let masks = gen_sign_masks_from_seed(42, 32, 1); + assert_eq!(masks.len(), 32); + + let mut rng = SplitMix64::new(42); + let word = rng.next_u64(); + let expected: Vec<_> = (0..32) + .map(|bit_idx| sign_mask_from_word(word, bit_idx)) + .collect(); + assert_eq!(masks, expected); + } + /// Verify roundtrip is exact to f32 precision across many dimensions and round counts, /// including non-power-of-two dimensions that require padding. #[rstest] @@ -318,7 +404,7 @@ mod tests { #[case(768, 3)] #[case(1024, 3)] fn roundtrip_exact(#[case] dim: usize, #[case] num_rounds: usize) -> VortexResult<()> { - let rot = RotationMatrix::try_new(42, dim, num_rounds)?; + let rot = SorfMatrix::try_new(42, dim, num_rounds)?; let padded_dim = rot.padded_dim(); let mut input = vec![0.0f32; padded_dim]; @@ -354,7 +440,7 @@ mod tests { #[case(128, 5)] #[case(768, 3)] fn preserves_norm(#[case] dim: usize, #[case] num_rounds: usize) -> VortexResult<()> { - let rot = RotationMatrix::try_new(7, dim, num_rounds)?; + let rot = SorfMatrix::try_new(7, dim, num_rounds)?; let padded_dim = rot.padded_dim(); let mut input = vec![0.0f32; padded_dim]; @@ -377,7 +463,7 @@ mod tests { Ok(()) } - /// Verify that export -> [`from_u8_slice`] produces identical rotation output. + /// Verify that export -> [`from_u8_slice`] produces identical transform output. #[rstest] #[case(64, 3)] #[case(128, 1)] @@ -388,11 +474,11 @@ mod tests { #[case] dim: usize, #[case] num_rounds: usize, ) -> VortexResult<()> { - let rot = RotationMatrix::try_new(42, dim, num_rounds)?; + let rot = SorfMatrix::try_new(42, dim, num_rounds)?; let padded_dim = rot.padded_dim(); let signs_u8 = rot.export_inverse_signs_u8(); - let rot2 = RotationMatrix::from_u8_slice(&signs_u8, dim, num_rounds)?; + let rot2 = SorfMatrix::from_u8_slice(&signs_u8, dim, num_rounds)?; let mut input = vec![0.0f32; padded_dim]; for i in 0..dim { @@ -403,12 +489,12 @@ mod tests { let mut out2 = vec![0.0f32; padded_dim]; rot.rotate(&input, &mut out1); rot2.rotate(&input, &mut out2); - assert_eq!(out1, out2, "Forward rotation mismatch after export/import"); + assert_eq!(out1, out2, "Forward transform mismatch after export/import"); rot.inverse_rotate(&out1, &mut out2); let mut out3 = vec![0.0f32; padded_dim]; rot2.inverse_rotate(&out1, &mut out3); - assert_eq!(out2, out3, "Inverse rotation mismatch after export/import"); + assert_eq!(out2, out3, "Inverse transform mismatch after export/import"); Ok(()) } diff --git a/vortex-tensor/src/scalar_fns/sorf_transform/splitmix64.rs b/vortex-tensor/src/scalar_fns/sorf_transform/splitmix64.rs new file mode 100644 index 00000000000..23345cbcb7d --- /dev/null +++ b/vortex-tensor/src/scalar_fns/sorf_transform/splitmix64.rs @@ -0,0 +1,73 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Frozen local SplitMix64 stream used to define SORF sign diagonals. +//! +//! This is a direct translation of the `splitmix64.c` reference implementation. The state is a +//! single `u64`, and `next_u64()` first adds [`SPLITMIX64_INCREMENT`] with wrapping arithmetic, +//! then applies the two reference mixing steps and final xor-shift. + +/// SplitMix64 additive constant from the reference implementation. +const SPLITMIX64_INCREMENT: u64 = 0x9E37_79B9_7F4A_7C15; + +/// First SplitMix64 mixing multiplier from the reference implementation. +const SPLITMIX64_MUL1: u64 = 0xBF58_476D_1CE4_E5B9; + +/// Second SplitMix64 mixing multiplier from the reference implementation. +const SPLITMIX64_MUL2: u64 = 0x94D0_49BB_1331_11EB; + +/// Frozen local SplitMix64 stream used to define SORF sign diagonals. +pub(crate) struct SplitMix64 { + state: u64, +} + +impl SplitMix64 { + pub(crate) fn new(seed: u64) -> Self { + Self { state: seed } + } + + pub(crate) fn next_u64(&mut self) -> u64 { + self.state = self.state.wrapping_add(SPLITMIX64_INCREMENT); + let mut z = self.state; + z = (z ^ (z >> 30)).wrapping_mul(SPLITMIX64_MUL1); + z = (z ^ (z >> 27)).wrapping_mul(SPLITMIX64_MUL2); + z ^ (z >> 31) + } +} + +#[cfg(test)] +mod tests { + use super::SplitMix64; + + const SPLITMIX64_SEED0_GOLDEN: [u64; 4] = [ + 0xE220_A839_7B1D_CDAF, + 0x6E78_9E6A_A1B9_65F4, + 0x06C4_5D18_8009_454F, + 0xF88B_B8A8_724C_81EC, + ]; + + const SPLITMIX64_SEED42_GOLDEN: [u64; 4] = [ + 0xBDD7_3226_2FEB_6E95, + 0x28EF_E333_B266_F103, + 0x4752_6757_130F_9F52, + 0x581C_E1FF_0E4A_E394, + ]; + + #[test] + fn splitmix64_seed0_matches_golden_outputs() { + let mut rng = SplitMix64::new(0); + let actual: Vec<_> = (0..SPLITMIX64_SEED0_GOLDEN.len()) + .map(|_| rng.next_u64()) + .collect(); + assert_eq!(actual, SPLITMIX64_SEED0_GOLDEN); + } + + #[test] + fn splitmix64_seed42_matches_golden_outputs() { + let mut rng = SplitMix64::new(42); + let actual: Vec<_> = (0..SPLITMIX64_SEED42_GOLDEN.len()) + .map(|_| rng.next_u64()) + .collect(); + assert_eq!(actual, SPLITMIX64_SEED42_GOLDEN); + } +} diff --git a/vortex-tensor/src/scalar_fns/sorf_transform/tests.rs b/vortex-tensor/src/scalar_fns/sorf_transform/tests.rs new file mode 100644 index 00000000000..46abc66db71 --- /dev/null +++ b/vortex-tensor/src/scalar_fns/sorf_transform/tests.rs @@ -0,0 +1,495 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Unit tests for the [`SorfTransform`] scalar function. + +#![allow(clippy::cast_possible_truncation)] + +use std::sync::Arc; + +use vortex_array::ArrayPlugin; +use vortex_array::ArrayRef; +use vortex_array::IntoArray; +use vortex_array::VortexSessionExecute; +use vortex_array::arrays::ExtensionArray; +use vortex_array::arrays::FixedSizeListArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::dict::DictArray; +use vortex_array::arrays::extension::ExtensionArrayExt; +use vortex_array::arrays::fixed_size_list::FixedSizeListArrayExt; +use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin; +use vortex_array::dtype::DType; +use vortex_array::dtype::Nullability; +use vortex_array::dtype::PType; +use vortex_array::dtype::extension::ExtDType; +use vortex_array::extension::EmptyMetadata; +use vortex_array::validity::Validity; +use vortex_buffer::Buffer; +use vortex_buffer::BufferMut; +use vortex_error::VortexExpect; +use vortex_error::VortexResult; + +use super::SorfOptions; +use super::SorfTransform; +use super::rotation::SorfMatrix; +use crate::encodings::turboquant::centroids::compute_centroid_boundaries; +use crate::encodings::turboquant::centroids::compute_or_get_centroids; +use crate::encodings::turboquant::centroids::find_nearest_centroid; +use crate::tests::SESSION; +use crate::types::vector::Vector; + +/// Build a unit-normalized input vector array and forward-transform + quantize it, returning +/// `(input_f32, Vector(FSL(Dict(codes, centroids))), padded_dim)`. +/// +/// This mimics what the TurboQuant compression pipeline does, but directly, so we can test +/// `SorfTransform` in isolation. +fn forward_rotate_and_quantize( + dim: usize, + num_rows: usize, + seed: u64, + num_rounds: usize, + bit_width: u8, +) -> VortexResult<(Vec, ExtensionArray, usize)> { + // Build simple unit-normalized input vectors. + let mut input_f32 = vec![0.0f32; num_rows * dim]; + for row in 0..num_rows { + let mut norm_sq = 0.0f32; + for i in 0..dim { + let val = ((row * dim + i) as f32 + 1.0) * 0.01; + input_f32[row * dim + i] = val; + norm_sq += val * val; + } + let norm = norm_sq.sqrt(); + for i in 0..dim { + input_f32[row * dim + i] /= norm; + } + } + + let rotation = SorfMatrix::try_new(seed, dim, num_rounds)?; + let padded_dim = rotation.padded_dim(); + let centroids = compute_or_get_centroids(padded_dim as u32, bit_width)?; + let boundaries = compute_centroid_boundaries(¢roids); + + let mut all_indices = BufferMut::::with_capacity(num_rows * padded_dim); + let mut padded = vec![0.0f32; padded_dim]; + let mut rotated = vec![0.0f32; padded_dim]; + + for row in 0..num_rows { + padded[..dim].copy_from_slice(&input_f32[row * dim..(row + 1) * dim]); + padded[dim..].fill(0.0); + rotation.rotate(&padded, &mut rotated); + for j in 0..padded_dim { + all_indices.push(find_nearest_centroid(rotated[j], &boundaries)); + } + } + + let codes = PrimitiveArray::new::(all_indices.freeze(), Validity::NonNullable); + let mut centroids_buf = BufferMut::::with_capacity(centroids.len()); + centroids_buf.extend_from_slice(¢roids); + let centroids_arr = PrimitiveArray::new::(centroids_buf.freeze(), Validity::NonNullable); + let dict = DictArray::try_new(codes.into_array(), centroids_arr.into_array())?; + let fsl = FixedSizeListArray::try_new( + dict.into_array(), + padded_dim as u32, + Validity::NonNullable, + num_rows, + )?; + let padded_vector = wrap_as_vector(fsl, Validity::NonNullable)?; + + Ok((input_f32, padded_vector, padded_dim)) +} + +/// Wrap an FSL in a Vector extension, optionally re-tagging its validity. This is used by tests +/// that need to adjust top-level nullability of a padded vector child. +fn wrap_as_vector(fsl: FixedSizeListArray, validity: Validity) -> VortexResult { + let list_size = fsl.list_size(); + let num_rows = fsl.len(); + let elements = fsl.elements().clone(); + let fsl = FixedSizeListArray::try_new(elements, list_size, validity, num_rows)?; + let ext_dtype = ExtDType::::try_new(EmptyMetadata, fsl.dtype().clone())?.erased(); + Ok(ExtensionArray::new(ext_dtype, fsl.into_array())) +} + +/// Helper to build `SorfOptions` with common defaults. +fn default_options(dim: u32, seed: u64) -> SorfOptions { + SorfOptions { + seed, + num_rounds: 3, + dimensions: dim, + element_ptype: PType::F32, + } +} + +/// Execute a `SorfTransform` array and return the decoded flat f32 elements. +fn execute_sorf( + options: &SorfOptions, + child: ExtensionArray, + num_rows: usize, +) -> VortexResult> { + let sorf = SorfTransform::try_new_array(options, child.into_array(), num_rows)?; + let mut ctx = SESSION.create_execution_ctx(); + let result: ExtensionArray = sorf.into_array().execute(&mut ctx)?; + let result_fsl: FixedSizeListArray = result.storage_array().clone().execute(&mut ctx)?; + let result_prim: PrimitiveArray = result_fsl.elements().clone().execute(&mut ctx)?; + Ok(result_prim.as_slice::().to_vec()) +} + +/// Build an empty `Vector` extension array wrapping an empty FSL. +fn empty_padded_vector(padded_dim: u32, validity: Validity) -> VortexResult { + let elements = PrimitiveArray::empty::(Nullability::NonNullable); + let fsl = FixedSizeListArray::try_new(elements.into_array(), padded_dim, validity, 0)?; + let ext_dtype = ExtDType::::try_new(EmptyMetadata, fsl.dtype().clone())?.erased(); + Ok(ExtensionArray::new(ext_dtype, fsl.into_array())) +} + +#[test] +fn roundtrip_recovery() -> VortexResult<()> { + let dim = 128; + let num_rows = 10; + let seed = 42u64; + let (input_f32, padded_vector, _) = forward_rotate_and_quantize(dim, num_rows, seed, 3, 8)?; + let options = default_options(dim as u32, seed); + let result = execute_sorf(&options, padded_vector, num_rows)?; + + assert_eq!(result.len(), num_rows * dim); + + // At 8-bit quantization, the reconstruction should be very close to the input. + for row in 0..num_rows { + let orig = &input_f32[row * dim..(row + 1) * dim]; + let recon = &result[row * dim..(row + 1) * dim]; + let err_sq: f32 = orig + .iter() + .zip(recon) + .map(|(&a, &b)| (a - b) * (a - b)) + .sum(); + let norm_sq: f32 = orig.iter().map(|&v| v * v).sum(); + assert!( + err_sq / norm_sq < 1e-3, + "row {row} MSE too high: {:.6}", + err_sq / norm_sq + ); + } + Ok(()) +} + +#[test] +fn empty_array_non_nullable() -> VortexResult<()> { + let dim = 128u32; + let padded_dim = dim.next_power_of_two(); + let options = default_options(dim, 42); + + // Build an empty Vector child. + let child = empty_padded_vector(padded_dim, Validity::NonNullable)?; + + let sorf = SorfTransform::try_new_array(&options, child.into_array(), 0)?; + let mut ctx = SESSION.create_execution_ctx(); + let result: ExtensionArray = sorf.into_array().execute(&mut ctx)?; + + assert_eq!(result.len(), 0); + + // Output should be non-nullable. + let result_fsl: FixedSizeListArray = result.storage_array().clone().execute(&mut ctx)?; + assert!(!result_fsl.dtype().is_nullable()); + + Ok(()) +} + +#[test] +fn empty_array_nullable() -> VortexResult<()> { + let dim = 128u32; + let padded_dim = dim.next_power_of_two(); + let options = default_options(dim, 42); + + // Build an empty but nullable Vector child. + let child = empty_padded_vector(padded_dim, Validity::from(Nullability::Nullable))?; + + let sorf = SorfTransform::try_new_array(&options, child.into_array(), 0)?; + let mut ctx = SESSION.create_execution_ctx(); + let result: ExtensionArray = sorf.into_array().execute(&mut ctx)?; + + assert_eq!(result.len(), 0); + + // Output should be nullable (matching the child). + let result_fsl: FixedSizeListArray = result.storage_array().clone().execute(&mut ctx)?; + assert!(result_fsl.dtype().is_nullable()); + + Ok(()) +} + +#[test] +fn nullable_validity_propagation() -> VortexResult<()> { + let dim = 128; + let num_rows = 4; + let seed = 42u64; + let (_, non_nullable_vector, padded_dim) = + forward_rotate_and_quantize(dim, num_rows, seed, 3, 8)?; + + // Re-wrap the underlying FSL with a validity mask: rows 0 and 2 are valid, rows 1 and 3 + // are null. + let validity = Validity::from_iter([true, false, true, false]); + let fsl_non_nullable: FixedSizeListArray = non_nullable_vector + .storage_array() + .clone() + .execute(&mut SESSION.create_execution_ctx())?; + let fsl_nullable = FixedSizeListArray::try_new( + fsl_non_nullable.elements().clone(), + padded_dim as u32, + validity.clone(), + num_rows, + )?; + let nullable_vector = wrap_as_vector(fsl_nullable, validity.clone())?; + + let options = default_options(dim as u32, seed); + let sorf = SorfTransform::try_new_array(&options, nullable_vector.into_array(), num_rows)?; + let mut ctx = SESSION.create_execution_ctx(); + let result: ExtensionArray = sorf.into_array().execute(&mut ctx)?; + let result_fsl: FixedSizeListArray = result.storage_array().clone().execute(&mut ctx)?; + + // The output FSL validity should match the input. + let output_validity = result_fsl.validity()?; + for row in 0..num_rows { + assert_eq!( + output_validity.is_valid(row)?, + validity.is_valid(row)?, + "validity mismatch at row {row}" + ); + } + + Ok(()) +} + +#[test] +fn dimension_truncation() -> VortexResult<()> { + // Use a non-power-of-2 dimension (padded 200 -> 256). + let dim = 200; + let num_rows = 3; + let seed = 42u64; + let (_, padded_vector, padded_dim) = forward_rotate_and_quantize(dim, num_rows, seed, 3, 8)?; + + assert_eq!(padded_dim, 256, "200 should pad to 256"); + + let options = default_options(dim as u32, seed); + let result = execute_sorf(&options, padded_vector, num_rows)?; + + // Output should have original dimension, not padded. + assert_eq!(result.len(), num_rows * dim); + + Ok(()) +} + +#[test] +fn return_dtype_is_vector_extension() -> VortexResult<()> { + let dim = 128u32; + let padded_dim = dim.next_power_of_two(); + let options = default_options(dim, 42); + + // Input must be a Vector extension dtype. + let child_elem_dtype = DType::Primitive(PType::F32, Nullability::NonNullable); + let child_storage_dtype = DType::FixedSizeList( + Arc::new(child_elem_dtype), + padded_dim, + Nullability::NonNullable, + ); + let child_ext_dtype = ExtDType::::try_new(EmptyMetadata, child_storage_dtype)?.erased(); + let child_dtype = DType::Extension(child_ext_dtype); + + use vortex_array::scalar_fn::ScalarFnVTable; + let return_dtype = SorfTransform.return_dtype(&options, &[child_dtype])?; + + // Should be a Vector extension type. + let ext = return_dtype + .as_extension_opt() + .expect("return dtype should be an extension type"); + assert!( + ext.metadata_opt::() + .is_some() + ); + + // Inner FSL should have the original (unpadded) dimension. + let DType::FixedSizeList(_, inner_dim, _) = ext.storage_dtype() else { + panic!("expected storage dtype to be FSL"); + }; + assert_eq!(*inner_dim, dim); + + Ok(()) +} + +#[test] +fn rejects_zero_rounds_at_construction() { + let options = SorfOptions { + seed: 42, + num_rounds: 0, + dimensions: 128, + element_ptype: PType::F32, + }; + let elements = PrimitiveArray::from_iter([0.0f32; 128]).into_array(); + let child = FixedSizeListArray::try_new(elements, 128, Validity::NonNullable, 1) + .expect("test child should be valid"); + + let err = SorfTransform::try_new_array(&options, child.into_array(), 1) + .expect_err("zero rounds should be rejected at construction time"); + assert!(err.to_string().contains("num_rounds")); +} + +#[test] +fn rejects_non_float_output_ptype_at_construction() { + let options = SorfOptions { + seed: 42, + num_rounds: 3, + dimensions: 128, + element_ptype: PType::U8, + }; + let elements = PrimitiveArray::from_iter([0.0f32; 128]).into_array(); + let child = FixedSizeListArray::try_new(elements, 128, Validity::NonNullable, 1) + .expect("test child should be valid"); + + let err = SorfTransform::try_new_array(&options, child.into_array(), 1) + .expect_err("non-float output ptypes should be rejected at construction time"); + assert!(err.to_string().contains("element_ptype")); +} + +#[test] +fn rejects_non_vector_extension_child_at_construction() { + let options = default_options(128, 42); + // A bare FSL child (not wrapped in a Vector extension) should be rejected. + let elements = PrimitiveArray::from_iter([0.0f32; 128]).into_array(); + let child = FixedSizeListArray::try_new(elements, 128, Validity::NonNullable, 1) + .expect("test child should be valid"); + + let err = SorfTransform::try_new_array(&options, child.into_array(), 1) + .expect_err("non-Vector-extension children should be rejected at construction time"); + assert!(err.to_string().contains("Vector extension")); +} + +#[test] +fn rejects_wrong_padded_dimension_at_construction() { + // Options say dimension=128 so padded_dim should be 128. Pass a Vector<256> instead. + let options = default_options(128, 42); + let elements = PrimitiveArray::from_iter([0.0f32; 256]).into_array(); + let fsl = FixedSizeListArray::try_new(elements, 256, Validity::NonNullable, 1) + .expect("test child should be valid"); + let child = wrap_as_vector(fsl, Validity::NonNullable).expect("wrap should succeed"); + + let err = SorfTransform::try_new_array(&options, child.into_array(), 1) + .expect_err("mismatched padded dimension should be rejected at construction time"); + assert!(err.to_string().contains("dimension")); +} + +#[test] +fn rejects_non_f32_child_storage_at_construction() { + // Options are valid and target f32 output. Pass a Vector<128> whose storage is f16 instead + // of f32 -- SorfTransform's f32-only input constraint should reject this. + let options = default_options(128, 42); + let elements = PrimitiveArray::from_iter([half::f16::from_f32(0.0); 128]).into_array(); + let fsl = FixedSizeListArray::try_new(elements, 128, Validity::NonNullable, 1) + .expect("test child should be valid"); + let child = wrap_as_vector(fsl, Validity::NonNullable).expect("wrap should succeed"); + + let err = SorfTransform::try_new_array(&options, child.into_array(), 1) + .expect_err("non-f32 Vector storage should be rejected at construction time"); + assert!(err.to_string().contains("f32")); +} + +#[test] +fn f16_output_type() -> VortexResult<()> { + let dim = 128; + let num_rows = 3; + let seed = 42u64; + let (_, padded_vector, _) = forward_rotate_and_quantize(dim, num_rows, seed, 3, 8)?; + + let options = SorfOptions { + seed, + num_rounds: 3, + dimensions: dim as u32, + element_ptype: PType::F16, + }; + let sorf = SorfTransform::try_new_array(&options, padded_vector.into_array(), num_rows)?; + let mut ctx = SESSION.create_execution_ctx(); + let result: ExtensionArray = sorf.into_array().execute(&mut ctx)?; + let result_fsl: FixedSizeListArray = result.storage_array().clone().execute(&mut ctx)?; + let result_prim: PrimitiveArray = result_fsl.elements().clone().execute(&mut ctx)?; + + assert_eq!(result_prim.ptype(), PType::F16); + assert_eq!(result_prim.as_slice::().len(), num_rows * dim); + + Ok(()) +} + +#[test] +fn f64_output_type() -> VortexResult<()> { + let dim = 128; + let num_rows = 3; + let seed = 42u64; + let (_, padded_vector, _) = forward_rotate_and_quantize(dim, num_rows, seed, 3, 8)?; + + let options = SorfOptions { + seed, + num_rounds: 3, + dimensions: dim as u32, + element_ptype: PType::F64, + }; + let sorf = SorfTransform::try_new_array(&options, padded_vector.into_array(), num_rows)?; + let mut ctx = SESSION.create_execution_ctx(); + let result: ExtensionArray = sorf.into_array().execute(&mut ctx)?; + let result_fsl: FixedSizeListArray = result.storage_array().clone().execute(&mut ctx)?; + let result_prim: PrimitiveArray = result_fsl.elements().clone().execute(&mut ctx)?; + + assert_eq!(result_prim.ptype(), PType::F64); + assert_eq!(result_prim.as_slice::().len(), num_rows * dim); + + Ok(()) +} + +/// Build a trivial `Vector>` child populated with zeroes. The values +/// are irrelevant for the serde round-trip test; only the dtype shape matters. +fn trivial_padded_vector(padded_dim: u32, num_rows: usize, validity: Validity) -> ArrayRef { + let elements = PrimitiveArray::new( + Buffer::::zeroed(num_rows * padded_dim as usize), + Validity::NonNullable, + ); + let fsl = FixedSizeListArray::try_new(elements.into_array(), padded_dim, validity, num_rows) + .vortex_expect("fsl must build"); + let ext_dtype = ExtDType::::try_new(EmptyMetadata, fsl.dtype().clone()) + .vortex_expect("ext dtype must build") + .erased(); + ExtensionArray::new(ext_dtype, fsl.into_array()).into_array() +} + +#[rstest::rstest] +// Non-power-of-two dimension to exercise `padded_dim = dim.next_power_of_two()`. +#[case::power_of_two_dim(128, Validity::NonNullable)] +#[case::non_power_of_two_dim(100, Validity::NonNullable)] +// Nullable top-level Vector to verify child nullability is reconstructed from the parent output. +#[case::nullable_child(100, Validity::AllValid)] +fn serde_round_trip(#[case] dimensions: u32, #[case] validity: Validity) -> VortexResult<()> { + let padded_dim = dimensions.next_power_of_two(); + let num_rows = 4; + let options = SorfOptions { + seed: 42, + num_rounds: 3, + dimensions, + element_ptype: PType::F32, + }; + let child = trivial_padded_vector(padded_dim, num_rows, validity); + let original = SorfTransform::try_new_array(&options, child.clone(), num_rows)?.into_array(); + + let plugin = ScalarFnArrayPlugin::new(SorfTransform); + let metadata = plugin + .serialize(&original, &SESSION)? + .expect("SorfTransform serialize must produce metadata"); + + let children = vec![child]; + let recovered = plugin.deserialize( + original.dtype(), + original.len(), + &metadata, + &[], + &children, + &SESSION, + )?; + + assert_eq!(recovered.dtype(), original.dtype()); + assert_eq!(recovered.len(), original.len()); + assert_eq!(recovered.encoding_id(), original.encoding_id()); + Ok(()) +} diff --git a/vortex-tensor/src/scalar_fns/sorf_transform/vtable.rs b/vortex-tensor/src/scalar_fns/sorf_transform/vtable.rs new file mode 100644 index 00000000000..827f8e6a796 --- /dev/null +++ b/vortex-tensor/src/scalar_fns/sorf_transform/vtable.rs @@ -0,0 +1,346 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! [`ScalarFnVTable`] implementation for [`SorfTransform`]. + +use std::fmt; +use std::fmt::Formatter; +use std::sync::Arc; + +use num_traits::Float; +use num_traits::FromPrimitive; +use prost::Message; +use vortex_array::ArrayRef; +use vortex_array::ExecutionCtx; +use vortex_array::IntoArray; +use vortex_array::arrays::ExtensionArray; +use vortex_array::arrays::FixedSizeListArray; +use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::ScalarFn as ScalarFnArrayEncoding; +use vortex_array::arrays::extension::ExtensionArrayExt; +use vortex_array::arrays::fixed_size_list::FixedSizeListArrayExt; +use vortex_array::arrays::scalar_fn::ScalarFnArrayExt; +use vortex_array::arrays::scalar_fn::ScalarFnArrayView; +use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayParts; +use vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayVTable; +use vortex_array::dtype::DType; +use vortex_array::dtype::NativePType; +use vortex_array::dtype::Nullability; +use vortex_array::dtype::PType; +use vortex_array::dtype::extension::ExtDType; +use vortex_array::dtype::proto::dtype as pb; +use vortex_array::expr::Expression; +use vortex_array::extension::EmptyMetadata; +use vortex_array::match_each_float_ptype; +use vortex_array::scalar_fn::Arity; +use vortex_array::scalar_fn::ChildName; +use vortex_array::scalar_fn::ExecutionArgs; +use vortex_array::scalar_fn::ScalarFnId; +use vortex_array::scalar_fn::ScalarFnVTable; +use vortex_array::serde::ArrayChildren; +use vortex_array::validity::Validity; +use vortex_buffer::BufferMut; +use vortex_error::VortexExpect; +use vortex_error::VortexResult; +use vortex_error::vortex_ensure_eq; +use vortex_error::vortex_err; +use vortex_session::VortexSession; + +use super::SorfOptions; +use super::SorfTransform; +use super::rotation::SorfMatrix; +use super::validate_sorf_options; +use crate::types::vector::AnyVector; +use crate::types::vector::Vector; + +impl ScalarFnVTable for SorfTransform { + type Options = SorfOptions; + + fn id(&self) -> ScalarFnId { + ScalarFnId::new("vortex.tensor.sorf_transform") + } + + fn arity(&self, _options: &Self::Options) -> Arity { + Arity::Exact(1) + } + + fn child_name(&self, _options: &Self::Options, child_idx: usize) -> ChildName { + match child_idx { + 0 => ChildName::from("rotated"), + _ => unreachable!("SorfTransform must have exactly one child"), + } + } + + fn fmt_sql( + &self, + options: &Self::Options, + expr: &Expression, + f: &mut Formatter<'_>, + ) -> fmt::Result { + write!(f, "sorf_transform(")?; + expr.child(0).fmt_sql(f)?; + write!(f, ", {options})") + } + + fn return_dtype(&self, options: &Self::Options, arg_dtypes: &[DType]) -> VortexResult { + validate_sorf_options(options)?; + + let child_dtype = &arg_dtypes[0]; + let vector_metadata = child_dtype + .as_extension_opt() + .and_then(|ext| ext.metadata_opt::()) + .ok_or_else(|| { + vortex_err!("SorfTransform child must be a Vector extension, got {child_dtype}") + })?; + + let expected_padded = options.dimensions.next_power_of_two(); + vortex_ensure_eq!( + vector_metadata.dimensions(), + expected_padded, + "SorfTransform child Vector must have dimension {expected_padded} (next power of two \ + for dimension {})", + options.dimensions, + ); + + // For now, the child Vector storage must be f32. TurboQuant stores its centroids as f32, + // and the SORF transform itself operates in f32, so any other input type would require an + // implicit cast that we do not yet support. The output element type is independently + // specified via `options.element_ptype` and is built below. + vortex_ensure_eq!( + vector_metadata.element_ptype(), + PType::F32, + "SorfTransform child Vector storage must be f32 (for now), got {}", + vector_metadata.element_ptype(), + ); + + let output_elem_dtype = DType::Primitive(options.element_ptype, Nullability::NonNullable); + let storage_dtype = DType::FixedSizeList( + Arc::new(output_elem_dtype), + options.dimensions, + child_dtype.nullability(), + ); + + let _ = vector_metadata; + let ext_dtype = ExtDType::::try_new(EmptyMetadata, storage_dtype)?.erased(); + + Ok(DType::Extension(ext_dtype)) + } + + fn execute( + &self, + options: &Self::Options, + args: &dyn ExecutionArgs, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + let dim = options.dimensions as usize; + let num_rows = args.row_count(); + + if num_rows == 0 { + let child_dtype = args.get(0)?.dtype().clone(); + let validity = Validity::from(child_dtype.nullability()); + + return match_each_float_ptype!(options.element_ptype, |T| { + let elements = PrimitiveArray::empty::(Nullability::NonNullable); + let fsl = FixedSizeListArray::try_new( + elements.into_array(), + options.dimensions, + validity, + 0, + )?; + Vector::try_new_vector_array(fsl.into_array()) + }); + } + + // Execute the child to get the Vector extension wrapping an FSL of f32 coordinates. The + // `return_dtype` check guarantees the child is a `Vector`, so the + // materialized FSL elements are always f32. + let child_ext: ExtensionArray = args.get(0)?.execute(ctx)?; + let child_validity = child_ext.as_ref().validity()?; + let child_fsl: FixedSizeListArray = child_ext.storage_array().clone().execute(ctx)?; + let padded_dim = + usize::try_from(child_fsl.list_size()).vortex_expect("list_size fits usize"); + + let elements_prim: PrimitiveArray = child_fsl.elements().clone().execute(ctx)?; + let f32_elements = elements_prim.into_buffer::(); + + // Reconstruct the orthogonal transform matrix from the seed. + let rotation = SorfMatrix::try_new(options.seed, dim, options.num_rounds as usize)?; + + // Inverse transform each row, truncate to original dimension, cast to target type. + match_each_float_ptype!(options.element_ptype, |T| { + inverse_rotate_typed::( + &f32_elements, + &rotation, + dim, + padded_dim, + num_rows, + child_validity, + ) + }) + } + + fn validity( + &self, + _options: &Self::Options, + expression: &Expression, + ) -> VortexResult> { + Ok(Some(expression.child(0).validity()?)) + } + + fn is_null_sensitive(&self, _options: &Self::Options) -> bool { + false + } + + fn is_fallible(&self, _options: &Self::Options) -> bool { + false + } +} + +/// Metadata for a serialized [`SorfTransform`] array. +/// +/// Stores the full [`SorfOptions`] inline along with the child [`DType`]. Older metadata omitted +/// this field; deserialization derives the legacy plain-`Vector` child dtype from the parent dtype +/// in that case. +#[derive(Clone, prost::Message)] +pub(super) struct SorfTransformMetadata { + #[prost(uint64, tag = "1")] + seed: u64, + /// Rust `u8` widened to `u32` for protobuf (no `u8` on the wire). + #[prost(uint32, tag = "2")] + num_rounds: u32, + #[prost(uint32, tag = "3")] + dimension: u32, + #[prost(enumeration = "PType", tag = "4")] + element_ptype: i32, + #[prost(message, optional, tag = "5")] + child_dtype: Option, +} + +impl ScalarFnArrayVTable for SorfTransform { + fn serialize( + &self, + view: &ScalarFnArrayView, + _session: &VortexSession, + ) -> VortexResult>> { + let scalar_fn_array = view.as_::(); + let child_dtype = Some(scalar_fn_array.child_at(0).dtype().try_into()?); + let metadata = SorfTransformMetadata { + child_dtype, + ..SorfTransformMetadata::from(view.options) + }; + Ok(Some(metadata.encode_to_vec())) + } + + fn deserialize( + &self, + dtype: &DType, + len: usize, + metadata: &[u8], + children: &dyn ArrayChildren, + session: &VortexSession, + ) -> VortexResult> { + let metadata = SorfTransformMetadata::decode(metadata) + .map_err(|e| vortex_err!("Failed to decode SorfTransformMetadata: {e}"))?; + let options = metadata.to_options()?; + + // `return_dtype` sets the output FSL's nullability to the child's nullability (see + // `return_dtype` above), so we read the child nullability back from the parent dtype. + let child_nullability = dtype + .as_extension_opt() + .ok_or_else(|| { + vortex_err!("SorfTransform parent dtype must be a Vector extension, got {dtype}") + })? + .storage_dtype() + .nullability(); + let padded_dim = options.dimensions.next_power_of_two(); + let child_storage = DType::FixedSizeList( + Arc::new(DType::Primitive(PType::F32, Nullability::NonNullable)), + padded_dim, + child_nullability, + ); + let child_dtype = match metadata.child_dtype.as_ref() { + Some(dtype) => DType::from_proto(dtype, session)?, + None => { + let child_ext = ExtDType::::try_new(EmptyMetadata, child_storage)?.erased(); + DType::Extension(child_ext) + } + }; + let child = children.get(0, &child_dtype, len)?; + + Ok(ScalarFnArrayParts { + options, + children: vec![child], + }) + } +} + +/// Convert an f32 value to a float type `T`. +/// +/// `FromPrimitive::from_f32` is infallible for all Vortex float types: f16 saturates via the +/// inherent `f16::from_f32()`, f32 is identity, f64 is lossless widening. +fn float_from_f32(v: f32) -> T { + FromPrimitive::from_f32(v).vortex_expect("f32-to-float conversion is infallible") +} + +/// Apply the inverse SORF transform on f32 data, truncate to the original dimension, cast each +/// element to `T`, and build a plain [`Vector`](crate::vector::Vector) extension array. +fn inverse_rotate_typed( + f32_elements: &[f32], + rotation: &SorfMatrix, + dim: usize, + padded_dim: usize, + num_rows: usize, + validity: Validity, +) -> VortexResult { + let dim_u32 = u32::try_from(dim).vortex_expect("dimension fits u32"); + let mut output = BufferMut::::with_capacity(num_rows * dim); + let mut unrotated = vec![0.0f32; padded_dim]; + + for row in 0..num_rows { + let row_data = &f32_elements[row * padded_dim..(row + 1) * padded_dim]; + + rotation.inverse_rotate(row_data, &mut unrotated); + + for idx in 0..dim { + // SAFETY: We allocated enough memory above. + unsafe { output.push_unchecked(float_from_f32::(unrotated[idx])) }; + } + } + + let elements = PrimitiveArray::new::(output.freeze(), Validity::NonNullable); + let fsl = FixedSizeListArray::try_new(elements.into_array(), dim_u32, validity, num_rows)?; + Vector::try_new_vector_array(fsl.into_array()) +} + +impl From<&SorfOptions> for SorfTransformMetadata { + fn from(options: &SorfOptions) -> Self { + Self { + seed: options.seed, + num_rounds: u32::from(options.num_rounds), + dimension: options.dimensions, + element_ptype: options.element_ptype as i32, + child_dtype: None, + } + } +} + +impl SorfTransformMetadata { + /// Rebuild the [`SorfOptions`] this metadata was serialized from, validating that the wire + /// values are in range. + fn to_options(&self) -> VortexResult { + let num_rounds = u8::try_from(self.num_rounds).map_err(|_| { + vortex_err!( + "SorfTransform num_rounds {} does not fit in u8", + self.num_rounds + ) + })?; + let options = SorfOptions { + seed: self.seed, + num_rounds, + dimensions: self.dimension, + element_ptype: self.element_ptype(), + }; + validate_sorf_options(&options)?; + Ok(options) + } +} diff --git a/vortex-tensor/src/fixed_shape/matcher.rs b/vortex-tensor/src/types/fixed_shape_tensor/matcher.rs similarity index 92% rename from vortex-tensor/src/fixed_shape/matcher.rs rename to vortex-tensor/src/types/fixed_shape_tensor/matcher.rs index 5248ca6514b..2cd7d7c8e4c 100644 --- a/vortex-tensor/src/fixed_shape/matcher.rs +++ b/vortex-tensor/src/types/fixed_shape_tensor/matcher.rs @@ -8,8 +8,8 @@ use vortex_array::dtype::extension::Matcher; use vortex_error::VortexExpect; use vortex_error::vortex_panic; -use crate::fixed_shape::FixedShapeTensor; -use crate::fixed_shape::FixedShapeTensorMetadata; +use crate::types::fixed_shape_tensor::FixedShapeTensor; +use crate::types::fixed_shape_tensor::FixedShapeTensorMetadata; pub struct AnyFixedShapeTensor; @@ -31,7 +31,7 @@ pub struct FixedShapeTensorMatcherMetadata<'a> { /// /// This matches the `FixedSizeList` list size in the storage dtype, which is the product of /// the logical shape dimensions. - flat_list_size: usize, + flat_list_size: u32, } impl Matcher for AnyFixedShapeTensor { @@ -64,7 +64,7 @@ impl Matcher for AnyFixedShapeTensor { Some(FixedShapeTensorMatcherMetadata { metadata, element_ptype: element_dtype.as_ptype(), - flat_list_size: *list_size as usize, + flat_list_size: *list_size, }) } } @@ -81,7 +81,7 @@ impl FixedShapeTensorMatcherMetadata<'_> { } /// Returns the flattened element count for each tensor row. - pub fn list_size(&self) -> usize { + pub fn flat_list_size(&self) -> u32 { self.flat_list_size } } @@ -98,7 +98,7 @@ mod tests { use vortex_error::VortexResult; use super::*; - use crate::vector::Vector; + use crate::types::vector::Vector; fn tensor_storage_dtype(element_ptype: PType, list_size: u32) -> DType { DType::FixedSizeList( @@ -118,7 +118,7 @@ mod tests { let metadata = ext_dtype.metadata::(); assert_eq!(metadata.element_ptype(), PType::F32); - assert_eq!(metadata.list_size(), 24); + assert_eq!(metadata.flat_list_size(), 24); assert_eq!(metadata.metadata().logical_shape(), &[2, 3, 4]); Ok(()) } diff --git a/vortex-tensor/src/fixed_shape/metadata.rs b/vortex-tensor/src/types/fixed_shape_tensor/metadata.rs similarity index 92% rename from vortex-tensor/src/fixed_shape/metadata.rs rename to vortex-tensor/src/types/fixed_shape_tensor/metadata.rs index 264d18453c4..757138d3e50 100644 --- a/vortex-tensor/src/fixed_shape/metadata.rs +++ b/vortex-tensor/src/types/fixed_shape_tensor/metadata.rs @@ -215,6 +215,7 @@ impl fmt::Display for FixedShapeTensorMetadata { } if let Some(perm) = &self.permutation { + write!(f, ", [")?; for (i, p) in perm.iter().enumerate() { if i > 0 { write!(f, ", ")?; @@ -353,6 +354,44 @@ mod tests { Ok(()) } + // -- Display -- + + #[test] + fn display_shape_only() { + let m = FixedShapeTensorMetadata::new(vec![2, 3, 4]); + assert_eq!(m.to_string(), "Tensor(2, 3, 4)"); + } + + #[test] + fn display_scalar_0d() { + let m = FixedShapeTensorMetadata::new(vec![]); + assert_eq!(m.to_string(), "Tensor()"); + } + + #[test] + fn display_with_dim_names() -> VortexResult<()> { + let m = FixedShapeTensorMetadata::new(vec![3, 4]) + .with_dim_names(vec!["rows".into(), "cols".into()])?; + assert_eq!(m.to_string(), "Tensor(rows: 3, cols: 4)"); + Ok(()) + } + + #[test] + fn display_with_permutation() -> VortexResult<()> { + let m = FixedShapeTensorMetadata::new(vec![2, 3, 4]).with_permutation(vec![1, 0, 2])?; + assert_eq!(m.to_string(), "Tensor(2, 3, 4, [1, 0, 2])"); + Ok(()) + } + + #[test] + fn display_with_dim_names_and_permutation() -> VortexResult<()> { + let m = FixedShapeTensorMetadata::new(vec![2, 3, 4]) + .with_dim_names(vec!["x".into(), "y".into(), "z".into()])? + .with_permutation(vec![1, 2, 0])?; + assert_eq!(m.to_string(), "Tensor(x: 2, y: 3, z: 4, [1, 2, 0])"); + Ok(()) + } + #[test] fn dim_names_wrong_length() { let result = FixedShapeTensorMetadata::new(vec![2, 3]).with_dim_names(vec!["x".into()]); diff --git a/vortex-tensor/src/fixed_shape/mod.rs b/vortex-tensor/src/types/fixed_shape_tensor/mod.rs similarity index 100% rename from vortex-tensor/src/fixed_shape/mod.rs rename to vortex-tensor/src/types/fixed_shape_tensor/mod.rs diff --git a/vortex-tensor/src/fixed_shape/proto.rs b/vortex-tensor/src/types/fixed_shape_tensor/proto.rs similarity index 98% rename from vortex-tensor/src/fixed_shape/proto.rs rename to vortex-tensor/src/types/fixed_shape_tensor/proto.rs index 9d89c56bcec..2bd608d717d 100644 --- a/vortex-tensor/src/fixed_shape/proto.rs +++ b/vortex-tensor/src/types/fixed_shape_tensor/proto.rs @@ -8,7 +8,7 @@ use vortex_error::VortexExpect; use vortex_error::VortexResult; use vortex_error::vortex_err; -use crate::fixed_shape::FixedShapeTensorMetadata; +use crate::types::fixed_shape_tensor::FixedShapeTensorMetadata; /// Protobuf representation of [`FixedShapeTensorMetadata`]. /// diff --git a/vortex-tensor/src/fixed_shape/vtable.rs b/vortex-tensor/src/types/fixed_shape_tensor/vtable.rs similarity index 50% rename from vortex-tensor/src/fixed_shape/vtable.rs rename to vortex-tensor/src/types/fixed_shape_tensor/vtable.rs index 21ab1ef5336..741d6b71801 100644 --- a/vortex-tensor/src/fixed_shape/vtable.rs +++ b/vortex-tensor/src/types/fixed_shape_tensor/vtable.rs @@ -11,9 +11,9 @@ use vortex_error::vortex_bail; use vortex_error::vortex_ensure; use vortex_error::vortex_ensure_eq; -use crate::fixed_shape::FixedShapeTensor; -use crate::fixed_shape::FixedShapeTensorMetadata; -use crate::fixed_shape::proto; +use crate::types::fixed_shape_tensor::FixedShapeTensor; +use crate::types::fixed_shape_tensor::FixedShapeTensorMetadata; +use crate::types::fixed_shape_tensor::proto; impl ExtVTable for FixedShapeTensor { type Metadata = FixedShapeTensorMetadata; @@ -22,7 +22,7 @@ impl ExtVTable for FixedShapeTensor { type NativeValue<'a> = &'a ScalarValue; fn id(&self) -> ExtId { - ExtId::new_ref("vortex.tensor.fixed_shape_tensor") + ExtId::new("vortex.tensor.fixed_shape_tensor") } fn serialize_metadata(&self, metadata: &Self::Metadata) -> VortexResult> { @@ -33,6 +33,22 @@ impl ExtVTable for FixedShapeTensor { proto::deserialize(metadata) } + fn least_supertype(ext_dtype: &ExtDType, other: &DType) -> Option { + let DType::Extension(other_ext) = other else { + return None; + }; + // Only element dtype may widen — shape, dim_names, and permutation must match exactly. + let other_metadata = other_ext.metadata_opt::()?; + if ext_dtype.metadata() != other_metadata { + return None; + } + let widened = ext_dtype + .storage_dtype() + .least_supertype(other_ext.storage_dtype())?; + let ext = ExtDType::::try_new(ext_dtype.metadata().clone(), widened).ok()?; + Some(DType::Extension(ext.erased())) + } + fn validate_dtype(ext_dtype: &ExtDType) -> VortexResult<()> { let storage_dtype = ext_dtype.storage_dtype(); let DType::FixedSizeList(element_dtype, list_size, _nullability) = storage_dtype else { @@ -76,12 +92,18 @@ impl ExtVTable for FixedShapeTensor { #[cfg(test)] mod tests { + use std::sync::Arc; + use rstest::rstest; + use vortex_array::dtype::DType; + use vortex_array::dtype::Nullability; + use vortex_array::dtype::PType; + use vortex_array::dtype::extension::ExtDType; use vortex_array::dtype::extension::ExtVTable; use vortex_error::VortexResult; - use crate::fixed_shape::FixedShapeTensor; - use crate::fixed_shape::FixedShapeTensorMetadata; + use crate::types::fixed_shape_tensor::FixedShapeTensor; + use crate::types::fixed_shape_tensor::FixedShapeTensorMetadata; /// Serializes and deserializes the given metadata through protobuf, asserting equality. fn assert_roundtrip(metadata: &FixedShapeTensorMetadata) -> VortexResult<()> { @@ -118,4 +140,70 @@ mod tests { ) -> VortexResult<()> { assert_roundtrip(&metadata?) } + + /// Constructs a `FixedShapeTensor` ext dtype wrapped in `DType::Extension`. + fn tensor_dtype( + metadata: FixedShapeTensorMetadata, + element: PType, + list_size: u32, + ) -> VortexResult { + let storage = DType::FixedSizeList( + Arc::new(DType::Primitive(element, Nullability::NonNullable)), + list_size, + Nullability::NonNullable, + ); + Ok(DType::Extension( + ExtDType::::try_new(metadata, storage)?.erased(), + )) + } + + #[test] + fn tensor_widens_element_when_metadata_matches() -> VortexResult<()> { + let metadata = FixedShapeTensorMetadata::new(vec![2, 3]); + let lhs = tensor_dtype(metadata.clone(), PType::F32, 6)?; + let rhs = tensor_dtype(metadata.clone(), PType::F64, 6)?; + let expected = tensor_dtype(metadata, PType::F64, 6)?; + assert_eq!(lhs.least_supertype(&rhs), Some(expected)); + Ok(()) + } + + #[test] + fn tensor_different_shape_returns_none() -> VortexResult<()> { + let lhs = tensor_dtype(FixedShapeTensorMetadata::new(vec![2, 3]), PType::F32, 6)?; + let rhs = tensor_dtype(FixedShapeTensorMetadata::new(vec![3, 2]), PType::F32, 6)?; + assert_eq!(lhs.least_supertype(&rhs), None); + Ok(()) + } + + #[test] + fn tensor_different_permutation_returns_none() -> VortexResult<()> { + let lhs_metadata = + FixedShapeTensorMetadata::new(vec![2, 3]).with_permutation(vec![0, 1])?; + let rhs_metadata = + FixedShapeTensorMetadata::new(vec![2, 3]).with_permutation(vec![1, 0])?; + let lhs = tensor_dtype(lhs_metadata, PType::F32, 6)?; + let rhs = tensor_dtype(rhs_metadata, PType::F32, 6)?; + assert_eq!(lhs.least_supertype(&rhs), None); + Ok(()) + } + + #[test] + fn tensor_different_dim_names_returns_none() -> VortexResult<()> { + let lhs_metadata = FixedShapeTensorMetadata::new(vec![2, 3]) + .with_dim_names(vec!["x".into(), "y".into()])?; + let rhs_metadata = FixedShapeTensorMetadata::new(vec![2, 3]) + .with_dim_names(vec!["rows".into(), "cols".into()])?; + let lhs = tensor_dtype(lhs_metadata, PType::F32, 6)?; + let rhs = tensor_dtype(rhs_metadata, PType::F32, 6)?; + assert_eq!(lhs.least_supertype(&rhs), None); + Ok(()) + } + + #[test] + fn tensor_vs_non_extension_returns_none() -> VortexResult<()> { + let lhs = tensor_dtype(FixedShapeTensorMetadata::new(vec![2, 3]), PType::F32, 6)?; + let rhs = DType::Primitive(PType::F32, Nullability::NonNullable); + assert_eq!(lhs.least_supertype(&rhs), None); + Ok(()) + } } diff --git a/vortex-tensor/src/types/mod.rs b/vortex-tensor/src/types/mod.rs new file mode 100644 index 00000000000..47dcabdb36d --- /dev/null +++ b/vortex-tensor/src/types/mod.rs @@ -0,0 +1,7 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Internal homes for tensor extension types. + +pub mod fixed_shape_tensor; +pub mod vector; diff --git a/vortex-tensor/src/vector/matcher.rs b/vortex-tensor/src/types/vector/matcher.rs similarity index 96% rename from vortex-tensor/src/vector/matcher.rs rename to vortex-tensor/src/types/vector/matcher.rs index 0da5384f303..b0e641ba45b 100644 --- a/vortex-tensor/src/vector/matcher.rs +++ b/vortex-tensor/src/types/vector/matcher.rs @@ -10,7 +10,7 @@ use vortex_error::VortexResult; use vortex_error::vortex_ensure; use vortex_error::vortex_panic; -use crate::vector::Vector; +use crate::types::vector::Vector; pub struct AnyVector; @@ -49,7 +49,7 @@ impl Matcher for AnyVector { let dimensions = *list_size; - assert!(element_dtype.is_float(), "element dtype must be primitive"); + assert!(element_dtype.is_float(), "element dtype must be float"); assert!( !element_dtype.is_nullable(), "element dtype must be non-nullable" @@ -101,8 +101,8 @@ mod tests { use vortex_error::VortexResult; use super::*; - use crate::fixed_shape::FixedShapeTensor; - use crate::fixed_shape::FixedShapeTensorMetadata; + use crate::types::fixed_shape_tensor::FixedShapeTensor; + use crate::types::fixed_shape_tensor::FixedShapeTensorMetadata; fn vector_storage_dtype(element_ptype: PType, dimensions: u32) -> DType { DType::FixedSizeList( diff --git a/vortex-tensor/src/types/vector/mod.rs b/vortex-tensor/src/types/vector/mod.rs new file mode 100644 index 00000000000..3763220fc82 --- /dev/null +++ b/vortex-tensor/src/types/vector/mod.rs @@ -0,0 +1,83 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Vector extension type for fixed-length float vectors (e.g., embeddings). + +use vortex_array::ArrayRef; +use vortex_array::IntoArray; +use vortex_array::arrays::ConstantArray; +use vortex_array::arrays::ExtensionArray; +use vortex_array::dtype::DType; +use vortex_array::dtype::NativePType; +use vortex_array::dtype::Nullability; +use vortex_array::extension::EmptyMetadata; +use vortex_array::scalar::PValue; +use vortex_array::scalar::Scalar; +use vortex_error::VortexResult; +use vortex_error::vortex_bail; +use vortex_error::vortex_ensure; + +/// Validates that `storage` is a valid storage dtype for a [`Vector`] or +/// [`NormalizedVector`](crate::normalized_vector::NormalizedVector) extension type. +/// +/// The storage must be a `FixedSizeList` with non-nullable float +/// elements. The outer nullability is not constrained. +pub(crate) fn validate_vector_storage_dtype(storage: &DType) -> VortexResult<()> { + let DType::FixedSizeList(element_dtype, _list_size, _nullability) = storage else { + vortex_bail!("Vector storage dtype must be a FixedSizeList, got {storage}"); + }; + + vortex_ensure!( + element_dtype.is_float(), + "Vector element dtype must be a float, got {element_dtype}" + ); + vortex_ensure!( + !element_dtype.is_nullable(), + "Vector element dtype must be non-nullable" + ); + + Ok(()) +} + +/// The Vector extension type. +#[derive(Clone, Debug, Default, PartialEq, Eq, Hash)] +pub struct Vector; + +impl Vector { + /// Helper function for creating a new [`Vector`] [`ExtensionArray`]. + /// + /// # Errors + /// + /// Returns an error if the [`Vector`] extension dtype rejects the storage array. + pub(crate) fn try_new_vector_array(storage: ArrayRef) -> VortexResult { + ExtensionArray::try_new_from_vtable(Vector, EmptyMetadata, storage) + .map(|ext| ext.into_array()) + } + + /// Helper function to build a [`Vector`] [`ExtensionArray`] whose storage is a + /// [`ConstantArray`], broadcasting a single vector `elements` across `len` rows. + /// + /// # Errors + /// + /// Returns an error if the [`Vector`] extension dtype rejects the constructed storage dtype. + pub(crate) fn constant_array>( + elements: &[T], + len: usize, + ) -> VortexResult { + let element_dtype = DType::Primitive(T::PTYPE, Nullability::NonNullable); + let children: Vec = elements + .iter() + .map(|&v| Scalar::primitive(v, Nullability::NonNullable)) + .collect(); + let storage_scalar = + Scalar::fixed_size_list(element_dtype, children, Nullability::NonNullable); + Self::try_new_vector_array(ConstantArray::new(storage_scalar, len).into_array()) + } +} + +mod matcher; + +pub use matcher::AnyVector; +pub use matcher::VectorMatcherMetadata; + +mod vtable; diff --git a/vortex-tensor/src/vector/vtable.rs b/vortex-tensor/src/types/vector/vtable.rs similarity index 55% rename from vortex-tensor/src/vector/vtable.rs rename to vortex-tensor/src/types/vector/vtable.rs index 2dda05b7363..c80f17665f2 100644 --- a/vortex-tensor/src/vector/vtable.rs +++ b/vortex-tensor/src/types/vector/vtable.rs @@ -8,10 +8,9 @@ use vortex_array::dtype::extension::ExtVTable; use vortex_array::extension::EmptyMetadata; use vortex_array::scalar::ScalarValue; use vortex_error::VortexResult; -use vortex_error::vortex_bail; -use vortex_error::vortex_ensure; -use crate::vector::Vector; +use crate::types::vector::Vector; +use crate::types::vector::validate_vector_storage_dtype; impl ExtVTable for Vector { type Metadata = EmptyMetadata; @@ -20,7 +19,7 @@ impl ExtVTable for Vector { type NativeValue<'a> = &'a ScalarValue; fn id(&self) -> ExtId { - ExtId::new_ref("vortex.tensor.vector") + ExtId::new("vortex.tensor.vector") } fn serialize_metadata(&self, _metadata: &Self::Metadata) -> VortexResult> { @@ -31,22 +30,22 @@ impl ExtVTable for Vector { Ok(EmptyMetadata) } - fn validate_dtype(ext_dtype: &ExtDType) -> VortexResult<()> { - let storage_dtype = ext_dtype.storage_dtype(); - let DType::FixedSizeList(element_dtype, _list_size, _nullability) = storage_dtype else { - vortex_bail!("Vector storage dtype must be a FixedSizeList, got {storage_dtype}"); + fn least_supertype(ext_dtype: &ExtDType, other: &DType) -> Option { + let DType::Extension(other_ext) = other else { + return None; }; + if !other_ext.is::() { + return None; + } + let widened = ext_dtype + .storage_dtype() + .least_supertype(other_ext.storage_dtype())?; + let ext = ExtDType::::try_new(EmptyMetadata, widened).ok()?; + Some(DType::Extension(ext.erased())) + } - vortex_ensure!( - element_dtype.is_float(), - "Vector element dtype must be a float, got {element_dtype}" - ); - vortex_ensure!( - !element_dtype.is_nullable(), - "Vector element dtype must be non-nullable" - ); - - Ok(()) + fn validate_dtype(ext_dtype: &ExtDType) -> VortexResult<()> { + validate_vector_storage_dtype(ext_dtype.storage_dtype()) } fn unpack_native<'a>( @@ -70,7 +69,7 @@ mod tests { use vortex_array::extension::EmptyMetadata; use vortex_error::VortexResult; - use crate::vector::Vector; + use crate::types::vector::Vector; /// Constructs a `FixedSizeList` storage dtype with the given float [`PType`], list size, and /// [`Nullability`]. @@ -138,4 +137,61 @@ mod tests { assert_eq!(deserialized, EmptyMetadata); Ok(()) } + + /// Constructs a `Vector` ext dtype wrapped in `DType::Extension`. + fn vector_dtype(ptype: PType, dims: u32) -> VortexResult { + vector_dtype_with_outer(ptype, dims, Nullability::NonNullable) + } + + /// Constructs a `Vector` ext dtype with the given outer `Nullability`, wrapped in + /// `DType::Extension`. + fn vector_dtype_with_outer(ptype: PType, dims: u32, outer: Nullability) -> VortexResult { + let storage = vector_storage_dtype(ptype, dims, outer); + Ok(DType::Extension( + ExtDType::::try_new(EmptyMetadata, storage)?.erased(), + )) + } + + #[test] + fn vector_widens_float_precision() -> VortexResult<()> { + let lhs = vector_dtype(PType::F32, 768)?; + let rhs = vector_dtype(PType::F64, 768)?; + let expected = vector_dtype(PType::F64, 768)?; + assert_eq!(lhs.least_supertype(&rhs), Some(expected)); + Ok(()) + } + + #[test] + fn vector_dim_mismatch_returns_none() -> VortexResult<()> { + let lhs = vector_dtype(PType::F32, 768)?; + let rhs = vector_dtype(PType::F32, 1024)?; + assert_eq!(lhs.least_supertype(&rhs), None); + Ok(()) + } + + #[test] + fn vector_vs_non_extension_returns_none() -> VortexResult<()> { + let lhs = vector_dtype(PType::F32, 768)?; + let rhs = DType::Primitive(PType::F32, Nullability::NonNullable); + assert_eq!(lhs.least_supertype(&rhs), None); + Ok(()) + } + + #[test] + fn vector_unions_outer_nullability_with_float_widening() -> VortexResult<()> { + let lhs = vector_dtype_with_outer(PType::F32, 4, Nullability::NonNullable)?; + let rhs = vector_dtype_with_outer(PType::F64, 4, Nullability::Nullable)?; + let expected = vector_dtype_with_outer(PType::F64, 4, Nullability::Nullable)?; + assert_eq!(lhs.least_supertype(&rhs), Some(expected)); + Ok(()) + } + + #[test] + fn vector_same_ptype_unions_outer_nullability() -> VortexResult<()> { + let lhs = vector_dtype_with_outer(PType::F32, 4, Nullability::NonNullable)?; + let rhs = vector_dtype_with_outer(PType::F32, 4, Nullability::Nullable)?; + let expected = vector_dtype_with_outer(PType::F32, 4, Nullability::Nullable)?; + assert_eq!(lhs.least_supertype(&rhs), Some(expected)); + Ok(()) + } } diff --git a/vortex-tensor/src/utils.rs b/vortex-tensor/src/utils.rs index 4d78597c962..e6d2cce453b 100644 --- a/vortex-tensor/src/utils.rs +++ b/vortex-tensor/src/utils.rs @@ -1,6 +1,8 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +use half::f16; +use prost::Message; use vortex_array::ArrayRef; use vortex_array::ExecutionCtx; use vortex_array::IntoArray; @@ -8,20 +10,70 @@ use vortex_array::arrays::Constant; use vortex_array::arrays::ConstantArray; use vortex_array::arrays::FixedSizeListArray; use vortex_array::arrays::PrimitiveArray; +use vortex_array::arrays::ScalarFn; use vortex_array::arrays::fixed_size_list::FixedSizeListArrayExt; +use vortex_array::arrays::primitive::PrimitiveArrayExt; use vortex_array::arrays::scalar_fn::ExactScalarFn; +use vortex_array::arrays::scalar_fn::ScalarFnArrayExt; +use vortex_array::arrays::scalar_fn::ScalarFnArrayView; use vortex_array::dtype::DType; use vortex_array::dtype::NativePType; use vortex_array::dtype::PType; +use vortex_array::dtype::proto::dtype as pb; +use vortex_array::scalar_fn::ScalarFnVTable; +use vortex_buffer::Buffer; use vortex_error::VortexExpect; use vortex_error::VortexResult; +use vortex_error::vortex_bail; use vortex_error::vortex_ensure; use vortex_error::vortex_err; +use vortex_session::VortexSession; use crate::matcher::AnyTensor; use crate::matcher::TensorMatch; use crate::scalar_fns::l2_denorm::L2Denorm; +/// Safety factor for unit-norm tolerance. Applied as a constant multiplier on the probabilistic +/// `√d · ε` bound so that legitimate round-off noise clears the check with headroom. +pub(crate) const SAFETY_FACTOR: usize = 10; + +/// Returns the acceptable unit-norm drift for the given element precision and dimension count. +/// +/// Uses the `c · √d · ε` bound where ε is machine epsilon and d is the vector dimension. Under +/// IEEE 754 round-to-nearest the probabilistic (RMS-case) forward error for computing ‖x‖₂ grows +/// as `O(√d · ε)` rather than the worst-case `O(d · ε)` from the classical Wilkinson bound, +/// assuming near-independent rounding errors across the d-term summation. +/// +/// Reference: Croci, Fasi, Higham, Mary, Mikaitis (2022). "Stochastic rounding: implementation, +/// error analysis and applications." Royal Society Open Science, 9: 211631, §6.1 "Probabilistic +/// error analysis." https://doi.org/10.1098/rsos.211631 +pub fn unit_norm_tolerance(element_ptype: PType, dimensions: usize) -> f64 { + let machine_epsilon: f64 = match element_ptype { + PType::F64 => f64::EPSILON, + PType::F32 => f32::EPSILON as f64, + PType::F16 => f16::EPSILON.to_f64_const(), + _ => unreachable!("unit_norm_tolerance requires a float ptype, got {element_ptype:?}"), + }; + + let dimensions_root = (dimensions as f64).sqrt(); + + SAFETY_FACTOR as f64 * machine_epsilon * dimensions_root +} + +/// Extracts the `(normalized, norms)` children from an [`L2Denorm`] scalar function array. +/// +/// [`L2Denorm`]: crate::scalar_fns::l2_denorm::L2Denorm +pub fn extract_l2_denorm_children(array: &ArrayRef) -> (ArrayRef, ArrayRef) { + let sfn = array + .as_opt::>() + .vortex_expect("expected ScalarFnArray wrapping L2Denorm"); + ( + sfn.nth_child(0) + .vortex_expect("L2Denorm missing normalized array"), + sfn.nth_child(1).vortex_expect("L2Denorm missing norms"), + ) +} + /// Validates that `input_dtype` is a float-valued tensor-like extension dtype. pub fn validate_tensor_float_input(input_dtype: &DType) -> VortexResult> { let ext = input_dtype @@ -41,15 +93,62 @@ pub fn validate_tensor_float_input(input_dtype: &DType) -> VortexResult( + lhs: &'a DType, + rhs: &DType, +) -> VortexResult> { + vortex_ensure!( + lhs.eq_ignore_nullability(rhs), + "binary tensor expression expects inputs to have the same dtype, got {lhs} and {rhs}" + ); + validate_tensor_float_input(lhs) +} + +/// Cast a float [`PrimitiveArray`] to a `Buffer`. +/// +/// Several operations in this crate (SORF transform, TurboQuant quantization) work exclusively +/// in f32. This function handles the cast from any float ptype: +/// +/// - f16: losslessly widened to f32. +/// - f32: zero-copy buffer extraction. +/// - f64: truncated to f32 precision. Values outside f32 range become +/- infinity. This is +/// acceptable because callers of this function operate in f32 and document this constraint. +pub fn cast_to_f32(prim: PrimitiveArray) -> VortexResult> { + match prim.ptype() { + PType::F16 => Ok(prim + .as_slice::() + .iter() + .map(|&v| f32::from(v)) + .collect()), + PType::F32 => Ok(prim.into_buffer()), + PType::F64 => Ok(prim + .as_slice::() + .iter() + .map(|&v| { + #[expect( + clippy::cast_possible_truncation, + reason = "f64 values outside f32 range become infinity, which is acceptable \ + because callers operate in f32 and document this constraint" + )] + let v = v as f32; + v + }) + .collect()), + other => vortex_bail!("expected float elements, got {other:?}"), + } +} + /// The flat primitive elements of a tensor storage array, with typed row access. /// /// This struct hides the stride detail that arises from the [`ConstantArray`] optimization: a -/// constant input materializes only a single row (stride=0), while a full array uses -/// stride=list_size. +/// constant-backed input materializes only a single row that every index reads (`is_constant = +/// true`), while a full array stores one row per index. pub struct FlatElements { elems: PrimitiveArray, - stride: usize, list_size: usize, + is_constant: bool, } impl FlatElements { @@ -60,153 +159,260 @@ impl FlatElements { } /// Returns the `i`-th row as a typed slice of length `list_size`. + /// + /// When the source was a constant-backed storage, all indices resolve to the single stored + /// row. #[must_use] pub fn row(&self, i: usize) -> &[T] { + let row_idx = if self.is_constant { 0 } else { i }; let slice = self.elems.as_slice::(); - &slice[i * self.stride..][..self.list_size] + &slice[row_idx * self.list_size..][..self.list_size] } } /// Extracts the flat primitive elements from a tensor storage array (FixedSizeList). /// /// When the input is a [`ConstantArray`] (e.g., a literal query vector), only a single row is -/// materialized to avoid expanding it to the full column length. +/// materialized to avoid expanding it to the full column length. Callers that have already +/// confirmed the storage is constant-backed should prefer [`extract_constant_flat_row`]. pub fn extract_flat_elements( storage: &ArrayRef, list_size: usize, ctx: &mut ExecutionCtx, ) -> VortexResult { - if let Some(constant) = storage.as_opt::() { - // Rewrite the array as a length 1 array so when we canonicalize, we do not duplicate a huge - // amount of data. + // Constant-backed storage: materialize just the single stored row so canonicalization does + // not expand the array to the full column length. + let (source, is_constant) = if let Some(constant) = storage.as_opt::() { let single = ConstantArray::new(constant.scalar().clone(), 1).into_array(); - let fsl: FixedSizeListArray = single.execute(ctx)?; - let elems: PrimitiveArray = fsl.elements().clone().execute(ctx)?; - return Ok(FlatElements { - elems, - stride: 0, - list_size, - }); - } + (single, true) + } else { + (storage.clone(), false) + }; - // Otherwise we have to fully expand all of the data. - let fsl: FixedSizeListArray = storage.clone().execute(ctx)?; + let fsl: FixedSizeListArray = source.execute(ctx)?; let elems: PrimitiveArray = fsl.elements().clone().execute(ctx)?; + vortex_ensure!( + !elems.nullability().is_nullable(), + "tensor storage elements must be non-nullable, got {}", + elems.dtype(), + ); Ok(FlatElements { elems, - stride: list_size, list_size, + is_constant, }) } -/// Extracts the `(normalized, norms)` children from an [`L2Denorm`] scalar function array. +/// The single stored row of a constant-backed tensor storage array. /// -/// [`L2Denorm`]: crate::scalar_fns::l2_denorm::L2Denorm -pub fn extract_l2_denorm_children(array: &ArrayRef) -> (ArrayRef, ArrayRef) { - let sfn = array - .as_opt::>() - .vortex_expect("expected ScalarFnArray wrapping L2Denorm"); - ( - sfn.nth_child(0) - .vortex_expect("L2Denorm missing normalized array"), - sfn.nth_child(1).vortex_expect("L2Denorm missing norms"), - ) +/// Contrast with [`FlatElements`], which exposes arbitrary row indices: a `FlatRow` statically +/// encodes "there is exactly one row available," so call sites that have gated on a constant input +/// read the row via [`Self::as_slice`] instead of `row(0)`. +pub struct FlatRow { + elems: PrimitiveArray, +} + +impl FlatRow { + /// Returns the [`PType`] of the underlying elements. + #[must_use] + pub fn ptype(&self) -> PType { + self.elems.ptype() + } + + /// Returns the stored row as a typed slice. Its length equals the storage scalar's + /// fixed-size-list size. + #[must_use] + pub fn as_slice(&self) -> &[T] { + self.elems.as_slice::() + } +} + +/// Extracts the single stored row from a [`Constant`]-backed tensor storage array. +/// +/// The caller must have confirmed that `storage` is a [`Constant`] encoding whose scalar is a +/// non-null fixed-size list. This is the fast path for constant query vectors: exactly one row is +/// materialized regardless of the column length. +/// +/// # Panics +/// +/// Panics if `storage` is not a [`Constant`] encoding. +pub fn extract_constant_flat_row( + storage: &ArrayRef, + ctx: &mut ExecutionCtx, +) -> VortexResult { + let constant = storage + .as_opt::() + .vortex_expect("extract_constant_flat_row requires Constant-backed storage"); + let single = ConstantArray::new(constant.scalar().clone(), 1).into_array(); + let fsl: FixedSizeListArray = single.execute(ctx)?; + let elems: PrimitiveArray = fsl.elements().clone().execute(ctx)?; + vortex_ensure!( + !elems.nullability().is_nullable(), + "tensor storage elements must be non-nullable, got {}", + elems.dtype(), + ); + Ok(FlatRow { elems }) +} + +/// Metadata for a serialized binary tensor-op array (shared by [`InnerProduct`] and +/// [`CosineSimilarity`]). Both operands share the same extension dtype up to nullability +/// (enforced by their `return_dtype` checks), but their individual nullabilities are lost in the +/// parent's unioned output, so both are persisted. +/// +/// [`CosineSimilarity`]: crate::scalar_fns::cosine_similarity::CosineSimilarity +#[derive(Clone, prost::Message)] +pub(crate) struct BinaryTensorOpMetadata { + #[prost(message, optional, tag = "1")] + pub(crate) lhs_dtype: Option, + #[prost(message, optional, tag = "2")] + pub(crate) rhs_dtype: Option, +} + +impl BinaryTensorOpMetadata { + /// Encodes the two children of `view` into a [`BinaryTensorOpMetadata`] byte blob. + pub(crate) fn encode_from_view( + view: &ScalarFnArrayView, + ) -> VortexResult> { + let scalar_fn_array = view.as_::(); + let lhs_dtype = Some(scalar_fn_array.child_at(0).dtype().try_into()?); + let rhs_dtype = Some(scalar_fn_array.child_at(1).dtype().try_into()?); + Ok(Self { + lhs_dtype, + rhs_dtype, + } + .encode_to_vec()) + } + + /// Decodes `metadata` and fetches both children from `children` using the decoded dtypes, + /// validating that `lhs` and `rhs` are compatible tensor operands. + pub(crate) fn decode_children( + metadata: &[u8], + len: usize, + children: &dyn vortex_array::serde::ArrayChildren, + session: &VortexSession, + ) -> VortexResult> { + let metadata = Self::decode(metadata) + .map_err(|e| vortex_err!("Failed to decode BinaryTensorOpMetadata: {e}"))?; + let lhs_pb = metadata + .lhs_dtype + .as_ref() + .ok_or_else(|| vortex_err!("metadata missing lhs_dtype"))?; + let rhs_pb = metadata + .rhs_dtype + .as_ref() + .ok_or_else(|| vortex_err!("metadata missing rhs_dtype"))?; + + let lhs_dtype = DType::from_proto(lhs_pb, session)?; + let rhs_dtype = DType::from_proto(rhs_pb, session)?; + validate_binary_tensor_float_inputs(&lhs_dtype, &rhs_dtype)?; + + let lhs = children.get(0, &lhs_dtype, len)?; + let rhs = children.get(1, &rhs_dtype, len)?; + Ok(vec![lhs, rhs]) + } } #[cfg(test)] pub mod test_helpers { use vortex_array::ArrayRef; + use vortex_array::ExecutionCtx; use vortex_array::IntoArray; use vortex_array::arrays::ConstantArray; use vortex_array::arrays::ExtensionArray; use vortex_array::arrays::FixedSizeListArray; + use vortex_array::arrays::PrimitiveArray; use vortex_array::dtype::DType; + use vortex_array::dtype::NativePType; use vortex_array::dtype::Nullability; - use vortex_array::dtype::PType; use vortex_array::dtype::extension::ExtDType; - use vortex_array::extension::EmptyMetadata; + use vortex_array::scalar::PValue; use vortex_array::scalar::Scalar; use vortex_array::validity::Validity; use vortex_buffer::Buffer; use vortex_error::VortexResult; - use crate::fixed_shape::FixedShapeTensor; - use crate::fixed_shape::FixedShapeTensorMetadata; - use crate::vector::Vector; + use crate::scalar_fns::l2_denorm::L2Denorm; + use crate::types::fixed_shape_tensor::FixedShapeTensor; + use crate::types::fixed_shape_tensor::FixedShapeTensorMetadata; + use crate::types::vector::Vector; + + /// Builds a `FixedSizeList` storage array from flat `elements`. The row count is + /// inferred from `elements.len() / list_size`. + fn flat_fsl(elements: &[T], list_size: u32) -> ArrayRef { + let row_count = elements.len() / list_size as usize; + let elems: ArrayRef = Buffer::copy_from(elements).into_array(); + FixedSizeListArray::new(elems, list_size, Validity::NonNullable, row_count).into_array() + } - /// Builds a [`FixedShapeTensor`] extension array from flat f64 elements and a logical shape. + /// Builds an FSL-valued [`Scalar`] from `elements` for use as a constant query. + fn fsl_scalar>(elements: &[T]) -> Scalar { + let element_dtype = DType::Primitive(T::PTYPE, Nullability::NonNullable); + let children: Vec = elements + .iter() + .map(|&v| Scalar::primitive(v, Nullability::NonNullable)) + .collect(); + Scalar::fixed_size_list(element_dtype, children, Nullability::NonNullable) + } + + /// Builds a [`FixedShapeTensor`] extension array from flat `elements` and a logical shape. /// /// The number of rows is inferred from the total element count divided by the product of the /// shape dimensions. For 0-dimensional tensors (scalar), each element is one row. - pub fn tensor_array(shape: &[usize], elements: &[f64]) -> VortexResult { + pub fn tensor_array(shape: &[usize], elements: &[T]) -> VortexResult { let list_size: u32 = shape.iter().product::().max(1).try_into().unwrap(); - let row_count = elements.len() / list_size as usize; - - let elems: ArrayRef = Buffer::copy_from(elements).into_array(); - let fsl = FixedSizeListArray::new(elems, list_size, Validity::NonNullable, row_count); - + let storage = flat_fsl(elements, list_size); let metadata = FixedShapeTensorMetadata::new(shape.to_vec()); let ext_dtype = - ExtDType::::try_new(metadata, fsl.dtype().clone())?.erased(); - - Ok(ExtensionArray::new(ext_dtype, fsl.into_array()).into_array()) + ExtDType::::try_new(metadata, storage.dtype().clone())?.erased(); + Ok(ExtensionArray::new(ext_dtype, storage).into_array()) } - /// Builds a [`Vector`] extension array from flat f64 elements and a vector dimension size. - pub fn vector_array(dim: u32, elements: &[f64]) -> VortexResult { - let row_count = elements.len() / dim as usize; - - let elems: ArrayRef = Buffer::copy_from(elements).into_array(); - let fsl = FixedSizeListArray::new(elems, dim, Validity::NonNullable, row_count); - - let ext_dtype = ExtDType::::try_new(EmptyMetadata, fsl.dtype().clone())?.erased(); - - Ok(ExtensionArray::new(ext_dtype, fsl.into_array()).into_array()) + /// Builds a [`Vector`] extension array from flat `elements` and a vector dimension size. + pub fn vector_array(dim: u32, elements: &[T]) -> VortexResult { + Vector::try_new_vector_array(flat_fsl(elements, dim)) } /// Builds a [`FixedShapeTensor`] extension array whose storage is a [`ConstantArray`], /// representing a single query tensor broadcast to `len` rows. - pub fn constant_tensor_array( + pub fn constant_tensor_array>( shape: &[usize], - elements: &[f64], + elements: &[T], len: usize, ) -> VortexResult { - let element_dtype = DType::Primitive(PType::F64, Nullability::NonNullable); - - let children: Vec = elements - .iter() - .map(|&v| Scalar::primitive(v, Nullability::NonNullable)) - .collect(); - let storage_scalar = - Scalar::fixed_size_list(element_dtype, children, Nullability::NonNullable); - - let storage = ConstantArray::new(storage_scalar, len).into_array(); - + let storage = ConstantArray::new(fsl_scalar(elements), len).into_array(); let metadata = FixedShapeTensorMetadata::new(shape.to_vec()); let ext_dtype = ExtDType::::try_new(metadata, storage.dtype().clone())?.erased(); - Ok(ExtensionArray::new(ext_dtype, storage).into_array()) } - /// Builds a [`Vector`] extension array whose storage is a [`ConstantArray`], representing a - /// single query vector broadcast to `len` rows. - pub fn constant_vector_array(elements: &[f64], len: usize) -> VortexResult { - let element_dtype = DType::Primitive(PType::F64, Nullability::NonNullable); - - let children: Vec = elements - .iter() - .map(|&v| Scalar::primitive(v, Nullability::NonNullable)) - .collect(); - let storage_scalar = - Scalar::fixed_size_list(element_dtype, children, Nullability::NonNullable); - - let storage = ConstantArray::new(storage_scalar, len).into_array(); - - let ext_dtype = - ExtDType::::try_new(EmptyMetadata, storage.dtype().clone())?.erased(); + /// Builds a [`ConstantArray`] whose scalar is itself a [`Vector`] extension scalar, broadcast + /// to `len` rows. This is the shape produced by an `lit(vector_scalar)` literal expression — + /// the constant lives at the extension level rather than inside the FSL storage, in contrast + /// to [`Vector::constant_array`]. + pub fn literal_vector_array>( + elements: &[T], + len: usize, + ) -> ArrayRef { + use vortex_array::extension::EmptyMetadata; + let ext_scalar = Scalar::extension::(EmptyMetadata, fsl_scalar(elements)); + ConstantArray::new(ext_scalar, len).into_array() + } - Ok(ExtensionArray::new(ext_dtype, storage).into_array()) + /// Creates an [`L2Denorm`] scalar function array from pre-normalized tensor elements and + /// matching norms. The caller must ensure every row of `normalized_elements` is unit-norm or + /// zero. + pub fn l2_denorm_array( + shape: &[usize], + normalized_elements: &[T], + norms: &[T], + ctx: &mut ExecutionCtx, + ) -> VortexResult { + let len = norms.len(); + let normalized = tensor_array(shape, normalized_elements)?; + let norms = + PrimitiveArray::new(Buffer::copy_from(norms), Validity::NonNullable).into_array(); + Ok(L2Denorm::try_new_array(normalized, norms, len, ctx)?.into_array()) } /// Asserts that each element in `actual` is within `1e-10` of the corresponding `expected` diff --git a/vortex-tensor/src/vector/mod.rs b/vortex-tensor/src/vector/mod.rs deleted file mode 100644 index 3c6a8a8c8cc..00000000000 --- a/vortex-tensor/src/vector/mod.rs +++ /dev/null @@ -1,15 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -//! Vector extension type for fixed-length float vectors (e.g., embeddings). - -/// The Vector extension type. -#[derive(Clone, Debug, Default, PartialEq, Eq, Hash)] -pub struct Vector; - -mod matcher; - -pub use matcher::AnyVector; -pub use matcher::VectorMatcherMetadata; - -mod vtable; diff --git a/vortex-tensor/src/vector_search.rs b/vortex-tensor/src/vector_search.rs new file mode 100644 index 00000000000..2a036accdfe --- /dev/null +++ b/vortex-tensor/src/vector_search.rs @@ -0,0 +1,175 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Reusable helpers for building brute-force vector similarity search expressions over +//! [`Vector`] extension arrays. +//! +//! [`build_similarity_search_tree`] broadcasts the query into the shape expected by +//! [`CosineSimilarity`] via `Vector::constant_array` and returns a lazy +//! `Binary(Gt, [CosineSimilarity(data, query), threshold])` expression. The caller is responsible +//! for preparing `data` (e.g. by running it through [`turboquant_encode`]); this builder does not +//! compress. +//! +//! Executing the tree into a [`BoolArray`] yields one boolean per row indicating whether that row's +//! cosine similarity to the query exceeds `threshold`. +//! +//! # Example +//! +//! ```ignore +//! use vortex_array::{ArrayRef, VortexSessionExecute}; +//! use vortex_array::arrays::BoolArray; +//! use vortex_session::VortexSession; +//! use vortex_tensor::encodings::turboquant::{TurboQuantConfig, turboquant_encode}; +//! use vortex_tensor::vector_search::build_similarity_search_tree; +//! +//! fn run(session: &VortexSession, data: ArrayRef, query: &[f32]) -> anyhow::Result<()> { +//! let mut ctx = session.create_execution_ctx(); +//! let data = turboquant_encode(data, &TurboQuantConfig::default(), &mut ctx)?; +//! let tree = build_similarity_search_tree(data, query, 0.8)?; +//! let _matches: BoolArray = tree.execute(&mut ctx)?; +//! Ok(()) +//! } +//! ``` +//! +//! [`Vector`]: crate::vector::Vector +//! [`CosineSimilarity`]: crate::scalar_fns::cosine_similarity::CosineSimilarity +//! [`turboquant_encode`]: crate::encodings::turboquant::turboquant_encode +//! [`BoolArray`]: vortex_array::arrays::BoolArray + +use vortex_array::ArrayRef; +use vortex_array::IntoArray; +use vortex_array::arrays::ConstantArray; +use vortex_array::builtins::ArrayBuiltins; +use vortex_array::dtype::NativePType; +use vortex_array::dtype::Nullability; +use vortex_array::scalar::PValue; +use vortex_array::scalar::Scalar; +use vortex_array::scalar_fn::fns::operators::Operator; +use vortex_error::VortexResult; + +use crate::scalar_fns::cosine_similarity::CosineSimilarity; +use crate::types::vector::Vector; + +/// Build the lazy similarity-search expression tree for a prepared database array and a +/// single query vector. +/// +/// The returned array is a lazy boolean expression of length `data.len()` whose position `i` +/// is `true` iff `cosine_similarity(data[i], query) > threshold`. Executing it into a +/// [`BoolArray`](vortex_array::arrays::BoolArray) runs the full scan. +/// +/// The tree shape is: +/// +/// ```text +/// Binary(Gt, [ +/// CosineSimilarity([data, ConstantArray(query_vec, n)]), +/// ConstantArray(threshold, n), +/// ]) +/// ``` +/// +/// The element type is inferred from `T` and must match the element type of `data`'s +/// [`Vector`] extension dtype. +/// +/// This function performs no execution; it is safe to call inside a benchmark setup closure. +/// +/// # Errors +/// +/// Returns an error if `query` has a length incompatible with `data`'s vector dimension, or +/// if any of the intermediate array constructors fails. +pub fn build_similarity_search_tree>( + data: ArrayRef, + query: &[T], + threshold: T, +) -> VortexResult { + let num_rows = data.len(); + let query_vec = Vector::constant_array(query, num_rows)?; + + let cosine = CosineSimilarity::try_new_array(data, query_vec, num_rows)?.into_array(); + + let threshold_scalar = Scalar::primitive(threshold, Nullability::NonNullable); + let threshold_array = ConstantArray::new(threshold_scalar, num_rows).into_array(); + + cosine.binary(threshold_array, Operator::Gt) +} + +#[cfg(test)] +mod tests { + use vortex_array::VortexSessionExecute; + use vortex_array::arrays::BoolArray; + use vortex_array::arrays::bool::BoolArrayExt; + use vortex_error::VortexResult; + + use super::build_similarity_search_tree; + use crate::encodings::turboquant::TurboQuantConfig; + use crate::encodings::turboquant::turboquant_encode; + use crate::tests::SESSION; + use crate::utils::test_helpers::vector_array; + + #[test] + fn similarity_search_tree_executes_to_bool_array() -> VortexResult<()> { + // 4 rows of 3-dim vectors; the first and last match the query [1, 0, 0]. + let data = vector_array( + 3, + &[ + 1.0f32, 0.0, 0.0, // + 0.0, 1.0, 0.0, // + 0.0, 0.0, 1.0, // + 1.0, 0.0, 0.0, // + ], + )?; + let query = [1.0f32, 0.0, 0.0]; + + let tree = build_similarity_search_tree(data, &query, 0.5)?; + let mut ctx = SESSION.create_execution_ctx(); + let result: BoolArray = tree.execute(&mut ctx)?; + + let bits = result.to_bit_buffer(); + assert_eq!(bits.len(), 4); + assert!(bits.value(0)); + assert!(!bits.value(1)); + assert!(!bits.value(2)); + assert!(bits.value(3)); + Ok(()) + } + + #[test] + fn turboquant_roundtrip_preserves_ranking() -> VortexResult<()> { + // Build 6 rows of 128-dim vectors where row 0 is highly correlated with the query. + // TurboQuant should preserve the "row 0 is the best match" ordering. + const DIM: u32 = 128; + const NUM_ROWS: usize = 6; + + let mut values = Vec::::with_capacity(NUM_ROWS * DIM as usize); + let query: Vec = (0..DIM as usize) + .map(|i| ((i as f32) * 0.017).sin()) + .collect(); + + // Row 0: identical to query (cosine=1.0) + values.extend_from_slice(&query); + // Row 1: query + noise + for (i, q) in query.iter().enumerate() { + values.push(q + 0.05 * ((i as f32) * 0.03).cos()); + } + // Rows 2..6: unrelated patterns + for row in 2..NUM_ROWS { + for i in 0..DIM as usize { + values.push(((row as f32 * 1.3 + i as f32) * 0.07).sin()); + } + } + + let data = vector_array(DIM, &values)?; + let mut ctx = SESSION.create_execution_ctx(); + let compressed = turboquant_encode(data, &TurboQuantConfig::default(), &mut ctx)?; + assert_eq!(compressed.len(), NUM_ROWS); + + // Build a tree with a low threshold so row 0 (cosine=1.0 exact) matches. + let tree = build_similarity_search_tree(compressed, &query, 0.95)?; + let result: BoolArray = tree.execute(&mut ctx)?; + let bits = result.to_bit_buffer(); + assert_eq!(bits.len(), NUM_ROWS); + assert!( + bits.value(0), + "row 0 (identical to query) must match at threshold 0.95 even after TurboQuant" + ); + Ok(()) + } +} diff --git a/vortex-test/compat-gen/Cargo.toml b/vortex-test/compat-gen/Cargo.toml index 62906843e8b..dd8bec4703a 100644 --- a/vortex-test/compat-gen/Cargo.toml +++ b/vortex-test/compat-gen/Cargo.toml @@ -45,6 +45,7 @@ tokio = { workspace = true, features = ["full"] } reqwest = { workspace = true } # CLI + serialization +base16ct = { workspace = true } clap = { workspace = true, features = ["derive"] } serde = { workspace = true, features = ["derive"] } serde_json = { workspace = true } diff --git a/vortex-test/compat-gen/src/adapter.rs b/vortex-test/compat-gen/src/adapter.rs index 9e29d6d69f0..a97399dac79 100644 --- a/vortex-test/compat-gen/src/adapter.rs +++ b/vortex-test/compat-gen/src/adapter.rs @@ -13,7 +13,9 @@ use futures::stream; use tokio::runtime::Runtime; use vortex::VortexSessionDefault; use vortex::array::ArrayRef; +use vortex::array::LEGACY_SESSION; use vortex::array::MaskFuture; +use vortex::array::VortexSessionExecute; use vortex::array::expr::root; use vortex::file::OpenOptionsSessionExt; use vortex::file::WriteOptionsSessionExt; @@ -40,8 +42,9 @@ fn runtime() -> VortexResult { /// all stats so they are present in the serialized output. pub fn compute_all_stats(array: &ArrayRef) -> VortexResult<()> { let all_stats: Vec = Stat::all().collect(); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); for node in array.depth_first_traversal() { - let computed = node.statistics().compute_all(&all_stats)?; + let computed = node.statistics().compute_all(&all_stats, &mut ctx)?; node.statistics().set_iter(computed.into_iter()); } Ok(()) diff --git a/vortex-test/compat-gen/src/fixtures/arrays/datasets/mod.rs b/vortex-test/compat-gen/src/fixtures/arrays/datasets/mod.rs index c799bc7a380..929e416fe14 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/datasets/mod.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/datasets/mod.rs @@ -2,7 +2,6 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors mod clickbench; -#[allow(clippy::cast_possible_truncation)] mod tpch; use crate::fixtures::DatasetFixture; diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/bool.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/bool.rs index e710438a2ce..97a05f8458b 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/bool.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/bool.rs @@ -3,6 +3,7 @@ use vortex_array::ArrayId; use vortex_array::ArrayRef; +use vortex_array::ArrayVTable; use vortex_array::IntoArray; use vortex_array::arrays::Bool; use vortex_array::arrays::BoolArray; @@ -25,7 +26,7 @@ impl FlatLayoutFixture for BooleansFixture { } fn expected_encodings(&self) -> Vec { - vec![Bool::ID] + vec![Bool.id()] } fn build(&self) -> VortexResult { diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/chunked.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/chunked.rs index 683d5b809a3..8f5782fb36e 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/chunked.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/chunked.rs @@ -3,6 +3,7 @@ use vortex_array::ArrayId; use vortex_array::ArrayRef; +use vortex_array::ArrayVTable; use vortex_array::IntoArray; use vortex_array::arrays::ChunkedArray; use vortex_array::arrays::Primitive; @@ -26,7 +27,7 @@ impl FlatLayoutFixture for ChunkedFixture { } fn expected_encodings(&self) -> Vec { - vec![Primitive::ID] + vec![Primitive.id()] } fn build(&self) -> VortexResult { diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/datetime.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/datetime.rs index 9571136ca38..792901d2bec 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/datetime.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/datetime.rs @@ -5,6 +5,7 @@ use std::sync::Arc; use vortex_array::ArrayId; use vortex_array::ArrayRef; +use vortex_array::ArrayVTable; use vortex_array::IntoArray; use vortex_array::arrays::Extension; use vortex_array::arrays::PrimitiveArray; @@ -30,7 +31,7 @@ impl FlatLayoutFixture for DateTimeFixture { } fn expected_encodings(&self) -> Vec { - vec![Extension::ID] + vec![Extension.id()] } fn build(&self) -> VortexResult { diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/decimal.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/decimal.rs index d7e261cdbf4..98f857f5988 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/decimal.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/decimal.rs @@ -3,6 +3,7 @@ use vortex_array::ArrayId; use vortex_array::ArrayRef; +use vortex_array::ArrayVTable; use vortex_array::IntoArray; use vortex_array::arrays::Decimal; use vortex_array::arrays::DecimalArray; @@ -27,7 +28,7 @@ impl FlatLayoutFixture for DecimalFixture { } fn expected_encodings(&self) -> Vec { - vec![Decimal::ID] + vec![Decimal.id()] } fn build(&self) -> VortexResult { diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/fixed_size_list.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/fixed_size_list.rs index c1c2e615b76..4b10ea607cf 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/fixed_size_list.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/fixed_size_list.rs @@ -3,6 +3,7 @@ use vortex_array::ArrayId; use vortex_array::ArrayRef; +use vortex_array::ArrayVTable; use vortex_array::IntoArray; use vortex_array::arrays::FixedSizeList; use vortex_array::arrays::FixedSizeListArray; @@ -27,7 +28,7 @@ impl FlatLayoutFixture for FixedSizeListFixture { } fn expected_encodings(&self) -> Vec { - vec![FixedSizeList::ID] + vec![FixedSizeList.id()] } fn build(&self) -> VortexResult { diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/list.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/list.rs index dd09e002630..ab844ab01b7 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/list.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/list.rs @@ -3,6 +3,7 @@ use vortex_array::ArrayId; use vortex_array::ArrayRef; +use vortex_array::ArrayVTable; use vortex_array::IntoArray; use vortex_array::arrays::List; use vortex_array::arrays::ListArray; @@ -28,7 +29,7 @@ impl FlatLayoutFixture for ListFixture { } fn expected_encodings(&self) -> Vec { - vec![List::ID] + vec![List.id()] } fn build(&self) -> VortexResult { diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/listview.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/listview.rs index 0a9ab2191ef..d81aaadfa5d 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/listview.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/listview.rs @@ -3,6 +3,7 @@ use vortex_array::ArrayId; use vortex_array::ArrayRef; +use vortex_array::ArrayVTable; use vortex_array::IntoArray; use vortex_array::arrays::ListView; use vortex_array::arrays::ListViewArray; @@ -28,7 +29,7 @@ impl FlatLayoutFixture for ListViewFixture { } fn expected_encodings(&self) -> Vec { - vec![ListView::ID] + vec![ListView.id()] } fn build(&self) -> VortexResult { diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/null.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/null.rs index 0d937a1c225..1c7dfa25f84 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/null.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/null.rs @@ -3,6 +3,7 @@ use vortex_array::ArrayId; use vortex_array::ArrayRef; +use vortex_array::ArrayVTable; use vortex_array::IntoArray; use vortex_array::arrays::Null; use vortex_array::arrays::NullArray; @@ -27,7 +28,7 @@ impl FlatLayoutFixture for NullFixture { } fn expected_encodings(&self) -> Vec { - vec![Null::ID] + vec![Null.id()] } fn build(&self) -> VortexResult { diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/primitive.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/primitive.rs index f77d5f42f39..a84c59f3fb5 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/primitive.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/primitive.rs @@ -3,6 +3,7 @@ use vortex_array::ArrayId; use vortex_array::ArrayRef; +use vortex_array::ArrayVTable; use vortex_array::IntoArray; use vortex_array::arrays::Primitive; use vortex_array::arrays::PrimitiveArray; @@ -26,7 +27,7 @@ impl FlatLayoutFixture for PrimitivesFixture { } fn expected_encodings(&self) -> Vec { - vec![Primitive::ID] + vec![Primitive.id()] } fn build(&self) -> VortexResult { diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/struct_nested.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/struct_nested.rs index cc02d856aab..d5d3eb4bd08 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/struct_nested.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/struct_nested.rs @@ -3,6 +3,7 @@ use vortex_array::ArrayId; use vortex_array::ArrayRef; +use vortex_array::ArrayVTable; use vortex_array::IntoArray; use vortex_array::arrays::PrimitiveArray; use vortex_array::arrays::Struct; @@ -27,7 +28,7 @@ impl FlatLayoutFixture for StructNestedFixture { } fn expected_encodings(&self) -> Vec { - vec![Struct::ID] + vec![Struct.id()] } fn build(&self) -> VortexResult { diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/varbin.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/varbin.rs index 72ac683ee44..26bbd3d554e 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/varbin.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/varbin.rs @@ -3,6 +3,7 @@ use vortex_array::ArrayId; use vortex_array::ArrayRef; +use vortex_array::ArrayVTable; use vortex_array::IntoArray; use vortex_array::arrays::StructArray; use vortex_array::arrays::VarBin; @@ -25,7 +26,7 @@ impl FlatLayoutFixture for VarBinFixture { } fn expected_encodings(&self) -> Vec { - vec![VarBin::ID] + vec![VarBin.id()] } fn build(&self) -> VortexResult { diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/varbinview.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/varbinview.rs index f795dafce49..ae32129a082 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/varbinview.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/arrays/varbinview.rs @@ -3,6 +3,7 @@ use vortex_array::ArrayId; use vortex_array::ArrayRef; +use vortex_array::ArrayVTable; use vortex_array::IntoArray; use vortex_array::arrays::StructArray; use vortex_array::arrays::VarBinView; @@ -25,7 +26,7 @@ impl FlatLayoutFixture for VarBinViewFixture { } fn expected_encodings(&self) -> Vec { - vec![VarBinView::ID] + vec![VarBinView.id()] } fn build(&self) -> VortexResult { diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/alp.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/alp.rs index cf20093cf21..4c4175b773d 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/alp.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/alp.rs @@ -3,7 +3,10 @@ use vortex::array::ArrayId; use vortex::array::ArrayRef; +use vortex::array::ArrayVTable; use vortex::array::IntoArray; +use vortex::array::LEGACY_SESSION; +use vortex::array::VortexSessionExecute; use vortex::array::arrays::PrimitiveArray; use vortex::array::arrays::StructArray; use vortex::array::dtype::FieldNames; @@ -55,7 +58,7 @@ impl FlatLayoutFixture for AlpFixture { } fn expected_encodings(&self) -> Vec { - vec![ALP::ID] + vec![ALP.id()] } fn build(&self) -> VortexResult { @@ -129,17 +132,72 @@ impl FlatLayoutFixture for AlpFixture { "f64_boundary_specials", ]), vec![ - alp_encode(&f64_prices, None)?.into_array(), - alp_encode(&f32_near_int, None)?.into_array(), - alp_encode(&f64_negative_near_int, None)?.into_array(), - alp_encode(&f64_currency, None)?.into_array(), - alp_encode(&f64_nullable, None)?.into_array(), - alp_encode(&f64_patched, None)?.into_array(), - alp_encode(&f64_patch_heavy, None)?.into_array(), - alp_encode(&f64_special_values, None)?.into_array(), - alp_encode(&f32_special_values, None)?.into_array(), - alp_encode(&f64_extremes, None)?.into_array(), - alp_encode(&f64_boundary_specials, None)?.into_array(), + alp_encode( + f64_prices.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + )? + .into_array(), + alp_encode( + f32_near_int.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + )? + .into_array(), + alp_encode( + f64_negative_near_int.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + )? + .into_array(), + alp_encode( + f64_currency.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + )? + .into_array(), + alp_encode( + f64_nullable.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + )? + .into_array(), + alp_encode( + f64_patched.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + )? + .into_array(), + alp_encode( + f64_patch_heavy.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + )? + .into_array(), + alp_encode( + f64_special_values.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + )? + .into_array(), + alp_encode( + f32_special_values.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + )? + .into_array(), + alp_encode( + f64_extremes.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + )? + .into_array(), + alp_encode( + f64_boundary_specials.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + )? + .into_array(), ], N, Validity::NonNullable, diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/alprd.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/alprd.rs index a6ca54363b6..97af909b966 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/alprd.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/alprd.rs @@ -3,7 +3,10 @@ use vortex::array::ArrayId; use vortex::array::ArrayRef; +use vortex::array::ArrayVTable; use vortex::array::IntoArray; +use vortex::array::LEGACY_SESSION; +use vortex::array::VortexSessionExecute; use vortex::array::arrays::PrimitiveArray; use vortex::array::arrays::StructArray; use vortex::array::dtype::FieldNames; @@ -41,10 +44,13 @@ impl FlatLayoutFixture for AlprdFixture { } fn expected_encodings(&self) -> Vec { - vec![ALPRD::ID] + vec![ALPRD.id()] } fn build(&self) -> VortexResult { + // NOTE: `FlatLayoutFixture::build` has a fixed trait signature without `ExecutionCtx`, so + // we construct a legacy ctx locally at this trait boundary. + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let sensor: PrimitiveArray = (0..N) .map(|i| { let noise = ((i * 7 + 13) % 100) as f64 / 1000.0; @@ -149,16 +155,32 @@ impl FlatLayoutFixture for AlprdFixture { "nullable_specials", ]), vec![ - sensor_enc.encode(&sensor).into_array(), - drift_enc.encode(&drift).into_array(), - constant_enc.encode(&constant_series).into_array(), - decreasing_enc.encode(&decreasing).into_array(), - oscillating_enc.encode(&oscillating).into_array(), - periodic_resets_enc.encode(&periodic_resets).into_array(), - nullable_enc.encode(&sensor_nullable).into_array(), - special_enc.encode(&special_values).into_array(), - boundary_enc.encode(&boundary_specials).into_array(), - nullable_special_enc.encode(&nullable_specials).into_array(), + sensor_enc.encode(sensor.as_view(), &mut ctx).into_array(), + drift_enc.encode(drift.as_view(), &mut ctx).into_array(), + constant_enc + .encode(constant_series.as_view(), &mut ctx) + .into_array(), + decreasing_enc + .encode(decreasing.as_view(), &mut ctx) + .into_array(), + oscillating_enc + .encode(oscillating.as_view(), &mut ctx) + .into_array(), + periodic_resets_enc + .encode(periodic_resets.as_view(), &mut ctx) + .into_array(), + nullable_enc + .encode(sensor_nullable.as_view(), &mut ctx) + .into_array(), + special_enc + .encode(special_values.as_view(), &mut ctx) + .into_array(), + boundary_enc + .encode(boundary_specials.as_view(), &mut ctx) + .into_array(), + nullable_special_enc + .encode(nullable_specials.as_view(), &mut ctx) + .into_array(), ], N, Validity::NonNullable, diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/bitpacked.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/bitpacked.rs index c0dc53817b3..f242ebc4e0a 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/bitpacked.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/bitpacked.rs @@ -3,7 +3,10 @@ use vortex::array::ArrayId; use vortex::array::ArrayRef; +use vortex::array::ArrayVTable; use vortex::array::IntoArray; +use vortex::array::LEGACY_SESSION; +use vortex::array::VortexSessionExecute; use vortex::array::arrays::PrimitiveArray; use vortex::array::arrays::StructArray; use vortex::array::dtype::FieldNames; @@ -27,10 +30,13 @@ impl FlatLayoutFixture for BitPackedFixture { } fn expected_encodings(&self) -> Vec { - vec![BitPacked::ID] + vec![BitPacked.id()] } fn build(&self) -> VortexResult { + // NOTE: `FlatLayoutFixture::build` has a fixed `(&self)` trait signature, so we cannot + // plumb `ExecutionCtx` from a caller; construct one from `LEGACY_SESSION` here. + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let u32_8bit: PrimitiveArray = (0..N as u32).map(|i| i % 256).collect(); let u64_12bit: PrimitiveArray = (0..N as u64).map(|i| i % 4096).collect(); let u16_4bit: PrimitiveArray = (0..N as u16).map(|i| i % 16).collect(); @@ -79,21 +85,21 @@ impl FlatLayoutFixture for BitPackedFixture { "u16_head_tail_nulls", ]), vec![ - bitpack_encode(&u32_8bit, 8, None)?.into_array(), - bitpack_encode(&u64_12bit, 12, None)?.into_array(), - bitpack_encode(&u16_4bit, 4, None)?.into_array(), - bitpack_encode(&u16_1bit, 1, None)?.into_array(), - bitpack_encode(&u32_nullable, 7, None)?.into_array(), - bitpack_encode(&u32_all_zero, 1, None)?.into_array(), - bitpack_encode(&u16_all_equal, 3, None)?.into_array(), - bitpack_encode(&u16_15bit, 15, None)?.into_array(), - bitpack_encode(&u32_31bit, 31, None)?.into_array(), - bitpack_encode(&u64_63bit, 63, None)?.into_array(), - bitpack_encode(&u8_3bit, 3, None)?.into_array(), - bitpack_encode(&u8_5bit, 5, None)?.into_array(), - bitpack_encode(&u16_9bit, 9, None)?.into_array(), - bitpack_encode(&u32_17bit, 17, None)?.into_array(), - bitpack_encode(&u16_head_tail_nulls, 5, None)?.into_array(), + bitpack_encode(&u32_8bit, 8, None, &mut ctx)?.into_array(), + bitpack_encode(&u64_12bit, 12, None, &mut ctx)?.into_array(), + bitpack_encode(&u16_4bit, 4, None, &mut ctx)?.into_array(), + bitpack_encode(&u16_1bit, 1, None, &mut ctx)?.into_array(), + bitpack_encode(&u32_nullable, 7, None, &mut ctx)?.into_array(), + bitpack_encode(&u32_all_zero, 1, None, &mut ctx)?.into_array(), + bitpack_encode(&u16_all_equal, 3, None, &mut ctx)?.into_array(), + bitpack_encode(&u16_15bit, 15, None, &mut ctx)?.into_array(), + bitpack_encode(&u32_31bit, 31, None, &mut ctx)?.into_array(), + bitpack_encode(&u64_63bit, 63, None, &mut ctx)?.into_array(), + bitpack_encode(&u8_3bit, 3, None, &mut ctx)?.into_array(), + bitpack_encode(&u8_5bit, 5, None, &mut ctx)?.into_array(), + bitpack_encode(&u16_9bit, 9, None, &mut ctx)?.into_array(), + bitpack_encode(&u32_17bit, 17, None, &mut ctx)?.into_array(), + bitpack_encode(&u16_head_tail_nulls, 5, None, &mut ctx)?.into_array(), ], N, Validity::NonNullable, diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/bytebool.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/bytebool.rs index 53cf79463d6..a410d2354c0 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/bytebool.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/bytebool.rs @@ -3,6 +3,7 @@ use vortex::array::ArrayId; use vortex::array::ArrayRef; +use vortex::array::ArrayVTable; use vortex::array::IntoArray; use vortex::array::arrays::BoolArray; use vortex::array::arrays::StructArray; @@ -27,7 +28,7 @@ impl FlatLayoutFixture for ByteBoolFixture { } fn expected_encodings(&self) -> Vec { - vec![ByteBool::ID] + vec![ByteBool.id()] } fn build(&self) -> VortexResult { diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/constant.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/constant.rs index 64feb357093..161f6d3c28e 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/constant.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/constant.rs @@ -3,7 +3,10 @@ use vortex::array::ArrayId; use vortex::array::ArrayRef; +use vortex::array::ArrayVTable; use vortex::array::IntoArray; +use vortex::array::LEGACY_SESSION; +use vortex::array::VortexSessionExecute; use vortex::array::arrays::Constant; use vortex::array::arrays::ConstantArray; use vortex::array::arrays::PrimitiveArray; @@ -34,7 +37,7 @@ impl FlatLayoutFixture for ConstantFixture { } fn expected_encodings(&self) -> Vec { - vec![Constant::ID] + vec![Constant.id()] } fn build(&self) -> VortexResult { @@ -75,7 +78,7 @@ impl FlatLayoutFixture for ConstantFixture { Some("UTC".into()), ) .into_array() - .scalar_at(0)?; + .execute_scalar(0, &mut LEGACY_SESSION.create_execution_ctx())?; let const_timestamp = ConstantArray::new(timestamp_scalar, N); let arr = StructArray::try_new( diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/datetimeparts.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/datetimeparts.rs index bfbf2cba21c..0bfb9985e1d 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/datetimeparts.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/datetimeparts.rs @@ -3,7 +3,10 @@ use vortex::array::ArrayId; use vortex::array::ArrayRef; +use vortex::array::ArrayVTable; use vortex::array::IntoArray; +use vortex::array::LEGACY_SESSION; +use vortex::array::VortexSessionExecute; use vortex::array::arrays::PrimitiveArray; use vortex::array::arrays::StructArray; use vortex::array::arrays::TemporalArray; @@ -21,7 +24,7 @@ pub struct DateTimePartsFixture; fn encode_temporal(temporal: TemporalArray) -> VortexResult { let dtype = temporal.dtype().clone(); - let parts = split_temporal(temporal)?; + let parts = split_temporal(temporal, &mut LEGACY_SESSION.create_execution_ctx())?; Ok(DateTimeParts::try_new(dtype, parts.days, parts.seconds, parts.subseconds)?.into_array()) } @@ -35,7 +38,7 @@ impl FlatLayoutFixture for DateTimePartsFixture { } fn expected_encodings(&self) -> Vec { - vec![DateTimeParts::ID] + vec![DateTimeParts.id()] } fn build(&self) -> VortexResult { diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/decimal_byte_parts.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/decimal_byte_parts.rs index 723df6c5378..79f6659c065 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/decimal_byte_parts.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/decimal_byte_parts.rs @@ -3,6 +3,7 @@ use vortex::array::ArrayId; use vortex::array::ArrayRef; +use vortex::array::ArrayVTable; use vortex::array::IntoArray; use vortex::array::arrays::PrimitiveArray; use vortex::array::arrays::StructArray; @@ -27,7 +28,7 @@ impl FlatLayoutFixture for DecimalBytePartsFixture { } fn expected_encodings(&self) -> Vec { - vec![DecimalByteParts::ID] + vec![DecimalByteParts.id()] } fn build(&self) -> VortexResult { diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/delta.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/delta.rs index f69e54c4631..21519d2a3b1 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/delta.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/delta.rs @@ -4,6 +4,7 @@ use vortex::VortexSessionDefault; use vortex::array::ArrayId; use vortex::array::ArrayRef; +use vortex::array::ArrayVTable; use vortex::array::IntoArray; use vortex::array::VortexSessionExecute; use vortex::array::arrays::PrimitiveArray; @@ -30,7 +31,7 @@ impl FlatLayoutFixture for DeltaFixture { } fn expected_encodings(&self) -> Vec { - vec![Delta::ID] + vec![Delta.id()] } fn build(&self) -> VortexResult { diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/dict.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/dict.rs index 84107b08b2a..add08b5f8e3 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/dict.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/dict.rs @@ -3,6 +3,7 @@ use vortex::array::ArrayId; use vortex::array::ArrayRef; +use vortex::array::ArrayVTable; use vortex::array::IntoArray; use vortex::array::arrays::Dict; use vortex::array::arrays::PrimitiveArray; @@ -28,7 +29,7 @@ impl FlatLayoutFixture for DictFixture { } fn expected_encodings(&self) -> Vec { - vec![Dict::ID] + vec![Dict.id()] } fn build(&self) -> VortexResult { diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/for_.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/for_.rs index 3c1a96b6bb1..70fe9d8d8ec 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/for_.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/for_.rs @@ -3,6 +3,7 @@ use vortex::array::ArrayId; use vortex::array::ArrayRef; +use vortex::array::ArrayVTable; use vortex::array::IntoArray; use vortex::array::arrays::PrimitiveArray; use vortex::array::arrays::StructArray; @@ -26,7 +27,7 @@ impl FlatLayoutFixture for FoRFixture { } fn expected_encodings(&self) -> Vec { - vec![FoR::ID] + vec![FoR.id()] } fn build(&self) -> VortexResult { diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/fsst.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/fsst.rs index 91d0f8dce07..17314629737 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/fsst.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/fsst.rs @@ -3,7 +3,10 @@ use vortex::array::ArrayId; use vortex::array::ArrayRef; +use vortex::array::ArrayVTable; use vortex::array::IntoArray; +use vortex::array::LEGACY_SESSION; +use vortex::array::VortexSessionExecute; use vortex::array::arrays::StructArray; use vortex::array::arrays::VarBinArray; use vortex::array::dtype::FieldNames; @@ -28,7 +31,7 @@ impl FlatLayoutFixture for FsstFixture { } fn expected_encodings(&self) -> Vec { - vec![FSST::ID] + vec![FSST.id()] } fn build(&self) -> VortexResult { @@ -108,6 +111,7 @@ impl FlatLayoutFixture for FsstFixture { let high_entropy_comp = fsst_train_compressor(&high_entropy_col); let all_null_clustered_comp = fsst_train_compressor(&all_null_clustered); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let arr = StructArray::try_new( FieldNames::from([ "urls", @@ -120,22 +124,44 @@ impl FlatLayoutFixture for FsstFixture { "all_null_clustered", ]), vec![ - fsst_compress(&url_col, url_col.len(), url_col.dtype(), &url_comp).into_array(), - fsst_compress(&log_col, log_col.len(), log_col.dtype(), &log_comp).into_array(), + fsst_compress( + &url_col, + url_col.len(), + url_col.dtype(), + &url_comp, + &mut ctx, + ) + .into_array(), + fsst_compress( + &log_col, + log_col.len(), + log_col.dtype(), + &log_comp, + &mut ctx, + ) + .into_array(), fsst_compress( &nullable_col, nullable_col.len(), nullable_col.dtype(), &nullable_comp, + &mut ctx, + ) + .into_array(), + fsst_compress( + &short_col, + short_col.len(), + short_col.dtype(), + &short_comp, + &mut ctx, ) .into_array(), - fsst_compress(&short_col, short_col.len(), short_col.dtype(), &short_comp) - .into_array(), fsst_compress( &empty_and_unicode_col, empty_and_unicode_col.len(), empty_and_unicode_col.dtype(), &empty_and_unicode_comp, + &mut ctx, ) .into_array(), fsst_compress( @@ -143,6 +169,7 @@ impl FlatLayoutFixture for FsstFixture { suffix_shared_col.len(), suffix_shared_col.dtype(), &suffix_shared_comp, + &mut ctx, ) .into_array(), fsst_compress( @@ -150,6 +177,7 @@ impl FlatLayoutFixture for FsstFixture { high_entropy_col.len(), high_entropy_col.dtype(), &high_entropy_comp, + &mut ctx, ) .into_array(), fsst_compress( @@ -157,6 +185,7 @@ impl FlatLayoutFixture for FsstFixture { all_null_clustered.len(), all_null_clustered.dtype(), &all_null_clustered_comp, + &mut ctx, ) .into_array(), ], diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/mod.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/mod.rs index b6c18064e0f..830b50450da 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/mod.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/mod.rs @@ -43,7 +43,8 @@ pub fn fixtures() -> Vec> { Box::new(dict::DictFixture), Box::new(fsst::FsstFixture), Box::new(for_::FoRFixture), - Box::new(patched::PatchedFixture), + // TODO(aduffy): add back once we stabilized Patched array + // Box::new(patched::PatchedFixture), Box::new(pco::PcoFixture), Box::new(rle::RleFixture), Box::new(runend::RunEndFixture), diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/patched.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/patched.rs index b9a43dfb1e2..fea0c990173 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/patched.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/patched.rs @@ -14,6 +14,10 @@ use vortex_session::VortexSession; use crate::fixtures::FlatLayoutFixture; +#[expect( + dead_code, + reason = "This will be unused until we stabilize Patched array" +)] pub struct PatchedFixture; impl FlatLayoutFixture for PatchedFixture { diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/pco.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/pco.rs index 33dd4b78e37..7c3f4edaf72 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/pco.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/pco.rs @@ -3,7 +3,10 @@ use vortex::array::ArrayId; use vortex::array::ArrayRef; +use vortex::array::ArrayVTable; use vortex::array::IntoArray; +use vortex::array::LEGACY_SESSION; +use vortex::array::VortexSessionExecute; use vortex::array::arrays::PrimitiveArray; use vortex::array::arrays::StructArray; use vortex::array::dtype::FieldNames; @@ -26,10 +29,13 @@ impl FlatLayoutFixture for PcoFixture { } fn expected_encodings(&self) -> Vec { - vec![Pco::ID] + vec![Pco.id()] } fn build(&self) -> VortexResult { + // NOTE: `FlatLayoutFixture::build` has a fixed trait signature without `ExecutionCtx`, so + // we construct a legacy ctx locally at this trait boundary. + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let irregular_i64: PrimitiveArray = (0..N as i64).map(|i| i * i + (i % 17) * 1000).collect(); let smooth_f64: PrimitiveArray = (0..N) @@ -69,14 +75,14 @@ impl FlatLayoutFixture for PcoFixture { "narrow_i16", ]), vec![ - Pco::from_primitive(&irregular_i64, 8, 0)?.into_array(), - Pco::from_primitive(&smooth_f64, 8, 0)?.into_array(), - Pco::from_primitive(&pattern_u32, 8, 0)?.into_array(), - Pco::from_primitive(&nullable_f32, 8, 0)?.into_array(), - Pco::from_primitive(&negative_i32, 8, 0)?.into_array(), - Pco::from_primitive(&constant_u16, 8, 0)?.into_array(), - Pco::from_primitive(&spike_outliers, 8, 0)?.into_array(), - Pco::from_primitive(&narrow_i16, 8, 0)?.into_array(), + Pco::from_primitive(irregular_i64.as_view(), 8, 0, &mut ctx)?.into_array(), + Pco::from_primitive(smooth_f64.as_view(), 8, 0, &mut ctx)?.into_array(), + Pco::from_primitive(pattern_u32.as_view(), 8, 0, &mut ctx)?.into_array(), + Pco::from_primitive(nullable_f32.as_view(), 8, 0, &mut ctx)?.into_array(), + Pco::from_primitive(negative_i32.as_view(), 8, 0, &mut ctx)?.into_array(), + Pco::from_primitive(constant_u16.as_view(), 8, 0, &mut ctx)?.into_array(), + Pco::from_primitive(spike_outliers.as_view(), 8, 0, &mut ctx)?.into_array(), + Pco::from_primitive(narrow_i16.as_view(), 8, 0, &mut ctx)?.into_array(), ], N, Validity::NonNullable, diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/rle.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/rle.rs index 46235ed1fa0..5c077386516 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/rle.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/rle.rs @@ -1,15 +1,19 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors +use vortex::VortexSessionDefault; use vortex::array::ArrayId; use vortex::array::ArrayRef; +use vortex::array::ArrayVTable; use vortex::array::IntoArray; +use vortex::array::VortexSessionExecute; use vortex::array::arrays::PrimitiveArray; use vortex::array::arrays::StructArray; use vortex::array::dtype::FieldNames; use vortex::array::validity::Validity; use vortex::encodings::fastlanes::RLE; use vortex::error::VortexResult; +use vortex_session::VortexSession; use super::N; use crate::fixtures::FlatLayoutFixture; @@ -26,10 +30,13 @@ impl FlatLayoutFixture for RleFixture { } fn expected_encodings(&self) -> Vec { - vec![RLE::ID] + vec![RLE.id()] } fn build(&self) -> VortexResult { + let session = VortexSession::default(); + let mut ctx = session.create_execution_ctx(); + let runs_i32: PrimitiveArray = (0..N as i32).map(|i| i / 64).collect(); let single_run: PrimitiveArray = std::iter::repeat_n(42u64, N).collect(); let nullable_runs = PrimitiveArray::from_option_iter( @@ -68,14 +75,14 @@ impl FlatLayoutFixture for RleFixture { "short_runs_u8", ]), vec![ - RLE::encode(&runs_i32)?.into_array(), - RLE::encode(&single_run)?.into_array(), - RLE::encode(&nullable_runs)?.into_array(), - RLE::encode(&alternating_singletons)?.into_array(), - RLE::encode(&exact_boundary_runs)?.into_array(), - RLE::encode(&giant_final_run)?.into_array(), - RLE::encode(&all_null_i32)?.into_array(), - RLE::encode(&short_runs_u8)?.into_array(), + RLE::encode(runs_i32.as_view(), &mut ctx)?.into_array(), + RLE::encode(single_run.as_view(), &mut ctx)?.into_array(), + RLE::encode(nullable_runs.as_view(), &mut ctx)?.into_array(), + RLE::encode(alternating_singletons.as_view(), &mut ctx)?.into_array(), + RLE::encode(exact_boundary_runs.as_view(), &mut ctx)?.into_array(), + RLE::encode(giant_final_run.as_view(), &mut ctx)?.into_array(), + RLE::encode(all_null_i32.as_view(), &mut ctx)?.into_array(), + RLE::encode(short_runs_u8.as_view(), &mut ctx)?.into_array(), ], N, Validity::NonNullable, diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/runend.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/runend.rs index e5594edcc7f..c14553ea11a 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/runend.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/runend.rs @@ -3,7 +3,10 @@ use vortex::array::ArrayId; use vortex::array::ArrayRef; +use vortex::array::ArrayVTable; use vortex::array::IntoArray; +use vortex::array::LEGACY_SESSION; +use vortex::array::VortexSessionExecute; use vortex::array::arrays::BoolArray; use vortex::array::arrays::PrimitiveArray; use vortex::array::arrays::StructArray; @@ -29,10 +32,11 @@ impl FlatLayoutFixture for RunEndFixture { } fn expected_encodings(&self) -> Vec { - vec![RunEnd::ID] + vec![RunEnd.id()] } fn build(&self) -> VortexResult { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let run_lengths = [1usize, 5, 10, 50, 100]; let mut values = Vec::with_capacity(N); let mut run_idx = 0i64; @@ -46,8 +50,8 @@ impl FlatLayoutFixture for RunEndFixture { rl_idx += 1; } let run_prim: PrimitiveArray = values.into_iter().collect(); - let (run_ends, run_values) = runend_encode(run_prim.as_view()); - let run_col = RunEnd::try_new(run_ends.into_array(), run_values)?; + let (run_ends, run_values) = runend_encode(run_prim.as_view(), &mut ctx); + let run_col = RunEnd::try_new(run_ends.into_array(), run_values, &mut ctx)?; let statuses = ["open", "closed", "pending", "cancelled"]; let mut status_values = Vec::new(); @@ -67,16 +71,17 @@ impl FlatLayoutFixture for RunEndFixture { let status_col = RunEnd::try_new( status_ends_prim.into_array(), VarBinArray::from_strs(status_values).into_array(), + &mut ctx, )?; let uniform_prim: PrimitiveArray = (0..N as i32).map(|i| i / 64).collect(); - let (uniform_ends, uniform_values) = runend_encode(uniform_prim.as_view()); - let uniform_col = RunEnd::try_new(uniform_ends.into_array(), uniform_values)?; + let (uniform_ends, uniform_values) = runend_encode(uniform_prim.as_view(), &mut ctx); + let uniform_col = RunEnd::try_new(uniform_ends.into_array(), uniform_values, &mut ctx)?; let bool_ends: PrimitiveArray = (1..=N / 32).map(|i| (i * 32) as u16).collect(); let bool_values = BoolArray::from_iter((0..bool_ends.len()).map(|i| i % 2 == 0)).into_array(); - let bool_runs = RunEnd::try_new(bool_ends.into_array(), bool_values)?; + let bool_runs = RunEnd::try_new(bool_ends.into_array(), bool_values, &mut ctx)?; let nullable_run_values = PrimitiveArray::from_option_iter([ Some(10i32), None, @@ -88,15 +93,20 @@ impl FlatLayoutFixture for RunEndFixture { let nullable_runs = RunEnd::try_new( PrimitiveArray::from_iter([16u16, 64, 128, 256, 512, N as u16]).into_array(), nullable_run_values.into_array(), + &mut ctx, )?; let single_run = RunEnd::try_new( PrimitiveArray::from_iter([N as u64]).into_array(), PrimitiveArray::from_iter([1234i64]).into_array(), + &mut ctx, )?; let singleton_values: PrimitiveArray = (0..N as i16).map(|i| i - 512).collect(); let singleton_ends: PrimitiveArray = (1..=N as u16).collect(); - let alternating_singletons = - RunEnd::try_new(singleton_ends.into_array(), singleton_values.into_array())?; + let alternating_singletons = RunEnd::try_new( + singleton_ends.into_array(), + singleton_values.into_array(), + &mut ctx, + )?; let arr = StructArray::try_new( FieldNames::from([ diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/sequence.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/sequence.rs index 61bd87252fb..9d9493f99c1 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/sequence.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/sequence.rs @@ -3,6 +3,7 @@ use vortex::array::ArrayId; use vortex::array::ArrayRef; +use vortex::array::ArrayVTable; use vortex::array::IntoArray; use vortex::array::arrays::StructArray; use vortex::array::dtype::FieldNames; @@ -26,7 +27,7 @@ impl FlatLayoutFixture for SequenceFixture { } fn expected_encodings(&self) -> Vec { - vec![Sequence::ID] + vec![Sequence.id()] } fn build(&self) -> VortexResult { diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/sparse.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/sparse.rs index eeff8c36c1b..951f9a50a23 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/sparse.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/sparse.rs @@ -3,7 +3,10 @@ use vortex::array::ArrayId; use vortex::array::ArrayRef; +use vortex::array::ArrayVTable; use vortex::array::IntoArray; +use vortex::array::LEGACY_SESSION; +use vortex::array::VortexSessionExecute; use vortex::array::arrays::BoolArray; use vortex::array::arrays::ConstantArray; use vortex::array::arrays::PrimitiveArray; @@ -31,10 +34,13 @@ impl FlatLayoutFixture for SparseFixture { } fn expected_encodings(&self) -> Vec { - vec![Sparse::ID] + vec![Sparse.id()] } fn build(&self) -> VortexResult { + // `FlatLayoutFixture::build` has a fixed trait signature that can't accept a ctx, + // so we instantiate one here. + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let sparse_i64_col = PrimitiveArray::from_option_iter( (0..N as i64).map(|i| (i % 50 == 0).then_some(i * 1000)), ); @@ -88,22 +94,28 @@ impl FlatLayoutFixture for SparseFixture { "mixed_null_and_values", ]), vec![ - Sparse::encode(&sparse_i64_col.into_array(), None)?, - Sparse::encode(&sparse_str_col.into_array(), None)?, - Sparse::encode(&sparse_bool_col.into_array(), None)?, - Sparse::encode(&sparse_f64.into_array(), None)?, - Sparse::encode(&sparse_boundary.into_array(), None)?, + Sparse::encode(&sparse_i64_col.into_array(), None, &mut ctx)?, + Sparse::encode(&sparse_str_col.into_array(), None, &mut ctx)?, + Sparse::encode(&sparse_bool_col.into_array(), None, &mut ctx)?, + Sparse::encode(&sparse_f64.into_array(), None, &mut ctx)?, + Sparse::encode(&sparse_boundary.into_array(), None, &mut ctx)?, Sparse::encode( &explicit_fill_values.into_array(), Some(Scalar::primitive(10i32, Nullability::Nullable)), + &mut ctx, )?, - Sparse::encode(&all_default, Some(Scalar::from(10i32)))?, - Sparse::encode(&clustered_edges.into_array(), None)?, + Sparse::encode(&all_default, Some(Scalar::from(10i32)), &mut ctx)?, + Sparse::encode(&clustered_edges.into_array(), None, &mut ctx)?, Sparse::encode( &almost_dense.into_array(), Some(Scalar::primitive(0i32, Nullability::Nullable)), + &mut ctx, + )?, + Sparse::encode( + &mixed_null_and_values.into_array(), + Some(mixed_null_fill), + &mut ctx, )?, - Sparse::encode(&mixed_null_and_values.into_array(), Some(mixed_null_fill))?, ], N, Validity::NonNullable, diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/zigzag.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/zigzag.rs index 2e1bb972f5d..16e553854a0 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/zigzag.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/zigzag.rs @@ -3,6 +3,7 @@ use vortex::array::ArrayId; use vortex::array::ArrayRef; +use vortex::array::ArrayVTable; use vortex::array::IntoArray; use vortex::array::arrays::PrimitiveArray; use vortex::array::arrays::StructArray; @@ -27,7 +28,7 @@ impl FlatLayoutFixture for ZigZagFixture { } fn expected_encodings(&self) -> Vec { - vec![ZigZag::ID] + vec![ZigZag.id()] } fn build(&self) -> VortexResult { @@ -80,17 +81,17 @@ impl FlatLayoutFixture for ZigZagFixture { "head_tail_nulls", ]), vec![ - zigzag_encode(alternating_i32)?.into_array(), - zigzag_encode(small_i64)?.into_array(), - zigzag_encode(deltas_i32)?.into_array(), - zigzag_encode(small_i16)?.into_array(), - zigzag_encode(small_i8)?.into_array(), - zigzag_encode(nullable_zigzag)?.into_array(), - zigzag_encode(extremes_i32)?.into_array(), - zigzag_encode(zero_heavy_outliers)?.into_array(), - zigzag_encode(repeated_negative)?.into_array(), - zigzag_encode(zero_crossing)?.into_array(), - zigzag_encode(head_tail_nulls)?.into_array(), + zigzag_encode(alternating_i32.as_view())?.into_array(), + zigzag_encode(small_i64.as_view())?.into_array(), + zigzag_encode(deltas_i32.as_view())?.into_array(), + zigzag_encode(small_i16.as_view())?.into_array(), + zigzag_encode(small_i8.as_view())?.into_array(), + zigzag_encode(nullable_zigzag.as_view())?.into_array(), + zigzag_encode(extremes_i32.as_view())?.into_array(), + zigzag_encode(zero_heavy_outliers.as_view())?.into_array(), + zigzag_encode(repeated_negative.as_view())?.into_array(), + zigzag_encode(zero_crossing.as_view())?.into_array(), + zigzag_encode(head_tail_nulls.as_view())?.into_array(), ], N, Validity::NonNullable, diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/zstd.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/zstd.rs index 655cbba8c06..aa55bb78eb9 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/zstd.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/encodings/zstd.rs @@ -3,7 +3,10 @@ use vortex::array::ArrayId; use vortex::array::ArrayRef; +use vortex::array::ArrayVTable; use vortex::array::IntoArray; +use vortex::array::LEGACY_SESSION; +use vortex::array::VortexSessionExecute; use vortex::array::arrays::PrimitiveArray; use vortex::array::arrays::StructArray; use vortex::array::arrays::VarBinViewArray; @@ -27,10 +30,11 @@ impl FlatLayoutFixture for ZstdFixture { } fn expected_encodings(&self) -> Vec { - vec![Zstd::ID] + vec![Zstd.id()] } fn build(&self) -> VortexResult { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let ints: PrimitiveArray = (0..N as i32).map(|i| i / 8).collect(); let floats: PrimitiveArray = (0..N) .map(|i| { @@ -86,14 +90,14 @@ impl FlatLayoutFixture for ZstdFixture { "pseudo_random", ]), vec![ - Zstd::from_primitive(&ints, 3, 128)?.into_array(), - Zstd::from_primitive(&floats, 3, 128)?.into_array(), - Zstd::from_primitive(&nullable_i64, 3, 128)?.into_array(), - Zstd::from_var_bin_view(&utf8, 3, 128)?.into_array(), - Zstd::from_var_bin_view(&nullable_utf8, 3, 128)?.into_array(), - Zstd::from_primitive(&all_zeros, 3, 128)?.into_array(), - Zstd::from_primitive(&all_null_i32, 3, 128)?.into_array(), - Zstd::from_primitive(&pseudo_random, 3, 128)?.into_array(), + Zstd::from_primitive(&ints, 3, 128, &mut ctx)?.into_array(), + Zstd::from_primitive(&floats, 3, 128, &mut ctx)?.into_array(), + Zstd::from_primitive(&nullable_i64, 3, 128, &mut ctx)?.into_array(), + Zstd::from_var_bin_view(&utf8, 3, 128, &mut ctx)?.into_array(), + Zstd::from_var_bin_view(&nullable_utf8, 3, 128, &mut ctx)?.into_array(), + Zstd::from_primitive(&all_zeros, 3, 128, &mut ctx)?.into_array(), + Zstd::from_primitive(&all_null_i32, 3, 128, &mut ctx)?.into_array(), + Zstd::from_primitive(&pseudo_random, 3, 128, &mut ctx)?.into_array(), ], N, Validity::NonNullable, diff --git a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/mod.rs b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/mod.rs index 3c3ffcb80d6..dd707779aea 100644 --- a/vortex-test/compat-gen/src/fixtures/arrays/synthetic/mod.rs +++ b/vortex-test/compat-gen/src/fixtures/arrays/synthetic/mod.rs @@ -2,7 +2,7 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors mod arrays; -#[allow(clippy::cast_possible_truncation)] +#[expect(clippy::cast_possible_truncation)] mod encodings; use crate::fixtures::FlatLayoutFixture; diff --git a/vortex-test/compat-gen/src/generate.rs b/vortex-test/compat-gen/src/generate.rs index 1393866c757..7490540f75e 100644 --- a/vortex-test/compat-gen/src/generate.rs +++ b/vortex-test/compat-gen/src/generate.rs @@ -3,6 +3,7 @@ use std::path::Path; +use base16ct::HexDisplay; use serde::Serialize; use sha2::Digest; use sha2::Sha256; @@ -50,7 +51,7 @@ pub fn write_fixtures(output_dir: &Path, exclude: &[String]) -> VortexResultrelease()` to drop the data allocated from the Rust side -#![allow(clippy::unwrap_used, clippy::expect_used)] +#![expect(clippy::unwrap_used, clippy::expect_used)] use std::mem; use std::sync::Arc; diff --git a/vortex-tui/Cargo.toml b/vortex-tui/Cargo.toml index ad4d0b052f2..33cd1958ba2 100644 --- a/vortex-tui/Cargo.toml +++ b/vortex-tui/Cargo.toml @@ -30,6 +30,7 @@ native = [ "vortex/tokio", "vortex/zstd", ] +unstable_encodings = ["vortex/unstable_encodings"] [lib] crate-type = ["cdylib", "rlib"] @@ -90,5 +91,13 @@ web-sys = { version = "0.3.81", features = [ "Window", ] } +[package.metadata.binstall] +pkg-url = "{ repo }/releases/download/{ version }/vx-{ target }{ archive-suffix }" +bin-dir = "{ bin }{ binary-ext }" +pkg-fmt = "tgz" + +[package.metadata.binstall.overrides.'cfg(target_os = "windows")'] +pkg-fmt = "zip" + [lints] workspace = true diff --git a/vortex-tui/src/browse/mod.rs b/vortex-tui/src/browse/mod.rs index fb3d279bfa0..a5746e4c61b 100644 --- a/vortex-tui/src/browse/mod.rs +++ b/vortex-tui/src/browse/mod.rs @@ -56,7 +56,6 @@ fn navigate_layout_down(app: &mut AppState, amount: usize) { /// Handle a key event in normal input mode. /// /// Returns [`HandleResult::Exit`] if the user pressed the quit key. -#[allow(clippy::cognitive_complexity)] pub(crate) fn handle_normal_mode(app: &mut AppState, event: InputEvent) -> HandleResult { // Check if we're in Query tab with SQL input focus - handle text input first #[cfg(feature = "native")] diff --git a/vortex-tui/src/browse/ui/layouts.rs b/vortex-tui/src/browse/ui/layouts.rs index a00c363610e..c3245446197 100644 --- a/vortex-tui/src/browse/ui/layouts.rs +++ b/vortex-tui/src/browse/ui/layouts.rs @@ -27,7 +27,8 @@ use ratatui::widgets::Table; use ratatui::widgets::Widget; use ratatui::widgets::Wrap; use vortex::array::ArrayRef; -use vortex::array::ToCanonical; +use vortex::array::VortexSessionExecute; +use vortex::array::arrays::StructArray; use vortex::array::arrays::struct_::StructArrayExt; use vortex::error::VortexExpect; use vortex::layout::layouts::flat::Flat; @@ -140,7 +141,11 @@ fn render_array(app: &AppState, area: Rect, buf: &mut Buffer, is_stats_table: bo if is_stats_table { // Render the stats table horizontally - let struct_array = array.to_struct(); + let mut ctx = app.session.create_execution_ctx(); + let struct_array = array + .clone() + .execute::(&mut ctx) + .vortex_expect("failed to canonicalize stats array to StructArray"); // add 1 for the chunk column let field_count = struct_array.struct_fields().nfields() + 1; let header = std::iter::once("chunk") @@ -159,17 +164,18 @@ fn render_array(app: &AppState, area: Rect, buf: &mut Buffer, is_stats_table: bo let field_arrays: Vec = struct_array.unmasked_fields().to_vec(); // TODO: trim the number of displayed rows and allow paging through column stats. - let rows = (0..array.len()).map(|chunk_id| { - std::iter::once(Cell::from(Text::from(format!("{chunk_id}")))) - .chain(field_arrays.iter().map(|arr| { - Cell::from(Text::from( - arr.scalar_at(chunk_id) - .vortex_expect("scalar_at failed") - .to_string(), - )) - })) - .collect::() - }); + let mut rows = Vec::with_capacity(array.len()); + for chunk_id in 0..array.len() { + let mut cells: Vec = Vec::with_capacity(field_count); + cells.push(Cell::from(Text::from(format!("{chunk_id}")))); + for arr in &field_arrays { + let scalar = arr + .execute_scalar(chunk_id, &mut ctx) + .vortex_expect("scalar_at failed"); + cells.push(Cell::from(Text::from(scalar.to_string()))); + } + rows.push(cells.into_iter().collect::()); + } Widget::render( Table::new(rows, (0..field_count).map(|_| Constraint::Min(6))).header(header), diff --git a/vortex-tui/src/browse/ui/query.rs b/vortex-tui/src/browse/ui/query.rs index 3c626b47207..709acf746b5 100644 --- a/vortex-tui/src/browse/ui/query.rs +++ b/vortex-tui/src/browse/ui/query.rs @@ -481,7 +481,7 @@ async fn get_row_count( { use arrow_array::Int64Array; if let Some(arr) = batch.column(0).as_any().downcast_ref::() { - #[allow(clippy::cast_sign_loss, clippy::cast_possible_truncation)] + #[expect(clippy::cast_sign_loss, clippy::cast_possible_truncation)] return Ok(arr.value(0) as usize); } } @@ -562,9 +562,9 @@ fn render_results_table(app: &mut AppState, area: Rect, buf: &mut Buffer) { // Show status in title let title = if app.query_state.running { "Results (running...)".to_string() - } else if let Some(ref error) = app.query_state.error { + } else if let Some(error) = &app.query_state.error { format!("Results (error: {})", truncate_str(error, 50)) - } else if let Some(ref _results) = app.query_state.results { + } else if let Some(_results) = &app.query_state.results { let total_rows = app.query_state.total_row_count.unwrap_or(0); let total_pages = app.query_state.total_pages(); format!( @@ -586,7 +586,7 @@ fn render_results_table(app: &mut AppState, area: Rect, buf: &mut Buffer) { let inner = block.inner(area); block.render(area, buf); - if let Some(ref error) = app.query_state.error { + if let Some(error) = &app.query_state.error { let error_text = Paragraph::new(error.as_str()) .style(Style::default().fg(Color::Red)) .wrap(ratatui::widgets::Wrap { trim: true }); @@ -594,7 +594,7 @@ fn render_results_table(app: &mut AppState, area: Rect, buf: &mut Buffer) { return; } - let Some(ref results) = app.query_state.results else { + let Some(results) = &app.query_state.results else { let help = Paragraph::new("Enter a SQL query above and press Enter to execute.\nThe table is available as 'data'.\n\nExample: SELECT * FROM data WHERE column > 10 LIMIT 100") .style(Style::default().fg(Color::Gray)); help.render(inner, buf); @@ -637,7 +637,7 @@ fn render_results_table(app: &mut AppState, area: Rect, buf: &mut Buffer) { let rows = get_all_rows(results, &app.query_state); // Calculate column widths - #[allow(clippy::cast_possible_truncation)] + #[expect(clippy::cast_possible_truncation)] let widths: Vec = results .column_names .iter() diff --git a/vortex-tui/src/datafusion_helper.rs b/vortex-tui/src/datafusion_helper.rs index 9205ca0bede..d4ca3714009 100644 --- a/vortex-tui/src/datafusion_helper.rs +++ b/vortex-tui/src/datafusion_helper.rs @@ -76,7 +76,7 @@ pub async fn create_vortex_context( /// /// Panics if the array type doesn't match the expected Arrow array type during downcast. /// This should not happen for well-formed Arrow arrays. -#[allow(clippy::unwrap_used)] +#[expect(clippy::unwrap_used)] pub fn arrow_value_to_json(array: &dyn ArrowArray, idx: usize) -> serde_json::Value { use arrow_array::*; use arrow_schema::DataType; diff --git a/vortex-tui/src/inspect.rs b/vortex-tui/src/inspect.rs index bc2e920aea4..5543716fb2b 100644 --- a/vortex-tui/src/inspect.rs +++ b/vortex-tui/src/inspect.rs @@ -509,13 +509,13 @@ impl SegmentInfo { impl PostscriptInfo { fn display(&self) { println!("\n=== Postscript ==="); - if let Some(ref dtype) = self.dtype { + if let Some(dtype) = &self.dtype { dtype.display("DType"); } else { println!(" DType: "); } self.layout.display("Layout"); - if let Some(ref stats) = self.statistics { + if let Some(stats) = &self.statistics { stats.display("Statistics"); } else { println!(" Statistics: "); diff --git a/vortex-utils/Cargo.toml b/vortex-utils/Cargo.toml index 0a1dd1ad0c8..66b2576f98d 100644 --- a/vortex-utils/Cargo.toml +++ b/vortex-utils/Cargo.toml @@ -24,4 +24,3 @@ workspace = true [features] dyn-traits = [] -_test-harness = ["dashmap", "parking_lot"] diff --git a/vortex-utils/public-api.lock b/vortex-utils/public-api.lock index 185029d5627..9247fa547e8 100644 --- a/vortex-utils/public-api.lock +++ b/vortex-utils/public-api.lock @@ -38,11 +38,11 @@ pub struct vortex_utils::aliases::StringEscape<'a>(pub &'a str) impl core::fmt::Display for vortex_utils::aliases::StringEscape<'_> -pub fn vortex_utils::aliases::StringEscape<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_utils::aliases::StringEscape<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl<'a> core::fmt::Debug for vortex_utils::aliases::StringEscape<'a> -pub fn vortex_utils::aliases::StringEscape<'a>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_utils::aliases::StringEscape<'a>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub mod vortex_utils::debug_with @@ -50,36 +50,40 @@ pub struct vortex_utils::debug_with::DebugWith where F: core::ops::function:: impl core::fmt::Debug for vortex_utils::debug_with::DebugWith where F: core::ops::function::Fn(&mut core::fmt::Formatter<'_>) -> core::fmt::Result -pub fn vortex_utils::debug_with::DebugWith::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_utils::debug_with::DebugWith::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub mod vortex_utils::dyn_traits pub trait vortex_utils::dyn_traits::DynEq: core::any::Any + vortex_utils::dyn_traits::private::SealedEq -pub fn vortex_utils::dyn_traits::DynEq::dyn_eq(&self, other: &dyn core::any::Any) -> bool +pub fn vortex_utils::dyn_traits::DynEq::dyn_eq(&self, &dyn core::any::Any) -> bool impl vortex_utils::dyn_traits::DynEq for T -pub fn T::dyn_eq(&self, other: &dyn core::any::Any) -> bool +pub fn T::dyn_eq(&self, &dyn core::any::Any) -> bool pub trait vortex_utils::dyn_traits::DynHash: vortex_utils::dyn_traits::private::SealedHash -pub fn vortex_utils::dyn_traits::DynHash::dyn_hash(&self, state: &mut dyn core::hash::Hasher) +pub fn vortex_utils::dyn_traits::DynHash::dyn_hash(&self, &mut dyn core::hash::Hasher) impl vortex_utils::dyn_traits::DynHash for T -pub fn T::dyn_hash(&self, state: &mut dyn core::hash::Hasher) +pub fn T::dyn_hash(&self, &mut dyn core::hash::Hasher) pub mod vortex_utils::iter pub trait vortex_utils::iter::ReduceBalancedIterExt: core::iter::traits::iterator::Iterator -pub fn vortex_utils::iter::ReduceBalancedIterExt::reduce_balanced(self, combine: F) -> core::option::Option where Self::Item: core::clone::Clone, F: core::ops::function::Fn(Self::Item, Self::Item) -> Self::Item +pub fn vortex_utils::iter::ReduceBalancedIterExt::reduce_balanced(self, F) -> core::option::Option where Self::Item: core::clone::Clone, F: core::ops::function::Fn(Self::Item, Self::Item) -> Self::Item -pub fn vortex_utils::iter::ReduceBalancedIterExt::try_reduce_balanced(self, combine: F) -> core::result::Result, E> where Self::Item: core::clone::Clone, F: core::ops::function::Fn(Self::Item, Self::Item) -> core::result::Result +pub fn vortex_utils::iter::ReduceBalancedIterExt::try_reduce_balanced(self, F) -> core::result::Result, E> where Self::Item: core::clone::Clone, F: core::ops::function::Fn(Self::Item, Self::Item) -> core::result::Result impl vortex_utils::iter::ReduceBalancedIterExt for I -pub fn I::reduce_balanced(self, combine: F) -> core::option::Option where Self::Item: core::clone::Clone, F: core::ops::function::Fn(Self::Item, Self::Item) -> Self::Item +pub fn I::reduce_balanced(self, F) -> core::option::Option where Self::Item: core::clone::Clone, F: core::ops::function::Fn(Self::Item, Self::Item) -> Self::Item -pub fn I::try_reduce_balanced(self, combine: F) -> core::result::Result, E> where Self::Item: core::clone::Clone, F: core::ops::function::Fn(Self::Item, Self::Item) -> core::result::Result +pub fn I::try_reduce_balanced(self, F) -> core::result::Result, E> where Self::Item: core::clone::Clone, F: core::ops::function::Fn(Self::Item, Self::Item) -> core::result::Result + +pub mod vortex_utils::parallelism + +pub fn vortex_utils::parallelism::get_available_parallelism() -> core::option::Option diff --git a/vortex-utils/src/env.rs b/vortex-utils/src/env.rs deleted file mode 100644 index 19e78e1f12b..00000000000 --- a/vortex-utils/src/env.rs +++ /dev/null @@ -1,197 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -//! Environment variable utilities for testing. - -use dashmap::DashMap; -use parking_lot::Mutex; -use vortex_error::vortex_panic; - -/// Global registry of locks per environment variable key. -/// -/// Each mutex is lazily created and leaked to get a 'static lifetime. -/// This is acceptable for test utilities that live for the process duration. -static ENV_LOCKS: std::sync::LazyLock>> = - std::sync::LazyLock::new(DashMap::new); - -/// Get or create a static mutex for the given key. -fn get_or_create_lock(key: &'static str) -> &'static Mutex<()> { - *ENV_LOCKS - .entry(key) - .or_insert_with(|| Box::leak(Box::new(Mutex::new(())))) -} - -/// RAII guard to set/remove an environment variable for the duration of a scope. -/// -/// Removes the variable when dropped, ensuring test isolation. -/// -/// This guard holds a mutex lock for the specific environment variable key, -/// ensuring that only one guard can exist for a given key at a time. This -/// prevents tests from accidentally having overlapping guards for the same -/// env var. -/// -/// # Example -/// -/// ``` -/// use vortex_utils::env::EnvVarGuard; -/// -/// // Set an env var for the duration of this scope -/// let _guard = EnvVarGuard::set("MY_TEST_VAR", "1"); -/// assert_eq!(std::env::var("MY_TEST_VAR").ok(), Some("1".to_string())); -/// -/// // Or remove an env var -/// let _guard2 = EnvVarGuard::remove("OTHER_VAR"); -/// assert!(std::env::var("OTHER_VAR").is_err()); -/// ``` -/// -/// # Panics -/// -/// Panics if a guard already exists for the same environment variable key -/// (detected via mutex lock contention). -/// -/// # Safety -/// -/// Environment variable modification is inherently unsafe in multi-threaded contexts. -/// This guard is intended for use in tests that are run serially or where env var -/// races are acceptable. The per-key locking ensures that the same env var isn't -/// modified concurrently by multiple guards. -pub struct EnvVarGuard { - key: &'static str, - /// We store this to ensure the mutex stays locked for our lifetime. - /// The () is just a dummy value - we only care about the lock. - #[expect(dead_code)] - lock_guard: parking_lot::MutexGuard<'static, ()>, -} - -/// Timeout for waiting on an env var lock. -const LOCK_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10); - -impl EnvVarGuard { - /// Acquire the lock for this key, waiting up to 10 seconds. - /// - /// If another guard holds the lock, this will wait for it to be released. - /// If the lock isn't released within 10 seconds, this panics to avoid deadlocks. - fn acquire_lock(key: &'static str) -> parking_lot::MutexGuard<'static, ()> { - let mutex = get_or_create_lock(key); - match mutex.try_lock_for(LOCK_TIMEOUT) { - Some(guard) => guard, - None => vortex_panic!( - "EnvVarGuard: timed out after {LOCK_TIMEOUT:?} waiting for environment variable '{key}'. \ - This likely indicates a deadlock - ensure guards for the same key are properly scoped \ - taken in lexicographical order and dropped before acquiring a new one." - ), - } - } - - /// Set an environment variable for the duration of this guard's lifetime. - pub fn set(key: &'static str, value: &str) -> Self { - let lock_guard = Self::acquire_lock(key); - - // SAFETY: We hold an exclusive lock for this key. - unsafe { - std::env::set_var(key, value); - } - - Self { key, lock_guard } - } - - /// Remove an environment variable for the duration of this guard's lifetime. - pub fn remove(key: &'static str) -> Self { - let lock_guard = Self::acquire_lock(key); - - // SAFETY: We hold an exclusive lock for this key. - unsafe { - std::env::remove_var(key); - } - - Self { key, lock_guard } - } -} - -impl Drop for EnvVarGuard { - fn drop(&mut self) { - // SAFETY: We hold an exclusive lock for this key. - unsafe { - std::env::remove_var(self.key); - } - } -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn test_set_and_remove() { - let key = "VORTEX_TEST_ENV_VAR_SET"; - - // Initially not set - assert!(std::env::var(key).is_err()); - - { - let _guard = EnvVarGuard::set(key, "test_value"); - assert_eq!(std::env::var(key).unwrap(), "test_value"); - } - - // After guard drops, var is removed - assert!(std::env::var(key).is_err()); - } - - #[test] - fn test_remove() { - let key = "VORTEX_TEST_ENV_VAR_REMOVE"; - - // Set it first - unsafe { - std::env::set_var(key, "initial"); - } - - { - let _guard = EnvVarGuard::remove(key); - assert!(std::env::var(key).is_err()); - } - - // After guard drops, var is still removed - assert!(std::env::var(key).is_err()); - } - - /// Test that a second guard waits for the first to be released. - #[test] - fn test_second_guard_waits_for_first() { - use std::sync::Arc; - use std::sync::atomic::AtomicBool; - use std::sync::atomic::Ordering; - use std::thread; - use std::time::Duration; - - let key = "VORTEX_TEST_ENV_VAR_WAIT"; - let second_acquired = Arc::new(AtomicBool::new(false)); - let second_acquired_clone = Arc::clone(&second_acquired); - - // First guard in main thread - let _guard1 = EnvVarGuard::set(key, "first"); - assert_eq!(std::env::var(key).unwrap(), "first"); - - // Spawn thread that will wait for the lock - let handle = thread::spawn(move || { - let _guard2 = EnvVarGuard::set(key, "second"); - second_acquired_clone.store(true, Ordering::SeqCst); - assert_eq!(std::env::var(key).unwrap(), "second"); - }); - - // Give the thread time to start waiting - thread::sleep(Duration::from_millis(50)); - - // Second guard should NOT have acquired yet (still waiting) - assert!(!second_acquired.load(Ordering::SeqCst)); - - // Drop the first guard - this should allow the second to proceed - drop(_guard1); - - // Wait for second thread to complete - handle.join().unwrap(); - - // Now the second guard should have acquired and set the value - assert!(second_acquired.load(Ordering::SeqCst)); - } -} diff --git a/vortex-utils/src/lib.rs b/vortex-utils/src/lib.rs index ff1610ddead..31decd406ff 100644 --- a/vortex-utils/src/lib.rs +++ b/vortex-utils/src/lib.rs @@ -9,6 +9,5 @@ pub mod aliases; pub mod debug_with; #[cfg(feature = "dyn-traits")] pub mod dyn_traits; -#[cfg(feature = "_test-harness")] -pub mod env; pub mod iter; +pub mod parallelism; diff --git a/vortex-utils/src/parallelism.rs b/vortex-utils/src/parallelism.rs new file mode 100644 index 00000000000..14b251e8bfb --- /dev/null +++ b/vortex-utils/src/parallelism.rs @@ -0,0 +1,19 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Useful utilities for discovering the desired level of parallelism + +use std::sync::LazyLock; + +/// Estimates the degree of parallelism the program should use, caching the result after the first call. +/// +/// This is currently implemented using [`std::thread::available_parallelism`], but might change in the future. +/// +/// Returns `None` if the underlying functions fails. +pub fn get_available_parallelism() -> Option { + #[allow(clippy::disallowed_methods)] + static PARALLELISM: LazyLock> = + LazyLock::new(|| std::thread::available_parallelism().ok().map(|n| n.get())); + + *PARALLELISM +} diff --git a/vortex-web/crate/Cargo.toml b/vortex-web/crate/Cargo.toml index 25b16583380..1b65fe832eb 100644 --- a/vortex-web/crate/Cargo.toml +++ b/vortex-web/crate/Cargo.toml @@ -2,7 +2,7 @@ name = "vortex-web-wasm" version = "0.1.0" edition = "2024" -rust-version = "1.90" +rust-version = "1.91.0" license = "Apache-2.0" description = "WASM bindings for the Vortex web explorer" publish = false diff --git a/vortex-web/eslint.config.ts b/vortex-web/eslint.config.ts index ecc2fe28df0..8c6a8e00920 100644 --- a/vortex-web/eslint.config.ts +++ b/vortex-web/eslint.config.ts @@ -21,7 +21,8 @@ export default tseslint.config({ ignores: ["dist", "storybook-static"] }, { "react-refresh": reactRefresh, }, rules: { - ...reactHooks.configs.recommended.rules, + "react-hooks/rules-of-hooks": "error", + "react-hooks/exhaustive-deps": "warn", "react-refresh/only-export-components": [ "warn", { allowConstantExport: true }, diff --git a/vortex-web/package-lock.json b/vortex-web/package-lock.json index 1fd57153b0f..fb211e202b2 100644 --- a/vortex-web/package-lock.json +++ b/vortex-web/package-lock.json @@ -16,26 +16,26 @@ "react-dom": "^19.1.0" }, "devDependencies": { - "@eslint/js": "^9.25.0", - "@storybook/addon-docs": "^10.3.3", - "@storybook/react-vite": "^10.3.3", - "@tailwindcss/vite": "^4.1.4", + "@eslint/js": "^10.0.1", + "@storybook/addon-docs": "^10.3.5", + "@storybook/react-vite": "^10.3.5", + "@tailwindcss/vite": "^4.2.4", "@types/d3-hierarchy": "^3.1.7", - "@types/react": "^19.1.2", - "@types/react-dom": "^19.1.2", - "@vitejs/plugin-react": "^4.4.1", - "eslint": "^9.25.0", + "@types/react": "^19.2.14", + "@types/react-dom": "^19.2.3", + "@vitejs/plugin-react": "^6.0.1", + "eslint": "^10.2.1", "eslint-config-prettier": "^10.1.8", - "eslint-plugin-react-hooks": "^5.2.0", - "eslint-plugin-react-refresh": "^0.4.20", - "eslint-plugin-storybook": "^10.3.3", - "globals": "^16.0.0", - "prettier": "^3.8.1", - "storybook": "^10.3.3", - "tailwindcss": "^4.1.4", - "typescript": "~5.7.0", - "typescript-eslint": "^8.30.1", - "vite": "^6.3.2" + "eslint-plugin-react-hooks": "^7.1.1", + "eslint-plugin-react-refresh": "^0.5.2", + "eslint-plugin-storybook": "^10.3.5", + "globals": "^17.5.0", + "prettier": "^3.8.3", + "storybook": "^10.3.5", + "tailwindcss": "^4.2.4", + "typescript": "~6.0.0", + "typescript-eslint": "^8.59.0", + "vite": "^8.0.10" } }, "node_modules/@adobe/css-tools": { @@ -177,16 +177,6 @@ "@babel/core": "^7.0.0" } }, - "node_modules/@babel/helper-plugin-utils": { - "version": "7.28.6", - "resolved": "https://registry.npmjs.org/@babel/helper-plugin-utils/-/helper-plugin-utils-7.28.6.tgz", - "integrity": "sha512-S9gzZ/bz83GRysI7gAD4wPT/AI3uCnY+9xn+Mx/KPs2JwHJIz1W8PZkg2cqyt3RNOBM8ejcXhV6y8Og7ly/Dug==", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=6.9.0" - } - }, "node_modules/@babel/helper-string-parser": { "version": "7.27.1", "resolved": "https://registry.npmjs.org/@babel/helper-string-parser/-/helper-string-parser-7.27.1.tgz", @@ -247,38 +237,6 @@ "node": ">=6.0.0" } }, - "node_modules/@babel/plugin-transform-react-jsx-self": { - "version": "7.27.1", - "resolved": "https://registry.npmjs.org/@babel/plugin-transform-react-jsx-self/-/plugin-transform-react-jsx-self-7.27.1.tgz", - "integrity": "sha512-6UzkCs+ejGdZ5mFFC/OCUrv028ab2fp1znZmCZjAOBKiBK2jXD1O+BPSfX8X2qjJ75fZBMSnQn3Rq2mrBJK2mw==", - "dev": true, - "license": "MIT", - "dependencies": { - "@babel/helper-plugin-utils": "^7.27.1" - }, - "engines": { - "node": ">=6.9.0" - }, - "peerDependencies": { - "@babel/core": "^7.0.0-0" - } - }, - "node_modules/@babel/plugin-transform-react-jsx-source": { - "version": "7.27.1", - "resolved": "https://registry.npmjs.org/@babel/plugin-transform-react-jsx-source/-/plugin-transform-react-jsx-source-7.27.1.tgz", - "integrity": "sha512-zbwoTsBruTeKB9hSq73ha66iFeJHuaFkUbwvqElnygoNbj/jHRsSeokowZFN3CZ64IvEqcmmkVe89OPXc7ldAw==", - "dev": true, - "license": "MIT", - "dependencies": { - "@babel/helper-plugin-utils": "^7.27.1" - }, - "engines": { - "node": ">=6.9.0" - }, - "peerDependencies": { - "@babel/core": "^7.0.0-0" - } - }, "node_modules/@babel/runtime": { "version": "7.29.2", "resolved": "https://registry.npmjs.org/@babel/runtime/-/runtime-7.29.2.tgz", @@ -338,10 +296,44 @@ "node": ">=6.9.0" } }, + "node_modules/@emnapi/core": { + "version": "1.10.0", + "resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.10.0.tgz", + "integrity": "sha512-yq6OkJ4p82CAfPl0u9mQebQHKPJkY7WrIuk205cTYnYe+k2Z8YBh11FrbRG/H6ihirqcacOgl2BIO8oyMQLeXw==", + "dev": true, + "license": "MIT", + "optional": true, + "dependencies": { + "@emnapi/wasi-threads": "1.2.1", + "tslib": "^2.4.0" + } + }, + "node_modules/@emnapi/runtime": { + "version": "1.10.0", + "resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.10.0.tgz", + "integrity": "sha512-ewvYlk86xUoGI0zQRNq/mC+16R1QeDlKQy21Ki3oSYXNgLb45GV1P6A0M+/s6nyCuNDqe5VpaY84BzXGwVbwFA==", + "dev": true, + "license": "MIT", + "optional": true, + "dependencies": { + "tslib": "^2.4.0" + } + }, + "node_modules/@emnapi/wasi-threads": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/@emnapi/wasi-threads/-/wasi-threads-1.2.1.tgz", + "integrity": "sha512-uTII7OYF+/Mes/MrcIOYp5yOtSMLBWSIoLPpcgwipoiKbli6k322tcoFsxoIIxPDqW01SQGAgko4EzZi2BNv2w==", + "dev": true, + "license": "MIT", + "optional": true, + "dependencies": { + "tslib": "^2.4.0" + } + }, "node_modules/@esbuild/aix-ppc64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.25.12.tgz", - "integrity": "sha512-Hhmwd6CInZ3dwpuGTF8fJG6yoWmsToE+vYgD4nytZVxcu1ulHpUQRAB1UJ8+N1Am3Mz4+xOByoQoSZf4D+CpkA==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.27.7.tgz", + "integrity": "sha512-EKX3Qwmhz1eMdEJokhALr0YiD0lhQNwDqkPYyPhiSwKrh7/4KRjQc04sZ8db+5DVVnZ1LmbNDI1uAMPEUBnQPg==", "cpu": [ "ppc64" ], @@ -356,9 +348,9 @@ } }, "node_modules/@esbuild/android-arm": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.25.12.tgz", - "integrity": "sha512-VJ+sKvNA/GE7Ccacc9Cha7bpS8nyzVv0jdVgwNDaR4gDMC/2TTRc33Ip8qrNYUcpkOHUT5OZ0bUcNNVZQ9RLlg==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.27.7.tgz", + "integrity": "sha512-jbPXvB4Yj2yBV7HUfE2KHe4GJX51QplCN1pGbYjvsyCZbQmies29EoJbkEc+vYuU5o45AfQn37vZlyXy4YJ8RQ==", "cpu": [ "arm" ], @@ -373,9 +365,9 @@ } }, "node_modules/@esbuild/android-arm64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.25.12.tgz", - "integrity": "sha512-6AAmLG7zwD1Z159jCKPvAxZd4y/VTO0VkprYy+3N2FtJ8+BQWFXU+OxARIwA46c5tdD9SsKGZ/1ocqBS/gAKHg==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.27.7.tgz", + "integrity": "sha512-62dPZHpIXzvChfvfLJow3q5dDtiNMkwiRzPylSCfriLvZeq0a1bWChrGx/BbUbPwOrsWKMn8idSllklzBy+dgQ==", "cpu": [ "arm64" ], @@ -390,9 +382,9 @@ } }, "node_modules/@esbuild/android-x64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.25.12.tgz", - "integrity": "sha512-5jbb+2hhDHx5phYR2By8GTWEzn6I9UqR11Kwf22iKbNpYrsmRB18aX/9ivc5cabcUiAT/wM+YIZ6SG9QO6a8kg==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.27.7.tgz", + "integrity": "sha512-x5VpMODneVDb70PYV2VQOmIUUiBtY3D3mPBG8NxVk5CogneYhkR7MmM3yR/uMdITLrC1ml/NV1rj4bMJuy9MCg==", "cpu": [ "x64" ], @@ -407,9 +399,9 @@ } }, "node_modules/@esbuild/darwin-arm64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.25.12.tgz", - "integrity": "sha512-N3zl+lxHCifgIlcMUP5016ESkeQjLj/959RxxNYIthIg+CQHInujFuXeWbWMgnTo4cp5XVHqFPmpyu9J65C1Yg==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.27.7.tgz", + "integrity": "sha512-5lckdqeuBPlKUwvoCXIgI2D9/ABmPq3Rdp7IfL70393YgaASt7tbju3Ac+ePVi3KDH6N2RqePfHnXkaDtY9fkw==", "cpu": [ "arm64" ], @@ -424,9 +416,9 @@ } }, "node_modules/@esbuild/darwin-x64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.25.12.tgz", - "integrity": "sha512-HQ9ka4Kx21qHXwtlTUVbKJOAnmG1ipXhdWTmNXiPzPfWKpXqASVcWdnf2bnL73wgjNrFXAa3yYvBSd9pzfEIpA==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.27.7.tgz", + "integrity": "sha512-rYnXrKcXuT7Z+WL5K980jVFdvVKhCHhUwid+dDYQpH+qu+TefcomiMAJpIiC2EM3Rjtq0sO3StMV/+3w3MyyqQ==", "cpu": [ "x64" ], @@ -441,9 +433,9 @@ } }, "node_modules/@esbuild/freebsd-arm64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.25.12.tgz", - "integrity": "sha512-gA0Bx759+7Jve03K1S0vkOu5Lg/85dou3EseOGUes8flVOGxbhDDh/iZaoek11Y8mtyKPGF3vP8XhnkDEAmzeg==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.27.7.tgz", + "integrity": "sha512-B48PqeCsEgOtzME2GbNM2roU29AMTuOIN91dsMO30t+Ydis3z/3Ngoj5hhnsOSSwNzS+6JppqWsuhTp6E82l2w==", "cpu": [ "arm64" ], @@ -458,9 +450,9 @@ } }, "node_modules/@esbuild/freebsd-x64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.25.12.tgz", - "integrity": "sha512-TGbO26Yw2xsHzxtbVFGEXBFH0FRAP7gtcPE7P5yP7wGy7cXK2oO7RyOhL5NLiqTlBh47XhmIUXuGciXEqYFfBQ==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.27.7.tgz", + "integrity": "sha512-jOBDK5XEjA4m5IJK3bpAQF9/Lelu/Z9ZcdhTRLf4cajlB+8VEhFFRjWgfy3M1O4rO2GQ/b2dLwCUGpiF/eATNQ==", "cpu": [ "x64" ], @@ -475,9 +467,9 @@ } }, "node_modules/@esbuild/linux-arm": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.25.12.tgz", - "integrity": "sha512-lPDGyC1JPDou8kGcywY0YILzWlhhnRjdof3UlcoqYmS9El818LLfJJc3PXXgZHrHCAKs/Z2SeZtDJr5MrkxtOw==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.27.7.tgz", + "integrity": "sha512-RkT/YXYBTSULo3+af8Ib0ykH8u2MBh57o7q/DAs3lTJlyVQkgQvlrPTnjIzzRPQyavxtPtfg0EopvDyIt0j1rA==", "cpu": [ "arm" ], @@ -492,9 +484,9 @@ } }, "node_modules/@esbuild/linux-arm64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.25.12.tgz", - "integrity": "sha512-8bwX7a8FghIgrupcxb4aUmYDLp8pX06rGh5HqDT7bB+8Rdells6mHvrFHHW2JAOPZUbnjUpKTLg6ECyzvas2AQ==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.27.7.tgz", + "integrity": "sha512-RZPHBoxXuNnPQO9rvjh5jdkRmVizktkT7TCDkDmQ0W2SwHInKCAV95GRuvdSvA7w4VMwfCjUiPwDi0ZO6Nfe9A==", "cpu": [ "arm64" ], @@ -509,9 +501,9 @@ } }, "node_modules/@esbuild/linux-ia32": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.25.12.tgz", - "integrity": "sha512-0y9KrdVnbMM2/vG8KfU0byhUN+EFCny9+8g202gYqSSVMonbsCfLjUO+rCci7pM0WBEtz+oK/PIwHkzxkyharA==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.27.7.tgz", + "integrity": "sha512-GA48aKNkyQDbd3KtkplYWT102C5sn/EZTY4XROkxONgruHPU72l+gW+FfF8tf2cFjeHaRbWpOYa/uRBz/Xq1Pg==", "cpu": [ "ia32" ], @@ -526,9 +518,9 @@ } }, "node_modules/@esbuild/linux-loong64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.25.12.tgz", - "integrity": "sha512-h///Lr5a9rib/v1GGqXVGzjL4TMvVTv+s1DPoxQdz7l/AYv6LDSxdIwzxkrPW438oUXiDtwM10o9PmwS/6Z0Ng==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.27.7.tgz", + "integrity": "sha512-a4POruNM2oWsD4WKvBSEKGIiWQF8fZOAsycHOt6JBpZ+JN2n2JH9WAv56SOyu9X5IqAjqSIPTaJkqN8F7XOQ5Q==", "cpu": [ "loong64" ], @@ -543,9 +535,9 @@ } }, "node_modules/@esbuild/linux-mips64el": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.25.12.tgz", - "integrity": "sha512-iyRrM1Pzy9GFMDLsXn1iHUm18nhKnNMWscjmp4+hpafcZjrr2WbT//d20xaGljXDBYHqRcl8HnxbX6uaA/eGVw==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.27.7.tgz", + "integrity": "sha512-KabT5I6StirGfIz0FMgl1I+R1H73Gp0ofL9A3nG3i/cYFJzKHhouBV5VWK1CSgKvVaG4q1RNpCTR2LuTVB3fIw==", "cpu": [ "mips64el" ], @@ -560,9 +552,9 @@ } }, "node_modules/@esbuild/linux-ppc64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.25.12.tgz", - "integrity": "sha512-9meM/lRXxMi5PSUqEXRCtVjEZBGwB7P/D4yT8UG/mwIdze2aV4Vo6U5gD3+RsoHXKkHCfSxZKzmDssVlRj1QQA==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.27.7.tgz", + "integrity": "sha512-gRsL4x6wsGHGRqhtI+ifpN/vpOFTQtnbsupUF5R5YTAg+y/lKelYR1hXbnBdzDjGbMYjVJLJTd2OFmMewAgwlQ==", "cpu": [ "ppc64" ], @@ -577,9 +569,9 @@ } }, "node_modules/@esbuild/linux-riscv64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.25.12.tgz", - "integrity": "sha512-Zr7KR4hgKUpWAwb1f3o5ygT04MzqVrGEGXGLnj15YQDJErYu/BGg+wmFlIDOdJp0PmB0lLvxFIOXZgFRrdjR0w==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.27.7.tgz", + "integrity": "sha512-hL25LbxO1QOngGzu2U5xeXtxXcW+/GvMN3ejANqXkxZ/opySAZMrc+9LY/WyjAan41unrR3YrmtTsUpwT66InQ==", "cpu": [ "riscv64" ], @@ -594,9 +586,9 @@ } }, "node_modules/@esbuild/linux-s390x": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.25.12.tgz", - "integrity": "sha512-MsKncOcgTNvdtiISc/jZs/Zf8d0cl/t3gYWX8J9ubBnVOwlk65UIEEvgBORTiljloIWnBzLs4qhzPkJcitIzIg==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.27.7.tgz", + "integrity": "sha512-2k8go8Ycu1Kb46vEelhu1vqEP+UeRVj2zY1pSuPdgvbd5ykAw82Lrro28vXUrRmzEsUV0NzCf54yARIK8r0fdw==", "cpu": [ "s390x" ], @@ -611,9 +603,9 @@ } }, "node_modules/@esbuild/linux-x64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.25.12.tgz", - "integrity": "sha512-uqZMTLr/zR/ed4jIGnwSLkaHmPjOjJvnm6TVVitAa08SLS9Z0VM8wIRx7gWbJB5/J54YuIMInDquWyYvQLZkgw==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.27.7.tgz", + "integrity": "sha512-hzznmADPt+OmsYzw1EE33ccA+HPdIqiCRq7cQeL1Jlq2gb1+OyWBkMCrYGBJ+sxVzve2ZJEVeePbLM2iEIZSxA==", "cpu": [ "x64" ], @@ -628,9 +620,9 @@ } }, "node_modules/@esbuild/netbsd-arm64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/netbsd-arm64/-/netbsd-arm64-0.25.12.tgz", - "integrity": "sha512-xXwcTq4GhRM7J9A8Gv5boanHhRa/Q9KLVmcyXHCTaM4wKfIpWkdXiMog/KsnxzJ0A1+nD+zoecuzqPmCRyBGjg==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/netbsd-arm64/-/netbsd-arm64-0.27.7.tgz", + "integrity": "sha512-b6pqtrQdigZBwZxAn1UpazEisvwaIDvdbMbmrly7cDTMFnw/+3lVxxCTGOrkPVnsYIosJJXAsILG9XcQS+Yu6w==", "cpu": [ "arm64" ], @@ -645,9 +637,9 @@ } }, "node_modules/@esbuild/netbsd-x64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.25.12.tgz", - "integrity": "sha512-Ld5pTlzPy3YwGec4OuHh1aCVCRvOXdH8DgRjfDy/oumVovmuSzWfnSJg+VtakB9Cm0gxNO9BzWkj6mtO1FMXkQ==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.27.7.tgz", + "integrity": "sha512-OfatkLojr6U+WN5EDYuoQhtM+1xco+/6FSzJJnuWiUw5eVcicbyK3dq5EeV/QHT1uy6GoDhGbFpprUiHUYggrw==", "cpu": [ "x64" ], @@ -662,9 +654,9 @@ } }, "node_modules/@esbuild/openbsd-arm64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/openbsd-arm64/-/openbsd-arm64-0.25.12.tgz", - "integrity": "sha512-fF96T6KsBo/pkQI950FARU9apGNTSlZGsv1jZBAlcLL1MLjLNIWPBkj5NlSz8aAzYKg+eNqknrUJ24QBybeR5A==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/openbsd-arm64/-/openbsd-arm64-0.27.7.tgz", + "integrity": "sha512-AFuojMQTxAz75Fo8idVcqoQWEHIXFRbOc1TrVcFSgCZtQfSdc1RXgB3tjOn/krRHENUB4j00bfGjyl2mJrU37A==", "cpu": [ "arm64" ], @@ -679,9 +671,9 @@ } }, "node_modules/@esbuild/openbsd-x64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.25.12.tgz", - "integrity": "sha512-MZyXUkZHjQxUvzK7rN8DJ3SRmrVrke8ZyRusHlP+kuwqTcfWLyqMOE3sScPPyeIXN/mDJIfGXvcMqCgYKekoQw==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.27.7.tgz", + "integrity": "sha512-+A1NJmfM8WNDv5CLVQYJ5PshuRm/4cI6WMZRg1by1GwPIQPCTs1GLEUHwiiQGT5zDdyLiRM/l1G0Pv54gvtKIg==", "cpu": [ "x64" ], @@ -696,9 +688,9 @@ } }, "node_modules/@esbuild/openharmony-arm64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/openharmony-arm64/-/openharmony-arm64-0.25.12.tgz", - "integrity": "sha512-rm0YWsqUSRrjncSXGA7Zv78Nbnw4XL6/dzr20cyrQf7ZmRcsovpcRBdhD43Nuk3y7XIoW2OxMVvwuRvk9XdASg==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/openharmony-arm64/-/openharmony-arm64-0.27.7.tgz", + "integrity": "sha512-+KrvYb/C8zA9CU/g0sR6w2RBw7IGc5J2BPnc3dYc5VJxHCSF1yNMxTV5LQ7GuKteQXZtspjFbiuW5/dOj7H4Yw==", "cpu": [ "arm64" ], @@ -713,9 +705,9 @@ } }, "node_modules/@esbuild/sunos-x64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.25.12.tgz", - "integrity": "sha512-3wGSCDyuTHQUzt0nV7bocDy72r2lI33QL3gkDNGkod22EsYl04sMf0qLb8luNKTOmgF/eDEDP5BFNwoBKH441w==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.27.7.tgz", + "integrity": "sha512-ikktIhFBzQNt/QDyOL580ti9+5mL/YZeUPKU2ivGtGjdTYoqz6jObj6nOMfhASpS4GU4Q/Clh1QtxWAvcYKamA==", "cpu": [ "x64" ], @@ -730,9 +722,9 @@ } }, "node_modules/@esbuild/win32-arm64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.25.12.tgz", - "integrity": "sha512-rMmLrur64A7+DKlnSuwqUdRKyd3UE7oPJZmnljqEptesKM8wx9J8gx5u0+9Pq0fQQW8vqeKebwNXdfOyP+8Bsg==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.27.7.tgz", + "integrity": "sha512-7yRhbHvPqSpRUV7Q20VuDwbjW5kIMwTHpptuUzV+AA46kiPze5Z7qgt6CLCK3pWFrHeNfDd1VKgyP4O+ng17CA==", "cpu": [ "arm64" ], @@ -747,9 +739,9 @@ } }, "node_modules/@esbuild/win32-ia32": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.25.12.tgz", - "integrity": "sha512-HkqnmmBoCbCwxUKKNPBixiWDGCpQGVsrQfJoVGYLPT41XWF8lHuE5N6WhVia2n4o5QK5M4tYr21827fNhi4byQ==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.27.7.tgz", + "integrity": "sha512-SmwKXe6VHIyZYbBLJrhOoCJRB/Z1tckzmgTLfFYOfpMAx63BJEaL9ExI8x7v0oAO3Zh6D/Oi1gVxEYr5oUCFhw==", "cpu": [ "ia32" ], @@ -764,9 +756,9 @@ } }, "node_modules/@esbuild/win32-x64": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.25.12.tgz", - "integrity": "sha512-alJC0uCZpTFrSL0CCDjcgleBXPnCrEAhTBILpeAp7M/OFgoqtAetfBzX0xM00MUsVVPpVjlPuMbREqnZCXaTnA==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.27.7.tgz", + "integrity": "sha512-56hiAJPhwQ1R4i+21FVF7V8kSD5zZTdHcVuRFMW0hn753vVfQN8xlx4uOPT4xoGH0Z/oVATuR82AiqSTDIpaHg==", "cpu": [ "x64" ], @@ -823,144 +815,129 @@ } }, "node_modules/@eslint/config-array": { - "version": "0.21.2", - "resolved": "https://registry.npmjs.org/@eslint/config-array/-/config-array-0.21.2.tgz", - "integrity": "sha512-nJl2KGTlrf9GjLimgIru+V/mzgSK0ABCDQRvxw5BjURL7WfH5uoWmizbH7QB6MmnMBd8cIC9uceWnezL1VZWWw==", + "version": "0.23.5", + "resolved": "https://registry.npmjs.org/@eslint/config-array/-/config-array-0.23.5.tgz", + "integrity": "sha512-Y3kKLvC1dvTOT+oGlqNQ1XLqK6D1HU2YXPc52NmAlJZbMMWDzGYXMiPRJ8TYD39muD/OTjlZmNJ4ib7dvSrMBA==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@eslint/object-schema": "^2.1.7", + "@eslint/object-schema": "^3.0.5", "debug": "^4.3.1", - "minimatch": "^3.1.5" + "minimatch": "^10.2.4" }, "engines": { - "node": "^18.18.0 || ^20.9.0 || >=21.1.0" + "node": "^20.19.0 || ^22.13.0 || >=24" } }, "node_modules/@eslint/config-helpers": { - "version": "0.4.2", - "resolved": "https://registry.npmjs.org/@eslint/config-helpers/-/config-helpers-0.4.2.tgz", - "integrity": "sha512-gBrxN88gOIf3R7ja5K9slwNayVcZgK6SOUORm2uBzTeIEfeVaIhOpCtTox3P6R7o2jLFwLFTLnC7kU/RGcYEgw==", + "version": "0.5.5", + "resolved": "https://registry.npmjs.org/@eslint/config-helpers/-/config-helpers-0.5.5.tgz", + "integrity": "sha512-eIJYKTCECbP/nsKaaruF6LW967mtbQbsw4JTtSVkUQc9MneSkbrgPJAbKl9nWr0ZeowV8BfsarBmPpBzGelA2w==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@eslint/core": "^0.17.0" + "@eslint/core": "^1.2.1" }, "engines": { - "node": "^18.18.0 || ^20.9.0 || >=21.1.0" + "node": "^20.19.0 || ^22.13.0 || >=24" } }, "node_modules/@eslint/core": { - "version": "0.17.0", - "resolved": "https://registry.npmjs.org/@eslint/core/-/core-0.17.0.tgz", - "integrity": "sha512-yL/sLrpmtDaFEiUj1osRP4TI2MDz1AddJL+jZ7KSqvBuliN4xqYY54IfdN8qD8Toa6g1iloph1fxQNkjOxrrpQ==", + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/@eslint/core/-/core-1.2.1.tgz", + "integrity": "sha512-MwcE1P+AZ4C6DWlpin/OmOA54mmIZ/+xZuJiQd4SyB29oAJjN30UW9wkKNptW2ctp4cEsvhlLY/CsQ1uoHDloQ==", "dev": true, "license": "Apache-2.0", "dependencies": { "@types/json-schema": "^7.0.15" }, "engines": { - "node": "^18.18.0 || ^20.9.0 || >=21.1.0" - } - }, - "node_modules/@eslint/eslintrc": { - "version": "3.3.5", - "resolved": "https://registry.npmjs.org/@eslint/eslintrc/-/eslintrc-3.3.5.tgz", - "integrity": "sha512-4IlJx0X0qftVsN5E+/vGujTRIFtwuLbNsVUe7TO6zYPDR1O6nFwvwhIKEKSrl6dZchmYBITazxKoUYOjdtjlRg==", - "dev": true, - "license": "MIT", - "dependencies": { - "ajv": "^6.14.0", - "debug": "^4.3.2", - "espree": "^10.0.1", - "globals": "^14.0.0", - "ignore": "^5.2.0", - "import-fresh": "^3.2.1", - "js-yaml": "^4.1.1", - "minimatch": "^3.1.5", - "strip-json-comments": "^3.1.1" - }, - "engines": { - "node": "^18.18.0 || ^20.9.0 || >=21.1.0" - }, - "funding": { - "url": "https://opencollective.com/eslint" - } - }, - "node_modules/@eslint/eslintrc/node_modules/globals": { - "version": "14.0.0", - "resolved": "https://registry.npmjs.org/globals/-/globals-14.0.0.tgz", - "integrity": "sha512-oahGvuMGQlPw/ivIYBjVSrWAfWLBeku5tpPE2fOPLi+WHffIWbuh2tCjhyQhTBPMf5E9jDEH4FOmTYgYwbKwtQ==", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=18" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" + "node": "^20.19.0 || ^22.13.0 || >=24" } }, "node_modules/@eslint/js": { - "version": "9.39.4", - "resolved": "https://registry.npmjs.org/@eslint/js/-/js-9.39.4.tgz", - "integrity": "sha512-nE7DEIchvtiFTwBw4Lfbu59PG+kCofhjsKaCWzxTpt4lfRjRMqG6uMBzKXuEcyXhOHoUp9riAm7/aWYGhXZ9cw==", + "version": "10.0.1", + "resolved": "https://registry.npmjs.org/@eslint/js/-/js-10.0.1.tgz", + "integrity": "sha512-zeR9k5pd4gxjZ0abRoIaxdc7I3nDktoXZk2qOv9gCNWx3mVwEn32VRhyLaRsDiJjTs0xq/T8mfPtyuXu7GWBcA==", "dev": true, "license": "MIT", "engines": { - "node": "^18.18.0 || ^20.9.0 || >=21.1.0" + "node": "^20.19.0 || ^22.13.0 || >=24" }, "funding": { "url": "https://eslint.org/donate" + }, + "peerDependencies": { + "eslint": "^10.0.0" + }, + "peerDependenciesMeta": { + "eslint": { + "optional": true + } } }, "node_modules/@eslint/object-schema": { - "version": "2.1.7", - "resolved": "https://registry.npmjs.org/@eslint/object-schema/-/object-schema-2.1.7.tgz", - "integrity": "sha512-VtAOaymWVfZcmZbp6E2mympDIHvyjXs/12LqWYjVw6qjrfF+VK+fyG33kChz3nnK+SU5/NeHOqrTEHS8sXO3OA==", + "version": "3.0.5", + "resolved": "https://registry.npmjs.org/@eslint/object-schema/-/object-schema-3.0.5.tgz", + "integrity": "sha512-vqTaUEgxzm+YDSdElad6PiRoX4t8VGDjCtt05zn4nU810UIx/uNEV7/lZJ6KwFThKZOzOxzXy48da+No7HZaMw==", "dev": true, "license": "Apache-2.0", "engines": { - "node": "^18.18.0 || ^20.9.0 || >=21.1.0" + "node": "^20.19.0 || ^22.13.0 || >=24" } }, "node_modules/@eslint/plugin-kit": { - "version": "0.4.1", - "resolved": "https://registry.npmjs.org/@eslint/plugin-kit/-/plugin-kit-0.4.1.tgz", - "integrity": "sha512-43/qtrDUokr7LJqoF2c3+RInu/t4zfrpYdoSDfYyhg52rwLV6TnOvdG4fXm7IkSB3wErkcmJS9iEhjVtOSEjjA==", + "version": "0.7.1", + "resolved": "https://registry.npmjs.org/@eslint/plugin-kit/-/plugin-kit-0.7.1.tgz", + "integrity": "sha512-rZAP3aVgB9ds9KOeUSL+zZ21hPmo8dh6fnIFwRQj5EAZl9gzR7wxYbYXYysAM8CTqGmUGyp2S4kUdV17MnGuWQ==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@eslint/core": "^0.17.0", + "@eslint/core": "^1.2.1", "levn": "^0.4.1" }, "engines": { - "node": "^18.18.0 || ^20.9.0 || >=21.1.0" + "node": "^20.19.0 || ^22.13.0 || >=24" } }, "node_modules/@humanfs/core": { - "version": "0.19.1", - "resolved": "https://registry.npmjs.org/@humanfs/core/-/core-0.19.1.tgz", - "integrity": "sha512-5DyQ4+1JEUzejeK1JGICcideyfUbGixgS9jNgex5nqkW+cY7WZhxBigmieN5Qnw9ZosSNVC9KQKyb+GUaGyKUA==", + "version": "0.19.2", + "resolved": "https://registry.npmjs.org/@humanfs/core/-/core-0.19.2.tgz", + "integrity": "sha512-UhXNm+CFMWcbChXywFwkmhqjs3PRCmcSa/hfBgLIb7oQ5HNb1wS0icWsGtSAUNgefHeI+eBrA8I1fxmbHsGdvA==", "dev": true, "license": "Apache-2.0", + "dependencies": { + "@humanfs/types": "^0.15.0" + }, "engines": { "node": ">=18.18.0" } }, "node_modules/@humanfs/node": { - "version": "0.16.7", - "resolved": "https://registry.npmjs.org/@humanfs/node/-/node-0.16.7.tgz", - "integrity": "sha512-/zUx+yOsIrG4Y43Eh2peDeKCxlRt/gET6aHfaKpuq267qXdYDFViVHfMaLyygZOnl0kGWxFIgsBy8QFuTLUXEQ==", + "version": "0.16.8", + "resolved": "https://registry.npmjs.org/@humanfs/node/-/node-0.16.8.tgz", + "integrity": "sha512-gE1eQNZ3R++kTzFUpdGlpmy8kDZD/MLyHqDwqjkVQI0JMdI1D51sy1H958PNXYkM2rAac7e5/CnIKZrHtPh3BQ==", "dev": true, "license": "Apache-2.0", "dependencies": { - "@humanfs/core": "^0.19.1", + "@humanfs/core": "^0.19.2", + "@humanfs/types": "^0.15.0", "@humanwhocodes/retry": "^0.4.0" }, "engines": { "node": ">=18.18.0" } }, + "node_modules/@humanfs/types": { + "version": "0.15.0", + "resolved": "https://registry.npmjs.org/@humanfs/types/-/types-0.15.0.tgz", + "integrity": "sha512-ZZ1w0aoQkwuUuC7Yf+7sdeaNfqQiiLcSRbfI08oAxqLtpXQr9AIVX7Ay7HLDuiLYAaFPu8oBYNq/QIi9URHJ3Q==", + "dev": true, + "license": "Apache-2.0", + "engines": { + "node": ">=18.18.0" + } + }, "node_modules/@humanwhocodes/module-importer": { "version": "1.0.1", "resolved": "https://registry.npmjs.org/@humanwhocodes/module-importer/-/module-importer-1.0.1.tgz", @@ -990,9 +967,9 @@ } }, "node_modules/@joshwooding/vite-plugin-react-docgen-typescript": { - "version": "0.6.4", - "resolved": "https://registry.npmjs.org/@joshwooding/vite-plugin-react-docgen-typescript/-/vite-plugin-react-docgen-typescript-0.6.4.tgz", - "integrity": "sha512-6PyZBYKnnVNqOSB0YFly+62R7dmov8segT27A+RVTBVd4iAE6kbW9QBJGlyR2yG4D4ohzhZSTIu7BK1UTtmFFA==", + "version": "0.7.0", + "resolved": "https://registry.npmjs.org/@joshwooding/vite-plugin-react-docgen-typescript/-/vite-plugin-react-docgen-typescript-0.7.0.tgz", + "integrity": "sha512-qvsTEwEFefhdirGOPnu9Wp6ChfIwy2dBCRuETU3uE+4cC+PFoxMSiiEhxk4lOluA34eARHA0OxqsEUYDqRMgeQ==", "dev": true, "license": "MIT", "dependencies": { @@ -1001,7 +978,7 @@ }, "peerDependencies": { "typescript": ">= 4.3.x", - "vite": "^3.0.0 || ^4.0.0 || ^5.0.0 || ^6.0.0 || ^7.0.0" + "vite": "^3.0.0 || ^4.0.0 || ^5.0.0 || ^6.0.0 || ^7.0.0 || ^8.0.0" }, "peerDependenciesMeta": { "typescript": { @@ -1077,54 +1054,39 @@ "react": ">=16" } }, - "node_modules/@rolldown/pluginutils": { - "version": "1.0.0-beta.27", - "resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.0-beta.27.tgz", - "integrity": "sha512-+d0F4MKMCbeVUJwG96uQ4SgAznZNSq93I3V+9NHA4OpvqG8mRCpGdKmK8l/dl02h2CCDHwW2FqilnTyDcAnqjA==", - "dev": true, - "license": "MIT" - }, - "node_modules/@rollup/pluginutils": { - "version": "5.3.0", - "resolved": "https://registry.npmjs.org/@rollup/pluginutils/-/pluginutils-5.3.0.tgz", - "integrity": "sha512-5EdhGZtnu3V88ces7s53hhfK5KSASnJZv8Lulpc04cWO3REESroJXg73DFsOmgbU2BhwV0E20bu2IDZb3VKW4Q==", + "node_modules/@napi-rs/wasm-runtime": { + "version": "1.1.4", + "resolved": "https://registry.npmjs.org/@napi-rs/wasm-runtime/-/wasm-runtime-1.1.4.tgz", + "integrity": "sha512-3NQNNgA1YSlJb/kMH1ildASP9HW7/7kYnRI2szWJaofaS1hWmbGI4H+d3+22aGzXXN9IJ+n+GiFVcGipJP18ow==", "dev": true, "license": "MIT", + "optional": true, "dependencies": { - "@types/estree": "^1.0.0", - "estree-walker": "^2.0.2", - "picomatch": "^4.0.2" + "@tybys/wasm-util": "^0.10.1" }, - "engines": { - "node": ">=14.0.0" + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" }, "peerDependencies": { - "rollup": "^1.20.0||^2.0.0||^3.0.0||^4.0.0" - }, - "peerDependenciesMeta": { - "rollup": { - "optional": true - } + "@emnapi/core": "^1.7.1", + "@emnapi/runtime": "^1.7.1" } }, - "node_modules/@rollup/rollup-android-arm-eabi": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm-eabi/-/rollup-android-arm-eabi-4.60.0.tgz", - "integrity": "sha512-WOhNW9K8bR3kf4zLxbfg6Pxu2ybOUbB2AjMDHSQx86LIF4rH4Ft7vmMwNt0loO0eonglSNy4cpD3MKXXKQu0/A==", - "cpu": [ - "arm" - ], + "node_modules/@oxc-project/types": { + "version": "0.127.0", + "resolved": "https://registry.npmjs.org/@oxc-project/types/-/types-0.127.0.tgz", + "integrity": "sha512-aIYXQBo4lCbO4z0R3FHeucQHpF46l2LbMdxRvqvuRuW2OxdnSkcng5B8+K12spgLDj93rtN3+J2Vac/TIO+ciQ==", "dev": true, "license": "MIT", - "optional": true, - "os": [ - "android" - ] + "funding": { + "url": "https://github.com/sponsors/Boshen" + } }, - "node_modules/@rollup/rollup-android-arm64": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm64/-/rollup-android-arm64-4.60.0.tgz", - "integrity": "sha512-u6JHLll5QKRvjciE78bQXDmqRqNs5M/3GVqZeMwvmjaNODJih/WIrJlFVEihvV0MiYFmd+ZyPr9wxOVbPAG2Iw==", + "node_modules/@rolldown/binding-android-arm64": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-android-arm64/-/binding-android-arm64-1.0.0-rc.17.tgz", + "integrity": "sha512-s70pVGhw4zqGeFnXWvAzJDlvxhlRollagdCCKRgOsgUOH3N1l0LIxf83AtGzmb5SiVM4Hjl5HyarMRfdfj3DaQ==", "cpu": [ "arm64" ], @@ -1133,12 +1095,15 @@ "optional": true, "os": [ "android" - ] + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-darwin-arm64": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-arm64/-/rollup-darwin-arm64-4.60.0.tgz", - "integrity": "sha512-qEF7CsKKzSRc20Ciu2Zw1wRrBz4g56F7r/vRwY430UPp/nt1x21Q/fpJ9N5l47WWvJlkNCPJz3QRVw008fi7yA==", + "node_modules/@rolldown/binding-darwin-arm64": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-arm64/-/binding-darwin-arm64-1.0.0-rc.17.tgz", + "integrity": "sha512-4ksWc9n0mhlZpZ9PMZgTGjeOPRu8MB1Z3Tz0Mo02eWfWCHMW1zN82Qz/pL/rC+yQa+8ZnutMF0JjJe7PjwasYw==", "cpu": [ "arm64" ], @@ -1147,12 +1112,15 @@ "optional": true, "os": [ "darwin" - ] + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-darwin-x64": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-x64/-/rollup-darwin-x64-4.60.0.tgz", - "integrity": "sha512-WADYozJ4QCnXCH4wPB+3FuGmDPoFseVCUrANmA5LWwGmC6FL14BWC7pcq+FstOZv3baGX65tZ378uT6WG8ynTw==", + "node_modules/@rolldown/binding-darwin-x64": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-darwin-x64/-/binding-darwin-x64-1.0.0-rc.17.tgz", + "integrity": "sha512-SUSDOI6WwUVNcWxd02QEBjLdY1VPHvlEkw6T/8nYG322iYWCTxRb1vzk4E+mWWYehTp7ERibq54LSJGjmouOsw==", "cpu": [ "x64" ], @@ -1161,26 +1129,15 @@ "optional": true, "os": [ "darwin" - ] - }, - "node_modules/@rollup/rollup-freebsd-arm64": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-arm64/-/rollup-freebsd-arm64-4.60.0.tgz", - "integrity": "sha512-6b8wGHJlDrGeSE3aH5mGNHBjA0TTkxdoNHik5EkvPHCt351XnigA4pS7Wsj/Eo9Y8RBU6f35cjN9SYmCFBtzxw==", - "cpu": [ - "arm64" ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "freebsd" - ] + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-freebsd-x64": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-x64/-/rollup-freebsd-x64-4.60.0.tgz", - "integrity": "sha512-h25Ga0t4jaylMB8M/JKAyrvvfxGRjnPQIR8lnCayyzEjEOx2EJIlIiMbhpWxDRKGKF8jbNH01NnN663dH638mA==", + "node_modules/@rolldown/binding-freebsd-x64": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-freebsd-x64/-/binding-freebsd-x64-1.0.0-rc.17.tgz", + "integrity": "sha512-hwnz3nw9dbJ05EDO/PvcjaaewqqDy7Y1rn1UO81l8iIK1GjenME75dl16ajbvSSMfv66WXSRCYKIqfgq2KCfxw==", "cpu": [ "x64" ], @@ -1189,26 +1146,15 @@ "optional": true, "os": [ "freebsd" - ] - }, - "node_modules/@rollup/rollup-linux-arm-gnueabihf": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-gnueabihf/-/rollup-linux-arm-gnueabihf-4.60.0.tgz", - "integrity": "sha512-RzeBwv0B3qtVBWtcuABtSuCzToo2IEAIQrcyB/b2zMvBWVbjo8bZDjACUpnaafaxhTw2W+imQbP2BD1usasK4g==", - "cpu": [ - "arm" ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ] + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-linux-arm-musleabihf": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-musleabihf/-/rollup-linux-arm-musleabihf-4.60.0.tgz", - "integrity": "sha512-Sf7zusNI2CIU1HLzuu9Tc5YGAHEZs5Lu7N1ssJG4Tkw6e0MEsN7NdjUDDfGNHy2IU+ENyWT+L2obgWiguWibWQ==", + "node_modules/@rolldown/binding-linux-arm-gnueabihf": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-1.0.0-rc.17.tgz", + "integrity": "sha512-IS+W7epTcwANmFSQFrS1SivEXHtl1JtuQA9wlxrZTcNi6mx+FDOYrakGevvvTwgj2JvWiK8B29/qD9BELZPyXQ==", "cpu": [ "arm" ], @@ -1217,26 +1163,15 @@ "optional": true, "os": [ "linux" - ] - }, - "node_modules/@rollup/rollup-linux-arm64-gnu": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-gnu/-/rollup-linux-arm64-gnu-4.60.0.tgz", - "integrity": "sha512-DX2x7CMcrJzsE91q7/O02IJQ5/aLkVtYFryqCjduJhUfGKG6yJV8hxaw8pZa93lLEpPTP/ohdN4wFz7yp/ry9A==", - "cpu": [ - "arm64" ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ] + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-linux-arm64-musl": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-musl/-/rollup-linux-arm64-musl-4.60.0.tgz", - "integrity": "sha512-09EL+yFVbJZlhcQfShpswwRZ0Rg+z/CsSELFCnPt3iK+iqwGsI4zht3secj5vLEs957QvFFXnzAT0FFPIxSrkQ==", + "node_modules/@rolldown/binding-linux-arm64-gnu": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-1.0.0-rc.17.tgz", + "integrity": "sha512-e6usGaHKW5BMNZOymS1UcEYGowQMWcgZ71Z17Sl/h2+ZziNJ1a9n3Zvcz6LdRyIW5572wBCTH/Z+bKuZouGk9Q==", "cpu": [ "arm64" ], @@ -1245,54 +1180,32 @@ "optional": true, "os": [ "linux" - ] - }, - "node_modules/@rollup/rollup-linux-loong64-gnu": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-loong64-gnu/-/rollup-linux-loong64-gnu-4.60.0.tgz", - "integrity": "sha512-i9IcCMPr3EXm8EQg5jnja0Zyc1iFxJjZWlb4wr7U2Wx/GrddOuEafxRdMPRYVaXjgbhvqalp6np07hN1w9kAKw==", - "cpu": [ - "loong64" ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ] + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-linux-loong64-musl": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-loong64-musl/-/rollup-linux-loong64-musl-4.60.0.tgz", - "integrity": "sha512-DGzdJK9kyJ+B78MCkWeGnpXJ91tK/iKA6HwHxF4TAlPIY7GXEvMe8hBFRgdrR9Ly4qebR/7gfUs9y2IoaVEyog==", + "node_modules/@rolldown/binding-linux-arm64-musl": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-arm64-musl/-/binding-linux-arm64-musl-1.0.0-rc.17.tgz", + "integrity": "sha512-b/CgbwAJpmrRLp02RPfhbudf5tZnN9nsPWK82znefso832etkem8H7FSZwxrOI9djcdTP7U6YfNhbRnh7djErg==", "cpu": [ - "loong64" + "arm64" ], "dev": true, "license": "MIT", "optional": true, "os": [ "linux" - ] - }, - "node_modules/@rollup/rollup-linux-ppc64-gnu": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-ppc64-gnu/-/rollup-linux-ppc64-gnu-4.60.0.tgz", - "integrity": "sha512-RwpnLsqC8qbS8z1H1AxBA1H6qknR4YpPR9w2XX0vo2Sz10miu57PkNcnHVaZkbqyw/kUWfKMI73jhmfi9BRMUQ==", - "cpu": [ - "ppc64" ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ] + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-linux-ppc64-musl": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-ppc64-musl/-/rollup-linux-ppc64-musl-4.60.0.tgz", - "integrity": "sha512-Z8pPf54Ly3aqtdWC3G4rFigZgNvd+qJlOE52fmko3KST9SoGfAdSRCwyoyG05q1HrrAblLbk1/PSIV+80/pxLg==", + "node_modules/@rolldown/binding-linux-ppc64-gnu": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-1.0.0-rc.17.tgz", + "integrity": "sha512-4EII1iNGRUN5WwGbF/kOh/EIkoDN9HsupgLQoXfY+D1oyJm7/F4t5PYU5n8SWZgG0FEwakyM8pGgwcBYruGTlA==", "cpu": [ "ppc64" ], @@ -1301,40 +1214,15 @@ "optional": true, "os": [ "linux" - ] - }, - "node_modules/@rollup/rollup-linux-riscv64-gnu": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-riscv64-gnu/-/rollup-linux-riscv64-gnu-4.60.0.tgz", - "integrity": "sha512-3a3qQustp3COCGvnP4SvrMHnPQ9d1vzCakQVRTliaz8cIp/wULGjiGpbcqrkv0WrHTEp8bQD/B3HBjzujVWLOA==", - "cpu": [ - "riscv64" - ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ] - }, - "node_modules/@rollup/rollup-linux-riscv64-musl": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-riscv64-musl/-/rollup-linux-riscv64-musl-4.60.0.tgz", - "integrity": "sha512-pjZDsVH/1VsghMJ2/kAaxt6dL0psT6ZexQVrijczOf+PeP2BUqTHYejk3l6TlPRydggINOeNRhvpLa0AYpCWSQ==", - "cpu": [ - "riscv64" ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "linux" - ] + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-linux-s390x-gnu": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-s390x-gnu/-/rollup-linux-s390x-gnu-4.60.0.tgz", - "integrity": "sha512-3ObQs0BhvPgiUVZrN7gqCSvmFuMWvWvsjG5ayJ3Lraqv+2KhOsp+pUbigqbeWqueGIsnn+09HBw27rJ+gYK4VQ==", + "node_modules/@rolldown/binding-linux-s390x-gnu": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-1.0.0-rc.17.tgz", + "integrity": "sha512-AH8oq3XqQo4IibpVXvPeLDI5pzkpYn0WiZAfT05kFzoJ6tQNzwRdDYQ45M8I/gslbodRZwW8uxLhbSBbkv96rA==", "cpu": [ "s390x" ], @@ -1343,12 +1231,15 @@ "optional": true, "os": [ "linux" - ] + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-linux-x64-gnu": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-gnu/-/rollup-linux-x64-gnu-4.60.0.tgz", - "integrity": "sha512-EtylprDtQPdS5rXvAayrNDYoJhIz1/vzN2fEubo3yLE7tfAw+948dO0g4M0vkTVFhKojnF+n6C8bDNe+gDRdTg==", + "node_modules/@rolldown/binding-linux-x64-gnu": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-gnu/-/binding-linux-x64-gnu-1.0.0-rc.17.tgz", + "integrity": "sha512-cLnjV3xfo7KslbU41Z7z8BH/E1y5mzUYzAqih1d1MDaIGZRCMqTijqLv76/P7fyHuvUcfGsIpqCdddbxLLK9rA==", "cpu": [ "x64" ], @@ -1357,12 +1248,15 @@ "optional": true, "os": [ "linux" - ] + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-linux-x64-musl": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-musl/-/rollup-linux-x64-musl-4.60.0.tgz", - "integrity": "sha512-k09oiRCi/bHU9UVFqD17r3eJR9bn03TyKraCrlz5ULFJGdJGi7VOmm9jl44vOJvRJ6P7WuBi/s2A97LxxHGIdw==", + "node_modules/@rolldown/binding-linux-x64-musl": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-linux-x64-musl/-/binding-linux-x64-musl-1.0.0-rc.17.tgz", + "integrity": "sha512-0phclDw1spsL7dUB37sIARuis2tAgomCJXAHZlpt8PXZ4Ba0dRP1e+66lsRqrfhISeN9bEGNjQs+T/Fbd7oYGw==", "cpu": [ "x64" ], @@ -1371,26 +1265,15 @@ "optional": true, "os": [ "linux" - ] - }, - "node_modules/@rollup/rollup-openbsd-x64": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-openbsd-x64/-/rollup-openbsd-x64-4.60.0.tgz", - "integrity": "sha512-1o/0/pIhozoSaDJoDcec+IVLbnRtQmHwPV730+AOD29lHEEo4F5BEUB24H0OBdhbBBDwIOSuf7vgg0Ywxdfiiw==", - "cpu": [ - "x64" ], - "dev": true, - "license": "MIT", - "optional": true, - "os": [ - "openbsd" - ] + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-openharmony-arm64": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-openharmony-arm64/-/rollup-openharmony-arm64-4.60.0.tgz", - "integrity": "sha512-pESDkos/PDzYwtyzB5p/UoNU/8fJo68vcXM9ZW2V0kjYayj1KaaUfi1NmTUTUpMn4UhU4gTuK8gIaFO4UGuMbA==", + "node_modules/@rolldown/binding-openharmony-arm64": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-openharmony-arm64/-/binding-openharmony-arm64-1.0.0-rc.17.tgz", + "integrity": "sha512-0ag/hEgXOwgw4t8QyQvUCxvEg+V0KBcA6YuOx9g0r02MprutRF5dyljgm3EmR02O292UX7UeS6HzWHAl6KgyhA==", "cpu": [ "arm64" ], @@ -1399,40 +1282,51 @@ "optional": true, "os": [ "openharmony" - ] + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-win32-arm64-msvc": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-arm64-msvc/-/rollup-win32-arm64-msvc-4.60.0.tgz", - "integrity": "sha512-hj1wFStD7B1YBeYmvY+lWXZ7ey73YGPcViMShYikqKT1GtstIKQAtfUI6yrzPjAy/O7pO0VLXGmUVWXQMaYgTQ==", + "node_modules/@rolldown/binding-wasm32-wasi": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-wasm32-wasi/-/binding-wasm32-wasi-1.0.0-rc.17.tgz", + "integrity": "sha512-LEXei6vo0E5wTGwpkJ4KoT3OZJRnglwldt5ziLzOlc6qqb55z4tWNq2A+PFqCJuvWWdP53CVhG1Z9NtToDPJrA==", "cpu": [ - "arm64" + "wasm32" ], "dev": true, "license": "MIT", "optional": true, - "os": [ - "win32" - ] + "dependencies": { + "@emnapi/core": "1.10.0", + "@emnapi/runtime": "1.10.0", + "@napi-rs/wasm-runtime": "^1.1.4" + }, + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-win32-ia32-msvc": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-ia32-msvc/-/rollup-win32-ia32-msvc-4.60.0.tgz", - "integrity": "sha512-SyaIPFoxmUPlNDq5EHkTbiKzmSEmq/gOYFI/3HHJ8iS/v1mbugVa7dXUzcJGQfoytp9DJFLhHH4U3/eTy2Bq4w==", + "node_modules/@rolldown/binding-win32-arm64-msvc": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.0.0-rc.17.tgz", + "integrity": "sha512-gUmyzBl3SPMa6hrqFUth9sVfcLBlYsbMzBx5PlexMroZStgzGqlZ26pYG89rBb45Mnia+oil6YAIFeEWGWhoZA==", "cpu": [ - "ia32" + "arm64" ], "dev": true, "license": "MIT", "optional": true, "os": [ "win32" - ] + ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } }, - "node_modules/@rollup/rollup-win32-x64-gnu": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-x64-gnu/-/rollup-win32-x64-gnu-4.60.0.tgz", - "integrity": "sha512-RdcryEfzZr+lAr5kRm2ucN9aVlCCa2QNq4hXelZxb8GG0NJSazq44Z3PCCc8wISRuCVnGs0lQJVX5Vp6fKA+IA==", + "node_modules/@rolldown/binding-win32-x64-msvc": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/binding-win32-x64-msvc/-/binding-win32-x64-msvc-1.0.0-rc.17.tgz", + "integrity": "sha512-3hkiolcUAvPB9FLb3UZdfjVVNWherN1f/skkGWJP/fgSQhYUZpSIRr0/I8ZK9TkF3F7kxvJAk0+IcKvPHk9qQg==", "cpu": [ "x64" ], @@ -1441,33 +1335,52 @@ "optional": true, "os": [ "win32" - ] - }, - "node_modules/@rollup/rollup-win32-x64-msvc": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-x64-msvc/-/rollup-win32-x64-msvc-4.60.0.tgz", - "integrity": "sha512-PrsWNQ8BuE00O3Xsx3ALh2Df8fAj9+cvvX9AIA6o4KpATR98c9mud4XtDWVvsEuyia5U4tVSTKygawyJkjm60w==", - "cpu": [ - "x64" ], + "engines": { + "node": "^20.19.0 || >=22.12.0" + } + }, + "node_modules/@rolldown/pluginutils": { + "version": "1.0.0-rc.7", + "resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.0-rc.7.tgz", + "integrity": "sha512-qujRfC8sFVInYSPPMLQByRh7zhwkGFS4+tyMQ83srV1qrxL4g8E2tyxVVyxd0+8QeBM1mIk9KbWxkegRr76XzA==", + "dev": true, + "license": "MIT" + }, + "node_modules/@rollup/pluginutils": { + "version": "5.3.0", + "resolved": "https://registry.npmjs.org/@rollup/pluginutils/-/pluginutils-5.3.0.tgz", + "integrity": "sha512-5EdhGZtnu3V88ces7s53hhfK5KSASnJZv8Lulpc04cWO3REESroJXg73DFsOmgbU2BhwV0E20bu2IDZb3VKW4Q==", "dev": true, "license": "MIT", - "optional": true, - "os": [ - "win32" - ] + "dependencies": { + "@types/estree": "^1.0.0", + "estree-walker": "^2.0.2", + "picomatch": "^4.0.2" + }, + "engines": { + "node": ">=14.0.0" + }, + "peerDependencies": { + "rollup": "^1.20.0||^2.0.0||^3.0.0||^4.0.0" + }, + "peerDependenciesMeta": { + "rollup": { + "optional": true + } + } }, "node_modules/@storybook/addon-docs": { - "version": "10.3.3", - "resolved": "https://registry.npmjs.org/@storybook/addon-docs/-/addon-docs-10.3.3.tgz", - "integrity": "sha512-trJQTpOtuOEuNv1Rn8X2Sopp5hSPpb0u0soEJ71BZAbxe4d2Y1d/1MYcxBdRKwncum6sCTsnxTpqQ/qvSJKlTQ==", + "version": "10.3.5", + "resolved": "https://registry.npmjs.org/@storybook/addon-docs/-/addon-docs-10.3.5.tgz", + "integrity": "sha512-WuHbxia/o5TX4Rg/IFD0641K5qId/Nk0dxhmAUNoFs5L0+yfZUwh65XOBbzXqrkYmYmcVID4v7cgDRmzstQNkA==", "dev": true, "license": "MIT", "dependencies": { "@mdx-js/react": "^3.0.0", - "@storybook/csf-plugin": "10.3.3", + "@storybook/csf-plugin": "10.3.5", "@storybook/icons": "^2.0.1", - "@storybook/react-dom-shim": "10.3.3", + "@storybook/react-dom-shim": "10.3.5", "react": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0", "react-dom": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0", "ts-dedent": "^2.0.0" @@ -1477,17 +1390,17 @@ "url": "https://opencollective.com/storybook" }, "peerDependencies": { - "storybook": "^10.3.3" + "storybook": "^10.3.5" } }, "node_modules/@storybook/builder-vite": { - "version": "10.3.3", - "resolved": "https://registry.npmjs.org/@storybook/builder-vite/-/builder-vite-10.3.3.tgz", - "integrity": "sha512-awspKCTZvXyeV3KabL0id62mFbxR5u/5yyGQultwCiSb2/yVgBfip2MAqLyS850pvTiB6QFVM9deOyd2/G/bEA==", + "version": "10.3.5", + "resolved": "https://registry.npmjs.org/@storybook/builder-vite/-/builder-vite-10.3.5.tgz", + "integrity": "sha512-i4KwCOKbhtlbQIbhm53+Kk7bMnxa0cwTn1pxmtA/x5wm1Qu7FrrBQV0V0DNjkUqzcSKo1CjspASJV/HlY0zYlw==", "dev": true, "license": "MIT", "dependencies": { - "@storybook/csf-plugin": "10.3.3", + "@storybook/csf-plugin": "10.3.5", "ts-dedent": "^2.0.0" }, "funding": { @@ -1495,14 +1408,14 @@ "url": "https://opencollective.com/storybook" }, "peerDependencies": { - "storybook": "^10.3.3", + "storybook": "^10.3.5", "vite": "^5.0.0 || ^6.0.0 || ^7.0.0 || ^8.0.0" } }, "node_modules/@storybook/csf-plugin": { - "version": "10.3.3", - "resolved": "https://registry.npmjs.org/@storybook/csf-plugin/-/csf-plugin-10.3.3.tgz", - "integrity": "sha512-Utlh7zubm+4iOzBBfzLW4F4vD99UBtl2Do4edlzK2F7krQIcFvR2ontjAE8S1FQVLZAC3WHalCOS+Ch8zf3knA==", + "version": "10.3.5", + "resolved": "https://registry.npmjs.org/@storybook/csf-plugin/-/csf-plugin-10.3.5.tgz", + "integrity": "sha512-qlEzNKxOjq86pvrbuMwiGD/bylnsXk1dg7ve0j77YFjEEchqtl7qTlrXvFdNaLA89GhW6D/EV6eOCu/eobPDgw==", "dev": true, "license": "MIT", "dependencies": { @@ -1515,7 +1428,7 @@ "peerDependencies": { "esbuild": "*", "rollup": "*", - "storybook": "^10.3.3", + "storybook": "^10.3.5", "vite": "*", "webpack": "*" }, @@ -1553,14 +1466,14 @@ } }, "node_modules/@storybook/react": { - "version": "10.3.3", - "resolved": "https://registry.npmjs.org/@storybook/react/-/react-10.3.3.tgz", - "integrity": "sha512-cGG5TbR8Tdx9zwlpsWyBEfWrejm5iWdYF26EwIhwuKq9GFUTAVrQzo0Rs7Tqc3ZyVhRS/YfsRiWSEH+zmq2JiQ==", + "version": "10.3.5", + "resolved": "https://registry.npmjs.org/@storybook/react/-/react-10.3.5.tgz", + "integrity": "sha512-tpLTLaVGoA6fLK3ReyGzZUricq7lyPaV2hLPpj5wqdXLV/LpRtAHClUpNoPDYSBjlnSjL81hMZijbkGC3mA+gw==", "dev": true, "license": "MIT", "dependencies": { "@storybook/global": "^5.0.0", - "@storybook/react-dom-shim": "10.3.3", + "@storybook/react-dom-shim": "10.3.5", "react-docgen": "^8.0.2", "react-docgen-typescript": "^2.2.2" }, @@ -1571,7 +1484,7 @@ "peerDependencies": { "react": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0", "react-dom": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0", - "storybook": "^10.3.3", + "storybook": "^10.3.5", "typescript": ">= 4.9.x" }, "peerDependenciesMeta": { @@ -1581,9 +1494,9 @@ } }, "node_modules/@storybook/react-dom-shim": { - "version": "10.3.3", - "resolved": "https://registry.npmjs.org/@storybook/react-dom-shim/-/react-dom-shim-10.3.3.tgz", - "integrity": "sha512-lkhuh4G3UTreU9M3Iz5Dt32c6U+l/4XuvqLtbe1sDHENZH6aPj7y0b5FwnfHyvuTvYRhtbo29xZrF5Bp9kCC0w==", + "version": "10.3.5", + "resolved": "https://registry.npmjs.org/@storybook/react-dom-shim/-/react-dom-shim-10.3.5.tgz", + "integrity": "sha512-Gw8R7XZm0zSUH0XAuxlQJhmizsLzyD6x00KOlP6l7oW9eQHXGfxg3seNDG3WrSAcW07iP1/P422kuiriQlOv7g==", "dev": true, "license": "MIT", "funding": { @@ -1593,20 +1506,20 @@ "peerDependencies": { "react": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0", "react-dom": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0", - "storybook": "^10.3.3" + "storybook": "^10.3.5" } }, "node_modules/@storybook/react-vite": { - "version": "10.3.3", - "resolved": "https://registry.npmjs.org/@storybook/react-vite/-/react-vite-10.3.3.tgz", - "integrity": "sha512-qHdlBe1hjqFAGXa8JL7bWTLbP/gDqXbWDm+SYCB646NHh5yvVDkZLwigP5Y+UL7M2ASfqFtosnroUK9tcCM2dw==", + "version": "10.3.5", + "resolved": "https://registry.npmjs.org/@storybook/react-vite/-/react-vite-10.3.5.tgz", + "integrity": "sha512-UB5sJHeh26bfd8sNMx2YPGYRYmErIdTRaLOT28m4bykQIa1l9IgVktsYg/geW7KsJU0lXd3oTbnUjLD+enpi3w==", "dev": true, "license": "MIT", "dependencies": { - "@joshwooding/vite-plugin-react-docgen-typescript": "^0.6.4", + "@joshwooding/vite-plugin-react-docgen-typescript": "^0.7.0", "@rollup/pluginutils": "^5.0.2", - "@storybook/builder-vite": "10.3.3", - "@storybook/react": "10.3.3", + "@storybook/builder-vite": "10.3.5", + "@storybook/react": "10.3.5", "empathic": "^2.0.0", "magic-string": "^0.30.0", "react-docgen": "^8.0.0", @@ -1620,23 +1533,23 @@ "peerDependencies": { "react": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0", "react-dom": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0", - "storybook": "^10.3.3", + "storybook": "^10.3.5", "vite": "^5.0.0 || ^6.0.0 || ^7.0.0 || ^8.0.0" } }, "node_modules/@swc/helpers": { - "version": "0.5.20", - "resolved": "https://registry.npmjs.org/@swc/helpers/-/helpers-0.5.20.tgz", - "integrity": "sha512-2egEBHUMasdypIzrprsu8g+OEVd7Vp2MM3a2eVlM/cyFYto0nGz5BX5BTgh/ShZZI9ed+ozEq+Ngt+rgmUs8tw==", + "version": "0.5.21", + "resolved": "https://registry.npmjs.org/@swc/helpers/-/helpers-0.5.21.tgz", + "integrity": "sha512-jI/VAmtdjB/RnI8GTnokyX7Ug8c+g+ffD6QRLa6XQewtnGyukKkKSk3wLTM3b5cjt1jNh9x0jfVlagdN2gDKQg==", "license": "Apache-2.0", "dependencies": { "tslib": "^2.8.0" } }, "node_modules/@tailwindcss/node": { - "version": "4.2.2", - "resolved": "https://registry.npmjs.org/@tailwindcss/node/-/node-4.2.2.tgz", - "integrity": "sha512-pXS+wJ2gZpVXqFaUEjojq7jzMpTGf8rU6ipJz5ovJV6PUGmlJ+jvIwGrzdHdQ80Sg+wmQxUFuoW1UAAwHNEdFA==", + "version": "4.2.4", + "resolved": "https://registry.npmjs.org/@tailwindcss/node/-/node-4.2.4.tgz", + "integrity": "sha512-Ai7+yQPxz3ddrDQzFfBKdHEVBg0w3Zl83jnjuwxnZOsnH9pGn93QHQtpU0p/8rYWxvbFZHneni6p1BSLK4DkGA==", "dev": true, "license": "MIT", "dependencies": { @@ -1646,37 +1559,37 @@ "lightningcss": "1.32.0", "magic-string": "^0.30.21", "source-map-js": "^1.2.1", - "tailwindcss": "4.2.2" + "tailwindcss": "4.2.4" } }, "node_modules/@tailwindcss/oxide": { - "version": "4.2.2", - "resolved": "https://registry.npmjs.org/@tailwindcss/oxide/-/oxide-4.2.2.tgz", - "integrity": "sha512-qEUA07+E5kehxYp9BVMpq9E8vnJuBHfJEC0vPC5e7iL/hw7HR61aDKoVoKzrG+QKp56vhNZe4qwkRmMC0zDLvg==", + "version": "4.2.4", + "resolved": "https://registry.npmjs.org/@tailwindcss/oxide/-/oxide-4.2.4.tgz", + "integrity": "sha512-9El/iI069DKDSXwTvB9J4BwdO5JhRrOweGaK25taBAvBXyXqJAX+Jqdvs8r8gKpsI/1m0LeJLyQYTf/WLrBT1Q==", "dev": true, "license": "MIT", "engines": { "node": ">= 20" }, "optionalDependencies": { - "@tailwindcss/oxide-android-arm64": "4.2.2", - "@tailwindcss/oxide-darwin-arm64": "4.2.2", - "@tailwindcss/oxide-darwin-x64": "4.2.2", - "@tailwindcss/oxide-freebsd-x64": "4.2.2", - "@tailwindcss/oxide-linux-arm-gnueabihf": "4.2.2", - "@tailwindcss/oxide-linux-arm64-gnu": "4.2.2", - "@tailwindcss/oxide-linux-arm64-musl": "4.2.2", - "@tailwindcss/oxide-linux-x64-gnu": "4.2.2", - "@tailwindcss/oxide-linux-x64-musl": "4.2.2", - "@tailwindcss/oxide-wasm32-wasi": "4.2.2", - "@tailwindcss/oxide-win32-arm64-msvc": "4.2.2", - "@tailwindcss/oxide-win32-x64-msvc": "4.2.2" + "@tailwindcss/oxide-android-arm64": "4.2.4", + "@tailwindcss/oxide-darwin-arm64": "4.2.4", + "@tailwindcss/oxide-darwin-x64": "4.2.4", + "@tailwindcss/oxide-freebsd-x64": "4.2.4", + "@tailwindcss/oxide-linux-arm-gnueabihf": "4.2.4", + "@tailwindcss/oxide-linux-arm64-gnu": "4.2.4", + "@tailwindcss/oxide-linux-arm64-musl": "4.2.4", + "@tailwindcss/oxide-linux-x64-gnu": "4.2.4", + "@tailwindcss/oxide-linux-x64-musl": "4.2.4", + "@tailwindcss/oxide-wasm32-wasi": "4.2.4", + "@tailwindcss/oxide-win32-arm64-msvc": "4.2.4", + "@tailwindcss/oxide-win32-x64-msvc": "4.2.4" } }, "node_modules/@tailwindcss/oxide-android-arm64": { - "version": "4.2.2", - "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-android-arm64/-/oxide-android-arm64-4.2.2.tgz", - "integrity": "sha512-dXGR1n+P3B6748jZO/SvHZq7qBOqqzQ+yFrXpoOWWALWndF9MoSKAT3Q0fYgAzYzGhxNYOoysRvYlpixRBBoDg==", + "version": "4.2.4", + "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-android-arm64/-/oxide-android-arm64-4.2.4.tgz", + "integrity": "sha512-e7MOr1SAn9U8KlZzPi1ZXGZHeC5anY36qjNwmZv9pOJ8E4Q6jmD1vyEHkQFmNOIN7twGPEMXRHmitN4zCMN03g==", "cpu": [ "arm64" ], @@ -1691,9 +1604,9 @@ } }, "node_modules/@tailwindcss/oxide-darwin-arm64": { - "version": "4.2.2", - "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-darwin-arm64/-/oxide-darwin-arm64-4.2.2.tgz", - "integrity": "sha512-iq9Qjr6knfMpZHj55/37ouZeykwbDqF21gPFtfnhCCKGDcPI/21FKC9XdMO/XyBM7qKORx6UIhGgg6jLl7BZlg==", + "version": "4.2.4", + "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-darwin-arm64/-/oxide-darwin-arm64-4.2.4.tgz", + "integrity": "sha512-tSC/Kbqpz/5/o/C2sG7QvOxAKqyd10bq+ypZNf+9Fi2TvbVbv1zNpcEptcsU7DPROaSbVgUXmrzKhurFvo5eDg==", "cpu": [ "arm64" ], @@ -1708,9 +1621,9 @@ } }, "node_modules/@tailwindcss/oxide-darwin-x64": { - "version": "4.2.2", - "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-darwin-x64/-/oxide-darwin-x64-4.2.2.tgz", - "integrity": "sha512-BlR+2c3nzc8f2G639LpL89YY4bdcIdUmiOOkv2GQv4/4M0vJlpXEa0JXNHhCHU7VWOKWT/CjqHdTP8aUuDJkuw==", + "version": "4.2.4", + "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-darwin-x64/-/oxide-darwin-x64-4.2.4.tgz", + "integrity": "sha512-yPyUXn3yO/ufR6+Kzv0t4fCg2qNr90jxXc5QqBpjlPNd0NqyDXcmQb/6weunH/MEDXW5dhyEi+agTDiqa3WsGg==", "cpu": [ "x64" ], @@ -1725,9 +1638,9 @@ } }, "node_modules/@tailwindcss/oxide-freebsd-x64": { - "version": "4.2.2", - "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-freebsd-x64/-/oxide-freebsd-x64-4.2.2.tgz", - "integrity": "sha512-YUqUgrGMSu2CDO82hzlQ5qSb5xmx3RUrke/QgnoEx7KvmRJHQuZHZmZTLSuuHwFf0DJPybFMXMYf+WJdxHy/nQ==", + "version": "4.2.4", + "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-freebsd-x64/-/oxide-freebsd-x64-4.2.4.tgz", + "integrity": "sha512-BoMIB4vMQtZsXdGLVc2z+P9DbETkiopogfWZKbWwM8b/1Vinbs4YcUwo+kM/KeLkX3Ygrf4/PsRndKaYhS8Eiw==", "cpu": [ "x64" ], @@ -1742,9 +1655,9 @@ } }, "node_modules/@tailwindcss/oxide-linux-arm-gnueabihf": { - "version": "4.2.2", - "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-linux-arm-gnueabihf/-/oxide-linux-arm-gnueabihf-4.2.2.tgz", - "integrity": "sha512-FPdhvsW6g06T9BWT0qTwiVZYE2WIFo2dY5aCSpjG/S/u1tby+wXoslXS0kl3/KXnULlLr1E3NPRRw0g7t2kgaQ==", + "version": "4.2.4", + "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-linux-arm-gnueabihf/-/oxide-linux-arm-gnueabihf-4.2.4.tgz", + "integrity": "sha512-7pIHBLTHYRAlS7V22JNuTh33yLH4VElwKtB3bwchK/UaKUPpQ0lPQiOWcbm4V3WP2I6fNIJ23vABIvoy2izdwA==", "cpu": [ "arm" ], @@ -1759,9 +1672,9 @@ } }, "node_modules/@tailwindcss/oxide-linux-arm64-gnu": { - "version": "4.2.2", - "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-linux-arm64-gnu/-/oxide-linux-arm64-gnu-4.2.2.tgz", - "integrity": "sha512-4og1V+ftEPXGttOO7eCmW7VICmzzJWgMx+QXAJRAhjrSjumCwWqMfkDrNu1LXEQzNAwz28NCUpucgQPrR4S2yw==", + "version": "4.2.4", + "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-linux-arm64-gnu/-/oxide-linux-arm64-gnu-4.2.4.tgz", + "integrity": "sha512-+E4wxJ0ZGOzSH325reXTWB48l42i93kQqMvDyz5gqfRzRZ7faNhnmvlV4EPGJU3QJM/3Ab5jhJ5pCRUsKn6OQw==", "cpu": [ "arm64" ], @@ -1776,9 +1689,9 @@ } }, "node_modules/@tailwindcss/oxide-linux-arm64-musl": { - "version": "4.2.2", - "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-linux-arm64-musl/-/oxide-linux-arm64-musl-4.2.2.tgz", - "integrity": "sha512-oCfG/mS+/+XRlwNjnsNLVwnMWYH7tn/kYPsNPh+JSOMlnt93mYNCKHYzylRhI51X+TbR+ufNhhKKzm6QkqX8ag==", + "version": "4.2.4", + "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-linux-arm64-musl/-/oxide-linux-arm64-musl-4.2.4.tgz", + "integrity": "sha512-bBADEGAbo4ASnppIziaQJelekCxdMaxisrk+fB7Thit72IBnALp9K6ffA2G4ruj90G9XRS2VQ6q2bCKbfFV82g==", "cpu": [ "arm64" ], @@ -1793,9 +1706,9 @@ } }, "node_modules/@tailwindcss/oxide-linux-x64-gnu": { - "version": "4.2.2", - "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-linux-x64-gnu/-/oxide-linux-x64-gnu-4.2.2.tgz", - "integrity": "sha512-rTAGAkDgqbXHNp/xW0iugLVmX62wOp2PoE39BTCGKjv3Iocf6AFbRP/wZT/kuCxC9QBh9Pu8XPkv/zCZB2mcMg==", + "version": "4.2.4", + "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-linux-x64-gnu/-/oxide-linux-x64-gnu-4.2.4.tgz", + "integrity": "sha512-7Mx25E4WTfnht0TVRTyC00j3i0M+EeFe7wguMDTlX4mRxafznw0CA8WJkFjWYH5BlgELd1kSjuU2JiPnNZbJDA==", "cpu": [ "x64" ], @@ -1810,9 +1723,9 @@ } }, "node_modules/@tailwindcss/oxide-linux-x64-musl": { - "version": "4.2.2", - "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-linux-x64-musl/-/oxide-linux-x64-musl-4.2.2.tgz", - "integrity": "sha512-XW3t3qwbIwiSyRCggeO2zxe3KWaEbM0/kW9e8+0XpBgyKU4ATYzcVSMKteZJ1iukJ3HgHBjbg9P5YPRCVUxlnQ==", + "version": "4.2.4", + "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-linux-x64-musl/-/oxide-linux-x64-musl-4.2.4.tgz", + "integrity": "sha512-2wwJRF7nyhOR0hhHoChc04xngV3iS+akccHTGtz965FwF0up4b2lOdo6kI1EbDaEXKgvcrFBYcYQQ/rrnWFVfA==", "cpu": [ "x64" ], @@ -1827,9 +1740,9 @@ } }, "node_modules/@tailwindcss/oxide-wasm32-wasi": { - "version": "4.2.2", - "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-wasm32-wasi/-/oxide-wasm32-wasi-4.2.2.tgz", - "integrity": "sha512-eKSztKsmEsn1O5lJ4ZAfyn41NfG7vzCg496YiGtMDV86jz1q/irhms5O0VrY6ZwTUkFy/EKG3RfWgxSI3VbZ8Q==", + "version": "4.2.4", + "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-wasm32-wasi/-/oxide-wasm32-wasi-4.2.4.tgz", + "integrity": "sha512-FQsqApeor8Fo6gUEklzmaa9994orJZZDBAlQpK2Mq+DslRKFJeD6AjHpBQ0kZFQohVr8o85PPh8eOy86VlSCmw==", "bundleDependencies": [ "@napi-rs/wasm-runtime", "@emnapi/core", @@ -1921,9 +1834,9 @@ "optional": true }, "node_modules/@tailwindcss/oxide-win32-arm64-msvc": { - "version": "4.2.2", - "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-win32-arm64-msvc/-/oxide-win32-arm64-msvc-4.2.2.tgz", - "integrity": "sha512-qPmaQM4iKu5mxpsrWZMOZRgZv1tOZpUm+zdhhQP0VhJfyGGO3aUKdbh3gDZc/dPLQwW4eSqWGrrcWNBZWUWaXQ==", + "version": "4.2.4", + "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-win32-arm64-msvc/-/oxide-win32-arm64-msvc-4.2.4.tgz", + "integrity": "sha512-L9BXqxC4ToVgwMFqj3pmZRqyHEztulpUJzCxUtLjobMCzTPsGt1Fa9enKbOpY2iIyVtaHNeNvAK8ERP/64sqGQ==", "cpu": [ "arm64" ], @@ -1938,9 +1851,9 @@ } }, "node_modules/@tailwindcss/oxide-win32-x64-msvc": { - "version": "4.2.2", - "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-win32-x64-msvc/-/oxide-win32-x64-msvc-4.2.2.tgz", - "integrity": "sha512-1T/37VvI7WyH66b+vqHj/cLwnCxt7Qt3WFu5Q8hk65aOvlwAhs7rAp1VkulBJw/N4tMirXjVnylTR72uI0HGcA==", + "version": "4.2.4", + "resolved": "https://registry.npmjs.org/@tailwindcss/oxide-win32-x64-msvc/-/oxide-win32-x64-msvc-4.2.4.tgz", + "integrity": "sha512-ESlKG0EpVJQwRjXDDa9rLvhEAh0mhP1sF7sap9dNZT0yyl9SAG6T7gdP09EH0vIv0UNTlo6jPWyujD6559fZvw==", "cpu": [ "x64" ], @@ -1955,15 +1868,15 @@ } }, "node_modules/@tailwindcss/vite": { - "version": "4.2.2", - "resolved": "https://registry.npmjs.org/@tailwindcss/vite/-/vite-4.2.2.tgz", - "integrity": "sha512-mEiF5HO1QqCLXoNEfXVA1Tzo+cYsrqV7w9Juj2wdUFyW07JRenqMG225MvPwr3ZD9N1bFQj46X7r33iHxLUW0w==", + "version": "4.2.4", + "resolved": "https://registry.npmjs.org/@tailwindcss/vite/-/vite-4.2.4.tgz", + "integrity": "sha512-pCvohwOCspk3ZFn6eJzrrX3g4n2JY73H6MmYC87XfGPyTty4YsCjYTMArRZm/zOI8dIt3+EcrLHAFPe5A4bgtw==", "dev": true, "license": "MIT", "dependencies": { - "@tailwindcss/node": "4.2.2", - "@tailwindcss/oxide": "4.2.2", - "tailwindcss": "4.2.2" + "@tailwindcss/node": "4.2.4", + "@tailwindcss/oxide": "4.2.4", + "tailwindcss": "4.2.4" }, "peerDependencies": { "vite": "^5.2.0 || ^6 || ^7 || ^8" @@ -1990,12 +1903,12 @@ } }, "node_modules/@tanstack/react-virtual": { - "version": "3.13.23", - "resolved": "https://registry.npmjs.org/@tanstack/react-virtual/-/react-virtual-3.13.23.tgz", - "integrity": "sha512-XnMRnHQ23piOVj2bzJqHrRrLg4r+F86fuBcwteKfbIjJrtGxb4z7tIvPVAe4B+4UVwo9G4Giuz5fmapcrnZ0OQ==", + "version": "3.13.24", + "resolved": "https://registry.npmjs.org/@tanstack/react-virtual/-/react-virtual-3.13.24.tgz", + "integrity": "sha512-aIJvz5OSkhNIhZIpYivrxrPTKYsjW9Uzy+sP/mx0S3sev2HyvPb7xmjbYvokzEpfgYHy/HjzJ2zFAETuUfgCpg==", "license": "MIT", "dependencies": { - "@tanstack/virtual-core": "3.13.23" + "@tanstack/virtual-core": "3.14.0" }, "funding": { "type": "github", @@ -2020,9 +1933,9 @@ } }, "node_modules/@tanstack/virtual-core": { - "version": "3.13.23", - "resolved": "https://registry.npmjs.org/@tanstack/virtual-core/-/virtual-core-3.13.23.tgz", - "integrity": "sha512-zSz2Z2HNyLjCplANTDyl3BcdQJc2k1+yyFoKhNRmCr7V7dY8o8q5m8uFTI1/Pg1kL+Hgrz6u3Xo6eFUB7l66cg==", + "version": "3.14.0", + "resolved": "https://registry.npmjs.org/@tanstack/virtual-core/-/virtual-core-3.14.0.tgz", + "integrity": "sha512-JLANqGy/D6k4Ujmh8Tr25lGimuOXNiaVyXaCAZS0W+1390sADdGnyUdSWNIfd49gebtIxGMij4IktRVzrdr12Q==", "license": "MIT", "funding": { "type": "github", @@ -2091,6 +2004,17 @@ "@testing-library/dom": ">=7.21.4" } }, + "node_modules/@tybys/wasm-util": { + "version": "0.10.1", + "resolved": "https://registry.npmjs.org/@tybys/wasm-util/-/wasm-util-0.10.1.tgz", + "integrity": "sha512-9tTaPJLSiejZKx+Bmog4uSubteqTvFrVrURwkmHixBo0G4seD0zUxp98E1DzUBJxLQ3NPwXrGKDiVjwx/DpPsg==", + "dev": true, + "license": "MIT", + "optional": true, + "dependencies": { + "tslib": "^2.4.0" + } + }, "node_modules/@types/aria-query": { "version": "5.0.4", "resolved": "https://registry.npmjs.org/@types/aria-query/-/aria-query-5.0.4.tgz", @@ -2188,6 +2112,13 @@ "dev": true, "license": "MIT" }, + "node_modules/@types/esrecurse": { + "version": "4.3.1", + "resolved": "https://registry.npmjs.org/@types/esrecurse/-/esrecurse-4.3.1.tgz", + "integrity": "sha512-xJBAbDifo5hpffDBuHl0Y8ywswbiAp/Wi7Y/GtAgSlZyIABppyurxVueOPE8LUQOxdlgi6Zqce7uoEpqNTeiUw==", + "dev": true, + "license": "MIT" + }, "node_modules/@types/estree": { "version": "1.0.8", "resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.8.tgz", @@ -2210,9 +2141,9 @@ "license": "MIT" }, "node_modules/@types/node": { - "version": "24.12.0", - "resolved": "https://registry.npmjs.org/@types/node/-/node-24.12.0.tgz", - "integrity": "sha512-GYDxsZi3ChgmckRT9HPU0WEhKLP08ev/Yfcq2AstjrDASOYCSXeyjDsHg4v5t4jOj7cyDX3vmprafKlWIG9MXQ==", + "version": "24.12.2", + "resolved": "https://registry.npmjs.org/@types/node/-/node-24.12.2.tgz", + "integrity": "sha512-A1sre26ke7HDIuY/M23nd9gfB+nrmhtYyMINbjI1zHJxYteKR6qSMX56FsmjMcDb3SMcjJg5BiRRgOCC/yBD0g==", "license": "MIT", "dependencies": { "undici-types": "~7.16.0" @@ -2246,20 +2177,20 @@ "license": "MIT" }, "node_modules/@typescript-eslint/eslint-plugin": { - "version": "8.57.2", - "resolved": "https://registry.npmjs.org/@typescript-eslint/eslint-plugin/-/eslint-plugin-8.57.2.tgz", - "integrity": "sha512-NZZgp0Fm2IkD+La5PR81sd+g+8oS6JwJje+aRWsDocxHkjyRw0J5L5ZTlN3LI1LlOcGL7ph3eaIUmTXMIjLk0w==", + "version": "8.59.0", + "resolved": "https://registry.npmjs.org/@typescript-eslint/eslint-plugin/-/eslint-plugin-8.59.0.tgz", + "integrity": "sha512-HyAZtpdkgZwpq8Sz3FSUvCR4c+ScbuWa9AksK2Jweub7w4M3yTz4O11AqVJzLYjy/B9ZWPyc81I+mOdJU/bDQw==", "dev": true, "license": "MIT", "dependencies": { "@eslint-community/regexpp": "^4.12.2", - "@typescript-eslint/scope-manager": "8.57.2", - "@typescript-eslint/type-utils": "8.57.2", - "@typescript-eslint/utils": "8.57.2", - "@typescript-eslint/visitor-keys": "8.57.2", + "@typescript-eslint/scope-manager": "8.59.0", + "@typescript-eslint/type-utils": "8.59.0", + "@typescript-eslint/utils": "8.59.0", + "@typescript-eslint/visitor-keys": "8.59.0", "ignore": "^7.0.5", "natural-compare": "^1.4.0", - "ts-api-utils": "^2.4.0" + "ts-api-utils": "^2.5.0" }, "engines": { "node": "^18.18.0 || ^20.9.0 || >=21.1.0" @@ -2269,9 +2200,9 @@ "url": "https://opencollective.com/typescript-eslint" }, "peerDependencies": { - "@typescript-eslint/parser": "^8.57.2", + "@typescript-eslint/parser": "^8.59.0", "eslint": "^8.57.0 || ^9.0.0 || ^10.0.0", - "typescript": ">=4.8.4 <6.0.0" + "typescript": ">=4.8.4 <6.1.0" } }, "node_modules/@typescript-eslint/eslint-plugin/node_modules/ignore": { @@ -2285,16 +2216,16 @@ } }, "node_modules/@typescript-eslint/parser": { - "version": "8.57.2", - "resolved": "https://registry.npmjs.org/@typescript-eslint/parser/-/parser-8.57.2.tgz", - "integrity": "sha512-30ScMRHIAD33JJQkgfGW1t8CURZtjc2JpTrq5n2HFhOefbAhb7ucc7xJwdWcrEtqUIYJ73Nybpsggii6GtAHjA==", + "version": "8.59.0", + "resolved": "https://registry.npmjs.org/@typescript-eslint/parser/-/parser-8.59.0.tgz", + "integrity": "sha512-TI1XGwKbDpo9tRW8UDIXCOeLk55qe9ZFGs8MTKU6/M08HWTw52DD/IYhfQtOEhEdPhLMT26Ka/x7p70nd3dzDg==", "dev": true, "license": "MIT", "dependencies": { - "@typescript-eslint/scope-manager": "8.57.2", - "@typescript-eslint/types": "8.57.2", - "@typescript-eslint/typescript-estree": "8.57.2", - "@typescript-eslint/visitor-keys": "8.57.2", + "@typescript-eslint/scope-manager": "8.59.0", + "@typescript-eslint/types": "8.59.0", + "@typescript-eslint/typescript-estree": "8.59.0", + "@typescript-eslint/visitor-keys": "8.59.0", "debug": "^4.4.3" }, "engines": { @@ -2306,18 +2237,18 @@ }, "peerDependencies": { "eslint": "^8.57.0 || ^9.0.0 || ^10.0.0", - "typescript": ">=4.8.4 <6.0.0" + "typescript": ">=4.8.4 <6.1.0" } }, "node_modules/@typescript-eslint/project-service": { - "version": "8.57.2", - "resolved": "https://registry.npmjs.org/@typescript-eslint/project-service/-/project-service-8.57.2.tgz", - "integrity": "sha512-FuH0wipFywXRTHf+bTTjNyuNQQsQC3qh/dYzaM4I4W0jrCqjCVuUh99+xd9KamUfmCGPvbO8NDngo/vsnNVqgw==", + "version": "8.59.0", + "resolved": "https://registry.npmjs.org/@typescript-eslint/project-service/-/project-service-8.59.0.tgz", + "integrity": "sha512-Lw5ITrR5s5TbC19YSvlr63ZfLaJoU6vtKTHyB0GQOpX0W7d5/Ir6vUahWi/8Sps/nOukZQ0IB3SmlxZnjaKVnw==", "dev": true, "license": "MIT", "dependencies": { - "@typescript-eslint/tsconfig-utils": "^8.57.2", - "@typescript-eslint/types": "^8.57.2", + "@typescript-eslint/tsconfig-utils": "^8.59.0", + "@typescript-eslint/types": "^8.59.0", "debug": "^4.4.3" }, "engines": { @@ -2328,18 +2259,18 @@ "url": "https://opencollective.com/typescript-eslint" }, "peerDependencies": { - "typescript": ">=4.8.4 <6.0.0" + "typescript": ">=4.8.4 <6.1.0" } }, "node_modules/@typescript-eslint/scope-manager": { - "version": "8.57.2", - "resolved": "https://registry.npmjs.org/@typescript-eslint/scope-manager/-/scope-manager-8.57.2.tgz", - "integrity": "sha512-snZKH+W4WbWkrBqj4gUNRIGb/jipDW3qMqVJ4C9rzdFc+wLwruxk+2a5D+uoFcKPAqyqEnSb4l2ULuZf95eSkw==", + "version": "8.59.0", + "resolved": "https://registry.npmjs.org/@typescript-eslint/scope-manager/-/scope-manager-8.59.0.tgz", + "integrity": "sha512-UzR16Ut8IpA3Mc4DbgAShlPPkVm8xXMWafXxB0BocaVRHs8ZGakAxGRskF7FId3sdk9lgGD73GSFaWmWFDE4dg==", "dev": true, "license": "MIT", "dependencies": { - "@typescript-eslint/types": "8.57.2", - "@typescript-eslint/visitor-keys": "8.57.2" + "@typescript-eslint/types": "8.59.0", + "@typescript-eslint/visitor-keys": "8.59.0" }, "engines": { "node": "^18.18.0 || ^20.9.0 || >=21.1.0" @@ -2350,9 +2281,9 @@ } }, "node_modules/@typescript-eslint/tsconfig-utils": { - "version": "8.57.2", - "resolved": "https://registry.npmjs.org/@typescript-eslint/tsconfig-utils/-/tsconfig-utils-8.57.2.tgz", - "integrity": "sha512-3Lm5DSM+DCowsUOJC+YqHHnKEfFh5CoGkj5Z31NQSNF4l5wdOwqGn99wmwN/LImhfY3KJnmordBq/4+VDe2eKw==", + "version": "8.59.0", + "resolved": "https://registry.npmjs.org/@typescript-eslint/tsconfig-utils/-/tsconfig-utils-8.59.0.tgz", + "integrity": "sha512-91Sbl3s4Kb3SybliIY6muFBmHVv+pYXfybC4Oolp3dvk8BvIE3wOPc+403CWIT7mJNkfQRGtdqghzs2+Z91Tqg==", "dev": true, "license": "MIT", "engines": { @@ -2363,21 +2294,21 @@ "url": "https://opencollective.com/typescript-eslint" }, "peerDependencies": { - "typescript": ">=4.8.4 <6.0.0" + "typescript": ">=4.8.4 <6.1.0" } }, "node_modules/@typescript-eslint/type-utils": { - "version": "8.57.2", - "resolved": "https://registry.npmjs.org/@typescript-eslint/type-utils/-/type-utils-8.57.2.tgz", - "integrity": "sha512-Co6ZCShm6kIbAM/s+oYVpKFfW7LBc6FXoPXjTRQ449PPNBY8U0KZXuevz5IFuuUj2H9ss40atTaf9dlGLzbWZg==", + "version": "8.59.0", + "resolved": "https://registry.npmjs.org/@typescript-eslint/type-utils/-/type-utils-8.59.0.tgz", + "integrity": "sha512-3TRiZaQSltGqGeNrJzzr1+8YcEobKH9rHnqIp/1psfKFmhRQDNMGP5hBufanYTGznwShzVLs3Mz+gDN7HkWfXg==", "dev": true, "license": "MIT", "dependencies": { - "@typescript-eslint/types": "8.57.2", - "@typescript-eslint/typescript-estree": "8.57.2", - "@typescript-eslint/utils": "8.57.2", + "@typescript-eslint/types": "8.59.0", + "@typescript-eslint/typescript-estree": "8.59.0", + "@typescript-eslint/utils": "8.59.0", "debug": "^4.4.3", - "ts-api-utils": "^2.4.0" + "ts-api-utils": "^2.5.0" }, "engines": { "node": "^18.18.0 || ^20.9.0 || >=21.1.0" @@ -2387,89 +2318,50 @@ "url": "https://opencollective.com/typescript-eslint" }, "peerDependencies": { - "eslint": "^8.57.0 || ^9.0.0 || ^10.0.0", - "typescript": ">=4.8.4 <6.0.0" - } - }, - "node_modules/@typescript-eslint/types": { - "version": "8.57.2", - "resolved": "https://registry.npmjs.org/@typescript-eslint/types/-/types-8.57.2.tgz", - "integrity": "sha512-/iZM6FnM4tnx9csuTxspMW4BOSegshwX5oBDznJ7S4WggL7Vczz5d2W11ecc4vRrQMQHXRSxzrCsyG5EsPPTbA==", - "dev": true, - "license": "MIT", - "engines": { - "node": "^18.18.0 || ^20.9.0 || >=21.1.0" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/typescript-eslint" - } - }, - "node_modules/@typescript-eslint/typescript-estree": { - "version": "8.57.2", - "resolved": "https://registry.npmjs.org/@typescript-eslint/typescript-estree/-/typescript-estree-8.57.2.tgz", - "integrity": "sha512-2MKM+I6g8tJxfSmFKOnHv2t8Sk3T6rF20A1Puk0svLK+uVapDZB/4pfAeB7nE83uAZrU6OxW+HmOd5wHVdXwXA==", - "dev": true, - "license": "MIT", - "dependencies": { - "@typescript-eslint/project-service": "8.57.2", - "@typescript-eslint/tsconfig-utils": "8.57.2", - "@typescript-eslint/types": "8.57.2", - "@typescript-eslint/visitor-keys": "8.57.2", - "debug": "^4.4.3", - "minimatch": "^10.2.2", - "semver": "^7.7.3", - "tinyglobby": "^0.2.15", - "ts-api-utils": "^2.4.0" - }, - "engines": { - "node": "^18.18.0 || ^20.9.0 || >=21.1.0" - }, - "funding": { - "type": "opencollective", - "url": "https://opencollective.com/typescript-eslint" - }, - "peerDependencies": { - "typescript": ">=4.8.4 <6.0.0" - } - }, - "node_modules/@typescript-eslint/typescript-estree/node_modules/balanced-match": { - "version": "4.0.4", - "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-4.0.4.tgz", - "integrity": "sha512-BLrgEcRTwX2o6gGxGOCNyMvGSp35YofuYzw9h1IMTRmKqttAZZVU67bdb9Pr2vUHA8+j3i2tJfjO6C6+4myGTA==", - "dev": true, - "license": "MIT", - "engines": { - "node": "18 || 20 || >=22" + "eslint": "^8.57.0 || ^9.0.0 || ^10.0.0", + "typescript": ">=4.8.4 <6.1.0" } }, - "node_modules/@typescript-eslint/typescript-estree/node_modules/brace-expansion": { - "version": "5.0.5", - "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.5.tgz", - "integrity": "sha512-VZznLgtwhn+Mact9tfiwx64fA9erHH/MCXEUfB/0bX/6Fz6ny5EGTXYltMocqg4xFAQZtnO3DHWWXi8RiuN7cQ==", + "node_modules/@typescript-eslint/types": { + "version": "8.59.0", + "resolved": "https://registry.npmjs.org/@typescript-eslint/types/-/types-8.59.0.tgz", + "integrity": "sha512-nLzdsT1gdOgFxxxwrlNVUBzSNBEEHJ86bblmk4QAS6stfig7rcJzWKqCyxFy3YRRHXDWEkb2NralA1nOYkkm/A==", "dev": true, "license": "MIT", - "dependencies": { - "balanced-match": "^4.0.2" - }, "engines": { - "node": "18 || 20 || >=22" + "node": "^18.18.0 || ^20.9.0 || >=21.1.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/typescript-eslint" } }, - "node_modules/@typescript-eslint/typescript-estree/node_modules/minimatch": { - "version": "10.2.4", - "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-10.2.4.tgz", - "integrity": "sha512-oRjTw/97aTBN0RHbYCdtF1MQfvusSIBQM0IZEgzl6426+8jSC0nF1a/GmnVLpfB9yyr6g6FTqWqiZVbxrtaCIg==", + "node_modules/@typescript-eslint/typescript-estree": { + "version": "8.59.0", + "resolved": "https://registry.npmjs.org/@typescript-eslint/typescript-estree/-/typescript-estree-8.59.0.tgz", + "integrity": "sha512-O9Re9P1BmBLFJyikRbQpLku/QA3/AueZNO9WePLBwQrvkixTmDe8u76B6CYUAITRl/rHawggEqUGn5QIkVRLMw==", "dev": true, - "license": "BlueOak-1.0.0", + "license": "MIT", "dependencies": { - "brace-expansion": "^5.0.2" + "@typescript-eslint/project-service": "8.59.0", + "@typescript-eslint/tsconfig-utils": "8.59.0", + "@typescript-eslint/types": "8.59.0", + "@typescript-eslint/visitor-keys": "8.59.0", + "debug": "^4.4.3", + "minimatch": "^10.2.2", + "semver": "^7.7.3", + "tinyglobby": "^0.2.15", + "ts-api-utils": "^2.5.0" }, "engines": { - "node": "18 || 20 || >=22" + "node": "^18.18.0 || ^20.9.0 || >=21.1.0" }, "funding": { - "url": "https://github.com/sponsors/isaacs" + "type": "opencollective", + "url": "https://opencollective.com/typescript-eslint" + }, + "peerDependencies": { + "typescript": ">=4.8.4 <6.1.0" } }, "node_modules/@typescript-eslint/typescript-estree/node_modules/semver": { @@ -2486,16 +2378,16 @@ } }, "node_modules/@typescript-eslint/utils": { - "version": "8.57.2", - "resolved": "https://registry.npmjs.org/@typescript-eslint/utils/-/utils-8.57.2.tgz", - "integrity": "sha512-krRIbvPK1ju1WBKIefiX+bngPs+odIQUtR7kymzPfo1POVw3jlF+nLkmexdSSd4UCbDcQn+wMBATOOmpBbqgKg==", + "version": "8.59.0", + "resolved": "https://registry.npmjs.org/@typescript-eslint/utils/-/utils-8.59.0.tgz", + "integrity": "sha512-I1R/K7V07XsMJ12Oaxg/O9GfrysGTmCRhvZJBv0RE0NcULMzjqVpR5kRRQjHsz3J/bElU7HwCO7zkqL+MSUz+g==", "dev": true, "license": "MIT", "dependencies": { "@eslint-community/eslint-utils": "^4.9.1", - "@typescript-eslint/scope-manager": "8.57.2", - "@typescript-eslint/types": "8.57.2", - "@typescript-eslint/typescript-estree": "8.57.2" + "@typescript-eslint/scope-manager": "8.59.0", + "@typescript-eslint/types": "8.59.0", + "@typescript-eslint/typescript-estree": "8.59.0" }, "engines": { "node": "^18.18.0 || ^20.9.0 || >=21.1.0" @@ -2506,17 +2398,17 @@ }, "peerDependencies": { "eslint": "^8.57.0 || ^9.0.0 || ^10.0.0", - "typescript": ">=4.8.4 <6.0.0" + "typescript": ">=4.8.4 <6.1.0" } }, "node_modules/@typescript-eslint/visitor-keys": { - "version": "8.57.2", - "resolved": "https://registry.npmjs.org/@typescript-eslint/visitor-keys/-/visitor-keys-8.57.2.tgz", - "integrity": "sha512-zhahknjobV2FiD6Ee9iLbS7OV9zi10rG26odsQdfBO/hjSzUQbkIYgda+iNKK1zNiW2ey+Lf8MU5btN17V3dUw==", + "version": "8.59.0", + "resolved": "https://registry.npmjs.org/@typescript-eslint/visitor-keys/-/visitor-keys-8.59.0.tgz", + "integrity": "sha512-/uejZt4dSere1bx12WLlPfv8GktzcaDtuJ7s42/HEZ5zGj9oxRaD4bj7qwSunXkf+pbAhFt2zjpHYUiT5lHf0Q==", "dev": true, "license": "MIT", "dependencies": { - "@typescript-eslint/types": "8.57.2", + "@typescript-eslint/types": "8.59.0", "eslint-visitor-keys": "^5.0.0" }, "engines": { @@ -2527,38 +2419,30 @@ "url": "https://opencollective.com/typescript-eslint" } }, - "node_modules/@typescript-eslint/visitor-keys/node_modules/eslint-visitor-keys": { - "version": "5.0.1", - "resolved": "https://registry.npmjs.org/eslint-visitor-keys/-/eslint-visitor-keys-5.0.1.tgz", - "integrity": "sha512-tD40eHxA35h0PEIZNeIjkHoDR4YjjJp34biM0mDvplBe//mB+IHCqHDGV7pxF+7MklTvighcCPPZC7ynWyjdTA==", - "dev": true, - "license": "Apache-2.0", - "engines": { - "node": "^20.19.0 || ^22.13.0 || >=24" - }, - "funding": { - "url": "https://opencollective.com/eslint" - } - }, "node_modules/@vitejs/plugin-react": { - "version": "4.7.0", - "resolved": "https://registry.npmjs.org/@vitejs/plugin-react/-/plugin-react-4.7.0.tgz", - "integrity": "sha512-gUu9hwfWvvEDBBmgtAowQCojwZmJ5mcLn3aufeCsitijs3+f2NsrPtlAWIR6OPiqljl96GVCUbLe0HyqIpVaoA==", + "version": "6.0.1", + "resolved": "https://registry.npmjs.org/@vitejs/plugin-react/-/plugin-react-6.0.1.tgz", + "integrity": "sha512-l9X/E3cDb+xY3SWzlG1MOGt2usfEHGMNIaegaUGFsLkb3RCn/k8/TOXBcab+OndDI4TBtktT8/9BwwW8Vi9KUQ==", "dev": true, "license": "MIT", "dependencies": { - "@babel/core": "^7.28.0", - "@babel/plugin-transform-react-jsx-self": "^7.27.1", - "@babel/plugin-transform-react-jsx-source": "^7.27.1", - "@rolldown/pluginutils": "1.0.0-beta.27", - "@types/babel__core": "^7.20.5", - "react-refresh": "^0.17.0" + "@rolldown/pluginutils": "1.0.0-rc.7" }, "engines": { - "node": "^14.18.0 || >=16.0.0" + "node": "^20.19.0 || >=22.12.0" }, "peerDependencies": { - "vite": "^4.2.0 || ^5.0.0 || ^6.0.0 || ^7.0.0" + "@rolldown/plugin-babel": "^0.1.7 || ^0.2.0", + "babel-plugin-react-compiler": "^1.0.0", + "vite": "^8.0.0" + }, + "peerDependenciesMeta": { + "@rolldown/plugin-babel": { + "optional": true + }, + "babel-plugin-react-compiler": { + "optional": true + } } }, "node_modules/@vitest/expect": { @@ -2578,7 +2462,7 @@ "url": "https://opencollective.com/vitest" } }, - "node_modules/@vitest/expect/node_modules/@vitest/pretty-format": { + "node_modules/@vitest/pretty-format": { "version": "3.2.4", "resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-3.2.4.tgz", "integrity": "sha512-IVNZik8IVRJRTr9fxlitMKeJeXFFFN0JaB9PHPGQ8NKQbGpfjlTx9zO4RefN8gp7eqjNy8nyK3NZmBzOPeIxtA==", @@ -2591,7 +2475,7 @@ "url": "https://opencollective.com/vitest" } }, - "node_modules/@vitest/expect/node_modules/@vitest/spy": { + "node_modules/@vitest/spy": { "version": "3.2.4", "resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-3.2.4.tgz", "integrity": "sha512-vAfasCOe6AIK70iP5UD11Ac4siNUNJ9i/9PZ3NKx07sG6sUxeag1LWdNrMWeKKYBLlzuK+Gn65Yd5nyL6ds+nw==", @@ -2604,7 +2488,7 @@ "url": "https://opencollective.com/vitest" } }, - "node_modules/@vitest/expect/node_modules/@vitest/utils": { + "node_modules/@vitest/utils": { "version": "3.2.4", "resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-3.2.4.tgz", "integrity": "sha512-fB2V0JFrQSMsCo9HiSq3Ezpdv4iYaXRG1Sx8edX3MwxfyNn83mKiGzOcH+Fkxt4MHxr3y42fQi1oeAInqgX2QA==", @@ -2619,15 +2503,12 @@ "url": "https://opencollective.com/vitest" } }, - "node_modules/@vitest/expect/node_modules/tinyrainbow": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/tinyrainbow/-/tinyrainbow-2.0.0.tgz", - "integrity": "sha512-op4nsTR47R6p0vMUUoYl/a+ljLFVtlfaXkLQmqfLR1qHma1h/ysYk4hEXZ880bf2CYgTskvTa/e196Vd5dDQXw==", + "node_modules/@webcontainer/env": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/@webcontainer/env/-/env-1.1.1.tgz", + "integrity": "sha512-6aN99yL695Hi9SuIk1oC88l9o0gmxL1nGWWQ/kNy81HigJ0FoaoTXpytCj6ItzgyCEwA9kF1wixsTuv5cjsgng==", "dev": true, - "license": "MIT", - "engines": { - "node": ">=14.0.0" - } + "license": "MIT" }, "node_modules/acorn": { "version": "8.16.0", @@ -2653,9 +2534,9 @@ } }, "node_modules/ajv": { - "version": "6.14.0", - "resolved": "https://registry.npmjs.org/ajv/-/ajv-6.14.0.tgz", - "integrity": "sha512-IWrosm/yrn43eiKqkfkHis7QioDleaXQHdDVPKg0FSwwd/DuvyX79TZnFOnYpB7dcsFAMmtFztZuXPDvSePkFw==", + "version": "6.15.0", + "resolved": "https://registry.npmjs.org/ajv/-/ajv-6.15.0.tgz", + "integrity": "sha512-fgFx7Hfoq60ytK2c7DhnF8jIvzYgOMxfugjLOSMHjLIPgenqa7S7oaagATUq99mV6IYvN2tRmC0wnTYX6iPbMw==", "dev": true, "license": "MIT", "dependencies": { @@ -2715,13 +2596,6 @@ "arrow2csv": "bin/arrow2csv.js" } }, - "node_modules/argparse": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/argparse/-/argparse-2.0.1.tgz", - "integrity": "sha512-8+9WqebbFzpX9OR+Wa6O29asIogeRMzcGtAINdpMHHyAg10f05aSFVBbcEqGf/PXw1EjAZ+q2/bEBg3DvurK3Q==", - "dev": true, - "license": "Python-2.0" - }, "node_modules/aria-query": { "version": "5.3.0", "resolved": "https://registry.npmjs.org/aria-query/-/aria-query-5.3.0.tgz", @@ -2765,16 +2639,19 @@ } }, "node_modules/balanced-match": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz", - "integrity": "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==", + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-4.0.4.tgz", + "integrity": "sha512-BLrgEcRTwX2o6gGxGOCNyMvGSp35YofuYzw9h1IMTRmKqttAZZVU67bdb9Pr2vUHA8+j3i2tJfjO6C6+4myGTA==", "dev": true, - "license": "MIT" + "license": "MIT", + "engines": { + "node": "18 || 20 || >=22" + } }, "node_modules/baseline-browser-mapping": { - "version": "2.10.10", - "resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.10.10.tgz", - "integrity": "sha512-sUoJ3IMxx4AyRqO4MLeHlnGDkyXRoUG0/AI9fjK+vS72ekpV0yWVY7O0BVjmBcRtkNcsAO2QDZ4tdKKGoI6YaQ==", + "version": "2.10.21", + "resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.10.21.tgz", + "integrity": "sha512-Q+rUQ7Uz8AHM7DEaNdwvfFCTq7a43lNTzuS94eiWqwyxfV/wJv+oUivef51T91mmRY4d4A1u9rcSvkeufCVXlA==", "dev": true, "license": "Apache-2.0", "bin": { @@ -2785,20 +2662,22 @@ } }, "node_modules/brace-expansion": { - "version": "1.1.12", - "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.12.tgz", - "integrity": "sha512-9T9UjW3r0UW5c1Q7GTwllptXwhvYmEzFhzMfZ9H7FQWt+uZePjZPjBP/W1ZEyZ1twGWom5/56TF4lPcqjnDHcg==", + "version": "5.0.5", + "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.5.tgz", + "integrity": "sha512-VZznLgtwhn+Mact9tfiwx64fA9erHH/MCXEUfB/0bX/6Fz6ny5EGTXYltMocqg4xFAQZtnO3DHWWXi8RiuN7cQ==", "dev": true, "license": "MIT", "dependencies": { - "balanced-match": "^1.0.0", - "concat-map": "0.0.1" + "balanced-match": "^4.0.2" + }, + "engines": { + "node": "18 || 20 || >=22" } }, "node_modules/browserslist": { - "version": "4.28.1", - "resolved": "https://registry.npmjs.org/browserslist/-/browserslist-4.28.1.tgz", - "integrity": "sha512-ZC5Bd0LgJXgwGqUknZY/vkUQ04r8NXnJZ3yYi4vDmSiZmC/pdSN0NbNRPxZpbtO4uAfDUAFffO8IZoM3Gj8IkA==", + "version": "4.28.2", + "resolved": "https://registry.npmjs.org/browserslist/-/browserslist-4.28.2.tgz", + "integrity": "sha512-48xSriZYYg+8qXna9kwqjIVzuQxi+KYWp2+5nCYnYKPTr0LvD89Jqk2Or5ogxz0NUMfIjhh2lIUX/LyX9B4oIg==", "dev": true, "funding": [ { @@ -2816,11 +2695,11 @@ ], "license": "MIT", "dependencies": { - "baseline-browser-mapping": "^2.9.0", - "caniuse-lite": "^1.0.30001759", - "electron-to-chromium": "^1.5.263", - "node-releases": "^2.0.27", - "update-browserslist-db": "^1.2.0" + "baseline-browser-mapping": "^2.10.12", + "caniuse-lite": "^1.0.30001782", + "electron-to-chromium": "^1.5.328", + "node-releases": "^2.0.36", + "update-browserslist-db": "^1.2.3" }, "bin": { "browserslist": "cli.js" @@ -2845,20 +2724,10 @@ "url": "https://github.com/sponsors/sindresorhus" } }, - "node_modules/callsites": { - "version": "3.1.0", - "resolved": "https://registry.npmjs.org/callsites/-/callsites-3.1.0.tgz", - "integrity": "sha512-P8BjAsXvZS+VIDUI11hHCQEv74YT67YUi5JJFNWIqL235sBmjX4+qx9Muvls5ivyNENctx46xQLQ3aTuE7ssaQ==", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=6" - } - }, "node_modules/caniuse-lite": { - "version": "1.0.30001781", - "resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001781.tgz", - "integrity": "sha512-RdwNCyMsNBftLjW6w01z8bKEvT6e/5tpPVEgtn22TiLGlstHOVecsX2KHFkD5e/vRnIE4EGzpuIODb3mtswtkw==", + "version": "1.0.30001790", + "resolved": "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001790.tgz", + "integrity": "sha512-bOoxfJPyYo+ds6W0YfptaCWbFnJYjh2Y1Eow5lRv+vI2u8ganPZqNm1JwNh0t2ELQCqIWg4B3dWEusgAmsoyOw==", "dev": true, "funding": [ { @@ -2990,13 +2859,6 @@ "node": ">=12.20.0" } }, - "node_modules/concat-map": { - "version": "0.0.1", - "resolved": "https://registry.npmjs.org/concat-map/-/concat-map-0.0.1.tgz", - "integrity": "sha512-/Srv4dswyQNBfohGpz9o6Yb3Gz3SrUDqBH5rTuhGR7ahtlbYKnVxw2bCFMRljaA7EXHaXZ8wsHdodFvbkhKmqg==", - "dev": true, - "license": "MIT" - }, "node_modules/convert-source-map": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/convert-source-map/-/convert-source-map-2.0.0.tgz", @@ -3162,9 +3024,9 @@ "peer": true }, "node_modules/electron-to-chromium": { - "version": "1.5.322", - "resolved": "https://registry.npmjs.org/electron-to-chromium/-/electron-to-chromium-1.5.322.tgz", - "integrity": "sha512-vFU34OcrvMcH66T+dYC3G4nURmgfDVewMIu6Q2urXpumAPSMmzvcn04KVVV8Opikq8Vs5nUbO/8laNhNRqSzYw==", + "version": "1.5.343", + "resolved": "https://registry.npmjs.org/electron-to-chromium/-/electron-to-chromium-1.5.343.tgz", + "integrity": "sha512-YHnQ3MXI08icvL9ZKnEBy05F2EQ8ob01UaMOuMbM8l+4UcAq6MPPbBTJBbsBUg3H8JeZNt+O4fjsoWth3p6IFg==", "dev": true, "license": "ISC" }, @@ -3192,10 +3054,20 @@ "node": ">=10.13.0" } }, + "node_modules/es-errors": { + "version": "1.3.0", + "resolved": "https://registry.npmjs.org/es-errors/-/es-errors-1.3.0.tgz", + "integrity": "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 0.4" + } + }, "node_modules/esbuild": { - "version": "0.25.12", - "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.25.12.tgz", - "integrity": "sha512-bbPBYYrtZbkt6Os6FiTLCTFxvq4tt3JKall1vRwshA3fdVztsLAatFaZobhkBC8/BrPetoa0oksYoKXoG4ryJg==", + "version": "0.27.7", + "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.27.7.tgz", + "integrity": "sha512-IxpibTjyVnmrIQo5aqNpCgoACA/dTKLTlhMHihVHhdkxKyPO1uBBthumT0rdHmcsk9uMonIWS0m4FljWzILh3w==", "dev": true, "hasInstallScript": true, "license": "MIT", @@ -3206,32 +3078,32 @@ "node": ">=18" }, "optionalDependencies": { - "@esbuild/aix-ppc64": "0.25.12", - "@esbuild/android-arm": "0.25.12", - "@esbuild/android-arm64": "0.25.12", - "@esbuild/android-x64": "0.25.12", - "@esbuild/darwin-arm64": "0.25.12", - "@esbuild/darwin-x64": "0.25.12", - "@esbuild/freebsd-arm64": "0.25.12", - "@esbuild/freebsd-x64": "0.25.12", - "@esbuild/linux-arm": "0.25.12", - "@esbuild/linux-arm64": "0.25.12", - "@esbuild/linux-ia32": "0.25.12", - "@esbuild/linux-loong64": "0.25.12", - "@esbuild/linux-mips64el": "0.25.12", - "@esbuild/linux-ppc64": "0.25.12", - "@esbuild/linux-riscv64": "0.25.12", - "@esbuild/linux-s390x": "0.25.12", - "@esbuild/linux-x64": "0.25.12", - "@esbuild/netbsd-arm64": "0.25.12", - "@esbuild/netbsd-x64": "0.25.12", - "@esbuild/openbsd-arm64": "0.25.12", - "@esbuild/openbsd-x64": "0.25.12", - "@esbuild/openharmony-arm64": "0.25.12", - "@esbuild/sunos-x64": "0.25.12", - "@esbuild/win32-arm64": "0.25.12", - "@esbuild/win32-ia32": "0.25.12", - "@esbuild/win32-x64": "0.25.12" + "@esbuild/aix-ppc64": "0.27.7", + "@esbuild/android-arm": "0.27.7", + "@esbuild/android-arm64": "0.27.7", + "@esbuild/android-x64": "0.27.7", + "@esbuild/darwin-arm64": "0.27.7", + "@esbuild/darwin-x64": "0.27.7", + "@esbuild/freebsd-arm64": "0.27.7", + "@esbuild/freebsd-x64": "0.27.7", + "@esbuild/linux-arm": "0.27.7", + "@esbuild/linux-arm64": "0.27.7", + "@esbuild/linux-ia32": "0.27.7", + "@esbuild/linux-loong64": "0.27.7", + "@esbuild/linux-mips64el": "0.27.7", + "@esbuild/linux-ppc64": "0.27.7", + "@esbuild/linux-riscv64": "0.27.7", + "@esbuild/linux-s390x": "0.27.7", + "@esbuild/linux-x64": "0.27.7", + "@esbuild/netbsd-arm64": "0.27.7", + "@esbuild/netbsd-x64": "0.27.7", + "@esbuild/openbsd-arm64": "0.27.7", + "@esbuild/openbsd-x64": "0.27.7", + "@esbuild/openharmony-arm64": "0.27.7", + "@esbuild/sunos-x64": "0.27.7", + "@esbuild/win32-arm64": "0.27.7", + "@esbuild/win32-ia32": "0.27.7", + "@esbuild/win32-x64": "0.27.7" } }, "node_modules/escalade": { @@ -3258,33 +3130,30 @@ } }, "node_modules/eslint": { - "version": "9.39.4", - "resolved": "https://registry.npmjs.org/eslint/-/eslint-9.39.4.tgz", - "integrity": "sha512-XoMjdBOwe/esVgEvLmNsD3IRHkm7fbKIUGvrleloJXUZgDHig2IPWNniv+GwjyJXzuNqVjlr5+4yVUZjycJwfQ==", + "version": "10.2.1", + "resolved": "https://registry.npmjs.org/eslint/-/eslint-10.2.1.tgz", + "integrity": "sha512-wiyGaKsDgqXvF40P8mDwiUp/KQjE1FdrIEJsM8PZ3XCiniTMXS3OHWWUe5FI5agoCnr8x4xPrTDZuxsBlNHl+Q==", "dev": true, "license": "MIT", "dependencies": { "@eslint-community/eslint-utils": "^4.8.0", - "@eslint-community/regexpp": "^4.12.1", - "@eslint/config-array": "^0.21.2", - "@eslint/config-helpers": "^0.4.2", - "@eslint/core": "^0.17.0", - "@eslint/eslintrc": "^3.3.5", - "@eslint/js": "9.39.4", - "@eslint/plugin-kit": "^0.4.1", + "@eslint-community/regexpp": "^4.12.2", + "@eslint/config-array": "^0.23.5", + "@eslint/config-helpers": "^0.5.5", + "@eslint/core": "^1.2.1", + "@eslint/plugin-kit": "^0.7.1", "@humanfs/node": "^0.16.6", "@humanwhocodes/module-importer": "^1.0.1", "@humanwhocodes/retry": "^0.4.2", "@types/estree": "^1.0.6", "ajv": "^6.14.0", - "chalk": "^4.0.0", "cross-spawn": "^7.0.6", "debug": "^4.3.2", "escape-string-regexp": "^4.0.0", - "eslint-scope": "^8.4.0", - "eslint-visitor-keys": "^4.2.1", - "espree": "^10.4.0", - "esquery": "^1.5.0", + "eslint-scope": "^9.1.2", + "eslint-visitor-keys": "^5.0.1", + "espree": "^11.2.0", + "esquery": "^1.7.0", "esutils": "^2.0.2", "fast-deep-equal": "^3.1.3", "file-entry-cache": "^8.0.0", @@ -3294,8 +3163,7 @@ "imurmurhash": "^0.1.4", "is-glob": "^4.0.0", "json-stable-stringify-without-jsonify": "^1.0.1", - "lodash.merge": "^4.6.2", - "minimatch": "^3.1.5", + "minimatch": "^10.2.4", "natural-compare": "^1.4.0", "optionator": "^0.9.3" }, @@ -3303,7 +3171,7 @@ "eslint": "bin/eslint.js" }, "engines": { - "node": "^18.18.0 || ^20.9.0 || >=21.1.0" + "node": "^20.19.0 || ^22.13.0 || >=24" }, "funding": { "url": "https://eslint.org/donate" @@ -3334,32 +3202,39 @@ } }, "node_modules/eslint-plugin-react-hooks": { - "version": "5.2.0", - "resolved": "https://registry.npmjs.org/eslint-plugin-react-hooks/-/eslint-plugin-react-hooks-5.2.0.tgz", - "integrity": "sha512-+f15FfK64YQwZdJNELETdn5ibXEUQmW1DZL6KXhNnc2heoy/sg9VJJeT7n8TlMWouzWqSWavFkIhHyIbIAEapg==", + "version": "7.1.1", + "resolved": "https://registry.npmjs.org/eslint-plugin-react-hooks/-/eslint-plugin-react-hooks-7.1.1.tgz", + "integrity": "sha512-f2I7Gw6JbvCexzIInuSbZpfdQ44D7iqdWX01FKLvrPgqxoE7oMj8clOfto8U6vYiz4yd5oKu39rRSVOe1zRu0g==", "dev": true, "license": "MIT", + "dependencies": { + "@babel/core": "^7.24.4", + "@babel/parser": "^7.24.4", + "hermes-parser": "^0.25.1", + "zod": "^3.25.0 || ^4.0.0", + "zod-validation-error": "^3.5.0 || ^4.0.0" + }, "engines": { - "node": ">=10" + "node": ">=18" }, "peerDependencies": { - "eslint": "^3.0.0 || ^4.0.0 || ^5.0.0 || ^6.0.0 || ^7.0.0 || ^8.0.0-0 || ^9.0.0" + "eslint": "^3.0.0 || ^4.0.0 || ^5.0.0 || ^6.0.0 || ^7.0.0 || ^8.0.0-0 || ^9.0.0 || ^10.0.0" } }, "node_modules/eslint-plugin-react-refresh": { - "version": "0.4.26", - "resolved": "https://registry.npmjs.org/eslint-plugin-react-refresh/-/eslint-plugin-react-refresh-0.4.26.tgz", - "integrity": "sha512-1RETEylht2O6FM/MvgnyvT+8K21wLqDNg4qD51Zj3guhjt433XbnnkVttHMyaVyAFD03QSV4LPS5iE3VQmO7XQ==", + "version": "0.5.2", + "resolved": "https://registry.npmjs.org/eslint-plugin-react-refresh/-/eslint-plugin-react-refresh-0.5.2.tgz", + "integrity": "sha512-hmgTH57GfzoTFjVN0yBwTggnsVUF2tcqi7RJZHqi9lIezSs4eFyAMktA68YD4r5kNw1mxyY4dmkyoFDb3FIqrA==", "dev": true, "license": "MIT", "peerDependencies": { - "eslint": ">=8.40" + "eslint": "^9 || ^10" } }, "node_modules/eslint-plugin-storybook": { - "version": "10.3.3", - "resolved": "https://registry.npmjs.org/eslint-plugin-storybook/-/eslint-plugin-storybook-10.3.3.tgz", - "integrity": "sha512-jo8wZvKaJlxxrNvf4hCsROJP3CdlpaLiYewAs5Ww+PJxCrLelIi5XVHWOAgBvvr3H9WDKvUw8xuvqPYqAlpkFg==", + "version": "10.3.5", + "resolved": "https://registry.npmjs.org/eslint-plugin-storybook/-/eslint-plugin-storybook-10.3.5.tgz", + "integrity": "sha512-rEFkfU3ypF44GpB4tiJ9EFDItueoGvGi3+weLHZax2ON2MB7VIDsxdSUGvIU5tMURg+oWYlpzCyLm4TpDq2deA==", "dev": true, "license": "MIT", "dependencies": { @@ -3367,52 +3242,54 @@ }, "peerDependencies": { "eslint": ">=8", - "storybook": "^10.3.3" + "storybook": "^10.3.5" } }, "node_modules/eslint-scope": { - "version": "8.4.0", - "resolved": "https://registry.npmjs.org/eslint-scope/-/eslint-scope-8.4.0.tgz", - "integrity": "sha512-sNXOfKCn74rt8RICKMvJS7XKV/Xk9kA7DyJr8mJik3S7Cwgy3qlkkmyS2uQB3jiJg6VNdZd/pDBJu0nvG2NlTg==", + "version": "9.1.2", + "resolved": "https://registry.npmjs.org/eslint-scope/-/eslint-scope-9.1.2.tgz", + "integrity": "sha512-xS90H51cKw0jltxmvmHy2Iai1LIqrfbw57b79w/J7MfvDfkIkFZ+kj6zC3BjtUwh150HsSSdxXZcsuv72miDFQ==", "dev": true, "license": "BSD-2-Clause", "dependencies": { + "@types/esrecurse": "^4.3.1", + "@types/estree": "^1.0.8", "esrecurse": "^4.3.0", "estraverse": "^5.2.0" }, "engines": { - "node": "^18.18.0 || ^20.9.0 || >=21.1.0" + "node": "^20.19.0 || ^22.13.0 || >=24" }, "funding": { "url": "https://opencollective.com/eslint" } }, "node_modules/eslint-visitor-keys": { - "version": "4.2.1", - "resolved": "https://registry.npmjs.org/eslint-visitor-keys/-/eslint-visitor-keys-4.2.1.tgz", - "integrity": "sha512-Uhdk5sfqcee/9H/rCOJikYz67o0a2Tw2hGRPOG2Y1R2dg7brRe1uG0yaNQDHu+TO/uQPF/5eCapvYSmHUjt7JQ==", + "version": "5.0.1", + "resolved": "https://registry.npmjs.org/eslint-visitor-keys/-/eslint-visitor-keys-5.0.1.tgz", + "integrity": "sha512-tD40eHxA35h0PEIZNeIjkHoDR4YjjJp34biM0mDvplBe//mB+IHCqHDGV7pxF+7MklTvighcCPPZC7ynWyjdTA==", "dev": true, "license": "Apache-2.0", "engines": { - "node": "^18.18.0 || ^20.9.0 || >=21.1.0" + "node": "^20.19.0 || ^22.13.0 || >=24" }, "funding": { "url": "https://opencollective.com/eslint" } }, "node_modules/espree": { - "version": "10.4.0", - "resolved": "https://registry.npmjs.org/espree/-/espree-10.4.0.tgz", - "integrity": "sha512-j6PAQ2uUr79PZhBjP5C5fhl8e39FmRnOjsD5lGnWrFU8i2G776tBK7+nP8KuQUTTyAZUwfQqXAgrVH5MbH9CYQ==", + "version": "11.2.0", + "resolved": "https://registry.npmjs.org/espree/-/espree-11.2.0.tgz", + "integrity": "sha512-7p3DrVEIopW1B1avAGLuCSh1jubc01H2JHc8B4qqGblmg5gI9yumBgACjWo4JlIc04ufug4xJ3SQI8HkS/Rgzw==", "dev": true, "license": "BSD-2-Clause", "dependencies": { - "acorn": "^8.15.0", + "acorn": "^8.16.0", "acorn-jsx": "^5.3.2", - "eslint-visitor-keys": "^4.2.1" + "eslint-visitor-keys": "^5.0.1" }, "engines": { - "node": "^18.18.0 || ^20.9.0 || >=21.1.0" + "node": "^20.19.0 || ^22.13.0 || >=24" }, "funding": { "url": "https://opencollective.com/eslint" @@ -3664,49 +3541,10 @@ "node": ">=10.13.0" } }, - "node_modules/glob/node_modules/balanced-match": { - "version": "4.0.4", - "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-4.0.4.tgz", - "integrity": "sha512-BLrgEcRTwX2o6gGxGOCNyMvGSp35YofuYzw9h1IMTRmKqttAZZVU67bdb9Pr2vUHA8+j3i2tJfjO6C6+4myGTA==", - "dev": true, - "license": "MIT", - "engines": { - "node": "18 || 20 || >=22" - } - }, - "node_modules/glob/node_modules/brace-expansion": { - "version": "5.0.5", - "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.5.tgz", - "integrity": "sha512-VZznLgtwhn+Mact9tfiwx64fA9erHH/MCXEUfB/0bX/6Fz6ny5EGTXYltMocqg4xFAQZtnO3DHWWXi8RiuN7cQ==", - "dev": true, - "license": "MIT", - "dependencies": { - "balanced-match": "^4.0.2" - }, - "engines": { - "node": "18 || 20 || >=22" - } - }, - "node_modules/glob/node_modules/minimatch": { - "version": "10.2.4", - "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-10.2.4.tgz", - "integrity": "sha512-oRjTw/97aTBN0RHbYCdtF1MQfvusSIBQM0IZEgzl6426+8jSC0nF1a/GmnVLpfB9yyr6g6FTqWqiZVbxrtaCIg==", - "dev": true, - "license": "BlueOak-1.0.0", - "dependencies": { - "brace-expansion": "^5.0.2" - }, - "engines": { - "node": "18 || 20 || >=22" - }, - "funding": { - "url": "https://github.com/sponsors/isaacs" - } - }, "node_modules/globals": { - "version": "16.5.0", - "resolved": "https://registry.npmjs.org/globals/-/globals-16.5.0.tgz", - "integrity": "sha512-c/c15i26VrJ4IRt5Z89DnIzCGDn9EcebibhAOjw5ibqEHsE1wLUgkPn9RDmNcUKyU87GeaL633nyJ+pplFR2ZQ==", + "version": "17.5.0", + "resolved": "https://registry.npmjs.org/globals/-/globals-17.5.0.tgz", + "integrity": "sha512-qoV+HK2yFl/366t2/Cb3+xxPUo5BuMynomoDmiaZBIdbs+0pYbjfZU+twLhGKp4uCZ/+NbtpVepH5bGCxRyy2g==", "dev": true, "license": "MIT", "engines": { @@ -3733,9 +3571,9 @@ } }, "node_modules/hasown": { - "version": "2.0.2", - "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.2.tgz", - "integrity": "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ==", + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.3.tgz", + "integrity": "sha512-ej4AhfhfL2Q2zpMmLo7U1Uv9+PyhIZpgQLGT1F9miIGmiCJIoCgSmczFdrc97mWT4kVY72KA+WnnhJ5pghSvSg==", "dev": true, "license": "MIT", "dependencies": { @@ -3745,6 +3583,23 @@ "node": ">= 0.4" } }, + "node_modules/hermes-estree": { + "version": "0.25.1", + "resolved": "https://registry.npmjs.org/hermes-estree/-/hermes-estree-0.25.1.tgz", + "integrity": "sha512-0wUoCcLp+5Ev5pDW2OriHC2MJCbwLwuRx+gAqMTOkGKJJiBCLjtrvy4PWUGn6MIVefecRpzoOZ/UV6iGdOr+Cw==", + "dev": true, + "license": "MIT" + }, + "node_modules/hermes-parser": { + "version": "0.25.1", + "resolved": "https://registry.npmjs.org/hermes-parser/-/hermes-parser-0.25.1.tgz", + "integrity": "sha512-6pEjquH3rqaI6cYAXYPcz9MS4rY6R4ngRgrgfDshRptUZIc3lw0MCIJIGDj9++mfySOuPTHB4nrSW99BCvOPIA==", + "dev": true, + "license": "MIT", + "dependencies": { + "hermes-estree": "0.25.1" + } + }, "node_modules/ignore": { "version": "5.3.2", "resolved": "https://registry.npmjs.org/ignore/-/ignore-5.3.2.tgz", @@ -3755,23 +3610,6 @@ "node": ">= 4" } }, - "node_modules/import-fresh": { - "version": "3.3.1", - "resolved": "https://registry.npmjs.org/import-fresh/-/import-fresh-3.3.1.tgz", - "integrity": "sha512-TR3KfrTZTYLPB6jUjfx6MF9WcWrHL9su5TObK4ZkYgBdWKPOFoSoQIdEuTuR82pmtxH2spWG9h6etwfr1pLBqQ==", - "dev": true, - "license": "MIT", - "dependencies": { - "parent-module": "^1.0.0", - "resolve-from": "^4.0.0" - }, - "engines": { - "node": ">=6" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, "node_modules/imurmurhash": { "version": "0.1.4", "resolved": "https://registry.npmjs.org/imurmurhash/-/imurmurhash-0.1.4.tgz", @@ -3906,19 +3744,6 @@ "dev": true, "license": "MIT" }, - "node_modules/js-yaml": { - "version": "4.1.1", - "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-4.1.1.tgz", - "integrity": "sha512-qQKT4zQxXl8lLwBtHMWwaTcGfFOZviOJet3Oy/xmGk2gZH677CJM9EvtfdSkgWcATZhj/55JZ0rmy3myCT5lsA==", - "dev": true, - "license": "MIT", - "dependencies": { - "argparse": "^2.0.1" - }, - "bin": { - "js-yaml": "bin/js-yaml.js" - } - }, "node_modules/jsesc": { "version": "3.1.0", "resolved": "https://registry.npmjs.org/jsesc/-/jsesc-3.1.0.tgz", @@ -4281,13 +4106,6 @@ "integrity": "sha512-TwuEnCnxbc3rAvhf/LbG7tJUDzhqXyFnv3dtzLOPgCG/hODL7WFnsbwktkD7yUV0RrreP/l1PALq/YSg6VvjlA==", "license": "MIT" }, - "node_modules/lodash.merge": { - "version": "4.6.2", - "resolved": "https://registry.npmjs.org/lodash.merge/-/lodash.merge-4.6.2.tgz", - "integrity": "sha512-0KpjqXRVvrYyCsX1swR/XTK0va6VQkQM6MNo7PqW77ByjAhoARA8EfrP1N4+KlKj8YS0ZUCtRT/YUuhyYDujIQ==", - "dev": true, - "license": "MIT" - }, "node_modules/loupe": { "version": "3.2.1", "resolved": "https://registry.npmjs.org/loupe/-/loupe-3.2.1.tgz", @@ -4337,16 +4155,19 @@ } }, "node_modules/minimatch": { - "version": "3.1.5", - "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-3.1.5.tgz", - "integrity": "sha512-VgjWUsnnT6n+NUk6eZq77zeFdpW2LWDzP6zFGrCbHXiYNul5Dzqk2HHQ5uFH2DNW5Xbp8+jVzaeNt94ssEEl4w==", + "version": "10.2.5", + "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-10.2.5.tgz", + "integrity": "sha512-MULkVLfKGYDFYejP07QOurDLLQpcjk7Fw+7jXS2R2czRQzR56yHRveU5NDJEOviH+hETZKSkIk5c+T23GjFUMg==", "dev": true, - "license": "ISC", + "license": "BlueOak-1.0.0", "dependencies": { - "brace-expansion": "^1.1.7" + "brace-expansion": "^5.0.5" }, "engines": { - "node": "*" + "node": "18 || 20 || >=22" + }, + "funding": { + "url": "https://github.com/sponsors/isaacs" } }, "node_modules/minimist": { @@ -4403,9 +4224,9 @@ "license": "MIT" }, "node_modules/node-releases": { - "version": "2.0.36", - "resolved": "https://registry.npmjs.org/node-releases/-/node-releases-2.0.36.tgz", - "integrity": "sha512-TdC8FSgHz8Mwtw9g5L4gR/Sh9XhSP/0DEkQxfEFXOpiul5IiHgHan2VhYYb6agDSfp4KuvltmGApc8HMgUrIkA==", + "version": "2.0.38", + "resolved": "https://registry.npmjs.org/node-releases/-/node-releases-2.0.38.tgz", + "integrity": "sha512-3qT/88Y3FbH/Kx4szpQQ4HzUbVrHPKTLVpVocKiLfoYvw9XSGOX2FmD2d6DrXbVYyAQTF2HeF6My8jmzx7/CRw==", "dev": true, "license": "MIT" }, @@ -4478,19 +4299,6 @@ "url": "https://github.com/sponsors/sindresorhus" } }, - "node_modules/parent-module": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/parent-module/-/parent-module-1.0.1.tgz", - "integrity": "sha512-GQ2EWRpQV8/o+Aw8YqtfZZPfNRWZYkbidE9k5rpl/hC3vtHHBfGm2Ifi6qWV+coDGkrUKZAxE3Lot5kcsRlh+g==", - "dev": true, - "license": "MIT", - "dependencies": { - "callsites": "^3.0.0" - }, - "engines": { - "node": ">=6" - } - }, "node_modules/path-exists": { "version": "4.0.0", "resolved": "https://registry.npmjs.org/path-exists/-/path-exists-4.0.0.tgz", @@ -4536,9 +4344,9 @@ } }, "node_modules/path-scurry/node_modules/lru-cache": { - "version": "11.2.7", - "resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-11.2.7.tgz", - "integrity": "sha512-aY/R+aEsRelme17KGQa/1ZSIpLpNYYrhcrepKTZgE+W3WM16YMCaPwOHLHsmopZHELU0Ojin1lPVxKR0MihncA==", + "version": "11.3.5", + "resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-11.3.5.tgz", + "integrity": "sha512-NxVFwLAnrd9i7KUBxC4DrUhmgjzOs+1Qm50D3oF1/oL+r1NpZ4gA7xvG0/zJ8evR7zIKn4vLf7qTNduWFtCrRw==", "dev": true, "license": "BlueOak-1.0.0", "engines": { @@ -4576,9 +4384,9 @@ } }, "node_modules/postcss": { - "version": "8.5.8", - "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.8.tgz", - "integrity": "sha512-OW/rX8O/jXnm82Ey1k44pObPtdblfiuWnrd8X7GJ7emImCOstunGbXUpp7HdBrFQX6rJzn3sPT397Wp5aCwCHg==", + "version": "8.5.10", + "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.10.tgz", + "integrity": "sha512-pMMHxBOZKFU6HgAZ4eyGnwXF/EvPGGqUr0MnZ5+99485wwW41kW91A4LOGxSHhgugZmSChL5AlElNdwlNgcnLQ==", "dev": true, "funding": [ { @@ -4615,9 +4423,9 @@ } }, "node_modules/prettier": { - "version": "3.8.1", - "resolved": "https://registry.npmjs.org/prettier/-/prettier-3.8.1.tgz", - "integrity": "sha512-UOnG6LftzbdaHZcKoPFtOcCKztrQ57WkHDeRD9t/PTQtmT0NHSeWWepj6pS0z/N7+08BHFDQVUrfmfMRcZwbMg==", + "version": "3.8.3", + "resolved": "https://registry.npmjs.org/prettier/-/prettier-3.8.3.tgz", + "integrity": "sha512-7igPTM53cGHMW8xWuVTydi2KO233VFiTNyF5hLJqpilHfmn8C8gPf+PS7dUT64YcXFbiMGZxS9pCSxL/Dxm/Jw==", "dev": true, "license": "MIT", "bin": { @@ -4671,9 +4479,9 @@ } }, "node_modules/react": { - "version": "19.2.4", - "resolved": "https://registry.npmjs.org/react/-/react-19.2.4.tgz", - "integrity": "sha512-9nfp2hYpCwOjAN+8TZFGhtWEwgvWHXqESH8qT89AT/lWklpLON22Lc8pEtnpsZz7VmawabSU0gCjnj8aC0euHQ==", + "version": "19.2.5", + "resolved": "https://registry.npmjs.org/react/-/react-19.2.5.tgz", + "integrity": "sha512-llUJLzz1zTUBrskt2pwZgLq59AemifIftw4aB7JxOqf1HY2FDaGDxgwpAPVzHU1kdWabH7FauP4i1oEeer2WCA==", "license": "MIT", "engines": { "node": ">=0.10.0" @@ -4712,15 +4520,15 @@ } }, "node_modules/react-dom": { - "version": "19.2.4", - "resolved": "https://registry.npmjs.org/react-dom/-/react-dom-19.2.4.tgz", - "integrity": "sha512-AXJdLo8kgMbimY95O2aKQqsz2iWi9jMgKJhRBAxECE4IFxfcazB2LmzloIoibJI3C12IlY20+KFaLv+71bUJeQ==", + "version": "19.2.5", + "resolved": "https://registry.npmjs.org/react-dom/-/react-dom-19.2.5.tgz", + "integrity": "sha512-J5bAZz+DXMMwW/wV3xzKke59Af6CHY7G4uYLN1OvBcKEsWOs4pQExj86BBKamxl/Ik5bx9whOrvBlSDfWzgSag==", "license": "MIT", "dependencies": { "scheduler": "^0.27.0" }, "peerDependencies": { - "react": "^19.2.4" + "react": "^19.2.5" } }, "node_modules/react-is": { @@ -4731,16 +4539,6 @@ "license": "MIT", "peer": true }, - "node_modules/react-refresh": { - "version": "0.17.0", - "resolved": "https://registry.npmjs.org/react-refresh/-/react-refresh-0.17.0.tgz", - "integrity": "sha512-z6F7K9bV85EfseRCp2bzrpyQ0Gkw1uLoCel9XBVWPg/TjRj94SkJzUTGfOa4bs7iJvBWtQG0Wq7wnI0syw3EBQ==", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=0.10.0" - } - }, "node_modules/recast": { "version": "0.23.11", "resolved": "https://registry.npmjs.org/recast/-/recast-0.23.11.tgz", @@ -4786,12 +4584,13 @@ } }, "node_modules/resolve": { - "version": "1.22.11", - "resolved": "https://registry.npmjs.org/resolve/-/resolve-1.22.11.tgz", - "integrity": "sha512-RfqAvLnMl313r7c9oclB1HhUEAezcpLjz95wFH4LVuhk9JF/r22qmVP9AMmOU4vMX7Q8pN8jwNg/CSpdFnMjTQ==", + "version": "1.22.12", + "resolved": "https://registry.npmjs.org/resolve/-/resolve-1.22.12.tgz", + "integrity": "sha512-TyeJ1zif53BPfHootBGwPRYT1RUt6oGWsaQr8UyZW/eAm9bKoijtvruSDEmZHm92CwS9nj7/fWttqPCgzep8CA==", "dev": true, "license": "MIT", "dependencies": { + "es-errors": "^1.3.0", "is-core-module": "^2.16.1", "path-parse": "^1.0.7", "supports-preserve-symlinks-flag": "^1.0.0" @@ -4806,60 +4605,46 @@ "url": "https://github.com/sponsors/ljharb" } }, - "node_modules/resolve-from": { - "version": "4.0.0", - "resolved": "https://registry.npmjs.org/resolve-from/-/resolve-from-4.0.0.tgz", - "integrity": "sha512-pb/MYmXstAkysRFx8piNI1tGFNQIFA3vkE3Gq4EuA1dF6gHp/+vgZqsCGJapvy8N3Q+4o7FwvquPJcnZ7RYy4g==", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=4" - } - }, - "node_modules/rollup": { - "version": "4.60.0", - "resolved": "https://registry.npmjs.org/rollup/-/rollup-4.60.0.tgz", - "integrity": "sha512-yqjxruMGBQJ2gG4HtjZtAfXArHomazDHoFwFFmZZl0r7Pdo7qCIXKqKHZc8yeoMgzJJ+pO6pEEHa+V7uzWlrAQ==", + "node_modules/rolldown": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.0.0-rc.17.tgz", + "integrity": "sha512-ZrT53oAKrtA4+YtBWPQbtPOxIbVDbxT0orcYERKd63VJTF13zPcgXTvD4843L8pcsI7M6MErt8QtON6lrB9tyA==", "dev": true, "license": "MIT", "dependencies": { - "@types/estree": "1.0.8" + "@oxc-project/types": "=0.127.0", + "@rolldown/pluginutils": "1.0.0-rc.17" }, "bin": { - "rollup": "dist/bin/rollup" + "rolldown": "bin/cli.mjs" }, "engines": { - "node": ">=18.0.0", - "npm": ">=8.0.0" + "node": "^20.19.0 || >=22.12.0" }, "optionalDependencies": { - "@rollup/rollup-android-arm-eabi": "4.60.0", - "@rollup/rollup-android-arm64": "4.60.0", - "@rollup/rollup-darwin-arm64": "4.60.0", - "@rollup/rollup-darwin-x64": "4.60.0", - "@rollup/rollup-freebsd-arm64": "4.60.0", - "@rollup/rollup-freebsd-x64": "4.60.0", - "@rollup/rollup-linux-arm-gnueabihf": "4.60.0", - "@rollup/rollup-linux-arm-musleabihf": "4.60.0", - "@rollup/rollup-linux-arm64-gnu": "4.60.0", - "@rollup/rollup-linux-arm64-musl": "4.60.0", - "@rollup/rollup-linux-loong64-gnu": "4.60.0", - "@rollup/rollup-linux-loong64-musl": "4.60.0", - "@rollup/rollup-linux-ppc64-gnu": "4.60.0", - "@rollup/rollup-linux-ppc64-musl": "4.60.0", - "@rollup/rollup-linux-riscv64-gnu": "4.60.0", - "@rollup/rollup-linux-riscv64-musl": "4.60.0", - "@rollup/rollup-linux-s390x-gnu": "4.60.0", - "@rollup/rollup-linux-x64-gnu": "4.60.0", - "@rollup/rollup-linux-x64-musl": "4.60.0", - "@rollup/rollup-openbsd-x64": "4.60.0", - "@rollup/rollup-openharmony-arm64": "4.60.0", - "@rollup/rollup-win32-arm64-msvc": "4.60.0", - "@rollup/rollup-win32-ia32-msvc": "4.60.0", - "@rollup/rollup-win32-x64-gnu": "4.60.0", - "@rollup/rollup-win32-x64-msvc": "4.60.0", - "fsevents": "~2.3.2" - } + "@rolldown/binding-android-arm64": "1.0.0-rc.17", + "@rolldown/binding-darwin-arm64": "1.0.0-rc.17", + "@rolldown/binding-darwin-x64": "1.0.0-rc.17", + "@rolldown/binding-freebsd-x64": "1.0.0-rc.17", + "@rolldown/binding-linux-arm-gnueabihf": "1.0.0-rc.17", + "@rolldown/binding-linux-arm64-gnu": "1.0.0-rc.17", + "@rolldown/binding-linux-arm64-musl": "1.0.0-rc.17", + "@rolldown/binding-linux-ppc64-gnu": "1.0.0-rc.17", + "@rolldown/binding-linux-s390x-gnu": "1.0.0-rc.17", + "@rolldown/binding-linux-x64-gnu": "1.0.0-rc.17", + "@rolldown/binding-linux-x64-musl": "1.0.0-rc.17", + "@rolldown/binding-openharmony-arm64": "1.0.0-rc.17", + "@rolldown/binding-wasm32-wasi": "1.0.0-rc.17", + "@rolldown/binding-win32-arm64-msvc": "1.0.0-rc.17", + "@rolldown/binding-win32-x64-msvc": "1.0.0-rc.17" + } + }, + "node_modules/rolldown/node_modules/@rolldown/pluginutils": { + "version": "1.0.0-rc.17", + "resolved": "https://registry.npmjs.org/@rolldown/pluginutils/-/pluginutils-1.0.0-rc.17.tgz", + "integrity": "sha512-n8iosDOt6Ig1UhJ2AYqoIhHWh/isz0xpicHTzpKBeotdVsTEcxsSA/i3EVM7gQAj0rU27OLAxCjzlj15IWY7bg==", + "dev": true, + "license": "MIT" }, "node_modules/run-applescript": { "version": "7.1.0", @@ -4934,9 +4719,9 @@ } }, "node_modules/storybook": { - "version": "10.3.3", - "resolved": "https://registry.npmjs.org/storybook/-/storybook-10.3.3.tgz", - "integrity": "sha512-tMoRAts9EVqf+mEMPLC6z1DPyHbcPe+CV1MhLN55IKsl0HxNjvVGK44rVPSePbltPE6vIsn4bdRj6CCUt8SJwQ==", + "version": "10.3.5", + "resolved": "https://registry.npmjs.org/storybook/-/storybook-10.3.5.tgz", + "integrity": "sha512-uBSZu/GZa9aEIW3QMGvdQPMZWhGxSe4dyRWU8B3/Vd47Gy/XLC7tsBxRr13txmmPOEDHZR94uLuq0H50fvuqBw==", "dev": true, "license": "MIT", "dependencies": { @@ -4946,6 +4731,7 @@ "@testing-library/user-event": "^14.6.1", "@vitest/expect": "3.2.4", "@vitest/spy": "3.2.4", + "@webcontainer/env": "^1.1.1", "esbuild": "^0.18.0 || ^0.19.0 || ^0.20.0 || ^0.21.0 || ^0.22.0 || ^0.23.0 || ^0.24.0 || ^0.25.0 || ^0.26.0 || ^0.27.0", "open": "^10.2.0", "recast": "^0.23.5", @@ -4969,19 +4755,6 @@ } } }, - "node_modules/storybook/node_modules/@vitest/spy": { - "version": "3.2.4", - "resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-3.2.4.tgz", - "integrity": "sha512-vAfasCOe6AIK70iP5UD11Ac4siNUNJ9i/9PZ3NKx07sG6sUxeag1LWdNrMWeKKYBLlzuK+Gn65Yd5nyL6ds+nw==", - "dev": true, - "license": "MIT", - "dependencies": { - "tinyspy": "^4.0.3" - }, - "funding": { - "url": "https://opencollective.com/vitest" - } - }, "node_modules/storybook/node_modules/semver": { "version": "7.7.4", "resolved": "https://registry.npmjs.org/semver/-/semver-7.7.4.tgz", @@ -5018,19 +4791,6 @@ "url": "https://github.com/sponsors/sindresorhus" } }, - "node_modules/strip-json-comments": { - "version": "3.1.1", - "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-3.1.1.tgz", - "integrity": "sha512-6fPc+R4ihwqP6N/aIv2f1gMH8lOVtWQHoqC4yK6oSDVVocumAsfCqjkXnqiYMhmMwS/mEHLp7Vehlt3ql6lEig==", - "dev": true, - "license": "MIT", - "engines": { - "node": ">=8" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, "node_modules/supports-color": { "version": "7.2.0", "resolved": "https://registry.npmjs.org/supports-color/-/supports-color-7.2.0.tgz", @@ -5070,16 +4830,16 @@ } }, "node_modules/tailwindcss": { - "version": "4.2.2", - "resolved": "https://registry.npmjs.org/tailwindcss/-/tailwindcss-4.2.2.tgz", - "integrity": "sha512-KWBIxs1Xb6NoLdMVqhbhgwZf2PGBpPEiwOqgI4pFIYbNTfBXiKYyWoTsXgBQ9WFg/OlhnvHaY+AEpW7wSmFo2Q==", + "version": "4.2.4", + "resolved": "https://registry.npmjs.org/tailwindcss/-/tailwindcss-4.2.4.tgz", + "integrity": "sha512-HhKppgO81FQof5m6TEnuBWCZGgfRAWbaeOaGT00KOy/Pf/j6oUihdvBpA7ltCeAvZpFhW3j0PTclkxsd4IXYDA==", "dev": true, "license": "MIT" }, "node_modules/tapable": { - "version": "2.3.2", - "resolved": "https://registry.npmjs.org/tapable/-/tapable-2.3.2.tgz", - "integrity": "sha512-1MOpMXuhGzGL5TTCZFItxCc0AARf1EZFQkGqMm7ERKj8+Hgr5oLvJOVFcC+lRmR8hCe2S3jC4T5D7Vg/d7/fhA==", + "version": "2.3.3", + "resolved": "https://registry.npmjs.org/tapable/-/tapable-2.3.3.tgz", + "integrity": "sha512-uxc/zpqFg6x7C8vOE7lh6Lbda8eEL9zmVm/PLeTPBRhh1xCgdWaQ+J1CUieGpIfm2HdtsUpRv+HshiasBMcc6A==", "dev": true, "license": "MIT", "engines": { @@ -5098,14 +4858,14 @@ "license": "MIT" }, "node_modules/tinyglobby": { - "version": "0.2.15", - "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.15.tgz", - "integrity": "sha512-j2Zq4NyQYG5XMST4cbs02Ak8iJUdxRM0XI5QyxXuZOzKOINmWurp3smXu3y5wDcJrptwpSjgXHzIQxR0omXljQ==", + "version": "0.2.16", + "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.16.tgz", + "integrity": "sha512-pn99VhoACYR8nFHhxqix+uvsbXineAasWm5ojXoN8xEwK5Kd3/TrhNn1wByuD52UxWRLy8pu+kRMniEi6Eq9Zg==", "dev": true, "license": "MIT", "dependencies": { "fdir": "^6.5.0", - "picomatch": "^4.0.3" + "picomatch": "^4.0.4" }, "engines": { "node": ">=12.0.0" @@ -5114,6 +4874,16 @@ "url": "https://github.com/sponsors/SuperchupuDev" } }, + "node_modules/tinyrainbow": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/tinyrainbow/-/tinyrainbow-2.0.0.tgz", + "integrity": "sha512-op4nsTR47R6p0vMUUoYl/a+ljLFVtlfaXkLQmqfLR1qHma1h/ysYk4hEXZ880bf2CYgTskvTa/e196Vd5dDQXw==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=14.0.0" + } + }, "node_modules/tinyspy": { "version": "4.0.4", "resolved": "https://registry.npmjs.org/tinyspy/-/tinyspy-4.0.4.tgz", @@ -5182,9 +4952,9 @@ } }, "node_modules/typescript": { - "version": "5.7.3", - "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.7.3.tgz", - "integrity": "sha512-84MVSjMEHP+FQRPy3pX9sTVV/INIex71s9TL2Gm5FG/WG1SqXeKyZ0k7/blY/4FdOzI12CBy1vGc4og/eus0fw==", + "version": "6.0.3", + "resolved": "https://registry.npmjs.org/typescript/-/typescript-6.0.3.tgz", + "integrity": "sha512-y2TvuxSZPDyQakkFRPZHKFm+KKVqIisdg9/CZwm9ftvKXLP8NRWj38/ODjNbr43SsoXqNuAisEf1GdCxqWcdBw==", "dev": true, "license": "Apache-2.0", "bin": { @@ -5196,16 +4966,16 @@ } }, "node_modules/typescript-eslint": { - "version": "8.57.2", - "resolved": "https://registry.npmjs.org/typescript-eslint/-/typescript-eslint-8.57.2.tgz", - "integrity": "sha512-VEPQ0iPgWO/sBaZOU1xo4nuNdODVOajPnTIbog2GKYr31nIlZ0fWPoCQgGfF3ETyBl1vn63F/p50Um9Z4J8O8A==", + "version": "8.59.0", + "resolved": "https://registry.npmjs.org/typescript-eslint/-/typescript-eslint-8.59.0.tgz", + "integrity": "sha512-BU3ONW9X+v90EcCH9ZS6LMackcVtxRLlI3XrYyqZIwVSHIk7Qf7bFw1z0M9Q0IUxhTMZCf8piY9hTYaNEIASrw==", "dev": true, "license": "MIT", "dependencies": { - "@typescript-eslint/eslint-plugin": "8.57.2", - "@typescript-eslint/parser": "8.57.2", - "@typescript-eslint/typescript-estree": "8.57.2", - "@typescript-eslint/utils": "8.57.2" + "@typescript-eslint/eslint-plugin": "8.59.0", + "@typescript-eslint/parser": "8.59.0", + "@typescript-eslint/typescript-estree": "8.59.0", + "@typescript-eslint/utils": "8.59.0" }, "engines": { "node": "^18.18.0 || ^20.9.0 || >=21.1.0" @@ -5216,7 +4986,7 @@ }, "peerDependencies": { "eslint": "^8.57.0 || ^9.0.0 || ^10.0.0", - "typescript": ">=4.8.4 <6.0.0" + "typescript": ">=4.8.4 <6.1.0" } }, "node_modules/typical": { @@ -5302,24 +5072,23 @@ } }, "node_modules/vite": { - "version": "6.4.2", - "resolved": "https://registry.npmjs.org/vite/-/vite-6.4.2.tgz", - "integrity": "sha512-2N/55r4JDJ4gdrCvGgINMy+HH3iRpNIz8K6SFwVsA+JbQScLiC+clmAxBgwiSPgcG9U15QmvqCGWzMbqda5zGQ==", + "version": "8.0.10", + "resolved": "https://registry.npmjs.org/vite/-/vite-8.0.10.tgz", + "integrity": "sha512-rZuUu9j6J5uotLDs+cAA4O5H4K1SfPliUlQwqa6YEwSrWDZzP4rhm00oJR5snMewjxF5V/K3D4kctsUTsIU9Mw==", "dev": true, "license": "MIT", "dependencies": { - "esbuild": "^0.25.0", - "fdir": "^6.4.4", - "picomatch": "^4.0.2", - "postcss": "^8.5.3", - "rollup": "^4.34.9", - "tinyglobby": "^0.2.13" + "lightningcss": "^1.32.0", + "picomatch": "^4.0.4", + "postcss": "^8.5.10", + "rolldown": "1.0.0-rc.17", + "tinyglobby": "^0.2.16" }, "bin": { "vite": "bin/vite.js" }, "engines": { - "node": "^18.0.0 || ^20.0.0 || >=22.0.0" + "node": "^20.19.0 || >=22.12.0" }, "funding": { "url": "https://github.com/vitejs/vite?sponsor=1" @@ -5328,14 +5097,15 @@ "fsevents": "~2.3.3" }, "peerDependencies": { - "@types/node": "^18.0.0 || ^20.0.0 || >=22.0.0", + "@types/node": "^20.19.0 || >=22.12.0", + "@vitejs/devtools": "^0.1.0", + "esbuild": "^0.27.0 || ^0.28.0", "jiti": ">=1.21.0", - "less": "*", - "lightningcss": "^1.21.0", - "sass": "*", - "sass-embedded": "*", - "stylus": "*", - "sugarss": "*", + "less": "^4.0.0", + "sass": "^1.70.0", + "sass-embedded": "^1.70.0", + "stylus": ">=0.54.8", + "sugarss": "^5.0.0", "terser": "^5.16.0", "tsx": "^4.8.1", "yaml": "^2.4.2" @@ -5344,13 +5114,16 @@ "@types/node": { "optional": true }, - "jiti": { + "@vitejs/devtools": { "optional": true }, - "less": { + "esbuild": { + "optional": true + }, + "jiti": { "optional": true }, - "lightningcss": { + "less": { "optional": true }, "sass": { @@ -5475,6 +5248,29 @@ "funding": { "url": "https://github.com/sponsors/sindresorhus" } + }, + "node_modules/zod": { + "version": "4.3.6", + "resolved": "https://registry.npmjs.org/zod/-/zod-4.3.6.tgz", + "integrity": "sha512-rftlrkhHZOcjDwkGlnUtZZkvaPHCsDATp4pGpuOOMDaTdDDXF91wuVDJoWoPsKX/3YPQ5fHuF3STjcYyKr+Qhg==", + "dev": true, + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/colinhacks" + } + }, + "node_modules/zod-validation-error": { + "version": "4.0.2", + "resolved": "https://registry.npmjs.org/zod-validation-error/-/zod-validation-error-4.0.2.tgz", + "integrity": "sha512-Q6/nZLe6jxuU80qb/4uJ4t5v2VEZ44lzQjPDhYJNztRQ4wyWc6VF3D3Kb/fAuPetZQnhS3hnajCf9CsWesghLQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=18.0.0" + }, + "peerDependencies": { + "zod": "^3.25.0 || ^4.0.0" + } } } } diff --git a/vortex-web/package.json b/vortex-web/package.json index ae1ea81bd5a..9f21562641d 100644 --- a/vortex-web/package.json +++ b/vortex-web/package.json @@ -27,25 +27,25 @@ "react-dom": "^19.1.0" }, "devDependencies": { - "@eslint/js": "^9.25.0", - "@storybook/addon-docs": "^10.3.3", - "@storybook/react-vite": "^10.3.3", - "@tailwindcss/vite": "^4.1.4", + "@eslint/js": "^10.0.1", + "@storybook/addon-docs": "^10.3.5", + "@storybook/react-vite": "^10.3.5", + "@tailwindcss/vite": "^4.2.4", "@types/d3-hierarchy": "^3.1.7", - "@types/react": "^19.1.2", - "@types/react-dom": "^19.1.2", - "@vitejs/plugin-react": "^4.4.1", - "eslint": "^9.25.0", + "@types/react": "^19.2.14", + "@types/react-dom": "^19.2.3", + "@vitejs/plugin-react": "^6.0.1", + "eslint": "^10.2.1", "eslint-config-prettier": "^10.1.8", - "eslint-plugin-react-hooks": "^5.2.0", - "eslint-plugin-react-refresh": "^0.4.20", - "eslint-plugin-storybook": "^10.3.3", - "globals": "^16.0.0", - "prettier": "^3.8.1", - "storybook": "^10.3.3", - "tailwindcss": "^4.1.4", - "typescript": "~5.7.0", - "typescript-eslint": "^8.30.1", - "vite": "^6.3.2" + "eslint-plugin-react-hooks": "^7.1.1", + "eslint-plugin-react-refresh": "^0.5.2", + "eslint-plugin-storybook": "^10.3.5", + "globals": "^17.5.0", + "prettier": "^3.8.3", + "storybook": "^10.3.5", + "tailwindcss": "^4.2.4", + "typescript": "~6.0.0", + "typescript-eslint": "^8.59.0", + "vite": "^8.0.10" } } diff --git a/vortex/Cargo.toml b/vortex/Cargo.toml index f042f568c11..982127a4035 100644 --- a/vortex/Cargo.toml +++ b/vortex/Cargo.toml @@ -54,6 +54,7 @@ anyhow = { workspace = true } arrow-array = { workspace = true } divan = { workspace = true } fastlanes = { workspace = true } +futures = { workspace = true } mimalloc = { workspace = true } parquet = { workspace = true } paste = { workspace = true } @@ -64,6 +65,7 @@ tokio = { workspace = true, features = ["full"] } tracing = { workspace = true } tracing-subscriber = { workspace = true } vortex = { path = ".", features = ["tokio"] } +vortex-bench = { workspace = true, features = ["unstable_encodings"] } vortex-tensor = { workspace = true } [features] @@ -83,6 +85,10 @@ unstable_encodings = [ "vortex-zstd?/unstable_encodings", ] +[[example]] +name = "turboquant_vector_search" +required-features = ["files", "tokio", "unstable_encodings"] + [[bench]] name = "single_encoding_throughput" harness = false diff --git a/vortex/benches/common_encoding_tree_throughput.rs b/vortex/benches/common_encoding_tree_throughput.rs index 077fda0a536..552f66c321f 100644 --- a/vortex/benches/common_encoding_tree_throughput.rs +++ b/vortex/benches/common_encoding_tree_throughput.rs @@ -1,8 +1,7 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] -#![allow(unexpected_cfgs)] +#![expect(clippy::unwrap_used)] use std::fmt; use std::ops::Deref; @@ -14,8 +13,10 @@ use mimalloc::MiMalloc; use rand::RngExt; use rand::SeedableRng; use vortex::array::ArrayRef; +use vortex::array::Canonical; use vortex::array::IntoArray; -use vortex::array::ToCanonical; +use vortex::array::LEGACY_SESSION; +use vortex::array::VortexSessionExecute; use vortex::array::arrays::DictArray; use vortex::array::arrays::PrimitiveArray; use vortex::array::arrays::TemporalArray; @@ -71,6 +72,7 @@ mod setup { use super::*; fn setup_primitive_arrays() -> (PrimitiveArray, PrimitiveArray, PrimitiveArray) { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let mut rng = StdRng::seed_from_u64(0); let uint_array = PrimitiveArray::from_iter((0..NUM_VALUES).map(|_| rng.random_range(42u32..256))); @@ -79,22 +81,25 @@ mod setup { .into_array() .cast(PType::I32.into()) .unwrap() - .to_primitive(); + .execute::(&mut ctx) + .unwrap(); let float_array = uint_array .clone() .into_array() .cast(PType::F64.into()) .unwrap() - .to_primitive(); + .execute::(&mut ctx) + .unwrap(); (uint_array, int_array, float_array) } /// Create FoR <- BitPacked encoding tree for u64 pub fn for_bp_u64() -> ArrayRef { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let (uint_array, ..) = setup_primitive_arrays(); let compressed = FoR::encode(uint_array).unwrap(); let inner = compressed.encoded(); - let bp = BitPacked::encode(inner, 8).unwrap(); + let bp = BitPacked::encode(inner, 8, &mut ctx).unwrap(); FoR::try_new(bp.into_array(), compressed.reference_scalar().clone()) .unwrap() .into_array() @@ -102,13 +107,19 @@ mod setup { /// Create ALP <- FoR <- BitPacked encoding tree for f64 pub fn alp_for_bp_f64() -> ArrayRef { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let (_, _, float_array) = setup_primitive_arrays(); - let alp_compressed = alp_encode(&float_array, None).unwrap(); + let alp_compressed = alp_encode(float_array.as_view(), None, &mut ctx).unwrap(); // Manually construct ALP <- FoR <- BitPacked tree - let for_array = FoR::encode(alp_compressed.encoded().to_primitive()).unwrap(); + let alp_encoded_prim = alp_compressed + .encoded() + .clone() + .execute::(&mut ctx) + .unwrap(); + let for_array = FoR::encode(alp_encoded_prim).unwrap(); let inner = for_array.encoded(); - let bp = BitPacked::encode(inner, 8).unwrap(); + let bp = BitPacked::encode(inner, 8, &mut ctx).unwrap(); let for_with_bp = FoR::try_new(bp.into_array(), for_array.reference_scalar().clone()).unwrap(); @@ -122,7 +133,7 @@ mod setup { } /// Create Dict <- VarBinView encoding tree for strings with BitPacked codes - #[allow(clippy::cast_possible_truncation)] + #[expect(clippy::cast_possible_truncation)] pub fn dict_varbinview_string() -> ArrayRef { let mut rng = StdRng::seed_from_u64(42); @@ -143,7 +154,8 @@ mod setup { let codes_prim = PrimitiveArray::from_iter(codes); // Compress codes with BitPacked (6 bits should be enough for ~50 unique values) - let codes_bp = BitPacked::encode(&codes_prim.into_array(), 6) + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let codes_bp = BitPacked::encode(&codes_prim.into_array(), 6, &mut ctx) .unwrap() .into_array(); @@ -156,7 +168,7 @@ mod setup { } /// Create RunEnd <- FoR <- BitPacked encoding tree for u32 - #[allow(clippy::cast_possible_truncation)] + #[expect(clippy::cast_possible_truncation)] pub fn runend_for_bp_u32() -> ArrayRef { let mut rng = StdRng::seed_from_u64(42); // Create data with runs of repeated values @@ -173,32 +185,41 @@ mod setup { run_length -= 1; } + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let prim_array = PrimitiveArray::from_iter(values); - let runend = RunEnd::encode(prim_array.into_array()).unwrap(); + let runend = RunEnd::encode(prim_array.into_array(), &mut ctx).unwrap(); // Compress the ends with FoR <- BitPacked - let ends_prim = runend.ends().to_primitive(); + let ends_prim = runend + .ends() + .clone() + .execute::(&mut ctx) + .unwrap(); let ends_for = FoR::encode(ends_prim).unwrap(); let ends_inner = ends_for.encoded(); - let ends_bp = BitPacked::encode(ends_inner, 8).unwrap(); + let ends_bp = BitPacked::encode(ends_inner, 8, &mut ctx).unwrap(); let compressed_ends = FoR::try_new(ends_bp.into_array(), ends_for.reference_scalar().clone()) .unwrap() .into_array(); // Compress the values with BitPacked - let values_prim = runend.values().to_primitive(); - let compressed_values = BitPacked::encode(&values_prim.into_array(), 8) + let values_prim = runend + .values() + .clone() + .execute::(&mut ctx) + .unwrap(); + let compressed_values = BitPacked::encode(&values_prim.into_array(), 8, &mut ctx) .unwrap() .into_array(); - RunEnd::try_new(compressed_ends, compressed_values) + RunEnd::try_new(compressed_ends, compressed_values, &mut ctx) .unwrap() .into_array() } /// Create Dict <- FSST <- VarBin encoding tree for strings - #[allow(clippy::cast_possible_truncation)] + #[expect(clippy::cast_possible_truncation)] pub fn dict_fsst_varbin_string() -> ArrayRef { let mut rng = StdRng::seed_from_u64(43); @@ -213,6 +234,7 @@ mod setup { .collect(); // Train and compress unique values with FSST + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let unique_varbinview = VarBinViewArray::from_iter_str(unique_strings); let fsst_compressor = fsst_train_compressor(&unique_varbinview); let fsst_values = fsst_compress( @@ -220,6 +242,7 @@ mod setup { unique_varbinview.len(), unique_varbinview.dtype(), &fsst_compressor, + &mut ctx, ); // Create codes array (random indices into unique values) @@ -235,7 +258,7 @@ mod setup { /// Create Dict <- FSST <- VarBin <- BitPacked encoding tree for strings /// Compress the VarBin offsets inside FSST with BitPacked - #[allow(clippy::cast_possible_truncation)] + #[expect(clippy::cast_possible_truncation)] pub fn dict_fsst_varbin_bp_string() -> ArrayRef { let mut rng = StdRng::seed_from_u64(45); @@ -250,6 +273,7 @@ mod setup { .collect(); // Train and compress unique values with FSST + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let unique_varbinview = VarBinViewArray::from_iter_str(unique_strings); let fsst_compressor = fsst_train_compressor(&unique_varbinview); let fsst = fsst_compress( @@ -257,12 +281,17 @@ mod setup { unique_varbinview.len(), unique_varbinview.dtype(), &fsst_compressor, + &mut ctx, ); // Compress the VarBin offsets with BitPacked let codes = fsst.codes(); - let offsets_prim = codes.offsets().to_primitive(); - let offsets_bp = BitPacked::encode(&offsets_prim.into_array(), 20).unwrap(); + let offsets_prim = codes + .offsets() + .clone() + .execute::(&mut ctx) + .unwrap(); + let offsets_bp = BitPacked::encode(&offsets_prim.into_array(), 20, &mut ctx).unwrap(); // Rebuild VarBin with compressed offsets let compressed_codes = VarBinArray::try_new( @@ -282,6 +311,7 @@ mod setup { fsst.symbol_lengths().clone(), compressed_codes, fsst.uncompressed_lengths().clone(), + &mut ctx, ) .unwrap(); @@ -311,23 +341,32 @@ mod setup { let temporal_array = TemporalArray::new_timestamp(ts_array, TimeUnit::Microseconds, None); // Split into days, seconds, subseconds - let parts = split_temporal(temporal_array.clone()).unwrap(); + let mut ctx = LEGACY_SESSION.create_execution_ctx(); + let parts = split_temporal(temporal_array.clone(), &mut ctx).unwrap(); // Compress days with FoR <- BitPacked - let days_prim = parts.days.to_primitive(); + let days_prim = parts + .days + .clone() + .execute::(&mut ctx) + .unwrap(); let days_for = FoR::encode(days_prim).unwrap(); let days_inner = days_for.encoded(); - let days_bp = BitPacked::encode(days_inner, 16).unwrap(); + let days_bp = BitPacked::encode(days_inner, 16, &mut ctx).unwrap(); let compressed_days = FoR::try_new(days_bp.into_array(), days_for.reference_scalar().clone()) .unwrap() .into_array(); // Compress seconds with FoR <- BitPacked - let seconds_prim = parts.seconds.to_primitive(); + let seconds_prim = parts + .seconds + .clone() + .execute::(&mut ctx) + .unwrap(); let seconds_for = FoR::encode(seconds_prim).unwrap(); let seconds_inner = seconds_for.encoded(); - let seconds_bp = BitPacked::encode(seconds_inner, 17).unwrap(); + let seconds_bp = BitPacked::encode(seconds_inner, 17, &mut ctx).unwrap(); let compressed_seconds = FoR::try_new( seconds_bp.into_array(), seconds_for.reference_scalar().clone(), @@ -336,10 +375,13 @@ mod setup { .into_array(); // Compress subseconds with FoR <- BitPacked - let subseconds_prim = parts.subseconds.to_primitive(); + let subseconds_prim = parts + .subseconds + .execute::(&mut ctx) + .unwrap(); let subseconds_for = FoR::encode(subseconds_prim).unwrap(); let subseconds_inner = subseconds_for.encoded(); - let subseconds_bp = BitPacked::encode(subseconds_inner, 20).unwrap(); + let subseconds_bp = BitPacked::encode(subseconds_inner, 20, &mut ctx).unwrap(); let compressed_subseconds = FoR::try_new( subseconds_bp.into_array(), subseconds_for.reference_scalar().clone(), @@ -412,6 +454,6 @@ fn decompress(bencher: Bencher, setup_fn: SetupFn) { let nbytes = compressed.nbytes(); with_byte_counter(bencher, nbytes) - .with_inputs(|| &compressed) - .bench_refs(|a| a.to_canonical()); + .with_inputs(|| (&compressed, LEGACY_SESSION.create_execution_ctx())) + .bench_refs(|(a, ctx)| (**a).clone().execute::(ctx)); } diff --git a/vortex/benches/pipeline.rs b/vortex/benches/pipeline.rs index d6c668e0607..5fa3f646c50 100644 --- a/vortex/benches/pipeline.rs +++ b/vortex/benches/pipeline.rs @@ -108,11 +108,7 @@ //! [1] //! [2] -#![allow( - clippy::unwrap_used, - clippy::uninit_vec, - clippy::cast_possible_truncation -)] +#![expect(clippy::unwrap_used, clippy::uninit_vec)] use divan::Bencher; use fastlanes::BitPacking; diff --git a/vortex/benches/single_encoding_throughput.rs b/vortex/benches/single_encoding_throughput.rs index 091b9a6747f..be253187956 100644 --- a/vortex/benches/single_encoding_throughput.rs +++ b/vortex/benches/single_encoding_throughput.rs @@ -1,9 +1,8 @@ // SPDX-License-Identifier: Apache-2.0 // SPDX-FileCopyrightText: Copyright the Vortex contributors -#![allow(clippy::unwrap_used)] -#![allow(clippy::cast_possible_truncation)] -#![allow(unexpected_cfgs)] +#![expect(clippy::unwrap_used)] +#![expect(clippy::cast_possible_truncation)] use std::sync::LazyLock; @@ -15,8 +14,10 @@ use rand::RngExt; use rand::SeedableRng; use rand::prelude::IndexedRandom; use rand::rngs::StdRng; +use vortex::array::Canonical; +use vortex::array::ExecutionCtx; use vortex::array::IntoArray; -use vortex::array::ToCanonical; +use vortex::array::LEGACY_SESSION; use vortex::array::arrays::PrimitiveArray; use vortex::array::arrays::VarBinViewArray; use vortex::array::builders::dict::dict_encode; @@ -39,6 +40,7 @@ use vortex::encodings::zigzag::zigzag_encode; use vortex::encodings::zstd::Zstd; use vortex::encodings::zstd::ZstdData; use vortex_array::VortexSessionExecute; +use vortex_error::VortexResult; use vortex_sequence::Sequence; use vortex_session::VortexSession; @@ -65,8 +67,13 @@ fn with_byte_counter<'a, 'b>(bencher: Bencher<'a, 'b>, bytes: u64) -> Bencher<'a } } +fn canonicalize(array: impl IntoArray, ctx: &mut ExecutionCtx) -> VortexResult { + array.into_array().execute::(ctx) +} + // Setup functions fn setup_primitive_arrays() -> (PrimitiveArray, PrimitiveArray, PrimitiveArray) { + let mut ctx = LEGACY_SESSION.create_execution_ctx(); let mut rng = StdRng::seed_from_u64(0); let uint_array = PrimitiveArray::from_iter((0..NUM_VALUES).map(|_| rng.random_range(42u32..256))); @@ -75,17 +82,19 @@ fn setup_primitive_arrays() -> (PrimitiveArray, PrimitiveArray, PrimitiveArray) .into_array() .cast(PType::I32.into()) .unwrap() - .to_primitive(); + .execute::(&mut ctx) + .unwrap(); let float_array = uint_array .clone() .into_array() .cast(PType::F64.into()) .unwrap() - .to_primitive(); + .execute::(&mut ctx) + .unwrap(); (uint_array, int_array, float_array) } -#[allow(clippy::cast_possible_truncation)] +#[expect(clippy::cast_possible_truncation)] fn gen_varbin_words(len: usize, uniqueness: f64) -> Vec { let mut rng = StdRng::seed_from_u64(0); let uniq_cnt = (len as f64 * uniqueness) as usize; @@ -120,13 +129,18 @@ fn bench_bitpacked_decompress_u32(bencher: Bencher) { let (uint_array, ..) = setup_primitive_arrays(); let bit_width = 8; - let compressed = bitpack_encode(&uint_array, bit_width, None) - .unwrap() - .into_array(); + let compressed = bitpack_encode( + &uint_array, + bit_width, + None, + &mut SESSION.create_execution_ctx(), + ) + .unwrap() + .into_array(); with_byte_counter(bencher, NUM_VALUES * 4) - .with_inputs(|| &compressed) - .bench_refs(|a| a.to_canonical()); + .with_inputs(|| (&compressed, SESSION.create_execution_ctx())) + .bench_refs(|(a, ctx)| canonicalize((**a).clone(), ctx)); } #[divan::bench(name = "runend_compress_u32")] @@ -134,18 +148,22 @@ fn bench_runend_compress_u32(bencher: Bencher) { let (uint_array, ..) = setup_primitive_arrays(); with_byte_counter(bencher, NUM_VALUES * 4) - .with_inputs(|| uint_array.clone()) - .bench_values(|a| RunEnd::encode(a.into_array()).unwrap()); + .with_inputs(|| (uint_array.clone(), LEGACY_SESSION.create_execution_ctx())) + .bench_values(|(a, mut ctx)| RunEnd::encode(a.into_array(), &mut ctx).unwrap()); } #[divan::bench(name = "runend_decompress_u32")] fn bench_runend_decompress_u32(bencher: Bencher) { let (uint_array, ..) = setup_primitive_arrays(); - let compressed = RunEnd::encode(uint_array.into_array()).unwrap(); + let compressed = RunEnd::encode( + uint_array.into_array(), + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); with_byte_counter(bencher, NUM_VALUES * 4) - .with_inputs(|| &compressed) - .bench_refs(|a| a.to_canonical()); + .with_inputs(|| (&compressed, SESSION.create_execution_ctx())) + .bench_refs(|(a, ctx)| canonicalize((**a).clone(), ctx)); } #[divan::bench(name = "delta_compress_u32")] @@ -153,9 +171,9 @@ fn bench_delta_compress_u32(bencher: Bencher) { let (uint_array, ..) = setup_primitive_arrays(); with_byte_counter(bencher, NUM_VALUES * 4) - .with_inputs(|| &uint_array) - .bench_refs(|a| { - let (_bases, _deltas) = delta_compress(a, &mut SESSION.create_execution_ctx()).unwrap(); + .with_inputs(|| (&uint_array, SESSION.create_execution_ctx())) + .bench_refs(|(a, ctx)| { + let (_bases, _deltas) = delta_compress(a, ctx).unwrap(); DeltaData::try_new(0).unwrap() }); } @@ -169,8 +187,8 @@ fn bench_delta_decompress_u32(bencher: Bencher) { .into_array(); with_byte_counter(bencher, NUM_VALUES * 4) - .with_inputs(|| &compressed) - .bench_refs(|a| a.to_canonical()); + .with_inputs(|| (&compressed, SESSION.create_execution_ctx())) + .bench_refs(|(a, ctx)| canonicalize((**a).clone(), ctx)); } #[divan::bench(name = "for_compress_i32")] @@ -188,8 +206,8 @@ fn bench_for_decompress_i32(bencher: Bencher) { let compressed = FoR::encode(int_array).unwrap(); with_byte_counter(bencher, NUM_VALUES * 4) - .with_inputs(|| &compressed) - .bench_refs(|a| a.to_canonical()); + .with_inputs(|| (&compressed, SESSION.create_execution_ctx())) + .bench_refs(|(a, ctx)| canonicalize((**a).clone(), ctx)); } #[divan::bench(name = "dict_compress_u32")] @@ -207,8 +225,8 @@ fn bench_dict_decompress_u32(bencher: Bencher) { let compressed = dict_encode(&uint_array.into_array()).unwrap(); with_byte_counter(bencher, NUM_VALUES * 4) - .with_inputs(|| &compressed) - .bench_refs(|a| a.to_canonical()); + .with_inputs(|| (&compressed, SESSION.create_execution_ctx())) + .bench_refs(|(a, ctx)| canonicalize((**a).clone(), ctx)); } #[divan::bench(name = "zigzag_compress_i32")] @@ -217,17 +235,17 @@ fn bench_zigzag_compress_i32(bencher: Bencher) { with_byte_counter(bencher, NUM_VALUES * 4) .with_inputs(|| int_array.clone()) - .bench_values(|a| zigzag_encode(a).unwrap()); + .bench_values(|a| zigzag_encode(a.as_view()).unwrap()); } #[divan::bench(name = "zigzag_decompress_i32")] fn bench_zigzag_decompress_i32(bencher: Bencher) { let (_, int_array, _) = setup_primitive_arrays(); - let compressed = zigzag_encode(int_array).unwrap().into_array(); + let compressed = zigzag_encode(int_array.as_view()).unwrap().into_array(); with_byte_counter(bencher, NUM_VALUES * 4) - .with_inputs(|| &compressed) - .bench_refs(|a| a.to_canonical()); + .with_inputs(|| (&compressed, SESSION.create_execution_ctx())) + .bench_refs(|(a, ctx)| canonicalize((**a).clone(), ctx)); } #[expect(clippy::cast_possible_truncation)] @@ -236,8 +254,8 @@ fn bench_sequence_compress_u32(bencher: Bencher) { let seq_array = PrimitiveArray::from_iter(0..NUM_VALUES as u32); with_byte_counter(bencher, NUM_VALUES * 4) - .with_inputs(|| seq_array.clone()) - .bench_values(|a| sequence_encode(&a).unwrap().unwrap()); + .with_inputs(|| (seq_array.clone(), SESSION.create_execution_ctx())) + .bench_values(|(a, mut ctx)| sequence_encode(a.as_view(), &mut ctx).unwrap().unwrap()); } #[expect(clippy::cast_possible_truncation)] @@ -248,8 +266,8 @@ fn bench_sequence_decompress_u32(bencher: Bencher) { .into_array(); with_byte_counter(bencher, NUM_VALUES * 4) - .with_inputs(|| &compressed) - .bench_refs(|a| a.to_canonical()); + .with_inputs(|| (&compressed, SESSION.create_execution_ctx())) + .bench_refs(|(a, ctx)| canonicalize((**a).clone(), ctx)); } #[divan::bench(name = "alp_compress_f64")] @@ -258,17 +276,29 @@ fn bench_alp_compress_f64(bencher: Bencher) { with_byte_counter(bencher, NUM_VALUES * 8) .with_inputs(|| &float_array) - .bench_refs(|a| alp_encode(a, None).unwrap()); + .bench_refs(|a| { + alp_encode( + a.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap() + }); } #[divan::bench(name = "alp_decompress_f64")] fn bench_alp_decompress_f64(bencher: Bencher) { let (_, _, float_array) = setup_primitive_arrays(); - let compressed = alp_encode(&float_array, None).unwrap(); + let compressed = alp_encode( + float_array.as_view(), + None, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(); with_byte_counter(bencher, NUM_VALUES * 8) - .with_inputs(|| &compressed) - .bench_refs(|a| a.to_canonical()); + .with_inputs(|| (&compressed, SESSION.create_execution_ctx())) + .bench_refs(|(a, ctx)| canonicalize((**a).clone(), ctx)); } #[divan::bench(name = "alp_rd_compress_f64")] @@ -276,10 +306,10 @@ fn bench_alp_rd_compress_f64(bencher: Bencher) { let (_, _, float_array) = setup_primitive_arrays(); with_byte_counter(bencher, NUM_VALUES * 8) - .with_inputs(|| &float_array) - .bench_refs(|a| { + .with_inputs(|| (&float_array, SESSION.create_execution_ctx())) + .bench_refs(|(a, ctx)| { let encoder = RDEncoder::new(a.as_slice::()); - encoder.encode(a) + encoder.encode(a.as_view(), ctx) }); } @@ -287,11 +317,11 @@ fn bench_alp_rd_compress_f64(bencher: Bencher) { fn bench_alp_rd_decompress_f64(bencher: Bencher) { let (_, _, float_array) = setup_primitive_arrays(); let encoder = RDEncoder::new(float_array.as_slice::()); - let compressed = encoder.encode(&float_array); + let compressed = encoder.encode(float_array.as_view(), &mut SESSION.create_execution_ctx()); with_byte_counter(bencher, NUM_VALUES * 8) - .with_inputs(|| &compressed) - .bench_refs(|a| a.to_canonical()); + .with_inputs(|| (&compressed, SESSION.create_execution_ctx())) + .bench_refs(|(a, ctx)| canonicalize((**a).clone(), ctx)); } #[divan::bench(name = "pcodec_compress_f64")] @@ -299,18 +329,24 @@ fn bench_pcodec_compress_f64(bencher: Bencher) { let (_, _, float_array) = setup_primitive_arrays(); with_byte_counter(bencher, NUM_VALUES * 8) - .with_inputs(|| &float_array) - .bench_refs(|a| Pco::from_primitive(a, 3, 0).unwrap()); + .with_inputs(|| (&float_array, SESSION.create_execution_ctx())) + .bench_refs(|(a, ctx)| Pco::from_primitive(a.as_view(), 3, 0, ctx).unwrap()); } #[divan::bench(name = "pcodec_decompress_f64")] fn bench_pcodec_decompress_f64(bencher: Bencher) { let (_, _, float_array) = setup_primitive_arrays(); - let compressed = Pco::from_primitive(&float_array, 3, 0).unwrap(); + let compressed = Pco::from_primitive( + float_array.as_view(), + 3, + 0, + &mut SESSION.create_execution_ctx(), + ) + .unwrap(); with_byte_counter(bencher, NUM_VALUES * 8) - .with_inputs(|| &compressed) - .bench_refs(|a| a.to_canonical()); + .with_inputs(|| (&compressed, SESSION.create_execution_ctx())) + .bench_refs(|(a, ctx)| canonicalize((**a).clone(), ctx)); } #[cfg(feature = "zstd")] @@ -321,7 +357,9 @@ fn bench_zstd_compress_u32(bencher: Bencher) { with_byte_counter(bencher, NUM_VALUES * 4) .with_inputs(|| array.clone()) - .bench_values(|a| ZstdData::from_array(a, 3, 8192).unwrap()); + .bench_values(|a| { + ZstdData::from_array(a, 3, 8192, &mut LEGACY_SESSION.create_execution_ctx()).unwrap() + }); } #[cfg(feature = "zstd")] @@ -332,15 +370,21 @@ fn bench_zstd_decompress_u32(bencher: Bencher) { let validity = uint_array.validity().unwrap(); let compressed = Zstd::try_new( dtype, - ZstdData::from_array(uint_array.into_array(), 3, 8192).unwrap(), + ZstdData::from_array( + uint_array.into_array(), + 3, + 8192, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(), validity, ) .unwrap() .into_array(); with_byte_counter(bencher, NUM_VALUES * 4) - .with_inputs(|| &compressed) - .bench_refs(|a| a.to_canonical()); + .with_inputs(|| (&compressed, SESSION.create_execution_ctx())) + .bench_refs(|(a, ctx)| canonicalize((**a).clone(), ctx)); } // String compression benchmarks @@ -363,8 +407,8 @@ fn bench_dict_decompress_string(bencher: Bencher) { let nbytes = varbinview_arr.into_array().nbytes() as u64; with_byte_counter(bencher, nbytes) - .with_inputs(|| &dict) - .bench_refs(|a| a.to_canonical()); + .with_inputs(|| (&dict, SESSION.create_execution_ctx())) + .bench_refs(|(a, ctx)| canonicalize((**a).clone(), ctx)); } #[divan::bench(name = "fsst_compress_string")] @@ -375,8 +419,8 @@ fn bench_fsst_compress_string(bencher: Bencher) { let nbytes = varbinview_arr.nbytes() as u64; with_byte_counter(bencher, nbytes) - .with_inputs(|| &varbinview_arr) - .bench_refs(|a| fsst_compress(*a, a.len(), a.dtype(), &fsst_compressor)); + .with_inputs(|| (&varbinview_arr, LEGACY_SESSION.create_execution_ctx())) + .bench_refs(|(a, ctx)| fsst_compress(*a, a.len(), a.dtype(), &fsst_compressor, ctx)); } #[divan::bench(name = "fsst_decompress_string")] @@ -389,12 +433,13 @@ fn bench_fsst_decompress_string(bencher: Bencher) { varbinview_arr.len(), varbinview_arr.dtype(), &fsst_compressor, + &mut LEGACY_SESSION.create_execution_ctx(), ); let nbytes = varbinview_arr.into_array().nbytes() as u64; with_byte_counter(bencher, nbytes) - .with_inputs(|| &fsst_array) - .bench_refs(|a| a.to_canonical()); + .with_inputs(|| (&fsst_array, SESSION.create_execution_ctx())) + .bench_refs(|(a, ctx)| canonicalize((**a).clone(), ctx)); } #[cfg(feature = "zstd")] @@ -407,7 +452,9 @@ fn bench_zstd_compress_string(bencher: Bencher) { with_byte_counter(bencher, nbytes) .with_inputs(|| array.clone()) - .bench_values(|a| ZstdData::from_array(a, 3, 8192).unwrap()); + .bench_values(|a| { + ZstdData::from_array(a, 3, 8192, &mut LEGACY_SESSION.create_execution_ctx()).unwrap() + }); } #[cfg(feature = "zstd")] @@ -419,7 +466,13 @@ fn bench_zstd_decompress_string(bencher: Bencher) { let validity = varbinview_arr.validity().unwrap(); let compressed = Zstd::try_new( dtype, - ZstdData::from_array(varbinview_arr.clone().into_array(), 3, 8192).unwrap(), + ZstdData::from_array( + varbinview_arr.clone().into_array(), + 3, + 8192, + &mut LEGACY_SESSION.create_execution_ctx(), + ) + .unwrap(), validity, ) .unwrap() @@ -427,10 +480,11 @@ fn bench_zstd_decompress_string(bencher: Bencher) { let nbytes = varbinview_arr.into_array().nbytes() as u64; with_byte_counter(bencher, nbytes) - .with_inputs(|| &compressed) - .bench_refs(|a| a.to_canonical()); + .with_inputs(|| (&compressed, SESSION.create_execution_ctx())) + .bench_refs(|(a, ctx)| canonicalize((**a).clone(), ctx)); } +// TODO(connor): Remove this. // TurboQuant vector quantization benchmarks. #[cfg(feature = "unstable_encodings")] mod turboquant_benches { @@ -451,7 +505,6 @@ mod turboquant_benches { use vortex_buffer::BufferMut; use vortex_tensor::encodings::turboquant::TurboQuantConfig; use vortex_tensor::encodings::turboquant::turboquant_encode_unchecked; - use vortex_tensor::scalar_fns::ApproxOptions; use vortex_tensor::scalar_fns::l2_denorm::normalize_as_l2_denorm; use vortex_tensor::vector::Vector; @@ -491,7 +544,7 @@ mod turboquant_benches { fn turboquant_config(bit_width: u8) -> TurboQuantConfig { TurboQuantConfig { bit_width, - seed: Some(123), + seed: 123, num_rounds: 3, } } @@ -499,7 +552,7 @@ mod turboquant_benches { fn setup_normalized_vector_ext(dim: usize) -> ExtensionArray { let ext = setup_vector_ext(dim); let mut ctx = SESSION.create_execution_ctx(); - let normalized = normalize_as_l2_denorm(&ApproxOptions::Exact, ext.into_array(), &mut ctx) + let normalized = normalize_as_l2_denorm(ext.into_array(), &mut ctx) .unwrap() .child_at(0) .clone(); @@ -509,21 +562,20 @@ mod turboquant_benches { macro_rules! turboquant_bench { (compress, $dim:literal, $bits:literal, $name:ident) => { paste! { - #[divan::bench(name = concat!("turboquant_compress_dim", stringify!($dim), "_", stringify!($bits), "bit"))] + #[divan::bench(name = concat!("turboquant_encode_dim", stringify!($dim), "_", stringify!($bits), "bit"))] fn $name(bencher: Bencher) { let normalized_ext = setup_normalized_vector_ext($dim); let config = turboquant_config($bits); with_byte_counter(bencher, (NUM_VECTORS * $dim * 4) as u64) - .with_inputs(|| normalized_ext.clone()) - .bench_refs(|a| { - let mut ctx = SESSION.create_execution_ctx(); + .with_inputs(|| (normalized_ext.clone(), SESSION.create_execution_ctx())) + .bench_refs(|(a, ctx)| { let normalized = a .as_ref() .as_opt::() .expect("normalized benchmark input should be an Extension array"); // SAFETY: Benchmark inputs are normalized once up front so the timed // region measures only TurboQuant encoding. - unsafe { turboquant_encode_unchecked(normalized, &config, &mut ctx) } + unsafe { turboquant_encode_unchecked(normalized, &config, ctx) } .unwrap() }); } @@ -541,12 +593,11 @@ mod turboquant_benches { } .unwrap(); with_byte_counter(bencher, (NUM_VECTORS * $dim * 4) as u64) - .with_inputs(|| &compressed) - .bench_refs(|a| { - let mut ctx = SESSION.create_execution_ctx(); - a.clone() + .with_inputs(|| (&compressed, SESSION.create_execution_ctx())) + .bench_refs(|(a, ctx)| { + (**a).clone() .into_array() - .execute::(&mut ctx) + .execute::(ctx) .unwrap() }); } diff --git a/vortex/examples/compression_showcase.rs b/vortex/examples/compression_showcase.rs index 7560cc377c8..6aa9cf08218 100644 --- a/vortex/examples/compression_showcase.rs +++ b/vortex/examples/compression_showcase.rs @@ -8,8 +8,10 @@ //! //! Run with: cargo run --example compression_showcase +use vortex::VortexSessionDefault; use vortex::array::ArrayRef; use vortex::array::IntoArray; +use vortex::array::VortexSessionExecute; use vortex::array::arrays::PrimitiveArray; use vortex::array::arrays::StructArray; use vortex::array::arrays::VarBinArray; @@ -17,6 +19,7 @@ use vortex::array::validity::Validity; use vortex::compressor::BtrBlocksCompressor; use vortex::dtype::DType; use vortex::dtype::Nullability; +use vortex::session::VortexSession; use vortex_buffer::Buffer; fn main() -> Result<(), Box> { @@ -25,35 +28,37 @@ fn main() -> Result<(), Box> { println!("This example demonstrates how Vortex automatically selects"); println!("optimal compression strategies for different data patterns.\n"); + let session = VortexSession::default(); + // 1. Compress sequential/monotonic data println!("1. Sequential Data Compression:"); - compress_sequential_data()?; + compress_sequential_data(&session)?; // 2. Compress repetitive data println!("\n2. Repetitive Data Compression:"); - compress_repetitive_data()?; + compress_repetitive_data(&session)?; // 3. Compress string data println!("\n3. String Data Compression:"); - compress_string_data()?; + compress_string_data(&session)?; // 4. Compress floating-point data println!("\n4. Floating-Point Data Compression:"); - compress_float_data()?; + compress_float_data(&session)?; // 5. Compress sparse data println!("\n5. Sparse Data Compression:"); - compress_sparse_data()?; + compress_sparse_data(&session)?; // 6. Compress structured data println!("\n6. Structured Data Compression:"); - compress_structured_data()?; + compress_structured_data(&session)?; println!("\n=== Compression showcase completed! ==="); Ok(()) } -fn compress_sequential_data() -> Result<(), Box> { +fn compress_sequential_data(session: &VortexSession) -> Result<(), Box> { // Create sequential data (e.g., timestamps, IDs) let sequential: PrimitiveArray = (1000..11000).map(|i| i as u64).collect(); @@ -63,7 +68,10 @@ fn compress_sequential_data() -> Result<(), Box> { // Compress using default strategy let compressor = BtrBlocksCompressor::default(); - let compressed = compressor.compress(&sequential.into_array())?; + let compressed = compressor.compress( + &sequential.into_array(), + &mut session.create_execution_ctx(), + )?; let compressed_size = compressed.nbytes(); let ratio = uncompressed_size as f64 / compressed_size as f64; @@ -76,7 +84,7 @@ fn compress_sequential_data() -> Result<(), Box> { Ok(()) } -fn compress_repetitive_data() -> Result<(), Box> { +fn compress_repetitive_data(session: &VortexSession) -> Result<(), Box> { // Create highly repetitive data (run-length encoding opportunity) let mut repetitive = Vec::new(); for i in 0..100 { @@ -91,7 +99,8 @@ fn compress_repetitive_data() -> Result<(), Box> { println!(" Uncompressed size: ~{} bytes", uncompressed_size); let compressor = BtrBlocksCompressor::default(); - let compressed = compressor.compress(&array.into_array())?; + let compressed = + compressor.compress(&array.into_array(), &mut session.create_execution_ctx())?; let compressed_size = compressed.nbytes(); let ratio = uncompressed_size as f64 / compressed_size as f64; @@ -104,7 +113,7 @@ fn compress_repetitive_data() -> Result<(), Box> { Ok(()) } -fn compress_string_data() -> Result<(), Box> { +fn compress_string_data(session: &VortexSession) -> Result<(), Box> { // Create string data with patterns let categories = vec!["Electronics", "Clothing", "Food", "Books"]; let mut strings = Vec::new(); @@ -123,7 +132,8 @@ fn compress_string_data() -> Result<(), Box> { println!(" Uncompressed size: ~{} bytes", uncompressed_size); let compressor = BtrBlocksCompressor::default(); - let compressed = compressor.compress(&array.into_array())?; + let compressed = + compressor.compress(&array.into_array(), &mut session.create_execution_ctx())?; let compressed_size = compressed.nbytes(); let ratio = uncompressed_size as f64 / compressed_size as f64; @@ -136,7 +146,7 @@ fn compress_string_data() -> Result<(), Box> { Ok(()) } -fn compress_float_data() -> Result<(), Box> { +fn compress_float_data(session: &VortexSession) -> Result<(), Box> { // Create floating-point data with patterns let floats: Buffer = (0..10000).map(|i| (i as f64) * 0.1 + 100.0).collect(); let array = floats.into_array(); @@ -146,7 +156,7 @@ fn compress_float_data() -> Result<(), Box> { println!(" Uncompressed size: ~{} bytes", uncompressed_size); let compressor = BtrBlocksCompressor::default(); - let compressed = compressor.compress(&array)?; + let compressed = compressor.compress(&array, &mut session.create_execution_ctx())?; let compressed_size = compressed.nbytes(); let ratio = uncompressed_size as f64 / compressed_size as f64; @@ -159,7 +169,7 @@ fn compress_float_data() -> Result<(), Box> { Ok(()) } -fn compress_sparse_data() -> Result<(), Box> { +fn compress_sparse_data(session: &VortexSession) -> Result<(), Box> { // Create sparse data (mostly zeros with few non-zero values) let mut sparse = vec![0i64; 10000]; for i in (0..10000).step_by(100) { @@ -172,7 +182,8 @@ fn compress_sparse_data() -> Result<(), Box> { println!(" Uncompressed size: ~{} bytes", uncompressed_size); let compressor = BtrBlocksCompressor::default(); - let compressed = compressor.compress(&array.into_array())?; + let compressed = + compressor.compress(&array.into_array(), &mut session.create_execution_ctx())?; let compressed_size = compressed.nbytes(); let ratio = uncompressed_size as f64 / compressed_size as f64; @@ -185,7 +196,7 @@ fn compress_sparse_data() -> Result<(), Box> { Ok(()) } -fn compress_structured_data() -> Result<(), Box> { +fn compress_structured_data(session: &VortexSession) -> Result<(), Box> { // Create a struct array with multiple columns let size = 5000; @@ -222,7 +233,10 @@ fn compress_structured_data() -> Result<(), Box> { println!(" Uncompressed size: ~{} bytes", uncompressed_size); let compressor = BtrBlocksCompressor::default(); - let compressed = compressor.compress(&struct_array.into_array())?; + let compressed = compressor.compress( + &struct_array.into_array(), + &mut session.create_execution_ctx(), + )?; let compressed_size = compressed.nbytes(); let ratio = uncompressed_size as f64 / compressed_size as f64; @@ -236,7 +250,7 @@ fn compress_structured_data() -> Result<(), Box> { } /// Estimate the size of an array in bytes (approximation) -#[allow(clippy::cast_possible_truncation)] +#[expect(clippy::cast_possible_truncation)] fn estimate_size(array: &ArrayRef) -> usize { array.nbytes() as usize } diff --git a/vortex/examples/tracing_vortex.rs b/vortex/examples/tracing_vortex.rs index 8d7dd6b2701..f1227e2e017 100644 --- a/vortex/examples/tracing_vortex.rs +++ b/vortex/examples/tracing_vortex.rs @@ -8,7 +8,7 @@ //! //! Run with: cargo run --example tracing_vortex --features tokio -#![allow( +#![expect( clippy::disallowed_types, clippy::unwrap_used, clippy::cast_possible_truncation @@ -91,8 +91,11 @@ async fn main() -> Result<(), Box> { Ok(()) } -/// Simulates application activity with various log levels and spans -#[allow(clippy::cognitive_complexity)] +/// Simulates application activity with various log levels and spans. +#[allow( + clippy::cognitive_complexity, + reason = "tracing sometimes triggers this" +)] async fn simulate_application_activity(user_id: u32) { // Simulate HTTP request handling let request_span = span!( diff --git a/vortex/examples/turboquant_vector_search.rs b/vortex/examples/turboquant_vector_search.rs new file mode 100644 index 00000000000..47e1e56ba3a --- /dev/null +++ b/vortex/examples/turboquant_vector_search.rs @@ -0,0 +1,399 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! TurboQuant vector-search roundtrip on a vector-embedding dataset. +//! +//! Load a parquet dataset (cohere-small), wrap the `emb` column as a `Vector` +//! extension, compress with BtrBlocks + TurboQuant, write to an in-memory Vortex file, then read +//! the file back twice: +//! +//! 1. plain scan — decode to canonical `FixedSizeList` and verify the per-element diff +//! against the original. TurboQuant is lossy, so we only check the reconstructed values are +//! within a tolerance. +//! 2. scan with a pushed-down cosine-similarity filter `cosine_similarity(emb, query) > thresh`. +//! The `CosineSimilarity` scalar fn is expressed directly as a filter `Expression`, so row +//! selection happens inside the scan rather than after materialization. +//! +//! The parquet file is cached under `vortex-bench/data//` after the first download. Run +//! with: +//! +//! ```sh +//! cargo run --example turboquant_vector_search \ +//! -p vortex --features unstable_encodings --release +//! ``` + +use std::path::PathBuf; +use std::time::Instant; + +use anyhow::Result; +use anyhow::bail; +use anyhow::ensure; +use futures::TryStreamExt; +use vortex::VortexSessionDefault; +use vortex::array::ArrayRef; +use vortex::array::IntoArray; +use vortex::array::VortexSessionExecute; +use vortex::array::arrays::ChunkedArray; +use vortex::array::arrays::ExtensionArray; +use vortex::array::arrays::FixedSizeListArray; +use vortex::array::arrays::PrimitiveArray; +use vortex::array::arrays::StructArray; +use vortex::array::arrays::extension::ExtensionArrayExt; +use vortex::array::arrays::fixed_size_list::FixedSizeListArrayExt; +use vortex::array::arrays::struct_::StructArrayExt; +use vortex::array::expr::col; +use vortex::array::expr::gt; +use vortex::array::expr::lit; +use vortex::array::extension::EmptyMetadata; +use vortex::array::scalar::Scalar; +use vortex::array::scalar_fn::EmptyOptions; +use vortex::array::scalar_fn::ScalarFnVTable; +use vortex::array::scalar_fn::ScalarFnVTableExt; +use vortex::buffer::ByteBuffer; +use vortex::buffer::ByteBufferMut; +use vortex::dtype::DType; +use vortex::dtype::Nullability; +use vortex::dtype::PType; +use vortex::file::ALLOWED_ENCODINGS; +use vortex::file::OpenOptionsSessionExt; +use vortex::file::WriteOptionsSessionExt; +use vortex::file::WriteStrategyBuilder; +use vortex::io::session::RuntimeSessionExt; +use vortex::session::VortexSession; +use vortex_array::ExecutionCtx; +use vortex_array::builtins::ArrayBuiltins; +use vortex_bench::conversions::parquet_to_vortex_chunks; +use vortex_bench::vector_dataset; +use vortex_bench::vector_dataset::TrainLayout; +use vortex_bench::vector_dataset::VectorDataset; +use vortex_bench::vector_dataset::list_to_vector_ext; +use vortex_btrblocks::BtrBlocksCompressorBuilder; +use vortex_error::VortexExpect; +use vortex_tensor::scalar_fns::cosine_similarity::CosineSimilarity; +use vortex_tensor::scalar_fns::l2_denorm::L2Denorm; +use vortex_tensor::scalar_fns::sorf_transform::SorfTransform; +use vortex_tensor::vector::AnyVector; +use vortex_tensor::vector::Vector; + +/// Cosine threshold for the demo filter. The query comes from the test split, so it may or may not +/// have nearby rows in the train split. +const COSINE_THRESHOLD: f32 = 0.85; + +/// Slack for checking decoded rows against a predicate that was evaluated on TurboQuant's lossy +/// readthrough representation. +const COSINE_THRESHOLD_TOL: f32 = 0.02; + +/// Regression ceiling on the decoded vs original max-abs-diff for 8-bit TurboQuant on 768-dim f32 +/// embeddings. Observed on cohere-small: ~0.10. Pinned with slack so the check catches large +/// quality regressions without flapping on normal run-to-run variation. +const MAX_ABS_DIFF_TOL: f32 = 0.2; + +#[tokio::main] +async fn main() -> Result<()> { + // Opt in to registering the tensor scalar-fn array plugins before building the session. + // Without this, the TurboQuant-compressed `emb` column cannot be serialized into the Vortex + // file or deserialized on read. + // + // SAFETY: single-threaded setup before any other thread exists. + unsafe { std::env::set_var(vortex_tensor::SCALAR_FN_ARRAY_TENSOR_PLUGIN_ENV, "1") }; + + let session = VortexSession::default().with_tokio(); + vortex_tensor::initialize(&session); + let mut ctx = session.create_execution_ctx(); + println!("session initialized with tensor plugins"); + + let dataset = VectorDataset::CohereSmall100k; // This is one of the smaller datasets. + + // Download the source parquet files. + let dataset_paths = vector_dataset::download(dataset, TrainLayout::Single).await?; + let (_id, query_vector) = get_query_vector(dataset_paths.test, &mut ctx).await?; + println!( + "query vector selected (id = {_id}, dim = {})", + query_vector.len() + ); + + // Bring the parquet file into memory so that we can write it as a vortex file (after prep). + let single_train_file = dataset_paths + .train_files + .first() + .vortex_expect("we know that there must be a file here") + .clone(); + + println!("reading parquet into chunked array..."); + let chunked_table = parquet_to_vortex_chunks(single_train_file) + .await? + .into_array(); + let len = chunked_table.len(); + println!("parquet loaded: {len} rows"); + + let id = chunked_table.get_item("id")?; + let emb = chunked_table.get_item("emb")?; + + println!("converting emb column to Vector extension type..."); + let vector_array = list_to_vector_ext(emb)?; + + let fields = [("id", id), ("emb", vector_array)]; + let struct_array = StructArray::from_fields(&fields)?.into_array(); + + println!("compressing with TurboQuant and writing to in-memory Vortex file..."); + let bytes = write_turboquant(&session, struct_array.clone().into_array()).await?; + println!("vortex file written: {} bytes", bytes.len()); + + println!("verifying roundtrip fidelity..."); + verify_roundtrip(&session, &bytes, struct_array.clone()).await?; + + println!("verifying filter pushdown with cosine similarity..."); + verify_filter_pushdown(&session, &bytes, &query_vector, struct_array, &mut ctx).await?; + + println!("all checks passed!"); + Ok(()) +} + +async fn get_query_vector( + query_vectors_path: PathBuf, + ctx: &mut ExecutionCtx, +) -> Result<(usize, Vec)> { + let test_vectors = parquet_to_vortex_chunks(query_vectors_path).await?; + + // Get a random query vector. + let idx = rand::random_range(0..test_vectors.len()); + let struct_scalar = test_vectors.execute_scalar(idx, ctx)?; + let id_scalar = struct_scalar + .as_struct() + .field("id") + .vortex_expect("test parquet file missing `id` field"); + + ensure!( + id_scalar + .as_primitive() + .as_::() + .vortex_expect("id was not a i64") + == idx as i64 + ); + + let emb_scalar = struct_scalar + .as_struct() + .field("emb") + .vortex_expect("test parquet file missing `emb` field"); + + // Pack into a `Vec`. + let query_vector: Vec = emb_scalar + .as_list() + .elements() + .vortex_expect("somehow had a null test vector") + .iter() + .map(|element| { + element + .as_primitive() + .as_::() + .vortex_expect("value was not a f32") + }) + .collect(); + + Ok((idx, query_vector)) +} + +async fn write_turboquant(session: &VortexSession, array: ArrayRef) -> Result { + let compressor = BtrBlocksCompressorBuilder::default() + .with_turboquant() + .build(); + + // TurboQuant produces `L2Denorm(SorfTransform(FSL(Dict(...))), norms)`. The default write + // allow-list only covers canonical/compressed array encodings, so the tensor scalar-fn + // encodings it emits get rejected during normalization. Extend the set with the two encoding + // IDs this scheme actually uses. + let mut allowed = ALLOWED_ENCODINGS.clone(); + allowed.insert(L2Denorm.id()); + allowed.insert(SorfTransform.id()); + + let strategy = WriteStrategyBuilder::default() + .with_compressor(compressor) + .with_allow_encodings(allowed) + .build(); + + let mut buf = ByteBufferMut::empty(); + session + .write_options() + .with_strategy(strategy) + .write(&mut buf, array.to_array_stream()) + .await?; + Ok(buf.freeze()) +} + +async fn verify_roundtrip( + session: &VortexSession, + bytes: &ByteBuffer, + original: ArrayRef, +) -> Result<()> { + let chunks: Vec = session + .open_options() + .open_buffer(bytes.clone())? + .scan()? + .into_array_stream()? + .try_collect() + .await?; + + let mut ctx = session.create_execution_ctx(); + + let read: StructArray = ChunkedArray::try_new(chunks, original.dtype().clone())? + .into_array() + .execute(&mut ctx)?; + let original: StructArray = original.execute(&mut ctx)?; + ensure!(read.len() == original.len()); + + let read_emb = read.unmasked_field_by_name("emb")?.clone(); + let original_emb = original.unmasked_field_by_name("emb")?.clone(); + + let decoded = flatten_vector_column(read_emb, &mut ctx)?; + let original_decoded = flatten_vector_column(original_emb, &mut ctx)?; + + let (max_abs, mean_abs) = diff_stats(&original_decoded, &decoded); + println!( + "roundtrip fidelity: max_abs_diff = {max_abs:.6}, mean_abs_diff = {mean_abs:.6} \ + (tol = {MAX_ABS_DIFF_TOL})" + ); + if max_abs > MAX_ABS_DIFF_TOL { + bail!("TurboQuant max_abs_diff {max_abs} exceeds tolerance {MAX_ABS_DIFF_TOL}"); + } + + Ok(()) +} + +async fn verify_filter_pushdown( + session: &VortexSession, + bytes: &ByteBuffer, + query: &[f32], + original: ArrayRef, + ctx: &mut ExecutionCtx, +) -> Result<()> { + // Build the filter as `cosine_similarity(emb, ) > threshold`. The RHS of + // `CosineSimilarity` is a `lit(...)` wrapping a `Vector` scalar; during scan + // evaluation the Literal expands to a ConstantArray whose row count matches the current batch, + // satisfying `CosineSimilarity`'s same-length requirement. The entire expression is pushed + // through `with_filter`, so row selection happens inside the scan rather than after the whole + // column is materialized. + println!("query: {}", preview_vector(query)); + + let query_scalar = build_query_vector_scalar(query)?; + let cosine_expr = CosineSimilarity.new_expr(EmptyOptions, [col("emb"), lit(query_scalar)]); + let filter = gt(cosine_expr, lit(COSINE_THRESHOLD)); + + let scan_start = Instant::now(); + let chunks: Vec = session + .open_options() + .open_buffer(bytes.clone())? + .scan()? + .with_filter(filter) + .into_array_stream()? + .try_collect() + .await?; + let scan_ms = scan_start.elapsed().as_secs_f64() * 1e3; + + let hits: usize = chunks.iter().map(|c| c.len()).sum(); + println!( + "pushed down `cosine_similarity(emb, query) > {COSINE_THRESHOLD}`: {hits} rows survived \ + in {scan_ms:.2} ms" + ); + if hits == 0 { + println!(" no rows survived the filter for this random query"); + return Ok(()); + } + + // Materialize the matching rows and dump each `emb` vector so the reader can see what the + // pushed-down filter actually selected. Vectors are truncated to the first few elements since + // DIM is typically large. + let filtered: StructArray = ChunkedArray::try_new(chunks, original.dtype().clone())? + .into_array() + .execute(ctx)?; + + let emb = filtered.unmasked_field_by_name("emb")?.clone(); + let flat = flatten_vector_column(emb, ctx)?; + + let dim = query.len(); + for (i, row) in flat.chunks_exact(dim).enumerate() { + let cos = cosine_similarity(query, row); + ensure!( + cos >= COSINE_THRESHOLD - COSINE_THRESHOLD_TOL, + "filtered row {i} had decoded cosine {cos:+.6}, below threshold {COSINE_THRESHOLD} \ + by more than tolerance {COSINE_THRESHOLD_TOL}" + ); + println!(" match {i}: cos = {cos:+.6} {}", preview_vector(row)); + } + + Ok(()) +} + +/// Plain `dot(a, b) / (||a|| * ||b||)` over two equal-length f32 slices. Used purely for reporting +/// — the actual row selection is done inside the scan by the pushed-down `CosineSimilarity` +/// expression. This lets the reader cross-check that the surviving rows really do clear the +/// threshold once decoded. +fn cosine_similarity(a: &[f32], b: &[f32]) -> f32 { + assert_eq!(a.len(), b.len()); + let mut dot = 0.0f32; + let mut a_sq = 0.0f32; + let mut b_sq = 0.0f32; + for (&x, &y) in a.iter().zip(b) { + dot += x * y; + a_sq += x * x; + b_sq += y * y; + } + dot / (a_sq.sqrt() * b_sq.sqrt()) +} + +/// Render a vector as `[v0, v1, ..., vN-1, vN]` with the first 4 and last 1 elements at 4-decimal +/// precision. Keeps the output compact for high-dim embeddings while still giving the reader +/// something concrete to eyeball. +fn preview_vector(row: &[f32]) -> String { + let dim = row.len(); + if dim <= 5 { + return format!("[{}] (dim = {dim})", fmt_slice(row)); + } + format!( + "[{}, ..., {}] (dim = {dim})", + fmt_slice(&row[..4]), + fmt_slice(&row[dim - 1..]) + ) +} + +fn fmt_slice(s: &[f32]) -> String { + s.iter() + .map(|v| format!("{v:+.4}")) + .collect::>() + .join(", ") +} + +/// Wrap a query vector in a `Vector` extension scalar suitable for use as the RHS of a +/// `CosineSimilarity` filter expression via `lit(...)`. +fn build_query_vector_scalar(query: &[f32]) -> Result { + let element_dtype = DType::Primitive(PType::F32, Nullability::NonNullable); + let children: Vec = query + .iter() + .map(|&v| Scalar::primitive(v, Nullability::NonNullable)) + .collect(); + let fsl_scalar = Scalar::fixed_size_list(element_dtype, children, Nullability::NonNullable); + Ok(Scalar::extension::(EmptyMetadata, fsl_scalar)) +} + +/// Decode a `Vector` extension array's storage down to its flat f32 buffer. +fn flatten_vector_column(emb: ArrayRef, ctx: &mut ExecutionCtx) -> Result> { + let ext: ExtensionArray = emb.execute(ctx)?; + ensure!(ext.ext_dtype().is::()); + + let fsl: FixedSizeListArray = ext.storage_array().clone().execute(ctx)?; + let elements: PrimitiveArray = fsl.elements().clone().execute(ctx)?; + Ok(elements.as_slice::().to_vec()) +} + +fn diff_stats(original: &[f32], decoded: &[f32]) -> (f32, f32) { + assert_eq!(original.len(), decoded.len()); + let (sum_abs, max_abs) = + original + .iter() + .zip(decoded) + .fold((0.0f32, 0.0f32), |(sum, peak), (&orig, &dec)| { + let diff = (orig - dec).abs(); + (sum + diff, peak.max(diff)) + }); + let mean_abs = sum_abs / original.len() as f32; + (max_abs, mean_abs) +} diff --git a/vortex/src/lib.rs b/vortex/src/lib.rs index 9053dbc7505..ae803ee98ae 100644 --- a/vortex/src/lib.rs +++ b/vortex/src/lib.rs @@ -12,6 +12,7 @@ use vortex_array::dtype::session::DTypeSession; // vortex::expr is in the process of having its dependencies inverted, and will eventually be // pulled back out into a vortex_expr crate. pub use vortex_array::expr; +use vortex_array::optimizer::kernels::ArrayKernels; pub use vortex_array::scalar_fn; use vortex_array::scalar_fn::session::ScalarFnSession; use vortex_array::session::ArraySession; @@ -154,7 +155,8 @@ pub mod encodings { /// Extension trait to create a default Vortex session. pub trait VortexSessionDefault { - /// Creates a default Vortex session with the standard arrays, layouts, and expressions. + /// Creates a default Vortex session with standard arrays, layouts, scalar functions, + /// optimizer kernels, expressions, aggregate functions, and runtime support. fn default() -> VortexSession; } @@ -165,6 +167,7 @@ impl VortexSessionDefault for VortexSession { .with::() .with::() .with::() + .with::() .with::() .with::(); @@ -246,7 +249,11 @@ mod test { let array = PrimitiveArray::new(buffer![42u64; 100_000], Validity::NonNullable); // You can compress an array in-memory with the BtrBlocks compressor - let compressed = BtrBlocksCompressor::default().compress(&array.clone().into_array())?; + let session = VortexSession::default(); + let compressed = BtrBlocksCompressor::default().compress( + &array.clone().into_array(), + &mut session.create_execution_ctx(), + )?; println!( "BtrBlocks size: {} / {}", compressed.nbytes(), diff --git a/wasm-test/src/main.rs b/wasm-test/src/main.rs index 67617c37955..964d3a36c9a 100644 --- a/wasm-test/src/main.rs +++ b/wasm-test/src/main.rs @@ -2,10 +2,13 @@ // SPDX-FileCopyrightText: Copyright the Vortex contributors use vortex::array::IntoArray; +use vortex::array::VortexSessionExecute; use vortex::array::arrays::PrimitiveArray; use vortex::array::validity::Validity; use vortex::buffer::buffer; use vortex::compressor::BtrBlocksCompressor; +use vortex::session::VortexSession; +use vortex::VortexSessionDefault; //use wasm_bindgen::prelude::*; @@ -13,7 +16,10 @@ pub fn main() { // Extremely simple test of compression/decompression and a few compute functions. let array = PrimitiveArray::new(buffer![1i32; 1024], Validity::AllValid).into_array(); - let compressed = BtrBlocksCompressor::default().compress(&array).unwrap(); + let session = VortexSession::default(); + let compressed = BtrBlocksCompressor::default() + .compress(&array, &mut session.create_execution_ctx()) + .unwrap(); println!("Compressed size: {}", compressed.len()); println!("Tree view: {}", compressed.display_tree()); } diff --git a/xtask/src/java_test_files.rs b/xtask/src/java_test_files.rs deleted file mode 100644 index 4a1f8f0b876..00000000000 --- a/xtask/src/java_test_files.rs +++ /dev/null @@ -1,10 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 -// SPDX-FileCopyrightText: Copyright the Vortex contributors - -use xshell::Shell; - -pub fn java_test_files() -> anyhow::Result<()> { - let sh = Shell::new()?; - xshell::cmd!(sh, "cargo run --manifest-path java/testfiles/Cargo.toml").run()?; - Ok(()) -} diff --git a/xtask/src/main.rs b/xtask/src/main.rs index e44022c6ada..3f909d92fea 100644 --- a/xtask/src/main.rs +++ b/xtask/src/main.rs @@ -3,14 +3,12 @@ mod generate_fbs; mod generate_proto; -mod java_test_files; mod public_api; use clap::Parser; use crate::generate_fbs::generate_fbs; use crate::generate_proto::generate_proto; -use crate::java_test_files::java_test_files; use crate::public_api::public_api; #[derive(clap::Parser)] @@ -27,9 +25,6 @@ enum Commands { /// Subcommand to regenerate protobuf language bindings for the Rust project. #[command(name = "generate-proto")] GenerateProto, - /// Subcommand to generate files for Java integration tests. - #[command(name = "java-test-files")] - JavaTestFiles, /// Regenerate public-api.lock files for all published crates. #[command(name = "public-api")] PublicApi, @@ -40,7 +35,6 @@ fn main() -> anyhow::Result<()> { match cli.command { Commands::GenerateFlatbuffers => generate_fbs()?, Commands::GenerateProto => generate_proto()?, - Commands::JavaTestFiles => java_test_files()?, Commands::PublicApi => public_api()?, } Ok(()) diff --git a/xtask/src/public_api.rs b/xtask/src/public_api.rs index 1541b331d7a..8e2c7853f87 100644 --- a/xtask/src/public_api.rs +++ b/xtask/src/public_api.rs @@ -68,6 +68,7 @@ pub fn public_api() -> anyhow::Result<()> { let api = public_api::Builder::from_rustdoc_json(&json_path) .omit_blanket_impls(true) .omit_auto_trait_impls(true) + .omit_param_names(true) .build()?; // Insert blank lines between items to reduce git conflicts.

#[repr(C, align(16))] pub union vortex_array::arrays::varbinview::BinaryView @@ -4844,9 +5032,9 @@ pub fn vortex_array::arrays::varbinview::BinaryView::is_inlined(&self) -> bool pub fn vortex_array::arrays::varbinview::BinaryView::len(&self) -> u32 -pub fn vortex_array::arrays::varbinview::BinaryView::make_view(value: &[u8], block: u32, offset: u32) -> Self +pub fn vortex_array::arrays::varbinview::BinaryView::make_view(&[u8], u32, u32) -> Self -pub fn vortex_array::arrays::varbinview::BinaryView::new_inlined(value: &[u8]) -> Self +pub fn vortex_array::arrays::varbinview::BinaryView::new_inlined(&[u8]) -> Self impl core::clone::Clone for vortex_array::arrays::varbinview::BinaryView @@ -4856,15 +5044,15 @@ impl core::cmp::Eq for vortex_array::arrays::varbinview::BinaryView impl core::cmp::PartialEq for vortex_array::arrays::varbinview::BinaryView -pub fn vortex_array::arrays::varbinview::BinaryView::eq(&self, other: &Self) -> bool +pub fn vortex_array::arrays::varbinview::BinaryView::eq(&self, &Self) -> bool impl core::convert::From for vortex_array::arrays::varbinview::BinaryView -pub fn vortex_array::arrays::varbinview::BinaryView::from(value: u128) -> Self +pub fn vortex_array::arrays::varbinview::BinaryView::from(u128) -> Self impl core::convert::From for vortex_array::arrays::varbinview::BinaryView -pub fn vortex_array::arrays::varbinview::BinaryView::from(value: vortex_array::arrays::varbinview::Ref) -> Self +pub fn vortex_array::arrays::varbinview::BinaryView::from(vortex_array::arrays::varbinview::Ref) -> Self impl core::default::Default for vortex_array::arrays::varbinview::BinaryView @@ -4872,11 +5060,11 @@ pub fn vortex_array::arrays::varbinview::BinaryView::default() -> Self impl core::fmt::Debug for vortex_array::arrays::varbinview::BinaryView -pub fn vortex_array::arrays::varbinview::BinaryView::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::varbinview::BinaryView::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::arrays::varbinview::BinaryView -pub fn vortex_array::arrays::varbinview::BinaryView::hash(&self, state: &mut H) +pub fn vortex_array::arrays::varbinview::BinaryView::hash(&self, &mut H) impl core::marker::Copy for vortex_array::arrays::varbinview::BinaryView @@ -4898,11 +5086,11 @@ impl core::cmp::Eq for vortex_array::arrays::varbinview::Inlined impl core::cmp::PartialEq for vortex_array::arrays::varbinview::Inlined -pub fn vortex_array::arrays::varbinview::Inlined::eq(&self, other: &vortex_array::arrays::varbinview::Inlined) -> bool +pub fn vortex_array::arrays::varbinview::Inlined::eq(&self, &vortex_array::arrays::varbinview::Inlined) -> bool impl core::fmt::Debug for vortex_array::arrays::varbinview::Inlined -pub fn vortex_array::arrays::varbinview::Inlined::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::varbinview::Inlined::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_array::arrays::varbinview::Inlined @@ -4922,7 +5110,7 @@ impl vortex_array::arrays::varbinview::Ref pub fn vortex_array::arrays::varbinview::Ref::as_range(&self) -> core::ops::range::Range -pub fn vortex_array::arrays::varbinview::Ref::with_buffer_and_offset(&self, buffer_index: u32, offset: u32) -> vortex_array::arrays::varbinview::Ref +pub fn vortex_array::arrays::varbinview::Ref::with_buffer_and_offset(&self, u32, u32) -> vortex_array::arrays::varbinview::Ref impl core::clone::Clone for vortex_array::arrays::varbinview::Ref @@ -4930,31 +5118,27 @@ pub fn vortex_array::arrays::varbinview::Ref::clone(&self) -> vortex_array::arra impl core::convert::From for vortex_array::arrays::varbinview::BinaryView -pub fn vortex_array::arrays::varbinview::BinaryView::from(value: vortex_array::arrays::varbinview::Ref) -> Self +pub fn vortex_array::arrays::varbinview::BinaryView::from(vortex_array::arrays::varbinview::Ref) -> Self impl core::fmt::Debug for vortex_array::arrays::varbinview::Ref -pub fn vortex_array::arrays::varbinview::Ref::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::varbinview::Ref::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_array::arrays::varbinview::Ref pub struct vortex_array::arrays::varbinview::VarBinView -impl vortex_array::arrays::VarBinView - -pub const vortex_array::arrays::VarBinView::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::VarBinView pub fn vortex_array::arrays::VarBinView::clone(&self) -> vortex_array::arrays::VarBinView impl core::fmt::Debug for vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::VarBinView::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBinView::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::VarBinView @@ -4964,99 +5148,103 @@ pub type vortex_array::arrays::VarBinView::OperationsVTable = vortex_array::arra pub type vortex_array::arrays::VarBinView::ValidityVTable = vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::VarBinView::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::VarBinView::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::VarBinView::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::VarBinView::buffer_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::VarBinView::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::VarBinView::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::VarBinView::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::VarBinView::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::VarBinView::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::VarBinView::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBinView::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBinView::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::VarBinView::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::VarBinView::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::VarBinView::nbuffers(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::VarBinView::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::VarBinView::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::VarBinView::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::VarBinView::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBinView::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBinView::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::VarBinView::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::VarBinView::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::VarBinView::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::VarBinView::validate(&self, data: &vortex_array::arrays::varbinview::VarBinViewData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::VarBinView::validate(&self, &vortex_array::arrays::varbinview::VarBinViewData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBinView::validity(vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::take(vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::VarBinView + +pub fn vortex_array::arrays::VarBinView::cast(vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::cast(vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::mask(vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::zip::ZipKernel for vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::zip(if_true: vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, if_false: &vortex_array::ArrayRef, mask: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::zip(vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, &vortex_array::ArrayRef, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub struct vortex_array::arrays::varbinview::VarBinViewData impl vortex_array::arrays::varbinview::VarBinViewData -pub fn vortex_array::arrays::varbinview::VarBinViewData::buffer(&self, idx: usize) -> &vortex_buffer::ByteBuffer +pub fn vortex_array::arrays::varbinview::VarBinViewData::buffer(&self, usize) -> &vortex_buffer::ByteBuffer -pub fn vortex_array::arrays::varbinview::VarBinViewData::bytes_at(&self, index: usize) -> vortex_buffer::ByteBuffer +pub fn vortex_array::arrays::varbinview::VarBinViewData::bytes_at(&self, usize) -> vortex_buffer::ByteBuffer pub fn vortex_array::arrays::varbinview::VarBinViewData::data_buffers(&self) -> &alloc::sync::Arc<[vortex_array::buffer::BufferHandle]> -pub fn vortex_array::arrays::varbinview::VarBinViewData::from_iter, I: core::iter::traits::collect::IntoIterator>>(iter: I, dtype: vortex_array::dtype::DType) -> Self +pub fn vortex_array::arrays::varbinview::VarBinViewData::from_iter, I: core::iter::traits::collect::IntoIterator>>(I, vortex_array::dtype::DType) -> Self -pub fn vortex_array::arrays::varbinview::VarBinViewData::from_iter_bin, I: core::iter::traits::collect::IntoIterator>(iter: I) -> Self +pub fn vortex_array::arrays::varbinview::VarBinViewData::from_iter_bin, I: core::iter::traits::collect::IntoIterator>(I) -> Self -pub fn vortex_array::arrays::varbinview::VarBinViewData::from_iter_nullable_bin, I: core::iter::traits::collect::IntoIterator>>(iter: I) -> Self +pub fn vortex_array::arrays::varbinview::VarBinViewData::from_iter_nullable_bin, I: core::iter::traits::collect::IntoIterator>>(I) -> Self -pub fn vortex_array::arrays::varbinview::VarBinViewData::from_iter_nullable_str, I: core::iter::traits::collect::IntoIterator>>(iter: I) -> Self +pub fn vortex_array::arrays::varbinview::VarBinViewData::from_iter_nullable_str, I: core::iter::traits::collect::IntoIterator>>(I) -> Self -pub fn vortex_array::arrays::varbinview::VarBinViewData::from_iter_str, I: core::iter::traits::collect::IntoIterator>(iter: I) -> Self +pub fn vortex_array::arrays::varbinview::VarBinViewData::from_iter_str, I: core::iter::traits::collect::IntoIterator>(I) -> Self pub fn vortex_array::arrays::varbinview::VarBinViewData::is_empty(&self) -> bool pub fn vortex_array::arrays::varbinview::VarBinViewData::len(&self) -> usize -pub fn vortex_array::arrays::varbinview::VarBinViewData::new(views: vortex_buffer::buffer::Buffer, buffers: alloc::sync::Arc<[vortex_buffer::ByteBuffer]>, dtype: vortex_array::dtype::DType, validity: vortex_array::validity::Validity) -> Self +pub fn vortex_array::arrays::varbinview::VarBinViewData::new(vortex_buffer::buffer::Buffer, alloc::sync::Arc<[vortex_buffer::ByteBuffer]>, vortex_array::dtype::DType, vortex_array::validity::Validity) -> Self -pub fn vortex_array::arrays::varbinview::VarBinViewData::new_handle(views: vortex_array::buffer::BufferHandle, buffers: alloc::sync::Arc<[vortex_array::buffer::BufferHandle]>, dtype: vortex_array::dtype::DType, validity: vortex_array::validity::Validity) -> Self +pub fn vortex_array::arrays::varbinview::VarBinViewData::new_handle(vortex_array::buffer::BufferHandle, alloc::sync::Arc<[vortex_array::buffer::BufferHandle]>, vortex_array::dtype::DType, vortex_array::validity::Validity) -> Self -pub unsafe fn vortex_array::arrays::varbinview::VarBinViewData::new_handle_unchecked(views: vortex_array::buffer::BufferHandle, buffers: alloc::sync::Arc<[vortex_array::buffer::BufferHandle]>, dtype: vortex_array::dtype::DType, _validity: vortex_array::validity::Validity) -> Self +pub unsafe fn vortex_array::arrays::varbinview::VarBinViewData::new_handle_unchecked(vortex_array::buffer::BufferHandle, alloc::sync::Arc<[vortex_array::buffer::BufferHandle]>, vortex_array::dtype::DType, vortex_array::validity::Validity) -> Self -pub unsafe fn vortex_array::arrays::varbinview::VarBinViewData::new_unchecked(views: vortex_buffer::buffer::Buffer, buffers: alloc::sync::Arc<[vortex_buffer::ByteBuffer]>, dtype: vortex_array::dtype::DType, validity: vortex_array::validity::Validity) -> Self +pub unsafe fn vortex_array::arrays::varbinview::VarBinViewData::new_unchecked(vortex_buffer::buffer::Buffer, alloc::sync::Arc<[vortex_buffer::ByteBuffer]>, vortex_array::dtype::DType, vortex_array::validity::Validity) -> Self -pub fn vortex_array::arrays::varbinview::VarBinViewData::try_new(views: vortex_buffer::buffer::Buffer, buffers: alloc::sync::Arc<[vortex_buffer::ByteBuffer]>, dtype: vortex_array::dtype::DType, validity: vortex_array::validity::Validity) -> vortex_error::VortexResult +pub fn vortex_array::arrays::varbinview::VarBinViewData::try_new(vortex_buffer::buffer::Buffer, alloc::sync::Arc<[vortex_buffer::ByteBuffer]>, vortex_array::dtype::DType, vortex_array::validity::Validity) -> vortex_error::VortexResult -pub fn vortex_array::arrays::varbinview::VarBinViewData::try_new_handle(views: vortex_array::buffer::BufferHandle, buffers: alloc::sync::Arc<[vortex_array::buffer::BufferHandle]>, dtype: vortex_array::dtype::DType, validity: vortex_array::validity::Validity) -> vortex_error::VortexResult +pub fn vortex_array::arrays::varbinview::VarBinViewData::try_new_handle(vortex_array::buffer::BufferHandle, alloc::sync::Arc<[vortex_array::buffer::BufferHandle]>, vortex_array::dtype::DType, vortex_array::validity::Validity) -> vortex_error::VortexResult -pub fn vortex_array::arrays::varbinview::VarBinViewData::validate(views: &vortex_buffer::buffer::Buffer, buffers: &alloc::sync::Arc<[vortex_buffer::ByteBuffer]>, dtype: &vortex_array::dtype::DType, validity: &vortex_array::validity::Validity) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::varbinview::VarBinViewData::validate(&vortex_buffer::buffer::Buffer, &alloc::sync::Arc<[vortex_buffer::ByteBuffer]>, &vortex_array::dtype::DType, &vortex_array::validity::Validity) -> vortex_error::VortexResult<()> pub fn vortex_array::arrays::varbinview::VarBinViewData::views(&self) -> &[vortex_array::arrays::varbinview::BinaryView] @@ -5068,35 +5256,35 @@ pub fn vortex_array::arrays::varbinview::VarBinViewData::clone(&self) -> vortex_ impl core::fmt::Debug for vortex_array::arrays::varbinview::VarBinViewData -pub fn vortex_array::arrays::varbinview::VarBinViewData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::varbinview::VarBinViewData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::arrays::varbinview::VarBinViewData -pub fn vortex_array::arrays::varbinview::VarBinViewData::fmt(&self, _f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::varbinview::VarBinViewData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::iter::traits::collect::FromIterator> for vortex_array::arrays::varbinview::VarBinViewData -pub fn vortex_array::arrays::varbinview::VarBinViewData::from_iter>>(iter: T) -> Self +pub fn vortex_array::arrays::varbinview::VarBinViewData::from_iter>>(T) -> Self impl core::iter::traits::collect::FromIterator>> for vortex_array::arrays::varbinview::VarBinViewData -pub fn vortex_array::arrays::varbinview::VarBinViewData::from_iter>>>(iter: T) -> Self +pub fn vortex_array::arrays::varbinview::VarBinViewData::from_iter>>>(T) -> Self impl vortex_array::ArrayEq for vortex_array::arrays::varbinview::VarBinViewData -pub fn vortex_array::arrays::varbinview::VarBinViewData::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::varbinview::VarBinViewData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayHash for vortex_array::arrays::varbinview::VarBinViewData -pub fn vortex_array::arrays::varbinview::VarBinViewData::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn vortex_array::arrays::varbinview::VarBinViewData::array_hash(&self, &mut H, vortex_array::Precision) impl<'a> core::iter::traits::collect::FromIterator> for vortex_array::arrays::varbinview::VarBinViewData -pub fn vortex_array::arrays::varbinview::VarBinViewData::from_iter>>(iter: T) -> Self +pub fn vortex_array::arrays::varbinview::VarBinViewData::from_iter>>(T) -> Self impl<'a> core::iter::traits::collect::FromIterator> for vortex_array::arrays::varbinview::VarBinViewData -pub fn vortex_array::arrays::varbinview::VarBinViewData::from_iter>>(iter: T) -> Self +pub fn vortex_array::arrays::varbinview::VarBinViewData::from_iter>>(T) -> Self pub struct vortex_array::arrays::varbinview::VarBinViewDataParts @@ -5114,37 +5302,29 @@ pub fn vortex_array::arrays::varbinview::VarBinViewArrayExt::dtype_parts(&self) pub fn vortex_array::arrays::varbinview::VarBinViewArrayExt::varbinview_validity(&self) -> vortex_array::validity::Validity -pub fn vortex_array::arrays::varbinview::VarBinViewArrayExt::varbinview_validity_mask(&self) -> vortex_mask::Mask - impl> vortex_array::arrays::varbinview::VarBinViewArrayExt for T pub fn T::dtype_parts(&self) -> (bool, vortex_array::dtype::Nullability) pub fn T::varbinview_validity(&self) -> vortex_array::validity::Validity -pub fn T::varbinview_validity_mask(&self) -> vortex_mask::Mask - pub type vortex_array::arrays::varbinview::VarBinViewArray = vortex_array::Array pub mod vortex_array::arrays::variant pub struct vortex_array::arrays::variant::Variant -impl vortex_array::arrays::Variant - -pub const vortex_array::arrays::Variant::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::Variant pub fn vortex_array::arrays::Variant::clone(&self) -> vortex_array::arrays::Variant impl core::fmt::Debug for vortex_array::arrays::Variant -pub fn vortex_array::arrays::Variant::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::Variant::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::Variant -pub fn vortex_array::arrays::Variant::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Variant>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Variant::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Variant>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::Variant @@ -5154,41 +5334,41 @@ pub type vortex_array::arrays::Variant::OperationsVTable = vortex_array::arrays: pub type vortex_array::arrays::Variant::ValidityVTable = vortex_array::arrays::Variant -pub fn vortex_array::arrays::Variant::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Variant::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Variant::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Variant::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Variant::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Variant::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Variant::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Variant::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Variant::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Variant::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Variant::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Variant::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Variant::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Variant::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Variant::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Variant::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Variant::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Variant::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Variant::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Variant::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Variant::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Variant::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Variant::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Variant::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Variant::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Variant::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Variant::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Variant::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Variant::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Variant::validate(&self, _data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Variant::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::Variant -pub fn vortex_array::arrays::Variant::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Variant>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Variant::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Variant>) -> vortex_error::VortexResult pub trait vortex_array::arrays::variant::VariantArrayExt: vortex_array::TypedArrayRef @@ -5202,21 +5382,17 @@ pub type vortex_array::arrays::variant::VariantArray = vortex_array::Array vortex_array::arrays::Bool impl core::fmt::Debug for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::Bool::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Bool::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::Bool @@ -5226,89 +5402,89 @@ pub type vortex_array::arrays::Bool::OperationsVTable = vortex_array::arrays::Bo pub type vortex_array::arrays::Bool::ValidityVTable = vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Bool::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Bool::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Bool::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Bool::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Bool::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Bool::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Bool::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Bool::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Bool::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Bool::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Bool::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Bool::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Bool::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Bool::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Bool::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Bool::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Bool::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Bool::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Bool::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Bool::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Bool::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Bool::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Bool::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Bool::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Bool::validate(&self, data: &vortex_array::arrays::bool::BoolData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Bool::validate(&self, &vortex_array::arrays::bool::BoolData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Bool::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::take(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterReduce for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::filter(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::filter(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::slice(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::slice(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::arrays::bool::BoolMaskedValidityRule pub type vortex_array::arrays::bool::BoolMaskedValidityRule::Parent = vortex_array::arrays::Masked -pub fn vortex_array::arrays::bool::BoolMaskedValidityRule::reduce_parent(&self, array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, parent: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::bool::BoolMaskedValidityRule::reduce_parent(&self, vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, usize) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::Bool + +pub fn vortex_array::arrays::Bool::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::fill_null::FillNullKernel for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::fill_null(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, fill_value: &vortex_array::scalar::Scalar, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::fill_null(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, &vortex_array::scalar::Scalar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::mask(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> pub struct vortex_array::arrays::Chunked -impl vortex_array::arrays::Chunked - -pub const vortex_array::arrays::Chunked::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::Chunked pub fn vortex_array::arrays::Chunked::clone(&self) -> vortex_array::arrays::Chunked impl core::fmt::Debug for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::Chunked::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Chunked::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::Chunked @@ -5318,78 +5494,74 @@ pub type vortex_array::arrays::Chunked::OperationsVTable = vortex_array::arrays: pub type vortex_array::arrays::Chunked::ValidityVTable = vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Chunked::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Chunked::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Chunked::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Chunked::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Chunked::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Chunked::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Chunked::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Chunked::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Chunked::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Chunked::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Chunked::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Chunked::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Chunked::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Chunked::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Chunked::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Chunked::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Chunked::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Chunked::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Chunked::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Chunked::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Chunked::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Chunked::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Chunked::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Chunked::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Chunked::validate(&self, data: &vortex_array::arrays::chunked::ChunkedData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Chunked::validate(&self, &vortex_array::arrays::chunked::ChunkedData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Chunked::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::take(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterKernel for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::filter(array: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, mask: &vortex_mask::Mask, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::filter(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, &vortex_mask::Mask, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceKernel for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::fill_null::FillNullReduce for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::fill_null(array: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, fill_value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::fill_null(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskKernel for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, mask: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::mask(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::zip::ZipKernel for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::zip(if_true: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, if_false: &vortex_array::ArrayRef, mask: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::zip(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, &vortex_array::ArrayRef, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub struct vortex_array::arrays::Constant impl vortex_array::arrays::Constant -pub const vortex_array::arrays::Constant::ID: vortex_array::ArrayId - -impl vortex_array::arrays::Constant - pub const vortex_array::arrays::Constant::TAKE_RULES: vortex_array::optimizer::rules::ParentRuleSet impl core::clone::Clone for vortex_array::arrays::Constant @@ -5398,11 +5570,11 @@ pub fn vortex_array::arrays::Constant::clone(&self) -> vortex_array::arrays::Con impl core::fmt::Debug for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::Constant::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, _index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Constant::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::Constant @@ -5412,87 +5584,83 @@ pub type vortex_array::arrays::Constant::OperationsVTable = vortex_array::arrays pub type vortex_array::arrays::Constant::ValidityVTable = vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Constant::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Constant::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Constant::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Constant::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Constant::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Constant::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Constant::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Constant::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Constant::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Constant::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, _metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Constant::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Constant::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Constant::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Constant::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Constant::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Constant::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Constant::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Constant::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Constant::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Constant::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Constant::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Constant::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Constant::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Constant::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Constant::validate(&self, data: &vortex_array::arrays::constant::ConstantData, dtype: &vortex_array::dtype::DType, _len: usize, _slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Constant::validate(&self, &vortex_array::arrays::constant::ConstantData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Constant::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeReduce for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, indices: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::take(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterReduce for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::filter(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::filter(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::between::BetweenReduce for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::between(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, lower: &vortex_array::ArrayRef, upper: &vortex_array::ArrayRef, options: &vortex_array::scalar_fn::fns::between::BetweenOptions) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::between(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, &vortex_array::ArrayRef, &vortex_array::ArrayRef, &vortex_array::scalar_fn::fns::between::BetweenOptions) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::fill_null::FillNullReduce for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::fill_null(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, fill_value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::fill_null(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::not::NotReduce for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::invert(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::invert(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>) -> vortex_error::VortexResult> pub struct vortex_array::arrays::Decimal -impl vortex_array::arrays::Decimal - -pub const vortex_array::arrays::Decimal::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::Decimal pub fn vortex_array::arrays::Decimal::clone(&self) -> vortex_array::arrays::Decimal impl core::fmt::Debug for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::Decimal::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Decimal::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::Decimal @@ -5502,89 +5670,89 @@ pub type vortex_array::arrays::Decimal::OperationsVTable = vortex_array::arrays: pub type vortex_array::arrays::Decimal::ValidityVTable = vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Decimal::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Decimal::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Decimal::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Decimal::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Decimal::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Decimal::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Decimal::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Decimal::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Decimal::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Decimal::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Decimal::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Decimal::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Decimal::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Decimal::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Decimal::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Decimal::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Decimal::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Decimal::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Decimal::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Decimal::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Decimal::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Decimal::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Decimal::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Decimal::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Decimal::validate(&self, data: &vortex_array::arrays::decimal::DecimalData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Decimal::validate(&self, &vortex_array::arrays::decimal::DecimalData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Decimal::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::take(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::arrays::decimal::DecimalMaskedValidityRule pub type vortex_array::arrays::decimal::DecimalMaskedValidityRule::Parent = vortex_array::arrays::Masked -pub fn vortex_array::arrays::decimal::DecimalMaskedValidityRule::reduce_parent(&self, array: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, parent: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, _child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::decimal::DecimalMaskedValidityRule::reduce_parent(&self, vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, usize) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::between::BetweenKernel for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::between(arr: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, lower: &vortex_array::ArrayRef, upper: &vortex_array::ArrayRef, options: &vortex_array::scalar_fn::fns::between::BetweenOptions, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::between(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, &vortex_array::ArrayRef, &vortex_array::ArrayRef, &vortex_array::scalar_fn::fns::between::BetweenOptions, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, dtype: &vortex_array::dtype::DType, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::Decimal + +pub fn vortex_array::arrays::Decimal::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::fill_null::FillNullKernel for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::fill_null(array: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, fill_value: &vortex_array::scalar::Scalar, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::fill_null(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, &vortex_array::scalar::Scalar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::mask(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> pub struct vortex_array::arrays::Dict -impl vortex_array::arrays::dict::Dict - -pub const vortex_array::arrays::dict::Dict::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::dict::Dict pub fn vortex_array::arrays::dict::Dict::clone(&self) -> vortex_array::arrays::dict::Dict impl core::fmt::Debug for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::dict::Dict::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::dict::Dict::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::dict::Dict @@ -5594,177 +5762,169 @@ pub type vortex_array::arrays::dict::Dict::OperationsVTable = vortex_array::arra pub type vortex_array::arrays::dict::Dict::ValidityVTable = vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::dict::Dict::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::dict::Dict::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::dict::Dict::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::dict::Dict::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::dict::Dict::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::dict::Dict::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::dict::Dict::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::dict::Dict::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::dict::Dict::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::dict::Dict::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::dict::Dict::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::dict::Dict::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::dict::Dict::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::dict::Dict::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::dict::Dict::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::dict::Dict::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::dict::Dict::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::dict::Dict::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::dict::Dict::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::dict::Dict::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::dict::Dict::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::dict::Dict::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::dict::Dict::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::dict::Dict::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::dict::Dict::validate(&self, _data: &vortex_array::arrays::dict::DictData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::dict::Dict::validate(&self, &vortex_array::arrays::dict::DictData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::dict::Dict::validity(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, indices: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::take(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterReduce for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::filter(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::filter(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::binary::CompareKernel for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::compare(lhs: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, rhs: &vortex_array::ArrayRef, operator: vortex_array::scalar_fn::fns::operators::CompareOperator, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::compare(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_array::ArrayRef, vortex_array::scalar_fn::fns::operators::CompareOperator, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::cast(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::fill_null::FillNullKernel for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::fill_null(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, fill_value: &vortex_array::scalar::Scalar, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::fill_null(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_array::scalar::Scalar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::like::LikeReduce for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::like(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, pattern: &vortex_array::ArrayRef, options: vortex_array::scalar_fn::fns::like::LikeOptions) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::like(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_array::ArrayRef, vortex_array::scalar_fn::fns::like::LikeOptions) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::mask(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> pub struct vortex_array::arrays::Extension -impl vortex_array::arrays::Extension - -pub const vortex_array::arrays::Extension::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::Extension pub fn vortex_array::arrays::Extension::clone(&self) -> vortex_array::arrays::Extension impl core::fmt::Debug for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::Extension::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Extension::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::Extension -pub type vortex_array::arrays::Extension::ArrayData = vortex_array::arrays::extension::ExtensionData +pub type vortex_array::arrays::Extension::ArrayData = vortex_array::EmptyArrayData pub type vortex_array::arrays::Extension::OperationsVTable = vortex_array::arrays::Extension pub type vortex_array::arrays::Extension::ValidityVTable = vortex_array::ValidityVTableFromChild -pub fn vortex_array::arrays::Extension::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Extension::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Extension::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Extension::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Extension::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Extension::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Extension::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Extension::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Extension::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Extension::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Extension::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Extension::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Extension::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Extension::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Extension::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Extension::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Extension::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Extension::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Extension::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Extension::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Extension::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Extension::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Extension::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Extension::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Extension::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Extension::validate(&self, data: &vortex_array::arrays::extension::ExtensionData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Extension::validate(&self, &vortex_array::EmptyArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityChild for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::validity_child(array: vortex_array::ArrayView<'_, vortex_array::arrays::Extension>) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Extension::validity_child(vortex_array::ArrayView<'_, vortex_array::arrays::Extension>) -> vortex_array::ArrayRef impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, indices: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::take(vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterReduce for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::filter(array: vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::filter(vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::binary::CompareKernel for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::compare(lhs: vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, rhs: &vortex_array::ArrayRef, operator: vortex_array::scalar_fn::fns::operators::CompareOperator, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::compare(vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, &vortex_array::ArrayRef, vortex_array::scalar_fn::fns::operators::CompareOperator, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::mask(vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> pub struct vortex_array::arrays::Filter -impl vortex_array::arrays::Filter - -pub const vortex_array::arrays::Filter::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::Filter pub fn vortex_array::arrays::Filter::clone(&self) -> vortex_array::arrays::Filter impl core::fmt::Debug for vortex_array::arrays::Filter -pub fn vortex_array::arrays::Filter::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::Filter::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::Filter -pub fn vortex_array::arrays::Filter::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Filter>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Filter::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Filter>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::Filter @@ -5774,59 +5934,55 @@ pub type vortex_array::arrays::Filter::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::Filter::ValidityVTable = vortex_array::arrays::Filter -pub fn vortex_array::arrays::Filter::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Filter::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Filter::buffer(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Filter::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Filter::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Filter::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Filter::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Filter::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Filter::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Filter::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Filter::deserialize(&self, _dtype: &vortex_array::dtype::DType, _len: usize, _metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Filter::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Filter::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Filter::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Filter::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Filter::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Filter::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Filter::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Filter::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Filter::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Filter::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Filter::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Filter::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Filter::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Filter::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Filter::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Filter::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Filter::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Filter::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Filter::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Filter::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::Filter -pub fn vortex_array::arrays::Filter::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Filter>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Filter::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Filter>) -> vortex_error::VortexResult pub struct vortex_array::arrays::FixedSizeList -impl vortex_array::arrays::FixedSizeList - -pub const vortex_array::arrays::FixedSizeList::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::FixedSizeList pub fn vortex_array::arrays::FixedSizeList::clone(&self) -> vortex_array::arrays::FixedSizeList impl core::fmt::Debug for vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::FixedSizeList::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::FixedSizeList::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::FixedSizeList @@ -5836,75 +5992,75 @@ pub type vortex_array::arrays::FixedSizeList::OperationsVTable = vortex_array::a pub type vortex_array::arrays::FixedSizeList::ValidityVTable = vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::FixedSizeList::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::FixedSizeList::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::FixedSizeList::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::FixedSizeList::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::FixedSizeList::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::FixedSizeList::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::FixedSizeList::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::FixedSizeList::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::FixedSizeList::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::FixedSizeList::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::FixedSizeList::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::FixedSizeList::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::FixedSizeList::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::FixedSizeList::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::FixedSizeList::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::FixedSizeList::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::FixedSizeList::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::FixedSizeList::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::FixedSizeList::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::FixedSizeList::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::FixedSizeList::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::FixedSizeList::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::FixedSizeList::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::FixedSizeList::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::FixedSizeList::validate(&self, data: &vortex_array::arrays::fixed_size_list::FixedSizeListData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::FixedSizeList::validate(&self, &vortex_array::arrays::fixed_size_list::FixedSizeListData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::FixedSizeList::validity(vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::take(vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::FixedSizeList + +pub fn vortex_array::arrays::FixedSizeList::cast(vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::cast(vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::mask(vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> pub struct vortex_array::arrays::List -impl vortex_array::arrays::List - -pub const vortex_array::arrays::List::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::List pub fn vortex_array::arrays::List::clone(&self) -> vortex_array::arrays::List impl core::fmt::Debug for vortex_array::arrays::List -pub fn vortex_array::arrays::List::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::List::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::List -pub fn vortex_array::arrays::List::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::List>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::List::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::List>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::List @@ -5914,79 +6070,79 @@ pub type vortex_array::arrays::List::OperationsVTable = vortex_array::arrays::Li pub type vortex_array::arrays::List::ValidityVTable = vortex_array::arrays::List -pub fn vortex_array::arrays::List::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::List::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::List::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::List::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::List::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::List::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::List::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::List::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::List::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::List::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::List::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::List::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::List::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::List::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::List::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::List::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::List::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::List::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::List::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::List::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::List::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::List::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::List::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::List::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::List::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::List::validate(&self, _data: &vortex_array::arrays::list::ListData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::List::validate(&self, &vortex_array::arrays::list::ListData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::List -pub fn vortex_array::arrays::List::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::List>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::List::validity(vortex_array::ArrayView<'_, vortex_array::arrays::List>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::List -pub fn vortex_array::arrays::List::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::List>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::take(vortex_array::ArrayView<'_, vortex_array::arrays::List>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterKernel for vortex_array::arrays::List -pub fn vortex_array::arrays::List::filter(array: vortex_array::ArrayView<'_, vortex_array::arrays::List>, mask: &vortex_mask::Mask, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::filter(vortex_array::ArrayView<'_, vortex_array::arrays::List>, &vortex_mask::Mask, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::List -pub fn vortex_array::arrays::List::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::List + +pub fn vortex_array::arrays::List::cast(vortex_array::ArrayView<'_, vortex_array::arrays::List>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::List -pub fn vortex_array::arrays::List::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::List>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::cast(vortex_array::ArrayView<'_, vortex_array::arrays::List>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::List -pub fn vortex_array::arrays::List::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::List>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::mask(vortex_array::ArrayView<'_, vortex_array::arrays::List>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> pub struct vortex_array::arrays::ListView -impl vortex_array::arrays::ListView - -pub const vortex_array::arrays::ListView::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::ListView pub fn vortex_array::arrays::ListView::clone(&self) -> vortex_array::arrays::ListView impl core::fmt::Debug for vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::ListView::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::ListView::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::ListView @@ -5996,75 +6152,79 @@ pub type vortex_array::arrays::ListView::OperationsVTable = vortex_array::arrays pub type vortex_array::arrays::ListView::ValidityVTable = vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::ListView::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::ListView::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::ListView::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::ListView::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::ListView::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::ListView::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::ListView::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::ListView::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::ListView::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::ListView::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::ListView::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::ListView::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::ListView::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::ListView::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::ListView::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::ListView::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize + +pub fn vortex_array::arrays::ListView::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::ListView::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::ListView::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::ListView::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::ListView::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::ListView::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::ListView::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::ListView::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::ListView::validate(&self, &vortex_array::arrays::listview::ListViewData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> + +impl vortex_array::ValidityVTable for vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::validate(&self, _data: &vortex_array::arrays::listview::ListViewData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::ListView::validity(vortex_array::ArrayView<'_, vortex_array::arrays::ListView>) -> vortex_error::VortexResult -impl vortex_array::ValidityVTable for vortex_array::arrays::ListView +impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::ListView>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::ListView::take(vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::dict::TakeReduce for vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, indices: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::take(vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::ListView + +pub fn vortex_array::arrays::ListView::cast(vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::cast(vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::mask(vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> pub struct vortex_array::arrays::Masked -impl vortex_array::arrays::Masked - -pub const vortex_array::arrays::Masked::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::Masked pub fn vortex_array::arrays::Masked::clone(&self) -> vortex_array::arrays::Masked impl core::fmt::Debug for vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::Masked::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Masked::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::Masked @@ -6074,66 +6234,62 @@ pub type vortex_array::arrays::Masked::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::Masked::ValidityVTable = vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Masked::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Masked::buffer(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Masked::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Masked::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Masked::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Masked::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Masked::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Masked::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Masked::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Masked::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Masked::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Masked::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Masked::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Masked::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Masked::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Masked::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Masked::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Masked::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Masked::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Masked::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Masked::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Masked::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Masked::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Masked::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Masked::validate(&self, _data: &vortex_array::arrays::masked::MaskedData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Masked::validate(&self, &vortex_array::arrays::masked::MaskedData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Masked::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Masked>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeReduce for vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, indices: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::take(vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterReduce for vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::filter(array: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::filter(vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::mask(vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> pub struct vortex_array::arrays::Null impl vortex_array::arrays::null::Null -pub const vortex_array::arrays::null::Null::ID: vortex_array::ArrayId - -impl vortex_array::arrays::null::Null - pub const vortex_array::arrays::null::Null::TAKE_RULES: vortex_array::optimizer::rules::ParentRuleSet impl core::clone::Clone for vortex_array::arrays::null::Null @@ -6142,11 +6298,11 @@ pub fn vortex_array::arrays::null::Null::clone(&self) -> vortex_array::arrays::n impl core::fmt::Debug for vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::null::Null::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::scalar_at(_array: vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, _index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::null::Null::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::null::Null @@ -6156,67 +6312,67 @@ pub type vortex_array::arrays::null::Null::OperationsVTable = vortex_array::arra pub type vortex_array::arrays::null::Null::ValidityVTable = vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::null::Null::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::null::Null::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::null::Null::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::null::Null::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::null::Null::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::null::Null::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::null::Null::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::null::Null::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::null::Null::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::null::Null::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::null::Null::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::null::Null::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::null::Null::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::null::Null::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::null::Null::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::null::Null::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::null::Null::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::null::Null::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::null::Null::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::null::Null::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::null::Null::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::null::Null::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::null::Null::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::null::Null::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::null::Null::validate(&self, _data: &vortex_array::EmptyArrayData, dtype: &vortex_array::dtype::DType, _len: usize, _slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::null::Null::validate(&self, &vortex_array::EmptyArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::validity(_array: vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::null::Null::validity(vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeReduce for vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, indices: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::take(vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterReduce for vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::filter(_array: vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::filter(vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::slice(_array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::cast(vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, _mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::mask(vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> pub struct vortex_array::arrays::Patched impl vortex_array::arrays::patched::Patched -pub fn vortex_array::arrays::patched::Patched::from_array_and_patches(inner: vortex_array::ArrayRef, patches: &vortex_array::patches::Patches, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::from_array_and_patches(vortex_array::ArrayRef, &vortex_array::patches::Patches, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl core::clone::Clone for vortex_array::arrays::patched::Patched @@ -6224,11 +6380,11 @@ pub fn vortex_array::arrays::patched::Patched::clone(&self) -> vortex_array::arr impl core::fmt::Debug for vortex_array::arrays::patched::Patched -pub fn vortex_array::arrays::patched::Patched::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::patched::Patched::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::patched::Patched -pub fn vortex_array::arrays::patched::Patched::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::patched::Patched>, index: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::patched::Patched::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::patched::Patched>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::patched::Patched @@ -6238,75 +6394,71 @@ pub type vortex_array::arrays::patched::Patched::OperationsVTable = vortex_array pub type vortex_array::arrays::patched::Patched::ValidityVTable = vortex_array::ValidityVTableFromChild -pub fn vortex_array::arrays::patched::Patched::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::patched::Patched::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::patched::Patched::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::patched::Patched::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::patched::Patched::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::patched::Patched::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::patched::Patched::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::patched::Patched::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::patched::Patched::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::patched::Patched::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::patched::Patched::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::patched::Patched::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::patched::Patched::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::patched::Patched::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::patched::Patched::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::patched::Patched::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::patched::Patched::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::patched::Patched::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::patched::Patched::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::patched::Patched::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::patched::Patched::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::patched::Patched::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::patched::Patched::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::patched::Patched::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::patched::Patched::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::patched::Patched::validate(&self, data: &vortex_array::arrays::patched::PatchedData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::patched::Patched::validate(&self, &vortex_array::arrays::patched::PatchedData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityChild for vortex_array::arrays::patched::Patched -pub fn vortex_array::arrays::patched::Patched::validity_child(array: vortex_array::ArrayView<'_, vortex_array::arrays::patched::Patched>) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::patched::Patched::validity_child(vortex_array::ArrayView<'_, vortex_array::arrays::patched::Patched>) -> vortex_array::ArrayRef impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::patched::Patched -pub fn vortex_array::arrays::patched::Patched::take(array: vortex_array::ArrayView<'_, Self>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::take(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterReduce for vortex_array::arrays::patched::Patched -pub fn vortex_array::arrays::patched::Patched::filter(array: vortex_array::ArrayView<'_, Self>, mask: &vortex_mask::Mask) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::filter(vortex_array::ArrayView<'_, Self>, &vortex_mask::Mask) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::patched::Patched -pub fn vortex_array::arrays::patched::Patched::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::binary::CompareKernel for vortex_array::arrays::patched::Patched -pub fn vortex_array::arrays::patched::Patched::compare(lhs: vortex_array::ArrayView<'_, Self>, rhs: &vortex_array::ArrayRef, operator: vortex_array::scalar_fn::fns::operators::CompareOperator, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::compare(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, vortex_array::scalar_fn::fns::operators::CompareOperator, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub struct vortex_array::arrays::Primitive -impl vortex_array::arrays::Primitive - -pub const vortex_array::arrays::Primitive::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::Primitive pub fn vortex_array::arrays::Primitive::clone(&self) -> vortex_array::arrays::Primitive impl core::fmt::Debug for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::Primitive::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Primitive::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::Primitive @@ -6316,147 +6468,147 @@ pub type vortex_array::arrays::Primitive::OperationsVTable = vortex_array::array pub type vortex_array::arrays::Primitive::ValidityVTable = vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Primitive::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Primitive::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Primitive::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Primitive::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Primitive::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Primitive::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Primitive::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Primitive::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Primitive::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Primitive::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Primitive::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Primitive::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Primitive::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Primitive::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Primitive::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Primitive::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Primitive::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Primitive::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Primitive::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Primitive::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Primitive::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Primitive::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Primitive::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Primitive::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Primitive::validate(&self, data: &vortex_array::arrays::primitive::PrimitiveData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Primitive::validate(&self, &vortex_array::arrays::primitive::PrimitiveData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Primitive::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::take(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::arrays::primitive::PrimitiveMaskedValidityRule pub type vortex_array::arrays::primitive::PrimitiveMaskedValidityRule::Parent = vortex_array::arrays::Masked -pub fn vortex_array::arrays::primitive::PrimitiveMaskedValidityRule::reduce_parent(&self, array: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, parent: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, _child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::primitive::PrimitiveMaskedValidityRule::reduce_parent(&self, vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, usize) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::between::BetweenKernel for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::between(arr: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, lower: &vortex_array::ArrayRef, upper: &vortex_array::ArrayRef, options: &vortex_array::scalar_fn::fns::between::BetweenOptions, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::between(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, &vortex_array::ArrayRef, &vortex_array::ArrayRef, &vortex_array::scalar_fn::fns::between::BetweenOptions, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, dtype: &vortex_array::dtype::DType, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::Primitive + +pub fn vortex_array::arrays::Primitive::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::fill_null::FillNullKernel for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::fill_null(array: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, fill_value: &vortex_array::scalar::Scalar, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::fill_null(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, &vortex_array::scalar::Scalar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::mask(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> -pub struct vortex_array::arrays::ScalarFnVTable +pub struct vortex_array::arrays::ScalarFn -impl core::clone::Clone for vortex_array::arrays::scalar_fn::ScalarFnVTable +impl core::clone::Clone for vortex_array::arrays::scalar_fn::ScalarFn -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::clone(&self) -> vortex_array::arrays::scalar_fn::ScalarFnVTable +pub fn vortex_array::arrays::scalar_fn::ScalarFn::clone(&self) -> vortex_array::arrays::scalar_fn::ScalarFn -impl core::fmt::Debug for vortex_array::arrays::scalar_fn::ScalarFnVTable +impl core::fmt::Debug for vortex_array::arrays::scalar_fn::ScalarFn -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::scalar_fn::ScalarFn::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result -impl vortex_array::OperationsVTable for vortex_array::arrays::scalar_fn::ScalarFnVTable +impl vortex_array::OperationsVTable for vortex_array::arrays::scalar_fn::ScalarFn -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::scalar_fn::ScalarFnVTable>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::scalar_fn::ScalarFn::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::scalar_fn::ScalarFn>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -impl vortex_array::VTable for vortex_array::arrays::scalar_fn::ScalarFnVTable +impl vortex_array::VTable for vortex_array::arrays::scalar_fn::ScalarFn -pub type vortex_array::arrays::scalar_fn::ScalarFnVTable::ArrayData = vortex_array::arrays::scalar_fn::ScalarFnData +pub type vortex_array::arrays::scalar_fn::ScalarFn::ArrayData = vortex_array::arrays::scalar_fn::array::ScalarFnData -pub type vortex_array::arrays::scalar_fn::ScalarFnVTable::OperationsVTable = vortex_array::arrays::scalar_fn::ScalarFnVTable +pub type vortex_array::arrays::scalar_fn::ScalarFn::OperationsVTable = vortex_array::arrays::scalar_fn::ScalarFn -pub type vortex_array::arrays::scalar_fn::ScalarFnVTable::ValidityVTable = vortex_array::arrays::scalar_fn::ScalarFnVTable +pub type vortex_array::arrays::scalar_fn::ScalarFn::ValidityVTable = vortex_array::arrays::scalar_fn::ScalarFn -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::scalar_fn::ScalarFn::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::scalar_fn::ScalarFn::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::scalar_fn::ScalarFn::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::scalar_fn::ScalarFn::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::deserialize(&self, _dtype: &vortex_array::dtype::DType, _len: usize, _metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::scalar_fn::ScalarFn::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::id(&self) -> vortex_array::ArrayId +pub fn vortex_array::arrays::scalar_fn::ScalarFn::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::scalar_fn::ScalarFn::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::scalar_fn::ScalarFn::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::slot_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::scalar_fn::ScalarFn::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::validate(&self, data: &vortex_array::arrays::scalar_fn::ScalarFnData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::validate(&self, &vortex_array::arrays::scalar_fn::array::ScalarFnData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> -impl vortex_array::ValidityVTable for vortex_array::arrays::scalar_fn::ScalarFnVTable +impl vortex_array::ValidityVTable for vortex_array::arrays::scalar_fn::ScalarFn -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::scalar_fn::ScalarFnVTable>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::scalar_fn::ScalarFn::validity(vortex_array::ArrayView<'_, vortex_array::arrays::scalar_fn::ScalarFn>) -> vortex_error::VortexResult pub struct vortex_array::arrays::Shared -impl vortex_array::arrays::Shared - -pub const vortex_array::arrays::Shared::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::Shared pub fn vortex_array::arrays::Shared::clone(&self) -> vortex_array::arrays::Shared impl core::fmt::Debug for vortex_array::arrays::Shared -pub fn vortex_array::arrays::Shared::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::Shared::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::Shared -pub fn vortex_array::arrays::Shared::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Shared>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Shared::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Shared>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::Shared @@ -6466,59 +6618,55 @@ pub type vortex_array::arrays::Shared::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::Shared::ValidityVTable = vortex_array::arrays::Shared -pub fn vortex_array::arrays::Shared::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Shared::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Shared::buffer(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Shared::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Shared::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Shared::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Shared::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Shared::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Shared::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Shared::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Shared::deserialize(&self, _dtype: &vortex_array::dtype::DType, _len: usize, _metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Shared::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Shared::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Shared::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Shared::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Shared::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Shared::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Shared::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Shared::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Shared::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Shared::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Shared::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Shared::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Shared::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Shared::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Shared::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Shared::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Shared::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Shared::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Shared::validate(&self, _data: &vortex_array::arrays::shared::SharedData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Shared::validate(&self, &vortex_array::arrays::shared::SharedData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::Shared -pub fn vortex_array::arrays::Shared::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Shared>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Shared::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Shared>) -> vortex_error::VortexResult pub struct vortex_array::arrays::Slice -impl vortex_array::arrays::slice::Slice - -pub const vortex_array::arrays::slice::Slice::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::slice::Slice pub fn vortex_array::arrays::slice::Slice::clone(&self) -> vortex_array::arrays::slice::Slice impl core::fmt::Debug for vortex_array::arrays::slice::Slice -pub fn vortex_array::arrays::slice::Slice::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::slice::Slice::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::slice::Slice -pub fn vortex_array::arrays::slice::Slice::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::slice::Slice>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::slice::Slice::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::slice::Slice>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::slice::Slice @@ -6528,63 +6676,59 @@ pub type vortex_array::arrays::slice::Slice::OperationsVTable = vortex_array::ar pub type vortex_array::arrays::slice::Slice::ValidityVTable = vortex_array::arrays::slice::Slice -pub fn vortex_array::arrays::slice::Slice::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::slice::Slice::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::slice::Slice::buffer(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::slice::Slice::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::slice::Slice::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::slice::Slice::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::slice::Slice::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::slice::Slice::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::slice::Slice::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::slice::Slice::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::slice::Slice::deserialize(&self, _dtype: &vortex_array::dtype::DType, _len: usize, _metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::slice::Slice::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::slice::Slice::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::slice::Slice::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::slice::Slice::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::slice::Slice::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::slice::Slice::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::slice::Slice::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::slice::Slice::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::slice::Slice::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::slice::Slice::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::slice::Slice::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::slice::Slice::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::slice::Slice::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::slice::Slice::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::slice::Slice::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::slice::Slice::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::slice::Slice -pub fn vortex_array::arrays::slice::Slice::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::slice::Slice>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::slice::Slice::validity(vortex_array::ArrayView<'_, vortex_array::arrays::slice::Slice>) -> vortex_error::VortexResult impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::slice::Slice -pub fn vortex_array::arrays::slice::Slice::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> pub struct vortex_array::arrays::Struct -impl vortex_array::arrays::Struct - -pub const vortex_array::arrays::Struct::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::Struct pub fn vortex_array::arrays::Struct::clone(&self) -> vortex_array::arrays::Struct impl core::fmt::Debug for vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::Struct::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Struct::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::Struct @@ -6594,71 +6738,71 @@ pub type vortex_array::arrays::Struct::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::Struct::ValidityVTable = vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Struct::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Struct::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Struct::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Struct::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Struct::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Struct::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Struct::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Struct::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Struct::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Struct::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Struct::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Struct::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Struct::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Struct::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Struct::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Struct::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Struct::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Struct::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Struct::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Struct::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Struct::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Struct::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Struct::slot_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Struct::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Struct::validate(&self, _data: &vortex_array::EmptyArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Struct::validate(&self, &vortex_array::EmptyArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Struct>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Struct::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Struct>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeReduce for vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, indices: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::take(vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, dtype: &vortex_array::dtype::DType, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::Struct + +pub fn vortex_array::arrays::Struct::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::mask(vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::zip::ZipKernel for vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::zip(if_true: vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, if_false: &vortex_array::ArrayRef, mask: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::zip(vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, &vortex_array::ArrayRef, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub struct vortex_array::arrays::VarBin impl vortex_array::arrays::VarBin -pub const vortex_array::arrays::VarBin::ID: vortex_array::ArrayId - -impl vortex_array::arrays::VarBin - -pub fn vortex_array::arrays::VarBin::_slice(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, range: core::ops::range::Range) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBin::_slice(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, core::ops::range::Range) -> vortex_error::VortexResult impl core::clone::Clone for vortex_array::arrays::VarBin @@ -6666,11 +6810,11 @@ pub fn vortex_array::arrays::VarBin::clone(&self) -> vortex_array::arrays::VarBi impl core::fmt::Debug for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::VarBin::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBin::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::VarBin @@ -6680,83 +6824,83 @@ pub type vortex_array::arrays::VarBin::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::VarBin::ValidityVTable = vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::VarBin::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::VarBin::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::VarBin::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::VarBin::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::VarBin::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::VarBin::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::VarBin::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::VarBin::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::VarBin::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::VarBin::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBin::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBin::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::VarBin::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::VarBin::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::VarBin::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::VarBin::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::VarBin::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::VarBin::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::VarBin::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBin::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBin::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::VarBin::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::VarBin::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::VarBin::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::VarBin::validate(&self, _data: &vortex_array::arrays::varbin::VarBinData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::VarBin::validate(&self, &vortex_array::arrays::varbin::VarBinData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBin::validity(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::take(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::filter::FilterKernel for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::filter(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, mask: &vortex_mask::Mask, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::filter(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, &vortex_mask::Mask, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::binary::CompareKernel for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::compare(lhs: vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, rhs: &vortex_array::ArrayRef, operator: vortex_array::scalar_fn::fns::operators::CompareOperator, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::compare(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, &vortex_array::ArrayRef, vortex_array::scalar_fn::fns::operators::CompareOperator, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::VarBin + +pub fn vortex_array::arrays::VarBin::cast(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::cast(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::mask(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> pub struct vortex_array::arrays::VarBinView -impl vortex_array::arrays::VarBinView - -pub const vortex_array::arrays::VarBinView::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::VarBinView pub fn vortex_array::arrays::VarBinView::clone(&self) -> vortex_array::arrays::VarBinView impl core::fmt::Debug for vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::VarBinView::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBinView::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::VarBinView @@ -6766,79 +6910,79 @@ pub type vortex_array::arrays::VarBinView::OperationsVTable = vortex_array::arra pub type vortex_array::arrays::VarBinView::ValidityVTable = vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::VarBinView::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::VarBinView::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::VarBinView::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::VarBinView::buffer_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::VarBinView::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::VarBinView::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::VarBinView::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::VarBinView::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::VarBinView::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::VarBinView::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBinView::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBinView::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::VarBinView::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::VarBinView::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::VarBinView::nbuffers(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::VarBinView::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::VarBinView::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::VarBinView::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::VarBinView::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBinView::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBinView::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::VarBinView::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::VarBinView::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::VarBinView::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::VarBinView::validate(&self, data: &vortex_array::arrays::varbinview::VarBinViewData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::VarBinView::validate(&self, &vortex_array::arrays::varbinview::VarBinViewData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBinView::validity(vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>) -> vortex_error::VortexResult impl vortex_array::arrays::dict::TakeExecute for vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::take(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::take(vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrays::slice::SliceReduce for vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::slice(array: vortex_array::ArrayView<'_, Self>, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::slice(vortex_array::ArrayView<'_, Self>, core::ops::range::Range) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::VarBinView + +pub fn vortex_array::arrays::VarBinView::cast(vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::cast(vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::mask(vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::zip::ZipKernel for vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::zip(if_true: vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, if_false: &vortex_array::ArrayRef, mask: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::zip(vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, &vortex_array::ArrayRef, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub struct vortex_array::arrays::Variant -impl vortex_array::arrays::Variant - -pub const vortex_array::arrays::Variant::ID: vortex_array::ArrayId - impl core::clone::Clone for vortex_array::arrays::Variant pub fn vortex_array::arrays::Variant::clone(&self) -> vortex_array::arrays::Variant impl core::fmt::Debug for vortex_array::arrays::Variant -pub fn vortex_array::arrays::Variant::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrays::Variant::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::OperationsVTable for vortex_array::arrays::Variant -pub fn vortex_array::arrays::Variant::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Variant>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Variant::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Variant>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::VTable for vortex_array::arrays::Variant @@ -6848,41 +6992,41 @@ pub type vortex_array::arrays::Variant::OperationsVTable = vortex_array::arrays: pub type vortex_array::arrays::Variant::ValidityVTable = vortex_array::arrays::Variant -pub fn vortex_array::arrays::Variant::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Variant::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Variant::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Variant::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Variant::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Variant::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Variant::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Variant::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Variant::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Variant::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Variant::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Variant::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Variant::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Variant::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Variant::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Variant::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Variant::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Variant::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Variant::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Variant::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Variant::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Variant::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Variant::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Variant::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Variant::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Variant::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Variant::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Variant::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Variant::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Variant::validate(&self, _data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Variant::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::ValidityVTable for vortex_array::arrays::Variant -pub fn vortex_array::arrays::Variant::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Variant>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Variant::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Variant>) -> vortex_error::VortexResult pub type vortex_array::arrays::BoolArray = vortex_array::Array @@ -6912,7 +7056,7 @@ pub type vortex_array::arrays::PatchedArray = vortex_array::Array -pub type vortex_array::arrays::ScalarFnArray = vortex_array::Array +pub type vortex_array::arrays::ScalarFnArray = vortex_array::Array pub type vortex_array::arrays::SharedArray = vortex_array::Array @@ -6932,27 +7076,27 @@ pub mod vortex_array::arrow pub mod vortex_array::arrow::bool -pub fn vortex_array::arrow::bool::canonical_bool_to_arrow(array: &vortex_array::arrays::BoolArray) -> vortex_error::VortexResult +pub fn vortex_array::arrow::bool::canonical_bool_to_arrow(&vortex_array::arrays::BoolArray, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub mod vortex_array::arrow::byte_view -pub fn vortex_array::arrow::byte_view::canonical_varbinview_to_arrow(array: &vortex_array::arrays::VarBinViewArray) -> vortex_error::VortexResult +pub fn vortex_array::arrow::byte_view::canonical_varbinview_to_arrow(&vortex_array::arrays::VarBinViewArray, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrow::byte_view::execute_varbinview_to_arrow(array: &vortex_array::arrays::VarBinViewArray, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrow::byte_view::execute_varbinview_to_arrow(&vortex_array::arrays::VarBinViewArray, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub mod vortex_array::arrow::null -pub fn vortex_array::arrow::null::canonical_null_to_arrow(array: &vortex_array::arrays::null::NullArray) -> arrow_array::array::ArrayRef +pub fn vortex_array::arrow::null::canonical_null_to_arrow(&vortex_array::arrays::null::NullArray) -> arrow_array::array::ArrayRef pub mod vortex_array::arrow::primitive -pub fn vortex_array::arrow::primitive::canonical_primitive_to_arrow(array: vortex_array::arrays::PrimitiveArray) -> vortex_error::VortexResult where ::Native: vortex_array::dtype::NativePType +pub fn vortex_array::arrow::primitive::canonical_primitive_to_arrow(vortex_array::arrays::PrimitiveArray, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult where ::Native: vortex_array::dtype::NativePType pub struct vortex_array::arrow::ArrowArrayStreamAdapter impl vortex_array::arrow::ArrowArrayStreamAdapter -pub fn vortex_array::arrow::ArrowArrayStreamAdapter::new(stream: arrow_array::ffi_stream::ArrowArrayStreamReader, dtype: vortex_array::dtype::DType) -> Self +pub fn vortex_array::arrow::ArrowArrayStreamAdapter::new(arrow_array::ffi_stream::ArrowArrayStreamReader, vortex_array::dtype::DType) -> Self impl core::iter::traits::iterator::Iterator for vortex_array::arrow::ArrowArrayStreamAdapter @@ -6970,11 +7114,11 @@ impl vortex_array::arrow::Datum pub fn vortex_array::arrow::Datum::data_type(&self) -> &arrow_schema::datatype::DataType -pub fn vortex_array::arrow::Datum::try_new(array: &vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::arrow::Datum::try_new(&vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrow::Datum::try_new_array(array: &vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::arrow::Datum::try_new_array(&vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrow::Datum::try_new_with_target_datatype(array: &vortex_array::ArrayRef, target_datatype: &arrow_schema::datatype::DataType) -> vortex_error::VortexResult +pub fn vortex_array::arrow::Datum::try_new_with_target_datatype(&vortex_array::ArrayRef, &arrow_schema::datatype::DataType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl arrow_array::scalar::Datum for vortex_array::arrow::Datum @@ -6982,197 +7126,197 @@ pub fn vortex_array::arrow::Datum::get(&self) -> (&dyn arrow_array::array::Array impl core::fmt::Debug for vortex_array::arrow::Datum -pub fn vortex_array::arrow::Datum::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::arrow::Datum::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub trait vortex_array::arrow::ArrowArrayExecutor: core::marker::Sized -pub fn vortex_array::arrow::ArrowArrayExecutor::execute_arrow(self, data_type: core::option::Option<&arrow_schema::datatype::DataType>, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrow::ArrowArrayExecutor::execute_arrow(self, core::option::Option<&arrow_schema::datatype::DataType>, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrow::ArrowArrayExecutor::execute_record_batch(self, schema: &arrow_schema::schema::Schema, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrow::ArrowArrayExecutor::execute_record_batch(self, &arrow_schema::schema::Schema, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrow::ArrowArrayExecutor::execute_record_batches(self, schema: &arrow_schema::schema::Schema, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrow::ArrowArrayExecutor::execute_record_batches(self, &arrow_schema::schema::Schema, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrow::ArrowArrayExecutor for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::execute_arrow(self, data_type: core::option::Option<&arrow_schema::datatype::DataType>, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::execute_arrow(self, core::option::Option<&arrow_schema::datatype::DataType>, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::execute_record_batch(self, schema: &arrow_schema::schema::Schema, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::execute_record_batch(self, &arrow_schema::schema::Schema, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::execute_record_batches(self, schema: &arrow_schema::schema::Schema, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::ArrayRef::execute_record_batches(self, &arrow_schema::schema::Schema, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub trait vortex_array::arrow::FromArrowArray -pub fn vortex_array::arrow::FromArrowArray::from_arrow(array: A, nullable: bool) -> vortex_error::VortexResult where Self: core::marker::Sized +pub fn vortex_array::arrow::FromArrowArray::from_arrow(A, bool) -> vortex_error::VortexResult where Self: core::marker::Sized impl vortex_array::arrow::FromArrowArray<&arrow_array::array::boolean_array::BooleanArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::boolean_array::BooleanArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::boolean_array::BooleanArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::fixed_size_list_array::FixedSizeListArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(array: &arrow_array::array::fixed_size_list_array::FixedSizeListArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::fixed_size_list_array::FixedSizeListArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::map_array::MapArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::map_array::MapArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::map_array::MapArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::null_array::NullArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::null_array::NullArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::null_array::NullArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(array: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(array: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(array: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(array: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::struct_array::StructArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::struct_array::StructArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::struct_array::StructArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::record_batch::RecordBatch> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(array: &arrow_array::record_batch::RecordBatch, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::record_batch::RecordBatch, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&dyn arrow_array::array::Array> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(array: &dyn arrow_array::array::Array, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&dyn arrow_array::array::Array, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(array: arrow_array::record_batch::RecordBatch, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(arrow_array::record_batch::RecordBatch, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::dictionary_array::DictionaryArray> for vortex_array::arrays::dict::DictArray -pub fn vortex_array::arrays::dict::DictArray::from_arrow(array: &arrow_array::array::dictionary_array::DictionaryArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::arrays::dict::DictArray::from_arrow(&arrow_array::array::dictionary_array::DictionaryArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::list_view_array::GenericListViewArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(array: &arrow_array::array::list_view_array::GenericListViewArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::list_view_array::GenericListViewArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::list_array::GenericListArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::list_array::GenericListArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::list_array::GenericListArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::byte_array::GenericByteArray> for vortex_array::ArrayRef where ::Offset: vortex_array::dtype::IntegerPType -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::byte_array::GenericByteArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::byte_array::GenericByteArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::byte_view_array::GenericByteViewArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::byte_view_array::GenericByteViewArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::byte_view_array::GenericByteViewArray, bool) -> vortex_error::VortexResult pub trait vortex_array::arrow::IntoArrowArray -pub fn vortex_array::arrow::IntoArrowArray::into_arrow(self, data_type: &arrow_schema::datatype::DataType) -> vortex_error::VortexResult +pub fn vortex_array::arrow::IntoArrowArray::into_arrow(self, &arrow_schema::datatype::DataType) -> vortex_error::VortexResult pub fn vortex_array::arrow::IntoArrowArray::into_arrow_preferred(self) -> vortex_error::VortexResult impl vortex_array::arrow::IntoArrowArray for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::into_arrow(self, data_type: &arrow_schema::datatype::DataType) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::into_arrow(self, &arrow_schema::datatype::DataType) -> vortex_error::VortexResult pub fn vortex_array::ArrayRef::into_arrow_preferred(self) -> vortex_error::VortexResult -pub fn vortex_array::arrow::from_arrow_array_with_len(array: A, len: usize, nullable: bool) -> vortex_error::VortexResult where vortex_array::ArrayRef: vortex_array::arrow::FromArrowArray +pub fn vortex_array::arrow::from_arrow_array_with_len(A, usize, bool) -> vortex_error::VortexResult where vortex_array::ArrayRef: vortex_array::arrow::FromArrowArray -pub fn vortex_array::arrow::to_arrow_null_buffer(validity: vortex_array::validity::Validity, len: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrow::to_arrow_null_buffer(vortex_array::validity::Validity, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> -pub fn vortex_array::arrow::to_null_buffer(mask: vortex_mask::Mask) -> core::option::Option +pub fn vortex_array::arrow::to_null_buffer(vortex_mask::Mask) -> core::option::Option pub mod vortex_array::buffer @@ -7190,13 +7334,13 @@ pub fn vortex_array::buffer::BufferHandle::as_host(&self) -> &vortex_buffer::Byt pub fn vortex_array::buffer::BufferHandle::as_host_opt(&self) -> core::option::Option<&vortex_buffer::ByteBuffer> -pub fn vortex_array::buffer::BufferHandle::ensure_aligned(self, alignment: vortex_buffer::alignment::Alignment) -> vortex_error::VortexResult +pub fn vortex_array::buffer::BufferHandle::ensure_aligned(self, vortex_buffer::alignment::Alignment) -> vortex_error::VortexResult pub fn vortex_array::buffer::BufferHandle::into_host(self) -> futures_core::future::BoxFuture<'static, vortex_buffer::ByteBuffer> pub fn vortex_array::buffer::BufferHandle::into_host_sync(self) -> vortex_buffer::ByteBuffer -pub fn vortex_array::buffer::BufferHandle::is_aligned_to(&self, alignment: vortex_buffer::alignment::Alignment) -> bool +pub fn vortex_array::buffer::BufferHandle::is_aligned_to(&self, vortex_buffer::alignment::Alignment) -> bool pub fn vortex_array::buffer::BufferHandle::is_empty(&self) -> bool @@ -7206,9 +7350,9 @@ pub fn vortex_array::buffer::BufferHandle::is_on_host(&self) -> bool pub fn vortex_array::buffer::BufferHandle::len(&self) -> usize -pub fn vortex_array::buffer::BufferHandle::slice(&self, range: core::ops::range::Range) -> Self +pub fn vortex_array::buffer::BufferHandle::slice(&self, core::ops::range::Range) -> Self -pub fn vortex_array::buffer::BufferHandle::slice_typed(&self, range: core::ops::range::Range) -> Self +pub fn vortex_array::buffer::BufferHandle::slice_typed(&self, core::ops::range::Range) -> Self pub fn vortex_array::buffer::BufferHandle::to_host(&self) -> futures_core::future::BoxFuture<'static, vortex_buffer::ByteBuffer> @@ -7228,9 +7372,9 @@ pub fn vortex_array::buffer::BufferHandle::unwrap_host(self) -> vortex_buffer::B impl vortex_array::buffer::BufferHandle -pub fn vortex_array::buffer::BufferHandle::new_device(device: alloc::sync::Arc) -> Self +pub fn vortex_array::buffer::BufferHandle::new_device(alloc::sync::Arc) -> Self -pub fn vortex_array::buffer::BufferHandle::new_host(byte_buffer: vortex_buffer::ByteBuffer) -> Self +pub fn vortex_array::buffer::BufferHandle::new_host(vortex_buffer::ByteBuffer) -> Self impl core::clone::Clone for vortex_array::buffer::BufferHandle @@ -7240,45 +7384,45 @@ impl core::convert::TryFrom for vortex_array pub type vortex_array::serde::SerializedArray::Error = vortex_error::VortexError -pub fn vortex_array::serde::SerializedArray::try_from(value: vortex_array::buffer::BufferHandle) -> core::result::Result +pub fn vortex_array::serde::SerializedArray::try_from(vortex_array::buffer::BufferHandle) -> core::result::Result impl core::fmt::Debug for vortex_array::buffer::BufferHandle -pub fn vortex_array::buffer::BufferHandle::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::buffer::BufferHandle::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::ArrayEq for vortex_array::buffer::BufferHandle -pub fn vortex_array::buffer::BufferHandle::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn vortex_array::buffer::BufferHandle::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayHash for vortex_array::buffer::BufferHandle -pub fn vortex_array::buffer::BufferHandle::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn vortex_array::buffer::BufferHandle::array_hash(&self, &mut H, vortex_array::Precision) pub trait vortex_array::buffer::DeviceBuffer: 'static + core::marker::Send + core::marker::Sync + core::fmt::Debug + vortex_utils::dyn_traits::DynEq + vortex_utils::dyn_traits::DynHash -pub fn vortex_array::buffer::DeviceBuffer::aligned(self: alloc::sync::Arc, alignment: vortex_buffer::alignment::Alignment) -> vortex_error::VortexResult> +pub fn vortex_array::buffer::DeviceBuffer::aligned(alloc::sync::Arc, vortex_buffer::alignment::Alignment) -> vortex_error::VortexResult> pub fn vortex_array::buffer::DeviceBuffer::alignment(&self) -> vortex_buffer::alignment::Alignment pub fn vortex_array::buffer::DeviceBuffer::as_any(&self) -> &dyn core::any::Any -pub fn vortex_array::buffer::DeviceBuffer::copy_to_host(&self, alignment: vortex_buffer::alignment::Alignment) -> vortex_error::VortexResult>> +pub fn vortex_array::buffer::DeviceBuffer::copy_to_host(&self, vortex_buffer::alignment::Alignment) -> vortex_error::VortexResult>> -pub fn vortex_array::buffer::DeviceBuffer::copy_to_host_sync(&self, alignment: vortex_buffer::alignment::Alignment) -> vortex_error::VortexResult +pub fn vortex_array::buffer::DeviceBuffer::copy_to_host_sync(&self, vortex_buffer::alignment::Alignment) -> vortex_error::VortexResult pub fn vortex_array::buffer::DeviceBuffer::is_empty(&self) -> bool pub fn vortex_array::buffer::DeviceBuffer::len(&self) -> usize -pub fn vortex_array::buffer::DeviceBuffer::slice(&self, range: core::ops::range::Range) -> alloc::sync::Arc +pub fn vortex_array::buffer::DeviceBuffer::slice(&self, core::ops::range::Range) -> alloc::sync::Arc pub trait vortex_array::buffer::DeviceBufferExt: vortex_array::buffer::DeviceBuffer -pub fn vortex_array::buffer::DeviceBufferExt::slice_typed(&self, range: core::ops::range::Range) -> alloc::sync::Arc +pub fn vortex_array::buffer::DeviceBufferExt::slice_typed(&self, core::ops::range::Range) -> alloc::sync::Arc impl vortex_array::buffer::DeviceBufferExt for B -pub fn B::slice_typed(&self, range: core::ops::range::Range) -> alloc::sync::Arc +pub fn B::slice_typed(&self, core::ops::range::Range) -> alloc::sync::Arc pub mod vortex_array::builders @@ -7300,15 +7444,15 @@ pub trait vortex_array::builders::dict::DictEncoder: core::marker::Send pub fn vortex_array::builders::dict::DictEncoder::codes_ptype(&self) -> vortex_array::dtype::PType -pub fn vortex_array::builders::dict::DictEncoder::encode(&mut self, array: &vortex_array::ArrayRef) -> vortex_array::ArrayRef +pub fn vortex_array::builders::dict::DictEncoder::encode(&mut self, &vortex_array::ArrayRef) -> vortex_array::ArrayRef pub fn vortex_array::builders::dict::DictEncoder::reset(&mut self) -> vortex_array::ArrayRef -pub fn vortex_array::builders::dict::dict_encode(array: &vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::builders::dict::dict_encode(&vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::builders::dict::dict_encode_with_constraints(array: &vortex_array::ArrayRef, constraints: &vortex_array::builders::dict::DictConstraints) -> vortex_error::VortexResult +pub fn vortex_array::builders::dict::dict_encode_with_constraints(&vortex_array::ArrayRef, &vortex_array::builders::dict::DictConstraints) -> vortex_error::VortexResult -pub fn vortex_array::builders::dict::dict_encoder(array: &vortex_array::ArrayRef, constraints: &vortex_array::builders::dict::DictConstraints) -> alloc::boxed::Box +pub fn vortex_array::builders::dict::dict_encoder(&vortex_array::ArrayRef, &vortex_array::builders::dict::DictConstraints) -> alloc::boxed::Box pub enum vortex_array::builders::BufferGrowthStrategy @@ -7324,9 +7468,9 @@ pub vortex_array::builders::BufferGrowthStrategy::Fixed::size: u32 impl vortex_array::builders::BufferGrowthStrategy -pub fn vortex_array::builders::BufferGrowthStrategy::exponential(initial_size: u32, max_size: u32) -> Self +pub fn vortex_array::builders::BufferGrowthStrategy::exponential(u32, u32) -> Self -pub fn vortex_array::builders::BufferGrowthStrategy::fixed(size: u32) -> Self +pub fn vortex_array::builders::BufferGrowthStrategy::fixed(u32) -> Self pub fn vortex_array::builders::BufferGrowthStrategy::next_size(&mut self) -> u32 @@ -7340,7 +7484,7 @@ pub fn vortex_array::builders::BufferGrowthStrategy::default() -> Self impl core::fmt::Debug for vortex_array::builders::BufferGrowthStrategy -pub fn vortex_array::builders::BufferGrowthStrategy::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::builders::BufferGrowthStrategy::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub enum vortex_array::builders::CompletedBuffers @@ -7356,33 +7500,33 @@ pub struct vortex_array::builders::BoolBuilder impl vortex_array::builders::BoolBuilder -pub fn vortex_array::builders::BoolBuilder::append_value(&mut self, value: bool) +pub fn vortex_array::builders::BoolBuilder::append_value(&mut self, bool) -pub fn vortex_array::builders::BoolBuilder::append_values(&mut self, value: bool, n: usize) +pub fn vortex_array::builders::BoolBuilder::append_values(&mut self, bool, usize) pub fn vortex_array::builders::BoolBuilder::finish_into_bool(&mut self) -> vortex_array::arrays::BoolArray -pub fn vortex_array::builders::BoolBuilder::new(nullability: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::builders::BoolBuilder::new(vortex_array::dtype::Nullability) -> Self -pub fn vortex_array::builders::BoolBuilder::with_capacity(nullability: vortex_array::dtype::Nullability, capacity: usize) -> Self +pub fn vortex_array::builders::BoolBuilder::with_capacity(vortex_array::dtype::Nullability, usize) -> Self impl vortex_array::builders::ArrayBuilder for vortex_array::builders::BoolBuilder pub fn vortex_array::builders::BoolBuilder::append_default(&mut self) -pub fn vortex_array::builders::BoolBuilder::append_defaults(&mut self, n: usize) +pub fn vortex_array::builders::BoolBuilder::append_defaults(&mut self, usize) pub fn vortex_array::builders::BoolBuilder::append_null(&mut self) -pub fn vortex_array::builders::BoolBuilder::append_nulls(&mut self, n: usize) +pub fn vortex_array::builders::BoolBuilder::append_nulls(&mut self, usize) -pub unsafe fn vortex_array::builders::BoolBuilder::append_nulls_unchecked(&mut self, n: usize) +pub unsafe fn vortex_array::builders::BoolBuilder::append_nulls_unchecked(&mut self, usize) -pub fn vortex_array::builders::BoolBuilder::append_scalar(&mut self, scalar: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::BoolBuilder::append_scalar(&mut self, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::BoolBuilder::append_zero(&mut self) -pub fn vortex_array::builders::BoolBuilder::append_zeros(&mut self, n: usize) +pub fn vortex_array::builders::BoolBuilder::append_zeros(&mut self, usize) pub fn vortex_array::builders::BoolBuilder::as_any(&self) -> &dyn core::any::Any @@ -7390,9 +7534,9 @@ pub fn vortex_array::builders::BoolBuilder::as_any_mut(&mut self) -> &mut dyn co pub fn vortex_array::builders::BoolBuilder::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::builders::BoolBuilder::extend_from_array(&mut self, array: &vortex_array::ArrayRef) +pub fn vortex_array::builders::BoolBuilder::extend_from_array(&mut self, &vortex_array::ArrayRef) -pub unsafe fn vortex_array::builders::BoolBuilder::extend_from_array_unchecked(&mut self, array: &vortex_array::ArrayRef) +pub unsafe fn vortex_array::builders::BoolBuilder::extend_from_array_unchecked(&mut self, &vortex_array::ArrayRef) pub fn vortex_array::builders::BoolBuilder::finish(&mut self) -> vortex_array::ArrayRef @@ -7402,45 +7546,45 @@ pub fn vortex_array::builders::BoolBuilder::is_empty(&self) -> bool pub fn vortex_array::builders::BoolBuilder::len(&self) -> usize -pub fn vortex_array::builders::BoolBuilder::reserve_exact(&mut self, additional: usize) +pub fn vortex_array::builders::BoolBuilder::reserve_exact(&mut self, usize) -pub fn vortex_array::builders::BoolBuilder::set_validity(&mut self, validity: vortex_mask::Mask) +pub fn vortex_array::builders::BoolBuilder::set_validity(&mut self, vortex_mask::Mask) -pub unsafe fn vortex_array::builders::BoolBuilder::set_validity_unchecked(&mut self, validity: vortex_mask::Mask) +pub unsafe fn vortex_array::builders::BoolBuilder::set_validity_unchecked(&mut self, vortex_mask::Mask) pub struct vortex_array::builders::DecimalBuilder impl vortex_array::builders::DecimalBuilder -pub fn vortex_array::builders::DecimalBuilder::append_n_values(&mut self, value: V, n: usize) +pub fn vortex_array::builders::DecimalBuilder::append_n_values(&mut self, V, usize) -pub fn vortex_array::builders::DecimalBuilder::append_value(&mut self, value: V) +pub fn vortex_array::builders::DecimalBuilder::append_value(&mut self, V) pub fn vortex_array::builders::DecimalBuilder::decimal_dtype(&self) -> &vortex_array::dtype::DecimalDType pub fn vortex_array::builders::DecimalBuilder::finish_into_decimal(&mut self) -> vortex_array::arrays::DecimalArray -pub fn vortex_array::builders::DecimalBuilder::new(decimal: vortex_array::dtype::DecimalDType, nullability: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::builders::DecimalBuilder::new(vortex_array::dtype::DecimalDType, vortex_array::dtype::Nullability) -> Self -pub fn vortex_array::builders::DecimalBuilder::with_capacity(capacity: usize, decimal: vortex_array::dtype::DecimalDType, nullability: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::builders::DecimalBuilder::with_capacity(usize, vortex_array::dtype::DecimalDType, vortex_array::dtype::Nullability) -> Self impl vortex_array::builders::ArrayBuilder for vortex_array::builders::DecimalBuilder pub fn vortex_array::builders::DecimalBuilder::append_default(&mut self) -pub fn vortex_array::builders::DecimalBuilder::append_defaults(&mut self, n: usize) +pub fn vortex_array::builders::DecimalBuilder::append_defaults(&mut self, usize) pub fn vortex_array::builders::DecimalBuilder::append_null(&mut self) -pub fn vortex_array::builders::DecimalBuilder::append_nulls(&mut self, n: usize) +pub fn vortex_array::builders::DecimalBuilder::append_nulls(&mut self, usize) -pub unsafe fn vortex_array::builders::DecimalBuilder::append_nulls_unchecked(&mut self, n: usize) +pub unsafe fn vortex_array::builders::DecimalBuilder::append_nulls_unchecked(&mut self, usize) -pub fn vortex_array::builders::DecimalBuilder::append_scalar(&mut self, scalar: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::DecimalBuilder::append_scalar(&mut self, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::DecimalBuilder::append_zero(&mut self) -pub fn vortex_array::builders::DecimalBuilder::append_zeros(&mut self, n: usize) +pub fn vortex_array::builders::DecimalBuilder::append_zeros(&mut self, usize) pub fn vortex_array::builders::DecimalBuilder::as_any(&self) -> &dyn core::any::Any @@ -7448,9 +7592,9 @@ pub fn vortex_array::builders::DecimalBuilder::as_any_mut(&mut self) -> &mut dyn pub fn vortex_array::builders::DecimalBuilder::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::builders::DecimalBuilder::extend_from_array(&mut self, array: &vortex_array::ArrayRef) +pub fn vortex_array::builders::DecimalBuilder::extend_from_array(&mut self, &vortex_array::ArrayRef) -pub unsafe fn vortex_array::builders::DecimalBuilder::extend_from_array_unchecked(&mut self, array: &vortex_array::ArrayRef) +pub unsafe fn vortex_array::builders::DecimalBuilder::extend_from_array_unchecked(&mut self, &vortex_array::ArrayRef) pub fn vortex_array::builders::DecimalBuilder::finish(&mut self) -> vortex_array::ArrayRef @@ -7460,11 +7604,11 @@ pub fn vortex_array::builders::DecimalBuilder::is_empty(&self) -> bool pub fn vortex_array::builders::DecimalBuilder::len(&self) -> usize -pub fn vortex_array::builders::DecimalBuilder::reserve_exact(&mut self, additional: usize) +pub fn vortex_array::builders::DecimalBuilder::reserve_exact(&mut self, usize) -pub fn vortex_array::builders::DecimalBuilder::set_validity(&mut self, validity: vortex_mask::Mask) +pub fn vortex_array::builders::DecimalBuilder::set_validity(&mut self, vortex_mask::Mask) -pub unsafe fn vortex_array::builders::DecimalBuilder::set_validity_unchecked(&mut self, validity: vortex_mask::Mask) +pub unsafe fn vortex_array::builders::DecimalBuilder::set_validity_unchecked(&mut self, vortex_mask::Mask) pub struct vortex_array::builders::DeduplicatedBuffers @@ -7476,31 +7620,31 @@ pub struct vortex_array::builders::ExtensionBuilder impl vortex_array::builders::ExtensionBuilder -pub fn vortex_array::builders::ExtensionBuilder::append_value(&mut self, value: vortex_array::scalar::ExtScalar<'_>) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::ExtensionBuilder::append_value(&mut self, vortex_array::scalar::ExtScalar<'_>) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::ExtensionBuilder::finish_into_extension(&mut self) -> vortex_array::arrays::ExtensionArray -pub fn vortex_array::builders::ExtensionBuilder::new(ext_dtype: vortex_array::dtype::extension::ExtDTypeRef) -> Self +pub fn vortex_array::builders::ExtensionBuilder::new(vortex_array::dtype::extension::ExtDTypeRef) -> Self -pub fn vortex_array::builders::ExtensionBuilder::with_capacity(ext_dtype: vortex_array::dtype::extension::ExtDTypeRef, capacity: usize) -> Self +pub fn vortex_array::builders::ExtensionBuilder::with_capacity(vortex_array::dtype::extension::ExtDTypeRef, usize) -> Self impl vortex_array::builders::ArrayBuilder for vortex_array::builders::ExtensionBuilder pub fn vortex_array::builders::ExtensionBuilder::append_default(&mut self) -pub fn vortex_array::builders::ExtensionBuilder::append_defaults(&mut self, n: usize) +pub fn vortex_array::builders::ExtensionBuilder::append_defaults(&mut self, usize) pub fn vortex_array::builders::ExtensionBuilder::append_null(&mut self) -pub fn vortex_array::builders::ExtensionBuilder::append_nulls(&mut self, n: usize) +pub fn vortex_array::builders::ExtensionBuilder::append_nulls(&mut self, usize) -pub unsafe fn vortex_array::builders::ExtensionBuilder::append_nulls_unchecked(&mut self, n: usize) +pub unsafe fn vortex_array::builders::ExtensionBuilder::append_nulls_unchecked(&mut self, usize) -pub fn vortex_array::builders::ExtensionBuilder::append_scalar(&mut self, scalar: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::ExtensionBuilder::append_scalar(&mut self, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::ExtensionBuilder::append_zero(&mut self) -pub fn vortex_array::builders::ExtensionBuilder::append_zeros(&mut self, n: usize) +pub fn vortex_array::builders::ExtensionBuilder::append_zeros(&mut self, usize) pub fn vortex_array::builders::ExtensionBuilder::as_any(&self) -> &dyn core::any::Any @@ -7508,9 +7652,9 @@ pub fn vortex_array::builders::ExtensionBuilder::as_any_mut(&mut self) -> &mut d pub fn vortex_array::builders::ExtensionBuilder::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::builders::ExtensionBuilder::extend_from_array(&mut self, array: &vortex_array::ArrayRef) +pub fn vortex_array::builders::ExtensionBuilder::extend_from_array(&mut self, &vortex_array::ArrayRef) -pub unsafe fn vortex_array::builders::ExtensionBuilder::extend_from_array_unchecked(&mut self, array: &vortex_array::ArrayRef) +pub unsafe fn vortex_array::builders::ExtensionBuilder::extend_from_array_unchecked(&mut self, &vortex_array::ArrayRef) pub fn vortex_array::builders::ExtensionBuilder::finish(&mut self) -> vortex_array::ArrayRef @@ -7520,19 +7664,19 @@ pub fn vortex_array::builders::ExtensionBuilder::is_empty(&self) -> bool pub fn vortex_array::builders::ExtensionBuilder::len(&self) -> usize -pub fn vortex_array::builders::ExtensionBuilder::reserve_exact(&mut self, capacity: usize) +pub fn vortex_array::builders::ExtensionBuilder::reserve_exact(&mut self, usize) -pub fn vortex_array::builders::ExtensionBuilder::set_validity(&mut self, validity: vortex_mask::Mask) +pub fn vortex_array::builders::ExtensionBuilder::set_validity(&mut self, vortex_mask::Mask) -pub unsafe fn vortex_array::builders::ExtensionBuilder::set_validity_unchecked(&mut self, validity: vortex_mask::Mask) +pub unsafe fn vortex_array::builders::ExtensionBuilder::set_validity_unchecked(&mut self, vortex_mask::Mask) pub struct vortex_array::builders::FixedSizeListBuilder impl vortex_array::builders::FixedSizeListBuilder -pub fn vortex_array::builders::FixedSizeListBuilder::append_array_as_list(&mut self, array: &vortex_array::ArrayRef) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::FixedSizeListBuilder::append_array_as_list(&mut self, &vortex_array::ArrayRef) -> vortex_error::VortexResult<()> -pub fn vortex_array::builders::FixedSizeListBuilder::append_value(&mut self, value: vortex_array::scalar::ListScalar<'_>) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::FixedSizeListBuilder::append_value(&mut self, vortex_array::scalar::ListScalar<'_>) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::FixedSizeListBuilder::element_dtype(&self) -> &vortex_array::dtype::DType @@ -7540,27 +7684,27 @@ pub fn vortex_array::builders::FixedSizeListBuilder::finish_into_fixed_size_list pub fn vortex_array::builders::FixedSizeListBuilder::list_size(&self) -> u32 -pub fn vortex_array::builders::FixedSizeListBuilder::new(element_dtype: alloc::sync::Arc, list_size: u32, nullability: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::builders::FixedSizeListBuilder::new(alloc::sync::Arc, u32, vortex_array::dtype::Nullability) -> Self -pub fn vortex_array::builders::FixedSizeListBuilder::with_capacity(element_dtype: alloc::sync::Arc, list_size: u32, nullability: vortex_array::dtype::Nullability, capacity: usize) -> Self +pub fn vortex_array::builders::FixedSizeListBuilder::with_capacity(alloc::sync::Arc, u32, vortex_array::dtype::Nullability, usize) -> Self impl vortex_array::builders::ArrayBuilder for vortex_array::builders::FixedSizeListBuilder pub fn vortex_array::builders::FixedSizeListBuilder::append_default(&mut self) -pub fn vortex_array::builders::FixedSizeListBuilder::append_defaults(&mut self, n: usize) +pub fn vortex_array::builders::FixedSizeListBuilder::append_defaults(&mut self, usize) pub fn vortex_array::builders::FixedSizeListBuilder::append_null(&mut self) -pub fn vortex_array::builders::FixedSizeListBuilder::append_nulls(&mut self, n: usize) +pub fn vortex_array::builders::FixedSizeListBuilder::append_nulls(&mut self, usize) -pub unsafe fn vortex_array::builders::FixedSizeListBuilder::append_nulls_unchecked(&mut self, n: usize) +pub unsafe fn vortex_array::builders::FixedSizeListBuilder::append_nulls_unchecked(&mut self, usize) -pub fn vortex_array::builders::FixedSizeListBuilder::append_scalar(&mut self, scalar: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::FixedSizeListBuilder::append_scalar(&mut self, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::FixedSizeListBuilder::append_zero(&mut self) -pub fn vortex_array::builders::FixedSizeListBuilder::append_zeros(&mut self, n: usize) +pub fn vortex_array::builders::FixedSizeListBuilder::append_zeros(&mut self, usize) pub fn vortex_array::builders::FixedSizeListBuilder::as_any(&self) -> &dyn core::any::Any @@ -7568,9 +7712,9 @@ pub fn vortex_array::builders::FixedSizeListBuilder::as_any_mut(&mut self) -> &m pub fn vortex_array::builders::FixedSizeListBuilder::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::builders::FixedSizeListBuilder::extend_from_array(&mut self, array: &vortex_array::ArrayRef) +pub fn vortex_array::builders::FixedSizeListBuilder::extend_from_array(&mut self, &vortex_array::ArrayRef) -pub unsafe fn vortex_array::builders::FixedSizeListBuilder::extend_from_array_unchecked(&mut self, array: &vortex_array::ArrayRef) +pub unsafe fn vortex_array::builders::FixedSizeListBuilder::extend_from_array_unchecked(&mut self, &vortex_array::ArrayRef) pub fn vortex_array::builders::FixedSizeListBuilder::finish(&mut self) -> vortex_array::ArrayRef @@ -7580,45 +7724,45 @@ pub fn vortex_array::builders::FixedSizeListBuilder::is_empty(&self) -> bool pub fn vortex_array::builders::FixedSizeListBuilder::len(&self) -> usize -pub fn vortex_array::builders::FixedSizeListBuilder::reserve_exact(&mut self, additional: usize) +pub fn vortex_array::builders::FixedSizeListBuilder::reserve_exact(&mut self, usize) -pub fn vortex_array::builders::FixedSizeListBuilder::set_validity(&mut self, validity: vortex_mask::Mask) +pub fn vortex_array::builders::FixedSizeListBuilder::set_validity(&mut self, vortex_mask::Mask) -pub unsafe fn vortex_array::builders::FixedSizeListBuilder::set_validity_unchecked(&mut self, validity: vortex_mask::Mask) +pub unsafe fn vortex_array::builders::FixedSizeListBuilder::set_validity_unchecked(&mut self, vortex_mask::Mask) pub struct vortex_array::builders::ListBuilder impl vortex_array::builders::ListBuilder -pub fn vortex_array::builders::ListBuilder::append_array_as_list(&mut self, array: &vortex_array::ArrayRef) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::ListBuilder::append_array_as_list(&mut self, &vortex_array::ArrayRef) -> vortex_error::VortexResult<()> -pub fn vortex_array::builders::ListBuilder::append_value(&mut self, value: vortex_array::scalar::ListScalar<'_>) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::ListBuilder::append_value(&mut self, vortex_array::scalar::ListScalar<'_>) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::ListBuilder::element_dtype(&self) -> &vortex_array::dtype::DType pub fn vortex_array::builders::ListBuilder::finish_into_list(&mut self) -> vortex_array::arrays::ListArray -pub fn vortex_array::builders::ListBuilder::new(value_dtype: alloc::sync::Arc, nullability: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::builders::ListBuilder::new(alloc::sync::Arc, vortex_array::dtype::Nullability) -> Self -pub fn vortex_array::builders::ListBuilder::with_capacity(value_dtype: alloc::sync::Arc, nullability: vortex_array::dtype::Nullability, elements_capacity: usize, capacity: usize) -> Self +pub fn vortex_array::builders::ListBuilder::with_capacity(alloc::sync::Arc, vortex_array::dtype::Nullability, usize, usize) -> Self impl vortex_array::builders::ArrayBuilder for vortex_array::builders::ListBuilder pub fn vortex_array::builders::ListBuilder::append_default(&mut self) -pub fn vortex_array::builders::ListBuilder::append_defaults(&mut self, n: usize) +pub fn vortex_array::builders::ListBuilder::append_defaults(&mut self, usize) pub fn vortex_array::builders::ListBuilder::append_null(&mut self) -pub fn vortex_array::builders::ListBuilder::append_nulls(&mut self, n: usize) +pub fn vortex_array::builders::ListBuilder::append_nulls(&mut self, usize) -pub unsafe fn vortex_array::builders::ListBuilder::append_nulls_unchecked(&mut self, n: usize) +pub unsafe fn vortex_array::builders::ListBuilder::append_nulls_unchecked(&mut self, usize) -pub fn vortex_array::builders::ListBuilder::append_scalar(&mut self, scalar: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::ListBuilder::append_scalar(&mut self, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::ListBuilder::append_zero(&mut self) -pub fn vortex_array::builders::ListBuilder::append_zeros(&mut self, n: usize) +pub fn vortex_array::builders::ListBuilder::append_zeros(&mut self, usize) pub fn vortex_array::builders::ListBuilder::as_any(&self) -> &dyn core::any::Any @@ -7626,9 +7770,9 @@ pub fn vortex_array::builders::ListBuilder::as_any_mut(&mut self) -> &mut dyn pub fn vortex_array::builders::ListBuilder::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::builders::ListBuilder::extend_from_array(&mut self, array: &vortex_array::ArrayRef) +pub fn vortex_array::builders::ListBuilder::extend_from_array(&mut self, &vortex_array::ArrayRef) -pub unsafe fn vortex_array::builders::ListBuilder::extend_from_array_unchecked(&mut self, array: &vortex_array::ArrayRef) +pub unsafe fn vortex_array::builders::ListBuilder::extend_from_array_unchecked(&mut self, &vortex_array::ArrayRef) pub fn vortex_array::builders::ListBuilder::finish(&mut self) -> vortex_array::ArrayRef @@ -7638,45 +7782,45 @@ pub fn vortex_array::builders::ListBuilder::is_empty(&self) -> bool pub fn vortex_array::builders::ListBuilder::len(&self) -> usize -pub fn vortex_array::builders::ListBuilder::reserve_exact(&mut self, additional: usize) +pub fn vortex_array::builders::ListBuilder::reserve_exact(&mut self, usize) -pub fn vortex_array::builders::ListBuilder::set_validity(&mut self, validity: vortex_mask::Mask) +pub fn vortex_array::builders::ListBuilder::set_validity(&mut self, vortex_mask::Mask) -pub unsafe fn vortex_array::builders::ListBuilder::set_validity_unchecked(&mut self, validity: vortex_mask::Mask) +pub unsafe fn vortex_array::builders::ListBuilder::set_validity_unchecked(&mut self, vortex_mask::Mask) pub struct vortex_array::builders::ListViewBuilder impl vortex_array::builders::ListViewBuilder -pub fn vortex_array::builders::ListViewBuilder::append_array_as_list(&mut self, array: &vortex_array::ArrayRef) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::ListViewBuilder::append_array_as_list(&mut self, &vortex_array::ArrayRef) -> vortex_error::VortexResult<()> -pub fn vortex_array::builders::ListViewBuilder::append_value(&mut self, value: vortex_array::scalar::ListScalar<'_>) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::ListViewBuilder::append_value(&mut self, vortex_array::scalar::ListScalar<'_>) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::ListViewBuilder::element_dtype(&self) -> &vortex_array::dtype::DType pub fn vortex_array::builders::ListViewBuilder::finish_into_listview(&mut self) -> vortex_array::arrays::ListViewArray -pub fn vortex_array::builders::ListViewBuilder::new(element_dtype: alloc::sync::Arc, nullability: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::builders::ListViewBuilder::new(alloc::sync::Arc, vortex_array::dtype::Nullability) -> Self -pub fn vortex_array::builders::ListViewBuilder::with_capacity(element_dtype: alloc::sync::Arc, nullability: vortex_array::dtype::Nullability, elements_capacity: usize, capacity: usize) -> Self +pub fn vortex_array::builders::ListViewBuilder::with_capacity(alloc::sync::Arc, vortex_array::dtype::Nullability, usize, usize) -> Self impl vortex_array::builders::ArrayBuilder for vortex_array::builders::ListViewBuilder pub fn vortex_array::builders::ListViewBuilder::append_default(&mut self) -pub fn vortex_array::builders::ListViewBuilder::append_defaults(&mut self, n: usize) +pub fn vortex_array::builders::ListViewBuilder::append_defaults(&mut self, usize) pub fn vortex_array::builders::ListViewBuilder::append_null(&mut self) -pub fn vortex_array::builders::ListViewBuilder::append_nulls(&mut self, n: usize) +pub fn vortex_array::builders::ListViewBuilder::append_nulls(&mut self, usize) -pub unsafe fn vortex_array::builders::ListViewBuilder::append_nulls_unchecked(&mut self, n: usize) +pub unsafe fn vortex_array::builders::ListViewBuilder::append_nulls_unchecked(&mut self, usize) -pub fn vortex_array::builders::ListViewBuilder::append_scalar(&mut self, scalar: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::ListViewBuilder::append_scalar(&mut self, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::ListViewBuilder::append_zero(&mut self) -pub fn vortex_array::builders::ListViewBuilder::append_zeros(&mut self, n: usize) +pub fn vortex_array::builders::ListViewBuilder::append_zeros(&mut self, usize) pub fn vortex_array::builders::ListViewBuilder::as_any(&self) -> &dyn core::any::Any @@ -7684,9 +7828,9 @@ pub fn vortex_array::builders::ListViewBuilder::as_any_mut(&mut self) -> & pub fn vortex_array::builders::ListViewBuilder::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::builders::ListViewBuilder::extend_from_array(&mut self, array: &vortex_array::ArrayRef) +pub fn vortex_array::builders::ListViewBuilder::extend_from_array(&mut self, &vortex_array::ArrayRef) -pub unsafe fn vortex_array::builders::ListViewBuilder::extend_from_array_unchecked(&mut self, array: &vortex_array::ArrayRef) +pub unsafe fn vortex_array::builders::ListViewBuilder::extend_from_array_unchecked(&mut self, &vortex_array::ArrayRef) pub fn vortex_array::builders::ListViewBuilder::finish(&mut self) -> vortex_array::ArrayRef @@ -7696,11 +7840,11 @@ pub fn vortex_array::builders::ListViewBuilder::is_empty(&self) -> bool pub fn vortex_array::builders::ListViewBuilder::len(&self) -> usize -pub fn vortex_array::builders::ListViewBuilder::reserve_exact(&mut self, capacity: usize) +pub fn vortex_array::builders::ListViewBuilder::reserve_exact(&mut self, usize) -pub fn vortex_array::builders::ListViewBuilder::set_validity(&mut self, validity: vortex_mask::Mask) +pub fn vortex_array::builders::ListViewBuilder::set_validity(&mut self, vortex_mask::Mask) -pub unsafe fn vortex_array::builders::ListViewBuilder::set_validity_unchecked(&mut self, validity: vortex_mask::Mask) +pub unsafe fn vortex_array::builders::ListViewBuilder::set_validity_unchecked(&mut self, vortex_mask::Mask) pub struct vortex_array::builders::NullBuilder @@ -7716,19 +7860,19 @@ impl vortex_array::builders::ArrayBuilder for vortex_array::builders::NullBuilde pub fn vortex_array::builders::NullBuilder::append_default(&mut self) -pub fn vortex_array::builders::NullBuilder::append_defaults(&mut self, n: usize) +pub fn vortex_array::builders::NullBuilder::append_defaults(&mut self, usize) pub fn vortex_array::builders::NullBuilder::append_null(&mut self) -pub fn vortex_array::builders::NullBuilder::append_nulls(&mut self, n: usize) +pub fn vortex_array::builders::NullBuilder::append_nulls(&mut self, usize) -pub unsafe fn vortex_array::builders::NullBuilder::append_nulls_unchecked(&mut self, n: usize) +pub unsafe fn vortex_array::builders::NullBuilder::append_nulls_unchecked(&mut self, usize) -pub fn vortex_array::builders::NullBuilder::append_scalar(&mut self, scalar: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::NullBuilder::append_scalar(&mut self, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::NullBuilder::append_zero(&mut self) -pub fn vortex_array::builders::NullBuilder::append_zeros(&mut self, n: usize) +pub fn vortex_array::builders::NullBuilder::append_zeros(&mut self, usize) pub fn vortex_array::builders::NullBuilder::as_any(&self) -> &dyn core::any::Any @@ -7736,9 +7880,9 @@ pub fn vortex_array::builders::NullBuilder::as_any_mut(&mut self) -> &mut dyn co pub fn vortex_array::builders::NullBuilder::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::builders::NullBuilder::extend_from_array(&mut self, array: &vortex_array::ArrayRef) +pub fn vortex_array::builders::NullBuilder::extend_from_array(&mut self, &vortex_array::ArrayRef) -pub unsafe fn vortex_array::builders::NullBuilder::extend_from_array_unchecked(&mut self, array: &vortex_array::ArrayRef) +pub unsafe fn vortex_array::builders::NullBuilder::extend_from_array_unchecked(&mut self, &vortex_array::ArrayRef) pub fn vortex_array::builders::NullBuilder::finish(&mut self) -> vortex_array::ArrayRef @@ -7748,51 +7892,51 @@ pub fn vortex_array::builders::NullBuilder::is_empty(&self) -> bool pub fn vortex_array::builders::NullBuilder::len(&self) -> usize -pub fn vortex_array::builders::NullBuilder::reserve_exact(&mut self, _additional: usize) +pub fn vortex_array::builders::NullBuilder::reserve_exact(&mut self, usize) -pub fn vortex_array::builders::NullBuilder::set_validity(&mut self, validity: vortex_mask::Mask) +pub fn vortex_array::builders::NullBuilder::set_validity(&mut self, vortex_mask::Mask) -pub unsafe fn vortex_array::builders::NullBuilder::set_validity_unchecked(&mut self, _validity: vortex_mask::Mask) +pub unsafe fn vortex_array::builders::NullBuilder::set_validity_unchecked(&mut self, vortex_mask::Mask) pub struct vortex_array::builders::PrimitiveBuilder impl vortex_array::builders::PrimitiveBuilder -pub fn vortex_array::builders::PrimitiveBuilder::append_n_values(&mut self, value: T, n: usize) +pub fn vortex_array::builders::PrimitiveBuilder::append_n_values(&mut self, T, usize) -pub fn vortex_array::builders::PrimitiveBuilder::append_value(&mut self, value: T) +pub fn vortex_array::builders::PrimitiveBuilder::append_value(&mut self, T) -pub fn vortex_array::builders::PrimitiveBuilder::extend_with_iterator(&mut self, iter: impl core::iter::traits::collect::IntoIterator, mask: vortex_mask::Mask) +pub fn vortex_array::builders::PrimitiveBuilder::extend_with_iterator(&mut self, impl core::iter::traits::collect::IntoIterator, vortex_mask::Mask) pub fn vortex_array::builders::PrimitiveBuilder::finish_into_primitive(&mut self) -> vortex_array::arrays::PrimitiveArray -pub fn vortex_array::builders::PrimitiveBuilder::new(nullability: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::builders::PrimitiveBuilder::new(vortex_array::dtype::Nullability) -> Self -pub fn vortex_array::builders::PrimitiveBuilder::uninit_range(&mut self, len: usize) -> vortex_array::builders::UninitRange<'_, T> +pub fn vortex_array::builders::PrimitiveBuilder::uninit_range(&mut self, usize) -> vortex_array::builders::UninitRange<'_, T> pub fn vortex_array::builders::PrimitiveBuilder::values(&self) -> &[T] pub fn vortex_array::builders::PrimitiveBuilder::values_mut(&mut self) -> &mut [T] -pub fn vortex_array::builders::PrimitiveBuilder::with_capacity(nullability: vortex_array::dtype::Nullability, capacity: usize) -> Self +pub fn vortex_array::builders::PrimitiveBuilder::with_capacity(vortex_array::dtype::Nullability, usize) -> Self impl vortex_array::builders::ArrayBuilder for vortex_array::builders::PrimitiveBuilder pub fn vortex_array::builders::PrimitiveBuilder::append_default(&mut self) -pub fn vortex_array::builders::PrimitiveBuilder::append_defaults(&mut self, n: usize) +pub fn vortex_array::builders::PrimitiveBuilder::append_defaults(&mut self, usize) pub fn vortex_array::builders::PrimitiveBuilder::append_null(&mut self) -pub fn vortex_array::builders::PrimitiveBuilder::append_nulls(&mut self, n: usize) +pub fn vortex_array::builders::PrimitiveBuilder::append_nulls(&mut self, usize) -pub unsafe fn vortex_array::builders::PrimitiveBuilder::append_nulls_unchecked(&mut self, n: usize) +pub unsafe fn vortex_array::builders::PrimitiveBuilder::append_nulls_unchecked(&mut self, usize) -pub fn vortex_array::builders::PrimitiveBuilder::append_scalar(&mut self, scalar: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::PrimitiveBuilder::append_scalar(&mut self, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::PrimitiveBuilder::append_zero(&mut self) -pub fn vortex_array::builders::PrimitiveBuilder::append_zeros(&mut self, n: usize) +pub fn vortex_array::builders::PrimitiveBuilder::append_zeros(&mut self, usize) pub fn vortex_array::builders::PrimitiveBuilder::as_any(&self) -> &dyn core::any::Any @@ -7800,9 +7944,9 @@ pub fn vortex_array::builders::PrimitiveBuilder::as_any_mut(&mut self) -> &mu pub fn vortex_array::builders::PrimitiveBuilder::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::builders::PrimitiveBuilder::extend_from_array(&mut self, array: &vortex_array::ArrayRef) +pub fn vortex_array::builders::PrimitiveBuilder::extend_from_array(&mut self, &vortex_array::ArrayRef) -pub unsafe fn vortex_array::builders::PrimitiveBuilder::extend_from_array_unchecked(&mut self, array: &vortex_array::ArrayRef) +pub unsafe fn vortex_array::builders::PrimitiveBuilder::extend_from_array_unchecked(&mut self, &vortex_array::ArrayRef) pub fn vortex_array::builders::PrimitiveBuilder::finish(&mut self) -> vortex_array::ArrayRef @@ -7812,43 +7956,43 @@ pub fn vortex_array::builders::PrimitiveBuilder::is_empty(&self) -> bool pub fn vortex_array::builders::PrimitiveBuilder::len(&self) -> usize -pub fn vortex_array::builders::PrimitiveBuilder::reserve_exact(&mut self, additional: usize) +pub fn vortex_array::builders::PrimitiveBuilder::reserve_exact(&mut self, usize) -pub fn vortex_array::builders::PrimitiveBuilder::set_validity(&mut self, validity: vortex_mask::Mask) +pub fn vortex_array::builders::PrimitiveBuilder::set_validity(&mut self, vortex_mask::Mask) -pub unsafe fn vortex_array::builders::PrimitiveBuilder::set_validity_unchecked(&mut self, validity: vortex_mask::Mask) +pub unsafe fn vortex_array::builders::PrimitiveBuilder::set_validity_unchecked(&mut self, vortex_mask::Mask) pub struct vortex_array::builders::StructBuilder impl vortex_array::builders::StructBuilder -pub fn vortex_array::builders::StructBuilder::append_value(&mut self, struct_scalar: vortex_array::scalar::StructScalar<'_>) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::StructBuilder::append_value(&mut self, vortex_array::scalar::StructScalar<'_>) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::StructBuilder::finish_into_struct(&mut self) -> vortex_array::arrays::StructArray -pub fn vortex_array::builders::StructBuilder::new(struct_dtype: vortex_array::dtype::StructFields, nullability: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::builders::StructBuilder::new(vortex_array::dtype::StructFields, vortex_array::dtype::Nullability) -> Self pub fn vortex_array::builders::StructBuilder::struct_fields(&self) -> &vortex_array::dtype::StructFields -pub fn vortex_array::builders::StructBuilder::with_capacity(struct_dtype: vortex_array::dtype::StructFields, nullability: vortex_array::dtype::Nullability, capacity: usize) -> Self +pub fn vortex_array::builders::StructBuilder::with_capacity(vortex_array::dtype::StructFields, vortex_array::dtype::Nullability, usize) -> Self impl vortex_array::builders::ArrayBuilder for vortex_array::builders::StructBuilder pub fn vortex_array::builders::StructBuilder::append_default(&mut self) -pub fn vortex_array::builders::StructBuilder::append_defaults(&mut self, n: usize) +pub fn vortex_array::builders::StructBuilder::append_defaults(&mut self, usize) pub fn vortex_array::builders::StructBuilder::append_null(&mut self) -pub fn vortex_array::builders::StructBuilder::append_nulls(&mut self, n: usize) +pub fn vortex_array::builders::StructBuilder::append_nulls(&mut self, usize) -pub unsafe fn vortex_array::builders::StructBuilder::append_nulls_unchecked(&mut self, n: usize) +pub unsafe fn vortex_array::builders::StructBuilder::append_nulls_unchecked(&mut self, usize) -pub fn vortex_array::builders::StructBuilder::append_scalar(&mut self, scalar: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::StructBuilder::append_scalar(&mut self, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::StructBuilder::append_zero(&mut self) -pub fn vortex_array::builders::StructBuilder::append_zeros(&mut self, n: usize) +pub fn vortex_array::builders::StructBuilder::append_zeros(&mut self, usize) pub fn vortex_array::builders::StructBuilder::as_any(&self) -> &dyn core::any::Any @@ -7856,9 +8000,9 @@ pub fn vortex_array::builders::StructBuilder::as_any_mut(&mut self) -> &mut dyn pub fn vortex_array::builders::StructBuilder::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::builders::StructBuilder::extend_from_array(&mut self, array: &vortex_array::ArrayRef) +pub fn vortex_array::builders::StructBuilder::extend_from_array(&mut self, &vortex_array::ArrayRef) -pub unsafe fn vortex_array::builders::StructBuilder::extend_from_array_unchecked(&mut self, array: &vortex_array::ArrayRef) +pub unsafe fn vortex_array::builders::StructBuilder::extend_from_array_unchecked(&mut self, &vortex_array::ArrayRef) pub fn vortex_array::builders::StructBuilder::finish(&mut self) -> vortex_array::ArrayRef @@ -7868,19 +8012,19 @@ pub fn vortex_array::builders::StructBuilder::is_empty(&self) -> bool pub fn vortex_array::builders::StructBuilder::len(&self) -> usize -pub fn vortex_array::builders::StructBuilder::reserve_exact(&mut self, capacity: usize) +pub fn vortex_array::builders::StructBuilder::reserve_exact(&mut self, usize) -pub fn vortex_array::builders::StructBuilder::set_validity(&mut self, validity: vortex_mask::Mask) +pub fn vortex_array::builders::StructBuilder::set_validity(&mut self, vortex_mask::Mask) -pub unsafe fn vortex_array::builders::StructBuilder::set_validity_unchecked(&mut self, validity: vortex_mask::Mask) +pub unsafe fn vortex_array::builders::StructBuilder::set_validity_unchecked(&mut self, vortex_mask::Mask) pub struct vortex_array::builders::UninitRange<'a, T> impl vortex_array::builders::UninitRange<'_, T> -pub unsafe fn vortex_array::builders::UninitRange<'_, T>::append_mask(&mut self, mask: vortex_mask::Mask) +pub unsafe fn vortex_array::builders::UninitRange<'_, T>::append_mask(&mut self, vortex_mask::Mask) -pub fn vortex_array::builders::UninitRange<'_, T>::copy_from_slice(&mut self, local_offset: usize, src: &[T]) where T: core::marker::Copy +pub fn vortex_array::builders::UninitRange<'_, T>::copy_from_slice(&mut self, usize, &[T]) where T: core::marker::Copy pub unsafe fn vortex_array::builders::UninitRange<'_, T>::finish(self) @@ -7888,19 +8032,19 @@ pub fn vortex_array::builders::UninitRange<'_, T>::is_empty(&self) -> bool pub fn vortex_array::builders::UninitRange<'_, T>::len(&self) -> usize -pub fn vortex_array::builders::UninitRange<'_, T>::set_validity_bit(&mut self, index: usize, v: bool) +pub fn vortex_array::builders::UninitRange<'_, T>::set_validity_bit(&mut self, usize, bool) -pub fn vortex_array::builders::UninitRange<'_, T>::set_value(&mut self, index: usize, value: T) +pub fn vortex_array::builders::UninitRange<'_, T>::set_value(&mut self, usize, T) -pub unsafe fn vortex_array::builders::UninitRange<'_, T>::slice_uninit_mut(&mut self, offset: usize, len: usize) -> &mut [core::mem::maybe_uninit::MaybeUninit] +pub unsafe fn vortex_array::builders::UninitRange<'_, T>::slice_uninit_mut(&mut self, usize, usize) -> &mut [core::mem::maybe_uninit::MaybeUninit] pub struct vortex_array::builders::VarBinViewBuilder impl vortex_array::builders::VarBinViewBuilder -pub fn vortex_array::builders::VarBinViewBuilder::append_n_values>(&mut self, value: S, n: usize) +pub fn vortex_array::builders::VarBinViewBuilder::append_n_values>(&mut self, S, usize) -pub fn vortex_array::builders::VarBinViewBuilder::append_value>(&mut self, value: S) +pub fn vortex_array::builders::VarBinViewBuilder::append_value>(&mut self, S) pub fn vortex_array::builders::VarBinViewBuilder::completed_block_count(&self) -> u32 @@ -7908,33 +8052,33 @@ pub fn vortex_array::builders::VarBinViewBuilder::finish_into_varbinview(&mut se pub fn vortex_array::builders::VarBinViewBuilder::in_progress(&self) -> bool -pub fn vortex_array::builders::VarBinViewBuilder::new(dtype: vortex_array::dtype::DType, capacity: usize, completed: vortex_array::builders::CompletedBuffers, growth_strategy: vortex_array::builders::BufferGrowthStrategy, compaction_threshold: f64) -> Self +pub fn vortex_array::builders::VarBinViewBuilder::new(vortex_array::dtype::DType, usize, vortex_array::builders::CompletedBuffers, vortex_array::builders::BufferGrowthStrategy, f64) -> Self -pub fn vortex_array::builders::VarBinViewBuilder::push_buffer_and_adjusted_views(&mut self, buffers: &[vortex_buffer::ByteBuffer], views: &vortex_buffer::buffer::Buffer, validity_mask: vortex_mask::Mask) +pub fn vortex_array::builders::VarBinViewBuilder::push_buffer_and_adjusted_views(&mut self, &[vortex_buffer::ByteBuffer], &vortex_buffer::buffer::Buffer, vortex_mask::Mask) -pub fn vortex_array::builders::VarBinViewBuilder::with_buffer_deduplication(dtype: vortex_array::dtype::DType, capacity: usize) -> Self +pub fn vortex_array::builders::VarBinViewBuilder::with_buffer_deduplication(vortex_array::dtype::DType, usize) -> Self -pub fn vortex_array::builders::VarBinViewBuilder::with_capacity(dtype: vortex_array::dtype::DType, capacity: usize) -> Self +pub fn vortex_array::builders::VarBinViewBuilder::with_capacity(vortex_array::dtype::DType, usize) -> Self -pub fn vortex_array::builders::VarBinViewBuilder::with_compaction(dtype: vortex_array::dtype::DType, capacity: usize, compaction_threshold: f64) -> Self +pub fn vortex_array::builders::VarBinViewBuilder::with_compaction(vortex_array::dtype::DType, usize, f64) -> Self impl vortex_array::builders::ArrayBuilder for vortex_array::builders::VarBinViewBuilder pub fn vortex_array::builders::VarBinViewBuilder::append_default(&mut self) -pub fn vortex_array::builders::VarBinViewBuilder::append_defaults(&mut self, n: usize) +pub fn vortex_array::builders::VarBinViewBuilder::append_defaults(&mut self, usize) pub fn vortex_array::builders::VarBinViewBuilder::append_null(&mut self) -pub fn vortex_array::builders::VarBinViewBuilder::append_nulls(&mut self, n: usize) +pub fn vortex_array::builders::VarBinViewBuilder::append_nulls(&mut self, usize) -pub unsafe fn vortex_array::builders::VarBinViewBuilder::append_nulls_unchecked(&mut self, n: usize) +pub unsafe fn vortex_array::builders::VarBinViewBuilder::append_nulls_unchecked(&mut self, usize) -pub fn vortex_array::builders::VarBinViewBuilder::append_scalar(&mut self, scalar: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::VarBinViewBuilder::append_scalar(&mut self, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::VarBinViewBuilder::append_zero(&mut self) -pub fn vortex_array::builders::VarBinViewBuilder::append_zeros(&mut self, n: usize) +pub fn vortex_array::builders::VarBinViewBuilder::append_zeros(&mut self, usize) pub fn vortex_array::builders::VarBinViewBuilder::as_any(&self) -> &dyn core::any::Any @@ -7942,9 +8086,9 @@ pub fn vortex_array::builders::VarBinViewBuilder::as_any_mut(&mut self) -> &mut pub fn vortex_array::builders::VarBinViewBuilder::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::builders::VarBinViewBuilder::extend_from_array(&mut self, array: &vortex_array::ArrayRef) +pub fn vortex_array::builders::VarBinViewBuilder::extend_from_array(&mut self, &vortex_array::ArrayRef) -pub unsafe fn vortex_array::builders::VarBinViewBuilder::extend_from_array_unchecked(&mut self, array: &vortex_array::ArrayRef) +pub unsafe fn vortex_array::builders::VarBinViewBuilder::extend_from_array_unchecked(&mut self, &vortex_array::ArrayRef) pub fn vortex_array::builders::VarBinViewBuilder::finish(&mut self) -> vortex_array::ArrayRef @@ -7954,11 +8098,11 @@ pub fn vortex_array::builders::VarBinViewBuilder::is_empty(&self) -> bool pub fn vortex_array::builders::VarBinViewBuilder::len(&self) -> usize -pub fn vortex_array::builders::VarBinViewBuilder::reserve_exact(&mut self, additional: usize) +pub fn vortex_array::builders::VarBinViewBuilder::reserve_exact(&mut self, usize) -pub fn vortex_array::builders::VarBinViewBuilder::set_validity(&mut self, validity: vortex_mask::Mask) +pub fn vortex_array::builders::VarBinViewBuilder::set_validity(&mut self, vortex_mask::Mask) -pub unsafe fn vortex_array::builders::VarBinViewBuilder::set_validity_unchecked(&mut self, validity: vortex_mask::Mask) +pub unsafe fn vortex_array::builders::VarBinViewBuilder::set_validity_unchecked(&mut self, vortex_mask::Mask) pub const vortex_array::builders::DEFAULT_BUILDER_CAPACITY: usize @@ -7966,19 +8110,19 @@ pub trait vortex_array::builders::ArrayBuilder: core::marker::Send pub fn vortex_array::builders::ArrayBuilder::append_default(&mut self) -pub fn vortex_array::builders::ArrayBuilder::append_defaults(&mut self, n: usize) +pub fn vortex_array::builders::ArrayBuilder::append_defaults(&mut self, usize) pub fn vortex_array::builders::ArrayBuilder::append_null(&mut self) -pub fn vortex_array::builders::ArrayBuilder::append_nulls(&mut self, n: usize) +pub fn vortex_array::builders::ArrayBuilder::append_nulls(&mut self, usize) -pub unsafe fn vortex_array::builders::ArrayBuilder::append_nulls_unchecked(&mut self, n: usize) +pub unsafe fn vortex_array::builders::ArrayBuilder::append_nulls_unchecked(&mut self, usize) -pub fn vortex_array::builders::ArrayBuilder::append_scalar(&mut self, scalar: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::ArrayBuilder::append_scalar(&mut self, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::ArrayBuilder::append_zero(&mut self) -pub fn vortex_array::builders::ArrayBuilder::append_zeros(&mut self, n: usize) +pub fn vortex_array::builders::ArrayBuilder::append_zeros(&mut self, usize) pub fn vortex_array::builders::ArrayBuilder::as_any(&self) -> &dyn core::any::Any @@ -7986,9 +8130,9 @@ pub fn vortex_array::builders::ArrayBuilder::as_any_mut(&mut self) -> &mut dyn c pub fn vortex_array::builders::ArrayBuilder::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::builders::ArrayBuilder::extend_from_array(&mut self, array: &vortex_array::ArrayRef) +pub fn vortex_array::builders::ArrayBuilder::extend_from_array(&mut self, &vortex_array::ArrayRef) -pub unsafe fn vortex_array::builders::ArrayBuilder::extend_from_array_unchecked(&mut self, array: &vortex_array::ArrayRef) +pub unsafe fn vortex_array::builders::ArrayBuilder::extend_from_array_unchecked(&mut self, &vortex_array::ArrayRef) pub fn vortex_array::builders::ArrayBuilder::finish(&mut self) -> vortex_array::ArrayRef @@ -7998,29 +8142,29 @@ pub fn vortex_array::builders::ArrayBuilder::is_empty(&self) -> bool pub fn vortex_array::builders::ArrayBuilder::len(&self) -> usize -pub fn vortex_array::builders::ArrayBuilder::reserve_exact(&mut self, additional: usize) +pub fn vortex_array::builders::ArrayBuilder::reserve_exact(&mut self, usize) -pub fn vortex_array::builders::ArrayBuilder::set_validity(&mut self, validity: vortex_mask::Mask) +pub fn vortex_array::builders::ArrayBuilder::set_validity(&mut self, vortex_mask::Mask) -pub unsafe fn vortex_array::builders::ArrayBuilder::set_validity_unchecked(&mut self, validity: vortex_mask::Mask) +pub unsafe fn vortex_array::builders::ArrayBuilder::set_validity_unchecked(&mut self, vortex_mask::Mask) impl vortex_array::builders::ArrayBuilder for vortex_array::builders::BoolBuilder pub fn vortex_array::builders::BoolBuilder::append_default(&mut self) -pub fn vortex_array::builders::BoolBuilder::append_defaults(&mut self, n: usize) +pub fn vortex_array::builders::BoolBuilder::append_defaults(&mut self, usize) pub fn vortex_array::builders::BoolBuilder::append_null(&mut self) -pub fn vortex_array::builders::BoolBuilder::append_nulls(&mut self, n: usize) +pub fn vortex_array::builders::BoolBuilder::append_nulls(&mut self, usize) -pub unsafe fn vortex_array::builders::BoolBuilder::append_nulls_unchecked(&mut self, n: usize) +pub unsafe fn vortex_array::builders::BoolBuilder::append_nulls_unchecked(&mut self, usize) -pub fn vortex_array::builders::BoolBuilder::append_scalar(&mut self, scalar: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::BoolBuilder::append_scalar(&mut self, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::BoolBuilder::append_zero(&mut self) -pub fn vortex_array::builders::BoolBuilder::append_zeros(&mut self, n: usize) +pub fn vortex_array::builders::BoolBuilder::append_zeros(&mut self, usize) pub fn vortex_array::builders::BoolBuilder::as_any(&self) -> &dyn core::any::Any @@ -8028,9 +8172,9 @@ pub fn vortex_array::builders::BoolBuilder::as_any_mut(&mut self) -> &mut dyn co pub fn vortex_array::builders::BoolBuilder::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::builders::BoolBuilder::extend_from_array(&mut self, array: &vortex_array::ArrayRef) +pub fn vortex_array::builders::BoolBuilder::extend_from_array(&mut self, &vortex_array::ArrayRef) -pub unsafe fn vortex_array::builders::BoolBuilder::extend_from_array_unchecked(&mut self, array: &vortex_array::ArrayRef) +pub unsafe fn vortex_array::builders::BoolBuilder::extend_from_array_unchecked(&mut self, &vortex_array::ArrayRef) pub fn vortex_array::builders::BoolBuilder::finish(&mut self) -> vortex_array::ArrayRef @@ -8040,29 +8184,29 @@ pub fn vortex_array::builders::BoolBuilder::is_empty(&self) -> bool pub fn vortex_array::builders::BoolBuilder::len(&self) -> usize -pub fn vortex_array::builders::BoolBuilder::reserve_exact(&mut self, additional: usize) +pub fn vortex_array::builders::BoolBuilder::reserve_exact(&mut self, usize) -pub fn vortex_array::builders::BoolBuilder::set_validity(&mut self, validity: vortex_mask::Mask) +pub fn vortex_array::builders::BoolBuilder::set_validity(&mut self, vortex_mask::Mask) -pub unsafe fn vortex_array::builders::BoolBuilder::set_validity_unchecked(&mut self, validity: vortex_mask::Mask) +pub unsafe fn vortex_array::builders::BoolBuilder::set_validity_unchecked(&mut self, vortex_mask::Mask) impl vortex_array::builders::ArrayBuilder for vortex_array::builders::DecimalBuilder pub fn vortex_array::builders::DecimalBuilder::append_default(&mut self) -pub fn vortex_array::builders::DecimalBuilder::append_defaults(&mut self, n: usize) +pub fn vortex_array::builders::DecimalBuilder::append_defaults(&mut self, usize) pub fn vortex_array::builders::DecimalBuilder::append_null(&mut self) -pub fn vortex_array::builders::DecimalBuilder::append_nulls(&mut self, n: usize) +pub fn vortex_array::builders::DecimalBuilder::append_nulls(&mut self, usize) -pub unsafe fn vortex_array::builders::DecimalBuilder::append_nulls_unchecked(&mut self, n: usize) +pub unsafe fn vortex_array::builders::DecimalBuilder::append_nulls_unchecked(&mut self, usize) -pub fn vortex_array::builders::DecimalBuilder::append_scalar(&mut self, scalar: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::DecimalBuilder::append_scalar(&mut self, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::DecimalBuilder::append_zero(&mut self) -pub fn vortex_array::builders::DecimalBuilder::append_zeros(&mut self, n: usize) +pub fn vortex_array::builders::DecimalBuilder::append_zeros(&mut self, usize) pub fn vortex_array::builders::DecimalBuilder::as_any(&self) -> &dyn core::any::Any @@ -8070,9 +8214,9 @@ pub fn vortex_array::builders::DecimalBuilder::as_any_mut(&mut self) -> &mut dyn pub fn vortex_array::builders::DecimalBuilder::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::builders::DecimalBuilder::extend_from_array(&mut self, array: &vortex_array::ArrayRef) +pub fn vortex_array::builders::DecimalBuilder::extend_from_array(&mut self, &vortex_array::ArrayRef) -pub unsafe fn vortex_array::builders::DecimalBuilder::extend_from_array_unchecked(&mut self, array: &vortex_array::ArrayRef) +pub unsafe fn vortex_array::builders::DecimalBuilder::extend_from_array_unchecked(&mut self, &vortex_array::ArrayRef) pub fn vortex_array::builders::DecimalBuilder::finish(&mut self) -> vortex_array::ArrayRef @@ -8082,29 +8226,29 @@ pub fn vortex_array::builders::DecimalBuilder::is_empty(&self) -> bool pub fn vortex_array::builders::DecimalBuilder::len(&self) -> usize -pub fn vortex_array::builders::DecimalBuilder::reserve_exact(&mut self, additional: usize) +pub fn vortex_array::builders::DecimalBuilder::reserve_exact(&mut self, usize) -pub fn vortex_array::builders::DecimalBuilder::set_validity(&mut self, validity: vortex_mask::Mask) +pub fn vortex_array::builders::DecimalBuilder::set_validity(&mut self, vortex_mask::Mask) -pub unsafe fn vortex_array::builders::DecimalBuilder::set_validity_unchecked(&mut self, validity: vortex_mask::Mask) +pub unsafe fn vortex_array::builders::DecimalBuilder::set_validity_unchecked(&mut self, vortex_mask::Mask) impl vortex_array::builders::ArrayBuilder for vortex_array::builders::ExtensionBuilder pub fn vortex_array::builders::ExtensionBuilder::append_default(&mut self) -pub fn vortex_array::builders::ExtensionBuilder::append_defaults(&mut self, n: usize) +pub fn vortex_array::builders::ExtensionBuilder::append_defaults(&mut self, usize) pub fn vortex_array::builders::ExtensionBuilder::append_null(&mut self) -pub fn vortex_array::builders::ExtensionBuilder::append_nulls(&mut self, n: usize) +pub fn vortex_array::builders::ExtensionBuilder::append_nulls(&mut self, usize) -pub unsafe fn vortex_array::builders::ExtensionBuilder::append_nulls_unchecked(&mut self, n: usize) +pub unsafe fn vortex_array::builders::ExtensionBuilder::append_nulls_unchecked(&mut self, usize) -pub fn vortex_array::builders::ExtensionBuilder::append_scalar(&mut self, scalar: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::ExtensionBuilder::append_scalar(&mut self, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::ExtensionBuilder::append_zero(&mut self) -pub fn vortex_array::builders::ExtensionBuilder::append_zeros(&mut self, n: usize) +pub fn vortex_array::builders::ExtensionBuilder::append_zeros(&mut self, usize) pub fn vortex_array::builders::ExtensionBuilder::as_any(&self) -> &dyn core::any::Any @@ -8112,9 +8256,9 @@ pub fn vortex_array::builders::ExtensionBuilder::as_any_mut(&mut self) -> &mut d pub fn vortex_array::builders::ExtensionBuilder::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::builders::ExtensionBuilder::extend_from_array(&mut self, array: &vortex_array::ArrayRef) +pub fn vortex_array::builders::ExtensionBuilder::extend_from_array(&mut self, &vortex_array::ArrayRef) -pub unsafe fn vortex_array::builders::ExtensionBuilder::extend_from_array_unchecked(&mut self, array: &vortex_array::ArrayRef) +pub unsafe fn vortex_array::builders::ExtensionBuilder::extend_from_array_unchecked(&mut self, &vortex_array::ArrayRef) pub fn vortex_array::builders::ExtensionBuilder::finish(&mut self) -> vortex_array::ArrayRef @@ -8124,29 +8268,29 @@ pub fn vortex_array::builders::ExtensionBuilder::is_empty(&self) -> bool pub fn vortex_array::builders::ExtensionBuilder::len(&self) -> usize -pub fn vortex_array::builders::ExtensionBuilder::reserve_exact(&mut self, capacity: usize) +pub fn vortex_array::builders::ExtensionBuilder::reserve_exact(&mut self, usize) -pub fn vortex_array::builders::ExtensionBuilder::set_validity(&mut self, validity: vortex_mask::Mask) +pub fn vortex_array::builders::ExtensionBuilder::set_validity(&mut self, vortex_mask::Mask) -pub unsafe fn vortex_array::builders::ExtensionBuilder::set_validity_unchecked(&mut self, validity: vortex_mask::Mask) +pub unsafe fn vortex_array::builders::ExtensionBuilder::set_validity_unchecked(&mut self, vortex_mask::Mask) impl vortex_array::builders::ArrayBuilder for vortex_array::builders::FixedSizeListBuilder pub fn vortex_array::builders::FixedSizeListBuilder::append_default(&mut self) -pub fn vortex_array::builders::FixedSizeListBuilder::append_defaults(&mut self, n: usize) +pub fn vortex_array::builders::FixedSizeListBuilder::append_defaults(&mut self, usize) pub fn vortex_array::builders::FixedSizeListBuilder::append_null(&mut self) -pub fn vortex_array::builders::FixedSizeListBuilder::append_nulls(&mut self, n: usize) +pub fn vortex_array::builders::FixedSizeListBuilder::append_nulls(&mut self, usize) -pub unsafe fn vortex_array::builders::FixedSizeListBuilder::append_nulls_unchecked(&mut self, n: usize) +pub unsafe fn vortex_array::builders::FixedSizeListBuilder::append_nulls_unchecked(&mut self, usize) -pub fn vortex_array::builders::FixedSizeListBuilder::append_scalar(&mut self, scalar: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::FixedSizeListBuilder::append_scalar(&mut self, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::FixedSizeListBuilder::append_zero(&mut self) -pub fn vortex_array::builders::FixedSizeListBuilder::append_zeros(&mut self, n: usize) +pub fn vortex_array::builders::FixedSizeListBuilder::append_zeros(&mut self, usize) pub fn vortex_array::builders::FixedSizeListBuilder::as_any(&self) -> &dyn core::any::Any @@ -8154,9 +8298,9 @@ pub fn vortex_array::builders::FixedSizeListBuilder::as_any_mut(&mut self) -> &m pub fn vortex_array::builders::FixedSizeListBuilder::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::builders::FixedSizeListBuilder::extend_from_array(&mut self, array: &vortex_array::ArrayRef) +pub fn vortex_array::builders::FixedSizeListBuilder::extend_from_array(&mut self, &vortex_array::ArrayRef) -pub unsafe fn vortex_array::builders::FixedSizeListBuilder::extend_from_array_unchecked(&mut self, array: &vortex_array::ArrayRef) +pub unsafe fn vortex_array::builders::FixedSizeListBuilder::extend_from_array_unchecked(&mut self, &vortex_array::ArrayRef) pub fn vortex_array::builders::FixedSizeListBuilder::finish(&mut self) -> vortex_array::ArrayRef @@ -8166,29 +8310,29 @@ pub fn vortex_array::builders::FixedSizeListBuilder::is_empty(&self) -> bool pub fn vortex_array::builders::FixedSizeListBuilder::len(&self) -> usize -pub fn vortex_array::builders::FixedSizeListBuilder::reserve_exact(&mut self, additional: usize) +pub fn vortex_array::builders::FixedSizeListBuilder::reserve_exact(&mut self, usize) -pub fn vortex_array::builders::FixedSizeListBuilder::set_validity(&mut self, validity: vortex_mask::Mask) +pub fn vortex_array::builders::FixedSizeListBuilder::set_validity(&mut self, vortex_mask::Mask) -pub unsafe fn vortex_array::builders::FixedSizeListBuilder::set_validity_unchecked(&mut self, validity: vortex_mask::Mask) +pub unsafe fn vortex_array::builders::FixedSizeListBuilder::set_validity_unchecked(&mut self, vortex_mask::Mask) impl vortex_array::builders::ArrayBuilder for vortex_array::builders::NullBuilder pub fn vortex_array::builders::NullBuilder::append_default(&mut self) -pub fn vortex_array::builders::NullBuilder::append_defaults(&mut self, n: usize) +pub fn vortex_array::builders::NullBuilder::append_defaults(&mut self, usize) pub fn vortex_array::builders::NullBuilder::append_null(&mut self) -pub fn vortex_array::builders::NullBuilder::append_nulls(&mut self, n: usize) +pub fn vortex_array::builders::NullBuilder::append_nulls(&mut self, usize) -pub unsafe fn vortex_array::builders::NullBuilder::append_nulls_unchecked(&mut self, n: usize) +pub unsafe fn vortex_array::builders::NullBuilder::append_nulls_unchecked(&mut self, usize) -pub fn vortex_array::builders::NullBuilder::append_scalar(&mut self, scalar: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::NullBuilder::append_scalar(&mut self, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::NullBuilder::append_zero(&mut self) -pub fn vortex_array::builders::NullBuilder::append_zeros(&mut self, n: usize) +pub fn vortex_array::builders::NullBuilder::append_zeros(&mut self, usize) pub fn vortex_array::builders::NullBuilder::as_any(&self) -> &dyn core::any::Any @@ -8196,9 +8340,9 @@ pub fn vortex_array::builders::NullBuilder::as_any_mut(&mut self) -> &mut dyn co pub fn vortex_array::builders::NullBuilder::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::builders::NullBuilder::extend_from_array(&mut self, array: &vortex_array::ArrayRef) +pub fn vortex_array::builders::NullBuilder::extend_from_array(&mut self, &vortex_array::ArrayRef) -pub unsafe fn vortex_array::builders::NullBuilder::extend_from_array_unchecked(&mut self, array: &vortex_array::ArrayRef) +pub unsafe fn vortex_array::builders::NullBuilder::extend_from_array_unchecked(&mut self, &vortex_array::ArrayRef) pub fn vortex_array::builders::NullBuilder::finish(&mut self) -> vortex_array::ArrayRef @@ -8208,29 +8352,29 @@ pub fn vortex_array::builders::NullBuilder::is_empty(&self) -> bool pub fn vortex_array::builders::NullBuilder::len(&self) -> usize -pub fn vortex_array::builders::NullBuilder::reserve_exact(&mut self, _additional: usize) +pub fn vortex_array::builders::NullBuilder::reserve_exact(&mut self, usize) -pub fn vortex_array::builders::NullBuilder::set_validity(&mut self, validity: vortex_mask::Mask) +pub fn vortex_array::builders::NullBuilder::set_validity(&mut self, vortex_mask::Mask) -pub unsafe fn vortex_array::builders::NullBuilder::set_validity_unchecked(&mut self, _validity: vortex_mask::Mask) +pub unsafe fn vortex_array::builders::NullBuilder::set_validity_unchecked(&mut self, vortex_mask::Mask) impl vortex_array::builders::ArrayBuilder for vortex_array::builders::StructBuilder pub fn vortex_array::builders::StructBuilder::append_default(&mut self) -pub fn vortex_array::builders::StructBuilder::append_defaults(&mut self, n: usize) +pub fn vortex_array::builders::StructBuilder::append_defaults(&mut self, usize) pub fn vortex_array::builders::StructBuilder::append_null(&mut self) -pub fn vortex_array::builders::StructBuilder::append_nulls(&mut self, n: usize) +pub fn vortex_array::builders::StructBuilder::append_nulls(&mut self, usize) -pub unsafe fn vortex_array::builders::StructBuilder::append_nulls_unchecked(&mut self, n: usize) +pub unsafe fn vortex_array::builders::StructBuilder::append_nulls_unchecked(&mut self, usize) -pub fn vortex_array::builders::StructBuilder::append_scalar(&mut self, scalar: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::StructBuilder::append_scalar(&mut self, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::StructBuilder::append_zero(&mut self) -pub fn vortex_array::builders::StructBuilder::append_zeros(&mut self, n: usize) +pub fn vortex_array::builders::StructBuilder::append_zeros(&mut self, usize) pub fn vortex_array::builders::StructBuilder::as_any(&self) -> &dyn core::any::Any @@ -8238,9 +8382,9 @@ pub fn vortex_array::builders::StructBuilder::as_any_mut(&mut self) -> &mut dyn pub fn vortex_array::builders::StructBuilder::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::builders::StructBuilder::extend_from_array(&mut self, array: &vortex_array::ArrayRef) +pub fn vortex_array::builders::StructBuilder::extend_from_array(&mut self, &vortex_array::ArrayRef) -pub unsafe fn vortex_array::builders::StructBuilder::extend_from_array_unchecked(&mut self, array: &vortex_array::ArrayRef) +pub unsafe fn vortex_array::builders::StructBuilder::extend_from_array_unchecked(&mut self, &vortex_array::ArrayRef) pub fn vortex_array::builders::StructBuilder::finish(&mut self) -> vortex_array::ArrayRef @@ -8250,29 +8394,29 @@ pub fn vortex_array::builders::StructBuilder::is_empty(&self) -> bool pub fn vortex_array::builders::StructBuilder::len(&self) -> usize -pub fn vortex_array::builders::StructBuilder::reserve_exact(&mut self, capacity: usize) +pub fn vortex_array::builders::StructBuilder::reserve_exact(&mut self, usize) -pub fn vortex_array::builders::StructBuilder::set_validity(&mut self, validity: vortex_mask::Mask) +pub fn vortex_array::builders::StructBuilder::set_validity(&mut self, vortex_mask::Mask) -pub unsafe fn vortex_array::builders::StructBuilder::set_validity_unchecked(&mut self, validity: vortex_mask::Mask) +pub unsafe fn vortex_array::builders::StructBuilder::set_validity_unchecked(&mut self, vortex_mask::Mask) impl vortex_array::builders::ArrayBuilder for vortex_array::builders::VarBinViewBuilder pub fn vortex_array::builders::VarBinViewBuilder::append_default(&mut self) -pub fn vortex_array::builders::VarBinViewBuilder::append_defaults(&mut self, n: usize) +pub fn vortex_array::builders::VarBinViewBuilder::append_defaults(&mut self, usize) pub fn vortex_array::builders::VarBinViewBuilder::append_null(&mut self) -pub fn vortex_array::builders::VarBinViewBuilder::append_nulls(&mut self, n: usize) +pub fn vortex_array::builders::VarBinViewBuilder::append_nulls(&mut self, usize) -pub unsafe fn vortex_array::builders::VarBinViewBuilder::append_nulls_unchecked(&mut self, n: usize) +pub unsafe fn vortex_array::builders::VarBinViewBuilder::append_nulls_unchecked(&mut self, usize) -pub fn vortex_array::builders::VarBinViewBuilder::append_scalar(&mut self, scalar: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::VarBinViewBuilder::append_scalar(&mut self, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::VarBinViewBuilder::append_zero(&mut self) -pub fn vortex_array::builders::VarBinViewBuilder::append_zeros(&mut self, n: usize) +pub fn vortex_array::builders::VarBinViewBuilder::append_zeros(&mut self, usize) pub fn vortex_array::builders::VarBinViewBuilder::as_any(&self) -> &dyn core::any::Any @@ -8280,9 +8424,9 @@ pub fn vortex_array::builders::VarBinViewBuilder::as_any_mut(&mut self) -> &mut pub fn vortex_array::builders::VarBinViewBuilder::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::builders::VarBinViewBuilder::extend_from_array(&mut self, array: &vortex_array::ArrayRef) +pub fn vortex_array::builders::VarBinViewBuilder::extend_from_array(&mut self, &vortex_array::ArrayRef) -pub unsafe fn vortex_array::builders::VarBinViewBuilder::extend_from_array_unchecked(&mut self, array: &vortex_array::ArrayRef) +pub unsafe fn vortex_array::builders::VarBinViewBuilder::extend_from_array_unchecked(&mut self, &vortex_array::ArrayRef) pub fn vortex_array::builders::VarBinViewBuilder::finish(&mut self) -> vortex_array::ArrayRef @@ -8292,29 +8436,29 @@ pub fn vortex_array::builders::VarBinViewBuilder::is_empty(&self) -> bool pub fn vortex_array::builders::VarBinViewBuilder::len(&self) -> usize -pub fn vortex_array::builders::VarBinViewBuilder::reserve_exact(&mut self, additional: usize) +pub fn vortex_array::builders::VarBinViewBuilder::reserve_exact(&mut self, usize) -pub fn vortex_array::builders::VarBinViewBuilder::set_validity(&mut self, validity: vortex_mask::Mask) +pub fn vortex_array::builders::VarBinViewBuilder::set_validity(&mut self, vortex_mask::Mask) -pub unsafe fn vortex_array::builders::VarBinViewBuilder::set_validity_unchecked(&mut self, validity: vortex_mask::Mask) +pub unsafe fn vortex_array::builders::VarBinViewBuilder::set_validity_unchecked(&mut self, vortex_mask::Mask) impl vortex_array::builders::ArrayBuilder for vortex_array::builders::ListViewBuilder pub fn vortex_array::builders::ListViewBuilder::append_default(&mut self) -pub fn vortex_array::builders::ListViewBuilder::append_defaults(&mut self, n: usize) +pub fn vortex_array::builders::ListViewBuilder::append_defaults(&mut self, usize) pub fn vortex_array::builders::ListViewBuilder::append_null(&mut self) -pub fn vortex_array::builders::ListViewBuilder::append_nulls(&mut self, n: usize) +pub fn vortex_array::builders::ListViewBuilder::append_nulls(&mut self, usize) -pub unsafe fn vortex_array::builders::ListViewBuilder::append_nulls_unchecked(&mut self, n: usize) +pub unsafe fn vortex_array::builders::ListViewBuilder::append_nulls_unchecked(&mut self, usize) -pub fn vortex_array::builders::ListViewBuilder::append_scalar(&mut self, scalar: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::ListViewBuilder::append_scalar(&mut self, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::ListViewBuilder::append_zero(&mut self) -pub fn vortex_array::builders::ListViewBuilder::append_zeros(&mut self, n: usize) +pub fn vortex_array::builders::ListViewBuilder::append_zeros(&mut self, usize) pub fn vortex_array::builders::ListViewBuilder::as_any(&self) -> &dyn core::any::Any @@ -8322,9 +8466,9 @@ pub fn vortex_array::builders::ListViewBuilder::as_any_mut(&mut self) -> & pub fn vortex_array::builders::ListViewBuilder::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::builders::ListViewBuilder::extend_from_array(&mut self, array: &vortex_array::ArrayRef) +pub fn vortex_array::builders::ListViewBuilder::extend_from_array(&mut self, &vortex_array::ArrayRef) -pub unsafe fn vortex_array::builders::ListViewBuilder::extend_from_array_unchecked(&mut self, array: &vortex_array::ArrayRef) +pub unsafe fn vortex_array::builders::ListViewBuilder::extend_from_array_unchecked(&mut self, &vortex_array::ArrayRef) pub fn vortex_array::builders::ListViewBuilder::finish(&mut self) -> vortex_array::ArrayRef @@ -8334,29 +8478,29 @@ pub fn vortex_array::builders::ListViewBuilder::is_empty(&self) -> bool pub fn vortex_array::builders::ListViewBuilder::len(&self) -> usize -pub fn vortex_array::builders::ListViewBuilder::reserve_exact(&mut self, capacity: usize) +pub fn vortex_array::builders::ListViewBuilder::reserve_exact(&mut self, usize) -pub fn vortex_array::builders::ListViewBuilder::set_validity(&mut self, validity: vortex_mask::Mask) +pub fn vortex_array::builders::ListViewBuilder::set_validity(&mut self, vortex_mask::Mask) -pub unsafe fn vortex_array::builders::ListViewBuilder::set_validity_unchecked(&mut self, validity: vortex_mask::Mask) +pub unsafe fn vortex_array::builders::ListViewBuilder::set_validity_unchecked(&mut self, vortex_mask::Mask) impl vortex_array::builders::ArrayBuilder for vortex_array::builders::ListBuilder pub fn vortex_array::builders::ListBuilder::append_default(&mut self) -pub fn vortex_array::builders::ListBuilder::append_defaults(&mut self, n: usize) +pub fn vortex_array::builders::ListBuilder::append_defaults(&mut self, usize) pub fn vortex_array::builders::ListBuilder::append_null(&mut self) -pub fn vortex_array::builders::ListBuilder::append_nulls(&mut self, n: usize) +pub fn vortex_array::builders::ListBuilder::append_nulls(&mut self, usize) -pub unsafe fn vortex_array::builders::ListBuilder::append_nulls_unchecked(&mut self, n: usize) +pub unsafe fn vortex_array::builders::ListBuilder::append_nulls_unchecked(&mut self, usize) -pub fn vortex_array::builders::ListBuilder::append_scalar(&mut self, scalar: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::ListBuilder::append_scalar(&mut self, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::ListBuilder::append_zero(&mut self) -pub fn vortex_array::builders::ListBuilder::append_zeros(&mut self, n: usize) +pub fn vortex_array::builders::ListBuilder::append_zeros(&mut self, usize) pub fn vortex_array::builders::ListBuilder::as_any(&self) -> &dyn core::any::Any @@ -8364,9 +8508,9 @@ pub fn vortex_array::builders::ListBuilder::as_any_mut(&mut self) -> &mut dyn pub fn vortex_array::builders::ListBuilder::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::builders::ListBuilder::extend_from_array(&mut self, array: &vortex_array::ArrayRef) +pub fn vortex_array::builders::ListBuilder::extend_from_array(&mut self, &vortex_array::ArrayRef) -pub unsafe fn vortex_array::builders::ListBuilder::extend_from_array_unchecked(&mut self, array: &vortex_array::ArrayRef) +pub unsafe fn vortex_array::builders::ListBuilder::extend_from_array_unchecked(&mut self, &vortex_array::ArrayRef) pub fn vortex_array::builders::ListBuilder::finish(&mut self) -> vortex_array::ArrayRef @@ -8376,29 +8520,29 @@ pub fn vortex_array::builders::ListBuilder::is_empty(&self) -> bool pub fn vortex_array::builders::ListBuilder::len(&self) -> usize -pub fn vortex_array::builders::ListBuilder::reserve_exact(&mut self, additional: usize) +pub fn vortex_array::builders::ListBuilder::reserve_exact(&mut self, usize) -pub fn vortex_array::builders::ListBuilder::set_validity(&mut self, validity: vortex_mask::Mask) +pub fn vortex_array::builders::ListBuilder::set_validity(&mut self, vortex_mask::Mask) -pub unsafe fn vortex_array::builders::ListBuilder::set_validity_unchecked(&mut self, validity: vortex_mask::Mask) +pub unsafe fn vortex_array::builders::ListBuilder::set_validity_unchecked(&mut self, vortex_mask::Mask) impl vortex_array::builders::ArrayBuilder for vortex_array::builders::PrimitiveBuilder pub fn vortex_array::builders::PrimitiveBuilder::append_default(&mut self) -pub fn vortex_array::builders::PrimitiveBuilder::append_defaults(&mut self, n: usize) +pub fn vortex_array::builders::PrimitiveBuilder::append_defaults(&mut self, usize) pub fn vortex_array::builders::PrimitiveBuilder::append_null(&mut self) -pub fn vortex_array::builders::PrimitiveBuilder::append_nulls(&mut self, n: usize) +pub fn vortex_array::builders::PrimitiveBuilder::append_nulls(&mut self, usize) -pub unsafe fn vortex_array::builders::PrimitiveBuilder::append_nulls_unchecked(&mut self, n: usize) +pub unsafe fn vortex_array::builders::PrimitiveBuilder::append_nulls_unchecked(&mut self, usize) -pub fn vortex_array::builders::PrimitiveBuilder::append_scalar(&mut self, scalar: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> +pub fn vortex_array::builders::PrimitiveBuilder::append_scalar(&mut self, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult<()> pub fn vortex_array::builders::PrimitiveBuilder::append_zero(&mut self) -pub fn vortex_array::builders::PrimitiveBuilder::append_zeros(&mut self, n: usize) +pub fn vortex_array::builders::PrimitiveBuilder::append_zeros(&mut self, usize) pub fn vortex_array::builders::PrimitiveBuilder::as_any(&self) -> &dyn core::any::Any @@ -8406,9 +8550,9 @@ pub fn vortex_array::builders::PrimitiveBuilder::as_any_mut(&mut self) -> &mu pub fn vortex_array::builders::PrimitiveBuilder::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::builders::PrimitiveBuilder::extend_from_array(&mut self, array: &vortex_array::ArrayRef) +pub fn vortex_array::builders::PrimitiveBuilder::extend_from_array(&mut self, &vortex_array::ArrayRef) -pub unsafe fn vortex_array::builders::PrimitiveBuilder::extend_from_array_unchecked(&mut self, array: &vortex_array::ArrayRef) +pub unsafe fn vortex_array::builders::PrimitiveBuilder::extend_from_array_unchecked(&mut self, &vortex_array::ArrayRef) pub fn vortex_array::builders::PrimitiveBuilder::finish(&mut self) -> vortex_array::ArrayRef @@ -8418,101 +8562,109 @@ pub fn vortex_array::builders::PrimitiveBuilder::is_empty(&self) -> bool pub fn vortex_array::builders::PrimitiveBuilder::len(&self) -> usize -pub fn vortex_array::builders::PrimitiveBuilder::reserve_exact(&mut self, additional: usize) +pub fn vortex_array::builders::PrimitiveBuilder::reserve_exact(&mut self, usize) -pub fn vortex_array::builders::PrimitiveBuilder::set_validity(&mut self, validity: vortex_mask::Mask) +pub fn vortex_array::builders::PrimitiveBuilder::set_validity(&mut self, vortex_mask::Mask) -pub unsafe fn vortex_array::builders::PrimitiveBuilder::set_validity_unchecked(&mut self, validity: vortex_mask::Mask) +pub unsafe fn vortex_array::builders::PrimitiveBuilder::set_validity_unchecked(&mut self, vortex_mask::Mask) -pub fn vortex_array::builders::builder_with_capacity(dtype: &vortex_array::dtype::DType, capacity: usize) -> alloc::boxed::Box +pub fn vortex_array::builders::builder_with_capacity(&vortex_array::dtype::DType, usize) -> alloc::boxed::Box -pub fn vortex_array::builders::builder_with_capacity_in(allocator: vortex_array::memory::HostAllocatorRef, dtype: &vortex_array::dtype::DType, capacity: usize) -> alloc::boxed::Box +pub fn vortex_array::builders::builder_with_capacity_in(vortex_array::memory::HostAllocatorRef, &vortex_array::dtype::DType, usize) -> alloc::boxed::Box pub mod vortex_array::builtins pub trait vortex_array::builtins::ArrayBuiltins: core::marker::Sized -pub fn vortex_array::builtins::ArrayBuiltins::between(self, lower: vortex_array::ArrayRef, upper: vortex_array::ArrayRef, options: vortex_array::scalar_fn::fns::between::BetweenOptions) -> vortex_error::VortexResult +pub fn vortex_array::builtins::ArrayBuiltins::between(self, vortex_array::ArrayRef, vortex_array::ArrayRef, vortex_array::scalar_fn::fns::between::BetweenOptions) -> vortex_error::VortexResult + +pub fn vortex_array::builtins::ArrayBuiltins::binary(&self, vortex_array::ArrayRef, vortex_array::scalar_fn::fns::operators::Operator) -> vortex_error::VortexResult -pub fn vortex_array::builtins::ArrayBuiltins::binary(&self, rhs: vortex_array::ArrayRef, op: vortex_array::scalar_fn::fns::operators::Operator) -> vortex_error::VortexResult +pub fn vortex_array::builtins::ArrayBuiltins::cast(&self, vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::builtins::ArrayBuiltins::cast(&self, dtype: vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::builtins::ArrayBuiltins::fill_null(&self, impl core::convert::Into) -> vortex_error::VortexResult -pub fn vortex_array::builtins::ArrayBuiltins::fill_null(&self, fill_value: impl core::convert::Into) -> vortex_error::VortexResult +pub fn vortex_array::builtins::ArrayBuiltins::get_item(&self, impl core::convert::Into) -> vortex_error::VortexResult -pub fn vortex_array::builtins::ArrayBuiltins::get_item(&self, field_name: impl core::convert::Into) -> vortex_error::VortexResult +pub fn vortex_array::builtins::ArrayBuiltins::is_not_null(&self) -> vortex_error::VortexResult pub fn vortex_array::builtins::ArrayBuiltins::is_null(&self) -> vortex_error::VortexResult -pub fn vortex_array::builtins::ArrayBuiltins::list_contains(&self, value: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::builtins::ArrayBuiltins::list_contains(&self, vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::builtins::ArrayBuiltins::mask(self, mask: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::builtins::ArrayBuiltins::mask(self, vortex_array::ArrayRef) -> vortex_error::VortexResult pub fn vortex_array::builtins::ArrayBuiltins::not(&self) -> vortex_error::VortexResult -pub fn vortex_array::builtins::ArrayBuiltins::zip(&self, if_true: vortex_array::ArrayRef, if_false: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::builtins::ArrayBuiltins::zip(&self, vortex_array::ArrayRef, vortex_array::ArrayRef) -> vortex_error::VortexResult impl vortex_array::builtins::ArrayBuiltins for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::between(self, lower: vortex_array::ArrayRef, upper: vortex_array::ArrayRef, options: vortex_array::scalar_fn::fns::between::BetweenOptions) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::between(self, vortex_array::ArrayRef, vortex_array::ArrayRef, vortex_array::scalar_fn::fns::between::BetweenOptions) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::binary(&self, rhs: vortex_array::ArrayRef, op: vortex_array::scalar_fn::fns::operators::Operator) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::binary(&self, vortex_array::ArrayRef, vortex_array::scalar_fn::fns::operators::Operator) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::cast(&self, dtype: vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::cast(&self, vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::fill_null(&self, fill_value: impl core::convert::Into) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::fill_null(&self, impl core::convert::Into) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::get_item(&self, field_name: impl core::convert::Into) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::get_item(&self, impl core::convert::Into) -> vortex_error::VortexResult + +pub fn vortex_array::ArrayRef::is_not_null(&self) -> vortex_error::VortexResult pub fn vortex_array::ArrayRef::is_null(&self) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::list_contains(&self, value: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::list_contains(&self, vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::mask(self, mask: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::mask(self, vortex_array::ArrayRef) -> vortex_error::VortexResult pub fn vortex_array::ArrayRef::not(&self) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::zip(&self, if_true: vortex_array::ArrayRef, if_false: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::zip(&self, vortex_array::ArrayRef, vortex_array::ArrayRef) -> vortex_error::VortexResult pub trait vortex_array::builtins::ExprBuiltins: core::marker::Sized -pub fn vortex_array::builtins::ExprBuiltins::binary(&self, rhs: vortex_array::expr::Expression, op: vortex_array::scalar_fn::fns::operators::Operator) -> vortex_error::VortexResult +pub fn vortex_array::builtins::ExprBuiltins::binary(&self, vortex_array::expr::Expression, vortex_array::scalar_fn::fns::operators::Operator) -> vortex_error::VortexResult + +pub fn vortex_array::builtins::ExprBuiltins::cast(&self, vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::builtins::ExprBuiltins::cast(&self, dtype: vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::builtins::ExprBuiltins::fill_null(&self, vortex_array::expr::Expression) -> vortex_error::VortexResult -pub fn vortex_array::builtins::ExprBuiltins::fill_null(&self, fill_value: vortex_array::expr::Expression) -> vortex_error::VortexResult +pub fn vortex_array::builtins::ExprBuiltins::get_item(&self, impl core::convert::Into) -> vortex_error::VortexResult -pub fn vortex_array::builtins::ExprBuiltins::get_item(&self, field_name: impl core::convert::Into) -> vortex_error::VortexResult +pub fn vortex_array::builtins::ExprBuiltins::is_not_null(&self) -> vortex_error::VortexResult pub fn vortex_array::builtins::ExprBuiltins::is_null(&self) -> vortex_error::VortexResult -pub fn vortex_array::builtins::ExprBuiltins::list_contains(&self, value: vortex_array::expr::Expression) -> vortex_error::VortexResult +pub fn vortex_array::builtins::ExprBuiltins::list_contains(&self, vortex_array::expr::Expression) -> vortex_error::VortexResult -pub fn vortex_array::builtins::ExprBuiltins::mask(&self, mask: vortex_array::expr::Expression) -> vortex_error::VortexResult +pub fn vortex_array::builtins::ExprBuiltins::mask(&self, vortex_array::expr::Expression) -> vortex_error::VortexResult pub fn vortex_array::builtins::ExprBuiltins::not(&self) -> vortex_error::VortexResult -pub fn vortex_array::builtins::ExprBuiltins::zip(&self, if_true: vortex_array::expr::Expression, if_false: vortex_array::expr::Expression) -> vortex_error::VortexResult +pub fn vortex_array::builtins::ExprBuiltins::zip(&self, vortex_array::expr::Expression, vortex_array::expr::Expression) -> vortex_error::VortexResult impl vortex_array::builtins::ExprBuiltins for vortex_array::expr::Expression -pub fn vortex_array::expr::Expression::binary(&self, rhs: vortex_array::expr::Expression, op: vortex_array::scalar_fn::fns::operators::Operator) -> vortex_error::VortexResult +pub fn vortex_array::expr::Expression::binary(&self, vortex_array::expr::Expression, vortex_array::scalar_fn::fns::operators::Operator) -> vortex_error::VortexResult -pub fn vortex_array::expr::Expression::cast(&self, dtype: vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::expr::Expression::cast(&self, vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::expr::Expression::fill_null(&self, fill_value: vortex_array::expr::Expression) -> vortex_error::VortexResult +pub fn vortex_array::expr::Expression::fill_null(&self, vortex_array::expr::Expression) -> vortex_error::VortexResult -pub fn vortex_array::expr::Expression::get_item(&self, field_name: impl core::convert::Into) -> vortex_error::VortexResult +pub fn vortex_array::expr::Expression::get_item(&self, impl core::convert::Into) -> vortex_error::VortexResult + +pub fn vortex_array::expr::Expression::is_not_null(&self) -> vortex_error::VortexResult pub fn vortex_array::expr::Expression::is_null(&self) -> vortex_error::VortexResult -pub fn vortex_array::expr::Expression::list_contains(&self, value: vortex_array::expr::Expression) -> vortex_error::VortexResult +pub fn vortex_array::expr::Expression::list_contains(&self, vortex_array::expr::Expression) -> vortex_error::VortexResult -pub fn vortex_array::expr::Expression::mask(&self, mask: vortex_array::expr::Expression) -> vortex_error::VortexResult +pub fn vortex_array::expr::Expression::mask(&self, vortex_array::expr::Expression) -> vortex_error::VortexResult pub fn vortex_array::expr::Expression::not(&self) -> vortex_error::VortexResult -pub fn vortex_array::expr::Expression::zip(&self, if_true: vortex_array::expr::Expression, if_false: vortex_array::expr::Expression) -> vortex_error::VortexResult +pub fn vortex_array::expr::Expression::zip(&self, vortex_array::expr::Expression, vortex_array::expr::Expression) -> vortex_error::VortexResult pub mod vortex_array::compute @@ -8544,27 +8696,27 @@ pub vortex_array::display::BufferExtractor::show_percent: bool impl vortex_array::display::TreeExtractor for vortex_array::display::BufferExtractor -pub fn vortex_array::display::BufferExtractor::write_details(&self, array: &vortex_array::ArrayRef, _ctx: &vortex_array::display::TreeContext, f: &mut vortex_array::display::IndentedFormatter<'_, '_>) -> core::fmt::Result +pub fn vortex_array::display::BufferExtractor::write_details(&self, &vortex_array::ArrayRef, &vortex_array::display::TreeContext, &mut vortex_array::display::IndentedFormatter<'_, '_>) -> core::fmt::Result -pub fn vortex_array::display::BufferExtractor::write_header(&self, array: &vortex_array::ArrayRef, ctx: &vortex_array::display::TreeContext, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::display::BufferExtractor::write_header(&self, &vortex_array::ArrayRef, &vortex_array::display::TreeContext, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_array::display::DisplayArrayAs<'a>(pub &'a vortex_array::ArrayRef, pub vortex_array::display::DisplayOptions) impl core::fmt::Display for vortex_array::display::DisplayArrayAs<'_> -pub fn vortex_array::display::DisplayArrayAs<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::display::DisplayArrayAs<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_array::display::EncodingSummaryExtractor impl vortex_array::display::EncodingSummaryExtractor -pub fn vortex_array::display::EncodingSummaryExtractor::write(array: &vortex_array::ArrayRef, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::display::EncodingSummaryExtractor::write(&vortex_array::ArrayRef, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::display::TreeExtractor for vortex_array::display::EncodingSummaryExtractor -pub fn vortex_array::display::EncodingSummaryExtractor::write_details(&self, array: &vortex_array::ArrayRef, ctx: &vortex_array::display::TreeContext, f: &mut vortex_array::display::IndentedFormatter<'_, '_>) -> core::fmt::Result +pub fn vortex_array::display::EncodingSummaryExtractor::write_details(&self, &vortex_array::ArrayRef, &vortex_array::display::TreeContext, &mut vortex_array::display::IndentedFormatter<'_, '_>) -> core::fmt::Result -pub fn vortex_array::display::EncodingSummaryExtractor::write_header(&self, array: &vortex_array::ArrayRef, _ctx: &vortex_array::display::TreeContext, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::display::EncodingSummaryExtractor::write_header(&self, &vortex_array::ArrayRef, &vortex_array::display::TreeContext, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_array::display::IndentedFormatter<'a, 'b> @@ -8580,25 +8732,25 @@ pub struct vortex_array::display::MetadataExtractor impl vortex_array::display::TreeExtractor for vortex_array::display::MetadataExtractor -pub fn vortex_array::display::MetadataExtractor::write_details(&self, array: &vortex_array::ArrayRef, _ctx: &vortex_array::display::TreeContext, f: &mut vortex_array::display::IndentedFormatter<'_, '_>) -> core::fmt::Result +pub fn vortex_array::display::MetadataExtractor::write_details(&self, &vortex_array::ArrayRef, &vortex_array::display::TreeContext, &mut vortex_array::display::IndentedFormatter<'_, '_>) -> core::fmt::Result -pub fn vortex_array::display::MetadataExtractor::write_header(&self, array: &vortex_array::ArrayRef, ctx: &vortex_array::display::TreeContext, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::display::MetadataExtractor::write_header(&self, &vortex_array::ArrayRef, &vortex_array::display::TreeContext, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_array::display::NbytesExtractor impl vortex_array::display::TreeExtractor for vortex_array::display::NbytesExtractor -pub fn vortex_array::display::NbytesExtractor::write_details(&self, array: &vortex_array::ArrayRef, ctx: &vortex_array::display::TreeContext, f: &mut vortex_array::display::IndentedFormatter<'_, '_>) -> core::fmt::Result +pub fn vortex_array::display::NbytesExtractor::write_details(&self, &vortex_array::ArrayRef, &vortex_array::display::TreeContext, &mut vortex_array::display::IndentedFormatter<'_, '_>) -> core::fmt::Result -pub fn vortex_array::display::NbytesExtractor::write_header(&self, array: &vortex_array::ArrayRef, ctx: &vortex_array::display::TreeContext, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::display::NbytesExtractor::write_header(&self, &vortex_array::ArrayRef, &vortex_array::display::TreeContext, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_array::display::StatsExtractor impl vortex_array::display::TreeExtractor for vortex_array::display::StatsExtractor -pub fn vortex_array::display::StatsExtractor::write_details(&self, array: &vortex_array::ArrayRef, ctx: &vortex_array::display::TreeContext, f: &mut vortex_array::display::IndentedFormatter<'_, '_>) -> core::fmt::Result +pub fn vortex_array::display::StatsExtractor::write_details(&self, &vortex_array::ArrayRef, &vortex_array::display::TreeContext, &mut vortex_array::display::IndentedFormatter<'_, '_>) -> core::fmt::Result -pub fn vortex_array::display::StatsExtractor::write_header(&self, array: &vortex_array::ArrayRef, _ctx: &vortex_array::display::TreeContext, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::display::StatsExtractor::write_header(&self, &vortex_array::ArrayRef, &vortex_array::display::TreeContext, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_array::display::TreeContext @@ -8610,53 +8762,53 @@ pub struct vortex_array::display::TreeDisplay impl vortex_array::display::TreeDisplay -pub fn vortex_array::display::TreeDisplay::default_display(array: vortex_array::ArrayRef) -> Self +pub fn vortex_array::display::TreeDisplay::default_display(vortex_array::ArrayRef) -> Self -pub fn vortex_array::display::TreeDisplay::new(array: vortex_array::ArrayRef) -> Self +pub fn vortex_array::display::TreeDisplay::new(vortex_array::ArrayRef) -> Self -pub fn vortex_array::display::TreeDisplay::with(self, extractor: E) -> Self +pub fn vortex_array::display::TreeDisplay::with(self, E) -> Self -pub fn vortex_array::display::TreeDisplay::with_boxed(self, extractor: alloc::boxed::Box) -> Self +pub fn vortex_array::display::TreeDisplay::with_boxed(self, alloc::boxed::Box) -> Self impl core::fmt::Display for vortex_array::display::TreeDisplay -pub fn vortex_array::display::TreeDisplay::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::display::TreeDisplay::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub trait vortex_array::display::TreeExtractor: core::marker::Send + core::marker::Sync -pub fn vortex_array::display::TreeExtractor::write_details(&self, array: &vortex_array::ArrayRef, ctx: &vortex_array::display::TreeContext, f: &mut vortex_array::display::IndentedFormatter<'_, '_>) -> core::fmt::Result +pub fn vortex_array::display::TreeExtractor::write_details(&self, &vortex_array::ArrayRef, &vortex_array::display::TreeContext, &mut vortex_array::display::IndentedFormatter<'_, '_>) -> core::fmt::Result -pub fn vortex_array::display::TreeExtractor::write_header(&self, array: &vortex_array::ArrayRef, ctx: &vortex_array::display::TreeContext, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::display::TreeExtractor::write_header(&self, &vortex_array::ArrayRef, &vortex_array::display::TreeContext, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::display::TreeExtractor for vortex_array::display::BufferExtractor -pub fn vortex_array::display::BufferExtractor::write_details(&self, array: &vortex_array::ArrayRef, _ctx: &vortex_array::display::TreeContext, f: &mut vortex_array::display::IndentedFormatter<'_, '_>) -> core::fmt::Result +pub fn vortex_array::display::BufferExtractor::write_details(&self, &vortex_array::ArrayRef, &vortex_array::display::TreeContext, &mut vortex_array::display::IndentedFormatter<'_, '_>) -> core::fmt::Result -pub fn vortex_array::display::BufferExtractor::write_header(&self, array: &vortex_array::ArrayRef, ctx: &vortex_array::display::TreeContext, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::display::BufferExtractor::write_header(&self, &vortex_array::ArrayRef, &vortex_array::display::TreeContext, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::display::TreeExtractor for vortex_array::display::EncodingSummaryExtractor -pub fn vortex_array::display::EncodingSummaryExtractor::write_details(&self, array: &vortex_array::ArrayRef, ctx: &vortex_array::display::TreeContext, f: &mut vortex_array::display::IndentedFormatter<'_, '_>) -> core::fmt::Result +pub fn vortex_array::display::EncodingSummaryExtractor::write_details(&self, &vortex_array::ArrayRef, &vortex_array::display::TreeContext, &mut vortex_array::display::IndentedFormatter<'_, '_>) -> core::fmt::Result -pub fn vortex_array::display::EncodingSummaryExtractor::write_header(&self, array: &vortex_array::ArrayRef, _ctx: &vortex_array::display::TreeContext, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::display::EncodingSummaryExtractor::write_header(&self, &vortex_array::ArrayRef, &vortex_array::display::TreeContext, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::display::TreeExtractor for vortex_array::display::MetadataExtractor -pub fn vortex_array::display::MetadataExtractor::write_details(&self, array: &vortex_array::ArrayRef, _ctx: &vortex_array::display::TreeContext, f: &mut vortex_array::display::IndentedFormatter<'_, '_>) -> core::fmt::Result +pub fn vortex_array::display::MetadataExtractor::write_details(&self, &vortex_array::ArrayRef, &vortex_array::display::TreeContext, &mut vortex_array::display::IndentedFormatter<'_, '_>) -> core::fmt::Result -pub fn vortex_array::display::MetadataExtractor::write_header(&self, array: &vortex_array::ArrayRef, ctx: &vortex_array::display::TreeContext, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::display::MetadataExtractor::write_header(&self, &vortex_array::ArrayRef, &vortex_array::display::TreeContext, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::display::TreeExtractor for vortex_array::display::NbytesExtractor -pub fn vortex_array::display::NbytesExtractor::write_details(&self, array: &vortex_array::ArrayRef, ctx: &vortex_array::display::TreeContext, f: &mut vortex_array::display::IndentedFormatter<'_, '_>) -> core::fmt::Result +pub fn vortex_array::display::NbytesExtractor::write_details(&self, &vortex_array::ArrayRef, &vortex_array::display::TreeContext, &mut vortex_array::display::IndentedFormatter<'_, '_>) -> core::fmt::Result -pub fn vortex_array::display::NbytesExtractor::write_header(&self, array: &vortex_array::ArrayRef, ctx: &vortex_array::display::TreeContext, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::display::NbytesExtractor::write_header(&self, &vortex_array::ArrayRef, &vortex_array::display::TreeContext, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::display::TreeExtractor for vortex_array::display::StatsExtractor -pub fn vortex_array::display::StatsExtractor::write_details(&self, array: &vortex_array::ArrayRef, ctx: &vortex_array::display::TreeContext, f: &mut vortex_array::display::IndentedFormatter<'_, '_>) -> core::fmt::Result +pub fn vortex_array::display::StatsExtractor::write_details(&self, &vortex_array::ArrayRef, &vortex_array::display::TreeContext, &mut vortex_array::display::IndentedFormatter<'_, '_>) -> core::fmt::Result -pub fn vortex_array::display::StatsExtractor::write_header(&self, array: &vortex_array::ArrayRef, _ctx: &vortex_array::display::TreeContext, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::display::StatsExtractor::write_header(&self, &vortex_array::ArrayRef, &vortex_array::display::TreeContext, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub mod vortex_array::dtype @@ -8666,39 +8818,39 @@ pub mod vortex_array::dtype::arrow pub trait vortex_array::dtype::arrow::FromArrowType: core::marker::Sized -pub fn vortex_array::dtype::arrow::FromArrowType::from_arrow(value: T) -> Self +pub fn vortex_array::dtype::arrow::FromArrowType::from_arrow(T) -> Self impl vortex_array::dtype::arrow::FromArrowType<&arrow_schema::field::Field> for vortex_array::dtype::DType -pub fn vortex_array::dtype::DType::from_arrow(field: &arrow_schema::field::Field) -> Self +pub fn vortex_array::dtype::DType::from_arrow(&arrow_schema::field::Field) -> Self impl vortex_array::dtype::arrow::FromArrowType<&arrow_schema::fields::Fields> for vortex_array::dtype::StructFields -pub fn vortex_array::dtype::StructFields::from_arrow(value: &arrow_schema::fields::Fields) -> Self +pub fn vortex_array::dtype::StructFields::from_arrow(&arrow_schema::fields::Fields) -> Self impl vortex_array::dtype::arrow::FromArrowType<&arrow_schema::schema::Schema> for vortex_array::dtype::DType -pub fn vortex_array::dtype::DType::from_arrow(value: &arrow_schema::schema::Schema) -> Self +pub fn vortex_array::dtype::DType::from_arrow(&arrow_schema::schema::Schema) -> Self impl vortex_array::dtype::arrow::FromArrowType<(&arrow_schema::datatype::DataType, vortex_array::dtype::Nullability)> for vortex_array::dtype::DType -pub fn vortex_array::dtype::DType::from_arrow((data_type, nullability): (&arrow_schema::datatype::DataType, vortex_array::dtype::Nullability)) -> Self +pub fn vortex_array::dtype::DType::from_arrow((&arrow_schema::datatype::DataType, vortex_array::dtype::Nullability)) -> Self impl vortex_array::dtype::arrow::FromArrowType> for vortex_array::dtype::DType -pub fn vortex_array::dtype::DType::from_arrow(value: arrow_schema::schema::SchemaRef) -> Self +pub fn vortex_array::dtype::DType::from_arrow(arrow_schema::schema::SchemaRef) -> Self pub trait vortex_array::dtype::arrow::TryFromArrowType: core::marker::Sized -pub fn vortex_array::dtype::arrow::TryFromArrowType::try_from_arrow(value: T) -> vortex_error::VortexResult +pub fn vortex_array::dtype::arrow::TryFromArrowType::try_from_arrow(T) -> vortex_error::VortexResult impl vortex_array::dtype::arrow::TryFromArrowType<&arrow_schema::datatype::DataType> for vortex_array::dtype::DecimalDType -pub fn vortex_array::dtype::DecimalDType::try_from_arrow(value: &arrow_schema::datatype::DataType) -> vortex_error::VortexResult +pub fn vortex_array::dtype::DecimalDType::try_from_arrow(&arrow_schema::datatype::DataType) -> vortex_error::VortexResult impl vortex_array::dtype::arrow::TryFromArrowType<&arrow_schema::datatype::DataType> for vortex_array::dtype::PType -pub fn vortex_array::dtype::PType::try_from_arrow(value: &arrow_schema::datatype::DataType) -> vortex_error::VortexResult +pub fn vortex_array::dtype::PType::try_from_arrow(&arrow_schema::datatype::DataType) -> vortex_error::VortexResult pub mod vortex_array::dtype::extension @@ -8706,19 +8858,19 @@ pub struct vortex_array::dtype::extension::ExtDType vortex_array::dtype::extension::ExtDType -pub fn vortex_array::dtype::extension::ExtDType::try_new(metadata: ::Metadata, storage_dtype: vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::dtype::extension::ExtDType::try_new(::Metadata, vortex_array::dtype::DType) -> vortex_error::VortexResult impl vortex_array::dtype::extension::ExtDType -pub fn vortex_array::dtype::extension::ExtDType::can_coerce_from(&self, other: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::dtype::extension::ExtDType::can_coerce_from(&self, &vortex_array::dtype::DType) -> bool -pub fn vortex_array::dtype::extension::ExtDType::can_coerce_to(&self, other: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::dtype::extension::ExtDType::can_coerce_to(&self, &vortex_array::dtype::DType) -> bool pub fn vortex_array::dtype::extension::ExtDType::erased(self) -> vortex_array::dtype::extension::ExtDTypeRef pub fn vortex_array::dtype::extension::ExtDType::id(&self) -> vortex_array::dtype::extension::ExtId -pub fn vortex_array::dtype::extension::ExtDType::least_supertype(&self, other: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::dtype::extension::ExtDType::least_supertype(&self, &vortex_array::dtype::DType) -> core::option::Option pub fn vortex_array::dtype::extension::ExtDType::metadata(&self) -> &::Metadata @@ -8726,13 +8878,13 @@ pub fn vortex_array::dtype::extension::ExtDType::serialize_metadata(&self) -> pub fn vortex_array::dtype::extension::ExtDType::storage_dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::dtype::extension::ExtDType::try_with_vtable(vtable: V, metadata: ::Metadata, storage_dtype: vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::dtype::extension::ExtDType::try_with_vtable(V, ::Metadata, vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::dtype::extension::ExtDType::validate_scalar_value(&self, storage_value: &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult<()> +pub fn vortex_array::dtype::extension::ExtDType::validate_scalar_value(&self, &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult<()> pub fn vortex_array::dtype::extension::ExtDType::vtable(&self) -> &V -pub fn vortex_array::dtype::extension::ExtDType::with_nullability(&self, nullability: vortex_array::dtype::Nullability) -> vortex_array::dtype::extension::ExtDTypeRef +pub fn vortex_array::dtype::extension::ExtDType::with_nullability(&self, vortex_array::dtype::Nullability) -> vortex_array::dtype::extension::ExtDTypeRef impl core::clone::Clone for vortex_array::dtype::extension::ExtDType where ::Metadata: core::clone::Clone @@ -8742,15 +8894,15 @@ impl core::cmp::Eq impl core::cmp::PartialEq for vortex_array::dtype::extension::ExtDType where ::Metadata: core::cmp::PartialEq -pub fn vortex_array::dtype::extension::ExtDType::eq(&self, other: &vortex_array::dtype::extension::ExtDType) -> bool +pub fn vortex_array::dtype::extension::ExtDType::eq(&self, &vortex_array::dtype::extension::ExtDType) -> bool impl core::fmt::Debug for vortex_array::dtype::extension::ExtDType where ::Metadata: core::fmt::Debug -pub fn vortex_array::dtype::extension::ExtDType::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::extension::ExtDType::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::dtype::extension::ExtDType where ::Metadata: core::hash::Hash -pub fn vortex_array::dtype::extension::ExtDType::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::dtype::extension::ExtDType::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_array::dtype::extension::ExtDType @@ -8758,19 +8910,19 @@ pub struct vortex_array::dtype::extension::ExtDTypeRef(_) impl vortex_array::dtype::extension::ExtDTypeRef -pub fn vortex_array::dtype::extension::ExtDTypeRef::can_coerce_from(&self, other: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::dtype::extension::ExtDTypeRef::can_coerce_from(&self, &vortex_array::dtype::DType) -> bool -pub fn vortex_array::dtype::extension::ExtDTypeRef::can_coerce_to(&self, other: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::dtype::extension::ExtDTypeRef::can_coerce_to(&self, &vortex_array::dtype::DType) -> bool pub fn vortex_array::dtype::extension::ExtDTypeRef::display_metadata(&self) -> impl core::fmt::Display + '_ -pub fn vortex_array::dtype::extension::ExtDTypeRef::eq_ignore_nullability(&self, other: &Self) -> bool +pub fn vortex_array::dtype::extension::ExtDTypeRef::eq_ignore_nullability(&self, &Self) -> bool pub fn vortex_array::dtype::extension::ExtDTypeRef::id(&self) -> vortex_array::dtype::extension::ExtId pub fn vortex_array::dtype::extension::ExtDTypeRef::is_nullable(&self) -> bool -pub fn vortex_array::dtype::extension::ExtDTypeRef::least_supertype(&self, other: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::dtype::extension::ExtDTypeRef::least_supertype(&self, &vortex_array::dtype::DType) -> core::option::Option pub fn vortex_array::dtype::extension::ExtDTypeRef::nullability(&self) -> vortex_array::dtype::Nullability @@ -8778,7 +8930,7 @@ pub fn vortex_array::dtype::extension::ExtDTypeRef::serialize_metadata(&self) -> pub fn vortex_array::dtype::extension::ExtDTypeRef::storage_dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::dtype::extension::ExtDTypeRef::with_nullability(&self, nullability: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::dtype::extension::ExtDTypeRef::with_nullability(&self, vortex_array::dtype::Nullability) -> Self impl vortex_array::dtype::extension::ExtDTypeRef @@ -8800,31 +8952,31 @@ impl core::cmp::Eq for vortex_array::dtype::extension::ExtDTypeRef impl core::cmp::PartialEq for vortex_array::dtype::extension::ExtDTypeRef -pub fn vortex_array::dtype::extension::ExtDTypeRef::eq(&self, other: &Self) -> bool +pub fn vortex_array::dtype::extension::ExtDTypeRef::eq(&self, &Self) -> bool impl core::fmt::Debug for vortex_array::dtype::extension::ExtDTypeRef -pub fn vortex_array::dtype::extension::ExtDTypeRef::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::extension::ExtDTypeRef::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::dtype::extension::ExtDTypeRef -pub fn vortex_array::dtype::extension::ExtDTypeRef::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::extension::ExtDTypeRef::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::dtype::extension::ExtDTypeRef -pub fn vortex_array::dtype::extension::ExtDTypeRef::hash(&self, state: &mut H) +pub fn vortex_array::dtype::extension::ExtDTypeRef::hash(&self, &mut H) pub trait vortex_array::dtype::extension::ExtDTypePlugin: 'static + core::marker::Send + core::marker::Sync + core::fmt::Debug -pub fn vortex_array::dtype::extension::ExtDTypePlugin::deserialize(&self, data: &[u8], storage_dtype: vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::dtype::extension::ExtDTypePlugin::deserialize(&self, &[u8], vortex_array::dtype::DType) -> vortex_error::VortexResult pub fn vortex_array::dtype::extension::ExtDTypePlugin::id(&self) -> vortex_array::dtype::extension::ExtId impl vortex_array::dtype::extension::ExtDTypePlugin for V -pub fn V::deserialize(&self, data: &[u8], storage_dtype: vortex_array::dtype::DType) -> core::result::Result +pub fn V::deserialize(&self, &[u8], vortex_array::dtype::DType) -> core::result::Result -pub fn V::id(&self) -> arcref::ArcRef +pub fn V::id(&self) -> vortex_session::registry::Id pub trait vortex_array::dtype::extension::ExtVTable: 'static + core::marker::Sized + core::marker::Send + core::marker::Sync + core::clone::Clone + core::fmt::Debug + core::cmp::Eq + core::hash::Hash @@ -8832,23 +8984,23 @@ pub type vortex_array::dtype::extension::ExtVTable::Metadata: 'static + core::ma pub type vortex_array::dtype::extension::ExtVTable::NativeValue<'a>: core::fmt::Display -pub fn vortex_array::dtype::extension::ExtVTable::can_coerce_from(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::dtype::extension::ExtVTable::can_coerce_from(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> bool -pub fn vortex_array::dtype::extension::ExtVTable::can_coerce_to(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::dtype::extension::ExtVTable::can_coerce_to(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> bool -pub fn vortex_array::dtype::extension::ExtVTable::deserialize_metadata(&self, metadata: &[u8]) -> vortex_error::VortexResult +pub fn vortex_array::dtype::extension::ExtVTable::deserialize_metadata(&self, &[u8]) -> vortex_error::VortexResult pub fn vortex_array::dtype::extension::ExtVTable::id(&self) -> vortex_array::dtype::extension::ExtId -pub fn vortex_array::dtype::extension::ExtVTable::least_supertype(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::dtype::extension::ExtVTable::least_supertype(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::dtype::extension::ExtVTable::serialize_metadata(&self, metadata: &Self::Metadata) -> vortex_error::VortexResult> +pub fn vortex_array::dtype::extension::ExtVTable::serialize_metadata(&self, &Self::Metadata) -> vortex_error::VortexResult> -pub fn vortex_array::dtype::extension::ExtVTable::unpack_native<'a>(ext_dtype: &'a vortex_array::dtype::extension::ExtDType, storage_value: &'a vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult +pub fn vortex_array::dtype::extension::ExtVTable::unpack_native<'a>(&'a vortex_array::dtype::extension::ExtDType, &'a vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult -pub fn vortex_array::dtype::extension::ExtVTable::validate_dtype(ext_dtype: &vortex_array::dtype::extension::ExtDType) -> vortex_error::VortexResult<()> +pub fn vortex_array::dtype::extension::ExtVTable::validate_dtype(&vortex_array::dtype::extension::ExtDType) -> vortex_error::VortexResult<()> -pub fn vortex_array::dtype::extension::ExtVTable::validate_scalar_value(ext_dtype: &vortex_array::dtype::extension::ExtDType, storage_value: &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult<()> +pub fn vortex_array::dtype::extension::ExtVTable::validate_scalar_value(&vortex_array::dtype::extension::ExtDType, &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult<()> impl vortex_array::dtype::extension::ExtVTable for vortex_array::extension::datetime::Date @@ -8856,23 +9008,23 @@ pub type vortex_array::extension::datetime::Date::Metadata = vortex_array::exten pub type vortex_array::extension::datetime::Date::NativeValue<'a> = vortex_array::extension::datetime::DateValue -pub fn vortex_array::extension::datetime::Date::can_coerce_from(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::extension::datetime::Date::can_coerce_from(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> bool -pub fn vortex_array::extension::datetime::Date::can_coerce_to(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::extension::datetime::Date::can_coerce_to(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> bool -pub fn vortex_array::extension::datetime::Date::deserialize_metadata(&self, metadata: &[u8]) -> vortex_error::VortexResult +pub fn vortex_array::extension::datetime::Date::deserialize_metadata(&self, &[u8]) -> vortex_error::VortexResult pub fn vortex_array::extension::datetime::Date::id(&self) -> vortex_array::dtype::extension::ExtId -pub fn vortex_array::extension::datetime::Date::least_supertype(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::extension::datetime::Date::least_supertype(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::extension::datetime::Date::serialize_metadata(&self, metadata: &Self::Metadata) -> vortex_error::VortexResult> +pub fn vortex_array::extension::datetime::Date::serialize_metadata(&self, &Self::Metadata) -> vortex_error::VortexResult> -pub fn vortex_array::extension::datetime::Date::unpack_native<'a>(ext_dtype: &'a vortex_array::dtype::extension::ExtDType, storage_value: &'a vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult +pub fn vortex_array::extension::datetime::Date::unpack_native<'a>(&'a vortex_array::dtype::extension::ExtDType, &'a vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult -pub fn vortex_array::extension::datetime::Date::validate_dtype(ext_dtype: &vortex_array::dtype::extension::ExtDType) -> vortex_error::VortexResult<()> +pub fn vortex_array::extension::datetime::Date::validate_dtype(&vortex_array::dtype::extension::ExtDType) -> vortex_error::VortexResult<()> -pub fn vortex_array::extension::datetime::Date::validate_scalar_value(ext_dtype: &vortex_array::dtype::extension::ExtDType, storage_value: &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult<()> +pub fn vortex_array::extension::datetime::Date::validate_scalar_value(&vortex_array::dtype::extension::ExtDType, &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult<()> impl vortex_array::dtype::extension::ExtVTable for vortex_array::extension::datetime::Time @@ -8880,23 +9032,23 @@ pub type vortex_array::extension::datetime::Time::Metadata = vortex_array::exten pub type vortex_array::extension::datetime::Time::NativeValue<'a> = vortex_array::extension::datetime::TimeValue -pub fn vortex_array::extension::datetime::Time::can_coerce_from(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::extension::datetime::Time::can_coerce_from(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> bool -pub fn vortex_array::extension::datetime::Time::can_coerce_to(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::extension::datetime::Time::can_coerce_to(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> bool -pub fn vortex_array::extension::datetime::Time::deserialize_metadata(&self, data: &[u8]) -> vortex_error::VortexResult +pub fn vortex_array::extension::datetime::Time::deserialize_metadata(&self, &[u8]) -> vortex_error::VortexResult pub fn vortex_array::extension::datetime::Time::id(&self) -> vortex_array::dtype::extension::ExtId -pub fn vortex_array::extension::datetime::Time::least_supertype(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::extension::datetime::Time::least_supertype(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::extension::datetime::Time::serialize_metadata(&self, metadata: &Self::Metadata) -> vortex_error::VortexResult> +pub fn vortex_array::extension::datetime::Time::serialize_metadata(&self, &Self::Metadata) -> vortex_error::VortexResult> -pub fn vortex_array::extension::datetime::Time::unpack_native<'a>(ext_dtype: &'a vortex_array::dtype::extension::ExtDType, storage_value: &'a vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult +pub fn vortex_array::extension::datetime::Time::unpack_native<'a>(&'a vortex_array::dtype::extension::ExtDType, &'a vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult -pub fn vortex_array::extension::datetime::Time::validate_dtype(ext_dtype: &vortex_array::dtype::extension::ExtDType) -> vortex_error::VortexResult<()> +pub fn vortex_array::extension::datetime::Time::validate_dtype(&vortex_array::dtype::extension::ExtDType) -> vortex_error::VortexResult<()> -pub fn vortex_array::extension::datetime::Time::validate_scalar_value(ext_dtype: &vortex_array::dtype::extension::ExtDType, storage_value: &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult<()> +pub fn vortex_array::extension::datetime::Time::validate_scalar_value(&vortex_array::dtype::extension::ExtDType, &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult<()> impl vortex_array::dtype::extension::ExtVTable for vortex_array::extension::datetime::Timestamp @@ -8904,23 +9056,23 @@ pub type vortex_array::extension::datetime::Timestamp::Metadata = vortex_array:: pub type vortex_array::extension::datetime::Timestamp::NativeValue<'a> = vortex_array::extension::datetime::TimestampValue<'a> -pub fn vortex_array::extension::datetime::Timestamp::can_coerce_from(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::extension::datetime::Timestamp::can_coerce_from(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> bool -pub fn vortex_array::extension::datetime::Timestamp::can_coerce_to(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::extension::datetime::Timestamp::can_coerce_to(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> bool -pub fn vortex_array::extension::datetime::Timestamp::deserialize_metadata(&self, data: &[u8]) -> vortex_error::VortexResult +pub fn vortex_array::extension::datetime::Timestamp::deserialize_metadata(&self, &[u8]) -> vortex_error::VortexResult pub fn vortex_array::extension::datetime::Timestamp::id(&self) -> vortex_array::dtype::extension::ExtId -pub fn vortex_array::extension::datetime::Timestamp::least_supertype(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::extension::datetime::Timestamp::least_supertype(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::extension::datetime::Timestamp::serialize_metadata(&self, metadata: &Self::Metadata) -> vortex_error::VortexResult> +pub fn vortex_array::extension::datetime::Timestamp::serialize_metadata(&self, &Self::Metadata) -> vortex_error::VortexResult> -pub fn vortex_array::extension::datetime::Timestamp::unpack_native<'a>(ext_dtype: &'a vortex_array::dtype::extension::ExtDType, storage_value: &'a vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult +pub fn vortex_array::extension::datetime::Timestamp::unpack_native<'a>(&'a vortex_array::dtype::extension::ExtDType, &'a vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult -pub fn vortex_array::extension::datetime::Timestamp::validate_dtype(ext_dtype: &vortex_array::dtype::extension::ExtDType) -> vortex_error::VortexResult<()> +pub fn vortex_array::extension::datetime::Timestamp::validate_dtype(&vortex_array::dtype::extension::ExtDType) -> vortex_error::VortexResult<()> -pub fn vortex_array::extension::datetime::Timestamp::validate_scalar_value(ext_dtype: &vortex_array::dtype::extension::ExtDType, storage_value: &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult<()> +pub fn vortex_array::extension::datetime::Timestamp::validate_scalar_value(&vortex_array::dtype::extension::ExtDType, &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult<()> impl vortex_array::dtype::extension::ExtVTable for vortex_array::extension::uuid::Uuid @@ -8928,51 +9080,51 @@ pub type vortex_array::extension::uuid::Uuid::Metadata = vortex_array::extension pub type vortex_array::extension::uuid::Uuid::NativeValue<'a> = uuid::Uuid -pub fn vortex_array::extension::uuid::Uuid::can_coerce_from(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::extension::uuid::Uuid::can_coerce_from(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> bool -pub fn vortex_array::extension::uuid::Uuid::can_coerce_to(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::extension::uuid::Uuid::can_coerce_to(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> bool -pub fn vortex_array::extension::uuid::Uuid::deserialize_metadata(&self, metadata: &[u8]) -> vortex_error::VortexResult +pub fn vortex_array::extension::uuid::Uuid::deserialize_metadata(&self, &[u8]) -> vortex_error::VortexResult pub fn vortex_array::extension::uuid::Uuid::id(&self) -> vortex_array::dtype::extension::ExtId -pub fn vortex_array::extension::uuid::Uuid::least_supertype(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::extension::uuid::Uuid::least_supertype(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::extension::uuid::Uuid::serialize_metadata(&self, metadata: &Self::Metadata) -> vortex_error::VortexResult> +pub fn vortex_array::extension::uuid::Uuid::serialize_metadata(&self, &Self::Metadata) -> vortex_error::VortexResult> -pub fn vortex_array::extension::uuid::Uuid::unpack_native<'a>(ext_dtype: &vortex_array::dtype::extension::ExtDType, storage_value: &'a vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult +pub fn vortex_array::extension::uuid::Uuid::unpack_native<'a>(&vortex_array::dtype::extension::ExtDType, &'a vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult -pub fn vortex_array::extension::uuid::Uuid::validate_dtype(ext_dtype: &vortex_array::dtype::extension::ExtDType) -> vortex_error::VortexResult<()> +pub fn vortex_array::extension::uuid::Uuid::validate_dtype(&vortex_array::dtype::extension::ExtDType) -> vortex_error::VortexResult<()> -pub fn vortex_array::extension::uuid::Uuid::validate_scalar_value(ext_dtype: &vortex_array::dtype::extension::ExtDType, storage_value: &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult<()> +pub fn vortex_array::extension::uuid::Uuid::validate_scalar_value(&vortex_array::dtype::extension::ExtDType, &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult<()> pub trait vortex_array::dtype::extension::Matcher pub type vortex_array::dtype::extension::Matcher::Match<'a> -pub fn vortex_array::dtype::extension::Matcher::matches(item: &vortex_array::dtype::extension::ExtDTypeRef) -> bool +pub fn vortex_array::dtype::extension::Matcher::matches(&vortex_array::dtype::extension::ExtDTypeRef) -> bool -pub fn vortex_array::dtype::extension::Matcher::try_match<'a>(item: &'a vortex_array::dtype::extension::ExtDTypeRef) -> core::option::Option +pub fn vortex_array::dtype::extension::Matcher::try_match<'a>(&'a vortex_array::dtype::extension::ExtDTypeRef) -> core::option::Option impl vortex_array::dtype::extension::Matcher for vortex_array::extension::datetime::AnyTemporal pub type vortex_array::extension::datetime::AnyTemporal::Match<'a> = vortex_array::extension::datetime::TemporalMetadata<'a> -pub fn vortex_array::extension::datetime::AnyTemporal::matches(item: &vortex_array::dtype::extension::ExtDTypeRef) -> bool +pub fn vortex_array::extension::datetime::AnyTemporal::matches(&vortex_array::dtype::extension::ExtDTypeRef) -> bool -pub fn vortex_array::extension::datetime::AnyTemporal::try_match<'a>(item: &'a vortex_array::dtype::extension::ExtDTypeRef) -> core::option::Option +pub fn vortex_array::extension::datetime::AnyTemporal::try_match<'a>(&'a vortex_array::dtype::extension::ExtDTypeRef) -> core::option::Option impl vortex_array::dtype::extension::Matcher for V pub type V::Match<'a> = &'a ::Metadata -pub fn V::matches(ext_dtype: &vortex_array::dtype::extension::ExtDTypeRef) -> bool +pub fn V::matches(&vortex_array::dtype::extension::ExtDTypeRef) -> bool -pub fn V::try_match<'a>(ext_dtype: &'a vortex_array::dtype::extension::ExtDTypeRef) -> core::option::Option<::Match> +pub fn V::try_match<'a>(&'a vortex_array::dtype::extension::ExtDTypeRef) -> core::option::Option<::Match> pub type vortex_array::dtype::extension::ExtDTypePluginRef = alloc::sync::Arc -pub type vortex_array::dtype::extension::ExtId = arcref::ArcRef +pub type vortex_array::dtype::extension::ExtId = vortex_session::registry::Id pub mod vortex_array::dtype::flatbuffers @@ -8990,7 +9142,7 @@ pub struct vortex_array::dtype::session::DTypeSession impl vortex_array::dtype::session::DTypeSession -pub fn vortex_array::dtype::session::DTypeSession::register(&self, vtable: V) +pub fn vortex_array::dtype::session::DTypeSession::register(&self, V) pub fn vortex_array::dtype::session::DTypeSession::registry(&self) -> &vortex_array::dtype::session::ExtDTypeRegistry @@ -9000,7 +9152,13 @@ pub fn vortex_array::dtype::session::DTypeSession::default() -> Self impl core::fmt::Debug for vortex_array::dtype::session::DTypeSession -pub fn vortex_array::dtype::session::DTypeSession::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::session::DTypeSession::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +impl vortex_session::SessionVar for vortex_array::dtype::session::DTypeSession + +pub fn vortex_array::dtype::session::DTypeSession::as_any(&self) -> &dyn core::any::Any + +pub fn vortex_array::dtype::session::DTypeSession::as_any_mut(&mut self) -> &mut dyn core::any::Any pub trait vortex_array::dtype::session::DTypeSessionExt: vortex_session::SessionExt @@ -9064,11 +9222,11 @@ pub fn vortex_array::dtype::DType::as_struct_fields_opt(&self) -> core::option:: pub fn vortex_array::dtype::DType::element_size(&self) -> core::option::Option -pub fn vortex_array::dtype::DType::eq_ignore_nullability(&self, other: &Self) -> bool +pub fn vortex_array::dtype::DType::eq_ignore_nullability(&self, &Self) -> bool -pub fn vortex_array::dtype::DType::eq_with_nullability_subset(&self, other: &Self) -> bool +pub fn vortex_array::dtype::DType::eq_with_nullability_subset(&self, &Self) -> bool -pub fn vortex_array::dtype::DType::eq_with_nullability_superset(&self, other: &Self) -> bool +pub fn vortex_array::dtype::DType::eq_with_nullability_superset(&self, &Self) -> bool pub fn vortex_array::dtype::DType::into_any_size_list_element_opt(self) -> core::option::Option> @@ -9114,45 +9272,45 @@ pub fn vortex_array::dtype::DType::is_utf8(&self) -> bool pub fn vortex_array::dtype::DType::is_variant(&self) -> bool -pub fn vortex_array::dtype::DType::list(dtype: impl core::convert::Into, nullability: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::dtype::DType::list(impl core::convert::Into, vortex_array::dtype::Nullability) -> Self pub fn vortex_array::dtype::DType::nullability(&self) -> vortex_array::dtype::Nullability -pub fn vortex_array::dtype::DType::struct_, impl core::convert::Into)>>(iter: I, nullability: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::dtype::DType::struct_, impl core::convert::Into)>>(I, vortex_array::dtype::Nullability) -> Self -pub fn vortex_array::dtype::DType::union_nullability(&self, other: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::dtype::DType::union_nullability(&self, vortex_array::dtype::Nullability) -> Self -pub fn vortex_array::dtype::DType::with_nullability(&self, nullability: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::dtype::DType::with_nullability(&self, vortex_array::dtype::Nullability) -> Self impl vortex_array::dtype::DType -pub fn vortex_array::dtype::DType::all_coercible_to(types: &[vortex_array::dtype::DType], target: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::dtype::DType::all_coercible_to(&[vortex_array::dtype::DType], &vortex_array::dtype::DType) -> bool -pub fn vortex_array::dtype::DType::are_coercible(types: &[vortex_array::dtype::DType]) -> bool +pub fn vortex_array::dtype::DType::are_coercible(&[vortex_array::dtype::DType]) -> bool -pub fn vortex_array::dtype::DType::can_coerce_from(&self, other: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::dtype::DType::can_coerce_from(&self, &vortex_array::dtype::DType) -> bool -pub fn vortex_array::dtype::DType::can_coerce_to(&self, other: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::dtype::DType::can_coerce_to(&self, &vortex_array::dtype::DType) -> bool -pub fn vortex_array::dtype::DType::coerce_all_to(types: &[vortex_array::dtype::DType], target: &vortex_array::dtype::DType) -> core::option::Option> +pub fn vortex_array::dtype::DType::coerce_all_to(&[vortex_array::dtype::DType], &vortex_array::dtype::DType) -> core::option::Option> -pub fn vortex_array::dtype::DType::coerce_to_supertype(types: &[vortex_array::dtype::DType]) -> core::option::Option> +pub fn vortex_array::dtype::DType::coerce_to_supertype(&[vortex_array::dtype::DType]) -> core::option::Option> pub fn vortex_array::dtype::DType::is_numeric(&self) -> bool pub fn vortex_array::dtype::DType::is_temporal(&self) -> bool -pub fn vortex_array::dtype::DType::least_supertype(&self, other: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::dtype::DType::least_supertype(&self, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::dtype::DType::least_supertype_of(types: &[vortex_array::dtype::DType]) -> core::option::Option +pub fn vortex_array::dtype::DType::least_supertype_of(&[vortex_array::dtype::DType]) -> core::option::Option impl vortex_array::dtype::DType -pub fn vortex_array::dtype::DType::from_flatbuffer(buffer: vortex_flatbuffers::FlatBuffer, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::dtype::DType::from_flatbuffer(vortex_flatbuffers::FlatBuffer, &vortex_session::VortexSession) -> vortex_error::VortexResult impl vortex_array::dtype::DType -pub fn vortex_array::dtype::DType::from_proto(value: &vortex_proto::dtype::DType, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::dtype::DType::from_proto(&vortex_proto::dtype::DType, &vortex_session::VortexSession) -> vortex_error::VortexResult impl vortex_array::dtype::DType @@ -9168,73 +9326,71 @@ impl core::cmp::Eq for vortex_array::dtype::DType impl core::cmp::PartialEq for vortex_array::dtype::DType -pub fn vortex_array::dtype::DType::eq(&self, other: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::dtype::DType::eq(&self, &Self) -> bool impl core::convert::From for vortex_array::dtype::FieldDType -pub fn vortex_array::dtype::FieldDType::from(value: vortex_array::dtype::DType) -> Self +pub fn vortex_array::dtype::FieldDType::from(vortex_array::dtype::DType) -> Self impl core::convert::From for &vortex_array::dtype::DType -pub fn &vortex_array::dtype::DType::from(item: vortex_array::dtype::PType) -> Self +pub fn &vortex_array::dtype::DType::from(vortex_array::dtype::PType) -> Self impl core::convert::From for vortex_array::dtype::DType -pub fn vortex_array::dtype::DType::from(item: vortex_array::dtype::PType) -> Self +pub fn vortex_array::dtype::DType::from(vortex_array::dtype::PType) -> Self impl core::convert::TryFrom<&vortex_array::dtype::DType> for vortex_array::dtype::DecimalDType pub type vortex_array::dtype::DecimalDType::Error = vortex_error::VortexError -pub fn vortex_array::dtype::DecimalDType::try_from(value: &vortex_array::dtype::DType) -> core::result::Result +pub fn vortex_array::dtype::DecimalDType::try_from(&vortex_array::dtype::DType) -> core::result::Result impl core::convert::TryFrom<&vortex_array::dtype::DType> for vortex_array::dtype::PType pub type vortex_array::dtype::PType::Error = vortex_error::VortexError -pub fn vortex_array::dtype::PType::try_from(value: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::dtype::PType::try_from(&vortex_array::dtype::DType) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::dtype::DType> for vortex_proto::dtype::DType pub type vortex_proto::dtype::DType::Error = vortex_error::VortexError -pub fn vortex_proto::dtype::DType::try_from(value: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_proto::dtype::DType::try_from(&vortex_array::dtype::DType) -> vortex_error::VortexResult impl core::convert::TryFrom for vortex_array::dtype::DecimalDType pub type vortex_array::dtype::DecimalDType::Error = vortex_error::VortexError -pub fn vortex_array::dtype::DecimalDType::try_from(value: vortex_array::dtype::DType) -> core::result::Result +pub fn vortex_array::dtype::DecimalDType::try_from(vortex_array::dtype::DType) -> core::result::Result impl core::fmt::Debug for vortex_array::dtype::DType -pub fn vortex_array::dtype::DType::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::DType::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::dtype::DType -pub fn vortex_array::dtype::DType::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::DType::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::dtype::DType -pub fn vortex_array::dtype::DType::hash<__H: core::hash::Hasher>(&self, state: &mut __H) - -impl core::marker::StructuralPartialEq for vortex_array::dtype::DType +pub fn vortex_array::dtype::DType::hash<__H: core::hash::Hasher>(&self, &mut __H) impl vortex_array::dtype::arrow::FromArrowType<&arrow_schema::field::Field> for vortex_array::dtype::DType -pub fn vortex_array::dtype::DType::from_arrow(field: &arrow_schema::field::Field) -> Self +pub fn vortex_array::dtype::DType::from_arrow(&arrow_schema::field::Field) -> Self impl vortex_array::dtype::arrow::FromArrowType<&arrow_schema::schema::Schema> for vortex_array::dtype::DType -pub fn vortex_array::dtype::DType::from_arrow(value: &arrow_schema::schema::Schema) -> Self +pub fn vortex_array::dtype::DType::from_arrow(&arrow_schema::schema::Schema) -> Self impl vortex_array::dtype::arrow::FromArrowType<(&arrow_schema::datatype::DataType, vortex_array::dtype::Nullability)> for vortex_array::dtype::DType -pub fn vortex_array::dtype::DType::from_arrow((data_type, nullability): (&arrow_schema::datatype::DataType, vortex_array::dtype::Nullability)) -> Self +pub fn vortex_array::dtype::DType::from_arrow((&arrow_schema::datatype::DataType, vortex_array::dtype::Nullability)) -> Self impl vortex_array::dtype::arrow::FromArrowType> for vortex_array::dtype::DType -pub fn vortex_array::dtype::DType::from_arrow(value: arrow_schema::schema::SchemaRef) -> Self +pub fn vortex_array::dtype::DType::from_arrow(arrow_schema::schema::SchemaRef) -> Self impl vortex_flatbuffers::FlatBufferRoot for vortex_array::dtype::DType @@ -9242,7 +9398,7 @@ impl vortex_flatbuffers::WriteFlatBuffer for vortex_array::dtype::DType pub type vortex_array::dtype::DType::Target<'a> = vortex_flatbuffers::dtype::DType<'a> -pub fn vortex_array::dtype::DType::write_flatbuffer<'fb>(&self, fbb: &mut flatbuffers::builder::FlatBufferBuilder<'fb>) -> vortex_error::VortexResult> +pub fn vortex_array::dtype::DType::write_flatbuffer<'fb>(&self, &mut flatbuffers::builder::FlatBufferBuilder<'fb>) -> vortex_error::VortexResult> #[repr(u8)] pub enum vortex_array::dtype::DecimalType @@ -9262,15 +9418,15 @@ impl vortex_array::dtype::DecimalType pub fn vortex_array::dtype::DecimalType::byte_width(&self) -> usize -pub fn vortex_array::dtype::DecimalType::is_compatible_decimal_value_type(self, dtype: vortex_array::dtype::DecimalDType) -> bool +pub fn vortex_array::dtype::DecimalType::is_compatible_decimal_value_type(self, vortex_array::dtype::DecimalDType) -> bool -pub fn vortex_array::dtype::DecimalType::smallest_decimal_value_type(decimal_dtype: &vortex_array::dtype::DecimalDType) -> vortex_array::dtype::DecimalType +pub fn vortex_array::dtype::DecimalType::smallest_decimal_value_type(&vortex_array::dtype::DecimalDType) -> vortex_array::dtype::DecimalType impl vortex_array::dtype::DecimalType -pub fn vortex_array::dtype::DecimalType::from_i32(value: i32) -> core::option::Option +pub fn vortex_array::dtype::DecimalType::from_i32(i32) -> core::option::Option -pub fn vortex_array::dtype::DecimalType::is_valid(value: i32) -> bool +pub fn vortex_array::dtype::DecimalType::is_valid(i32) -> bool impl core::clone::Clone for vortex_array::dtype::DecimalType @@ -9280,31 +9436,31 @@ impl core::cmp::Eq for vortex_array::dtype::DecimalType impl core::cmp::Ord for vortex_array::dtype::DecimalType -pub fn vortex_array::dtype::DecimalType::cmp(&self, other: &vortex_array::dtype::DecimalType) -> core::cmp::Ordering +pub fn vortex_array::dtype::DecimalType::cmp(&self, &vortex_array::dtype::DecimalType) -> core::cmp::Ordering impl core::cmp::PartialEq for vortex_array::dtype::DecimalType -pub fn vortex_array::dtype::DecimalType::eq(&self, other: &vortex_array::dtype::DecimalType) -> bool +pub fn vortex_array::dtype::DecimalType::eq(&self, &vortex_array::dtype::DecimalType) -> bool impl core::cmp::PartialOrd for vortex_array::dtype::DecimalType -pub fn vortex_array::dtype::DecimalType::partial_cmp(&self, other: &vortex_array::dtype::DecimalType) -> core::option::Option +pub fn vortex_array::dtype::DecimalType::partial_cmp(&self, &vortex_array::dtype::DecimalType) -> core::option::Option impl core::convert::From for i32 -pub fn i32::from(value: vortex_array::dtype::DecimalType) -> i32 +pub fn i32::from(vortex_array::dtype::DecimalType) -> i32 impl core::convert::TryFrom for vortex_array::dtype::DecimalType pub type vortex_array::dtype::DecimalType::Error = prost::error::UnknownEnumValue -pub fn vortex_array::dtype::DecimalType::try_from(value: i32) -> core::result::Result +pub fn vortex_array::dtype::DecimalType::try_from(i32) -> core::result::Result impl core::convert::TryFrom for vortex_array::dtype::DecimalType pub type vortex_array::dtype::DecimalType::Error = vortex_error::VortexError -pub fn vortex_array::dtype::DecimalType::try_from(value: vortex_array::dtype::PType) -> core::result::Result +pub fn vortex_array::dtype::DecimalType::try_from(vortex_array::dtype::PType) -> core::result::Result impl core::default::Default for vortex_array::dtype::DecimalType @@ -9312,15 +9468,15 @@ pub fn vortex_array::dtype::DecimalType::default() -> vortex_array::dtype::Decim impl core::fmt::Debug for vortex_array::dtype::DecimalType -pub fn vortex_array::dtype::DecimalType::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::DecimalType::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::dtype::DecimalType -pub fn vortex_array::dtype::DecimalType::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::DecimalType::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::dtype::DecimalType -pub fn vortex_array::dtype::DecimalType::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::dtype::DecimalType::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_array::dtype::DecimalType @@ -9346,39 +9502,39 @@ impl core::cmp::Eq for vortex_array::dtype::Field impl core::cmp::PartialEq for vortex_array::dtype::Field -pub fn vortex_array::dtype::Field::eq(&self, other: &vortex_array::dtype::Field) -> bool +pub fn vortex_array::dtype::Field::eq(&self, &vortex_array::dtype::Field) -> bool impl core::convert::From<&str> for vortex_array::dtype::Field -pub fn vortex_array::dtype::Field::from(value: &str) -> Self +pub fn vortex_array::dtype::Field::from(&str) -> Self impl core::convert::From> for vortex_array::dtype::Field -pub fn vortex_array::dtype::Field::from(value: alloc::sync::Arc) -> Self +pub fn vortex_array::dtype::Field::from(alloc::sync::Arc) -> Self impl core::convert::From for vortex_array::dtype::FieldPath -pub fn vortex_array::dtype::FieldPath::from(value: vortex_array::dtype::Field) -> Self +pub fn vortex_array::dtype::FieldPath::from(vortex_array::dtype::Field) -> Self impl core::convert::From for vortex_array::dtype::Field -pub fn vortex_array::dtype::Field::from(value: vortex_array::dtype::FieldName) -> Self +pub fn vortex_array::dtype::Field::from(vortex_array::dtype::FieldName) -> Self impl core::fmt::Debug for vortex_array::dtype::Field -pub fn vortex_array::dtype::Field::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::Field::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::dtype::Field -pub fn vortex_array::dtype::Field::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::Field::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::dtype::Field -pub fn vortex_array::dtype::Field::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::dtype::Field::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::iter::traits::collect::FromIterator for vortex_array::dtype::FieldPath -pub fn vortex_array::dtype::FieldPath::from_iter>(iter: T) -> Self +pub fn vortex_array::dtype::FieldPath::from_iter>(T) -> Self impl core::marker::StructuralPartialEq for vortex_array::dtype::Field @@ -9408,11 +9564,11 @@ impl core::cmp::Eq for vortex_array::dtype::FieldMask impl core::cmp::PartialEq for vortex_array::dtype::FieldMask -pub fn vortex_array::dtype::FieldMask::eq(&self, other: &vortex_array::dtype::FieldMask) -> bool +pub fn vortex_array::dtype::FieldMask::eq(&self, &vortex_array::dtype::FieldMask) -> bool impl core::fmt::Debug for vortex_array::dtype::FieldMask -pub fn vortex_array::dtype::FieldMask::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::FieldMask::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::StructuralPartialEq for vortex_array::dtype::FieldMask @@ -9434,23 +9590,23 @@ impl core::cmp::Eq for vortex_array::dtype::Nullability impl core::cmp::PartialEq for vortex_array::dtype::Nullability -pub fn vortex_array::dtype::Nullability::eq(&self, other: &vortex_array::dtype::Nullability) -> bool +pub fn vortex_array::dtype::Nullability::eq(&self, &vortex_array::dtype::Nullability) -> bool impl core::convert::From<&vortex_array::dtype::Nullability> for vortex_array::validity::Validity -pub fn vortex_array::validity::Validity::from(value: &vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::validity::Validity::from(&vortex_array::dtype::Nullability) -> Self impl core::convert::From for vortex_array::dtype::Nullability -pub fn vortex_array::dtype::Nullability::from(value: bool) -> Self +pub fn vortex_array::dtype::Nullability::from(bool) -> Self impl core::convert::From for bool -pub fn bool::from(value: vortex_array::dtype::Nullability) -> Self +pub fn bool::from(vortex_array::dtype::Nullability) -> Self impl core::convert::From for vortex_array::validity::Validity -pub fn vortex_array::validity::Validity::from(value: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::validity::Validity::from(vortex_array::dtype::Nullability) -> Self impl core::default::Default for vortex_array::dtype::Nullability @@ -9458,15 +9614,15 @@ pub fn vortex_array::dtype::Nullability::default() -> vortex_array::dtype::Nulla impl core::fmt::Debug for vortex_array::dtype::Nullability -pub fn vortex_array::dtype::Nullability::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::Nullability::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::dtype::Nullability -pub fn vortex_array::dtype::Nullability::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::Nullability::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::dtype::Nullability -pub fn vortex_array::dtype::Nullability::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::dtype::Nullability::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_array::dtype::Nullability @@ -9476,11 +9632,11 @@ impl core::ops::bit::BitOr for vortex_array::dtype::Nullability pub type vortex_array::dtype::Nullability::Output = vortex_array::dtype::Nullability -pub fn vortex_array::dtype::Nullability::bitor(self, rhs: Self) -> Self::Output +pub fn vortex_array::dtype::Nullability::bitor(self, Self) -> Self::Output impl core::ops::bit::BitOrAssign for vortex_array::dtype::Nullability -pub fn vortex_array::dtype::Nullability::bitor_assign(&mut self, rhs: Self) +pub fn vortex_array::dtype::Nullability::bitor_assign(&mut self, Self) #[repr(u8)] pub enum vortex_array::dtype::PType @@ -9520,15 +9676,15 @@ pub const fn vortex_array::dtype::PType::is_signed_int(self) -> bool pub const fn vortex_array::dtype::PType::is_unsigned_int(self) -> bool -pub const fn vortex_array::dtype::PType::max_signed_ptype(self, other: Self) -> Self +pub const fn vortex_array::dtype::PType::max_signed_ptype(self, Self) -> Self -pub const fn vortex_array::dtype::PType::max_unsigned_ptype(self, other: Self) -> Self +pub const fn vortex_array::dtype::PType::max_unsigned_ptype(self, Self) -> Self pub fn vortex_array::dtype::PType::max_value_as_u64(&self) -> u64 -pub const fn vortex_array::dtype::PType::min_signed_ptype_for_value(value: i64) -> Self +pub const fn vortex_array::dtype::PType::min_signed_ptype_for_value(i64) -> Self -pub const fn vortex_array::dtype::PType::min_unsigned_ptype_for_value(value: u64) -> Self +pub const fn vortex_array::dtype::PType::min_unsigned_ptype_for_value(u64) -> Self pub const fn vortex_array::dtype::PType::to_signed(self) -> Self @@ -9536,13 +9692,13 @@ pub const fn vortex_array::dtype::PType::to_unsigned(self) -> Self impl vortex_array::dtype::PType -pub fn vortex_array::dtype::PType::from_i32(value: i32) -> core::option::Option +pub fn vortex_array::dtype::PType::from_i32(i32) -> core::option::Option -pub fn vortex_array::dtype::PType::is_valid(value: i32) -> bool +pub fn vortex_array::dtype::PType::is_valid(i32) -> bool impl vortex_array::dtype::PType -pub fn vortex_array::dtype::PType::least_supertype(self, other: vortex_array::dtype::PType) -> core::option::Option +pub fn vortex_array::dtype::PType::least_supertype(self, vortex_array::dtype::PType) -> core::option::Option impl core::clone::Clone for vortex_array::dtype::PType @@ -9552,63 +9708,63 @@ impl core::cmp::Eq for vortex_array::dtype::PType impl core::cmp::PartialEq for vortex_array::dtype::PType -pub fn vortex_array::dtype::PType::eq(&self, other: &vortex_array::dtype::PType) -> bool +pub fn vortex_array::dtype::PType::eq(&self, &vortex_array::dtype::PType) -> bool impl core::cmp::PartialOrd for vortex_array::dtype::PType -pub fn vortex_array::dtype::PType::partial_cmp(&self, other: &vortex_array::dtype::PType) -> core::option::Option +pub fn vortex_array::dtype::PType::partial_cmp(&self, &vortex_array::dtype::PType) -> core::option::Option impl core::convert::From for &vortex_array::dtype::DType -pub fn &vortex_array::dtype::DType::from(item: vortex_array::dtype::PType) -> Self +pub fn &vortex_array::dtype::DType::from(vortex_array::dtype::PType) -> Self impl core::convert::From for i32 -pub fn i32::from(value: vortex_array::dtype::PType) -> i32 +pub fn i32::from(vortex_array::dtype::PType) -> i32 impl core::convert::From for vortex_array::dtype::DType -pub fn vortex_array::dtype::DType::from(item: vortex_array::dtype::PType) -> Self +pub fn vortex_array::dtype::DType::from(vortex_array::dtype::PType) -> Self impl core::convert::From for vortex_array::dtype::FieldDType -pub fn vortex_array::dtype::FieldDType::from(value: vortex_array::dtype::PType) -> Self +pub fn vortex_array::dtype::FieldDType::from(vortex_array::dtype::PType) -> Self impl core::convert::From for vortex_flatbuffers::dtype::PType -pub fn vortex_flatbuffers::dtype::PType::from(value: vortex_array::dtype::PType) -> Self +pub fn vortex_flatbuffers::dtype::PType::from(vortex_array::dtype::PType) -> Self impl core::convert::From for vortex_proto::dtype::PType -pub fn vortex_proto::dtype::PType::from(value: vortex_array::dtype::PType) -> Self +pub fn vortex_proto::dtype::PType::from(vortex_array::dtype::PType) -> Self impl core::convert::From for vortex_array::dtype::PType -pub fn vortex_array::dtype::PType::from(value: vortex_proto::dtype::PType) -> Self +pub fn vortex_array::dtype::PType::from(vortex_proto::dtype::PType) -> Self impl core::convert::TryFrom<&vortex_array::dtype::DType> for vortex_array::dtype::PType pub type vortex_array::dtype::PType::Error = vortex_error::VortexError -pub fn vortex_array::dtype::PType::try_from(value: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::dtype::PType::try_from(&vortex_array::dtype::DType) -> vortex_error::VortexResult impl core::convert::TryFrom for vortex_array::dtype::PType pub type vortex_array::dtype::PType::Error = prost::error::UnknownEnumValue -pub fn vortex_array::dtype::PType::try_from(value: i32) -> core::result::Result +pub fn vortex_array::dtype::PType::try_from(i32) -> core::result::Result impl core::convert::TryFrom for vortex_array::dtype::DecimalType pub type vortex_array::dtype::DecimalType::Error = vortex_error::VortexError -pub fn vortex_array::dtype::DecimalType::try_from(value: vortex_array::dtype::PType) -> core::result::Result +pub fn vortex_array::dtype::DecimalType::try_from(vortex_array::dtype::PType) -> core::result::Result impl core::convert::TryFrom for vortex_array::dtype::PType pub type vortex_array::dtype::PType::Error = vortex_error::VortexError -pub fn vortex_array::dtype::PType::try_from(value: vortex_flatbuffers::dtype::PType) -> core::result::Result +pub fn vortex_array::dtype::PType::try_from(vortex_flatbuffers::dtype::PType) -> core::result::Result impl core::default::Default for vortex_array::dtype::PType @@ -9616,15 +9772,15 @@ pub fn vortex_array::dtype::PType::default() -> vortex_array::dtype::PType impl core::fmt::Debug for vortex_array::dtype::PType -pub fn vortex_array::dtype::PType::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::PType::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::dtype::PType -pub fn vortex_array::dtype::PType::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::PType::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::dtype::PType -pub fn vortex_array::dtype::PType::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::dtype::PType::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_array::dtype::PType @@ -9632,13 +9788,13 @@ impl core::marker::StructuralPartialEq for vortex_array::dtype::PType impl vortex_array::dtype::arrow::TryFromArrowType<&arrow_schema::datatype::DataType> for vortex_array::dtype::PType -pub fn vortex_array::dtype::PType::try_from_arrow(value: &arrow_schema::datatype::DataType) -> vortex_error::VortexResult +pub fn vortex_array::dtype::PType::try_from_arrow(&arrow_schema::datatype::DataType) -> vortex_error::VortexResult pub struct vortex_array::dtype::DecimalDType impl vortex_array::dtype::DecimalDType -pub fn vortex_array::dtype::DecimalDType::new(precision: u8, scale: i8) -> Self +pub fn vortex_array::dtype::DecimalDType::new(u8, i8) -> Self pub fn vortex_array::dtype::DecimalDType::precision(&self) -> u8 @@ -9646,7 +9802,7 @@ pub fn vortex_array::dtype::DecimalDType::required_bit_width(&self) -> usize pub fn vortex_array::dtype::DecimalDType::scale(&self) -> i8 -pub fn vortex_array::dtype::DecimalDType::try_new(precision: u8, scale: i8) -> vortex_error::VortexResult +pub fn vortex_array::dtype::DecimalDType::try_new(u8, i8) -> vortex_error::VortexResult impl core::clone::Clone for vortex_array::dtype::DecimalDType @@ -9656,31 +9812,31 @@ impl core::cmp::Eq for vortex_array::dtype::DecimalDType impl core::cmp::PartialEq for vortex_array::dtype::DecimalDType -pub fn vortex_array::dtype::DecimalDType::eq(&self, other: &vortex_array::dtype::DecimalDType) -> bool +pub fn vortex_array::dtype::DecimalDType::eq(&self, &vortex_array::dtype::DecimalDType) -> bool impl core::convert::TryFrom<&vortex_array::dtype::DType> for vortex_array::dtype::DecimalDType pub type vortex_array::dtype::DecimalDType::Error = vortex_error::VortexError -pub fn vortex_array::dtype::DecimalDType::try_from(value: &vortex_array::dtype::DType) -> core::result::Result +pub fn vortex_array::dtype::DecimalDType::try_from(&vortex_array::dtype::DType) -> core::result::Result impl core::convert::TryFrom for vortex_array::dtype::DecimalDType pub type vortex_array::dtype::DecimalDType::Error = vortex_error::VortexError -pub fn vortex_array::dtype::DecimalDType::try_from(value: vortex_array::dtype::DType) -> core::result::Result +pub fn vortex_array::dtype::DecimalDType::try_from(vortex_array::dtype::DType) -> core::result::Result impl core::fmt::Debug for vortex_array::dtype::DecimalDType -pub fn vortex_array::dtype::DecimalDType::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::DecimalDType::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::dtype::DecimalDType -pub fn vortex_array::dtype::DecimalDType::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::DecimalDType::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::dtype::DecimalDType -pub fn vortex_array::dtype::DecimalDType::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::dtype::DecimalDType::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_array::dtype::DecimalDType @@ -9688,17 +9844,17 @@ impl core::marker::StructuralPartialEq for vortex_array::dtype::DecimalDType impl vortex_array::dtype::arrow::TryFromArrowType<&arrow_schema::datatype::DataType> for vortex_array::dtype::DecimalDType -pub fn vortex_array::dtype::DecimalDType::try_from_arrow(value: &arrow_schema::datatype::DataType) -> vortex_error::VortexResult +pub fn vortex_array::dtype::DecimalDType::try_from_arrow(&arrow_schema::datatype::DataType) -> vortex_error::VortexResult impl core::convert::From> for vortex_array::dtype::DecimalDType -pub fn vortex_array::dtype::DecimalDType::from(value: vortex_array::dtype::PrecisionScale) -> Self +pub fn vortex_array::dtype::DecimalDType::from(vortex_array::dtype::PrecisionScale) -> Self impl core::convert::TryFrom<&vortex_array::dtype::DecimalDType> for vortex_array::dtype::PrecisionScale pub type vortex_array::dtype::PrecisionScale::Error = vortex_error::VortexError -pub fn vortex_array::dtype::PrecisionScale::try_from(value: &vortex_array::dtype::DecimalDType) -> vortex_error::VortexResult +pub fn vortex_array::dtype::PrecisionScale::try_from(&vortex_array::dtype::DecimalDType) -> vortex_error::VortexResult pub struct vortex_array::dtype::FieldDType @@ -9714,23 +9870,23 @@ impl core::cmp::Eq for vortex_array::dtype::FieldDType impl core::cmp::PartialEq for vortex_array::dtype::FieldDType -pub fn vortex_array::dtype::FieldDType::eq(&self, other: &vortex_array::dtype::FieldDType) -> bool +pub fn vortex_array::dtype::FieldDType::eq(&self, &vortex_array::dtype::FieldDType) -> bool impl core::convert::From for vortex_array::dtype::FieldDType -pub fn vortex_array::dtype::FieldDType::from(value: vortex_array::dtype::DType) -> Self +pub fn vortex_array::dtype::FieldDType::from(vortex_array::dtype::DType) -> Self impl core::convert::From for vortex_array::dtype::FieldDType -pub fn vortex_array::dtype::FieldDType::from(value: vortex_array::dtype::PType) -> Self +pub fn vortex_array::dtype::FieldDType::from(vortex_array::dtype::PType) -> Self impl core::fmt::Debug for vortex_array::dtype::FieldDType -pub fn vortex_array::dtype::FieldDType::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::FieldDType::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::dtype::FieldDType -pub fn vortex_array::dtype::FieldDType::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::dtype::FieldDType::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_array::dtype::FieldDType @@ -9756,43 +9912,43 @@ impl core::cmp::Eq for vortex_array::dtype::FieldName impl core::cmp::Ord for vortex_array::dtype::FieldName -pub fn vortex_array::dtype::FieldName::cmp(&self, other: &vortex_array::dtype::FieldName) -> core::cmp::Ordering +pub fn vortex_array::dtype::FieldName::cmp(&self, &vortex_array::dtype::FieldName) -> core::cmp::Ordering impl core::cmp::PartialEq for vortex_array::dtype::FieldName -pub fn vortex_array::dtype::FieldName::eq(&self, other: &vortex_array::dtype::FieldName) -> bool +pub fn vortex_array::dtype::FieldName::eq(&self, &Self) -> bool impl core::cmp::PartialEq<&alloc::string::String> for vortex_array::dtype::FieldName -pub fn vortex_array::dtype::FieldName::eq(&self, other: &&alloc::string::String) -> bool +pub fn vortex_array::dtype::FieldName::eq(&self, &&alloc::string::String) -> bool impl core::cmp::PartialEq<&str> for &vortex_array::dtype::FieldName -pub fn &vortex_array::dtype::FieldName::eq(&self, other: &&str) -> bool +pub fn &vortex_array::dtype::FieldName::eq(&self, &&str) -> bool impl core::cmp::PartialEq<&str> for vortex_array::dtype::FieldName -pub fn vortex_array::dtype::FieldName::eq(&self, other: &&str) -> bool +pub fn vortex_array::dtype::FieldName::eq(&self, &&str) -> bool impl core::cmp::PartialEq<&vortex_array::dtype::FieldName> for vortex_array::dtype::FieldName -pub fn vortex_array::dtype::FieldName::eq(&self, other: &&vortex_array::dtype::FieldName) -> bool +pub fn vortex_array::dtype::FieldName::eq(&self, &&vortex_array::dtype::FieldName) -> bool impl core::cmp::PartialEq for vortex_array::dtype::FieldName -pub fn vortex_array::dtype::FieldName::eq(&self, other: &alloc::string::String) -> bool +pub fn vortex_array::dtype::FieldName::eq(&self, &alloc::string::String) -> bool impl core::cmp::PartialEq for &vortex_array::dtype::FieldName -pub fn &vortex_array::dtype::FieldName::eq(&self, other: &vortex_array::dtype::FieldName) -> bool +pub fn &vortex_array::dtype::FieldName::eq(&self, &vortex_array::dtype::FieldName) -> bool impl core::cmp::PartialEq for str -pub fn str::eq(&self, other: &vortex_array::dtype::FieldName) -> bool +pub fn str::eq(&self, &vortex_array::dtype::FieldName) -> bool impl core::cmp::PartialOrd for vortex_array::dtype::FieldName -pub fn vortex_array::dtype::FieldName::partial_cmp(&self, other: &vortex_array::dtype::FieldName) -> core::option::Option +pub fn vortex_array::dtype::FieldName::partial_cmp(&self, &vortex_array::dtype::FieldName) -> core::option::Option impl core::convert::AsRef for vortex_array::dtype::FieldName @@ -9800,41 +9956,39 @@ pub fn vortex_array::dtype::FieldName::as_ref(&self) -> &str impl core::convert::From<&str> for vortex_array::dtype::FieldName -pub fn vortex_array::dtype::FieldName::from(value: &str) -> Self +pub fn vortex_array::dtype::FieldName::from(&str) -> Self impl core::convert::From for vortex_array::dtype::FieldName -pub fn vortex_array::dtype::FieldName::from(value: alloc::string::String) -> Self +pub fn vortex_array::dtype::FieldName::from(alloc::string::String) -> Self impl core::convert::From> for vortex_array::dtype::FieldName -pub fn vortex_array::dtype::FieldName::from(value: alloc::sync::Arc) -> Self +pub fn vortex_array::dtype::FieldName::from(alloc::sync::Arc) -> Self impl core::convert::From for alloc::string::String -pub fn alloc::string::String::from(value: vortex_array::dtype::FieldName) -> Self +pub fn alloc::string::String::from(vortex_array::dtype::FieldName) -> Self impl core::convert::From for alloc::sync::Arc -pub fn alloc::sync::Arc::from(value: vortex_array::dtype::FieldName) -> Self +pub fn alloc::sync::Arc::from(vortex_array::dtype::FieldName) -> Self impl core::convert::From for vortex_array::dtype::Field -pub fn vortex_array::dtype::Field::from(value: vortex_array::dtype::FieldName) -> Self +pub fn vortex_array::dtype::Field::from(vortex_array::dtype::FieldName) -> Self impl core::fmt::Debug for vortex_array::dtype::FieldName -pub fn vortex_array::dtype::FieldName::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::FieldName::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::dtype::FieldName -pub fn vortex_array::dtype::FieldName::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::FieldName::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::dtype::FieldName -pub fn vortex_array::dtype::FieldName::hash<__H: core::hash::Hasher>(&self, state: &mut __H) - -impl core::marker::StructuralPartialEq for vortex_array::dtype::FieldName +pub fn vortex_array::dtype::FieldName::hash<__H: core::hash::Hasher>(&self, &mut __H) pub struct vortex_array::dtype::FieldNames(_) @@ -9842,9 +9996,9 @@ impl vortex_array::dtype::FieldNames pub fn vortex_array::dtype::FieldNames::empty() -> Self -pub fn vortex_array::dtype::FieldNames::find(&self, name: impl core::convert::AsRef) -> core::option::Option +pub fn vortex_array::dtype::FieldNames::find(&self, impl core::convert::AsRef) -> core::option::Option -pub fn vortex_array::dtype::FieldNames::get(&self, index: usize) -> core::option::Option<&vortex_array::dtype::FieldName> +pub fn vortex_array::dtype::FieldNames::get(&self, usize) -> core::option::Option<&vortex_array::dtype::FieldName> pub fn vortex_array::dtype::FieldNames::is_empty(&self) -> bool @@ -9860,27 +10014,27 @@ impl core::cmp::Eq for vortex_array::dtype::FieldNames impl core::cmp::PartialEq for vortex_array::dtype::FieldNames -pub fn vortex_array::dtype::FieldNames::eq(&self, other: &vortex_array::dtype::FieldNames) -> bool +pub fn vortex_array::dtype::FieldNames::eq(&self, &Self) -> bool impl core::cmp::PartialEq<&[&str]> for &vortex_array::dtype::FieldNames -pub fn &vortex_array::dtype::FieldNames::eq(&self, other: &&[&str]) -> bool +pub fn &vortex_array::dtype::FieldNames::eq(&self, &&[&str]) -> bool impl core::cmp::PartialEq<&[&str]> for vortex_array::dtype::FieldNames -pub fn vortex_array::dtype::FieldNames::eq(&self, other: &&[&str]) -> bool +pub fn vortex_array::dtype::FieldNames::eq(&self, &&[&str]) -> bool impl core::cmp::PartialEq<&[vortex_array::dtype::FieldName]> for &vortex_array::dtype::FieldNames -pub fn &vortex_array::dtype::FieldNames::eq(&self, other: &&[vortex_array::dtype::FieldName]) -> bool +pub fn &vortex_array::dtype::FieldNames::eq(&self, &&[vortex_array::dtype::FieldName]) -> bool impl core::cmp::PartialEq<&[vortex_array::dtype::FieldName]> for vortex_array::dtype::FieldNames -pub fn vortex_array::dtype::FieldNames::eq(&self, other: &&[vortex_array::dtype::FieldName]) -> bool +pub fn vortex_array::dtype::FieldNames::eq(&self, &&[vortex_array::dtype::FieldName]) -> bool impl core::cmp::PartialEq<&vortex_array::dtype::FieldNames> for vortex_array::dtype::FieldNames -pub fn vortex_array::dtype::FieldNames::eq(&self, other: &&vortex_array::dtype::FieldNames) -> bool +pub fn vortex_array::dtype::FieldNames::eq(&self, &&vortex_array::dtype::FieldNames) -> bool impl core::convert::AsRef<[vortex_array::dtype::FieldName]> for vortex_array::dtype::FieldNames @@ -9888,23 +10042,23 @@ pub fn vortex_array::dtype::FieldNames::as_ref(&self) -> &[vortex_array::dtype:: impl core::convert::From<&[&'static str]> for vortex_array::dtype::FieldNames -pub fn vortex_array::dtype::FieldNames::from(value: &[&'static str]) -> Self +pub fn vortex_array::dtype::FieldNames::from(&[&'static str]) -> Self impl core::convert::From<&[vortex_array::dtype::FieldName]> for vortex_array::dtype::FieldNames -pub fn vortex_array::dtype::FieldNames::from(value: &[vortex_array::dtype::FieldName]) -> Self +pub fn vortex_array::dtype::FieldNames::from(&[vortex_array::dtype::FieldName]) -> Self impl core::convert::From> for vortex_array::dtype::FieldNames -pub fn vortex_array::dtype::FieldNames::from(value: alloc::vec::Vec<&str>) -> Self +pub fn vortex_array::dtype::FieldNames::from(alloc::vec::Vec<&str>) -> Self impl core::convert::From>> for vortex_array::dtype::FieldNames -pub fn vortex_array::dtype::FieldNames::from(value: alloc::vec::Vec>) -> Self +pub fn vortex_array::dtype::FieldNames::from(alloc::vec::Vec>) -> Self impl core::convert::From> for vortex_array::dtype::FieldNames -pub fn vortex_array::dtype::FieldNames::from(value: alloc::vec::Vec) -> Self +pub fn vortex_array::dtype::FieldNames::from(alloc::vec::Vec) -> Self impl core::default::Default for vortex_array::dtype::FieldNames @@ -9912,15 +10066,15 @@ pub fn vortex_array::dtype::FieldNames::default() -> vortex_array::dtype::FieldN impl core::fmt::Debug for vortex_array::dtype::FieldNames -pub fn vortex_array::dtype::FieldNames::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::FieldNames::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::dtype::FieldNames -pub fn vortex_array::dtype::FieldNames::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::FieldNames::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::dtype::FieldNames -pub fn vortex_array::dtype::FieldNames::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::dtype::FieldNames::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::iter::traits::collect::IntoIterator for vortex_array::dtype::FieldNames @@ -9930,37 +10084,35 @@ pub type vortex_array::dtype::FieldNames::Item = vortex_array::dtype::FieldName pub fn vortex_array::dtype::FieldNames::into_iter(self) -> Self::IntoIter -impl core::marker::StructuralPartialEq for vortex_array::dtype::FieldNames - impl core::ops::index::Index for vortex_array::dtype::FieldNames pub type vortex_array::dtype::FieldNames::Output = vortex_array::dtype::FieldName -pub fn vortex_array::dtype::FieldNames::index(&self, index: usize) -> &Self::Output +pub fn vortex_array::dtype::FieldNames::index(&self, usize) -> &Self::Output impl> core::iter::traits::collect::FromIterator for vortex_array::dtype::FieldNames -pub fn vortex_array::dtype::FieldNames::from_iter>(iter: T) -> Self +pub fn vortex_array::dtype::FieldNames::from_iter>(T) -> Self impl core::cmp::PartialEq<[&str; N]> for &vortex_array::dtype::FieldNames -pub fn &vortex_array::dtype::FieldNames::eq(&self, other: &[&str; N]) -> bool +pub fn &vortex_array::dtype::FieldNames::eq(&self, &[&str; N]) -> bool impl core::cmp::PartialEq<[&str; N]> for vortex_array::dtype::FieldNames -pub fn vortex_array::dtype::FieldNames::eq(&self, other: &[&str; N]) -> bool +pub fn vortex_array::dtype::FieldNames::eq(&self, &[&str; N]) -> bool impl core::convert::From<&[&str; N]> for vortex_array::dtype::FieldNames -pub fn vortex_array::dtype::FieldNames::from(value: &[&str; N]) -> Self +pub fn vortex_array::dtype::FieldNames::from(&[&str; N]) -> Self impl core::convert::From<[&str; N]> for vortex_array::dtype::FieldNames -pub fn vortex_array::dtype::FieldNames::from(value: [&str; N]) -> Self +pub fn vortex_array::dtype::FieldNames::from([&str; N]) -> Self impl core::convert::From<[vortex_array::dtype::FieldName; N]> for vortex_array::dtype::FieldNames -pub fn vortex_array::dtype::FieldNames::from(value: [vortex_array::dtype::FieldName; N]) -> Self +pub fn vortex_array::dtype::FieldNames::from([vortex_array::dtype::FieldName; N]) -> Self pub struct vortex_array::dtype::FieldNamesIntoIter @@ -9990,23 +10142,23 @@ pub struct vortex_array::dtype::FieldPath(_) impl vortex_array::dtype::FieldPath -pub fn vortex_array::dtype::FieldPath::exists_in(&self, dtype: vortex_array::dtype::DType) -> bool +pub fn vortex_array::dtype::FieldPath::exists_in(&self, vortex_array::dtype::DType) -> bool -pub fn vortex_array::dtype::FieldPath::from_name>(name: F) -> Self +pub fn vortex_array::dtype::FieldPath::from_name>(F) -> Self pub fn vortex_array::dtype::FieldPath::is_root(&self) -> bool -pub fn vortex_array::dtype::FieldPath::overlap(&self, other: &vortex_array::dtype::FieldPath) -> bool +pub fn vortex_array::dtype::FieldPath::overlap(&self, &vortex_array::dtype::FieldPath) -> bool pub fn vortex_array::dtype::FieldPath::parts(&self) -> &[vortex_array::dtype::Field] -pub fn vortex_array::dtype::FieldPath::push>(self, field: F) -> Self +pub fn vortex_array::dtype::FieldPath::push>(self, F) -> Self -pub fn vortex_array::dtype::FieldPath::resolve(&self, dtype: vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::dtype::FieldPath::resolve(&self, vortex_array::dtype::DType) -> core::option::Option pub fn vortex_array::dtype::FieldPath::root() -> Self -pub fn vortex_array::dtype::FieldPath::starts_with_field(&self, field: &vortex_array::dtype::Field) -> bool +pub fn vortex_array::dtype::FieldPath::starts_with_field(&self, &vortex_array::dtype::Field) -> bool pub fn vortex_array::dtype::FieldPath::step_into(self) -> core::option::Option @@ -10018,21 +10170,21 @@ impl core::cmp::Eq for vortex_array::dtype::FieldPath impl core::cmp::PartialEq for vortex_array::dtype::FieldPath -pub fn vortex_array::dtype::FieldPath::eq(&self, other: &vortex_array::dtype::FieldPath) -> bool +pub fn vortex_array::dtype::FieldPath::eq(&self, &vortex_array::dtype::FieldPath) -> bool impl core::convert::From> for vortex_array::dtype::FieldPath -pub fn vortex_array::dtype::FieldPath::from(value: alloc::vec::Vec) -> Self +pub fn vortex_array::dtype::FieldPath::from(alloc::vec::Vec) -> Self impl core::convert::From for vortex_array::dtype::FieldPath -pub fn vortex_array::dtype::FieldPath::from(value: vortex_array::dtype::Field) -> Self +pub fn vortex_array::dtype::FieldPath::from(vortex_array::dtype::Field) -> Self impl core::convert::TryFrom<&vortex_proto::dtype::FieldPath> for vortex_array::dtype::FieldPath pub type vortex_array::dtype::FieldPath::Error = vortex_error::VortexError -pub fn vortex_array::dtype::FieldPath::try_from(value: &vortex_proto::dtype::FieldPath) -> core::result::Result +pub fn vortex_array::dtype::FieldPath::try_from(&vortex_proto::dtype::FieldPath) -> core::result::Result impl core::default::Default for vortex_array::dtype::FieldPath @@ -10040,23 +10192,23 @@ pub fn vortex_array::dtype::FieldPath::default() -> vortex_array::dtype::FieldPa impl core::fmt::Debug for vortex_array::dtype::FieldPath -pub fn vortex_array::dtype::FieldPath::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::FieldPath::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::dtype::FieldPath -pub fn vortex_array::dtype::FieldPath::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::FieldPath::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::dtype::FieldPath -pub fn vortex_array::dtype::FieldPath::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::dtype::FieldPath::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::iter::traits::collect::FromIterator for vortex_array::dtype::FieldPath -pub fn vortex_array::dtype::FieldPath::from_iter>(iter: T) -> Self +pub fn vortex_array::dtype::FieldPath::from_iter>(T) -> Self impl core::iter::traits::collect::FromIterator for vortex_array::dtype::FieldPathSet -pub fn vortex_array::dtype::FieldPathSet::from_iter>(iter: T) -> Self +pub fn vortex_array::dtype::FieldPathSet::from_iter>(T) -> Self impl core::marker::StructuralPartialEq for vortex_array::dtype::FieldPath @@ -10064,7 +10216,7 @@ pub struct vortex_array::dtype::FieldPathSet impl vortex_array::dtype::FieldPathSet -pub fn vortex_array::dtype::FieldPathSet::contains(&self, path: &vortex_array::dtype::FieldPath) -> bool +pub fn vortex_array::dtype::FieldPathSet::contains(&self, &vortex_array::dtype::FieldPath) -> bool impl core::clone::Clone for vortex_array::dtype::FieldPathSet @@ -10076,27 +10228,27 @@ pub fn vortex_array::dtype::FieldPathSet::default() -> vortex_array::dtype::Fiel impl core::fmt::Debug for vortex_array::dtype::FieldPathSet -pub fn vortex_array::dtype::FieldPathSet::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::FieldPathSet::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::iter::traits::collect::FromIterator for vortex_array::dtype::FieldPathSet -pub fn vortex_array::dtype::FieldPathSet::from_iter>(iter: T) -> Self +pub fn vortex_array::dtype::FieldPathSet::from_iter>(T) -> Self pub struct vortex_array::dtype::PrecisionScale impl vortex_array::dtype::PrecisionScale -pub fn vortex_array::dtype::PrecisionScale::is_valid(&self, value: D) -> bool +pub fn vortex_array::dtype::PrecisionScale::is_valid(&self, D) -> bool -pub fn vortex_array::dtype::PrecisionScale::new(precision: u8, scale: i8) -> Self +pub fn vortex_array::dtype::PrecisionScale::new(u8, i8) -> Self -pub unsafe fn vortex_array::dtype::PrecisionScale::new_unchecked(precision: u8, scale: i8) -> Self +pub unsafe fn vortex_array::dtype::PrecisionScale::new_unchecked(u8, i8) -> Self pub fn vortex_array::dtype::PrecisionScale::precision(&self) -> u8 pub fn vortex_array::dtype::PrecisionScale::scale(&self) -> i8 -pub fn vortex_array::dtype::PrecisionScale::try_new(precision: u8, scale: i8) -> vortex_error::VortexResult +pub fn vortex_array::dtype::PrecisionScale::try_new(u8, i8) -> vortex_error::VortexResult impl core::clone::Clone for vortex_array::dtype::PrecisionScale @@ -10106,31 +10258,31 @@ impl core::cmp::Eq for vortex_array::dtype::PrecisionScale impl core::cmp::PartialEq for vortex_array::dtype::PrecisionScale -pub fn vortex_array::dtype::PrecisionScale::eq(&self, other: &vortex_array::dtype::PrecisionScale) -> bool +pub fn vortex_array::dtype::PrecisionScale::eq(&self, &vortex_array::dtype::PrecisionScale) -> bool impl core::fmt::Debug for vortex_array::dtype::PrecisionScale -pub fn vortex_array::dtype::PrecisionScale::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::PrecisionScale::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::dtype::PrecisionScale -pub fn vortex_array::dtype::PrecisionScale::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::dtype::PrecisionScale::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_array::dtype::PrecisionScale impl core::convert::From> for vortex_array::dtype::DecimalDType -pub fn vortex_array::dtype::DecimalDType::from(value: vortex_array::dtype::PrecisionScale) -> Self +pub fn vortex_array::dtype::DecimalDType::from(vortex_array::dtype::PrecisionScale) -> Self impl core::convert::TryFrom<&vortex_array::dtype::DecimalDType> for vortex_array::dtype::PrecisionScale pub type vortex_array::dtype::PrecisionScale::Error = vortex_error::VortexError -pub fn vortex_array::dtype::PrecisionScale::try_from(value: &vortex_array::dtype::DecimalDType) -> vortex_error::VortexResult +pub fn vortex_array::dtype::PrecisionScale::try_from(&vortex_array::dtype::DecimalDType) -> vortex_error::VortexResult impl core::fmt::Display for vortex_array::dtype::PrecisionScale -pub fn vortex_array::dtype::PrecisionScale::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::PrecisionScale::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::StructuralPartialEq for vortex_array::dtype::PrecisionScale @@ -10138,31 +10290,31 @@ pub struct vortex_array::dtype::StructFields(_) impl vortex_array::dtype::StructFields -pub fn vortex_array::dtype::StructFields::disjoint_merge(&self, other: &Self) -> vortex_error::VortexResult +pub fn vortex_array::dtype::StructFields::disjoint_merge(&self, &Self) -> vortex_error::VortexResult pub fn vortex_array::dtype::StructFields::empty() -> Self -pub fn vortex_array::dtype::StructFields::field(&self, name: impl core::convert::AsRef) -> core::option::Option +pub fn vortex_array::dtype::StructFields::field(&self, impl core::convert::AsRef) -> core::option::Option -pub fn vortex_array::dtype::StructFields::field_by_index(&self, index: usize) -> core::option::Option +pub fn vortex_array::dtype::StructFields::field_by_index(&self, usize) -> core::option::Option -pub fn vortex_array::dtype::StructFields::field_name(&self, index: usize) -> core::option::Option<&vortex_array::dtype::FieldName> +pub fn vortex_array::dtype::StructFields::field_name(&self, usize) -> core::option::Option<&vortex_array::dtype::FieldName> pub fn vortex_array::dtype::StructFields::fields(&self) -> impl core::iter::traits::exact_size::ExactSizeIterator + '_ -pub fn vortex_array::dtype::StructFields::find(&self, name: impl core::convert::AsRef) -> core::option::Option +pub fn vortex_array::dtype::StructFields::find(&self, impl core::convert::AsRef) -> core::option::Option -pub fn vortex_array::dtype::StructFields::from_fields(names: vortex_array::dtype::FieldNames, dtypes: alloc::vec::Vec) -> Self +pub fn vortex_array::dtype::StructFields::from_fields(vortex_array::dtype::FieldNames, alloc::vec::Vec) -> Self pub fn vortex_array::dtype::StructFields::names(&self) -> &vortex_array::dtype::FieldNames -pub fn vortex_array::dtype::StructFields::new(names: vortex_array::dtype::FieldNames, dtypes: alloc::vec::Vec) -> Self +pub fn vortex_array::dtype::StructFields::new(vortex_array::dtype::FieldNames, alloc::vec::Vec) -> Self pub fn vortex_array::dtype::StructFields::nfields(&self) -> usize -pub fn vortex_array::dtype::StructFields::project(&self, projection: &[vortex_array::dtype::FieldName]) -> vortex_error::VortexResult +pub fn vortex_array::dtype::StructFields::project(&self, &[vortex_array::dtype::FieldName]) -> vortex_error::VortexResult -pub fn vortex_array::dtype::StructFields::without_field(&self, index: usize) -> vortex_error::VortexResult +pub fn vortex_array::dtype::StructFields::without_field(&self, usize) -> vortex_error::VortexResult impl core::clone::Clone for vortex_array::dtype::StructFields @@ -10172,7 +10324,7 @@ impl core::cmp::Eq for vortex_array::dtype::StructFields impl core::cmp::PartialEq for vortex_array::dtype::StructFields -pub fn vortex_array::dtype::StructFields::eq(&self, other: &vortex_array::dtype::StructFields) -> bool +pub fn vortex_array::dtype::StructFields::eq(&self, &Self) -> bool impl core::default::Default for vortex_array::dtype::StructFields @@ -10180,25 +10332,23 @@ pub fn vortex_array::dtype::StructFields::default() -> Self impl core::fmt::Debug for vortex_array::dtype::StructFields -pub fn vortex_array::dtype::StructFields::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::StructFields::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::dtype::StructFields -pub fn vortex_array::dtype::StructFields::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::StructFields::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::dtype::StructFields -pub fn vortex_array::dtype::StructFields::hash<__H: core::hash::Hasher>(&self, state: &mut __H) - -impl core::marker::StructuralPartialEq for vortex_array::dtype::StructFields +pub fn vortex_array::dtype::StructFields::hash<__H: core::hash::Hasher>(&self, &mut __H) impl vortex_array::dtype::arrow::FromArrowType<&arrow_schema::fields::Fields> for vortex_array::dtype::StructFields -pub fn vortex_array::dtype::StructFields::from_arrow(value: &arrow_schema::fields::Fields) -> Self +pub fn vortex_array::dtype::StructFields::from_arrow(&arrow_schema::fields::Fields) -> Self impl core::iter::traits::collect::FromIterator<(T, V)> for vortex_array::dtype::StructFields where T: core::convert::Into, V: core::convert::Into -pub fn vortex_array::dtype::StructFields::from_iter>(iter: I) -> Self +pub fn vortex_array::dtype::StructFields::from_iter>(I) -> Self #[repr(transparent)] pub struct vortex_array::dtype::i256(_) @@ -10212,13 +10362,13 @@ pub const vortex_array::dtype::i256::ONE: Self pub const vortex_array::dtype::i256::ZERO: Self -pub fn vortex_array::dtype::i256::checked_pow(&self, exp: u32) -> core::option::Option +pub fn vortex_array::dtype::i256::checked_pow(&self, u32) -> core::option::Option -pub const fn vortex_array::dtype::i256::from_i128(i: i128) -> Self +pub const fn vortex_array::dtype::i256::from_i128(i128) -> Self -pub const fn vortex_array::dtype::i256::from_le_bytes(bytes: [u8; 32]) -> Self +pub const fn vortex_array::dtype::i256::from_le_bytes([u8; 32]) -> Self -pub const fn vortex_array::dtype::i256::from_parts(lower: u128, upper: i128) -> Self +pub const fn vortex_array::dtype::i256::from_parts(u128, i128) -> Self pub const fn vortex_array::dtype::i256::into_parts(self) -> (u128, i128) @@ -10230,9 +10380,9 @@ pub const fn vortex_array::dtype::i256::to_le_bytes(&self) -> [u8; 32] pub const fn vortex_array::dtype::i256::to_parts(&self) -> (u128, i128) -pub fn vortex_array::dtype::i256::wrapping_add(&self, other: Self) -> Self +pub fn vortex_array::dtype::i256::wrapping_add(&self, Self) -> Self -pub fn vortex_array::dtype::i256::wrapping_pow(&self, exp: u32) -> Self +pub fn vortex_array::dtype::i256::wrapping_pow(&self, u32) -> Self impl core::clone::Clone for vortex_array::dtype::i256 @@ -10242,33 +10392,33 @@ impl core::cmp::Eq for vortex_array::dtype::i256 impl core::cmp::Ord for vortex_array::dtype::i256 -pub fn vortex_array::dtype::i256::cmp(&self, other: &vortex_array::dtype::i256) -> core::cmp::Ordering +pub fn vortex_array::dtype::i256::cmp(&self, &vortex_array::dtype::i256) -> core::cmp::Ordering impl core::cmp::PartialEq for vortex_array::dtype::i256 -pub fn vortex_array::dtype::i256::eq(&self, other: &vortex_array::dtype::i256) -> bool +pub fn vortex_array::dtype::i256::eq(&self, &vortex_array::dtype::i256) -> bool impl core::cmp::PartialOrd for vortex_array::dtype::i256 -pub fn vortex_array::dtype::i256::partial_cmp(&self, other: &vortex_array::dtype::i256) -> core::option::Option +pub fn vortex_array::dtype::i256::partial_cmp(&self, &vortex_array::dtype::i256) -> core::option::Option impl core::convert::From for vortex_array::dtype::i256 -pub fn vortex_array::dtype::i256::from(i: arrow_buffer::bigint::i256) -> Self +pub fn vortex_array::dtype::i256::from(arrow_buffer::bigint::i256) -> Self impl core::convert::From for arrow_buffer::bigint::i256 -pub fn arrow_buffer::bigint::i256::from(i: vortex_array::dtype::i256) -> Self +pub fn arrow_buffer::bigint::i256::from(vortex_array::dtype::i256) -> Self impl core::convert::From for vortex_array::scalar::DecimalValue -pub fn vortex_array::scalar::DecimalValue::from(value: vortex_array::dtype::i256) -> Self +pub fn vortex_array::scalar::DecimalValue::from(vortex_array::dtype::i256) -> Self impl core::convert::TryFrom> for vortex_array::dtype::i256 pub type vortex_array::dtype::i256::Error = vortex_error::VortexError -pub fn vortex_array::dtype::i256::try_from(value: vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result +pub fn vortex_array::dtype::i256::try_from(vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result impl core::default::Default for vortex_array::dtype::i256 @@ -10276,15 +10426,15 @@ pub fn vortex_array::dtype::i256::default() -> vortex_array::dtype::i256 impl core::fmt::Debug for vortex_array::dtype::i256 -pub fn vortex_array::dtype::i256::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::i256::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::dtype::i256 -pub fn vortex_array::dtype::i256::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::dtype::i256::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::dtype::i256 -pub fn vortex_array::dtype::i256::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::dtype::i256::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_array::dtype::i256 @@ -10294,23 +10444,23 @@ impl core::ops::arith::Add for vortex_array::dtype::i256 pub type vortex_array::dtype::i256::Output = vortex_array::dtype::i256 -pub fn vortex_array::dtype::i256::add(self, rhs: Self) -> Self::Output +pub fn vortex_array::dtype::i256::add(self, Self) -> Self::Output impl core::ops::arith::AddAssign for vortex_array::dtype::i256 -pub fn vortex_array::dtype::i256::add_assign(&mut self, rhs: Self) +pub fn vortex_array::dtype::i256::add_assign(&mut self, Self) impl core::ops::arith::Div for vortex_array::dtype::i256 pub type vortex_array::dtype::i256::Output = vortex_array::dtype::i256 -pub fn vortex_array::dtype::i256::div(self, rhs: Self) -> Self::Output +pub fn vortex_array::dtype::i256::div(self, Self) -> Self::Output impl core::ops::arith::Mul for vortex_array::dtype::i256 pub type vortex_array::dtype::i256::Output = vortex_array::dtype::i256 -pub fn vortex_array::dtype::i256::mul(self, rhs: Self) -> Self::Output +pub fn vortex_array::dtype::i256::mul(self, Self) -> Self::Output impl core::ops::arith::Neg for vortex_array::dtype::i256 @@ -10322,31 +10472,31 @@ impl core::ops::arith::Rem for vortex_array::dtype::i256 pub type vortex_array::dtype::i256::Output = vortex_array::dtype::i256 -pub fn vortex_array::dtype::i256::rem(self, rhs: Self) -> Self::Output +pub fn vortex_array::dtype::i256::rem(self, Self) -> Self::Output impl core::ops::arith::Sub for vortex_array::dtype::i256 pub type vortex_array::dtype::i256::Output = vortex_array::dtype::i256 -pub fn vortex_array::dtype::i256::sub(self, rhs: Self) -> Self::Output +pub fn vortex_array::dtype::i256::sub(self, Self) -> Self::Output impl core::ops::bit::BitOr for vortex_array::dtype::i256 pub type vortex_array::dtype::i256::Output = vortex_array::dtype::i256 -pub fn vortex_array::dtype::i256::bitor(self, rhs: Self) -> Self::Output +pub fn vortex_array::dtype::i256::bitor(self, Self) -> Self::Output impl core::ops::bit::Shl for vortex_array::dtype::i256 pub type vortex_array::dtype::i256::Output = vortex_array::dtype::i256 -pub fn vortex_array::dtype::i256::shl(self, rhs: usize) -> Self::Output +pub fn vortex_array::dtype::i256::shl(self, usize) -> Self::Output impl core::ops::bit::Shr for vortex_array::dtype::i256 pub type vortex_array::dtype::i256::Output = vortex_array::dtype::i256 -pub fn vortex_array::dtype::i256::shr(self, rhs: Self) -> Self::Output +pub fn vortex_array::dtype::i256::shr(self, Self) -> Self::Output impl num_traits::cast::AsPrimitive for vortex_array::dtype::i256 @@ -10422,31 +10572,31 @@ pub fn vortex_array::dtype::i256::zero() -> Self impl num_traits::ops::checked::CheckedAdd for vortex_array::dtype::i256 -pub fn vortex_array::dtype::i256::checked_add(&self, v: &Self) -> core::option::Option +pub fn vortex_array::dtype::i256::checked_add(&self, &Self) -> core::option::Option impl num_traits::ops::checked::CheckedDiv for vortex_array::dtype::i256 -pub fn vortex_array::dtype::i256::checked_div(&self, v: &Self) -> core::option::Option +pub fn vortex_array::dtype::i256::checked_div(&self, &Self) -> core::option::Option impl num_traits::ops::checked::CheckedMul for vortex_array::dtype::i256 -pub fn vortex_array::dtype::i256::checked_mul(&self, v: &Self) -> core::option::Option +pub fn vortex_array::dtype::i256::checked_mul(&self, &Self) -> core::option::Option impl num_traits::ops::checked::CheckedSub for vortex_array::dtype::i256 -pub fn vortex_array::dtype::i256::checked_sub(&self, v: &Self) -> core::option::Option +pub fn vortex_array::dtype::i256::checked_sub(&self, &Self) -> core::option::Option impl num_traits::ops::wrapping::WrappingAdd for vortex_array::dtype::i256 -pub fn vortex_array::dtype::i256::wrapping_add(&self, v: &Self) -> Self +pub fn vortex_array::dtype::i256::wrapping_add(&self, &Self) -> Self impl num_traits::ops::wrapping::WrappingSub for vortex_array::dtype::i256 -pub fn vortex_array::dtype::i256::wrapping_sub(&self, v: &Self) -> Self +pub fn vortex_array::dtype::i256::wrapping_sub(&self, &Self) -> Self impl vortex_array::dtype::BigCast for vortex_array::dtype::i256 -pub fn vortex_array::dtype::i256::from(n: T) -> core::option::Option +pub fn vortex_array::dtype::i256::from(T) -> core::option::Option impl vortex_array::dtype::NativeDecimalType for vortex_array::dtype::i256 @@ -10460,9 +10610,9 @@ pub const vortex_array::dtype::i256::MAX_SCALE: i8 pub const vortex_array::dtype::i256::MIN_BY_PRECISION: &'static [Self] -pub fn vortex_array::dtype::i256::downcast(visitor: V) -> ::Output +pub fn vortex_array::dtype::i256::downcast(V) -> ::Output -pub fn vortex_array::dtype::i256::upcast(input: ::Input) -> V +pub fn vortex_array::dtype::i256::upcast(::Input) -> V impl vortex_array::dtype::ToI256 for vortex_array::dtype::i256 @@ -10474,51 +10624,51 @@ pub const vortex_array::dtype::MAX_SCALE: i8 pub trait vortex_array::dtype::BigCast: core::marker::Sized + num_traits::cast::ToPrimitive + vortex_array::dtype::ToI256 -pub fn vortex_array::dtype::BigCast::from(n: T) -> core::option::Option +pub fn vortex_array::dtype::BigCast::from(T) -> core::option::Option impl vortex_array::dtype::BigCast for i128 -pub fn i128::from(n: T) -> core::option::Option +pub fn i128::from(T) -> core::option::Option impl vortex_array::dtype::BigCast for i16 -pub fn i16::from(n: T) -> core::option::Option +pub fn i16::from(T) -> core::option::Option impl vortex_array::dtype::BigCast for i32 -pub fn i32::from(n: T) -> core::option::Option +pub fn i32::from(T) -> core::option::Option impl vortex_array::dtype::BigCast for i64 -pub fn i64::from(n: T) -> core::option::Option +pub fn i64::from(T) -> core::option::Option impl vortex_array::dtype::BigCast for i8 -pub fn i8::from(n: T) -> core::option::Option +pub fn i8::from(T) -> core::option::Option impl vortex_array::dtype::BigCast for u128 -pub fn u128::from(n: T) -> core::option::Option +pub fn u128::from(T) -> core::option::Option impl vortex_array::dtype::BigCast for u16 -pub fn u16::from(n: T) -> core::option::Option +pub fn u16::from(T) -> core::option::Option impl vortex_array::dtype::BigCast for u32 -pub fn u32::from(n: T) -> core::option::Option +pub fn u32::from(T) -> core::option::Option impl vortex_array::dtype::BigCast for u64 -pub fn u64::from(n: T) -> core::option::Option +pub fn u64::from(T) -> core::option::Option impl vortex_array::dtype::BigCast for u8 -pub fn u8::from(n: T) -> core::option::Option +pub fn u8::from(T) -> core::option::Option impl vortex_array::dtype::BigCast for vortex_array::dtype::i256 -pub fn vortex_array::dtype::i256::from(n: T) -> core::option::Option +pub fn vortex_array::dtype::i256::from(T) -> core::option::Option pub trait vortex_array::dtype::DecimalTypeDowncast @@ -10540,69 +10690,69 @@ pub trait vortex_array::dtype::DecimalTypeUpcast pub type vortex_array::dtype::DecimalTypeUpcast::Input -pub fn vortex_array::dtype::DecimalTypeUpcast::from_i128(input: Self::Input) -> Self +pub fn vortex_array::dtype::DecimalTypeUpcast::from_i128(Self::Input) -> Self -pub fn vortex_array::dtype::DecimalTypeUpcast::from_i16(input: Self::Input) -> Self +pub fn vortex_array::dtype::DecimalTypeUpcast::from_i16(Self::Input) -> Self -pub fn vortex_array::dtype::DecimalTypeUpcast::from_i256(input: Self::Input) -> Self +pub fn vortex_array::dtype::DecimalTypeUpcast::from_i256(Self::Input) -> Self -pub fn vortex_array::dtype::DecimalTypeUpcast::from_i32(input: Self::Input) -> Self +pub fn vortex_array::dtype::DecimalTypeUpcast::from_i32(Self::Input) -> Self -pub fn vortex_array::dtype::DecimalTypeUpcast::from_i64(input: Self::Input) -> Self +pub fn vortex_array::dtype::DecimalTypeUpcast::from_i64(Self::Input) -> Self -pub fn vortex_array::dtype::DecimalTypeUpcast::from_i8(input: Self::Input) -> Self +pub fn vortex_array::dtype::DecimalTypeUpcast::from_i8(Self::Input) -> Self pub trait vortex_array::dtype::FromPrimitiveOrF16: num_traits::cast::FromPrimitive -pub fn vortex_array::dtype::FromPrimitiveOrF16::from_f16(v: half::binary16::f16) -> core::option::Option +pub fn vortex_array::dtype::FromPrimitiveOrF16::from_f16(half::binary16::f16) -> core::option::Option impl vortex_array::dtype::FromPrimitiveOrF16 for f32 -pub fn f32::from_f16(v: half::binary16::f16) -> core::option::Option +pub fn f32::from_f16(half::binary16::f16) -> core::option::Option impl vortex_array::dtype::FromPrimitiveOrF16 for f64 -pub fn f64::from_f16(v: half::binary16::f16) -> core::option::Option +pub fn f64::from_f16(half::binary16::f16) -> core::option::Option impl vortex_array::dtype::FromPrimitiveOrF16 for half::binary16::f16 -pub fn half::binary16::f16::from_f16(v: half::binary16::f16) -> core::option::Option +pub fn half::binary16::f16::from_f16(half::binary16::f16) -> core::option::Option impl vortex_array::dtype::FromPrimitiveOrF16 for i16 -pub fn i16::from_f16(_: half::binary16::f16) -> core::option::Option +pub fn i16::from_f16(half::binary16::f16) -> core::option::Option impl vortex_array::dtype::FromPrimitiveOrF16 for i32 -pub fn i32::from_f16(_: half::binary16::f16) -> core::option::Option +pub fn i32::from_f16(half::binary16::f16) -> core::option::Option impl vortex_array::dtype::FromPrimitiveOrF16 for i64 -pub fn i64::from_f16(_: half::binary16::f16) -> core::option::Option +pub fn i64::from_f16(half::binary16::f16) -> core::option::Option impl vortex_array::dtype::FromPrimitiveOrF16 for i8 -pub fn i8::from_f16(_: half::binary16::f16) -> core::option::Option +pub fn i8::from_f16(half::binary16::f16) -> core::option::Option impl vortex_array::dtype::FromPrimitiveOrF16 for u16 -pub fn u16::from_f16(value: half::binary16::f16) -> core::option::Option +pub fn u16::from_f16(half::binary16::f16) -> core::option::Option impl vortex_array::dtype::FromPrimitiveOrF16 for u32 -pub fn u32::from_f16(value: half::binary16::f16) -> core::option::Option +pub fn u32::from_f16(half::binary16::f16) -> core::option::Option impl vortex_array::dtype::FromPrimitiveOrF16 for u64 -pub fn u64::from_f16(value: half::binary16::f16) -> core::option::Option +pub fn u64::from_f16(half::binary16::f16) -> core::option::Option impl vortex_array::dtype::FromPrimitiveOrF16 for u8 -pub fn u8::from_f16(value: half::binary16::f16) -> core::option::Option +pub fn u8::from_f16(half::binary16::f16) -> core::option::Option impl vortex_array::dtype::FromPrimitiveOrF16 for usize -pub fn usize::from_f16(value: half::binary16::f16) -> core::option::Option +pub fn usize::from_f16(half::binary16::f16) -> core::option::Option pub trait vortex_array::dtype::IntegerPType: vortex_array::dtype::NativePType + num_traits::int::PrimInt + num_traits::cast::ToPrimitive + num_traits::bounds::Bounded + core::ops::arith::AddAssign + num_traits::cast::AsPrimitive @@ -10700,9 +10850,9 @@ pub const vortex_array::dtype::NativeDecimalType::MAX_SCALE: i8 pub const vortex_array::dtype::NativeDecimalType::MIN_BY_PRECISION: &'static [Self] -pub fn vortex_array::dtype::NativeDecimalType::downcast(visitor: V) -> ::Output +pub fn vortex_array::dtype::NativeDecimalType::downcast(V) -> ::Output -pub fn vortex_array::dtype::NativeDecimalType::upcast(input: ::Input) -> V +pub fn vortex_array::dtype::NativeDecimalType::upcast(::Input) -> V impl vortex_array::dtype::NativeDecimalType for i128 @@ -10716,9 +10866,9 @@ pub const i128::MAX_SCALE: i8 pub const i128::MIN_BY_PRECISION: &'static [Self] -pub fn i128::downcast(visitor: V) -> ::Output +pub fn i128::downcast(V) -> ::Output -pub fn i128::upcast(input: ::Input) -> V +pub fn i128::upcast(::Input) -> V impl vortex_array::dtype::NativeDecimalType for i16 @@ -10732,9 +10882,9 @@ pub const i16::MAX_SCALE: i8 pub const i16::MIN_BY_PRECISION: &'static [Self] -pub fn i16::downcast(visitor: V) -> ::Output +pub fn i16::downcast(V) -> ::Output -pub fn i16::upcast(input: ::Input) -> V +pub fn i16::upcast(::Input) -> V impl vortex_array::dtype::NativeDecimalType for i32 @@ -10748,9 +10898,9 @@ pub const i32::MAX_SCALE: i8 pub const i32::MIN_BY_PRECISION: &'static [Self] -pub fn i32::downcast(visitor: V) -> ::Output +pub fn i32::downcast(V) -> ::Output -pub fn i32::upcast(input: ::Input) -> V +pub fn i32::upcast(::Input) -> V impl vortex_array::dtype::NativeDecimalType for i64 @@ -10764,9 +10914,9 @@ pub const i64::MAX_SCALE: i8 pub const i64::MIN_BY_PRECISION: &'static [Self] -pub fn i64::downcast(visitor: V) -> ::Output +pub fn i64::downcast(V) -> ::Output -pub fn i64::upcast(input: ::Input) -> V +pub fn i64::upcast(::Input) -> V impl vortex_array::dtype::NativeDecimalType for i8 @@ -10780,9 +10930,9 @@ pub const i8::MAX_SCALE: i8 pub const i8::MIN_BY_PRECISION: &'static [Self] -pub fn i8::downcast(visitor: V) -> ::Output +pub fn i8::downcast(V) -> ::Output -pub fn i8::upcast(input: ::Input) -> V +pub fn i8::upcast(::Input) -> V impl vortex_array::dtype::NativeDecimalType for vortex_array::dtype::i256 @@ -10796,297 +10946,297 @@ pub const vortex_array::dtype::i256::MAX_SCALE: i8 pub const vortex_array::dtype::i256::MIN_BY_PRECISION: &'static [Self] -pub fn vortex_array::dtype::i256::downcast(visitor: V) -> ::Output +pub fn vortex_array::dtype::i256::downcast(V) -> ::Output -pub fn vortex_array::dtype::i256::upcast(input: ::Input) -> V +pub fn vortex_array::dtype::i256::upcast(::Input) -> V pub trait vortex_array::dtype::NativePType: core::marker::Send + core::marker::Sync + core::clone::Clone + core::marker::Copy + core::fmt::Debug + core::fmt::Display + core::default::Default + core::panic::unwind_safe::RefUnwindSafe + num_traits::Num + num_traits::cast::NumCast + vortex_array::dtype::FromPrimitiveOrF16 + vortex_array::dtype::ToBytes + vortex_array::dtype::TryFromBytes + vortex_array::dtype::ptype::private::Sealed + 'static pub const vortex_array::dtype::NativePType::PTYPE: vortex_array::dtype::PType -pub fn vortex_array::dtype::NativePType::downcast(visitor: V) -> ::Output +pub fn vortex_array::dtype::NativePType::downcast(V) -> ::Output -pub fn vortex_array::dtype::NativePType::is_eq(self, other: Self) -> bool +pub fn vortex_array::dtype::NativePType::is_eq(self, Self) -> bool -pub fn vortex_array::dtype::NativePType::is_ge(self, other: Self) -> bool +pub fn vortex_array::dtype::NativePType::is_ge(self, Self) -> bool -pub fn vortex_array::dtype::NativePType::is_gt(self, other: Self) -> bool +pub fn vortex_array::dtype::NativePType::is_gt(self, Self) -> bool pub fn vortex_array::dtype::NativePType::is_infinite(self) -> bool -pub fn vortex_array::dtype::NativePType::is_le(self, other: Self) -> bool +pub fn vortex_array::dtype::NativePType::is_le(self, Self) -> bool -pub fn vortex_array::dtype::NativePType::is_lt(self, other: Self) -> bool +pub fn vortex_array::dtype::NativePType::is_lt(self, Self) -> bool pub fn vortex_array::dtype::NativePType::is_nan(self) -> bool -pub fn vortex_array::dtype::NativePType::total_compare(self, other: Self) -> core::cmp::Ordering +pub fn vortex_array::dtype::NativePType::total_compare(self, Self) -> core::cmp::Ordering -pub fn vortex_array::dtype::NativePType::upcast(input: ::Input) -> V +pub fn vortex_array::dtype::NativePType::upcast(::Input) -> V impl vortex_array::dtype::NativePType for f32 pub const f32::PTYPE: vortex_array::dtype::PType -pub fn f32::downcast(visitor: V) -> ::Output +pub fn f32::downcast(V) -> ::Output -pub fn f32::is_eq(self, other: Self) -> bool +pub fn f32::is_eq(self, Self) -> bool -pub fn f32::is_ge(self, other: Self) -> bool +pub fn f32::is_ge(self, Self) -> bool -pub fn f32::is_gt(self, other: Self) -> bool +pub fn f32::is_gt(self, Self) -> bool pub fn f32::is_infinite(self) -> bool -pub fn f32::is_le(self, other: Self) -> bool +pub fn f32::is_le(self, Self) -> bool -pub fn f32::is_lt(self, other: Self) -> bool +pub fn f32::is_lt(self, Self) -> bool pub fn f32::is_nan(self) -> bool -pub fn f32::total_compare(self, other: Self) -> core::cmp::Ordering +pub fn f32::total_compare(self, Self) -> core::cmp::Ordering -pub fn f32::upcast(input: ::Input) -> V +pub fn f32::upcast(::Input) -> V impl vortex_array::dtype::NativePType for f64 pub const f64::PTYPE: vortex_array::dtype::PType -pub fn f64::downcast(visitor: V) -> ::Output +pub fn f64::downcast(V) -> ::Output -pub fn f64::is_eq(self, other: Self) -> bool +pub fn f64::is_eq(self, Self) -> bool -pub fn f64::is_ge(self, other: Self) -> bool +pub fn f64::is_ge(self, Self) -> bool -pub fn f64::is_gt(self, other: Self) -> bool +pub fn f64::is_gt(self, Self) -> bool pub fn f64::is_infinite(self) -> bool -pub fn f64::is_le(self, other: Self) -> bool +pub fn f64::is_le(self, Self) -> bool -pub fn f64::is_lt(self, other: Self) -> bool +pub fn f64::is_lt(self, Self) -> bool pub fn f64::is_nan(self) -> bool -pub fn f64::total_compare(self, other: Self) -> core::cmp::Ordering +pub fn f64::total_compare(self, Self) -> core::cmp::Ordering -pub fn f64::upcast(input: ::Input) -> V +pub fn f64::upcast(::Input) -> V impl vortex_array::dtype::NativePType for half::binary16::f16 pub const half::binary16::f16::PTYPE: vortex_array::dtype::PType -pub fn half::binary16::f16::downcast(visitor: V) -> ::Output +pub fn half::binary16::f16::downcast(V) -> ::Output -pub fn half::binary16::f16::is_eq(self, other: Self) -> bool +pub fn half::binary16::f16::is_eq(self, Self) -> bool -pub fn half::binary16::f16::is_ge(self, other: Self) -> bool +pub fn half::binary16::f16::is_ge(self, Self) -> bool -pub fn half::binary16::f16::is_gt(self, other: Self) -> bool +pub fn half::binary16::f16::is_gt(self, Self) -> bool pub fn half::binary16::f16::is_infinite(self) -> bool -pub fn half::binary16::f16::is_le(self, other: Self) -> bool +pub fn half::binary16::f16::is_le(self, Self) -> bool -pub fn half::binary16::f16::is_lt(self, other: Self) -> bool +pub fn half::binary16::f16::is_lt(self, Self) -> bool pub fn half::binary16::f16::is_nan(self) -> bool -pub fn half::binary16::f16::total_compare(self, other: Self) -> core::cmp::Ordering +pub fn half::binary16::f16::total_compare(self, Self) -> core::cmp::Ordering -pub fn half::binary16::f16::upcast(input: ::Input) -> V +pub fn half::binary16::f16::upcast(::Input) -> V impl vortex_array::dtype::NativePType for i16 pub const i16::PTYPE: vortex_array::dtype::PType -pub fn i16::downcast(visitor: V) -> ::Output +pub fn i16::downcast(V) -> ::Output -pub fn i16::is_eq(self, other: Self) -> bool +pub fn i16::is_eq(self, Self) -> bool -pub fn i16::is_ge(self, other: Self) -> bool +pub fn i16::is_ge(self, Self) -> bool -pub fn i16::is_gt(self, other: Self) -> bool +pub fn i16::is_gt(self, Self) -> bool pub fn i16::is_infinite(self) -> bool -pub fn i16::is_le(self, other: Self) -> bool +pub fn i16::is_le(self, Self) -> bool -pub fn i16::is_lt(self, other: Self) -> bool +pub fn i16::is_lt(self, Self) -> bool pub fn i16::is_nan(self) -> bool -pub fn i16::total_compare(self, other: Self) -> core::cmp::Ordering +pub fn i16::total_compare(self, Self) -> core::cmp::Ordering -pub fn i16::upcast(input: ::Input) -> V +pub fn i16::upcast(::Input) -> V impl vortex_array::dtype::NativePType for i32 pub const i32::PTYPE: vortex_array::dtype::PType -pub fn i32::downcast(visitor: V) -> ::Output +pub fn i32::downcast(V) -> ::Output -pub fn i32::is_eq(self, other: Self) -> bool +pub fn i32::is_eq(self, Self) -> bool -pub fn i32::is_ge(self, other: Self) -> bool +pub fn i32::is_ge(self, Self) -> bool -pub fn i32::is_gt(self, other: Self) -> bool +pub fn i32::is_gt(self, Self) -> bool pub fn i32::is_infinite(self) -> bool -pub fn i32::is_le(self, other: Self) -> bool +pub fn i32::is_le(self, Self) -> bool -pub fn i32::is_lt(self, other: Self) -> bool +pub fn i32::is_lt(self, Self) -> bool pub fn i32::is_nan(self) -> bool -pub fn i32::total_compare(self, other: Self) -> core::cmp::Ordering +pub fn i32::total_compare(self, Self) -> core::cmp::Ordering -pub fn i32::upcast(input: ::Input) -> V +pub fn i32::upcast(::Input) -> V impl vortex_array::dtype::NativePType for i64 pub const i64::PTYPE: vortex_array::dtype::PType -pub fn i64::downcast(visitor: V) -> ::Output +pub fn i64::downcast(V) -> ::Output -pub fn i64::is_eq(self, other: Self) -> bool +pub fn i64::is_eq(self, Self) -> bool -pub fn i64::is_ge(self, other: Self) -> bool +pub fn i64::is_ge(self, Self) -> bool -pub fn i64::is_gt(self, other: Self) -> bool +pub fn i64::is_gt(self, Self) -> bool pub fn i64::is_infinite(self) -> bool -pub fn i64::is_le(self, other: Self) -> bool +pub fn i64::is_le(self, Self) -> bool -pub fn i64::is_lt(self, other: Self) -> bool +pub fn i64::is_lt(self, Self) -> bool pub fn i64::is_nan(self) -> bool -pub fn i64::total_compare(self, other: Self) -> core::cmp::Ordering +pub fn i64::total_compare(self, Self) -> core::cmp::Ordering -pub fn i64::upcast(input: ::Input) -> V +pub fn i64::upcast(::Input) -> V impl vortex_array::dtype::NativePType for i8 pub const i8::PTYPE: vortex_array::dtype::PType -pub fn i8::downcast(visitor: V) -> ::Output +pub fn i8::downcast(V) -> ::Output -pub fn i8::is_eq(self, other: Self) -> bool +pub fn i8::is_eq(self, Self) -> bool -pub fn i8::is_ge(self, other: Self) -> bool +pub fn i8::is_ge(self, Self) -> bool -pub fn i8::is_gt(self, other: Self) -> bool +pub fn i8::is_gt(self, Self) -> bool pub fn i8::is_infinite(self) -> bool -pub fn i8::is_le(self, other: Self) -> bool +pub fn i8::is_le(self, Self) -> bool -pub fn i8::is_lt(self, other: Self) -> bool +pub fn i8::is_lt(self, Self) -> bool pub fn i8::is_nan(self) -> bool -pub fn i8::total_compare(self, other: Self) -> core::cmp::Ordering +pub fn i8::total_compare(self, Self) -> core::cmp::Ordering -pub fn i8::upcast(input: ::Input) -> V +pub fn i8::upcast(::Input) -> V impl vortex_array::dtype::NativePType for u16 pub const u16::PTYPE: vortex_array::dtype::PType -pub fn u16::downcast(visitor: V) -> ::Output +pub fn u16::downcast(V) -> ::Output -pub fn u16::is_eq(self, other: Self) -> bool +pub fn u16::is_eq(self, Self) -> bool -pub fn u16::is_ge(self, other: Self) -> bool +pub fn u16::is_ge(self, Self) -> bool -pub fn u16::is_gt(self, other: Self) -> bool +pub fn u16::is_gt(self, Self) -> bool pub fn u16::is_infinite(self) -> bool -pub fn u16::is_le(self, other: Self) -> bool +pub fn u16::is_le(self, Self) -> bool -pub fn u16::is_lt(self, other: Self) -> bool +pub fn u16::is_lt(self, Self) -> bool pub fn u16::is_nan(self) -> bool -pub fn u16::total_compare(self, other: Self) -> core::cmp::Ordering +pub fn u16::total_compare(self, Self) -> core::cmp::Ordering -pub fn u16::upcast(input: ::Input) -> V +pub fn u16::upcast(::Input) -> V impl vortex_array::dtype::NativePType for u32 pub const u32::PTYPE: vortex_array::dtype::PType -pub fn u32::downcast(visitor: V) -> ::Output +pub fn u32::downcast(V) -> ::Output -pub fn u32::is_eq(self, other: Self) -> bool +pub fn u32::is_eq(self, Self) -> bool -pub fn u32::is_ge(self, other: Self) -> bool +pub fn u32::is_ge(self, Self) -> bool -pub fn u32::is_gt(self, other: Self) -> bool +pub fn u32::is_gt(self, Self) -> bool pub fn u32::is_infinite(self) -> bool -pub fn u32::is_le(self, other: Self) -> bool +pub fn u32::is_le(self, Self) -> bool -pub fn u32::is_lt(self, other: Self) -> bool +pub fn u32::is_lt(self, Self) -> bool pub fn u32::is_nan(self) -> bool -pub fn u32::total_compare(self, other: Self) -> core::cmp::Ordering +pub fn u32::total_compare(self, Self) -> core::cmp::Ordering -pub fn u32::upcast(input: ::Input) -> V +pub fn u32::upcast(::Input) -> V impl vortex_array::dtype::NativePType for u64 pub const u64::PTYPE: vortex_array::dtype::PType -pub fn u64::downcast(visitor: V) -> ::Output +pub fn u64::downcast(V) -> ::Output -pub fn u64::is_eq(self, other: Self) -> bool +pub fn u64::is_eq(self, Self) -> bool -pub fn u64::is_ge(self, other: Self) -> bool +pub fn u64::is_ge(self, Self) -> bool -pub fn u64::is_gt(self, other: Self) -> bool +pub fn u64::is_gt(self, Self) -> bool pub fn u64::is_infinite(self) -> bool -pub fn u64::is_le(self, other: Self) -> bool +pub fn u64::is_le(self, Self) -> bool -pub fn u64::is_lt(self, other: Self) -> bool +pub fn u64::is_lt(self, Self) -> bool pub fn u64::is_nan(self) -> bool -pub fn u64::total_compare(self, other: Self) -> core::cmp::Ordering +pub fn u64::total_compare(self, Self) -> core::cmp::Ordering -pub fn u64::upcast(input: ::Input) -> V +pub fn u64::upcast(::Input) -> V impl vortex_array::dtype::NativePType for u8 pub const u8::PTYPE: vortex_array::dtype::PType -pub fn u8::downcast(visitor: V) -> ::Output +pub fn u8::downcast(V) -> ::Output -pub fn u8::is_eq(self, other: Self) -> bool +pub fn u8::is_eq(self, Self) -> bool -pub fn u8::is_ge(self, other: Self) -> bool +pub fn u8::is_ge(self, Self) -> bool -pub fn u8::is_gt(self, other: Self) -> bool +pub fn u8::is_gt(self, Self) -> bool pub fn u8::is_infinite(self) -> bool -pub fn u8::is_le(self, other: Self) -> bool +pub fn u8::is_le(self, Self) -> bool -pub fn u8::is_lt(self, other: Self) -> bool +pub fn u8::is_lt(self, Self) -> bool pub fn u8::is_nan(self) -> bool -pub fn u8::total_compare(self, other: Self) -> core::cmp::Ordering +pub fn u8::total_compare(self, Self) -> core::cmp::Ordering -pub fn u8::upcast(input: ::Input) -> V +pub fn u8::upcast(::Input) -> V pub trait vortex_array::dtype::PTypeDowncast @@ -11126,27 +11276,27 @@ pub trait vortex_array::dtype::PTypeUpcast pub type vortex_array::dtype::PTypeUpcast::Input -pub fn vortex_array::dtype::PTypeUpcast::from_f16(input: Self::Input) -> Self +pub fn vortex_array::dtype::PTypeUpcast::from_f16(Self::Input) -> Self -pub fn vortex_array::dtype::PTypeUpcast::from_f32(input: Self::Input) -> Self +pub fn vortex_array::dtype::PTypeUpcast::from_f32(Self::Input) -> Self -pub fn vortex_array::dtype::PTypeUpcast::from_f64(input: Self::Input) -> Self +pub fn vortex_array::dtype::PTypeUpcast::from_f64(Self::Input) -> Self -pub fn vortex_array::dtype::PTypeUpcast::from_i16(input: Self::Input) -> Self +pub fn vortex_array::dtype::PTypeUpcast::from_i16(Self::Input) -> Self -pub fn vortex_array::dtype::PTypeUpcast::from_i32(input: Self::Input) -> Self +pub fn vortex_array::dtype::PTypeUpcast::from_i32(Self::Input) -> Self -pub fn vortex_array::dtype::PTypeUpcast::from_i64(input: Self::Input) -> Self +pub fn vortex_array::dtype::PTypeUpcast::from_i64(Self::Input) -> Self -pub fn vortex_array::dtype::PTypeUpcast::from_i8(input: Self::Input) -> Self +pub fn vortex_array::dtype::PTypeUpcast::from_i8(Self::Input) -> Self -pub fn vortex_array::dtype::PTypeUpcast::from_u16(input: Self::Input) -> Self +pub fn vortex_array::dtype::PTypeUpcast::from_u16(Self::Input) -> Self -pub fn vortex_array::dtype::PTypeUpcast::from_u32(input: Self::Input) -> Self +pub fn vortex_array::dtype::PTypeUpcast::from_u32(Self::Input) -> Self -pub fn vortex_array::dtype::PTypeUpcast::from_u64(input: Self::Input) -> Self +pub fn vortex_array::dtype::PTypeUpcast::from_u64(Self::Input) -> Self -pub fn vortex_array::dtype::PTypeUpcast::from_u8(input: Self::Input) -> Self +pub fn vortex_array::dtype::PTypeUpcast::from_u8(Self::Input) -> Self pub trait vortex_array::dtype::PhysicalPType: vortex_array::dtype::NativePType @@ -11286,51 +11436,51 @@ pub fn vortex_array::dtype::i256::to_i256(&self) -> core::option::Option vortex_error::VortexResult +pub fn vortex_array::dtype::TryFromBytes::try_from_le_bytes(&[u8]) -> vortex_error::VortexResult impl vortex_array::dtype::TryFromBytes for f32 -pub fn f32::try_from_le_bytes(bytes: &[u8]) -> vortex_error::VortexResult +pub fn f32::try_from_le_bytes(&[u8]) -> vortex_error::VortexResult impl vortex_array::dtype::TryFromBytes for f64 -pub fn f64::try_from_le_bytes(bytes: &[u8]) -> vortex_error::VortexResult +pub fn f64::try_from_le_bytes(&[u8]) -> vortex_error::VortexResult impl vortex_array::dtype::TryFromBytes for half::binary16::f16 -pub fn half::binary16::f16::try_from_le_bytes(bytes: &[u8]) -> vortex_error::VortexResult +pub fn half::binary16::f16::try_from_le_bytes(&[u8]) -> vortex_error::VortexResult impl vortex_array::dtype::TryFromBytes for i16 -pub fn i16::try_from_le_bytes(bytes: &[u8]) -> vortex_error::VortexResult +pub fn i16::try_from_le_bytes(&[u8]) -> vortex_error::VortexResult impl vortex_array::dtype::TryFromBytes for i32 -pub fn i32::try_from_le_bytes(bytes: &[u8]) -> vortex_error::VortexResult +pub fn i32::try_from_le_bytes(&[u8]) -> vortex_error::VortexResult impl vortex_array::dtype::TryFromBytes for i64 -pub fn i64::try_from_le_bytes(bytes: &[u8]) -> vortex_error::VortexResult +pub fn i64::try_from_le_bytes(&[u8]) -> vortex_error::VortexResult impl vortex_array::dtype::TryFromBytes for i8 -pub fn i8::try_from_le_bytes(bytes: &[u8]) -> vortex_error::VortexResult +pub fn i8::try_from_le_bytes(&[u8]) -> vortex_error::VortexResult impl vortex_array::dtype::TryFromBytes for u16 -pub fn u16::try_from_le_bytes(bytes: &[u8]) -> vortex_error::VortexResult +pub fn u16::try_from_le_bytes(&[u8]) -> vortex_error::VortexResult impl vortex_array::dtype::TryFromBytes for u32 -pub fn u32::try_from_le_bytes(bytes: &[u8]) -> vortex_error::VortexResult +pub fn u32::try_from_le_bytes(&[u8]) -> vortex_error::VortexResult impl vortex_array::dtype::TryFromBytes for u64 -pub fn u64::try_from_le_bytes(bytes: &[u8]) -> vortex_error::VortexResult +pub fn u64::try_from_le_bytes(&[u8]) -> vortex_error::VortexResult impl vortex_array::dtype::TryFromBytes for u8 -pub fn u8::try_from_le_bytes(bytes: &[u8]) -> vortex_error::VortexResult +pub fn u8::try_from_le_bytes(&[u8]) -> vortex_error::VortexResult pub trait vortex_array::dtype::UnsignedPType: vortex_array::dtype::IntegerPType + num_traits::sign::Unsigned @@ -11360,17 +11510,17 @@ impl vortex_array::expr::AnnotationFn for F where A: vortex_array::expr::A pub type F::Annotation = A -pub fn vortex_array::expr::analysis::annotation::descendent_annotations(expr: &vortex_array::expr::Expression, annotate: A) -> vortex_array::expr::Annotations<'_, ::Annotation> +pub fn vortex_array::expr::analysis::annotation::descendent_annotations(&vortex_array::expr::Expression, A) -> vortex_array::expr::Annotations<'_, ::Annotation> pub type vortex_array::expr::analysis::annotation::Annotations<'a, A> = vortex_utils::aliases::hash_map::HashMap<&'a vortex_array::expr::Expression, vortex_utils::aliases::hash_set::HashSet> pub mod vortex_array::expr::analysis::immediate_access -pub fn vortex_array::expr::analysis::immediate_access::immediate_scope_access<'a>(expr: &'a vortex_array::expr::Expression, scope: &'a vortex_array::dtype::StructFields) -> vortex_utils::aliases::hash_set::HashSet +pub fn vortex_array::expr::analysis::immediate_access::immediate_scope_access<'a>(&'a vortex_array::expr::Expression, &'a vortex_array::dtype::StructFields) -> vortex_utils::aliases::hash_set::HashSet -pub fn vortex_array::expr::analysis::immediate_access::immediate_scope_accesses<'a>(expr: &'a vortex_array::expr::Expression, scope: &'a vortex_array::dtype::StructFields) -> vortex_array::expr::FieldAccesses<'a> +pub fn vortex_array::expr::analysis::immediate_access::immediate_scope_accesses<'a>(&'a vortex_array::expr::Expression, &'a vortex_array::dtype::StructFields) -> vortex_array::expr::FieldAccesses<'a> -pub fn vortex_array::expr::analysis::immediate_access::make_free_field_annotator(scope: &vortex_array::dtype::StructFields) -> impl vortex_array::expr::AnnotationFn +pub fn vortex_array::expr::analysis::immediate_access::make_free_field_annotator(&vortex_array::dtype::StructFields) -> impl vortex_array::expr::AnnotationFn pub type vortex_array::expr::analysis::immediate_access::FieldAccesses<'a> = vortex_array::expr::Annotations<'a, vortex_array::dtype::FieldName> @@ -11386,19 +11536,19 @@ impl vortex_array::expr::AnnotationFn for F where A: vortex_array::expr::A pub type F::Annotation = A -pub fn vortex_array::expr::analysis::descendent_annotations(expr: &vortex_array::expr::Expression, annotate: A) -> vortex_array::expr::Annotations<'_, ::Annotation> +pub fn vortex_array::expr::analysis::descendent_annotations(&vortex_array::expr::Expression, A) -> vortex_array::expr::Annotations<'_, ::Annotation> -pub fn vortex_array::expr::analysis::immediate_scope_access<'a>(expr: &'a vortex_array::expr::Expression, scope: &'a vortex_array::dtype::StructFields) -> vortex_utils::aliases::hash_set::HashSet +pub fn vortex_array::expr::analysis::immediate_scope_access<'a>(&'a vortex_array::expr::Expression, &'a vortex_array::dtype::StructFields) -> vortex_utils::aliases::hash_set::HashSet -pub fn vortex_array::expr::analysis::immediate_scope_accesses<'a>(expr: &'a vortex_array::expr::Expression, scope: &'a vortex_array::dtype::StructFields) -> vortex_array::expr::FieldAccesses<'a> +pub fn vortex_array::expr::analysis::immediate_scope_accesses<'a>(&'a vortex_array::expr::Expression, &'a vortex_array::dtype::StructFields) -> vortex_array::expr::FieldAccesses<'a> -pub fn vortex_array::expr::analysis::label_is_fallible(expr: &vortex_array::expr::Expression) -> vortex_array::expr::BooleanLabels<'_> +pub fn vortex_array::expr::analysis::label_is_fallible(&vortex_array::expr::Expression) -> vortex_array::expr::BooleanLabels<'_> -pub fn vortex_array::expr::analysis::label_null_sensitive(expr: &vortex_array::expr::Expression) -> vortex_array::expr::BooleanLabels<'_> +pub fn vortex_array::expr::analysis::label_null_sensitive(&vortex_array::expr::Expression) -> vortex_array::expr::BooleanLabels<'_> -pub fn vortex_array::expr::analysis::label_tree(expr: &vortex_array::expr::Expression, self_label: impl core::ops::function::Fn(&vortex_array::expr::Expression) -> L, merge_child: impl core::ops::function::FnMut(L, &L) -> L) -> vortex_utils::aliases::hash_map::HashMap<&vortex_array::expr::Expression, L> +pub fn vortex_array::expr::analysis::label_tree(&vortex_array::expr::Expression, impl core::ops::function::Fn(&vortex_array::expr::Expression) -> L, impl core::ops::function::FnMut(L, &L) -> L) -> vortex_utils::aliases::hash_map::HashMap<&vortex_array::expr::Expression, L> -pub fn vortex_array::expr::analysis::make_free_field_annotator(scope: &vortex_array::dtype::StructFields) -> impl vortex_array::expr::AnnotationFn +pub fn vortex_array::expr::analysis::make_free_field_annotator(&vortex_array::dtype::StructFields) -> impl vortex_array::expr::AnnotationFn pub type vortex_array::expr::analysis::Annotations<'a, A> = vortex_utils::aliases::hash_map::HashMap<&'a vortex_array::expr::Expression, vortex_utils::aliases::hash_set::HashSet> @@ -11420,7 +11570,7 @@ impl vortex_array::expr::AnnotationFn for F where A: vortex_array::expr::A pub type F::Annotation = A -pub fn vortex_array::expr::annotation::descendent_annotations(expr: &vortex_array::expr::Expression, annotate: A) -> vortex_array::expr::Annotations<'_, ::Annotation> +pub fn vortex_array::expr::annotation::descendent_annotations(&vortex_array::expr::Expression, A) -> vortex_array::expr::Annotations<'_, ::Annotation> pub type vortex_array::expr::annotation::Annotations<'a, A> = vortex_utils::aliases::hash_map::HashMap<&'a vortex_array::expr::Expression, vortex_utils::aliases::hash_set::HashSet> @@ -11436,19 +11586,19 @@ pub struct vortex_array::expr::display::DisplayTreeExpr<'a>(pub &'a vortex_array impl core::fmt::Display for vortex_array::expr::display::DisplayTreeExpr<'_> -pub fn vortex_array::expr::display::DisplayTreeExpr<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::expr::display::DisplayTreeExpr<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub mod vortex_array::expr::forms -pub fn vortex_array::expr::forms::conjuncts(expr: &vortex_array::expr::Expression) -> alloc::vec::Vec +pub fn vortex_array::expr::forms::conjuncts(&vortex_array::expr::Expression) -> alloc::vec::Vec pub mod vortex_array::expr::immediate_access -pub fn vortex_array::expr::immediate_access::immediate_scope_access<'a>(expr: &'a vortex_array::expr::Expression, scope: &'a vortex_array::dtype::StructFields) -> vortex_utils::aliases::hash_set::HashSet +pub fn vortex_array::expr::immediate_access::immediate_scope_access<'a>(&'a vortex_array::expr::Expression, &'a vortex_array::dtype::StructFields) -> vortex_utils::aliases::hash_set::HashSet -pub fn vortex_array::expr::immediate_access::immediate_scope_accesses<'a>(expr: &'a vortex_array::expr::Expression, scope: &'a vortex_array::dtype::StructFields) -> vortex_array::expr::FieldAccesses<'a> +pub fn vortex_array::expr::immediate_access::immediate_scope_accesses<'a>(&'a vortex_array::expr::Expression, &'a vortex_array::dtype::StructFields) -> vortex_array::expr::FieldAccesses<'a> -pub fn vortex_array::expr::immediate_access::make_free_field_annotator(scope: &vortex_array::dtype::StructFields) -> impl vortex_array::expr::AnnotationFn +pub fn vortex_array::expr::immediate_access::make_free_field_annotator(&vortex_array::dtype::StructFields) -> impl vortex_array::expr::AnnotationFn pub type vortex_array::expr::immediate_access::FieldAccesses<'a> = vortex_array::expr::Annotations<'a, vortex_array::dtype::FieldName> @@ -11462,7 +11612,7 @@ impl vortex_array::expr::proto::ExprSerializeProtoExt for vortex_array::expr::Ex pub fn vortex_array::expr::Expression::serialize_proto(&self) -> vortex_error::VortexResult -pub fn vortex_array::expr::proto::deserialize_expr_proto(expr: &vortex_proto::expr::Expr, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::expr::proto::deserialize_expr_proto(&vortex_proto::expr::Expr, &vortex_session::VortexSession) -> vortex_error::VortexResult pub mod vortex_array::expr::pruning @@ -11470,7 +11620,7 @@ pub struct vortex_array::expr::pruning::Relation impl vortex_array::expr::pruning::Relation -pub fn vortex_array::expr::pruning::Relation::insert(&mut self, k: K, v: V) +pub fn vortex_array::expr::pruning::Relation::insert(&mut self, K, V) pub fn vortex_array::expr::pruning::Relation::map(&self) -> &vortex_utils::aliases::hash_map::HashMap> @@ -11478,7 +11628,7 @@ pub fn vortex_array::expr::pruning::Relation::new() -> Self impl core::convert::From>> for vortex_array::expr::pruning::Relation -pub fn vortex_array::expr::pruning::Relation::from(value: vortex_utils::aliases::hash_map::HashMap>) -> Self +pub fn vortex_array::expr::pruning::Relation::from(vortex_utils::aliases::hash_map::HashMap>) -> Self impl core::iter::traits::collect::IntoIterator for vortex_array::expr::pruning::Relation @@ -11494,7 +11644,7 @@ pub fn vortex_array::expr::pruning::Relation::clone(&self) -> vortex_array impl core::fmt::Debug for vortex_array::expr::pruning::Relation -pub fn vortex_array::expr::pruning::Relation::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::expr::pruning::Relation::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::default::Default for vortex_array::expr::pruning::Relation @@ -11502,9 +11652,9 @@ pub fn vortex_array::expr::pruning::Relation::default() -> Self pub trait vortex_array::expr::pruning::StatsCatalog -pub fn vortex_array::expr::pruning::StatsCatalog::stats_ref(&self, _field_path: &vortex_array::dtype::FieldPath, _stat: vortex_array::expr::stats::Stat) -> core::option::Option +pub fn vortex_array::expr::pruning::StatsCatalog::stats_ref(&self, &vortex_array::dtype::FieldPath, vortex_array::expr::stats::Stat) -> core::option::Option -pub fn vortex_array::expr::pruning::checked_pruning_expr(expr: &vortex_array::expr::Expression, available_stats: &vortex_array::dtype::FieldPathSet) -> core::option::Option<(vortex_array::expr::Expression, vortex_array::expr::pruning::RequiredStats)> +pub fn vortex_array::expr::pruning::checked_pruning_expr(&vortex_array::expr::Expression, &vortex_array::dtype::FieldPathSet) -> core::option::Option<(vortex_array::expr::Expression, vortex_array::expr::pruning::RequiredStats)> pub type vortex_array::expr::pruning::RequiredStats = vortex_array::expr::pruning::Relation @@ -11518,7 +11668,7 @@ pub vortex_array::expr::stats::IntersectionResult::Value(T) impl vortex_array::expr::stats::IntersectionResult -pub fn vortex_array::expr::stats::IntersectionResult::ok_or_else(self, err: F) -> vortex_error::VortexResult where F: core::ops::function::FnOnce() -> vortex_error::VortexError +pub fn vortex_array::expr::stats::IntersectionResult::ok_or_else(self, F) -> vortex_error::VortexResult where F: core::ops::function::FnOnce() -> vortex_error::VortexError impl core::clone::Clone for vortex_array::expr::stats::IntersectionResult @@ -11528,11 +11678,11 @@ impl core::cmp::Eq for vortex_array::expr::stats::Intersection impl core::cmp::PartialEq for vortex_array::expr::stats::IntersectionResult -pub fn vortex_array::expr::stats::IntersectionResult::eq(&self, other: &vortex_array::expr::stats::IntersectionResult) -> bool +pub fn vortex_array::expr::stats::IntersectionResult::eq(&self, &vortex_array::expr::stats::IntersectionResult) -> bool impl core::fmt::Debug for vortex_array::expr::stats::IntersectionResult -pub fn vortex_array::expr::stats::IntersectionResult::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::expr::stats::IntersectionResult::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::StructuralPartialEq for vortex_array::expr::stats::IntersectionResult @@ -11544,11 +11694,11 @@ pub vortex_array::expr::stats::Precision::Inexact(T) impl vortex_array::expr::stats::Precision<&vortex_array::scalar::ScalarValue> -pub fn vortex_array::expr::stats::Precision<&vortex_array::scalar::ScalarValue>::into_scalar(self, dtype: vortex_array::dtype::DType) -> vortex_array::expr::stats::Precision +pub fn vortex_array::expr::stats::Precision<&vortex_array::scalar::ScalarValue>::into_scalar(self, vortex_array::dtype::DType) -> vortex_array::expr::stats::Precision impl vortex_array::expr::stats::Precision -pub fn vortex_array::expr::stats::Precision::into_scalar(self, dtype: vortex_array::dtype::DType) -> vortex_array::expr::stats::Precision +pub fn vortex_array::expr::stats::Precision::into_scalar(self, vortex_array::dtype::DType) -> vortex_array::expr::stats::Precision impl vortex_array::expr::stats::Precision where T: core::marker::Copy @@ -11562,9 +11712,9 @@ pub fn vortex_array::expr::stats::Precision::as_inexact(self) -> core::option pub fn vortex_array::expr::stats::Precision::as_ref(&self) -> vortex_array::expr::stats::Precision<&T> -pub fn vortex_array::expr::stats::Precision::exact>(s: S) -> vortex_array::expr::stats::Precision +pub fn vortex_array::expr::stats::Precision::exact>(S) -> vortex_array::expr::stats::Precision -pub fn vortex_array::expr::stats::Precision::inexact>(s: S) -> vortex_array::expr::stats::Precision +pub fn vortex_array::expr::stats::Precision::inexact>(S) -> vortex_array::expr::stats::Precision pub fn vortex_array::expr::stats::Precision::into_inexact(self) -> Self @@ -11572,11 +11722,11 @@ pub fn vortex_array::expr::stats::Precision::into_inner(self) -> T pub fn vortex_array::expr::stats::Precision::is_exact(&self) -> bool -pub fn vortex_array::expr::stats::Precision::map U>(self, f: F) -> vortex_array::expr::stats::Precision +pub fn vortex_array::expr::stats::Precision::map U>(self, F) -> vortex_array::expr::stats::Precision -pub fn vortex_array::expr::stats::Precision::try_map vortex_error::VortexResult>(self, f: F) -> vortex_error::VortexResult> +pub fn vortex_array::expr::stats::Precision::try_map vortex_error::VortexResult>(self, F) -> vortex_error::VortexResult> -pub fn vortex_array::expr::stats::Precision::zip(self, other: vortex_array::expr::stats::Precision) -> vortex_array::expr::stats::Precision<(T, U)> +pub fn vortex_array::expr::stats::Precision::zip(self, vortex_array::expr::stats::Precision) -> vortex_array::expr::stats::Precision<(T, U)> impl vortex_array::expr::stats::Precision @@ -11594,31 +11744,31 @@ impl core::cmp::Eq for vortex_array::expr::stats::Precision impl core::cmp::PartialEq for vortex_array::expr::stats::Precision -pub fn vortex_array::expr::stats::Precision::eq(&self, other: &vortex_array::expr::stats::Precision) -> bool +pub fn vortex_array::expr::stats::Precision::eq(&self, &vortex_array::expr::stats::Precision) -> bool impl core::cmp::PartialEq for vortex_array::expr::stats::Precision -pub fn vortex_array::expr::stats::Precision::eq(&self, other: &T) -> bool +pub fn vortex_array::expr::stats::Precision::eq(&self, &T) -> bool impl vortex_array::expr::stats::StatBound for vortex_array::expr::stats::Precision -pub fn vortex_array::expr::stats::Precision::intersection(&self, other: &Self) -> core::option::Option> +pub fn vortex_array::expr::stats::Precision::intersection(&self, &Self) -> core::option::Option> pub fn vortex_array::expr::stats::Precision::into_value(self) -> vortex_array::expr::stats::Precision -pub fn vortex_array::expr::stats::Precision::lift(value: vortex_array::expr::stats::Precision) -> Self +pub fn vortex_array::expr::stats::Precision::lift(vortex_array::expr::stats::Precision) -> Self pub fn vortex_array::expr::stats::Precision::to_exact(&self) -> core::option::Option<&T> -pub fn vortex_array::expr::stats::Precision::union(&self, other: &Self) -> core::option::Option +pub fn vortex_array::expr::stats::Precision::union(&self, &Self) -> core::option::Option impl core::fmt::Debug for vortex_array::expr::stats::Precision -pub fn vortex_array::expr::stats::Precision::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::expr::stats::Precision::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::expr::stats::Precision -pub fn vortex_array::expr::stats::Precision::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::expr::stats::Precision::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_array::expr::stats::Precision @@ -11648,7 +11798,7 @@ impl vortex_array::expr::stats::Stat pub fn vortex_array::expr::stats::Stat::all() -> impl core::iter::traits::iterator::Iterator -pub fn vortex_array::expr::stats::Stat::dtype(&self, data_type: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::expr::stats::Stat::dtype(&self, &vortex_array::dtype::DType) -> core::option::Option pub fn vortex_array::expr::stats::Stat::has_same_dtype_as_array(&self) -> bool @@ -11664,37 +11814,37 @@ impl core::cmp::Eq for vortex_array::expr::stats::Stat impl core::cmp::Ord for vortex_array::expr::stats::Stat -pub fn vortex_array::expr::stats::Stat::cmp(&self, other: &vortex_array::expr::stats::Stat) -> core::cmp::Ordering +pub fn vortex_array::expr::stats::Stat::cmp(&self, &vortex_array::expr::stats::Stat) -> core::cmp::Ordering impl core::cmp::PartialEq for vortex_array::expr::stats::Stat -pub fn vortex_array::expr::stats::Stat::eq(&self, other: &vortex_array::expr::stats::Stat) -> bool +pub fn vortex_array::expr::stats::Stat::eq(&self, &vortex_array::expr::stats::Stat) -> bool impl core::cmp::PartialOrd for vortex_array::expr::stats::Stat -pub fn vortex_array::expr::stats::Stat::partial_cmp(&self, other: &vortex_array::expr::stats::Stat) -> core::option::Option +pub fn vortex_array::expr::stats::Stat::partial_cmp(&self, &vortex_array::expr::stats::Stat) -> core::option::Option impl core::convert::From for u8 -pub fn u8::from(enum_value: vortex_array::expr::stats::Stat) -> Self +pub fn u8::from(vortex_array::expr::stats::Stat) -> Self impl core::convert::TryFrom for vortex_array::expr::stats::Stat pub type vortex_array::expr::stats::Stat::Error = num_enum::TryFromPrimitiveError -pub fn vortex_array::expr::stats::Stat::try_from(number: u8) -> core::result::Result> +pub fn vortex_array::expr::stats::Stat::try_from(u8) -> core::result::Result> impl core::fmt::Debug for vortex_array::expr::stats::Stat -pub fn vortex_array::expr::stats::Stat::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::expr::stats::Stat::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::expr::stats::Stat -pub fn vortex_array::expr::stats::Stat::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::expr::stats::Stat::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::expr::stats::Stat -pub fn vortex_array::expr::stats::Stat::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::expr::stats::Stat::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_array::expr::stats::Stat @@ -11720,7 +11870,7 @@ pub type vortex_array::expr::stats::Stat::Primitive = u8 pub const vortex_array::expr::stats::Stat::NAME: &'static str -pub fn vortex_array::expr::stats::Stat::try_from_primitive(number: Self::Primitive) -> core::result::Result> +pub fn vortex_array::expr::stats::Stat::try_from_primitive(Self::Primitive) -> core::result::Result> pub struct vortex_array::expr::stats::IsConstant @@ -11760,31 +11910,31 @@ impl core::cmp::Eq for vortex_array::expr::stats::LowerBound core::cmp::PartialEq for vortex_array::expr::stats::LowerBound -pub fn vortex_array::expr::stats::LowerBound::eq(&self, other: &vortex_array::expr::stats::LowerBound) -> bool +pub fn vortex_array::expr::stats::LowerBound::eq(&self, &vortex_array::expr::stats::LowerBound) -> bool impl vortex_array::expr::stats::StatBound for vortex_array::expr::stats::LowerBound -pub fn vortex_array::expr::stats::LowerBound::intersection(&self, other: &Self) -> core::option::Option>> +pub fn vortex_array::expr::stats::LowerBound::intersection(&self, &Self) -> core::option::Option>> pub fn vortex_array::expr::stats::LowerBound::into_value(self) -> vortex_array::expr::stats::Precision -pub fn vortex_array::expr::stats::LowerBound::lift(value: vortex_array::expr::stats::Precision) -> Self +pub fn vortex_array::expr::stats::LowerBound::lift(vortex_array::expr::stats::Precision) -> Self pub fn vortex_array::expr::stats::LowerBound::to_exact(&self) -> core::option::Option<&T> -pub fn vortex_array::expr::stats::LowerBound::union(&self, other: &Self) -> core::option::Option> +pub fn vortex_array::expr::stats::LowerBound::union(&self, &Self) -> core::option::Option> impl core::cmp::PartialEq for vortex_array::expr::stats::LowerBound -pub fn vortex_array::expr::stats::LowerBound::eq(&self, other: &T) -> bool +pub fn vortex_array::expr::stats::LowerBound::eq(&self, &T) -> bool impl core::cmp::PartialOrd for vortex_array::expr::stats::LowerBound -pub fn vortex_array::expr::stats::LowerBound::partial_cmp(&self, other: &T) -> core::option::Option +pub fn vortex_array::expr::stats::LowerBound::partial_cmp(&self, &T) -> core::option::Option impl core::fmt::Debug for vortex_array::expr::stats::LowerBound -pub fn vortex_array::expr::stats::LowerBound::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::expr::stats::LowerBound::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::StructuralPartialEq for vortex_array::expr::stats::LowerBound @@ -11846,81 +11996,81 @@ impl core::cmp::Eq for vortex_array::expr::stats::UpperBound core::cmp::PartialEq for vortex_array::expr::stats::UpperBound -pub fn vortex_array::expr::stats::UpperBound::eq(&self, other: &vortex_array::expr::stats::UpperBound) -> bool +pub fn vortex_array::expr::stats::UpperBound::eq(&self, &vortex_array::expr::stats::UpperBound) -> bool impl vortex_array::expr::stats::StatBound for vortex_array::expr::stats::UpperBound -pub fn vortex_array::expr::stats::UpperBound::intersection(&self, other: &Self) -> core::option::Option>> +pub fn vortex_array::expr::stats::UpperBound::intersection(&self, &Self) -> core::option::Option>> pub fn vortex_array::expr::stats::UpperBound::into_value(self) -> vortex_array::expr::stats::Precision -pub fn vortex_array::expr::stats::UpperBound::lift(value: vortex_array::expr::stats::Precision) -> Self +pub fn vortex_array::expr::stats::UpperBound::lift(vortex_array::expr::stats::Precision) -> Self pub fn vortex_array::expr::stats::UpperBound::to_exact(&self) -> core::option::Option<&T> -pub fn vortex_array::expr::stats::UpperBound::union(&self, other: &Self) -> core::option::Option> +pub fn vortex_array::expr::stats::UpperBound::union(&self, &Self) -> core::option::Option> impl core::cmp::PartialEq for vortex_array::expr::stats::UpperBound -pub fn vortex_array::expr::stats::UpperBound::eq(&self, other: &T) -> bool +pub fn vortex_array::expr::stats::UpperBound::eq(&self, &T) -> bool impl core::cmp::PartialOrd for vortex_array::expr::stats::UpperBound -pub fn vortex_array::expr::stats::UpperBound::partial_cmp(&self, other: &T) -> core::option::Option +pub fn vortex_array::expr::stats::UpperBound::partial_cmp(&self, &T) -> core::option::Option impl core::fmt::Debug for vortex_array::expr::stats::UpperBound -pub fn vortex_array::expr::stats::UpperBound::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::expr::stats::UpperBound::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::StructuralPartialEq for vortex_array::expr::stats::UpperBound pub trait vortex_array::expr::stats::StatBound: core::marker::Sized -pub fn vortex_array::expr::stats::StatBound::intersection(&self, other: &Self) -> core::option::Option> +pub fn vortex_array::expr::stats::StatBound::intersection(&self, &Self) -> core::option::Option> pub fn vortex_array::expr::stats::StatBound::into_value(self) -> vortex_array::expr::stats::Precision -pub fn vortex_array::expr::stats::StatBound::lift(value: vortex_array::expr::stats::Precision) -> Self +pub fn vortex_array::expr::stats::StatBound::lift(vortex_array::expr::stats::Precision) -> Self pub fn vortex_array::expr::stats::StatBound::to_exact(&self) -> core::option::Option<&T> -pub fn vortex_array::expr::stats::StatBound::union(&self, other: &Self) -> core::option::Option +pub fn vortex_array::expr::stats::StatBound::union(&self, &Self) -> core::option::Option impl vortex_array::expr::stats::StatBound for vortex_array::expr::stats::LowerBound -pub fn vortex_array::expr::stats::LowerBound::intersection(&self, other: &Self) -> core::option::Option>> +pub fn vortex_array::expr::stats::LowerBound::intersection(&self, &Self) -> core::option::Option>> pub fn vortex_array::expr::stats::LowerBound::into_value(self) -> vortex_array::expr::stats::Precision -pub fn vortex_array::expr::stats::LowerBound::lift(value: vortex_array::expr::stats::Precision) -> Self +pub fn vortex_array::expr::stats::LowerBound::lift(vortex_array::expr::stats::Precision) -> Self pub fn vortex_array::expr::stats::LowerBound::to_exact(&self) -> core::option::Option<&T> -pub fn vortex_array::expr::stats::LowerBound::union(&self, other: &Self) -> core::option::Option> +pub fn vortex_array::expr::stats::LowerBound::union(&self, &Self) -> core::option::Option> impl vortex_array::expr::stats::StatBound for vortex_array::expr::stats::Precision -pub fn vortex_array::expr::stats::Precision::intersection(&self, other: &Self) -> core::option::Option> +pub fn vortex_array::expr::stats::Precision::intersection(&self, &Self) -> core::option::Option> pub fn vortex_array::expr::stats::Precision::into_value(self) -> vortex_array::expr::stats::Precision -pub fn vortex_array::expr::stats::Precision::lift(value: vortex_array::expr::stats::Precision) -> Self +pub fn vortex_array::expr::stats::Precision::lift(vortex_array::expr::stats::Precision) -> Self pub fn vortex_array::expr::stats::Precision::to_exact(&self) -> core::option::Option<&T> -pub fn vortex_array::expr::stats::Precision::union(&self, other: &Self) -> core::option::Option +pub fn vortex_array::expr::stats::Precision::union(&self, &Self) -> core::option::Option impl vortex_array::expr::stats::StatBound for vortex_array::expr::stats::UpperBound -pub fn vortex_array::expr::stats::UpperBound::intersection(&self, other: &Self) -> core::option::Option>> +pub fn vortex_array::expr::stats::UpperBound::intersection(&self, &Self) -> core::option::Option>> pub fn vortex_array::expr::stats::UpperBound::into_value(self) -> vortex_array::expr::stats::Precision -pub fn vortex_array::expr::stats::UpperBound::lift(value: vortex_array::expr::stats::Precision) -> Self +pub fn vortex_array::expr::stats::UpperBound::lift(vortex_array::expr::stats::Precision) -> Self pub fn vortex_array::expr::stats::UpperBound::to_exact(&self) -> core::option::Option<&T> -pub fn vortex_array::expr::stats::UpperBound::union(&self, other: &Self) -> core::option::Option> +pub fn vortex_array::expr::stats::UpperBound::union(&self, &Self) -> core::option::Option> pub trait vortex_array::expr::stats::StatType @@ -11984,7 +12134,7 @@ pub const vortex_array::expr::stats::UncompressedSizeInBytes::STAT: vortex_array pub trait vortex_array::expr::stats::StatsProvider -pub fn vortex_array::expr::stats::StatsProvider::get(&self, stat: vortex_array::expr::stats::Stat) -> core::option::Option> +pub fn vortex_array::expr::stats::StatsProvider::get(&self, vortex_array::expr::stats::Stat) -> core::option::Option> pub fn vortex_array::expr::stats::StatsProvider::is_empty(&self) -> bool @@ -11992,7 +12142,7 @@ pub fn vortex_array::expr::stats::StatsProvider::len(&self) -> usize impl vortex_array::expr::stats::StatsProvider for vortex_array::stats::MutTypedStatsSetRef<'_, '_> -pub fn vortex_array::stats::MutTypedStatsSetRef<'_, '_>::get(&self, stat: vortex_array::expr::stats::Stat) -> core::option::Option> +pub fn vortex_array::stats::MutTypedStatsSetRef<'_, '_>::get(&self, vortex_array::expr::stats::Stat) -> core::option::Option> pub fn vortex_array::stats::MutTypedStatsSetRef<'_, '_>::is_empty(&self) -> bool @@ -12000,7 +12150,7 @@ pub fn vortex_array::stats::MutTypedStatsSetRef<'_, '_>::len(&self) -> usize impl vortex_array::expr::stats::StatsProvider for vortex_array::stats::StatsSetRef<'_> -pub fn vortex_array::stats::StatsSetRef<'_>::get(&self, stat: vortex_array::expr::stats::Stat) -> core::option::Option> +pub fn vortex_array::stats::StatsSetRef<'_>::get(&self, vortex_array::expr::stats::Stat) -> core::option::Option> pub fn vortex_array::stats::StatsSetRef<'_>::is_empty(&self) -> bool @@ -12008,7 +12158,7 @@ pub fn vortex_array::stats::StatsSetRef<'_>::len(&self) -> usize impl vortex_array::expr::stats::StatsProvider for vortex_array::stats::TypedStatsSetRef<'_, '_> -pub fn vortex_array::stats::TypedStatsSetRef<'_, '_>::get(&self, stat: vortex_array::expr::stats::Stat) -> core::option::Option> +pub fn vortex_array::stats::TypedStatsSetRef<'_, '_>::get(&self, vortex_array::expr::stats::Stat) -> core::option::Option> pub fn vortex_array::stats::TypedStatsSetRef<'_, '_>::is_empty(&self) -> bool @@ -12016,7 +12166,7 @@ pub fn vortex_array::stats::TypedStatsSetRef<'_, '_>::len(&self) -> usize pub trait vortex_array::expr::stats::StatsProviderExt: vortex_array::expr::stats::StatsProvider -pub fn vortex_array::expr::stats::StatsProviderExt::get_as core::convert::TryFrom<&'a vortex_array::scalar::Scalar, Error = vortex_error::VortexError>>(&self, stat: vortex_array::expr::stats::Stat) -> core::option::Option> +pub fn vortex_array::expr::stats::StatsProviderExt::get_as core::convert::TryFrom<&'a vortex_array::scalar::Scalar, Error = vortex_error::VortexError>>(&self, vortex_array::expr::stats::Stat) -> core::option::Option> pub fn vortex_array::expr::stats::StatsProviderExt::get_as_bound(&self) -> core::option::Option<::Bound> where S: vortex_array::expr::stats::StatType, U: for<'a> core::convert::TryFrom<&'a vortex_array::scalar::Scalar, Error = vortex_error::VortexError> @@ -12024,7 +12174,7 @@ pub fn vortex_array::expr::stats::StatsProviderExt::get_scalar_bound vortex_array::expr::stats::StatsProviderExt for S where S: vortex_array::expr::stats::StatsProvider -pub fn S::get_as core::convert::TryFrom<&'a vortex_array::scalar::Scalar, Error = vortex_error::VortexError>>(&self, stat: vortex_array::expr::stats::Stat) -> core::option::Option> +pub fn S::get_as core::convert::TryFrom<&'a vortex_array::scalar::Scalar, Error = vortex_error::VortexError>>(&self, vortex_array::expr::stats::Stat) -> core::option::Option> pub fn S::get_as_bound(&self) -> core::option::Option<::Bound> where S: vortex_array::expr::stats::StatType, U: for<'a> core::convert::TryFrom<&'a vortex_array::scalar::Scalar, Error = vortex_error::VortexError> @@ -12046,23 +12196,23 @@ pub vortex_array::expr::transform::PartitionedExpr::root: vortex_array::expr::Ex impl vortex_array::expr::transform::PartitionedExpr where vortex_array::dtype::FieldName: core::convert::From -pub fn vortex_array::expr::transform::PartitionedExpr::find_partition(&self, id: &A) -> core::option::Option<&vortex_array::expr::Expression> +pub fn vortex_array::expr::transform::PartitionedExpr::find_partition(&self, &A) -> core::option::Option<&vortex_array::expr::Expression> impl core::fmt::Debug for vortex_array::expr::transform::PartitionedExpr -pub fn vortex_array::expr::transform::PartitionedExpr::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::expr::transform::PartitionedExpr::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::expr::transform::PartitionedExpr -pub fn vortex_array::expr::transform::PartitionedExpr::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::expr::transform::PartitionedExpr::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result -pub fn vortex_array::expr::transform::coerce_expression(expr: vortex_array::expr::Expression, scope: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::expr::transform::coerce_expression(vortex_array::expr::Expression, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::expr::transform::partition(expr: vortex_array::expr::Expression, scope: &vortex_array::dtype::DType, annotate_fn: A) -> vortex_error::VortexResult::Annotation>> where ::Annotation: core::fmt::Display, vortex_array::dtype::FieldName: core::convert::From<::Annotation> +pub fn vortex_array::expr::transform::partition(vortex_array::expr::Expression, &vortex_array::dtype::DType, A) -> vortex_error::VortexResult::Annotation>> where ::Annotation: core::fmt::Display, vortex_array::dtype::FieldName: core::convert::From<::Annotation> -pub fn vortex_array::expr::transform::replace(expr: vortex_array::expr::Expression, needle: &vortex_array::expr::Expression, replacement: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::transform::replace(vortex_array::expr::Expression, &vortex_array::expr::Expression, vortex_array::expr::Expression) -> vortex_array::expr::Expression -pub fn vortex_array::expr::transform::replace_root_fields(expr: vortex_array::expr::Expression, fields: &vortex_array::dtype::StructFields) -> vortex_array::expr::Expression +pub fn vortex_array::expr::transform::replace_root_fields(vortex_array::expr::Expression, &vortex_array::dtype::StructFields) -> vortex_array::expr::Expression pub mod vortex_array::expr::traversal @@ -12076,7 +12226,7 @@ pub vortex_array::expr::traversal::FoldDown::Stop(R) impl core::fmt::Debug for vortex_array::expr::traversal::FoldDown -pub fn vortex_array::expr::traversal::FoldDown::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::expr::traversal::FoldDown::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub enum vortex_array::expr::traversal::FoldDownContext @@ -12088,7 +12238,7 @@ pub vortex_array::expr::traversal::FoldDownContext::Stop(R) impl core::fmt::Debug for vortex_array::expr::traversal::FoldDownContext -pub fn vortex_array::expr::traversal::FoldDownContext::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::expr::traversal::FoldDownContext::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub enum vortex_array::expr::traversal::FoldUp @@ -12102,7 +12252,7 @@ pub fn vortex_array::expr::traversal::FoldUp::value(self) -> R impl core::fmt::Debug for vortex_array::expr::traversal::FoldUp -pub fn vortex_array::expr::traversal::FoldUp::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::expr::traversal::FoldUp::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub enum vortex_array::expr::traversal::TraversalOrder @@ -12114,9 +12264,9 @@ pub vortex_array::expr::traversal::TraversalOrder::Stop impl vortex_array::expr::traversal::TraversalOrder -pub fn vortex_array::expr::traversal::TraversalOrder::visit_children vortex_error::VortexResult>(self, f: F) -> vortex_error::VortexResult +pub fn vortex_array::expr::traversal::TraversalOrder::visit_children vortex_error::VortexResult>(self, F) -> vortex_error::VortexResult -pub fn vortex_array::expr::traversal::TraversalOrder::visit_parent vortex_error::VortexResult>(self, f: F) -> vortex_error::VortexResult +pub fn vortex_array::expr::traversal::TraversalOrder::visit_parent vortex_error::VortexResult>(self, F) -> vortex_error::VortexResult impl core::clone::Clone for vortex_array::expr::traversal::TraversalOrder @@ -12126,11 +12276,11 @@ impl core::cmp::Eq for vortex_array::expr::traversal::TraversalOrder impl core::cmp::PartialEq for vortex_array::expr::traversal::TraversalOrder -pub fn vortex_array::expr::traversal::TraversalOrder::eq(&self, other: &vortex_array::expr::traversal::TraversalOrder) -> bool +pub fn vortex_array::expr::traversal::TraversalOrder::eq(&self, &vortex_array::expr::traversal::TraversalOrder) -> bool impl core::fmt::Debug for vortex_array::expr::traversal::TraversalOrder -pub fn vortex_array::expr::traversal::TraversalOrder::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::expr::traversal::TraversalOrder::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::StructuralPartialEq for vortex_array::expr::traversal::TraversalOrder @@ -12142,7 +12292,7 @@ pub fn vortex_array::expr::traversal::ReferenceCollector::into_fields(self) -> v pub fn vortex_array::expr::traversal::ReferenceCollector::new() -> Self -pub fn vortex_array::expr::traversal::ReferenceCollector::with_set(set: vortex_utils::aliases::hash_set::HashSet) -> Self +pub fn vortex_array::expr::traversal::ReferenceCollector::with_set(vortex_utils::aliases::hash_set::HashSet) -> Self impl core::default::Default for vortex_array::expr::traversal::ReferenceCollector @@ -12152,9 +12302,9 @@ impl vortex_array::expr::traversal::NodeVisitor<'_> for vortex_array::expr::trav pub type vortex_array::expr::traversal::ReferenceCollector::NodeTy = vortex_array::expr::Expression -pub fn vortex_array::expr::traversal::ReferenceCollector::visit_down(&mut self, node: &'a Self::NodeTy) -> vortex_error::VortexResult +pub fn vortex_array::expr::traversal::ReferenceCollector::visit_down(&mut self, &'a Self::NodeTy) -> vortex_error::VortexResult -pub fn vortex_array::expr::traversal::ReferenceCollector::visit_up(&mut self, node: &vortex_array::expr::Expression) -> vortex_error::VortexResult +pub fn vortex_array::expr::traversal::ReferenceCollector::visit_up(&mut self, &vortex_array::expr::Expression) -> vortex_error::VortexResult pub struct vortex_array::expr::traversal::Transformed @@ -12168,11 +12318,11 @@ impl vortex_array::expr::traversal::Transformed pub fn vortex_array::expr::traversal::Transformed::into_inner(self) -> T -pub fn vortex_array::expr::traversal::Transformed::map O>(self, f: F) -> vortex_array::expr::traversal::Transformed +pub fn vortex_array::expr::traversal::Transformed::map O>(self, F) -> vortex_array::expr::traversal::Transformed -pub fn vortex_array::expr::traversal::Transformed::no(value: T) -> Self +pub fn vortex_array::expr::traversal::Transformed::no(T) -> Self -pub fn vortex_array::expr::traversal::Transformed::yes(value: T) -> Self +pub fn vortex_array::expr::traversal::Transformed::yes(T) -> Self impl core::clone::Clone for vortex_array::expr::traversal::Transformed @@ -12180,95 +12330,95 @@ pub fn vortex_array::expr::traversal::Transformed::clone(&self) -> vortex_arr impl core::fmt::Debug for vortex_array::expr::traversal::Transformed -pub fn vortex_array::expr::traversal::Transformed::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::expr::traversal::Transformed::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub trait vortex_array::expr::traversal::Node: core::marker::Sized + core::clone::Clone -pub fn vortex_array::expr::traversal::Node::apply_children<'a, F: core::ops::function::FnMut(&'a Self) -> vortex_error::VortexResult>(&'a self, f: F) -> vortex_error::VortexResult +pub fn vortex_array::expr::traversal::Node::apply_children<'a, F: core::ops::function::FnMut(&'a Self) -> vortex_error::VortexResult>(&'a self, F) -> vortex_error::VortexResult pub fn vortex_array::expr::traversal::Node::children_count(&self) -> usize -pub fn vortex_array::expr::traversal::Node::iter_children(&self, f: impl core::ops::function::FnOnce(&mut dyn core::iter::traits::iterator::Iterator) -> T) -> T +pub fn vortex_array::expr::traversal::Node::iter_children(&self, impl core::ops::function::FnOnce(&mut dyn core::iter::traits::iterator::Iterator) -> T) -> T -pub fn vortex_array::expr::traversal::Node::map_children vortex_error::VortexResult>>(self, f: F) -> vortex_error::VortexResult> +pub fn vortex_array::expr::traversal::Node::map_children vortex_error::VortexResult>>(self, F) -> vortex_error::VortexResult> impl vortex_array::expr::traversal::Node for vortex_array::expr::Expression -pub fn vortex_array::expr::Expression::apply_children<'a, F: core::ops::function::FnMut(&'a Self) -> vortex_error::VortexResult>(&'a self, f: F) -> vortex_error::VortexResult +pub fn vortex_array::expr::Expression::apply_children<'a, F: core::ops::function::FnMut(&'a Self) -> vortex_error::VortexResult>(&'a self, F) -> vortex_error::VortexResult pub fn vortex_array::expr::Expression::children_count(&self) -> usize -pub fn vortex_array::expr::Expression::iter_children(&self, f: impl core::ops::function::FnOnce(&mut dyn core::iter::traits::iterator::Iterator) -> T) -> T +pub fn vortex_array::expr::Expression::iter_children(&self, impl core::ops::function::FnOnce(&mut dyn core::iter::traits::iterator::Iterator) -> T) -> T -pub fn vortex_array::expr::Expression::map_children vortex_error::VortexResult>>(self, f: F) -> vortex_error::VortexResult> +pub fn vortex_array::expr::Expression::map_children vortex_error::VortexResult>>(self, F) -> vortex_error::VortexResult> pub trait vortex_array::expr::traversal::NodeContainer<'a, T: 'a>: core::marker::Sized -pub fn vortex_array::expr::traversal::NodeContainer::apply_elements vortex_error::VortexResult>(&'a self, f: F) -> vortex_error::VortexResult +pub fn vortex_array::expr::traversal::NodeContainer::apply_elements vortex_error::VortexResult>(&'a self, F) -> vortex_error::VortexResult -pub fn vortex_array::expr::traversal::NodeContainer::map_elements vortex_error::VortexResult>>(self, f: F) -> vortex_error::VortexResult> +pub fn vortex_array::expr::traversal::NodeContainer::map_elements vortex_error::VortexResult>>(self, F) -> vortex_error::VortexResult> impl<'a, T, C> vortex_array::expr::traversal::NodeContainer<'a, T> for alloc::sync::Arc where T: 'a, C: vortex_array::expr::traversal::NodeContainer<'a, T> + core::clone::Clone -pub fn alloc::sync::Arc::apply_elements vortex_error::VortexResult>(&'a self, f: F) -> vortex_error::VortexResult +pub fn alloc::sync::Arc::apply_elements vortex_error::VortexResult>(&'a self, F) -> vortex_error::VortexResult -pub fn alloc::sync::Arc::map_elements vortex_error::VortexResult>>(self, f: F) -> vortex_error::VortexResult>> +pub fn alloc::sync::Arc::map_elements vortex_error::VortexResult>>(self, F) -> vortex_error::VortexResult>> impl<'a, T: 'a, C: vortex_array::expr::traversal::NodeContainer<'a, T>> vortex_array::expr::traversal::NodeContainer<'a, T> for [C; 2] -pub fn [C; 2]::apply_elements vortex_error::VortexResult>(&'a self, f: F) -> vortex_error::VortexResult +pub fn [C; 2]::apply_elements vortex_error::VortexResult>(&'a self, F) -> vortex_error::VortexResult -pub fn [C; 2]::map_elements vortex_error::VortexResult>>(self, f: F) -> vortex_error::VortexResult> +pub fn [C; 2]::map_elements vortex_error::VortexResult>>(self, F) -> vortex_error::VortexResult> impl<'a, T: 'a, C: vortex_array::expr::traversal::NodeContainer<'a, T>> vortex_array::expr::traversal::NodeContainer<'a, T> for alloc::boxed::Box -pub fn alloc::boxed::Box::apply_elements vortex_error::VortexResult>(&'a self, f: F) -> vortex_error::VortexResult +pub fn alloc::boxed::Box::apply_elements vortex_error::VortexResult>(&'a self, F) -> vortex_error::VortexResult -pub fn alloc::boxed::Box::map_elements vortex_error::VortexResult>>(self, f: F) -> vortex_error::VortexResult>> +pub fn alloc::boxed::Box::map_elements vortex_error::VortexResult>>(self, F) -> vortex_error::VortexResult>> impl<'a, T: 'a, C: vortex_array::expr::traversal::NodeContainer<'a, T>> vortex_array::expr::traversal::NodeContainer<'a, T> for alloc::vec::Vec -pub fn alloc::vec::Vec::apply_elements vortex_error::VortexResult>(&'a self, f: F) -> vortex_error::VortexResult +pub fn alloc::vec::Vec::apply_elements vortex_error::VortexResult>(&'a self, F) -> vortex_error::VortexResult -pub fn alloc::vec::Vec::map_elements vortex_error::VortexResult>>(self, f: F) -> vortex_error::VortexResult> +pub fn alloc::vec::Vec::map_elements vortex_error::VortexResult>>(self, F) -> vortex_error::VortexResult> impl<'a> vortex_array::expr::traversal::NodeContainer<'a, vortex_array::expr::Expression> for vortex_array::expr::Expression -pub fn vortex_array::expr::Expression::apply_elements vortex_error::VortexResult>(&'a self, f: F) -> vortex_error::VortexResult +pub fn vortex_array::expr::Expression::apply_elements vortex_error::VortexResult>(&'a self, F) -> vortex_error::VortexResult -pub fn vortex_array::expr::Expression::map_elements vortex_error::VortexResult>>(self, f: F) -> vortex_error::VortexResult> +pub fn vortex_array::expr::Expression::map_elements vortex_error::VortexResult>>(self, F) -> vortex_error::VortexResult> pub trait vortex_array::expr::traversal::NodeExt: vortex_array::expr::traversal::Node -pub fn vortex_array::expr::traversal::NodeExt::accept<'a, V: vortex_array::expr::traversal::NodeVisitor<'a, NodeTy = Self>>(&'a self, visitor: &mut V) -> vortex_error::VortexResult +pub fn vortex_array::expr::traversal::NodeExt::accept<'a, V: vortex_array::expr::traversal::NodeVisitor<'a, NodeTy = Self>>(&'a self, &mut V) -> vortex_error::VortexResult -pub fn vortex_array::expr::traversal::NodeExt::fold>(self, folder: &mut F) -> vortex_error::VortexResult> +pub fn vortex_array::expr::traversal::NodeExt::fold>(self, &mut F) -> vortex_error::VortexResult> -pub fn vortex_array::expr::traversal::NodeExt::fold_context>(self, ctx: &::Context, folder: &mut F) -> vortex_error::VortexResult> +pub fn vortex_array::expr::traversal::NodeExt::fold_context>(self, &::Context, &mut F) -> vortex_error::VortexResult> -pub fn vortex_array::expr::traversal::NodeExt::rewrite>(self, rewriter: &mut R) -> vortex_error::VortexResult> +pub fn vortex_array::expr::traversal::NodeExt::rewrite>(self, &mut R) -> vortex_error::VortexResult> -pub fn vortex_array::expr::traversal::NodeExt::transform(self, down: F, up: G) -> vortex_error::VortexResult> where F: core::ops::function::FnMut(Self) -> vortex_error::VortexResult>, G: core::ops::function::FnMut(Self) -> vortex_error::VortexResult> +pub fn vortex_array::expr::traversal::NodeExt::transform(self, F, G) -> vortex_error::VortexResult> where F: core::ops::function::FnMut(Self) -> vortex_error::VortexResult>, G: core::ops::function::FnMut(Self) -> vortex_error::VortexResult> -pub fn vortex_array::expr::traversal::NodeExt::transform_down vortex_error::VortexResult>>(self, f: F) -> vortex_error::VortexResult> +pub fn vortex_array::expr::traversal::NodeExt::transform_down vortex_error::VortexResult>>(self, F) -> vortex_error::VortexResult> -pub fn vortex_array::expr::traversal::NodeExt::transform_up vortex_error::VortexResult>>(self, f: F) -> vortex_error::VortexResult> +pub fn vortex_array::expr::traversal::NodeExt::transform_up vortex_error::VortexResult>>(self, F) -> vortex_error::VortexResult> impl vortex_array::expr::traversal::NodeExt for T -pub fn T::accept<'a, V: vortex_array::expr::traversal::NodeVisitor<'a, NodeTy = Self>>(&'a self, visitor: &mut V) -> vortex_error::VortexResult +pub fn T::accept<'a, V: vortex_array::expr::traversal::NodeVisitor<'a, NodeTy = Self>>(&'a self, &mut V) -> vortex_error::VortexResult -pub fn T::fold>(self, folder: &mut F) -> vortex_error::VortexResult> +pub fn T::fold>(self, &mut F) -> vortex_error::VortexResult> -pub fn T::fold_context>(self, ctx: &::Context, folder: &mut F) -> vortex_error::VortexResult> +pub fn T::fold_context>(self, &::Context, &mut F) -> vortex_error::VortexResult> -pub fn T::rewrite>(self, rewriter: &mut R) -> vortex_error::VortexResult> +pub fn T::rewrite>(self, &mut R) -> vortex_error::VortexResult> -pub fn T::transform(self, down: F, up: G) -> vortex_error::VortexResult> where F: core::ops::function::FnMut(Self) -> vortex_error::VortexResult>, G: core::ops::function::FnMut(Self) -> vortex_error::VortexResult> +pub fn T::transform(self, F, G) -> vortex_error::VortexResult> where F: core::ops::function::FnMut(Self) -> vortex_error::VortexResult>, G: core::ops::function::FnMut(Self) -> vortex_error::VortexResult> -pub fn T::transform_down vortex_error::VortexResult>>(self, f: F) -> vortex_error::VortexResult> +pub fn T::transform_down vortex_error::VortexResult>>(self, F) -> vortex_error::VortexResult> -pub fn T::transform_up vortex_error::VortexResult>>(self, f: F) -> vortex_error::VortexResult> +pub fn T::transform_up vortex_error::VortexResult>>(self, F) -> vortex_error::VortexResult> pub trait vortex_array::expr::traversal::NodeFolder @@ -12276,9 +12426,9 @@ pub type vortex_array::expr::traversal::NodeFolder::NodeTy: vortex_array::expr:: pub type vortex_array::expr::traversal::NodeFolder::Result -pub fn vortex_array::expr::traversal::NodeFolder::visit_down(&mut self, _node: &Self::NodeTy) -> vortex_error::VortexResult> +pub fn vortex_array::expr::traversal::NodeFolder::visit_down(&mut self, &Self::NodeTy) -> vortex_error::VortexResult> -pub fn vortex_array::expr::traversal::NodeFolder::visit_up(&mut self, _node: Self::NodeTy, _children: alloc::vec::Vec) -> vortex_error::VortexResult> +pub fn vortex_array::expr::traversal::NodeFolder::visit_up(&mut self, Self::NodeTy, alloc::vec::Vec) -> vortex_error::VortexResult> pub trait vortex_array::expr::traversal::NodeFolderContext @@ -12288,45 +12438,45 @@ pub type vortex_array::expr::traversal::NodeFolderContext::NodeTy: vortex_array: pub type vortex_array::expr::traversal::NodeFolderContext::Result -pub fn vortex_array::expr::traversal::NodeFolderContext::visit_down(&mut self, _ctx: &Self::Context, _node: &Self::NodeTy) -> vortex_error::VortexResult> +pub fn vortex_array::expr::traversal::NodeFolderContext::visit_down(&mut self, &Self::Context, &Self::NodeTy) -> vortex_error::VortexResult> -pub fn vortex_array::expr::traversal::NodeFolderContext::visit_up(&mut self, _node: Self::NodeTy, _context: &Self::Context, _children: alloc::vec::Vec) -> vortex_error::VortexResult> +pub fn vortex_array::expr::traversal::NodeFolderContext::visit_up(&mut self, Self::NodeTy, &Self::Context, alloc::vec::Vec) -> vortex_error::VortexResult> pub trait vortex_array::expr::traversal::NodeRefContainer<'a, T: 'a>: core::marker::Sized -pub fn vortex_array::expr::traversal::NodeRefContainer::apply_ref_elements vortex_error::VortexResult>(&self, f: F) -> vortex_error::VortexResult +pub fn vortex_array::expr::traversal::NodeRefContainer::apply_ref_elements vortex_error::VortexResult>(&self, F) -> vortex_error::VortexResult impl<'a, T: 'a, C: vortex_array::expr::traversal::NodeContainer<'a, T>> vortex_array::expr::traversal::NodeRefContainer<'a, T> for &'a [C] -pub fn &'a [C]::apply_ref_elements vortex_error::VortexResult>(&self, f: F) -> vortex_error::VortexResult +pub fn &'a [C]::apply_ref_elements vortex_error::VortexResult>(&self, F) -> vortex_error::VortexResult pub trait vortex_array::expr::traversal::NodeRewriter: core::marker::Sized pub type vortex_array::expr::traversal::NodeRewriter::NodeTy: vortex_array::expr::traversal::Node -pub fn vortex_array::expr::traversal::NodeRewriter::visit_down(&mut self, node: Self::NodeTy) -> vortex_error::VortexResult> +pub fn vortex_array::expr::traversal::NodeRewriter::visit_down(&mut self, Self::NodeTy) -> vortex_error::VortexResult> -pub fn vortex_array::expr::traversal::NodeRewriter::visit_up(&mut self, node: Self::NodeTy) -> vortex_error::VortexResult> +pub fn vortex_array::expr::traversal::NodeRewriter::visit_up(&mut self, Self::NodeTy) -> vortex_error::VortexResult> pub trait vortex_array::expr::traversal::NodeVisitor<'a> pub type vortex_array::expr::traversal::NodeVisitor::NodeTy: vortex_array::expr::traversal::Node -pub fn vortex_array::expr::traversal::NodeVisitor::visit_down(&mut self, node: &'a Self::NodeTy) -> vortex_error::VortexResult +pub fn vortex_array::expr::traversal::NodeVisitor::visit_down(&mut self, &'a Self::NodeTy) -> vortex_error::VortexResult -pub fn vortex_array::expr::traversal::NodeVisitor::visit_up(&mut self, node: &'a Self::NodeTy) -> vortex_error::VortexResult +pub fn vortex_array::expr::traversal::NodeVisitor::visit_up(&mut self, &'a Self::NodeTy) -> vortex_error::VortexResult impl vortex_array::expr::traversal::NodeVisitor<'_> for vortex_array::expr::traversal::ReferenceCollector pub type vortex_array::expr::traversal::ReferenceCollector::NodeTy = vortex_array::expr::Expression -pub fn vortex_array::expr::traversal::ReferenceCollector::visit_down(&mut self, node: &'a Self::NodeTy) -> vortex_error::VortexResult +pub fn vortex_array::expr::traversal::ReferenceCollector::visit_down(&mut self, &'a Self::NodeTy) -> vortex_error::VortexResult -pub fn vortex_array::expr::traversal::ReferenceCollector::visit_up(&mut self, node: &vortex_array::expr::Expression) -> vortex_error::VortexResult +pub fn vortex_array::expr::traversal::ReferenceCollector::visit_up(&mut self, &vortex_array::expr::Expression) -> vortex_error::VortexResult -pub fn vortex_array::expr::traversal::pre_order_visit_down<'a, T: 'a + vortex_array::expr::traversal::Node>(tree: &'a T, f: impl core::ops::function::FnMut(&'a T) -> vortex_error::VortexResult) -> vortex_error::VortexResult<()> +pub fn vortex_array::expr::traversal::pre_order_visit_down<'a, T: 'a + vortex_array::expr::traversal::Node>(&'a T, impl core::ops::function::FnMut(&'a T) -> vortex_error::VortexResult) -> vortex_error::VortexResult<()> -pub fn vortex_array::expr::traversal::pre_order_visit_up<'a, T: 'a + vortex_array::expr::traversal::Node>(tree: &'a T, f: impl core::ops::function::FnMut(&'a T) -> vortex_error::VortexResult) -> vortex_error::VortexResult<()> +pub fn vortex_array::expr::traversal::pre_order_visit_up<'a, T: 'a + vortex_array::expr::traversal::Node>(&'a T, impl core::ops::function::FnMut(&'a T) -> vortex_error::VortexResult) -> vortex_error::VortexResult<()> pub struct vortex_array::expr::ExactExpr(pub vortex_array::expr::Expression) @@ -12338,59 +12488,59 @@ impl core::cmp::Eq for vortex_array::expr::ExactExpr impl core::cmp::PartialEq for vortex_array::expr::ExactExpr -pub fn vortex_array::expr::ExactExpr::eq(&self, other: &Self) -> bool +pub fn vortex_array::expr::ExactExpr::eq(&self, &Self) -> bool impl core::hash::Hash for vortex_array::expr::ExactExpr -pub fn vortex_array::expr::ExactExpr::hash(&self, state: &mut H) +pub fn vortex_array::expr::ExactExpr::hash(&self, &mut H) pub struct vortex_array::expr::Expression impl vortex_array::expr::Expression -pub fn vortex_array::expr::Expression::child(&self, n: usize) -> &vortex_array::expr::Expression +pub fn vortex_array::expr::Expression::child(&self, usize) -> &vortex_array::expr::Expression pub fn vortex_array::expr::Expression::children(&self) -> &alloc::sync::Arc> pub fn vortex_array::expr::Expression::display_tree(&self) -> impl core::fmt::Display -pub fn vortex_array::expr::Expression::fmt_sql(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::expr::Expression::fmt_sql(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result -pub fn vortex_array::expr::Expression::return_dtype(&self, scope: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::expr::Expression::return_dtype(&self, &vortex_array::dtype::DType) -> vortex_error::VortexResult pub fn vortex_array::expr::Expression::scalar_fn(&self) -> &vortex_array::scalar_fn::ScalarFnRef -pub fn vortex_array::expr::Expression::stat_expression(&self, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::expr::Expression::stat_expression(&self, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::expr::Expression::stat_falsification(&self, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::expr::Expression::stat_falsification(&self, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::expr::Expression::stat_max(&self, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::expr::Expression::stat_max(&self, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::expr::Expression::stat_min(&self, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::expr::Expression::stat_min(&self, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::expr::Expression::try_new(scalar_fn: vortex_array::scalar_fn::ScalarFnRef, children: impl core::iter::traits::collect::IntoIterator) -> vortex_error::VortexResult +pub fn vortex_array::expr::Expression::try_new(vortex_array::scalar_fn::ScalarFnRef, impl core::iter::traits::collect::IntoIterator) -> vortex_error::VortexResult pub fn vortex_array::expr::Expression::validity(&self) -> vortex_error::VortexResult -pub fn vortex_array::expr::Expression::with_children(self, children: impl core::iter::traits::collect::IntoIterator) -> vortex_error::VortexResult +pub fn vortex_array::expr::Expression::with_children(self, impl core::iter::traits::collect::IntoIterator) -> vortex_error::VortexResult impl vortex_array::expr::Expression -pub fn vortex_array::expr::Expression::from_proto(expr: &vortex_proto::expr::Expr, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::expr::Expression::from_proto(&vortex_proto::expr::Expr, &vortex_session::VortexSession) -> vortex_error::VortexResult impl vortex_array::expr::Expression -pub fn vortex_array::expr::Expression::optimize(&self, scope: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::expr::Expression::optimize(&self, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::expr::Expression::optimize_recursive(&self, scope: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::expr::Expression::optimize_recursive(&self, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::expr::Expression::simplify(&self, scope: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::expr::Expression::simplify(&self, &vortex_array::dtype::DType) -> vortex_error::VortexResult pub fn vortex_array::expr::Expression::simplify_untyped(&self) -> vortex_error::VortexResult -pub fn vortex_array::expr::Expression::try_optimize(&self, scope: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::expr::Expression::try_optimize(&self, &vortex_array::dtype::DType) -> vortex_error::VortexResult> -pub fn vortex_array::expr::Expression::try_optimize_recursive(&self, scope: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::expr::Expression::try_optimize_recursive(&self, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl core::clone::Clone for vortex_array::expr::Expression @@ -12400,19 +12550,19 @@ impl core::cmp::Eq for vortex_array::expr::Expression impl core::cmp::PartialEq for vortex_array::expr::Expression -pub fn vortex_array::expr::Expression::eq(&self, other: &vortex_array::expr::Expression) -> bool +pub fn vortex_array::expr::Expression::eq(&self, &vortex_array::expr::Expression) -> bool impl core::fmt::Debug for vortex_array::expr::Expression -pub fn vortex_array::expr::Expression::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::expr::Expression::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::expr::Expression -pub fn vortex_array::expr::Expression::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::expr::Expression::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::expr::Expression -pub fn vortex_array::expr::Expression::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::expr::Expression::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_array::expr::Expression @@ -12428,23 +12578,25 @@ pub fn vortex_array::expr::Expression::drop(&mut self) impl vortex_array::builtins::ExprBuiltins for vortex_array::expr::Expression -pub fn vortex_array::expr::Expression::binary(&self, rhs: vortex_array::expr::Expression, op: vortex_array::scalar_fn::fns::operators::Operator) -> vortex_error::VortexResult +pub fn vortex_array::expr::Expression::binary(&self, vortex_array::expr::Expression, vortex_array::scalar_fn::fns::operators::Operator) -> vortex_error::VortexResult -pub fn vortex_array::expr::Expression::cast(&self, dtype: vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::expr::Expression::cast(&self, vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::expr::Expression::fill_null(&self, fill_value: vortex_array::expr::Expression) -> vortex_error::VortexResult +pub fn vortex_array::expr::Expression::fill_null(&self, vortex_array::expr::Expression) -> vortex_error::VortexResult -pub fn vortex_array::expr::Expression::get_item(&self, field_name: impl core::convert::Into) -> vortex_error::VortexResult +pub fn vortex_array::expr::Expression::get_item(&self, impl core::convert::Into) -> vortex_error::VortexResult + +pub fn vortex_array::expr::Expression::is_not_null(&self) -> vortex_error::VortexResult pub fn vortex_array::expr::Expression::is_null(&self) -> vortex_error::VortexResult -pub fn vortex_array::expr::Expression::list_contains(&self, value: vortex_array::expr::Expression) -> vortex_error::VortexResult +pub fn vortex_array::expr::Expression::list_contains(&self, vortex_array::expr::Expression) -> vortex_error::VortexResult -pub fn vortex_array::expr::Expression::mask(&self, mask: vortex_array::expr::Expression) -> vortex_error::VortexResult +pub fn vortex_array::expr::Expression::mask(&self, vortex_array::expr::Expression) -> vortex_error::VortexResult pub fn vortex_array::expr::Expression::not(&self) -> vortex_error::VortexResult -pub fn vortex_array::expr::Expression::zip(&self, if_true: vortex_array::expr::Expression, if_false: vortex_array::expr::Expression) -> vortex_error::VortexResult +pub fn vortex_array::expr::Expression::zip(&self, vortex_array::expr::Expression, vortex_array::expr::Expression) -> vortex_error::VortexResult impl vortex_array::expr::VortexExprExt for vortex_array::expr::Expression @@ -12456,19 +12608,19 @@ pub fn vortex_array::expr::Expression::serialize_proto(&self) -> vortex_error::V impl vortex_array::expr::traversal::Node for vortex_array::expr::Expression -pub fn vortex_array::expr::Expression::apply_children<'a, F: core::ops::function::FnMut(&'a Self) -> vortex_error::VortexResult>(&'a self, f: F) -> vortex_error::VortexResult +pub fn vortex_array::expr::Expression::apply_children<'a, F: core::ops::function::FnMut(&'a Self) -> vortex_error::VortexResult>(&'a self, F) -> vortex_error::VortexResult pub fn vortex_array::expr::Expression::children_count(&self) -> usize -pub fn vortex_array::expr::Expression::iter_children(&self, f: impl core::ops::function::FnOnce(&mut dyn core::iter::traits::iterator::Iterator) -> T) -> T +pub fn vortex_array::expr::Expression::iter_children(&self, impl core::ops::function::FnOnce(&mut dyn core::iter::traits::iterator::Iterator) -> T) -> T -pub fn vortex_array::expr::Expression::map_children vortex_error::VortexResult>>(self, f: F) -> vortex_error::VortexResult> +pub fn vortex_array::expr::Expression::map_children vortex_error::VortexResult>>(self, F) -> vortex_error::VortexResult> impl<'a> vortex_array::expr::traversal::NodeContainer<'a, vortex_array::expr::Expression> for vortex_array::expr::Expression -pub fn vortex_array::expr::Expression::apply_elements vortex_error::VortexResult>(&'a self, f: F) -> vortex_error::VortexResult +pub fn vortex_array::expr::Expression::apply_elements vortex_error::VortexResult>(&'a self, F) -> vortex_error::VortexResult -pub fn vortex_array::expr::Expression::map_elements vortex_error::VortexResult>>(self, f: F) -> vortex_error::VortexResult> +pub fn vortex_array::expr::Expression::map_elements vortex_error::VortexResult>>(self, F) -> vortex_error::VortexResult> pub trait vortex_array::expr::Annotation: core::clone::Clone + core::hash::Hash + core::cmp::Eq @@ -12484,7 +12636,7 @@ pub type F::Annotation = A pub trait vortex_array::expr::StatsCatalog -pub fn vortex_array::expr::StatsCatalog::stats_ref(&self, _field_path: &vortex_array::dtype::FieldPath, _stat: vortex_array::expr::stats::Stat) -> core::option::Option +pub fn vortex_array::expr::StatsCatalog::stats_ref(&self, &vortex_array::dtype::FieldPath, vortex_array::expr::stats::Stat) -> core::option::Option pub trait vortex_array::expr::VortexExprExt @@ -12494,95 +12646,97 @@ impl vortex_array::expr::VortexExprExt for vortex_array::expr::Expression pub fn vortex_array::expr::Expression::field_references(&self) -> vortex_utils::aliases::hash_set::HashSet -pub fn vortex_array::expr::and(lhs: vortex_array::expr::Expression, rhs: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::and(vortex_array::expr::Expression, vortex_array::expr::Expression) -> vortex_array::expr::Expression + +pub fn vortex_array::expr::and_collect(I) -> core::option::Option where I: core::iter::traits::collect::IntoIterator -pub fn vortex_array::expr::and_collect(iter: I) -> core::option::Option where I: core::iter::traits::collect::IntoIterator +pub fn vortex_array::expr::between(vortex_array::expr::Expression, vortex_array::expr::Expression, vortex_array::expr::Expression, vortex_array::scalar_fn::fns::between::BetweenOptions) -> vortex_array::expr::Expression -pub fn vortex_array::expr::between(arr: vortex_array::expr::Expression, lower: vortex_array::expr::Expression, upper: vortex_array::expr::Expression, options: vortex_array::scalar_fn::fns::between::BetweenOptions) -> vortex_array::expr::Expression +pub fn vortex_array::expr::case_when(vortex_array::expr::Expression, vortex_array::expr::Expression, vortex_array::expr::Expression) -> vortex_array::expr::Expression -pub fn vortex_array::expr::case_when(condition: vortex_array::expr::Expression, then_value: vortex_array::expr::Expression, else_value: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::case_when_no_else(vortex_array::expr::Expression, vortex_array::expr::Expression) -> vortex_array::expr::Expression -pub fn vortex_array::expr::case_when_no_else(condition: vortex_array::expr::Expression, then_value: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::cast(vortex_array::expr::Expression, vortex_array::dtype::DType) -> vortex_array::expr::Expression -pub fn vortex_array::expr::cast(child: vortex_array::expr::Expression, target: vortex_array::dtype::DType) -> vortex_array::expr::Expression +pub fn vortex_array::expr::checked_add(vortex_array::expr::Expression, vortex_array::expr::Expression) -> vortex_array::expr::Expression -pub fn vortex_array::expr::checked_add(lhs: vortex_array::expr::Expression, rhs: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::col(impl core::convert::Into) -> vortex_array::expr::Expression -pub fn vortex_array::expr::col(field: impl core::convert::Into) -> vortex_array::expr::Expression +pub fn vortex_array::expr::descendent_annotations(&vortex_array::expr::Expression, A) -> vortex_array::expr::Annotations<'_, ::Annotation> -pub fn vortex_array::expr::descendent_annotations(expr: &vortex_array::expr::Expression, annotate: A) -> vortex_array::expr::Annotations<'_, ::Annotation> +pub fn vortex_array::expr::dynamic(vortex_array::scalar_fn::fns::operators::CompareOperator, impl core::ops::function::Fn() -> core::option::Option + core::marker::Send + core::marker::Sync + 'static, vortex_array::dtype::DType, bool, vortex_array::expr::Expression) -> vortex_array::expr::Expression -pub fn vortex_array::expr::dynamic(operator: vortex_array::scalar_fn::fns::operators::CompareOperator, rhs_value: impl core::ops::function::Fn() -> core::option::Option + core::marker::Send + core::marker::Sync + 'static, rhs_dtype: vortex_array::dtype::DType, default: bool, lhs: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::eq(vortex_array::expr::Expression, vortex_array::expr::Expression) -> vortex_array::expr::Expression -pub fn vortex_array::expr::eq(lhs: vortex_array::expr::Expression, rhs: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::fill_null(vortex_array::expr::Expression, vortex_array::expr::Expression) -> vortex_array::expr::Expression -pub fn vortex_array::expr::fill_null(child: vortex_array::expr::Expression, fill_value: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::get_item(impl core::convert::Into, vortex_array::expr::Expression) -> vortex_array::expr::Expression -pub fn vortex_array::expr::get_item(field: impl core::convert::Into, child: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::gt(vortex_array::expr::Expression, vortex_array::expr::Expression) -> vortex_array::expr::Expression -pub fn vortex_array::expr::gt(lhs: vortex_array::expr::Expression, rhs: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::gt_eq(vortex_array::expr::Expression, vortex_array::expr::Expression) -> vortex_array::expr::Expression -pub fn vortex_array::expr::gt_eq(lhs: vortex_array::expr::Expression, rhs: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::ilike(vortex_array::expr::Expression, vortex_array::expr::Expression) -> vortex_array::expr::Expression -pub fn vortex_array::expr::ilike(child: vortex_array::expr::Expression, pattern: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::immediate_scope_access<'a>(&'a vortex_array::expr::Expression, &'a vortex_array::dtype::StructFields) -> vortex_utils::aliases::hash_set::HashSet -pub fn vortex_array::expr::immediate_scope_access<'a>(expr: &'a vortex_array::expr::Expression, scope: &'a vortex_array::dtype::StructFields) -> vortex_utils::aliases::hash_set::HashSet +pub fn vortex_array::expr::immediate_scope_accesses<'a>(&'a vortex_array::expr::Expression, &'a vortex_array::dtype::StructFields) -> vortex_array::expr::FieldAccesses<'a> -pub fn vortex_array::expr::immediate_scope_accesses<'a>(expr: &'a vortex_array::expr::Expression, scope: &'a vortex_array::dtype::StructFields) -> vortex_array::expr::FieldAccesses<'a> +pub fn vortex_array::expr::is_not_null(vortex_array::expr::Expression) -> vortex_array::expr::Expression -pub fn vortex_array::expr::is_null(child: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::is_null(vortex_array::expr::Expression) -> vortex_array::expr::Expression -pub fn vortex_array::expr::is_root(expr: &vortex_array::expr::Expression) -> bool +pub fn vortex_array::expr::is_root(&vortex_array::expr::Expression) -> bool -pub fn vortex_array::expr::label_is_fallible(expr: &vortex_array::expr::Expression) -> vortex_array::expr::BooleanLabels<'_> +pub fn vortex_array::expr::label_is_fallible(&vortex_array::expr::Expression) -> vortex_array::expr::BooleanLabels<'_> -pub fn vortex_array::expr::label_null_sensitive(expr: &vortex_array::expr::Expression) -> vortex_array::expr::BooleanLabels<'_> +pub fn vortex_array::expr::label_null_sensitive(&vortex_array::expr::Expression) -> vortex_array::expr::BooleanLabels<'_> -pub fn vortex_array::expr::label_tree(expr: &vortex_array::expr::Expression, self_label: impl core::ops::function::Fn(&vortex_array::expr::Expression) -> L, merge_child: impl core::ops::function::FnMut(L, &L) -> L) -> vortex_utils::aliases::hash_map::HashMap<&vortex_array::expr::Expression, L> +pub fn vortex_array::expr::label_tree(&vortex_array::expr::Expression, impl core::ops::function::Fn(&vortex_array::expr::Expression) -> L, impl core::ops::function::FnMut(L, &L) -> L) -> vortex_utils::aliases::hash_map::HashMap<&vortex_array::expr::Expression, L> -pub fn vortex_array::expr::like(child: vortex_array::expr::Expression, pattern: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::like(vortex_array::expr::Expression, vortex_array::expr::Expression) -> vortex_array::expr::Expression -pub fn vortex_array::expr::list_contains(list: vortex_array::expr::Expression, value: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::list_contains(vortex_array::expr::Expression, vortex_array::expr::Expression) -> vortex_array::expr::Expression -pub fn vortex_array::expr::lit(value: impl core::convert::Into) -> vortex_array::expr::Expression +pub fn vortex_array::expr::lit(impl core::convert::Into) -> vortex_array::expr::Expression -pub fn vortex_array::expr::lt(lhs: vortex_array::expr::Expression, rhs: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::lt(vortex_array::expr::Expression, vortex_array::expr::Expression) -> vortex_array::expr::Expression -pub fn vortex_array::expr::lt_eq(lhs: vortex_array::expr::Expression, rhs: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::lt_eq(vortex_array::expr::Expression, vortex_array::expr::Expression) -> vortex_array::expr::Expression -pub fn vortex_array::expr::make_free_field_annotator(scope: &vortex_array::dtype::StructFields) -> impl vortex_array::expr::AnnotationFn +pub fn vortex_array::expr::make_free_field_annotator(&vortex_array::dtype::StructFields) -> impl vortex_array::expr::AnnotationFn -pub fn vortex_array::expr::mask(array: vortex_array::expr::Expression, mask: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::mask(vortex_array::expr::Expression, vortex_array::expr::Expression) -> vortex_array::expr::Expression -pub fn vortex_array::expr::merge(elements: impl core::iter::traits::collect::IntoIterator>) -> vortex_array::expr::Expression +pub fn vortex_array::expr::merge(impl core::iter::traits::collect::IntoIterator>) -> vortex_array::expr::Expression -pub fn vortex_array::expr::merge_opts(elements: impl core::iter::traits::collect::IntoIterator>, duplicate_handling: vortex_array::scalar_fn::fns::merge::DuplicateHandling) -> vortex_array::expr::Expression +pub fn vortex_array::expr::merge_opts(impl core::iter::traits::collect::IntoIterator>, vortex_array::scalar_fn::fns::merge::DuplicateHandling) -> vortex_array::expr::Expression -pub fn vortex_array::expr::nested_case_when(when_then_pairs: alloc::vec::Vec<(vortex_array::expr::Expression, vortex_array::expr::Expression)>, else_value: core::option::Option) -> vortex_array::expr::Expression +pub fn vortex_array::expr::nested_case_when(alloc::vec::Vec<(vortex_array::expr::Expression, vortex_array::expr::Expression)>, core::option::Option) -> vortex_array::expr::Expression -pub fn vortex_array::expr::not(operand: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::not(vortex_array::expr::Expression) -> vortex_array::expr::Expression -pub fn vortex_array::expr::not_eq(lhs: vortex_array::expr::Expression, rhs: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::not_eq(vortex_array::expr::Expression, vortex_array::expr::Expression) -> vortex_array::expr::Expression -pub fn vortex_array::expr::not_ilike(child: vortex_array::expr::Expression, pattern: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::not_ilike(vortex_array::expr::Expression, vortex_array::expr::Expression) -> vortex_array::expr::Expression -pub fn vortex_array::expr::not_like(child: vortex_array::expr::Expression, pattern: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::not_like(vortex_array::expr::Expression, vortex_array::expr::Expression) -> vortex_array::expr::Expression -pub fn vortex_array::expr::or(lhs: vortex_array::expr::Expression, rhs: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::or(vortex_array::expr::Expression, vortex_array::expr::Expression) -> vortex_array::expr::Expression -pub fn vortex_array::expr::or_collect(iter: I) -> core::option::Option where I: core::iter::traits::collect::IntoIterator +pub fn vortex_array::expr::or_collect(I) -> core::option::Option where I: core::iter::traits::collect::IntoIterator -pub fn vortex_array::expr::pack(elements: impl core::iter::traits::collect::IntoIterator, vortex_array::expr::Expression)>, nullability: vortex_array::dtype::Nullability) -> vortex_array::expr::Expression +pub fn vortex_array::expr::pack(impl core::iter::traits::collect::IntoIterator, vortex_array::expr::Expression)>, vortex_array::dtype::Nullability) -> vortex_array::expr::Expression pub fn vortex_array::expr::root() -> vortex_array::expr::Expression -pub fn vortex_array::expr::select(field_names: impl core::convert::Into, child: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::select(impl core::convert::Into, vortex_array::expr::Expression) -> vortex_array::expr::Expression -pub fn vortex_array::expr::select_exclude(fields: impl core::convert::Into, child: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::select_exclude(impl core::convert::Into, vortex_array::expr::Expression) -> vortex_array::expr::Expression -pub fn vortex_array::expr::split_conjunction(expr: &vortex_array::expr::Expression) -> alloc::vec::Vec +pub fn vortex_array::expr::split_conjunction(&vortex_array::expr::Expression) -> alloc::vec::Vec -pub fn vortex_array::expr::zip_expr(mask: vortex_array::expr::Expression, if_true: vortex_array::expr::Expression, if_false: vortex_array::expr::Expression) -> vortex_array::expr::Expression +pub fn vortex_array::expr::zip_expr(vortex_array::expr::Expression, vortex_array::expr::Expression, vortex_array::expr::Expression) -> vortex_array::expr::Expression pub type vortex_array::expr::Annotations<'a, A> = vortex_utils::aliases::hash_map::HashMap<&'a vortex_array::expr::Expression, vortex_utils::aliases::hash_set::HashSet> @@ -12602,7 +12756,7 @@ pub vortex_array::extension::datetime::DateValue::Milliseconds(i64) impl core::fmt::Display for vortex_array::extension::datetime::DateValue -pub fn vortex_array::extension::datetime::DateValue::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::extension::datetime::DateValue::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub enum vortex_array::extension::datetime::TemporalJiff @@ -12616,7 +12770,7 @@ pub vortex_array::extension::datetime::TemporalJiff::Zoned(jiff::zoned::Zoned) impl core::fmt::Display for vortex_array::extension::datetime::TemporalJiff -pub fn vortex_array::extension::datetime::TemporalJiff::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::extension::datetime::TemporalJiff::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub enum vortex_array::extension::datetime::TemporalMetadata<'a> @@ -12630,17 +12784,17 @@ impl vortex_array::extension::datetime::TemporalMetadata<'_> pub fn vortex_array::extension::datetime::TemporalMetadata<'_>::time_unit(&self) -> vortex_array::extension::datetime::TimeUnit -pub fn vortex_array::extension::datetime::TemporalMetadata<'_>::to_jiff(&self, v: i64) -> vortex_error::VortexResult +pub fn vortex_array::extension::datetime::TemporalMetadata<'_>::to_jiff(&self, i64) -> vortex_error::VortexResult impl<'a> core::cmp::Eq for vortex_array::extension::datetime::TemporalMetadata<'a> impl<'a> core::cmp::PartialEq for vortex_array::extension::datetime::TemporalMetadata<'a> -pub fn vortex_array::extension::datetime::TemporalMetadata<'a>::eq(&self, other: &vortex_array::extension::datetime::TemporalMetadata<'a>) -> bool +pub fn vortex_array::extension::datetime::TemporalMetadata<'a>::eq(&self, &vortex_array::extension::datetime::TemporalMetadata<'a>) -> bool impl<'a> core::fmt::Debug for vortex_array::extension::datetime::TemporalMetadata<'a> -pub fn vortex_array::extension::datetime::TemporalMetadata<'a>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::extension::datetime::TemporalMetadata<'a>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl<'a> core::marker::StructuralPartialEq for vortex_array::extension::datetime::TemporalMetadata<'a> @@ -12658,7 +12812,7 @@ pub vortex_array::extension::datetime::TimeUnit::Seconds = 3 impl vortex_array::extension::datetime::TimeUnit -pub fn vortex_array::extension::datetime::TimeUnit::to_jiff_span(&self, v: i64) -> vortex_error::VortexResult +pub fn vortex_array::extension::datetime::TimeUnit::to_jiff_span(&self, i64) -> vortex_error::VortexResult impl core::clone::Clone for vortex_array::extension::datetime::TimeUnit @@ -12668,51 +12822,51 @@ impl core::cmp::Eq for vortex_array::extension::datetime::TimeUnit impl core::cmp::Ord for vortex_array::extension::datetime::TimeUnit -pub fn vortex_array::extension::datetime::TimeUnit::cmp(&self, other: &vortex_array::extension::datetime::TimeUnit) -> core::cmp::Ordering +pub fn vortex_array::extension::datetime::TimeUnit::cmp(&self, &vortex_array::extension::datetime::TimeUnit) -> core::cmp::Ordering impl core::cmp::PartialEq for vortex_array::extension::datetime::TimeUnit -pub fn vortex_array::extension::datetime::TimeUnit::eq(&self, other: &vortex_array::extension::datetime::TimeUnit) -> bool +pub fn vortex_array::extension::datetime::TimeUnit::eq(&self, &vortex_array::extension::datetime::TimeUnit) -> bool impl core::cmp::PartialOrd for vortex_array::extension::datetime::TimeUnit -pub fn vortex_array::extension::datetime::TimeUnit::partial_cmp(&self, other: &vortex_array::extension::datetime::TimeUnit) -> core::option::Option +pub fn vortex_array::extension::datetime::TimeUnit::partial_cmp(&self, &vortex_array::extension::datetime::TimeUnit) -> core::option::Option impl core::convert::From<&arrow_schema::datatype::TimeUnit> for vortex_array::extension::datetime::TimeUnit -pub fn vortex_array::extension::datetime::TimeUnit::from(value: &arrow_schema::datatype::TimeUnit) -> Self +pub fn vortex_array::extension::datetime::TimeUnit::from(&arrow_schema::datatype::TimeUnit) -> Self impl core::convert::From for vortex_array::extension::datetime::TimeUnit -pub fn vortex_array::extension::datetime::TimeUnit::from(value: arrow_schema::datatype::TimeUnit) -> Self +pub fn vortex_array::extension::datetime::TimeUnit::from(arrow_schema::datatype::TimeUnit) -> Self impl core::convert::From for u8 -pub fn u8::from(enum_value: vortex_array::extension::datetime::TimeUnit) -> Self +pub fn u8::from(vortex_array::extension::datetime::TimeUnit) -> Self impl core::convert::TryFrom for vortex_array::extension::datetime::TimeUnit pub type vortex_array::extension::datetime::TimeUnit::Error = vortex_error::VortexError -pub fn vortex_array::extension::datetime::TimeUnit::try_from(value: u8) -> core::result::Result +pub fn vortex_array::extension::datetime::TimeUnit::try_from(u8) -> core::result::Result impl core::convert::TryFrom for arrow_schema::datatype::TimeUnit pub type arrow_schema::datatype::TimeUnit::Error = vortex_error::VortexError -pub fn arrow_schema::datatype::TimeUnit::try_from(value: vortex_array::extension::datetime::TimeUnit) -> vortex_error::VortexResult +pub fn arrow_schema::datatype::TimeUnit::try_from(vortex_array::extension::datetime::TimeUnit) -> vortex_error::VortexResult impl core::fmt::Debug for vortex_array::extension::datetime::TimeUnit -pub fn vortex_array::extension::datetime::TimeUnit::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::extension::datetime::TimeUnit::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::extension::datetime::TimeUnit -pub fn vortex_array::extension::datetime::TimeUnit::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::extension::datetime::TimeUnit::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::extension::datetime::TimeUnit -pub fn vortex_array::extension::datetime::TimeUnit::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::extension::datetime::TimeUnit::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_array::extension::datetime::TimeUnit @@ -12730,7 +12884,7 @@ pub vortex_array::extension::datetime::TimeValue::Seconds(i32) impl core::fmt::Display for vortex_array::extension::datetime::TimeValue -pub fn vortex_array::extension::datetime::TimeValue::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::extension::datetime::TimeValue::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub enum vortex_array::extension::datetime::TimestampValue<'a> @@ -12744,7 +12898,7 @@ pub vortex_array::extension::datetime::TimestampValue::Seconds(i64, core::option impl core::fmt::Display for vortex_array::extension::datetime::TimestampValue<'_> -pub fn vortex_array::extension::datetime::TimestampValue<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::extension::datetime::TimestampValue<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_array::extension::datetime::AnyTemporal @@ -12752,17 +12906,17 @@ impl vortex_array::dtype::extension::Matcher for vortex_array::extension::dateti pub type vortex_array::extension::datetime::AnyTemporal::Match<'a> = vortex_array::extension::datetime::TemporalMetadata<'a> -pub fn vortex_array::extension::datetime::AnyTemporal::matches(item: &vortex_array::dtype::extension::ExtDTypeRef) -> bool +pub fn vortex_array::extension::datetime::AnyTemporal::matches(&vortex_array::dtype::extension::ExtDTypeRef) -> bool -pub fn vortex_array::extension::datetime::AnyTemporal::try_match<'a>(item: &'a vortex_array::dtype::extension::ExtDTypeRef) -> core::option::Option +pub fn vortex_array::extension::datetime::AnyTemporal::try_match<'a>(&'a vortex_array::dtype::extension::ExtDTypeRef) -> core::option::Option pub struct vortex_array::extension::datetime::Date impl vortex_array::extension::datetime::Date -pub fn vortex_array::extension::datetime::Date::new(time_unit: vortex_array::extension::datetime::TimeUnit, nullability: vortex_array::dtype::Nullability) -> vortex_array::dtype::extension::ExtDType +pub fn vortex_array::extension::datetime::Date::new(vortex_array::extension::datetime::TimeUnit, vortex_array::dtype::Nullability) -> vortex_array::dtype::extension::ExtDType -pub fn vortex_array::extension::datetime::Date::try_new(time_unit: vortex_array::extension::datetime::TimeUnit, nullability: vortex_array::dtype::Nullability) -> vortex_error::VortexResult> +pub fn vortex_array::extension::datetime::Date::try_new(vortex_array::extension::datetime::TimeUnit, vortex_array::dtype::Nullability) -> vortex_error::VortexResult> impl core::clone::Clone for vortex_array::extension::datetime::Date @@ -12772,7 +12926,7 @@ impl core::cmp::Eq for vortex_array::extension::datetime::Date impl core::cmp::PartialEq for vortex_array::extension::datetime::Date -pub fn vortex_array::extension::datetime::Date::eq(&self, other: &vortex_array::extension::datetime::Date) -> bool +pub fn vortex_array::extension::datetime::Date::eq(&self, &vortex_array::extension::datetime::Date) -> bool impl core::default::Default for vortex_array::extension::datetime::Date @@ -12780,11 +12934,11 @@ pub fn vortex_array::extension::datetime::Date::default() -> vortex_array::exten impl core::fmt::Debug for vortex_array::extension::datetime::Date -pub fn vortex_array::extension::datetime::Date::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::extension::datetime::Date::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::extension::datetime::Date -pub fn vortex_array::extension::datetime::Date::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::extension::datetime::Date::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_array::extension::datetime::Date @@ -12794,31 +12948,31 @@ pub type vortex_array::extension::datetime::Date::Metadata = vortex_array::exten pub type vortex_array::extension::datetime::Date::NativeValue<'a> = vortex_array::extension::datetime::DateValue -pub fn vortex_array::extension::datetime::Date::can_coerce_from(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::extension::datetime::Date::can_coerce_from(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> bool -pub fn vortex_array::extension::datetime::Date::can_coerce_to(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::extension::datetime::Date::can_coerce_to(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> bool -pub fn vortex_array::extension::datetime::Date::deserialize_metadata(&self, metadata: &[u8]) -> vortex_error::VortexResult +pub fn vortex_array::extension::datetime::Date::deserialize_metadata(&self, &[u8]) -> vortex_error::VortexResult pub fn vortex_array::extension::datetime::Date::id(&self) -> vortex_array::dtype::extension::ExtId -pub fn vortex_array::extension::datetime::Date::least_supertype(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::extension::datetime::Date::least_supertype(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::extension::datetime::Date::serialize_metadata(&self, metadata: &Self::Metadata) -> vortex_error::VortexResult> +pub fn vortex_array::extension::datetime::Date::serialize_metadata(&self, &Self::Metadata) -> vortex_error::VortexResult> -pub fn vortex_array::extension::datetime::Date::unpack_native<'a>(ext_dtype: &'a vortex_array::dtype::extension::ExtDType, storage_value: &'a vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult +pub fn vortex_array::extension::datetime::Date::unpack_native<'a>(&'a vortex_array::dtype::extension::ExtDType, &'a vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult -pub fn vortex_array::extension::datetime::Date::validate_dtype(ext_dtype: &vortex_array::dtype::extension::ExtDType) -> vortex_error::VortexResult<()> +pub fn vortex_array::extension::datetime::Date::validate_dtype(&vortex_array::dtype::extension::ExtDType) -> vortex_error::VortexResult<()> -pub fn vortex_array::extension::datetime::Date::validate_scalar_value(ext_dtype: &vortex_array::dtype::extension::ExtDType, storage_value: &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult<()> +pub fn vortex_array::extension::datetime::Date::validate_scalar_value(&vortex_array::dtype::extension::ExtDType, &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult<()> pub struct vortex_array::extension::datetime::Time impl vortex_array::extension::datetime::Time -pub fn vortex_array::extension::datetime::Time::new(time_unit: vortex_array::extension::datetime::TimeUnit, nullability: vortex_array::dtype::Nullability) -> vortex_array::dtype::extension::ExtDType +pub fn vortex_array::extension::datetime::Time::new(vortex_array::extension::datetime::TimeUnit, vortex_array::dtype::Nullability) -> vortex_array::dtype::extension::ExtDType -pub fn vortex_array::extension::datetime::Time::try_new(time_unit: vortex_array::extension::datetime::TimeUnit, nullability: vortex_array::dtype::Nullability) -> vortex_error::VortexResult> +pub fn vortex_array::extension::datetime::Time::try_new(vortex_array::extension::datetime::TimeUnit, vortex_array::dtype::Nullability) -> vortex_error::VortexResult> impl core::clone::Clone for vortex_array::extension::datetime::Time @@ -12828,7 +12982,7 @@ impl core::cmp::Eq for vortex_array::extension::datetime::Time impl core::cmp::PartialEq for vortex_array::extension::datetime::Time -pub fn vortex_array::extension::datetime::Time::eq(&self, other: &vortex_array::extension::datetime::Time) -> bool +pub fn vortex_array::extension::datetime::Time::eq(&self, &vortex_array::extension::datetime::Time) -> bool impl core::default::Default for vortex_array::extension::datetime::Time @@ -12836,11 +12990,11 @@ pub fn vortex_array::extension::datetime::Time::default() -> vortex_array::exten impl core::fmt::Debug for vortex_array::extension::datetime::Time -pub fn vortex_array::extension::datetime::Time::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::extension::datetime::Time::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::extension::datetime::Time -pub fn vortex_array::extension::datetime::Time::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::extension::datetime::Time::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_array::extension::datetime::Time @@ -12850,33 +13004,33 @@ pub type vortex_array::extension::datetime::Time::Metadata = vortex_array::exten pub type vortex_array::extension::datetime::Time::NativeValue<'a> = vortex_array::extension::datetime::TimeValue -pub fn vortex_array::extension::datetime::Time::can_coerce_from(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::extension::datetime::Time::can_coerce_from(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> bool -pub fn vortex_array::extension::datetime::Time::can_coerce_to(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::extension::datetime::Time::can_coerce_to(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> bool -pub fn vortex_array::extension::datetime::Time::deserialize_metadata(&self, data: &[u8]) -> vortex_error::VortexResult +pub fn vortex_array::extension::datetime::Time::deserialize_metadata(&self, &[u8]) -> vortex_error::VortexResult pub fn vortex_array::extension::datetime::Time::id(&self) -> vortex_array::dtype::extension::ExtId -pub fn vortex_array::extension::datetime::Time::least_supertype(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::extension::datetime::Time::least_supertype(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::extension::datetime::Time::serialize_metadata(&self, metadata: &Self::Metadata) -> vortex_error::VortexResult> +pub fn vortex_array::extension::datetime::Time::serialize_metadata(&self, &Self::Metadata) -> vortex_error::VortexResult> -pub fn vortex_array::extension::datetime::Time::unpack_native<'a>(ext_dtype: &'a vortex_array::dtype::extension::ExtDType, storage_value: &'a vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult +pub fn vortex_array::extension::datetime::Time::unpack_native<'a>(&'a vortex_array::dtype::extension::ExtDType, &'a vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult -pub fn vortex_array::extension::datetime::Time::validate_dtype(ext_dtype: &vortex_array::dtype::extension::ExtDType) -> vortex_error::VortexResult<()> +pub fn vortex_array::extension::datetime::Time::validate_dtype(&vortex_array::dtype::extension::ExtDType) -> vortex_error::VortexResult<()> -pub fn vortex_array::extension::datetime::Time::validate_scalar_value(ext_dtype: &vortex_array::dtype::extension::ExtDType, storage_value: &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult<()> +pub fn vortex_array::extension::datetime::Time::validate_scalar_value(&vortex_array::dtype::extension::ExtDType, &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult<()> pub struct vortex_array::extension::datetime::Timestamp impl vortex_array::extension::datetime::Timestamp -pub fn vortex_array::extension::datetime::Timestamp::new(time_unit: vortex_array::extension::datetime::TimeUnit, nullability: vortex_array::dtype::Nullability) -> vortex_array::dtype::extension::ExtDType +pub fn vortex_array::extension::datetime::Timestamp::new(vortex_array::extension::datetime::TimeUnit, vortex_array::dtype::Nullability) -> vortex_array::dtype::extension::ExtDType -pub fn vortex_array::extension::datetime::Timestamp::new_with_options(options: vortex_array::extension::datetime::TimestampOptions, nullability: vortex_array::dtype::Nullability) -> vortex_array::dtype::extension::ExtDType +pub fn vortex_array::extension::datetime::Timestamp::new_with_options(vortex_array::extension::datetime::TimestampOptions, vortex_array::dtype::Nullability) -> vortex_array::dtype::extension::ExtDType -pub fn vortex_array::extension::datetime::Timestamp::new_with_tz(time_unit: vortex_array::extension::datetime::TimeUnit, timezone: core::option::Option>, nullability: vortex_array::dtype::Nullability) -> vortex_array::dtype::extension::ExtDType +pub fn vortex_array::extension::datetime::Timestamp::new_with_tz(vortex_array::extension::datetime::TimeUnit, core::option::Option>, vortex_array::dtype::Nullability) -> vortex_array::dtype::extension::ExtDType impl core::clone::Clone for vortex_array::extension::datetime::Timestamp @@ -12886,7 +13040,7 @@ impl core::cmp::Eq for vortex_array::extension::datetime::Timestamp impl core::cmp::PartialEq for vortex_array::extension::datetime::Timestamp -pub fn vortex_array::extension::datetime::Timestamp::eq(&self, other: &vortex_array::extension::datetime::Timestamp) -> bool +pub fn vortex_array::extension::datetime::Timestamp::eq(&self, &vortex_array::extension::datetime::Timestamp) -> bool impl core::default::Default for vortex_array::extension::datetime::Timestamp @@ -12894,11 +13048,11 @@ pub fn vortex_array::extension::datetime::Timestamp::default() -> vortex_array:: impl core::fmt::Debug for vortex_array::extension::datetime::Timestamp -pub fn vortex_array::extension::datetime::Timestamp::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::extension::datetime::Timestamp::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::extension::datetime::Timestamp -pub fn vortex_array::extension::datetime::Timestamp::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::extension::datetime::Timestamp::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_array::extension::datetime::Timestamp @@ -12908,23 +13062,23 @@ pub type vortex_array::extension::datetime::Timestamp::Metadata = vortex_array:: pub type vortex_array::extension::datetime::Timestamp::NativeValue<'a> = vortex_array::extension::datetime::TimestampValue<'a> -pub fn vortex_array::extension::datetime::Timestamp::can_coerce_from(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::extension::datetime::Timestamp::can_coerce_from(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> bool -pub fn vortex_array::extension::datetime::Timestamp::can_coerce_to(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::extension::datetime::Timestamp::can_coerce_to(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> bool -pub fn vortex_array::extension::datetime::Timestamp::deserialize_metadata(&self, data: &[u8]) -> vortex_error::VortexResult +pub fn vortex_array::extension::datetime::Timestamp::deserialize_metadata(&self, &[u8]) -> vortex_error::VortexResult pub fn vortex_array::extension::datetime::Timestamp::id(&self) -> vortex_array::dtype::extension::ExtId -pub fn vortex_array::extension::datetime::Timestamp::least_supertype(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::extension::datetime::Timestamp::least_supertype(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::extension::datetime::Timestamp::serialize_metadata(&self, metadata: &Self::Metadata) -> vortex_error::VortexResult> +pub fn vortex_array::extension::datetime::Timestamp::serialize_metadata(&self, &Self::Metadata) -> vortex_error::VortexResult> -pub fn vortex_array::extension::datetime::Timestamp::unpack_native<'a>(ext_dtype: &'a vortex_array::dtype::extension::ExtDType, storage_value: &'a vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult +pub fn vortex_array::extension::datetime::Timestamp::unpack_native<'a>(&'a vortex_array::dtype::extension::ExtDType, &'a vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult -pub fn vortex_array::extension::datetime::Timestamp::validate_dtype(ext_dtype: &vortex_array::dtype::extension::ExtDType) -> vortex_error::VortexResult<()> +pub fn vortex_array::extension::datetime::Timestamp::validate_dtype(&vortex_array::dtype::extension::ExtDType) -> vortex_error::VortexResult<()> -pub fn vortex_array::extension::datetime::Timestamp::validate_scalar_value(ext_dtype: &vortex_array::dtype::extension::ExtDType, storage_value: &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult<()> +pub fn vortex_array::extension::datetime::Timestamp::validate_scalar_value(&vortex_array::dtype::extension::ExtDType, &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult<()> pub struct vortex_array::extension::datetime::TimestampOptions @@ -12940,19 +13094,19 @@ impl core::cmp::Eq for vortex_array::extension::datetime::TimestampOptions impl core::cmp::PartialEq for vortex_array::extension::datetime::TimestampOptions -pub fn vortex_array::extension::datetime::TimestampOptions::eq(&self, other: &vortex_array::extension::datetime::TimestampOptions) -> bool +pub fn vortex_array::extension::datetime::TimestampOptions::eq(&self, &vortex_array::extension::datetime::TimestampOptions) -> bool impl core::fmt::Debug for vortex_array::extension::datetime::TimestampOptions -pub fn vortex_array::extension::datetime::TimestampOptions::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::extension::datetime::TimestampOptions::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::extension::datetime::TimestampOptions -pub fn vortex_array::extension::datetime::TimestampOptions::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::extension::datetime::TimestampOptions::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::extension::datetime::TimestampOptions -pub fn vortex_array::extension::datetime::TimestampOptions::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::extension::datetime::TimestampOptions::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_array::extension::datetime::TimestampOptions @@ -12968,7 +13122,7 @@ impl core::cmp::Eq for vortex_array::extension::uuid::Uuid impl core::cmp::PartialEq for vortex_array::extension::uuid::Uuid -pub fn vortex_array::extension::uuid::Uuid::eq(&self, other: &vortex_array::extension::uuid::Uuid) -> bool +pub fn vortex_array::extension::uuid::Uuid::eq(&self, &vortex_array::extension::uuid::Uuid) -> bool impl core::default::Default for vortex_array::extension::uuid::Uuid @@ -12976,11 +13130,11 @@ pub fn vortex_array::extension::uuid::Uuid::default() -> vortex_array::extension impl core::fmt::Debug for vortex_array::extension::uuid::Uuid -pub fn vortex_array::extension::uuid::Uuid::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::extension::uuid::Uuid::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::extension::uuid::Uuid -pub fn vortex_array::extension::uuid::Uuid::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::extension::uuid::Uuid::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_array::extension::uuid::Uuid @@ -12990,23 +13144,23 @@ pub type vortex_array::extension::uuid::Uuid::Metadata = vortex_array::extension pub type vortex_array::extension::uuid::Uuid::NativeValue<'a> = uuid::Uuid -pub fn vortex_array::extension::uuid::Uuid::can_coerce_from(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::extension::uuid::Uuid::can_coerce_from(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> bool -pub fn vortex_array::extension::uuid::Uuid::can_coerce_to(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::extension::uuid::Uuid::can_coerce_to(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> bool -pub fn vortex_array::extension::uuid::Uuid::deserialize_metadata(&self, metadata: &[u8]) -> vortex_error::VortexResult +pub fn vortex_array::extension::uuid::Uuid::deserialize_metadata(&self, &[u8]) -> vortex_error::VortexResult pub fn vortex_array::extension::uuid::Uuid::id(&self) -> vortex_array::dtype::extension::ExtId -pub fn vortex_array::extension::uuid::Uuid::least_supertype(ext_dtype: &vortex_array::dtype::extension::ExtDType, other: &vortex_array::dtype::DType) -> core::option::Option +pub fn vortex_array::extension::uuid::Uuid::least_supertype(&vortex_array::dtype::extension::ExtDType, &vortex_array::dtype::DType) -> core::option::Option -pub fn vortex_array::extension::uuid::Uuid::serialize_metadata(&self, metadata: &Self::Metadata) -> vortex_error::VortexResult> +pub fn vortex_array::extension::uuid::Uuid::serialize_metadata(&self, &Self::Metadata) -> vortex_error::VortexResult> -pub fn vortex_array::extension::uuid::Uuid::unpack_native<'a>(ext_dtype: &vortex_array::dtype::extension::ExtDType, storage_value: &'a vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult +pub fn vortex_array::extension::uuid::Uuid::unpack_native<'a>(&vortex_array::dtype::extension::ExtDType, &'a vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult -pub fn vortex_array::extension::uuid::Uuid::validate_dtype(ext_dtype: &vortex_array::dtype::extension::ExtDType) -> vortex_error::VortexResult<()> +pub fn vortex_array::extension::uuid::Uuid::validate_dtype(&vortex_array::dtype::extension::ExtDType) -> vortex_error::VortexResult<()> -pub fn vortex_array::extension::uuid::Uuid::validate_scalar_value(ext_dtype: &vortex_array::dtype::extension::ExtDType, storage_value: &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult<()> +pub fn vortex_array::extension::uuid::Uuid::validate_scalar_value(&vortex_array::dtype::extension::ExtDType, &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult<()> pub struct vortex_array::extension::uuid::UuidMetadata @@ -13020,7 +13174,7 @@ impl core::cmp::Eq for vortex_array::extension::uuid::UuidMetadata impl core::cmp::PartialEq for vortex_array::extension::uuid::UuidMetadata -pub fn vortex_array::extension::uuid::UuidMetadata::eq(&self, other: &Self) -> bool +pub fn vortex_array::extension::uuid::UuidMetadata::eq(&self, &Self) -> bool impl core::default::Default for vortex_array::extension::uuid::UuidMetadata @@ -13028,15 +13182,15 @@ pub fn vortex_array::extension::uuid::UuidMetadata::default() -> vortex_array::e impl core::fmt::Debug for vortex_array::extension::uuid::UuidMetadata -pub fn vortex_array::extension::uuid::UuidMetadata::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::extension::uuid::UuidMetadata::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::extension::uuid::UuidMetadata -pub fn vortex_array::extension::uuid::UuidMetadata::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::extension::uuid::UuidMetadata::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::extension::uuid::UuidMetadata -pub fn vortex_array::extension::uuid::UuidMetadata::hash(&self, state: &mut H) +pub fn vortex_array::extension::uuid::UuidMetadata::hash(&self, &mut H) pub struct vortex_array::extension::EmptyMetadata @@ -13048,19 +13202,19 @@ impl core::cmp::Eq for vortex_array::extension::EmptyMetadata impl core::cmp::PartialEq for vortex_array::extension::EmptyMetadata -pub fn vortex_array::extension::EmptyMetadata::eq(&self, other: &vortex_array::extension::EmptyMetadata) -> bool +pub fn vortex_array::extension::EmptyMetadata::eq(&self, &vortex_array::extension::EmptyMetadata) -> bool impl core::fmt::Debug for vortex_array::extension::EmptyMetadata -pub fn vortex_array::extension::EmptyMetadata::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::extension::EmptyMetadata::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::extension::EmptyMetadata -pub fn vortex_array::extension::EmptyMetadata::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::extension::EmptyMetadata::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::extension::EmptyMetadata -pub fn vortex_array::extension::EmptyMetadata::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::extension::EmptyMetadata::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_array::extension::EmptyMetadata @@ -13074,7 +13228,7 @@ pub struct vortex_array::iter::ArrayIteratorAdapter impl vortex_array::iter::ArrayIteratorAdapter -pub fn vortex_array::iter::ArrayIteratorAdapter::new(dtype: vortex_array::dtype::DType, inner: I) -> Self +pub fn vortex_array::iter::ArrayIteratorAdapter::new(vortex_array::dtype::DType, I) -> Self impl core::iter::traits::iterator::Iterator for vortex_array::iter::ArrayIteratorAdapter where I: core::iter::traits::iterator::Iterator> @@ -13120,113 +13274,113 @@ pub struct vortex_array::kernel::ParentKernelAdapter impl> core::fmt::Debug for vortex_array::kernel::ParentKernelAdapter -pub fn vortex_array::kernel::ParentKernelAdapter::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::kernel::ParentKernelAdapter::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl> vortex_array::kernel::DynParentKernel for vortex_array::kernel::ParentKernelAdapter -pub fn vortex_array::kernel::ParentKernelAdapter::execute_parent(&self, child: vortex_array::ArrayView<'_, V>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::kernel::ParentKernelAdapter::execute_parent(&self, vortex_array::ArrayView<'_, V>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> -pub fn vortex_array::kernel::ParentKernelAdapter::matches(&self, parent: &vortex_array::ArrayRef) -> bool +pub fn vortex_array::kernel::ParentKernelAdapter::matches(&self, &vortex_array::ArrayRef) -> bool pub struct vortex_array::kernel::ParentKernelSet impl vortex_array::kernel::ParentKernelSet -pub fn vortex_array::kernel::ParentKernelSet::execute(&self, child: vortex_array::ArrayView<'_, V>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::kernel::ParentKernelSet::execute(&self, vortex_array::ArrayView<'_, V>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> -pub const fn vortex_array::kernel::ParentKernelSet::lift>(kernel: &'static K) -> &'static dyn vortex_array::kernel::DynParentKernel +pub const fn vortex_array::kernel::ParentKernelSet::lift>(&'static K) -> &'static dyn vortex_array::kernel::DynParentKernel -pub const fn vortex_array::kernel::ParentKernelSet::new(kernels: &'static [&'static dyn vortex_array::kernel::DynParentKernel]) -> Self +pub const fn vortex_array::kernel::ParentKernelSet::new(&'static [&'static dyn vortex_array::kernel::DynParentKernel]) -> Self pub trait vortex_array::kernel::DynParentKernel: core::marker::Send + core::marker::Sync -pub fn vortex_array::kernel::DynParentKernel::execute_parent(&self, child: vortex_array::ArrayView<'_, V>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::kernel::DynParentKernel::execute_parent(&self, vortex_array::ArrayView<'_, V>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> -pub fn vortex_array::kernel::DynParentKernel::matches(&self, parent: &vortex_array::ArrayRef) -> bool +pub fn vortex_array::kernel::DynParentKernel::matches(&self, &vortex_array::ArrayRef) -> bool impl> vortex_array::kernel::DynParentKernel for vortex_array::kernel::ParentKernelAdapter -pub fn vortex_array::kernel::ParentKernelAdapter::execute_parent(&self, child: vortex_array::ArrayView<'_, V>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::kernel::ParentKernelAdapter::execute_parent(&self, vortex_array::ArrayView<'_, V>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> -pub fn vortex_array::kernel::ParentKernelAdapter::matches(&self, parent: &vortex_array::ArrayRef) -> bool +pub fn vortex_array::kernel::ParentKernelAdapter::matches(&self, &vortex_array::ArrayRef) -> bool pub trait vortex_array::kernel::ExecuteParentKernel: core::fmt::Debug + core::marker::Send + core::marker::Sync + 'static pub type vortex_array::kernel::ExecuteParentKernel::Parent: vortex_array::matcher::Matcher -pub fn vortex_array::kernel::ExecuteParentKernel::execute_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: ::Match, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::kernel::ExecuteParentKernel::execute_parent(&self, vortex_array::ArrayView<'_, V>, ::Match, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::kernel::ExecuteParentKernel for vortex_array::arrays::dict::TakeExecuteAdaptor where V: vortex_array::arrays::dict::TakeExecute pub type vortex_array::arrays::dict::TakeExecuteAdaptor::Parent = vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::TakeExecuteAdaptor::execute_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: ::Match, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::TakeExecuteAdaptor::execute_parent(&self, vortex_array::ArrayView<'_, V>, ::Match, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::kernel::ExecuteParentKernel for vortex_array::arrays::filter::FilterExecuteAdaptor where V: vortex_array::arrays::filter::FilterKernel pub type vortex_array::arrays::filter::FilterExecuteAdaptor::Parent = vortex_array::arrays::Filter -pub fn vortex_array::arrays::filter::FilterExecuteAdaptor::execute_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: ::Match, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::filter::FilterExecuteAdaptor::execute_parent(&self, vortex_array::ArrayView<'_, V>, ::Match, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::kernel::ExecuteParentKernel for vortex_array::arrays::slice::SliceExecuteAdaptor where V: vortex_array::arrays::slice::SliceKernel pub type vortex_array::arrays::slice::SliceExecuteAdaptor::Parent = vortex_array::arrays::slice::Slice -pub fn vortex_array::arrays::slice::SliceExecuteAdaptor::execute_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: ::Match, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::SliceExecuteAdaptor::execute_parent(&self, vortex_array::ArrayView<'_, V>, ::Match, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::kernel::ExecuteParentKernel for vortex_array::scalar_fn::fns::between::BetweenExecuteAdaptor where V: vortex_array::scalar_fn::fns::between::BetweenKernel pub type vortex_array::scalar_fn::fns::between::BetweenExecuteAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::between::BetweenExecuteAdaptor::execute_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::between::Between>, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::between::BetweenExecuteAdaptor::execute_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::between::Between>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::kernel::ExecuteParentKernel for vortex_array::scalar_fn::fns::binary::CompareExecuteAdaptor where V: vortex_array::scalar_fn::fns::binary::CompareKernel pub type vortex_array::scalar_fn::fns::binary::CompareExecuteAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::binary::CompareExecuteAdaptor::execute_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::binary::Binary>, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::binary::CompareExecuteAdaptor::execute_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::binary::Binary>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::kernel::ExecuteParentKernel for vortex_array::scalar_fn::fns::cast::CastExecuteAdaptor where V: vortex_array::scalar_fn::fns::cast::CastKernel pub type vortex_array::scalar_fn::fns::cast::CastExecuteAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::cast::CastExecuteAdaptor::execute_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: ::Match, _child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::cast::CastExecuteAdaptor::execute_parent(&self, vortex_array::ArrayView<'_, V>, ::Match, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::kernel::ExecuteParentKernel for vortex_array::scalar_fn::fns::fill_null::FillNullExecuteAdaptor where V: vortex_array::scalar_fn::fns::fill_null::FillNullKernel pub type vortex_array::scalar_fn::fns::fill_null::FillNullExecuteAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::fill_null::FillNullExecuteAdaptor::execute_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::fill_null::FillNull>, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::fill_null::FillNullExecuteAdaptor::execute_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::fill_null::FillNull>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::kernel::ExecuteParentKernel for vortex_array::scalar_fn::fns::like::LikeExecuteAdaptor where V: vortex_array::scalar_fn::fns::like::LikeKernel pub type vortex_array::scalar_fn::fns::like::LikeExecuteAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::like::LikeExecuteAdaptor::execute_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::like::Like>, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::like::LikeExecuteAdaptor::execute_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::like::Like>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::kernel::ExecuteParentKernel for vortex_array::scalar_fn::fns::list_contains::ListContainsElementExecuteAdaptor where V: vortex_array::scalar_fn::fns::list_contains::ListContainsElementKernel pub type vortex_array::scalar_fn::fns::list_contains::ListContainsElementExecuteAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::list_contains::ListContainsElementExecuteAdaptor::execute_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::list_contains::ListContains>, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::list_contains::ListContainsElementExecuteAdaptor::execute_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::list_contains::ListContains>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::kernel::ExecuteParentKernel for vortex_array::scalar_fn::fns::mask::MaskExecuteAdaptor where V: vortex_array::scalar_fn::fns::mask::MaskKernel pub type vortex_array::scalar_fn::fns::mask::MaskExecuteAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::mask::MaskExecuteAdaptor::execute_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::mask::Mask>, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::mask::MaskExecuteAdaptor::execute_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::mask::Mask>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::kernel::ExecuteParentKernel for vortex_array::scalar_fn::fns::not::NotExecuteAdaptor where V: vortex_array::scalar_fn::fns::not::NotKernel pub type vortex_array::scalar_fn::fns::not::NotExecuteAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::not::NotExecuteAdaptor::execute_parent(&self, array: vortex_array::ArrayView<'_, V>, _parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::not::Not>, _child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::not::NotExecuteAdaptor::execute_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::not::Not>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::kernel::ExecuteParentKernel for vortex_array::scalar_fn::fns::zip::ZipExecuteAdaptor where V: vortex_array::scalar_fn::fns::zip::ZipKernel pub type vortex_array::scalar_fn::fns::zip::ZipExecuteAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::zip::ZipExecuteAdaptor::execute_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::zip::Zip>, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::zip::ZipExecuteAdaptor::execute_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::zip::Zip>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub mod vortex_array::mask @@ -13236,71 +13390,71 @@ pub struct vortex_array::matcher::AnyArray impl core::fmt::Debug for vortex_array::matcher::AnyArray -pub fn vortex_array::matcher::AnyArray::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::matcher::AnyArray::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::matcher::Matcher for vortex_array::matcher::AnyArray pub type vortex_array::matcher::AnyArray::Match<'a> = &'a vortex_array::ArrayRef -pub fn vortex_array::matcher::AnyArray::matches(_array: &vortex_array::ArrayRef) -> bool +pub fn vortex_array::matcher::AnyArray::matches(&vortex_array::ArrayRef) -> bool -pub fn vortex_array::matcher::AnyArray::try_match(array: &vortex_array::ArrayRef) -> core::option::Option +pub fn vortex_array::matcher::AnyArray::try_match(&vortex_array::ArrayRef) -> core::option::Option pub trait vortex_array::matcher::Matcher pub type vortex_array::matcher::Matcher::Match<'a> -pub fn vortex_array::matcher::Matcher::matches(array: &vortex_array::ArrayRef) -> bool +pub fn vortex_array::matcher::Matcher::matches(&vortex_array::ArrayRef) -> bool -pub fn vortex_array::matcher::Matcher::try_match<'a>(array: &'a vortex_array::ArrayRef) -> core::option::Option +pub fn vortex_array::matcher::Matcher::try_match<'a>(&'a vortex_array::ArrayRef) -> core::option::Option impl vortex_array::matcher::Matcher for vortex_array::AnyCanonical pub type vortex_array::AnyCanonical::Match<'a> = vortex_array::CanonicalView<'a> -pub fn vortex_array::AnyCanonical::matches(array: &vortex_array::ArrayRef) -> bool +pub fn vortex_array::AnyCanonical::matches(&vortex_array::ArrayRef) -> bool -pub fn vortex_array::AnyCanonical::try_match<'a>(array: &'a vortex_array::ArrayRef) -> core::option::Option +pub fn vortex_array::AnyCanonical::try_match<'a>(&'a vortex_array::ArrayRef) -> core::option::Option impl vortex_array::matcher::Matcher for vortex_array::AnyColumnar pub type vortex_array::AnyColumnar::Match<'a> = vortex_array::ColumnarView<'a> -pub fn vortex_array::AnyColumnar::matches(array: &vortex_array::ArrayRef) -> bool +pub fn vortex_array::AnyColumnar::matches(&vortex_array::ArrayRef) -> bool -pub fn vortex_array::AnyColumnar::try_match<'a>(array: &'a vortex_array::ArrayRef) -> core::option::Option +pub fn vortex_array::AnyColumnar::try_match<'a>(&'a vortex_array::ArrayRef) -> core::option::Option impl vortex_array::matcher::Matcher for vortex_array::arrays::scalar_fn::AnyScalarFn -pub type vortex_array::arrays::scalar_fn::AnyScalarFn::Match<'a> = vortex_array::ArrayView<'a, vortex_array::arrays::scalar_fn::ScalarFnVTable> +pub type vortex_array::arrays::scalar_fn::AnyScalarFn::Match<'a> = vortex_array::ArrayView<'a, vortex_array::arrays::scalar_fn::ScalarFn> -pub fn vortex_array::arrays::scalar_fn::AnyScalarFn::matches(array: &vortex_array::ArrayRef) -> bool +pub fn vortex_array::arrays::scalar_fn::AnyScalarFn::matches(&vortex_array::ArrayRef) -> bool -pub fn vortex_array::arrays::scalar_fn::AnyScalarFn::try_match(array: &vortex_array::ArrayRef) -> core::option::Option +pub fn vortex_array::arrays::scalar_fn::AnyScalarFn::try_match(&vortex_array::ArrayRef) -> core::option::Option impl vortex_array::matcher::Matcher for vortex_array::matcher::AnyArray pub type vortex_array::matcher::AnyArray::Match<'a> = &'a vortex_array::ArrayRef -pub fn vortex_array::matcher::AnyArray::matches(_array: &vortex_array::ArrayRef) -> bool +pub fn vortex_array::matcher::AnyArray::matches(&vortex_array::ArrayRef) -> bool -pub fn vortex_array::matcher::AnyArray::try_match(array: &vortex_array::ArrayRef) -> core::option::Option +pub fn vortex_array::matcher::AnyArray::try_match(&vortex_array::ArrayRef) -> core::option::Option impl vortex_array::matcher::Matcher for vortex_array::arrays::scalar_fn::ExactScalarFn pub type vortex_array::arrays::scalar_fn::ExactScalarFn::Match<'a> = vortex_array::arrays::scalar_fn::ScalarFnArrayView<'a, F> -pub fn vortex_array::arrays::scalar_fn::ExactScalarFn::matches(array: &vortex_array::ArrayRef) -> bool +pub fn vortex_array::arrays::scalar_fn::ExactScalarFn::matches(&vortex_array::ArrayRef) -> bool -pub fn vortex_array::arrays::scalar_fn::ExactScalarFn::try_match(array: &vortex_array::ArrayRef) -> core::option::Option +pub fn vortex_array::arrays::scalar_fn::ExactScalarFn::try_match(&vortex_array::ArrayRef) -> core::option::Option impl vortex_array::matcher::Matcher for V pub type V::Match<'a> = vortex_array::ArrayView<'a, V> -pub fn V::matches(array: &vortex_array::ArrayRef) -> bool +pub fn V::matches(&vortex_array::ArrayRef) -> bool -pub fn V::try_match<'a>(array: &'a vortex_array::ArrayRef) -> core::option::Option> +pub fn V::try_match<'a>(&'a vortex_array::ArrayRef) -> core::option::Option> pub mod vortex_array::memory @@ -13312,11 +13466,11 @@ pub fn vortex_array::memory::DefaultHostAllocator::default() -> vortex_array::me impl core::fmt::Debug for vortex_array::memory::DefaultHostAllocator -pub fn vortex_array::memory::DefaultHostAllocator::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::memory::DefaultHostAllocator::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::memory::HostAllocator for vortex_array::memory::DefaultHostAllocator -pub fn vortex_array::memory::DefaultHostAllocator::allocate(&self, len: usize, alignment: vortex_buffer::alignment::Alignment) -> vortex_error::VortexResult +pub fn vortex_array::memory::DefaultHostAllocator::allocate(&self, usize, vortex_buffer::alignment::Alignment) -> vortex_error::VortexResult pub struct vortex_array::memory::MemorySession @@ -13324,9 +13478,9 @@ impl vortex_array::memory::MemorySession pub fn vortex_array::memory::MemorySession::allocator(&self) -> vortex_array::memory::HostAllocatorRef -pub fn vortex_array::memory::MemorySession::new(allocator: vortex_array::memory::HostAllocatorRef) -> Self +pub fn vortex_array::memory::MemorySession::new(vortex_array::memory::HostAllocatorRef) -> Self -pub fn vortex_array::memory::MemorySession::set_allocator(&mut self, allocator: vortex_array::memory::HostAllocatorRef) +pub fn vortex_array::memory::MemorySession::set_allocator(&mut self, vortex_array::memory::HostAllocatorRef) impl core::default::Default for vortex_array::memory::MemorySession @@ -13334,7 +13488,13 @@ pub fn vortex_array::memory::MemorySession::default() -> Self impl core::fmt::Debug for vortex_array::memory::MemorySession -pub fn vortex_array::memory::MemorySession::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::memory::MemorySession::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +impl vortex_session::SessionVar for vortex_array::memory::MemorySession + +pub fn vortex_array::memory::MemorySession::as_any(&self) -> &dyn core::any::Any + +pub fn vortex_array::memory::MemorySession::as_any_mut(&mut self) -> &mut dyn core::any::Any pub struct vortex_array::memory::WritableHostBuffer @@ -13354,27 +13514,27 @@ pub fn vortex_array::memory::WritableHostBuffer::is_empty(&self) -> bool pub fn vortex_array::memory::WritableHostBuffer::len(&self) -> usize -pub fn vortex_array::memory::WritableHostBuffer::new(inner: alloc::boxed::Box) -> Self +pub fn vortex_array::memory::WritableHostBuffer::new(alloc::boxed::Box) -> Self impl core::fmt::Debug for vortex_array::memory::WritableHostBuffer -pub fn vortex_array::memory::WritableHostBuffer::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::memory::WritableHostBuffer::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub trait vortex_array::memory::HostAllocator: core::fmt::Debug + core::marker::Send + core::marker::Sync + 'static -pub fn vortex_array::memory::HostAllocator::allocate(&self, len: usize, alignment: vortex_buffer::alignment::Alignment) -> vortex_error::VortexResult +pub fn vortex_array::memory::HostAllocator::allocate(&self, usize, vortex_buffer::alignment::Alignment) -> vortex_error::VortexResult impl vortex_array::memory::HostAllocator for vortex_array::memory::DefaultHostAllocator -pub fn vortex_array::memory::DefaultHostAllocator::allocate(&self, len: usize, alignment: vortex_buffer::alignment::Alignment) -> vortex_error::VortexResult +pub fn vortex_array::memory::DefaultHostAllocator::allocate(&self, usize, vortex_buffer::alignment::Alignment) -> vortex_error::VortexResult pub trait vortex_array::memory::HostAllocatorExt: vortex_array::memory::HostAllocator -pub fn vortex_array::memory::HostAllocatorExt::allocate_typed(&self, len: usize) -> vortex_error::VortexResult +pub fn vortex_array::memory::HostAllocatorExt::allocate_typed(&self, usize) -> vortex_error::VortexResult impl vortex_array::memory::HostAllocatorExt for A -pub fn A::allocate_typed(&self, len: usize) -> vortex_error::VortexResult +pub fn A::allocate_typed(&self, usize) -> vortex_error::VortexResult pub trait vortex_array::memory::HostBufferMut: core::marker::Send + 'static @@ -13382,7 +13542,7 @@ pub fn vortex_array::memory::HostBufferMut::alignment(&self) -> vortex_buffer::a pub fn vortex_array::memory::HostBufferMut::as_mut_slice(&mut self) -> &mut [u8] -pub fn vortex_array::memory::HostBufferMut::freeze(self: alloc::boxed::Box) -> vortex_buffer::ByteBuffer +pub fn vortex_array::memory::HostBufferMut::freeze(alloc::boxed::Box) -> vortex_buffer::ByteBuffer pub fn vortex_array::memory::HostBufferMut::is_empty(&self) -> bool @@ -13422,155 +13582,195 @@ pub vortex_array::normalize::NormalizeOptions::operation: vortex_array::normaliz pub mod vortex_array::optimizer +pub mod vortex_array::optimizer::kernels + +pub struct vortex_array::optimizer::kernels::ArrayKernels + +impl vortex_array::optimizer::kernels::ArrayKernels + +pub fn vortex_array::optimizer::kernels::ArrayKernels::empty() -> Self + +pub fn vortex_array::optimizer::kernels::ArrayKernels::find_reduce_parent(&self, vortex_session::registry::Id, vortex_session::registry::Id) -> core::option::Option> + +pub fn vortex_array::optimizer::kernels::ArrayKernels::register_reduce_parent>(&self, vortex_session::registry::Id, vortex_session::registry::Id, I) + +impl core::default::Default for vortex_array::optimizer::kernels::ArrayKernels + +pub fn vortex_array::optimizer::kernels::ArrayKernels::default() -> vortex_array::optimizer::kernels::ArrayKernels + +impl core::fmt::Debug for vortex_array::optimizer::kernels::ArrayKernels + +pub fn vortex_array::optimizer::kernels::ArrayKernels::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +impl vortex_session::SessionVar for vortex_array::optimizer::kernels::ArrayKernels + +pub fn vortex_array::optimizer::kernels::ArrayKernels::as_any(&self) -> &dyn core::any::Any + +pub fn vortex_array::optimizer::kernels::ArrayKernels::as_any_mut(&mut self) -> &mut dyn core::any::Any + +pub trait vortex_array::optimizer::kernels::ArrayKernelsExt: vortex_session::SessionExt + +pub fn vortex_array::optimizer::kernels::ArrayKernelsExt::kernels(&self) -> vortex_session::Ref<'_, vortex_array::optimizer::kernels::ArrayKernels> + +impl vortex_array::optimizer::kernels::ArrayKernelsExt for S + +pub fn S::kernels(&self) -> vortex_session::Ref<'_, vortex_array::optimizer::kernels::ArrayKernels> + +pub type vortex_array::optimizer::kernels::ReduceParentFn = fn(&vortex_array::ArrayRef, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> + pub mod vortex_array::optimizer::rules pub struct vortex_array::optimizer::rules::ParentReduceRuleAdapter impl> vortex_array::optimizer::rules::DynArrayParentReduceRule for vortex_array::optimizer::rules::ParentReduceRuleAdapter -pub fn vortex_array::optimizer::rules::ParentReduceRuleAdapter::matches(&self, parent: &vortex_array::ArrayRef) -> bool +pub fn vortex_array::optimizer::rules::ParentReduceRuleAdapter::matches(&self, &vortex_array::ArrayRef) -> bool -pub fn vortex_array::optimizer::rules::ParentReduceRuleAdapter::reduce_parent(&self, child: vortex_array::ArrayView<'_, V>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::optimizer::rules::ParentReduceRuleAdapter::reduce_parent(&self, vortex_array::ArrayView<'_, V>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> impl> core::fmt::Debug for vortex_array::optimizer::rules::ParentReduceRuleAdapter -pub fn vortex_array::optimizer::rules::ParentReduceRuleAdapter::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::optimizer::rules::ParentReduceRuleAdapter::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_array::optimizer::rules::ParentRuleSet impl vortex_array::optimizer::rules::ParentRuleSet -pub fn vortex_array::optimizer::rules::ParentRuleSet::evaluate(&self, child: vortex_array::ArrayView<'_, V>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::optimizer::rules::ParentRuleSet::evaluate(&self, vortex_array::ArrayView<'_, V>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub const fn vortex_array::optimizer::rules::ParentRuleSet::lift>(rule: &'static R) -> &'static dyn vortex_array::optimizer::rules::DynArrayParentReduceRule +pub const fn vortex_array::optimizer::rules::ParentRuleSet::lift>(&'static R) -> &'static dyn vortex_array::optimizer::rules::DynArrayParentReduceRule -pub const fn vortex_array::optimizer::rules::ParentRuleSet::new(rules: &'static [&'static dyn vortex_array::optimizer::rules::DynArrayParentReduceRule]) -> Self +pub const fn vortex_array::optimizer::rules::ParentRuleSet::new(&'static [&'static dyn vortex_array::optimizer::rules::DynArrayParentReduceRule]) -> Self pub struct vortex_array::optimizer::rules::ReduceRuleSet impl vortex_array::optimizer::rules::ReduceRuleSet -pub fn vortex_array::optimizer::rules::ReduceRuleSet::evaluate(&self, array: vortex_array::ArrayView<'_, V>) -> vortex_error::VortexResult> +pub fn vortex_array::optimizer::rules::ReduceRuleSet::evaluate(&self, vortex_array::ArrayView<'_, V>) -> vortex_error::VortexResult> -pub const fn vortex_array::optimizer::rules::ReduceRuleSet::new(rules: &'static [&'static dyn vortex_array::optimizer::rules::ArrayReduceRule]) -> Self +pub const fn vortex_array::optimizer::rules::ReduceRuleSet::new(&'static [&'static dyn vortex_array::optimizer::rules::ArrayReduceRule]) -> Self pub trait vortex_array::optimizer::rules::ArrayParentReduceRule: core::fmt::Debug + core::marker::Send + core::marker::Sync + 'static pub type vortex_array::optimizer::rules::ArrayParentReduceRule::Parent: vortex_array::matcher::Matcher -pub fn vortex_array::optimizer::rules::ArrayParentReduceRule::reduce_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: ::Match, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::optimizer::rules::ArrayParentReduceRule::reduce_parent(&self, vortex_array::ArrayView<'_, V>, ::Match, usize) -> vortex_error::VortexResult> impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::arrays::bool::BoolMaskedValidityRule pub type vortex_array::arrays::bool::BoolMaskedValidityRule::Parent = vortex_array::arrays::Masked -pub fn vortex_array::arrays::bool::BoolMaskedValidityRule::reduce_parent(&self, array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, parent: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::bool::BoolMaskedValidityRule::reduce_parent(&self, vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, usize) -> vortex_error::VortexResult> impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::arrays::decimal::DecimalMaskedValidityRule pub type vortex_array::arrays::decimal::DecimalMaskedValidityRule::Parent = vortex_array::arrays::Masked -pub fn vortex_array::arrays::decimal::DecimalMaskedValidityRule::reduce_parent(&self, array: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, parent: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, _child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::decimal::DecimalMaskedValidityRule::reduce_parent(&self, vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, usize) -> vortex_error::VortexResult> impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::arrays::primitive::PrimitiveMaskedValidityRule pub type vortex_array::arrays::primitive::PrimitiveMaskedValidityRule::Parent = vortex_array::arrays::Masked -pub fn vortex_array::arrays::primitive::PrimitiveMaskedValidityRule::reduce_parent(&self, array: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, parent: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, _child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::primitive::PrimitiveMaskedValidityRule::reduce_parent(&self, vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, usize) -> vortex_error::VortexResult> impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::arrays::dict::TakeReduceAdaptor where V: vortex_array::arrays::dict::TakeReduce pub type vortex_array::arrays::dict::TakeReduceAdaptor::Parent = vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::TakeReduceAdaptor::reduce_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::TakeReduceAdaptor::reduce_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, usize) -> vortex_error::VortexResult> impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::arrays::filter::FilterReduceAdaptor where V: vortex_array::arrays::filter::FilterReduce pub type vortex_array::arrays::filter::FilterReduceAdaptor::Parent = vortex_array::arrays::Filter -pub fn vortex_array::arrays::filter::FilterReduceAdaptor::reduce_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::ArrayView<'_, vortex_array::arrays::Filter>, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::filter::FilterReduceAdaptor::reduce_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::ArrayView<'_, vortex_array::arrays::Filter>, usize) -> vortex_error::VortexResult> impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::arrays::slice::SliceReduceAdaptor where V: vortex_array::arrays::slice::SliceReduce pub type vortex_array::arrays::slice::SliceReduceAdaptor::Parent = vortex_array::arrays::slice::Slice -pub fn vortex_array::arrays::slice::SliceReduceAdaptor::reduce_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: ::Match, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::SliceReduceAdaptor::reduce_parent(&self, vortex_array::ArrayView<'_, V>, ::Match, usize) -> vortex_error::VortexResult> impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::scalar_fn::fns::between::BetweenReduceAdaptor where V: vortex_array::scalar_fn::fns::between::BetweenReduce pub type vortex_array::scalar_fn::fns::between::BetweenReduceAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::between::BetweenReduceAdaptor::reduce_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::between::Between>, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::between::BetweenReduceAdaptor::reduce_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::between::Between>, usize) -> vortex_error::VortexResult> impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::scalar_fn::fns::cast::CastReduceAdaptor where V: vortex_array::scalar_fn::fns::cast::CastReduce pub type vortex_array::scalar_fn::fns::cast::CastReduceAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::cast::CastReduceAdaptor::reduce_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::cast::Cast>, _child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::cast::CastReduceAdaptor::reduce_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::cast::Cast>, usize) -> vortex_error::VortexResult> impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::scalar_fn::fns::fill_null::FillNullReduceAdaptor where V: vortex_array::scalar_fn::fns::fill_null::FillNullReduce pub type vortex_array::scalar_fn::fns::fill_null::FillNullReduceAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::fill_null::FillNullReduceAdaptor::reduce_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::fill_null::FillNull>, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::fill_null::FillNullReduceAdaptor::reduce_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::fill_null::FillNull>, usize) -> vortex_error::VortexResult> impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::scalar_fn::fns::like::LikeReduceAdaptor where V: vortex_array::scalar_fn::fns::like::LikeReduce pub type vortex_array::scalar_fn::fns::like::LikeReduceAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::like::LikeReduceAdaptor::reduce_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::like::Like>, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::like::LikeReduceAdaptor::reduce_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::like::Like>, usize) -> vortex_error::VortexResult> impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::scalar_fn::fns::list_contains::ListContainsElementReduceAdaptor where V: vortex_array::scalar_fn::fns::list_contains::ListContainsElementReduce pub type vortex_array::scalar_fn::fns::list_contains::ListContainsElementReduceAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::list_contains::ListContainsElementReduceAdaptor::reduce_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::list_contains::ListContains>, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::list_contains::ListContainsElementReduceAdaptor::reduce_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::list_contains::ListContains>, usize) -> vortex_error::VortexResult> impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::scalar_fn::fns::mask::MaskReduceAdaptor where V: vortex_array::scalar_fn::fns::mask::MaskReduce pub type vortex_array::scalar_fn::fns::mask::MaskReduceAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::mask::MaskReduceAdaptor::reduce_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::mask::Mask>, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::mask::MaskReduceAdaptor::reduce_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::mask::Mask>, usize) -> vortex_error::VortexResult> impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::scalar_fn::fns::not::NotReduceAdaptor where V: vortex_array::scalar_fn::fns::not::NotReduce pub type vortex_array::scalar_fn::fns::not::NotReduceAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::not::NotReduceAdaptor::reduce_parent(&self, array: vortex_array::ArrayView<'_, V>, _parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::not::Not>, _child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::not::NotReduceAdaptor::reduce_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::not::Not>, usize) -> vortex_error::VortexResult> impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::scalar_fn::fns::zip::ZipReduceAdaptor where V: vortex_array::scalar_fn::fns::zip::ZipReduce pub type vortex_array::scalar_fn::fns::zip::ZipReduceAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::zip::ZipReduceAdaptor::reduce_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::zip::Zip>, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::zip::ZipReduceAdaptor::reduce_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::zip::Zip>, usize) -> vortex_error::VortexResult> pub trait vortex_array::optimizer::rules::ArrayReduceRule: core::fmt::Debug + core::marker::Send + core::marker::Sync + 'static -pub fn vortex_array::optimizer::rules::ArrayReduceRule::reduce(&self, array: vortex_array::ArrayView<'_, V>) -> vortex_error::VortexResult> +pub fn vortex_array::optimizer::rules::ArrayReduceRule::reduce(&self, vortex_array::ArrayView<'_, V>) -> vortex_error::VortexResult> pub trait vortex_array::optimizer::rules::DynArrayParentReduceRule: core::fmt::Debug + core::marker::Send + core::marker::Sync -pub fn vortex_array::optimizer::rules::DynArrayParentReduceRule::matches(&self, parent: &vortex_array::ArrayRef) -> bool +pub fn vortex_array::optimizer::rules::DynArrayParentReduceRule::matches(&self, &vortex_array::ArrayRef) -> bool -pub fn vortex_array::optimizer::rules::DynArrayParentReduceRule::reduce_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::optimizer::rules::DynArrayParentReduceRule::reduce_parent(&self, vortex_array::ArrayView<'_, V>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> impl> vortex_array::optimizer::rules::DynArrayParentReduceRule for vortex_array::optimizer::rules::ParentReduceRuleAdapter -pub fn vortex_array::optimizer::rules::ParentReduceRuleAdapter::matches(&self, parent: &vortex_array::ArrayRef) -> bool +pub fn vortex_array::optimizer::rules::ParentReduceRuleAdapter::matches(&self, &vortex_array::ArrayRef) -> bool -pub fn vortex_array::optimizer::rules::ParentReduceRuleAdapter::reduce_parent(&self, child: vortex_array::ArrayView<'_, V>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::optimizer::rules::ParentReduceRuleAdapter::reduce_parent(&self, vortex_array::ArrayView<'_, V>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> pub trait vortex_array::optimizer::ArrayOptimizer pub fn vortex_array::optimizer::ArrayOptimizer::optimize(&self) -> vortex_error::VortexResult -pub fn vortex_array::optimizer::ArrayOptimizer::optimize_recursive(&self) -> vortex_error::VortexResult +pub fn vortex_array::optimizer::ArrayOptimizer::optimize_ctx(&self, &vortex_session::VortexSession) -> vortex_error::VortexResult + +pub fn vortex_array::optimizer::ArrayOptimizer::optimize_recursive(&self, &vortex_session::VortexSession) -> vortex_error::VortexResult impl vortex_array::optimizer::ArrayOptimizer for vortex_array::ArrayRef pub fn vortex_array::ArrayRef::optimize(&self) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::optimize_recursive(&self) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::optimize_ctx(&self, &vortex_session::VortexSession) -> vortex_error::VortexResult + +pub fn vortex_array::ArrayRef::optimize_recursive(&self, &vortex_session::VortexSession) -> vortex_error::VortexResult pub mod vortex_array::patches @@ -13578,21 +13778,19 @@ pub struct vortex_array::patches::Patches impl vortex_array::patches::Patches -pub unsafe fn vortex_array::patches::Patches::apply_to_buffer(&self, buffer: &mut [P], validity: &mut vortex_mask::mask_mut::MaskMut, ctx: &mut vortex_array::ExecutionCtx) - pub fn vortex_array::patches::Patches::array_len(&self) -> usize -pub fn vortex_array::patches::Patches::cast_values(self, values_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::patches::Patches::cast_values(self, &vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::patches::Patches::chunk_offset_at(&self, idx: usize) -> vortex_error::VortexResult +pub fn vortex_array::patches::Patches::chunk_offset_at(&self, usize) -> vortex_error::VortexResult pub fn vortex_array::patches::Patches::chunk_offsets(&self) -> &core::option::Option pub fn vortex_array::patches::Patches::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::patches::Patches::filter(&self, mask: &vortex_mask::Mask, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::patches::Patches::filter(&self, &vortex_mask::Mask, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> -pub fn vortex_array::patches::Patches::get_patched(&self, index: usize) -> vortex_error::VortexResult> +pub fn vortex_array::patches::Patches::get_patched(&self, usize) -> vortex_error::VortexResult> pub fn vortex_array::patches::Patches::indices(&self) -> &vortex_array::ArrayRef @@ -13604,17 +13802,17 @@ pub fn vortex_array::patches::Patches::into_indices(self) -> vortex_array::Array pub fn vortex_array::patches::Patches::into_values(self) -> vortex_array::ArrayRef -pub fn vortex_array::patches::Patches::map_values(self, f: F) -> vortex_error::VortexResult where F: core::ops::function::FnOnce(vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::patches::Patches::map_values(self, F) -> vortex_error::VortexResult where F: core::ops::function::FnOnce(vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::patches::Patches::mask(&self, mask: &vortex_mask::Mask, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::patches::Patches::mask(&self, &vortex_mask::Mask, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::patches::Patches::max_index(&self) -> vortex_error::VortexResult pub fn vortex_array::patches::Patches::min_index(&self) -> vortex_error::VortexResult -pub fn vortex_array::patches::Patches::new(array_len: usize, offset: usize, indices: vortex_array::ArrayRef, values: vortex_array::ArrayRef, chunk_offsets: core::option::Option) -> vortex_error::VortexResult +pub fn vortex_array::patches::Patches::new(usize, usize, vortex_array::ArrayRef, vortex_array::ArrayRef, core::option::Option) -> vortex_error::VortexResult -pub unsafe fn vortex_array::patches::Patches::new_unchecked(array_len: usize, offset: usize, indices: vortex_array::ArrayRef, values: vortex_array::ArrayRef, chunk_offsets: core::option::Option, offset_within_chunk: core::option::Option) -> Self +pub unsafe fn vortex_array::patches::Patches::new_unchecked(usize, usize, vortex_array::ArrayRef, vortex_array::ArrayRef, core::option::Option, core::option::Option) -> Self pub fn vortex_array::patches::Patches::num_patches(&self) -> usize @@ -13622,19 +13820,19 @@ pub fn vortex_array::patches::Patches::offset(&self) -> usize pub fn vortex_array::patches::Patches::offset_within_chunk(&self) -> core::option::Option -pub fn vortex_array::patches::Patches::search_index(&self, index: usize) -> vortex_error::VortexResult +pub fn vortex_array::patches::Patches::search_index(&self, usize) -> vortex_error::VortexResult -pub fn vortex_array::patches::Patches::slice(&self, range: core::ops::range::Range) -> vortex_error::VortexResult> +pub fn vortex_array::patches::Patches::slice(&self, core::ops::range::Range) -> vortex_error::VortexResult> -pub fn vortex_array::patches::Patches::take(&self, take_indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::patches::Patches::take(&self, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> -pub fn vortex_array::patches::Patches::take_map(&self, take_indices: vortex_array::arrays::PrimitiveArray, include_nulls: bool, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::patches::Patches::take_map(&self, vortex_array::arrays::PrimitiveArray, bool, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> -pub fn vortex_array::patches::Patches::take_search(&self, take_indices: vortex_array::arrays::PrimitiveArray, include_nulls: bool, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::patches::Patches::take_search(&self, vortex_array::arrays::PrimitiveArray, bool, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> -pub fn vortex_array::patches::Patches::take_with_nulls(&self, take_indices: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::patches::Patches::take_with_nulls(&self, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> -pub fn vortex_array::patches::Patches::to_metadata(&self, len: usize, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::patches::Patches::to_metadata(&self, usize, &vortex_array::dtype::DType) -> vortex_error::VortexResult pub fn vortex_array::patches::Patches::values(&self) -> &vortex_array::ArrayRef @@ -13646,15 +13844,15 @@ pub fn vortex_array::patches::Patches::clone(&self) -> vortex_array::patches::Pa impl core::fmt::Debug for vortex_array::patches::Patches -pub fn vortex_array::patches::Patches::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::patches::Patches::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::ArrayEq for vortex_array::patches::Patches -pub fn vortex_array::patches::Patches::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn vortex_array::patches::Patches::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayHash for vortex_array::patches::Patches -pub fn vortex_array::patches::Patches::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn vortex_array::patches::Patches::array_hash(&self, &mut H, vortex_array::Precision) pub struct vortex_array::patches::PatchesMetadata @@ -13668,7 +13866,7 @@ pub fn vortex_array::patches::PatchesMetadata::is_empty(&self) -> bool pub fn vortex_array::patches::PatchesMetadata::len(&self) -> vortex_error::VortexResult -pub fn vortex_array::patches::PatchesMetadata::new(len: usize, offset: usize, indices_ptype: vortex_array::dtype::PType, chunk_offsets_len: core::option::Option, chunk_offsets_ptype: core::option::Option, offset_within_chunk: core::option::Option) -> Self +pub fn vortex_array::patches::PatchesMetadata::new(usize, usize, vortex_array::dtype::PType, core::option::Option, core::option::Option, core::option::Option) -> Self pub fn vortex_array::patches::PatchesMetadata::offset(&self) -> vortex_error::VortexResult @@ -13682,9 +13880,9 @@ pub fn vortex_array::patches::PatchesMetadata::indices_ptype(&self) -> vortex_ar pub fn vortex_array::patches::PatchesMetadata::offset_within_chunk(&self) -> u64 -pub fn vortex_array::patches::PatchesMetadata::set_chunk_offsets_ptype(&mut self, value: vortex_array::dtype::PType) +pub fn vortex_array::patches::PatchesMetadata::set_chunk_offsets_ptype(&mut self, vortex_array::dtype::PType) -pub fn vortex_array::patches::PatchesMetadata::set_indices_ptype(&mut self, value: vortex_array::dtype::PType) +pub fn vortex_array::patches::PatchesMetadata::set_indices_ptype(&mut self, vortex_array::dtype::PType) impl core::clone::Clone for vortex_array::patches::PatchesMetadata @@ -13696,7 +13894,7 @@ pub fn vortex_array::patches::PatchesMetadata::default() -> Self impl core::fmt::Debug for vortex_array::patches::PatchesMetadata -pub fn vortex_array::patches::PatchesMetadata::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::patches::PatchesMetadata::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_array::patches::PatchesMetadata @@ -13728,23 +13926,23 @@ impl vortex_array::scalar::DecimalValue pub fn vortex_array::scalar::DecimalValue::cast(&self) -> core::option::Option -pub fn vortex_array::scalar::DecimalValue::checked_add(&self, other: &Self) -> core::option::Option +pub fn vortex_array::scalar::DecimalValue::checked_add(&self, &Self) -> core::option::Option -pub fn vortex_array::scalar::DecimalValue::checked_div(&self, other: &Self) -> core::option::Option +pub fn vortex_array::scalar::DecimalValue::checked_div(&self, &Self) -> core::option::Option -pub fn vortex_array::scalar::DecimalValue::checked_mul(&self, other: &Self) -> core::option::Option +pub fn vortex_array::scalar::DecimalValue::checked_mul(&self, &Self) -> core::option::Option -pub fn vortex_array::scalar::DecimalValue::checked_sub(&self, other: &Self) -> core::option::Option +pub fn vortex_array::scalar::DecimalValue::checked_sub(&self, &Self) -> core::option::Option pub fn vortex_array::scalar::DecimalValue::decimal_dtype(&self) -> vortex_array::dtype::DecimalDType pub fn vortex_array::scalar::DecimalValue::decimal_type(&self) -> vortex_array::dtype::DecimalType -pub fn vortex_array::scalar::DecimalValue::fits_in_precision(&self, decimal_type: vortex_array::dtype::DecimalDType) -> bool +pub fn vortex_array::scalar::DecimalValue::fits_in_precision(&self, vortex_array::dtype::DecimalDType) -> bool pub fn vortex_array::scalar::DecimalValue::is_zero(&self) -> bool -pub fn vortex_array::scalar::DecimalValue::zero(decimal_type: &vortex_array::dtype::DecimalDType) -> Self +pub fn vortex_array::scalar::DecimalValue::zero(&vortex_array::dtype::DecimalDType) -> Self impl core::clone::Clone for vortex_array::scalar::DecimalValue @@ -13754,83 +13952,83 @@ impl core::cmp::Eq for vortex_array::scalar::DecimalValue impl core::cmp::PartialEq for vortex_array::scalar::DecimalValue -pub fn vortex_array::scalar::DecimalValue::eq(&self, other: &Self) -> bool +pub fn vortex_array::scalar::DecimalValue::eq(&self, &Self) -> bool impl core::cmp::PartialOrd for vortex_array::scalar::DecimalValue -pub fn vortex_array::scalar::DecimalValue::partial_cmp(&self, other: &Self) -> core::option::Option +pub fn vortex_array::scalar::DecimalValue::partial_cmp(&self, &Self) -> core::option::Option impl core::convert::From for vortex_array::scalar::DecimalValue -pub fn vortex_array::scalar::DecimalValue::from(value: i128) -> Self +pub fn vortex_array::scalar::DecimalValue::from(i128) -> Self impl core::convert::From for vortex_array::scalar::DecimalValue -pub fn vortex_array::scalar::DecimalValue::from(value: i16) -> Self +pub fn vortex_array::scalar::DecimalValue::from(i16) -> Self impl core::convert::From for vortex_array::scalar::DecimalValue -pub fn vortex_array::scalar::DecimalValue::from(value: i32) -> Self +pub fn vortex_array::scalar::DecimalValue::from(i32) -> Self impl core::convert::From for vortex_array::scalar::DecimalValue -pub fn vortex_array::scalar::DecimalValue::from(value: i64) -> Self +pub fn vortex_array::scalar::DecimalValue::from(i64) -> Self impl core::convert::From for vortex_array::scalar::DecimalValue -pub fn vortex_array::scalar::DecimalValue::from(value: i8) -> Self +pub fn vortex_array::scalar::DecimalValue::from(i8) -> Self impl core::convert::From for vortex_array::scalar::DecimalValue -pub fn vortex_array::scalar::DecimalValue::from(value: u16) -> Self +pub fn vortex_array::scalar::DecimalValue::from(u16) -> Self impl core::convert::From for vortex_array::scalar::DecimalValue -pub fn vortex_array::scalar::DecimalValue::from(value: u32) -> Self +pub fn vortex_array::scalar::DecimalValue::from(u32) -> Self impl core::convert::From for vortex_array::scalar::DecimalValue -pub fn vortex_array::scalar::DecimalValue::from(value: u64) -> Self +pub fn vortex_array::scalar::DecimalValue::from(u64) -> Self impl core::convert::From for vortex_array::scalar::DecimalValue -pub fn vortex_array::scalar::DecimalValue::from(value: u8) -> Self +pub fn vortex_array::scalar::DecimalValue::from(u8) -> Self impl core::convert::From for vortex_array::scalar::DecimalValue -pub fn vortex_array::scalar::DecimalValue::from(value: vortex_array::dtype::i256) -> Self +pub fn vortex_array::scalar::DecimalValue::from(vortex_array::dtype::i256) -> Self impl core::convert::From for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: vortex_array::scalar::DecimalValue) -> Self +pub fn vortex_array::scalar::Scalar::from(vortex_array::scalar::DecimalValue) -> Self impl core::convert::From for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::from(value: vortex_array::scalar::DecimalValue) -> Self +pub fn vortex_array::scalar::ScalarValue::from(vortex_array::scalar::DecimalValue) -> Self impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for vortex_array::scalar::DecimalValue pub type vortex_array::scalar::DecimalValue::Error = vortex_error::VortexError -pub fn vortex_array::scalar::DecimalValue::try_from(scalar: &vortex_array::scalar::Scalar) -> core::result::Result +pub fn vortex_array::scalar::DecimalValue::try_from(&vortex_array::scalar::Scalar) -> core::result::Result impl core::convert::TryFrom for vortex_array::scalar::DecimalValue pub type vortex_array::scalar::DecimalValue::Error = vortex_error::VortexError -pub fn vortex_array::scalar::DecimalValue::try_from(scalar: vortex_array::scalar::Scalar) -> core::result::Result +pub fn vortex_array::scalar::DecimalValue::try_from(vortex_array::scalar::Scalar) -> core::result::Result impl core::fmt::Debug for vortex_array::scalar::DecimalValue -pub fn vortex_array::scalar::DecimalValue::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::DecimalValue::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::scalar::DecimalValue -pub fn vortex_array::scalar::DecimalValue::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::DecimalValue::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::scalar::DecimalValue -pub fn vortex_array::scalar::DecimalValue::hash(&self, state: &mut H) +pub fn vortex_array::scalar::DecimalValue::hash(&self, &mut H) impl core::marker::Copy for vortex_array::scalar::DecimalValue @@ -13852,19 +14050,19 @@ impl core::cmp::Eq for vortex_array::scalar::NumericOperator impl core::cmp::PartialEq for vortex_array::scalar::NumericOperator -pub fn vortex_array::scalar::NumericOperator::eq(&self, other: &vortex_array::scalar::NumericOperator) -> bool +pub fn vortex_array::scalar::NumericOperator::eq(&self, &vortex_array::scalar::NumericOperator) -> bool impl core::convert::From for vortex_array::scalar_fn::fns::operators::Operator -pub fn vortex_array::scalar_fn::fns::operators::Operator::from(op: vortex_array::scalar::NumericOperator) -> Self +pub fn vortex_array::scalar_fn::fns::operators::Operator::from(vortex_array::scalar::NumericOperator) -> Self impl core::fmt::Debug for vortex_array::scalar::NumericOperator -pub fn vortex_array::scalar::NumericOperator::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::NumericOperator::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::scalar::NumericOperator -pub fn vortex_array::scalar::NumericOperator::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::NumericOperator::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_array::scalar::NumericOperator @@ -13920,7 +14118,7 @@ pub fn vortex_array::scalar::PValue::as_u8(self) -> core::option::Option pub fn vortex_array::scalar::PValue::cast(&self) -> vortex_error::VortexResult -pub fn vortex_array::scalar::PValue::is_instance_of(&self, ptype: &vortex_array::dtype::PType) -> bool +pub fn vortex_array::scalar::PValue::is_instance_of(&self, &vortex_array::dtype::PType) -> bool pub fn vortex_array::scalar::PValue::is_nan(&self) -> bool @@ -13928,9 +14126,9 @@ pub fn vortex_array::scalar::PValue::is_zero(&self) -> bool pub fn vortex_array::scalar::PValue::ptype(&self) -> vortex_array::dtype::PType -pub fn vortex_array::scalar::PValue::reinterpret_cast(&self, ptype: vortex_array::dtype::PType) -> Self +pub fn vortex_array::scalar::PValue::reinterpret_cast(&self, vortex_array::dtype::PType) -> Self -pub fn vortex_array::scalar::PValue::zero(ptype: &vortex_array::dtype::PType) -> vortex_array::scalar::PValue +pub fn vortex_array::scalar::PValue::zero(&vortex_array::dtype::PType) -> vortex_array::scalar::PValue impl core::clone::Clone for vortex_array::scalar::PValue @@ -13940,151 +14138,151 @@ impl core::cmp::Eq for vortex_array::scalar::PValue impl core::cmp::PartialEq for vortex_array::scalar::PValue -pub fn vortex_array::scalar::PValue::eq(&self, other: &Self) -> bool +pub fn vortex_array::scalar::PValue::eq(&self, &Self) -> bool impl core::cmp::PartialOrd for vortex_array::scalar::PValue -pub fn vortex_array::scalar::PValue::partial_cmp(&self, other: &Self) -> core::option::Option +pub fn vortex_array::scalar::PValue::partial_cmp(&self, &Self) -> core::option::Option impl core::convert::From<&vortex_array::scalar::PValue> for vortex_proto::scalar::ScalarValue -pub fn vortex_proto::scalar::ScalarValue::from(value: &vortex_array::scalar::PValue) -> Self +pub fn vortex_proto::scalar::ScalarValue::from(&vortex_array::scalar::PValue) -> Self impl core::convert::From for vortex_array::scalar::PValue -pub fn vortex_array::scalar::PValue::from(value: f32) -> Self +pub fn vortex_array::scalar::PValue::from(f32) -> Self impl core::convert::From for vortex_array::scalar::PValue -pub fn vortex_array::scalar::PValue::from(value: f64) -> Self +pub fn vortex_array::scalar::PValue::from(f64) -> Self impl core::convert::From for vortex_array::scalar::PValue -pub fn vortex_array::scalar::PValue::from(value: half::binary16::f16) -> Self +pub fn vortex_array::scalar::PValue::from(half::binary16::f16) -> Self impl core::convert::From for vortex_array::scalar::PValue -pub fn vortex_array::scalar::PValue::from(value: i16) -> Self +pub fn vortex_array::scalar::PValue::from(i16) -> Self impl core::convert::From for vortex_array::scalar::PValue -pub fn vortex_array::scalar::PValue::from(value: i32) -> Self +pub fn vortex_array::scalar::PValue::from(i32) -> Self impl core::convert::From for vortex_array::scalar::PValue -pub fn vortex_array::scalar::PValue::from(value: i64) -> Self +pub fn vortex_array::scalar::PValue::from(i64) -> Self impl core::convert::From for vortex_array::scalar::PValue -pub fn vortex_array::scalar::PValue::from(value: i8) -> Self +pub fn vortex_array::scalar::PValue::from(i8) -> Self impl core::convert::From for vortex_array::scalar::PValue -pub fn vortex_array::scalar::PValue::from(value: u16) -> Self +pub fn vortex_array::scalar::PValue::from(u16) -> Self impl core::convert::From for vortex_array::scalar::PValue -pub fn vortex_array::scalar::PValue::from(value: u32) -> Self +pub fn vortex_array::scalar::PValue::from(u32) -> Self impl core::convert::From for vortex_array::scalar::PValue -pub fn vortex_array::scalar::PValue::from(value: u64) -> Self +pub fn vortex_array::scalar::PValue::from(u64) -> Self impl core::convert::From for vortex_array::scalar::PValue -pub fn vortex_array::scalar::PValue::from(value: u8) -> Self +pub fn vortex_array::scalar::PValue::from(u8) -> Self impl core::convert::From for vortex_array::scalar::PValue -pub fn vortex_array::scalar::PValue::from(value: usize) -> vortex_array::scalar::PValue +pub fn vortex_array::scalar::PValue::from(usize) -> vortex_array::scalar::PValue impl core::convert::From for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::from(value: vortex_array::scalar::PValue) -> Self +pub fn vortex_array::scalar::ScalarValue::from(vortex_array::scalar::PValue) -> Self impl core::convert::TryFrom for f32 pub type f32::Error = vortex_error::VortexError -pub fn f32::try_from(value: vortex_array::scalar::PValue) -> core::result::Result +pub fn f32::try_from(vortex_array::scalar::PValue) -> core::result::Result impl core::convert::TryFrom for f64 pub type f64::Error = vortex_error::VortexError -pub fn f64::try_from(value: vortex_array::scalar::PValue) -> core::result::Result +pub fn f64::try_from(vortex_array::scalar::PValue) -> core::result::Result impl core::convert::TryFrom for half::binary16::f16 pub type half::binary16::f16::Error = vortex_error::VortexError -pub fn half::binary16::f16::try_from(value: vortex_array::scalar::PValue) -> core::result::Result +pub fn half::binary16::f16::try_from(vortex_array::scalar::PValue) -> core::result::Result impl core::convert::TryFrom for i16 pub type i16::Error = vortex_error::VortexError -pub fn i16::try_from(value: vortex_array::scalar::PValue) -> core::result::Result +pub fn i16::try_from(vortex_array::scalar::PValue) -> core::result::Result impl core::convert::TryFrom for i32 pub type i32::Error = vortex_error::VortexError -pub fn i32::try_from(value: vortex_array::scalar::PValue) -> core::result::Result +pub fn i32::try_from(vortex_array::scalar::PValue) -> core::result::Result impl core::convert::TryFrom for i64 pub type i64::Error = vortex_error::VortexError -pub fn i64::try_from(value: vortex_array::scalar::PValue) -> core::result::Result +pub fn i64::try_from(vortex_array::scalar::PValue) -> core::result::Result impl core::convert::TryFrom for i8 pub type i8::Error = vortex_error::VortexError -pub fn i8::try_from(value: vortex_array::scalar::PValue) -> core::result::Result +pub fn i8::try_from(vortex_array::scalar::PValue) -> core::result::Result impl core::convert::TryFrom for u16 pub type u16::Error = vortex_error::VortexError -pub fn u16::try_from(value: vortex_array::scalar::PValue) -> core::result::Result +pub fn u16::try_from(vortex_array::scalar::PValue) -> core::result::Result impl core::convert::TryFrom for u32 pub type u32::Error = vortex_error::VortexError -pub fn u32::try_from(value: vortex_array::scalar::PValue) -> core::result::Result +pub fn u32::try_from(vortex_array::scalar::PValue) -> core::result::Result impl core::convert::TryFrom for u64 pub type u64::Error = vortex_error::VortexError -pub fn u64::try_from(value: vortex_array::scalar::PValue) -> core::result::Result +pub fn u64::try_from(vortex_array::scalar::PValue) -> core::result::Result impl core::convert::TryFrom for u8 pub type u8::Error = vortex_error::VortexError -pub fn u8::try_from(value: vortex_array::scalar::PValue) -> core::result::Result +pub fn u8::try_from(vortex_array::scalar::PValue) -> core::result::Result impl core::convert::TryFrom for usize pub type usize::Error = vortex_error::VortexError -pub fn usize::try_from(value: vortex_array::scalar::PValue) -> core::result::Result +pub fn usize::try_from(vortex_array::scalar::PValue) -> core::result::Result impl core::fmt::Debug for vortex_array::scalar::PValue -pub fn vortex_array::scalar::PValue::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::PValue::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::scalar::PValue -pub fn vortex_array::scalar::PValue::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::PValue::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::scalar::PValue -pub fn vortex_array::scalar::PValue::hash(&self, state: &mut H) +pub fn vortex_array::scalar::PValue::hash(&self, &mut H) impl core::marker::Copy for vortex_array::scalar::PValue @@ -14094,17 +14292,17 @@ pub fn vortex_array::scalar::PValue::to_le_bytes(&self) -> &[u8] impl vortex_array::search_sorted::IndexOrd for vortex_array::variants::PrimitiveTyped<'_> -pub fn vortex_array::variants::PrimitiveTyped<'_>::index_cmp(&self, idx: usize, elem: &vortex_array::scalar::PValue) -> vortex_error::VortexResult> +pub fn vortex_array::variants::PrimitiveTyped<'_>::index_cmp(&self, usize, &vortex_array::scalar::PValue) -> vortex_error::VortexResult> -pub fn vortex_array::variants::PrimitiveTyped<'_>::index_ge(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::variants::PrimitiveTyped<'_>::index_ge(&self, usize, &V) -> vortex_error::VortexResult -pub fn vortex_array::variants::PrimitiveTyped<'_>::index_gt(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::variants::PrimitiveTyped<'_>::index_gt(&self, usize, &V) -> vortex_error::VortexResult -pub fn vortex_array::variants::PrimitiveTyped<'_>::index_le(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::variants::PrimitiveTyped<'_>::index_le(&self, usize, &V) -> vortex_error::VortexResult pub fn vortex_array::variants::PrimitiveTyped<'_>::index_len(&self) -> usize -pub fn vortex_array::variants::PrimitiveTyped<'_>::index_lt(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::variants::PrimitiveTyped<'_>::index_lt(&self, usize, &V) -> vortex_error::VortexResult pub enum vortex_array::scalar::ScalarValue @@ -14114,10 +14312,10 @@ pub vortex_array::scalar::ScalarValue::Bool(bool) pub vortex_array::scalar::ScalarValue::Decimal(vortex_array::scalar::DecimalValue) -pub vortex_array::scalar::ScalarValue::List(alloc::vec::Vec>) - pub vortex_array::scalar::ScalarValue::Primitive(vortex_array::scalar::PValue) +pub vortex_array::scalar::ScalarValue::Tuple(alloc::vec::Vec>) + pub vortex_array::scalar::ScalarValue::Utf8(vortex_buffer::string::BufferString) pub vortex_array::scalar::ScalarValue::Variant(alloc::boxed::Box) @@ -14154,15 +14352,15 @@ pub fn vortex_array::scalar::ScalarValue::into_variant(self) -> vortex_array::sc impl vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::from_proto(value: &vortex_proto::scalar::ScalarValue, dtype: &vortex_array::dtype::DType, session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::scalar::ScalarValue::from_proto(&vortex_proto::scalar::ScalarValue, &vortex_array::dtype::DType, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::scalar::ScalarValue::from_proto_bytes(bytes: &[u8], dtype: &vortex_array::dtype::DType, session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::scalar::ScalarValue::from_proto_bytes(&[u8], &vortex_array::dtype::DType, &vortex_session::VortexSession) -> vortex_error::VortexResult> impl vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::to_proto(this: core::option::Option<&Self>) -> vortex_proto::scalar::ScalarValue +pub fn vortex_array::scalar::ScalarValue::to_proto(core::option::Option<&Self>) -> vortex_proto::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::to_proto_bytes(value: core::option::Option<&vortex_array::scalar::ScalarValue>) -> B +pub fn vortex_array::scalar::ScalarValue::to_proto_bytes(core::option::Option<&vortex_array::scalar::ScalarValue>) -> B impl core::clone::Clone for vortex_array::scalar::ScalarValue @@ -14172,185 +14370,181 @@ impl core::cmp::Eq for vortex_array::scalar::ScalarValue impl core::cmp::PartialEq for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::eq(&self, other: &vortex_array::scalar::ScalarValue) -> bool - -impl core::cmp::PartialOrd for vortex_array::scalar::ScalarValue - -pub fn vortex_array::scalar::ScalarValue::partial_cmp(&self, other: &Self) -> core::option::Option +pub fn vortex_array::scalar::ScalarValue::eq(&self, &vortex_array::scalar::ScalarValue) -> bool impl core::convert::From<&[u8]> for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::from(value: &[u8]) -> Self +pub fn vortex_array::scalar::ScalarValue::from(&[u8]) -> Self impl core::convert::From<&str> for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::from(value: &str) -> Self +pub fn vortex_array::scalar::ScalarValue::from(&str) -> Self impl core::convert::From<&vortex_array::scalar::ScalarValue> for vortex_proto::scalar::ScalarValue -pub fn vortex_proto::scalar::ScalarValue::from(value: &vortex_array::scalar::ScalarValue) -> Self +pub fn vortex_proto::scalar::ScalarValue::from(&vortex_array::scalar::ScalarValue) -> Self impl core::convert::From for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::from(value: alloc::string::String) -> Self +pub fn vortex_array::scalar::ScalarValue::from(alloc::string::String) -> Self impl core::convert::From for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::from(value: bool) -> Self +pub fn vortex_array::scalar::ScalarValue::from(bool) -> Self impl core::convert::From for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::from(value: f32) -> Self +pub fn vortex_array::scalar::ScalarValue::from(f32) -> Self impl core::convert::From for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::from(value: f64) -> Self +pub fn vortex_array::scalar::ScalarValue::from(f64) -> Self impl core::convert::From for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::from(value: half::binary16::f16) -> Self +pub fn vortex_array::scalar::ScalarValue::from(half::binary16::f16) -> Self impl core::convert::From for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::from(value: i16) -> Self +pub fn vortex_array::scalar::ScalarValue::from(i16) -> Self impl core::convert::From for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::from(value: i32) -> Self +pub fn vortex_array::scalar::ScalarValue::from(i32) -> Self impl core::convert::From for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::from(value: i64) -> Self +pub fn vortex_array::scalar::ScalarValue::from(i64) -> Self impl core::convert::From for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::from(value: i8) -> Self +pub fn vortex_array::scalar::ScalarValue::from(i8) -> Self impl core::convert::From for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::from(value: u16) -> Self +pub fn vortex_array::scalar::ScalarValue::from(u16) -> Self impl core::convert::From for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::from(value: u32) -> Self +pub fn vortex_array::scalar::ScalarValue::from(u32) -> Self impl core::convert::From for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::from(value: u64) -> Self +pub fn vortex_array::scalar::ScalarValue::from(u64) -> Self impl core::convert::From for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::from(value: u8) -> Self +pub fn vortex_array::scalar::ScalarValue::from(u8) -> Self impl core::convert::From for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::from(value: usize) -> Self +pub fn vortex_array::scalar::ScalarValue::from(usize) -> Self impl core::convert::From for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::from(value: vortex_array::scalar::DecimalValue) -> Self +pub fn vortex_array::scalar::ScalarValue::from(vortex_array::scalar::DecimalValue) -> Self impl core::convert::From for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::from(value: vortex_array::scalar::PValue) -> Self +pub fn vortex_array::scalar::ScalarValue::from(vortex_array::scalar::PValue) -> Self impl core::convert::From> for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::from(value: vortex_buffer::ByteBuffer) -> Self +pub fn vortex_array::scalar::ScalarValue::from(vortex_buffer::ByteBuffer) -> Self impl core::convert::From for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::from(value: vortex_buffer::string::BufferString) -> Self +pub fn vortex_array::scalar::ScalarValue::from(vortex_buffer::string::BufferString) -> Self impl core::convert::TryFrom<&vortex_array::scalar::ScalarValue> for f32 pub type f32::Error = vortex_error::VortexError -pub fn f32::try_from(value: &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult +pub fn f32::try_from(&vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::ScalarValue> for f64 pub type f64::Error = vortex_error::VortexError -pub fn f64::try_from(value: &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult +pub fn f64::try_from(&vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::ScalarValue> for half::binary16::f16 pub type half::binary16::f16::Error = vortex_error::VortexError -pub fn half::binary16::f16::try_from(value: &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult +pub fn half::binary16::f16::try_from(&vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::ScalarValue> for i16 pub type i16::Error = vortex_error::VortexError -pub fn i16::try_from(value: &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult +pub fn i16::try_from(&vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::ScalarValue> for i32 pub type i32::Error = vortex_error::VortexError -pub fn i32::try_from(value: &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult +pub fn i32::try_from(&vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::ScalarValue> for i64 pub type i64::Error = vortex_error::VortexError -pub fn i64::try_from(value: &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult +pub fn i64::try_from(&vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::ScalarValue> for i8 pub type i8::Error = vortex_error::VortexError -pub fn i8::try_from(value: &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult +pub fn i8::try_from(&vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::ScalarValue> for u16 pub type u16::Error = vortex_error::VortexError -pub fn u16::try_from(value: &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult +pub fn u16::try_from(&vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::ScalarValue> for u32 pub type u32::Error = vortex_error::VortexError -pub fn u32::try_from(value: &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult +pub fn u32::try_from(&vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::ScalarValue> for u64 pub type u64::Error = vortex_error::VortexError -pub fn u64::try_from(value: &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult +pub fn u64::try_from(&vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::ScalarValue> for u8 pub type u8::Error = vortex_error::VortexError -pub fn u8::try_from(value: &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult +pub fn u8::try_from(&vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::ScalarValue> for usize pub type usize::Error = vortex_error::VortexError -pub fn usize::try_from(value: &vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult +pub fn usize::try_from(&vortex_array::scalar::ScalarValue) -> vortex_error::VortexResult impl core::fmt::Debug for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::ScalarValue::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::ScalarValue::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::scalar::ScalarValue -pub fn vortex_array::scalar::ScalarValue::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::scalar::ScalarValue::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_array::scalar::ScalarValue impl core::convert::From> for vortex_array::scalar::ScalarValue where T: vortex_array::dtype::NativeDType, vortex_array::scalar::Scalar: core::convert::From -pub fn vortex_array::scalar::ScalarValue::from(vec: alloc::vec::Vec) -> Self +pub fn vortex_array::scalar::ScalarValue::from(alloc::vec::Vec) -> Self pub struct vortex_array::scalar::BinaryScalar<'a> @@ -14362,7 +14556,7 @@ pub fn vortex_array::scalar::BinaryScalar<'a>::is_empty(&self) -> core::option:: pub fn vortex_array::scalar::BinaryScalar<'a>::len(&self) -> core::option::Option -pub fn vortex_array::scalar::BinaryScalar<'a>::try_new(dtype: &'a vortex_array::dtype::DType, value: core::option::Option<&'a vortex_array::scalar::ScalarValue>) -> vortex_error::VortexResult +pub fn vortex_array::scalar::BinaryScalar<'a>::try_new(&'a vortex_array::dtype::DType, core::option::Option<&'a vortex_array::scalar::ScalarValue>) -> vortex_error::VortexResult pub fn vortex_array::scalar::BinaryScalar<'a>::value(&self) -> core::option::Option<&'a vortex_buffer::ByteBuffer> @@ -14370,19 +14564,19 @@ impl core::cmp::Eq for vortex_array::scalar::BinaryScalar<'_> impl core::cmp::Ord for vortex_array::scalar::BinaryScalar<'_> -pub fn vortex_array::scalar::BinaryScalar<'_>::cmp(&self, other: &Self) -> core::cmp::Ordering +pub fn vortex_array::scalar::BinaryScalar<'_>::cmp(&self, &Self) -> core::cmp::Ordering impl core::cmp::PartialEq for vortex_array::scalar::BinaryScalar<'_> -pub fn vortex_array::scalar::BinaryScalar<'_>::eq(&self, other: &Self) -> bool +pub fn vortex_array::scalar::BinaryScalar<'_>::eq(&self, &Self) -> bool impl core::cmp::PartialOrd for vortex_array::scalar::BinaryScalar<'_> -pub fn vortex_array::scalar::BinaryScalar<'_>::partial_cmp(&self, other: &Self) -> core::option::Option +pub fn vortex_array::scalar::BinaryScalar<'_>::partial_cmp(&self, &Self) -> core::option::Option impl core::fmt::Display for vortex_array::scalar::BinaryScalar<'_> -pub fn vortex_array::scalar::BinaryScalar<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::BinaryScalar<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl<'a> core::clone::Clone for vortex_array::scalar::BinaryScalar<'a> @@ -14392,15 +14586,17 @@ impl<'a> core::convert::TryFrom<&'a vortex_array::scalar::Scalar> for vortex_arr pub type vortex_array::scalar::BinaryScalar<'a>::Error = vortex_error::VortexError -pub fn vortex_array::scalar::BinaryScalar<'a>::try_from(value: &'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn vortex_array::scalar::BinaryScalar<'a>::try_from(&'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl<'a> core::fmt::Debug for vortex_array::scalar::BinaryScalar<'a> -pub fn vortex_array::scalar::BinaryScalar<'a>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::BinaryScalar<'a>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl<'a> core::hash::Hash for vortex_array::scalar::BinaryScalar<'a> -pub fn vortex_array::scalar::BinaryScalar<'a>::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::scalar::BinaryScalar<'a>::hash<__H: core::hash::Hasher>(&self, &mut __H) + +impl<'a> core::marker::Copy for vortex_array::scalar::BinaryScalar<'a> pub struct vortex_array::scalar::BoolScalar<'a> @@ -14412,25 +14608,25 @@ pub fn vortex_array::scalar::BoolScalar<'a>::into_scalar(self) -> vortex_array:: pub fn vortex_array::scalar::BoolScalar<'a>::invert(self) -> vortex_array::scalar::BoolScalar<'a> -pub fn vortex_array::scalar::BoolScalar<'a>::try_new(dtype: &'a vortex_array::dtype::DType, value: core::option::Option<&vortex_array::scalar::ScalarValue>) -> vortex_error::VortexResult +pub fn vortex_array::scalar::BoolScalar<'a>::try_new(&'a vortex_array::dtype::DType, core::option::Option<&vortex_array::scalar::ScalarValue>) -> vortex_error::VortexResult pub fn vortex_array::scalar::BoolScalar<'a>::value(&self) -> core::option::Option impl core::cmp::Ord for vortex_array::scalar::BoolScalar<'_> -pub fn vortex_array::scalar::BoolScalar<'_>::cmp(&self, other: &Self) -> core::cmp::Ordering +pub fn vortex_array::scalar::BoolScalar<'_>::cmp(&self, &Self) -> core::cmp::Ordering impl core::cmp::PartialEq for vortex_array::scalar::BoolScalar<'_> -pub fn vortex_array::scalar::BoolScalar<'_>::eq(&self, other: &Self) -> bool +pub fn vortex_array::scalar::BoolScalar<'_>::eq(&self, &Self) -> bool impl core::cmp::PartialOrd for vortex_array::scalar::BoolScalar<'_> -pub fn vortex_array::scalar::BoolScalar<'_>::partial_cmp(&self, other: &Self) -> core::option::Option +pub fn vortex_array::scalar::BoolScalar<'_>::partial_cmp(&self, &Self) -> core::option::Option impl core::fmt::Display for vortex_array::scalar::BoolScalar<'_> -pub fn vortex_array::scalar::BoolScalar<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::BoolScalar<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl<'a> core::clone::Clone for vortex_array::scalar::BoolScalar<'a> @@ -14442,21 +14638,23 @@ impl<'a> core::convert::TryFrom<&'a vortex_array::scalar::Scalar> for vortex_arr pub type vortex_array::scalar::BoolScalar<'a>::Error = vortex_error::VortexError -pub fn vortex_array::scalar::BoolScalar<'a>::try_from(value: &'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn vortex_array::scalar::BoolScalar<'a>::try_from(&'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl<'a> core::fmt::Debug for vortex_array::scalar::BoolScalar<'a> -pub fn vortex_array::scalar::BoolScalar<'a>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::BoolScalar<'a>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl<'a> core::hash::Hash for vortex_array::scalar::BoolScalar<'a> -pub fn vortex_array::scalar::BoolScalar<'a>::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::scalar::BoolScalar<'a>::hash<__H: core::hash::Hasher>(&self, &mut __H) + +impl<'a> core::marker::Copy for vortex_array::scalar::BoolScalar<'a> pub struct vortex_array::scalar::DecimalScalar<'a> impl<'a> vortex_array::scalar::DecimalScalar<'a> -pub fn vortex_array::scalar::DecimalScalar<'a>::checked_binary_numeric(&self, other: &vortex_array::scalar::DecimalScalar<'a>, op: vortex_array::scalar::NumericOperator) -> core::option::Option> +pub fn vortex_array::scalar::DecimalScalar<'a>::checked_binary_numeric(&self, &vortex_array::scalar::DecimalScalar<'a>, vortex_array::scalar::NumericOperator) -> core::option::Option> pub fn vortex_array::scalar::DecimalScalar<'a>::decimal_value(&self) -> core::option::Option @@ -14464,97 +14662,97 @@ pub fn vortex_array::scalar::DecimalScalar<'a>::dtype(&self) -> &'a vortex_array pub fn vortex_array::scalar::DecimalScalar<'a>::is_zero(&self) -> core::option::Option -pub fn vortex_array::scalar::DecimalScalar<'a>::try_new(dtype: &'a vortex_array::dtype::DType, value: core::option::Option<&vortex_array::scalar::ScalarValue>) -> vortex_error::VortexResult +pub fn vortex_array::scalar::DecimalScalar<'a>::try_new(&'a vortex_array::dtype::DType, core::option::Option<&vortex_array::scalar::ScalarValue>) -> vortex_error::VortexResult impl core::cmp::Eq for vortex_array::scalar::DecimalScalar<'_> impl core::cmp::PartialEq for vortex_array::scalar::DecimalScalar<'_> -pub fn vortex_array::scalar::DecimalScalar<'_>::eq(&self, other: &Self) -> bool +pub fn vortex_array::scalar::DecimalScalar<'_>::eq(&self, &Self) -> bool impl core::cmp::PartialOrd for vortex_array::scalar::DecimalScalar<'_> -pub fn vortex_array::scalar::DecimalScalar<'_>::partial_cmp(&self, other: &Self) -> core::option::Option +pub fn vortex_array::scalar::DecimalScalar<'_>::partial_cmp(&self, &Self) -> core::option::Option impl core::convert::From> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(ds: vortex_array::scalar::DecimalScalar<'_>) -> Self +pub fn vortex_array::scalar::Scalar::from(vortex_array::scalar::DecimalScalar<'_>) -> Self impl core::convert::TryFrom> for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(value: vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result +pub fn core::option::Option::try_from(vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result impl core::convert::TryFrom> for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(value: vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result +pub fn core::option::Option::try_from(vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result impl core::convert::TryFrom> for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(value: vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result +pub fn core::option::Option::try_from(vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result impl core::convert::TryFrom> for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(value: vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result +pub fn core::option::Option::try_from(vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result impl core::convert::TryFrom> for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(value: vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result +pub fn core::option::Option::try_from(vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result impl core::convert::TryFrom> for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(value: vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result +pub fn core::option::Option::try_from(vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result impl core::convert::TryFrom> for i128 pub type i128::Error = vortex_error::VortexError -pub fn i128::try_from(value: vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result +pub fn i128::try_from(vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result impl core::convert::TryFrom> for i16 pub type i16::Error = vortex_error::VortexError -pub fn i16::try_from(value: vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result +pub fn i16::try_from(vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result impl core::convert::TryFrom> for i32 pub type i32::Error = vortex_error::VortexError -pub fn i32::try_from(value: vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result +pub fn i32::try_from(vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result impl core::convert::TryFrom> for i64 pub type i64::Error = vortex_error::VortexError -pub fn i64::try_from(value: vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result +pub fn i64::try_from(vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result impl core::convert::TryFrom> for i8 pub type i8::Error = vortex_error::VortexError -pub fn i8::try_from(value: vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result +pub fn i8::try_from(vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result impl core::convert::TryFrom> for vortex_array::dtype::i256 pub type vortex_array::dtype::i256::Error = vortex_error::VortexError -pub fn vortex_array::dtype::i256::try_from(value: vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result +pub fn vortex_array::dtype::i256::try_from(vortex_array::scalar::DecimalScalar<'_>) -> core::result::Result impl core::fmt::Display for vortex_array::scalar::DecimalScalar<'_> -pub fn vortex_array::scalar::DecimalScalar<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::DecimalScalar<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl<'a> core::clone::Clone for vortex_array::scalar::DecimalScalar<'a> @@ -14564,15 +14762,15 @@ impl<'a> core::convert::TryFrom<&'a vortex_array::scalar::Scalar> for vortex_arr pub type vortex_array::scalar::DecimalScalar<'a>::Error = vortex_error::VortexError -pub fn vortex_array::scalar::DecimalScalar<'a>::try_from(value: &'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn vortex_array::scalar::DecimalScalar<'a>::try_from(&'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl<'a> core::fmt::Debug for vortex_array::scalar::DecimalScalar<'a> -pub fn vortex_array::scalar::DecimalScalar<'a>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::DecimalScalar<'a>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl<'a> core::hash::Hash for vortex_array::scalar::DecimalScalar<'a> -pub fn vortex_array::scalar::DecimalScalar<'a>::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::scalar::DecimalScalar<'a>::hash<__H: core::hash::Hasher>(&self, &mut __H) impl<'a> core::marker::Copy for vortex_array::scalar::DecimalScalar<'a> @@ -14590,19 +14788,19 @@ impl core::cmp::Eq for vortex_array::scalar::ExtScalar<'_> impl core::cmp::PartialEq for vortex_array::scalar::ExtScalar<'_> -pub fn vortex_array::scalar::ExtScalar<'_>::eq(&self, other: &Self) -> bool +pub fn vortex_array::scalar::ExtScalar<'_>::eq(&self, &Self) -> bool impl core::cmp::PartialOrd for vortex_array::scalar::ExtScalar<'_> -pub fn vortex_array::scalar::ExtScalar<'_>::partial_cmp(&self, other: &Self) -> core::option::Option +pub fn vortex_array::scalar::ExtScalar<'_>::partial_cmp(&self, &Self) -> core::option::Option impl core::fmt::Display for vortex_array::scalar::ExtScalar<'_> -pub fn vortex_array::scalar::ExtScalar<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::ExtScalar<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::scalar::ExtScalar<'_> -pub fn vortex_array::scalar::ExtScalar<'_>::hash(&self, state: &mut H) +pub fn vortex_array::scalar::ExtScalar<'_>::hash(&self, &mut H) impl<'a> core::clone::Clone for vortex_array::scalar::ExtScalar<'a> @@ -14612,11 +14810,13 @@ impl<'a> core::convert::TryFrom<&'a vortex_array::scalar::Scalar> for vortex_arr pub type vortex_array::scalar::ExtScalar<'a>::Error = vortex_error::VortexError -pub fn vortex_array::scalar::ExtScalar<'a>::try_from(value: &'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn vortex_array::scalar::ExtScalar<'a>::try_from(&'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl<'a> core::fmt::Debug for vortex_array::scalar::ExtScalar<'a> -pub fn vortex_array::scalar::ExtScalar<'a>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::ExtScalar<'a>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +impl<'a> core::marker::Copy for vortex_array::scalar::ExtScalar<'a> pub struct vortex_array::scalar::ListScalar<'a> @@ -14624,7 +14824,7 @@ impl<'a> vortex_array::scalar::ListScalar<'a> pub fn vortex_array::scalar::ListScalar<'a>::dtype(&self) -> &'a vortex_array::dtype::DType -pub fn vortex_array::scalar::ListScalar<'a>::element(&self, idx: usize) -> core::option::Option +pub fn vortex_array::scalar::ListScalar<'a>::element(&self, usize) -> core::option::Option pub fn vortex_array::scalar::ListScalar<'a>::element_dtype(&self) -> &vortex_array::dtype::DType @@ -14636,25 +14836,25 @@ pub fn vortex_array::scalar::ListScalar<'a>::is_null(&self) -> bool pub fn vortex_array::scalar::ListScalar<'a>::len(&self) -> usize -pub fn vortex_array::scalar::ListScalar<'a>::try_new(dtype: &'a vortex_array::dtype::DType, value: core::option::Option<&'a vortex_array::scalar::ScalarValue>) -> vortex_error::VortexResult +pub fn vortex_array::scalar::ListScalar<'a>::try_new(&'a vortex_array::dtype::DType, core::option::Option<&'a vortex_array::scalar::ScalarValue>) -> vortex_error::VortexResult impl core::cmp::Eq for vortex_array::scalar::ListScalar<'_> impl core::cmp::PartialEq for vortex_array::scalar::ListScalar<'_> -pub fn vortex_array::scalar::ListScalar<'_>::eq(&self, other: &Self) -> bool +pub fn vortex_array::scalar::ListScalar<'_>::eq(&self, &Self) -> bool impl core::cmp::PartialOrd for vortex_array::scalar::ListScalar<'_> -pub fn vortex_array::scalar::ListScalar<'_>::partial_cmp(&self, other: &Self) -> core::option::Option +pub fn vortex_array::scalar::ListScalar<'_>::partial_cmp(&self, &Self) -> core::option::Option impl core::fmt::Display for vortex_array::scalar::ListScalar<'_> -pub fn vortex_array::scalar::ListScalar<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::ListScalar<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::scalar::ListScalar<'_> -pub fn vortex_array::scalar::ListScalar<'_>::hash(&self, state: &mut H) +pub fn vortex_array::scalar::ListScalar<'_>::hash(&self, &mut H) impl<'a> core::clone::Clone for vortex_array::scalar::ListScalar<'a> @@ -14664,11 +14864,13 @@ impl<'a> core::convert::TryFrom<&'a vortex_array::scalar::Scalar> for vortex_arr pub type vortex_array::scalar::ListScalar<'a>::Error = vortex_error::VortexError -pub fn vortex_array::scalar::ListScalar<'a>::try_from(value: &'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn vortex_array::scalar::ListScalar<'a>::try_from(&'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl<'a> core::fmt::Debug for vortex_array::scalar::ListScalar<'a> -pub fn vortex_array::scalar::ListScalar<'a>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::ListScalar<'a>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +impl<'a> core::marker::Copy for vortex_array::scalar::ListScalar<'a> pub struct vortex_array::scalar::PrimitiveScalar<'a> @@ -14688,7 +14890,7 @@ pub fn vortex_array::scalar::PrimitiveScalar<'a>::ptype(&self) -> vortex_array:: pub fn vortex_array::scalar::PrimitiveScalar<'a>::pvalue(&self) -> core::option::Option -pub fn vortex_array::scalar::PrimitiveScalar<'a>::try_new(dtype: &'a vortex_array::dtype::DType, value: core::option::Option<&vortex_array::scalar::ScalarValue>) -> vortex_error::VortexResult +pub fn vortex_array::scalar::PrimitiveScalar<'a>::try_new(&'a vortex_array::dtype::DType, core::option::Option<&vortex_array::scalar::ScalarValue>) -> vortex_error::VortexResult pub fn vortex_array::scalar::PrimitiveScalar<'a>::try_typed_value(&self) -> vortex_error::VortexResult> @@ -14696,45 +14898,45 @@ pub fn vortex_array::scalar::PrimitiveScalar<'a>::typed_value vortex_array::scalar::PrimitiveScalar<'a> -pub fn vortex_array::scalar::PrimitiveScalar<'a>::checked_binary_numeric(&self, other: &vortex_array::scalar::PrimitiveScalar<'a>, op: vortex_array::scalar::NumericOperator) -> core::option::Option> +pub fn vortex_array::scalar::PrimitiveScalar<'a>::checked_binary_numeric(&self, &vortex_array::scalar::PrimitiveScalar<'a>, vortex_array::scalar::NumericOperator) -> core::option::Option> impl core::cmp::Eq for vortex_array::scalar::PrimitiveScalar<'_> impl core::cmp::PartialEq for vortex_array::scalar::PrimitiveScalar<'_> -pub fn vortex_array::scalar::PrimitiveScalar<'_>::eq(&self, other: &Self) -> bool +pub fn vortex_array::scalar::PrimitiveScalar<'_>::eq(&self, &Self) -> bool impl core::cmp::PartialOrd for vortex_array::scalar::PrimitiveScalar<'_> -pub fn vortex_array::scalar::PrimitiveScalar<'_>::partial_cmp(&self, other: &Self) -> core::option::Option +pub fn vortex_array::scalar::PrimitiveScalar<'_>::partial_cmp(&self, &Self) -> core::option::Option impl core::convert::From> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(ps: vortex_array::scalar::PrimitiveScalar<'_>) -> Self +pub fn vortex_array::scalar::Scalar::from(vortex_array::scalar::PrimitiveScalar<'_>) -> Self impl core::fmt::Display for vortex_array::scalar::PrimitiveScalar<'_> -pub fn vortex_array::scalar::PrimitiveScalar<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::PrimitiveScalar<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::ops::arith::Add for vortex_array::scalar::PrimitiveScalar<'_> pub type vortex_array::scalar::PrimitiveScalar<'_>::Output = vortex_array::scalar::PrimitiveScalar<'_> -pub fn vortex_array::scalar::PrimitiveScalar<'_>::add(self, rhs: Self) -> Self::Output +pub fn vortex_array::scalar::PrimitiveScalar<'_>::add(self, Self) -> Self::Output impl core::ops::arith::Sub for vortex_array::scalar::PrimitiveScalar<'_> pub type vortex_array::scalar::PrimitiveScalar<'_>::Output = vortex_array::scalar::PrimitiveScalar<'_> -pub fn vortex_array::scalar::PrimitiveScalar<'_>::sub(self, rhs: Self) -> Self::Output +pub fn vortex_array::scalar::PrimitiveScalar<'_>::sub(self, Self) -> Self::Output impl num_traits::ops::checked::CheckedAdd for vortex_array::scalar::PrimitiveScalar<'_> -pub fn vortex_array::scalar::PrimitiveScalar<'_>::checked_add(&self, rhs: &Self) -> core::option::Option +pub fn vortex_array::scalar::PrimitiveScalar<'_>::checked_add(&self, &Self) -> core::option::Option impl num_traits::ops::checked::CheckedSub for vortex_array::scalar::PrimitiveScalar<'_> -pub fn vortex_array::scalar::PrimitiveScalar<'_>::checked_sub(&self, rhs: &Self) -> core::option::Option +pub fn vortex_array::scalar::PrimitiveScalar<'_>::checked_sub(&self, &Self) -> core::option::Option impl<'a> core::clone::Clone for vortex_array::scalar::PrimitiveScalar<'a> @@ -14744,15 +14946,15 @@ impl<'a> core::convert::TryFrom<&'a vortex_array::scalar::Scalar> for vortex_arr pub type vortex_array::scalar::PrimitiveScalar<'a>::Error = vortex_error::VortexError -pub fn vortex_array::scalar::PrimitiveScalar<'a>::try_from(value: &'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn vortex_array::scalar::PrimitiveScalar<'a>::try_from(&'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl<'a> core::fmt::Debug for vortex_array::scalar::PrimitiveScalar<'a> -pub fn vortex_array::scalar::PrimitiveScalar<'a>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::PrimitiveScalar<'a>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl<'a> core::hash::Hash for vortex_array::scalar::PrimitiveScalar<'a> -pub fn vortex_array::scalar::PrimitiveScalar<'a>::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::scalar::PrimitiveScalar<'a>::hash<__H: core::hash::Hasher>(&self, &mut __H) impl<'a> core::marker::Copy for vortex_array::scalar::PrimitiveScalar<'a> @@ -14762,11 +14964,11 @@ impl vortex_array::scalar::Scalar pub fn vortex_array::scalar::Scalar::approx_nbytes(&self) -> usize -pub fn vortex_array::scalar::Scalar::default_value(dtype: &vortex_array::dtype::DType) -> Self +pub fn vortex_array::scalar::Scalar::default_value(&vortex_array::dtype::DType) -> Self pub fn vortex_array::scalar::Scalar::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::scalar::Scalar::eq_ignore_nullability(&self, other: &Self) -> bool +pub fn vortex_array::scalar::Scalar::eq_ignore_nullability(&self, &Self) -> bool pub fn vortex_array::scalar::Scalar::into_parts(self) -> (vortex_array::dtype::DType, core::option::Option) @@ -14778,19 +14980,19 @@ pub fn vortex_array::scalar::Scalar::is_valid(&self) -> bool pub fn vortex_array::scalar::Scalar::is_zero(&self) -> core::option::Option -pub unsafe fn vortex_array::scalar::Scalar::new_unchecked(dtype: vortex_array::dtype::DType, value: core::option::Option) -> Self +pub unsafe fn vortex_array::scalar::Scalar::new_unchecked(vortex_array::dtype::DType, core::option::Option) -> Self -pub fn vortex_array::scalar::Scalar::null(dtype: vortex_array::dtype::DType) -> Self +pub fn vortex_array::scalar::Scalar::null(vortex_array::dtype::DType) -> Self pub fn vortex_array::scalar::Scalar::null_native() -> Self -pub fn vortex_array::scalar::Scalar::primitive_reinterpret_cast(&self, ptype: vortex_array::dtype::PType) -> vortex_error::VortexResult +pub fn vortex_array::scalar::Scalar::primitive_reinterpret_cast(&self, vortex_array::dtype::PType) -> vortex_error::VortexResult -pub fn vortex_array::scalar::Scalar::try_new(dtype: vortex_array::dtype::DType, value: core::option::Option) -> vortex_error::VortexResult +pub fn vortex_array::scalar::Scalar::try_new(vortex_array::dtype::DType, core::option::Option) -> vortex_error::VortexResult pub fn vortex_array::scalar::Scalar::value(&self) -> core::option::Option<&vortex_array::scalar::ScalarValue> -pub fn vortex_array::scalar::Scalar::zero_value(dtype: &vortex_array::dtype::DType) -> Self +pub fn vortex_array::scalar::Scalar::zero_value(&vortex_array::dtype::DType) -> Self impl vortex_array::scalar::Scalar @@ -14832,53 +15034,53 @@ pub fn vortex_array::scalar::Scalar::as_variant_opt(&self) -> core::option::Opti impl vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::binary(buffer: impl core::convert::Into, nullability: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::scalar::Scalar::binary(impl core::convert::Into, vortex_array::dtype::Nullability) -> Self -pub fn vortex_array::scalar::Scalar::bool(value: bool, nullability: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::scalar::Scalar::bool(bool, vortex_array::dtype::Nullability) -> Self -pub fn vortex_array::scalar::Scalar::decimal(value: vortex_array::scalar::DecimalValue, decimal_type: vortex_array::dtype::DecimalDType, nullability: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::scalar::Scalar::decimal(vortex_array::scalar::DecimalValue, vortex_array::dtype::DecimalDType, vortex_array::dtype::Nullability) -> Self -pub fn vortex_array::scalar::Scalar::extension(options: ::Metadata, storage_scalar: vortex_array::scalar::Scalar) -> Self +pub fn vortex_array::scalar::Scalar::extension(::Metadata, vortex_array::scalar::Scalar) -> Self -pub fn vortex_array::scalar::Scalar::extension_ref(ext_dtype: vortex_array::dtype::extension::ExtDTypeRef, storage_scalar: vortex_array::scalar::Scalar) -> Self +pub fn vortex_array::scalar::Scalar::extension_ref(vortex_array::dtype::extension::ExtDTypeRef, vortex_array::scalar::Scalar) -> Self -pub fn vortex_array::scalar::Scalar::fixed_size_list(element_dtype: impl core::convert::Into>, children: alloc::vec::Vec, nullability: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::scalar::Scalar::fixed_size_list(impl core::convert::Into>, alloc::vec::Vec, vortex_array::dtype::Nullability) -> Self -pub fn vortex_array::scalar::Scalar::list(element_dtype: impl core::convert::Into>, children: alloc::vec::Vec, nullability: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::scalar::Scalar::list(impl core::convert::Into>, alloc::vec::Vec, vortex_array::dtype::Nullability) -> Self -pub fn vortex_array::scalar::Scalar::list_empty(element_dtype: alloc::sync::Arc, nullability: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::scalar::Scalar::list_empty(alloc::sync::Arc, vortex_array::dtype::Nullability) -> Self -pub fn vortex_array::scalar::Scalar::primitive>(value: T, nullability: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::scalar::Scalar::primitive>(T, vortex_array::dtype::Nullability) -> Self -pub fn vortex_array::scalar::Scalar::primitive_value(value: vortex_array::scalar::PValue, ptype: vortex_array::dtype::PType, nullability: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::scalar::Scalar::primitive_value(vortex_array::scalar::PValue, vortex_array::dtype::PType, vortex_array::dtype::Nullability) -> Self -pub fn vortex_array::scalar::Scalar::try_utf8(str: B, nullability: vortex_array::dtype::Nullability) -> core::result::Result>::Error> where B: core::convert::TryInto +pub fn vortex_array::scalar::Scalar::try_utf8(B, vortex_array::dtype::Nullability) -> core::result::Result>::Error> where B: core::convert::TryInto -pub fn vortex_array::scalar::Scalar::utf8(str: B, nullability: vortex_array::dtype::Nullability) -> Self where B: core::convert::Into +pub fn vortex_array::scalar::Scalar::utf8(B, vortex_array::dtype::Nullability) -> Self where B: core::convert::Into -pub fn vortex_array::scalar::Scalar::variant(value: vortex_array::scalar::Scalar) -> Self +pub fn vortex_array::scalar::Scalar::variant(vortex_array::scalar::Scalar) -> Self impl vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::cast(&self, target_dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::scalar::Scalar::cast(&self, &vortex_array::dtype::DType) -> vortex_error::VortexResult pub fn vortex_array::scalar::Scalar::into_nullable(self) -> vortex_array::scalar::Scalar impl vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from_proto(value: &vortex_proto::scalar::Scalar, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar::Scalar::from_proto(&vortex_proto::scalar::Scalar, &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar::Scalar::from_proto_value(value: &vortex_proto::scalar::ScalarValue, dtype: &vortex_array::dtype::DType, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar::Scalar::from_proto_value(&vortex_proto::scalar::ScalarValue, &vortex_array::dtype::DType, &vortex_session::VortexSession) -> vortex_error::VortexResult impl vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::struct_(dtype: vortex_array::dtype::DType, children: impl core::iter::traits::collect::IntoIterator) -> Self +pub fn vortex_array::scalar::Scalar::struct_(vortex_array::dtype::DType, impl core::iter::traits::collect::IntoIterator) -> Self -pub unsafe fn vortex_array::scalar::Scalar::struct_unchecked(dtype: vortex_array::dtype::DType, children: impl core::iter::traits::collect::IntoIterator) -> Self +pub unsafe fn vortex_array::scalar::Scalar::struct_unchecked(vortex_array::dtype::DType, impl core::iter::traits::collect::IntoIterator) -> Self impl vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::validate(dtype: &vortex_array::dtype::DType, value: core::option::Option<&vortex_array::scalar::ScalarValue>) -> vortex_error::VortexResult<()> +pub fn vortex_array::scalar::Scalar::validate(&vortex_array::dtype::DType, core::option::Option<&vortex_array::scalar::ScalarValue>) -> vortex_error::VortexResult<()> impl core::clone::Clone for vortex_array::scalar::Scalar @@ -14888,539 +15090,539 @@ impl core::cmp::Eq for vortex_array::scalar::Scalar impl core::cmp::PartialEq for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::eq(&self, other: &Self) -> bool +pub fn vortex_array::scalar::Scalar::eq(&self, &Self) -> bool impl core::cmp::PartialOrd for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::partial_cmp(&self, other: &Self) -> core::option::Option +pub fn vortex_array::scalar::Scalar::partial_cmp(&self, &Self) -> core::option::Option impl core::convert::From<&[u8]> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: &[u8]) -> Self +pub fn vortex_array::scalar::Scalar::from(&[u8]) -> Self impl core::convert::From<&str> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: &str) -> Self +pub fn vortex_array::scalar::Scalar::from(&str) -> Self impl core::convert::From<&vortex_array::scalar::Scalar> for vortex_proto::scalar::Scalar -pub fn vortex_proto::scalar::Scalar::from(value: &vortex_array::scalar::Scalar) -> Self +pub fn vortex_proto::scalar::Scalar::from(&vortex_array::scalar::Scalar) -> Self impl core::convert::From for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: alloc::string::String) -> Self +pub fn vortex_array::scalar::Scalar::from(alloc::string::String) -> Self impl core::convert::From for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: bool) -> Self +pub fn vortex_array::scalar::Scalar::from(bool) -> Self impl core::convert::From> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: core::option::Option<&[u8]>) -> Self +pub fn vortex_array::scalar::Scalar::from(core::option::Option<&[u8]>) -> Self impl core::convert::From> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: core::option::Option<&str>) -> Self +pub fn vortex_array::scalar::Scalar::from(core::option::Option<&str>) -> Self impl core::convert::From> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: core::option::Option) -> Self +pub fn vortex_array::scalar::Scalar::from(core::option::Option) -> Self impl core::convert::From> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: core::option::Option) -> Self +pub fn vortex_array::scalar::Scalar::from(core::option::Option) -> Self impl core::convert::From> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: core::option::Option) -> Self +pub fn vortex_array::scalar::Scalar::from(core::option::Option) -> Self impl core::convert::From> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: core::option::Option) -> Self +pub fn vortex_array::scalar::Scalar::from(core::option::Option) -> Self impl core::convert::From> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: core::option::Option) -> Self +pub fn vortex_array::scalar::Scalar::from(core::option::Option) -> Self impl core::convert::From> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: core::option::Option) -> Self +pub fn vortex_array::scalar::Scalar::from(core::option::Option) -> Self impl core::convert::From> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: core::option::Option) -> Self +pub fn vortex_array::scalar::Scalar::from(core::option::Option) -> Self impl core::convert::From> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: core::option::Option) -> Self +pub fn vortex_array::scalar::Scalar::from(core::option::Option) -> Self impl core::convert::From> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: core::option::Option) -> Self +pub fn vortex_array::scalar::Scalar::from(core::option::Option) -> Self impl core::convert::From> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: core::option::Option) -> Self +pub fn vortex_array::scalar::Scalar::from(core::option::Option) -> Self impl core::convert::From> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: core::option::Option) -> Self +pub fn vortex_array::scalar::Scalar::from(core::option::Option) -> Self impl core::convert::From> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: core::option::Option) -> Self +pub fn vortex_array::scalar::Scalar::from(core::option::Option) -> Self impl core::convert::From> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: core::option::Option) -> Self +pub fn vortex_array::scalar::Scalar::from(core::option::Option) -> Self impl core::convert::From> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: core::option::Option) -> Self +pub fn vortex_array::scalar::Scalar::from(core::option::Option) -> Self impl core::convert::From> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: core::option::Option) -> Self +pub fn vortex_array::scalar::Scalar::from(core::option::Option) -> Self impl core::convert::From>> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: core::option::Option) -> Self +pub fn vortex_array::scalar::Scalar::from(core::option::Option) -> Self impl core::convert::From> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: core::option::Option) -> Self +pub fn vortex_array::scalar::Scalar::from(core::option::Option) -> Self impl core::convert::From for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: f32) -> Self +pub fn vortex_array::scalar::Scalar::from(f32) -> Self impl core::convert::From for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: f64) -> Self +pub fn vortex_array::scalar::Scalar::from(f64) -> Self impl core::convert::From for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: half::binary16::f16) -> Self +pub fn vortex_array::scalar::Scalar::from(half::binary16::f16) -> Self impl core::convert::From for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: i16) -> Self +pub fn vortex_array::scalar::Scalar::from(i16) -> Self impl core::convert::From for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: i32) -> Self +pub fn vortex_array::scalar::Scalar::from(i32) -> Self impl core::convert::From for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: i64) -> Self +pub fn vortex_array::scalar::Scalar::from(i64) -> Self impl core::convert::From for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: i8) -> Self +pub fn vortex_array::scalar::Scalar::from(i8) -> Self impl core::convert::From for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: u16) -> Self +pub fn vortex_array::scalar::Scalar::from(u16) -> Self impl core::convert::From for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: u32) -> Self +pub fn vortex_array::scalar::Scalar::from(u32) -> Self impl core::convert::From for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: u64) -> Self +pub fn vortex_array::scalar::Scalar::from(u64) -> Self impl core::convert::From for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: u8) -> Self +pub fn vortex_array::scalar::Scalar::from(u8) -> Self impl core::convert::From for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: usize) -> Self +pub fn vortex_array::scalar::Scalar::from(usize) -> Self impl core::convert::From> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(ds: vortex_array::scalar::DecimalScalar<'_>) -> Self +pub fn vortex_array::scalar::Scalar::from(vortex_array::scalar::DecimalScalar<'_>) -> Self impl core::convert::From for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: vortex_array::scalar::DecimalValue) -> Self +pub fn vortex_array::scalar::Scalar::from(vortex_array::scalar::DecimalValue) -> Self impl core::convert::From> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(ps: vortex_array::scalar::PrimitiveScalar<'_>) -> Self +pub fn vortex_array::scalar::Scalar::from(vortex_array::scalar::PrimitiveScalar<'_>) -> Self impl core::convert::From> for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: vortex_buffer::ByteBuffer) -> Self +pub fn vortex_array::scalar::Scalar::from(vortex_buffer::ByteBuffer) -> Self impl core::convert::From for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::from(value: vortex_buffer::string::BufferString) -> Self +pub fn vortex_array::scalar::Scalar::from(vortex_buffer::string::BufferString) -> Self impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for alloc::sync::Arc pub type alloc::sync::Arc::Error = vortex_error::VortexError -pub fn alloc::sync::Arc::try_from(value: &vortex_array::scalar::Scalar) -> core::result::Result, Self::Error> +pub fn alloc::sync::Arc::try_from(&vortex_array::scalar::Scalar) -> core::result::Result, Self::Error> impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for bool pub type bool::Error = vortex_error::VortexError -pub fn bool::try_from(value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn bool::try_from(&vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn core::option::Option::try_from(&vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn core::option::Option::try_from(&vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn core::option::Option::try_from(&vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn core::option::Option::try_from(&vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn core::option::Option::try_from(&vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn core::option::Option::try_from(&vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn core::option::Option::try_from(&vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn core::option::Option::try_from(&vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn core::option::Option::try_from(&vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn core::option::Option::try_from(&vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn core::option::Option::try_from(&vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn core::option::Option::try_from(&vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(value: &vortex_array::scalar::Scalar) -> core::result::Result +pub fn core::option::Option::try_from(&vortex_array::scalar::Scalar) -> core::result::Result impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(scalar: &vortex_array::scalar::Scalar) -> core::result::Result +pub fn core::option::Option::try_from(&vortex_array::scalar::Scalar) -> core::result::Result impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for f32 pub type f32::Error = vortex_error::VortexError -pub fn f32::try_from(value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn f32::try_from(&vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for f64 pub type f64::Error = vortex_error::VortexError -pub fn f64::try_from(value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn f64::try_from(&vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for half::binary16::f16 pub type half::binary16::f16::Error = vortex_error::VortexError -pub fn half::binary16::f16::try_from(value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn half::binary16::f16::try_from(&vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for i16 pub type i16::Error = vortex_error::VortexError -pub fn i16::try_from(value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn i16::try_from(&vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for i32 pub type i32::Error = vortex_error::VortexError -pub fn i32::try_from(value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn i32::try_from(&vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for i64 pub type i64::Error = vortex_error::VortexError -pub fn i64::try_from(value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn i64::try_from(&vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for i8 pub type i8::Error = vortex_error::VortexError -pub fn i8::try_from(value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn i8::try_from(&vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for u16 pub type u16::Error = vortex_error::VortexError -pub fn u16::try_from(value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn u16::try_from(&vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for u32 pub type u32::Error = vortex_error::VortexError -pub fn u32::try_from(value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn u32::try_from(&vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for u64 pub type u64::Error = vortex_error::VortexError -pub fn u64::try_from(value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn u64::try_from(&vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for u8 pub type u8::Error = vortex_error::VortexError -pub fn u8::try_from(value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn u8::try_from(&vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for usize pub type usize::Error = vortex_error::VortexError -pub fn usize::try_from(value: &vortex_array::scalar::Scalar) -> core::result::Result +pub fn usize::try_from(&vortex_array::scalar::Scalar) -> core::result::Result impl core::convert::TryFrom<&vortex_array::scalar::Scalar> for vortex_array::scalar::DecimalValue pub type vortex_array::scalar::DecimalValue::Error = vortex_error::VortexError -pub fn vortex_array::scalar::DecimalValue::try_from(scalar: &vortex_array::scalar::Scalar) -> core::result::Result +pub fn vortex_array::scalar::DecimalValue::try_from(&vortex_array::scalar::Scalar) -> core::result::Result impl core::convert::TryFrom for alloc::string::String pub type alloc::string::String::Error = vortex_error::VortexError -pub fn alloc::string::String::try_from(value: vortex_array::scalar::Scalar) -> core::result::Result +pub fn alloc::string::String::try_from(vortex_array::scalar::Scalar) -> core::result::Result impl core::convert::TryFrom for bool pub type bool::Error = vortex_error::VortexError -pub fn bool::try_from(value: vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn bool::try_from(vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(value: vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn core::option::Option::try_from(vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(scalar: vortex_array::scalar::Scalar) -> core::result::Result +pub fn core::option::Option::try_from(vortex_array::scalar::Scalar) -> core::result::Result impl core::convert::TryFrom for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(scalar: vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn core::option::Option::try_from(vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(scalar: vortex_array::scalar::Scalar) -> core::result::Result +pub fn core::option::Option::try_from(vortex_array::scalar::Scalar) -> core::result::Result impl core::convert::TryFrom for vortex_array::scalar::DecimalValue pub type vortex_array::scalar::DecimalValue::Error = vortex_error::VortexError -pub fn vortex_array::scalar::DecimalValue::try_from(scalar: vortex_array::scalar::Scalar) -> core::result::Result +pub fn vortex_array::scalar::DecimalValue::try_from(vortex_array::scalar::Scalar) -> core::result::Result impl core::convert::TryFrom for vortex_buffer::ByteBuffer pub type vortex_buffer::ByteBuffer::Error = vortex_error::VortexError -pub fn vortex_buffer::ByteBuffer::try_from(scalar: vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn vortex_buffer::ByteBuffer::try_from(vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::TryFrom for vortex_buffer::string::BufferString pub type vortex_buffer::string::BufferString::Error = vortex_error::VortexError -pub fn vortex_buffer::string::BufferString::try_from(scalar: vortex_array::scalar::Scalar) -> core::result::Result +pub fn vortex_buffer::string::BufferString::try_from(vortex_array::scalar::Scalar) -> core::result::Result impl core::fmt::Debug for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::Scalar::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::Scalar::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::scalar::Scalar -pub fn vortex_array::scalar::Scalar::hash(&self, state: &mut H) +pub fn vortex_array::scalar::Scalar::hash(&self, &mut H) impl vortex_array::search_sorted::IndexOrd for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::index_cmp(&self, idx: usize, elem: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult> +pub fn vortex_array::ArrayRef::index_cmp(&self, usize, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult> -pub fn vortex_array::ArrayRef::index_ge(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::index_ge(&self, usize, &V) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::index_gt(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::index_gt(&self, usize, &V) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::index_le(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::index_le(&self, usize, &V) -> vortex_error::VortexResult pub fn vortex_array::ArrayRef::index_len(&self) -> usize -pub fn vortex_array::ArrayRef::index_lt(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::index_lt(&self, usize, &V) -> vortex_error::VortexResult impl<'a, T> core::convert::TryFrom<&'a vortex_array::scalar::Scalar> for alloc::vec::Vec where T: for<'b> core::convert::TryFrom<&'b vortex_array::scalar::Scalar, Error = vortex_error::VortexError> pub type alloc::vec::Vec::Error = vortex_error::VortexError -pub fn alloc::vec::Vec::try_from(value: &'a vortex_array::scalar::Scalar) -> core::result::Result +pub fn alloc::vec::Vec::try_from(&'a vortex_array::scalar::Scalar) -> core::result::Result impl<'a> core::convert::TryFrom<&'a vortex_array::scalar::Scalar> for alloc::string::String pub type alloc::string::String::Error = vortex_error::VortexError -pub fn alloc::string::String::try_from(value: &'a vortex_array::scalar::Scalar) -> core::result::Result +pub fn alloc::string::String::try_from(&'a vortex_array::scalar::Scalar) -> core::result::Result impl<'a> core::convert::TryFrom<&'a vortex_array::scalar::Scalar> for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(scalar: &'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn core::option::Option::try_from(&'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl<'a> core::convert::TryFrom<&'a vortex_array::scalar::Scalar> for core::option::Option pub type core::option::Option::Error = vortex_error::VortexError -pub fn core::option::Option::try_from(scalar: &'a vortex_array::scalar::Scalar) -> core::result::Result +pub fn core::option::Option::try_from(&'a vortex_array::scalar::Scalar) -> core::result::Result impl<'a> core::convert::TryFrom<&'a vortex_array::scalar::Scalar> for vortex_array::scalar::BinaryScalar<'a> pub type vortex_array::scalar::BinaryScalar<'a>::Error = vortex_error::VortexError -pub fn vortex_array::scalar::BinaryScalar<'a>::try_from(value: &'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn vortex_array::scalar::BinaryScalar<'a>::try_from(&'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl<'a> core::convert::TryFrom<&'a vortex_array::scalar::Scalar> for vortex_array::scalar::BoolScalar<'a> pub type vortex_array::scalar::BoolScalar<'a>::Error = vortex_error::VortexError -pub fn vortex_array::scalar::BoolScalar<'a>::try_from(value: &'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn vortex_array::scalar::BoolScalar<'a>::try_from(&'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl<'a> core::convert::TryFrom<&'a vortex_array::scalar::Scalar> for vortex_array::scalar::DecimalScalar<'a> pub type vortex_array::scalar::DecimalScalar<'a>::Error = vortex_error::VortexError -pub fn vortex_array::scalar::DecimalScalar<'a>::try_from(value: &'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn vortex_array::scalar::DecimalScalar<'a>::try_from(&'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl<'a> core::convert::TryFrom<&'a vortex_array::scalar::Scalar> for vortex_array::scalar::ExtScalar<'a> pub type vortex_array::scalar::ExtScalar<'a>::Error = vortex_error::VortexError -pub fn vortex_array::scalar::ExtScalar<'a>::try_from(value: &'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn vortex_array::scalar::ExtScalar<'a>::try_from(&'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl<'a> core::convert::TryFrom<&'a vortex_array::scalar::Scalar> for vortex_array::scalar::ListScalar<'a> pub type vortex_array::scalar::ListScalar<'a>::Error = vortex_error::VortexError -pub fn vortex_array::scalar::ListScalar<'a>::try_from(value: &'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn vortex_array::scalar::ListScalar<'a>::try_from(&'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl<'a> core::convert::TryFrom<&'a vortex_array::scalar::Scalar> for vortex_array::scalar::PrimitiveScalar<'a> pub type vortex_array::scalar::PrimitiveScalar<'a>::Error = vortex_error::VortexError -pub fn vortex_array::scalar::PrimitiveScalar<'a>::try_from(value: &'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn vortex_array::scalar::PrimitiveScalar<'a>::try_from(&'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl<'a> core::convert::TryFrom<&'a vortex_array::scalar::Scalar> for vortex_array::scalar::StructScalar<'a> pub type vortex_array::scalar::StructScalar<'a>::Error = vortex_error::VortexError -pub fn vortex_array::scalar::StructScalar<'a>::try_from(value: &'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn vortex_array::scalar::StructScalar<'a>::try_from(&'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl<'a> core::convert::TryFrom<&'a vortex_array::scalar::Scalar> for vortex_array::scalar::Utf8Scalar<'a> pub type vortex_array::scalar::Utf8Scalar<'a>::Error = vortex_error::VortexError -pub fn vortex_array::scalar::Utf8Scalar<'a>::try_from(value: &'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn vortex_array::scalar::Utf8Scalar<'a>::try_from(&'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl<'a> core::convert::TryFrom<&'a vortex_array::scalar::Scalar> for vortex_buffer::ByteBuffer pub type vortex_buffer::ByteBuffer::Error = vortex_error::VortexError -pub fn vortex_buffer::ByteBuffer::try_from(scalar: &'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn vortex_buffer::ByteBuffer::try_from(&'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl<'a> core::convert::TryFrom<&'a vortex_array::scalar::Scalar> for vortex_buffer::string::BufferString pub type vortex_buffer::string::BufferString::Error = vortex_error::VortexError -pub fn vortex_buffer::string::BufferString::try_from(scalar: &'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn vortex_buffer::string::BufferString::try_from(&'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl core::convert::From> for vortex_array::scalar::Scalar where T: vortex_array::dtype::NativeDType, vortex_array::scalar::Scalar: core::convert::From -pub fn vortex_array::scalar::Scalar::from(vec: alloc::vec::Vec) -> Self +pub fn vortex_array::scalar::Scalar::from(alloc::vec::Vec) -> Self impl core::convert::From>> for vortex_array::scalar::Scalar where T: vortex_array::dtype::NativeDType, vortex_array::scalar::Scalar: core::convert::From -pub fn vortex_array::scalar::Scalar::from(vec: core::option::Option>) -> Self +pub fn vortex_array::scalar::Scalar::from(core::option::Option>) -> Self impl core::convert::TryFrom for alloc::vec::Vec where T: for<'b> core::convert::TryFrom<&'b vortex_array::scalar::Scalar, Error = vortex_error::VortexError> pub type alloc::vec::Vec::Error = vortex_error::VortexError -pub fn alloc::vec::Vec::try_from(value: vortex_array::scalar::Scalar) -> core::result::Result +pub fn alloc::vec::Vec::try_from(vortex_array::scalar::Scalar) -> core::result::Result pub struct vortex_array::scalar::StructScalar<'a> impl<'a> vortex_array::scalar::StructScalar<'a> -pub fn vortex_array::scalar::StructScalar<'a>::cast(&self, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::scalar::StructScalar<'a>::cast(&self, &vortex_array::dtype::DType) -> vortex_error::VortexResult pub fn vortex_array::scalar::StructScalar<'a>::dtype(&self) -> &'a vortex_array::dtype::DType -pub fn vortex_array::scalar::StructScalar<'a>::field(&self, name: impl core::convert::AsRef) -> core::option::Option +pub fn vortex_array::scalar::StructScalar<'a>::field(&self, impl core::convert::AsRef) -> core::option::Option -pub fn vortex_array::scalar::StructScalar<'a>::field_by_idx(&self, idx: usize) -> core::option::Option +pub fn vortex_array::scalar::StructScalar<'a>::field_by_idx(&self, usize) -> core::option::Option pub fn vortex_array::scalar::StructScalar<'a>::fields_iter(&self) -> core::option::Option> @@ -15428,7 +15630,7 @@ pub fn vortex_array::scalar::StructScalar<'a>::is_null(&self) -> bool pub fn vortex_array::scalar::StructScalar<'a>::names(&self) -> &vortex_array::dtype::FieldNames -pub fn vortex_array::scalar::StructScalar<'a>::project(&self, projection: &[vortex_array::dtype::FieldName]) -> vortex_error::VortexResult +pub fn vortex_array::scalar::StructScalar<'a>::project(&self, &[vortex_array::dtype::FieldName]) -> vortex_error::VortexResult pub fn vortex_array::scalar::StructScalar<'a>::struct_fields(&self) -> &vortex_array::dtype::StructFields @@ -15436,19 +15638,19 @@ impl core::cmp::Eq for vortex_array::scalar::StructScalar<'_> impl core::cmp::PartialEq for vortex_array::scalar::StructScalar<'_> -pub fn vortex_array::scalar::StructScalar<'_>::eq(&self, other: &Self) -> bool +pub fn vortex_array::scalar::StructScalar<'_>::eq(&self, &Self) -> bool impl core::cmp::PartialOrd for vortex_array::scalar::StructScalar<'_> -pub fn vortex_array::scalar::StructScalar<'_>::partial_cmp(&self, other: &Self) -> core::option::Option +pub fn vortex_array::scalar::StructScalar<'_>::partial_cmp(&self, &Self) -> core::option::Option impl core::fmt::Display for vortex_array::scalar::StructScalar<'_> -pub fn vortex_array::scalar::StructScalar<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::StructScalar<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::scalar::StructScalar<'_> -pub fn vortex_array::scalar::StructScalar<'_>::hash(&self, state: &mut H) +pub fn vortex_array::scalar::StructScalar<'_>::hash(&self, &mut H) impl<'a> core::clone::Clone for vortex_array::scalar::StructScalar<'a> @@ -15458,11 +15660,13 @@ impl<'a> core::convert::TryFrom<&'a vortex_array::scalar::Scalar> for vortex_arr pub type vortex_array::scalar::StructScalar<'a>::Error = vortex_error::VortexError -pub fn vortex_array::scalar::StructScalar<'a>::try_from(value: &'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn vortex_array::scalar::StructScalar<'a>::try_from(&'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl<'a> core::fmt::Debug for vortex_array::scalar::StructScalar<'a> -pub fn vortex_array::scalar::StructScalar<'a>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::StructScalar<'a>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +impl<'a> core::marker::Copy for vortex_array::scalar::StructScalar<'a> pub struct vortex_array::scalar::Utf8Scalar<'a> @@ -15474,25 +15678,25 @@ pub fn vortex_array::scalar::Utf8Scalar<'a>::is_empty(&self) -> core::option::Op pub fn vortex_array::scalar::Utf8Scalar<'a>::len(&self) -> core::option::Option -pub fn vortex_array::scalar::Utf8Scalar<'a>::try_new(dtype: &'a vortex_array::dtype::DType, value: core::option::Option<&'a vortex_array::scalar::ScalarValue>) -> vortex_error::VortexResult +pub fn vortex_array::scalar::Utf8Scalar<'a>::try_new(&'a vortex_array::dtype::DType, core::option::Option<&'a vortex_array::scalar::ScalarValue>) -> vortex_error::VortexResult pub fn vortex_array::scalar::Utf8Scalar<'a>::value(&self) -> core::option::Option<&'a vortex_buffer::string::BufferString> impl core::cmp::Ord for vortex_array::scalar::Utf8Scalar<'_> -pub fn vortex_array::scalar::Utf8Scalar<'_>::cmp(&self, other: &Self) -> core::cmp::Ordering +pub fn vortex_array::scalar::Utf8Scalar<'_>::cmp(&self, &Self) -> core::cmp::Ordering impl core::cmp::PartialEq for vortex_array::scalar::Utf8Scalar<'_> -pub fn vortex_array::scalar::Utf8Scalar<'_>::eq(&self, other: &Self) -> bool +pub fn vortex_array::scalar::Utf8Scalar<'_>::eq(&self, &Self) -> bool impl core::cmp::PartialOrd for vortex_array::scalar::Utf8Scalar<'_> -pub fn vortex_array::scalar::Utf8Scalar<'_>::partial_cmp(&self, other: &Self) -> core::option::Option +pub fn vortex_array::scalar::Utf8Scalar<'_>::partial_cmp(&self, &Self) -> core::option::Option impl core::fmt::Display for vortex_array::scalar::Utf8Scalar<'_> -pub fn vortex_array::scalar::Utf8Scalar<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::Utf8Scalar<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl<'a> core::clone::Clone for vortex_array::scalar::Utf8Scalar<'a> @@ -15504,15 +15708,17 @@ impl<'a> core::convert::TryFrom<&'a vortex_array::scalar::Scalar> for vortex_arr pub type vortex_array::scalar::Utf8Scalar<'a>::Error = vortex_error::VortexError -pub fn vortex_array::scalar::Utf8Scalar<'a>::try_from(value: &'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult +pub fn vortex_array::scalar::Utf8Scalar<'a>::try_from(&'a vortex_array::scalar::Scalar) -> vortex_error::VortexResult impl<'a> core::fmt::Debug for vortex_array::scalar::Utf8Scalar<'a> -pub fn vortex_array::scalar::Utf8Scalar<'a>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::Utf8Scalar<'a>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl<'a> core::hash::Hash for vortex_array::scalar::Utf8Scalar<'a> -pub fn vortex_array::scalar::Utf8Scalar<'a>::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::scalar::Utf8Scalar<'a>::hash<__H: core::hash::Hasher>(&self, &mut __H) + +impl<'a> core::marker::Copy for vortex_array::scalar::Utf8Scalar<'a> pub struct vortex_array::scalar::VariantScalar<'a> @@ -15530,7 +15736,7 @@ pub fn vortex_array::scalar::VariantScalar<'a>::value(&self) -> core::option::Op impl core::fmt::Display for vortex_array::scalar::VariantScalar<'_> -pub fn vortex_array::scalar::VariantScalar<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::VariantScalar<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl<'a> core::clone::Clone for vortex_array::scalar::VariantScalar<'a> @@ -15538,45 +15744,45 @@ pub fn vortex_array::scalar::VariantScalar<'a>::clone(&self) -> vortex_array::sc impl<'a> core::fmt::Debug for vortex_array::scalar::VariantScalar<'a> -pub fn vortex_array::scalar::VariantScalar<'a>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar::VariantScalar<'a>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl<'a> core::marker::Copy for vortex_array::scalar::VariantScalar<'a> pub trait vortex_array::scalar::ScalarTruncation: core::marker::Send + core::marker::Sized -pub fn vortex_array::scalar::ScalarTruncation::from_scalar(value: vortex_array::scalar::Scalar) -> vortex_error::VortexResult> +pub fn vortex_array::scalar::ScalarTruncation::from_scalar(vortex_array::scalar::Scalar) -> vortex_error::VortexResult> -pub fn vortex_array::scalar::ScalarTruncation::into_scalar(self, nullability: vortex_array::dtype::Nullability) -> vortex_array::scalar::Scalar +pub fn vortex_array::scalar::ScalarTruncation::into_scalar(self, vortex_array::dtype::Nullability) -> vortex_array::scalar::Scalar pub fn vortex_array::scalar::ScalarTruncation::len(&self) -> usize -pub fn vortex_array::scalar::ScalarTruncation::lower_bound(self, max_length: usize) -> Self +pub fn vortex_array::scalar::ScalarTruncation::lower_bound(self, usize) -> Self -pub fn vortex_array::scalar::ScalarTruncation::upper_bound(self, max_length: usize) -> core::option::Option +pub fn vortex_array::scalar::ScalarTruncation::upper_bound(self, usize) -> core::option::Option impl vortex_array::scalar::ScalarTruncation for vortex_buffer::ByteBuffer -pub fn vortex_buffer::ByteBuffer::from_scalar(value: vortex_array::scalar::Scalar) -> vortex_error::VortexResult> +pub fn vortex_buffer::ByteBuffer::from_scalar(vortex_array::scalar::Scalar) -> vortex_error::VortexResult> -pub fn vortex_buffer::ByteBuffer::into_scalar(self, nullability: vortex_array::dtype::Nullability) -> vortex_array::scalar::Scalar +pub fn vortex_buffer::ByteBuffer::into_scalar(self, vortex_array::dtype::Nullability) -> vortex_array::scalar::Scalar pub fn vortex_buffer::ByteBuffer::len(&self) -> usize -pub fn vortex_buffer::ByteBuffer::lower_bound(self, max_length: usize) -> Self +pub fn vortex_buffer::ByteBuffer::lower_bound(self, usize) -> Self -pub fn vortex_buffer::ByteBuffer::upper_bound(self, max_length: usize) -> core::option::Option +pub fn vortex_buffer::ByteBuffer::upper_bound(self, usize) -> core::option::Option impl vortex_array::scalar::ScalarTruncation for vortex_buffer::string::BufferString -pub fn vortex_buffer::string::BufferString::from_scalar(value: vortex_array::scalar::Scalar) -> vortex_error::VortexResult> +pub fn vortex_buffer::string::BufferString::from_scalar(vortex_array::scalar::Scalar) -> vortex_error::VortexResult> -pub fn vortex_buffer::string::BufferString::into_scalar(self, nullability: vortex_array::dtype::Nullability) -> vortex_array::scalar::Scalar +pub fn vortex_buffer::string::BufferString::into_scalar(self, vortex_array::dtype::Nullability) -> vortex_array::scalar::Scalar pub fn vortex_buffer::string::BufferString::len(&self) -> usize -pub fn vortex_buffer::string::BufferString::lower_bound(self, max_length: usize) -> Self +pub fn vortex_buffer::string::BufferString::lower_bound(self, usize) -> Self -pub fn vortex_buffer::string::BufferString::upper_bound(self, max_length: usize) -> core::option::Option +pub fn vortex_buffer::string::BufferString::upper_bound(self, usize) -> core::option::Option pub trait vortex_array::scalar::StringLike: vortex_array::scalar::typed_view::utf8::private::Sealed + core::marker::Sized @@ -15590,9 +15796,9 @@ impl vortex_array::scalar::StringLike for vortex_buffer::string::BufferString pub fn vortex_buffer::string::BufferString::increment(self) -> core::result::Result -pub fn vortex_array::scalar::lower_bound(value: core::option::Option, max_length: usize, nullability: vortex_array::dtype::Nullability) -> core::option::Option<(vortex_array::scalar::Scalar, bool)> +pub fn vortex_array::scalar::lower_bound(core::option::Option, usize, vortex_array::dtype::Nullability) -> core::option::Option<(vortex_array::scalar::Scalar, bool)> -pub fn vortex_array::scalar::upper_bound(value: core::option::Option, max_length: usize, nullability: vortex_array::dtype::Nullability) -> core::option::Option<(vortex_array::scalar::Scalar, bool)> +pub fn vortex_array::scalar::upper_bound(core::option::Option, usize, vortex_array::dtype::Nullability) -> core::option::Option<(vortex_array::scalar::Scalar, bool)> pub mod vortex_array::scalar_fn @@ -15622,15 +15828,15 @@ impl core::cmp::Eq for vortex_array::scalar_fn::fns::between::StrictComparison impl core::cmp::PartialEq for vortex_array::scalar_fn::fns::between::StrictComparison -pub fn vortex_array::scalar_fn::fns::between::StrictComparison::eq(&self, other: &vortex_array::scalar_fn::fns::between::StrictComparison) -> bool +pub fn vortex_array::scalar_fn::fns::between::StrictComparison::eq(&self, &vortex_array::scalar_fn::fns::between::StrictComparison) -> bool impl core::fmt::Debug for vortex_array::scalar_fn::fns::between::StrictComparison -pub fn vortex_array::scalar_fn::fns::between::StrictComparison::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::between::StrictComparison::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::scalar_fn::fns::between::StrictComparison -pub fn vortex_array::scalar_fn::fns::between::StrictComparison::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::scalar_fn::fns::between::StrictComparison::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_array::scalar_fn::fns::between::StrictComparison @@ -15646,39 +15852,39 @@ impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::b pub type vortex_array::scalar_fn::fns::between::Between::Options = vortex_array::scalar_fn::fns::between::BetweenOptions -pub fn vortex_array::scalar_fn::fns::between::Between::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::between::Between::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::between::Between::child_name(&self, _instance: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::between::Between::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::between::Between::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::between::Between::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::between::Between::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::between::Between::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::between::Between::execute(&self, options: &Self::Options, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::between::Between::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::between::Between::fmt_sql(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::between::Between::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::between::Between::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::between::Between::is_fallible(&self, _options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::between::Between::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::between::Between::is_null_sensitive(&self, _instance: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::between::Between::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::between::Between::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::between::Between::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::between::Between::return_dtype(&self, _options: &Self::Options, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::between::Between::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::between::Between::serialize(&self, instance: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::between::Between::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::between::Between::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::between::Between::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::between::Between::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::between::Between::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::between::Between::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::between::Between::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::between::Between::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::between::Between::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::between::Between::validity(&self, options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::between::Between::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> pub struct vortex_array::scalar_fn::fns::between::BetweenExecuteAdaptor(pub V) @@ -15688,13 +15894,13 @@ pub fn vortex_array::scalar_fn::fns::between::BetweenExecuteAdaptor::default( impl core::fmt::Debug for vortex_array::scalar_fn::fns::between::BetweenExecuteAdaptor -pub fn vortex_array::scalar_fn::fns::between::BetweenExecuteAdaptor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::between::BetweenExecuteAdaptor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::kernel::ExecuteParentKernel for vortex_array::scalar_fn::fns::between::BetweenExecuteAdaptor where V: vortex_array::scalar_fn::fns::between::BetweenKernel pub type vortex_array::scalar_fn::fns::between::BetweenExecuteAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::between::BetweenExecuteAdaptor::execute_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::between::Between>, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::between::BetweenExecuteAdaptor::execute_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::between::Between>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub struct vortex_array::scalar_fn::fns::between::BetweenOptions @@ -15710,19 +15916,19 @@ impl core::cmp::Eq for vortex_array::scalar_fn::fns::between::BetweenOptions impl core::cmp::PartialEq for vortex_array::scalar_fn::fns::between::BetweenOptions -pub fn vortex_array::scalar_fn::fns::between::BetweenOptions::eq(&self, other: &vortex_array::scalar_fn::fns::between::BetweenOptions) -> bool +pub fn vortex_array::scalar_fn::fns::between::BetweenOptions::eq(&self, &vortex_array::scalar_fn::fns::between::BetweenOptions) -> bool impl core::fmt::Debug for vortex_array::scalar_fn::fns::between::BetweenOptions -pub fn vortex_array::scalar_fn::fns::between::BetweenOptions::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::between::BetweenOptions::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::scalar_fn::fns::between::BetweenOptions -pub fn vortex_array::scalar_fn::fns::between::BetweenOptions::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::between::BetweenOptions::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::scalar_fn::fns::between::BetweenOptions -pub fn vortex_array::scalar_fn::fns::between::BetweenOptions::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::scalar_fn::fns::between::BetweenOptions::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_array::scalar_fn::fns::between::BetweenOptions @@ -15734,33 +15940,33 @@ pub fn vortex_array::scalar_fn::fns::between::BetweenReduceAdaptor::default() impl core::fmt::Debug for vortex_array::scalar_fn::fns::between::BetweenReduceAdaptor -pub fn vortex_array::scalar_fn::fns::between::BetweenReduceAdaptor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::between::BetweenReduceAdaptor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::scalar_fn::fns::between::BetweenReduceAdaptor where V: vortex_array::scalar_fn::fns::between::BetweenReduce pub type vortex_array::scalar_fn::fns::between::BetweenReduceAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::between::BetweenReduceAdaptor::reduce_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::between::Between>, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::between::BetweenReduceAdaptor::reduce_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::between::Between>, usize) -> vortex_error::VortexResult> pub trait vortex_array::scalar_fn::fns::between::BetweenKernel: vortex_array::VTable -pub fn vortex_array::scalar_fn::fns::between::BetweenKernel::between(array: vortex_array::ArrayView<'_, Self>, lower: &vortex_array::ArrayRef, upper: &vortex_array::ArrayRef, options: &vortex_array::scalar_fn::fns::between::BetweenOptions, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::between::BetweenKernel::between(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, &vortex_array::ArrayRef, &vortex_array::scalar_fn::fns::between::BetweenOptions, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::between::BetweenKernel for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::between(arr: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, lower: &vortex_array::ArrayRef, upper: &vortex_array::ArrayRef, options: &vortex_array::scalar_fn::fns::between::BetweenOptions, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::between(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, &vortex_array::ArrayRef, &vortex_array::ArrayRef, &vortex_array::scalar_fn::fns::between::BetweenOptions, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::between::BetweenKernel for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::between(arr: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, lower: &vortex_array::ArrayRef, upper: &vortex_array::ArrayRef, options: &vortex_array::scalar_fn::fns::between::BetweenOptions, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::between(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, &vortex_array::ArrayRef, &vortex_array::ArrayRef, &vortex_array::scalar_fn::fns::between::BetweenOptions, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub trait vortex_array::scalar_fn::fns::between::BetweenReduce: vortex_array::VTable -pub fn vortex_array::scalar_fn::fns::between::BetweenReduce::between(array: vortex_array::ArrayView<'_, Self>, lower: &vortex_array::ArrayRef, upper: &vortex_array::ArrayRef, options: &vortex_array::scalar_fn::fns::between::BetweenOptions) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::between::BetweenReduce::between(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, &vortex_array::ArrayRef, &vortex_array::scalar_fn::fns::between::BetweenOptions) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::between::BetweenReduce for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::between(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, lower: &vortex_array::ArrayRef, upper: &vortex_array::ArrayRef, options: &vortex_array::scalar_fn::fns::between::BetweenOptions) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::between(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, &vortex_array::ArrayRef, &vortex_array::ArrayRef, &vortex_array::scalar_fn::fns::between::BetweenOptions) -> vortex_error::VortexResult> pub mod vortex_array::scalar_fn::fns::binary @@ -15774,39 +15980,39 @@ impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::b pub type vortex_array::scalar_fn::fns::binary::Binary::Options = vortex_array::scalar_fn::fns::operators::Operator -pub fn vortex_array::scalar_fn::fns::binary::Binary::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::binary::Binary::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::binary::Binary::child_name(&self, _instance: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::binary::Binary::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::binary::Binary::coerce_args(&self, operator: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::binary::Binary::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::binary::Binary::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::binary::Binary::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::binary::Binary::execute(&self, op: &vortex_array::scalar_fn::fns::operators::Operator, args: &dyn vortex_array::scalar_fn::ExecutionArgs, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::binary::Binary::execute(&self, &vortex_array::scalar_fn::fns::operators::Operator, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::binary::Binary::fmt_sql(&self, operator: &vortex_array::scalar_fn::fns::operators::Operator, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::binary::Binary::fmt_sql(&self, &vortex_array::scalar_fn::fns::operators::Operator, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::binary::Binary::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::binary::Binary::is_fallible(&self, operator: &vortex_array::scalar_fn::fns::operators::Operator) -> bool +pub fn vortex_array::scalar_fn::fns::binary::Binary::is_fallible(&self, &vortex_array::scalar_fn::fns::operators::Operator) -> bool -pub fn vortex_array::scalar_fn::fns::binary::Binary::is_null_sensitive(&self, _operator: &vortex_array::scalar_fn::fns::operators::Operator) -> bool +pub fn vortex_array::scalar_fn::fns::binary::Binary::is_null_sensitive(&self, &vortex_array::scalar_fn::fns::operators::Operator) -> bool -pub fn vortex_array::scalar_fn::fns::binary::Binary::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::binary::Binary::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::binary::Binary::return_dtype(&self, operator: &vortex_array::scalar_fn::fns::operators::Operator, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::binary::Binary::return_dtype(&self, &vortex_array::scalar_fn::fns::operators::Operator, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::binary::Binary::serialize(&self, instance: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::binary::Binary::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::binary::Binary::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::binary::Binary::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::binary::Binary::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::binary::Binary::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::binary::Binary::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::binary::Binary::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::binary::Binary::stat_falsification(&self, operator: &vortex_array::scalar_fn::fns::operators::Operator, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::binary::Binary::stat_falsification(&self, &vortex_array::scalar_fn::fns::operators::Operator, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::binary::Binary::validity(&self, operator: &vortex_array::scalar_fn::fns::operators::Operator, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::binary::Binary::validity(&self, &vortex_array::scalar_fn::fns::operators::Operator, &vortex_array::expr::Expression) -> vortex_error::VortexResult> pub struct vortex_array::scalar_fn::fns::binary::CompareExecuteAdaptor(pub V) @@ -15816,41 +16022,41 @@ pub fn vortex_array::scalar_fn::fns::binary::CompareExecuteAdaptor::default() impl core::fmt::Debug for vortex_array::scalar_fn::fns::binary::CompareExecuteAdaptor -pub fn vortex_array::scalar_fn::fns::binary::CompareExecuteAdaptor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::binary::CompareExecuteAdaptor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::kernel::ExecuteParentKernel for vortex_array::scalar_fn::fns::binary::CompareExecuteAdaptor where V: vortex_array::scalar_fn::fns::binary::CompareKernel pub type vortex_array::scalar_fn::fns::binary::CompareExecuteAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::binary::CompareExecuteAdaptor::execute_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::binary::Binary>, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::binary::CompareExecuteAdaptor::execute_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::binary::Binary>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub trait vortex_array::scalar_fn::fns::binary::CompareKernel: vortex_array::VTable -pub fn vortex_array::scalar_fn::fns::binary::CompareKernel::compare(lhs: vortex_array::ArrayView<'_, Self>, rhs: &vortex_array::ArrayRef, operator: vortex_array::scalar_fn::fns::operators::CompareOperator, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::binary::CompareKernel::compare(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, vortex_array::scalar_fn::fns::operators::CompareOperator, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::binary::CompareKernel for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::compare(lhs: vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, rhs: &vortex_array::ArrayRef, operator: vortex_array::scalar_fn::fns::operators::CompareOperator, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::compare(vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, &vortex_array::ArrayRef, vortex_array::scalar_fn::fns::operators::CompareOperator, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::binary::CompareKernel for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::compare(lhs: vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, rhs: &vortex_array::ArrayRef, operator: vortex_array::scalar_fn::fns::operators::CompareOperator, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::compare(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, &vortex_array::ArrayRef, vortex_array::scalar_fn::fns::operators::CompareOperator, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::binary::CompareKernel for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::compare(lhs: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, rhs: &vortex_array::ArrayRef, operator: vortex_array::scalar_fn::fns::operators::CompareOperator, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::compare(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_array::ArrayRef, vortex_array::scalar_fn::fns::operators::CompareOperator, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::binary::CompareKernel for vortex_array::arrays::patched::Patched -pub fn vortex_array::arrays::patched::Patched::compare(lhs: vortex_array::ArrayView<'_, Self>, rhs: &vortex_array::ArrayRef, operator: vortex_array::scalar_fn::fns::operators::CompareOperator, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::compare(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, vortex_array::scalar_fn::fns::operators::CompareOperator, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::binary::and_kleene(lhs: &vortex_array::ArrayRef, rhs: &vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::binary::and_kleene(&vortex_array::ArrayRef, &vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::binary::compare_nested_arrow_arrays(lhs: &dyn arrow_array::array::Array, rhs: &dyn arrow_array::array::Array, operator: vortex_array::scalar_fn::fns::operators::CompareOperator) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::binary::compare_nested_arrow_arrays(&dyn arrow_array::array::Array, &dyn arrow_array::array::Array, vortex_array::scalar_fn::fns::operators::CompareOperator) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::binary::or_kleene(lhs: &vortex_array::ArrayRef, rhs: &vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::binary::or_kleene(&vortex_array::ArrayRef, &vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::binary::scalar_cmp(lhs: &vortex_array::scalar::Scalar, rhs: &vortex_array::scalar::Scalar, operator: vortex_array::scalar_fn::fns::operators::CompareOperator) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::binary::scalar_cmp(&vortex_array::scalar::Scalar, &vortex_array::scalar::Scalar, vortex_array::scalar_fn::fns::operators::CompareOperator) -> vortex_error::VortexResult pub mod vortex_array::scalar_fn::fns::case_when @@ -15864,39 +16070,39 @@ impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::c pub type vortex_array::scalar_fn::fns::case_when::CaseWhen::Options = vortex_array::scalar_fn::fns::case_when::CaseWhenOptions -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::arity(&self, options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::child_name(&self, options: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::deserialize(&self, metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::execute(&self, options: &Self::Options, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::fmt_sql(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::is_fallible(&self, _options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::is_null_sensitive(&self, _options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::return_dtype(&self, options: &Self::Options, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::validity(&self, options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> pub struct vortex_array::scalar_fn::fns::case_when::CaseWhenOptions @@ -15916,19 +16122,19 @@ impl core::cmp::Eq for vortex_array::scalar_fn::fns::case_when::CaseWhenOptions impl core::cmp::PartialEq for vortex_array::scalar_fn::fns::case_when::CaseWhenOptions -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhenOptions::eq(&self, other: &vortex_array::scalar_fn::fns::case_when::CaseWhenOptions) -> bool +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhenOptions::eq(&self, &vortex_array::scalar_fn::fns::case_when::CaseWhenOptions) -> bool impl core::fmt::Debug for vortex_array::scalar_fn::fns::case_when::CaseWhenOptions -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhenOptions::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhenOptions::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::scalar_fn::fns::case_when::CaseWhenOptions -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhenOptions::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhenOptions::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::scalar_fn::fns::case_when::CaseWhenOptions -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhenOptions::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhenOptions::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_array::scalar_fn::fns::case_when::CaseWhenOptions @@ -15946,39 +16152,39 @@ impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::c pub type vortex_array::scalar_fn::fns::cast::Cast::Options = vortex_array::dtype::DType -pub fn vortex_array::scalar_fn::fns::cast::Cast::arity(&self, _options: &vortex_array::dtype::DType) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::cast::Cast::arity(&self, &vortex_array::dtype::DType) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::cast::Cast::child_name(&self, _instance: &vortex_array::dtype::DType, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::cast::Cast::child_name(&self, &vortex_array::dtype::DType, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::cast::Cast::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::cast::Cast::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::cast::Cast::deserialize(&self, _metadata: &[u8], session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::cast::Cast::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::cast::Cast::execute(&self, target_dtype: &vortex_array::dtype::DType, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::cast::Cast::execute(&self, &vortex_array::dtype::DType, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::cast::Cast::fmt_sql(&self, dtype: &vortex_array::dtype::DType, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::cast::Cast::fmt_sql(&self, &vortex_array::dtype::DType, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::cast::Cast::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::cast::Cast::is_fallible(&self, options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::cast::Cast::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::cast::Cast::is_null_sensitive(&self, _instance: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::scalar_fn::fns::cast::Cast::is_null_sensitive(&self, &vortex_array::dtype::DType) -> bool -pub fn vortex_array::scalar_fn::fns::cast::Cast::reduce(&self, target_dtype: &vortex_array::dtype::DType, node: &dyn vortex_array::scalar_fn::ReduceNode, _ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::cast::Cast::reduce(&self, &vortex_array::dtype::DType, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::cast::Cast::return_dtype(&self, dtype: &vortex_array::dtype::DType, _arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::cast::Cast::return_dtype(&self, &vortex_array::dtype::DType, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::cast::Cast::serialize(&self, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::cast::Cast::serialize(&self, &vortex_array::dtype::DType) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::cast::Cast::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::cast::Cast::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::cast::Cast::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::cast::Cast::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::cast::Cast::stat_expression(&self, dtype: &vortex_array::dtype::DType, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::cast::Cast::stat_expression(&self, &vortex_array::dtype::DType, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::cast::Cast::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::cast::Cast::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::cast::Cast::validity(&self, dtype: &vortex_array::dtype::DType, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::cast::Cast::validity(&self, &vortex_array::dtype::DType, &vortex_array::expr::Expression) -> vortex_error::VortexResult> pub struct vortex_array::scalar_fn::fns::cast::CastExecuteAdaptor(pub V) @@ -15988,13 +16194,13 @@ pub fn vortex_array::scalar_fn::fns::cast::CastExecuteAdaptor::default() -> v impl core::fmt::Debug for vortex_array::scalar_fn::fns::cast::CastExecuteAdaptor -pub fn vortex_array::scalar_fn::fns::cast::CastExecuteAdaptor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::cast::CastExecuteAdaptor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::kernel::ExecuteParentKernel for vortex_array::scalar_fn::fns::cast::CastExecuteAdaptor where V: vortex_array::scalar_fn::fns::cast::CastKernel pub type vortex_array::scalar_fn::fns::cast::CastExecuteAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::cast::CastExecuteAdaptor::execute_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: ::Match, _child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::cast::CastExecuteAdaptor::execute_parent(&self, vortex_array::ArrayView<'_, V>, ::Match, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub struct vortex_array::scalar_fn::fns::cast::CastReduceAdaptor(pub V) @@ -16004,77 +16210,113 @@ pub fn vortex_array::scalar_fn::fns::cast::CastReduceAdaptor::default() -> vo impl core::fmt::Debug for vortex_array::scalar_fn::fns::cast::CastReduceAdaptor -pub fn vortex_array::scalar_fn::fns::cast::CastReduceAdaptor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::cast::CastReduceAdaptor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::scalar_fn::fns::cast::CastReduceAdaptor where V: vortex_array::scalar_fn::fns::cast::CastReduce pub type vortex_array::scalar_fn::fns::cast::CastReduceAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::cast::CastReduceAdaptor::reduce_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::cast::Cast>, _child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::cast::CastReduceAdaptor::reduce_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::cast::Cast>, usize) -> vortex_error::VortexResult> pub trait vortex_array::scalar_fn::fns::cast::CastKernel: vortex_array::VTable -pub fn vortex_array::scalar_fn::fns::cast::CastKernel::cast(array: vortex_array::ArrayView<'_, Self>, dtype: &vortex_array::dtype::DType, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::cast::CastKernel::cast(vortex_array::ArrayView<'_, Self>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::Bool + +pub fn vortex_array::arrays::Bool::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, dtype: &vortex_array::dtype::DType, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::FixedSizeList + +pub fn vortex_array::arrays::FixedSizeList::cast(vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::List + +pub fn vortex_array::arrays::List::cast(vortex_array::ArrayView<'_, vortex_array::arrays::List>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::ListView + +pub fn vortex_array::arrays::ListView::cast(vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, dtype: &vortex_array::dtype::DType, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, dtype: &vortex_array::dtype::DType, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::VarBin + +pub fn vortex_array::arrays::VarBin::cast(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastKernel for vortex_array::arrays::VarBinView + +pub fn vortex_array::arrays::VarBinView::cast(vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, &vortex_array::dtype::DType, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub trait vortex_array::scalar_fn::fns::cast::CastReduce: vortex_array::VTable -pub fn vortex_array::scalar_fn::fns::cast::CastReduce::cast(array: vortex_array::ArrayView<'_, Self>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::cast::CastReduce::cast(vortex_array::ArrayView<'_, Self>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::Decimal + +pub fn vortex_array::arrays::Decimal::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::cast(vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::List -pub fn vortex_array::arrays::List::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::List>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::cast(vortex_array::ArrayView<'_, vortex_array::arrays::List>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::cast(vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::Primitive + +pub fn vortex_array::arrays::Primitive::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::Struct + +pub fn vortex_array::arrays::Struct::cast(vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::cast(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::cast(vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::cast(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::cast::CastReduce for vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::cast(array: vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::cast(vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, &vortex_array::dtype::DType) -> vortex_error::VortexResult> pub mod vortex_array::scalar_fn::fns::dynamic @@ -16088,39 +16330,39 @@ impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::d pub type vortex_array::scalar_fn::fns::dynamic::DynamicComparison::Options = vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::child_name(&self, _instance: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::execute(&self, data: &Self::Options, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::fmt_sql(&self, dynamic: &vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::fmt_sql(&self, &vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::is_fallible(&self, options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::is_null_sensitive(&self, _instance: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::return_dtype(&self, dynamic: &vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::return_dtype(&self, &vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::serialize(&self, options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::stat_falsification(&self, dynamic: &vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::stat_falsification(&self, &vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::validity(&self, options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> pub struct vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr @@ -16136,25 +16378,25 @@ impl core::cmp::Eq for vortex_array::scalar_fn::fns::dynamic::DynamicComparisonE impl core::cmp::PartialEq for vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr::eq(&self, other: &Self) -> bool +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr::eq(&self, &Self) -> bool impl core::fmt::Debug for vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr::hash(&self, state: &mut H) +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr::hash(&self, &mut H) pub struct vortex_array::scalar_fn::fns::dynamic::DynamicExprUpdates impl vortex_array::scalar_fn::fns::dynamic::DynamicExprUpdates -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicExprUpdates::new(expr: &vortex_array::expr::Expression) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicExprUpdates::new(&vortex_array::expr::Expression) -> core::option::Option pub fn vortex_array::scalar_fn::fns::dynamic::DynamicExprUpdates::version(&self) -> u64 @@ -16170,39 +16412,39 @@ impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::f pub type vortex_array::scalar_fn::fns::fill_null::FillNull::Options = vortex_array::scalar_fn::EmptyOptions -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::child_name(&self, _options: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::execute(&self, _options: &Self::Options, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::fmt_sql(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::is_fallible(&self, _options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::is_null_sensitive(&self, _options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::return_dtype(&self, _options: &Self::Options, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::simplify(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::validity(&self, _options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> pub struct vortex_array::scalar_fn::fns::fill_null::FillNullExecuteAdaptor(pub V) @@ -16212,13 +16454,13 @@ pub fn vortex_array::scalar_fn::fns::fill_null::FillNullExecuteAdaptor::defau impl core::fmt::Debug for vortex_array::scalar_fn::fns::fill_null::FillNullExecuteAdaptor -pub fn vortex_array::scalar_fn::fns::fill_null::FillNullExecuteAdaptor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::fill_null::FillNullExecuteAdaptor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::kernel::ExecuteParentKernel for vortex_array::scalar_fn::fns::fill_null::FillNullExecuteAdaptor where V: vortex_array::scalar_fn::fns::fill_null::FillNullKernel pub type vortex_array::scalar_fn::fns::fill_null::FillNullExecuteAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::fill_null::FillNullExecuteAdaptor::execute_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::fill_null::FillNull>, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::fill_null::FillNullExecuteAdaptor::execute_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::fill_null::FillNull>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub struct vortex_array::scalar_fn::fns::fill_null::FillNullReduceAdaptor(pub V) @@ -16228,45 +16470,45 @@ pub fn vortex_array::scalar_fn::fns::fill_null::FillNullReduceAdaptor::defaul impl core::fmt::Debug for vortex_array::scalar_fn::fns::fill_null::FillNullReduceAdaptor -pub fn vortex_array::scalar_fn::fns::fill_null::FillNullReduceAdaptor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::fill_null::FillNullReduceAdaptor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::scalar_fn::fns::fill_null::FillNullReduceAdaptor where V: vortex_array::scalar_fn::fns::fill_null::FillNullReduce pub type vortex_array::scalar_fn::fns::fill_null::FillNullReduceAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::fill_null::FillNullReduceAdaptor::reduce_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::fill_null::FillNull>, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::fill_null::FillNullReduceAdaptor::reduce_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::fill_null::FillNull>, usize) -> vortex_error::VortexResult> pub trait vortex_array::scalar_fn::fns::fill_null::FillNullKernel: vortex_array::VTable -pub fn vortex_array::scalar_fn::fns::fill_null::FillNullKernel::fill_null(array: vortex_array::ArrayView<'_, Self>, fill_value: &vortex_array::scalar::Scalar, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::fill_null::FillNullKernel::fill_null(vortex_array::ArrayView<'_, Self>, &vortex_array::scalar::Scalar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::fill_null::FillNullKernel for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::fill_null(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, fill_value: &vortex_array::scalar::Scalar, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::fill_null(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, &vortex_array::scalar::Scalar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::fill_null::FillNullKernel for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::fill_null(array: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, fill_value: &vortex_array::scalar::Scalar, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::fill_null(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, &vortex_array::scalar::Scalar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::fill_null::FillNullKernel for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::fill_null(array: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, fill_value: &vortex_array::scalar::Scalar, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::fill_null(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, &vortex_array::scalar::Scalar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::fill_null::FillNullKernel for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::fill_null(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, fill_value: &vortex_array::scalar::Scalar, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::fill_null(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_array::scalar::Scalar, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub trait vortex_array::scalar_fn::fns::fill_null::FillNullReduce: vortex_array::VTable -pub fn vortex_array::scalar_fn::fns::fill_null::FillNullReduce::fill_null(array: vortex_array::ArrayView<'_, Self>, fill_value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::fill_null::FillNullReduce::fill_null(vortex_array::ArrayView<'_, Self>, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::fill_null::FillNullReduce for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::fill_null(array: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, fill_value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::fill_null(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::fill_null::FillNullReduce for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::fill_null(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, fill_value: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::fill_null(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult> pub mod vortex_array::scalar_fn::fns::get_item @@ -16280,39 +16522,85 @@ impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::g pub type vortex_array::scalar_fn::fns::get_item::GetItem::Options = vortex_array::dtype::FieldName -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::arity(&self, _field_name: &vortex_array::dtype::FieldName) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::arity(&self, &vortex_array::dtype::FieldName) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::child_name(&self, _instance: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::execute(&self, field_name: &vortex_array::dtype::FieldName, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::execute(&self, &vortex_array::dtype::FieldName, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::fmt_sql(&self, field_name: &vortex_array::dtype::FieldName, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::fmt_sql(&self, &vortex_array::dtype::FieldName, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::get_item::GetItem::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::is_fallible(&self, _field_name: &vortex_array::dtype::FieldName) -> bool +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::is_fallible(&self, &vortex_array::dtype::FieldName) -> bool -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::is_null_sensitive(&self, _field_name: &vortex_array::dtype::FieldName) -> bool +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::is_null_sensitive(&self, &vortex_array::dtype::FieldName) -> bool -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::reduce(&self, field_name: &vortex_array::dtype::FieldName, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::reduce(&self, &vortex_array::dtype::FieldName, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::return_dtype(&self, field_name: &vortex_array::dtype::FieldName, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::return_dtype(&self, &vortex_array::dtype::FieldName, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::serialize(&self, instance: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::simplify_untyped(&self, field_name: &vortex_array::dtype::FieldName, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::simplify_untyped(&self, &vortex_array::dtype::FieldName, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::stat_expression(&self, field_name: &vortex_array::dtype::FieldName, _expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::stat_expression(&self, &vortex_array::dtype::FieldName, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::validity(&self, options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> + +pub mod vortex_array::scalar_fn::fns::is_not_null + +pub struct vortex_array::scalar_fn::fns::is_not_null::IsNotNull + +impl core::clone::Clone for vortex_array::scalar_fn::fns::is_not_null::IsNotNull + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::clone(&self) -> vortex_array::scalar_fn::fns::is_not_null::IsNotNull + +impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::is_not_null::IsNotNull + +pub type vortex_array::scalar_fn::fns::is_not_null::IsNotNull::Options = vortex_array::scalar_fn::EmptyOptions + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::id(&self) -> vortex_array::scalar_fn::ScalarFnId + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::is_fallible(&self, &Self::Options) -> bool + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::is_null_sensitive(&self, &Self::Options) -> bool + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> pub mod vortex_array::scalar_fn::fns::is_null @@ -16326,39 +16614,39 @@ impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::i pub type vortex_array::scalar_fn::fns::is_null::IsNull::Options = vortex_array::scalar_fn::EmptyOptions -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::child_name(&self, _instance: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::execute(&self, _data: &Self::Options, args: &dyn vortex_array::scalar_fn::ExecutionArgs, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::fmt_sql(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::is_null::IsNull::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::is_fallible(&self, _instance: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::is_null_sensitive(&self, _instance: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::return_dtype(&self, _options: &Self::Options, _arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::serialize(&self, _instance: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::stat_falsification(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::validity(&self, options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> pub mod vortex_array::scalar_fn::fns::like @@ -16372,39 +16660,39 @@ impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::l pub type vortex_array::scalar_fn::fns::like::Like::Options = vortex_array::scalar_fn::fns::like::LikeOptions -pub fn vortex_array::scalar_fn::fns::like::Like::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::like::Like::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::like::Like::child_name(&self, _instance: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::like::Like::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::like::Like::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::like::Like::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::like::Like::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::like::Like::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::like::Like::execute(&self, options: &Self::Options, args: &dyn vortex_array::scalar_fn::ExecutionArgs, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::like::Like::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::like::Like::fmt_sql(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::like::Like::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::like::Like::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::like::Like::is_fallible(&self, options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::like::Like::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::like::Like::is_null_sensitive(&self, _instance: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::like::Like::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::like::Like::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::like::Like::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::like::Like::return_dtype(&self, _options: &Self::Options, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::like::Like::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::like::Like::serialize(&self, instance: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::like::Like::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::like::Like::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::like::Like::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::like::Like::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::like::Like::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::like::Like::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::like::Like::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::like::Like::stat_falsification(&self, like_opts: &vortex_array::scalar_fn::fns::like::LikeOptions, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::like::Like::stat_falsification(&self, &vortex_array::scalar_fn::fns::like::LikeOptions, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::like::Like::validity(&self, _options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::like::Like::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> pub struct vortex_array::scalar_fn::fns::like::LikeExecuteAdaptor(pub V) @@ -16414,13 +16702,13 @@ pub fn vortex_array::scalar_fn::fns::like::LikeExecuteAdaptor::default() -> v impl core::fmt::Debug for vortex_array::scalar_fn::fns::like::LikeExecuteAdaptor -pub fn vortex_array::scalar_fn::fns::like::LikeExecuteAdaptor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::like::LikeExecuteAdaptor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::kernel::ExecuteParentKernel for vortex_array::scalar_fn::fns::like::LikeExecuteAdaptor where V: vortex_array::scalar_fn::fns::like::LikeKernel pub type vortex_array::scalar_fn::fns::like::LikeExecuteAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::like::LikeExecuteAdaptor::execute_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::like::Like>, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::like::LikeExecuteAdaptor::execute_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::like::Like>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub struct vortex_array::scalar_fn::fns::like::LikeOptions @@ -16436,7 +16724,7 @@ impl core::cmp::Eq for vortex_array::scalar_fn::fns::like::LikeOptions impl core::cmp::PartialEq for vortex_array::scalar_fn::fns::like::LikeOptions -pub fn vortex_array::scalar_fn::fns::like::LikeOptions::eq(&self, other: &vortex_array::scalar_fn::fns::like::LikeOptions) -> bool +pub fn vortex_array::scalar_fn::fns::like::LikeOptions::eq(&self, &vortex_array::scalar_fn::fns::like::LikeOptions) -> bool impl core::default::Default for vortex_array::scalar_fn::fns::like::LikeOptions @@ -16444,15 +16732,15 @@ pub fn vortex_array::scalar_fn::fns::like::LikeOptions::default() -> vortex_arra impl core::fmt::Debug for vortex_array::scalar_fn::fns::like::LikeOptions -pub fn vortex_array::scalar_fn::fns::like::LikeOptions::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::like::LikeOptions::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::scalar_fn::fns::like::LikeOptions -pub fn vortex_array::scalar_fn::fns::like::LikeOptions::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::like::LikeOptions::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::scalar_fn::fns::like::LikeOptions -pub fn vortex_array::scalar_fn::fns::like::LikeOptions::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::scalar_fn::fns::like::LikeOptions::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_array::scalar_fn::fns::like::LikeOptions @@ -16466,25 +16754,25 @@ pub fn vortex_array::scalar_fn::fns::like::LikeReduceAdaptor::default() -> vo impl core::fmt::Debug for vortex_array::scalar_fn::fns::like::LikeReduceAdaptor -pub fn vortex_array::scalar_fn::fns::like::LikeReduceAdaptor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::like::LikeReduceAdaptor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::scalar_fn::fns::like::LikeReduceAdaptor where V: vortex_array::scalar_fn::fns::like::LikeReduce pub type vortex_array::scalar_fn::fns::like::LikeReduceAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::like::LikeReduceAdaptor::reduce_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::like::Like>, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::like::LikeReduceAdaptor::reduce_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::like::Like>, usize) -> vortex_error::VortexResult> pub trait vortex_array::scalar_fn::fns::like::LikeKernel: vortex_array::VTable -pub fn vortex_array::scalar_fn::fns::like::LikeKernel::like(array: vortex_array::ArrayView<'_, Self>, pattern: &vortex_array::ArrayRef, options: vortex_array::scalar_fn::fns::like::LikeOptions, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::like::LikeKernel::like(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, vortex_array::scalar_fn::fns::like::LikeOptions, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub trait vortex_array::scalar_fn::fns::like::LikeReduce: vortex_array::VTable -pub fn vortex_array::scalar_fn::fns::like::LikeReduce::like(array: vortex_array::ArrayView<'_, Self>, pattern: &vortex_array::ArrayRef, options: vortex_array::scalar_fn::fns::like::LikeOptions) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::like::LikeReduce::like(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, vortex_array::scalar_fn::fns::like::LikeOptions) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::like::LikeReduce for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::like(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, pattern: &vortex_array::ArrayRef, options: vortex_array::scalar_fn::fns::like::LikeOptions) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::like(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_array::ArrayRef, vortex_array::scalar_fn::fns::like::LikeOptions) -> vortex_error::VortexResult> pub mod vortex_array::scalar_fn::fns::list_contains @@ -16498,39 +16786,39 @@ impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::l pub type vortex_array::scalar_fn::fns::list_contains::ListContains::Options = vortex_array::scalar_fn::EmptyOptions -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::child_name(&self, _instance: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::execute(&self, _options: &Self::Options, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::fmt_sql(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::is_fallible(&self, _options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::is_null_sensitive(&self, _instance: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::return_dtype(&self, _options: &Self::Options, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::serialize(&self, _instance: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::stat_falsification(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::validity(&self, options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> pub struct vortex_array::scalar_fn::fns::list_contains::ListContainsElementExecuteAdaptor(pub V) @@ -16540,13 +16828,13 @@ pub fn vortex_array::scalar_fn::fns::list_contains::ListContainsElementExecuteAd impl core::fmt::Debug for vortex_array::scalar_fn::fns::list_contains::ListContainsElementExecuteAdaptor -pub fn vortex_array::scalar_fn::fns::list_contains::ListContainsElementExecuteAdaptor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::list_contains::ListContainsElementExecuteAdaptor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::kernel::ExecuteParentKernel for vortex_array::scalar_fn::fns::list_contains::ListContainsElementExecuteAdaptor where V: vortex_array::scalar_fn::fns::list_contains::ListContainsElementKernel pub type vortex_array::scalar_fn::fns::list_contains::ListContainsElementExecuteAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::list_contains::ListContainsElementExecuteAdaptor::execute_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::list_contains::ListContains>, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::list_contains::ListContainsElementExecuteAdaptor::execute_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::list_contains::ListContains>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub struct vortex_array::scalar_fn::fns::list_contains::ListContainsElementReduceAdaptor(pub V) @@ -16556,21 +16844,21 @@ pub fn vortex_array::scalar_fn::fns::list_contains::ListContainsElementReduceAda impl core::fmt::Debug for vortex_array::scalar_fn::fns::list_contains::ListContainsElementReduceAdaptor -pub fn vortex_array::scalar_fn::fns::list_contains::ListContainsElementReduceAdaptor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::list_contains::ListContainsElementReduceAdaptor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::scalar_fn::fns::list_contains::ListContainsElementReduceAdaptor where V: vortex_array::scalar_fn::fns::list_contains::ListContainsElementReduce pub type vortex_array::scalar_fn::fns::list_contains::ListContainsElementReduceAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::list_contains::ListContainsElementReduceAdaptor::reduce_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::list_contains::ListContains>, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::list_contains::ListContainsElementReduceAdaptor::reduce_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::list_contains::ListContains>, usize) -> vortex_error::VortexResult> pub trait vortex_array::scalar_fn::fns::list_contains::ListContainsElementKernel: vortex_array::VTable -pub fn vortex_array::scalar_fn::fns::list_contains::ListContainsElementKernel::list_contains(list: &vortex_array::ArrayRef, element: vortex_array::ArrayView<'_, Self>, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::list_contains::ListContainsElementKernel::list_contains(&vortex_array::ArrayRef, vortex_array::ArrayView<'_, Self>, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub trait vortex_array::scalar_fn::fns::list_contains::ListContainsElementReduce: vortex_array::VTable -pub fn vortex_array::scalar_fn::fns::list_contains::ListContainsElementReduce::list_contains(list: &vortex_array::ArrayRef, element: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::list_contains::ListContainsElementReduce::list_contains(&vortex_array::ArrayRef, vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> pub mod vortex_array::scalar_fn::fns::literal @@ -16584,39 +16872,39 @@ impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::l pub type vortex_array::scalar_fn::fns::literal::Literal::Options = vortex_array::scalar::Scalar -pub fn vortex_array::scalar_fn::fns::literal::Literal::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::literal::Literal::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::literal::Literal::child_name(&self, _instance: &Self::Options, _child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::literal::Literal::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::literal::Literal::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::literal::Literal::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::literal::Literal::deserialize(&self, _metadata: &[u8], session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::literal::Literal::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::literal::Literal::execute(&self, scalar: &vortex_array::scalar::Scalar, args: &dyn vortex_array::scalar_fn::ExecutionArgs, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::literal::Literal::execute(&self, &vortex_array::scalar::Scalar, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::literal::Literal::fmt_sql(&self, scalar: &vortex_array::scalar::Scalar, _expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::literal::Literal::fmt_sql(&self, &vortex_array::scalar::Scalar, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::literal::Literal::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::literal::Literal::is_fallible(&self, _instance: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::literal::Literal::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::literal::Literal::is_null_sensitive(&self, _instance: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::literal::Literal::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::literal::Literal::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::literal::Literal::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::literal::Literal::return_dtype(&self, options: &Self::Options, _arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::literal::Literal::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::literal::Literal::serialize(&self, instance: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::literal::Literal::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::literal::Literal::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::literal::Literal::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::literal::Literal::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::literal::Literal::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::literal::Literal::stat_expression(&self, scalar: &vortex_array::scalar::Scalar, _expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, _catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::literal::Literal::stat_expression(&self, &vortex_array::scalar::Scalar, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::literal::Literal::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::literal::Literal::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::literal::Literal::validity(&self, scalar: &vortex_array::scalar::Scalar, _expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::literal::Literal::validity(&self, &vortex_array::scalar::Scalar, &vortex_array::expr::Expression) -> vortex_error::VortexResult> pub mod vortex_array::scalar_fn::fns::mask @@ -16630,39 +16918,39 @@ impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::m pub type vortex_array::scalar_fn::fns::mask::Mask::Options = vortex_array::scalar_fn::EmptyOptions -pub fn vortex_array::scalar_fn::fns::mask::Mask::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::mask::Mask::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::mask::Mask::child_name(&self, _options: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::mask::Mask::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::mask::Mask::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::mask::Mask::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::mask::Mask::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::mask::Mask::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::mask::Mask::execute(&self, _options: &Self::Options, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::mask::Mask::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::mask::Mask::fmt_sql(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::mask::Mask::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::mask::Mask::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::mask::Mask::is_fallible(&self, options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::mask::Mask::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::mask::Mask::is_null_sensitive(&self, options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::mask::Mask::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::mask::Mask::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::mask::Mask::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::mask::Mask::return_dtype(&self, _options: &Self::Options, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::mask::Mask::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::mask::Mask::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::mask::Mask::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::mask::Mask::simplify(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::mask::Mask::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::mask::Mask::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::mask::Mask::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::mask::Mask::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::mask::Mask::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::mask::Mask::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::mask::Mask::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::mask::Mask::validity(&self, _options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::mask::Mask::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> pub struct vortex_array::scalar_fn::fns::mask::MaskExecuteAdaptor(pub V) @@ -16672,13 +16960,13 @@ pub fn vortex_array::scalar_fn::fns::mask::MaskExecuteAdaptor::default() -> v impl core::fmt::Debug for vortex_array::scalar_fn::fns::mask::MaskExecuteAdaptor -pub fn vortex_array::scalar_fn::fns::mask::MaskExecuteAdaptor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::mask::MaskExecuteAdaptor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::kernel::ExecuteParentKernel for vortex_array::scalar_fn::fns::mask::MaskExecuteAdaptor where V: vortex_array::scalar_fn::fns::mask::MaskKernel pub type vortex_array::scalar_fn::fns::mask::MaskExecuteAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::mask::MaskExecuteAdaptor::execute_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::mask::Mask>, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::mask::MaskExecuteAdaptor::execute_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::mask::Mask>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub struct vortex_array::scalar_fn::fns::mask::MaskReduceAdaptor(pub V) @@ -16688,77 +16976,77 @@ pub fn vortex_array::scalar_fn::fns::mask::MaskReduceAdaptor::default() -> vo impl core::fmt::Debug for vortex_array::scalar_fn::fns::mask::MaskReduceAdaptor -pub fn vortex_array::scalar_fn::fns::mask::MaskReduceAdaptor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::mask::MaskReduceAdaptor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::scalar_fn::fns::mask::MaskReduceAdaptor where V: vortex_array::scalar_fn::fns::mask::MaskReduce pub type vortex_array::scalar_fn::fns::mask::MaskReduceAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::mask::MaskReduceAdaptor::reduce_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::mask::Mask>, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::mask::MaskReduceAdaptor::reduce_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::mask::Mask>, usize) -> vortex_error::VortexResult> pub trait vortex_array::scalar_fn::fns::mask::MaskKernel: vortex_array::VTable -pub fn vortex_array::scalar_fn::fns::mask::MaskKernel::mask(array: vortex_array::ArrayView<'_, Self>, mask: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::mask::MaskKernel::mask(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskKernel for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, mask: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::mask(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub trait vortex_array::scalar_fn::fns::mask::MaskReduce: vortex_array::VTable -pub fn vortex_array::scalar_fn::fns::mask::MaskReduce::mask(array: vortex_array::ArrayView<'_, Self>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::mask::MaskReduce::mask(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::mask(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::mask(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::mask(vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::mask(vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::List -pub fn vortex_array::arrays::List::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::List>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::mask(vortex_array::ArrayView<'_, vortex_array::arrays::List>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::mask(vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::mask(vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::mask(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::mask(vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::mask(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::mask(vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::mask(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::mask::MaskReduce for vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::mask(array: vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, _mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::mask(vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, &vortex_array::ArrayRef) -> vortex_error::VortexResult> pub mod vortex_array::scalar_fn::fns::merge @@ -16776,7 +17064,7 @@ impl core::cmp::Eq for vortex_array::scalar_fn::fns::merge::DuplicateHandling impl core::cmp::PartialEq for vortex_array::scalar_fn::fns::merge::DuplicateHandling -pub fn vortex_array::scalar_fn::fns::merge::DuplicateHandling::eq(&self, other: &vortex_array::scalar_fn::fns::merge::DuplicateHandling) -> bool +pub fn vortex_array::scalar_fn::fns::merge::DuplicateHandling::eq(&self, &vortex_array::scalar_fn::fns::merge::DuplicateHandling) -> bool impl core::default::Default for vortex_array::scalar_fn::fns::merge::DuplicateHandling @@ -16784,15 +17072,15 @@ pub fn vortex_array::scalar_fn::fns::merge::DuplicateHandling::default() -> vort impl core::fmt::Debug for vortex_array::scalar_fn::fns::merge::DuplicateHandling -pub fn vortex_array::scalar_fn::fns::merge::DuplicateHandling::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::merge::DuplicateHandling::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::scalar_fn::fns::merge::DuplicateHandling -pub fn vortex_array::scalar_fn::fns::merge::DuplicateHandling::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::merge::DuplicateHandling::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::scalar_fn::fns::merge::DuplicateHandling -pub fn vortex_array::scalar_fn::fns::merge::DuplicateHandling::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::scalar_fn::fns::merge::DuplicateHandling::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_array::scalar_fn::fns::merge::DuplicateHandling @@ -16808,39 +17096,39 @@ impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::m pub type vortex_array::scalar_fn::fns::merge::Merge::Options = vortex_array::scalar_fn::fns::merge::DuplicateHandling -pub fn vortex_array::scalar_fn::fns::merge::Merge::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::merge::Merge::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::merge::Merge::child_name(&self, _instance: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::merge::Merge::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::merge::Merge::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::merge::Merge::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::merge::Merge::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::merge::Merge::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::merge::Merge::execute(&self, options: &Self::Options, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::merge::Merge::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::merge::Merge::fmt_sql(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::merge::Merge::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::merge::Merge::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::merge::Merge::is_fallible(&self, instance: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::merge::Merge::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::merge::Merge::is_null_sensitive(&self, _instance: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::merge::Merge::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::merge::Merge::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::merge::Merge::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::merge::Merge::return_dtype(&self, options: &Self::Options, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::merge::Merge::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::merge::Merge::serialize(&self, instance: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::merge::Merge::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::merge::Merge::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::merge::Merge::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::merge::Merge::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::merge::Merge::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::merge::Merge::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::merge::Merge::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::merge::Merge::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::merge::Merge::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::merge::Merge::validity(&self, _options: &Self::Options, _expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::merge::Merge::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> pub mod vortex_array::scalar_fn::fns::not @@ -16854,39 +17142,39 @@ impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::n pub type vortex_array::scalar_fn::fns::not::Not::Options = vortex_array::scalar_fn::EmptyOptions -pub fn vortex_array::scalar_fn::fns::not::Not::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::not::Not::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::not::Not::child_name(&self, _options: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::not::Not::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::not::Not::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::not::Not::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::not::Not::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::not::Not::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::not::Not::execute(&self, _data: &Self::Options, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::not::Not::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::not::Not::fmt_sql(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::not::Not::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::not::Not::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::not::Not::is_fallible(&self, _options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::not::Not::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::not::Not::is_null_sensitive(&self, _options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::not::Not::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::not::Not::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::not::Not::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::not::Not::return_dtype(&self, _options: &Self::Options, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::not::Not::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::not::Not::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::not::Not::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::not::Not::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::not::Not::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::not::Not::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::not::Not::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::not::Not::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::not::Not::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::not::Not::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::not::Not::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::not::Not::validity(&self, options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::not::Not::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> pub struct vortex_array::scalar_fn::fns::not::NotExecuteAdaptor(pub V) @@ -16896,13 +17184,13 @@ pub fn vortex_array::scalar_fn::fns::not::NotExecuteAdaptor::default() -> vor impl core::fmt::Debug for vortex_array::scalar_fn::fns::not::NotExecuteAdaptor -pub fn vortex_array::scalar_fn::fns::not::NotExecuteAdaptor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::not::NotExecuteAdaptor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::kernel::ExecuteParentKernel for vortex_array::scalar_fn::fns::not::NotExecuteAdaptor where V: vortex_array::scalar_fn::fns::not::NotKernel pub type vortex_array::scalar_fn::fns::not::NotExecuteAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::not::NotExecuteAdaptor::execute_parent(&self, array: vortex_array::ArrayView<'_, V>, _parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::not::Not>, _child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::not::NotExecuteAdaptor::execute_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::not::Not>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub struct vortex_array::scalar_fn::fns::not::NotReduceAdaptor(pub V) @@ -16912,25 +17200,25 @@ pub fn vortex_array::scalar_fn::fns::not::NotReduceAdaptor::default() -> vort impl core::fmt::Debug for vortex_array::scalar_fn::fns::not::NotReduceAdaptor -pub fn vortex_array::scalar_fn::fns::not::NotReduceAdaptor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::not::NotReduceAdaptor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::scalar_fn::fns::not::NotReduceAdaptor where V: vortex_array::scalar_fn::fns::not::NotReduce pub type vortex_array::scalar_fn::fns::not::NotReduceAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::not::NotReduceAdaptor::reduce_parent(&self, array: vortex_array::ArrayView<'_, V>, _parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::not::Not>, _child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::not::NotReduceAdaptor::reduce_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::not::Not>, usize) -> vortex_error::VortexResult> pub trait vortex_array::scalar_fn::fns::not::NotKernel: vortex_array::VTable -pub fn vortex_array::scalar_fn::fns::not::NotKernel::invert(array: vortex_array::ArrayView<'_, Self>, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::not::NotKernel::invert(vortex_array::ArrayView<'_, Self>, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub trait vortex_array::scalar_fn::fns::not::NotReduce: vortex_array::VTable -pub fn vortex_array::scalar_fn::fns::not::NotReduce::invert(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::not::NotReduce::invert(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::not::NotReduce for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::invert(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::invert(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>) -> vortex_error::VortexResult> pub mod vortex_array::scalar_fn::fns::operators @@ -16962,33 +17250,33 @@ impl core::cmp::Eq for vortex_array::scalar_fn::fns::operators::CompareOperator impl core::cmp::PartialEq for vortex_array::scalar_fn::fns::operators::CompareOperator -pub fn vortex_array::scalar_fn::fns::operators::CompareOperator::eq(&self, other: &vortex_array::scalar_fn::fns::operators::CompareOperator) -> bool +pub fn vortex_array::scalar_fn::fns::operators::CompareOperator::eq(&self, &vortex_array::scalar_fn::fns::operators::CompareOperator) -> bool impl core::cmp::PartialOrd for vortex_array::scalar_fn::fns::operators::CompareOperator -pub fn vortex_array::scalar_fn::fns::operators::CompareOperator::partial_cmp(&self, other: &vortex_array::scalar_fn::fns::operators::CompareOperator) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::operators::CompareOperator::partial_cmp(&self, &vortex_array::scalar_fn::fns::operators::CompareOperator) -> core::option::Option impl core::convert::From for vortex_array::scalar_fn::fns::operators::Operator -pub fn vortex_array::scalar_fn::fns::operators::Operator::from(value: vortex_array::scalar_fn::fns::operators::CompareOperator) -> Self +pub fn vortex_array::scalar_fn::fns::operators::Operator::from(vortex_array::scalar_fn::fns::operators::CompareOperator) -> Self impl core::convert::TryFrom for vortex_array::scalar_fn::fns::operators::CompareOperator pub type vortex_array::scalar_fn::fns::operators::CompareOperator::Error = vortex_error::VortexError -pub fn vortex_array::scalar_fn::fns::operators::CompareOperator::try_from(value: vortex_array::scalar_fn::fns::operators::Operator) -> core::result::Result +pub fn vortex_array::scalar_fn::fns::operators::CompareOperator::try_from(vortex_array::scalar_fn::fns::operators::Operator) -> core::result::Result impl core::fmt::Debug for vortex_array::scalar_fn::fns::operators::CompareOperator -pub fn vortex_array::scalar_fn::fns::operators::CompareOperator::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::operators::CompareOperator::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::scalar_fn::fns::operators::CompareOperator -pub fn vortex_array::scalar_fn::fns::operators::CompareOperator::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::operators::CompareOperator::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::scalar_fn::fns::operators::CompareOperator -pub fn vortex_array::scalar_fn::fns::operators::CompareOperator::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::scalar_fn::fns::operators::CompareOperator::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_array::scalar_fn::fns::operators::CompareOperator @@ -17040,55 +17328,55 @@ impl core::cmp::Eq for vortex_array::scalar_fn::fns::operators::Operator impl core::cmp::PartialEq for vortex_array::scalar_fn::fns::operators::Operator -pub fn vortex_array::scalar_fn::fns::operators::Operator::eq(&self, other: &vortex_array::scalar_fn::fns::operators::Operator) -> bool +pub fn vortex_array::scalar_fn::fns::operators::Operator::eq(&self, &vortex_array::scalar_fn::fns::operators::Operator) -> bool impl core::cmp::PartialOrd for vortex_array::scalar_fn::fns::operators::Operator -pub fn vortex_array::scalar_fn::fns::operators::Operator::partial_cmp(&self, other: &vortex_array::scalar_fn::fns::operators::Operator) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::operators::Operator::partial_cmp(&self, &vortex_array::scalar_fn::fns::operators::Operator) -> core::option::Option impl core::convert::From for vortex_array::scalar_fn::fns::operators::Operator -pub fn vortex_array::scalar_fn::fns::operators::Operator::from(op: vortex_array::scalar::NumericOperator) -> Self +pub fn vortex_array::scalar_fn::fns::operators::Operator::from(vortex_array::scalar::NumericOperator) -> Self impl core::convert::From for vortex_array::scalar_fn::fns::operators::Operator -pub fn vortex_array::scalar_fn::fns::operators::Operator::from(value: vortex_array::scalar_fn::fns::operators::CompareOperator) -> Self +pub fn vortex_array::scalar_fn::fns::operators::Operator::from(vortex_array::scalar_fn::fns::operators::CompareOperator) -> Self impl core::convert::From for i32 -pub fn i32::from(value: vortex_array::scalar_fn::fns::operators::Operator) -> Self +pub fn i32::from(vortex_array::scalar_fn::fns::operators::Operator) -> Self impl core::convert::From for vortex_proto::expr::binary_opts::BinaryOp -pub fn vortex_proto::expr::binary_opts::BinaryOp::from(value: vortex_array::scalar_fn::fns::operators::Operator) -> Self +pub fn vortex_proto::expr::binary_opts::BinaryOp::from(vortex_array::scalar_fn::fns::operators::Operator) -> Self impl core::convert::From for vortex_array::scalar_fn::fns::operators::Operator -pub fn vortex_array::scalar_fn::fns::operators::Operator::from(value: vortex_proto::expr::binary_opts::BinaryOp) -> Self +pub fn vortex_array::scalar_fn::fns::operators::Operator::from(vortex_proto::expr::binary_opts::BinaryOp) -> Self impl core::convert::TryFrom for vortex_array::scalar_fn::fns::operators::Operator pub type vortex_array::scalar_fn::fns::operators::Operator::Error = vortex_error::VortexError -pub fn vortex_array::scalar_fn::fns::operators::Operator::try_from(value: i32) -> core::result::Result +pub fn vortex_array::scalar_fn::fns::operators::Operator::try_from(i32) -> core::result::Result impl core::convert::TryFrom for vortex_array::scalar_fn::fns::operators::CompareOperator pub type vortex_array::scalar_fn::fns::operators::CompareOperator::Error = vortex_error::VortexError -pub fn vortex_array::scalar_fn::fns::operators::CompareOperator::try_from(value: vortex_array::scalar_fn::fns::operators::Operator) -> core::result::Result +pub fn vortex_array::scalar_fn::fns::operators::CompareOperator::try_from(vortex_array::scalar_fn::fns::operators::Operator) -> core::result::Result impl core::fmt::Debug for vortex_array::scalar_fn::fns::operators::Operator -pub fn vortex_array::scalar_fn::fns::operators::Operator::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::operators::Operator::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::scalar_fn::fns::operators::Operator -pub fn vortex_array::scalar_fn::fns::operators::Operator::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::operators::Operator::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::scalar_fn::fns::operators::Operator -pub fn vortex_array::scalar_fn::fns::operators::Operator::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::scalar_fn::fns::operators::Operator::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::Copy for vortex_array::scalar_fn::fns::operators::Operator @@ -17106,39 +17394,39 @@ impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::p pub type vortex_array::scalar_fn::fns::pack::Pack::Options = vortex_array::scalar_fn::fns::pack::PackOptions -pub fn vortex_array::scalar_fn::fns::pack::Pack::arity(&self, options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::pack::Pack::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::pack::Pack::child_name(&self, instance: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::pack::Pack::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::pack::Pack::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::pack::Pack::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::pack::Pack::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::pack::Pack::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::pack::Pack::execute(&self, options: &Self::Options, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::pack::Pack::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::pack::Pack::fmt_sql(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::pack::Pack::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::pack::Pack::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::pack::Pack::is_fallible(&self, _instance: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::pack::Pack::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::pack::Pack::is_null_sensitive(&self, _instance: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::pack::Pack::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::pack::Pack::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::pack::Pack::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::pack::Pack::return_dtype(&self, options: &Self::Options, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::pack::Pack::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::pack::Pack::serialize(&self, instance: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::pack::Pack::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::pack::Pack::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::pack::Pack::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::pack::Pack::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::pack::Pack::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::pack::Pack::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::pack::Pack::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::pack::Pack::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::pack::Pack::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::pack::Pack::validity(&self, _options: &Self::Options, _expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::pack::Pack::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> pub struct vortex_array::scalar_fn::fns::pack::PackOptions @@ -17154,19 +17442,19 @@ impl core::cmp::Eq for vortex_array::scalar_fn::fns::pack::PackOptions impl core::cmp::PartialEq for vortex_array::scalar_fn::fns::pack::PackOptions -pub fn vortex_array::scalar_fn::fns::pack::PackOptions::eq(&self, other: &vortex_array::scalar_fn::fns::pack::PackOptions) -> bool +pub fn vortex_array::scalar_fn::fns::pack::PackOptions::eq(&self, &vortex_array::scalar_fn::fns::pack::PackOptions) -> bool impl core::fmt::Debug for vortex_array::scalar_fn::fns::pack::PackOptions -pub fn vortex_array::scalar_fn::fns::pack::PackOptions::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::pack::PackOptions::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::scalar_fn::fns::pack::PackOptions -pub fn vortex_array::scalar_fn::fns::pack::PackOptions::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::pack::PackOptions::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::scalar_fn::fns::pack::PackOptions -pub fn vortex_array::scalar_fn::fns::pack::PackOptions::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::scalar_fn::fns::pack::PackOptions::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_array::scalar_fn::fns::pack::PackOptions @@ -17182,39 +17470,39 @@ impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::r pub type vortex_array::scalar_fn::fns::root::Root::Options = vortex_array::scalar_fn::EmptyOptions -pub fn vortex_array::scalar_fn::fns::root::Root::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::root::Root::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::root::Root::child_name(&self, _instance: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::root::Root::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::root::Root::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::root::Root::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::root::Root::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::root::Root::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::root::Root::execute(&self, _data: &Self::Options, _args: &dyn vortex_array::scalar_fn::ExecutionArgs, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::root::Root::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::root::Root::fmt_sql(&self, _options: &Self::Options, _expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::root::Root::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::root::Root::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::root::Root::is_fallible(&self, _options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::root::Root::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::root::Root::is_null_sensitive(&self, _options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::root::Root::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::root::Root::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::root::Root::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::root::Root::return_dtype(&self, _options: &Self::Options, _arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::root::Root::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::root::Root::serialize(&self, _instance: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::root::Root::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::root::Root::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::root::Root::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::root::Root::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::root::Root::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::root::Root::stat_expression(&self, _options: &Self::Options, _expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::root::Root::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::root::Root::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::root::Root::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::root::Root::validity(&self, options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::root::Root::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> pub mod vortex_array::scalar_fn::fns::select @@ -17226,17 +17514,17 @@ pub vortex_array::scalar_fn::fns::select::FieldSelection::Include(vortex_array:: impl vortex_array::scalar_fn::fns::select::FieldSelection -pub fn vortex_array::scalar_fn::fns::select::FieldSelection::exclude(columns: vortex_array::dtype::FieldNames) -> Self +pub fn vortex_array::scalar_fn::fns::select::FieldSelection::exclude(vortex_array::dtype::FieldNames) -> Self pub fn vortex_array::scalar_fn::fns::select::FieldSelection::field_names(&self) -> &vortex_array::dtype::FieldNames -pub fn vortex_array::scalar_fn::fns::select::FieldSelection::include(columns: vortex_array::dtype::FieldNames) -> Self +pub fn vortex_array::scalar_fn::fns::select::FieldSelection::include(vortex_array::dtype::FieldNames) -> Self pub fn vortex_array::scalar_fn::fns::select::FieldSelection::is_exclude(&self) -> bool pub fn vortex_array::scalar_fn::fns::select::FieldSelection::is_include(&self) -> bool -pub fn vortex_array::scalar_fn::fns::select::FieldSelection::normalize_to_included_fields(&self, available_fields: &vortex_array::dtype::FieldNames) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::select::FieldSelection::normalize_to_included_fields(&self, &vortex_array::dtype::FieldNames) -> vortex_error::VortexResult impl core::clone::Clone for vortex_array::scalar_fn::fns::select::FieldSelection @@ -17246,19 +17534,19 @@ impl core::cmp::Eq for vortex_array::scalar_fn::fns::select::FieldSelection impl core::cmp::PartialEq for vortex_array::scalar_fn::fns::select::FieldSelection -pub fn vortex_array::scalar_fn::fns::select::FieldSelection::eq(&self, other: &vortex_array::scalar_fn::fns::select::FieldSelection) -> bool +pub fn vortex_array::scalar_fn::fns::select::FieldSelection::eq(&self, &vortex_array::scalar_fn::fns::select::FieldSelection) -> bool impl core::fmt::Debug for vortex_array::scalar_fn::fns::select::FieldSelection -pub fn vortex_array::scalar_fn::fns::select::FieldSelection::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::select::FieldSelection::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::scalar_fn::fns::select::FieldSelection -pub fn vortex_array::scalar_fn::fns::select::FieldSelection::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::select::FieldSelection::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::scalar_fn::fns::select::FieldSelection -pub fn vortex_array::scalar_fn::fns::select::FieldSelection::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::scalar_fn::fns::select::FieldSelection::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_array::scalar_fn::fns::select::FieldSelection @@ -17272,39 +17560,39 @@ impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::s pub type vortex_array::scalar_fn::fns::select::Select::Options = vortex_array::scalar_fn::fns::select::FieldSelection -pub fn vortex_array::scalar_fn::fns::select::Select::arity(&self, _options: &vortex_array::scalar_fn::fns::select::FieldSelection) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::select::Select::arity(&self, &vortex_array::scalar_fn::fns::select::FieldSelection) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::select::Select::child_name(&self, _instance: &vortex_array::scalar_fn::fns::select::FieldSelection, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::select::Select::child_name(&self, &vortex_array::scalar_fn::fns::select::FieldSelection, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::select::Select::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::select::Select::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::select::Select::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::select::Select::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::select::Select::execute(&self, selection: &vortex_array::scalar_fn::fns::select::FieldSelection, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::select::Select::execute(&self, &vortex_array::scalar_fn::fns::select::FieldSelection, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::select::Select::fmt_sql(&self, selection: &vortex_array::scalar_fn::fns::select::FieldSelection, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::select::Select::fmt_sql(&self, &vortex_array::scalar_fn::fns::select::FieldSelection, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::select::Select::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::select::Select::is_fallible(&self, _instance: &vortex_array::scalar_fn::fns::select::FieldSelection) -> bool +pub fn vortex_array::scalar_fn::fns::select::Select::is_fallible(&self, &vortex_array::scalar_fn::fns::select::FieldSelection) -> bool -pub fn vortex_array::scalar_fn::fns::select::Select::is_null_sensitive(&self, _instance: &vortex_array::scalar_fn::fns::select::FieldSelection) -> bool +pub fn vortex_array::scalar_fn::fns::select::Select::is_null_sensitive(&self, &vortex_array::scalar_fn::fns::select::FieldSelection) -> bool -pub fn vortex_array::scalar_fn::fns::select::Select::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::select::Select::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::select::Select::return_dtype(&self, selection: &vortex_array::scalar_fn::fns::select::FieldSelection, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::select::Select::return_dtype(&self, &vortex_array::scalar_fn::fns::select::FieldSelection, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::select::Select::serialize(&self, instance: &vortex_array::scalar_fn::fns::select::FieldSelection) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::select::Select::serialize(&self, &vortex_array::scalar_fn::fns::select::FieldSelection) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::select::Select::simplify(&self, selection: &vortex_array::scalar_fn::fns::select::FieldSelection, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::select::Select::simplify(&self, &vortex_array::scalar_fn::fns::select::FieldSelection, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::select::Select::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::select::Select::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::select::Select::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::select::Select::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::select::Select::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::select::Select::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::select::Select::validity(&self, options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::select::Select::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> pub mod vortex_array::scalar_fn::fns::zip @@ -17318,39 +17606,39 @@ impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::z pub type vortex_array::scalar_fn::fns::zip::Zip::Options = vortex_array::scalar_fn::EmptyOptions -pub fn vortex_array::scalar_fn::fns::zip::Zip::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::zip::Zip::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::zip::Zip::child_name(&self, _options: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::zip::Zip::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::zip::Zip::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::zip::Zip::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::zip::Zip::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::zip::Zip::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::zip::Zip::execute(&self, _options: &Self::Options, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::zip::Zip::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::zip::Zip::fmt_sql(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::zip::Zip::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::zip::Zip::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::zip::Zip::is_fallible(&self, _options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::zip::Zip::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::zip::Zip::is_null_sensitive(&self, _options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::zip::Zip::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::zip::Zip::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::zip::Zip::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::zip::Zip::return_dtype(&self, _options: &Self::Options, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::zip::Zip::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::zip::Zip::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::zip::Zip::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::zip::Zip::simplify(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, _ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::zip::Zip::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::zip::Zip::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::zip::Zip::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::zip::Zip::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::zip::Zip::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::zip::Zip::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::zip::Zip::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::zip::Zip::validity(&self, options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::zip::Zip::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> pub struct vortex_array::scalar_fn::fns::zip::ZipExecuteAdaptor(pub V) @@ -17360,13 +17648,13 @@ pub fn vortex_array::scalar_fn::fns::zip::ZipExecuteAdaptor::default() -> vor impl core::fmt::Debug for vortex_array::scalar_fn::fns::zip::ZipExecuteAdaptor -pub fn vortex_array::scalar_fn::fns::zip::ZipExecuteAdaptor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::zip::ZipExecuteAdaptor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::kernel::ExecuteParentKernel for vortex_array::scalar_fn::fns::zip::ZipExecuteAdaptor where V: vortex_array::scalar_fn::fns::zip::ZipKernel pub type vortex_array::scalar_fn::fns::zip::ZipExecuteAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::zip::ZipExecuteAdaptor::execute_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::zip::Zip>, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::zip::ZipExecuteAdaptor::execute_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::zip::Zip>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub struct vortex_array::scalar_fn::fns::zip::ZipReduceAdaptor(pub V) @@ -17376,33 +17664,85 @@ pub fn vortex_array::scalar_fn::fns::zip::ZipReduceAdaptor::default() -> vort impl core::fmt::Debug for vortex_array::scalar_fn::fns::zip::ZipReduceAdaptor -pub fn vortex_array::scalar_fn::fns::zip::ZipReduceAdaptor::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::zip::ZipReduceAdaptor::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::optimizer::rules::ArrayParentReduceRule for vortex_array::scalar_fn::fns::zip::ZipReduceAdaptor where V: vortex_array::scalar_fn::fns::zip::ZipReduce pub type vortex_array::scalar_fn::fns::zip::ZipReduceAdaptor::Parent = vortex_array::arrays::scalar_fn::ExactScalarFn -pub fn vortex_array::scalar_fn::fns::zip::ZipReduceAdaptor::reduce_parent(&self, array: vortex_array::ArrayView<'_, V>, parent: vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::zip::Zip>, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::zip::ZipReduceAdaptor::reduce_parent(&self, vortex_array::ArrayView<'_, V>, vortex_array::arrays::scalar_fn::ScalarFnArrayView<'_, vortex_array::scalar_fn::fns::zip::Zip>, usize) -> vortex_error::VortexResult> pub trait vortex_array::scalar_fn::fns::zip::ZipKernel: vortex_array::VTable -pub fn vortex_array::scalar_fn::fns::zip::ZipKernel::zip(array: vortex_array::ArrayView<'_, Self>, if_false: &vortex_array::ArrayRef, mask: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::zip::ZipKernel::zip(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::zip::ZipKernel for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::zip(if_true: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, if_false: &vortex_array::ArrayRef, mask: &vortex_array::ArrayRef, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::zip(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, &vortex_array::ArrayRef, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::zip::ZipKernel for vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::zip(if_true: vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, if_false: &vortex_array::ArrayRef, mask: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::zip(vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, &vortex_array::ArrayRef, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::fns::zip::ZipKernel for vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::zip(if_true: vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, if_false: &vortex_array::ArrayRef, mask: &vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::zip(vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, &vortex_array::ArrayRef, &vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub trait vortex_array::scalar_fn::fns::zip::ZipReduce: vortex_array::VTable -pub fn vortex_array::scalar_fn::fns::zip::ZipReduce::zip(array: vortex_array::ArrayView<'_, Self>, if_false: &vortex_array::ArrayRef, mask: &vortex_array::ArrayRef) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::zip::ZipReduce::zip(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, &vortex_array::ArrayRef) -> vortex_error::VortexResult> + +pub mod vortex_array::scalar_fn::internal + +pub mod vortex_array::scalar_fn::internal::row_count + +pub struct vortex_array::scalar_fn::internal::row_count::RowCount + +impl core::clone::Clone for vortex_array::scalar_fn::internal::row_count::RowCount + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::clone(&self) -> vortex_array::scalar_fn::internal::row_count::RowCount + +impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::internal::row_count::RowCount + +pub type vortex_array::scalar_fn::internal::row_count::RowCount::Options = vortex_array::scalar_fn::EmptyOptions + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::id(&self) -> vortex_array::scalar_fn::ScalarFnId + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::is_fallible(&self, &Self::Options) -> bool + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::is_null_sensitive(&self, &Self::Options) -> bool + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> + +pub fn vortex_array::scalar_fn::internal::row_count::contains_row_count(&vortex_array::ArrayRef) -> bool + +pub fn vortex_array::scalar_fn::internal::row_count::substitute_row_count(vortex_array::ArrayRef, &vortex_array::ArrayRef) -> vortex_error::VortexResult pub mod vortex_array::scalar_fn::session @@ -17410,7 +17750,7 @@ pub struct vortex_array::scalar_fn::session::ScalarFnSession impl vortex_array::scalar_fn::session::ScalarFnSession -pub fn vortex_array::scalar_fn::session::ScalarFnSession::register(&self, vtable: V) +pub fn vortex_array::scalar_fn::session::ScalarFnSession::register(&self, V) pub fn vortex_array::scalar_fn::session::ScalarFnSession::registry(&self) -> &vortex_array::scalar_fn::session::ScalarFnRegistry @@ -17420,7 +17760,13 @@ pub fn vortex_array::scalar_fn::session::ScalarFnSession::default() -> Self impl core::fmt::Debug for vortex_array::scalar_fn::session::ScalarFnSession -pub fn vortex_array::scalar_fn::session::ScalarFnSession::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::session::ScalarFnSession::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +impl vortex_session::SessionVar for vortex_array::scalar_fn::session::ScalarFnSession + +pub fn vortex_array::scalar_fn::session::ScalarFnSession::as_any(&self) -> &dyn core::any::Any + +pub fn vortex_array::scalar_fn::session::ScalarFnSession::as_any_mut(&mut self) -> &mut dyn core::any::Any pub trait vortex_array::scalar_fn::session::ScalarFnSessionExt: vortex_session::SessionExt @@ -17444,7 +17790,7 @@ pub vortex_array::scalar_fn::Arity::Variadic::min: usize impl vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::Arity::matches(&self, arg_count: usize) -> bool +pub fn vortex_array::scalar_fn::Arity::matches(&self, usize) -> bool impl core::clone::Clone for vortex_array::scalar_fn::Arity @@ -17454,15 +17800,15 @@ impl core::cmp::Eq for vortex_array::scalar_fn::Arity impl core::cmp::PartialEq for vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::Arity::eq(&self, other: &vortex_array::scalar_fn::Arity) -> bool +pub fn vortex_array::scalar_fn::Arity::eq(&self, &vortex_array::scalar_fn::Arity) -> bool impl core::fmt::Debug for vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::Arity::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::Arity::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::Arity::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::Arity::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_array::scalar_fn::Arity @@ -17478,19 +17824,19 @@ impl core::cmp::Eq for vortex_array::scalar_fn::EmptyOptions impl core::cmp::PartialEq for vortex_array::scalar_fn::EmptyOptions -pub fn vortex_array::scalar_fn::EmptyOptions::eq(&self, other: &vortex_array::scalar_fn::EmptyOptions) -> bool +pub fn vortex_array::scalar_fn::EmptyOptions::eq(&self, &vortex_array::scalar_fn::EmptyOptions) -> bool impl core::fmt::Debug for vortex_array::scalar_fn::EmptyOptions -pub fn vortex_array::scalar_fn::EmptyOptions::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::EmptyOptions::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::scalar_fn::EmptyOptions -pub fn vortex_array::scalar_fn::EmptyOptions::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::EmptyOptions::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::scalar_fn::EmptyOptions -pub fn vortex_array::scalar_fn::EmptyOptions::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::scalar_fn::EmptyOptions::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_array::scalar_fn::EmptyOptions @@ -17498,7 +17844,7 @@ pub struct vortex_array::scalar_fn::ForeignScalarFnOptions impl vortex_array::scalar_fn::ForeignScalarFnOptions -pub fn vortex_array::scalar_fn::ForeignScalarFnOptions::new(metadata: alloc::vec::Vec, arity: usize) -> Self +pub fn vortex_array::scalar_fn::ForeignScalarFnOptions::new(alloc::vec::Vec, usize) -> Self impl core::clone::Clone for vortex_array::scalar_fn::ForeignScalarFnOptions @@ -17508,19 +17854,19 @@ impl core::cmp::Eq for vortex_array::scalar_fn::ForeignScalarFnOptions impl core::cmp::PartialEq for vortex_array::scalar_fn::ForeignScalarFnOptions -pub fn vortex_array::scalar_fn::ForeignScalarFnOptions::eq(&self, other: &vortex_array::scalar_fn::ForeignScalarFnOptions) -> bool +pub fn vortex_array::scalar_fn::ForeignScalarFnOptions::eq(&self, &vortex_array::scalar_fn::ForeignScalarFnOptions) -> bool impl core::fmt::Debug for vortex_array::scalar_fn::ForeignScalarFnOptions -pub fn vortex_array::scalar_fn::ForeignScalarFnOptions::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::ForeignScalarFnOptions::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::scalar_fn::ForeignScalarFnOptions -pub fn vortex_array::scalar_fn::ForeignScalarFnOptions::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::ForeignScalarFnOptions::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::scalar_fn::ForeignScalarFnOptions -pub fn vortex_array::scalar_fn::ForeignScalarFnOptions::hash<__H: core::hash::Hasher>(&self, state: &mut __H) +pub fn vortex_array::scalar_fn::ForeignScalarFnOptions::hash<__H: core::hash::Hasher>(&self, &mut __H) impl core::marker::StructuralPartialEq for vortex_array::scalar_fn::ForeignScalarFnOptions @@ -17528,9 +17874,9 @@ pub struct vortex_array::scalar_fn::ForeignScalarFnVTable impl vortex_array::scalar_fn::ForeignScalarFnVTable -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::make_scalar_fn(id: vortex_array::scalar_fn::ScalarFnId, metadata: alloc::vec::Vec, arity: usize) -> vortex_array::scalar_fn::ScalarFnRef +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::make_scalar_fn(vortex_array::scalar_fn::ScalarFnId, alloc::vec::Vec, usize) -> vortex_array::scalar_fn::ScalarFnRef -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::new(id: vortex_array::scalar_fn::ScalarFnId) -> Self +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::new(vortex_array::scalar_fn::ScalarFnId) -> Self impl core::clone::Clone for vortex_array::scalar_fn::ForeignScalarFnVTable @@ -17538,57 +17884,45 @@ pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::clone(&self) -> vortex_ar impl core::fmt::Debug for vortex_array::scalar_fn::ForeignScalarFnVTable -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::ForeignScalarFnVTable pub type vortex_array::scalar_fn::ForeignScalarFnVTable::Options = vortex_array::scalar_fn::ForeignScalarFnOptions -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::arity(&self, options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::child_name(&self, _options: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::deserialize(&self, metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::execute(&self, _options: &Self::Options, _args: &dyn vortex_array::scalar_fn::ExecutionArgs, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::fmt_sql(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::is_fallible(&self, options: &Self::Options) -> bool - -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::is_null_sensitive(&self, options: &Self::Options) -> bool - -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> - -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::return_dtype(&self, _options: &Self::Options, _args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult - -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::serialize(&self, options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::validity(&self, options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub struct vortex_array::scalar_fn::ScalarFn +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -impl vortex_array::scalar_fn::ScalarFn +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::ScalarFn::erased(self) -> vortex_array::scalar_fn::ScalarFnRef +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::ScalarFn::new(vtable: V, options: ::Options) -> Self - -pub fn vortex_array::scalar_fn::ScalarFn::options(&self) -> &::Options - -pub fn vortex_array::scalar_fn::ScalarFn::vtable(&self) -> &V +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> pub struct vortex_array::scalar_fn::ScalarFnOptions<'a> @@ -17604,19 +17938,19 @@ impl core::cmp::Eq for vortex_array::scalar_fn::ScalarFnOptions<'_> impl core::cmp::PartialEq for vortex_array::scalar_fn::ScalarFnOptions<'_> -pub fn vortex_array::scalar_fn::ScalarFnOptions<'_>::eq(&self, other: &Self) -> bool +pub fn vortex_array::scalar_fn::ScalarFnOptions<'_>::eq(&self, &Self) -> bool impl core::fmt::Debug for vortex_array::scalar_fn::ScalarFnOptions<'_> -pub fn vortex_array::scalar_fn::ScalarFnOptions<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::ScalarFnOptions<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::scalar_fn::ScalarFnOptions<'_> -pub fn vortex_array::scalar_fn::ScalarFnOptions<'_>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::ScalarFnOptions<'_>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::scalar_fn::ScalarFnOptions<'_> -pub fn vortex_array::scalar_fn::ScalarFnOptions<'_>::hash(&self, state: &mut H) +pub fn vortex_array::scalar_fn::ScalarFnOptions<'_>::hash(&self, &mut H) pub struct vortex_array::scalar_fn::ScalarFnRef(_) @@ -17626,13 +17960,13 @@ pub fn vortex_array::scalar_fn::ScalarFnRef::as_(&self) -> core::option::Option<&::Options> -pub fn vortex_array::scalar_fn::ScalarFnRef::coerce_args(&self, arg_types: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::ScalarFnRef::coerce_args(&self, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::ScalarFnRef::downcast(self) -> alloc::sync::Arc> +pub fn vortex_array::scalar_fn::ScalarFnRef::downcast(self) -> alloc::sync::Arc> -pub fn vortex_array::scalar_fn::ScalarFnRef::downcast_ref(&self) -> core::option::Option<&vortex_array::scalar_fn::ScalarFn> +pub fn vortex_array::scalar_fn::ScalarFnRef::downcast_ref(&self) -> core::option::Option<&vortex_array::scalar_fn::TypedScalarFnInstance> -pub fn vortex_array::scalar_fn::ScalarFnRef::execute(&self, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::ScalarFnRef::execute(&self, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_array::scalar_fn::ScalarFnRef::id(&self) -> vortex_array::scalar_fn::ScalarFnId @@ -17640,15 +17974,15 @@ pub fn vortex_array::scalar_fn::ScalarFnRef::is vortex_array::scalar_fn::ScalarFnOptions<'_> -pub fn vortex_array::scalar_fn::ScalarFnRef::reduce(&self, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::ScalarFnRef::reduce(&self, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::ScalarFnRef::return_dtype(&self, arg_types: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::ScalarFnRef::return_dtype(&self, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult pub fn vortex_array::scalar_fn::ScalarFnRef::signature(&self) -> vortex_array::scalar_fn::ScalarFnSignature<'_> -pub fn vortex_array::scalar_fn::ScalarFnRef::try_downcast(self) -> core::result::Result>, vortex_array::scalar_fn::ScalarFnRef> +pub fn vortex_array::scalar_fn::ScalarFnRef::try_downcast(self) -> core::result::Result>, vortex_array::scalar_fn::ScalarFnRef> -pub fn vortex_array::scalar_fn::ScalarFnRef::validity(&self, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::ScalarFnRef::validity(&self, &vortex_array::expr::Expression) -> vortex_error::VortexResult impl core::clone::Clone for vortex_array::scalar_fn::ScalarFnRef @@ -17658,19 +17992,19 @@ impl core::cmp::Eq for vortex_array::scalar_fn::ScalarFnRef impl core::cmp::PartialEq for vortex_array::scalar_fn::ScalarFnRef -pub fn vortex_array::scalar_fn::ScalarFnRef::eq(&self, other: &Self) -> bool +pub fn vortex_array::scalar_fn::ScalarFnRef::eq(&self, &Self) -> bool impl core::fmt::Debug for vortex_array::scalar_fn::ScalarFnRef -pub fn vortex_array::scalar_fn::ScalarFnRef::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::ScalarFnRef::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::scalar_fn::ScalarFnRef -pub fn vortex_array::scalar_fn::ScalarFnRef::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::ScalarFnRef::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::hash::Hash for vortex_array::scalar_fn::ScalarFnRef -pub fn vortex_array::scalar_fn::ScalarFnRef::hash(&self, state: &mut H) +pub fn vortex_array::scalar_fn::ScalarFnRef::hash(&self, &mut H) pub struct vortex_array::scalar_fn::ScalarFnSignature<'a> @@ -17678,21 +18012,33 @@ impl vortex_array::scalar_fn::ScalarFnSignature<'_> pub fn vortex_array::scalar_fn::ScalarFnSignature<'_>::arity(&self) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::ScalarFnSignature<'_>::child_name(&self, index: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::ScalarFnSignature<'_>::child_name(&self, usize) -> vortex_array::scalar_fn::ChildName pub fn vortex_array::scalar_fn::ScalarFnSignature<'_>::is_fallible(&self) -> bool pub fn vortex_array::scalar_fn::ScalarFnSignature<'_>::is_null_sensitive(&self) -> bool +pub struct vortex_array::scalar_fn::TypedScalarFnInstance + +impl vortex_array::scalar_fn::TypedScalarFnInstance + +pub fn vortex_array::scalar_fn::TypedScalarFnInstance::erased(self) -> vortex_array::scalar_fn::ScalarFnRef + +pub fn vortex_array::scalar_fn::TypedScalarFnInstance::new(V, ::Options) -> Self + +pub fn vortex_array::scalar_fn::TypedScalarFnInstance::options(&self) -> &::Options + +pub fn vortex_array::scalar_fn::TypedScalarFnInstance::vtable(&self) -> &V + pub struct vortex_array::scalar_fn::VecExecutionArgs impl vortex_array::scalar_fn::VecExecutionArgs -pub fn vortex_array::scalar_fn::VecExecutionArgs::new(inputs: alloc::vec::Vec, row_count: usize) -> Self +pub fn vortex_array::scalar_fn::VecExecutionArgs::new(alloc::vec::Vec, usize) -> Self impl vortex_array::scalar_fn::ExecutionArgs for vortex_array::scalar_fn::VecExecutionArgs -pub fn vortex_array::scalar_fn::VecExecutionArgs::get(&self, index: usize) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::VecExecutionArgs::get(&self, usize) -> vortex_error::VortexResult pub fn vortex_array::scalar_fn::VecExecutionArgs::num_inputs(&self) -> usize @@ -17700,7 +18046,7 @@ pub fn vortex_array::scalar_fn::VecExecutionArgs::row_count(&self) -> usize pub trait vortex_array::scalar_fn::ExecutionArgs -pub fn vortex_array::scalar_fn::ExecutionArgs::get(&self, index: usize) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::ExecutionArgs::get(&self, usize) -> vortex_error::VortexResult pub fn vortex_array::scalar_fn::ExecutionArgs::num_inputs(&self) -> usize @@ -17708,7 +18054,7 @@ pub fn vortex_array::scalar_fn::ExecutionArgs::row_count(&self) -> usize impl vortex_array::scalar_fn::ExecutionArgs for vortex_array::scalar_fn::VecExecutionArgs -pub fn vortex_array::scalar_fn::VecExecutionArgs::get(&self, index: usize) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::VecExecutionArgs::get(&self, usize) -> vortex_error::VortexResult pub fn vortex_array::scalar_fn::VecExecutionArgs::num_inputs(&self) -> usize @@ -17716,13 +18062,13 @@ pub fn vortex_array::scalar_fn::VecExecutionArgs::row_count(&self) -> usize pub trait vortex_array::scalar_fn::ReduceCtx -pub fn vortex_array::scalar_fn::ReduceCtx::new_node(&self, scalar_fn: vortex_array::scalar_fn::ScalarFnRef, children: &[vortex_array::scalar_fn::ReduceNodeRef]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::ReduceCtx::new_node(&self, vortex_array::scalar_fn::ScalarFnRef, &[vortex_array::scalar_fn::ReduceNodeRef]) -> vortex_error::VortexResult pub trait vortex_array::scalar_fn::ReduceNode pub fn vortex_array::scalar_fn::ReduceNode::as_any(&self) -> &dyn core::any::Any -pub fn vortex_array::scalar_fn::ReduceNode::child(&self, idx: usize) -> vortex_array::scalar_fn::ReduceNodeRef +pub fn vortex_array::scalar_fn::ReduceNode::child(&self, usize) -> vortex_array::scalar_fn::ReduceNodeRef pub fn vortex_array::scalar_fn::ReduceNode::child_count(&self) -> usize @@ -17734,7 +18080,7 @@ impl vortex_array::scalar_fn::ReduceNode for vortex_array::ArrayRef pub fn vortex_array::ArrayRef::as_any(&self) -> &dyn core::any::Any -pub fn vortex_array::ArrayRef::child(&self, idx: usize) -> vortex_array::scalar_fn::ReduceNodeRef +pub fn vortex_array::ArrayRef::child(&self, usize) -> vortex_array::scalar_fn::ReduceNodeRef pub fn vortex_array::ArrayRef::child_count(&self) -> usize @@ -17744,801 +18090,877 @@ pub fn vortex_array::ArrayRef::scalar_fn(&self) -> core::option::Option<&vortex_ pub trait vortex_array::scalar_fn::ScalarFnPlugin: 'static + core::marker::Send + core::marker::Sync -pub fn vortex_array::scalar_fn::ScalarFnPlugin::deserialize(&self, metadata: &[u8], session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::ScalarFnPlugin::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult pub fn vortex_array::scalar_fn::ScalarFnPlugin::id(&self) -> vortex_array::scalar_fn::ScalarFnId impl vortex_array::scalar_fn::ScalarFnPlugin for V -pub fn V::deserialize(&self, metadata: &[u8], session: &vortex_session::VortexSession) -> core::result::Result +pub fn V::deserialize(&self, &[u8], &vortex_session::VortexSession) -> core::result::Result -pub fn V::id(&self) -> arcref::ArcRef +pub fn V::id(&self) -> vortex_session::registry::Id pub trait vortex_array::scalar_fn::ScalarFnVTable: 'static + core::marker::Sized + core::clone::Clone + core::marker::Send + core::marker::Sync pub type vortex_array::scalar_fn::ScalarFnVTable::Options: 'static + core::marker::Send + core::marker::Sync + core::clone::Clone + core::fmt::Debug + core::fmt::Display + core::cmp::PartialEq + core::cmp::Eq + core::hash::Hash -pub fn vortex_array::scalar_fn::ScalarFnVTable::arity(&self, options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::ScalarFnVTable::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::ScalarFnVTable::child_name(&self, options: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::ScalarFnVTable::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::ScalarFnVTable::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::ScalarFnVTable::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::ScalarFnVTable::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::ScalarFnVTable::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::ScalarFnVTable::execute(&self, options: &Self::Options, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::ScalarFnVTable::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::ScalarFnVTable::fmt_sql(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::ScalarFnVTable::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::ScalarFnVTable::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::ScalarFnVTable::is_fallible(&self, options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::ScalarFnVTable::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::ScalarFnVTable::is_null_sensitive(&self, options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::ScalarFnVTable::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::ScalarFnVTable::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::ScalarFnVTable::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::ScalarFnVTable::return_dtype(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::ScalarFnVTable::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::ScalarFnVTable::serialize(&self, options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::ScalarFnVTable::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::ScalarFnVTable::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::ScalarFnVTable::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::ScalarFnVTable::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::ScalarFnVTable::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::ScalarFnVTable::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::ScalarFnVTable::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::ScalarFnVTable::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::ScalarFnVTable::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::ScalarFnVTable::validity(&self, options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::ScalarFnVTable::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::ForeignScalarFnVTable pub type vortex_array::scalar_fn::ForeignScalarFnVTable::Options = vortex_array::scalar_fn::ForeignScalarFnOptions -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::arity(&self, options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::child_name(&self, _options: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::deserialize(&self, metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::execute(&self, _options: &Self::Options, _args: &dyn vortex_array::scalar_fn::ExecutionArgs, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::fmt_sql(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::is_fallible(&self, options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::is_null_sensitive(&self, options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::return_dtype(&self, _options: &Self::Options, _args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::serialize(&self, options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::validity(&self, options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::ForeignScalarFnVTable::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::between::Between pub type vortex_array::scalar_fn::fns::between::Between::Options = vortex_array::scalar_fn::fns::between::BetweenOptions -pub fn vortex_array::scalar_fn::fns::between::Between::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::between::Between::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::between::Between::child_name(&self, _instance: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::between::Between::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::between::Between::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::between::Between::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::between::Between::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::between::Between::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::between::Between::execute(&self, options: &Self::Options, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::between::Between::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::between::Between::fmt_sql(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::between::Between::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::between::Between::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::between::Between::is_fallible(&self, _options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::between::Between::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::between::Between::is_null_sensitive(&self, _instance: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::between::Between::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::between::Between::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::between::Between::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::between::Between::return_dtype(&self, _options: &Self::Options, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::between::Between::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::between::Between::serialize(&self, instance: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::between::Between::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::between::Between::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::between::Between::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::between::Between::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::between::Between::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::between::Between::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::between::Between::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::between::Between::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::between::Between::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::between::Between::validity(&self, options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::between::Between::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::binary::Binary pub type vortex_array::scalar_fn::fns::binary::Binary::Options = vortex_array::scalar_fn::fns::operators::Operator -pub fn vortex_array::scalar_fn::fns::binary::Binary::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::binary::Binary::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::binary::Binary::child_name(&self, _instance: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::binary::Binary::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::binary::Binary::coerce_args(&self, operator: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::binary::Binary::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::binary::Binary::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::binary::Binary::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::binary::Binary::execute(&self, op: &vortex_array::scalar_fn::fns::operators::Operator, args: &dyn vortex_array::scalar_fn::ExecutionArgs, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::binary::Binary::execute(&self, &vortex_array::scalar_fn::fns::operators::Operator, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::binary::Binary::fmt_sql(&self, operator: &vortex_array::scalar_fn::fns::operators::Operator, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::binary::Binary::fmt_sql(&self, &vortex_array::scalar_fn::fns::operators::Operator, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::binary::Binary::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::binary::Binary::is_fallible(&self, operator: &vortex_array::scalar_fn::fns::operators::Operator) -> bool +pub fn vortex_array::scalar_fn::fns::binary::Binary::is_fallible(&self, &vortex_array::scalar_fn::fns::operators::Operator) -> bool -pub fn vortex_array::scalar_fn::fns::binary::Binary::is_null_sensitive(&self, _operator: &vortex_array::scalar_fn::fns::operators::Operator) -> bool +pub fn vortex_array::scalar_fn::fns::binary::Binary::is_null_sensitive(&self, &vortex_array::scalar_fn::fns::operators::Operator) -> bool -pub fn vortex_array::scalar_fn::fns::binary::Binary::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::binary::Binary::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::binary::Binary::return_dtype(&self, operator: &vortex_array::scalar_fn::fns::operators::Operator, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::binary::Binary::return_dtype(&self, &vortex_array::scalar_fn::fns::operators::Operator, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::binary::Binary::serialize(&self, instance: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::binary::Binary::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::binary::Binary::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::binary::Binary::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::binary::Binary::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::binary::Binary::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::binary::Binary::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::binary::Binary::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::binary::Binary::stat_falsification(&self, operator: &vortex_array::scalar_fn::fns::operators::Operator, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::binary::Binary::stat_falsification(&self, &vortex_array::scalar_fn::fns::operators::Operator, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::binary::Binary::validity(&self, operator: &vortex_array::scalar_fn::fns::operators::Operator, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::binary::Binary::validity(&self, &vortex_array::scalar_fn::fns::operators::Operator, &vortex_array::expr::Expression) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::case_when::CaseWhen pub type vortex_array::scalar_fn::fns::case_when::CaseWhen::Options = vortex_array::scalar_fn::fns::case_when::CaseWhenOptions -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::arity(&self, options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::child_name(&self, options: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::deserialize(&self, metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::execute(&self, options: &Self::Options, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::fmt_sql(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::is_fallible(&self, _options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::is_null_sensitive(&self, _options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::return_dtype(&self, options: &Self::Options, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::validity(&self, options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::case_when::CaseWhen::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::cast::Cast pub type vortex_array::scalar_fn::fns::cast::Cast::Options = vortex_array::dtype::DType -pub fn vortex_array::scalar_fn::fns::cast::Cast::arity(&self, _options: &vortex_array::dtype::DType) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::cast::Cast::arity(&self, &vortex_array::dtype::DType) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::cast::Cast::child_name(&self, _instance: &vortex_array::dtype::DType, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::cast::Cast::child_name(&self, &vortex_array::dtype::DType, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::cast::Cast::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::cast::Cast::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::cast::Cast::deserialize(&self, _metadata: &[u8], session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::cast::Cast::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::cast::Cast::execute(&self, target_dtype: &vortex_array::dtype::DType, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::cast::Cast::execute(&self, &vortex_array::dtype::DType, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::cast::Cast::fmt_sql(&self, dtype: &vortex_array::dtype::DType, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::cast::Cast::fmt_sql(&self, &vortex_array::dtype::DType, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::cast::Cast::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::cast::Cast::is_fallible(&self, options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::cast::Cast::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::cast::Cast::is_null_sensitive(&self, _instance: &vortex_array::dtype::DType) -> bool +pub fn vortex_array::scalar_fn::fns::cast::Cast::is_null_sensitive(&self, &vortex_array::dtype::DType) -> bool -pub fn vortex_array::scalar_fn::fns::cast::Cast::reduce(&self, target_dtype: &vortex_array::dtype::DType, node: &dyn vortex_array::scalar_fn::ReduceNode, _ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::cast::Cast::reduce(&self, &vortex_array::dtype::DType, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::cast::Cast::return_dtype(&self, dtype: &vortex_array::dtype::DType, _arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::cast::Cast::return_dtype(&self, &vortex_array::dtype::DType, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::cast::Cast::serialize(&self, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::cast::Cast::serialize(&self, &vortex_array::dtype::DType) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::cast::Cast::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::cast::Cast::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::cast::Cast::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::cast::Cast::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::cast::Cast::stat_expression(&self, dtype: &vortex_array::dtype::DType, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::cast::Cast::stat_expression(&self, &vortex_array::dtype::DType, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::cast::Cast::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::cast::Cast::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::cast::Cast::validity(&self, dtype: &vortex_array::dtype::DType, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::cast::Cast::validity(&self, &vortex_array::dtype::DType, &vortex_array::expr::Expression) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::dynamic::DynamicComparison pub type vortex_array::scalar_fn::fns::dynamic::DynamicComparison::Options = vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::child_name(&self, _instance: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::execute(&self, data: &Self::Options, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::fmt_sql(&self, dynamic: &vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::fmt_sql(&self, &vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::is_fallible(&self, options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::is_null_sensitive(&self, _instance: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::return_dtype(&self, dynamic: &vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::return_dtype(&self, &vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::serialize(&self, options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::stat_falsification(&self, dynamic: &vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::stat_falsification(&self, &vortex_array::scalar_fn::fns::dynamic::DynamicComparisonExpr, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::validity(&self, options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::dynamic::DynamicComparison::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::fill_null::FillNull pub type vortex_array::scalar_fn::fns::fill_null::FillNull::Options = vortex_array::scalar_fn::EmptyOptions -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::child_name(&self, _options: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::execute(&self, _options: &Self::Options, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::fmt_sql(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::is_fallible(&self, _options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::is_null_sensitive(&self, _options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::return_dtype(&self, _options: &Self::Options, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::simplify(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::validity(&self, _options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::fill_null::FillNull::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::get_item::GetItem pub type vortex_array::scalar_fn::fns::get_item::GetItem::Options = vortex_array::dtype::FieldName -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::arity(&self, _field_name: &vortex_array::dtype::FieldName) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::arity(&self, &vortex_array::dtype::FieldName) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::child_name(&self, _instance: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::execute(&self, field_name: &vortex_array::dtype::FieldName, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::execute(&self, &vortex_array::dtype::FieldName, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::fmt_sql(&self, field_name: &vortex_array::dtype::FieldName, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::fmt_sql(&self, &vortex_array::dtype::FieldName, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::get_item::GetItem::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::is_fallible(&self, _field_name: &vortex_array::dtype::FieldName) -> bool +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::is_fallible(&self, &vortex_array::dtype::FieldName) -> bool -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::is_null_sensitive(&self, _field_name: &vortex_array::dtype::FieldName) -> bool +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::is_null_sensitive(&self, &vortex_array::dtype::FieldName) -> bool -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::reduce(&self, field_name: &vortex_array::dtype::FieldName, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::reduce(&self, &vortex_array::dtype::FieldName, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::return_dtype(&self, field_name: &vortex_array::dtype::FieldName, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::return_dtype(&self, &vortex_array::dtype::FieldName, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::serialize(&self, instance: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::simplify_untyped(&self, field_name: &vortex_array::dtype::FieldName, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::simplify_untyped(&self, &vortex_array::dtype::FieldName, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::stat_expression(&self, field_name: &vortex_array::dtype::FieldName, _expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::stat_expression(&self, &vortex_array::dtype::FieldName, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::get_item::GetItem::validity(&self, options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::get_item::GetItem::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::is_not_null::IsNotNull + +pub type vortex_array::scalar_fn::fns::is_not_null::IsNotNull::Options = vortex_array::scalar_fn::EmptyOptions + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::id(&self) -> vortex_array::scalar_fn::ScalarFnId + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::is_fallible(&self, &Self::Options) -> bool + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::is_null_sensitive(&self, &Self::Options) -> bool + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option + +pub fn vortex_array::scalar_fn::fns::is_not_null::IsNotNull::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::is_null::IsNull pub type vortex_array::scalar_fn::fns::is_null::IsNull::Options = vortex_array::scalar_fn::EmptyOptions -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::child_name(&self, _instance: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::execute(&self, _data: &Self::Options, args: &dyn vortex_array::scalar_fn::ExecutionArgs, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::fmt_sql(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::is_null::IsNull::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::is_fallible(&self, _instance: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::is_null_sensitive(&self, _instance: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::return_dtype(&self, _options: &Self::Options, _arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::serialize(&self, _instance: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::stat_falsification(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::is_null::IsNull::validity(&self, options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::is_null::IsNull::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::like::Like pub type vortex_array::scalar_fn::fns::like::Like::Options = vortex_array::scalar_fn::fns::like::LikeOptions -pub fn vortex_array::scalar_fn::fns::like::Like::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::like::Like::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::like::Like::child_name(&self, _instance: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::like::Like::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::like::Like::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::like::Like::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::like::Like::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::like::Like::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::like::Like::execute(&self, options: &Self::Options, args: &dyn vortex_array::scalar_fn::ExecutionArgs, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::like::Like::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::like::Like::fmt_sql(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::like::Like::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::like::Like::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::like::Like::is_fallible(&self, options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::like::Like::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::like::Like::is_null_sensitive(&self, _instance: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::like::Like::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::like::Like::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::like::Like::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::like::Like::return_dtype(&self, _options: &Self::Options, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::like::Like::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::like::Like::serialize(&self, instance: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::like::Like::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::like::Like::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::like::Like::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::like::Like::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::like::Like::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::like::Like::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::like::Like::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::like::Like::stat_falsification(&self, like_opts: &vortex_array::scalar_fn::fns::like::LikeOptions, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::like::Like::stat_falsification(&self, &vortex_array::scalar_fn::fns::like::LikeOptions, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::like::Like::validity(&self, _options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::like::Like::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::list_contains::ListContains pub type vortex_array::scalar_fn::fns::list_contains::ListContains::Options = vortex_array::scalar_fn::EmptyOptions -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::child_name(&self, _instance: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::execute(&self, _options: &Self::Options, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::fmt_sql(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::is_fallible(&self, _options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::is_null_sensitive(&self, _instance: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::return_dtype(&self, _options: &Self::Options, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::serialize(&self, _instance: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::stat_falsification(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::validity(&self, options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::list_contains::ListContains::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::literal::Literal pub type vortex_array::scalar_fn::fns::literal::Literal::Options = vortex_array::scalar::Scalar -pub fn vortex_array::scalar_fn::fns::literal::Literal::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::literal::Literal::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::literal::Literal::child_name(&self, _instance: &Self::Options, _child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::literal::Literal::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::literal::Literal::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::literal::Literal::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::literal::Literal::deserialize(&self, _metadata: &[u8], session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::literal::Literal::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::literal::Literal::execute(&self, scalar: &vortex_array::scalar::Scalar, args: &dyn vortex_array::scalar_fn::ExecutionArgs, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::literal::Literal::execute(&self, &vortex_array::scalar::Scalar, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::literal::Literal::fmt_sql(&self, scalar: &vortex_array::scalar::Scalar, _expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::literal::Literal::fmt_sql(&self, &vortex_array::scalar::Scalar, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::literal::Literal::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::literal::Literal::is_fallible(&self, _instance: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::literal::Literal::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::literal::Literal::is_null_sensitive(&self, _instance: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::literal::Literal::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::literal::Literal::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::literal::Literal::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::literal::Literal::return_dtype(&self, options: &Self::Options, _arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::literal::Literal::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::literal::Literal::serialize(&self, instance: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::literal::Literal::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::literal::Literal::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::literal::Literal::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::literal::Literal::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::literal::Literal::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::literal::Literal::stat_expression(&self, scalar: &vortex_array::scalar::Scalar, _expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, _catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::literal::Literal::stat_expression(&self, &vortex_array::scalar::Scalar, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::literal::Literal::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::literal::Literal::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::literal::Literal::validity(&self, scalar: &vortex_array::scalar::Scalar, _expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::literal::Literal::validity(&self, &vortex_array::scalar::Scalar, &vortex_array::expr::Expression) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::mask::Mask pub type vortex_array::scalar_fn::fns::mask::Mask::Options = vortex_array::scalar_fn::EmptyOptions -pub fn vortex_array::scalar_fn::fns::mask::Mask::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::mask::Mask::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::mask::Mask::child_name(&self, _options: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::mask::Mask::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::mask::Mask::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::mask::Mask::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::mask::Mask::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::mask::Mask::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::mask::Mask::execute(&self, _options: &Self::Options, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::mask::Mask::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::mask::Mask::fmt_sql(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::mask::Mask::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::mask::Mask::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::mask::Mask::is_fallible(&self, options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::mask::Mask::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::mask::Mask::is_null_sensitive(&self, options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::mask::Mask::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::mask::Mask::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::mask::Mask::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::mask::Mask::return_dtype(&self, _options: &Self::Options, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::mask::Mask::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::mask::Mask::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::mask::Mask::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::mask::Mask::simplify(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::mask::Mask::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::mask::Mask::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::mask::Mask::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::mask::Mask::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::mask::Mask::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::mask::Mask::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::mask::Mask::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::mask::Mask::validity(&self, _options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::mask::Mask::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::merge::Merge pub type vortex_array::scalar_fn::fns::merge::Merge::Options = vortex_array::scalar_fn::fns::merge::DuplicateHandling -pub fn vortex_array::scalar_fn::fns::merge::Merge::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::merge::Merge::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::merge::Merge::child_name(&self, _instance: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::merge::Merge::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::merge::Merge::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::merge::Merge::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::merge::Merge::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::merge::Merge::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::merge::Merge::execute(&self, options: &Self::Options, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::merge::Merge::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::merge::Merge::fmt_sql(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::merge::Merge::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::merge::Merge::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::merge::Merge::is_fallible(&self, instance: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::merge::Merge::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::merge::Merge::is_null_sensitive(&self, _instance: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::merge::Merge::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::merge::Merge::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::merge::Merge::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::merge::Merge::return_dtype(&self, options: &Self::Options, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::merge::Merge::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::merge::Merge::serialize(&self, instance: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::merge::Merge::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::merge::Merge::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::merge::Merge::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::merge::Merge::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::merge::Merge::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::merge::Merge::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::merge::Merge::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::merge::Merge::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::merge::Merge::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::merge::Merge::validity(&self, _options: &Self::Options, _expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::merge::Merge::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::not::Not pub type vortex_array::scalar_fn::fns::not::Not::Options = vortex_array::scalar_fn::EmptyOptions -pub fn vortex_array::scalar_fn::fns::not::Not::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::not::Not::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::not::Not::child_name(&self, _options: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::not::Not::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::not::Not::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::not::Not::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::not::Not::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::not::Not::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::not::Not::execute(&self, _data: &Self::Options, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::not::Not::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::not::Not::fmt_sql(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::not::Not::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::not::Not::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::not::Not::is_fallible(&self, _options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::not::Not::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::not::Not::is_null_sensitive(&self, _options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::not::Not::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::not::Not::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::not::Not::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::not::Not::return_dtype(&self, _options: &Self::Options, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::not::Not::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::not::Not::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::not::Not::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::not::Not::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::not::Not::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::not::Not::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::not::Not::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::not::Not::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::not::Not::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::not::Not::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::not::Not::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::not::Not::validity(&self, options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::not::Not::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::pack::Pack pub type vortex_array::scalar_fn::fns::pack::Pack::Options = vortex_array::scalar_fn::fns::pack::PackOptions -pub fn vortex_array::scalar_fn::fns::pack::Pack::arity(&self, options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::pack::Pack::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::pack::Pack::child_name(&self, instance: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::pack::Pack::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::pack::Pack::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::pack::Pack::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::pack::Pack::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::pack::Pack::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::pack::Pack::execute(&self, options: &Self::Options, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::pack::Pack::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::pack::Pack::fmt_sql(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::pack::Pack::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::pack::Pack::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::pack::Pack::is_fallible(&self, _instance: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::pack::Pack::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::pack::Pack::is_null_sensitive(&self, _instance: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::pack::Pack::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::pack::Pack::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::pack::Pack::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::pack::Pack::return_dtype(&self, options: &Self::Options, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::pack::Pack::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::pack::Pack::serialize(&self, instance: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::pack::Pack::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::pack::Pack::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::pack::Pack::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::pack::Pack::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::pack::Pack::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::pack::Pack::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::pack::Pack::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::pack::Pack::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::pack::Pack::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::pack::Pack::validity(&self, _options: &Self::Options, _expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::pack::Pack::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::root::Root pub type vortex_array::scalar_fn::fns::root::Root::Options = vortex_array::scalar_fn::EmptyOptions -pub fn vortex_array::scalar_fn::fns::root::Root::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::root::Root::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::root::Root::child_name(&self, _instance: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::root::Root::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::root::Root::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::root::Root::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::root::Root::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::root::Root::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::root::Root::execute(&self, _data: &Self::Options, _args: &dyn vortex_array::scalar_fn::ExecutionArgs, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::root::Root::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::root::Root::fmt_sql(&self, _options: &Self::Options, _expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::root::Root::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::root::Root::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::root::Root::is_fallible(&self, _options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::root::Root::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::root::Root::is_null_sensitive(&self, _options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::root::Root::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::root::Root::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::root::Root::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::root::Root::return_dtype(&self, _options: &Self::Options, _arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::root::Root::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::root::Root::serialize(&self, _instance: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::root::Root::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::root::Root::simplify(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::root::Root::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::root::Root::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::root::Root::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::root::Root::stat_expression(&self, _options: &Self::Options, _expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::root::Root::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::root::Root::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::root::Root::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::root::Root::validity(&self, options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::root::Root::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::select::Select pub type vortex_array::scalar_fn::fns::select::Select::Options = vortex_array::scalar_fn::fns::select::FieldSelection -pub fn vortex_array::scalar_fn::fns::select::Select::arity(&self, _options: &vortex_array::scalar_fn::fns::select::FieldSelection) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::select::Select::arity(&self, &vortex_array::scalar_fn::fns::select::FieldSelection) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::select::Select::child_name(&self, _instance: &vortex_array::scalar_fn::fns::select::FieldSelection, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::select::Select::child_name(&self, &vortex_array::scalar_fn::fns::select::FieldSelection, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::select::Select::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::select::Select::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::select::Select::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::select::Select::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::select::Select::execute(&self, selection: &vortex_array::scalar_fn::fns::select::FieldSelection, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::select::Select::execute(&self, &vortex_array::scalar_fn::fns::select::FieldSelection, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::select::Select::fmt_sql(&self, selection: &vortex_array::scalar_fn::fns::select::FieldSelection, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::select::Select::fmt_sql(&self, &vortex_array::scalar_fn::fns::select::FieldSelection, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::select::Select::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::select::Select::is_fallible(&self, _instance: &vortex_array::scalar_fn::fns::select::FieldSelection) -> bool +pub fn vortex_array::scalar_fn::fns::select::Select::is_fallible(&self, &vortex_array::scalar_fn::fns::select::FieldSelection) -> bool -pub fn vortex_array::scalar_fn::fns::select::Select::is_null_sensitive(&self, _instance: &vortex_array::scalar_fn::fns::select::FieldSelection) -> bool +pub fn vortex_array::scalar_fn::fns::select::Select::is_null_sensitive(&self, &vortex_array::scalar_fn::fns::select::FieldSelection) -> bool -pub fn vortex_array::scalar_fn::fns::select::Select::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::select::Select::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::select::Select::return_dtype(&self, selection: &vortex_array::scalar_fn::fns::select::FieldSelection, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::select::Select::return_dtype(&self, &vortex_array::scalar_fn::fns::select::FieldSelection, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::select::Select::serialize(&self, instance: &vortex_array::scalar_fn::fns::select::FieldSelection) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::fns::select::Select::serialize(&self, &vortex_array::scalar_fn::fns::select::FieldSelection) -> vortex_error::VortexResult>> -pub fn vortex_array::scalar_fn::fns::select::Select::simplify(&self, selection: &vortex_array::scalar_fn::fns::select::FieldSelection, expr: &vortex_array::expr::Expression, ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::select::Select::simplify(&self, &vortex_array::scalar_fn::fns::select::FieldSelection, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::select::Select::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::select::Select::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::select::Select::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::select::Select::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::select::Select::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::fns::select::Select::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option -pub fn vortex_array::scalar_fn::fns::select::Select::validity(&self, options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::select::Select::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::fns::zip::Zip pub type vortex_array::scalar_fn::fns::zip::Zip::Options = vortex_array::scalar_fn::EmptyOptions -pub fn vortex_array::scalar_fn::fns::zip::Zip::arity(&self, _options: &Self::Options) -> vortex_array::scalar_fn::Arity +pub fn vortex_array::scalar_fn::fns::zip::Zip::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity -pub fn vortex_array::scalar_fn::fns::zip::Zip::child_name(&self, _options: &Self::Options, child_idx: usize) -> vortex_array::scalar_fn::ChildName +pub fn vortex_array::scalar_fn::fns::zip::Zip::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::zip::Zip::coerce_args(&self, options: &Self::Options, args: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::fns::zip::Zip::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::zip::Zip::deserialize(&self, _metadata: &[u8], _session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::zip::Zip::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::zip::Zip::execute(&self, _options: &Self::Options, args: &dyn vortex_array::scalar_fn::ExecutionArgs, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::fns::zip::Zip::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::zip::Zip::fmt_sql(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::scalar_fn::fns::zip::Zip::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::scalar_fn::fns::zip::Zip::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::zip::Zip::is_fallible(&self, _options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::fns::zip::Zip::is_fallible(&self, &Self::Options) -> bool + +pub fn vortex_array::scalar_fn::fns::zip::Zip::is_null_sensitive(&self, &Self::Options) -> bool + +pub fn vortex_array::scalar_fn::fns::zip::Zip::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> + +pub fn vortex_array::scalar_fn::fns::zip::Zip::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult + +pub fn vortex_array::scalar_fn::fns::zip::Zip::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> + +pub fn vortex_array::scalar_fn::fns::zip::Zip::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> + +pub fn vortex_array::scalar_fn::fns::zip::Zip::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> + +pub fn vortex_array::scalar_fn::fns::zip::Zip::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option + +pub fn vortex_array::scalar_fn::fns::zip::Zip::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option + +pub fn vortex_array::scalar_fn::fns::zip::Zip::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> + +impl vortex_array::scalar_fn::ScalarFnVTable for vortex_array::scalar_fn::internal::row_count::RowCount + +pub type vortex_array::scalar_fn::internal::row_count::RowCount::Options = vortex_array::scalar_fn::EmptyOptions + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::arity(&self, &Self::Options) -> vortex_array::scalar_fn::Arity + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::child_name(&self, &Self::Options, usize) -> vortex_array::scalar_fn::ChildName -pub fn vortex_array::scalar_fn::fns::zip::Zip::is_null_sensitive(&self, _options: &Self::Options) -> bool +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::coerce_args(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::zip::Zip::reduce(&self, options: &Self::Options, node: &dyn vortex_array::scalar_fn::ReduceNode, ctx: &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::deserialize(&self, &[u8], &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::zip::Zip::return_dtype(&self, _options: &Self::Options, arg_dtypes: &[vortex_array::dtype::DType]) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::execute(&self, &Self::Options, &dyn vortex_array::scalar_fn::ExecutionArgs, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::scalar_fn::fns::zip::Zip::serialize(&self, _options: &Self::Options) -> vortex_error::VortexResult>> +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::fmt_sql(&self, &Self::Options, &vortex_array::expr::Expression, &mut core::fmt::Formatter<'_>) -> core::fmt::Result -pub fn vortex_array::scalar_fn::fns::zip::Zip::simplify(&self, _options: &Self::Options, expr: &vortex_array::expr::Expression, _ctx: &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::id(&self) -> vortex_array::scalar_fn::ScalarFnId -pub fn vortex_array::scalar_fn::fns::zip::Zip::simplify_untyped(&self, options: &Self::Options, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::is_fallible(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::zip::Zip::stat_expression(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, stat: vortex_array::expr::stats::Stat, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::is_null_sensitive(&self, &Self::Options) -> bool -pub fn vortex_array::scalar_fn::fns::zip::Zip::stat_falsification(&self, options: &Self::Options, expr: &vortex_array::expr::Expression, catalog: &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::reduce(&self, &Self::Options, &dyn vortex_array::scalar_fn::ReduceNode, &dyn vortex_array::scalar_fn::ReduceCtx) -> vortex_error::VortexResult> -pub fn vortex_array::scalar_fn::fns::zip::Zip::validity(&self, options: &Self::Options, expression: &vortex_array::expr::Expression) -> vortex_error::VortexResult> +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::return_dtype(&self, &Self::Options, &[vortex_array::dtype::DType]) -> vortex_error::VortexResult + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::serialize(&self, &Self::Options) -> vortex_error::VortexResult>> + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::simplify(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::scalar_fn::SimplifyCtx) -> vortex_error::VortexResult> + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::simplify_untyped(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::stat_expression(&self, &Self::Options, &vortex_array::expr::Expression, vortex_array::expr::stats::Stat, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::stat_falsification(&self, &Self::Options, &vortex_array::expr::Expression, &dyn vortex_array::expr::pruning::StatsCatalog) -> core::option::Option + +pub fn vortex_array::scalar_fn::internal::row_count::RowCount::validity(&self, &Self::Options, &vortex_array::expr::Expression) -> vortex_error::VortexResult> pub trait vortex_array::scalar_fn::ScalarFnVTableExt: vortex_array::scalar_fn::ScalarFnVTable -pub fn vortex_array::scalar_fn::ScalarFnVTableExt::bind(&self, options: Self::Options) -> vortex_array::scalar_fn::ScalarFnRef +pub fn vortex_array::scalar_fn::ScalarFnVTableExt::bind(&self, Self::Options) -> vortex_array::scalar_fn::ScalarFnRef -pub fn vortex_array::scalar_fn::ScalarFnVTableExt::new_expr(&self, options: Self::Options, children: impl core::iter::traits::collect::IntoIterator) -> vortex_array::expr::Expression +pub fn vortex_array::scalar_fn::ScalarFnVTableExt::new_expr(&self, Self::Options, impl core::iter::traits::collect::IntoIterator) -> vortex_array::expr::Expression -pub fn vortex_array::scalar_fn::ScalarFnVTableExt::try_new_expr(&self, options: Self::Options, children: impl core::iter::traits::collect::IntoIterator) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::ScalarFnVTableExt::try_new_expr(&self, Self::Options, impl core::iter::traits::collect::IntoIterator) -> vortex_error::VortexResult impl vortex_array::scalar_fn::ScalarFnVTableExt for V -pub fn V::bind(&self, options: Self::Options) -> vortex_array::scalar_fn::ScalarFnRef +pub fn V::bind(&self, Self::Options) -> vortex_array::scalar_fn::ScalarFnRef -pub fn V::new_expr(&self, options: Self::Options, children: impl core::iter::traits::collect::IntoIterator) -> vortex_array::expr::Expression +pub fn V::new_expr(&self, Self::Options, impl core::iter::traits::collect::IntoIterator) -> vortex_array::expr::Expression -pub fn V::try_new_expr(&self, options: Self::Options, children: impl core::iter::traits::collect::IntoIterator) -> vortex_error::VortexResult +pub fn V::try_new_expr(&self, Self::Options, impl core::iter::traits::collect::IntoIterator) -> vortex_error::VortexResult pub trait vortex_array::scalar_fn::SimplifyCtx -pub fn vortex_array::scalar_fn::SimplifyCtx::return_dtype(&self, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult +pub fn vortex_array::scalar_fn::SimplifyCtx::return_dtype(&self, &vortex_array::expr::Expression) -> vortex_error::VortexResult pub type vortex_array::scalar_fn::ChildName = arcref::ArcRef pub type vortex_array::scalar_fn::ReduceNodeRef = alloc::sync::Arc -pub type vortex_array::scalar_fn::ScalarFnId = arcref::ArcRef +pub type vortex_array::scalar_fn::ScalarFnId = vortex_session::registry::Id pub type vortex_array::scalar_fn::ScalarFnPluginRef = alloc::sync::Arc @@ -18552,13 +18974,13 @@ pub vortex_array::search_sorted::SearchResult::NotFound(usize) impl vortex_array::search_sorted::SearchResult -pub fn vortex_array::search_sorted::SearchResult::to_ends_index(self, len: usize) -> usize +pub fn vortex_array::search_sorted::SearchResult::to_ends_index(self, usize) -> usize pub fn vortex_array::search_sorted::SearchResult::to_found(self) -> core::option::Option pub fn vortex_array::search_sorted::SearchResult::to_index(self) -> usize -pub fn vortex_array::search_sorted::SearchResult::to_offsets_index(self, len: usize, side: vortex_array::search_sorted::SearchSortedSide) -> usize +pub fn vortex_array::search_sorted::SearchResult::to_offsets_index(self, usize, vortex_array::search_sorted::SearchSortedSide) -> usize impl core::clone::Clone for vortex_array::search_sorted::SearchResult @@ -18568,15 +18990,15 @@ impl core::cmp::Eq for vortex_array::search_sorted::SearchResult impl core::cmp::PartialEq for vortex_array::search_sorted::SearchResult -pub fn vortex_array::search_sorted::SearchResult::eq(&self, other: &vortex_array::search_sorted::SearchResult) -> bool +pub fn vortex_array::search_sorted::SearchResult::eq(&self, &vortex_array::search_sorted::SearchResult) -> bool impl core::fmt::Debug for vortex_array::search_sorted::SearchResult -pub fn vortex_array::search_sorted::SearchResult::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::search_sorted::SearchResult::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::search_sorted::SearchResult -pub fn vortex_array::search_sorted::SearchResult::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::search_sorted::SearchResult::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_array::search_sorted::SearchResult @@ -18596,15 +19018,15 @@ impl core::cmp::Eq for vortex_array::search_sorted::SearchSortedSide impl core::cmp::PartialEq for vortex_array::search_sorted::SearchSortedSide -pub fn vortex_array::search_sorted::SearchSortedSide::eq(&self, other: &vortex_array::search_sorted::SearchSortedSide) -> bool +pub fn vortex_array::search_sorted::SearchSortedSide::eq(&self, &vortex_array::search_sorted::SearchSortedSide) -> bool impl core::fmt::Debug for vortex_array::search_sorted::SearchSortedSide -pub fn vortex_array::search_sorted::SearchSortedSide::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::search_sorted::SearchSortedSide::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::search_sorted::SearchSortedSide -pub fn vortex_array::search_sorted::SearchSortedSide::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::search_sorted::SearchSortedSide::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_array::search_sorted::SearchSortedSide @@ -18612,85 +19034,85 @@ impl core::marker::StructuralPartialEq for vortex_array::search_sorted::SearchSo pub trait vortex_array::search_sorted::IndexOrd -pub fn vortex_array::search_sorted::IndexOrd::index_cmp(&self, idx: usize, elem: &V) -> vortex_error::VortexResult> +pub fn vortex_array::search_sorted::IndexOrd::index_cmp(&self, usize, &V) -> vortex_error::VortexResult> -pub fn vortex_array::search_sorted::IndexOrd::index_ge(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::search_sorted::IndexOrd::index_ge(&self, usize, &V) -> vortex_error::VortexResult -pub fn vortex_array::search_sorted::IndexOrd::index_gt(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::search_sorted::IndexOrd::index_gt(&self, usize, &V) -> vortex_error::VortexResult -pub fn vortex_array::search_sorted::IndexOrd::index_le(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::search_sorted::IndexOrd::index_le(&self, usize, &V) -> vortex_error::VortexResult pub fn vortex_array::search_sorted::IndexOrd::index_len(&self) -> usize -pub fn vortex_array::search_sorted::IndexOrd::index_lt(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::search_sorted::IndexOrd::index_lt(&self, usize, &V) -> vortex_error::VortexResult impl vortex_array::search_sorted::IndexOrd> for vortex_array::variants::PrimitiveTyped<'_> -pub fn vortex_array::variants::PrimitiveTyped<'_>::index_cmp(&self, idx: usize, elem: &core::option::Option) -> vortex_error::VortexResult> +pub fn vortex_array::variants::PrimitiveTyped<'_>::index_cmp(&self, usize, &core::option::Option) -> vortex_error::VortexResult> -pub fn vortex_array::variants::PrimitiveTyped<'_>::index_ge(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::variants::PrimitiveTyped<'_>::index_ge(&self, usize, &V) -> vortex_error::VortexResult -pub fn vortex_array::variants::PrimitiveTyped<'_>::index_gt(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::variants::PrimitiveTyped<'_>::index_gt(&self, usize, &V) -> vortex_error::VortexResult -pub fn vortex_array::variants::PrimitiveTyped<'_>::index_le(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::variants::PrimitiveTyped<'_>::index_le(&self, usize, &V) -> vortex_error::VortexResult pub fn vortex_array::variants::PrimitiveTyped<'_>::index_len(&self) -> usize -pub fn vortex_array::variants::PrimitiveTyped<'_>::index_lt(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::variants::PrimitiveTyped<'_>::index_lt(&self, usize, &V) -> vortex_error::VortexResult impl vortex_array::search_sorted::IndexOrd for vortex_array::variants::PrimitiveTyped<'_> -pub fn vortex_array::variants::PrimitiveTyped<'_>::index_cmp(&self, idx: usize, elem: &vortex_array::scalar::PValue) -> vortex_error::VortexResult> +pub fn vortex_array::variants::PrimitiveTyped<'_>::index_cmp(&self, usize, &vortex_array::scalar::PValue) -> vortex_error::VortexResult> -pub fn vortex_array::variants::PrimitiveTyped<'_>::index_ge(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::variants::PrimitiveTyped<'_>::index_ge(&self, usize, &V) -> vortex_error::VortexResult -pub fn vortex_array::variants::PrimitiveTyped<'_>::index_gt(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::variants::PrimitiveTyped<'_>::index_gt(&self, usize, &V) -> vortex_error::VortexResult -pub fn vortex_array::variants::PrimitiveTyped<'_>::index_le(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::variants::PrimitiveTyped<'_>::index_le(&self, usize, &V) -> vortex_error::VortexResult pub fn vortex_array::variants::PrimitiveTyped<'_>::index_len(&self) -> usize -pub fn vortex_array::variants::PrimitiveTyped<'_>::index_lt(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::variants::PrimitiveTyped<'_>::index_lt(&self, usize, &V) -> vortex_error::VortexResult impl vortex_array::search_sorted::IndexOrd for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::index_cmp(&self, idx: usize, elem: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult> +pub fn vortex_array::ArrayRef::index_cmp(&self, usize, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult> -pub fn vortex_array::ArrayRef::index_ge(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::index_ge(&self, usize, &V) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::index_gt(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::index_gt(&self, usize, &V) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::index_le(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::index_le(&self, usize, &V) -> vortex_error::VortexResult pub fn vortex_array::ArrayRef::index_len(&self) -> usize -pub fn vortex_array::ArrayRef::index_lt(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::index_lt(&self, usize, &V) -> vortex_error::VortexResult impl vortex_array::search_sorted::IndexOrd for [T] -pub fn [T]::index_cmp(&self, idx: usize, elem: &T) -> vortex_error::VortexResult> +pub fn [T]::index_cmp(&self, usize, &T) -> vortex_error::VortexResult> -pub fn [T]::index_ge(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn [T]::index_ge(&self, usize, &V) -> vortex_error::VortexResult -pub fn [T]::index_gt(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn [T]::index_gt(&self, usize, &V) -> vortex_error::VortexResult -pub fn [T]::index_le(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn [T]::index_le(&self, usize, &V) -> vortex_error::VortexResult pub fn [T]::index_len(&self) -> usize -pub fn [T]::index_lt(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn [T]::index_lt(&self, usize, &V) -> vortex_error::VortexResult pub trait vortex_array::search_sorted::SearchSorted -pub fn vortex_array::search_sorted::SearchSorted::search_sorted(&self, value: &T, side: vortex_array::search_sorted::SearchSortedSide) -> vortex_error::VortexResult where Self: vortex_array::search_sorted::IndexOrd +pub fn vortex_array::search_sorted::SearchSorted::search_sorted(&self, &T, vortex_array::search_sorted::SearchSortedSide) -> vortex_error::VortexResult where Self: vortex_array::search_sorted::IndexOrd -pub fn vortex_array::search_sorted::SearchSorted::search_sorted_by vortex_error::VortexResult, N: core::ops::function::FnMut(usize) -> vortex_error::VortexResult>(&self, find: F, side_find: N, side: vortex_array::search_sorted::SearchSortedSide) -> vortex_error::VortexResult +pub fn vortex_array::search_sorted::SearchSorted::search_sorted_by vortex_error::VortexResult, N: core::ops::function::FnMut(usize) -> vortex_error::VortexResult>(&self, F, N, vortex_array::search_sorted::SearchSortedSide) -> vortex_error::VortexResult impl vortex_array::search_sorted::SearchSorted for S where S: vortex_array::search_sorted::IndexOrd + ?core::marker::Sized -pub fn S::search_sorted(&self, value: &T, side: vortex_array::search_sorted::SearchSortedSide) -> vortex_error::VortexResult where Self: vortex_array::search_sorted::IndexOrd +pub fn S::search_sorted(&self, &T, vortex_array::search_sorted::SearchSortedSide) -> vortex_error::VortexResult where Self: vortex_array::search_sorted::IndexOrd -pub fn S::search_sorted_by(&self, find: F, side_find: N, side: vortex_array::search_sorted::SearchSortedSide) -> core::result::Result where F: core::ops::function::FnMut(usize) -> core::result::Result, N: core::ops::function::FnMut(usize) -> core::result::Result +pub fn S::search_sorted_by(&self, F, N, vortex_array::search_sorted::SearchSortedSide) -> core::result::Result where F: core::ops::function::FnMut(usize) -> core::result::Result, N: core::ops::function::FnMut(usize) -> core::result::Result pub mod vortex_array::serde @@ -18698,9 +19120,9 @@ pub struct vortex_array::serde::ArrayNodeFlatBuffer<'a> impl<'a> vortex_array::serde::ArrayNodeFlatBuffer<'a> -pub fn vortex_array::serde::ArrayNodeFlatBuffer<'a>::try_new(ctx: &'a vortex_array::ArrayContext, session: &'a vortex_session::VortexSession, array: &'a vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::serde::ArrayNodeFlatBuffer<'a>::try_new(&'a vortex_array::ArrayContext, &'a vortex_session::VortexSession, &'a vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::serde::ArrayNodeFlatBuffer<'a>::try_write_flatbuffer<'fb>(&self, fbb: &mut flatbuffers::builder::FlatBufferBuilder<'fb>) -> vortex_error::VortexResult>> +pub fn vortex_array::serde::ArrayNodeFlatBuffer<'a>::try_write_flatbuffer<'fb>(&self, &mut flatbuffers::builder::FlatBufferBuilder<'fb>) -> vortex_error::VortexResult>> pub struct vortex_array::serde::SerializeOptions @@ -18714,29 +19136,29 @@ pub fn vortex_array::serde::SerializeOptions::default() -> vortex_array::serde:: impl core::fmt::Debug for vortex_array::serde::SerializeOptions -pub fn vortex_array::serde::SerializeOptions::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::serde::SerializeOptions::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_array::serde::SerializedArray impl vortex_array::serde::SerializedArray -pub fn vortex_array::serde::SerializedArray::buffer(&self, idx: usize) -> vortex_error::VortexResult +pub fn vortex_array::serde::SerializedArray::buffer(&self, usize) -> vortex_error::VortexResult pub fn vortex_array::serde::SerializedArray::buffer_lengths(&self) -> alloc::vec::Vec -pub fn vortex_array::serde::SerializedArray::child(&self, idx: usize) -> vortex_array::serde::SerializedArray +pub fn vortex_array::serde::SerializedArray::child(&self, usize) -> vortex_array::serde::SerializedArray -pub fn vortex_array::serde::SerializedArray::decode(&self, dtype: &vortex_array::dtype::DType, len: usize, ctx: &vortex_session::registry::ReadContext, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::serde::SerializedArray::decode(&self, &vortex_array::dtype::DType, usize, &vortex_session::registry::ReadContext, &vortex_session::VortexSession) -> vortex_error::VortexResult pub fn vortex_array::serde::SerializedArray::encoding_id(&self) -> u16 -pub fn vortex_array::serde::SerializedArray::from_array_tree(array_tree: impl core::convert::Into) -> vortex_error::VortexResult +pub fn vortex_array::serde::SerializedArray::from_array_tree(impl core::convert::Into) -> vortex_error::VortexResult -pub fn vortex_array::serde::SerializedArray::from_flatbuffer_and_segment(array_tree: vortex_buffer::ByteBuffer, segment: vortex_array::buffer::BufferHandle) -> vortex_error::VortexResult +pub fn vortex_array::serde::SerializedArray::from_flatbuffer_and_segment(vortex_buffer::ByteBuffer, vortex_array::buffer::BufferHandle) -> vortex_error::VortexResult -pub fn vortex_array::serde::SerializedArray::from_flatbuffer_and_segment_with_overrides(array_tree: vortex_buffer::ByteBuffer, segment: vortex_array::buffer::BufferHandle, buffer_overrides: &vortex_utils::aliases::hash_map::HashMap) -> vortex_error::VortexResult +pub fn vortex_array::serde::SerializedArray::from_flatbuffer_and_segment_with_overrides(vortex_buffer::ByteBuffer, vortex_array::buffer::BufferHandle, &vortex_utils::aliases::hash_map::HashMap) -> vortex_error::VortexResult -pub fn vortex_array::serde::SerializedArray::from_flatbuffer_with_buffers(array_tree: impl core::convert::Into, buffers: alloc::vec::Vec) -> vortex_error::VortexResult +pub fn vortex_array::serde::SerializedArray::from_flatbuffer_with_buffers(impl core::convert::Into, alloc::vec::Vec) -> vortex_error::VortexResult pub fn vortex_array::serde::SerializedArray::metadata(&self) -> &[u8] @@ -18752,21 +19174,21 @@ impl core::convert::TryFrom for vortex_array pub type vortex_array::serde::SerializedArray::Error = vortex_error::VortexError -pub fn vortex_array::serde::SerializedArray::try_from(value: vortex_array::buffer::BufferHandle) -> core::result::Result +pub fn vortex_array::serde::SerializedArray::try_from(vortex_array::buffer::BufferHandle) -> core::result::Result impl core::convert::TryFrom> for vortex_array::serde::SerializedArray pub type vortex_array::serde::SerializedArray::Error = vortex_error::VortexError -pub fn vortex_array::serde::SerializedArray::try_from(value: vortex_buffer::ByteBuffer) -> core::result::Result +pub fn vortex_array::serde::SerializedArray::try_from(vortex_buffer::ByteBuffer) -> core::result::Result impl core::fmt::Debug for vortex_array::serde::SerializedArray -pub fn vortex_array::serde::SerializedArray::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::serde::SerializedArray::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub trait vortex_array::serde::ArrayChildren -pub fn vortex_array::serde::ArrayChildren::get(&self, index: usize, dtype: &vortex_array::dtype::DType, len: usize) -> vortex_error::VortexResult +pub fn vortex_array::serde::ArrayChildren::get(&self, usize, &vortex_array::dtype::DType, usize) -> vortex_error::VortexResult pub fn vortex_array::serde::ArrayChildren::is_empty(&self) -> bool @@ -18774,7 +19196,7 @@ pub fn vortex_array::serde::ArrayChildren::len(&self) -> usize impl> vortex_array::serde::ArrayChildren for T -pub fn T::get(&self, index: usize, dtype: &vortex_array::dtype::DType, len: usize) -> vortex_error::VortexResult +pub fn T::get(&self, usize, &vortex_array::dtype::DType, usize) -> vortex_error::VortexResult pub fn T::is_empty(&self) -> bool @@ -18788,7 +19210,7 @@ impl vortex_array::session::ArraySession pub fn vortex_array::session::ArraySession::empty() -> vortex_array::session::ArraySession -pub fn vortex_array::session::ArraySession::register(&self, plugin: P) +pub fn vortex_array::session::ArraySession::register(&self, P) pub fn vortex_array::session::ArraySession::registry(&self) -> &vortex_array::session::ArrayRegistry @@ -18798,17 +19220,23 @@ pub fn vortex_array::session::ArraySession::default() -> Self impl core::fmt::Debug for vortex_array::session::ArraySession -pub fn vortex_array::session::ArraySession::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::session::ArraySession::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result + +impl vortex_session::SessionVar for vortex_array::session::ArraySession + +pub fn vortex_array::session::ArraySession::as_any(&self) -> &dyn core::any::Any + +pub fn vortex_array::session::ArraySession::as_any_mut(&mut self) -> &mut dyn core::any::Any pub trait vortex_array::session::ArraySessionExt: vortex_session::SessionExt -pub fn vortex_array::session::ArraySessionExt::array_serialize(&self, array: &vortex_array::ArrayRef) -> vortex_error::VortexResult>> +pub fn vortex_array::session::ArraySessionExt::array_serialize(&self, &vortex_array::ArrayRef) -> vortex_error::VortexResult>> pub fn vortex_array::session::ArraySessionExt::arrays(&self) -> vortex_session::Ref<'_, vortex_array::session::ArraySession> impl vortex_array::session::ArraySessionExt for S -pub fn S::array_serialize(&self, array: &vortex_array::ArrayRef) -> vortex_error::VortexResult>> +pub fn S::array_serialize(&self, &vortex_array::ArrayRef) -> vortex_error::VortexResult>> pub fn S::arrays(&self) -> vortex_session::Ref<'_, vortex_array::session::ArraySession> @@ -18822,13 +19250,13 @@ pub struct vortex_array::stats::ArrayStats impl vortex_array::stats::ArrayStats -pub fn vortex_array::stats::ArrayStats::clear(&self, stat: vortex_array::expr::stats::Stat) +pub fn vortex_array::stats::ArrayStats::clear(&self, vortex_array::expr::stats::Stat) -pub fn vortex_array::stats::ArrayStats::retain(&self, stats: &[vortex_array::expr::stats::Stat]) +pub fn vortex_array::stats::ArrayStats::retain(&self, &[vortex_array::expr::stats::Stat]) -pub fn vortex_array::stats::ArrayStats::set(&self, stat: vortex_array::expr::stats::Stat, value: vortex_array::expr::stats::Precision) +pub fn vortex_array::stats::ArrayStats::set(&self, vortex_array::expr::stats::Stat, vortex_array::expr::stats::Precision) -pub fn vortex_array::stats::ArrayStats::to_ref<'a>(&'a self, array: &'a vortex_array::ArrayRef) -> vortex_array::stats::StatsSetRef<'a> +pub fn vortex_array::stats::ArrayStats::to_ref<'a>(&'a self, &'a vortex_array::ArrayRef) -> vortex_array::stats::StatsSetRef<'a> impl core::clone::Clone for vortex_array::stats::ArrayStats @@ -18836,11 +19264,11 @@ pub fn vortex_array::stats::ArrayStats::clone(&self) -> vortex_array::stats::Arr impl core::convert::From for vortex_array::stats::StatsSet -pub fn vortex_array::stats::StatsSet::from(value: vortex_array::stats::ArrayStats) -> Self +pub fn vortex_array::stats::StatsSet::from(vortex_array::stats::ArrayStats) -> Self impl core::convert::From for vortex_array::stats::ArrayStats -pub fn vortex_array::stats::ArrayStats::from(value: vortex_array::stats::StatsSet) -> Self +pub fn vortex_array::stats::ArrayStats::from(vortex_array::stats::StatsSet) -> Self impl core::default::Default for vortex_array::stats::ArrayStats @@ -18848,7 +19276,7 @@ pub fn vortex_array::stats::ArrayStats::default() -> vortex_array::stats::ArrayS impl core::fmt::Debug for vortex_array::stats::ArrayStats -pub fn vortex_array::stats::ArrayStats::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::stats::ArrayStats::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_array::stats::MutTypedStatsSetRef<'a, 'b> @@ -18858,21 +19286,21 @@ pub vortex_array::stats::MutTypedStatsSetRef::values: &'a mut vortex_array::stat impl vortex_array::stats::MutTypedStatsSetRef<'_, '_> -pub fn vortex_array::stats::MutTypedStatsSetRef<'_, '_>::clear(&mut self, stat: vortex_array::expr::stats::Stat) +pub fn vortex_array::stats::MutTypedStatsSetRef<'_, '_>::clear(&mut self, vortex_array::expr::stats::Stat) -pub fn vortex_array::stats::MutTypedStatsSetRef<'_, '_>::set(&mut self, stat: vortex_array::expr::stats::Stat, value: vortex_array::expr::stats::Precision) +pub fn vortex_array::stats::MutTypedStatsSetRef<'_, '_>::set(&mut self, vortex_array::expr::stats::Stat, vortex_array::expr::stats::Precision) impl vortex_array::stats::MutTypedStatsSetRef<'_, '_> -pub fn vortex_array::stats::MutTypedStatsSetRef<'_, '_>::combine_sets(&mut self, other: &vortex_array::stats::TypedStatsSetRef<'_, '_>) -> vortex_error::VortexResult<()> +pub fn vortex_array::stats::MutTypedStatsSetRef<'_, '_>::combine_sets(&mut self, &vortex_array::stats::TypedStatsSetRef<'_, '_>) -> vortex_error::VortexResult<()> -pub fn vortex_array::stats::MutTypedStatsSetRef<'_, '_>::merge_ordered(self, other: &vortex_array::stats::TypedStatsSetRef<'_, '_>) -> Self +pub fn vortex_array::stats::MutTypedStatsSetRef<'_, '_>::merge_ordered(self, &vortex_array::stats::TypedStatsSetRef<'_, '_>) -> Self -pub fn vortex_array::stats::MutTypedStatsSetRef<'_, '_>::merge_unordered(self, other: &vortex_array::stats::TypedStatsSetRef<'_, '_>) -> Self +pub fn vortex_array::stats::MutTypedStatsSetRef<'_, '_>::merge_unordered(self, &vortex_array::stats::TypedStatsSetRef<'_, '_>) -> Self impl vortex_array::expr::stats::StatsProvider for vortex_array::stats::MutTypedStatsSetRef<'_, '_> -pub fn vortex_array::stats::MutTypedStatsSetRef<'_, '_>::get(&self, stat: vortex_array::expr::stats::Stat) -> core::option::Option> +pub fn vortex_array::stats::MutTypedStatsSetRef<'_, '_>::get(&self, vortex_array::expr::stats::Stat) -> core::option::Option> pub fn vortex_array::stats::MutTypedStatsSetRef<'_, '_>::is_empty(&self) -> bool @@ -18882,21 +19310,21 @@ pub struct vortex_array::stats::StatsSet impl vortex_array::stats::StatsSet -pub fn vortex_array::stats::StatsSet::as_mut_typed_ref<'a, 'b>(&'a mut self, dtype: &'b vortex_array::dtype::DType) -> vortex_array::stats::MutTypedStatsSetRef<'a, 'b> +pub fn vortex_array::stats::StatsSet::as_mut_typed_ref<'a, 'b>(&'a mut self, &'b vortex_array::dtype::DType) -> vortex_array::stats::MutTypedStatsSetRef<'a, 'b> -pub fn vortex_array::stats::StatsSet::as_typed_ref<'a, 'b>(&'a self, dtype: &'b vortex_array::dtype::DType) -> vortex_array::stats::TypedStatsSetRef<'a, 'b> +pub fn vortex_array::stats::StatsSet::as_typed_ref<'a, 'b>(&'a self, &'b vortex_array::dtype::DType) -> vortex_array::stats::TypedStatsSetRef<'a, 'b> -pub unsafe fn vortex_array::stats::StatsSet::new_unchecked(values: alloc::vec::Vec<(vortex_array::expr::stats::Stat, vortex_array::expr::stats::Precision)>) -> Self +pub unsafe fn vortex_array::stats::StatsSet::new_unchecked(alloc::vec::Vec<(vortex_array::expr::stats::Stat, vortex_array::expr::stats::Precision)>) -> Self -pub fn vortex_array::stats::StatsSet::of(stat: vortex_array::expr::stats::Stat, value: vortex_array::expr::stats::Precision) -> Self +pub fn vortex_array::stats::StatsSet::of(vortex_array::expr::stats::Stat, vortex_array::expr::stats::Precision) -> Self impl vortex_array::stats::StatsSet -pub fn vortex_array::stats::StatsSet::clear(&mut self, stat: vortex_array::expr::stats::Stat) +pub fn vortex_array::stats::StatsSet::clear(&mut self, vortex_array::expr::stats::Stat) -pub fn vortex_array::stats::StatsSet::get(&self, stat: vortex_array::expr::stats::Stat) -> core::option::Option> +pub fn vortex_array::stats::StatsSet::get(&self, vortex_array::expr::stats::Stat) -> core::option::Option> -pub fn vortex_array::stats::StatsSet::get_as core::convert::TryFrom<&'a vortex_array::scalar::Scalar, Error = vortex_error::VortexError>>(&self, stat: vortex_array::expr::stats::Stat, dtype: &vortex_array::dtype::DType) -> core::option::Option> +pub fn vortex_array::stats::StatsSet::get_as core::convert::TryFrom<&'a vortex_array::scalar::Scalar, Error = vortex_error::VortexError>>(&self, vortex_array::expr::stats::Stat, &vortex_array::dtype::DType) -> core::option::Option> pub fn vortex_array::stats::StatsSet::is_empty(&self) -> bool @@ -18904,21 +19332,21 @@ pub fn vortex_array::stats::StatsSet::iter(&self) -> impl core::iter::traits::it pub fn vortex_array::stats::StatsSet::len(&self) -> usize -pub fn vortex_array::stats::StatsSet::retain_only(&mut self, stats: &[vortex_array::expr::stats::Stat]) +pub fn vortex_array::stats::StatsSet::retain_only(&mut self, &[vortex_array::expr::stats::Stat]) -pub fn vortex_array::stats::StatsSet::set(&mut self, stat: vortex_array::expr::stats::Stat, value: vortex_array::expr::stats::Precision) +pub fn vortex_array::stats::StatsSet::set(&mut self, vortex_array::expr::stats::Stat, vortex_array::expr::stats::Precision) impl vortex_array::stats::StatsSet -pub fn vortex_array::stats::StatsSet::combine_sets(&mut self, other: &Self, dtype: &vortex_array::dtype::DType) -> vortex_error::VortexResult<()> +pub fn vortex_array::stats::StatsSet::combine_sets(&mut self, &Self, &vortex_array::dtype::DType) -> vortex_error::VortexResult<()> -pub fn vortex_array::stats::StatsSet::merge_ordered(self, other: &Self, dtype: &vortex_array::dtype::DType) -> Self +pub fn vortex_array::stats::StatsSet::merge_ordered(self, &Self, &vortex_array::dtype::DType) -> Self -pub fn vortex_array::stats::StatsSet::merge_unordered(self, other: &Self, dtype: &vortex_array::dtype::DType) -> Self +pub fn vortex_array::stats::StatsSet::merge_unordered(self, &Self, &vortex_array::dtype::DType) -> Self impl vortex_array::stats::StatsSet -pub fn vortex_array::stats::StatsSet::from_flatbuffer<'a>(fb: &vortex_flatbuffers::array::ArrayStats<'a>, array_dtype: &vortex_array::dtype::DType, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::stats::StatsSet::from_flatbuffer<'a>(&vortex_flatbuffers::array::ArrayStats<'a>, &vortex_array::dtype::DType, &vortex_session::VortexSession) -> vortex_error::VortexResult impl core::clone::Clone for vortex_array::stats::StatsSet @@ -18926,11 +19354,11 @@ pub fn vortex_array::stats::StatsSet::clone(&self) -> vortex_array::stats::Stats impl core::convert::From for vortex_array::stats::StatsSet -pub fn vortex_array::stats::StatsSet::from(value: vortex_array::stats::ArrayStats) -> Self +pub fn vortex_array::stats::StatsSet::from(vortex_array::stats::ArrayStats) -> Self impl core::convert::From for vortex_array::stats::ArrayStats -pub fn vortex_array::stats::ArrayStats::from(value: vortex_array::stats::StatsSet) -> Self +pub fn vortex_array::stats::ArrayStats::from(vortex_array::stats::StatsSet) -> Self impl core::default::Default for vortex_array::stats::StatsSet @@ -18938,15 +19366,15 @@ pub fn vortex_array::stats::StatsSet::default() -> vortex_array::stats::StatsSet impl core::fmt::Debug for vortex_array::stats::StatsSet -pub fn vortex_array::stats::StatsSet::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::stats::StatsSet::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::iter::traits::collect::Extend<(vortex_array::expr::stats::Stat, vortex_array::expr::stats::Precision)> for vortex_array::stats::StatsSet -pub fn vortex_array::stats::StatsSet::extend)>>(&mut self, iter: T) +pub fn vortex_array::stats::StatsSet::extend)>>(&mut self, T) impl core::iter::traits::collect::FromIterator<(vortex_array::expr::stats::Stat, vortex_array::expr::stats::Precision)> for vortex_array::stats::StatsSet -pub fn vortex_array::stats::StatsSet::from_iter)>>(iter: T) -> Self +pub fn vortex_array::stats::StatsSet::from_iter)>>(T) -> Self impl core::iter::traits::collect::IntoIterator for vortex_array::stats::StatsSet @@ -18960,7 +19388,7 @@ impl vortex_flatbuffers::WriteFlatBuffer for vortex_array::stats::StatsSet pub type vortex_array::stats::StatsSet::Target<'t> = vortex_flatbuffers::array::ArrayStats<'t> -pub fn vortex_array::stats::StatsSet::write_flatbuffer<'fb>(&self, fbb: &mut flatbuffers::builder::FlatBufferBuilder<'fb>) -> vortex_error::VortexResult> +pub fn vortex_array::stats::StatsSet::write_flatbuffer<'fb>(&self, &mut flatbuffers::builder::FlatBufferBuilder<'fb>) -> vortex_error::VortexResult> pub struct vortex_array::stats::StatsSetIntoIter(_) @@ -18974,51 +19402,51 @@ pub struct vortex_array::stats::StatsSetRef<'a> impl vortex_array::stats::StatsSetRef<'_> -pub fn vortex_array::stats::StatsSetRef<'_>::clear(&self, stat: vortex_array::expr::stats::Stat) +pub fn vortex_array::stats::StatsSetRef<'_>::clear(&self, vortex_array::expr::stats::Stat) -pub fn vortex_array::stats::StatsSetRef<'_>::compute_as core::convert::TryFrom<&'a vortex_array::scalar::Scalar, Error = vortex_error::VortexError>>(&self, stat: vortex_array::expr::stats::Stat) -> core::option::Option +pub fn vortex_array::stats::StatsSetRef<'_>::compute_as core::convert::TryFrom<&'a vortex_array::scalar::Scalar, Error = vortex_error::VortexError>>(&self, vortex_array::expr::stats::Stat, &mut vortex_array::ExecutionCtx) -> core::option::Option -pub fn vortex_array::stats::StatsSetRef<'_>::compute_is_constant(&self) -> core::option::Option +pub fn vortex_array::stats::StatsSetRef<'_>::compute_is_constant(&self, &mut vortex_array::ExecutionCtx) -> core::option::Option -pub fn vortex_array::stats::StatsSetRef<'_>::compute_is_sorted(&self) -> core::option::Option +pub fn vortex_array::stats::StatsSetRef<'_>::compute_is_sorted(&self, &mut vortex_array::ExecutionCtx) -> core::option::Option -pub fn vortex_array::stats::StatsSetRef<'_>::compute_is_strict_sorted(&self) -> core::option::Option +pub fn vortex_array::stats::StatsSetRef<'_>::compute_is_strict_sorted(&self, &mut vortex_array::ExecutionCtx) -> core::option::Option -pub fn vortex_array::stats::StatsSetRef<'_>::compute_max core::convert::TryFrom<&'a vortex_array::scalar::Scalar, Error = vortex_error::VortexError>>(&self) -> core::option::Option +pub fn vortex_array::stats::StatsSetRef<'_>::compute_max core::convert::TryFrom<&'a vortex_array::scalar::Scalar, Error = vortex_error::VortexError>>(&self, &mut vortex_array::ExecutionCtx) -> core::option::Option -pub fn vortex_array::stats::StatsSetRef<'_>::compute_min core::convert::TryFrom<&'a vortex_array::scalar::Scalar, Error = vortex_error::VortexError>>(&self) -> core::option::Option +pub fn vortex_array::stats::StatsSetRef<'_>::compute_min core::convert::TryFrom<&'a vortex_array::scalar::Scalar, Error = vortex_error::VortexError>>(&self, &mut vortex_array::ExecutionCtx) -> core::option::Option -pub fn vortex_array::stats::StatsSetRef<'_>::compute_null_count(&self) -> core::option::Option +pub fn vortex_array::stats::StatsSetRef<'_>::compute_null_count(&self, &mut vortex_array::ExecutionCtx) -> core::option::Option -pub fn vortex_array::stats::StatsSetRef<'_>::compute_uncompressed_size_in_bytes(&self) -> core::option::Option +pub fn vortex_array::stats::StatsSetRef<'_>::compute_uncompressed_size_in_bytes(&self, &mut vortex_array::ExecutionCtx) -> core::option::Option -pub fn vortex_array::stats::StatsSetRef<'_>::set(&self, stat: vortex_array::expr::stats::Stat, value: vortex_array::expr::stats::Precision) +pub fn vortex_array::stats::StatsSetRef<'_>::set(&self, vortex_array::expr::stats::Stat, vortex_array::expr::stats::Precision) impl vortex_array::stats::StatsSetRef<'_> -pub fn vortex_array::stats::StatsSetRef<'_>::compute_all(&self, stats: &[vortex_array::expr::stats::Stat]) -> vortex_error::VortexResult +pub fn vortex_array::stats::StatsSetRef<'_>::compute_all(&self, &[vortex_array::expr::stats::Stat], &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::stats::StatsSetRef<'_>::compute_stat(&self, stat: vortex_array::expr::stats::Stat) -> vortex_error::VortexResult> +pub fn vortex_array::stats::StatsSetRef<'_>::compute_stat(&self, vortex_array::expr::stats::Stat, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> -pub fn vortex_array::stats::StatsSetRef<'_>::inherit<'a>(&self, iter: impl core::iter::traits::iterator::Iterator)>) +pub fn vortex_array::stats::StatsSetRef<'_>::inherit<'a>(&self, impl core::iter::traits::iterator::Iterator)>) -pub fn vortex_array::stats::StatsSetRef<'_>::inherit_from(&self, stats: vortex_array::stats::StatsSetRef<'_>) +pub fn vortex_array::stats::StatsSetRef<'_>::inherit_from(&self, vortex_array::stats::StatsSetRef<'_>) -pub fn vortex_array::stats::StatsSetRef<'_>::set_iter(&self, iter: vortex_array::stats::StatsSetIntoIter) +pub fn vortex_array::stats::StatsSetRef<'_>::set_iter(&self, vortex_array::stats::StatsSetIntoIter) pub fn vortex_array::stats::StatsSetRef<'_>::to_array_stats(&self) -> vortex_array::stats::ArrayStats pub fn vortex_array::stats::StatsSetRef<'_>::to_owned(&self) -> vortex_array::stats::StatsSet -pub fn vortex_array::stats::StatsSetRef<'_>::with_iter core::ops::function::FnOnce(&mut dyn core::iter::traits::iterator::Iterator)>) -> R, R>(&self, f: F) -> R +pub fn vortex_array::stats::StatsSetRef<'_>::with_iter core::ops::function::FnOnce(&mut dyn core::iter::traits::iterator::Iterator)>) -> R, R>(&self, F) -> R -pub fn vortex_array::stats::StatsSetRef<'_>::with_mut_typed_stats_set) -> U>(&self, apply: F) -> U +pub fn vortex_array::stats::StatsSetRef<'_>::with_mut_typed_stats_set) -> U>(&self, F) -> U -pub fn vortex_array::stats::StatsSetRef<'_>::with_typed_stats_set) -> U>(&self, apply: F) -> U +pub fn vortex_array::stats::StatsSetRef<'_>::with_typed_stats_set) -> U>(&self, F) -> U impl vortex_array::expr::stats::StatsProvider for vortex_array::stats::StatsSetRef<'_> -pub fn vortex_array::stats::StatsSetRef<'_>::get(&self, stat: vortex_array::expr::stats::Stat) -> core::option::Option> +pub fn vortex_array::stats::StatsSetRef<'_>::get(&self, vortex_array::expr::stats::Stat) -> core::option::Option> pub fn vortex_array::stats::StatsSetRef<'_>::is_empty(&self) -> bool @@ -19028,7 +19456,7 @@ impl vortex_flatbuffers::WriteFlatBuffer for vortex_array::stats::StatsSetRef<'_ pub type vortex_array::stats::StatsSetRef<'_>::Target<'t> = vortex_flatbuffers::array::ArrayStats<'t> -pub fn vortex_array::stats::StatsSetRef<'_>::write_flatbuffer<'fb>(&self, fbb: &mut flatbuffers::builder::FlatBufferBuilder<'fb>) -> vortex_error::VortexResult> +pub fn vortex_array::stats::StatsSetRef<'_>::write_flatbuffer<'fb>(&self, &mut flatbuffers::builder::FlatBufferBuilder<'fb>) -> vortex_error::VortexResult> pub struct vortex_array::stats::TypedStatsSetRef<'a, 'b> @@ -19038,7 +19466,7 @@ pub vortex_array::stats::TypedStatsSetRef::values: &'a vortex_array::stats::Stat impl vortex_array::expr::stats::StatsProvider for vortex_array::stats::TypedStatsSetRef<'_, '_> -pub fn vortex_array::stats::TypedStatsSetRef<'_, '_>::get(&self, stat: vortex_array::expr::stats::Stat) -> core::option::Option> +pub fn vortex_array::stats::TypedStatsSetRef<'_, '_>::get(&self, vortex_array::expr::stats::Stat) -> core::option::Option> pub fn vortex_array::stats::TypedStatsSetRef<'_, '_>::is_empty(&self) -> bool @@ -19046,9 +19474,9 @@ pub fn vortex_array::stats::TypedStatsSetRef<'_, '_>::len(&self) -> usize pub const vortex_array::stats::PRUNING_STATS: &[vortex_array::expr::stats::Stat] -pub fn vortex_array::stats::as_stat_bitset_bytes(stats: &[vortex_array::expr::stats::Stat]) -> alloc::vec::Vec +pub fn vortex_array::stats::as_stat_bitset_bytes(&[vortex_array::expr::stats::Stat]) -> alloc::vec::Vec -pub fn vortex_array::stats::stats_from_bitset_bytes(bytes: &[u8]) -> alloc::vec::Vec +pub fn vortex_array::stats::stats_from_bitset_bytes(&[u8]) -> alloc::vec::Vec pub mod vortex_array::stream @@ -19056,7 +19484,7 @@ pub struct vortex_array::stream::ArrayStreamAdapter impl vortex_array::stream::ArrayStreamAdapter where S: futures_core::stream::Stream> -pub fn vortex_array::stream::ArrayStreamAdapter::new(dtype: vortex_array::dtype::DType, inner: S) -> Self +pub fn vortex_array::stream::ArrayStreamAdapter::new(vortex_array::dtype::DType, S) -> Self impl<'__pin, S> core::marker::Unpin for vortex_array::stream::ArrayStreamAdapter where pin_project_lite::__private::PinnedFieldsOf<__Origin<'__pin, S>>: core::marker::Unpin @@ -19064,7 +19492,7 @@ impl futures_core::stream::Stream for vortex_array::stream::ArrayStreamAdapte pub type vortex_array::stream::ArrayStreamAdapter::Item = core::result::Result -pub fn vortex_array::stream::ArrayStreamAdapter::poll_next(self: core::pin::Pin<&mut Self>, cx: &mut core::task::wake::Context<'_>) -> core::task::poll::Poll> +pub fn vortex_array::stream::ArrayStreamAdapter::poll_next(core::pin::Pin<&mut Self>, &mut core::task::wake::Context<'_>) -> core::task::poll::Poll> pub fn vortex_array::stream::ArrayStreamAdapter::size_hint(&self) -> (usize, core::option::Option) @@ -19114,59 +19542,65 @@ impl vortex_array::validity::Validity pub const vortex_array::validity::Validity::DTYPE: vortex_array::dtype::DType -pub fn vortex_array::validity::Validity::and(self, rhs: vortex_array::validity::Validity) -> vortex_error::VortexResult +pub fn vortex_array::validity::Validity::and(self, vortex_array::validity::Validity) -> vortex_error::VortexResult pub fn vortex_array::validity::Validity::as_array(&self) -> core::option::Option<&vortex_array::ArrayRef> -pub fn vortex_array::validity::Validity::cast_nullability(self, nullability: vortex_array::dtype::Nullability, len: usize) -> vortex_error::VortexResult - -pub fn vortex_array::validity::Validity::copy_from_array(array: &vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::validity::Validity::cast_nullability(self, vortex_array::dtype::Nullability, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::validity::Validity::execute_mask(&self, length: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::validity::Validity::execute_mask(&self, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::validity::Validity::filter(&self, mask: &vortex_mask::Mask) -> vortex_error::VortexResult +pub fn vortex_array::validity::Validity::filter(&self, &vortex_mask::Mask) -> vortex_error::VortexResult pub fn vortex_array::validity::Validity::into_array(self) -> core::option::Option -pub fn vortex_array::validity::Validity::into_non_nullable(self, len: usize) -> core::option::Option +pub fn vortex_array::validity::Validity::into_non_nullable(self, usize, &mut vortex_array::ExecutionCtx) -> core::option::Option pub fn vortex_array::validity::Validity::into_nullable(self) -> vortex_array::validity::Validity -pub fn vortex_array::validity::Validity::is_null(&self, index: usize) -> vortex_error::VortexResult +pub fn vortex_array::validity::Validity::is_null(&self, usize) -> vortex_error::VortexResult -pub fn vortex_array::validity::Validity::is_valid(&self, index: usize) -> vortex_error::VortexResult +pub fn vortex_array::validity::Validity::is_valid(&self, usize) -> vortex_error::VortexResult -pub fn vortex_array::validity::Validity::mask_eq(&self, other: &vortex_array::validity::Validity, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::validity::Validity::mask_eq(&self, &vortex_array::validity::Validity, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_array::validity::Validity::maybe_len(&self) -> core::option::Option +pub fn vortex_array::validity::Validity::no_nulls(&self) -> bool + pub fn vortex_array::validity::Validity::not(&self) -> vortex_error::VortexResult pub fn vortex_array::validity::Validity::nullability(&self) -> vortex_array::dtype::Nullability -pub fn vortex_array::validity::Validity::patch(self, len: usize, indices_offset: usize, indices: &vortex_array::ArrayRef, patches: &vortex_array::validity::Validity, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::validity::Validity::patch(self, usize, usize, &vortex_array::ArrayRef, &vortex_array::validity::Validity, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult + +pub fn vortex_array::validity::Validity::slice(&self, core::ops::range::Range) -> vortex_error::VortexResult -pub fn vortex_array::validity::Validity::slice(&self, range: core::ops::range::Range) -> vortex_error::VortexResult +pub fn vortex_array::validity::Validity::take(&self, &vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::validity::Validity::take(&self, indices: &vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::validity::Validity::to_array(&self, usize) -> vortex_array::ArrayRef -pub fn vortex_array::validity::Validity::to_array(&self, len: usize) -> vortex_array::ArrayRef +pub fn vortex_array::validity::Validity::to_mask(&self, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::validity::Validity::to_mask(&self, length: usize) -> vortex_mask::Mask +pub fn vortex_array::validity::Validity::trivial_cast_nullability(self, vortex_array::dtype::Nullability, usize) -> vortex_error::VortexResult> -pub fn vortex_array::validity::Validity::uncompressed_size(&self) -> usize +pub fn vortex_array::validity::Validity::trivial_into_non_nullable(self, usize) -> vortex_error::VortexResult> -pub fn vortex_array::validity::Validity::union_nullability(self, nullability: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::validity::Validity::union_nullability(self, vortex_array::dtype::Nullability) -> Self impl vortex_array::validity::Validity -pub fn vortex_array::validity::Validity::execute(self, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::validity::Validity::concat(alloc::vec::Vec<(vortex_array::validity::Validity, usize)>) -> core::option::Option impl vortex_array::validity::Validity -pub fn vortex_array::validity::Validity::from_bit_buffer(buffer: vortex_buffer::bit::buf::BitBuffer, nullability: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::validity::Validity::execute(self, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::validity::Validity::from_mask(mask: vortex_mask::Mask, nullability: vortex_array::dtype::Nullability) -> Self +impl vortex_array::validity::Validity + +pub fn vortex_array::validity::Validity::from_bit_buffer(vortex_buffer::bit::buf::BitBuffer, vortex_array::dtype::Nullability) -> Self + +pub fn vortex_array::validity::Validity::from_mask(vortex_mask::Mask, vortex_array::dtype::Nullability) -> Self impl core::clone::Clone for vortex_array::validity::Validity @@ -19174,35 +19608,35 @@ pub fn vortex_array::validity::Validity::clone(&self) -> vortex_array::validity: impl core::convert::From<&vortex_array::dtype::Nullability> for vortex_array::validity::Validity -pub fn vortex_array::validity::Validity::from(value: &vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::validity::Validity::from(&vortex_array::dtype::Nullability) -> Self impl core::convert::From for vortex_array::validity::Validity -pub fn vortex_array::validity::Validity::from(value: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::validity::Validity::from(vortex_array::dtype::Nullability) -> Self impl core::convert::From for vortex_array::validity::Validity -pub fn vortex_array::validity::Validity::from(value: vortex_buffer::bit::buf::BitBuffer) -> Self +pub fn vortex_array::validity::Validity::from(vortex_buffer::bit::buf::BitBuffer) -> Self impl core::fmt::Debug for vortex_array::validity::Validity -pub fn vortex_array::validity::Validity::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::validity::Validity::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::iter::traits::collect::FromIterator for vortex_array::validity::Validity -pub fn vortex_array::validity::Validity::from_iter>(iter: T) -> Self +pub fn vortex_array::validity::Validity::from_iter>(T) -> Self impl core::iter::traits::collect::FromIterator for vortex_array::validity::Validity -pub fn vortex_array::validity::Validity::from_iter>(iter: T) -> Self +pub fn vortex_array::validity::Validity::from_iter>(T) -> Self impl vortex_array::ArrayEq for vortex_array::validity::Validity -pub fn vortex_array::validity::Validity::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn vortex_array::validity::Validity::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayHash for vortex_array::validity::Validity -pub fn vortex_array::validity::Validity::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn vortex_array::validity::Validity::array_hash(&self, &mut H, vortex_array::Precision) pub mod vortex_array::variants @@ -19232,37 +19666,37 @@ impl vortex_array::variants::PrimitiveTyped<'_> pub fn vortex_array::variants::PrimitiveTyped<'_>::ptype(&self) -> vortex_array::dtype::PType -pub fn vortex_array::variants::PrimitiveTyped<'_>::value(&self, idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::variants::PrimitiveTyped<'_>::value(&self, usize) -> vortex_error::VortexResult> -pub fn vortex_array::variants::PrimitiveTyped<'_>::value_unchecked(&self, idx: usize) -> vortex_error::VortexResult +pub fn vortex_array::variants::PrimitiveTyped<'_>::value_unchecked(&self, usize) -> vortex_error::VortexResult impl vortex_array::search_sorted::IndexOrd> for vortex_array::variants::PrimitiveTyped<'_> -pub fn vortex_array::variants::PrimitiveTyped<'_>::index_cmp(&self, idx: usize, elem: &core::option::Option) -> vortex_error::VortexResult> +pub fn vortex_array::variants::PrimitiveTyped<'_>::index_cmp(&self, usize, &core::option::Option) -> vortex_error::VortexResult> -pub fn vortex_array::variants::PrimitiveTyped<'_>::index_ge(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::variants::PrimitiveTyped<'_>::index_ge(&self, usize, &V) -> vortex_error::VortexResult -pub fn vortex_array::variants::PrimitiveTyped<'_>::index_gt(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::variants::PrimitiveTyped<'_>::index_gt(&self, usize, &V) -> vortex_error::VortexResult -pub fn vortex_array::variants::PrimitiveTyped<'_>::index_le(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::variants::PrimitiveTyped<'_>::index_le(&self, usize, &V) -> vortex_error::VortexResult pub fn vortex_array::variants::PrimitiveTyped<'_>::index_len(&self) -> usize -pub fn vortex_array::variants::PrimitiveTyped<'_>::index_lt(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::variants::PrimitiveTyped<'_>::index_lt(&self, usize, &V) -> vortex_error::VortexResult impl vortex_array::search_sorted::IndexOrd for vortex_array::variants::PrimitiveTyped<'_> -pub fn vortex_array::variants::PrimitiveTyped<'_>::index_cmp(&self, idx: usize, elem: &vortex_array::scalar::PValue) -> vortex_error::VortexResult> +pub fn vortex_array::variants::PrimitiveTyped<'_>::index_cmp(&self, usize, &vortex_array::scalar::PValue) -> vortex_error::VortexResult> -pub fn vortex_array::variants::PrimitiveTyped<'_>::index_ge(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::variants::PrimitiveTyped<'_>::index_ge(&self, usize, &V) -> vortex_error::VortexResult -pub fn vortex_array::variants::PrimitiveTyped<'_>::index_gt(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::variants::PrimitiveTyped<'_>::index_gt(&self, usize, &V) -> vortex_error::VortexResult -pub fn vortex_array::variants::PrimitiveTyped<'_>::index_le(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::variants::PrimitiveTyped<'_>::index_le(&self, usize, &V) -> vortex_error::VortexResult pub fn vortex_array::variants::PrimitiveTyped<'_>::index_len(&self) -> usize -pub fn vortex_array::variants::PrimitiveTyped<'_>::index_lt(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::variants::PrimitiveTyped<'_>::index_lt(&self, usize, &V) -> vortex_error::VortexResult pub struct vortex_array::variants::StructTyped<'a>(_) @@ -19290,53 +19724,67 @@ pub fn vortex_array::EmptyArrayData::default() -> vortex_array::EmptyArrayData impl core::fmt::Debug for vortex_array::EmptyArrayData -pub fn vortex_array::EmptyArrayData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::EmptyArrayData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::EmptyArrayData -pub fn vortex_array::EmptyArrayData::fmt(&self, _f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::EmptyArrayData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::ArrayEq for vortex_array::EmptyArrayData -pub fn vortex_array::EmptyArrayData::array_eq(&self, _other: &Self, _precision: vortex_array::Precision) -> bool +pub fn vortex_array::EmptyArrayData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayHash for vortex_array::EmptyArrayData -pub fn vortex_array::EmptyArrayData::array_hash(&self, _state: &mut H, _precision: vortex_array::Precision) +pub fn vortex_array::EmptyArrayData::array_hash(&self, &mut H, vortex_array::Precision) pub struct vortex_array::vtable::NotSupported impl vortex_array::OperationsVTable for vortex_array::NotSupported -pub fn vortex_array::NotSupported::scalar_at(array: vortex_array::ArrayView<'_, V>, _index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::NotSupported::scalar_at(vortex_array::ArrayView<'_, V>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub struct vortex_array::vtable::ValidityVTableFromChild impl vortex_array::ValidityVTable for vortex_array::ValidityVTableFromChild where V: vortex_array::ValidityChild + vortex_array::VTable -pub fn vortex_array::ValidityVTableFromChild::validity(array: vortex_array::ArrayView<'_, V>) -> vortex_error::VortexResult +pub fn vortex_array::ValidityVTableFromChild::validity(vortex_array::ArrayView<'_, V>) -> vortex_error::VortexResult pub struct vortex_array::vtable::ValidityVTableFromChildSliceHelper impl vortex_array::ValidityVTable for vortex_array::ValidityVTableFromChildSliceHelper where ::ArrayData: vortex_array::ValidityChildSliceHelper -pub fn vortex_array::ValidityVTableFromChildSliceHelper::validity(array: vortex_array::ArrayView<'_, V>) -> vortex_error::VortexResult +pub fn vortex_array::ValidityVTableFromChildSliceHelper::validity(vortex_array::ArrayView<'_, V>) -> vortex_error::VortexResult pub trait vortex_array::vtable::ArrayPlugin: 'static + core::marker::Send + core::marker::Sync -pub fn vortex_array::vtable::ArrayPlugin::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::vtable::ArrayPlugin::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult pub fn vortex_array::vtable::ArrayPlugin::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::vtable::ArrayPlugin::serialize(&self, array: &vortex_array::ArrayRef, session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::vtable::ArrayPlugin::is_supported_encoding(&self, &vortex_array::ArrayId) -> bool + +pub fn vortex_array::vtable::ArrayPlugin::serialize(&self, &vortex_array::ArrayRef, &vortex_session::VortexSession) -> vortex_error::VortexResult>> impl vortex_array::ArrayPlugin for V -pub fn V::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, session: &vortex_session::VortexSession) -> core::result::Result +pub fn V::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> core::result::Result -pub fn V::id(&self) -> arcref::ArcRef +pub fn V::id(&self) -> vortex_session::registry::Id -pub fn V::serialize(&self, array: &vortex_array::ArrayRef, session: &vortex_session::VortexSession) -> core::result::Result>, vortex_error::VortexError> +pub fn V::is_supported_encoding(&self, &vortex_array::ArrayId) -> bool + +pub fn V::serialize(&self, &vortex_array::ArrayRef, &vortex_session::VortexSession) -> core::result::Result>, vortex_error::VortexError> + +impl vortex_array::ArrayPlugin for vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin + +pub fn vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult + +pub fn vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin::id(&self) -> vortex_array::ArrayId + +pub fn vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin::is_supported_encoding(&self, &vortex_array::ArrayId) -> bool + +pub fn vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin::serialize(&self, &vortex_array::ArrayRef, &vortex_session::VortexSession) -> vortex_error::VortexResult>> pub trait vortex_array::vtable::ArrayVTable: 'static + core::clone::Clone + core::marker::Sized + core::marker::Send + core::marker::Sync + core::fmt::Debug @@ -19346,37 +19794,37 @@ pub type vortex_array::vtable::ArrayVTable::OperationsVTable: vortex_array::Oper pub type vortex_array::vtable::ArrayVTable::ValidityVTable: vortex_array::ValidityVTable -pub fn vortex_array::vtable::ArrayVTable::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::vtable::ArrayVTable::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::vtable::ArrayVTable::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::vtable::ArrayVTable::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::vtable::ArrayVTable::buffer_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::vtable::ArrayVTable::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::vtable::ArrayVTable::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::vtable::ArrayVTable::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::vtable::ArrayVTable::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::vtable::ArrayVTable::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::vtable::ArrayVTable::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::vtable::ArrayVTable::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::vtable::ArrayVTable::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::vtable::ArrayVTable::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::vtable::ArrayVTable::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::vtable::ArrayVTable::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::vtable::ArrayVTable::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::vtable::ArrayVTable::nbuffers(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::vtable::ArrayVTable::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::vtable::ArrayVTable::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::vtable::ArrayVTable::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::vtable::ArrayVTable::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::vtable::ArrayVTable::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::vtable::ArrayVTable::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::vtable::ArrayVTable::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::vtable::ArrayVTable::serialize(array: vortex_array::ArrayView<'_, Self>, session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::vtable::ArrayVTable::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::vtable::ArrayVTable::slot_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::vtable::ArrayVTable::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::vtable::ArrayVTable::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::vtable::ArrayVTable::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Bool @@ -19386,37 +19834,37 @@ pub type vortex_array::arrays::Bool::OperationsVTable = vortex_array::arrays::Bo pub type vortex_array::arrays::Bool::ValidityVTable = vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Bool::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Bool::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Bool::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Bool::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Bool::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Bool::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Bool::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Bool::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Bool::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Bool::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Bool::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Bool::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Bool::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Bool::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Bool::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Bool::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Bool::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Bool::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Bool::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Bool::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Bool::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Bool::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Bool::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Bool::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Bool::validate(&self, data: &vortex_array::arrays::bool::BoolData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Bool::validate(&self, &vortex_array::arrays::bool::BoolData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Chunked @@ -19426,37 +19874,37 @@ pub type vortex_array::arrays::Chunked::OperationsVTable = vortex_array::arrays: pub type vortex_array::arrays::Chunked::ValidityVTable = vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Chunked::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Chunked::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Chunked::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Chunked::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Chunked::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Chunked::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Chunked::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Chunked::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Chunked::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Chunked::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Chunked::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Chunked::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Chunked::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Chunked::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Chunked::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Chunked::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Chunked::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Chunked::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Chunked::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Chunked::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Chunked::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Chunked::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Chunked::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Chunked::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Chunked::validate(&self, data: &vortex_array::arrays::chunked::ChunkedData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Chunked::validate(&self, &vortex_array::arrays::chunked::ChunkedData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Constant @@ -19466,37 +19914,37 @@ pub type vortex_array::arrays::Constant::OperationsVTable = vortex_array::arrays pub type vortex_array::arrays::Constant::ValidityVTable = vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Constant::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Constant::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Constant::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Constant::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Constant::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Constant::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Constant::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Constant::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Constant::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Constant::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, _metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Constant::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Constant::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Constant::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Constant::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Constant::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Constant::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Constant::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Constant::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Constant::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Constant::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Constant::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Constant::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Constant::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Constant::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Constant::validate(&self, data: &vortex_array::arrays::constant::ConstantData, dtype: &vortex_array::dtype::DType, _len: usize, _slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Constant::validate(&self, &vortex_array::arrays::constant::ConstantData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Decimal @@ -19506,77 +19954,77 @@ pub type vortex_array::arrays::Decimal::OperationsVTable = vortex_array::arrays: pub type vortex_array::arrays::Decimal::ValidityVTable = vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Decimal::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Decimal::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Decimal::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Decimal::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Decimal::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Decimal::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Decimal::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Decimal::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Decimal::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Decimal::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Decimal::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Decimal::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Decimal::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Decimal::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Decimal::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Decimal::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Decimal::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Decimal::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Decimal::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Decimal::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Decimal::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Decimal::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Decimal::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Decimal::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Decimal::validate(&self, data: &vortex_array::arrays::decimal::DecimalData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Decimal::validate(&self, &vortex_array::arrays::decimal::DecimalData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Extension -pub type vortex_array::arrays::Extension::ArrayData = vortex_array::arrays::extension::ExtensionData +pub type vortex_array::arrays::Extension::ArrayData = vortex_array::EmptyArrayData pub type vortex_array::arrays::Extension::OperationsVTable = vortex_array::arrays::Extension pub type vortex_array::arrays::Extension::ValidityVTable = vortex_array::ValidityVTableFromChild -pub fn vortex_array::arrays::Extension::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Extension::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Extension::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Extension::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Extension::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Extension::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Extension::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Extension::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Extension::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Extension::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Extension::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Extension::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Extension::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Extension::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Extension::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Extension::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Extension::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Extension::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Extension::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Extension::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Extension::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Extension::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Extension::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Extension::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Extension::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Extension::validate(&self, data: &vortex_array::arrays::extension::ExtensionData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Extension::validate(&self, &vortex_array::EmptyArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Filter @@ -19586,37 +20034,37 @@ pub type vortex_array::arrays::Filter::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::Filter::ValidityVTable = vortex_array::arrays::Filter -pub fn vortex_array::arrays::Filter::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Filter::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Filter::buffer(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Filter::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Filter::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Filter::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Filter::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Filter::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Filter::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Filter::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Filter::deserialize(&self, _dtype: &vortex_array::dtype::DType, _len: usize, _metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Filter::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Filter::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Filter::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Filter::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Filter::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Filter::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Filter::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Filter::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Filter::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Filter::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Filter::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Filter::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Filter::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Filter::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Filter::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Filter::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Filter::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Filter::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Filter::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Filter::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::FixedSizeList @@ -19626,37 +20074,37 @@ pub type vortex_array::arrays::FixedSizeList::OperationsVTable = vortex_array::a pub type vortex_array::arrays::FixedSizeList::ValidityVTable = vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::FixedSizeList::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::FixedSizeList::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::FixedSizeList::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::FixedSizeList::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::FixedSizeList::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::FixedSizeList::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::FixedSizeList::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::FixedSizeList::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::FixedSizeList::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::FixedSizeList::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::FixedSizeList::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::FixedSizeList::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::FixedSizeList::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::FixedSizeList::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::FixedSizeList::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::FixedSizeList::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::FixedSizeList::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::FixedSizeList::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::FixedSizeList::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::FixedSizeList::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::FixedSizeList::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::FixedSizeList::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::FixedSizeList::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::FixedSizeList::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::FixedSizeList::validate(&self, data: &vortex_array::arrays::fixed_size_list::FixedSizeListData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::FixedSizeList::validate(&self, &vortex_array::arrays::fixed_size_list::FixedSizeListData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::List @@ -19666,37 +20114,37 @@ pub type vortex_array::arrays::List::OperationsVTable = vortex_array::arrays::Li pub type vortex_array::arrays::List::ValidityVTable = vortex_array::arrays::List -pub fn vortex_array::arrays::List::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::List::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::List::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::List::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::List::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::List::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::List::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::List::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::List::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::List::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::List::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::List::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::List::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::List::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::List::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::List::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::List::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::List::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::List::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::List::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::List::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::List::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::List::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::List::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::List::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::List::validate(&self, _data: &vortex_array::arrays::list::ListData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::List::validate(&self, &vortex_array::arrays::list::ListData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::ListView @@ -19706,37 +20154,37 @@ pub type vortex_array::arrays::ListView::OperationsVTable = vortex_array::arrays pub type vortex_array::arrays::ListView::ValidityVTable = vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::ListView::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::ListView::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::ListView::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::ListView::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::ListView::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::ListView::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::ListView::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::ListView::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::ListView::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::ListView::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::ListView::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::ListView::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::ListView::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::ListView::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::ListView::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::ListView::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::ListView::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::ListView::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::ListView::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::ListView::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::ListView::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::ListView::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::ListView::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::ListView::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::ListView::validate(&self, _data: &vortex_array::arrays::listview::ListViewData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::ListView::validate(&self, &vortex_array::arrays::listview::ListViewData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Masked @@ -19746,37 +20194,37 @@ pub type vortex_array::arrays::Masked::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::Masked::ValidityVTable = vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Masked::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Masked::buffer(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Masked::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Masked::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Masked::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Masked::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Masked::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Masked::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Masked::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Masked::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Masked::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Masked::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Masked::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Masked::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Masked::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Masked::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Masked::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Masked::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Masked::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Masked::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Masked::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Masked::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Masked::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Masked::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Masked::validate(&self, _data: &vortex_array::arrays::masked::MaskedData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Masked::validate(&self, &vortex_array::arrays::masked::MaskedData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Primitive @@ -19786,37 +20234,37 @@ pub type vortex_array::arrays::Primitive::OperationsVTable = vortex_array::array pub type vortex_array::arrays::Primitive::ValidityVTable = vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Primitive::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Primitive::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Primitive::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Primitive::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Primitive::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Primitive::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Primitive::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Primitive::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Primitive::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Primitive::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Primitive::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Primitive::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Primitive::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Primitive::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Primitive::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Primitive::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Primitive::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Primitive::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Primitive::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Primitive::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Primitive::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Primitive::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Primitive::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Primitive::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Primitive::validate(&self, data: &vortex_array::arrays::primitive::PrimitiveData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Primitive::validate(&self, &vortex_array::arrays::primitive::PrimitiveData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Shared @@ -19826,37 +20274,37 @@ pub type vortex_array::arrays::Shared::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::Shared::ValidityVTable = vortex_array::arrays::Shared -pub fn vortex_array::arrays::Shared::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Shared::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Shared::buffer(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Shared::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Shared::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Shared::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Shared::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Shared::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Shared::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Shared::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Shared::deserialize(&self, _dtype: &vortex_array::dtype::DType, _len: usize, _metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Shared::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Shared::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Shared::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Shared::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Shared::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Shared::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Shared::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Shared::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Shared::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Shared::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Shared::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Shared::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Shared::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Shared::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Shared::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Shared::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Shared::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Shared::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Shared::validate(&self, _data: &vortex_array::arrays::shared::SharedData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Shared::validate(&self, &vortex_array::arrays::shared::SharedData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Struct @@ -19866,37 +20314,37 @@ pub type vortex_array::arrays::Struct::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::Struct::ValidityVTable = vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Struct::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Struct::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Struct::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Struct::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Struct::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Struct::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Struct::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Struct::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Struct::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Struct::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Struct::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Struct::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Struct::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Struct::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Struct::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Struct::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Struct::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Struct::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Struct::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Struct::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Struct::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Struct::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Struct::slot_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Struct::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Struct::validate(&self, _data: &vortex_array::EmptyArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Struct::validate(&self, &vortex_array::EmptyArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::VarBin @@ -19906,37 +20354,37 @@ pub type vortex_array::arrays::VarBin::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::VarBin::ValidityVTable = vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::VarBin::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::VarBin::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::VarBin::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::VarBin::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::VarBin::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::VarBin::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::VarBin::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::VarBin::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::VarBin::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::VarBin::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBin::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBin::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::VarBin::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::VarBin::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::VarBin::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::VarBin::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::VarBin::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::VarBin::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::VarBin::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBin::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBin::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::VarBin::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::VarBin::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::VarBin::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::VarBin::validate(&self, _data: &vortex_array::arrays::varbin::VarBinData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::VarBin::validate(&self, &vortex_array::arrays::varbin::VarBinData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::VarBinView @@ -19946,37 +20394,37 @@ pub type vortex_array::arrays::VarBinView::OperationsVTable = vortex_array::arra pub type vortex_array::arrays::VarBinView::ValidityVTable = vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::VarBinView::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::VarBinView::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::VarBinView::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::VarBinView::buffer_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::VarBinView::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::VarBinView::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::VarBinView::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::VarBinView::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::VarBinView::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::VarBinView::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBinView::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBinView::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::VarBinView::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::VarBinView::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::VarBinView::nbuffers(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::VarBinView::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::VarBinView::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::VarBinView::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::VarBinView::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBinView::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBinView::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::VarBinView::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::VarBinView::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::VarBinView::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::VarBinView::validate(&self, data: &vortex_array::arrays::varbinview::VarBinViewData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::VarBinView::validate(&self, &vortex_array::arrays::varbinview::VarBinViewData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Variant @@ -19986,37 +20434,37 @@ pub type vortex_array::arrays::Variant::OperationsVTable = vortex_array::arrays: pub type vortex_array::arrays::Variant::ValidityVTable = vortex_array::arrays::Variant -pub fn vortex_array::arrays::Variant::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Variant::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Variant::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Variant::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Variant::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Variant::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Variant::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Variant::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Variant::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Variant::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Variant::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Variant::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Variant::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Variant::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Variant::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Variant::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Variant::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Variant::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Variant::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Variant::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Variant::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Variant::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Variant::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Variant::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Variant::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Variant::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Variant::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Variant::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Variant::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Variant::validate(&self, _data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Variant::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::dict::Dict @@ -20026,37 +20474,37 @@ pub type vortex_array::arrays::dict::Dict::OperationsVTable = vortex_array::arra pub type vortex_array::arrays::dict::Dict::ValidityVTable = vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::dict::Dict::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::dict::Dict::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::dict::Dict::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::dict::Dict::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::dict::Dict::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::dict::Dict::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::dict::Dict::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::dict::Dict::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::dict::Dict::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::dict::Dict::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::dict::Dict::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::dict::Dict::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::dict::Dict::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::dict::Dict::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::dict::Dict::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::dict::Dict::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::dict::Dict::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::dict::Dict::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::dict::Dict::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::dict::Dict::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::dict::Dict::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::dict::Dict::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::dict::Dict::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::dict::Dict::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::dict::Dict::validate(&self, _data: &vortex_array::arrays::dict::DictData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::dict::Dict::validate(&self, &vortex_array::arrays::dict::DictData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::null::Null @@ -20066,37 +20514,37 @@ pub type vortex_array::arrays::null::Null::OperationsVTable = vortex_array::arra pub type vortex_array::arrays::null::Null::ValidityVTable = vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::null::Null::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::null::Null::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::null::Null::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::null::Null::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::null::Null::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::null::Null::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::null::Null::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::null::Null::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::null::Null::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::null::Null::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::null::Null::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::null::Null::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::null::Null::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::null::Null::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::null::Null::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::null::Null::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::null::Null::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::null::Null::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::null::Null::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::null::Null::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::null::Null::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::null::Null::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::null::Null::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::null::Null::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::null::Null::validate(&self, _data: &vortex_array::EmptyArrayData, dtype: &vortex_array::dtype::DType, _len: usize, _slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::null::Null::validate(&self, &vortex_array::EmptyArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::patched::Patched @@ -20106,77 +20554,77 @@ pub type vortex_array::arrays::patched::Patched::OperationsVTable = vortex_array pub type vortex_array::arrays::patched::Patched::ValidityVTable = vortex_array::ValidityVTableFromChild -pub fn vortex_array::arrays::patched::Patched::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::patched::Patched::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::patched::Patched::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::patched::Patched::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::patched::Patched::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::patched::Patched::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::patched::Patched::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::patched::Patched::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::patched::Patched::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::patched::Patched::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::patched::Patched::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::patched::Patched::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::patched::Patched::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::patched::Patched::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::patched::Patched::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::patched::Patched::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::patched::Patched::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::patched::Patched::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::patched::Patched::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::patched::Patched::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::patched::Patched::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::patched::Patched::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::patched::Patched::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::patched::Patched::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::patched::Patched::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::patched::Patched::validate(&self, data: &vortex_array::arrays::patched::PatchedData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::patched::Patched::validate(&self, &vortex_array::arrays::patched::PatchedData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> -impl vortex_array::VTable for vortex_array::arrays::scalar_fn::ScalarFnVTable +impl vortex_array::VTable for vortex_array::arrays::scalar_fn::ScalarFn -pub type vortex_array::arrays::scalar_fn::ScalarFnVTable::ArrayData = vortex_array::arrays::scalar_fn::ScalarFnData +pub type vortex_array::arrays::scalar_fn::ScalarFn::ArrayData = vortex_array::arrays::scalar_fn::array::ScalarFnData -pub type vortex_array::arrays::scalar_fn::ScalarFnVTable::OperationsVTable = vortex_array::arrays::scalar_fn::ScalarFnVTable +pub type vortex_array::arrays::scalar_fn::ScalarFn::OperationsVTable = vortex_array::arrays::scalar_fn::ScalarFn -pub type vortex_array::arrays::scalar_fn::ScalarFnVTable::ValidityVTable = vortex_array::arrays::scalar_fn::ScalarFnVTable +pub type vortex_array::arrays::scalar_fn::ScalarFn::ValidityVTable = vortex_array::arrays::scalar_fn::ScalarFn -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::scalar_fn::ScalarFn::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::scalar_fn::ScalarFn::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::scalar_fn::ScalarFn::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::scalar_fn::ScalarFn::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::deserialize(&self, _dtype: &vortex_array::dtype::DType, _len: usize, _metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::scalar_fn::ScalarFn::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::id(&self) -> vortex_array::ArrayId +pub fn vortex_array::arrays::scalar_fn::ScalarFn::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::scalar_fn::ScalarFn::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::scalar_fn::ScalarFn::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::slot_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::scalar_fn::ScalarFn::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::validate(&self, data: &vortex_array::arrays::scalar_fn::ScalarFnData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::validate(&self, &vortex_array::arrays::scalar_fn::array::ScalarFnData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::slice::Slice @@ -20186,129 +20634,129 @@ pub type vortex_array::arrays::slice::Slice::OperationsVTable = vortex_array::ar pub type vortex_array::arrays::slice::Slice::ValidityVTable = vortex_array::arrays::slice::Slice -pub fn vortex_array::arrays::slice::Slice::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::slice::Slice::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::slice::Slice::buffer(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::slice::Slice::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::slice::Slice::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::slice::Slice::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::slice::Slice::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::slice::Slice::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::slice::Slice::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::slice::Slice::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::slice::Slice::deserialize(&self, _dtype: &vortex_array::dtype::DType, _len: usize, _metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::slice::Slice::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::slice::Slice::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::slice::Slice::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::slice::Slice::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::slice::Slice::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::slice::Slice::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::slice::Slice::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::slice::Slice::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::slice::Slice::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::slice::Slice::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::slice::Slice::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::slice::Slice::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::slice::Slice::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::slice::Slice::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::slice::Slice::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::slice::Slice::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> pub trait vortex_array::vtable::OperationsVTable -pub fn vortex_array::vtable::OperationsVTable::scalar_at(array: vortex_array::ArrayView<'_, V>, index: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::vtable::OperationsVTable::scalar_at(vortex_array::ArrayView<'_, V>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Bool::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Chunked::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, _index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Constant::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Decimal::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Extension::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::Filter -pub fn vortex_array::arrays::Filter::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Filter>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Filter::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Filter>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::FixedSizeList::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::List -pub fn vortex_array::arrays::List::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::List>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::List::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::List>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::ListView::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Masked::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Primitive::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::Shared -pub fn vortex_array::arrays::Shared::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Shared>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Shared::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Shared>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Struct::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBin::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBinView::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::Variant -pub fn vortex_array::arrays::Variant::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Variant>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Variant::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Variant>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::dict::Dict::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::scalar_at(_array: vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, _index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::null::Null::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::patched::Patched -pub fn vortex_array::arrays::patched::Patched::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::patched::Patched>, index: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::patched::Patched::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::patched::Patched>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -impl vortex_array::OperationsVTable for vortex_array::arrays::scalar_fn::ScalarFnVTable +impl vortex_array::OperationsVTable for vortex_array::arrays::scalar_fn::ScalarFn -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::scalar_fn::ScalarFnVTable>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::scalar_fn::ScalarFn::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::scalar_fn::ScalarFn>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::slice::Slice -pub fn vortex_array::arrays::slice::Slice::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::slice::Slice>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::slice::Slice::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::slice::Slice>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::NotSupported -pub fn vortex_array::NotSupported::scalar_at(array: vortex_array::ArrayView<'_, V>, _index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::NotSupported::scalar_at(vortex_array::ArrayView<'_, V>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub trait vortex_array::vtable::VTable: 'static + core::clone::Clone + core::marker::Sized + core::marker::Send + core::marker::Sync + core::fmt::Debug @@ -20318,37 +20766,37 @@ pub type vortex_array::vtable::VTable::OperationsVTable: vortex_array::Operation pub type vortex_array::vtable::VTable::ValidityVTable: vortex_array::ValidityVTable -pub fn vortex_array::vtable::VTable::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::vtable::VTable::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::vtable::VTable::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::vtable::VTable::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::vtable::VTable::buffer_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::vtable::VTable::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::vtable::VTable::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::vtable::VTable::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::vtable::VTable::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::vtable::VTable::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::vtable::VTable::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::vtable::VTable::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::vtable::VTable::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::vtable::VTable::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::vtable::VTable::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::vtable::VTable::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::vtable::VTable::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::vtable::VTable::nbuffers(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::vtable::VTable::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::vtable::VTable::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::vtable::VTable::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::vtable::VTable::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::vtable::VTable::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::vtable::VTable::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::vtable::VTable::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::vtable::VTable::serialize(array: vortex_array::ArrayView<'_, Self>, session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::vtable::VTable::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::vtable::VTable::slot_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::vtable::VTable::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::vtable::VTable::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::vtable::VTable::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Bool @@ -20358,37 +20806,37 @@ pub type vortex_array::arrays::Bool::OperationsVTable = vortex_array::arrays::Bo pub type vortex_array::arrays::Bool::ValidityVTable = vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Bool::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Bool::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Bool::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Bool::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Bool::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Bool::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Bool::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Bool::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Bool::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Bool::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Bool::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Bool::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Bool::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Bool::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Bool::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Bool::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Bool::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Bool::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Bool::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Bool::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Bool::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Bool::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Bool::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Bool::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Bool::validate(&self, data: &vortex_array::arrays::bool::BoolData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Bool::validate(&self, &vortex_array::arrays::bool::BoolData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Chunked @@ -20398,37 +20846,37 @@ pub type vortex_array::arrays::Chunked::OperationsVTable = vortex_array::arrays: pub type vortex_array::arrays::Chunked::ValidityVTable = vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Chunked::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Chunked::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Chunked::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Chunked::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Chunked::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Chunked::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Chunked::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Chunked::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Chunked::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Chunked::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Chunked::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Chunked::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Chunked::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Chunked::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Chunked::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Chunked::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Chunked::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Chunked::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Chunked::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Chunked::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Chunked::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Chunked::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Chunked::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Chunked::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Chunked::validate(&self, data: &vortex_array::arrays::chunked::ChunkedData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Chunked::validate(&self, &vortex_array::arrays::chunked::ChunkedData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Constant @@ -20438,37 +20886,37 @@ pub type vortex_array::arrays::Constant::OperationsVTable = vortex_array::arrays pub type vortex_array::arrays::Constant::ValidityVTable = vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Constant::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Constant::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Constant::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Constant::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Constant::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Constant::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Constant::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Constant::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Constant::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Constant::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, _metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Constant::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Constant::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Constant::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Constant::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Constant::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Constant::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Constant::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Constant::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Constant::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Constant::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Constant::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Constant::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Constant::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Constant::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Constant::validate(&self, data: &vortex_array::arrays::constant::ConstantData, dtype: &vortex_array::dtype::DType, _len: usize, _slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Constant::validate(&self, &vortex_array::arrays::constant::ConstantData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Decimal @@ -20478,77 +20926,77 @@ pub type vortex_array::arrays::Decimal::OperationsVTable = vortex_array::arrays: pub type vortex_array::arrays::Decimal::ValidityVTable = vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Decimal::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Decimal::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Decimal::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Decimal::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Decimal::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Decimal::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Decimal::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Decimal::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Decimal::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Decimal::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Decimal::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Decimal::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Decimal::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Decimal::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Decimal::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Decimal::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Decimal::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Decimal::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Decimal::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Decimal::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Decimal::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Decimal::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Decimal::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Decimal::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Decimal::validate(&self, data: &vortex_array::arrays::decimal::DecimalData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Decimal::validate(&self, &vortex_array::arrays::decimal::DecimalData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Extension -pub type vortex_array::arrays::Extension::ArrayData = vortex_array::arrays::extension::ExtensionData +pub type vortex_array::arrays::Extension::ArrayData = vortex_array::EmptyArrayData pub type vortex_array::arrays::Extension::OperationsVTable = vortex_array::arrays::Extension pub type vortex_array::arrays::Extension::ValidityVTable = vortex_array::ValidityVTableFromChild -pub fn vortex_array::arrays::Extension::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Extension::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Extension::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Extension::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Extension::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Extension::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Extension::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Extension::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Extension::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Extension::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Extension::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Extension::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Extension::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Extension::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Extension::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Extension::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Extension::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Extension::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Extension::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Extension::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Extension::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Extension::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Extension::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Extension::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Extension::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Extension::validate(&self, data: &vortex_array::arrays::extension::ExtensionData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Extension::validate(&self, &vortex_array::EmptyArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Filter @@ -20558,37 +21006,37 @@ pub type vortex_array::arrays::Filter::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::Filter::ValidityVTable = vortex_array::arrays::Filter -pub fn vortex_array::arrays::Filter::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Filter::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Filter::buffer(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Filter::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Filter::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Filter::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Filter::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Filter::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Filter::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Filter::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Filter::deserialize(&self, _dtype: &vortex_array::dtype::DType, _len: usize, _metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Filter::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Filter::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Filter::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Filter::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Filter::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Filter::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Filter::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Filter::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Filter::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Filter::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Filter::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Filter::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Filter::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Filter::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Filter::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Filter::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Filter::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Filter::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Filter::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Filter::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::FixedSizeList @@ -20598,37 +21046,37 @@ pub type vortex_array::arrays::FixedSizeList::OperationsVTable = vortex_array::a pub type vortex_array::arrays::FixedSizeList::ValidityVTable = vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::FixedSizeList::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::FixedSizeList::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::FixedSizeList::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::FixedSizeList::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::FixedSizeList::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::FixedSizeList::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::FixedSizeList::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::FixedSizeList::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::FixedSizeList::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::FixedSizeList::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::FixedSizeList::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::FixedSizeList::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::FixedSizeList::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::FixedSizeList::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::FixedSizeList::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::FixedSizeList::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::FixedSizeList::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::FixedSizeList::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::FixedSizeList::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::FixedSizeList::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::FixedSizeList::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::FixedSizeList::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::FixedSizeList::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::FixedSizeList::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::FixedSizeList::validate(&self, data: &vortex_array::arrays::fixed_size_list::FixedSizeListData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::FixedSizeList::validate(&self, &vortex_array::arrays::fixed_size_list::FixedSizeListData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::List @@ -20638,37 +21086,37 @@ pub type vortex_array::arrays::List::OperationsVTable = vortex_array::arrays::Li pub type vortex_array::arrays::List::ValidityVTable = vortex_array::arrays::List -pub fn vortex_array::arrays::List::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::List::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::List::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::List::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::List::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::List::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::List::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::List::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::List::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::List::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::List::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::List::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::List::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::List::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::List::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::List::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::List::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::List::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::List::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::List::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::List::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::List::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::List::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::List::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::List::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::List::validate(&self, _data: &vortex_array::arrays::list::ListData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::List::validate(&self, &vortex_array::arrays::list::ListData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::ListView @@ -20678,37 +21126,37 @@ pub type vortex_array::arrays::ListView::OperationsVTable = vortex_array::arrays pub type vortex_array::arrays::ListView::ValidityVTable = vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::ListView::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::ListView::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::ListView::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::ListView::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::ListView::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::ListView::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::ListView::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::ListView::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::ListView::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::ListView::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::ListView::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::ListView::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::ListView::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::ListView::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::ListView::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::ListView::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::ListView::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::ListView::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::ListView::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::ListView::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::ListView::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::ListView::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::ListView::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::ListView::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::ListView::validate(&self, _data: &vortex_array::arrays::listview::ListViewData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::ListView::validate(&self, &vortex_array::arrays::listview::ListViewData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Masked @@ -20718,37 +21166,37 @@ pub type vortex_array::arrays::Masked::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::Masked::ValidityVTable = vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Masked::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Masked::buffer(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Masked::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Masked::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Masked::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Masked::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Masked::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Masked::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Masked::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Masked::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Masked::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Masked::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Masked::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Masked::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Masked::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Masked::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Masked::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Masked::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Masked::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Masked::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Masked::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Masked::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Masked::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Masked::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Masked::validate(&self, _data: &vortex_array::arrays::masked::MaskedData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Masked::validate(&self, &vortex_array::arrays::masked::MaskedData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Primitive @@ -20758,37 +21206,37 @@ pub type vortex_array::arrays::Primitive::OperationsVTable = vortex_array::array pub type vortex_array::arrays::Primitive::ValidityVTable = vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Primitive::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Primitive::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Primitive::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Primitive::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Primitive::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Primitive::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Primitive::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Primitive::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Primitive::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Primitive::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Primitive::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Primitive::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Primitive::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Primitive::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Primitive::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Primitive::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Primitive::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Primitive::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Primitive::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Primitive::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Primitive::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Primitive::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Primitive::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Primitive::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Primitive::validate(&self, data: &vortex_array::arrays::primitive::PrimitiveData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Primitive::validate(&self, &vortex_array::arrays::primitive::PrimitiveData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Shared @@ -20798,37 +21246,37 @@ pub type vortex_array::arrays::Shared::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::Shared::ValidityVTable = vortex_array::arrays::Shared -pub fn vortex_array::arrays::Shared::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Shared::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Shared::buffer(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Shared::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Shared::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Shared::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Shared::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Shared::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Shared::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Shared::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Shared::deserialize(&self, _dtype: &vortex_array::dtype::DType, _len: usize, _metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Shared::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Shared::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Shared::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Shared::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Shared::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Shared::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Shared::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Shared::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Shared::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Shared::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Shared::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Shared::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Shared::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Shared::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Shared::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Shared::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Shared::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Shared::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Shared::validate(&self, _data: &vortex_array::arrays::shared::SharedData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Shared::validate(&self, &vortex_array::arrays::shared::SharedData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Struct @@ -20838,37 +21286,37 @@ pub type vortex_array::arrays::Struct::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::Struct::ValidityVTable = vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Struct::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Struct::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Struct::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Struct::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Struct::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Struct::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Struct::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Struct::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Struct::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Struct::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Struct::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Struct::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Struct::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Struct::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Struct::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Struct::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Struct::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Struct::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Struct::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Struct::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Struct::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Struct::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Struct::slot_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Struct::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Struct::validate(&self, _data: &vortex_array::EmptyArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Struct::validate(&self, &vortex_array::EmptyArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::VarBin @@ -20878,37 +21326,37 @@ pub type vortex_array::arrays::VarBin::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::VarBin::ValidityVTable = vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::VarBin::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::VarBin::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::VarBin::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::VarBin::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::VarBin::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::VarBin::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::VarBin::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::VarBin::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::VarBin::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::VarBin::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBin::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBin::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::VarBin::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::VarBin::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::VarBin::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::VarBin::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::VarBin::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::VarBin::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::VarBin::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBin::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBin::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::VarBin::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::VarBin::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::VarBin::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::VarBin::validate(&self, _data: &vortex_array::arrays::varbin::VarBinData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::VarBin::validate(&self, &vortex_array::arrays::varbin::VarBinData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::VarBinView @@ -20918,37 +21366,37 @@ pub type vortex_array::arrays::VarBinView::OperationsVTable = vortex_array::arra pub type vortex_array::arrays::VarBinView::ValidityVTable = vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::VarBinView::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::VarBinView::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::VarBinView::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::VarBinView::buffer_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::VarBinView::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::VarBinView::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::VarBinView::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::VarBinView::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::VarBinView::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::VarBinView::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBinView::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBinView::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::VarBinView::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::VarBinView::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::VarBinView::nbuffers(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::VarBinView::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::VarBinView::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::VarBinView::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::VarBinView::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBinView::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBinView::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::VarBinView::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::VarBinView::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::VarBinView::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::VarBinView::validate(&self, data: &vortex_array::arrays::varbinview::VarBinViewData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::VarBinView::validate(&self, &vortex_array::arrays::varbinview::VarBinViewData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Variant @@ -20958,37 +21406,37 @@ pub type vortex_array::arrays::Variant::OperationsVTable = vortex_array::arrays: pub type vortex_array::arrays::Variant::ValidityVTable = vortex_array::arrays::Variant -pub fn vortex_array::arrays::Variant::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Variant::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Variant::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Variant::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Variant::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Variant::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Variant::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Variant::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Variant::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Variant::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Variant::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Variant::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Variant::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Variant::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Variant::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Variant::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Variant::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Variant::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Variant::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Variant::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Variant::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Variant::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Variant::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Variant::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Variant::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Variant::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Variant::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Variant::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Variant::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Variant::validate(&self, _data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Variant::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::dict::Dict @@ -20998,37 +21446,37 @@ pub type vortex_array::arrays::dict::Dict::OperationsVTable = vortex_array::arra pub type vortex_array::arrays::dict::Dict::ValidityVTable = vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::dict::Dict::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::dict::Dict::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::dict::Dict::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::dict::Dict::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::dict::Dict::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::dict::Dict::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::dict::Dict::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::dict::Dict::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::dict::Dict::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::dict::Dict::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::dict::Dict::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::dict::Dict::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::dict::Dict::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::dict::Dict::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::dict::Dict::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::dict::Dict::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::dict::Dict::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::dict::Dict::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::dict::Dict::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::dict::Dict::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::dict::Dict::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::dict::Dict::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::dict::Dict::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::dict::Dict::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::dict::Dict::validate(&self, _data: &vortex_array::arrays::dict::DictData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::dict::Dict::validate(&self, &vortex_array::arrays::dict::DictData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::null::Null @@ -21038,37 +21486,37 @@ pub type vortex_array::arrays::null::Null::OperationsVTable = vortex_array::arra pub type vortex_array::arrays::null::Null::ValidityVTable = vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::null::Null::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::null::Null::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::null::Null::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::null::Null::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::null::Null::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::null::Null::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::null::Null::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::null::Null::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::null::Null::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::null::Null::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::null::Null::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::null::Null::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::null::Null::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::null::Null::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::null::Null::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::null::Null::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::null::Null::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::null::Null::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::null::Null::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::null::Null::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::null::Null::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::null::Null::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::null::Null::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::null::Null::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::null::Null::validate(&self, _data: &vortex_array::EmptyArrayData, dtype: &vortex_array::dtype::DType, _len: usize, _slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::null::Null::validate(&self, &vortex_array::EmptyArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::patched::Patched @@ -21078,77 +21526,77 @@ pub type vortex_array::arrays::patched::Patched::OperationsVTable = vortex_array pub type vortex_array::arrays::patched::Patched::ValidityVTable = vortex_array::ValidityVTableFromChild -pub fn vortex_array::arrays::patched::Patched::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::patched::Patched::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::patched::Patched::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::patched::Patched::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::patched::Patched::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::patched::Patched::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::patched::Patched::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::patched::Patched::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::patched::Patched::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::patched::Patched::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::patched::Patched::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::patched::Patched::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::patched::Patched::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::patched::Patched::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::patched::Patched::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::patched::Patched::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::patched::Patched::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::patched::Patched::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::patched::Patched::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::patched::Patched::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::patched::Patched::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::patched::Patched::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::patched::Patched::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::patched::Patched::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::patched::Patched::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::patched::Patched::validate(&self, data: &vortex_array::arrays::patched::PatchedData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::patched::Patched::validate(&self, &vortex_array::arrays::patched::PatchedData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> -impl vortex_array::VTable for vortex_array::arrays::scalar_fn::ScalarFnVTable +impl vortex_array::VTable for vortex_array::arrays::scalar_fn::ScalarFn -pub type vortex_array::arrays::scalar_fn::ScalarFnVTable::ArrayData = vortex_array::arrays::scalar_fn::ScalarFnData +pub type vortex_array::arrays::scalar_fn::ScalarFn::ArrayData = vortex_array::arrays::scalar_fn::array::ScalarFnData -pub type vortex_array::arrays::scalar_fn::ScalarFnVTable::OperationsVTable = vortex_array::arrays::scalar_fn::ScalarFnVTable +pub type vortex_array::arrays::scalar_fn::ScalarFn::OperationsVTable = vortex_array::arrays::scalar_fn::ScalarFn -pub type vortex_array::arrays::scalar_fn::ScalarFnVTable::ValidityVTable = vortex_array::arrays::scalar_fn::ScalarFnVTable +pub type vortex_array::arrays::scalar_fn::ScalarFn::ValidityVTable = vortex_array::arrays::scalar_fn::ScalarFn -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::scalar_fn::ScalarFn::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::scalar_fn::ScalarFn::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::scalar_fn::ScalarFn::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::scalar_fn::ScalarFn::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::deserialize(&self, _dtype: &vortex_array::dtype::DType, _len: usize, _metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::scalar_fn::ScalarFn::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::id(&self) -> vortex_array::ArrayId +pub fn vortex_array::arrays::scalar_fn::ScalarFn::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::scalar_fn::ScalarFn::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::scalar_fn::ScalarFn::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::slot_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::scalar_fn::ScalarFn::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::validate(&self, data: &vortex_array::arrays::scalar_fn::ScalarFnData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::validate(&self, &vortex_array::arrays::scalar_fn::array::ScalarFnData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::slice::Slice @@ -21158,49 +21606,49 @@ pub type vortex_array::arrays::slice::Slice::OperationsVTable = vortex_array::ar pub type vortex_array::arrays::slice::Slice::ValidityVTable = vortex_array::arrays::slice::Slice -pub fn vortex_array::arrays::slice::Slice::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::slice::Slice::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::slice::Slice::buffer(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::slice::Slice::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::slice::Slice::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::slice::Slice::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::slice::Slice::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::slice::Slice::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::slice::Slice::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::slice::Slice::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::slice::Slice::deserialize(&self, _dtype: &vortex_array::dtype::DType, _len: usize, _metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::slice::Slice::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::slice::Slice::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::slice::Slice::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::slice::Slice::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::slice::Slice::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::slice::Slice::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::slice::Slice::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::slice::Slice::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::slice::Slice::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::slice::Slice::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::slice::Slice::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::slice::Slice::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::slice::Slice::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::slice::Slice::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::slice::Slice::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::slice::Slice::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> pub trait vortex_array::vtable::ValidityChild -pub fn vortex_array::vtable::ValidityChild::validity_child(array: vortex_array::ArrayView<'_, V>) -> vortex_array::ArrayRef +pub fn vortex_array::vtable::ValidityChild::validity_child(vortex_array::ArrayView<'_, V>) -> vortex_array::ArrayRef impl vortex_array::ValidityChild for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::validity_child(array: vortex_array::ArrayView<'_, vortex_array::arrays::Extension>) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Extension::validity_child(vortex_array::ArrayView<'_, vortex_array::arrays::Extension>) -> vortex_array::ArrayRef impl vortex_array::ValidityChild for vortex_array::arrays::patched::Patched -pub fn vortex_array::arrays::patched::Patched::validity_child(array: vortex_array::ArrayView<'_, vortex_array::arrays::patched::Patched>) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::patched::Patched::validity_child(vortex_array::ArrayView<'_, vortex_array::arrays::patched::Patched>) -> vortex_array::ArrayRef pub trait vortex_array::vtable::ValidityChildSliceHelper @@ -21210,103 +21658,103 @@ pub fn vortex_array::vtable::ValidityChildSliceHelper::unsliced_child_and_slice( pub trait vortex_array::vtable::ValidityVTable -pub fn vortex_array::vtable::ValidityVTable::validity(array: vortex_array::ArrayView<'_, V>) -> vortex_error::VortexResult +pub fn vortex_array::vtable::ValidityVTable::validity(vortex_array::ArrayView<'_, V>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Bool::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Chunked::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Constant::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Decimal::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::Filter -pub fn vortex_array::arrays::Filter::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Filter>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Filter::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Filter>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::FixedSizeList::validity(vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::List -pub fn vortex_array::arrays::List::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::List>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::List::validity(vortex_array::ArrayView<'_, vortex_array::arrays::List>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::ListView>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::ListView::validity(vortex_array::ArrayView<'_, vortex_array::arrays::ListView>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Masked::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Masked>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Primitive::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::Shared -pub fn vortex_array::arrays::Shared::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Shared>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Shared::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Shared>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Struct>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Struct::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Struct>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBin::validity(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBinView::validity(vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::Variant -pub fn vortex_array::arrays::Variant::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Variant>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Variant::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Variant>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::dict::Dict::validity(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::validity(_array: vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::null::Null::validity(vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>) -> vortex_error::VortexResult -impl vortex_array::ValidityVTable for vortex_array::arrays::scalar_fn::ScalarFnVTable +impl vortex_array::ValidityVTable for vortex_array::arrays::scalar_fn::ScalarFn -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::scalar_fn::ScalarFnVTable>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::scalar_fn::ScalarFn::validity(vortex_array::ArrayView<'_, vortex_array::arrays::scalar_fn::ScalarFn>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::slice::Slice -pub fn vortex_array::arrays::slice::Slice::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::slice::Slice>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::slice::Slice::validity(vortex_array::ArrayView<'_, vortex_array::arrays::slice::Slice>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::ValidityVTableFromChildSliceHelper where ::ArrayData: vortex_array::ValidityChildSliceHelper -pub fn vortex_array::ValidityVTableFromChildSliceHelper::validity(array: vortex_array::ArrayView<'_, V>) -> vortex_error::VortexResult +pub fn vortex_array::ValidityVTableFromChildSliceHelper::validity(vortex_array::ArrayView<'_, V>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::ValidityVTableFromChild where V: vortex_array::ValidityChild + vortex_array::VTable -pub fn vortex_array::ValidityVTableFromChild::validity(array: vortex_array::ArrayView<'_, V>) -> vortex_error::VortexResult +pub fn vortex_array::ValidityVTableFromChild::validity(vortex_array::ArrayView<'_, V>) -> vortex_error::VortexResult -pub fn vortex_array::vtable::child_to_validity(child: &core::option::Option, nullability: vortex_array::dtype::Nullability) -> vortex_array::validity::Validity +pub fn vortex_array::vtable::child_to_validity(core::option::Option<&vortex_array::ArrayRef>, vortex_array::dtype::Nullability) -> vortex_array::validity::Validity -pub fn vortex_array::vtable::patches_child(patches: &vortex_array::patches::Patches, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::vtable::patches_child(&vortex_array::patches::Patches, usize) -> vortex_array::ArrayRef -pub fn vortex_array::vtable::patches_child_name(idx: usize) -> &'static str +pub fn vortex_array::vtable::patches_child_name(usize) -> &'static str -pub fn vortex_array::vtable::patches_nchildren(patches: &vortex_array::patches::Patches) -> usize +pub fn vortex_array::vtable::patches_nchildren(&vortex_array::patches::Patches) -> usize -pub fn vortex_array::vtable::validity_nchildren(validity: &vortex_array::validity::Validity) -> usize +pub fn vortex_array::vtable::validity_nchildren(&vortex_array::validity::Validity) -> usize -pub fn vortex_array::vtable::validity_to_child(validity: &vortex_array::validity::Validity, len: usize) -> core::option::Option +pub fn vortex_array::vtable::validity_to_child(&vortex_array::validity::Validity, usize) -> core::option::Option pub type vortex_array::vtable::ArrayPluginRef = alloc::sync::Arc @@ -21408,7 +21856,7 @@ impl vortex_array::Canonical pub fn vortex_array::Canonical::dtype(&self) -> &vortex_array::dtype::DType -pub fn vortex_array::Canonical::empty(dtype: &vortex_array::dtype::DType) -> vortex_array::Canonical +pub fn vortex_array::Canonical::empty(&vortex_array::dtype::DType) -> vortex_array::Canonical pub fn vortex_array::Canonical::is_empty(&self) -> bool @@ -21420,19 +21868,19 @@ pub fn vortex_array::Canonical::clone(&self) -> vortex_array::Canonical impl core::convert::From for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from(value: vortex_array::Canonical) -> Self +pub fn vortex_array::ArrayRef::from(vortex_array::Canonical) -> Self impl core::convert::From> for vortex_array::Canonical -pub fn vortex_array::Canonical::from(value: vortex_array::CanonicalView<'_>) -> Self +pub fn vortex_array::Canonical::from(vortex_array::CanonicalView<'_>) -> Self impl core::fmt::Debug for vortex_array::Canonical -pub fn vortex_array::Canonical::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::Canonical::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::Executable for vortex_array::Canonical -pub fn vortex_array::Canonical::execute(array: vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::Canonical::execute(vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::IntoArray for vortex_array::Canonical @@ -21466,7 +21914,7 @@ pub fn vortex_array::CanonicalView<'_>::to_array_ref(&self) -> vortex_array::Arr impl core::convert::From> for vortex_array::Canonical -pub fn vortex_array::Canonical::from(value: vortex_array::CanonicalView<'_>) -> Self +pub fn vortex_array::Canonical::from(vortex_array::CanonicalView<'_>) -> Self impl<'a> core::clone::Clone for vortex_array::CanonicalView<'a> @@ -21474,7 +21922,7 @@ pub fn vortex_array::CanonicalView<'a>::clone(&self) -> vortex_array::CanonicalV impl<'a> core::fmt::Debug for vortex_array::CanonicalView<'a> -pub fn vortex_array::CanonicalView<'a>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::CanonicalView<'a>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl<'a> core::marker::Copy for vortex_array::CanonicalView<'a> @@ -21486,7 +21934,7 @@ pub vortex_array::Columnar::Constant(vortex_array::arrays::ConstantArray) impl vortex_array::Columnar -pub fn vortex_array::Columnar::constant>(scalar: S, len: usize) -> Self +pub fn vortex_array::Columnar::constant>(S, usize) -> Self pub fn vortex_array::Columnar::dtype(&self) -> &vortex_array::dtype::DType @@ -21496,7 +21944,7 @@ pub fn vortex_array::Columnar::len(&self) -> usize impl vortex_array::Executable for vortex_array::Columnar -pub fn vortex_array::Columnar::execute(array: vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::Columnar::execute(vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::IntoArray for vortex_array::Columnar @@ -21510,13 +21958,15 @@ pub vortex_array::ColumnarView::Constant(vortex_array::ArrayView<'a, vortex_arra pub enum vortex_array::ExecutionStep +pub vortex_array::ExecutionStep::AppendChild(usize) + pub vortex_array::ExecutionStep::Done pub vortex_array::ExecutionStep::ExecuteSlot(usize, vortex_array::DonePredicate) impl core::fmt::Debug for vortex_array::ExecutionStep -pub fn vortex_array::ExecutionStep::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::ExecutionStep::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub enum vortex_array::Precision @@ -21530,7 +21980,7 @@ pub fn vortex_array::Precision::clone(&self) -> vortex_array::Precision impl core::fmt::Debug for vortex_array::Precision -pub fn vortex_array::Precision::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::Precision::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_array::Precision @@ -21540,9 +21990,9 @@ impl vortex_array::matcher::Matcher for vortex_array::AnyCanonical pub type vortex_array::AnyCanonical::Match<'a> = vortex_array::CanonicalView<'a> -pub fn vortex_array::AnyCanonical::matches(array: &vortex_array::ArrayRef) -> bool +pub fn vortex_array::AnyCanonical::matches(&vortex_array::ArrayRef) -> bool -pub fn vortex_array::AnyCanonical::try_match<'a>(array: &'a vortex_array::ArrayRef) -> core::option::Option +pub fn vortex_array::AnyCanonical::try_match<'a>(&'a vortex_array::ArrayRef) -> core::option::Option pub struct vortex_array::AnyColumnar @@ -21550,65 +22000,65 @@ impl vortex_array::matcher::Matcher for vortex_array::AnyColumnar pub type vortex_array::AnyColumnar::Match<'a> = vortex_array::ColumnarView<'a> -pub fn vortex_array::AnyColumnar::matches(array: &vortex_array::ArrayRef) -> bool +pub fn vortex_array::AnyColumnar::matches(&vortex_array::ArrayRef) -> bool -pub fn vortex_array::AnyColumnar::try_match<'a>(array: &'a vortex_array::ArrayRef) -> core::option::Option +pub fn vortex_array::AnyColumnar::try_match<'a>(&'a vortex_array::ArrayRef) -> core::option::Option pub struct vortex_array::Array impl vortex_array::Array -pub fn vortex_array::Array::from_indices>(length: usize, indices: I, validity: vortex_array::validity::Validity) -> Self +pub fn vortex_array::Array::from_indices>(usize, I, vortex_array::validity::Validity) -> Self pub fn vortex_array::Array::into_bit_buffer(self) -> vortex_buffer::bit::buf::BitBuffer -pub fn vortex_array::Array::new(bits: vortex_buffer::bit::buf::BitBuffer, validity: vortex_array::validity::Validity) -> Self +pub fn vortex_array::Array::new(vortex_buffer::bit::buf::BitBuffer, vortex_array::validity::Validity) -> Self -pub fn vortex_array::Array::new_handle(handle: vortex_array::buffer::BufferHandle, offset: usize, len: usize, validity: vortex_array::validity::Validity) -> Self +pub fn vortex_array::Array::new_handle(vortex_array::buffer::BufferHandle, usize, usize, vortex_array::validity::Validity) -> Self -pub unsafe fn vortex_array::Array::new_unchecked(bits: vortex_buffer::bit::buf::BitBuffer, validity: vortex_array::validity::Validity) -> Self +pub unsafe fn vortex_array::Array::new_unchecked(vortex_buffer::bit::buf::BitBuffer, vortex_array::validity::Validity) -> Self -pub fn vortex_array::Array::try_new(bits: vortex_buffer::bit::buf::BitBuffer, validity: vortex_array::validity::Validity) -> vortex_error::VortexResult +pub fn vortex_array::Array::try_new(vortex_buffer::bit::buf::BitBuffer, vortex_array::validity::Validity) -> vortex_error::VortexResult -pub fn vortex_array::Array::try_new_from_handle(bits: vortex_array::buffer::BufferHandle, offset: usize, len: usize, validity: vortex_array::validity::Validity) -> vortex_error::VortexResult +pub fn vortex_array::Array::try_new_from_handle(vortex_array::buffer::BufferHandle, usize, usize, vortex_array::validity::Validity) -> vortex_error::VortexResult -pub fn vortex_array::Array::validate(bits: &vortex_buffer::bit::buf::BitBuffer, validity: &vortex_array::validity::Validity) -> vortex_error::VortexResult<()> +pub fn vortex_array::Array::validate(&vortex_buffer::bit::buf::BitBuffer, &vortex_array::validity::Validity) -> vortex_error::VortexResult<()> impl vortex_array::Array -pub fn vortex_array::Array::patch(self, patches: &vortex_array::patches::Patches, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::Array::patch(self, &vortex_array::patches::Patches, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::Array -pub unsafe fn vortex_array::Array::new_unchecked(chunks: alloc::vec::Vec, dtype: vortex_array::dtype::DType) -> Self +pub unsafe fn vortex_array::Array::new_unchecked(alloc::vec::Vec, vortex_array::dtype::DType) -> Self -pub fn vortex_array::Array::rechunk(&self, target_bytesize: u64, target_rowsize: usize) -> vortex_error::VortexResult +pub fn vortex_array::Array::rechunk(&self, u64, usize) -> vortex_error::VortexResult -pub fn vortex_array::Array::try_new(chunks: alloc::vec::Vec, dtype: vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::Array::try_new(alloc::vec::Vec, vortex_array::dtype::DType) -> vortex_error::VortexResult impl vortex_array::Array -pub fn vortex_array::Array::new(scalar: S, len: usize) -> Self where S: core::convert::Into +pub fn vortex_array::Array::new(S, usize) -> Self where S: core::convert::Into impl vortex_array::Array -pub fn vortex_array::Array::from_iter>(iter: I, decimal_dtype: vortex_array::dtype::DecimalDType) -> Self +pub fn vortex_array::Array::from_iter>(I, vortex_array::dtype::DecimalDType) -> Self -pub fn vortex_array::Array::from_option_iter>>(iter: I, decimal_dtype: vortex_array::dtype::DecimalDType) -> Self +pub fn vortex_array::Array::from_option_iter>>(I, vortex_array::dtype::DecimalDType) -> Self -pub fn vortex_array::Array::new(buffer: vortex_buffer::buffer::Buffer, decimal_dtype: vortex_array::dtype::DecimalDType, validity: vortex_array::validity::Validity) -> Self +pub fn vortex_array::Array::new(vortex_buffer::buffer::Buffer, vortex_array::dtype::DecimalDType, vortex_array::validity::Validity) -> Self -pub fn vortex_array::Array::new_handle(values: vortex_array::buffer::BufferHandle, values_type: vortex_array::dtype::DecimalType, decimal_dtype: vortex_array::dtype::DecimalDType, validity: vortex_array::validity::Validity) -> Self +pub fn vortex_array::Array::new_handle(vortex_array::buffer::BufferHandle, vortex_array::dtype::DecimalType, vortex_array::dtype::DecimalDType, vortex_array::validity::Validity) -> Self -pub unsafe fn vortex_array::Array::new_unchecked(buffer: vortex_buffer::buffer::Buffer, decimal_dtype: vortex_array::dtype::DecimalDType, validity: vortex_array::validity::Validity) -> Self +pub unsafe fn vortex_array::Array::new_unchecked(vortex_buffer::buffer::Buffer, vortex_array::dtype::DecimalDType, vortex_array::validity::Validity) -> Self -pub unsafe fn vortex_array::Array::new_unchecked_handle(values: vortex_array::buffer::BufferHandle, values_type: vortex_array::dtype::DecimalType, decimal_dtype: vortex_array::dtype::DecimalDType, validity: vortex_array::validity::Validity) -> Self +pub unsafe fn vortex_array::Array::new_unchecked_handle(vortex_array::buffer::BufferHandle, vortex_array::dtype::DecimalType, vortex_array::dtype::DecimalDType, vortex_array::validity::Validity) -> Self -pub fn vortex_array::Array::patch(self, patches: &vortex_array::patches::Patches, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::Array::patch(self, &vortex_array::patches::Patches, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::Array::try_new(buffer: vortex_buffer::buffer::Buffer, decimal_dtype: vortex_array::dtype::DecimalDType, validity: vortex_array::validity::Validity) -> vortex_error::VortexResult +pub fn vortex_array::Array::try_new(vortex_buffer::buffer::Buffer, vortex_array::dtype::DecimalDType, vortex_array::validity::Validity) -> vortex_error::VortexResult -pub fn vortex_array::Array::try_new_handle(values: vortex_array::buffer::BufferHandle, values_type: vortex_array::dtype::DecimalType, decimal_dtype: vortex_array::dtype::DecimalDType, validity: vortex_array::validity::Validity) -> vortex_error::VortexResult +pub fn vortex_array::Array::try_new_handle(vortex_array::buffer::BufferHandle, vortex_array::dtype::DecimalType, vortex_array::dtype::DecimalDType, vortex_array::validity::Validity) -> vortex_error::VortexResult impl vortex_array::Array @@ -21616,83 +22066,85 @@ pub fn vortex_array::Array::into_data_parts(self) impl vortex_array::Array -pub fn vortex_array::Array::new(ext_dtype: vortex_array::dtype::extension::ExtDTypeRef, storage_array: vortex_array::ArrayRef) -> Self +pub fn vortex_array::Array::new(vortex_array::dtype::extension::ExtDTypeRef, vortex_array::ArrayRef) -> Self + +pub fn vortex_array::Array::try_new(vortex_array::dtype::extension::ExtDTypeRef, vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::Array::try_new(ext_dtype: vortex_array::dtype::extension::ExtDTypeRef, storage_array: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::Array::try_new_from_vtable(V, ::Metadata, vortex_array::ArrayRef) -> vortex_error::VortexResult impl vortex_array::Array -pub fn vortex_array::Array::new(array: vortex_array::ArrayRef, mask: vortex_mask::Mask) -> Self +pub fn vortex_array::Array::new(vortex_array::ArrayRef, vortex_mask::Mask) -> Self -pub fn vortex_array::Array::try_new(array: vortex_array::ArrayRef, mask: vortex_mask::Mask) -> vortex_error::VortexResult +pub fn vortex_array::Array::try_new(vortex_array::ArrayRef, vortex_mask::Mask) -> vortex_error::VortexResult impl vortex_array::Array pub fn vortex_array::Array::into_data_parts(self) -> vortex_array::arrays::fixed_size_list::FixedSizeListDataParts -pub fn vortex_array::Array::new(elements: vortex_array::ArrayRef, list_size: u32, validity: vortex_array::validity::Validity, len: usize) -> Self +pub fn vortex_array::Array::new(vortex_array::ArrayRef, u32, vortex_array::validity::Validity, usize) -> Self -pub unsafe fn vortex_array::Array::new_unchecked(elements: vortex_array::ArrayRef, list_size: u32, validity: vortex_array::validity::Validity, len: usize) -> Self +pub unsafe fn vortex_array::Array::new_unchecked(vortex_array::ArrayRef, u32, vortex_array::validity::Validity, usize) -> Self -pub fn vortex_array::Array::try_new(elements: vortex_array::ArrayRef, list_size: u32, validity: vortex_array::validity::Validity, len: usize) -> vortex_error::VortexResult +pub fn vortex_array::Array::try_new(vortex_array::ArrayRef, u32, vortex_array::validity::Validity, usize) -> vortex_error::VortexResult impl vortex_array::Array pub fn vortex_array::Array::into_data_parts(self) -> vortex_array::arrays::list::ListDataParts -pub fn vortex_array::Array::new(elements: vortex_array::ArrayRef, offsets: vortex_array::ArrayRef, validity: vortex_array::validity::Validity) -> Self +pub fn vortex_array::Array::new(vortex_array::ArrayRef, vortex_array::ArrayRef, vortex_array::validity::Validity) -> Self -pub unsafe fn vortex_array::Array::new_unchecked(elements: vortex_array::ArrayRef, offsets: vortex_array::ArrayRef, validity: vortex_array::validity::Validity) -> Self +pub unsafe fn vortex_array::Array::new_unchecked(vortex_array::ArrayRef, vortex_array::ArrayRef, vortex_array::validity::Validity) -> Self -pub fn vortex_array::Array::try_new(elements: vortex_array::ArrayRef, offsets: vortex_array::ArrayRef, validity: vortex_array::validity::Validity) -> vortex_error::VortexResult +pub fn vortex_array::Array::try_new(vortex_array::ArrayRef, vortex_array::ArrayRef, vortex_array::validity::Validity) -> vortex_error::VortexResult impl vortex_array::Array pub fn vortex_array::Array::into_data_parts(self) -> vortex_array::arrays::listview::ListViewDataParts -pub fn vortex_array::Array::new(elements: vortex_array::ArrayRef, offsets: vortex_array::ArrayRef, sizes: vortex_array::ArrayRef, validity: vortex_array::validity::Validity) -> Self +pub fn vortex_array::Array::new(vortex_array::ArrayRef, vortex_array::ArrayRef, vortex_array::ArrayRef, vortex_array::validity::Validity) -> Self -pub unsafe fn vortex_array::Array::new_unchecked(elements: vortex_array::ArrayRef, offsets: vortex_array::ArrayRef, sizes: vortex_array::ArrayRef, validity: vortex_array::validity::Validity) -> Self +pub unsafe fn vortex_array::Array::new_unchecked(vortex_array::ArrayRef, vortex_array::ArrayRef, vortex_array::ArrayRef, vortex_array::validity::Validity) -> Self -pub fn vortex_array::Array::try_new(elements: vortex_array::ArrayRef, offsets: vortex_array::ArrayRef, sizes: vortex_array::ArrayRef, validity: vortex_array::validity::Validity) -> vortex_error::VortexResult +pub fn vortex_array::Array::try_new(vortex_array::ArrayRef, vortex_array::ArrayRef, vortex_array::ArrayRef, vortex_array::validity::Validity) -> vortex_error::VortexResult -pub unsafe fn vortex_array::Array::with_zero_copy_to_list(self, is_zctl: bool) -> Self +pub unsafe fn vortex_array::Array::with_zero_copy_to_list(self, bool) -> Self impl vortex_array::Array -pub fn vortex_array::Array::rebuild(&self, mode: vortex_array::arrays::listview::ListViewRebuildMode) -> vortex_error::VortexResult +pub fn vortex_array::Array::rebuild(&self, vortex_array::arrays::listview::ListViewRebuildMode) -> vortex_error::VortexResult impl vortex_array::Array -pub fn vortex_array::Array::try_new(child: vortex_array::ArrayRef, validity: vortex_array::validity::Validity) -> vortex_error::VortexResult +pub fn vortex_array::Array::try_new(vortex_array::ArrayRef, vortex_array::validity::Validity) -> vortex_error::VortexResult impl vortex_array::Array -pub fn vortex_array::Array::empty(nullability: vortex_array::dtype::Nullability) -> Self +pub fn vortex_array::Array::empty(vortex_array::dtype::Nullability) -> Self -pub fn vortex_array::Array::from_buffer_handle(handle: vortex_array::buffer::BufferHandle, ptype: vortex_array::dtype::PType, validity: vortex_array::validity::Validity) -> Self +pub fn vortex_array::Array::from_buffer_handle(vortex_array::buffer::BufferHandle, vortex_array::dtype::PType, vortex_array::validity::Validity) -> Self -pub fn vortex_array::Array::from_byte_buffer(buffer: vortex_buffer::ByteBuffer, ptype: vortex_array::dtype::PType, validity: vortex_array::validity::Validity) -> Self +pub fn vortex_array::Array::from_byte_buffer(vortex_buffer::ByteBuffer, vortex_array::dtype::PType, vortex_array::validity::Validity) -> Self -pub fn vortex_array::Array::from_values_byte_buffer(valid_elems_buffer: vortex_buffer::ByteBuffer, ptype: vortex_array::dtype::PType, validity: vortex_array::validity::Validity, n_rows: usize) -> Self +pub fn vortex_array::Array::from_values_byte_buffer(vortex_buffer::ByteBuffer, vortex_array::dtype::PType, vortex_array::validity::Validity, usize) -> Self pub fn vortex_array::Array::into_data_parts(self) -> vortex_array::arrays::primitive::PrimitiveDataParts -pub fn vortex_array::Array::map_each_with_validity(self, f: F) -> vortex_error::VortexResult where T: vortex_array::dtype::NativePType, R: vortex_array::dtype::NativePType, F: core::ops::function::FnMut((T, bool)) -> R +pub fn vortex_array::Array::map_each_with_validity(self, F) -> vortex_error::VortexResult where T: vortex_array::dtype::NativePType, R: vortex_array::dtype::NativePType, F: core::ops::function::FnMut((T, bool)) -> R -pub fn vortex_array::Array::new(buffer: impl core::convert::Into>, validity: vortex_array::validity::Validity) -> Self +pub fn vortex_array::Array::new(impl core::convert::Into>, vortex_array::validity::Validity) -> Self -pub unsafe fn vortex_array::Array::new_unchecked(buffer: vortex_buffer::buffer::Buffer, validity: vortex_array::validity::Validity) -> Self +pub unsafe fn vortex_array::Array::new_unchecked(vortex_buffer::buffer::Buffer, vortex_array::validity::Validity) -> Self -pub unsafe fn vortex_array::Array::new_unchecked_from_handle(handle: vortex_array::buffer::BufferHandle, ptype: vortex_array::dtype::PType, validity: vortex_array::validity::Validity) -> Self +pub unsafe fn vortex_array::Array::new_unchecked_from_handle(vortex_array::buffer::BufferHandle, vortex_array::dtype::PType, vortex_array::validity::Validity) -> Self -pub fn vortex_array::Array::try_new(buffer: vortex_buffer::buffer::Buffer, validity: vortex_array::validity::Validity) -> vortex_error::VortexResult +pub fn vortex_array::Array::try_new(vortex_buffer::buffer::Buffer, vortex_array::validity::Validity) -> vortex_error::VortexResult -pub fn vortex_array::Array::validate(buffer: &vortex_buffer::buffer::Buffer, validity: &vortex_array::validity::Validity) -> vortex_error::VortexResult<()> +pub fn vortex_array::Array::validate(&vortex_buffer::buffer::Buffer, &vortex_array::validity::Validity) -> vortex_error::VortexResult<()> impl vortex_array::Array -pub fn vortex_array::Array::from_option_iter>>(iter: I) -> Self +pub fn vortex_array::Array::from_option_iter>>(I) -> Self pub fn vortex_array::Array::into_buffer(self) -> vortex_buffer::buffer::Buffer @@ -21704,7 +22156,7 @@ pub fn vortex_array::Array::try_into_buffer_mut impl vortex_array::Array -pub fn vortex_array::Array::patch(self, patches: &vortex_array::patches::Patches, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::Array::patch(self, &vortex_array::patches::Patches, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::Array @@ -21712,162 +22164,162 @@ pub fn vortex_array::Array::top_value(&self) -> impl vortex_array::Array -pub fn vortex_array::Array::new(source: vortex_array::ArrayRef) -> Self +pub fn vortex_array::Array::new(vortex_array::ArrayRef) -> Self impl vortex_array::Array -pub fn vortex_array::Array::from_fields>(items: &[(N, vortex_array::ArrayRef)]) -> vortex_error::VortexResult +pub fn vortex_array::Array::from_fields>(&[(N, vortex_array::ArrayRef)]) -> vortex_error::VortexResult pub fn vortex_array::Array::into_data_parts(self) -> vortex_array::arrays::struct_::StructDataParts -pub fn vortex_array::Array::new(names: vortex_array::dtype::FieldNames, fields: impl core::convert::Into>, length: usize, validity: vortex_array::validity::Validity) -> Self +pub fn vortex_array::Array::new(vortex_array::dtype::FieldNames, impl core::convert::Into>, usize, vortex_array::validity::Validity) -> Self -pub fn vortex_array::Array::new_fieldless_with_len(len: usize) -> Self +pub fn vortex_array::Array::new_fieldless_with_len(usize) -> Self -pub unsafe fn vortex_array::Array::new_unchecked(fields: impl core::convert::Into>, dtype: vortex_array::dtype::StructFields, length: usize, validity: vortex_array::validity::Validity) -> Self +pub unsafe fn vortex_array::Array::new_unchecked(impl core::convert::Into>, vortex_array::dtype::StructFields, usize, vortex_array::validity::Validity) -> Self -pub fn vortex_array::Array::project(&self, projection: &[vortex_array::dtype::FieldName]) -> vortex_error::VortexResult +pub fn vortex_array::Array::project(&self, &[vortex_array::dtype::FieldName]) -> vortex_error::VortexResult -pub fn vortex_array::Array::remove_column(&self, name: impl core::convert::Into) -> core::option::Option<(Self, vortex_array::ArrayRef)> +pub fn vortex_array::Array::remove_column(&self, impl core::convert::Into) -> core::option::Option<(Self, vortex_array::ArrayRef)> -pub fn vortex_array::Array::try_from_iter, A: vortex_array::IntoArray, T: core::iter::traits::collect::IntoIterator>(iter: T) -> vortex_error::VortexResult +pub fn vortex_array::Array::remove_column_owned(&self, impl core::convert::Into) -> core::option::Option<(Self, vortex_array::ArrayRef)> -pub fn vortex_array::Array::try_from_iter_with_validity, A: vortex_array::IntoArray, T: core::iter::traits::collect::IntoIterator>(iter: T, validity: vortex_array::validity::Validity) -> vortex_error::VortexResult +pub fn vortex_array::Array::try_concat(impl core::iter::traits::collect::IntoIterator) -> vortex_error::VortexResult where T: core::borrow::Borrow> -pub fn vortex_array::Array::try_new(names: vortex_array::dtype::FieldNames, fields: impl core::convert::Into>, length: usize, validity: vortex_array::validity::Validity) -> vortex_error::VortexResult +pub fn vortex_array::Array::try_from_iter, A: vortex_array::IntoArray, T: core::iter::traits::collect::IntoIterator>(T) -> vortex_error::VortexResult -pub fn vortex_array::Array::try_new_with_dtype(fields: impl core::convert::Into>, dtype: vortex_array::dtype::StructFields, length: usize, validity: vortex_array::validity::Validity) -> vortex_error::VortexResult +pub fn vortex_array::Array::try_from_iter_with_validity, A: vortex_array::IntoArray, T: core::iter::traits::collect::IntoIterator>(T, vortex_array::validity::Validity) -> vortex_error::VortexResult -impl vortex_array::Array +pub fn vortex_array::Array::try_new(vortex_array::dtype::FieldNames, impl core::convert::Into>, usize, vortex_array::validity::Validity) -> vortex_error::VortexResult -pub fn vortex_array::Array::into_record_batch_with_schema(self, schema: impl core::convert::AsRef) -> vortex_error::VortexResult +pub fn vortex_array::Array::try_new_with_dtype(impl core::convert::Into>, vortex_array::dtype::StructFields, usize, vortex_array::validity::Validity) -> vortex_error::VortexResult -impl vortex_array::Array +pub fn vortex_array::Array::with_column(&self, impl core::convert::Into, vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::Array::remove_column_owned(&self, name: impl core::convert::Into) -> core::option::Option<(Self, vortex_array::ArrayRef)> +impl vortex_array::Array -pub fn vortex_array::Array::with_column(&self, name: impl core::convert::Into, array: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::Array::into_record_batch_with_schema(self, impl core::convert::AsRef) -> vortex_error::VortexResult impl vortex_array::Array -pub fn vortex_array::Array::from_bytes(value: alloc::vec::Vec<&[u8]>) -> Self +pub fn vortex_array::Array::from_bytes(alloc::vec::Vec<&[u8]>) -> Self -pub fn vortex_array::Array::from_iter, I: core::iter::traits::collect::IntoIterator>>(iter: I, dtype: vortex_array::dtype::DType) -> Self +pub fn vortex_array::Array::from_iter, I: core::iter::traits::collect::IntoIterator>>(I, vortex_array::dtype::DType) -> Self -pub fn vortex_array::Array::from_iter_nonnull, I: core::iter::traits::collect::IntoIterator>(iter: I, dtype: vortex_array::dtype::DType) -> Self +pub fn vortex_array::Array::from_iter_nonnull, I: core::iter::traits::collect::IntoIterator>(I, vortex_array::dtype::DType) -> Self -pub fn vortex_array::Array::from_nullable_bytes(value: alloc::vec::Vec>) -> Self +pub fn vortex_array::Array::from_nullable_bytes(alloc::vec::Vec>) -> Self -pub fn vortex_array::Array::from_nullable_strs(value: alloc::vec::Vec>) -> Self +pub fn vortex_array::Array::from_nullable_strs(alloc::vec::Vec>) -> Self -pub fn vortex_array::Array::from_strs(value: alloc::vec::Vec<&str>) -> Self +pub fn vortex_array::Array::from_strs(alloc::vec::Vec<&str>) -> Self -pub fn vortex_array::Array::from_vec>(vec: alloc::vec::Vec, dtype: vortex_array::dtype::DType) -> Self +pub fn vortex_array::Array::from_vec>(alloc::vec::Vec, vortex_array::dtype::DType) -> Self pub fn vortex_array::Array::into_data_parts(self) -> vortex_array::arrays::varbin::VarBinDataParts impl vortex_array::Array -pub fn vortex_array::Array::new(offsets: vortex_array::ArrayRef, bytes: vortex_buffer::ByteBuffer, dtype: vortex_array::dtype::DType, validity: vortex_array::validity::Validity) -> Self +pub fn vortex_array::Array::new(vortex_array::ArrayRef, vortex_buffer::ByteBuffer, vortex_array::dtype::DType, vortex_array::validity::Validity) -> Self -pub unsafe fn vortex_array::Array::new_unchecked(offsets: vortex_array::ArrayRef, bytes: vortex_buffer::ByteBuffer, dtype: vortex_array::dtype::DType, validity: vortex_array::validity::Validity) -> Self +pub unsafe fn vortex_array::Array::new_unchecked(vortex_array::ArrayRef, vortex_buffer::ByteBuffer, vortex_array::dtype::DType, vortex_array::validity::Validity) -> Self -pub unsafe fn vortex_array::Array::new_unchecked_from_handle(offsets: vortex_array::ArrayRef, bytes: vortex_array::buffer::BufferHandle, dtype: vortex_array::dtype::DType, validity: vortex_array::validity::Validity) -> Self +pub unsafe fn vortex_array::Array::new_unchecked_from_handle(vortex_array::ArrayRef, vortex_array::buffer::BufferHandle, vortex_array::dtype::DType, vortex_array::validity::Validity) -> Self -pub fn vortex_array::Array::try_new(offsets: vortex_array::ArrayRef, bytes: vortex_buffer::ByteBuffer, dtype: vortex_array::dtype::DType, validity: vortex_array::validity::Validity) -> vortex_error::VortexResult +pub fn vortex_array::Array::try_new(vortex_array::ArrayRef, vortex_buffer::ByteBuffer, vortex_array::dtype::DType, vortex_array::validity::Validity) -> vortex_error::VortexResult impl vortex_array::Array pub fn vortex_array::Array::compact_buffers(&self) -> vortex_error::VortexResult -pub fn vortex_array::Array::compact_with_threshold(&self, buffer_utilization_threshold: f64) -> vortex_error::VortexResult +pub fn vortex_array::Array::compact_with_threshold(&self, f64) -> vortex_error::VortexResult impl vortex_array::Array -pub fn vortex_array::Array::from_iter, I: core::iter::traits::collect::IntoIterator>>(iter: I, dtype: vortex_array::dtype::DType) -> Self +pub fn vortex_array::Array::from_iter, I: core::iter::traits::collect::IntoIterator>>(I, vortex_array::dtype::DType) -> Self -pub fn vortex_array::Array::from_iter_bin, I: core::iter::traits::collect::IntoIterator>(iter: I) -> Self +pub fn vortex_array::Array::from_iter_bin, I: core::iter::traits::collect::IntoIterator>(I) -> Self -pub fn vortex_array::Array::from_iter_nullable_bin, I: core::iter::traits::collect::IntoIterator>>(iter: I) -> Self +pub fn vortex_array::Array::from_iter_nullable_bin, I: core::iter::traits::collect::IntoIterator>>(I) -> Self -pub fn vortex_array::Array::from_iter_nullable_str, I: core::iter::traits::collect::IntoIterator>>(iter: I) -> Self +pub fn vortex_array::Array::from_iter_nullable_str, I: core::iter::traits::collect::IntoIterator>>(I) -> Self -pub fn vortex_array::Array::from_iter_str, I: core::iter::traits::collect::IntoIterator>(iter: I) -> Self +pub fn vortex_array::Array::from_iter_str, I: core::iter::traits::collect::IntoIterator>(I) -> Self pub fn vortex_array::Array::into_data_parts(self) -> vortex_array::arrays::varbinview::VarBinViewDataParts -pub fn vortex_array::Array::new_handle(views: vortex_array::buffer::BufferHandle, buffers: alloc::sync::Arc<[vortex_array::buffer::BufferHandle]>, dtype: vortex_array::dtype::DType, validity: vortex_array::validity::Validity) -> Self +pub fn vortex_array::Array::new_handle(vortex_array::buffer::BufferHandle, alloc::sync::Arc<[vortex_array::buffer::BufferHandle]>, vortex_array::dtype::DType, vortex_array::validity::Validity) -> Self -pub unsafe fn vortex_array::Array::new_handle_unchecked(views: vortex_array::buffer::BufferHandle, buffers: alloc::sync::Arc<[vortex_array::buffer::BufferHandle]>, dtype: vortex_array::dtype::DType, validity: vortex_array::validity::Validity) -> Self +pub unsafe fn vortex_array::Array::new_handle_unchecked(vortex_array::buffer::BufferHandle, alloc::sync::Arc<[vortex_array::buffer::BufferHandle]>, vortex_array::dtype::DType, vortex_array::validity::Validity) -> Self -pub unsafe fn vortex_array::Array::new_unchecked(views: vortex_buffer::buffer::Buffer, buffers: alloc::sync::Arc<[vortex_buffer::ByteBuffer]>, dtype: vortex_array::dtype::DType, validity: vortex_array::validity::Validity) -> Self +pub unsafe fn vortex_array::Array::new_unchecked(vortex_buffer::buffer::Buffer, alloc::sync::Arc<[vortex_buffer::ByteBuffer]>, vortex_array::dtype::DType, vortex_array::validity::Validity) -> Self -pub fn vortex_array::Array::try_new(views: vortex_buffer::buffer::Buffer, buffers: alloc::sync::Arc<[vortex_buffer::ByteBuffer]>, dtype: vortex_array::dtype::DType, validity: vortex_array::validity::Validity) -> vortex_error::VortexResult +pub fn vortex_array::Array::try_new(vortex_buffer::buffer::Buffer, alloc::sync::Arc<[vortex_buffer::ByteBuffer]>, vortex_array::dtype::DType, vortex_array::validity::Validity) -> vortex_error::VortexResult impl vortex_array::Array -pub fn vortex_array::Array::new(child: vortex_array::ArrayRef) -> Self +pub fn vortex_array::Array::new(vortex_array::ArrayRef) -> Self impl vortex_array::Array -pub fn vortex_array::Array::new(codes: vortex_array::ArrayRef, values: vortex_array::ArrayRef) -> Self +pub fn vortex_array::Array::new(vortex_array::ArrayRef, vortex_array::ArrayRef) -> Self -pub unsafe fn vortex_array::Array::new_unchecked(codes: vortex_array::ArrayRef, values: vortex_array::ArrayRef) -> Self +pub unsafe fn vortex_array::Array::new_unchecked(vortex_array::ArrayRef, vortex_array::ArrayRef) -> Self -pub unsafe fn vortex_array::Array::set_all_values_referenced(self, all_values_referenced: bool) -> Self +pub unsafe fn vortex_array::Array::set_all_values_referenced(self, bool) -> Self -pub fn vortex_array::Array::try_new(codes: vortex_array::ArrayRef, values: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::Array::try_new(vortex_array::ArrayRef, vortex_array::ArrayRef) -> vortex_error::VortexResult impl vortex_array::Array -pub fn vortex_array::Array::new(len: usize) -> Self +pub fn vortex_array::Array::new(usize) -> Self -impl vortex_array::Array +impl vortex_array::Array -pub fn vortex_array::Array::try_new(scalar_fn: vortex_array::scalar_fn::ScalarFnRef, children: alloc::vec::Vec, len: usize) -> vortex_error::VortexResult +pub fn vortex_array::Array::try_new(vortex_array::scalar_fn::ScalarFnRef, alloc::vec::Vec, usize) -> vortex_error::VortexResult impl vortex_array::Array -pub fn vortex_array::Array::new(child: vortex_array::ArrayRef, range: core::ops::range::Range) -> Self +pub fn vortex_array::Array::new(vortex_array::ArrayRef, core::ops::range::Range) -> Self -pub fn vortex_array::Array::try_new(child: vortex_array::ArrayRef, range: core::ops::range::Range) -> vortex_error::VortexResult +pub fn vortex_array::Array::try_new(vortex_array::ArrayRef, core::ops::range::Range) -> vortex_error::VortexResult impl vortex_array::Array -pub fn vortex_array::Array::all_invalid(&self) -> vortex_error::VortexResult +pub fn vortex_array::Array::all_invalid(&self, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::Array::all_valid(&self) -> vortex_error::VortexResult +pub fn vortex_array::Array::all_valid(&self, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::Array::append_to_builder(&self, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::Array::append_to_builder(&self, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> pub fn vortex_array::Array::as_constant(&self) -> core::option::Option -pub fn vortex_array::Array::filter(&self, mask: vortex_mask::Mask) -> vortex_error::VortexResult +pub fn vortex_array::Array::execute_scalar(&self, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult + +pub fn vortex_array::Array::filter(&self, vortex_mask::Mask) -> vortex_error::VortexResult -pub fn vortex_array::Array::invalid_count(&self) -> vortex_error::VortexResult +pub fn vortex_array::Array::invalid_count(&self, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::Array::is_invalid(&self, index: usize) -> vortex_error::VortexResult +pub fn vortex_array::Array::is_invalid(&self, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::Array::is_valid(&self, index: usize) -> vortex_error::VortexResult +pub fn vortex_array::Array::is_valid(&self, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_array::Array::nbuffers(&self) -> usize pub fn vortex_array::Array::nbytes(&self) -> u64 -pub fn vortex_array::Array::scalar_at(&self, index: usize) -> vortex_error::VortexResult +pub fn vortex_array::Array::scalar_at(&self, usize) -> vortex_error::VortexResult -pub fn vortex_array::Array::slice(&self, range: core::ops::range::Range) -> vortex_error::VortexResult +pub fn vortex_array::Array::slice(&self, core::ops::range::Range) -> vortex_error::VortexResult -pub fn vortex_array::Array::take(&self, indices: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::Array::take(&self, vortex_array::ArrayRef) -> vortex_error::VortexResult pub fn vortex_array::Array::to_canonical(&self) -> vortex_error::VortexResult -pub fn vortex_array::Array::valid_count(&self) -> vortex_error::VortexResult +pub fn vortex_array::Array::valid_count(&self, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_array::Array::validity(&self) -> vortex_error::VortexResult -pub fn vortex_array::Array::validity_mask(&self) -> vortex_error::VortexResult - impl vortex_array::Array pub fn vortex_array::Array::as_array(&self) -> &vortex_array::ArrayRef @@ -21876,6 +22328,8 @@ pub fn vortex_array::Array::as_view(&self) -> vortex_array::ArrayView<'_, V> pub fn vortex_array::Array::data(&self) -> &::ArrayData +pub fn vortex_array::Array::data_mut(&mut self) -> core::option::Option<&mut ::ArrayData> + pub fn vortex_array::Array::dtype(&self) -> &vortex_array::dtype::DType pub fn vortex_array::Array::encoding_id(&self) -> vortex_array::ArrayId @@ -21890,87 +22344,91 @@ pub fn vortex_array::Array::slots(&self) -> &[core::option::Option::statistics(&self) -> vortex_array::stats::StatsSetRef<'_> -pub fn vortex_array::Array::try_from_array_ref(array: vortex_array::ArrayRef) -> core::result::Result +pub fn vortex_array::Array::try_from_array_ref(vortex_array::ArrayRef) -> core::result::Result -pub fn vortex_array::Array::try_from_parts(new: vortex_array::ArrayParts) -> vortex_error::VortexResult +pub fn vortex_array::Array::try_from_parts(vortex_array::ArrayParts) -> vortex_error::VortexResult pub fn vortex_array::Array::try_into_parts(self) -> core::result::Result, Self> -pub fn vortex_array::Array::with_stats_set(self, stats: vortex_array::stats::StatsSet) -> Self +pub fn vortex_array::Array::with_stats_set(self, vortex_array::stats::StatsSet) -> Self impl core::convert::From> for vortex_array::Array -pub fn vortex_array::Array::from(value: alloc::vec::Vec<&[u8]>) -> Self +pub fn vortex_array::Array::from(alloc::vec::Vec<&[u8]>) -> Self impl core::convert::From> for vortex_array::Array -pub fn vortex_array::Array::from(value: alloc::vec::Vec<&str>) -> Self +pub fn vortex_array::Array::from(alloc::vec::Vec<&str>) -> Self impl core::convert::From> for vortex_array::Array -pub fn vortex_array::Array::from(value: alloc::vec::Vec) -> Self +pub fn vortex_array::Array::from(alloc::vec::Vec) -> Self impl core::convert::From>> for vortex_array::Array -pub fn vortex_array::Array::from(value: alloc::vec::Vec>) -> Self +pub fn vortex_array::Array::from(alloc::vec::Vec>) -> Self impl core::convert::From>> for vortex_array::Array -pub fn vortex_array::Array::from(value: alloc::vec::Vec>) -> Self +pub fn vortex_array::Array::from(alloc::vec::Vec>) -> Self impl core::convert::From>> for vortex_array::Array -pub fn vortex_array::Array::from(value: alloc::vec::Vec>) -> Self +pub fn vortex_array::Array::from(alloc::vec::Vec>) -> Self impl core::convert::From>> for vortex_array::Array -pub fn vortex_array::Array::from(value: alloc::vec::Vec>) -> Self +pub fn vortex_array::Array::from(alloc::vec::Vec>) -> Self impl core::convert::From>>> for vortex_array::Array -pub fn vortex_array::Array::from(value: alloc::vec::Vec>>) -> Self +pub fn vortex_array::Array::from(alloc::vec::Vec>>) -> Self impl core::convert::TryFrom> for vortex_array::arrays::datetime::TemporalData pub type vortex_array::arrays::datetime::TemporalData::Error = vortex_error::VortexError -pub fn vortex_array::arrays::datetime::TemporalData::try_from(ext: vortex_array::arrays::ExtensionArray) -> core::result::Result +pub fn vortex_array::arrays::datetime::TemporalData::try_from(vortex_array::arrays::ExtensionArray) -> core::result::Result impl core::iter::traits::collect::FromIterator> for vortex_array::Array -pub fn vortex_array::Array::from_iter>>(iter: T) -> Self +pub fn vortex_array::Array::from_iter>>(T) -> Self impl core::iter::traits::collect::FromIterator> for vortex_array::Array -pub fn vortex_array::Array::from_iter>>(iter: T) -> Self +pub fn vortex_array::Array::from_iter>>(T) -> Self impl core::iter::traits::collect::FromIterator>> for vortex_array::Array -pub fn vortex_array::Array::from_iter>>>(iter: T) -> Self +pub fn vortex_array::Array::from_iter>>>(T) -> Self impl core::iter::traits::collect::FromIterator>> for vortex_array::Array -pub fn vortex_array::Array::from_iter>>>(iter: T) -> Self +pub fn vortex_array::Array::from_iter>>>(T) -> Self impl core::iter::traits::collect::FromIterator for vortex_array::Array -pub fn vortex_array::Array::from_iter>(iter: T) -> Self +pub fn vortex_array::Array::from_iter>(T) -> Self + +impl vortex_array::arrays::dict::DictOwnedExt for vortex_array::Array + +pub fn vortex_array::Array::into_parts(self) -> vortex_array::arrays::dict::DictParts impl<'a> core::iter::traits::collect::FromIterator> for vortex_array::Array -pub fn vortex_array::Array::from_iter>>(iter: T) -> Self +pub fn vortex_array::Array::from_iter>>(T) -> Self impl<'a> core::iter::traits::collect::FromIterator> for vortex_array::Array -pub fn vortex_array::Array::from_iter>>(iter: T) -> Self +pub fn vortex_array::Array::from_iter>>(T) -> Self impl<'a> core::iter::traits::collect::FromIterator> for vortex_array::Array -pub fn vortex_array::Array::from_iter>>(iter: T) -> Self +pub fn vortex_array::Array::from_iter>>(T) -> Self impl<'a> core::iter::traits::collect::FromIterator> for vortex_array::Array -pub fn vortex_array::Array::from_iter>>(iter: T) -> Self +pub fn vortex_array::Array::from_iter>>(T) -> Self impl core::clone::Clone for vortex_array::Array @@ -21982,11 +22440,11 @@ pub fn vortex_array::Array::as_ref(&self) -> &vortex_array::ArrayRef impl core::convert::From> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from(value: vortex_array::Array) -> vortex_array::ArrayRef +pub fn vortex_array::ArrayRef::from(vortex_array::Array) -> vortex_array::ArrayRef impl core::fmt::Debug for vortex_array::Array -pub fn vortex_array::Array::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::Array::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::ops::deref::Deref for vortex_array::Array @@ -22016,19 +22474,19 @@ pub vortex_array::ArrayParts::vtable: V impl vortex_array::ArrayParts -pub fn vortex_array::ArrayParts::new(vtable: V, dtype: vortex_array::dtype::DType, len: usize, data: ::ArrayData) -> Self +pub fn vortex_array::ArrayParts::new(V, vortex_array::dtype::DType, usize, ::ArrayData) -> Self -pub fn vortex_array::ArrayParts::with_slots(self, slots: alloc::vec::Vec>) -> Self +pub fn vortex_array::ArrayParts::with_slots(self, alloc::vec::Vec>) -> Self pub struct vortex_array::ArrayRef(_) impl vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::all_invalid(&self) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::all_invalid(&self, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::all_valid(&self) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::all_valid(&self, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::append_to_builder(&self, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::ArrayRef::append_to_builder(&self, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> pub fn vortex_array::ArrayRef::as_(&self) -> ::Match @@ -22056,13 +22514,15 @@ pub fn vortex_array::ArrayRef::dtype(&self) -> &vortex_array::dtype::DType pub fn vortex_array::ArrayRef::encoding_id(&self) -> vortex_array::ArrayId -pub fn vortex_array::ArrayRef::execute_parent(&self, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::ArrayRef::execute_parent(&self, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> -pub fn vortex_array::ArrayRef::filter(&self, mask: vortex_mask::Mask) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::execute_scalar(&self, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult + +pub fn vortex_array::ArrayRef::filter(&self, vortex_mask::Mask) -> vortex_error::VortexResult pub fn vortex_array::ArrayRef::into_canonical(self) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::invalid_count(&self) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::invalid_count(&self, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_array::ArrayRef::is(&self) -> bool @@ -22074,15 +22534,13 @@ pub fn vortex_array::ArrayRef::is_empty(&self) -> bool pub fn vortex_array::ArrayRef::is_host(&self) -> bool -pub fn vortex_array::ArrayRef::is_invalid(&self, index: usize) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::is_invalid(&self, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::is_valid(&self, index: usize) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::is_valid(&self, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_array::ArrayRef::len(&self) -> usize -pub fn vortex_array::ArrayRef::metadata(&self, session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> - -pub fn vortex_array::ArrayRef::metadata_fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::ArrayRef::metadata_fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn vortex_array::ArrayRef::named_buffers(&self) -> alloc::vec::Vec<(alloc::string::String, vortex_array::buffer::BufferHandle)> @@ -22096,41 +22554,39 @@ pub fn vortex_array::ArrayRef::nbytes(&self) -> u64 pub fn vortex_array::ArrayRef::nchildren(&self) -> usize -pub fn vortex_array::ArrayRef::nth_child(&self, idx: usize) -> core::option::Option +pub fn vortex_array::ArrayRef::nth_child(&self, usize) -> core::option::Option pub fn vortex_array::ArrayRef::reduce(&self) -> vortex_error::VortexResult> -pub fn vortex_array::ArrayRef::reduce_parent(&self, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::ArrayRef::reduce_parent(&self, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::ArrayRef::scalar_at(&self, index: usize) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::scalar_at(&self, usize) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::slice(&self, range: core::ops::range::Range) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::slice(&self, core::ops::range::Range) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::slot_name(&self, idx: usize) -> alloc::string::String +pub fn vortex_array::ArrayRef::slot_name(&self, usize) -> alloc::string::String pub fn vortex_array::ArrayRef::slots(&self) -> &[core::option::Option] pub fn vortex_array::ArrayRef::statistics(&self) -> vortex_array::stats::StatsSetRef<'_> -pub fn vortex_array::ArrayRef::take(&self, indices: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::take(&self, vortex_array::ArrayRef) -> vortex_error::VortexResult pub fn vortex_array::ArrayRef::to_canonical(&self) -> vortex_error::VortexResult pub fn vortex_array::ArrayRef::try_downcast(self) -> core::result::Result, vortex_array::ArrayRef> -pub fn vortex_array::ArrayRef::valid_count(&self) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::valid_count(&self, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub fn vortex_array::ArrayRef::validity(&self) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::validity_mask(&self) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::with_slot(self, usize, vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::with_slot(self, slot_idx: usize, replacement: vortex_array::ArrayRef) -> vortex_error::VortexResult - -pub fn vortex_array::ArrayRef::with_slots(self, slots: alloc::vec::Vec>) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::with_slots(self, alloc::vec::Vec>) -> vortex_error::VortexResult impl vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::apply(self, expr: &vortex_array::expr::Expression) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::apply(self, &vortex_array::expr::Expression) -> vortex_error::VortexResult impl vortex_array::ArrayRef @@ -22152,11 +22608,11 @@ pub fn vortex_array::ArrayRef::as_struct_typed(&self) -> vortex_array::variants: pub fn vortex_array::ArrayRef::as_utf8_typed(&self) -> vortex_array::variants::Utf8Typed<'_> -pub fn vortex_array::ArrayRef::try_to_mask_fill_null_false(&self, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::try_to_mask_fill_null_false(&self, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::display_as(&self, options: vortex_array::display::DisplayOptions) -> impl core::fmt::Display +pub fn vortex_array::ArrayRef::display_as(&self, vortex_array::display::DisplayOptions) -> impl core::fmt::Display pub fn vortex_array::ArrayRef::display_tree(&self) -> vortex_array::display::TreeDisplay @@ -22170,23 +22626,23 @@ pub fn vortex_array::ArrayRef::tree_display_builder(&self) -> vortex_array::disp impl vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::execute(self, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::execute(self, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::execute_as(self, _name: &'static str, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::execute_as(self, &'static str, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::execute_until(self, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::execute_until(self, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::normalize(self, options: &mut vortex_array::normalize::NormalizeOptions<'_>) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::normalize(self, &mut vortex_array::normalize::NormalizeOptions<'_>) -> vortex_error::VortexResult impl vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::ptr_eq(this: &vortex_array::ArrayRef, other: &vortex_array::ArrayRef) -> bool +pub fn vortex_array::ArrayRef::ptr_eq(&vortex_array::ArrayRef, &vortex_array::ArrayRef) -> bool impl vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::serialize(&self, ctx: &vortex_array::ArrayContext, session: &vortex_session::VortexSession, options: &vortex_array::serde::SerializeOptions) -> vortex_error::VortexResult> +pub fn vortex_array::ArrayRef::serialize(&self, &vortex_array::ArrayContext, &vortex_session::VortexSession, &vortex_array::serde::SerializeOptions) -> vortex_error::VortexResult> impl vortex_array::ArrayRef @@ -22202,47 +22658,41 @@ pub fn vortex_array::ArrayRef::clone(&self) -> vortex_array::ArrayRef impl core::convert::From for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from(value: vortex_array::Canonical) -> Self +pub fn vortex_array::ArrayRef::from(vortex_array::Canonical) -> Self impl core::convert::From for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from(value: vortex_array::arrays::datetime::TemporalData) -> Self - -impl core::convert::TryFrom<&vortex_array::ArrayRef> for arrow_array::record_batch::RecordBatch - -pub type arrow_array::record_batch::RecordBatch::Error = vortex_error::VortexError - -pub fn arrow_array::record_batch::RecordBatch::try_from(value: &vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from(vortex_array::arrays::datetime::TemporalData) -> Self impl core::convert::TryFrom for vortex_array::arrays::datetime::TemporalData pub type vortex_array::arrays::datetime::TemporalData::Error = vortex_error::VortexError -pub fn vortex_array::arrays::datetime::TemporalData::try_from(value: vortex_array::ArrayRef) -> core::result::Result +pub fn vortex_array::arrays::datetime::TemporalData::try_from(vortex_array::ArrayRef) -> core::result::Result impl core::fmt::Debug for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::ArrayRef::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::ArrayRef::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::iter::traits::collect::FromIterator for vortex_array::Array -pub fn vortex_array::Array::from_iter>(iter: T) -> Self +pub fn vortex_array::Array::from_iter>(T) -> Self impl vortex_array::ArrayEq for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn vortex_array::ArrayRef::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayHash for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn vortex_array::ArrayRef::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::Executable for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::execute(array: vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::execute(vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::IntoArray for vortex_array::ArrayRef @@ -22270,183 +22720,187 @@ pub fn vortex_array::ArrayRef::to_varbinview(&self) -> vortex_array::arrays::Var impl vortex_array::arrow::ArrowArrayExecutor for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::execute_arrow(self, data_type: core::option::Option<&arrow_schema::datatype::DataType>, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::execute_arrow(self, core::option::Option<&arrow_schema::datatype::DataType>, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::execute_record_batch(self, schema: &arrow_schema::schema::Schema, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::execute_record_batch(self, &arrow_schema::schema::Schema, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::execute_record_batches(self, schema: &arrow_schema::schema::Schema, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::ArrayRef::execute_record_batches(self, &arrow_schema::schema::Schema, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> impl vortex_array::arrow::FromArrowArray<&arrow_array::array::boolean_array::BooleanArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::boolean_array::BooleanArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::boolean_array::BooleanArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::fixed_size_list_array::FixedSizeListArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(array: &arrow_array::array::fixed_size_list_array::FixedSizeListArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::fixed_size_list_array::FixedSizeListArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::map_array::MapArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::map_array::MapArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::map_array::MapArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::null_array::NullArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::null_array::NullArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::null_array::NullArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(array: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(array: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(array: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(array: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::primitive_array::PrimitiveArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::primitive_array::PrimitiveArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::primitive_array::PrimitiveArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::struct_array::StructArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::struct_array::StructArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::struct_array::StructArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::record_batch::RecordBatch> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(array: &arrow_array::record_batch::RecordBatch, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::record_batch::RecordBatch, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&dyn arrow_array::array::Array> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(array: &dyn arrow_array::array::Array, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&dyn arrow_array::array::Array, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(array: arrow_array::record_batch::RecordBatch, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(arrow_array::record_batch::RecordBatch, bool) -> vortex_error::VortexResult impl vortex_array::arrow::IntoArrowArray for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::into_arrow(self, data_type: &arrow_schema::datatype::DataType) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::into_arrow(self, &arrow_schema::datatype::DataType) -> vortex_error::VortexResult pub fn vortex_array::ArrayRef::into_arrow_preferred(self) -> vortex_error::VortexResult impl vortex_array::builtins::ArrayBuiltins for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::between(self, lower: vortex_array::ArrayRef, upper: vortex_array::ArrayRef, options: vortex_array::scalar_fn::fns::between::BetweenOptions) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::between(self, vortex_array::ArrayRef, vortex_array::ArrayRef, vortex_array::scalar_fn::fns::between::BetweenOptions) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::binary(&self, rhs: vortex_array::ArrayRef, op: vortex_array::scalar_fn::fns::operators::Operator) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::binary(&self, vortex_array::ArrayRef, vortex_array::scalar_fn::fns::operators::Operator) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::cast(&self, dtype: vortex_array::dtype::DType) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::cast(&self, vortex_array::dtype::DType) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::fill_null(&self, fill_value: impl core::convert::Into) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::fill_null(&self, impl core::convert::Into) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::get_item(&self, field_name: impl core::convert::Into) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::get_item(&self, impl core::convert::Into) -> vortex_error::VortexResult + +pub fn vortex_array::ArrayRef::is_not_null(&self) -> vortex_error::VortexResult pub fn vortex_array::ArrayRef::is_null(&self) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::list_contains(&self, value: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::list_contains(&self, vortex_array::ArrayRef) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::mask(self, mask: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::mask(self, vortex_array::ArrayRef) -> vortex_error::VortexResult pub fn vortex_array::ArrayRef::not(&self) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::zip(&self, if_true: vortex_array::ArrayRef, if_false: vortex_array::ArrayRef) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::zip(&self, vortex_array::ArrayRef, vortex_array::ArrayRef) -> vortex_error::VortexResult impl vortex_array::optimizer::ArrayOptimizer for vortex_array::ArrayRef pub fn vortex_array::ArrayRef::optimize(&self) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::optimize_recursive(&self) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::optimize_ctx(&self, &vortex_session::VortexSession) -> vortex_error::VortexResult + +pub fn vortex_array::ArrayRef::optimize_recursive(&self, &vortex_session::VortexSession) -> vortex_error::VortexResult impl vortex_array::scalar_fn::ReduceNode for vortex_array::ArrayRef pub fn vortex_array::ArrayRef::as_any(&self) -> &dyn core::any::Any -pub fn vortex_array::ArrayRef::child(&self, idx: usize) -> vortex_array::scalar_fn::ReduceNodeRef +pub fn vortex_array::ArrayRef::child(&self, usize) -> vortex_array::scalar_fn::ReduceNodeRef pub fn vortex_array::ArrayRef::child_count(&self) -> usize @@ -22456,33 +22910,33 @@ pub fn vortex_array::ArrayRef::scalar_fn(&self) -> core::option::Option<&vortex_ impl vortex_array::search_sorted::IndexOrd for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::index_cmp(&self, idx: usize, elem: &vortex_array::scalar::Scalar) -> vortex_error::VortexResult> +pub fn vortex_array::ArrayRef::index_cmp(&self, usize, &vortex_array::scalar::Scalar) -> vortex_error::VortexResult> -pub fn vortex_array::ArrayRef::index_ge(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::index_ge(&self, usize, &V) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::index_gt(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::index_gt(&self, usize, &V) -> vortex_error::VortexResult -pub fn vortex_array::ArrayRef::index_le(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::index_le(&self, usize, &V) -> vortex_error::VortexResult pub fn vortex_array::ArrayRef::index_len(&self) -> usize -pub fn vortex_array::ArrayRef::index_lt(&self, idx: usize, elem: &V) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::index_lt(&self, usize, &V) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::list_view_array::GenericListViewArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(array: &arrow_array::array::list_view_array::GenericListViewArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::list_view_array::GenericListViewArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::list_array::GenericListArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::list_array::GenericListArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::list_array::GenericListArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::byte_array::GenericByteArray> for vortex_array::ArrayRef where ::Offset: vortex_array::dtype::IntegerPType -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::byte_array::GenericByteArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::byte_array::GenericByteArray, bool) -> vortex_error::VortexResult impl vortex_array::arrow::FromArrowArray<&arrow_array::array::byte_view_array::GenericByteViewArray> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from_arrow(value: &arrow_array::array::byte_view_array::GenericByteViewArray, nullable: bool) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::from_arrow(&arrow_array::array::byte_view_array::GenericByteViewArray, bool) -> vortex_error::VortexResult impl core::convert::AsRef for vortex_array::Array @@ -22494,7 +22948,7 @@ pub fn vortex_array::ArrayView<'_, V>::as_ref(&self) -> &vortex_array::ArrayRef impl core::convert::From> for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::from(value: vortex_array::Array) -> vortex_array::ArrayRef +pub fn vortex_array::ArrayRef::from(vortex_array::Array) -> vortex_array::ArrayRef pub struct vortex_array::ArrayView<'a, V: vortex_array::VTable> @@ -22530,7 +22984,7 @@ pub fn vortex_array::ArrayView<'_, V>::as_ref(&self) -> &vortex_array::ArrayRef impl core::fmt::Debug for vortex_array::ArrayView<'_, V> -pub fn vortex_array::ArrayView<'_, V>::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::ArrayView<'_, V>::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::marker::Copy for vortex_array::ArrayView<'_, V> @@ -22548,7 +23002,7 @@ pub struct vortex_array::CanonicalValidity(pub vortex_array::Canonical) impl vortex_array::Executable for vortex_array::CanonicalValidity -pub fn vortex_array::CanonicalValidity::execute(array: vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::CanonicalValidity::execute(vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub struct vortex_array::DepthFirstArrayIterator @@ -22570,31 +23024,31 @@ pub fn vortex_array::EmptyArrayData::default() -> vortex_array::EmptyArrayData impl core::fmt::Debug for vortex_array::EmptyArrayData -pub fn vortex_array::EmptyArrayData::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::EmptyArrayData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::EmptyArrayData -pub fn vortex_array::EmptyArrayData::fmt(&self, _f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::EmptyArrayData::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::ArrayEq for vortex_array::EmptyArrayData -pub fn vortex_array::EmptyArrayData::array_eq(&self, _other: &Self, _precision: vortex_array::Precision) -> bool +pub fn vortex_array::EmptyArrayData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayHash for vortex_array::EmptyArrayData -pub fn vortex_array::EmptyArrayData::array_hash(&self, _state: &mut H, _precision: vortex_array::Precision) +pub fn vortex_array::EmptyArrayData::array_hash(&self, &mut H, vortex_array::Precision) pub struct vortex_array::EmptyMetadata impl core::fmt::Debug for vortex_array::EmptyMetadata -pub fn vortex_array::EmptyMetadata::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::EmptyMetadata::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::DeserializeMetadata for vortex_array::EmptyMetadata pub type vortex_array::EmptyMetadata::Output = vortex_array::EmptyMetadata -pub fn vortex_array::EmptyMetadata::deserialize(metadata: &[u8]) -> vortex_error::VortexResult +pub fn vortex_array::EmptyMetadata::deserialize(&[u8]) -> vortex_error::VortexResult impl vortex_array::SerializeMetadata for vortex_array::EmptyMetadata @@ -22606,9 +23060,9 @@ impl vortex_array::ExecutionCtx pub fn vortex_array::ExecutionCtx::allocator(&self) -> vortex_array::memory::HostAllocatorRef -pub fn vortex_array::ExecutionCtx::log(&mut self, msg: core::fmt::Arguments<'_>) +pub fn vortex_array::ExecutionCtx::log(&mut self, core::fmt::Arguments<'_>) -pub fn vortex_array::ExecutionCtx::new(session: vortex_session::VortexSession) -> Self +pub fn vortex_array::ExecutionCtx::new(vortex_session::VortexSession) -> Self pub fn vortex_array::ExecutionCtx::session(&self) -> &vortex_session::VortexSession @@ -22618,11 +23072,11 @@ pub fn vortex_array::ExecutionCtx::clone(&self) -> vortex_array::ExecutionCtx impl core::fmt::Debug for vortex_array::ExecutionCtx -pub fn vortex_array::ExecutionCtx::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::ExecutionCtx::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::fmt::Display for vortex_array::ExecutionCtx -pub fn vortex_array::ExecutionCtx::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::ExecutionCtx::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::ops::drop::Drop for vortex_array::ExecutionCtx @@ -22632,11 +23086,13 @@ pub struct vortex_array::ExecutionResult impl vortex_array::ExecutionResult +pub fn vortex_array::ExecutionResult::append_child(impl vortex_array::IntoArray, usize) -> Self + pub fn vortex_array::ExecutionResult::array(&self) -> &vortex_array::ArrayRef -pub fn vortex_array::ExecutionResult::done(result: impl vortex_array::IntoArray) -> Self +pub fn vortex_array::ExecutionResult::done(impl vortex_array::IntoArray) -> Self -pub fn vortex_array::ExecutionResult::execute_slot(array: impl vortex_array::IntoArray, slot_idx: usize) -> Self +pub fn vortex_array::ExecutionResult::execute_slot(impl vortex_array::IntoArray, usize) -> Self pub fn vortex_array::ExecutionResult::into_parts(self) -> (vortex_array::ArrayRef, vortex_array::ExecutionStep) @@ -22644,25 +23100,25 @@ pub fn vortex_array::ExecutionResult::step(&self) -> &vortex_array::ExecutionSte impl core::fmt::Debug for vortex_array::ExecutionResult -pub fn vortex_array::ExecutionResult::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::ExecutionResult::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub struct vortex_array::MaskFuture impl vortex_array::MaskFuture -pub fn vortex_array::MaskFuture::inspect(self, f: impl core::ops::function::FnOnce(&vortex_error::SharedVortexResult) + 'static + core::marker::Send + core::marker::Sync) -> Self +pub fn vortex_array::MaskFuture::inspect(self, impl core::ops::function::FnOnce(&vortex_error::SharedVortexResult) + 'static + core::marker::Send + core::marker::Sync) -> Self pub fn vortex_array::MaskFuture::is_empty(&self) -> bool pub fn vortex_array::MaskFuture::len(&self) -> usize -pub fn vortex_array::MaskFuture::new(len: usize, fut: F) -> Self where F: core::future::future::Future> + core::marker::Send + 'static +pub fn vortex_array::MaskFuture::new(usize, F) -> Self where F: core::future::future::Future> + core::marker::Send + 'static -pub fn vortex_array::MaskFuture::new_true(row_count: usize) -> Self +pub fn vortex_array::MaskFuture::new_true(usize) -> Self -pub fn vortex_array::MaskFuture::ready(mask: vortex_mask::Mask) -> Self +pub fn vortex_array::MaskFuture::ready(vortex_mask::Mask) -> Self -pub fn vortex_array::MaskFuture::slice(&self, range: core::ops::range::Range) -> Self +pub fn vortex_array::MaskFuture::slice(&self, core::ops::range::Range) -> Self impl core::clone::Clone for vortex_array::MaskFuture @@ -22672,19 +23128,19 @@ impl core::future::future::Future for vortex_array::MaskFuture pub type vortex_array::MaskFuture::Output = core::result::Result -pub fn vortex_array::MaskFuture::poll(self: core::pin::Pin<&mut Self>, cx: &mut core::task::wake::Context<'_>) -> core::task::poll::Poll +pub fn vortex_array::MaskFuture::poll(core::pin::Pin<&mut Self>, &mut core::task::wake::Context<'_>) -> core::task::poll::Poll pub struct vortex_array::NotSupported impl vortex_array::OperationsVTable for vortex_array::NotSupported -pub fn vortex_array::NotSupported::scalar_at(array: vortex_array::ArrayView<'_, V>, _index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::NotSupported::scalar_at(vortex_array::ArrayView<'_, V>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub struct vortex_array::ProstMetadata(pub M) impl core::fmt::Debug for vortex_array::ProstMetadata -pub fn vortex_array::ProstMetadata::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::ProstMetadata::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl core::ops::deref::Deref for vortex_array::ProstMetadata @@ -22696,7 +23152,7 @@ impl vortex_array::DeserializeMetadata for vortex_array::ProstMetadata whe pub type vortex_array::ProstMetadata::Output = M -pub fn vortex_array::ProstMetadata::deserialize(metadata: &[u8]) -> vortex_error::VortexResult +pub fn vortex_array::ProstMetadata::deserialize(&[u8]) -> vortex_error::VortexResult impl vortex_array::SerializeMetadata for vortex_array::ProstMetadata where M: prost::message::Message @@ -22706,13 +23162,13 @@ pub struct vortex_array::RawMetadata(pub alloc::vec::Vec) impl core::fmt::Debug for vortex_array::RawMetadata -pub fn vortex_array::RawMetadata::fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result +pub fn vortex_array::RawMetadata::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result impl vortex_array::DeserializeMetadata for vortex_array::RawMetadata pub type vortex_array::RawMetadata::Output = alloc::vec::Vec -pub fn vortex_array::RawMetadata::deserialize(metadata: &[u8]) -> vortex_error::VortexResult +pub fn vortex_array::RawMetadata::deserialize(&[u8]) -> vortex_error::VortexResult impl vortex_array::SerializeMetadata for vortex_array::RawMetadata @@ -22722,277 +23178,289 @@ pub struct vortex_array::RecursiveCanonical(pub vortex_array::Canonical) impl vortex_array::Executable for vortex_array::RecursiveCanonical -pub fn vortex_array::RecursiveCanonical::execute(array: vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::RecursiveCanonical::execute(vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub struct vortex_array::ValidityVTableFromChild impl vortex_array::ValidityVTable for vortex_array::ValidityVTableFromChild where V: vortex_array::ValidityChild + vortex_array::VTable -pub fn vortex_array::ValidityVTableFromChild::validity(array: vortex_array::ArrayView<'_, V>) -> vortex_error::VortexResult +pub fn vortex_array::ValidityVTableFromChild::validity(vortex_array::ArrayView<'_, V>) -> vortex_error::VortexResult pub struct vortex_array::ValidityVTableFromChildSliceHelper impl vortex_array::ValidityVTable for vortex_array::ValidityVTableFromChildSliceHelper where ::ArrayData: vortex_array::ValidityChildSliceHelper -pub fn vortex_array::ValidityVTableFromChildSliceHelper::validity(array: vortex_array::ArrayView<'_, V>) -> vortex_error::VortexResult +pub fn vortex_array::ValidityVTableFromChildSliceHelper::validity(vortex_array::ArrayView<'_, V>) -> vortex_error::VortexResult pub static vortex_array::LEGACY_SESSION: std::sync::lazy_lock::LazyLock pub trait vortex_array::ArrayEq -pub fn vortex_array::ArrayEq::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn vortex_array::ArrayEq::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayEq for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn vortex_array::ArrayRef::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayEq for vortex_array::EmptyArrayData -pub fn vortex_array::EmptyArrayData::array_eq(&self, _other: &Self, _precision: vortex_array::Precision) -> bool +pub fn vortex_array::EmptyArrayData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayEq for vortex_array::arrays::bool::BoolData -pub fn vortex_array::arrays::bool::BoolData::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::bool::BoolData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayEq for vortex_array::arrays::chunked::ChunkedData -pub fn vortex_array::arrays::chunked::ChunkedData::array_eq(&self, _other: &Self, _precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::chunked::ChunkedData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayEq for vortex_array::arrays::constant::ConstantData -pub fn vortex_array::arrays::constant::ConstantData::array_eq(&self, other: &Self, _precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::constant::ConstantData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayEq for vortex_array::arrays::decimal::DecimalData -pub fn vortex_array::arrays::decimal::DecimalData::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::decimal::DecimalData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayEq for vortex_array::arrays::dict::DictData -pub fn vortex_array::arrays::dict::DictData::array_eq(&self, _other: &Self, _precision: vortex_array::Precision) -> bool - -impl vortex_array::ArrayEq for vortex_array::arrays::extension::ExtensionData - -pub fn vortex_array::arrays::extension::ExtensionData::array_eq(&self, _other: &Self, _precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::dict::DictData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayEq for vortex_array::arrays::filter::FilterData -pub fn vortex_array::arrays::filter::FilterData::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::filter::FilterData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayEq for vortex_array::arrays::fixed_size_list::FixedSizeListData -pub fn vortex_array::arrays::fixed_size_list::FixedSizeListData::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::fixed_size_list::FixedSizeListData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayEq for vortex_array::arrays::list::ListData -pub fn vortex_array::arrays::list::ListData::array_eq(&self, _other: &Self, _precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::list::ListData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayEq for vortex_array::arrays::listview::ListViewData -pub fn vortex_array::arrays::listview::ListViewData::array_eq(&self, other: &Self, _precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::listview::ListViewData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayEq for vortex_array::arrays::masked::MaskedData -pub fn vortex_array::arrays::masked::MaskedData::array_eq(&self, _other: &Self, _precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::masked::MaskedData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayEq for vortex_array::arrays::patched::PatchedData -pub fn vortex_array::arrays::patched::PatchedData::array_eq(&self, other: &Self, _precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::patched::PatchedData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayEq for vortex_array::arrays::primitive::PrimitiveData -pub fn vortex_array::arrays::primitive::PrimitiveData::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool - -impl vortex_array::ArrayEq for vortex_array::arrays::scalar_fn::ScalarFnData - -pub fn vortex_array::arrays::scalar_fn::ScalarFnData::array_eq(&self, other: &Self, _precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::primitive::PrimitiveData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayEq for vortex_array::arrays::shared::SharedData -pub fn vortex_array::arrays::shared::SharedData::array_eq(&self, _other: &Self, _precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::shared::SharedData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayEq for vortex_array::arrays::slice::SliceData -pub fn vortex_array::arrays::slice::SliceData::array_eq(&self, other: &Self, _precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::slice::SliceData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayEq for vortex_array::arrays::varbin::VarBinData -pub fn vortex_array::arrays::varbin::VarBinData::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::varbin::VarBinData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayEq for vortex_array::arrays::varbinview::VarBinViewData -pub fn vortex_array::arrays::varbinview::VarBinViewData::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn vortex_array::arrays::varbinview::VarBinViewData::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayEq for vortex_array::buffer::BufferHandle -pub fn vortex_array::buffer::BufferHandle::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn vortex_array::buffer::BufferHandle::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayEq for vortex_array::patches::Patches -pub fn vortex_array::patches::Patches::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn vortex_array::patches::Patches::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayEq for vortex_array::validity::Validity -pub fn vortex_array::validity::Validity::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn vortex_array::validity::Validity::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayEq for vortex_buffer::bit::buf::BitBuffer -pub fn vortex_buffer::bit::buf::BitBuffer::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn vortex_buffer::bit::buf::BitBuffer::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayEq for vortex_mask::Mask -pub fn vortex_mask::Mask::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn vortex_mask::Mask::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayEq for vortex_buffer::buffer::Buffer -pub fn vortex_buffer::buffer::Buffer::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn vortex_buffer::buffer::Buffer::array_eq(&self, &Self, vortex_array::Precision) -> bool impl vortex_array::ArrayEq for core::option::Option -pub fn core::option::Option::array_eq(&self, other: &Self, precision: vortex_array::Precision) -> bool +pub fn core::option::Option::array_eq(&self, &Self, vortex_array::Precision) -> bool pub trait vortex_array::ArrayHash -pub fn vortex_array::ArrayHash::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn vortex_array::ArrayHash::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::ArrayHash for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn vortex_array::ArrayRef::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::ArrayHash for vortex_array::EmptyArrayData -pub fn vortex_array::EmptyArrayData::array_hash(&self, _state: &mut H, _precision: vortex_array::Precision) +pub fn vortex_array::EmptyArrayData::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::ArrayHash for vortex_array::arrays::bool::BoolData -pub fn vortex_array::arrays::bool::BoolData::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn vortex_array::arrays::bool::BoolData::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::ArrayHash for vortex_array::arrays::chunked::ChunkedData -pub fn vortex_array::arrays::chunked::ChunkedData::array_hash(&self, _state: &mut H, _precision: vortex_array::Precision) +pub fn vortex_array::arrays::chunked::ChunkedData::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::ArrayHash for vortex_array::arrays::constant::ConstantData -pub fn vortex_array::arrays::constant::ConstantData::array_hash(&self, state: &mut H, _precision: vortex_array::Precision) +pub fn vortex_array::arrays::constant::ConstantData::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::ArrayHash for vortex_array::arrays::decimal::DecimalData -pub fn vortex_array::arrays::decimal::DecimalData::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn vortex_array::arrays::decimal::DecimalData::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::ArrayHash for vortex_array::arrays::dict::DictData -pub fn vortex_array::arrays::dict::DictData::array_hash(&self, _state: &mut H, _precision: vortex_array::Precision) - -impl vortex_array::ArrayHash for vortex_array::arrays::extension::ExtensionData - -pub fn vortex_array::arrays::extension::ExtensionData::array_hash(&self, _state: &mut H, _precision: vortex_array::Precision) +pub fn vortex_array::arrays::dict::DictData::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::ArrayHash for vortex_array::arrays::filter::FilterData -pub fn vortex_array::arrays::filter::FilterData::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn vortex_array::arrays::filter::FilterData::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::ArrayHash for vortex_array::arrays::fixed_size_list::FixedSizeListData -pub fn vortex_array::arrays::fixed_size_list::FixedSizeListData::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn vortex_array::arrays::fixed_size_list::FixedSizeListData::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::ArrayHash for vortex_array::arrays::list::ListData -pub fn vortex_array::arrays::list::ListData::array_hash(&self, _state: &mut H, _precision: vortex_array::Precision) +pub fn vortex_array::arrays::list::ListData::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::ArrayHash for vortex_array::arrays::listview::ListViewData -pub fn vortex_array::arrays::listview::ListViewData::array_hash(&self, state: &mut H, _precision: vortex_array::Precision) +pub fn vortex_array::arrays::listview::ListViewData::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::ArrayHash for vortex_array::arrays::masked::MaskedData -pub fn vortex_array::arrays::masked::MaskedData::array_hash(&self, _state: &mut H, _precision: vortex_array::Precision) +pub fn vortex_array::arrays::masked::MaskedData::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::ArrayHash for vortex_array::arrays::patched::PatchedData -pub fn vortex_array::arrays::patched::PatchedData::array_hash(&self, state: &mut H, _precision: vortex_array::Precision) +pub fn vortex_array::arrays::patched::PatchedData::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::ArrayHash for vortex_array::arrays::primitive::PrimitiveData -pub fn vortex_array::arrays::primitive::PrimitiveData::array_hash(&self, state: &mut H, precision: vortex_array::Precision) - -impl vortex_array::ArrayHash for vortex_array::arrays::scalar_fn::ScalarFnData - -pub fn vortex_array::arrays::scalar_fn::ScalarFnData::array_hash(&self, state: &mut H, _precision: vortex_array::Precision) +pub fn vortex_array::arrays::primitive::PrimitiveData::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::ArrayHash for vortex_array::arrays::shared::SharedData -pub fn vortex_array::arrays::shared::SharedData::array_hash(&self, _state: &mut H, _precision: vortex_array::Precision) +pub fn vortex_array::arrays::shared::SharedData::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::ArrayHash for vortex_array::arrays::slice::SliceData -pub fn vortex_array::arrays::slice::SliceData::array_hash(&self, state: &mut H, _precision: vortex_array::Precision) +pub fn vortex_array::arrays::slice::SliceData::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::ArrayHash for vortex_array::arrays::varbin::VarBinData -pub fn vortex_array::arrays::varbin::VarBinData::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn vortex_array::arrays::varbin::VarBinData::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::ArrayHash for vortex_array::arrays::varbinview::VarBinViewData -pub fn vortex_array::arrays::varbinview::VarBinViewData::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn vortex_array::arrays::varbinview::VarBinViewData::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::ArrayHash for vortex_array::buffer::BufferHandle -pub fn vortex_array::buffer::BufferHandle::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn vortex_array::buffer::BufferHandle::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::ArrayHash for vortex_array::patches::Patches -pub fn vortex_array::patches::Patches::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn vortex_array::patches::Patches::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::ArrayHash for vortex_array::validity::Validity -pub fn vortex_array::validity::Validity::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn vortex_array::validity::Validity::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::ArrayHash for vortex_buffer::bit::buf::BitBuffer -pub fn vortex_buffer::bit::buf::BitBuffer::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn vortex_buffer::bit::buf::BitBuffer::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::ArrayHash for vortex_mask::Mask -pub fn vortex_mask::Mask::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn vortex_mask::Mask::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::ArrayHash for vortex_buffer::buffer::Buffer -pub fn vortex_buffer::buffer::Buffer::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn vortex_buffer::buffer::Buffer::array_hash(&self, &mut H, vortex_array::Precision) impl vortex_array::ArrayHash for core::option::Option -pub fn core::option::Option::array_hash(&self, state: &mut H, precision: vortex_array::Precision) +pub fn core::option::Option::array_hash(&self, &mut H, vortex_array::Precision) pub trait vortex_array::ArrayPlugin: 'static + core::marker::Send + core::marker::Sync pub trait vortex_array::ArrayPlugin: 'static + core::marker::Send + core::marker::Sync -pub fn vortex_array::ArrayPlugin::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::ArrayPlugin::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult -pub fn vortex_array::ArrayPlugin::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, session: &vortex_session::VortexSession) -> vortex_error::VortexResult +pub fn vortex_array::ArrayPlugin::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult pub fn vortex_array::ArrayPlugin::id(&self) -> vortex_array::ArrayId pub fn vortex_array::ArrayPlugin::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::ArrayPlugin::serialize(&self, array: &vortex_array::ArrayRef, session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::ArrayPlugin::is_supported_encoding(&self, &vortex_array::ArrayId) -> bool + +pub fn vortex_array::ArrayPlugin::is_supported_encoding(&self, &vortex_array::ArrayId) -> bool -pub fn vortex_array::ArrayPlugin::serialize(&self, array: &vortex_array::ArrayRef, session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::ArrayPlugin::serialize(&self, &vortex_array::ArrayRef, &vortex_session::VortexSession) -> vortex_error::VortexResult>> + +pub fn vortex_array::ArrayPlugin::serialize(&self, &vortex_array::ArrayRef, &vortex_session::VortexSession) -> vortex_error::VortexResult>> impl vortex_array::ArrayPlugin for V impl vortex_array::ArrayPlugin for V -pub fn V::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, session: &vortex_session::VortexSession) -> core::result::Result +pub fn V::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> core::result::Result + +pub fn V::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> core::result::Result + +pub fn V::id(&self) -> vortex_session::registry::Id + +pub fn V::id(&self) -> vortex_session::registry::Id + +pub fn V::is_supported_encoding(&self, &vortex_array::ArrayId) -> bool + +pub fn V::is_supported_encoding(&self, &vortex_array::ArrayId) -> bool -pub fn V::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, session: &vortex_session::VortexSession) -> core::result::Result +pub fn V::serialize(&self, &vortex_array::ArrayRef, &vortex_session::VortexSession) -> core::result::Result>, vortex_error::VortexError> -pub fn V::id(&self) -> arcref::ArcRef +pub fn V::serialize(&self, &vortex_array::ArrayRef, &vortex_session::VortexSession) -> core::result::Result>, vortex_error::VortexError> -pub fn V::id(&self) -> arcref::ArcRef +impl vortex_array::ArrayPlugin for vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin -pub fn V::serialize(&self, array: &vortex_array::ArrayRef, session: &vortex_session::VortexSession) -> core::result::Result>, vortex_error::VortexError> +impl vortex_array::ArrayPlugin for vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin -pub fn V::serialize(&self, array: &vortex_array::ArrayRef, session: &vortex_session::VortexSession) -> core::result::Result>, vortex_error::VortexError> +pub fn vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult + +pub fn vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult + +pub fn vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin::id(&self) -> vortex_array::ArrayId + +pub fn vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin::id(&self) -> vortex_array::ArrayId + +pub fn vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin::is_supported_encoding(&self, &vortex_array::ArrayId) -> bool + +pub fn vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin::is_supported_encoding(&self, &vortex_array::ArrayId) -> bool + +pub fn vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin::serialize(&self, &vortex_array::ArrayRef, &vortex_session::VortexSession) -> vortex_error::VortexResult>> + +pub fn vortex_array::arrays::scalar_fn::plugin::ScalarFnArrayPlugin::serialize(&self, &vortex_array::ArrayRef, &vortex_session::VortexSession) -> vortex_error::VortexResult>> pub trait vortex_array::ArrayVTable: 'static + core::clone::Clone + core::marker::Sized + core::marker::Send + core::marker::Sync + core::fmt::Debug @@ -23002,37 +23470,37 @@ pub type vortex_array::ArrayVTable::OperationsVTable: vortex_array::OperationsVT pub type vortex_array::ArrayVTable::ValidityVTable: vortex_array::ValidityVTable -pub fn vortex_array::ArrayVTable::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::ArrayVTable::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::ArrayVTable::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::ArrayVTable::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::ArrayVTable::buffer_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::ArrayVTable::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::ArrayVTable::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::ArrayVTable::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::ArrayVTable::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::ArrayVTable::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::ArrayVTable::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::ArrayVTable::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::ArrayVTable::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::ArrayVTable::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::ArrayVTable::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::ArrayVTable::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::ArrayVTable::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::ArrayVTable::nbuffers(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::ArrayVTable::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::ArrayVTable::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::ArrayVTable::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::ArrayVTable::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::ArrayVTable::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::ArrayVTable::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::ArrayVTable::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::ArrayVTable::serialize(array: vortex_array::ArrayView<'_, Self>, session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::ArrayVTable::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::ArrayVTable::slot_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::ArrayVTable::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::ArrayVTable::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::ArrayVTable::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Bool @@ -23042,37 +23510,37 @@ pub type vortex_array::arrays::Bool::OperationsVTable = vortex_array::arrays::Bo pub type vortex_array::arrays::Bool::ValidityVTable = vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Bool::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Bool::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Bool::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Bool::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Bool::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Bool::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Bool::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Bool::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Bool::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Bool::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Bool::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Bool::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Bool::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Bool::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Bool::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Bool::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Bool::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Bool::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Bool::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Bool::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Bool::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Bool::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Bool::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Bool::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Bool::validate(&self, data: &vortex_array::arrays::bool::BoolData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Bool::validate(&self, &vortex_array::arrays::bool::BoolData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Chunked @@ -23082,37 +23550,37 @@ pub type vortex_array::arrays::Chunked::OperationsVTable = vortex_array::arrays: pub type vortex_array::arrays::Chunked::ValidityVTable = vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Chunked::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Chunked::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Chunked::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Chunked::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Chunked::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Chunked::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Chunked::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Chunked::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Chunked::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Chunked::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Chunked::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Chunked::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Chunked::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Chunked::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Chunked::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Chunked::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Chunked::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Chunked::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Chunked::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Chunked::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Chunked::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Chunked::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Chunked::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Chunked::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Chunked::validate(&self, data: &vortex_array::arrays::chunked::ChunkedData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Chunked::validate(&self, &vortex_array::arrays::chunked::ChunkedData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Constant @@ -23122,37 +23590,37 @@ pub type vortex_array::arrays::Constant::OperationsVTable = vortex_array::arrays pub type vortex_array::arrays::Constant::ValidityVTable = vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Constant::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Constant::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Constant::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Constant::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Constant::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Constant::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Constant::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Constant::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Constant::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Constant::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, _metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Constant::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Constant::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Constant::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Constant::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Constant::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Constant::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Constant::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Constant::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Constant::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Constant::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Constant::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Constant::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Constant::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Constant::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Constant::validate(&self, data: &vortex_array::arrays::constant::ConstantData, dtype: &vortex_array::dtype::DType, _len: usize, _slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Constant::validate(&self, &vortex_array::arrays::constant::ConstantData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Decimal @@ -23162,77 +23630,77 @@ pub type vortex_array::arrays::Decimal::OperationsVTable = vortex_array::arrays: pub type vortex_array::arrays::Decimal::ValidityVTable = vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Decimal::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Decimal::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Decimal::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Decimal::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Decimal::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Decimal::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Decimal::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Decimal::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Decimal::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Decimal::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Decimal::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Decimal::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Decimal::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Decimal::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Decimal::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Decimal::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Decimal::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Decimal::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Decimal::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Decimal::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Decimal::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Decimal::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Decimal::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Decimal::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Decimal::validate(&self, data: &vortex_array::arrays::decimal::DecimalData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Decimal::validate(&self, &vortex_array::arrays::decimal::DecimalData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Extension -pub type vortex_array::arrays::Extension::ArrayData = vortex_array::arrays::extension::ExtensionData +pub type vortex_array::arrays::Extension::ArrayData = vortex_array::EmptyArrayData pub type vortex_array::arrays::Extension::OperationsVTable = vortex_array::arrays::Extension pub type vortex_array::arrays::Extension::ValidityVTable = vortex_array::ValidityVTableFromChild -pub fn vortex_array::arrays::Extension::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Extension::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Extension::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Extension::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Extension::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Extension::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Extension::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Extension::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Extension::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Extension::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Extension::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Extension::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Extension::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Extension::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Extension::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Extension::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Extension::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Extension::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Extension::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Extension::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Extension::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Extension::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Extension::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Extension::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Extension::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Extension::validate(&self, data: &vortex_array::arrays::extension::ExtensionData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Extension::validate(&self, &vortex_array::EmptyArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Filter @@ -23242,37 +23710,37 @@ pub type vortex_array::arrays::Filter::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::Filter::ValidityVTable = vortex_array::arrays::Filter -pub fn vortex_array::arrays::Filter::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Filter::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Filter::buffer(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Filter::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Filter::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Filter::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Filter::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Filter::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Filter::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Filter::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Filter::deserialize(&self, _dtype: &vortex_array::dtype::DType, _len: usize, _metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Filter::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Filter::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Filter::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Filter::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Filter::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Filter::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Filter::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Filter::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Filter::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Filter::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Filter::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Filter::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Filter::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Filter::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Filter::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Filter::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Filter::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Filter::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Filter::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Filter::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::FixedSizeList @@ -23282,37 +23750,37 @@ pub type vortex_array::arrays::FixedSizeList::OperationsVTable = vortex_array::a pub type vortex_array::arrays::FixedSizeList::ValidityVTable = vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::FixedSizeList::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::FixedSizeList::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::FixedSizeList::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::FixedSizeList::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::FixedSizeList::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::FixedSizeList::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::FixedSizeList::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::FixedSizeList::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::FixedSizeList::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::FixedSizeList::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::FixedSizeList::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::FixedSizeList::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::FixedSizeList::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::FixedSizeList::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::FixedSizeList::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::FixedSizeList::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::FixedSizeList::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::FixedSizeList::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::FixedSizeList::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::FixedSizeList::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::FixedSizeList::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::FixedSizeList::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::FixedSizeList::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::FixedSizeList::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::FixedSizeList::validate(&self, data: &vortex_array::arrays::fixed_size_list::FixedSizeListData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::FixedSizeList::validate(&self, &vortex_array::arrays::fixed_size_list::FixedSizeListData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::List @@ -23322,37 +23790,37 @@ pub type vortex_array::arrays::List::OperationsVTable = vortex_array::arrays::Li pub type vortex_array::arrays::List::ValidityVTable = vortex_array::arrays::List -pub fn vortex_array::arrays::List::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::List::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::List::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::List::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::List::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::List::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::List::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::List::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::List::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::List::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::List::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::List::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::List::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::List::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::List::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::List::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::List::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::List::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::List::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::List::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::List::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::List::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::List::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::List::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::List::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::List::validate(&self, _data: &vortex_array::arrays::list::ListData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::List::validate(&self, &vortex_array::arrays::list::ListData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::ListView @@ -23362,37 +23830,37 @@ pub type vortex_array::arrays::ListView::OperationsVTable = vortex_array::arrays pub type vortex_array::arrays::ListView::ValidityVTable = vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::ListView::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::ListView::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::ListView::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::ListView::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::ListView::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::ListView::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::ListView::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::ListView::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::ListView::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::ListView::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::ListView::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::ListView::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::ListView::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::ListView::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::ListView::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::ListView::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::ListView::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::ListView::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::ListView::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::ListView::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::ListView::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::ListView::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::ListView::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::ListView::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::ListView::validate(&self, _data: &vortex_array::arrays::listview::ListViewData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::ListView::validate(&self, &vortex_array::arrays::listview::ListViewData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Masked @@ -23402,37 +23870,37 @@ pub type vortex_array::arrays::Masked::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::Masked::ValidityVTable = vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Masked::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Masked::buffer(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Masked::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Masked::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Masked::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Masked::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Masked::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Masked::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Masked::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Masked::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Masked::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Masked::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Masked::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Masked::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Masked::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Masked::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Masked::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Masked::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Masked::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Masked::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Masked::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Masked::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Masked::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Masked::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Masked::validate(&self, _data: &vortex_array::arrays::masked::MaskedData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Masked::validate(&self, &vortex_array::arrays::masked::MaskedData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Primitive @@ -23442,37 +23910,37 @@ pub type vortex_array::arrays::Primitive::OperationsVTable = vortex_array::array pub type vortex_array::arrays::Primitive::ValidityVTable = vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Primitive::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Primitive::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Primitive::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Primitive::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Primitive::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Primitive::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Primitive::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Primitive::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Primitive::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Primitive::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Primitive::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Primitive::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Primitive::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Primitive::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Primitive::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Primitive::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Primitive::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Primitive::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Primitive::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Primitive::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Primitive::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Primitive::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Primitive::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Primitive::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Primitive::validate(&self, data: &vortex_array::arrays::primitive::PrimitiveData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Primitive::validate(&self, &vortex_array::arrays::primitive::PrimitiveData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Shared @@ -23482,37 +23950,37 @@ pub type vortex_array::arrays::Shared::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::Shared::ValidityVTable = vortex_array::arrays::Shared -pub fn vortex_array::arrays::Shared::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Shared::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Shared::buffer(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Shared::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Shared::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Shared::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Shared::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Shared::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Shared::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Shared::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Shared::deserialize(&self, _dtype: &vortex_array::dtype::DType, _len: usize, _metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Shared::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Shared::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Shared::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Shared::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Shared::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Shared::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Shared::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Shared::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Shared::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Shared::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Shared::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Shared::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Shared::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Shared::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Shared::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Shared::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Shared::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Shared::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Shared::validate(&self, _data: &vortex_array::arrays::shared::SharedData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Shared::validate(&self, &vortex_array::arrays::shared::SharedData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Struct @@ -23522,37 +23990,37 @@ pub type vortex_array::arrays::Struct::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::Struct::ValidityVTable = vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Struct::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Struct::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Struct::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Struct::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Struct::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Struct::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Struct::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Struct::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Struct::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Struct::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Struct::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Struct::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Struct::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Struct::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Struct::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Struct::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Struct::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Struct::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Struct::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Struct::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Struct::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Struct::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Struct::slot_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Struct::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Struct::validate(&self, _data: &vortex_array::EmptyArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Struct::validate(&self, &vortex_array::EmptyArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::VarBin @@ -23562,37 +24030,37 @@ pub type vortex_array::arrays::VarBin::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::VarBin::ValidityVTable = vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::VarBin::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::VarBin::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::VarBin::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::VarBin::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::VarBin::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::VarBin::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::VarBin::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::VarBin::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::VarBin::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::VarBin::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBin::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBin::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::VarBin::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::VarBin::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::VarBin::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::VarBin::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::VarBin::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::VarBin::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::VarBin::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBin::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBin::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::VarBin::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::VarBin::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::VarBin::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::VarBin::validate(&self, _data: &vortex_array::arrays::varbin::VarBinData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::VarBin::validate(&self, &vortex_array::arrays::varbin::VarBinData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::VarBinView @@ -23602,37 +24070,37 @@ pub type vortex_array::arrays::VarBinView::OperationsVTable = vortex_array::arra pub type vortex_array::arrays::VarBinView::ValidityVTable = vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::VarBinView::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::VarBinView::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::VarBinView::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::VarBinView::buffer_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::VarBinView::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::VarBinView::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::VarBinView::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::VarBinView::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::VarBinView::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::VarBinView::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBinView::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBinView::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::VarBinView::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::VarBinView::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::VarBinView::nbuffers(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::VarBinView::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::VarBinView::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::VarBinView::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::VarBinView::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBinView::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBinView::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::VarBinView::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::VarBinView::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::VarBinView::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::VarBinView::validate(&self, data: &vortex_array::arrays::varbinview::VarBinViewData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::VarBinView::validate(&self, &vortex_array::arrays::varbinview::VarBinViewData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Variant @@ -23642,37 +24110,37 @@ pub type vortex_array::arrays::Variant::OperationsVTable = vortex_array::arrays: pub type vortex_array::arrays::Variant::ValidityVTable = vortex_array::arrays::Variant -pub fn vortex_array::arrays::Variant::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Variant::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Variant::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Variant::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Variant::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Variant::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Variant::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Variant::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Variant::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Variant::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Variant::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Variant::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Variant::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Variant::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Variant::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Variant::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Variant::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Variant::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Variant::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Variant::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Variant::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Variant::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Variant::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Variant::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Variant::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Variant::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Variant::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Variant::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Variant::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Variant::validate(&self, _data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Variant::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::dict::Dict @@ -23682,37 +24150,37 @@ pub type vortex_array::arrays::dict::Dict::OperationsVTable = vortex_array::arra pub type vortex_array::arrays::dict::Dict::ValidityVTable = vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::dict::Dict::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::dict::Dict::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::dict::Dict::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::dict::Dict::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::dict::Dict::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::dict::Dict::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::dict::Dict::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::dict::Dict::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::dict::Dict::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::dict::Dict::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::dict::Dict::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::dict::Dict::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::dict::Dict::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::dict::Dict::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::dict::Dict::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::dict::Dict::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::dict::Dict::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::dict::Dict::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::dict::Dict::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::dict::Dict::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::dict::Dict::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::dict::Dict::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::dict::Dict::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::dict::Dict::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::dict::Dict::validate(&self, _data: &vortex_array::arrays::dict::DictData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::dict::Dict::validate(&self, &vortex_array::arrays::dict::DictData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::null::Null @@ -23722,37 +24190,37 @@ pub type vortex_array::arrays::null::Null::OperationsVTable = vortex_array::arra pub type vortex_array::arrays::null::Null::ValidityVTable = vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::null::Null::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::null::Null::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::null::Null::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::null::Null::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::null::Null::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::null::Null::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::null::Null::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::null::Null::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::null::Null::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::null::Null::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::null::Null::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::null::Null::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::null::Null::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::null::Null::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::null::Null::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::null::Null::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::null::Null::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::null::Null::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::null::Null::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::null::Null::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::null::Null::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::null::Null::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::null::Null::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::null::Null::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::null::Null::validate(&self, _data: &vortex_array::EmptyArrayData, dtype: &vortex_array::dtype::DType, _len: usize, _slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::null::Null::validate(&self, &vortex_array::EmptyArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::patched::Patched @@ -23762,77 +24230,77 @@ pub type vortex_array::arrays::patched::Patched::OperationsVTable = vortex_array pub type vortex_array::arrays::patched::Patched::ValidityVTable = vortex_array::ValidityVTableFromChild -pub fn vortex_array::arrays::patched::Patched::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::patched::Patched::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::patched::Patched::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::patched::Patched::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::patched::Patched::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::patched::Patched::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::patched::Patched::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::patched::Patched::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::patched::Patched::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::patched::Patched::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::patched::Patched::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::patched::Patched::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::patched::Patched::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::patched::Patched::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::patched::Patched::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::patched::Patched::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::patched::Patched::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::patched::Patched::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::patched::Patched::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::patched::Patched::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::patched::Patched::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::patched::Patched::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::patched::Patched::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::patched::Patched::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::patched::Patched::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::patched::Patched::validate(&self, data: &vortex_array::arrays::patched::PatchedData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::patched::Patched::validate(&self, &vortex_array::arrays::patched::PatchedData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> -impl vortex_array::VTable for vortex_array::arrays::scalar_fn::ScalarFnVTable +impl vortex_array::VTable for vortex_array::arrays::scalar_fn::ScalarFn -pub type vortex_array::arrays::scalar_fn::ScalarFnVTable::ArrayData = vortex_array::arrays::scalar_fn::ScalarFnData +pub type vortex_array::arrays::scalar_fn::ScalarFn::ArrayData = vortex_array::arrays::scalar_fn::array::ScalarFnData -pub type vortex_array::arrays::scalar_fn::ScalarFnVTable::OperationsVTable = vortex_array::arrays::scalar_fn::ScalarFnVTable +pub type vortex_array::arrays::scalar_fn::ScalarFn::OperationsVTable = vortex_array::arrays::scalar_fn::ScalarFn -pub type vortex_array::arrays::scalar_fn::ScalarFnVTable::ValidityVTable = vortex_array::arrays::scalar_fn::ScalarFnVTable +pub type vortex_array::arrays::scalar_fn::ScalarFn::ValidityVTable = vortex_array::arrays::scalar_fn::ScalarFn -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::scalar_fn::ScalarFn::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::scalar_fn::ScalarFn::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::scalar_fn::ScalarFn::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::scalar_fn::ScalarFn::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::deserialize(&self, _dtype: &vortex_array::dtype::DType, _len: usize, _metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::scalar_fn::ScalarFn::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::id(&self) -> vortex_array::ArrayId +pub fn vortex_array::arrays::scalar_fn::ScalarFn::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::scalar_fn::ScalarFn::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::scalar_fn::ScalarFn::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::slot_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::scalar_fn::ScalarFn::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::validate(&self, data: &vortex_array::arrays::scalar_fn::ScalarFnData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::validate(&self, &vortex_array::arrays::scalar_fn::array::ScalarFnData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::slice::Slice @@ -23842,149 +24310,149 @@ pub type vortex_array::arrays::slice::Slice::OperationsVTable = vortex_array::ar pub type vortex_array::arrays::slice::Slice::ValidityVTable = vortex_array::arrays::slice::Slice -pub fn vortex_array::arrays::slice::Slice::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::slice::Slice::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::slice::Slice::buffer(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::slice::Slice::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::slice::Slice::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::slice::Slice::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::slice::Slice::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::slice::Slice::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::slice::Slice::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::slice::Slice::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::slice::Slice::deserialize(&self, _dtype: &vortex_array::dtype::DType, _len: usize, _metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::slice::Slice::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::slice::Slice::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::slice::Slice::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::slice::Slice::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::slice::Slice::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::slice::Slice::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::slice::Slice::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::slice::Slice::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::slice::Slice::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::slice::Slice::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::slice::Slice::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::slice::Slice::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::slice::Slice::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::slice::Slice::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::slice::Slice::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::slice::Slice::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> pub trait vortex_array::DeserializeMetadata where Self: core::marker::Sized pub type vortex_array::DeserializeMetadata::Output -pub fn vortex_array::DeserializeMetadata::deserialize(metadata: &[u8]) -> vortex_error::VortexResult +pub fn vortex_array::DeserializeMetadata::deserialize(&[u8]) -> vortex_error::VortexResult impl vortex_array::DeserializeMetadata for vortex_array::EmptyMetadata pub type vortex_array::EmptyMetadata::Output = vortex_array::EmptyMetadata -pub fn vortex_array::EmptyMetadata::deserialize(metadata: &[u8]) -> vortex_error::VortexResult +pub fn vortex_array::EmptyMetadata::deserialize(&[u8]) -> vortex_error::VortexResult impl vortex_array::DeserializeMetadata for vortex_array::RawMetadata pub type vortex_array::RawMetadata::Output = alloc::vec::Vec -pub fn vortex_array::RawMetadata::deserialize(metadata: &[u8]) -> vortex_error::VortexResult +pub fn vortex_array::RawMetadata::deserialize(&[u8]) -> vortex_error::VortexResult impl vortex_array::DeserializeMetadata for vortex_array::ProstMetadata where M: core::fmt::Debug + prost::message::Message + core::default::Default pub type vortex_array::ProstMetadata::Output = M -pub fn vortex_array::ProstMetadata::deserialize(metadata: &[u8]) -> vortex_error::VortexResult +pub fn vortex_array::ProstMetadata::deserialize(&[u8]) -> vortex_error::VortexResult pub trait vortex_array::DynArrayEq: vortex_array::hash::private::SealedEq -pub fn vortex_array::DynArrayEq::dyn_array_eq(&self, other: &dyn core::any::Any, precision: vortex_array::Precision) -> bool +pub fn vortex_array::DynArrayEq::dyn_array_eq(&self, &dyn core::any::Any, vortex_array::Precision) -> bool impl vortex_array::DynArrayEq for T -pub fn T::dyn_array_eq(&self, other: &(dyn core::any::Any + 'static), precision: vortex_array::Precision) -> bool +pub fn T::dyn_array_eq(&self, &(dyn core::any::Any + 'static), vortex_array::Precision) -> bool pub trait vortex_array::DynArrayHash: vortex_array::hash::private::SealedHash -pub fn vortex_array::DynArrayHash::dyn_array_hash(&self, state: &mut dyn core::hash::Hasher, precision: vortex_array::Precision) +pub fn vortex_array::DynArrayHash::dyn_array_hash(&self, &mut dyn core::hash::Hasher, vortex_array::Precision) impl vortex_array::DynArrayHash for T -pub fn T::dyn_array_hash(&self, state: &mut dyn core::hash::Hasher, precision: vortex_array::Precision) +pub fn T::dyn_array_hash(&self, &mut dyn core::hash::Hasher, vortex_array::Precision) pub trait vortex_array::Executable: core::marker::Sized -pub fn vortex_array::Executable::execute(array: vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::Executable::execute(vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::Executable for vortex_array::ArrayRef -pub fn vortex_array::ArrayRef::execute(array: vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::ArrayRef::execute(vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::Executable for vortex_array::Canonical -pub fn vortex_array::Canonical::execute(array: vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::Canonical::execute(vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::Executable for vortex_array::CanonicalValidity -pub fn vortex_array::CanonicalValidity::execute(array: vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::CanonicalValidity::execute(vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::Executable for vortex_array::Columnar -pub fn vortex_array::Columnar::execute(array: vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::Columnar::execute(vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::Executable for vortex_array::RecursiveCanonical -pub fn vortex_array::RecursiveCanonical::execute(array: vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::RecursiveCanonical::execute(vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::Executable for vortex_array::arrays::BoolArray -pub fn vortex_array::arrays::BoolArray::execute(array: vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::BoolArray::execute(vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::Executable for vortex_array::arrays::DecimalArray -pub fn vortex_array::arrays::DecimalArray::execute(array: vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::DecimalArray::execute(vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::Executable for vortex_array::arrays::ExtensionArray -pub fn vortex_array::arrays::ExtensionArray::execute(array: vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::ExtensionArray::execute(vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::Executable for vortex_array::arrays::FixedSizeListArray -pub fn vortex_array::arrays::FixedSizeListArray::execute(array: vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::FixedSizeListArray::execute(vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::Executable for vortex_array::arrays::ListViewArray -pub fn vortex_array::arrays::ListViewArray::execute(array: vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::ListViewArray::execute(vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::Executable for vortex_array::arrays::PrimitiveArray -pub fn vortex_array::arrays::PrimitiveArray::execute(array: vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::PrimitiveArray::execute(vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::Executable for vortex_array::arrays::StructArray -pub fn vortex_array::arrays::StructArray::execute(array: vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::StructArray::execute(vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::Executable for vortex_array::arrays::VarBinViewArray -pub fn vortex_array::arrays::VarBinViewArray::execute(array: vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBinViewArray::execute(vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::Executable for vortex_array::arrays::null::NullArray -pub fn vortex_array::arrays::null::NullArray::execute(array: vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::null::NullArray::execute(vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::Executable for vortex_buffer::bit::buf::BitBuffer -pub fn vortex_buffer::bit::buf::BitBuffer::execute(array: vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_buffer::bit::buf::BitBuffer::execute(vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::Executable for vortex_mask::Mask -pub fn vortex_mask::Mask::execute(array: vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_mask::Mask::execute(vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::Executable for vortex_buffer::buffer::Buffer -pub fn vortex_buffer::buffer::Buffer::execute(array: vortex_array::ArrayRef, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_buffer::buffer::Buffer::execute(vortex_array::ArrayRef, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub trait vortex_array::IntoArray @@ -24056,95 +24524,95 @@ pub fn vortex_array::Array::into_array(self) -> vortex_array::ArrayRef pub trait vortex_array::OperationsVTable -pub fn vortex_array::OperationsVTable::scalar_at(array: vortex_array::ArrayView<'_, V>, index: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::OperationsVTable::scalar_at(vortex_array::ArrayView<'_, V>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Bool::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Chunked::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, _index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Constant::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Decimal::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Extension::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Extension>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::Filter -pub fn vortex_array::arrays::Filter::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Filter>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Filter::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Filter>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::FixedSizeList::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::List -pub fn vortex_array::arrays::List::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::List>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::List::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::List>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::ListView::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::ListView>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Masked::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Masked>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Primitive::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::Shared -pub fn vortex_array::arrays::Shared::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Shared>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Shared::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Shared>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Struct::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Struct>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBin::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBinView::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::Variant -pub fn vortex_array::arrays::Variant::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::Variant>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Variant::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::Variant>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::dict::Dict::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::scalar_at(_array: vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, _index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::null::Null::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::patched::Patched -pub fn vortex_array::arrays::patched::Patched::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::patched::Patched>, index: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::patched::Patched::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::patched::Patched>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -impl vortex_array::OperationsVTable for vortex_array::arrays::scalar_fn::ScalarFnVTable +impl vortex_array::OperationsVTable for vortex_array::arrays::scalar_fn::ScalarFn -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::scalar_fn::ScalarFnVTable>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::scalar_fn::ScalarFn::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::scalar_fn::ScalarFn>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::arrays::slice::Slice -pub fn vortex_array::arrays::slice::Slice::scalar_at(array: vortex_array::ArrayView<'_, vortex_array::arrays::slice::Slice>, index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::slice::Slice::scalar_at(vortex_array::ArrayView<'_, vortex_array::arrays::slice::Slice>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult impl vortex_array::OperationsVTable for vortex_array::NotSupported -pub fn vortex_array::NotSupported::scalar_at(array: vortex_array::ArrayView<'_, V>, _index: usize, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::NotSupported::scalar_at(vortex_array::ArrayView<'_, V>, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult pub trait vortex_array::SerializeMetadata @@ -24222,37 +24690,37 @@ pub type vortex_array::VTable::OperationsVTable: vortex_array::OperationsVTable< pub type vortex_array::VTable::ValidityVTable: vortex_array::ValidityVTable -pub fn vortex_array::VTable::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::VTable::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::VTable::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::VTable::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::VTable::buffer_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::VTable::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::VTable::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::VTable::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::VTable::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::VTable::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::VTable::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::VTable::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::VTable::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::VTable::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::VTable::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::VTable::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::VTable::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::VTable::nbuffers(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::VTable::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::VTable::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::VTable::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::VTable::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::VTable::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::VTable::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::VTable::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::VTable::serialize(array: vortex_array::ArrayView<'_, Self>, session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::VTable::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::VTable::slot_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::VTable::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::VTable::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::VTable::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Bool @@ -24262,37 +24730,37 @@ pub type vortex_array::arrays::Bool::OperationsVTable = vortex_array::arrays::Bo pub type vortex_array::arrays::Bool::ValidityVTable = vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Bool::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Bool::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Bool::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Bool::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Bool::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Bool::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Bool::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Bool::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Bool::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Bool::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Bool::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Bool::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Bool::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Bool::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Bool::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Bool::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Bool::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Bool::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Bool::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Bool::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Bool::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Bool::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Bool::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Bool::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Bool::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Bool::validate(&self, data: &vortex_array::arrays::bool::BoolData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Bool::validate(&self, &vortex_array::arrays::bool::BoolData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Chunked @@ -24302,37 +24770,37 @@ pub type vortex_array::arrays::Chunked::OperationsVTable = vortex_array::arrays: pub type vortex_array::arrays::Chunked::ValidityVTable = vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Chunked::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Chunked::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Chunked::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Chunked::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Chunked::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Chunked::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Chunked::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Chunked::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Chunked::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Chunked::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Chunked::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Chunked::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Chunked::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Chunked::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Chunked::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Chunked::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Chunked::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Chunked::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Chunked::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Chunked::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Chunked::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Chunked::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Chunked::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Chunked::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Chunked::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Chunked::validate(&self, data: &vortex_array::arrays::chunked::ChunkedData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Chunked::validate(&self, &vortex_array::arrays::chunked::ChunkedData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Constant @@ -24342,37 +24810,37 @@ pub type vortex_array::arrays::Constant::OperationsVTable = vortex_array::arrays pub type vortex_array::arrays::Constant::ValidityVTable = vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Constant::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Constant::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Constant::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Constant::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Constant::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Constant::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Constant::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Constant::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Constant::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Constant::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, _metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Constant::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Constant::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Constant::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Constant::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Constant::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Constant::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Constant::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Constant::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Constant::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Constant::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Constant::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Constant::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Constant::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Constant::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Constant::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Constant::validate(&self, data: &vortex_array::arrays::constant::ConstantData, dtype: &vortex_array::dtype::DType, _len: usize, _slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Constant::validate(&self, &vortex_array::arrays::constant::ConstantData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Decimal @@ -24382,77 +24850,77 @@ pub type vortex_array::arrays::Decimal::OperationsVTable = vortex_array::arrays: pub type vortex_array::arrays::Decimal::ValidityVTable = vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Decimal::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Decimal::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Decimal::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Decimal::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Decimal::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Decimal::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Decimal::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Decimal::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Decimal::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Decimal::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Decimal::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Decimal::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Decimal::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Decimal::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Decimal::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Decimal::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Decimal::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Decimal::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Decimal::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Decimal::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Decimal::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Decimal::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Decimal::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Decimal::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Decimal::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Decimal::validate(&self, data: &vortex_array::arrays::decimal::DecimalData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Decimal::validate(&self, &vortex_array::arrays::decimal::DecimalData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Extension -pub type vortex_array::arrays::Extension::ArrayData = vortex_array::arrays::extension::ExtensionData +pub type vortex_array::arrays::Extension::ArrayData = vortex_array::EmptyArrayData pub type vortex_array::arrays::Extension::OperationsVTable = vortex_array::arrays::Extension pub type vortex_array::arrays::Extension::ValidityVTable = vortex_array::ValidityVTableFromChild -pub fn vortex_array::arrays::Extension::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Extension::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Extension::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Extension::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Extension::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Extension::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Extension::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Extension::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Extension::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Extension::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Extension::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Extension::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Extension::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Extension::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Extension::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Extension::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Extension::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Extension::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Extension::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Extension::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Extension::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Extension::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Extension::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Extension::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Extension::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Extension::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Extension::validate(&self, data: &vortex_array::arrays::extension::ExtensionData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Extension::validate(&self, &vortex_array::EmptyArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Filter @@ -24462,37 +24930,37 @@ pub type vortex_array::arrays::Filter::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::Filter::ValidityVTable = vortex_array::arrays::Filter -pub fn vortex_array::arrays::Filter::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Filter::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Filter::buffer(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Filter::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Filter::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Filter::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Filter::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Filter::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Filter::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Filter::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Filter::deserialize(&self, _dtype: &vortex_array::dtype::DType, _len: usize, _metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Filter::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Filter::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Filter::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Filter::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Filter::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Filter::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Filter::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Filter::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Filter::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Filter::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Filter::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Filter::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Filter::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Filter::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Filter::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Filter::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Filter::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Filter::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Filter::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Filter::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::FixedSizeList @@ -24502,37 +24970,37 @@ pub type vortex_array::arrays::FixedSizeList::OperationsVTable = vortex_array::a pub type vortex_array::arrays::FixedSizeList::ValidityVTable = vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::FixedSizeList::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::FixedSizeList::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::FixedSizeList::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::FixedSizeList::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::FixedSizeList::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::FixedSizeList::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::FixedSizeList::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::FixedSizeList::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::FixedSizeList::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::FixedSizeList::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::FixedSizeList::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::FixedSizeList::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::FixedSizeList::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::FixedSizeList::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::FixedSizeList::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::FixedSizeList::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::FixedSizeList::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::FixedSizeList::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::FixedSizeList::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::FixedSizeList::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::FixedSizeList::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::FixedSizeList::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::FixedSizeList::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::FixedSizeList::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::FixedSizeList::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::FixedSizeList::validate(&self, data: &vortex_array::arrays::fixed_size_list::FixedSizeListData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::FixedSizeList::validate(&self, &vortex_array::arrays::fixed_size_list::FixedSizeListData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::List @@ -24542,37 +25010,37 @@ pub type vortex_array::arrays::List::OperationsVTable = vortex_array::arrays::Li pub type vortex_array::arrays::List::ValidityVTable = vortex_array::arrays::List -pub fn vortex_array::arrays::List::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::List::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::List::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::List::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::List::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::List::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::List::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::List::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::List::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::List::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::List::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::List::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::List::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::List::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::List::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::List::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::List::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::List::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::List::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::List::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::List::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::List::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::List::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::List::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::List::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::List::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::List::validate(&self, _data: &vortex_array::arrays::list::ListData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::List::validate(&self, &vortex_array::arrays::list::ListData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::ListView @@ -24582,37 +25050,37 @@ pub type vortex_array::arrays::ListView::OperationsVTable = vortex_array::arrays pub type vortex_array::arrays::ListView::ValidityVTable = vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::ListView::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::ListView::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::ListView::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::ListView::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::ListView::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::ListView::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::ListView::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::ListView::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::ListView::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::ListView::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::ListView::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::ListView::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::ListView::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::ListView::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::ListView::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::ListView::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::ListView::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::ListView::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::ListView::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::ListView::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::ListView::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::ListView::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::ListView::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::ListView::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::ListView::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::ListView::validate(&self, _data: &vortex_array::arrays::listview::ListViewData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::ListView::validate(&self, &vortex_array::arrays::listview::ListViewData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Masked @@ -24622,37 +25090,37 @@ pub type vortex_array::arrays::Masked::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::Masked::ValidityVTable = vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Masked::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Masked::buffer(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Masked::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Masked::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Masked::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Masked::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Masked::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Masked::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Masked::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Masked::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Masked::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Masked::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Masked::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Masked::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Masked::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Masked::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Masked::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Masked::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Masked::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Masked::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Masked::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Masked::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Masked::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Masked::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Masked::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Masked::validate(&self, _data: &vortex_array::arrays::masked::MaskedData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Masked::validate(&self, &vortex_array::arrays::masked::MaskedData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Primitive @@ -24662,37 +25130,37 @@ pub type vortex_array::arrays::Primitive::OperationsVTable = vortex_array::array pub type vortex_array::arrays::Primitive::ValidityVTable = vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Primitive::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Primitive::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Primitive::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Primitive::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Primitive::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Primitive::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Primitive::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Primitive::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Primitive::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Primitive::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Primitive::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Primitive::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Primitive::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Primitive::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Primitive::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Primitive::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Primitive::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Primitive::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Primitive::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Primitive::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Primitive::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Primitive::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Primitive::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Primitive::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Primitive::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Primitive::validate(&self, data: &vortex_array::arrays::primitive::PrimitiveData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Primitive::validate(&self, &vortex_array::arrays::primitive::PrimitiveData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Shared @@ -24702,37 +25170,37 @@ pub type vortex_array::arrays::Shared::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::Shared::ValidityVTable = vortex_array::arrays::Shared -pub fn vortex_array::arrays::Shared::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Shared::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Shared::buffer(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Shared::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Shared::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Shared::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Shared::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Shared::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Shared::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Shared::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Shared::deserialize(&self, _dtype: &vortex_array::dtype::DType, _len: usize, _metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Shared::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Shared::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Shared::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Shared::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Shared::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Shared::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Shared::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Shared::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Shared::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Shared::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Shared::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Shared::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Shared::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Shared::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Shared::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Shared::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Shared::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Shared::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Shared::validate(&self, _data: &vortex_array::arrays::shared::SharedData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Shared::validate(&self, &vortex_array::arrays::shared::SharedData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Struct @@ -24742,37 +25210,37 @@ pub type vortex_array::arrays::Struct::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::Struct::ValidityVTable = vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Struct::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Struct::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Struct::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Struct::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Struct::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Struct::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Struct::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Struct::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Struct::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Struct::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Struct::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Struct::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Struct::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Struct::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Struct::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Struct::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Struct::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Struct::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Struct::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Struct::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Struct::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Struct::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Struct::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Struct::slot_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Struct::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Struct::validate(&self, _data: &vortex_array::EmptyArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Struct::validate(&self, &vortex_array::EmptyArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::VarBin @@ -24782,37 +25250,37 @@ pub type vortex_array::arrays::VarBin::OperationsVTable = vortex_array::arrays:: pub type vortex_array::arrays::VarBin::ValidityVTable = vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::VarBin::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::VarBin::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::VarBin::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::VarBin::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::VarBin::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::VarBin::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::VarBin::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::VarBin::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::VarBin::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::VarBin::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBin::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBin::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::VarBin::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::VarBin::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::VarBin::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::VarBin::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::VarBin::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::VarBin::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::VarBin::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBin::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBin::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBin::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::VarBin::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::VarBin::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::VarBin::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::VarBin::validate(&self, _data: &vortex_array::arrays::varbin::VarBinData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::VarBin::validate(&self, &vortex_array::arrays::varbin::VarBinData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::VarBinView @@ -24822,37 +25290,37 @@ pub type vortex_array::arrays::VarBinView::OperationsVTable = vortex_array::arra pub type vortex_array::arrays::VarBinView::ValidityVTable = vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::VarBinView::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::VarBinView::buffer(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::VarBinView::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::VarBinView::buffer_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::VarBinView::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::VarBinView::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::VarBinView::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::VarBinView::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::VarBinView::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::VarBinView::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBinView::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBinView::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::VarBinView::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::VarBinView::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::VarBinView::nbuffers(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::VarBinView::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::VarBinView::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::VarBinView::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::VarBinView::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBinView::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::VarBinView::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::VarBinView::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::VarBinView::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::VarBinView::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::VarBinView::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::VarBinView::validate(&self, data: &vortex_array::arrays::varbinview::VarBinViewData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::VarBinView::validate(&self, &vortex_array::arrays::varbinview::VarBinViewData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::Variant @@ -24862,37 +25330,37 @@ pub type vortex_array::arrays::Variant::OperationsVTable = vortex_array::arrays: pub type vortex_array::arrays::Variant::ValidityVTable = vortex_array::arrays::Variant -pub fn vortex_array::arrays::Variant::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Variant::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::Variant::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::Variant::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::Variant::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::Variant::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::Variant::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Variant::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::Variant::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Variant::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Variant::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Variant::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Variant::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Variant::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::Variant::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Variant::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::Variant::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::Variant::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Variant::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Variant::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::Variant::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::Variant::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Variant::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Variant::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::Variant::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::Variant::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::Variant::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::Variant::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::Variant::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::Variant::validate(&self, _data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::Variant::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::dict::Dict @@ -24902,37 +25370,37 @@ pub type vortex_array::arrays::dict::Dict::OperationsVTable = vortex_array::arra pub type vortex_array::arrays::dict::Dict::ValidityVTable = vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::dict::Dict::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::dict::Dict::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::dict::Dict::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::dict::Dict::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::dict::Dict::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::dict::Dict::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::dict::Dict::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::dict::Dict::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::dict::Dict::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::dict::Dict::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::dict::Dict::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::dict::Dict::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::dict::Dict::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::dict::Dict::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::dict::Dict::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::dict::Dict::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::dict::Dict::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::dict::Dict::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::dict::Dict::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::dict::Dict::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::dict::Dict::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::dict::Dict::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::dict::Dict::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::dict::Dict::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::dict::Dict::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::dict::Dict::validate(&self, _data: &vortex_array::arrays::dict::DictData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::dict::Dict::validate(&self, &vortex_array::arrays::dict::DictData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::null::Null @@ -24942,37 +25410,37 @@ pub type vortex_array::arrays::null::Null::OperationsVTable = vortex_array::arra pub type vortex_array::arrays::null::Null::ValidityVTable = vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::null::Null::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::null::Null::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::null::Null::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::null::Null::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::null::Null::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::null::Null::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::null::Null::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::null::Null::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::null::Null::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::null::Null::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::null::Null::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::null::Null::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::null::Null::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::null::Null::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::null::Null::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::null::Null::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::null::Null::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::null::Null::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::null::Null::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::null::Null::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::null::Null::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::null::Null::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::null::Null::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::null::Null::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::null::Null::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::null::Null::validate(&self, _data: &vortex_array::EmptyArrayData, dtype: &vortex_array::dtype::DType, _len: usize, _slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::null::Null::validate(&self, &vortex_array::EmptyArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::patched::Patched @@ -24982,77 +25450,77 @@ pub type vortex_array::arrays::patched::Patched::OperationsVTable = vortex_array pub type vortex_array::arrays::patched::Patched::ValidityVTable = vortex_array::ValidityVTableFromChild -pub fn vortex_array::arrays::patched::Patched::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::patched::Patched::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::patched::Patched::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::patched::Patched::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::patched::Patched::buffer_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> core::option::Option +pub fn vortex_array::arrays::patched::Patched::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::patched::Patched::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::patched::Patched::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::patched::Patched::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::patched::Patched::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::patched::Patched::deserialize(&self, dtype: &vortex_array::dtype::DType, len: usize, metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::patched::Patched::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::patched::Patched::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::patched::Patched::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::patched::Patched::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::patched::Patched::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::patched::Patched::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::patched::Patched::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::patched::Patched::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::patched::Patched::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::patched::Patched::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::patched::Patched::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::patched::Patched::serialize(array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::patched::Patched::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::patched::Patched::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::patched::Patched::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::patched::Patched::validate(&self, data: &vortex_array::arrays::patched::PatchedData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::patched::Patched::validate(&self, &vortex_array::arrays::patched::PatchedData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> -impl vortex_array::VTable for vortex_array::arrays::scalar_fn::ScalarFnVTable +impl vortex_array::VTable for vortex_array::arrays::scalar_fn::ScalarFn -pub type vortex_array::arrays::scalar_fn::ScalarFnVTable::ArrayData = vortex_array::arrays::scalar_fn::ScalarFnData +pub type vortex_array::arrays::scalar_fn::ScalarFn::ArrayData = vortex_array::arrays::scalar_fn::array::ScalarFnData -pub type vortex_array::arrays::scalar_fn::ScalarFnVTable::OperationsVTable = vortex_array::arrays::scalar_fn::ScalarFnVTable +pub type vortex_array::arrays::scalar_fn::ScalarFn::OperationsVTable = vortex_array::arrays::scalar_fn::ScalarFn -pub type vortex_array::arrays::scalar_fn::ScalarFnVTable::ValidityVTable = vortex_array::arrays::scalar_fn::ScalarFnVTable +pub type vortex_array::arrays::scalar_fn::ScalarFn::ValidityVTable = vortex_array::arrays::scalar_fn::ScalarFn -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::buffer(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::scalar_fn::ScalarFn::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::scalar_fn::ScalarFn::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::scalar_fn::ScalarFn::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::scalar_fn::ScalarFn::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::deserialize(&self, _dtype: &vortex_array::dtype::DType, _len: usize, _metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::execute(array: vortex_array::Array, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::scalar_fn::ScalarFn::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::id(&self) -> vortex_array::ArrayId +pub fn vortex_array::arrays::scalar_fn::ScalarFn::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::scalar_fn::ScalarFn::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::scalar_fn::ScalarFn::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::slot_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::scalar_fn::ScalarFn::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::validate(&self, data: &vortex_array::arrays::scalar_fn::ScalarFnData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::scalar_fn::ScalarFn::validate(&self, &vortex_array::arrays::scalar_fn::array::ScalarFnData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> impl vortex_array::VTable for vortex_array::arrays::slice::Slice @@ -25062,49 +25530,49 @@ pub type vortex_array::arrays::slice::Slice::OperationsVTable = vortex_array::ar pub type vortex_array::arrays::slice::Slice::ValidityVTable = vortex_array::arrays::slice::Slice -pub fn vortex_array::arrays::slice::Slice::append_to_builder(array: vortex_array::ArrayView<'_, Self>, builder: &mut dyn vortex_array::builders::ArrayBuilder, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::slice::Slice::append_to_builder(vortex_array::ArrayView<'_, Self>, &mut dyn vortex_array::builders::ArrayBuilder, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult<()> -pub fn vortex_array::arrays::slice::Slice::buffer(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> vortex_array::buffer::BufferHandle +pub fn vortex_array::arrays::slice::Slice::buffer(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::buffer::BufferHandle -pub fn vortex_array::arrays::slice::Slice::buffer_name(_array: vortex_array::ArrayView<'_, Self>, _idx: usize) -> core::option::Option +pub fn vortex_array::arrays::slice::Slice::buffer_name(vortex_array::ArrayView<'_, Self>, usize) -> core::option::Option -pub fn vortex_array::arrays::slice::Slice::child(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::slice::Slice::child(vortex_array::ArrayView<'_, Self>, usize) -> vortex_array::ArrayRef -pub fn vortex_array::arrays::slice::Slice::child_name(array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::slice::Slice::child_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::slice::Slice::deserialize(&self, _dtype: &vortex_array::dtype::DType, _len: usize, _metadata: &[u8], _buffers: &[vortex_array::buffer::BufferHandle], _children: &dyn vortex_array::serde::ArrayChildren, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::deserialize(&self, &vortex_array::dtype::DType, usize, &[u8], &[vortex_array::buffer::BufferHandle], &dyn vortex_array::serde::ArrayChildren, &vortex_session::VortexSession) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::slice::Slice::execute(array: vortex_array::Array, _ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult +pub fn vortex_array::arrays::slice::Slice::execute(vortex_array::Array, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult -pub fn vortex_array::arrays::slice::Slice::execute_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize, ctx: &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::execute_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> pub fn vortex_array::arrays::slice::Slice::id(&self) -> vortex_array::ArrayId -pub fn vortex_array::arrays::slice::Slice::nbuffers(_array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::slice::Slice::nbuffers(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::slice::Slice::nchildren(array: vortex_array::ArrayView<'_, Self>) -> usize +pub fn vortex_array::arrays::slice::Slice::nchildren(vortex_array::ArrayView<'_, Self>) -> usize -pub fn vortex_array::arrays::slice::Slice::reduce(array: vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::reduce(vortex_array::ArrayView<'_, Self>) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::slice::Slice::reduce_parent(array: vortex_array::ArrayView<'_, Self>, parent: &vortex_array::ArrayRef, child_idx: usize) -> vortex_error::VortexResult> +pub fn vortex_array::arrays::slice::Slice::reduce_parent(vortex_array::ArrayView<'_, Self>, &vortex_array::ArrayRef, usize) -> vortex_error::VortexResult> -pub fn vortex_array::arrays::slice::Slice::serialize(_array: vortex_array::ArrayView<'_, Self>, _session: &vortex_session::VortexSession) -> vortex_error::VortexResult>> +pub fn vortex_array::arrays::slice::Slice::serialize(vortex_array::ArrayView<'_, Self>, &vortex_session::VortexSession) -> vortex_error::VortexResult>> -pub fn vortex_array::arrays::slice::Slice::slot_name(_array: vortex_array::ArrayView<'_, Self>, idx: usize) -> alloc::string::String +pub fn vortex_array::arrays::slice::Slice::slot_name(vortex_array::ArrayView<'_, Self>, usize) -> alloc::string::String -pub fn vortex_array::arrays::slice::Slice::validate(&self, data: &Self::ArrayData, dtype: &vortex_array::dtype::DType, len: usize, slots: &[core::option::Option]) -> vortex_error::VortexResult<()> +pub fn vortex_array::arrays::slice::Slice::validate(&self, &Self::ArrayData, &vortex_array::dtype::DType, usize, &[core::option::Option]) -> vortex_error::VortexResult<()> pub trait vortex_array::ValidityChild -pub fn vortex_array::ValidityChild::validity_child(array: vortex_array::ArrayView<'_, V>) -> vortex_array::ArrayRef +pub fn vortex_array::ValidityChild::validity_child(vortex_array::ArrayView<'_, V>) -> vortex_array::ArrayRef impl vortex_array::ValidityChild for vortex_array::arrays::Extension -pub fn vortex_array::arrays::Extension::validity_child(array: vortex_array::ArrayView<'_, vortex_array::arrays::Extension>) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::Extension::validity_child(vortex_array::ArrayView<'_, vortex_array::arrays::Extension>) -> vortex_array::ArrayRef impl vortex_array::ValidityChild for vortex_array::arrays::patched::Patched -pub fn vortex_array::arrays::patched::Patched::validity_child(array: vortex_array::ArrayView<'_, vortex_array::arrays::patched::Patched>) -> vortex_array::ArrayRef +pub fn vortex_array::arrays::patched::Patched::validity_child(vortex_array::ArrayView<'_, vortex_array::arrays::patched::Patched>) -> vortex_array::ArrayRef pub trait vortex_array::ValidityChildSliceHelper @@ -25114,91 +25582,91 @@ pub fn vortex_array::ValidityChildSliceHelper::unsliced_child_and_slice(&self) - pub trait vortex_array::ValidityVTable -pub fn vortex_array::ValidityVTable::validity(array: vortex_array::ArrayView<'_, V>) -> vortex_error::VortexResult +pub fn vortex_array::ValidityVTable::validity(vortex_array::ArrayView<'_, V>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::Bool -pub fn vortex_array::arrays::Bool::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Bool>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Bool::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Bool>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::Chunked -pub fn vortex_array::arrays::Chunked::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Chunked::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Chunked>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::Constant -pub fn vortex_array::arrays::Constant::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Constant>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Constant::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Constant>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::Decimal -pub fn vortex_array::arrays::Decimal::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Decimal::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Decimal>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::Filter -pub fn vortex_array::arrays::Filter::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Filter>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Filter::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Filter>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::FixedSizeList -pub fn vortex_array::arrays::FixedSizeList::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::FixedSizeList::validity(vortex_array::ArrayView<'_, vortex_array::arrays::FixedSizeList>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::List -pub fn vortex_array::arrays::List::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::List>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::List::validity(vortex_array::ArrayView<'_, vortex_array::arrays::List>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::ListView -pub fn vortex_array::arrays::ListView::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::ListView>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::ListView::validity(vortex_array::ArrayView<'_, vortex_array::arrays::ListView>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::Masked -pub fn vortex_array::arrays::Masked::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Masked>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Masked::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Masked>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::Primitive -pub fn vortex_array::arrays::Primitive::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Primitive::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Primitive>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::Shared -pub fn vortex_array::arrays::Shared::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Shared>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Shared::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Shared>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::Struct -pub fn vortex_array::arrays::Struct::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Struct>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Struct::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Struct>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::VarBin -pub fn vortex_array::arrays::VarBin::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBin::validity(vortex_array::ArrayView<'_, vortex_array::arrays::VarBin>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::VarBinView -pub fn vortex_array::arrays::VarBinView::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::VarBinView::validity(vortex_array::ArrayView<'_, vortex_array::arrays::VarBinView>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::Variant -pub fn vortex_array::arrays::Variant::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::Variant>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::Variant::validity(vortex_array::ArrayView<'_, vortex_array::arrays::Variant>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::dict::Dict -pub fn vortex_array::arrays::dict::Dict::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::dict::Dict::validity(vortex_array::ArrayView<'_, vortex_array::arrays::dict::Dict>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::null::Null -pub fn vortex_array::arrays::null::Null::validity(_array: vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::null::Null::validity(vortex_array::ArrayView<'_, vortex_array::arrays::null::Null>) -> vortex_error::VortexResult -impl vortex_array::ValidityVTable for vortex_array::arrays::scalar_fn::ScalarFnVTable +impl vortex_array::ValidityVTable for vortex_array::arrays::scalar_fn::ScalarFn -pub fn vortex_array::arrays::scalar_fn::ScalarFnVTable::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::scalar_fn::ScalarFnVTable>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::scalar_fn::ScalarFn::validity(vortex_array::ArrayView<'_, vortex_array::arrays::scalar_fn::ScalarFn>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::arrays::slice::Slice -pub fn vortex_array::arrays::slice::Slice::validity(array: vortex_array::ArrayView<'_, vortex_array::arrays::slice::Slice>) -> vortex_error::VortexResult +pub fn vortex_array::arrays::slice::Slice::validity(vortex_array::ArrayView<'_, vortex_array::arrays::slice::Slice>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::ValidityVTableFromChildSliceHelper where ::ArrayData: vortex_array::ValidityChildSliceHelper -pub fn vortex_array::ValidityVTableFromChildSliceHelper::validity(array: vortex_array::ArrayView<'_, V>) -> vortex_error::VortexResult +pub fn vortex_array::ValidityVTableFromChildSliceHelper::validity(vortex_array::ArrayView<'_, V>) -> vortex_error::VortexResult impl vortex_array::ValidityVTable for vortex_array::ValidityVTableFromChild where V: vortex_array::ValidityChild + vortex_array::VTable -pub fn vortex_array::ValidityVTableFromChild::validity(array: vortex_array::ArrayView<'_, V>) -> vortex_error::VortexResult +pub fn vortex_array::ValidityVTableFromChild::validity(vortex_array::ArrayView<'_, V>) -> vortex_error::VortexResult pub trait vortex_array::VortexSessionExecute @@ -25208,21 +25676,23 @@ impl vortex_array::VortexSessionExecute for vortex_session::VortexSession pub fn vortex_session::VortexSession::create_execution_ctx(&self) -> vortex_array::ExecutionCtx -pub fn vortex_array::child_to_validity(child: &core::option::Option, nullability: vortex_array::dtype::Nullability) -> vortex_array::validity::Validity +pub fn vortex_array::child_to_validity(core::option::Option<&vortex_array::ArrayRef>, vortex_array::dtype::Nullability) -> vortex_array::validity::Validity + +pub fn vortex_array::execute_into_builder(vortex_array::ArrayRef, alloc::boxed::Box, &mut vortex_array::ExecutionCtx) -> vortex_error::VortexResult> -pub fn vortex_array::patches_child(patches: &vortex_array::patches::Patches, idx: usize) -> vortex_array::ArrayRef +pub fn vortex_array::patches_child(&vortex_array::patches::Patches, usize) -> vortex_array::ArrayRef -pub fn vortex_array::patches_child_name(idx: usize) -> &'static str +pub fn vortex_array::patches_child_name(usize) -> &'static str -pub fn vortex_array::patches_nchildren(patches: &vortex_array::patches::Patches) -> usize +pub fn vortex_array::patches_nchildren(&vortex_array::patches::Patches) -> usize -pub fn vortex_array::validity_nchildren(validity: &vortex_array::validity::Validity) -> usize +pub fn vortex_array::validity_nchildren(&vortex_array::validity::Validity) -> usize -pub fn vortex_array::validity_to_child(validity: &vortex_array::validity::Validity, len: usize) -> core::option::Option +pub fn vortex_array::validity_to_child(&vortex_array::validity::Validity, usize) -> core::option::Option pub type vortex_array::ArrayContext = vortex_session::registry::Context -pub type vortex_array::ArrayId = arcref::ArcRef +pub type vortex_array::ArrayId = vortex_session::registry::Id pub type vortex_array::ArrayPluginRef = alloc::sync::Arc diff --git a/vortex-array/src/aggregate_fn/accumulator.rs b/vortex-array/src/aggregate_fn/accumulator.rs index 00c80b38221..9c864eea6ca 100644 --- a/vortex-array/src/aggregate_fn/accumulator.rs +++ b/vortex-array/src/aggregate_fn/accumulator.rs @@ -14,7 +14,7 @@ use crate::aggregate_fn::AggregateFnRef; use crate::aggregate_fn::AggregateFnVTable; use crate::aggregate_fn::session::AggregateFnSessionExt; use crate::dtype::DType; -use crate::executor::MAX_ITERATIONS; +use crate::executor::max_iterations; use crate::scalar::Scalar; /// Reference-counted type-erased accumulator. @@ -108,7 +108,7 @@ impl DynAccumulator for Accumulator { let kernels = &session.aggregate_fns().kernels; let mut batch = batch.clone(); - for _ in 0..*MAX_ITERATIONS { + for _ in 0..max_iterations() { if batch.is::() { break; } @@ -116,7 +116,7 @@ impl DynAccumulator for Accumulator { let kernels_r = kernels.read(); let batch_id = batch.encoding_id(); if let Some(result) = kernels_r - .get(&(batch_id.clone(), Some(self.aggregate_fn.id()))) + .get(&(batch_id, Some(self.aggregate_fn.id()))) .or_else(|| kernels_r.get(&(batch_id, None))) .and_then(|kernel| { kernel diff --git a/vortex-array/src/aggregate_fn/accumulator_grouped.rs b/vortex-array/src/aggregate_fn/accumulator_grouped.rs index 24f8f0157ec..a751a7c5749 100644 --- a/vortex-array/src/aggregate_fn/accumulator_grouped.rs +++ b/vortex-array/src/aggregate_fn/accumulator_grouped.rs @@ -32,7 +32,7 @@ use crate::builders::builder_with_capacity; use crate::builtins::ArrayBuiltins; use crate::dtype::DType; use crate::dtype::IntegerPType; -use crate::executor::MAX_ITERATIONS; +use crate::executor::max_iterations; use crate::match_each_integer_ptype; /// Reference-counted type-erased grouped accumulator. @@ -165,7 +165,7 @@ impl GroupedAccumulator { let session = ctx.session().clone(); let kernels = &session.aggregate_fns().grouped_kernels; - for _ in 0..*MAX_ITERATIONS { + for _ in 0..max_iterations() { if elements.is::() { break; } @@ -311,7 +311,7 @@ impl GroupedAccumulator { if validity.value(i) { let group = elements.slice(offset..offset + size)?; accumulator.accumulate(&group, ctx)?; - states.append_scalar(&accumulator.finish()?)?; + states.append_scalar(&accumulator.flush()?)?; } else { states.append_null() } diff --git a/vortex-array/src/aggregate_fn/combined.rs b/vortex-array/src/aggregate_fn/combined.rs new file mode 100644 index 00000000000..ab0ebca5785 --- /dev/null +++ b/vortex-array/src/aggregate_fn/combined.rs @@ -0,0 +1,265 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +//! Generic adapter for aggregates whose result is computed from two child +//! aggregate functions, e.g. `Mean = Sum / Count`. + +use std::fmt::Debug; +use std::fmt::Display; +use std::fmt::Formatter; +use std::fmt::{self}; +use std::hash::Hash; + +use vortex_error::VortexResult; +use vortex_error::vortex_bail; +use vortex_error::vortex_err; +use vortex_session::VortexSession; + +use crate::ArrayRef; +use crate::Columnar; +use crate::ExecutionCtx; +use crate::aggregate_fn::AggregateFnId; +use crate::aggregate_fn::AggregateFnVTable; +use crate::builtins::ArrayBuiltins; +use crate::dtype::DType; +use crate::dtype::FieldName; +use crate::dtype::FieldNames; +use crate::dtype::Nullability; +use crate::dtype::StructFields; +use crate::scalar::Scalar; + +/// Pair of options for the two children of a [`BinaryCombined`] aggregate. +/// +/// Wrapper around `(L, R)` because the [`AggregateFnVTable::Options`] bound +/// requires `Display`, which tuples don't implement. +#[derive(Clone, Debug, PartialEq, Eq, Hash)] +pub struct PairOptions(pub L, pub R); + +impl Display for PairOptions { + fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result { + write!(f, "({}, {})", self.0, self.1) + } +} + +// Convenience aliases so signatures stay readable. +type LeftOptions = <::Left as AggregateFnVTable>::Options; +type RightOptions = <::Right as AggregateFnVTable>::Options; +type LeftPartial = <::Left as AggregateFnVTable>::Partial; +type RightPartial = <::Right as AggregateFnVTable>::Partial; +/// Combined options for a [`BinaryCombined`] aggregate. +pub type CombinedOptions = PairOptions, RightOptions>; + +/// Declare an aggregate function in terms of two child aggregates. +pub trait BinaryCombined: 'static + Send + Sync + Clone { + /// The left child aggregate vtable. + type Left: AggregateFnVTable; + /// The right child aggregate vtable. + type Right: AggregateFnVTable; + + /// Stable identifier for the combined aggregate. + fn id(&self) -> AggregateFnId; + + /// Construct the left child vtable. + fn left(&self) -> Self::Left; + + /// Construct the right child vtable. + fn right(&self) -> Self::Right; + + /// Field name for the left child in the partial struct dtype. + fn left_name(&self) -> &'static str { + "left" + } + + /// Field name for the right child in the partial struct dtype. + fn right_name(&self) -> &'static str { + "right" + } + + /// Return type of the combined aggregate. + fn return_dtype(&self, input_dtype: &DType) -> Option; + + /// Combine the finalized left and right results into the final aggregate. + fn finalize(&self, left: ArrayRef, right: ArrayRef) -> VortexResult; + + fn finalize_scalar(&self, left_scalar: Scalar, right_scalar: Scalar) -> VortexResult; + + /// Serialize the options for this combined aggregate. Default: not serializable. + fn serialize(&self, options: &CombinedOptions) -> VortexResult>> { + let _ = options; + Ok(None) + } + + /// Deserialize the options for this combined aggregate. Default: bails. + fn deserialize( + &self, + metadata: &[u8], + session: &VortexSession, + ) -> VortexResult> { + let _ = (metadata, session); + vortex_bail!( + "Combined aggregate function {} is not deserializable", + BinaryCombined::id(self) + ); + } + + /// Coerce the input type. Default: chains `right.coerce_args(left.coerce_args(input))`. + fn coerce_args( + &self, + options: &CombinedOptions, + input_dtype: &DType, + ) -> VortexResult { + let left_coerced = self.left().coerce_args(&options.0, input_dtype)?; + self.right().coerce_args(&options.1, &left_coerced) + } + + /// Build the partial struct dtype that wraps the two child partials. + fn partial_struct_dtype(&self, left: DType, right: DType) -> DType { + DType::Struct( + StructFields::new( + FieldNames::from_iter([ + FieldName::from(self.left_name()), + FieldName::from(self.right_name()), + ]), + vec![left, right], + ), + Nullability::NonNullable, + ) + } +} + +/// Adapter that exposes any [`BinaryCombined`] as an [`AggregateFnVTable`]. +#[derive(Clone, Debug)] +pub struct Combined(pub T); + +impl Combined { + /// Construct a new combined aggregate vtable. + pub fn new(inner: T) -> Self { + Self(inner) + } +} + +impl AggregateFnVTable for Combined { + type Options = CombinedOptions; + type Partial = (LeftPartial, RightPartial); + + fn id(&self) -> AggregateFnId { + self.0.id() + } + + fn serialize(&self, options: &Self::Options) -> VortexResult>> { + BinaryCombined::serialize(&self.0, options) + } + + fn deserialize(&self, metadata: &[u8], session: &VortexSession) -> VortexResult { + BinaryCombined::deserialize(&self.0, metadata, session) + } + + fn coerce_args(&self, options: &Self::Options, input_dtype: &DType) -> VortexResult { + BinaryCombined::coerce_args(&self.0, options, input_dtype) + } + + fn return_dtype(&self, _options: &Self::Options, input_dtype: &DType) -> Option { + BinaryCombined::return_dtype(&self.0, input_dtype) + } + + fn partial_dtype(&self, options: &Self::Options, input_dtype: &DType) -> Option { + let l = self.0.left().partial_dtype(&options.0, input_dtype)?; + let r = self.0.right().partial_dtype(&options.1, input_dtype)?; + Some(self.0.partial_struct_dtype(l, r)) + } + + fn empty_partial( + &self, + options: &Self::Options, + input_dtype: &DType, + ) -> VortexResult { + Ok(( + self.0.left().empty_partial(&options.0, input_dtype)?, + self.0.right().empty_partial(&options.1, input_dtype)?, + )) + } + + fn combine_partials(&self, partial: &mut Self::Partial, other: Scalar) -> VortexResult<()> { + if other.is_null() { + return Ok(()); + } + let s = other.as_struct(); + let lname = self.0.left_name(); + let rname = self.0.right_name(); + let l_field = s + .field(lname) + .ok_or_else(|| vortex_err!("BinaryCombined partial missing `{}` field", lname))?; + let r_field = s + .field(rname) + .ok_or_else(|| vortex_err!("BinaryCombined partial missing `{}` field", rname))?; + self.0.left().combine_partials(&mut partial.0, l_field)?; + self.0.right().combine_partials(&mut partial.1, r_field)?; + Ok(()) + } + + fn to_scalar(&self, partial: &Self::Partial) -> VortexResult { + let l_scalar = self.0.left().to_scalar(&partial.0)?; + let r_scalar = self.0.right().to_scalar(&partial.1)?; + let dtype = self + .0 + .partial_struct_dtype(l_scalar.dtype().clone(), r_scalar.dtype().clone()); + Ok(Scalar::struct_(dtype, vec![l_scalar, r_scalar])) + } + + fn reset(&self, partial: &mut Self::Partial) { + self.0.left().reset(&mut partial.0); + self.0.right().reset(&mut partial.1); + } + + fn is_saturated(&self, partial: &Self::Partial) -> bool { + self.0.left().is_saturated(&partial.0) && self.0.right().is_saturated(&partial.1) + } + + /// Fans out to each child's `try_accumulate`, falling back to `accumulate` + /// against a lazily-canonicalized batch. We always claim to handle the + /// batch ourselves so [`Self::accumulate`] is unreachable — this is the + /// same trick `Count` uses to opt out of the canonicalization path. + fn try_accumulate( + &self, + state: &mut Self::Partial, + batch: &ArrayRef, + ctx: &mut ExecutionCtx, + ) -> VortexResult { + let mut canonical: Option = None; + if !self.0.left().try_accumulate(&mut state.0, batch, ctx)? { + let c = canonical.insert(batch.clone().execute::(ctx)?); + self.0.left().accumulate(&mut state.0, c, ctx)?; + } + if !self.0.right().try_accumulate(&mut state.1, batch, ctx)? { + let c = match canonical.as_ref() { + Some(c) => c, + None => canonical.insert(batch.clone().execute::(ctx)?), + }; + self.0.right().accumulate(&mut state.1, c, ctx)?; + } + Ok(true) + } + + fn accumulate( + &self, + _state: &mut Self::Partial, + _batch: &Columnar, + _ctx: &mut ExecutionCtx, + ) -> VortexResult<()> { + unreachable!("Combined::try_accumulate handles all batches") + } + + fn finalize(&self, states: ArrayRef) -> VortexResult { + let l_field = states.get_item(FieldName::from(self.0.left_name()))?; + let r_field = states.get_item(FieldName::from(self.0.right_name()))?; + let l_finalized = self.0.left().finalize(l_field)?; + let r_finalized = self.0.right().finalize(r_field)?; + BinaryCombined::finalize(&self.0, l_finalized, r_finalized) + } + + fn finalize_scalar(&self, partial: &Self::Partial) -> VortexResult { + let l_scalar = self.0.left().finalize_scalar(&partial.0)?; + let r_scalar = self.0.right().finalize_scalar(&partial.1)?; + BinaryCombined::finalize_scalar(&self.0, l_scalar, r_scalar) + } +} diff --git a/vortex-array/src/aggregate_fn/fns/count/mod.rs b/vortex-array/src/aggregate_fn/fns/count/mod.rs index f0617e96bfe..e25c42e0845 100644 --- a/vortex-array/src/aggregate_fn/fns/count/mod.rs +++ b/vortex-array/src/aggregate_fn/fns/count/mod.rs @@ -27,7 +27,7 @@ impl AggregateFnVTable for Count { type Partial = u64; fn id(&self) -> AggregateFnId { - AggregateFnId::new_ref("vortex.count") + AggregateFnId::new("vortex.count") } fn serialize(&self, _options: &Self::Options) -> VortexResult>> { @@ -76,9 +76,9 @@ impl AggregateFnVTable for Count { &self, state: &mut Self::Partial, batch: &ArrayRef, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { - *state += batch.valid_count()? as u64; + *state += batch.valid_count(ctx)? as u64; Ok(true) } diff --git a/vortex-array/src/aggregate_fn/fns/first/mod.rs b/vortex-array/src/aggregate_fn/fns/first/mod.rs index 38e55607279..22e9ef7e39d 100644 --- a/vortex-array/src/aggregate_fn/fns/first/mod.rs +++ b/vortex-array/src/aggregate_fn/fns/first/mod.rs @@ -40,7 +40,7 @@ impl AggregateFnVTable for First { type Partial = FirstPartial; fn id(&self) -> AggregateFnId { - AggregateFnId::new_ref("vortex.first") + AggregateFnId::new("vortex.first") } fn serialize(&self, _options: &Self::Options) -> VortexResult>> { @@ -94,13 +94,13 @@ impl AggregateFnVTable for First { &self, partial: &mut Self::Partial, batch: &ArrayRef, - _ctx: &mut ExecutionCtx, + ctx: &mut ExecutionCtx, ) -> VortexResult { if partial.value.is_some() { return Ok(true); } - if let Some(idx) = batch.validity_mask()?.first() { - let scalar = batch.scalar_at(idx)?; + if let Some(idx) = batch.validity()?.execute_mask(batch.len(), ctx)?.first() { + let scalar = batch.execute_scalar(idx, ctx)?; partial.value = Some(scalar.into_nullable()); } Ok(true) diff --git a/vortex-array/src/aggregate_fn/fns/is_constant/list.rs b/vortex-array/src/aggregate_fn/fns/is_constant/list.rs index 533ffa6edf8..c33e325d463 100644 --- a/vortex-array/src/aggregate_fn/fns/is_constant/list.rs +++ b/vortex-array/src/aggregate_fn/fns/is_constant/list.rs @@ -4,6 +4,7 @@ use vortex_error::VortexResult; use super::arrays_value_equal; +use super::is_constant; use crate::ExecutionCtx; use crate::arrays::ListViewArray; use crate::arrays::listview::ListViewArrayExt; @@ -20,16 +21,24 @@ pub(super) fn check_listview_constant( return Ok(true); } + if !is_constant(l.sizes(), ctx)? { + // If sizes aren't all equal, can't be constant. + return Ok(false); + } + let first_size = l.size_at(0); - let first_elements = l.list_elements_at(0)?; + if first_size == 0 { + return Ok(true); + } + + if is_constant(l.offsets(), ctx)? { + // If all offsets are identical, every list references the same slice. + return Ok(true); + } + // Check each list individually, this can be expensive. + let first_elements = l.list_elements_at(0)?; for i in 1..l.len() { - if l.size_at(i) != first_size { - return Ok(false); - } - if first_size == 0 { - continue; - } let current_elements = l.list_elements_at(i)?; if !arrays_value_equal(&first_elements, ¤t_elements, ctx)? { return Ok(false); diff --git a/vortex-array/src/aggregate_fn/fns/is_constant/mod.rs b/vortex-array/src/aggregate_fn/fns/is_constant/mod.rs index 4df47b770de..078467ad92b 100644 --- a/vortex-array/src/aggregate_fn/fns/is_constant/mod.rs +++ b/vortex-array/src/aggregate_fn/fns/is_constant/mod.rs @@ -59,8 +59,8 @@ fn arrays_value_equal(a: &ArrayRef, b: &ArrayRef, ctx: &mut ExecutionCtx) -> Vor } // Check validity masks match (null positions must be identical). - let a_mask = a.validity_mask()?; - let b_mask = b.validity_mask()?; + let a_mask = a.validity()?.execute_mask(a.len(), ctx)?; + let b_mask = b.validity()?.execute_mask(b.len(), ctx)?; if a_mask != b_mask { return Ok(false); } @@ -114,7 +114,7 @@ pub fn is_constant(array: &ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult VortexResult VortexResult { + pub fn make_partial( + batch: &ArrayRef, + is_constant: bool, + ctx: &mut ExecutionCtx, + ) -> VortexResult { let partial_dtype = make_is_constant_partial_dtype(batch.dtype()); if is_constant { if batch.is_empty() { return Ok(Scalar::null(partial_dtype)); } - let first_value = batch.scalar_at(0)?.into_nullable(); + let first_value = batch.execute_scalar(0, ctx)?.into_nullable(); Ok(Scalar::struct_( partial_dtype, vec![Scalar::bool(true, Nullability::NonNullable), first_value], @@ -256,7 +260,7 @@ impl AggregateFnVTable for IsConstant { type Partial = IsConstantPartial; fn id(&self) -> AggregateFnId { - AggregateFnId::new_ref("vortex.is_constant") + AggregateFnId::new("vortex.is_constant") } fn serialize(&self, _options: &Self::Options) -> VortexResult>> { @@ -371,13 +375,13 @@ impl AggregateFnVTable for IsConstant { // Convert to ArrayRef for DynArray methods. let array_ref = c.clone().into_array(); - let all_invalid = array_ref.all_invalid()?; + let all_invalid = array_ref.all_invalid(ctx)?; if all_invalid { partial.check_value(Scalar::null(partial.element_dtype.as_nullable())); return Ok(()); } - let all_valid = array_ref.all_valid()?; + let all_valid = array_ref.all_valid(ctx)?; // Mixed nulls → not constant. if !all_valid && !all_invalid { partial.is_constant = false; @@ -386,7 +390,7 @@ impl AggregateFnVTable for IsConstant { // All valid from here. Check batch-level constancy. if c.len() == 1 { - partial.check_value(array_ref.scalar_at(0)?.into_nullable()); + partial.check_value(array_ref.execute_scalar(0, ctx)?.into_nullable()); return Ok(()); } @@ -410,7 +414,7 @@ impl AggregateFnVTable for IsConstant { return Ok(()); } - partial.check_value(array_ref.scalar_at(0)?.into_nullable()); + partial.check_value(array_ref.execute_scalar(0, ctx)?.into_nullable()); Ok(()) } } @@ -460,11 +464,13 @@ mod tests { let mut ctx = LEGACY_SESSION.create_execution_ctx(); let arr = buffer![0, 1].into_array(); - arr.statistics().compute_all(&[Stat::Min, Stat::Max])?; + arr.statistics() + .compute_all(&[Stat::Min, Stat::Max], &mut ctx)?; assert!(!is_constant(&arr, &mut ctx)?); let arr = buffer![0, 0].into_array(); - arr.statistics().compute_all(&[Stat::Min, Stat::Max])?; + arr.statistics() + .compute_all(&[Stat::Min, Stat::Max], &mut ctx)?; assert!(is_constant(&arr, &mut ctx)?); let arr = PrimitiveArray::from_option_iter([Some(0), Some(0)]).into_array(); @@ -477,13 +483,15 @@ mod tests { let mut ctx = LEGACY_SESSION.create_execution_ctx(); let arr = PrimitiveArray::from_iter([0.0, 0.0, f32::NAN]).into_array(); - arr.statistics().compute_all(&[Stat::Min, Stat::Max])?; + arr.statistics() + .compute_all(&[Stat::Min, Stat::Max], &mut ctx)?; assert!(!is_constant(&arr, &mut ctx)?); let arr = PrimitiveArray::from_option_iter([Some(f32::NEG_INFINITY), Some(f32::NEG_INFINITY)]) .into_array(); - arr.statistics().compute_all(&[Stat::Min, Stat::Max])?; + arr.statistics() + .compute_all(&[Stat::Min, Stat::Max], &mut ctx)?; assert!(is_constant(&arr, &mut ctx)?); Ok(()) } diff --git a/vortex-array/src/aggregate_fn/fns/is_sorted/bool.rs b/vortex-array/src/aggregate_fn/fns/is_sorted/bool.rs index c1efdc91563..afd18f15902 100644 --- a/vortex-array/src/aggregate_fn/fns/is_sorted/bool.rs +++ b/vortex-array/src/aggregate_fn/fns/is_sorted/bool.rs @@ -5,11 +5,20 @@ use vortex_error::VortexResult; use vortex_mask::Mask; use super::IsSortedIteratorExt; +use crate::ExecutionCtx; use crate::arrays::BoolArray; use crate::arrays::bool::BoolArrayExt; -pub(super) fn check_bool_sorted(array: &BoolArray, strict: bool) -> VortexResult { - match array.validity_mask()? { +pub(super) fn check_bool_sorted( + array: &BoolArray, + strict: bool, + ctx: &mut ExecutionCtx, +) -> VortexResult { + match array + .as_ref() + .validity()? + .execute_mask(array.as_ref().len(), ctx)? + { Mask::AllFalse(_) => Ok(!strict), Mask::AllTrue(_) => { let values = array.to_bit_buffer(); diff --git a/vortex-array/src/aggregate_fn/fns/is_sorted/decimal.rs b/vortex-array/src/aggregate_fn/fns/is_sorted/decimal.rs index e56858fd015..1a9fc96aad2 100644 --- a/vortex-array/src/aggregate_fn/fns/is_sorted/decimal.rs +++ b/vortex-array/src/aggregate_fn/fns/is_sorted/decimal.rs @@ -6,21 +6,34 @@ use vortex_error::VortexResult; use vortex_mask::Mask; use super::IsSortedIteratorExt; +use crate::ExecutionCtx; use crate::arrays::DecimalArray; use crate::dtype::NativeDecimalType; use crate::match_each_decimal_value_type; -pub(super) fn check_decimal_sorted(array: &DecimalArray, strict: bool) -> VortexResult { +pub(super) fn check_decimal_sorted( + array: &DecimalArray, + strict: bool, + ctx: &mut ExecutionCtx, +) -> VortexResult { match_each_decimal_value_type!(array.values_type(), |S| { - compute_is_sorted::(array, strict) + compute_is_sorted::(array, strict, ctx) }) } -fn compute_is_sorted(array: &DecimalArray, strict: bool) -> VortexResult +fn compute_is_sorted( + array: &DecimalArray, + strict: bool, + ctx: &mut ExecutionCtx, +) -> VortexResult where dyn Iterator: IsSortedIteratorExt, { - match array.validity_mask()? { + match array + .as_ref() + .validity()? + .execute_mask(array.as_ref().len(), ctx)? + { Mask::AllFalse(_) => Ok(!strict), Mask::AllTrue(_) => { let buf = array.buffer::(); diff --git a/vortex-array/src/aggregate_fn/fns/is_sorted/mod.rs b/vortex-array/src/aggregate_fn/fns/is_sorted/mod.rs index 780fc492baa..abc328d90df 100644 --- a/vortex-array/src/aggregate_fn/fns/is_sorted/mod.rs +++ b/vortex-array/src/aggregate_fn/fns/is_sorted/mod.rs @@ -108,13 +108,13 @@ fn is_sorted_impl(array: &ArrayRef, strict: bool, ctx: &mut ExecutionCtx) -> Vor // Enforce strictness before we even try to check if the array is sorted. if strict { - let invalid_count = array.invalid_count()?; + let invalid_count = array.invalid_count(ctx)?; match invalid_count { // We can keep going 0 => {} // If we have a potential null value - it has to be the first one. 1 => { - if !array.is_invalid(0)? { + if !array.is_invalid(0, ctx)? { cache_is_sorted(array, strict, false); return Ok(false); } @@ -171,13 +171,18 @@ impl IsSorted { /// Kernels that compute `is_sorted` by delegating to child arrays can call this /// to package the boolean result into the partial struct format expected by the /// accumulator, avoiding duplicated boilerplate. - pub fn make_partial(batch: &ArrayRef, is_sorted: bool, strict: bool) -> VortexResult { + pub fn make_partial( + batch: &ArrayRef, + is_sorted: bool, + strict: bool, + ctx: &mut ExecutionCtx, + ) -> VortexResult { let partial_dtype = make_is_sorted_partial_dtype(batch.dtype()); if batch.is_empty() { return Ok(Scalar::null(partial_dtype)); } - let first_value = batch.scalar_at(0)?.into_nullable(); - let last_value = batch.scalar_at(batch.len() - 1)?.into_nullable(); + let first_value = batch.execute_scalar(0, ctx)?.into_nullable(); + let last_value = batch.execute_scalar(batch.len() - 1, ctx)?.into_nullable(); // SAFETY: We constructed partial_dtype and the children match its field dtypes exactly. Ok(unsafe { Scalar::struct_unchecked( @@ -227,7 +232,7 @@ impl AggregateFnVTable for IsSorted { type Partial = IsSortedPartial; fn id(&self) -> AggregateFnId { - AggregateFnId::new_ref("vortex.is_sorted") + AggregateFnId::new("vortex.is_sorted") } fn serialize(&self, _options: &Self::Options) -> VortexResult>> { @@ -299,8 +304,8 @@ impl AggregateFnVTable for IsSorted { } // Check boundary: self.last_value vs other.first_value - if let Some(ref self_last) = partial.last_value - && let Some(ref other_first_val) = other_first + if let Some(self_last) = &partial.last_value + && let Some(other_first_val) = &other_first { if !self_last.is_null() && !other_first_val.is_null() { let boundary_ok = if partial.strict { @@ -399,7 +404,7 @@ impl AggregateFnVTable for IsSorted { } // Check boundary with previous chunk. - if let Some(ref self_last) = partial.last_value { + if let Some(self_last) = &partial.last_value { if !self_last.is_null() && !value.is_null() { let boundary_ok = if partial.strict { *self_last < value @@ -430,8 +435,8 @@ impl AggregateFnVTable for IsSorted { let array_ref = c.clone().into_array(); // Check boundary with previous chunk. - let first_value = array_ref.scalar_at(0)?.into_nullable(); - if let Some(ref self_last) = partial.last_value { + let first_value = array_ref.execute_scalar(0, ctx)?.into_nullable(); + if let Some(self_last) = &partial.last_value { if !self_last.is_null() && !first_value.is_null() { let boundary_ok = if partial.strict { *self_last < first_value @@ -440,8 +445,11 @@ impl AggregateFnVTable for IsSorted { }; if !boundary_ok { partial.is_sorted = false; - partial.last_value = - Some(array_ref.scalar_at(array_ref.len() - 1)?.into_nullable()); + partial.last_value = Some( + array_ref + .execute_scalar(array_ref.len() - 1, ctx)? + .into_nullable(), + ); if partial.first_value.is_none() { partial.first_value = Some(first_value); } @@ -451,8 +459,11 @@ impl AggregateFnVTable for IsSorted { || (self_last.is_null() && first_value.is_null() && partial.strict) { partial.is_sorted = false; - partial.last_value = - Some(array_ref.scalar_at(array_ref.len() - 1)?.into_nullable()); + partial.last_value = Some( + array_ref + .execute_scalar(array_ref.len() - 1, ctx)? + .into_nullable(), + ); if partial.first_value.is_none() { partial.first_value = Some(first_value); } @@ -462,10 +473,10 @@ impl AggregateFnVTable for IsSorted { // Check within-batch sortedness. let batch_is_sorted = match c { - Canonical::Primitive(p) => check_primitive_sorted(p, partial.strict)?, - Canonical::Bool(b) => check_bool_sorted(b, partial.strict)?, + Canonical::Primitive(p) => check_primitive_sorted(p, partial.strict, ctx)?, + Canonical::Bool(b) => check_bool_sorted(b, partial.strict, ctx)?, Canonical::VarBinView(v) => check_varbinview_sorted(v, partial.strict)?, - Canonical::Decimal(d) => check_decimal_sorted(d, partial.strict)?, + Canonical::Decimal(d) => check_decimal_sorted(d, partial.strict, ctx)?, Canonical::Extension(e) => check_extension_sorted(e, partial.strict, ctx)?, Canonical::Null(_) => !partial.strict, // Struct, List, FixedSizeList should have been filtered out by return_dtype @@ -479,8 +490,11 @@ impl AggregateFnVTable for IsSorted { if partial.first_value.is_none() { partial.first_value = Some(first_value); } - partial.last_value = - Some(array_ref.scalar_at(array_ref.len() - 1)?.into_nullable()); + partial.last_value = Some( + array_ref + .execute_scalar(array_ref.len() - 1, ctx)? + .into_nullable(), + ); Ok(()) } } @@ -648,10 +662,8 @@ mod tests { let mut ctx = LEGACY_SESSION.create_execution_ctx(); let dtype = DecimalDType::new(19, 2); - let i100 = - parse_decimal::("100.00", dtype.precision(), dtype.scale()).unwrap(); - let i200 = - parse_decimal::("200.00", dtype.precision(), dtype.scale()).unwrap(); + let i100 = parse_decimal::("100.00", dtype.precision(), dtype.scale())?; + let i200 = parse_decimal::("200.00", dtype.precision(), dtype.scale())?; let sorted = buffer![i100, i200, i200]; let unsorted = buffer![i200, i100, i200]; @@ -675,12 +687,9 @@ mod tests { let mut ctx = LEGACY_SESSION.create_execution_ctx(); let dtype = DecimalDType::new(19, 2); - let i100 = - parse_decimal::("100.00", dtype.precision(), dtype.scale()).unwrap(); - let i200 = - parse_decimal::("200.00", dtype.precision(), dtype.scale()).unwrap(); - let i300 = - parse_decimal::("300.00", dtype.precision(), dtype.scale()).unwrap(); + let i100 = parse_decimal::("100.00", dtype.precision(), dtype.scale())?; + let i200 = parse_decimal::("200.00", dtype.precision(), dtype.scale())?; + let i300 = parse_decimal::("300.00", dtype.precision(), dtype.scale())?; let strict_sorted = buffer![i100, i200, i300]; let sorted = buffer![i100, i200, i200]; diff --git a/vortex-array/src/aggregate_fn/fns/is_sorted/primitive.rs b/vortex-array/src/aggregate_fn/fns/is_sorted/primitive.rs index 21c80e7bd45..da9cc12b7aa 100644 --- a/vortex-array/src/aggregate_fn/fns/is_sorted/primitive.rs +++ b/vortex-array/src/aggregate_fn/fns/is_sorted/primitive.rs @@ -6,17 +6,32 @@ use vortex_error::VortexResult; use vortex_mask::Mask; use super::IsSortedIteratorExt; +use crate::ExecutionCtx; use crate::arrays::PrimitiveArray; use crate::arrays::primitive::NativeValue; use crate::dtype::NativePType; use crate::match_each_native_ptype; -pub(super) fn check_primitive_sorted(array: &PrimitiveArray, strict: bool) -> VortexResult { - match_each_native_ptype!(array.ptype(), |P| { compute_is_sorted::